Computational exploration of a viable route to the Kitaev-quantum spin liquid phase

in monolayer OsCl3
Qiangqiang Gu,1, 2 Shishir Kumar Pandey,2, ∗ and Yihao Lin3
1
School of Mathematical Science, Peking University, Beijing 100871, China
2
AI for Science Institute, Beijing, China
3
International Center for Quantum Materials, School of Physics,
Peking University, Beijing 100871, China
(Dated: June 6, 2023)
In this computational study, we explore a viable route to access the Kitaev-Quantum Spin Liquid
(QSL) state in recently synthesized monolayer of a so-called spin-orbit assisted Mott insulator OsCl3 .
In addition to other magnetic ground states in different regions, the small JH /U region of our
arXiv:2304.04257v2 [cond-mat.str-el] 4 Jun 2023

Hubbard U –Hund’s JH quantum phase diagram, obtained by combining second-order perturbation

and pseudo-Fermion renormalization group calculations, hosts Kitaev-QSL phase. Only Kitaev
interaction of a smaller magnitude is obtained in this region. Negligibly small farther neighbor
interactions appear as a distinct feature of monolayer OsCl3 , suggesting this material to be a better
candidate for the exploration of possible Kitaev-QSL state than earlier proposed materials. Insights
from our study might be useful to probe magnetic phase transitions by purportedly manipulating
U and JH with epitaxial strain in advanced crystal growth techniques.

The Kitaev honeycomb model with bond-dependent explore the Kitaev-QSL state in SOC-assisted Mott in-
Ising-type exchange interactions on a honeycomb lat- sulators through manipulation of the fundamental elec-
tice naturally hosts the novel quantum spin liquid (QSL) tronic parameters U and JH . Our study is also motivated
state1 . The realization of the Kitaev model in so-called in part by recent theoretical work that argues for the ex-
spin-orbit coupling (SOC) assisted Mott insulators was perimental tuning of U and JH parameters in perovskites
proposed by Jackeli and Khaliullin2 . This requires SOC with epitaxial strain, based on advancements in experi-
active transition-metal ions to be in edge-shared octahe- mental epitaxial growth techniques32 .
dral crystal field (CF) of anions, and Sr2 IrO4 was initially To realize our goal, we consider the OsCl3 monolayer
proposed as an example2 . Since then, many materials as a case study. Our material choice is based on the fact
like α-RuCl3 3–7 and Cobaltates8–17 , other Iridates18–25 it satisfies all the necessary criteria mentioned earlier to
are proposed as pertinent Kitaev-QSL candidates26,27 . host strong Kitaev interactions, viz Os+3 5d5 ions in the
In these materials, the formation of jeff = 1/2 Kramers edge-shared OsCl6 octahedron possess large SOC inter-
doublets, which form the low-energy space in the mate- action. In a recent experimental study by Kataoka et.
rials, results from the interplay of octahedral CF (∆CF ) al.33 , magnetization and heat capacity measurements on
and SOC. Os0.81 Cl3 revealed the absence of any long-range mag-
Although the dominant magnetic interaction between netic ordering of Os+3 spins arranged on the triangular
these pseudospin 1/2 states is proposed to be Kitaev (K) lattice (with local honeycomb domain formation) up to
type2 , other undesirable interactions such as isotropic 0.08 K temperature, which can be an indication of under-
Heisenberg (J) and off-diagonal terms (Γ), as well as far- lying strong magnetic frustration in this compound, also
ther neighbor couplings, also appear, causing long-range a building block for Kitaev physics. A previous computa-
magnetic order in the materials mentioned above, and tional attempt to study the topological properties of this
driving them away from a possible QSL ground state. For compound proposed a ferromagnetic (FM) ground state
example, the appearance of third nearest neighbor (3NN) for the monolayer OsCl3 34 , but a scrupulous investigation
J3 28 and strong neutron absorbing Ir ions along with of its magnetic properties is warranted in the light of the
rapidly decaying magnetic form factor with an increase fact that the iso-electronic α-RuCl3 has a zigzag (ZZ)
in momentum transfer in inelastic neutron scattering ex- antiferromagnetic (AFM) ground state. From a com-
periments on Iridates are some of the experimental chal- putational modeling perspective, electronic parameters
lenges21,27 . In another example, smaller SOC strength of U and JH are part of Hubbard-Kanamori Hint Hamilto-
Ru+3 ions as well a significant J3 in α-RuCl3 are the hur- nian in Eq. S4. The full model Hamiltonian H in Eq. S4
dles in the quest of QSL state in this material. Despite has been quite successful and hence, widely accepted for
such inevitable challenges, attempts have been made to the estimation of magnetic interactions in these materi-
access the QSL state using external knobs. For exam- als 7,16,28,35,36 . Thus, one can explore the access to the
ple, magnetic interactions in α-RuCl3 were manipulated Kitaev-QSL phase by tuning the U and JH parameters
by means of the external magnetic field to achieve the and calculating the corresponding magnetic interactions
QSL state29–31 . In another theoretical attempt by Liu which is the objective of this article.
et. al.11 , ∆CF
tri was proposed as a tunable parameter with We first used ab initio calculations of higher accuracy
pressure to realize QSL state in Cobaltates. Towards to show that the ZZ and FM configurations in this ma-
this goal, our computational study in this article aims to terial are nearly degenerate, indicating highly compet-
 2

ing magnetic interactions that may be a reason for the

absence of long-range magnetic ordering in this com-
pound. We then varied U and JH /U in the second-
order perturbation calculations to obtain magnetic in-
teractions and input them into pseudofermion functional
renormalization-group (pf-FRG) calculations, which al-
lowed us to obtain a U –JH /U quantum phase diagram.
At small values of JH /U , we found the emergence of only
K-type 1NN interaction, albeit of smaller magnitude,
resulting in the Kitaev-QSL state occupying a substan-
tial part of the phase diagram. In addition to the QSL
state, we also find FM, ZZ, and Néel states present in
different parts of the phase diagram. Much more advan-
tageous than previously suggested Kitaev-QSL material
candidates like Iridates, α-RuCl3 and Cobaltates, we find
that the second and third neighbor magnetic interactions
are negligibly smaller in the monolayer OsCl3 , suggesting
this material to be a better candidate for the exploration
of possible Kitaev-QSL state.
Crystal structure and magnetism – ab initio calcula-
tions: We consider the C2/m honeycomb crystal struc-
ture of the monolayer OsCl3 which is similar to α-
RuCl3 34,37 . Very recently, a triangular lattice with FIG. 1. (a) Top view of out-of-plane spin arrangement on
nanometer-sized honeycomb domains of Os atoms has a hexagon in ZZ magnetic state in monolayer OsCl3 . Spins
point towards the center of one of the Cl-Cl edges in an Os-Cl6
been observed experimentally33 . The structure is shown
octahedron. Big Cyan and small gray balls represent Os and
in Fig. 1(a) and Fig. S1 of supplementary material Cl atoms; x, y and z are local axes while a and b are crys-
(SM)38 . The full optimization of lattice constants within tallographic axes. Variation of the total energy of different
highly accurate ab initio calculations after imposition of magnetic configurations relative to ZZ state with U on Os d
both ZZ and FM state brings nearly identical crystal states for, (b) out-of-plane spin alignment, and (d) alignment
structure in both cases. Our phonon calculations with of spins along a axis. (c) Color-coded X, Y, and Z bonds up
this optimized crystal structure indicate that the honey- to 3NN are shown.
comb lattice of OsCl3 monolayer is at least dynamically
stable. Details of ab initio calculations, optimized crys-
tal structure, and phonon calculations can be found in
Sec. S1 of SM38 . states, like Néel and stripe, and ZZ state may indicate
that the exchange interactions in monolayer OsCl3 may
To understand the nature of magnetism and under-
not be weak. In the most simplistic second-order per-
lying frustration, highly accurate ab initio total energy
turbation theory, magnetic exchange interactions are ∝
calculations are performed, with tighter energy and force
t2 /U at large U values if one treats hopping t as a per-
convergence criterion, after imposing various magnetic
turbation. The expected 1/U behavior41,42 of magnetic
configurations like FM, ZZ, Néel, and Stripe. We varied
stability curves in Fig. 1 (b) and (d) at large values U
U on Os d states in the range 0.1–3 eV and the SOC
is apparent. We also find that the in-plane magnetic
effect is self-consistently included. The results, shown
anisotropy for monolayer OsCl3 is quite small (see Fig.
in Fig. 1(b) reveal that the ZZ state is lowest in energy
S3 of SM38 ). Next, we describe the electronic Hamil-
when the spins are canted in an out-of-plane direction
tonian considered in this study for exploration of U –JH
towards the center of one of the Cl-Cl edges in an OsCl6
quantum phase diagram in the quest of QSL state.
octahedron (see Fig. 1(a)), with the closely competing
FM state higher in energy. Other AFM configurations Model Hamiltonian: The full Hamiltonian for d5 sys-
like Néel and stripe are energetically much higher. Such tems is given as7,16,28,35 ,
orientation of magnetic moments was also proposed for
Na2 IrO3 39,40 . The energy difference between the ZZ and
FM state nearly vanishes for the in-plane arrangement of H = Hhop + Hcf + Hsoc + Hint (1)
spins along the a axis as shown in Fig. 1(d). This is con-
sistent with the previous study by Sheng et. al.34 . The
nearly degenerate FM and ZZ configurations, which dif- where terms on the right-hand side are hopping, CF,
fer by a single 1NN spin flip, are indicative of large mag- SOC, and Hubbard-Kanamori Hamiltonians, respec-
netic frustration present in the system, explaining the ab- tively. For Mott insulators, in limit t ≪ U , one can
sence of long-range magnetic ordering observed in experi- separate the “onsite” term Hi0 from H. Hi0 , represented
†
ment33 . The larger energy difference between other AFM in basis ψiσ = [d†z2 , d†xz , d†yz , d†x2 −y2 , d†xy ]iσ for site i and
 3

spin σ, then reads as, viously considered for 5d system such as Iridates in the
literature28 . Details of pf-FRG calculations are given in
Hi0 = Hcf + Hsoc + Hint Sec. S5 of SM38 . We will now discuss this phase diagram
X † X
= ψiσ ∆CF λLi · si in detail.
i ψiσ +
σ
UX U′ X TABLE I. Estimated bond-dependent magnetic interactions
+ niασ niασ′ + niα niβ
2 α 2 for monolayer OsCl3 using second-order perturbation method
α̸=β
at U = 1.7 eV, JH = 0.3 eV and λ = 0.355 eV. Values
JH X † † of Heisenberg J, Kitaev K, off-diagonal Γ, Γ′ and asym-
− ψiασ ψiασ′ ψiβσ ′ ψiβσ
2 metric DMI D terms along with diagonal and off-diagonal
σ,σ ′ ,α̸=β
anisotropic terms ξ and ζ are listed. 3NN interactions as well
J′ X † † as some 2NN interactions are < 0.05 meV and hence not listed
− ψiασ ψiβσ′ ψiασ ′ ψiβσ (2)
2 here.
σ̸=σ ′ ,α̸=β
Magnetic interactions (meV)
In above expression, U /U ′ are intraorbital/interorbital Bond J K Γ Γ′ ξ ζ
Coulomb interaction terms, and JH and J ′ are Hund’s
X1 -1.055 -14.600 5.662 -3.578 0.520 2.515
coupling and pair hopping interaction, respectively. U ′ = Y1 -0.961 -17.215 6.010 -0.748 -0.213 -4.320
U - 2JH and JH = J ′ is considered in the above equation. Z1 -0.844 -15.127 4.749 -2.342 -1.285 -4.567
To bring complete rotational invariance of Hi0 in the pres- X2 -0.294 0.533 0.206 -0.197 – -0.082
ence of eg orbitals, the inclusion of additional three and Y2 0.145 0.878 0.050 -0.150 – -0.167
four orbitals terms in Hint should be considered. How- Z2 -0.258 0.453 0.086 -0.246 – -0.050
ever, for large ∆CF
i , such additional terms have an imper- Dij
ceptible influence on magnetic interactions. A discussion
X2 ( 0, 0, 0)
around this point is included in Sec. 4(a) of SM38 . ∆CF i Y2 ( -0.350, 0, -0.216)
P †
and hopping amplitudes tij in Hhop = i̸=j,σ ψiσ Tij ψjσ Z2 ( -0.108, -0.125, 0 )
are obtained from a Wannier based tight-binding (TB)
model excluding SOC effect. The strength of SOC in
monolayer OsCl3 , λ = 0.355 eV, is estimated by a band- Various magnetic phases viz FM, ZZ and Néel and
structure fitting procedure (see Sec S238 for details). As QSL states occupy different parts of the phase diagram.
proposed earlier, it might be possible to experimentally At smaller values of U = 0.5 eV and JH /U = 0.075 - 0.15,
tune parameters U and JH by epitaxial strain in advance there is a (< 1.5 meV) 1NN FM J which is slightly larger
growth techniques32 . Hence, we vary these two param- than the AFM K coupling resulting in an FM ground
eters to explore the possibility of QSL state in mono- state with a competing ZZ phase. This behavior man-
layer OsCl3 . For a d5 system, the dimension of Hilbert ifested in a star-like shape of static spin susceptibility
spaces for Hi0 is C510 = 252 and by exactly diagonalizing (hereafter only susceptibility) plot in Fig. 2(c) (last plot).
it, we obtain the lowest two eigenstates as the jeff =1/2 Consistent with the FM phase, the major contribution to
Kramers doublet. We then use the second-order pertur- the susceptibility comes from the zone center while small
bation method, treating hopping as a perturbation, to es- spreads along M -points in the first BZ indicate a com-
timate magnetic interactions between these pseudo-spin peting ZZ AFM phase. Transition to later happens when
states. Details of the methodology and discussions on both couplings change their respective signs with FM K
the validity of low-energy jeff =1/2 picture and atomic > AFM J, which is consistence with a recent observation
features which can be observed in experiments for mono- on Cobaltates16 . At a typical phase point U = 0.5 eV
layer OsCl3 are presented in Sec. S3, S4 and S6 of SM38 . and JH /U = 0.1 in this region, depicted by white star in
Magnetic interactions and quantum phase diagram: Fig. 2(a), we obtained J = -0.5 meV K = 0.35 meV.
Estimated magnetic interactions up to 3NN (Fig. 1(c) In ZZ region, away from the FM-ZZ phase boundary,
depicts bonds up to 3NN) from second-order perturba- other terms like Γ, Γ′ , ξ, ζ as well as smaller farther neigh-
tion method are used as input for the pf-FRG calcula- bor interactions started to emerge in the ZZ region of the
tions. We then analyzed the ground states by plotting the phase diagram. However, 1NN dominant K and J terms
static spin correlation function, obtained from pf-FRG remain FM and AFM, respectively. We emphasize here
calculations, in the first and extended BZ of monolayer that 2NN and 3NN interactions are at least 2 ∼ 3 orders
OsCl3 . Our choice of the pf-FRG method as a quantum of magnitude smaller than 1NN and have no qualitative
toolkit is based on its earlier demonstrated success in ex- effect on the magnetic ground state. Values of U = 1.7
ploring quantum phase diagrams on different lattice and eV and JH = 0.3 eV bring ZZ state (white star in ZZ re-
spin models43,44 . Our obtained quantum phase diagram gion in Fig. 2(a)) and estimated interactions are listed in
for a range of U (0.5 - 3.0 eV) and JH /U (0.075 - 0.3) is Table I. The corresponding susceptibility plot in Fig. 2(c)
shown in Fig. 2(a). This range of parameters is chosen (second from the left) has dominant contributions from
so that the phase space is extended on either side of the M -points of the first BZ, consistent with the underlying
point U = 1.7 eV and JH = 0.3 eV (JH /U ∼ 0.176) pre- ZZ magnetic order. In the absence of inversion symmetry
 4

FIG. 2. (a) Magnetic quantum phase diagram in U –JH /U parameter space obtained using a combination of second-order
perturbation and pf-FRG calculations. The range considered is U = 0.5 – 3.0 eV and JH /U = 0.075 – 0.3 are considered.
Obtained magnetic phases are FM, ZZ, Néel, and QSL magnetic states. Static spin correlation plots in (c) correspond to the
points depicted by star symbols in (a). Intensity is normalized for each plot in (c), and the inner and outer hexagons in these
plots denote the first and extended BZ of monolayer OsCl3 . (b) The renormalization group flow of the magnetic correlations
(χ) for the two cases, ZZ (green curve) and QSL (red) states with an inset showing the flow at low frequencies for the QSL
state. Vertical dash lines show the break of the flow of χ for the ZZ state while its smooth flow for the QSL state is indicative
of no spontaneous symmetry breaking.

on the 2NN bonds, a small Dzyaloshinskii–Moriya inter- (leftmost) has a contribution from the whole first BZ,
action (DMI) appears for these magnetic neighbors. We with the highest intensity appearing at the Γ point. No
would like to mention that the value of U and JH obtained long-range magnetic ordering and hence, no spontaneous
from cRPA calculations also brings a ZZ state, details of symmetry breaking is expected for this state even at low
which are provided in Sec. S7 of SM38 . ZZ phase occupy- temperatures. To verify this, we can plot the renormal-
ing the largest part of the phase diagram quantitatively ization group flow of the magnetic correlations (χ) as a
differs in different regions and a related discussion is pro- function of the renormalization group cutoff (Λ). Spon-
vided in Sec. S7 of SM38 . Further increase of U (≈ 2 – 3 taneous symmetry breaking by the onset of a long-range
eV) and JH /U (≈ 0.2 – 0.3) stabilizes Néel AFM state, magnetic ordering at a Λ results in a kink or break of χ.
with 1NN AFM J being the dominant interaction while The smooth evolution of χ down to the lowest Λ can be
all other interactions are highly suppressed. For a typi- seen for the QSL state from the red curve in Fig. 2(b)
cal phase point U = 2.5 eV, JH /U = 0.3 (marked by a (see also the inset). For comparison, we also show a sim-
white star in Néel region), 1NN, 2NN, and 3NN J values ilar plot for the ZZ order (green curve) corresponding to
are 3.39, 0.39 and 0.03 meV, respectively. Contribution interactions listed in Table I. Clearly, χ, in this case, dis-
from Γ′ points of the extended BZ in the corresponding plays a breakdown at Λc ≈ 3.3 signaling the onset of the
susceptibility plot shown in Fig. 2(c) (third from the left) ZZ order.
is consistent with the underlying Néel magnetic order. Several remarks are in order. First, though off-
Most importantly, a substantial region of the phase di- diagonal magnetic coupling (e.g. Γ) is large compared to
agram, in the range of U = 1.0 - 3.0 eV, JH /U = 0.075 - Iridates28 , orders of magnitude smaller farther neighbor
0.1, is occupied by the Kitaev-QSL state. In this region, magnetic interactions (3NN interactions are all < |0.05|)
we only obtained a small but finite 1NN FM K term (∼ distinguishes monolayer OsCl3 these earlier proposed
1 meV) in our second-order perturbation calculations. Kitaev-QSL candidates including Iridates16,28 . Second,
The susceptibility plot for this state shown in Fig. 2(c) strain as an external knob, proposed in Ref.32 for tuning
 5

U and JH can also be utilized to tune trigonal distortions Combining second-order perturbation and pf-FRG cal-
as suggested in Ref.11 . The strain in this case can also al- culations, we presented a magnetic quantum phase dia-
ter the hopping amplitudes between d orbitals. Hence, it gram in U -JH /U space. A substantial part of this phase
would be interesting to probe the role of strain in tuning diagram hosts the Kitaev QSL state, stabilized by 1NN
the magnetic properties of these QSL candidate materi- Kitaev interaction only, with other magnetic phases like
als. Third, tuning of U and JH to access the QSL re- FM, ZZ, and Néel AFM states also appearing in differ-
gion in the phase diagram can also be achieved through ent regions of the phase diagram stabilized by changing
the choice of suitable elements from the periodic table. dominant magnetic interactions. Insights obtained from
Advancement in material synthesis techniques can be of our study may be helpful in designing experiments on
great help here. pertinent Kitaev-QSL candidate materials.
In conclusion, considering recently synthesized mono-
layer OsCl3 as an example, we have explored a route to
access Kitaev QSL state by varying electronic parame- ACKNOWLEDGMENTS
ters Hubbard U and Hund’s JH . These two parameters
purportedly can be tuned experimentally with epitax- QG and SKP contributed equally to this work. SKP
ial strain. Using highly accurate ab initio total energy gratefully acknowledges valuable feedback from Prof. D.
calculations, we have shown that the ZZ and FM are D. Sarma. SKP also acknowledges the careful reading
nearly degenerate indicating the large magnetic frustra- of the manuscript and correspondence on magnetic ex-
tion which might be the possible reason behind the ab- change couplings with Ramesh Dhakal and Dr. Stephen
sence of long-range magnetic ordering in this material. M. Winter.

∗
[email protected] C. Stock, Kitaev interactions in the Co honeycomb anti-
1
A. Kitaev, Anyons in an exactly solved model and beyond, ferromagnets Na3 Co2 SbO6 and Na2 Co2 TeO6 , Phys. Rev.
Ann. Phys. 321, 2 (2006), january Special Issue. B 102, 224429 (2020).
2 11
G. Jackeli and G. Khaliullin, Mott Insulators in the Strong H. Liu, J. c. v. Chaloupka, and G. Khaliullin, Kitaev Spin
Spin-Orbit Coupling Limit: From Heisenberg to a Quan- Liquid in 3d Transition Metal Compounds, Phys. Rev.
tum Compass and Kitaev Models, Phys. Rev. Lett. 102, Lett. 125, 047201 (2020).
12
017205 (2009). L. Viciu, Q. Huang, E. Morosan, H. Zandbergen, N. Green-
3
K. W. Plumb, J. P. Clancy, L. J. Sandilands, V. V. baum, T. McQueen, and R. Cava, Structure and basic
Shankar, Y. F. Hu, K. S. Burch, H.-Y. Kee, and Y.-J. magnetic properties of the honeycomb lattice compounds
Kim, α − RuCl3 : A spin-orbit assisted Mott insulator on Na2 Co2 TeO6 and Na3 Co2 SbO6 , J. Solid State Chem. 180,
a honeycomb lattice, Phys. Rev. B 90, 041112 (2014). 1060 (2007).
4 13
R. D. Johnson, S. C. Williams, A. A. Haghighirad, J. Sin- G. Xiao, Z. Xia, W. Zhang, X. Yue, S. Huang, X. Zhang,
gleton, V. Zapf, P. Manuel, I. I. Mazin, Y. Li, H. O. F. Yang, Y. Song, M. Wei, H. Deng, and D. Jiang, Crys-
Jeschke, R. Valentı́, and R. Coldea, Monoclinic crystal tal Growth and the Magnetic Properties of Na2 Co2 TeO6
structure of α−RuCl3 and the zigzag antiferromagnetic with Quasi-Two-Dimensional Honeycomb Lattice, Cryst.
ground state, Phys. Rev. B 92, 235119 (2015). Growth Des. 19, 2658 (2019).
5 14
A. Banerjee, C. A. Bridges, J.-Q. Yan, A. A. Aczel, L. Li, E. Lefrançois, M. Songvilay, J. Robert, G. Nataf, E. Jor-
M. B. Stone, G. E. Granroth, M. D. Lumsden, Y. Yiu, dan, L. Chaix, C. V. Colin, P. Lejay, A. Hadj-Azzem,
J. Knolle, S. Bhattacharjee, D. L. Kovrizhin, R. Moessner, R. Ballou, and V. Simonet, Magnetic properties of the
D. A. Tennant, D. G. Mandrus, and S. E. Nagler, Prox- honeycomb oxide Na2 Co2 TeO6 , Phys. Rev. B 94, 214416
imate Kitaev quantum spin liquid behaviour in a honey- (2016).
15
comb magnet, Nat. Mat. 15, 733 (2016). A. K. Bera, S. M. Yusuf, A. Kumar, and C. Ritter, Zigzag
6
H.-S. Kim, V. S. V., A. Catuneanu, and H.-Y. Kee, Kitaev antiferromagnetic ground state with anisotropic correla-
magnetism in honeycomb RuCl3 with intermediate spin- tion lengths in the quasi-two-dimensional honeycomb lat-
orbit coupling, Phys. Rev. B 91, 241110 (2015). tice compound Na2 Co2 TeO6 , Phys. Rev. B 95, 094424
7
W. Wang, Z.-Y. Dong, S.-L. Yu, and J.-X. Li, Theoreti- (2017).
16
cal investigation of magnetic dynamics in α−RuCl3 , Phys. S. K. Pandey and J. Feng, Spin interaction and magnetism
Rev. B 96, 115103 (2017). in cobaltate kitaev candidate materials: An ab initio and
8
H. Liu and G. Khaliullin, Pseudospin exchange interactions model hamiltonian approach, Phys. Rev. B 106, 174411
in d7 cobalt compounds: Possible realization of the Kitaev (2022).
17
model, Phys. Rev. B 97, 014407 (2018). W. Chen, X. Li, Z. Hu, Z. Hu, L. Yue, R. Sutarto, F. He,
9
R. Sano, Y. Kato, and Y. Motome, Kitaev-Heisenberg K. Iida, K. Kamazawa, W. Yu, X. Lin, and Y. Li, Spin-
Hamiltonian for high-spin d7 Mott insulators, Phys. Rev. orbit phase behavior of Na2 Co2 TeO6 at low temperatures,
B 97, 014408 (2018). Phys. Rev. B 103, L180404 (2021).
10 18
M. Songvilay, J. Robert, S. Petit, J. A. Rodriguez-Rivera, J. c. v. Chaloupka, G. Jackeli, and G. Khaliullin, Kitaev-
W. D. Ratcliff, F. Damay, V. Balédent, M. Jiménez-Ruiz, Heisenberg Model on a Honeycomb Lattice: Possible Ex-
P. Lejay, E. Pachoud, A. Hadj-Azzem, V. Simonet, and otic Phases in Iridium Oxides A2 IrO3 , Phys. Rev. Lett.
 6

105, 027204 (2010). transition metal trichloride OsCl3 : A playground for

19
Y. Singh, S. Manni, J. Reuther, T. Berlijn, R. Thomale, two-dimensional magnetism, room-temperature quantum
W. Ku, S. Trebst, and P. Gegenwart, Relevance of the anomalous hall effect, and topological phase transitions,
Heisenberg-Kitaev Model for the Honeycomb Lattice Iri- Phys. Rev. B 95, 201402 (2017).
35
dates A2 IrO3 , Phys. Rev. Lett. 108, 127203 (2012). J. G. Rau, E. K.-H. Lee, and H.-Y. Kee, Spin-orbit physics
20
Y. Singh and P. Gegenwart, Antiferromagnetic Mott in- giving rise to novel phases in correlated systems: Iridates
sulating state in single crystals of the honeycomb lattice and related materials, Annual Review of Condensed Mat-
material Na2 IrO3 , Phys. Rev. B 82, 064412 (2010). ter Physics 7, 195 (2016).
21 36
S. K. Choi, R. Coldea, A. N. Kolmogorov, T. Lancaster, S. K. Pandey, Q. Gu, Y. Lin, R. Tiwari, and J. Feng, Emer-
I. I. Mazin, S. J. Blundell, P. G. Radaelli, Y. Singh, gence of bond-dependent highly anisotropic magnetic in-
P. Gegenwart, K. R. Choi, S.-W. Cheong, P. J. Baker, teractions in sr4 rho6 : A theoretical study, Phys. Rev. B
C. Stock, and J. Taylor, Spin Waves and Revised Crystal 107, 115119 (2023).
37
Structure of Honeycomb Iridate Na2 IrO3 , Phys. Rev. Lett. P. H. Souza, D. P. d. A. Deus, W. H. Brito, and R. H. Miwa,
108, 127204 (2012). Magnetic anisotropy energies and metal-insulator transi-
22
A. Biffin, R. D. Johnson, S. Choi, F. Freund, S. Manni, tions in monolayers of α−RuCl3 and OsCl3 on graphene,
A. Bombardi, P. Manuel, P. Gegenwart, and R. Coldea, Phys. Rev. B 106, 155118 (2022).
38
Unconventional magnetic order on the hyperhoneycomb Q. Gu, S. K. Pandey, and Y. Lin, For crystal structure,
Kitaev lattice in β−Li2 IrO3 : Full solution via magnetic methodology, phonon dispersion, magnetic anisotropy
resonant x-ray diffraction, Phys. Rev. B 90, 205116 (2014). plot, validity of jeff = 1/2 picture and spin-wave spectra,
23
A. Biffin, R. D. Johnson, I. Kimchi, R. Morris, A. Bom- refer to the supplemental material. (2023).
39
bardi, J. G. Analytis, A. Vishwanath, and R. Coldea, Non- K. Hu, F. Wang, and J. Feng, First-principles study of
coplanar and Counterrotating Incommensurate Magnetic the magnetic structure of Na2 IrO3 , Phys. Rev. Lett. 115,
Order Stabilized by Kitaev Interactions in γ−Li2 IrO3 , 167204 (2015).
40
Phys. Rev. Lett. 113, 197201 (2014). S. Hwan Chun, J.-W. Kim, J. Kim, H. Zheng, C. C.
24
B. H. Kim, G. Khaliullin, and B. I. Min, Electronic ex- Stoumpos, C. Malliakas, J. Mitchell, K. Mehlawat,
citations in the edge-shared relativistic Mott insulator: Y. Singh, Y. Choi, et al., Direct evidence for dominant
Na2 IrO3 , Phys. Rev. B 89, 081109 (2014). bond-directional interactions in a honeycomb lattice iri-
25
H. Gretarsson, J. P. Clancy, X. Liu, J. P. Hill, E. Bozin, date Na2 IrO3 , Nat. Phys. 11, 462 (2015).
41
Y. Singh, S. Manni, P. Gegenwart, J. Kim, A. H. Said, S. Middey, A. K. Nandy, S. Pandey, P. Mahadevan, and
D. Casa, T. Gog, M. H. Upton, H.-S. Kim, J. Yu, V. M. D. Sarma, Route to high néel temperatures in 4d and 5d
Katukuri, L. Hozoi, J. van den Brink, and Y.-J. Kim, transition metal oxides, Phys. Rev. B 86, 104406 (2012).
42
Crystal-Field Splitting and Correlation Effect on the Elec- S. K. Pandey, P. Mahadevan, and D. D. Sarma, Doping
tronic Structure of A2 IrO3 , Phys. Rev. Lett. 110, 076402 an antiferromagnetic insulator: A route to an antiferro-
(2013). magnetic metallic phase, Europhysics Letters 117, 57003
26
L. Clark and A. H. Abdeldaim, Quantum spin liquids from (2017).
43
a materials perspective, Annual Review of Materials Re- J. Reuther, R. Thomale, and S. Trebst, Finite-temperature
search 51, 495 (2021). phase diagram of the heisenberg-kitaev model, Phys. Rev.
27
H. Takagi, T. Takayama, G. Jackeli, G. Khaliullin, and B 84, 100406 (2011); R. Suttner, C. Platt, J. Reuther, and
S. E. Nagler, Concept and realization of kitaev quantum R. Thomale, Renormalization group analysis of compet-
spin liquids, Nature Reviews Physics 1, 264 (2019). ing quantum phases in the J1 -J2 heisenberg model on the
28
S. M. Winter, Y. Li, H. O. Jeschke, and R. Valentı́, Chal- kagome lattice, Phys. Rev. B 89, 020408 (2014); M. Her-
lenges in design of Kitaev materials: Magnetic interactions ing, F. Ferrari, A. Razpopov, I. I. Mazin, R. Valentı́, H. O.
from competing energy scales, Phys. Rev. B 93, 214431 Jeschke, and J. Reuther, Phase diagram of a distorted
(2016). kagome antiferromagnet and application to y-kapellasite,
29
S.-H. Baek, S.-H. Do, K.-Y. Choi, Y. S. Kwon, A. U. B. npj Computational Materials 8, 10 (2022).
44
Wolter, S. Nishimoto, J. van den Brink, and B. Büchner, N. Astrakhantsev, F. Ferrari, N. Niggemann, T. Müller,
Evidence for a field-induced quantum spin liquid in α-rucl3 , A. Chauhan, A. Kshetrimayum, P. Ghosh, N. Regnault,
Phys. Rev. Lett. 119, 037201 (2017). R. Thomale, J. Reuther, T. Neupert, and Y. Iqbal, Pin-
30
J. S. Gordon, A. Catuneanu, E. S. Sørensen, and H.-Y. wheel valence bond crystal ground state of the spin-
1
Kee, Theory of the field-revealed Kitaev spin liquid, Nat. 2
heisenberg antiferromagnet on the shuriken lattice,
Commun. 10, 2470 (2019). Phys. Rev. B 104, L220408 (2021); Y. Iqbal, H. O.
31
L. Janssen and M. Vojta, Heisenberg–kitaev physics in Jeschke, J. Reuther, R. Valentı́, I. I. Mazin, M. Gre-
magnetic fields, Journal of Physics: Condensed Matter 31, iter, and R. Thomale, Paramagnetism in the kagome
423002 (2019). compounds (Zn,Mg,Cd)Cu3 (OH)6 Cl2 , Phys. Rev. B 92,
32
B. Kim, P. Liu, J. M. Tomczak, and C. Franchini, Strain- 220404 (2015); Y. Watanabe, S. Trebst, and C. Hickey,
induced tuning of the electronic coulomb interaction in Frustrated ferromagnetism of honeycomb cobaltates: In-
3d transition metal oxide perovskites, Phys. Rev. B 98, commensurate spirals, quantum disordered phases, and
075130 (2018). out-of-plane ising order (2022), arXiv:2212.14053 [cond-
33
K. Kataoka, D. Hirai, T. Yajima, D. Nishio-Hamane, mat.str-el].
45
R. Ishii, K.-Y. Choi, D. Wulferding, P. Lemmens, S. Kit- G. Kresse and D. Joubert, From ultrasoft pseudopotentials
taka, T. Sakakibara, et al., Kitaev spin liquid candidate to the projector augmented-wave method, Phys. Rev. B
Osx Cl3 comprised of honeycomb nano-domains, Journal 59, 1758 (1999).
46
of the Physical Society of Japan 89, 114709 (2020). P. E. Blöchl, Projector augmented-wave method, Phys.
34
X.-L. Sheng and B. K. Nikolić, Monolayer of the 5d Rev. B 50, 17953 (1994).
 7

47
G. Kresse and J. Hafner, Ab initio molecular dynamics for
liquid metals, Phys. Rev. B 47, 558 (1993).
48
G. Kresse and J. Furthmüller, Efficiency of ab-initio total
energy calculations for metals and semiconductors using a
plane-wave basis set, Comput. Mater. Sci. 6, 15 (1996).
49
G. Kresse and J. Furthmüller, Efficient iterative schemes
for ab initio total-energy calculations using a plane-wave
basis set, Phys. Rev. B 54, 11169 (1996).
50
J. P. Perdew, K. Burke, and M. Ernzerhof, Generalized
gradient approximation made simple, Phys. Rev. Lett. 77,
3865 (1996).
51
S. L. Dudarev, G. A. Botton, S. Y. Savrasov, C. J.
Humphreys, and A. P. Sutton, Electron-energy-loss spectra
and the structural stability of nickel oxide: An LSDA+U
study, Phys. Rev. B 57, 1505 (1998).
52
A. Togo and I. Tanaka, First principles phonon calculations
in materials science, Scr. Mater. 108, 1 (2015).
53
A. Togo, First-principles phonon calculations with
phonopy and phono3py, J. Phys. Soc. Jpn. 92, 012001
(2023).
54
A. A. Mostofi, J. R. Yates, Y.-S. Lee, I. Souza, D. Van-
derbilt, and N. Marzari, wannier90: A tool for obtain-
ing maximally-localised Wannier functions, Comput. Phys.
Commun. 178, 685 (2008).
55
C. Franchini, R. Kovacik, M. Marsman, S. S. Murthy,
J. He, C. Ederer, and G. Kresse, Maximally localized Wan-
nier functions in LaMnO3 within PBE+U, hybrid func-
tionals and partially self-consistent GW: an efficient route
to construct ab initio tight-binding parameters for eg per-
ovskites, Journal of Physics: Condensed Matter 24, 235602
(2012).
56
Q. Gu, S. K. Pandey, and R. Tiwari, A computational
method to estimate spin-orbital interaction strength in
solid state systems, Comput. Mater. Sci. 221, 112090
(2023).
57
S. L. Adler, Quantum theory of the dielectric constant in
real solids, Phys. Rev. 126, 413 (1962).
58
N. Wiser, Dielectric constant with local field effects in-
cluded, Phys. Rev. 129, 62 (1963).
59
F. Aryasetiawan, M. Imada, A. Georges, G. Kotliar,
S. Biermann, and A. I. Lichtenstein, Frequency-dependent
local interactions and low-energy effective models from
electronic structure calculations, Phys. Rev. B 70, 195104
(2004).
60
F. L. Buessen, The SpinParser software for pseudofermion
functional renormalization group calculations on quantum
magnets, SciPost Phys. Codebases , 5 (2022).
61
J. Reuther and P. Wölfle, J1 −J2 frustrated two-
dimensional heisenberg model: Random phase approxima-
tion and functional renormalization group, Phys. Rev. B
81, 144410 (2010).
62
F. L. Buessen, V. Noculak, S. Trebst, and J. Reuther,
Functional renormalization group for frustrated magnets
with nondiagonal spin interactions, Phys. Rev. B 100,
125164 (2019).
63
S. Toth and B. Lake, Linear spin wave theory for single-q
incommensurate magnetic structures, J. Phys.: Condens.
Matter 27, 166002 (2015).
 8

SUPPLEMENTARY MATERIALS

1. Ab initio calculations

a. Details of ab initio calculations

Density-functional theory (DFT) calculations have

been performed using projector-augmented wave
method45,46 , implemented within Vienna ab initio
simulation package (VASP)47–49 . The Perdew-Burke-
Ernzerhof functional50 is used for the exchange-
correlation functional within the GGA formalism. We FIG. S1. (a) Primitive unit cell of OsCl3 , (b) the 2 × 1 × 1
start with the monoclinic (C2/m) crystal structure supercell hosting ZZ magnetic order, and (c) BZ of the ZZ
for the monolayer OsCl3 with 2 Os atom unit cell supercell.
as shown in Fig. S1(a). Full structural relaxation is
performed on the 2 × 1 × 1 magnetic supercell with
FM and ZZ magnetic orders (shown in Fig. S1(b)).
this, we perform phonon calculations on a 2×4×1 super-
The energy cutoff for a plane-wave basis is set to 500
cell, while considering Hubbard U = 1 eV on Os d orbitals
eV and Γ-centered k-mesh of 4 × 8 × 1 is considered
and imposing ZZ magnetic configuration. We used the fi-
in our calculations. High accuracy of our calculations
nite displacement method to calculate the force constants
is ensured by tighter relaxation convergence criterion
and phonon dispersions obtained with and without SOC
wherein tolerance considered for total energy is 10−8 eV
effects are shown in Fig. S2. In both cases, crystal struc-
while for force it is 5 × 10−6 eV/Å. Full optimization of
ture is at least dynamically stable as no soft modes are
the crystal structure is performed after considering U =
found in our calculations. The post-processing of these
1 eV under Dudarev51 scheme and spin-orbit coupling
phonon calculations is done by Phonopy package52,53 .
(SOC) effect at the self-consistent level. For both the
cases of ZZ and FM states, optimization resulted in a
similar crystal structure and optimized lattice constants
are a = b = 6.1787 Å, vacuum c = 25 Å, and α = β =
90◦ and γ ≈ 120◦ and atomic coordinates of the final
structure are listed in the Table S1. The primitive unit
cell is shown in Fig. S1 (a) and the Brillouin zone (BZ)
for the zigzag (ZZ) supercell is shown in Fig. S1 (c).

TABLE S1. Atomic coordinates in magnetic unit cell of

monolayer OsCl3 with space group C2/m. Lattice constants
are a = b = 6.1787 Å, c = 25 Å, α = β = 90◦ and γ = 119.88◦
FIG. S2. Phonon dispersion plots for monolayer OsCl3 with
atom x/a y/b z/c ZZ magnetic order and U = 1 eV on Os d orbitals calculated
Os1 0.166797 0.833203 0.500000 without (a) and with (b) SOC effect.
Os2 0.833203 0.166797 0.500000
Cl1 0.855480 0.855480 0.446984
Cl2 0.144520 0.144520 0.553016
Cl3 0.499663 0.143222 0.446932
Cl4 0.856778 0.500337 0.553068
Cl5 0.500337 0.856778 0.553068
Cl6 0.143222 0.499663 0.446932 c. In-plane magnetic anisotropy

To obtain the magnetic anisotropy in monolayer OsCl3 ,

we start with magnetic moments aligned along a axis in
b. Phonon dispersion the FM state. We then rotate these moments by 30◦ in
the ab plane and perform total energy calculations. The
The pristine monolayer OsCl3 has not been observed upper limit of rotation was fixed at 180◦ . Obviously,
experimentally yet. As explained in the main text, tri- at 120◦ , magnetic moments are aligned along the b axis.
angular lattice of Os+3 ions with nanometer-sized honey- The obtained plot is shown in Fig. S3. From this plot, one
comb domains is observed recently in experiments33 This can conclude that while Os spins prefer to align along a
raises a natural question about the structural stability of axis, the in-plane anisotropic energy is small in monolayer
the resulting monolayer honeycomb lattice. To answer OsCl3 .
 9

where I2 is the two-dimensional Pidentity matrix and ⊗ is

the Kronecker product. Hsoc = i λLi · si is the onsite
SOC interaction term that couples the electronic spin (S)
and its orbital momentum (L) with a strength λ. Here,
the SOC strength λ is estimated by fitting the eigen-
values of H with those obtained from SOC included ab
initio band structure calculations as the target. The best
agreement between these two band structures, shown in
Fig. S4(b), is obtained for λ = 0.335 eV which is consid-
ered in further calculations.

FIG. S3. In-plane magnetic anisotropy in monolayer OsCl3

estimated considering FM state. While Os spins prefer to
align along a, in-plane magnetic anisotropy is small in this
material.

2. Estimation of electronic parameters

FIG. S4. Comparison of ab initio and TB model band struc-
tures for, (a) non-spin-polarized case, and (b) SOC included
a. Crystal field splitting and hopping amplitudes case. In (b), SOC effects in ab initio calculations are in-
cluded self-consistently, which then fitted with the TB Hamil-
Crystal field matrix and hopping amplitudes are esti- tonian obtained
P in (a) after explicitly adding onsite SOC term
mated from a non-spin-polarised, SOC excluded tight- Hsoc = i λLi · si to HTB .
binding (TB) Hamiltonian (HTB ) expressed in the local
axes framework depicted in Fig. 1(a). These two terms
can be given as,
HTB = Hcf + Hhop
The above Hamiltonian can be obtained by projecting the c. Estimation of Hubbard U and Hund’s JH within cRPA
ab initio band structure onto Wannier functions in which
all the five Os-d orbitals form the basis54,55 . The choice We estimate the Coulomb matrix elements Uijkl (ω =
of the local axes (x, y, z) in Fig. 1(a) along the oxygen 0) within the constrained random phase approximation
atoms is such that c = x + y + z lies along trigonal (cRPA) implemented with VASP57–59 . We neglect the
distortion axis of the OsCl6 octahedra. Close agreement screening effects for all the five Os d orbital states which
between the thus obtained TB model and ab initio band are energetically well-separated from other states (see
structure is shown in Fig. S4(a). Crystal field matrix on Fig. S4). The estimated parameters are U = 3.8 eV and
a site i (∆CFi ) in the above term Hcf is extracted from JH = 0.143 eV.
the onsite part of HTB and is given by,

2.6322 −0.0772 0.2578 −0.0122 0.0369

 
−0.0772 0.0479 −0.0268 0.0709 0.0272
∆CF =  0.2578 −0.0268 0.0707 0.1026 0.0366 (S1)
 
i
−0.0122 3. Validity of jeff = 1/2 picture
0.0709 0.1026 2.6244 0.2800
0.0369 0.0272 0.0366 0.2800 0.0748
Entries in the above matrix are in eV. Hopping ampli- It is crucial to analyze the effects of various electronic
tudes up to third neighbor are listed below in Table S2. interactions on the band structure of monolayer OsCl3 .
Such analysis can help us to examine the validity of low-
energy jeff = 1/2 picture, which is considered a building
b. Estimation of SOC strength (λ) block for the emergence of dominant Kitaev interaction.
To this end, ab initio band structure calculations are per-
To extract SOC strength in OsCl3 , we added an onsite formed for three cases, viz SOC, SOC + U , and SOC +
SOC term to the TB Hamiltonian as56 , U + ZZ magnetic state. Obtained results are then pro-
jected onto jeff = 1/2 and 3/2 states, eigenfunctions of
H = I2 ⊗ HTB + Hsoc which are given below in Eq. S2.
 10

TABLE S2. First, second and third neighbor Os-Os hopping amplitudes (in eV) on different types of bonds listed in the basis
ψ † = [d†z2 , d†xz , d†yz , d†x2 −y2 , d†xy ]
.

X1 -bond  
Y1 -bond  
Z1 -bond 
−0.0183 −0.0150 −0.1032 0.0291 −0.0521 0.0323 −0.0899 −0.0070 0.0027 0.0193 −0.0265 0.0477 −0.0163 −0.0228 0.2041
 −0.0150 0.0653 −0.0670 −0.0074 0.1817   −0.0899 −0.0784 0.0121 −0.1796 −0.0473   0.0477 0.0212 0.1853 0.0196 0.0170 
     
 −0.1032 −0.0670 −0.0762 0.1708 0.0028   −0.0070 0.0121 0.0100 −0.0476 0.1845   −0.0163 0.1853 0.0650 0.0179 −0.0533 
     
 0.0291 −0.0074 0.1708 0.0097 0.0405   0.0027 −0.1796 −0.0476 −0.0431 −0.0016   −0.0228 0.0196 0.0179 0.0141 0.0077 
−0.0521 0.1817 0.0028 0.0405 0.0151 0.0193 −0.0473 0.1845 −0.0016 0.0742 0.2041 0.0170 −0.0533 0.0077 −0.0782


X2 -bond  
Y2 -bond  
Z2 -bond 
−0.0213 0.0404 −0.0290 0.0126 −0.0045 0.0230 −0.0353 0.0376 0.0151 −0.0054 0.0181 −0.0222 −0.0433 −0.0306 0.0921

 0.0500 0.0033 −0.0083 −0.0206 −0.0514 

 −0.0712
 −0.0011 0.0109 −0.0742 −0.0063 

 −0.0316
 −0.0155 −0.0534 0.0241 0.0186 

 −0.0590 −0.0213 0.0000 0.0746 0.0163   0.0384 0.0196 −0.0104 0.0145 −0.0531   −0.0386 −0.0491 0.0015 −0.0277 −0.0185 
     
 0.0006 −0.0201 0.0977 0.0323 −0.0414   0.0279 −0.0885 0.0043 −0.0107 0.0570   −0.0189 0.0285 −0.0363 0.0006 0.0140 
−0.0125 −0.0479 0.0077 −0.0362 −0.0100 −0.0010 −0.0181 −0.0486 0.0486 −0.0034 0.0979 0.0110 −0.0065 −0.0239 −0.0004


X3 -bond  
Y3 -bond  
Z3 -bond 
0.0239 0.0031 −0.0241 0.0315 0.0131 0.0304 0.0003 −0.0019 −0.0269 0.0035 −0.0230 −0.0140 −0.0071 −0.0032 0.0241

 0.0031 0.0086 0.0073 −0.0044 −0.0108 


 0.0003 −0.0285 0.0049 −0.0284 0.0090 

 −0.0140
 0.0092 −0.0116 −0.0095 0.0054 

 −0.0241 0.0073 −0.0289 0.0149 0.0038   −0.0019 0.0049 0.0104 0.0155 −0.0120   −0.0071 −0.0116 0.0081 −0.0009 0.0089 
     
 0.0315 −0.0040 0.0149 −0.0039 −0.0062   −0.0269 −0.0288 0.0155 −0.0105 0.0066   −0.0032 −0.0095 −0.0009 0.0448 0.0010 
0.0131 −0.0108 0.0038 −0.0062 0.0103 0.0035 0.0090 −0.0120 0.0066 0.0073 0.0241 0.0054 0.0089 0.0100 −0.0292

where the terms on the right-hand side are hopping, CF,


1
   SOC, and Hubbard-Kanamori interaction Hamiltonians.
 ,±1 = 1 1 1 1
†
2 2
√
3
∓ |dxy ; ± ⟩ ∓ i |dxz ; ∓ ⟩ − |dyz ; ∓ ⟩
2 2 2 Using the basis ψiσ = [d†z2 ,d†xz , d†yz , d†x2 −y2 , d†xy ]iσ at site
i of Os and spin σ, above Hamiltonian then reads,
   
3
 ,±3 = 1
√
1 1
−i |dxz ; ± ⟩ ∓ |dyz ; ± ⟩ (S2)
† †
2 X X X
2 2 2 2 H= ψiσ Tij ψjσ + ψiσ ∆CF λLi · si
    i ψiσ +
3
 ,±1 = 1
√
1 1 1
2 |dxy ; ± ⟩ − i |dxz ; ∓ ⟩ ∓ |dyz ; ∓ ⟩
i̸=j,σ i,σ i

U′ X
2 2 6 2 2 2 U X
+ niασ niασ′ + niα niβ
2 i,α 2 i,α̸=β
Plots obtained for the three cases are shown in Fig. S5(a),
(S4)
(b) and (c) respectively. SOC effect is included at the −
JH X †
ψiασ †
ψiασ′ ψiβσ ′ ψiβσ
self-consistent level in all these calculations. 2
i,σ,σ ′ ,α̸=β
Certain features of the band structures can immedi- J′ † †
X
ately be identified. Firstly, one can find a clear separation − ψiασ ψiβσ′ ψiασ ′ ψiβσ
2
i,σ̸=σ ′ ,α̸=β
between narrow jeff = 1/2 and 3/2 bands for SOC-only
band structure shown in Fig. S5(a). In Fig. S5(b), for In the above expression, the first three terms are hop-
the SOC + U case, the jeff = 1/2 bands near the Fermi pings, CF and SOC, while the last four terms are the
level remain largely unaffected at this U = 1 eV (though Hubbard-Kanamori interactions. The U /U ′ are intraor-
a larger U = 2 eV opens a clear band gap which is not bital/interorbital Hartree energies and JH and J ′ are
shown here) and the system is nearly insulating in both Hund’s coupling and pair hopping interaction, respec-
cases. This differs from the α-RuCl3 picture presented tively. Rotational invariance in the isolated atom limit
by Kim et. al.6 where U = 1.5 eV was necessary to bring dictates the relationships: U ′ = U - 2JH and JH = J ′ .
out a clear jeff = 1/2-3/2 separation and to drive the Hopping amplitudes Tij and CF matrix ∆CF i can be ob-
system insulating. The weak response towards smaller U tained by fitting the ab initio band structure with a TB
along with the narrow bandwidth (∼ 0.3 eV) of jeff = model as described earlier.
1/2 states may be hinting that t and U parameters have In the limit t ≪ U , i.e. in the isolated atom limit, one
similar scales in this material. A constrained random can separate the “onsite” term Hi0 at a site i from H as,
phase approximation (cRPA) estimation brings U = 3.8
Hi0 = Hcf + Hsoc + Hint
eV and JH = 0.143 eV for monolayer OsCl3 . However,
the largest 1NN hopping magnitude |t| estimated from We drop index i since this Hamiltonian is the same for
the Wannier TB model is ≈ 0.2 eV. Based on these val- each site. The key here is that the d5 manifold has a
ues, one can still think of limit t ≪ U and hence the two-fold degenerate ground state, which forms a Kramers
formation of well-localized jeff states in this compound. doublet and can be treated as a pseudospin-1/2. In order
A wider band gap opens up once magnetism in the form to extract the magnetic interactions of these pseudospin
of ZZ state is included and obtained band structure for states, we first project the full TB Hamiltonian onto the
this case is shown in Fig. S5(c). pseudospin j1/2 space {ϕiα }, α =↑, ↓, where ↑, ↓ refer
to the SOC pseudospin-1/2 states. Starting from the
isolated limit we introduce Hhop as a perturbation. In
4. Second-order perturbation method the second-order perturbation theory, the Hamiltonian is
written as,
The full Hamiltonian for the d5 Os+3 ions is given by, X X
H (2) = H(i, j)αβα′ β ′ |iα, jβ⟩ ⟨iα′ , jβ ′ |,
H = Hhop + Hcf + Hsoc + Hint (S3) ij αβα′ β ′
 11

FIG. S5. Band structure plots of monolayer OsCl3 for, (a) SOC included, (b) SOC + U , and (c) SOC + U + ZZ magnetic
state are shown. Red and green spheres depicted the weight of jeff =1/2 and 3/2 states, respectively. Fermi energy is set to
zero in these plots.

Only these symmetric components of interaction ma-

XX 1 trices appear on the first and third neighbor bonds, while
H(i, j)αβα′ β ′ = ⟨iα, jβ|Hhop |kγ, lλ⟩ on the second neighbor bonds, due to the absence of in-
∆E (S5)
kl γλ version symmetry, an additional antisymmetric compo-
′ ′ nent appears in the form of Dzyaloshinskii–Moriya in-
× ⟨kγ, lλ|Hhop |iα , jβ ⟩,
teraction (DMI). The matrix from these antisymmetric
where 1/∆E = 21 [1/(Eiα +Ejβ −Ekλ −Elγ )+1/(Eiα′ + interactions is given in Ref.28 .
Ejβ ′ − Ekλ − Elγ )]. Here, |iα, jβ⟩ and |iα′ , jβ ′ ⟩ are two-
site states in the jeff = 1/2 ground states, and |kλ, lγ⟩
are two-site excited states, both in the isolated atom a. Three and four orbital terms
limit. Hhop connects a two-site ground state to an ex-
cited state with (d4 , d6 ) configuration, the dimensions of Since we consider both t2g and eg orbitals in our basis
whose Hilbert spaces is 210. The eigenstates of isolated above, to ensure the rotational invariance of the “on-
Os with 4, 5, and 6 d electrons are obtained by exact site” Hamiltonian Hi0 , one should include three and four
diagonalization. One can represent the pseudo-spins j1/2 orbital terms in the Hubbard-Kanamori Hamiltonian.
µ
as S µ = ⟨iα|ji,eff |iβ⟩ which are the expectation values of Three orbital terms can be given,
µ 0
pseudospin jeff operators with µ = 0, x, y, z. Here, jeff = 3orb V1 X † †
0
I2 is the matrix representation of operator jeff . Eq. (S5) Hint = ψiασ ψiβσ ψiγσ ′ ψiγσ ′
2
σ,σ ′ ,α̸=β
can be mapped to a spin Hamiltonian of the form,
V1 X † †
+ ψiασ ψiγσ ψiγσ ′ ψiβσ ′
Hspin = Siµ η(i, j)µν Sjν 2
σ,σ ′ ,α̸=β
(S6) (S9)
1 V2
= η(i, j)µν ϕ†iα σααµ ν † † †
X
′ ϕiα′ ϕjβ σββ ′ ϕjβ , + ψiασ ψiγσ ψiβσ ′ ψiγσ ′
4 2
σ̸=σ ′ ,α̸=β

where µ, ν = 0, x, y, z, σ µ are Pauli matrices, and sum- +

V2 X †
ψiγσ †
ψiασ ψiγσ ′ ψiβσ ′
mation over all repeated indexes is implied. The map 2
σ̸=σ ′ ,α̸=β
can be achieved by solving the linear equations
In the above equation, indices α, β = dz2 /dx2 −y2 and γ =
1 µ ν µν dxz /dyz . Magnitude of V1 , V2 are usually quite small (<
− σαα′ σββ ′ η(ij) = H(i, j)αβα′ β ′ (S7)
4 10−2 eV). Four orbital terms in the Hubbard-Kanamori
Thus, the most general form of exchange interaction Hamiltonian can be given as,
matrix on the three Os-Os bonds can be written as, 4orb
Hint =
V X †
ψiασ †
ψiδσ ψiβσ ′ ψiγσ ′ (S10)
2
σ,σ ′ ,α̸=β
J + K Γ′ + ζ Γ′ − ζ
 

η X =  Γ′ + ζ J + ξ Γ  In the above equation, the four distinct orbital indices α,

Γ′ − ζ Γ J −ξ β, γ, and δ run over all the five d orbitals in our basis.
The magnitude of V is usually of the order of ∼ 10−3 eV.
J + ξ Γ′ + ζ
 
Γ 3orb 4orb
Addition of Hint and Hint to Hint term of Eq. 2 in the
ηY = Γ′ + ζ J + K Γ′ − ζ  (S8)
main text modify the magnetic interactions mostly at the
Γ Γ′ − ζ J − ξ
third decimal places. This is consistent with the fact that
Γ′ + ζ
 
J +ξ Γ these additional terms are redundant in the presence of
ηZ =  Γ J − ξ Γ′ − ζ  large crystal field splitting(∆CFi in Eq. S1) which is also
Γ + ζ Γ′ − ζ J + K
′
the case of monolayer OsCl3 . Since these significantly
 12

smaller changes in magnetic interactions do not bring from diagonalization of many-body “onsite” Hamiltonian
any substantial changes in the magnetic ground state, H 0 . Hence, we discuss here the nature of CF splitting
we used only two orbital terms of Eq. 2 for the phase in monolayer OsCl3 and its effect on the electronic struc-
diagram in Fig. 2(a) of the main text. ture. The CF matrix is given in Eq. S1.
Examining eigenvalues of CF matrix (∆i ) in Eq. S1
t −e
5. Pseudo-fermion functional renormalization reveals that t2g -eg ∆i2g g ≈ 2.60 eV and there is addi-
group calculations i ≈ 64 meV of the octahe-
tional trigonal distortion ∆tri
dra. Eigenvalues obtained after exact diagonalization of
Hi0 with this ∆CF can be compared with the single-point
We use SpinParser package60 to perform pseud- i
excitations observed in RIXS experiments. In our cal-
ofermion functional renormalization group (pf-FRG) cal-
culation, we find jeff =1/2 - 3/2 separation to be 0.580
culations61,62 on generalized Kitaev spin model obtained
eV which is approximately 23 λ for λ = 0.355 eV esti-
in the previous section. We consider honeycomb lat-
mated from band structure fitting of monolayer OsCl3 .
tice to initialize vertex functions in order to capture
The four-fold jeff =3/2 states are split into two doublets
two-particle interactions on lattice sites separated by up
which are separated by ≈ 56 meV. This splitting is a di-
to 7 lattice bonds. In pf-FRG calculations, the single-
rect consequence of ∆trii present and is also found in Iri-
particle vertex ΣΛ (ω) and two-particle vertex basis func-
dates and α-RuCl3 . By putting ∆trii = 0 eV, as expected,
tions ΓΛ (ω, ω ′ , ω ′′ ) are parameterized by a frequencies ω
jeff =3/2 states regained the four-fold degeneracy.
and the renormalization group cutoff Λ. Here, then we
use a logarithmically spaced mesh of discrete frequencies
as
  n
ωmax Nω −1
ωn = ωmin ; n = (0, 1, · · · , Nω − 1).
ωmin 7. Magnetic interaction and ground state from
cRPA parameters
where ωmin and ωmax are the limit points of frequency
on the positive half-axis which are set to be 0.01 and
35.0 respectively. Nω is the overall number of positive Below in Table S3, we provide estimated magnetic in-
frequencies and is chosen to be 32 in our case. As for teractions corresponding to parameters obtained from
the negative half-axis, the frequencies are generated au- cRPA, i.e. U = 3.8 eV and JH = 0.143 eV.
tomatically by symmetry. The discretization of the cut-
off parameter Λ is also done on a logarithmically spaced
mesh given by, TABLE S3. Estimated bond-dependent magnetic interac-
   tions for cRPA estimated U = 3.8 eV, JH = 0.143 eV. Here
n Λmin also, λ = 0.355 eV. Values of Heisenberg J, Kitaev K, off-
Λn = Λmax b ; n = 0, . . . , logb
Λmax diagonal Γ, and Γ′ terms along with diagonal and off-diagonal
anisotropic terms ξ and ζ are listed. 3NN interactions as well
where the Λmin and Λmax are the lower and upper bound- as some 2NN interactions are < 0.05 meV and hence not listed
aries of Λ which are set to be 0.01 and 50, respectively. here.
The parameter b is step size and is set to be 0.98 in our Magnetic interactions (meV)
calculations. Though we have considered up to 3NN in-
Bond J K Γ Γ′ ξ ζ
teractions in our pf-FRG calculations, negligibly small
2NN and 3NN magnetic interactions do not affect the X1 0.057 -1.318 0.428 -0.334 – 0.235
magnetic ground state obtained with this method at a Y1 0.068 -1.571 0.465 -0.051 – -0.364
qualitative level. We find that most part of the phase Z1 0.043 -1.396 0.372 -0.212 – -0.388
X2 – 0.075 – – – –
points in Fig. 2(a) of the main text can be described by
Y2 0.132 0.126 – – – –
dominant 1NN magnetic interactions only. Z2 – 0.068 – – – –
Dij
6. Atomic features X2
Y2 < |0.04|
Z2
One of the quantities which can be measured from the
resonant inelastic X-ray scattering (RIXS) experiments
are single-point excitations represented by sharp peaks
in the scattering intensity in the relevant energy range. We use these interactions from Table S3 in the pf-FRG
It can be a direct probe for cubic symmetry lowering of calculations and the static spin correlation for the corre-
the Rh-O6 octahedra in a material. Theoretically, such sponding ZZ ground state is shown in Fig. S6. One of
a low-lying crystal field-assisted many-body excitations the M points of the first BZ contributes to this ground
bear a close resemblance with the eigenvalues obtained state indicating some sort of “anisotropic” ZZ state.
 13

uration is obtained from interactions listed in Table S3.

Corresponding spin-wave spectrum, calculated using lin-
ear spin-wave theory as implemented in SpinW63 , are
also shown in Fig. S7(c) and (d).
As one can see that the spectra are gapless in Fig. S7(c)
for the in-plane alignment of spins and are gapped in
Fig. S7(d) for the out-of-plane tilting of spins. This dis-
tinctive feature of the spectra can be understood as fol-
lows. In the absence of magnetic couplings like Γ and
Γ′ , any in-plane spin alignment would classically have
the same energy resulting in a gapless mode. This can
be seen for the spin alignment of Fig. S7(a) results in a
gapless mode at Y point. However, Γ and Γ′ terms dic-
tate the out-of-plane tilting of spins and pinning down of
moments in a specific direction (along one of the Cl-Cl
edges in Os-Cl6 octahedra, see Fig. 1 (a)). This intro-

FIG. S6. Static spin correlations plot, shown in the inset, ob-
tained from pf-FRG calculations using magnetic interactions
from Table S3. Intensity contribution from one of the M
points of the BZ indicates a ZZ ground state. The renormal-
ization group flow of the magnetic correlation (χ) represented
by the green curve breaks at ≈ 0.3 Λ indicating the onset of
the long-range magnetic order.

8. Quantitatively different zigzag states in phase

diagram

As we have mentioned that different part of the phase

diagram in Fig. 2(a) of the main text hosts different mag-
netic ground state depending upon the nature of domi- FIG. S7. Optimized classical magnetic ground states for in-
nant magnetic interactions. The largest part of the phase teractions corresponding to, (a) J = 0.4 meV and K = -18.2
diagram is occupied by the ZZ magnetic state which dif- meV, and (b) interaction listed in Table S3. Spin wave spec-
fers quantitatively in different regions of the phase space. tra obtained using linear spin wave theory for, (c) in-plane
To demonstrate this point, we plot two of them in Fig. S7. configuration shown in (a), and (d) out-of-plane shown in (b)
Spin orientations in these two ZZ states are obtained by
optimization of classical ground state using SpinW pack-
age63 . As one can see in Fig. S7(a), spin alignment is duces additional anisotropy in the Hamiltonian causes
in-plane while spins are tilted in the out-of-plane direc- the spectra to be gapped. Diagonal anisotropic term ξ
tion in Fig. S7(b). In-plane configuration corresponds to may lead to further anisotropy in the system, in-tern ac-
J = 0.4 meV and K = -18.2 meV and out-of-plane config- tively contributing to gapped spectra.