# Anisotropic magnetic interactions in hexagonal AB-stacked kagome lattice
structures: Applications to $\mathrm{Mn}_{3}\mathrm{X}$ ($\mathrm{X}$ =
$\mathrm{Ge}$, $\mathrm{Sn}$, $\mathrm{Ga}$) compounds

A. Zelenskiy  Department of Physics and Atmospheric Science, Dalhousie
University, Halifax, Nova Scotia, Canada B3H 3J5  T. L. Monchesky  Department
of Physics and Atmospheric Science, Dalhousie University, Halifax, Nova
Scotia, Canada B3H 3J5  M. L. Plumer  Department of Physics and Atmospheric
Science, Dalhousie University, Halifax, Nova Scotia, Canada B3H 3J5
Department of Physics and Physical Oceanography, Memorial University of
Newfoundland, St. John’s, Newfoundland, A1B 3X7, Canada  B. W. Southern
Department of Physics and Astronomy, University of Manitoba, Winnipeg,
Manitoba, Canada R3T 2N2

###### Abstract

$\mathrm{Mn}_{3}\mathrm{X}$ compounds in which the magnetic $\mathrm{Mn}$
atoms form AB-stacked kagome lattices have received a tremendous amount of
attention since the observation of the anomalous Hall effect in
$\mathrm{Mn}_{3}\mathrm{Ge}$ and $\mathrm{Mn}_{3}\mathrm{Sn}$. Although the
magnetic ground state has been known for some time to be an inverse triangular
structure with an induced in-plane magnetic moment, there have been several
controversies about the minimal magnetic Hamiltonian. We present a general
symmetry-based model for these compounds that includes a previously unreported
interplane Dzyaloshinskii-Moriya interaction, as well as anisotropic exchange
interactions. The latter are shown to compete with the single-ion anisotropy
which strongly affects the ground state configurations and elementary spin-
wave excitations. Finally, we present the calculated elastic and inelastic
neutron scattering intensities and point to experimental assessment of the
types of magnetic anisotropy in these compounds that may be important.

###### pacs:

Valid PACS appear here

##  I Introduction

Kagome lattice antiferromagnets have been the center of attention in many
branches of condensed matter physics due to their rich electronic and magnetic
properties. In the quantum limit, these materials are believed to provide a
promising platform for experimental realisation of quantum spin liquids and
other unconventional phases Fujihala  _et al._ (2020); Hirschberger  _et al._
(2019); Jiang  _et al._ (2019); Mendels and Bert (2016); Iqbal  _et al._
(2013); Depenbrock  _et al._ (2012); Isakov  _et al._ (2006); Wen  _et al._
(2019); Yan  _et al._ (2011). On the other hand, the interest in semi-
classical non-collinear magnets on kagome lattices has been renewed by recent
studies of their interactions with electric currents. In particular, very
recently a few hexagonal D$0_{19}$ compounds with general formula
$\mathrm{Mn}_{3}\mathrm{X}$,
($\mathrm{X}=\mathrm{Ge},\mathrm{Sn},\mathrm{Ga}$) were predicted and then
experimentally shown to display a large anomalous Hall effect (AHE) Chen  _et
al._ (2014); Nakatsuji  _et al._ (2015); Nayak  _et al._ (2016); Kiyohara  _et
al._ (2016) and topological Hall effect Yan  _et al._ (2019). AHE in
ferromagnetic materials has been studied extensively over the years Nagaosa
_et al._ (2010). However, more recently it was discovered that it depends not
only on the broken time-reversal symmetry but also on the particular type of
the magnetic order and the underlying magnetic interactions. Thus, for
example, it has been well established that in compounds with collinear
ferromagnetic order, spin-orbit coupling is crucial for the AHE Berger (1970).
Unlike collinear antiferromagnets, non-collinear antiferromagnetic structures
have been shown to induce AHE even without spin-orbit coupling. However,
little is known about the types of antiferromagnetic order that can yield AHE
and the majority of the existing theories have been based on the previous
studies of $\mathrm{Mn}_{3}\mathrm{Sn}$ and $\mathrm{Mn}_{3}\mathrm{Ge}$ Chen
_et al._ (2014); Kübler and Felser (2014); Busch  _et al._ (2020). From a
technological standpoint these emergent transport properties of
$\mathrm{Mn}_{3}\mathrm{X}$ compounds are very attractive in the development
of antiferromagnetic spintronics and memory devices since the size of these
materials is not limited by the demagnetizing fields as in the case of
ferromagnets.

Novel techniques have recently been proposed for imaging and writing of
magnetic domains in $\mathrm{Mn}_{3}\mathrm{Sn}$ Reichlova  _et al._ (2019).

Figure 1: Magnetic exchange pathways in the $\mathrm{Mn}_{3}\mathrm{X}$ AB-
stacked kagome crystals. Here, the circles represent the $\mathrm{Mn}$ atoms,
and light blue and dark red indicate atoms with $z=\frac{1}{4}$ and
$z=\frac{3}{4}$ respectively. The numbers further label the six sublattices.

Consequently, there have been several experimental and theoretical studies
focused on determination of the magnetic ground state of
$\mathrm{Mn}_{3}\mathrm{X}$ compounds. Yasukochi Yasukōchi  _et al._ (1961)
and Ohoyama Ohoyama  _et al._ (1961) were the first to identify weak
ferromagnetism in $\mathrm{Mn}_{3}\mathrm{Sn}$ and
$\mathrm{Mn}_{3}\mathrm{Ge}$. Later, the first neutron diffraction studies of
these two compounds led to determination of the non-collinear 120∘ structure
Kouvel and Kasper (1965). Consecutive powder neutron diffraction experiments
on $\mathrm{Mn}_{3}\mathrm{Ga}$ Krén and Kádár (1970),
$\mathrm{Mn}_{3}\mathrm{Ge}$ Kádár and Krén (1971), and
$\mathrm{Mn}_{3}\mathrm{Sn}$ Zimmer  _et al._ (1972) also determined that the
antiferromagnetic order and the induced magnetic moment are restricted to the
plane perpendicular to the $\hat{c}$-axis of the compound. More recent studies
revealed that of the possible triangular magnetic structures, the ground state
of these $\mathrm{Mn}_{3}\mathrm{X}$ compounds is the antichiral “inverse
triangular” structure Brown  _et al._ (1990); Soh  _et al._ (2020); *Soh_supp;
Chen  _et al._ (2020); Duan  _et al._ (2015).

The main challenge in the modelling of the magnetic properties of these
compounds is the abundance of magnetic interactions, as evident from Fig. 1.
The lattice structure, which consists of corner-sharing equilateral triangles,
establishes competing antiferromagnetic exchange interactions leading to
geometric frustration. The exchange couplings originate predominantly from the
Ruderman-Kittel-Kasuya-Yosida (RKKY) interactions which is characteristic of
the metallic magnetic materials with itinerant $d$-electrons Park  _et al._
(2018). Unlike regular kagome lattices, the crystal structure of
$\mathrm{Mn}_{3}\mathrm{X}$ family is referred to as breathing kagome as it
contains adjacent triangles of slightly different size. This results in
anisotropy in the exchange interactions between the spins belonging to
different triangles. From the values of the crystallographic parameters Chen
_et al._ (2020); Park  _et al._ (2018); Brown  _et al._ (1990); Khmelevskyi
_et al._ (2016); TAO-FAN and CHING-KWEI (1966), the differences in bond
lengths for the two types of triangles in $\mathrm{Mn}_{3}\mathrm{Ge}$,
$\mathrm{Mn}_{3}\mathrm{Ga}$ and $\mathrm{Mn}_{3}\mathrm{Sn}$ are
approximately $0.008$ Å, $0.08$ Å, and $0.2$ Å respectively. As a result, the
breathing anisotropy is expected to be largest in $\mathrm{Mn}_{3}\mathrm{Sn}$
and nearly negligible in the other two compounds. In the present work, the
effects of this type of anisotropy will be omitted. All previous studies
included both in-plane ($J_{2}$ and $J_{3}$) and out-of-plane ($J_{1}$)
nearest-neighbour (NN) exchange interactions; however, some studies also
indicate the importance of the next-nearest-neighbour (NNN) exchange
interactions Cable  _et al._ (1993); Park  _et al._ (2018); Chen  _et al._
(2020). Most studies also included in-plane NN Dzyaloshinskii-Moriya (DM)
interactions, although the form of the DM vector has been inconsistent in some
of the recent literature: In most cases the DM vector is chosen to be
perpendicular to the kagome planes,
$\mathbf{D_{ij}}\parallel\mathbf{\hat{z}}$. However, Ref. Liu and Balents
(2017) also includes a term with a DM vector along the in-plane bond
directions.

Lastly, there is an ongoing controversy about the role of the
magnetocrystalline anisotropy in these compounds. It is widely known that due
to geometric frustration, the ground state of a 2D kagome antiferromagnet with
NN interactions only is a 120∘ structure with a macroscopic $U(1)$ degeneracy.
Early inelastic neutron scattering experiments Radhakrishna and Tomiyoshi
(1991); Cable  _et al._ (1993) revealed that the excitation spectrum contains
an anisotropy gap that is associated with the in-plane spin fluctuations. In
order to produce this energy gap, some of the previous studies Cable  _et al._
(1993); Park  _et al._ (2018); Soh  _et al._ (2020); *Soh_supp included 6
order single-ion anisotropy since the 2 and 4 order terms cannot break the
continuous manifold of the 120∘ ground state configuration Cable  _et al._
(1993); Kiyohara  _et al._ (2016); Nagamiya  _et al._ (1982); Liu and Balents
(2017). Nevertheless, it has been reported that due to the deviations from the
120∘ structure induced by weak ferromagnetism in $\mathrm{Mn}_{3}\mathrm{X}$
compounds, the previous arguments no longer apply and 2 order anisotropy is
expected to be sufficient to break the $U(1)$ degeneracy Chen  _et al._
(2020). This observation is relevant to the present study.

In this paper, we present a general magnetic Hamiltonian model for the AB-
stacked hexagonal family of compounds, derived from symmetry considerations in
hopes of resolving some of the existing controversies about the magnetic
interactions in these $\mathrm{Mn}_{3}\mathrm{X}$ compounds. This model is
then used to investigate the relative effects of the single-ion and the
exchange anisotropy on the magnetic structure of the ground state spin
configurations, and in particular on the induced in-plane magnetic moment.
Based on these results, we provide calculations of the elastic neutron
scattering intensities for the systems with different types of magnetic
anisotropy. The impact of these anisotropies on spin waves and inelastic
neutron scattering intensities is then examined.

##  II Model

###  II.1  Structural details

Hexagonal D$0_{19}$ compounds belong to P6${}_{3}/mmc$ (No. 194) space group.
In the case of the $\mathrm{Mn}_{3}\mathrm{X}$ family, the six $\mathrm{Mn}$
atoms are located at the $6h$ Wyckoff positions and form the AB-stacked
breathing kagome lattice planes, while the non-magnetic $\mathrm{X}$ atoms sit
in the centers of the $\mathrm{Mn}$ hexagons (Wyckoff position $2c$). The
atomic coordinates of the six $\mathrm{Mn}$ atoms are
$\left(x,2x,\frac{1}{4}\right)$, $\left(x,\bar{x},\frac{1}{4}\right)$,
$\left(2\bar{x},\bar{x},\frac{1}{4}\right)$,
$\left(\bar{x},2\bar{x},\frac{3}{4}\right)$,
$\left(\bar{x},x,\frac{3}{4}\right)$, $\left(2x,x,\frac{3}{4}\right)$ where
$x$ determines the breathing amplitude of the lattice. When $x=\frac{5}{6}$,
the structure simplifies to a perfect kagome lattice.

###  II.2  Magnetic Hamiltonian

Since the overall spin energy must be invariant under all symmetry
transformations of the space group of the crystal system, the magnetic model
is constructed by identifying all of the spin invariants. For the purpose of
this paper, only terms quadratic in spin components were considered. The full
derivation of the model is provided in the Supplemental Material. The
corresponding spin Hamiltonian is given by

$\displaystyle\mathcal{H}=\mathcal{H}_{K}+\mathcal{H}_{J}+\mathcal{H}_{D}+\mathcal{H}_{A}$
(1)
$\displaystyle\mathcal{H}_{K}=\sum_{\mathbf{r}}\sum_{i}\sum_{\alpha}K_{\alpha}\left(\mathbf{\hat{n}}_{i\alpha}\cdot\mathbf{S}_{i}(\mathbf{r})\right)^{2}$
$\displaystyle\mathcal{H}_{J}=\sum_{\mathbf{r}\mathbf{r^{\prime}}}\sum_{ij}J_{ij}(\mathbf{r}-\mathbf{r}^{\prime})\mathbf{S}_{i}(\mathbf{r})\cdot\mathbf{S}_{j}(\mathbf{r^{\prime}})$
$\displaystyle\mathcal{H}_{D}=\sum_{\mathbf{r}\mathbf{r^{\prime}}}\sum_{ij}D_{ij}(\mathbf{r}-\mathbf{r}^{\prime})\mathbf{\hat{z}}\cdot\left(\mathbf{S}_{i}(\mathbf{r})\times\mathbf{S}_{j}(\mathbf{r^{\prime}})\right)$
$\displaystyle\mathcal{H}_{A}=\sum_{\mathbf{r}\mathbf{r^{\prime}}}\sum_{ij}\sum_{\alpha}A_{ij\alpha}(\mathbf{r}-\mathbf{r^{\prime}})\left(\mathbf{n}_{i\alpha}\cdot\mathbf{S}_{i}(\mathbf{r})\right)\left(\mathbf{n}_{j\alpha}\cdot\mathbf{S}_{j}(\mathbf{r^{\prime}})\right),$

where $\mathcal{H}_{K}$ is the 2 order single-ion anisotropy,
$\mathcal{H}_{J}$ is the isotropic Heisenberg exchange, $\mathcal{H}_{D}$ is
the DM interaction, and $\mathcal{H}_{A}$ is the symmetric, anisotropic
exchange interaction. The latter interactions have also been derived for two-
dimensional kagome planes Essafi  _et al._ (2017). However, the interplane
exchange anisotropies have not been reported before. Sum indices $\mathbf{r}$,
$\mathbf{r^{\prime}}$ label unit cells, $i,j\in\\{1,...,6\\}$ label atoms in
each unit cell, and $\alpha\in\\{x,y,z\\}$ labels the spin vector components.
Vectors $\mathbf{n}_{i\alpha}$ represent local anisotropy axes and are shown
in Fig. 2.

Figure 2: Local anisotropy axes for the six sublattices of
$\mathrm{Mn}_{3}\mathrm{X}$. The local $z$-axes have the same direction (out
of the page). The full vector expressions are given in Supplemental Material.

In total, we identify three 2 order single-ion anisotropy terms with
anisotropy constants $K_{x}$, $K_{y}$, and $K_{z}$, five isotropic exchange
interactions with coupling constants $J_{1}-J_{5}$, three DM interaction terms
with DM vectors $\mathbf{D}_{1}$, $\mathbf{D}_{2}$, and $\mathbf{D}_{3}$ and
ten anisotropic exchange interactions with coupling constants
$A_{i}=A_{ix}=-A_{iy}$, $A_{iz}$ with $i\in{1,...,5}$. Note that in the case
of single-ion anisotropy, there are only two independent coupling constants
since the magnitude of the spins is taken to be fixed. The notation used
throughout this paper is chosen based on the distance between the magnetic
ions: index 1 labels out-of-plane NN interactions, 2 and 3 label in-plane NN
interactions, and 4,5 label NNN interactions. Note also that the symmetry of
the lattice restricts all of the DM vectors to point perpendicular to the
kagome planes. As mentioned in the Introduction, the in-plane DM interactions
have already been implemented in some of the previous studies, however, to our
knowledge, the out-of-plane DM interaction has not been considered before.
Similarly, the exchange anisotropy has not been used in any of the previous
studies of $\mathrm{Mn}_{3}\mathrm{X}$ systems.

The exchange anisotropy, also called bond-dependent anisotropy, typically
originates from the strong spin-orbit coupling Shekhtman  _et al._ (1993).
However, unlike DM interactions, the existence of exchange anisotropy does not
depend on the inversion symmetry of a crystal. In magnetic insulators, similar
terms have been considered, such as the compass and Kitaev interactions
Trousselet  _et al._ (2010); Nussinov and van den Brink (2015); Nikolaev  _et
al._ (2018). In triangular lattices, these interactions have been shown to
stabilize spiral and multi-$Q$ spin configurations Hayami and Motome (2020).

##  III Magnetic ground state

###  III.1 Single layer

Previous studies have established that the inverse-triangular ground state in
two-dimensional kagome systems is stabilized by in-plane DM interaction with a
negative DM constant Soh  _et al._ (2020); *Soh_supp; Chen  _et al._ (2020).
More generally, we find that this state is stable whenever $J_{2}>0$,
$J_{3}>0$, and $D_{2}<0$, $D_{3}<0$. Therefore, throughout the paper we focus
on $\mathrm{Mn}_{3}\mathrm{Ge}$ as a prototype for hexagonal AB-stacked
$\mathrm{Mn}_{3}\mathrm{X}$ compounds, in which case it is reasonable to set
$J_{3}=J_{2}$, $D_{3}=D_{2}$, and $A_{3}=A_{2}$. The spin Hamiltonian for a
single kagome plane simplifies to

$\displaystyle\mathcal{H}_{p}$
$\displaystyle=K_{x}\sum_{\mathbf{r}}\Big{[}(\mathbf{\hat{n}}_{1x}\cdot\mathbf{S}_{1}(\mathbf{r}))^{2}+(\mathbf{\hat{n}}_{2x}\cdot\mathbf{S}_{2}(\mathbf{r}))^{2}+(\mathbf{\hat{n}}_{3x}\cdot\mathbf{S}_{3}(\mathbf{r}))^{2}\Big{]}+J_{2}\sum_{\langle\mathbf{r}\mathbf{r}^{\prime}\rangle}\Big{[}\mathbf{S}_{1}(\mathbf{r})\cdot\mathbf{S}_{2}(\mathbf{r}^{\prime})+\mathbf{S}_{1}(\mathbf{r})\cdot\mathbf{S}_{3}(\mathbf{r}^{\prime})+\mathbf{S}_{2}(\mathbf{r})\cdot\mathbf{S}_{3}(\mathbf{r}^{\prime})\Big{]}$
$\displaystyle+D_{2}\sum_{\langle\mathbf{r}\mathbf{r}^{\prime}\rangle}\mathbf{\hat{z}}\cdot\Big{[}\mathbf{S}_{1}(\mathbf{r})\times\mathbf{S}_{2}(\mathbf{r}^{\prime})-\mathbf{S}_{1}(\mathbf{r})\times\mathbf{S}_{3}(\mathbf{r}^{\prime})+\mathbf{S}_{2}(\mathbf{r})\times\mathbf{S}_{3}(\mathbf{r}^{\prime})\Big{]}$
$\displaystyle+A_{2}\sum_{\langle\mathbf{r}\mathbf{r^{\prime}}\rangle}\Big{[}\left(\mathbf{n}_{1x}\cdot\mathbf{S}_{1}(\mathbf{r})\right)\left(\mathbf{n}_{2x}\cdot\mathbf{S}_{2}(\mathbf{r^{\prime}})\right)+\left(\mathbf{n}_{1x}\cdot\mathbf{S}_{1}(\mathbf{r})\right)\left(\mathbf{n}_{3x}\cdot\mathbf{S}_{3}(\mathbf{r^{\prime}})\right)+\left(\mathbf{n}_{2x}\cdot\mathbf{S}_{2}(\mathbf{r})\right)\left(\mathbf{n}_{3x}\cdot\mathbf{S}_{3}(\mathbf{r^{\prime}})\right)\Big{]}$
$\displaystyle-
A_{2}\sum_{\langle\mathbf{r}\mathbf{r}^{\prime}\rangle}\Big{[}\left(\mathbf{n}_{1y}\cdot\mathbf{S}_{1}(\mathbf{r})\right)\left(\mathbf{n}_{2y}\cdot\mathbf{S}_{2}(\mathbf{r^{\prime}})\right)+\left(\mathbf{n}_{1y}\cdot\mathbf{S}_{1}(\mathbf{r})\right)\left(\mathbf{n}_{3y}\cdot\mathbf{S}_{3}(\mathbf{r^{\prime}})\right)+\left(\mathbf{n}_{2y}\cdot\mathbf{S}_{2}(\mathbf{r})\right)\left(\mathbf{n}_{3y}\cdot\mathbf{S}_{3}(\mathbf{r^{\prime}})\right)\Big{]},$
(2)

where $\langle\cdots\rangle$ represents sums over nearest neighbours.

Figure 3: Inverse-triangular 120∘ structure. The two magnetic configurations
shown here are orthogonal and form a two-dimensional order parameter
$\mathbf{A}=A(\phi)$ which transforms according to $E_{1g}$ irreducible
representation of the point group $D_{6h}$.

The magnetic ground states can be calculated by minimizing the target
Hamiltonian using Monte-Carlo simulated annealing Rutenbar (1989). The
simulations were performed on a system with $6^{3}$ unit cells with runs at a
given temperature consisting of $10^{4}$ Monte-Carlo steps. In the case of a
continuous ordering phase transition, the ordered phase must transform as one
of the irreducible representations of the underlying symmetry group. The
magnetic moments on the magnetic $\mathrm{Mn}$ sites were previously shown to
form an 18-dimensional representation Soh  _et al._ (2020); *Soh_supp; Chen
_et al._ (2021). However, since the experimental results indicate that the
ground state spin configurations are planar, and that the spins in the
$z=\frac{1}{4}$ sublattice are parallel to their inversion-related partners in
the $z=\frac{3}{4}$ sublattice, the representation of the spins can be
expressed in a 6-dimensional form. As outlined in the Supplementary notes of
Ref. Soh  _et al._ (2020); *Soh_supp, this 6-dimensional representation can be
decomposed into a combination of three irreducible representations:
$B_{1g}\oplus B_{2g}\oplus 2E_{1g}$, where $B_{1g}$ and $B_{2g}$ are
1-dimensional, and $E_{1g}$ is 2-dimensional irreducible representations of
the point group $D_{6h}$. When the anisotropic terms in the Hamiltonian are
zero ($K_{x}=0$, $A_{2}=0$), the ground state spin configuration is the
inverse-triangular structure, shown labeled as a two-dimensional vector
$\mathbf{A}$ in Fig. 3. An important feature of the hexagonal
$\mathrm{Mn}_{3}\mathrm{X}$ compounds is that both the in-plane magnetization
$\mathbf{M}$ and the order parameter $\mathbf{A}$ transform according to the
$E_{1g}$ irreducible representation. This results in an invariant coupling of
the order parameter to the magnetization, which has been previously shown to
be $\propto\mathbf{A}\cdot\mathbf{M}$ Soh  _et al._ (2020); *Soh_supp. As a
result, the spin configuration shown in Fig. 3 may be distorted by acquiring
an in-plane magnetic moment without changing the symmetry of the ground state.
The consequences of the induced magnetic moment are far-reaching. When the
ground state spin configuration corresponds to order parameter $\mathbf{A}$,
the constraint that the spin vectors have fixed magnitudes implies that one
can write $\mathbf{A}=A(\phi)$ as function of a single parameter, $\phi$, the
angle of the global in-plane spin rotation.

Figure 4: Distorted inverse-triangular structure, $\mathbf{A}+\mathbf{M}$,
which results from non-zero single-ion and exchange anisotropy interactions.
(a) $K_{x}<0$, $A_{2}=0$ (b) $K_{x}>0$, $A_{2}=0$, (c) $K_{x}=0$, $A_{2}<0$,
and (d) $K_{x}=0$, $A_{2}>0$. The blue arrow indicates the direction of the
induced magnetic moment.

When this spin configuration is inserted in (2), one finds that the energy per
spin is

Figure 5: The magnitude of the induced in-plane magnetic moment per spin as
function of one of the anisotropy parameters, $K_{x}$ (black dots) or $A_{2}$
(red dots), while keeping the other parameter zero.

$E_{\mathbf{A}}=\sqrt{3}D_{2}-J_{2}+\frac{K_{x}}{2},$ (3)

which is independent of $\phi$ and $A_{2}$. Now, when the 120∘ configuration
is distorted, two spins in the unit cell rotate towards (or away from) each
other by a small angle $\theta$, inducing an in-plane magnetic moment. The
magnetic energy can then be written as

$E_{\mathbf{A}+\mathbf{M}}=E_{1}(J_{2},D_{2},\theta)+E_{2}(K_{x},A_{2},\phi,\theta).$
(4)

Assuming that the value of the distortion angle is small, these terms can be
written as

$\displaystyle E_{1}(J_{2},D_{2},\theta)$
$\displaystyle\approx(J_{2}-\sqrt{3}D_{2})\theta^{2},$ (5) $\displaystyle
E_{2}(K_{x},A_{2},\phi,\theta)$
$\displaystyle\approx\cos{2\phi}\left[\frac{K_{x}-2A_{2}}{\sqrt{3}}\theta+\frac{A_{2}+K_{x}}{3}\theta^{2}\right].$
(6)

As a result, the energy has a $\phi$-dependent term which depends on the
strength of the anisotropic interactions, $K_{x}$ and $A_{2}$. The minima of
$E_{\mathbf{A}+\mathbf{M}}$ are determined by the signs and relative
magnitudes of the anisotropic coupling constants as presented in Fig. 4.
Moreover, it can be shown that $\theta=0$ only when both $K_{x}$ and $A_{2}$
are zero. The explicit forms of expressions in (4) are presented in the
Supplemental Material. The important result is that the inclusion of
anisotropic interactions sets a competition with the in-plane DM interaction
that favours the inverse triangular structure. This, in turn, induces an in-
plane magnetic moment, which removes the continuous degeneracy of the
$A(\phi)$ configuration associated with the $U(1)$ symmetry. This is confirmed
by the numerical calculations that were also used to determine the magnitude
of the induced moment per spin, $m=\frac{|\mathbf{M}|}{N}$, as function of the
anisotropy parameters, $K_{x}$ and $A_{2}$, where $N$ represents the number of
spins. Fig. 5 shows the results for a system with $J_{2}=1$ and $D_{2}=-0.2$.
In the high-anisotropy limit ($|K_{x}|>2$ or $|A_{2}|>0.35$), the ground state
configuration changes to a perfect 120∘ structure with spins pointing along
the corresponding local anisotropy axes. In both cases, the relationship
between the anisotropic parameters and the magnitude of the magnetic moment is
approximately linear, with the slope of the $m$ vs $A_{2}$ line approximately
twice as large as that of the $m$ vs $K_{x}$ line that is as expected from
(6).

###  III.2 AB-stacked layers

When the kagome planes are coupled to each other via the out-of-plane
interactions, the Hamiltonian becomes

$\displaystyle\mathcal{H}$
$\displaystyle=\sum_{p_{1},p_{2}}\left[\mathcal{H}_{p_{1}}+\mathcal{H}_{p_{2}}\right]+J_{1}\sum_{\langle\mathbf{r}\mathbf{r}^{\prime}\rangle}\Big{[}\mathbf{S}_{1}(\mathbf{r})\cdot\mathbf{S}_{5}(\mathbf{r}^{\prime})+\mathbf{S}_{1}(\mathbf{r})\cdot\mathbf{S}_{6}(\mathbf{r}^{\prime})+\mathbf{S}_{2}(\mathbf{r})\cdot\mathbf{S}_{4}(\mathbf{r}^{\prime})+\mathbf{S}_{2}(\mathbf{r})\cdot\mathbf{S}_{6}(\mathbf{r}^{\prime})+\mathbf{S}_{3}(\mathbf{r})\cdot\mathbf{S}_{4}(\mathbf{r}^{\prime})+\mathbf{S}_{3}(\mathbf{r})\cdot\mathbf{S}_{5}(\mathbf{r}^{\prime})\Big{]}$
$\displaystyle+D_{1}\sum_{\langle\mathbf{r}\mathbf{r}^{\prime}\rangle}\mathbf{\hat{z}}\cdot\Big{[}-\mathbf{S}_{1}(\mathbf{r})\times\mathbf{S}_{5}(\mathbf{r}^{\prime})+\mathbf{S}_{1}(\mathbf{r})\times\mathbf{S}_{6}(\mathbf{r}^{\prime})+\mathbf{S}_{2}(\mathbf{r})\times\mathbf{S}_{4}(\mathbf{r}^{\prime})-\mathbf{S}_{2}(\mathbf{r})\times\mathbf{S}_{6}(\mathbf{r}^{\prime})-\mathbf{S}_{3}(\mathbf{r})\times\mathbf{S}_{4}(\mathbf{r}^{\prime})+\mathbf{S}_{3}(\mathbf{r})\times\mathbf{S}_{5}(\mathbf{r}^{\prime})\Big{]}$
$\displaystyle+J_{4}\sum_{\langle\mathbf{r}\mathbf{r}^{\prime}\rangle}\Big{[}\mathbf{S}_{1}(\mathbf{r})\cdot\mathbf{S}_{4}(\mathbf{r}^{\prime})+\mathbf{S}_{2}(\mathbf{r})\cdot\mathbf{S}_{5}(\mathbf{r}^{\prime})+\mathbf{S}_{3}(\mathbf{r})\cdot\mathbf{S}_{6}(\mathbf{r}^{\prime})\Big{]},$
(7)

where $p_{1}$ and $p_{2}$ correspond to planes with spins $\mathbf{S}_{1}$,
$\mathbf{S}_{2}$, $\mathbf{S}_{3}$, and $\mathbf{S}_{4}$, $\mathbf{S}_{5}$,
$\mathbf{S}_{6}$ respectively. As in the previous section, we have assumed
perfect kagome planes, which corresponds to setting $J_{4}=J_{5}$. The out-of-
plane exchange anisotropy ($A_{1}$, $A_{4}$) was ignored for simplicity.
First, we investigate the effects of the interplane exchange, $J_{1}$, and DM
interaction, $D_{1}$, on the magnitude of the induced magnetic moment. As
shown in Fig. 6, the antiferromagnetic ($J_{1}>0$) NN interplane coupling
reduces the value of $m$, until at $J_{1}\approx 1.7$ the ground state changes
from $\mathbf{A}+\mathbf{M}$ configuration to a structure with magnetic
wavevector $\mathbf{Q}=\left(\frac{1}{3},\frac{1}{3},0\right)$.

Figure 6: Magnitude of the induced magnetic moment as function of the NN
interplane exchange and DM coupling constants. The values of $K_{x}$ and
$A_{2}$ were chosen to give similar values of $m$ in the decoupled system.
When $J_{1}=0$, the directions of the magnetic moments in different planes are
uncorrelated and the net magnetization is zero on average.

Similarly, for $D_{1}>0$ and $J_{1}=0.5$, the magnitude of the magnetic moment
slowly decreases. In the absence of the anisotropic interactions, $D_{1}>0$
stabilizes the $\mathbf{A}$ configuration, and so we find that even when
$D_{1}=10J_{1}$, the magnitude of the magnetic moment is small but non-zero.
When the interplane coupling is ferromagnetic ($J_{1}<0$), the in-plane
magnetic moments align antiparallel along the $c$-axis such that the total
magnetization is zero (Fig. 7).

Figure 7: Magnetic ground states for AB-stacked kagome planes coupled via NN
out-of-plane exchange interactions ($J_{1}$). (a) $K_{x}>0$, $A_{2}=0$,
$J_{1}>0$, (b) $K_{x}>0$, $A_{2}=0$, $J_{1}<0$, (c) $K_{x}=0$, $A_{2}>0$,
$J_{1}>0$, (d) $K_{x}=0$, $A_{2}>0$, $J_{1}<0$. For all cases, two spins in in
each triangle rotate towards or away from each other, inducing magnetic
moment. Note, however, that the induced in-plane moments cancel in (b) and
(d).

For $D_{1}<0$, the orientation of the spins changes to point along the local
anisotropy axes. Note that the ferromagnetic NNN interactions ($J_{4}<0$),
suggested in previous studies Cable  _et al._ (1993); Chen  _et al._ (2020);
Park  _et al._ (2018) do not introduce any additional energetic competitions
and hence do not change the spin structure presented above.

###  III.3 Elastic neutron scattering

The effects of the anisotropic interactions on the magnetic ground states can
be studied with elastic neutron scattering. In the following, we ignore the
effects of the temperature and consider the case of a single magnetic domain.

Figure 8: (a) magnetic elastic neutron scattering intensity without
anisotropic interactions. (b) effects of the single-ion and exchange
anisotropy on the intensity. The intensity difference, $\Delta I(\bm{\kappa})$
is calculated by subtracting the intensity of the magnetic system without
anisotropic interactions. For both panels, $J_{2}=0.5$ and $D_{2}=0$ were
used.

The elastic scattering differential cross-section is proportional to the
quantity Marshall and Lovesey (1972); Squires (2012)

$I(\bm{\kappa})=|F(\bm{\kappa})|^{2}\sum_{\alpha\beta}\mathcal{S}_{\alpha\beta}(\bm{\kappa})(\delta_{\alpha\beta}-\hat{\kappa}_{\alpha}\hat{\kappa}_{\beta}),$
(8)

where $\bm{\kappa}$ is the scattering vector, $F(\bm{\kappa})$ is the magnetic
form factor, obained from the dipolar approximation Squires (2012); Marshall
and Lovesey (1972), and $\mathcal{S}_{\alpha\beta}(\bm{\kappa})$ is the static
magnetic structure factor:

$\mathcal{S}_{\alpha\beta}(\bm{\kappa})=\sum_{ij}\langle
S_{i\alpha}S_{j\beta}\rangle
e^{i\bm{\kappa}\cdot(\mathbf{r}_{i}-\mathbf{r}_{j})},$ (9)

This expression is calculated assuming that the induced magnetic moment points
along $\hat{\mathbf{n}}_{4y}$ (same as in Fig. 7(c) and (d)). Fig. 8 shows the
effects of the anisotropic interactions on the elastic scattering intensity.
The peaks correspond to a summation over the multiplicity for a given set of
$h$, $k$, and $l$ values and therefore would be appropriate for a powder
sample. In the absence of anisotropic interactions, the spectrum displays six
peaks dictated by the Bragg reflection conditions. When the anisotropic terms
are included and an in-plane magnetic moment is induced, three additional
peaks appear at $\\{20\bar{2}0\\}$, $\\{22\bar{4}0\\}$, and
$\\{22\bar{4}2\\}$. Note that the intensity of these new peaks is much smaller
than the principal peaks. Nevertheless, it could potentially be enhanced with
an applied magnetic field. Comparing the relative intensity ratios of the
principal peaks might allow one to differentiate between the types of magnetic
anisotropy in a given material; however, inelastic neutron scattering may give
better qualitative signatures of the two types of the anisotropy, as discussed
below.

##  IV Spin-wave excitations

We study the impact of the single ion and exchange anisotropies on the spin-
wave fluctuations about the ground state configurations by considering the
plane-wave solutions of the linearized spin torque equations.

Figure 9: Effects of anisotropic interactions on the spin-wave excitations for
a single kagome plane. (a) and (b) represent systems with weak single type of
anisotropy; (c) and (d) show systems with strong single type of anisotropy,
and (e) and (f) display systems with mixed types of anisotropy (one dominating
over another). The dashed blue lines correspond to $K_{x}=0$, $A_{2}=0$, and
the solid red lines correspond to the dispersion with anisotropic
interactions.

In order to perform the spin-wave analysis, it is convenient to introduce a
local coordinate system on each sublattice site such that the equilibrium
directions of the spins coincide with the $z-$components of the local
coordinates LeBlanc  _et al._ (2014). This is done with the use of six
transformation matrices, $\mathbf{U}_{i}$, which allow one to transform the
local spin coordinates into the global ones:
$\mathbf{S}_{i}(\mathbf{r},t)=\mathbf{U}_{i}\mathbf{\tilde{S}}_{i}(\mathbf{r},t)$.
Here, the local coordinates are indicated by the tildes. The collective spin-
wave modes are then described with the use of the Fourier transform of the
spin density:
$\mathbf{\tilde{S}}_{i}(\mathbf{r},t)=\frac{1}{\sqrt{N}}\sum_{\mathbf{q}}\mathbf{\tilde{S}}_{i}(\mathbf{q})e^{i\mathbf{q}\cdot{\mathbf{r}}-i\omega
t}$. The linearized equations of motion simplify to

$-i\omega\tilde{S}_{\alpha i}(\mathbf{q})=\Gamma_{\alpha\beta
ij}\tilde{S}_{\beta j}(\mathbf{q}),$ (10)

where Einstein summation is implied. Combining the spin component and
sublattice indices, (10) becomes an eigenvalue problem, which in general must
be solved numerically. In all of the following calculations, the ground state
magnetic structures are assumed to have an induced magnetic moment in the
$\hat{\mathbf{n}}_{4y}$ direction.

###  IV.1  Single layer

First, we consider the effects of single-ion and exchange anisotropies in a
single kagome plane where the magnetic structure is determined by minimizing
(2). The solutions of the linearized equations of motion (10) correspond to
three spin-wave modes. When only isotropic exchange interaction are present in
the system, all three modes are gapless with a single dispesionless mode,
which reflects the macroscopic degeneracy of the 120∘ ground state Harris  _et
al._ (1992); LeBlanc  _et al._ (2014); Chernyshev and Zhitomirsky (2015).
Intraplanar NN DM coupling ($D_{2}$) lifts the energies of two of the modes
resulting in one acoustic and two optical modes Chernyshev and Zhitomirsky
(2015). The corresponding frequencies are

$\displaystyle\omega_{1,2}^{2}$
$\displaystyle=3(J_{2}-\sqrt{3}D_{2})^{2}-J_{2}(J_{2}-\sqrt{3}D_{2})f(\mathbf{q})$
$\displaystyle\pm D_{2}(J_{2}-\sqrt{3}D_{2})\sqrt{9+6f(\mathbf{q}))},$ (11)
$\displaystyle\omega_{3}^{2}$ $\displaystyle=6D_{2}(3D_{2}-\sqrt{3}J_{2}).$
(12)

where

$f(\mathbf{q})=\cos(2q_{x})+\cos\left(q_{x}+\sqrt{3}q_{y}\right)+\cos\left(q_{x}-\sqrt{3}q_{y}\right)$
(13)

where $q_{x}$ and $q_{y}$ lie within the first Brillouin zone and the NN
lattice parameter, $a$, was set to 1. Note that the DM interactions lift the
dispersionless mode to a finite frequency. Near the $\Gamma$ point,
$\mathbf{q}=0$, the expressions for the frequencies become

$\displaystyle\omega_{1}\approx\sqrt{(J_{2}-\sqrt{3}D_{2})(3J_{2}-\sqrt{3}D_{2})}|\mathbf{q}|$
(14)
$\displaystyle\omega_{2}\approx\sqrt{6D_{2}(3D_{2}-\sqrt{3}J_{2})+(J_{2}-\sqrt{3}D_{2})(3J_{2}+\sqrt{3}D_{2})|\mathbf{q}|^{2}},$
(15) $\displaystyle\omega_{3}=\sqrt{6D_{2}(3D_{2}-\sqrt{3}J_{2})}.$ (16)

Thus, $D_{2}<0$ enhances the velocity of $\omega_{1}$ and $\omega_{2}$ at the
$\Gamma$ point. The spin-wave modes with and without anisotropy are presented
in Fig. 9. When $D_{2}$ is zero, these expressions simplify to
$\omega_{1,2}=\sqrt{3}J_{2}|\mathbf{q}|$ and $\omega_{3}=0$, which is
consistent with Eq. 69 in Ref. Dasgupta and Tchernyshyov (2020). The inclusion
of anisotropic interactions breaks the degeneracy of the doublet and
introduces a gap in the dispersion of the acoustic mode. Here,
$K_{1}=\Big{(}\frac{1}{3},\frac{1}{3},0\Big{)}$,
$K_{2}=\Big{(}\frac{2}{3},\bar{\frac{1}{3}},0\Big{)}$,
$M_{1}=\Big{(}\frac{1}{2},0,0\Big{)}$,
$M_{2}=\Big{(}\frac{1}{2},\bar{\frac{1}{2}},0\Big{)}$. The different symmetry
points were chosen to be at 60 degrees to each other. An important feature
that can be observed throughout these results is that these modes break the
six-fold rotational symmetry of the material, which can be seen by comparing
the dispersion in the $\Gamma K_{1}$ and $\Gamma K_{2}$ regions. The reason
for this symmetry breaking is the fact that the induced magnetic moment pins
the ground state configuration with only one of the sublattices oriented
parallel to its respective local axis, determined by the anisotropic
interactions. The results in Fig. 9 are presented in such a way as to compare
the dispersion for both weak and strong single type (either single ion or
exchange) anisotropy as well as the modes corresponding to the systems with
mixed anisotropic interactions. It is clear that the qualitative features of
the spin-wave dispersion (such as energy gaps) depend strongly on the
anisotropic terms even when the corresponding coupling constants are small.
The calculations of the analytical expressions for the energy gaps at the
$\Gamma$ point are given in the Supplemental Material.

###  IV.2  AB-stacked layers planes

When the AB-stacked kagome planes become coupled, the equations of motion
produce six independent modes, three of which are acoustic and three are
optical. When both $D_{1}$ and $D_{2}$ are zero, the acoustic modes are
gapless, and the optic modes form a singlet and a doublet. The velocities of
these modes at the $\Gamma$ point depend on the relative strengths of the
interplanar coupling constants. The velocity of the lowest energy mode is
given by

$\displaystyle v_{1}^{xy}$
$\displaystyle=a\sqrt{\frac{3(J_{2}-4J_{4})(J_{1}+J_{2})(3J_{2}+2J_{1}-12J_{4})}{3J_{2}+J_{1}-12J_{4}}},$
(17) $\displaystyle v_{1}^{z}$
$\displaystyle=c\sqrt{6(J_{1}+J_{2})(J_{1}-3J_{4})}.$ (18)

Note that the velocities of the spin-wave modes have been previously
calculated in Refs. Chen  _et al._ (2020); Dasgupta and Tchernyshyov (2020),
however these references used different notation for the NNN coupling
constants: $J_{4}$ and $J_{5}$ constants in the present work were labelled as
$J_{3}$ and $J_{4}$ respectively. Furthermore, the references mentioned above
set $J_{4}=0$ ($J_{3}=0$ in the alternative notation), whereas, as mentioned
previously, we have set $J_{4}=J_{5}$ throughout the paper. As a result, the
expression for the out-of-plane velocity ($v_{1}^{(z)}$) presented here is
equivalent to the previously reported expressions (Eq. 7 in Ref. Chen  _et
al._ (2020) and Eq. 73 in Ref. Dasgupta and Tchernyshyov (2020)), within the
assumptions made in regards to the NNN coupling constants. However, the
expression for the in-plane velocity ($v_{1}^{(xy)}$) is very different from
those presented previously, although it yields similar numerical values. The
origins of this discrepancy are unclear. The velocities for the remaining
modes are given in the Supplemental Material. DM interactions lift the energy
of two of the acoustic modes leading to a gapless singlet and a gapped
doublet. The corresponding squared frequencies are

Figure 10: Spin-wave modes in AB-stacked kagome antiferromagnet. Here,
$A=\Big{(}0,0,\frac{1}{2}\Big{)}$. In (a), (c), (e) the anisotropy parameters
are set to $K_{x}=0.5$, $A_{2}=0.05$, and in (b), (d), (f) they are
$K_{x}=0.1$, $A_{2}=0.25$. (a) and (b) show the dispersion of a system of
kagome planes coupled via NN exchange ($J_{1}=0.2$) only, (c) and (d) include
also the NNN inter-planar exchange coupling ($J_{4}=-0.5$), and finally, (e)
and (f) also have interplane DM interaction ($D_{1}=0.1$). The dashed blue
lines correspond to $K_{x}=0$, $A_{2}=0$, and the solid red lines correspond
to the dispersion with anisotropic interactions. Figure 11: Relative magnitude
of the inelastic scattering function $\mathcal{S}(\mathbf{q},\omega)$
(colorbar scale) near the $\Gamma$ point, $\mathbf{q}=[11\bar{1}0]$, assuming
that the induced magnetic moment is parallel to $\hat{\mathbf{n}}_{4y}$. (a),
(e) and (c), (g) correspond to single kind of anisotropy (single-ion and
exchange respectively), (b), (f) and (d), (h) have both kinds of anisotropy.
We have set $J_{1}=0.2$, $J_{4}=-0.5$, and $D_{1}=0$.

$\displaystyle\omega_{1}^{2}$ $\displaystyle=0,$ (19)
$\displaystyle\omega_{2,3}^{2}$
$\displaystyle=6\sqrt{3}(J_{1}+J_{2})(D_{1}-D_{2})+18(D1-D2)^{2}$ (20)
$\displaystyle\omega_{4,5}^{2}$
$\displaystyle=6D_{1}^{2}-24D_{1}D_{2}+6\sqrt{3}D_{1}J_{1}+6\sqrt{3}D_{1}J_{2}+144J_{4}^{2}$
$\displaystyle-36\sqrt{3}D_{1}J_{4}+18D_{2}^{2}-14\sqrt{3}D_{2}J_{1}-6\sqrt{3}D_{2}J_{2}$
$\displaystyle+60\sqrt{3}D_{2}J_{4}+4J_{1}^{2}+12J_{1}J_{2}-60J_{1}J_{4}-36J_{2}J_{4},$
(21) $\displaystyle\omega_{6}^{2}$
$\displaystyle=24D_{1}^{2}-24D_{1}D_{2}+24\sqrt{3}D_{1}J_{2}-72\sqrt{3}D_{1}J_{4}$
$\displaystyle-8\sqrt{3}D_{2}J_{1}+24\sqrt{3}D_{2}J_{4}-8J_{1}^{2}+24J_{1}J_{2}-24J_{1}J_{4}$
$\displaystyle-72J_{2}J_{4}+144J_{4}^{2}.$ (22)

Frequencies $\omega_{1-3}$ correspond to acoustic modes and $\omega_{4-6}$ –
to optical modes. Note that the NNN exchange interactions ($J_{4}$) only enter
the expressions 19 \- 22 for the optic modes. The dispersion of the spin-wave
modes for AB-staked kagome planes are presented in Fig. 10. We choose the
values of the interplane coupling based on the previous estimates of the
exchange parameters for $\mathrm{Mn}_{3}\mathrm{Ge}$ Cable  _et al._ (1993);
Park  _et al._ (2018); Chen  _et al._ (2020). Thus, we choose $J_{4}$ to be
large (and negative), and $J_{1}$ to be small (and positive). The value of
$D_{1}$ was chosen to be half that of $J_{1}$. The two cases considered here
correspond to one type of anisotropy dominating over another with the values
of $K_{x}$ and $A_{2}$ chosen to give similar magnitudes of induced magnetic
moment. In both cases, the strong ferromagnetic NNN exchange ($J_{4}$)
interactions lead to a large energy gap between the three lowest and three
highest energy modes. In an actual experiment, the latter typically appear at
very high energies and are often unobservable Cable  _et al._ (1993);
Radhakrishna and Tomiyoshi (1991); Sukhanov  _et al._ (2019); Chen  _et al._
(2020). The main qualitative differences between the two anisotropy regimes
correspond to the branch crossings between the three lowest energy modes. The
small inter-planar DM interaction ($D_{1}$) moves the second and third lowest
energy modes higher in energy and closer together, eventually making them
nearly degenerate.

Figure 12: Relative magnitude of dynamic structure factor,
$\mathcal{S}(\mathbf{q},\omega)$, (colorbar scale) near
$\mathbf{q}=[11\bar{1}0]$ where we have set $J_{1}=0.2$, $J_{4}=-0.5$, and
$D_{1}=0.1$.

##  V Inelastic neutron scattering

The inelastic scattering cross-section can be written as

$\frac{d^{2}\sigma}{d\Omega dE^{\prime}}=\left(\frac{\gamma
e^{2}}{m_{e}c^{2}}\right)^{2}\frac{k^{\prime}}{k}\sum_{\alpha\beta}(\delta_{\alpha\beta}-\hat{q}_{\alpha}\hat{q}_{\beta})\mathcal{S}_{\alpha\beta}(\mathbf{q},\omega),$
(23)

where

$\mathcal{S}_{\alpha\beta}(\mathbf{q},\omega)=\frac{1}{2\pi\hbar}\int_{-\infty}^{\infty}dte^{-i\omega
t}\langle T_{\alpha}(\mathbf{q},t)T_{\beta}(-\mathbf{q},t)\rangle$ (24)

is the dynamic structure factor, and

$T_{\alpha}(\mathbf{q},t)=F(q)\sum_{\mathbf{r}}\sum_{i}\sum_{a}e^{-i\mathbf{q}\cdot(\mathbf{r}+\mathbf{r}_{i})}\Lambda_{\alpha
ai}\tilde{S}_{ai}(\mathbf{r},t).$ (25)

The dynamic structure factor can be calculated using standard Green’s function
methods LeBlanc  _et al._ (2014); Marshall and Lovesey (1972); Squires (2012).
As mentioned earlier, previous inelastic scattering experiments for
$\mathrm{Mn}_{3}\mathrm{Ge}$ Cable  _et al._ (1993); Radhakrishna and
Tomiyoshi (1991); Sukhanov  _et al._ (2019); Chen  _et al._ (2020) and
$\mathrm{Mn}_{3}\mathrm{Sn}$ Park  _et al._ (2018) indicated that the high
energy modes are found at $E\sim 100$ meV and are often hard to resolve.
Therefore, we focus on the qualitative features of the lower energy branches.
Fig. 11 shows $\mathcal{S}(\mathbf{q},\omega)$ calculated assuming that the
induced magnetic moment is oriented along the $\hat{\mathbf{n}}_{4y}$
direction with $J_{2}=1$, $D_{2}=-0.2$, $J_{1}=0.2$, $D_{1}=0.1$ and
$J_{4}=-0.5$. The intensity is largest for $\mathbf{q}$-vectors which give the
elastic peaks, but for smaller wave vectors it drops rapidly; however, the
lower energy modes should be distinguishable in an experiment. Importantly,
the characteristic features of the dispersion curves in Fig. 10 are clearly
seen and can be used to deduce the types of anisotropic interactions in the
$\mathrm{Mn}_{3}\mathrm{X}$ compounds. First of all, the dominant anisotropic
interactions can be deduced from the separation of spin-wave branches and the
asymmetry of the modes around the $\Gamma$ point. From there, the possibility
of mixed types of anisotropic interactions can be investigated by comparing
the relative energy gaps and the velocities of the spin-wave modes, as well
as, where possible, the qualitative features like the branch crossings.

Fig. 12 shows the effects of the interplane DM interaction on the relative
intensities of the dynamic structure factor. As stated in the previous
section, the second and third lowest energy modes are pushed closer to each
other, and could potentially appear as a single line in an experiment. This
situation could be the case for the spin-wave spectra for
$\mathrm{Mn}_{3}\mathrm{Ge}$ Sukhanov  _et al._ (2019); Chen  _et al._ (2020),
although further experimental studies might be illuminating.

##  VI Conclusions

In summary, we have used symmetry considerations in order to construct a
general magnetic Hamiltonian for AB-stacked magnetic kagome planes with
hexagonal symmetry, with a focus on $\mathrm{Mn}_{3}\mathrm{X}$ compounds. In
addition to the previously known interactions we have also derived from
symmetry an additional NN inter-planar DM coupling, as well as symmetric
anisotropic exchange interactions. The magnetic ground state of the
$\mathrm{Mn}_{3}\mathrm{X}$ systems, which corresponds to the distorted
inverse-triangular structure, was shown to depend strongly on the anisotropic
terms in the the model. In particular, the magnitude and direction of the in-
plane magnetic moment, induced by the distortion of the 120∘ state, is
determined by the relative strengths of the single-ion and exchange
anisotropies. In either case, the anisotropy pins the ground state removing
the continuous degeneracy of the 120∘ configuration.

Bond-dependent anisotropic exchange interactions in bulk magnetic systems have
been, for the most part, neglected in the literature, despite having similar
physical origin as the DM interaction. Nevertheless, as some recent studies
indicate, this type of interaction can be crucial for understanding the
magnetic properties of some materials. In particular, Kitaev-type interactions
in honeycomb $\mathrm{Na}_{2}\mathrm{IrO}_{3}$ have been shown to dominant
over isotropic antiferromagnetic exchange Sizyuk  _et al._ (2014); Katukuri
_et al._ (2014). Another example is the recently synthesized compound
$\mathrm{YbMgGaO}_{4}$ with triangular lattice structure where the exchange
anisotropy was argued to stabilize the quantum spin liquid ground state Li
_et al._ (2015); Maksimov  _et al._ (2019).

The two types of the anisotropic interactions have opposing effects on the
elastic scattering intensity; however distinguishing between the different
kinds of anisotropic interactions from the elastic scattering experiments only
may be challenging. On the contrary, the spin-wave excitations were shown to
be very sensitive to even small changes in anisotropic coupling constants
which makes the inelastic neutron scattering a better candidate for studying
the anisotropic effects in these compounds. Both kinds of anisotropy break the
degeneracy of the optical modes and introduce a gap in the acoustic mode. The
dispersion was also shown to break the six-fold rotational symmetry,
reflecting the “pinning” of the ground state. Most importantly, we have shown
that the characteristic features of the excitation spectra for systems with
one or two kinds of magnetic anisotropy should be accessible in an
experimental setting.

##  VII  Acknowledgements

The authors would like to thank J. S. R. McCoombs and S. H. Curnoe for very
valuable discussions and suggestions. This work was supported by the Natural
Sciences and Engineering Research Council of Canada (NSERC).

## References

  * Fujihala  _et al._ (2020) M. Fujihala, K. Morita, R. Mole, S. Mitsuda, T. Tohyama, S.-i. Yano, D. Yu, S. Sota, T. Kuwai, A. Koda, H. Okabe, H. Lee, S. Itoh, T. Hawai, T. Masuda, H. Sagayama, A. Matsuo, K. Kindo, S. Ohira-Kawamura, and K. Nakajima, Nature Communications 11, 3429 (2020).
  * Hirschberger  _et al._ (2019) M. Hirschberger, T. Nakajima, S. Gao, L. Peng, A. Kikkawa, T. Kurumaji, M. Kriener, Y. Yamasaki, H. Sagayama, H. Nakao, K. Ohishi, K. Kakurai, Y. Taguchi, X. Yu, T.-h. Arima, and Y. Tokura, Nature Communications 10, 5831 (2019).
  * Jiang  _et al._ (2019) S. Jiang, P. Kim, J. H. Han, and Y. Ran, SciPost Physics 7 (2019), 10.21468/scipostphys.7.1.006.
  * Mendels and Bert (2016) P. Mendels and F. Bert, Comptes Rendus Physique 17, 455 (2016), physique de la matière condensée au XXIe siècle: l’héritage de Jacques Friedel.
  * Iqbal  _et al._ (2013) Y. Iqbal, F. Becca, S. Sorella, and D. Poilblanc, Phys. Rev. B 87, 060405 (2013).
  * Depenbrock  _et al._ (2012) S. Depenbrock, I. P. McCulloch, and U. Schollwöck, Phys. Rev. Lett. 109, 067201 (2012).
  * Isakov  _et al._ (2006) S. V. Isakov, S. Wessel, R. G. Melko, K. Sengupta, and Y. B. Kim, Phys. Rev. Lett. 97, 147202 (2006).
  * Wen  _et al._ (2019) J. Wen, S.-L. Yu, S. Li, W. Yu, and J.-X. Li, npj Quantum Materials 4, 12 (2019).
  * Yan  _et al._ (2011) S. Yan, D. A. Huse, and S. R. White, Science 332, 1173 (2011).
  * Chen  _et al._ (2014) H. Chen, Q. Niu, and A. H. MacDonald, Phys. Rev. Lett. 112, 017205 (2014).
  * Nakatsuji  _et al._ (2015) S. Nakatsuji, N. Kiyohara, and T. Higo, Nature 527, 212 (2015).
  * Nayak  _et al._ (2016) A. K. Nayak, J. E. Fischer, Y. Sun, B. Yan, J. Karel, A. C. Komarek, C. Shekhar, N. Kumar, W. Schnelle, J. Kübler, C. Felser, and S. S. P. Parkin, Science Advances 2 (2016), 10.1126/sciadv.1501870.
  * Kiyohara  _et al._ (2016) N. Kiyohara, T. Tomita, and S. Nakatsuji, Phys. Rev. Applied 5, 064009 (2016).
  * Yan  _et al._ (2019) J. Yan, X. Luo, H. Y. Lv, Y. Sun, P. Tong, W. J. Lu, X. B. Zhu, W. H. Song, and Y. P. Sun, Applied Physics Letters 115, 102404 (2019), https://doi.org/10.1063/1.5119838 .
  * Nagaosa  _et al._ (2010) N. Nagaosa, J. Sinova, S. Onoda, A. H. MacDonald, and N. P. Ong, Rev. Mod. Phys. 82, 1539 (2010).
  * Berger (1970) L. Berger, Phys. Rev. B 2, 4559 (1970).
  * Kübler and Felser (2014) J. Kübler and C. Felser, EPL (Europhysics Letters) 108, 67001 (2014).
  * Busch  _et al._ (2020) O. Busch, B. Göbel, and I. Mertig, Phys. Rev. Research 2, 033112 (2020).
  * Reichlova  _et al._ (2019) H. Reichlova, T. Janda, J. Godinho, A. Markou, D. Kriegner, R. Schlitz, J. Zelezny, Z. Soban, M. Bejarano, H. Schultheiss, P. Nemec, T. Jungwirth, C. Felser, J. Wunderlich, and S. T. B. Goennenwein, Nature Communications 10, 5459 (2019).
  * Yasukōchi  _et al._ (1961) K. Yasukōchi, K. Kanematsu, and T. Ohoyama, Journal of the Physical Society of Japan 16, 1123 (1961), https://doi.org/10.1143/JPSJ.16.1123 .
  * Ohoyama  _et al._ (1961) T. Ohoyama, K. Yasukōchi, and K. Kanematsu, Journal of the Physical Society of Japan 16, 352 (1961), https://doi.org/10.1143/JPSJ.16.352 .
  * Kouvel and Kasper (1965) J. Kouvel and J. Kasper, Proceedings of the International Conference on Magnetism, Nottingham (1965).
  * Krén and Kádár (1970) E. Krén and G. Kádár, Solid State Communications 8, 1653 (1970).
  * Kádár and Krén (1971) G. Kádár and E. Krén, Int. J. Magn 1, 143 (1971).
  * Zimmer  _et al._ (1972) G. J. Zimmer, E. Kren, C. D. Graham, and J. J. Rhyne, AIP Conference Proceedings 5, 513 (1972), https://aip.scitation.org/doi/pdf/10.1063/1.3699489 .
  * Brown  _et al._ (1990) P. J. Brown, V. Nunez, F. Tasset, J. B. Forsyth, and P. Radhakrishna, Journal of Physics: Condensed Matter 2, 9409 (1990).
  * Soh  _et al._ (2020) J.-R. Soh, F. de Juan, N. Qureshi, H. Jacobsen, H.-Y. Wang, Y.-F. Guo, and A. T. Boothroyd, Phys. Rev. B 101, 140411 (2020).
  * (28) J.-R. Soh, F. de Juan, N. Qureshi, H. Jacobsen, H.-Y. Wang, Y.-F. Guo, and A. T. Boothroyd, “Supplemental material,” http://link.aps.org/supplemental/10.1103/PhysRevB.101.140411.
  * Chen  _et al._ (2020) Y. Chen, J. Gaudet, S. Dasgupta, G. G. Marcus, J. Lin, T. Chen, T. Tomita, M. Ikhlas, Y. Zhao, W. C. Chen, M. B. Stone, O. Tchernyshyov, S. Nakatsuji, and C. Broholm, Phys. Rev. B 102, 054403 (2020).
  * Duan  _et al._ (2015) T. F. Duan, W. J. Ren, W. L. Liu, S. J. Li, W. Liu, and Z. D. Zhang, Applied Physics Letters 107, 082403 (2015), https://doi.org/10.1063/1.4929447 .
  * Park  _et al._ (2018) P. Park, J. Oh, K. Uhlířová, J. Jackson, A. Deák, L. Szunyogh, K. H. Lee, H. Cho, H.-L. Kim, H. C. Walker, D. Adroja, V. Sechovský, and J.-G. Park, npj Quantum Materials 3, 63 (2018).
  * Khmelevskyi  _et al._ (2016) S. Khmelevskyi, A. B. Shick, and P. Mohn, Applied Physics Letters 109, 222402 (2016), https://doi.org/10.1063/1.4970691 .
  * TAO-FAN and CHING-KWEI (1966) C. TAO-FAN and L. CHING-KWEI, Acta Physica Sinica 22, 1004 (1966).
  * Cable  _et al._ (1993) J. W. Cable, N. Wakabayashi, and P. Radhakrishna, Phys. Rev. B 48, 6159 (1993).
  * Liu and Balents (2017) J. Liu and L. Balents, Phys. Rev. Lett. 119, 087202 (2017).
  * Radhakrishna and Tomiyoshi (1991) P. Radhakrishna and S. Tomiyoshi, Journal of Physics: Condensed Matter 3, 2523 (1991).
  * Nagamiya  _et al._ (1982) T. Nagamiya, S. Tomiyoshi, and Y. Yamaguchi, Solid State Communications 42, 385 (1982).
  * Essafi  _et al._ (2017) K. Essafi, O. Benton, and L. D. C. Jaubert, Phys. Rev. B 96, 205126 (2017).
  * Shekhtman  _et al._ (1993) L. Shekhtman, A. Aharony, and O. Entin-Wohlman, Phys. Rev. B 47, 174 (1993).
  * Trousselet  _et al._ (2010) F. Trousselet, A. M. Oleś, and P. Horsch, EPL (Europhysics Letters) 91, 40005 (2010).
  * Nussinov and van den Brink (2015) Z. Nussinov and J. van den Brink, Rev. Mod. Phys. 87, 1 (2015).
  * Nikolaev  _et al._ (2018) S. A. Nikolaev, I. V. Solovyev, A. N. Ignatenko, V. Y. Irkhin, and S. V. Streltsov, Phys. Rev. B 98, 201106 (2018).
  * Hayami and Motome (2020) S. Hayami and Y. Motome, “Noncoplanar multiple-$q$ spin textures by itinerant frustration: Effects of single-ion anisotropy and bond-dependent anisotropy,” (2020), arXiv:2011.13040 [cond-mat.str-el] .
  * Rutenbar (1989) R. A. Rutenbar, IEEE Circuits and Devices Magazine 5, 19 (1989).
  * Chen  _et al._ (2021) T. Chen, T. Tomita, S. Minami, M. Fu, T. Koretsune, M. Kitatani, I. Muhammad, D. Nishio-Hamane, R. Ishii, F. Ishii, R. Arita, and S. Nakatsuji, Nature Communications 12, 572 (2021).
  * Marshall and Lovesey (1972) W. Marshall and S. Lovesey,  _Theory of thermal neutron scattering: the use of neutrons for the investigation of condensed_, Vol. 12 (Oxford University Press, 1972).
  * Squires (2012) G. L. Squires,  _Introduction to the Theory of Thermal Neutron Scattering_, 3rd ed. (Cambridge University Press, 2012).
  * LeBlanc  _et al._ (2014) M. D. LeBlanc, B. W. Southern, M. L. Plumer, and J. P. Whitehead, Phys. Rev. B 90, 144403 (2014).
  * Harris  _et al._ (1992) A. B. Harris, C. Kallin, and A. J. Berlinsky, Phys. Rev. B 45, 2899 (1992).
  * Chernyshev and Zhitomirsky (2015) A. L. Chernyshev and M. E. Zhitomirsky, Phys. Rev. B 92, 144415 (2015).
  * Dasgupta and Tchernyshyov (2020) S. Dasgupta and O. Tchernyshyov, Phys. Rev. B 102, 144417 (2020).
  * Sukhanov  _et al._ (2019) A. S. Sukhanov, M. S. Pavlovskii, P. Bourges, H. C. Walker, K. Manna, C. Felser, and D. S. Inosov, Phys. Rev. B 99, 214445 (2019).
  * Sizyuk  _et al._ (2014) Y. Sizyuk, C. Price, P. Wölfle, and N. B. Perkins, Phys. Rev. B 90, 155126 (2014).
  * Katukuri  _et al._ (2014) V. M. Katukuri, S. Nishimoto, V. Yushankhai, A. Stoyanova, H. Kandpal, S. Choi, R. Coldea, I. Rousochatzakis, L. Hozoi, and J. van den Brink, New Journal of Physics 16, 013056 (2014).
  * Li  _et al._ (2015) Y. Li, G. Chen, W. Tong, L. Pi, J. Liu, Z. Yang, X. Wang, and Q. Zhang, Phys. Rev. Lett. 115, 167203 (2015).
  * Maksimov  _et al._ (2019) P. A. Maksimov, Z. Zhu, S. R. White, and A. L. Chernyshev, Phys. Rev. X 9, 021017 (2019).