# Supplemental Material for Boundary obstructed topological superconductor in
buckled honeycomb lattice under perpendicular electric field

Rasoul Ghadimi  Seunghun Lee  Bohm-Jung Yang

###### Contents

  1. S1 The derivation of the Bogoliubov-de-Gennes (BdG) Hamiltonian
    1. S1.1 normal state
    2. S1.2 pairing potential
    3. S1.3 BdG Hamiltonian
    4. S1.4 Symmetries
    5. S1.5 Basis transformation
  2. S2 Edge Hamiltonian and symmetry representation
  3. S3 Edge Theory
  4. S4 Large Sublattice Potential Limit: Effective Hamiltonian
  5. S5 winding number as a topological invariant for effective Kitaev chain
  6. S6 Spin polarized case
  7. S7 f-wave spin-triplet pairing and quantum spin hall effect, large spin-orbit coupling limit
  8. S8 f-wave spin-triplet pairing and valley quantum hall effect
  9. S9 topological invariant for the strip of the honeycomb lattice
  10. S10 BOTS at dangling edge

##  S1 The derivation of the Bogoliubov-de-Gennes (BdG) Hamiltonian

###  S1.1 normal state

The general Hamiltonian of a honeycomb material with $p_{z}$ orbital in the
normal state reads as,

$\displaystyle
H=-t_{1}\sum_{\expectationvalue{i,j},s}c^{\dagger}_{is}c_{js}+t_{2}\sum_{\expectationvalue{\expectationvalue{i,j}},s}c^{\dagger}_{is}c_{js}-\mu\sum_{i,s}c^{\dagger}_{is}c_{is}$
$\displaystyle+M\sum_{i,s}\kappa_{i}c^{\dagger}_{is}c_{is}+i\lambda_{SO}\sum_{\expectationvalue{\expectationvalue{i,j}},ss^{\prime}}v_{ij}c^{\dagger}_{is}[s_{z}]_{ss^{\prime}}c_{js^{\prime}}$
$\displaystyle+i\lambda_{R}\sum_{\expectationvalue{\expectationvalue{i,j}},ss^{\prime}}c^{\dagger}_{is}[(\vec{s}\times\hat{d}_{ij})_{\hat{z}}]_{ss^{\prime}}c_{js^{\prime}},$
(S1)

where, $t_{1}$, $t_{2}$, $\mu$, $\lambda_{SO}$, $\lambda_{R}$, and $M$ are
nearest-neighbor hopping, next-nearest-neighbor hopping, chemical potential,
intrinsic spin-orbit coupling, Rashba spin-orbit coupling, and sublattice
potential, respectively. In Eq. (S1.1), $\expectationvalue{i,j}$, and
$\expectationvalue{\expectationvalue{i,j}}$ indicate nearest-neighbor and
next-nearest-neighbor links in the honeycomb structure. The $s_{\alpha=x,y,z}$
are usual Pauli matrices that act on the spin degree of freedom, and
$c^{\dagger}_{is}$, $c_{is}$ are the creation and annihilation operator of an
electron in $i$’th site with spin $s=\uparrow,\downarrow$, respectively. We
set $\kappa_{i\in\text{A}}=1$ and $\kappa_{i\in\text{B}}=-1$ where A and B
denote the two sublattices of the honeycomb lattice (Fig. 1(a)). In Eq.
(S1.1), $\hat{d}_{ij}$ is the direction that connect $i$ to $j$ site and
$v_{ij}=\tfrac{\vec{d}_{ik}\times\vec{d}_{kj}}{|\vec{d}_{ik}\times\vec{d}_{kj}|}$,
where $k$ is the intermediate site that connect $i$ to site $j$ by a nearest-
neighbor path. By employing Fourier transformation
$c_{is}^{\dagger}=\frac{1}{\sqrt{N}}\sum_{\mathbf{k}\in
B.Z}e^{i\mathbf{k}.r_{i}}c_{\mathbf{k}\kappa_{i}s}^{\dagger}$,
$c_{is}=\frac{1}{\sqrt{N}}\sum_{\mathbf{k}\in
B.Z}e^{-i\mathbf{k}.r_{i}}c_{\mathbf{k}\kappa_{i}s}$, we obtain

$H=\sum_{\mathbf{k}}\sum_{\kappa\kappa^{\prime}=\\{\text{A},\text{B}\\}}\sum_{s,s^{\prime}=\\{\uparrow,\downarrow\\}}{c}^{\dagger}_{s\kappa\mathbf{k}}[H_{\text{N}}(\mathbf{k})]_{s\kappa,s^{\prime}\kappa^{\prime}}{c}_{s^{\prime}\kappa^{\prime}\mathbf{k}},$
(S2)

where $\mathbf{k}=(k_{x},k_{y})$, and

$\displaystyle
H_{\text{N}}(\mathbf{k})=t_{1}H^{\text{++}}_{1}(\mathbf{k})\sigma_{x}+t_{1}H^{\text{-+}}_{1^{\prime}}(\mathbf{k})\sigma_{y}+t_{2}H^{\text{++}}_{2}(\mathbf{k})-\mu$
$\displaystyle+\lambda_{SO}H^{\text{+-}}_{3}(\mathbf{k})\mathbf{s}_{z}\sigma_{z}+\lambda_{R}H^{\text{+-}}_{4}(\mathbf{k})s_{x}+\lambda_{R}H^{\text{-+}}_{5}(\mathbf{k})s_{y}+M\sigma_{z}.$
(S3)

In Eq. (S1.1), $\sigma_{\alpha=0,x,y,z}$ are the identity and Pauli matrices
that act on the sublattice degree of freedom. Furthermore, we suppress the
direct product notation $\otimes$ between Pauli matrices, and $2\times 2$
identity matrices, and define
$H^{\text{++}}_{1}(\mathbf{k})=-2\text{cos}\tfrac{k_{x}}{2}\text{cos}\tfrac{\sqrt{3}k_{y}}{2}-\text{cos}k_{x}$,
$H^{\text{-+}}_{1^{\prime}}(\mathbf{k})=2\text{sin}\tfrac{k_{x}}{2}\text{cos}\tfrac{k_{x}}{2}-2\text{sin}\tfrac{k_{x}}{2}\text{cos}\tfrac{\sqrt{3}k_{y}}{2}$,
$H^{\text{++}}_{2}(\mathbf{k})=4\text{cos}\tfrac{3k_{x}}{2}\text{cos}\tfrac{\sqrt{3}k_{y}}{2}+2\text{cos}\sqrt{3}k_{y}$,
$H^{\text{+-}}_{3}(\mathbf{k})=2\text{sin}\sqrt{3}k_{y}-4\text{cos}\tfrac{3k_{x}}{2}\text{sin}\tfrac{\sqrt{3}k_{y}}{2}$,
$H^{\text{+-}}_{4}(\mathbf{k})=-2\text{cos}\tfrac{3k_{x}}{2}\text{sin}\tfrac{\sqrt{3}k_{y}}{2}-2\text{sin}\sqrt{3}k_{y}$,
and
$H^{\text{-+}}_{5}(\mathbf{k})=2\sqrt{3}\text{sin}\frac{3k_{x}}{2}\text{cos}\frac{\sqrt{3}k_{y}}{2}$,
respectively.

Figure S1:  Energy dispersion (normal state) of the honeycomb lattice for
positive energy, when $M\neq 0$ and $\lambda_{\text{SO}}\neq 0$. The
intersection of energy dispersion (with some $\mu$) is shown with blue and red
contours, indicating their spin characteristic. The wiggly green lines show
the instability toward non-FFLO superconductivity between opposite spins.

###  S1.2 pairing potential

The energy dispersion of the Eq. (S1.1) in the presence of both $\mu$ and
$\lambda_{\text{SO}}$ but without $\lambda_{\text{R}}=0$ is shown in Fig. S1.
As we can see, the system has two valleys, and the Fermi contours along them
are spin-polarized, and its polarization flips in different valleys.
Considering the Fermi-contour geometry (red and blue contours in Fig. S1), and
in the absence of Fulde–Ferrell–Larkin–Ovchinnikov (FFLO) superconductivity,
the pairing can occur among electrons with opposite spin and momenta which can
lead to both singlet and triplet pairing instabilities. Furthermore, in the
buckled honeycomb structure, such as silicene, we can tune both $\mu$ and $M$
using a double-gate setup (see Fig. 1(b)), which may enhance or change
superconductivity instability. Indeed it is shown that the triplet f-wave
pairing can be dominated over a singlet d-wave channel in the presence of
large $M$ by polarizing wave function in one of the sublattices [1] (see Fig.
1(c)). The f-wave pairing has $\text{sin}3\theta_{ij}$ characteristic, where
$\theta_{ij}$ is the azimuthal angle from the horizontal direction. Therefore,
the pairing potential between the nearest-neighbor vertices is zero, while for
the next-nearest-neighbors, the sign of pairing flips six times by rotating
around a given site (see Fig. 1(a)). In the main letter, we assume that
$\lambda_{R},\lambda_{SO}\ll M$ which can be satisfied for low buckled
honeycomb structure under a strong electric field, and we expect f-wave
pairing can be dominated even in the presence of small spin-orbit coupling.
Because of $M\neq 0$, the f-wave pairing potential $\Delta_{A}$, $\Delta_{B}$
can be different for A and B sublattices, respectively. The f-wave pairing in
real space can be modeled by

$H_{sc}=\sum_{\expectationvalue{\expectationvalue{i,j}}}\sum_{ss^{\prime}}v_{ij}(\kappa_{i}\Delta_{s}+\Delta_{a})c^{\dagger}_{is}[s_{x}]_{ss^{\prime}}c^{\dagger}_{is^{\prime}},$
(S4)

where we define symmetric $\Delta_{s}=(\Delta_{A}+\Delta_{B})/2$, and
asymmetric $\Delta_{a}=(\Delta_{A}-\Delta_{B})/2$ pairing potential. Note that
$v_{ij}=\kappa_{i}\text{sin}3\theta_{ij}$. We can multiply Eq. (S4) by any
arbitrary phase factor, which although may change the form of the Hamiltonian
and symmetries representation, does not have any physical importance.

###  S1.3 BdG Hamiltonian

By obtaining Fourier transformation of Eq. (S4), $H_{sc}=-i\sum
c^{\dagger}_{s\kappa\mathbf{k}}H^{\text{+-}}_{3}(\mathbf{k})(k)[(\Delta_{s}+\sigma_{z}\Delta_{a})s_{x}]_{s\alpha,s^{\prime}\alpha^{\prime}}c^{\dagger}_{s^{\prime}\kappa\mathbf{k}}$.
We can write the Bogoliubov–de Gennes (BdG) Hamiltonian as

$H_{\text{BdG}}=\sum
a^{\dagger}_{s\kappa\tau\mathbf{k}}[H_{\text{BdG}}(\mathbf{k})]_{s\kappa\tau,s^{\prime}\kappa^{\prime}\tau^{\prime}}a_{s^{\prime}\kappa^{\prime}\tau^{\prime}\mathbf{k}},$
(S5)

where $\tau=\pm 1$ indicate electron and hole in the Nambu space;
$a^{\dagger}_{s\kappa(+1)\mathbf{k}}=c^{\dagger}_{s\kappa\mathbf{k}}$,
$a^{\dagger}_{s\kappa(-1)\mathbf{k}}=c_{s\kappa-\mathbf{k}}$,
$a_{s\kappa(+1)\mathbf{k}}=c_{s\kappa\mathbf{k}}$,
$a_{s\kappa(-1)\mathbf{k}}=c^{\dagger}_{s\kappa-\mathbf{k}}$, and we assume
summation over repeated indices. The $H_{\text{BdG}}(\mathbf{k})$ is given as
follow

$\displaystyle
H_{\text{BdG}}(\mathbf{k})=t_{1}H^{\text{++}}_{1}(\mathbf{k})\sigma_{x}\tau_{z}+t_{1}H^{\text{-+}}_{1^{\prime}}(\mathbf{k})\sigma_{y}\tau_{z}+t_{2}H^{\text{++}}_{2}(\mathbf{k})\tau_{z}$
$\displaystyle+\lambda_{SO}H^{\text{+-}}_{3}(\mathbf{k})s_{z}\sigma_{z}+\lambda_{R}H^{\text{+-}}_{4}(\mathbf{k})s_{x}+\lambda_{R}H^{\text{-+}}_{5}(\mathbf{k})s_{y}\tau_{z}$
$\displaystyle-\mu\tau_{z}+M\sigma_{z}\tau_{z}+H^{\text{+-}}_{3}(\mathbf{k})(\Delta_{s}+\sigma_{z}\Delta_{a})s_{x}\tau_{y},$
(S6)

where $\tau_{0,x,y,z}$ are identity and the Pauli matrices that act on the
electron-hole degrees of freedom.

###  S1.4 Symmetries

In this section, we present the extended symmetries of (S1.3). For simplicity
lets first assume that honeycomb lattice is unbuckled. Equation (S1.3)
generally (in the existence of all terms) has particle-hole symmetry
$\Xi=\tau_{x}\mathcal{K}[\mathbf{k}\rightarrow-\mathbf{k}]$, time-reversal
symmetry $\Theta=s_{y}\tau_{z}\mathcal{K}[\mathbf{k}\rightarrow-\mathbf{k}]$,
and chiral symmetry $\Pi=s_{y}\tau_{y}$, where $\mathcal{K}$ is the complex
conjugate operator.

Moreover, Eq. (S1.3) is invariant under a three-fold rotational Symmetry
$\mathcal{C}_{3z}=e^{i\tfrac{\pi}{3}s_{z}\tau_{z}}[\mathbf{k}\rightarrow\mathcal{R}_{2\pi/3}\mathbf{k}]$,
mirror symmetry
$\mathcal{M}_{y}=s_{y}[{(k_{x},k_{y})}\rightarrow{(k_{x},-k_{y})}]$, and a
two-fold rotational Symmetry
$\mathcal{C}_{2x}=s_{x}\tau_{z}[{(k_{x},k_{y})}\rightarrow{(k_{x},-k_{y})}]$
($\lambda_{R}=0$), in addition to $U(1)$ spin rotation around the out-of-plane
direction, whose generator is given by $\mathcal{S}=s_{z}\tau_{z}$
($\lambda_{R}=0$).

Furthermore, when $\Delta_{a}=M=0$, Eq. (S1.3) has inversion symmetry
$\mathcal{P}=\tau_{z}\sigma_{x}[\mathbf{k}\rightarrow-\mathbf{k}]$
($\lambda_{R}=0$), a mirror symmetry
$\mathcal{M}_{x}=s_{x}\sigma_{x}[{(k_{x},k_{y})}\rightarrow{(-k_{x},k_{y})}]$,
and a six-fold rotational Symmetry
$\mathcal{C}_{6z}=e^{i\tfrac{\pi}{6}s_{z}\tau_{z}}\sigma_{x}\tau_{z}[\mathbf{k}\rightarrow\mathcal{R}_{2\pi/6}\mathbf{k}]$
and two two-fold rotation symmetries
$\mathcal{C}_{2z}=s_{z}\sigma_{x}[\mathbf{k}\rightarrow-\mathbf{k}]$ and
$\mathcal{C}_{2y}=s_{y}\sigma_{x}\tau_{z}[{(k_{x},k_{y})}\rightarrow{(-k_{x},k_{y})}]$
($\lambda_{R}=0$).

When $\mu{=}M{=}t_{2}{=}0$, Eq. (S1.3) has an additional local symmetry
$\mathcal{L}=s_{x}\sigma_{z}\tau_{y}$ ($\lambda_{so}=0$) that commutes with
$\mathcal{S}$.

On the other hand, when $\Delta_{a}=\Delta_{s}=0$, Eq. (S1.3) has two
additional symmetries regarding $U(1)$ gauge symmetry $\mathcal{T}=\tau_{z}$
and spin rotational Symmetry $\tilde{\mathcal{S}}=s_{z}$ ($\lambda_{R}=0$),
where $\mathcal{S}=\tilde{\mathcal{S}}\mathcal{T}$.

Note that considering buckling property, some symmetry operators such as
$\mathcal{M}_{x}$, and $\mathcal{C}_{2x}$ are not true spatial symmetry of the
system. However, they are related to the other true spatial symmetry of the
system by multiplication of $\mathcal{S}$, for instance,
$\mathcal{M}_{x}=-i\mathcal{C}_{2y}\mathcal{S}$, and
$\mathcal{C}_{2x}=i\mathcal{M}_{y}\mathcal{S}$, respectively.

###  S1.5 Basis transformation

In studying the first order topological phase and deriving the edge
Hamiltonian we rewrite Eq. (S1.3) in a basis that $\mathcal{L}$,
$\mathcal{S}$, and $\sigma_{z}$ are diagonal. we use

$U=e^{i\pi
s_{x}\sigma_{z}(-\tfrac{1}{4}+\tfrac{1}{3\sqrt{3}}\tau_{x}+\tfrac{1}{3\sqrt{3}}\tau_{z})+i\pi\tfrac{1}{3\sqrt{3}}\tau_{y}}$
(S7)

and get ($\lambda_{SO}=\lambda_{R}=\Delta_{a}=0$)

$\displaystyle U^{\dagger}H_{\text{BdG}}(\mathbf{k})U=$ $\displaystyle
t_{1}H^{\text{++}}_{1}(\mathbf{k})\sigma_{x}+t_{1}H^{\text{-+}}_{1^{\prime}}(\mathbf{k})\sigma_{y}+\tau_{z}H^{\text{+-}}_{3}(k)\Delta_{s}\sigma_{z}+$
$\displaystyle(t_{2}H^{\text{++}}_{2}(k)-\mu)\tau_{x}+M\sigma_{z}\tau_{x}.$
(S8)

| bulk representation | algebra between symmetry operators: $XX^{\prime}-sX^{\prime}X=0$ | edge representation | algebra
---|---|---|---|---
$X$ | operation | $\mathbf{k}\rightarrow$ | $X^{2}$ | $\Xi$ | $\Theta$ | $\mathcal{L}$ | $\mathcal{S}$ | $\mathcal{M}_{y}$ | $\mathcal{C}_{2x}$ | $\mathcal{P}$ | $\mathcal{M}_{x}$ | $\mathcal{C}_{2z}$ | $\mathcal{C}_{2y}$ | $\mathcal{C}_{3z}$ | $\mathcal{C}_{6z}$ | operation | $\theta\rightarrow$ | $k_{\parallel}\rightarrow$ | $\tau_{z}$ | $\tau_{x}$
$\Xi$ | $is_{x}\tau_{z}\sigma_{z}\mathcal{K}$ | $-\mathbf{k}$ | $+1$ | $+1$ | $+1$ | $+1$ | $-1$ | $-1$ | $-1$ | $-1$ | $+1$ | $+1$ | $+1$ | $+1$ | $-1$ | $s_{x}\tau_{z}\mathcal{K}$ | $\theta$ | $-k_{\parallel}$ | $+1$ | $-1$
$\Theta$ | $s_{y}\tau_{x}\mathcal{K}$ | $-\mathbf{k}$ | $-1$ | $+1$ | $+1$ | $-1$ | $-1$ | $-1$ | $-1$ | $+1$ | $-1$ | $-1$ | $-1$ | $+1$ | $+1$ | $is_{y}\tau_{x}\mathcal{K}$ | $\theta$ | $-k_{\parallel}$ | $-1$ | $+1$
$\mathcal{L}$ | $\tau_{z}$ | $\mathbf{k}$ | $+1$ | $+1$ | $-1$ | $+1$ | $+1$ | $-1$ | $-1$ | $+1$ | $-1$ | $+1$ | $-1$ | $+1$ | $+1$ | $\tau_{z}$ | $\theta$ | $k_{\parallel}$ | $+1$ | $-1$
$\mathcal{S}$ | $s_{z}$ | $\mathbf{k}$ | $+1$ | $-1$ | $-1$ | $+1$ | $+1$ | $-1$ | $-1$ | $+1$ | $-1$ | $+1$ | $-1$ | $+1$ | $+1$ | $s_{z}$ | $\theta$ | $k_{\parallel}$ | $+1$ | $+1$
$\mathcal{M}_{y}$ | $s_{y}\tau_{x}$ | $(k_{x},-k_{y})$ | $+1$ | $-1$ | $-1$ | $-1$ | $-1$ | $+1$ | $-1$ | $+1$ | $-1$ | $-1$ | $+1$ | $\dots$ | $\dots$ | $s_{y}\tau_{x}$ | $-\theta$ | $-k_{\parallel}$ | $-1$ | $+1$
$\mathcal{C}_{2x}$ | $s_{x}\tau_{x}$ | $(k_{x},-k_{y})$ | $+1$ | $-1$ | $-1$ | $-1$ | $-1$ | $-1$ | $+1$ | $+1$ | $+1$ | $-1$ | $-1$ | $\dots$ | $\dots$ | $s_{x}\tau_{x}$ | $-\theta$ | $-k_{\parallel}$ | $-1$ | $+1$
$\mathcal{P}$ | $\sigma_{x}$ | $-\mathbf{k}$ | $+1$ | $-1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $s_{z}\tau_{z}$ | $\theta+\pi$ | $k_{\parallel}$ | $+1$ | $-1$
$\mathcal{M}_{x}$ | $s_{x}\tau_{x}\sigma_{x}$ | $(-k_{x},k_{y})$ | $+1$ | $+1$ | $-1$ | $-1$ | $-1$ | $-1$ | $+1$ | $+1$ | $+1$ | $-1$ | $-1$ | $\dots$ | $\dots$ | $s_{y}\tau_{y}$ | $\pi-\theta$ | $-k_{\parallel}$ | $-1$ | $-1$
$\mathcal{C}_{2z}$ | $s_{z}\sigma_{x}$ | $-\mathbf{k}$ | $+1$ | $+1$ | $-1$ | $+1$ | $+1$ | $-1$ | $-1$ | $+1$ | $-1$ | $+1$ | $-1$ | $+1$ | $+1$ | $\tau_{z}$ | $\theta+\pi$ | $k_{\parallel}$ | $+1$ | $-1$
$\mathcal{C}_{2y}$ | $s_{y}\tau_{x}\sigma_{x}$ | $(-k_{x},k_{y})$ | $+1$ | $+1$ | $-1$ | $-1$ | $-1$ | $+1$ | $-1$ | $+1$ | $-1$ | $-1$ | $+1$ | $\dots$ | $\dots$ | $s_{x}\tau_{y}$ | $\pi-\theta$ | $-k_{\parallel}$ | $-1$ | $-1$
$\mathcal{C}_{3z}$ | $\tfrac{\mathbbm{1}+i\sqrt{3}s_{z}}{2}$ | $\mathcal{R}_{2\pi/3}\mathbf{k}$ | $-\mathcal{C}^{-1}_{3z}$ | $+1$ | $+1$ | $+1$ | $+1$ | $\dots$ | $\dots$ | $+1$ | $\dots$ | $+1$ | $\dots$ | $+1$ | $+1$ | $\tfrac{\mathbbm{1}+i\sqrt{3}s_{z}}{2}$ | $\theta+2\pi/3$ | $k_{\parallel}$ | $+1$ | $+1$
$\mathcal{C}_{6z}$ | $\tfrac{\sqrt{3}\sigma_{x}+is_{z}\sigma_{x}}{2}$ | $\mathcal{R}_{2\pi/6}\mathbf{k}$ | $\mathcal{C}_{3z}$ | $-1$ | $+1$ | $+1$ | $+1$ | $\dots$ | $\dots$ | $+1$ | $\dots$ | $+1$ | $\dots$ | $+1$ | $+1$ | $\tfrac{\sqrt{3}s_{z}\tau_{z}+i\tau_{z}}{2}$ | $\theta+2\pi/6$ | $k_{\parallel}$ | $+1$ | $-1$
Table S1: Symmetry operators (in the bulk representation Eq. (S1.5) and edge
representation Eq. (S9)) and their algebra, where their commutation or
anticommutation $XX^{\prime}-sX^{\prime}X=0$ are shown with $s=\pm 1$,
respectively. For the cases where they do not either commute or anticommuting
we leave their blocks empty. The last two columns show the algebra between
symmetry operators (edge representation) and $\tau_{z}$, $\tau_{x}$.

##  S2 Edge Hamiltonian and symmetry representation

In the main text, we argue that due to the chiral propagating of the edge
modes in opposite directions, the edge-Hamiltonian is given by,

$H_{\text{edge}}(\theta,k_{\parallel})=v_{\theta}\tau_{z}s_{0}k_{\parallel}$
(S9)

We can justify this edge Hamiltonian as follows. First $s_{z}$ is absent from
Eq. (S1.5). Thus, the edge Hamiltonian is independent of $s_{z}$. Second, we
knew that in Eq. (S1.5) if we change the sign of $\Delta_{s}$ (when $\mu=M=0$)
the propagation of chiral modes flips and therefore the edge Hamiltonian has
to proportional to $\tau_{z}$.

In the following, we obtain a representation of the symmetry operators for the
edge Hamiltonian. To derive a representation of symmetry operators we first
note that $\mathcal{L}^{\text{edge}}=\tau_{z}$, and
$\mathcal{S}^{\text{edge}}=s_{z}$. Next, we determine the particle-hole
operator, regarding the edge-Hamiltonian (anticommuting with Hamiltonian and
flip the sign of $k_{\parallel}$), and its relation with
$\mathcal{L}^{\text{edge}}$ and $\mathcal{S}^{\text{edge}}$ (given in Table.
S1). However, the edge Hamiltonian and $\mathcal{L}^{\text{edge}}$ and
$\mathcal{S}^{\text{edge}}$ are invariant under
$\exp(i\phi+i\alpha\tau_{z}+i\beta s_{z}+i\gamma\tau_{z}s_{z})$. Therefore we
have the freedom to choose particle-hole symmetry representation. After fixing
the particle-hole symmetry representation, we can uniquely determine a
representation for time-reversal symmetry by considering its definition
regarding the edge Hamiltonian (commuting with Hamiltonian and flipping the
sign of $k_{\parallel}$), and its relation with $\mathcal{S}^{\text{edge}}$,
$\mathcal{L}^{\text{edge}}$, and $\Xi^{\text{edge}}$. We choose
$\Xi^{\text{edge}}=s_{x}\tau_{z}\mathcal{K}[k_{\parallel}\rightarrow-
k_{\parallel}]$, and obtain
$\Theta^{\text{edge}}=is_{y}\tau_{x}\mathcal{K}[k_{\parallel}\rightarrow-
k_{\parallel}]$.

Next, we determine the representation of other spatial symmetries. Generally,
all of the spatial symmetry operators, have to commute with the edge
Hamiltonian. To obtain them, we should know how reflection, rotations, and
inversion transform $k_{\parallel}$. In Table S1 we show the effect of
symmetry operators on $k_{\parallel}$, which can be easily found by
considering Fig. 3(a) of the main text (we illustrate rotation and reflection
symmetries in Fig. 3(a)). If a given symmetry operator flips (keeps) the sign
of $k_{\parallel}$, it has to anticommuting (commute) with $\tau_{z}$ (we have
tabulated them in Table S1). For continuing, we need to find the algebra or
relation among different symmetry operators from their bulk definition (given
in Table S1) and then construct the edge representation for them by
considering the same algebra. We start with $\mathcal{M}^{\text{edge}}_{y}$.
Considering definition of $\mathcal{M}^{\text{edge}}_{y}$ regarding the edge
Hamiltonian, and its relations with $\mathcal{S}^{\text{edge}}$,
$\mathcal{L}^{\text{edge}}$, $\Xi^{\text{edge}}$, and $\Theta^{\text{edge}}$,
we obtain arbitrary combination of two option. This is because still
$H_{\text{edge}}$, $\mathcal{L}^{\text{edge}}$, $\mathcal{S}^{\text{edge}}$,
$\Xi^{\text{edge}}$, and $\Theta^{\text{edge}}$ are invariant under
$\exp(i\beta s_{z})$. We choose
$\mathcal{M}^{\text{edge}}_{y}=s_{y}\tau_{x}[k_{\parallel}\rightarrow-
k_{\parallel}]$ and the representation of other symmetry operators can be
determined uniquely by doing the same procedure (all given in Table S1).

As we have discussed in the main letter, the only edge mass that respects both
particle-hole and time-reversal operators is $m(\theta)\tau_{x}$. If we assume
that in the bulk, a $\mathcal{L}$-symmetry-breaking term commute with certain
symmetry operations, then similarly its corresponding mass $m(\theta)\tau_{x}$
has to commute with the edge representation of those symmetry operators.
Therefore if the representation of that symmetry operator anticommuting
(commute) with $\tau_{x}$, then $m(\theta)$ should flip (keeps) its sign on
the symmetric partner $m(\theta_{\text{s.p}})=-m(\theta)$
[$m(\theta_{\text{s.p}})=m(\theta)$]. We also tabulate the commutation or
anticommutation of edge representation of symmetry operators with $\tau_{x}$
in Table S1.

Furthermore, under time-reversal and particle-hole symmetry, we can also add
$m^{\prime}\tau_{z}s_{z}$ term to the edge Hamiltonian Eq. (S9). Considering
Eq. (S9) and edge mass and $m^{\prime}\tau_{z}s_{z}$ we can write

$H^{\prime}_{\text{edge}}(\theta,k_{\parallel})=v_{\theta}\tau_{z}s_{0}k_{\parallel}+m\tau_{x}+m^{\prime}\tau_{z}s_{z}$
(S10)

The energy spectrum of Eq. (S10) is given by

$E(k)=\pm\sqrt{(k\pm m^{\prime})^{2}+m^{2}}$ (S11)

The topological phase transition occurs when $k=\pm m^{\prime}$ and $m=0$.
Nonzero values of $m^{\prime}$ alter the BOTS transition in momentum space. It
is noteworthy that, in the presence of $\lambda_{\text{SO}}$, the BOTS
transition does not occur at the time-reversal invariant point (i.e., $k=0$ in
our low energy Dirac theory), as demonstrated in Section S6. This observation
suggests that spin-orbit coupling introduces additional terms in the effective
edge Hamiltonian.

##  S3 Edge Theory

We can find the zero energy modes of Eq. (S1.5) by expanding the Hamiltonian
up to the linear order of $k_{x}$ around $M_{1}$ [2]. To describe the zero
energy edge modes, we substitute $k_{x}\to-i\partial_{x}$, where $x=0$ is the
position of the edge which separates interior region $x<0$ from its exterior
$x>0$, achieving

$\displaystyle\left(-t_{1}\zeta(x)\tfrac{\sigma_{x}+\sqrt{3}\sigma_{y}}{2}-i\partial_{x}(\sqrt{3}\sigma_{x}-\sigma_{y})t_{1}\right)\psi_{s_{z}=\pm
1,\tau_{z}=\pm 1}(x)=0$ (S12)

where, we introduce $\zeta(x)=-\text{sgn}(x)$, $|x|\gg 0$. The following wave
functions satisfy this equation

$\displaystyle\psi_{s_{z}=\pm 1,\tau_{z}=\pm
1}(x){=}\tfrac{1}{\mathcal{N}}e^{\tfrac{1}{2}\int\zeta(x)dx}\ket{s_{z}=\pm
1,\tau_{z}=\pm 1,\sigma_{z}=1}$ (S13)

where $\mathcal{N}$ is the normalization factor. By projecting $\mu$, $M$, and
$t_{2}$ onto $\psi_{s_{z}=\pm 1,\tau_{z}=\pm 1}(x)$ subspace , we obtain
$m_{e4}\equiv m(\theta=0)=-\mu-2t_{2}+M$.

##  S4 Large Sublattice Potential Limit: Effective Hamiltonian

In the large sublattice potential limit, we can decompose the Hamiltonian of
the honeycomb lattice into two triangular lattices. For simplicity, consider
the Hamiltonian of a honeycomb lattice (normal state) with sublattice
potential and nearest-neighbor hopping terms,

$\displaystyle t_{ij}\psi_{j}^{A}-M\psi_{i}^{B}=E\psi_{i}^{B},$ (S14)
$\displaystyle t_{ij}\psi_{j}^{B}+M\psi_{i}^{A}=E\psi_{i}^{A},$ (S15)

where $\psi_{i}^{A}$, and $\psi_{i}^{B}$ are wave function at site $i$ and
sublattice $A$, and $B$ respectively. We want to rewrite Eq. (S14)-(S15) in a
way that A and B sublattice sectors are decomposed. To do this, we can rewrite
Eq. (S14)-(S15) as

$\displaystyle\psi_{i}^{B}=\tfrac{1}{E+M}t_{ij}\psi_{j}^{A},$ (S16)
$\displaystyle\psi_{i}^{A}=\tfrac{1}{E-M}t_{ij}\psi_{j}^{B},$ (S17)

which lead to

$\displaystyle
t_{ij}\tfrac{1}{E-M}t_{jk}\psi_{k}^{B}-M\psi_{i}^{B}=E\psi_{i}^{B},$ (S18)
$\displaystyle
t_{ij}\tfrac{1}{E+M}t_{jk}\psi_{k}^{A}+M\psi_{i}^{A}=E\psi_{i}^{A}.$ (S19)

In the large sublattice limit, we can approximate $E=\mp M$ in the denominator
of Eq. S18, and Eq. S19, respectively,

$\displaystyle
t_{ij}\tfrac{1}{-2M}t_{jk}\psi_{k}^{B}-M\psi_{i}^{B}=E\psi_{i}^{B},$ (S20)
$\displaystyle
t_{ij}\tfrac{1}{+2M}t_{jk}\psi_{k}^{A}+M\psi_{i}^{A}=E\psi_{i}^{A}.$ (S21)

These Hamiltonians describe two triangular lattices with effective onsite and
nearest-neighbor hopping. We can read the effective hopping ($k\neq i$) and
onsite potential ($k=i$) as $t^{2}/(\kappa 2M)$, and $zt^{2}/(\kappa
2M)+\kappa M$, respectively, where $z$ is the coordination number.

##  S5 winding number as a topological invariant for effective Kitaev chain

The topological phase of the effective Kitaev chain also is protected by the
chiral symmetry, when $\lambda_{R}=0$. To see this we rewrite the effective
Kitaev Hamiltonian in the basis where both chiral symmetry $\Pi=s_{y}\tau_{y}$
and $\mathcal{S}=s_{z}\tau_{z}$ are diagonal ($\ket{\Pi=1,\mathcal{S}=1}$,
$\ket{\Pi=1,\mathcal{S}=-1}$, $\ket{\Pi=-1,\mathcal{S}=1}$,
$\ket{\Pi=-1,\mathcal{S}=-1}$), getting a block off-diagonalized matrix

$U^{\dagger}H_{\text{Kitaev}}(k)U=\left(\begin{matrix}0&h(k)\\\
h^{\dagger}(k)&0\end{matrix}\right),$ (S22)

where $U$ is a unitary matrix and

$h(k)=\left(\begin{matrix}-H_{N}(k)-H_{\text{SO}}(k)+iH_{\Delta}(k)&H_{\text{R}}(k)\\\
H_{\text{R}}(k)&-H_{N}(k)+H_{\text{SO}}(k)-iH_{\Delta}(k)\end{matrix}\right).$
(S23)

When $\lambda_{R}=0$, the topological invariant of the effective Kitaev chains
is given by the winding number of each diagonal element of $h(k)$, or
equivalently $\nu_{\mathcal{S}=\pm 1}=\tfrac{1}{2\pi}\int
dkh^{-1}_{\mathcal{S}\mathcal{S}}(k)\partial_{k}h_{\mathcal{S}\mathcal{S}}(k)$.
In the topological phase, the two winding numbers are nonzero but have
opposite signs $\nu_{1}=-\nu_{2}=\pm 1$. After turning on $\lambda_{R}\neq 0$,
we have to calculate the total winding number of whole $h(k)$ or equivalently
$\nu_{t}=\tfrac{1}{2\pi}\int dk\tr(h^{-1}(k)\partial_{k}h(k))$, which already
gives zero for $\lambda_{R}=0$. Therefore in the presence of $\lambda_{R}$,
winding number cannot diagnosis the topological phase.

##  S6 Spin polarized case

In the Dirac half-metal material, one spin component is gapped out, and the
remaining Dirac cones are gapless and spin-polarized. Introducing sublattice
potential gapped out the remaining Dirac cone. Alternatively, we can also
generalize this material to gapped Dirac half metal material, where Dirac
cones in both spin sectors are gapped out with a spin-dependent sublattice
potential $M_{s}\sigma_{z}$, but sublattice potential for one spin sector is
bigger than the other, for instance, $M_{\uparrow}\gg M_{\downarrow}$.
Therefore, the Fermi contours are polarized in one spin sector, and triplet
pairing is the only possible pairing instability. As we have discussed before,
the existence of normal states (edge modes in the absence of
superconductivity) is the key to realizing the BOTS (when superconductivity is
turned on), and we can write down an effective Hamiltonian for the spin-
polarized Kitaev chain

$\displaystyle
H_{\text{S.P.K.C}}(k)=H_{\text{N}}(k)\tau_{z}+H_{\text{SO}}(k)\tau_{0}+H_{\Delta}(k)\tau_{y}$
(S24)

In Eq. (S24) $H_{\text{N}}(k)=-\mu+\kappa
M_{\downarrow}+2t_{2}\cos(k)+F^{\kappa}_{\text{boundary}}(k)$,
$H_{\text{SO}}(k)=2\kappa\lambda_{\text{SO}}\sin(k)$,
$H_{\text{R}}(k)=2\lambda_{\text{R}}\sin(k)$,
$H_{\Delta}(k)=2\Delta_{\kappa}\sin(k)$ and
$F_{\text{boundary}}(k)=\sum_{n=0}^{n_{c}}t_{n}^{\prime\kappa}\cos(nk)$ where
$t_{n}^{\prime\kappa}$ are the effective hopping terms mediated by other (e.g.
bulk) vertices. The particle-hole and time-reversal symmetry are given by
$\Xi=\tau_{x}\mathcal{K}[k\rightarrow-k]$, and
$\Theta=\mathcal{K}[k\rightarrow-k]$, respectively. In the absence of spin
orbit coupling, Eq. (S24) is invariant under a chiral symmetry $\tau_{x}$, and
$\mathcal{C}_{2x}=\tau_{z}[k\to-k]$ symmetry. The chiral symmetry allows us to
define the winding number and similar to the spin-full calculation, it gives
the nontrivial topological phase if $|M|<|\mu+2t|$. On the other hand, $Z_{2}$
topological invariant can be determined by parity information on $k=0,\pi$,
which gives similar results to the winding number calculation. In the presence
of SOC, we cannot use the winding number, but the $Z_{2}$ topological
invariant is well-defined and cannot be changed until the edge Hamiltonian
goes to gap-closing.

##  S7 f-wave spin-triplet pairing and quantum spin hall effect, large spin-
orbit coupling limit

| bulk representation | algebra between symmetry operators: $XX^{\prime}-sX^{\prime}X=0$ | edge representation | algebra
---|---|---|---|---
$X$ | operation | $\mathbf{k}\rightarrow$ | $X^{2}$ | $\Xi$ | $\Theta$ | $\mathcal{T}$ | $\tilde{\mathcal{S}}$ | $\mathcal{M}_{y}$ | $\mathcal{C}_{2x}$ | $\mathcal{P}$ | $\mathcal{M}_{x}$ | $\mathcal{C}_{2z}$ | $\mathcal{C}_{2y}$ | $\mathcal{C}_{3z}$ | $\mathcal{C}_{6z}$ | operation | $\theta\rightarrow$ | $k_{\parallel}\rightarrow$ | $s_{z}$ | $s_{x}\tau_{x}$
$\Xi$ | $-i\tau_{x}\sigma_{z}\mathcal{K}$ | $-\mathbf{k}$ | $+1$ | $+1$ | $+1$ | $-1$ | $+1$ | $-1$ | $-1$ | $-1$ | $+1$ | $+1$ | $+1$ | $+1$ | $-1$ | $\tau_{x}\mathcal{K}$ | $\theta$ | $-k_{\parallel}$ | $+1$ | $+1$
$\Theta$ | $s_{y}\mathcal{K}$ | $-\mathbf{k}$ | $-1$ | $+1$ | $+1$ | $+1$ | $-1$ | $-1$ | $-1$ | $+1$ | $-1$ | $-1$ | $-1$ | $+1$ | $+1$ | $is_{y}\mathcal{K}$ | $\theta$ | $-k_{\parallel}$ | $-1$ | $-1$
$\mathcal{T}$ | $\tau_{z}$ | $\mathbf{k}$ | $+1$ | $-1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $\tau_{z}$ | $\theta$ | $k_{\parallel}$ | $+1$ | $-1$
$\tilde{\mathcal{S}}$ | $s_{z}$ | $\mathbf{k}$ | $+1$ | $+1$ | $-1$ | $+1$ | $+1$ | $-1$ | $-1$ | $+1$ | $-1$ | $+1$ | $-1$ | $+1$ | $+1$ | $s_{z}$ | $\theta$ | $k_{\parallel}$ | $+1$ | $-1$
$\mathcal{M}_{y}$ | $s_{y}$ | $(k_{x},-k_{y})$ | $+1$ | $-1$ | $-1$ | $+1$ | $-1$ | $+1$ | $-1$ | $+1$ | $-1$ | $-1$ | $+1$ | $\dots$ | $\dots$ | $s_{y}$ | $-\theta$ | $-k_{\parallel}$ | $-1$ | $-1$
$\mathcal{C}_{2x}$ | $s_{x}\tau_{z}$ | $(k_{x},-k_{y})$ | $+1$ | $-1$ | $-1$ | $+1$ | $-1$ | $-1$ | $+1$ | $+1$ | $+1$ | $-1$ | $-1$ | $\dots$ | $\dots$ | $s_{x}\tau_{z}$ | $-\theta$ | $-k_{\parallel}$ | $-1$ | $+1$
$\mathcal{P}$ | $\sigma_{x}$ | $-\mathbf{k}$ | $+1$ | $-1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $+1$ | $\tau_{z}$ | $\theta+\pi$ | $k_{\parallel}$ | $+1$ | $-1$
$\mathcal{M}_{x}$ | $s_{x}\tau_{z}\sigma_{x}$ | $(-k_{x},k_{y})$ | $+1$ | $+1$ | $-1$ | $+1$ | $-1$ | $-1$ | $+1$ | $+1$ | $+1$ | $-1$ | $-1$ | $\dots$ | $\dots$ | $s_{x}$ | $\pi-\theta$ | $-k_{\parallel}$ | $-1$ | $+1$
$\mathcal{C}_{2z}$ | $s_{z}\tau_{z}\sigma_{x}$ | $-\mathbf{k}$ | $+1$ | $+1$ | $-1$ | $+1$ | $+1$ | $-1$ | $-1$ | $+1$ | $-1$ | $+1$ | $-1$ | $+1$ | $+1$ | $s_{z}$ | $\theta+\pi$ | $k_{\parallel}$ | $+1$ | $-1$
$\mathcal{C}_{2y}$ | $s_{y}\sigma_{x}$ | $(-k_{x},k_{y})$ | $+1$ | $+1$ | $-1$ | $+1$ | $-1$ | $+1$ | $-1$ | $+1$ | $-1$ | $-1$ | $+1$ | $\dots$ | $\dots$ | $s_{y}\tau_{z}$ | $\pi-\theta$ | $-k_{\parallel}$ | $-1$ | $+1$
$\mathcal{C}_{3z}$ | $\tfrac{\mathbbm{1}+i\sqrt{3}s_{z}\tau_{z}}{2}$ | $\mathcal{R}_{2\pi/3}\mathbf{k}$ | $-\mathcal{C}^{-1}_{3z}$ | $+1$ | $+1$ | $+1$ | $+1$ | $\dots$ | $\dots$ | $+1$ | $\dots$ | $+1$ | $\dots$ | $+1$ | $+1$ | $\tfrac{\mathbbm{1}+i\sqrt{3}s_{z}\tau_{z}}{2}$ | $\theta+2\pi/3$ | $k_{\parallel}$ | $+1$ | $+1$
$\mathcal{C}_{6z}$ | $\tfrac{\sqrt{3}\sigma_{x}+is_{z}\sigma_{x}\tau_{z}}{2}$ | $\mathcal{R}_{2\pi/6}\mathbf{k}$ | $\mathcal{C}_{3z}$ | $-1$ | $+1$ | $+1$ | $+1$ | $\dots$ | $\dots$ | $+1$ | $\dots$ | $+1$ | $\dots$ | $+1$ | $+1$ | $\tfrac{\sqrt{3}\tau_{z}+is_{z}}{2}$ | $\theta+2\pi/6$ | $k_{\parallel}$ | $+1$ | $-1$
Table S2: Symmetry operators (in the bulk representation Eq. (S27) and edge
representation Eq. (S28)) and their algebra, where their commutation or
anticommutation $XX^{\prime}-sX^{\prime}X=0$ are shown with $s=\pm 1$,
respectively. For the cases where they do not either commute or anticommuting
we leave their blocks empty. The last two columns show the algebra between
symmetry operators (edge representation) and $s_{z}$, $s_{x}\tau_{x}$.

In the large $\lambda_{\text{SO}}>M$ limit, the normal state of the system has
a nontrivial topological phase, which is known as the quantum spin hall effect
[3]. In the following, we derive the edge Hamiltonian using symmetry analysis.
When ($\mu=t_{2}=M=\lambda_{R}=\Delta_{s}=\Delta_{a}=0$), Eq. (S1.3) is
reduced to

$H_{\text{BdG}}(\mathbf{k})=t_{1}H^{\text{++}}_{1}(\mathbf{k})\sigma_{x}\tau_{z}+t_{1}H^{\text{-+}}_{1^{\prime}}(\mathbf{k})\sigma_{y}\tau_{z}+\lambda_{SO}H^{\text{+-}}_{3}(\mathbf{k})\mathbf{s}_{z}\sigma_{z}$
(S25)

This Hamiltonian is invariant under $\Xi$, $\Theta$, $\mathcal{C}_{6z}$,
$\mathcal{C}_{2z}$, $\mathcal{C}_{2y}$, $\mathcal{M}_{x}$, $\mathcal{P}$,
$\mathcal{C}_{3z}$, $\mathcal{C}_{2x}$, $\mathcal{M}_{y}$, $\mathcal{S}$,
$\mathcal{T}$, and $\tilde{\mathcal{S}}$. In this section for simplicity, we
assume $\lambda_{R}=0$.

It is helpful to rewrite Eq. (S25) in a new basis using a unitary
transformation

$U=\exp^{\tfrac{1}{4}i\pi(\sigma_{z}-\sigma_{z}\tau_{z})},$ (S26)

which leads to

$\displaystyle
U^{\dagger}H_{\text{BdG}}(\mathbf{k})U=t_{1}H^{\text{++}}_{1}(\mathbf{k})\sigma_{x}+t_{1}H^{\text{-+}}_{1^{\prime}}(\mathbf{k})\sigma_{y}+\lambda_{\text{SO}}H^{\text{+-}}_{3}(k)\sigma_{z}s_{z}.$
(S27)

The Eq. (S27) describes two copies ( $\tau_{z}=\pm 1$) of the celebrated Kane-
Mele model, and hosts two chiral boundary modes propagating in opposite
directions. Therefore the edge Hamiltonian can be described by

$H^{0}_{\text{edge}}(\theta,k_{\parallel})=v_{\theta}k_{\parallel}s_{z}\tau_{0}.$
(S28)

We obtain the symmetries representation of Eq. (S28), using the same procedure
that we have done in Sec. S2, where we tabulate them in Table. S2.

In the following, we try to find a general edge Hamiltonian regarding nonzero
$\Delta_{s}$, $\mu$, $M$, and $\Delta_{a}$. Each of these terms has a certain
commutation or anticommutation relation with symmetry operators and we assume
that the main effect of them also respects those relations in the edges
Hamiltonian.

The $\mu$, $t_{2}$ and $M$ respect all of the local symmetries ($\Xi$,
$\Theta$, $\mathcal{T}$, $\tilde{\mathcal{S}}$, and $\mathcal{S}$). The only
$k_{\parallel}$ independent term that can satisfy these symmetries is

$W_{\alpha}(\theta)\tau_{z},$ (S29)

where $\alpha=\mu,t_{2},M$. These terms, are symmetric under
$\mathcal{M}_{y}$, $\mathcal{C}_{2x}$ and $\mathcal{C}_{3z}$. Accordingly and
using Table. S2,
$W_{\alpha}(\theta)=W_{\alpha}(-\theta)=W_{\alpha}(\theta+2\pi/3)$. However,
although $\mu$, and $t_{2}$ are symmetric under $\mathcal{M}_{x}$,
$\mathcal{C}_{2y}$, $\mathcal{P}$, $\mathcal{C}_{2z}$ and $\mathcal{C}_{6z}$
(commute with them), but $M$ not (anticommute with them). Thus, $\mu$ and
$t_{2}$ leads to (where $\alpha=\mu,t_{2}$)

$W_{\alpha}(\theta)=W_{\alpha}(\pi-\theta)=W_{\alpha}(\pi+\theta)=W_{\alpha}(\theta+2\pi/6),$
(S30)

while $M$ leads to

$W_{M}(\theta)=-W_{M}(\pi-\theta)=-W_{M}(\pi+\theta)=-W_{M}(\theta+2\pi/6).$
(S31)

This means that in the $\mathcal{M}_{x}$ symmetric edges (armchair edges)
$W_{M}(\theta)=0$. Note that $W_{\mu,t_{2},M}(\theta)\tau_{z}$, does not
commute with Hamiltonian and cannot gap out the edge Hamiltonian (see Fig.
S2(a1), and Fig. S2(b1)).

As we discussed before, nonzero $\Delta_{s}\neq 0$ (or $\Delta_{a}\neq 0$)
breaks $\mathcal{T}$, and $\tilde{\mathcal{S}}$, but still system is invariant
under $\mathcal{S}$. If we assume three remaining local symmetries $\Xi$,
$\Theta$, and $\mathcal{S}$, we can add the following terms to the edge
Hamiltonian, that break $\tilde{\mathcal{S}}$, and $\mathcal{T}$

$\delta_{\alpha}(\theta)s_{y}\tau_{y}+\Delta_{\alpha}^{\text{edge}}(\theta)k_{\parallel}s_{x}\tau_{x},\qquad\alpha=\Delta_{s},\Delta_{a}.$
(S32)

These terms anticommuting with Eq. (S28) and can gap it out (Fig. S2(a2)).
Note that even with nonzero $\Delta_{s}$ or $\Delta_{a}$, the system is
invariant under $\mathcal{M}_{y}$, $\mathcal{C}_{2x}$ and $\mathcal{C}_{3z}$.
Using Table. S2 we obtain

$\delta_{\alpha}(\theta)=\delta_{\alpha}(-\theta)=\delta_{\alpha}(\theta+2\pi/3),$
(S33)

and

$\Delta_{\alpha}^{\text{edge}}(\theta)=\Delta_{\alpha}^{\text{edge}}(-\theta)=\Delta_{\alpha}^{\text{edge}}(\theta+2\pi/3).$
(S34)

Although $\Delta_{s}$ is symmetric (commute with them) under
$\mathcal{M}_{x}$, $\mathcal{C}_{2y}$, $\mathcal{P}$, $\mathcal{C}_{2z}$ and
$\mathcal{C}_{6z}$, but $\Delta_{a}$ are not (anticommute with them). Using
Table. S2, we obtain

$\delta_{\Delta_{s}}(\theta)=-\delta_{\Delta_{s}}(\pi+\theta)=-\delta_{\Delta_{s}}(\pi-\theta),$
(S35)
$\Delta^{\text{edge}}_{\Delta_{s}}(\theta)=-\Delta^{\text{edge}}_{\Delta_{s}}(\pi+\theta)=-\Delta^{\text{edge}}_{\Delta_{s}}(\pi-\theta),$
(S36)
$\delta_{\Delta_{a}}(\theta)=\delta_{\Delta_{a}}(\pi+\theta)=\delta_{\Delta_{a}}(\pi-\theta),$
(S37)

and

$\Delta^{\text{edge}}_{\Delta_{a}}(\theta)=\Delta^{\text{edge}}_{\Delta_{a}}(\pi+\theta)=\Delta^{\text{edge}}_{\Delta_{a}}(\pi-\theta).$
(S38)

This means that at the $\mathcal{M}_{x}$ symmetric edges (armchair edges)
$\delta_{\Delta_{s}}(\theta)=\Delta^{\text{edge}}_{\Delta_{s}}(\theta)=0$.

We can write general edge Hamiltonian as follows

$H_{\text{edge}}(\theta,k_{\parallel})=v_{\theta}k_{\parallel}s_{z}+W(\theta)\tau_{z}+\delta(\theta)s_{y}\tau_{y}+\Delta^{\text{edge}}(\theta)k_{\parallel}s_{x}\tau_{x},$
(S39)

where

$\displaystyle
W(\theta)=W_{\mu}(\theta)+W_{t_{2}}(\theta)+W_{M}(\theta)+W_{\text{other
corrections}}(\theta),$ (S40)
$\displaystyle\delta(\theta)=\delta_{\Delta_{s}}(\theta)+\delta_{\Delta_{a}}(\theta)+\delta_{\text{other
corrections}}(\theta),$ (S41)
$\displaystyle\Delta^{\text{edge}}(\theta)=\Delta^{\text{edge}}_{\Delta_{s}}(\theta)+\Delta^{\text{edge}}_{\Delta_{a}}(\theta)+\Delta^{\text{edge}}_{\text{other
corrections}}(\theta).$ (S42)

As we have discussed before, $\mathcal{M}_{y}$, $\mathcal{C}_{2x}$ and
$\mathcal{C}_{3z}$ gives

$\displaystyle W(\theta)=W(-\theta)=W(\theta+2\pi/3),$ (S43)
$\displaystyle\delta(\theta)=\delta(-\theta)=\delta(\theta+2\pi/3),$ (S44)
$\displaystyle\Delta^{\text{edge}}(\theta)=\Delta^{\text{edge}}(-\theta)=\Delta^{\text{edge}}(\theta+2\pi/3).$
(S45)

However, when $M=\Delta_{a}=0$ then $\mathcal{M}_{x}$, $\mathcal{C}_{2y}$,
$\mathcal{C}_{2z}$, $\mathcal{P}$ and $\mathcal{C}_{6z}$ gives

$\displaystyle W(\theta)=W(\pi-\theta)=W(\pi+\theta)=W(\theta+2\pi/6),$ (S46)
$\displaystyle\delta(\theta)=-\delta(\pi-\theta)=-\delta(\pi+\theta)=-\delta(\theta+2\pi/6),$
(S47)
$\displaystyle\Delta^{\text{edge}}(\theta){=}-\Delta^{\text{edge}}(\pi{-}\theta){=}-\Delta^{\text{edge}}(\pi{+}\theta)=-\Delta^{\text{edge}}(\theta{+}\frac{2\pi}{6}).$
(S48)

In the following, we discuss different topological phases of the system by
studying the gapless state of the general edge Hamiltonian Eq. (S39) that is
forced by symmetries relations. We first discuss the case of
$\mu=t_{2}=M=\Delta_{a}=0$, then we turn on them and discuss possible
topological phases.

When $\mu=t_{2}=M=0$ (but nonzero $\Delta_{s}$ and $\Delta_{a}$), the bulk
Hamiltonian is invariant under two other local symmetries
$\mathcal{O}_{1}=\tau_{x}\sigma_{z}s_{z}$, and
$\mathcal{O}_{2}=\tau_{y}\sigma_{z}$ (in the representation of Eq. (S1.3)). We
obtain their edge Hamiltonian representation as
$\mathcal{O}^{\text{edge}}_{1}=\tau_{x}$ and
$\mathcal{O}^{\text{edge}}_{2}=s_{z}\tau_{y}$, by using their relationships
(commutation or anticommutation) with other local symmetries $\Xi$, $\Theta$,
etc. Note that $\mathcal{O}^{\text{edge}}_{1}$ and
$\mathcal{O}^{\text{edge}}_{2}$ force $W(\theta)=\delta(\theta)=0$ in Eq.
(S39). Consequently, Eq. (S39) remains gapless and f-wave pairing potential
can not gap out the normal state, and the system can be considered as a first-
order topological phase, which indicates gapless states exist along all
boundaries. This phase is connected ($\Delta_{a}=0$, $\lambda_{R}\rightarrow
0$) to the first-order topological state that we discuss in the main letter.

Figure S2:  Energy dispersion (superconducting states, or BdG Hamiltonian) of
a strip geometry of the honeycomb lattice with f-wave spin-triplet pairing and
spin-orbit coupling.

If we turn on $\mu\neq 0$ or $t_{2}\neq 0$, but keep $M=\Delta_{a}=0$, a
nonzero $W(\theta)$, $\delta(\theta)$, and $\Delta^{\text{edge}}(\theta)$ can
be induced in the edge Hamiltonian. However, symmetry constraints (type-1
symmetry group) force sign changing for
$\Delta^{\text{edge}}(\theta_{s.p})=-\Delta(\theta)$, and
$\delta(\theta_{s.p})=-\delta^{\text{edge}}(\theta)$, which means that they
vanish at the symmetric edges (armchair edges). Therefore, the normal modes
remain gapless, and f-wave spin-triplet pairing does not gap them out in the
$\mathcal{M}_{x}$ symmetric edges (compare Fig. S2(a2), and Fig. S2(b2)). We
can call this system, a crystalline topological superconductor. Furthermore,
due to the sign changing of $\delta(\theta)$, and
$\Delta^{\text{edge}}(\theta)$, this system becomes a HOTS in the absence of
symmetric edges[4].

By introducing $M\neq 0$, or $\Delta_{a}\neq 0$, type-1 symmetries will be
broken, and there is no constraint that force gapless states for Eq. (S39).
Therefore, the system is not any more HOTS, and crystalline topological
superconductivity breaks down. Consequently, gapless edge modes along
$\mathcal{M}_{x}$ symmetric edges are gaped out (see Fig. S2(b3)). However,
the system still can obtain a nontrivial BOTS phase, where the topological
phase transition from the nontrivial phase to the trivial phase mediated by
gap closing at a $\mathcal{M}_{y}$ symmetric edge. The energy spectrum of Eq.
(S39) is zero (gapless) if
$W(\theta)=-v(\theta)\tfrac{\delta(\theta)}{\Delta^{\text{edge}}(\theta)}$,
and $k_{\parallel}=\tfrac{\delta(\theta)}{\Delta^{\text{edge}}(\theta)}$. Now
consider two $\mathcal{M}_{y}$ symmetric edges that are located at
$\theta=0,\pi$. Doing the same procedure that we have done in the main letter,
we can find $W(0)\approx-(\mu+2t_{2})+M$, while
$W(\pi)\approx-(\mu+2t_{2})-M$. Therefore, in the absence of
$\delta(0,\pi)=0$, the BOTS phase transition is given by $|M|=|\mu+2t_{2}|$,
which is the same as that we obtain in other limits. This BOTS phase
transition is not dependent on spin-orbit coupling strength, so reducing
$\lambda_{so}$ does not change it. Finally, a trivial insulator is deriving
from the topological quantum spin hall phase by reducing $\lambda_{so}$.
Though in this process the normal state undergoes a bulk gap closing, in the
superconducting phase this process is always gapped. This phase is thus
connected to the large sublattice potential limit that we discussed in the
main letter.

In Fig. S2(a3) we plot the BOTS phase transition, which appears for $k\neq
0,\pi$ (corresponding to $k=0$ in our low energy Dirac theory), indicates
nonzero $\delta(\theta)$. Note that $\delta(\theta)$ breaks $\mathcal{T}$
symmetry and is a superconducting pairing potential. Because
$\delta(\theta)s_{y}\tau_{y}$ is $k_{\parallel}$ independent, we can interpret
it as an effective spin-singlet pairing potential. We confirm that in the
absence of $\lambda_{so}$ the BOTS transition happens in $k=0$, indicating
$\delta(\theta)$ is emerging from a combination of spin-orbit coupling, f-wave
spin-triplet pairing, and sublattice potential processes.

##  S8 f-wave spin-triplet pairing and valley quantum hall effect

Figure S3:  The f-wave spin-triplet pairing and valley quantum hall effect.
(a) The double gate setup with an opposite electric field on the opposite
halves of the system. (b), (d) The energy dispersion (normal state) of a
zigzag strip geometry with an opposite sublattice mass $|M|=0.3t_{1}$
($t_{1}=1$) on the two sides of the system. (c), (e) Majorana zero-energy
modes (red spots), when we turn on f-wave spin-triplet pairing
$\Delta_{s}=0.1t_{1}$. We set (c) $\mu=-0.4$, (e) $\mu=0.4$ which is shown by
green dashed lines in (b), and (d), respectively. The number of Majorana zero
modes in (c) and (e) amount to 12, and 4, respectively.

Boundary modes in the honeycomb structures can be achieved by applying
opposite sublattice potential on the two sides of the system. This can be done
by applying an opposite electric field on two opposite sides of a buckled
honeycomb lattice (Fig. S3(a)), in which a boundary mode appears in the middle
of them (Fig. S3(b)). Actually, this boundary mode is related to the edge mode
of a topological phase with a nonzero valley Chern number (equal to one)
called the valley quantum hall effect and is protected by the absence of
scattering between two valleys [5]. One can imagine this boundary mode as a 1D
chain that lives in the middle of the system. We show in Fig. S3(c), and Fig.
S3(e) that the Majorana zero-energy modes appear (when $\Delta_{s}\neq 0$) at
the ends of this effective 1D (Kitaev) chain. The corner modes in Fig. S3(c),
are related to the normal modes that appear at the zigzag edges (two effective
Kitaev chain that lives on the two sides of the system). These normal modes
obtain the same energy due to opposite sublattice potential on the two sides
of the system (see degenerate blue and red edge states of the energy spectrum
in Fig. S3(b)).

##  S9 topological invariant for the strip of the honeycomb lattice

Figure S4:  The BOTS transition in a strip geometry where (a) two edges are
zigzag, and (b) one is zigzag and the other is dangling. We plot energy
dispersion (normal state), the edge resolved winding number, and the $Z_{2}$
topological invariant in the left, center, and right panels. In this
calculation, we set $M=0.3t_{1}$, $\Delta_{s}=0.1t_{1}$, and other parameters
to zero. We also show the energy gap (in the superconducting state) in the
right panels.

We can generalize the $Z_{2}$ topological invariant or winding number to
calculate the topological states of the strip geometry. To confirm the Kitaev
chain picture, we define edge resolved winding-number $\nu_{\mathcal{S}=\pm
1}=\nu^{\text{L}}_{\mathcal{S}}+\nu^{\text{R}}_{\mathcal{S}}$, where
$\nu^{L,R}_{\mathcal{S}}=\tfrac{1}{2\pi}\int
dk\text{Tr}^{L,R}(\tilde{h}^{-1}_{\mathcal{S}}(k)\partial_{k}\tilde{h}_{\mathcal{S}}(k))$,
$\tilde{h}_{\mathcal{S}}$ is the off-diagonal part of the BdG Hamiltonian in
the basis of chiral symmetry and $\mathcal{S}$. The $\text{Tr}^{L,R}$ is the
partial trace that only sum over indices that belongs to the left (L) or right
(R) part of the system (two opposite half of the strip geometry, see Fig.
1(a)) [6]. Note that although $\nu_{\mathcal{S}}$ is a quantize number, but
$\nu^{R,L}_{\mathcal{S}}$ are not. However, the jumping of
$\nu^{R,L}_{\mathcal{S}}$ across the topological phase transition is a good
signature for topological phase transition [6]. Furthermore, we numerically
calculate the $Z_{2}$ topological invariant for the strip geometry (see Ref. 7
for the procedure). In Fig. S4(a1)-(a3) we compare the $Z_{2}$, winding number
and energy gap for a system with zigzag edges, which confirms the
applicability of these topological invariants for diagnosing topological
phases.

##  S10 BOTS at dangling edge

The fact that the existence of a nontrivial topological phase with corner
modes depends on the normal state condition, leads us to design different edge
terminations and therefore manipulate corner modes. For instance, suppose in
designing strip geometry, we instead terminate two edges with the same
sublattice atoms (for instance remove the red vertices on the right side of
Fig. 1(a)). In this case, one edge is zigzag and the other is dangling. The
coordination number on the dangling edge is $z=1$ and we expect normal modes
for them. However, its energy is the same as the zigzag normal modes (compare
Fig. S4(a1), Fig. S4(b1)). Therefore, the two edges obtain the same
topological state simultaneously, where the jump of edge-resolved winding
number shows topological phase transition for both edges (see Fig. S4(b2)).
Note that, the $Z_{2}$ and $\nu_{\mathcal{S}}$ cannot diagnose this
topological phase transition [see Fig. S4(b3)]. Remarkably, because of two
nontrivial Kitaev chains on both sides of the system, there is no guarantee
that the topological phase transition happen at a symmetric edge.

## References

  * Zhang  _et al._ [2015] L.-D. Zhang, F. Yang, and Y. Yao, Possible electric-field-induced superconducting states in doped silicene, Scientific Reports 5, 8203 (2015).
  * Winkler and Deshpande [2017] R. Winkler and H. Deshpande, Effective hamiltonian for protected edge states in graphene, Phys. Rev. B 95, 235312 (2017).
  * Kane and Mele [2005] C. L. Kane and E. J. Mele, ${Z}_{2}$ topological order and the quantum spin hall effect, Phys. Rev. Lett. 95, 146802 (2005).
  * Scammell  _et al._ [2022] H. D. Scammell, J. Ingham, M. Geier, and T. Li, Intrinsic first- and higher-order topological superconductivity in a doped topological insulator, Phys. Rev. B 105, 195149 (2022).
  * Kim  _et al._ [2014] Y. Kim, K. Choi, J. Ihm, and H. Jin, Topological domain walls and quantum valley hall effects in silicene, Phys. Rev. B 89, 085429 (2014).
  * Wu  _et al._ [2020] X. Wu, W. A. Benalcazar, Y. Li, R. Thomale, C.-X. Liu, and J. Hu, Boundary-obstructed topological high-${\mathit{t}}_{c}$ superconductivity in iron pnictides, Phys. Rev. X 10, 041014 (2020).
  * Budich and Ardonne [2013] J. C. Budich and E. Ardonne, Topological invariant for generic one-dimensional time-reversal-symmetric superconductors in class diii, Phys. Rev. B 88, 134523 (2013).