# complex hyperbolic triangle groups with 2-fold symmetry

John R. Parker and Li-jie Sun  University of Durham
South Road
Durham DH1 3LE UK<EMAIL_ADDRESS>Tokyo Institute of Technology
2-12-1 O-okayama Meguro-ku
Tokyo 152-8552 Japan<EMAIL_ADDRESS>
###### Abstract.

In this paper we will consider the 2-fold symmetric complex hyperbolic
triangle groups generated by three complex reflections through angle $2\pi/p$
with $p\geqslant 2$. We will mainly concentrate on the groups where some
elements are elliptic of finite order. Then we will classify all such groups
which are candidates for being discrete. There are only 4 types.

MSC2000: 20H10, 22E40, 51M10

Keywords: complex hyperbolic triangle groups

##  1\.  Introduction

A complex hyperbolic triangle is a triple $(C_{1},C_{2},C_{3})$ of complex
geodesics in ${\textbf{H}_{\mathbb{C}}^{2}}$. If each pair of complex
geodesics intersects in
${\textbf{H}_{\mathbb{C}}^{2}}\cup\partial{\textbf{H}_{\mathbb{C}}^{2}}$ and
the angles between $C_{k-1}$ and $C_{k}$ for $k=1,2,3$ (the indices are taken
mod 3) are $\pi/p_{1},\pi/p_{2},\pi/p_{3}$, where
$p_{1},p_{2},p_{3}\in\mathbb{N}\cup\\{\infty\\}$, we call the triangle
$(C_{1},C_{2},C_{3})$ a $(p_{1},p_{2},p_{3})$-$triangle$. The intersection
points of pairs of complex geodesics are called the vertices of the complex
hyperbolic triangle. A group $\Gamma$ is called a
$(p_{1},p_{2},p_{3})-$_triangle group_ , if $\Gamma$ is generated by three
complex reflections $R_{1},R_{2},R_{3}$ fixing sides $C_{1},C_{2},C_{3}$ of
$(p_{1},p_{2},p_{3})$-$triangle$. Note that a complex reflection may have
order greater than $2$. In what follows we suppose that $R_{1}$, $R_{2}$ and
$R_{3}$ all have order $p\in{\mathbb{Z}}$ with $p\geqslant 2$.

Any two real hyperbolic triangle groups with the same intersection angles are
conjugate in $\textrm{Isom}^{+}(\textbf{H}^{3})$, which is the orientation
preserving isometry group of 3-dimensional hyperbolic space. If we consider
the groups in $\textrm{PU}(2,1)=\textrm{Aut}({\textbf{H}_{\mathbb{C}}^{2}})$,
we will get the nontrivial deformations. The deformation theory of complex
hyperbolic triangle groups was begun in [3] in which they investigated
$\Gamma$ of type $(\infty,\infty,\infty)$ with $p=2$ (complex hyperbolic ideal
triangle groups). Since then, there have been many developments referring to
other types, such as [13, 5, 11] among which they mainly gave the necessary
conditions of $\Gamma$ to be discrete. Especially Parker and Paupert in [9]
and [10] investigated the equilateral triangle group generated by three
complex reflections with finite order. These include _Deraux’s lattice,
Livné’s lattices, Mostow’s lattices_. Our starting point is a result given by
Thompson [12] where he investigated the non-equilateral triangle groups
generated by three complex involutions (that is the order of the reflections
is $p=2$). He obtained his result using a computer search. Using [10] we see
that Thompson’s results apply to groups with $p>2$ as well. In what follows we
will give the specific case about the triangles group with 2-fold symmetry and
we give a rigorous proof.

We will restrict to the complex hyperbolic triangle groups generated by three
complex reflections with finite order $p\geq 2$. Suppose that the polar vector
of a complex geodesic $C_{1}$ is $\bf v_{1}$ (see Section 2 for a more precise
explanation). We consider the complex reflection $R_{1}$ in the complex
geodesic $C_{1}$. This map sends $\bf v_{1}$ to $e^{i\phi}\bf v_{1}$ and acts
as the identity on the orthogonal complement of ${\bf v}_{1}$, that is on
vectors that project to $C_{1}$. We will always restrict to the case where
$\phi=2\pi/p$ and so $R_{1}$ has order $p\geqslant 2$.

(1.1) $R_{1}({\bf z})={\bf z}+(e^{i\phi}-1)\frac{\langle{\bf z},{\bf
v_{1}}\rangle}{\langle{\bf v_{1}},{\bf v_{1}}\rangle}{\bf v_{1}}.$

In order to convert $R_{1}$ into a matrix with determinant 1, we need to
multiply the expression in (1.1) by $e^{-i\phi/3}$. The ambiguity involved in
this choice is precisely the ambiguity involved in lifting an isometry in
$\textrm{PU}(2,1)$ to a matrix in $\textrm{SU}(2,1)$.

Here we recall the terminology for _braid relations_ between group elements
(see Section 2.2 of Mostow [7]). Let $G$ be a group and $a,\,b\in G$. Then $a$
and $b$ satisfy a braid relation of length $l\in\mathbb{Z}_{+}$ if

$(ab)^{l/2}=(ba)^{l/2},$

where powers means that the corresponding alternating product of $a$ and $b$
should have $l$ factors. For example, $(ab)^{3/2}=aba$, $(ba)^{2}=baba$. We
denote the _braid length_ $l$ by $br(a,b)$ to be the minimum length of a braid
relation satisfied by $a$ and $b$.

We define the $(l_{1},l_{2},l_{3};l_{4})$-triangle groups to be the triangle
groups with the following braid relations:

$\begin{array}[]{ll}br(R_{2},R_{3})=l_{1},&br(R_{1},R_{3})=l_{2},\\\
br(R_{1},R_{2})=l_{3},&br(R_{1},R_{3}^{-1}R_{2}R_{3})=l_{4},\end{array}$

where $R_{j}$ is of order $p$.

In this paper we aim to list the candidates of discrete triangle groups
generated by $R_{1},R_{2},R_{3}$ with $l_{1}=l_{2}$ and $l_{3}=l_{4}$ as
stated in Theorem 2.4.

##  2\. The parameter space, traces and main result

Firstly we recall some fundamentals about complex hyperbolic 2-space. Please
refer to [2, 8] for more details about the complex hyperbolic space. Let
$\mathbb{C}^{2,1}$ denote the vector space $\mathbb{C}^{3}$ equipped with the
Hermitian form

$\langle{\bf z},{\bf
w}\rangle=z_{1}\overline{w}_{1}+z_{2}\overline{w}_{2}-z_{3}\overline{w}_{3}$

of signature (2,1), where ${\bf z}=[z_{1},z_{2},z_{3}]^{t}$ and ${\bf
w}=[w_{1},w_{2},w_{3}]^{t}.$ The Hermitian form divides $\mathbb{C}^{2,1}$
into three parts $V_{-},V_{0}$ and $V_{+}$, which are

$\displaystyle V_{-}=\\{{\bf z}\in\mathbb{C}^{2,1}|\langle{\bf z},{\bf
z}\rangle<0\\},$ $\displaystyle V_{0}=\\{{\bf
z}\in\mathbb{C}^{2,1}|\langle{\bf z},{\bf z}\rangle=0\\},$ $\displaystyle
V_{+}=\\{{\bf z}\in\mathbb{C}^{2,1}|\langle{\bf z},{\bf z}\rangle>0\\}.$

We denote by $\mathbb{CP}^{2}$ the complex projectivisation of
$\mathbb{C}^{2,1}$ and by $\mathbb{P}:$
$\mathbb{C}^{2,1}\setminus\\{0\\}\rightarrow\mathbb{CP}^{2}$ the natural
projectivisation map. The _complex hyperbolic 2-space_
${\textbf{H}_{\mathbb{C}}^{2}}$ is defined as $\mathbb{P}(V_{-})$. It is
called the standard projective model of the complex hyperbolic space.
Correspondingly the boundary of ${\textbf{H}_{\mathbb{C}}^{2}}$ is
$\partial{\textbf{H}_{\mathbb{C}}^{2}}=\mathbb{P}({V_{0}\setminus\\{0\\}})$.
One can also consider the _unit ball model_ whose boundary is the sphere
$\mathbb{S}^{3}$ by taking $z_{3}=1,$ which can be simply written as
$\\{(z_{1},z_{2})\in\mathbb{C}^{2}:|z_{1}|^{2}+|z_{2}|^{2}<1\\}.$

The complex hyperbolic plane ${\textbf{H}_{\mathbb{C}}^{2}}$ is a
$K\ddot{a}hler$ manifold of constant holomorphic sectional curvature $-1$. The
holomorphic automorphism group of ${\textbf{H}_{\mathbb{C}}^{2}}$ is the
projectivisation $\textrm{PU}(2,1)$ of the group $\textrm{U}(2,1)$ of complex
linear transformations on $\mathbb{C}^{2,1}$, which preserve the Hermitian
form. Especially $\textrm{SU}(2,1)$ is the subgroup of $\textrm{U}(2,1)$ with
the determinant of each element being 1.

Let $x,y\in{\textbf{H}_{\mathbb{C}}^{2}}$ be points corresponding to vectors
${\bf x},{\bf y}\in\mathbb{C}^{2,1}\setminus\\{0\\}$. Then the Bergman metric
$\rho$ on ${\textbf{H}_{\mathbb{C}}^{2}}$ is given by

$\cosh^{2}\Big{(}\frac{\rho({x},{y})}{2}\Big{)}=\frac{\langle{\bf x},{\bf
y}\rangle\langle{\bf y},{\bf x}\rangle}{\langle{\bf x},{\bf
x}\rangle\langle{\bf y},{\bf y}\rangle},$

where ${\bf x}$, ${\bf y}{\in V_{-}}$ are the lifts of $x,$ $y$ respectively.
It is easy to check that this definition is independent of the choice of
lifts.

Given two points $x$ and $y$ in
${\textbf{H}_{\mathbb{C}}^{2}}\cup\partial{\textbf{H}_{\mathbb{C}}^{2}}$, with
lifts ${\bf x}$ and ${\bf y}$ to $\mathbb{C}^{2,1}$ respectively, the complex
span of ${\bf x}$ and ${\bf y}$ projects to a _complex line_ in
$\mathbb{CP}^{2}$ passing through $x$ and $y$. The intersection of a complex
line with ${\textbf{H}_{\mathbb{C}}^{2}}$ will be called a _complex geodesic_
$C$, which can be uniquely determined by a positive vector ${\bf v}\in V_{+},$
i.e. $C=\mathbb{P}(\\{{\bf z}\in\mathbb{C}^{2,1}\setminus\\{0\\}|\langle{\bf
z},{\bf v}\rangle=0\\})$. We call ${\bf v}$ a _polar vector_ to $C.$ As stated
in Section 1, we will consider $(l_{1},l_{2},l_{3};l_{4})$-triangle groups
$\Gamma$ generated by three complex reflections (see (1.1)) through angle
$\phi$ in three complex geodesics.

Throughout this paper, we assume that $R_{1},\,R_{2},\,R_{3}$ are three
complex reflections in complex geodesics $C_{1},\,C_{2},\,C_{3}$ respectively.
We parameterize the triangle groups generated by $R_{1},\,R_{2},\,R_{3}$ by
three complex numbers $\rho$, $\sigma$ and $\tau$. Up to the action of ${\rm
PU}(2,1)$, we can parameterize the collection of three pairwise distinct
complex lines in ${\bf H}^{2}_{\mathbb{C}}$ by four real parameters. The
parameters we choose are $|\rho|$, $|\sigma|$, $|\tau|$ and ${\rm
arg}(\rho\sigma\tau)$. In particular, we can freely choose the argument of two
out of the three parameters.

Write $u=e^{i\phi/3}=e^{2\pi i/3p}$. The group $\Gamma$ has generators given
by

(2.2) $R_{1}=\left(\begin{matrix}u^{2}&\rho&-u\bar{\tau}\\\ 0&\bar{u}&0\\\
0&0&\bar{u}\end{matrix}\right),\\\ R_{2}=\left(\begin{matrix}\bar{u}&0&0\\\
-u\bar{\rho}&u^{2}&\sigma\\\ 0&0&\bar{u}\end{matrix}\right),\\\
R_{3}=\left(\begin{matrix}\bar{u}&0&0\\\ 0&\bar{u}&0\\\
\tau&-u\bar{\sigma}&u^{2}\end{matrix}\right)$

which preserve the Hermitian form

(2.3) $H=\left(\begin{matrix}\alpha&\beta_{1}&\overline{\beta}_{3}\\\
\overline{\beta}_{1}&\alpha&\beta_{2}\\\
\beta_{3}&\overline{\beta}_{2}&\alpha\end{matrix}\right)$

where $\alpha=\sqrt{2-u^{3}-\bar{u}^{3}}$, $\beta_{1}=-i\bar{u}^{1/2}\rho$,
$\beta_{2}=-i\bar{u}^{1/2}\sigma$, $\beta_{3}=-i\bar{u}^{1/2}\tau$ (note that
here we take $\bar{u}^{1/2}=e^{-\pi i/3p}$).

This Hermitian form has signature $(2,1)$ if and only if ${\rm det}(H)<0$.
That is,

$\displaystyle 0$ $\displaystyle<$
$\displaystyle\alpha|\beta_{1}|^{2}+\alpha|\beta_{2}|^{2}+\alpha|\beta_{3}|^{2}-\alpha^{3}-\beta_{1}\beta_{2}\beta_{3}-\bar{\beta}_{1}\bar{\beta}_{2}\bar{\beta}_{3}$
$\displaystyle=$
$\displaystyle\alpha^{2}|\rho|^{2}+\alpha^{2}|\sigma|^{2}+\alpha^{2}|\tau|^{2}-\alpha^{3}-i\bar{u}^{3/2}\rho\sigma\tau+iu^{3/2}\bar{\rho}\bar{\sigma}\bar{\tau}.$

In terms of these parameters

(2.4) $\begin{array}[]{ll}&{\rm
tr}(R_{1}R_{2})=u(2-|\rho|^{2})+\bar{u}^{2},\quad{\rm
tr}(R_{2}R_{3})=u(2-|\sigma|^{2})+\bar{u}^{2},\\\ &{\rm
tr}(R_{1}R_{3})=u(2-|\tau|^{2})+\bar{u}^{2},\\\ &{\rm
tr}(R_{1}R_{3}^{-1}R_{2}R_{3})=u(2-|\sigma\tau-\bar{\rho}|^{2})+\bar{u}^{2}.\end{array}$

######  Lemma 2.1.

[10, Corollary 2.5] If $|\rho|=2\cos\zeta$, then the three eigenvalues of
$R_{1}R_{2}$ will be $\bar{u}^{2},\,-ue^{2i\zeta},\,-ue^{-2i\zeta}$.

###### Proof.

Each point on $C_{1}$ is a $\bar{u}=e^{-i\phi/3}$ eigenvector of $R_{1}$ and
each point on $C_{2}$ is a $\bar{u}=e^{-i\phi/3}$ eigenvector of $R_{2}$, see
(1.1). Therefore if ${\bf z}\in C_{1}\cap C_{2}$, then we will get that

$R_{1}R_{2}({\bf z})=e^{-i\phi/3}R_{1}({\bf z})=e^{-2i\phi/3}\bf z$

Hence $\bf z$ is a $\bar{u}^{2}=e^{-2i\phi 3}$ eigenvector of $R_{1}R_{2}$.
Hence the sum of the other two eigenvalues of $R_{1}R_{2}$ is
$u(2-|\rho|^{2})$. By the assumption $|\rho|=2\cos\zeta$, we know that
$R_{1}R_{2}$ is not loxodromic, see Section 6.2 in [2]. Therefore each
eigenvalue of $R_{1}R_{2}$ is of modulus one. Then we can get that the three
eigenvalues of $R_{1}R_{2}$ will be
$\bar{u}^{2},\,-ue^{2i\zeta},\,-ue^{-2i\zeta}$ from the form of
$\textrm{tr}(R_{1}R_{2})$ in (2.4). ∎

######  Remark 2.2.

If we suppose that $m\in\mathbb{N}$, $m\geqslant 2$, then
$|\rho|=2\cos(\pi/m)$ if and only if $br(R_{1},R_{2})=m$, see Section 2.2 in
[7] for details or more precisely [1, prop 2.3]. In particular if
$R_{1},\,R_{2}$ are complex involutions ($p=2$), then the order of
$R_{1}R_{2}$ will be of $m$.

Assume that

$br(R_{1},\,R_{2})=br(R_{1},\,R_{3}^{-1}R_{2}R_{3}),\qquad
br(R_{2},\,R_{3})=br(R_{1},\,R_{3}).$

From Remark 2.2 and (2.4), our hypothesis on braiding implies that

$|\rho|=|\sigma\tau-\bar{\rho}|,\qquad|\sigma|=|\tau|.$

Since we are free to choose the argument of two of the three parameters, we
impose the condition that $\sigma$ and $\tau$ should be real and non-negative,
which means that ${\rm Im}(\rho)={\rm Im}(\sigma\tau-\bar{\rho})$. So the
condition $|\rho|=|\sigma\tau-\bar{\rho}|$ becomes either
$\sigma\tau=\rho+\bar{\rho}$ or $\sigma\tau=0$. In the latter case the group
is reducible, so we do not consider it. Hence we suppose ${\rm Re}(\rho)>0$
and $\sigma=\tau=\sqrt{\rho+\bar{\rho}}$.

We suppose that $|\rho|=2\cos(\pi/m)$ and $\sigma=\tau=2\cos(\pi/n)$, where
$m,\,n\in\mathbb{N}$ and $m,\,n\geqslant 3$. Therefore the matrices in (2.2)
become:

(2.5) $\displaystyle R_{1}$ $\displaystyle=$
$\displaystyle\left(\begin{matrix}u^{2}&\rho&-u\sqrt{\rho+\bar{\rho}}\\\
0&\bar{u}&0\\\ 0&0&\bar{u}\end{matrix}\right),$ (2.6) $\displaystyle R_{2}$
$\displaystyle=$ $\displaystyle\left(\begin{matrix}\bar{u}&0&0\\\
-u\bar{\rho}&u^{2}&\sqrt{\rho+\bar{\rho}}\\\ 0&0&\bar{u}\end{matrix}\right),$
(2.7) $\displaystyle R_{3}$ $\displaystyle=$
$\displaystyle\left(\begin{matrix}\bar{u}&0&0\\\ 0&\bar{u}&0\\\
\sqrt{\rho+\bar{\rho}}&-u\sqrt{\rho+\bar{\rho}}&u^{2}\end{matrix}\right).$

Furthermore, the Hermitian form $H$ (2.3) has signature $(2,1)$ if and only if

(2.8) $\displaystyle 0$
$\displaystyle<\alpha|\rho|^{2}+2\alpha(\rho+\bar{\rho})-\alpha^{3}-i\bar{u}^{3/2}(\rho^{2}+|\rho|^{2})+iu^{3/2}(\bar{\rho}^{2}+|\rho|^{2})$
$\displaystyle=2\alpha(\rho+\bar{\rho})-\alpha^{3}-i\bar{u}^{3/2}\rho^{2}+iu^{3/2}\bar{\rho}^{2}.$

######  Proposition 2.3.

Let

$S=\left(\begin{matrix}\rho&u(1-\rho-\bar{\rho})&u^{2}\sqrt{\rho+\bar{\rho}}\\\
\bar{u}&0&0\\\ 0&\bar{u}\sqrt{\rho+\bar{\rho}}&-1\end{matrix}\right).$

Then

  1. (a)

$S^{2}=R_{1}R_{2}R_{3}$,

  2. (b)

$\displaystyle SR_{1}S^{-1}$ $\displaystyle=$ $\displaystyle
R_{1}R_{2}R_{1}^{-1},$ $\displaystyle SR_{2}S^{-1}$ $\displaystyle=$
$\displaystyle R_{1}R_{3}R_{1}R_{3}^{-1}R_{1}^{-1},$ $\displaystyle
SR_{3}S^{-1}$ $\displaystyle=$ $\displaystyle R_{1}R_{3}R_{1}^{-1}.$

In particular,

$S(R_{2}R_{3})S^{-1}=R_{1}R_{3},\quad
S(R_{1}R_{3}^{-1}R_{2}R_{3})S^{-1}=R_{1}R_{2}.$

###### Proof.

The first part follows by a simple matrix calculation. The second part could
be verified similarly, but it is easy to observe that

$S{\bf v}_{1}=R_{1}{\bf v}_{2},\quad S{\bf v}_{2}=R_{1}R_{3}{\bf v}_{1},\quad
S{\bf v}_{3}=-uR_{1}{\bf v}_{3},$

where ${\bf v}_{1}=[1,0,0]^{t},$ ${\bf v}_{2}=[0,1,0]^{t}$ and ${\bf
v}_{3}=[0,0,1]^{t}$ are the polar vectors of $R_{1},\,R_{2},\,R_{3}$
respectively. The result follows. ∎

In the following we will classify all discrete triangle groups generated by
$R_{1},R_{2},R_{3}$ with the 2-fold symmetry given by $S$ satisfying the
conditions (a) and (b) in Proposition 2.3.

######  Theorem 2.4.

Let $R_{1},R_{2},R_{3}$ be three complex reflections of order $p$ (with
$p\geqslant 2$) in $\textrm{SU}(2,1)$ so that $R_{i}$ keeps a complex geodesic
$C_{i}$ $(i=1,\,2,\,3)$ invariant. Assume that there is $S\in\textrm{SU}(2,1)$
such that

$SR_{1}S^{-1}=R_{1}R_{2}R_{1}^{-1},~{}SR_{2}S^{-1}=R_{1}R_{3}R_{1}R_{3}^{-1}R_{1}^{-1},SR_{3}S^{-1}=R_{1}R_{3}R_{1}^{-1},$
$S^{2}=R_{1}R_{2}R_{3}.$

Suppose that $br(R_{1},R_{3})=n$, $br(R_{1},R_{2})=m$ (where
$m,\,n\in\mathbb{N}$ and $m,\,n\geqslant 3$) and $R_{1}R_{2}R_{3}$ is of
finite order. Then the candidates for $(n,m)$ will be $(3,4)$, $(3,5)$,
$(4,3)$, $(5,4)$, $(8,6)$ and $(k,k)$ ($k\in\mathbb{N}$ and $k\geqslant 3$).

Note that the solutions correspond to the following parameter values, or their
complex conjugates:

$\begin{array}[]{|l|lll|}\hline\cr(n,m)&\rho&s=\rho-1&\sigma=\tau\\\
\hline\cr(3,4)&(1+i\sqrt{7})/2&e^{2\pi i/7}+e^{4\pi i/7}+e^{-6\pi i/7}&1\\\
(3,5)&2e^{2\pi i/5}\cos(\pi/5)&e^{2\pi i/5}+e^{7\pi i/15}+e^{-13\pi i/15}&1\\\
(4,3)&1&0&\sqrt{2}\\\ (5,4)&(1+i\sqrt{3})(\sqrt{5}-i\sqrt{3})/4&e^{-2\pi
i/3}+e^{2\pi i/15}+e^{8\pi i/15}&(1+\sqrt{5})/2\\\
(8,6)&(1+i)(1-i/\sqrt{2})&e^{\pi i/2}+e^{\pi i/12}+e^{-7\pi
i/12}&\sqrt{2+\sqrt{2}}\\\ (k,k)&2e^{i\pi/k}\cos(\pi/k)&e^{2\pi
i/k}&2\cos(\pi/k)\\\ \hline\cr\end{array}$

##  3\. The proof

Firstly a direct computation will show that the symmetry $S$ conjugates
$R_{1}R_{2}$ to $R_{1}R_{3}^{-1}R_{2}R_{3}$ and conjugates $R_{2}R_{3}$ to
$R_{1}R_{3}$. It means that

$br(R_{1},\,R_{2})=br(R_{1},\,R_{3}^{-1}R_{2}R_{3}),\qquad
br(R_{2},\,R_{3})=br(R_{1},\,R_{3})$

by recalling Remark 2.2 and (2.4). By the parameterization of the triangle
groups in Section 2 and the assumption in Theorem 2.4, one could get the
matrix representation of $H,\,R_{1},\,R_{2},\,R_{3}$ as (2.3), (2.5), (2.6),
(2.7), where

$|\rho|=2\cos(\pi/m),\qquad\sigma=\tau=2\cos(\pi/n).$

Throughout the proof we let $\zeta=\pi/m$ and $\eta=\pi/n$.

Because of $S^{2}=R_{1}R_{2}R_{3}$, we can restrict ourselves to $S$, which is
elliptic of finite order. Equivalently, there exist $a$ and $b$ that are
rational multiples of $\pi$ for which:

(3.9) $\textrm{tr}(S)=-1+\rho=e^{ia}+e^{ib}+e^{-i(a+b)}.$

Observe that there is some ambiguity in the choice of $a$ and $b$. First, we
can permute the three terms in this expression, and so permute
$\\{a,\,b,\,-a-b\\}$; secondly we can change the sign of all three terms and,
finally, since $\textrm{tr}(S)$ is only defined up to multiplying by a cube
root of unity, we can add the same integer multiple of $2\pi/3$ to both $a$
and $b$. We will use these operations to simplify things in our calculations
below.

We denote $\textrm{tr}(S)$ by $s$, then get that

(3.10) $\displaystyle|s|^{2}=1+|\rho|^{2}-2\,{\operatorname{Re}\,}(\rho)$
$\displaystyle=$ $\displaystyle|e^{ia}+e^{ib}+e^{-i(a+b)}|^{2},$ (3.11)
$\displaystyle\,{\operatorname{Re}\,}(s)=-1+\,{\operatorname{Re}\,}(\rho)$
$\displaystyle=$ $\displaystyle\cos(a)+\cos(b)+\cos(a+b),$

Recall that

$\displaystyle|\rho|^{2}$ $\displaystyle=$ $\displaystyle
4\cos^{2}\zeta=2\cos(2\zeta)+2,$ $\displaystyle\,{\operatorname{Re}\,}(\rho)$
$\displaystyle=$
$\displaystyle\frac{\sigma\tau}{2}=2\cos^{2}\eta=\cos(2\eta)+1.$

The above two equations can be simplified to

(3.12) $\displaystyle 1$ $\displaystyle=$
$\displaystyle\cos(2\zeta)-\cos(2\eta)-\cos(a-b)-\cos(a+2b)-\cos(2a+b),\qquad$
(3.13) $\displaystyle 0$ $\displaystyle=$
$\displaystyle\cos(2\eta)-\cos(a)-\cos(b)-\cos(a+b).$

In what follows we will repeatedly use the following result given by A.
Monaghan, which generalises the result of Conway and Jones for vanishing sums
of cosines of rational multiples of $\pi$.

######  Proposition 3.1.

[6, Theorem 2.4.3.1] Suppose that we have at most five distinct rational
numbers of $\pi$, for which some rational linear combination of their cosines
is rational but no proper subset has this property. If $\phi\in(0,\pi)$ and
all other angles are normalised to lie in $(0,\frac{\pi}{2}),$ then the
appropriate linear combination is proportional to one of the following:
_(a)_ $0=\sum_{j=0}^{2}\cos(\phi+\frac{2j\pi}{3})$,
_(b)_
$0=\cos(\phi)+\cos(\phi\pm\frac{2\pi}{5})-\cos(\phi\pm\frac{2\pi}{15})+\cos(\phi\pm\frac{7\pi}{15}),$
_(c)_
$0=\cos(\phi)-\cos(\phi\pm\frac{\pi}{5})+\cos(\phi\pm\frac{\pi}{15})-\cos(\phi\pm\frac{4\pi}{15}),$
_(d)_ $\frac{1}{2}=\cos(\frac{\pi}{3})$,
_(e)_ $\frac{1}{2}=\cos(\frac{\pi}{5})-\cos(\frac{2\pi}{5})$,
_(f)_
$\frac{1}{2}=\cos(\frac{\pi}{5})-\cos(\frac{\pi}{15})+\cos(\frac{4\pi}{15})$,
_(g)_
$\frac{1}{2}=-\cos(\frac{2\pi}{5})+\cos(\frac{2\pi}{15})-\cos(\frac{7\pi}{15})$,
_(h)_
$\frac{1}{2}=-\cos(\frac{\pi}{15})+\cos(\frac{2\pi}{15})+\cos(\frac{4\pi}{15})-\cos(\frac{7\pi}{15})$,
_(i)_
$\frac{1}{2}=\cos(\frac{\pi}{7})-\cos(\frac{2\pi}{7})+\cos(\frac{3\pi}{7})$,
_(j)_
$\frac{1}{2}=\cos(\frac{\pi}{7})-\cos(\frac{2\pi}{7})+\cos(\frac{2\pi}{21})-\cos(\frac{5\pi}{21})$,
_(k)_
$\frac{1}{2}=\cos(\frac{\pi}{7})+\cos(\frac{3\pi}{7})-\cos(\frac{\pi}{21})+\cos(\frac{8\pi}{21})$,
_(l)_
$\frac{1}{2}=-\cos(\frac{2\pi}{7})+\cos(\frac{3\pi}{7})+\cos(\frac{4\pi}{21})+\cos(\frac{10\pi}{21})$,
_(m)_
$\frac{1}{2}=\cos(\frac{\pi}{7})-\cos(\frac{\pi}{21})+\cos(\frac{2\pi}{21})-\cos(\frac{5\pi}{21})+\cos(\frac{8\pi}{21})$,
_(n)_
$\frac{1}{2}=-\cos(\frac{2\pi}{7})+\cos(\frac{2\pi}{21})+\cos(\frac{4\pi}{21})-\cos(\frac{5\pi}{21})+\cos(\frac{10\pi}{21})$,
_(o)_
$\frac{1}{2}=\cos(\frac{3\pi}{7})-\cos(\frac{\pi}{21})+\cos(\frac{4\pi}{21})+\cos(\frac{8\pi}{21})+\cos(\frac{10\pi}{21})$.

Since the right hand side of equation (3.12) is $1$ (rather than $0$ or
$1/2$), Monaghan’s theorem implies that it must be a sum of (at least) two
similar sums involving fewer cosines. We begin by showing that at least one of
the cosines must itself be rational.

######  Proposition 3.2.

Suppose that $\zeta=\pi/m$, $\eta=\pi/n$ and $a$, $b$ are rational multiples
of $\pi$ so that equations (3.12) and (3.13) hold. Then one of the cosines in
equation (3.12) must be rational.

###### Proof.

Suppose that none of the cosines are rational. Then (3.12) splits into two
rational sums, one of length two and the other of length three, neither of
which has a rational subsum. By inspection from prop 3.1 we see that these two
sums must have the value $0$, $\pm 1/2$. Since they sum to $1$, they must both
be $1/2$. Therefore, the sum of length 2 must be (e) and the sum of length 3
must be one of (f), (g) or (i).

  1. (1)

$1/2=\cos(2\zeta)-\cos(2\eta)=-\cos(a-b)-\cos(a+2b)-\cos(2a+b)$. Since
$\zeta=\pi/m$ and $\eta=\pi/n$ the sum (e) implies that $2\zeta=\pi/5$ and
$2\eta=2\pi/5$. For the second equation, there are certain symmetry operations
on $a$ and $b$ described in the paragraph after equation (3.9) above. Up to
these operations, we now list the possible values of $a$ and $b$:

$\begin{array}[]{|c|c|c|c|c|}\hline\cr a-b&a+2b&2a+b&a&b\\\
\hline\cr\pi/15&11\pi/15&4\pi/5&13\pi/45&2\pi/9\\\
2\pi/5&7\pi/15&13\pi/15&19\pi/45&\pi/45\\\
2\pi/7&4\pi/7&6\pi/7&-2\pi/7&-4\pi/7\\\ \hline\cr\end{array}$

Using $2\eta=2\pi/5$, we see that none of the values in this table satisfy
(3.13). Therefore we get no solutions.

  2. (2)

$1/2=\cos(2\zeta)-\cos(a-b)=-\cos(2\eta)-\cos(a+2b)-\cos(2a+b)$. The first
equation gives $2\zeta=\pi/5$ as in case (1) and so $a-b=2\pi/5$. Since
$a-b=(2a+b)-(a+2b)$ the difference of two of the angles in the second equation
must be $2\pi/5$. By inspection, we see the only solution is $a+2b=7\pi/15$
and $2a+b=13\pi/15$. This means $2\eta=2\pi/5$ and we are back in case (1).

  3. (3)

$1/2=-\cos(2\eta)-\cos(a-b)=\cos(2\zeta)-\cos(a+2b)-\cos(2a+b)$. The first
equation gives $2\eta=2\pi/5$ as in (1) and so $a-b=4\pi/5$. Substituting in
the second equation, we see $a+2b=-\pi/15$ and $2a+b=11\pi/15$. Thus
$2\zeta=\pi/5$ and we are back in case (1) again.

  4. (4)

$1/2=-\cos(a-b)-\cos(a+2b)=\cos(2\zeta)-\cos(2\eta)-\cos(2a+b)$. Up to
symmetries of $a$, $b$ and $-a-b$, the first sum implies that $a-b=2\pi/5$ and
$a+2b=4\pi/5$. Hence $2a+b=6\pi/5$ and so the second sum must be (f). Thus
$\cos(2\zeta)=\cos(2\pi/m)=\cos(4\pi/15)$ or
$\cos(2\eta)=\cos(2\pi/n)=-\cos(4\pi/15)$ so either $m$ or $n$ is not an
integer. Therefore there are no solutions.

∎

As a consequence of this result, we can consider separate cases where each of
the cosines in (3.12) is rational. If either $\cos(2\zeta)$ or $\cos(2\eta)$
is rational it must be $0$ or $\pm 1/2$ since $\zeta=\pi/m$ and $\eta=\pi/n$
where $m$ and $n$ are at least 3. If one of the other three cosines is
rational we can use the allowable symmetries of $a$ and $b$, we to assume that
$\cos(a-b)$ is rational. We treat each of these cases separately below. First
we eliminate a simple situation which gives us many solutions and will recur
in the different cases.

######  Lemma 3.3.

Suppose that $\cos(2\zeta)=\cos(2\eta)$, or equivalently $m=n$, then putting
$s=e^{\pm 2\pi i/m}$ gives a solution to equations (3.12) and (3.13) for all
$m\geqslant 3$.

###### Proof.

Substituting $\cos(2\zeta)=\cos(2\eta)$ into (3.12) gives: Now we obtain that

$\displaystyle 0$ $\displaystyle=$ $\displaystyle
1+\cos(a-b)+\cos(a+2b)+\cos(2a+b)$ $\displaystyle=$ $\displaystyle
2\cos^{2}\bigl{(}(a-b)/2\bigr{)}+2\cos\bigl{(}(a-b)/2\bigr{)}\cos\bigl{(}3(a+b)/2\bigr{)}$
$\displaystyle=$ $\displaystyle
4\cos\bigl{(}(a-b)/2\bigr{)}\cos\bigl{(}(a+2b)/2\bigr{)}\cos\bigl{(}(2a+b)/2\bigr{)}.$

Therefore one of $(a-b)$, $(a+2b)$ or $(2a+b)$ is an odd multiple of $\pi$.
Without loss of generality, we suppose that $a+2b=(2k+1)\pi$. Then we get
$-a-b=b-(2k+1)\pi$ which yields $s=e^{ia},$ where $a$ is a rational multiple
of $\pi$. Because
$\,{\operatorname{Re}\,}(s)=-1+\,{\operatorname{Re}\,}(\rho)=-1+\frac{|\sigma|^{2}}{2}=\cos(2\eta)$,
we see that $\cos(a)=\cos(2\pi/m)=\cos(2\pi/n)$.

Now we consider the signature of the Hermitian form

$\textrm{Det}(H)=ie^{-\frac{4\theta+3\phi}{2}i}(-1+e^{(2\theta+\phi)i})(e^{i\theta}+e^{i\phi})^{2}.$

Table 1. Signature of Hermitian form $s$ | (2, 1) | degenerate | (3, 0)
---|---|---|---
$e^{\frac{2\pi i}{3}}$ ($m=n=3$) | $p\geqslant 4$ | $p=3$ | $p=2$
$e^{\frac{2\pi i}{4}}$ ($m=n=4$) | $p\geqslant 3$ | $p=2$ | none
$e^{\frac{2\pi i}{k}}$ ($m=n=k\geqslant 5$) | $p\geqslant 2$ | none | none

In this case, we get the solution $n=m$.
∎

We now consider the cases where $\cos(2\zeta)$, $\cos(2\eta)$ or $\cos(a-b)$
are rational. We will use the following result proved by Parker when he was
analysing the triangle groups with 3-fold symmetry [9]. In [9] the last two
cases were missed out, but this was corrected in [1].

######  Proposition 3.4.

[9, prop 3.2] Let $\theta$, $a$ and $b$ be rational multiples of $\pi$. Write
$s=e^{ia}+e^{ib}+e^{-i(a+b)}$. Then the only possible solutions to the
equation

$\cos(2\theta)-\cos(a-b)-\cos(a+2b)-\cos(2a+b)=\frac{1}{2}$

give rise to the following values of $\theta$ and $s$, up to changing the sign
of $\theta$ and up to conjugating $s$ and multiplying it by a power of
$\omega=e^{2\pi i/3}$:

  1. (i)

$2\theta=2\pi/3$ and $s=-e^{-i\psi/3}$ for some angle $\psi$ that is a
rational multiple of $\pi$;

  2. (ii)

$2\theta=\psi$ and
$s=e^{2i\psi/3}+e^{-i\psi/3}=e^{i\psi/6}2\cos\frac{\psi}{2}$ for some angle
$\psi$ that is a rational multiple of $\pi$;

  3. (iii)

$2\theta=\pi/3$ and $s=e^{i\pi/3}+e^{-i\pi/6}2\cos\frac{\pi}{4}$;

  4. (iv)

$2\theta=\pi/5$ and $s=e^{i\pi/3}+e^{-i\pi/6}2\cos\frac{\pi}{5}$;

  5. (v)

$2\theta=3\pi/5$ and $s=e^{i\pi/3}+e^{-i\pi/6}2\cos\frac{2\pi}{5}$;

  6. (vi)

$2\theta=\pi/2$ and $s=e^{2\pi i/7}+e^{4\pi i/7}+e^{-6\pi i/7}$;

  7. (vii)

$2\theta=\pi/2$ and $s=e^{2\pi i/9}+e^{-i\pi/9}2\cos\frac{2\pi}{5}$;

  8. (viii)

$2\theta=\pi/2$ and $s=e^{2\pi i/9}+e^{-i\pi/9}2\cos\frac{4\pi}{5}$;

  9. (ix)

$2\theta=\pi/7$ and $s=e^{2\pi i/9}+e^{-i\pi/9}2\cos\frac{2\pi}{7}$;

  10. (x)

$2\theta=5\pi/7$ and $s=e^{2\pi i/9}+e^{-i\pi/9}2\cos\frac{4\pi}{7}$;

  11. (xi)

$2\theta=3\pi/7$ and $s=e^{2\pi i/9}+e^{-i\pi/9}2\cos\frac{6\pi}{7}$;

  12. (xii)

$2\theta=2\pi/5$ and $s=1+2\cos\frac{2\pi}{5}$;

  13. (xiii)

$2\theta=4\pi/5$ and $s=1+2\cos\frac{4\pi}{5}$.

Note that for the groups Parker was considering $s=e^{ia}+e^{ib}+e^{-ia-ib}$
was the trace of $R_{1}J$, whereas in our case it is the trace of $S$. In the
cases where $\cos(2\zeta)=1/2$ or $\cos(2\eta)=-1/2$ then equation (3.12)
reduces to the equation from prop 3.4, and we can use that result to find
solutions.

######  Lemma 3.5.

Suppose that $\cos(2\eta)$ is rational. Then the only solutions to (3.12) and
(3.13) are $\cos(2\zeta)=\cos(2\pi/m)$ and $\cos(2\eta)=\cos(2\pi/n)$ where
$(n,m)$ is one of $(3,3)$, $(3,4)$, $(3,5)$, $(4,3)$, $(4,4)$ or $(6,6)$.

###### Proof.

Since $\cos(2\eta)$ is rational and not equal to $\pm 1$ it can only be $0$ or
$\pm 1/2$. We treat each case separately.

  1. (1)

$\cos(2\eta)=-\frac{1}{2}$, which gives $n=3.$ Note that

$\frac{1}{2}=\cos(2\eta)+1=\,{\operatorname{Re}\,}(\rho)=\,{\operatorname{Re}\,}(s)+1$

and so $\,{\operatorname{Re}\,}(s)=-1/2$. We rewrite (3.12) to give the
equation from prop 3.4 with $\theta=\zeta$. By direct calculation, we just
need to consider cases (i), (ii) and (vi) because of
$\,{\operatorname{Re}\,}(s)=-1/2$.

    1. (i)

$s=-e^{-i\psi/3}$ and so $|s|=1$. This yields that $|\rho|=2\cos(\pi/m)=1$,
and so $m=3$. By considering $\,{\operatorname{Re}\,}(s)=-\cos(\theta/3)$, we
know that $\theta=\pm\pi+6k\pi$ ($k\in\mathbb{Z}$) which means that $s=-e^{\mp
i\pi/3}$. From (2.8), we get that

$\textrm{Det}(H)=\mp\sqrt{3}\cos(\phi/2)+\sin(\phi/2)-2\sin(3\phi/2).$

We list the corresponding signature of Hermitian form for different $s$ in
Table 3.2.

Table 2. Signature of Hermitian form $s$ | (2, 1) | degenerate | (3, 0)
---|---|---|---
$-e^{-i\pi/3}$ | $p\geqslant 4$ | $p=3$ | $p=2$
$-e^{i\pi/3}$ | none | $p=6$ | $p\neq 6$

In this case, we get that $n=m=3$.

    2. (ii)

$s=e^{2i\psi/3}+e^{-i\psi/3}=e^{i\psi/6}2\cos(\psi/2)$ where $\psi=2\theta$.
By solving

$\displaystyle-1/2$ $\displaystyle=$
$\displaystyle\,{\operatorname{Re}\,}(s)=\cos(4\theta/3)+\cos(2\theta/3)$
$\displaystyle=$ $\displaystyle 2\cos^{2}(2\theta/3)+\cos(2\theta/3)-1$

we obtain $\cos(2\theta/3)=(-1\pm\sqrt{5})/4$. That is, $2\theta/3=\pm
2\pi/5+2k\pi$ or $\pm 4\pi/5+2k\pi$. Hence $\theta=\pm 3\pi/5+3k\pi$ or $\pm
6\pi/5+3k\pi$. The only solution to $|s|=2|\cos(\theta)|=2\cos(\pi/m)$ is
$m=5$ (coming from $2\theta/3=4\pi/5-2\pi$). Therefore $s=e^{2\pi i/5}+e^{4\pi
i/5}$ or $s=e^{-2\pi i/5}+e^{-4\pi i/5}$. In these cases we find,
respectively, that:

$\displaystyle\textrm{Det}(H)$
$\displaystyle=-\sqrt{5+2\sqrt{5}}\cos{\frac{\phi}{2}}-(2+\sqrt{5}+4\cos{\phi})\sin{\frac{\phi}{2}},$
$\displaystyle\textrm{Det}(H)$
$\displaystyle=\sqrt{5+2\sqrt{5}}\cos{\frac{\phi}{2}}-(2+\sqrt{5}+4\cos{\phi})\sin{\frac{\phi}{2}}.$

Table 3. Signature of Hermitian form $s$ | (2, 1) | degenerate | (3, 0)
---|---|---|---
$e^{-\frac{2\pi i}{5}}+e^{-\frac{4\pi i}{5}}$ | $p\leqslant 7$ | none | $p\geqslant 8$
$e^{\frac{2\pi i}{5}}+e^{\frac{4\pi i}{5}}$ | $p\geqslant 2$ | none | none

In this case, we get that $n=3,$ $m=5$.

    3. (vi)

$s=e^{2\pi i/7}+e^{4\pi i/7}+e^{-6\pi i/7}=(-1+\sqrt{7}i)/2$. It follows that
$\,{\operatorname{Re}\,}(s)=-1/2$ and $|s|=\sqrt{2}$ which indicates that
$m=4.$ A simple calculation yields that

$\textrm{Det}(H)=\frac{1}{2}(1-8\cos\phi)\sin\frac{\phi}{2}$

from which it follows that the signature of the Hermitian form will be of
$(2,1)$ for $p\geqslant 5$, otherwise it will be positive. In this case, we
get that $n=3$, $m=4$.

Therefore we obtain the solutions $(n,m)=(3,3)$, $(3,4)$ and $(3,5)$.

  2. (2)

$\cos(2\eta)=0.$ Now we have $|\sigma|^{2}=2$ which yields
$\,{\operatorname{Re}\,}(s)=0$. Therefore one can get the following two
equations

(3.14) $\left\\{\begin{aligned}
&\cos(2\zeta)-\cos(a-b)-\cos(a+2b)-\cos(2a+b)=1,\\\ &\cos a+\cos
b+\cos(a+b)=0.\end{aligned}\right.$

Since the first of these has 1 on the right hand side, it must split as the
sum of (at least) two minimal subsums. Treating these case by case we see that
the only possibilities are $\cos(2\zeta)=0$, which yields $m=n=4$ and
$\cos(2\zeta)=-1/2$, which gives $n=4$ and $m=3$. The former case is a
particular instance of lem 3.3. In the latter case we rewrite (3.10) as

$|s|^{2}=1+|\rho|^{2}-2\,{\operatorname{Re}\,}(\rho)=1+2\cos(2\zeta)-2\cos(2\eta)=0.$

Therefore, the only solution is $s=0$, or equivalently $\rho=1$. This implies
that

$\textrm{Det}(H)=-2\sin\frac{3\phi}{2}=-2\sin\frac{3\pi}{p},$

and the signature of the Hermitian form will be positive if $p=2$, degenerate
if $p=3$, negative (of signature (2,1)) if $p\geqslant 4.$ Therefore in this
case we get that $n=4$, $m=3$. Hence the only solutions we get in this case
are $(n,m)=(4,3)$ and $(4,4)$.

  3. (3)

$\cos(2\eta)=1/2.$ Now we have $|\sigma|^{2}=3$ from which it follows that
$\,{\operatorname{Re}\,}(s)=1/2$. We rewrite the two equations

(3.15) $\left\\{\begin{aligned}
&\cos(2\zeta)-\cos(a-b)-\cos(a+2b)-\cos(2a+b)=\frac{3}{2},\\\ &\cos a+\cos
b+\cos(a+b)=\frac{1}{2}.\end{aligned}\right.$

If the second equation is irreducible, then it must be one of prop 3.1 parts
(f), (g) or (i). We see in each case that the angles involved do not sum to
$0$ (making each cosine positive, the sum is $\pi$ times the ratio of two odd
integers for each choice of sign). If the second equation splits as the sum of
two rational subsums then, without loss of generality, $\cos(a)$ is rational.
Hence it is in the set $\\{0,\pm 1/2.\pm 1\\}$. Simple trigonometry shows that

$\displaystyle 2\cos(a/2)\cos(a/2+b)$ $\displaystyle=$
$\displaystyle\cos(b)+\cos(a+b)$ $\displaystyle=$ $\displaystyle 1/2-\cos(a),$
$\displaystyle\cos(a+2b)+1$ $\displaystyle=$ $\displaystyle 2\cos^{2}(a/2+b)$
$\displaystyle=$
$\displaystyle\bigl{(}1/2-\cos(a)\bigr{)}^{2}/\bigl{(}1+\cos(a)\bigr{)},$
$\displaystyle\cos(a-b)+\cos(2a+b)$ $\displaystyle=$ $\displaystyle
2\cos(3a/2)\cos(a/2+b)$ $\displaystyle=$
$\displaystyle-2\bigl{(}1/2-\cos(a)\bigr{)}^{2}.$

Substituting these identities in the first equation, we see that
$\cos(2\zeta)$ is a rational function of $\cos(a)$, and so is rational.
Substituting the different values of $\cos(a)$ gives a solution with
$\zeta=\pi/m$ only when $\cos(a)=\pm 1/2$. In both cases, $\cos(2\zeta)=1/2$
and so $m=6$. Thus we obtain the solution $(n,m)=(6,6)$.

∎

######  Lemma 3.6.

Suppose that $\cos(2\zeta)$ is rational. Then the only solutions to (3.12) and
(3.13) are $\cos(2\zeta)=\cos(2\pi/m)$ and $\cos(2\eta)=\cos(2\pi/n)$ where
$(n,m)$ is one of $(3,3)$, $(4,3)$, $(4,4)$, $(5,4)$, $(6,6)$ or $(8,6)$.

###### Proof.

Since $\cos(2\zeta)$ is rational and not equal to $\pm 1$ it can only be $0$
or $\pm 1/2$. We treat each case separately.

  1. (1)

$\cos(2\zeta)=1/2$, which gives $m=6$. In this case, we know

$|s|^{2}=1+2\cos(2\zeta)-2\cos(2\eta)=2-2\cos(2\eta).$

In this case, we rewrite equation (3.12) to give the equation from prop 3.4
with $2\theta=\pi-2\eta$. Checking one by one, we will find that there is no
value of $s$ in prop 3.4 satisfying (3.13) except the cases (i) and (ii). For
(i) we have $2\eta=\pi-2\theta=\pi/3$ and so $n=6$ (we have analysed this case
previously). For (ii), we have $\psi=\pi-2\eta$ and
$s=e^{2i\pi/3-4i\eta/3}+e^{-i\pi/3+2i\eta/3}$. Substituting in equation (3.13)
gives

$\displaystyle 0$ $\displaystyle=$ $\displaystyle\cos(2\eta)-{\rm Re}(s)$
$\displaystyle=$
$\displaystyle-\cos(\pi-2\eta)-\cos(2\pi/3-4\eta/3)-\cos(\pi/3-2\eta/3)$
$\displaystyle=$
$\displaystyle-\cos(2\pi/3-4\eta/3)\bigl{(}1+2\cos(\pi/3-2\eta/3)\bigr{)}.$

The only solution with $\eta=\pi/n$ is when $2\pi/3-4\eta/3=\pi/2$. That is,
$n=8$. By calculating $\textrm{Det}(H)=-2\cos(\phi)(1+2\sin\phi)$, we see that
$H$ is of signature $(3,0)$ for $p=2$ and is of signature $(2,1)$ for any
$p\geqslant 3$. In this case, we get $(n,m)=(6,6)$ or $(8,6)$.

  2. (2)

$\cos(2\zeta)=0$, which gives $m=4$. Then we get that $|\rho|^{2}=2$ and
$|s|^{2}=3-|\sigma|^{2}$. Also (3.12) can be replaced by

$-\cos(2\eta)-\cos(a-b)-\cos(a+2b)-\cos(2a+b)=1.$

We have already analysed the case where $\cos(2\eta)=0$ or $-1/2$, which lead
to the solution $(n,m)=(3,4)$ or $(4,4)$. If $\cos(2\eta)=1/2$, then
$|\sigma|^{2}=3$ induces $s=0$, which contradicts
$\,{\operatorname{Re}\,}(s)=-1+|\sigma|^{2}/2=1/2$. Then it suffices for us to
consider the following possible values due to $\eta=\pi/n$,

$2\eta$ | $a-b$ | $a+2b$ | $2a+b$ | $a$ | $b$
---|---|---|---|---|---
$2\pi/5$ | $2\pi/3$ | $7\pi/15$ | $17\pi/15$ | $3\pi/5$ | $-\pi/15$
$2\pi/5$ | $2\pi/5$ | $4\pi/5$ | $6\pi/5$ | $8\pi/5$ | $2\pi/15$

From this table, we know that $n=5$ and the pair values $a=3\pi/5$ and
$b=-\pi/15$ do not satisfy the equation (3.13). However the second line
$a=8\pi/5$ and $b=2\pi/15$ satisfy the equation (3.13) by applying the
equation (g) in prop 3.1. Then we calculate the signature of the Hermitian
form $H$ using (2.8). We see that $H$ is of signature $(3,0)$ for $p=2$ and is
of signature $(2,1)$ for any $p\geqslant 3$. In this case, we get that $m=4$
and $n=5$.

  3. (3)

$\cos(2\zeta)=-\frac{1}{2}.$ It follows that $m=3$ and

$-\cos(2\eta)-\cos(a-b)-\cos(a+2b)-\cos(2a+b)=\frac{3}{2}.$

Also, $\cos(2\zeta)=-1/2$ implies $|\rho|=1$ and so
$\cos(2\eta)+1=\,{\operatorname{Re}\,}(\rho)\leqslant 1$. This means that
$\cos(2\eta)\leq 0$ and so either $\cos(2\eta)=\cos(2\pi/n)=-1/2$ or $0$. We
have analysed both of these cases already. These give solutions $(n,m)=(3,3)$
or $(4,3)$.

∎

Now we begin to consider the remaining case in which $\cos(a-b)$ is rational.

######  Lemma 3.7.

Suppose that $\cos(a-b)=-1$, then $\cos(2\zeta)-\cos(2\eta)=0$, and the
possible solutions are given in lem 3.3 in which $n=m$.

###### Proof.

$\cos(a-b)=-1$ which gives $b=a+(2k+1)\pi$. Hence we have
$\cos(a+2b)=\cos(3a)$ and $\cos(2a+b)=-\cos(3a)$. Therefore, equation (3.12)
reduces to $\cos(2\zeta)-\cos(2\eta)=0$, which we have already treated in lem
3.3. ∎

######  Lemma 3.8.

Suppose that $\cos(a-b)=-1/2$, $\cos(2\zeta)$ and $\cos(2\eta)$ are not
rational, $\cos(2\zeta)-\cos(2\eta)\neq 0$. Then we get no solutions for $n$,
$m$ such that (3.12) and (3.13) hold.

###### Proof.

$\cos(a-b)=-1/2$, which gives $b=a\pm 2\pi/3+2k\pi$. Hence we have
$\cos(a+2b)=\cos(3a\mp 2\pi/3)$ and $\cos(2a+b)=\cos(3a\pm 2\pi/3)$. Therefore
equation (3.12) becomes

$\displaystyle 1/2$ $\displaystyle=$ $\displaystyle 1+\cos(a-b)$
$\displaystyle=$ $\displaystyle\cos(2\zeta)-\cos(2\eta)-\cos(a+2b)-\cos(2a+b)$
$\displaystyle=$ $\displaystyle\cos(2\zeta)-\cos(2\eta)+\cos(3a).$

Since we have supposed that $\cos(2\zeta)$ and $\cos(2\eta)$ are not rational,
the only way this equation can split into to rational subsums is for
$\cos(3a)$ to be rational. Investigating the different possibilities, we see
that (3.13) then implies $\cos(2\eta)$ is rational.

Now suppose the equation does not split into two rational sums of cosines. We
list the possible values of $2\zeta,\ 2\eta,\ a,\,b$, up to the allowable
symmetries of $a$ and $b$.

$2\zeta$ | $2\eta$ | $3a$ | $a-b$ | $a$ | $b$
---|---|---|---|---|---
$\pi/5$ | $2\pi/5$ | $\pi/2$ | $2\pi/3$ | $\pi/6$ | $-\pi/2$
$2\pi/5$ | $\pi/5$ | $0$ | $2\pi/3$ | $0$ | $-2\pi/3$
$\pi/5$ | $\pi/15$ | $4\pi/15$ | $2\pi/3$ | $4\pi/45$ | $-26\pi/45$
$2\pi/15$ | $2\pi/5$ | $8\pi/15$ | $2\pi/3$ | $8\pi/45$ | $-22\pi/45$
$\pi/7$ | $2\pi/7$ | $3\pi/7$ | $2\pi/3$ | $\pi/7$ | $-11\pi/21$

However, we note that there are no values of $2\eta,\ a,\,b$ in this list
satisfying (3.13). Therefore there are no solutions for $n,m$. ∎

######  Lemma 3.9.

Suppose that $\cos(a-b)=0$, $1/2$ or $1$, $\cos(2\zeta)$ and $\cos(2\eta)$ are
not rational, $\cos(2\zeta)-\cos(2\eta)\neq 0$. Then there are no solutions
for $n$, $m$ satisfying both (3.12) and (3.13).

###### Proof.

We immediately get that

(3.16) $\cos(2\zeta)-\cos(2\eta)-\cos(a+2b)-\cos(2a+b)=1\hbox{ or
}\frac{3}{2}\hbox{ or }2.$

Since the right hand side is not $0$, $\pm 1/2$, we see that this sum must
split into shorter rational sums of cosines.We break down into the following
three cases.

  1. (1)

$\cos(2\zeta)-\cos(2\eta)=\pm 1/2$.

    1. (i)

$\cos(2\zeta)-\cos(2\eta)=1/2$. Note that $\zeta=\pi/m$ and $\eta=\pi/n$,
where $m,n\in\mathbb{N}$. Therefore we know that $(n,m)$ is $(5,10)$ and

$\cos(a-b)+\cos(a+2b)+\cos(2a+b)=-\frac{1}{2}.$

We have supposed that $\cos(a-b)$ is rational, then using elementary
trigonometry arguments, we see that

$2\cos\bigl{(}(a-b)/2\bigr{)}\cos\bigl{(}3(a+b)/2\bigr{)}=-\frac{1}{2}-\cos(a-b).$

Squaring both sides and rearranging gives

$\cos(3a+3b)=\frac{\cos^{2}(a-b)-3/4}{\cos(a-b)+1}.$

We have assumed that $\cos(a-b)=0$ or $\cos(a-b)=1/2$ or $\cos(a-b)=1$, which
means that $\cos(3a+3b)=-3/4$ or $-1/3$ or $-1/8$. It gives a contradiction
here.

    2. (ii)

$\cos(2\zeta)-\cos(2\eta)=-1/2$, i.e.

$\cos(a+2b)-\cos(2a+b)=-\frac{3}{2}\hbox{ or }-2\hbox{ or }-\frac{5}{2}.$

This sum must again split and so both cosines are rational. Therefore the
possible values for $\cos(a+2b)$ are just $-1$ or $-1/2$ which are equivalent
to the case where $\cos(a-b)$ is this value, see lem 3.7 and lem 3.8. However
we have assumed that $\cos(2\zeta)-\cos(2\eta)\neq 0$, therefore there are no
solutions for $n$, $m$ satisfying both (3.12) and (3.13).

  2. (2)

$\cos(2\zeta)-\cos(x)$ (or $\cos(2\eta)+\cos(y)$) is $1/2$ or $-1/2$, where
$x,\,y\in\\{a+2b,\,2a+b\\}$.

Recalling the equation (3.16), $\cos(2\zeta)-\cos(x)=\pm 1/2$ means that
$\cos(2\eta)+\cos(y)$ is one of the values
$\\{-5/2,~{}-2,~{}-3/2,~{}-1,~{}-1/2\\}$. We just need to consider the case
$\cos(2\eta)+\cos(y)=-1/2$, because other values of $\cos(2\eta)+\cos(y)$ mean
that $\cos(2\eta)$ will be rational. Without loss of generality, we suppose
that $x=a+2b$, $y=2a+b$ and list the values of $2\zeta$, $a+2b$, $2\eta$,
$2a+b$ and corresponding $a-b$ .

$2\zeta$ | $a+2b$ | $2\eta$ | $2a+b$ | $a-b$
---|---|---|---|---
$\pi/5$ | $2\pi/5$ | $2\pi/5$ | $4\pi/5$ | $2\pi/5$
$\pi/5$ | $-2\pi/5$ | $2\pi/5$ | $4\pi/5$ | $6\pi/5$

There are no values of $a-b$ such that $\cos(a-b)=0$ or $\cos(a-b)=1/2$ or
$\cos(a-b)=1$.
Therefore there are no solutions for $n$, $m$ satisfying both (3.12) and
(3.13) in this case.

  3. (3)

$\cos(x)$ is rational, where $x\in\\{a+2b,\,2a+b\\}$. By suitable changes of
$a$ and $b$, the cases $\cos(a+2b)$ or $\cos(2a+b)$ is $-1/2$ or $-1$ are
equivalent to the cases in lem 3.7 and lem 3.8. Therefore there are no
solutions for $n$, $m$ because we supposed that $\cos(2\zeta)-\cos(2\eta)\neq
0$.

Then we consider the condition for $\cos(x)$ to be $0$, $1/2$ or $1$ and
suppose that $x=a+2b.$ We get that

$\cos(2\zeta)-\cos(2\eta)-\cos(2a+b)\in\left\\{1,~{}\frac{3}{2},~{}2,~{}\frac{5}{2},~{}3\right\\},$

which can be reduced to $\cos(2\zeta)-\cos(2a+b)$ or $\cos(2\eta)+\cos(2a+b)$
is rational which has been considered above.

Now we can get that there are no solutions for $n$, $m$ satisfying both (3.12)
and (3.13) under the conditions in lem 3.9. ∎

We sum up all the candidates for $n$, $m$ from above process,

lem 3.3 $n=m\geqslant 3$;

lem 3.5 $(n,m)\in(3,3)$, $(3,4)$, $(3,5)$, $(4,3)$, $(4,4)$ or $(6,6)$;

lem 3.6 $(n,m)\in(3,3)$, $(4,3)$, $(4,4)$, $(5,4)$, $(6,6)$ or $(8,6)$;
which we desired.

######  Remark 3.10.

Note that the new candidates for $(n,m)$ to be $(5,4)$, $(4,3)$ and $(8,6)$ do
not appear on Thompson’s list in [12]. However referring to [1], in what
follows we will see that the triangle groups for $(n,m)$ to be $(5,4)$
corresponds to Thompson groups ${\bf S}_{2}$ and the triangle groups for
$(m,n)$ to be $(4,3)$ is of actually Mostow groups with braiding $(2,3,4;4)$.
The pair $(n,m)=(8,6)$ was also found by Deraux when he was making a similar
computer search to Thompson (private communication). 1\. $(n,m)=(5,4)$.
Suppose that $M_{1},\,M_{2},\,M_{3}$ are three complex reflections of order
$p$, which satisfy

$br(M_{1},M_{2})=4,\quad br(M_{1},M_{3})=br(M_{2},M_{3})=3,\quad
br(M_{1},M_{3}^{-1}M_{2}M_{3})=5.$

Actually, $M_{1},\,M_{2},\,M_{3}$ will be Thompson group ${\bf S}_{2}$. Write
$R_{1}=M_{2}^{-1}M_{1}M_{2}$, $R_{2}=M_{1}M_{2}M_{1}^{-1}$, $R_{3}=M_{3}$. We
claim that

$br(R_{1},R_{2})=br(R_{1},R_{3}^{-1}R_{2}R_{3})=4,\quad
br(R_{1},R_{3})=br(R_{2},R_{3})=5.$

First, observe, we also have
$br(M_{2}^{-1}M_{1}M_{2},M_{3})=br(M_{1}^{-1}M_{2}M_{1},M_{3})=5$. Thus

$br(R_{1},R_{3})=br(M_{2}^{-1}M_{1}M_{2},M_{3})=5,\quad
br(R_{2},R_{3})=br(M_{1}M_{2}M_{1}^{-1},M_{3})=5.$

Using $br(M_{1},M_{2})=4$, we have

$R_{1}R_{2}=(M_{2}^{-1}M_{1}M_{2})(M_{1}M_{2}M_{1}^{-1})=M_{2}^{-1}(M_{2}M_{1}M_{2}M_{1})M_{1}^{-1}=M_{1}M_{2}.$

Hence $br(R_{1},R_{2})=br(M_{1},M_{2})=4.$ We denote $M_{1}$, $M_{1}^{-1}$ by
$1$, $\bar{1}$ simply and so on. Now we consider

$\displaystyle R_{1}R_{3}^{-1}R_{2}R_{3}$
$\displaystyle=M_{2}^{-1}M_{1}M_{2}M_{3}^{-1}M_{1}M_{2}M_{1}^{-1}M_{3}$
$\displaystyle=\bar{2}12\bar{3}12\bar{1}3$
$\displaystyle=(123123)(\bar{3}\bar{2}\bar{1}\bar{3}\bar{2}\bar{1}\cdot\bar{2}12\bar{3}12\bar{1}3\cdot
123123)\bar{3}\bar{2}\bar{1}\bar{3}\bar{2}\bar{1}$
$\displaystyle=(123123)(\bar{3}\bar{2}\bar{1}\bar{3}\cdot
12\bar{1}\cdot\bar{3}12\bar{1}3\cdot
123123)\bar{3}\bar{2}\bar{1}\bar{3}\bar{2}\bar{1}$
$\displaystyle=(123123)(\bar{3}\bar{2}(\bar{1}\bar{3}1)2(\bar{1}\bar{3}1)2(\bar{1}31)23123)\bar{3}\bar{2}\bar{1}\bar{3}\bar{2}\bar{1}$
$\displaystyle=(123123)(\bar{3}\bar{1}31\bar{2}\bar{1}\bar{3}13123)\bar{3}\bar{2}\bar{1}\bar{3}\bar{2}\bar{1}$
$\displaystyle=(123123)(1\bar{3}\bar{2}323)\bar{3}\bar{2}\bar{1}\bar{3}\bar{2}\bar{1}$
$\displaystyle=(123123)(12)\bar{3}\bar{2}\bar{1}\bar{3}\bar{2}\bar{1}$

Since $R_{1}R_{3}^{-1}R_{2}R_{3}$ is conjugate to $M_{1}M_{2}$ we see that
$br(R_{1},R_{3}^{-1}R_{2}R_{3})=br(M_{1},M_{2})=4$ as claimed. In particular
this shows that this case is equivalent to Thompson groups ${\bf S}_{2}$. 2\.
$(n,m)=(4,3)$
In this case, it is easy to check that $R_{2}$, $R_{3}$ (also $R_{3}$,
$R_{1}$) braid with length 4, $R_{1}$, $R_{2}$ (also $R_{1}$,
$R_{3}^{-1}R_{2}R_{3}$) braid with length 3, $R_{1}$, $R_{2}R_{3}R_{2}^{-1}$
(also $R_{3}$, $R_{1}R_{2}R_{1}^{-1}$) braid with length 2 (i.e. they commute)
and $R_{1}R_{2}R_{3}$ is regular elliptic of order 3. Note that
$\textrm{Det}(H)<0$ when $p\geqslant 4$.

As the same fashion in [4], we define $\iota$ by the reflection of group that
acts on the generating set ($R_{1},\,R_{2},\,R_{3}$) as follows,

$\iota(R_{1})=R_{1},\quad\iota(R_{2})=R_{1}R_{2}R_{1}^{-1},\quad\iota(R_{3})=R_{3}.$

Under the action of $\iota$, the (4, 4, 3; 3)-triangle groups will be sent to
the triangle groups with braiding (2, 3, 4; 4)

$\displaystyle\langle\iota(R_{1}),\,\iota(R_{2})\,\iota(R_{3}):$
$\displaystyle\iota(R_{2}R_{3})=\iota(R_{3}R_{2}),(\iota(R_{1}R_{2}))^{\frac{3}{2}}=(\iota(R_{2}R_{1}))^{\frac{3}{2}},$
$\displaystyle(\iota(R_{1}R_{3}))^{2}=(\iota(R_{3}R_{1}))^{2},$
$\displaystyle(\iota(R_{1}R_{2}R_{3}R_{2}^{-1}))^{2}=(\iota(R_{2}R_{3}R_{2}^{-1}R_{1}))^{2}\rangle.$

Recall the Mostow groups $\Gamma(p,t)$ mentioned in [7, 9]. For Mostow groups,
there exists a complex hyperbolic isometry $J$ of order 3 so that
$R_{j+1}=JR_{j}J^{-1}$ and $R_{i}R_{i+1}R_{i}=R_{i+1}R_{i}R_{i+1}$. We could
rewrite them as triangle groups with braiding (2, 3, 4; 4) as follows

$\displaystyle\langle R_{1},R_{2},J(R_{1}R_{2})^{-1}:$ $\displaystyle
R_{2}J(R_{1}R_{2})^{-1}=J(R_{1}R_{2})^{-1}R_{2},(R_{1}R_{2})^{\frac{3}{2}}=(R_{2}R_{1})^{\frac{3}{2}},$
$\displaystyle(R_{1}J(R_{1}R_{2})^{-1})^{2}=(J(R_{1}R_{2})^{-1}R_{1})^{2},$
$\displaystyle(R_{1}R_{2}J(R_{1}R_{2})^{-1}R_{2}^{-1})^{2}=(R_{2}J(R_{1}R_{2})^{-1}R_{2}^{-1}R_{1})^{2}\rangle.$

Acknowledgement This research is carried out while John Parker was visiting
Department of Mathematical and Computing Sciences at Tokyo Institute of
Technology. The first author is supported by a JSPS Invitation Fellowship. The
second author is supported by Grant-in-Aid Scientific Research (B) No.
15H03619 (Sadayoshi Kojima, Tokyo Institute of Technology) and Grant-in-Aid
Scientific Research (S) No. 24224002 (Takashi Tsuboi, University of Tokyo). We
also thank Martin Deraux for several useful conversations.

## References

  * [1] M. Deraux, J. R. Parker, and J. Paupert, On commensurability classes of non-arithmetic complex hyperbolic lattices. arXiv:1611.00330.
  * [2] W. M. Goldman, Complex hyperbolic geometry, Oxford University Press, 1999.
  * [3] W. M. Goldman and J. R. Parker, Complex hyperbolic ideal triangle groups, J. Reine Angew. Math., 425 (1992), pp. 71–86.
  * [4] S. Kamiya, J. R. Parker, and J. M. Thompson, Notes on complex hyperbolic triangle groups, Conform. Geom. Dyn., 14 (2010), pp. 202–218.
  * [5] , Non-discrete complex hyperbolic triangle groups of type $(n,n,\infty;k)$, Canad. Math. Bull., 55 (2012), pp. 329–338.
  * [6] A. Monaghan, Complex hyperbolic triangle groups.  Doctoral thesis, 2013.
  * [7] G. D. Mostow, On a remarkable class of polyhedra in complex hyperbolic space, Pacific J. Math., 86 (1980), pp. 171–276.
  * [8] J. R. Parker, Complex hyperbolic kleinian groups.  Preprint.
  * [9] J. R. Parker, Unfaithful complex hyperbolic triangle groups. I. Involutions, Pacific J. Math., 238 (2008), pp. 145–169.
  * [10] J. R. Parker and J. Paupert, Unfaithful complex hyperbolic triangle groups. II. Higher order reflections, Pacific J. Math., 239 (2009), pp. 357–389.
  * [11] L.-J. Sun, Notes on complex hyperbolic triangle groups of type $(m,n,\infty)$.  To appear in Advances in Geometry.
  * [12] J. M. Thompson, Complex hyperbolic triangle groups.  Doctoral thesis, 2010.
  * [13] J. O. Wyss-Gallifent, Complex hyperbolic triangle groups, ProQuest LLC, Ann Arbor, MI, 2000.  Thesis (Ph.D.)–University of Maryland, College Park.