# ON THE EIGENVALUES OF THE LAPLACIAN ON ELLIPSOIDS OBTAINED AS PERTURBATION
OF UNIT SPHERE

ANANDATEERTHA MANGASULI AND ADITYA TIWARI

Abstract. We study the eigenvalues of the Laplacian on ellipsoids that are
obtained as perturbations of the standard Euclidean unit sphere in dimension
two. A comparison of these eigenvalues with those of the standard Euclidean
unit sphere is obtained under a Gaussian curvature condition, in line with the
Lichnerowicz theorem on the first positive eigenvalue on a compact Riemannian
manifold.

##  1\. Introduction

Let $(N,g)$ be a compact $n$-Riemannian manifold without boundary. Then the
Laplacian in local coordinate system $(U,x^{1},x^{2},\dots,x^{n})$ is given by

$-\Delta_{g}:=-\frac{1}{\sqrt{|det(g)|}}{\sum_{i,j=1}^{n}}\frac{\partial}{\partial
x^{i}}\bigg{(}g^{ij}\sqrt{|det(g)|}\frac{\partial}{\partial x^{j}}\bigg{)},$

where $(g^{ij})$ is the inverse of the Riemannian metric $g=(g_{ij})$ in this
coordinate system and the Laplacian has a sequence of the eigenvalues ordered
as below:

$0=\Lambda_{0}<\Lambda_{1}\leq\Lambda_{2}\leq\Lambda_{3}\leq\cdots\cdots\cdots\leq\Lambda_{n}\leq\cdots\nearrow+\infty,$

where each eigenvalue has finite multiplicity with corresponding eigenspaces
orthogonal to each other and the direct sum of these eigenspaces is
$L^{2}(N)$. Lichnerowicz in his famous theorem states that

######  Theorem 1.1 (Lichnerowicz).

Let $(N,g)$ be a compact Riemannian $n$-manifold. If there exists a positive
number $k$ such that

$Ric_{g}\geq kg,$

where $Ric_{g}$ being the Ricci tensor of $g$, then

$\Lambda_{1}(g)\geq\frac{n}{n-1}k,$

where $\Lambda_{1}(g)$ is the first non-zero eigenvalue of the Laplacian
acting on smooth functions on $N$.

The spectrum of the Laplacian on $S^{n}$ is explicitly known with the first
eigenvalue being $n$. Therefore, the Lichnerowicz theorem compares the first
eigenvalue of the Laplacian on any compact $n$-manifold with the first
eigenvalue of the Laplacian on the standard unit sphere $(S^{n},g_{0})$.
Suppose $(\mathcal{M},g)$ is a compact $n$-manifold such that

$Ric_{g}\geq(n-1)g,$

then we have $\Lambda_{1}(g)\geq\Lambda_{1}(g_{0})$. Obata showed that if the
first eigenvalue of the Laplacian on $(\mathcal{M},g)$ is $n$, then
$(\mathcal{M},g)$ must be isometric to $(S^{n},g_{0})$. This comparison of the
first eigenvalue gave a significant amount of information about the Riemannian
manifold. Hence, it raises an interest to compare the higher eigenvalues of
the Laplacian. A natural question which can be asked is “Does the inequality
$\Lambda_{i}(g)\geq\Lambda_{i}(g_{0})$ holds for all $i\geq 0$?” in the above
case.
The answer to the above question in general is NO. It was shown by Donnelly in
[2]. He constructed a counterexample in dimension four where the above
eigenvalue inequality fails. Furthermore, the methods used in constructing the
above example can be adopted in higher dimensions but not in dimensions two
and three, and hence it became interesting to know about the inequalities of
the higher eigenvalues in dimensions two and three. In [5], Mangasuli showed
that for all $j\geq 1$, the above inequality holds for left-invariant metrics
on $S^{3}$. Furthermore, in [4], he showed that if the underlying manifold is
$S^{2}$, then this inequality is also satisfied for certain metrics
sufficiently close to $g_{0}$, which are obtained by analytically perturbing
$g_{0}$ through rotationally symmetric and conformal metrics.
In this article, we show given $L\geq 1$, there exist $j\in\mathbb{N}$ such
that the above inequality holds for all $i\leq j$, when the underlying
manifold is an ellipsoid with the Gaussian curvature greater than or equal to
one at every point.
The first section contains a motivation behind the problem, while the notation
and some calculations regarding curvature are discussed in section two. The
article concludes with stating and proving the main result in section three.

##  2\. Ellipsoid

In case of $2$-manifolds, $Ric_{g}$ equals to $Kg$, where $K$ denotes the
Gaussian curvature. Therefore, $2$-manifolds with $K\geq 1$, satisfy
$Ric_{g}\geq(2-1)g$. In this section, we define the notations and obtain
certain conditions on semi-axes lengths for the Gaussian curvature of the
ellipsoid to be greater than equal to one at all points. An ellipsoid is a
2-dimensional manifold represented as:

$\bigg{\\{}(x,y,z)\in\mathbb{R}^{3}\bigg{|}\frac{x^{2}}{A^{2}}+\frac{y^{2}}{B^{2}}+\frac{z^{2}}{C^{2}}=1\bigg{\\}},$

where $A$, $B$ and $C$ $>$ 0 are semi-axes lengths of the ellipsoid. We will
use the following notation to represent an ellipsoid. Given $A,B$ and $C>0$,

$E_{A,B,C}:=\bigg{\\{}(x,y,z)\in\mathbb{R}^{3}\bigg{|}\frac{x^{2}}{A^{2}}+\frac{y^{2}}{B^{2}}+\frac{z^{2}}{C^{2}}=1\bigg{\\}}.$

When all the semi-axes lengths are unequal, the corresponding ellipsoid is
called a triaxial ellipsoid. When any of the two semi-axes lengths are equal,
we will call the ellipsoid a biaxial ellipsoid. When all are equal, it is a
sphere. Let $p=(x,y,z)\in E_{A,B,C}$, then the Gaussian curvature at $p$ is
given by

(1)
$K(p)=\frac{1}{A^{2}B^{2}C^{2}(\frac{x^{2}}{A^{4}}+\frac{y^{2}}{B^{4}}+\frac{z^{2}}{C^{4}})^{2}},$

where $A,B$ and $C>0.$ We will find the minimum of the Gaussian curvature on
the ellipsoid and use it to compute the condition on axes lengths for $K\geq
1$.

###  2.1. Maxima and Minima of Curvature on $E_{A,B,C}$:

Using the method of Lagrange’s multipliers to find the minimum curvature on
the ellipsoid $E_{A,B,C}$, we get the points of extrema as

$\\{(\pm A,0,0),(0,\pm B,0),(0,0,\pm C)\\},$

with corresponding extrema values as

$h(\pm A,0,0)=\frac{A^{2}}{B^{2}C^{2}},$ $h(0,\pm
B,0)=\frac{B^{2}}{A^{2}C^{2}},$ $h(0,0,\pm C)=\frac{C^{2}}{A^{2}B^{2}}.$

In order to obtain the minimum curvature, we specify an ordering of the axes
lengths $A$, $B$ and $C$. Suppose $A\geq B\geq C>0$. Then,

(2) $\displaystyle K_{max}(E_{A,B,C}):=\max_{p\in E_{A,B,C}}K(p)=K(\pm
A,0,0)=\max_{(x,y,z)\in S}h(x,y,z)=\frac{A^{2}}{B^{2}C^{2}},$ (3)
$\displaystyle K_{min}(E_{A,B,C}):=\min_{p\in E_{A,B,C}}K(p)=K(0,0,\pm
C)=\min_{(x,y,z)\in S}h(x,y,z)=\frac{C^{2}}{A^{2}B^{2}}.$

We revise our notation as follows: $E_{A,B,C}$ will denote all the ellipsoids
with semi-axes lengths $A\geq B\geq C>0$ i.e; $E_{A,B,C}$ will represent six
different ellipsoids. The minimum and maximum curvatures of these ellipsoids
are given as, $K_{min}(E_{A,B,C})=\frac{C^{2}}{A^{2}B^{2}}$ and
$K_{max}(E_{A,B,C})=\frac{A^{2}}{B^{2}C^{2}}$. With the revised notation,
there will be two types of biaxial ellipsoids $E_{A,B,B}$ and $E_{A,A,B}$,
where $A\geq B>0$.

###  2.2. Condition on semi-axes lengths for $K_{min}\geq 1$.

####  2.2.1. Biaxial ellipsoids of type $E_{A,B,B}$.

We have $K_{min}(E_{A,B,B})=\frac{1}{A^{2}}.$ Since $A>B>0$, therefore

(4) $K_{min}(E_{A,B,B})\geq 1\iff 1\geq A>B>0.$

####  2.2.2. Biaxial ellipsoids of type $E_{A,A,B}$.

We have $K_{min}(E_{A,A,B})=\frac{B^{2}}{A^{4}}.$ Therefore,

(5) $K_{min}(E_{A,A,B})\geq 1\iff 1\geq A>B\geq A^{2}>0.$

####  2.2.3. Triaxial ellipsoids of type $E_{A,B,C}$.

We have $K_{min}(E_{A,B,C})=\frac{C^{2}}{A^{2}B^{2}}.$ Therefore,

(6) $K_{min}(E_{A,B,C})\geq 1\iff 1\geq A>B>C\geq AB>0.$

###  2.3. Ellipsoids as perturbations of Sphere

Let $A:=1+\alpha\epsilon\geq$ $B:=1+\beta\epsilon\geq$ $C:=1+\gamma\epsilon>0$
for $\epsilon>0$ where $\alpha,\beta$ and $\gamma$ are fixed real numbers.
Then the family of ellipsoids represented as below

$E_{A,B,C}(\epsilon):=\bigg{\\{}(x,y,z)\in\mathbb{R}^{3}\bigg{|}\frac{x^{2}}{(1+\alpha\epsilon)^{2}}+\frac{y^{2}}{(1+\beta\epsilon)^{2}}+\frac{z^{2}}{(1+\gamma\epsilon)^{2}}=1\bigg{\\}},$

is close to $E_{1,1,1}$ for $\epsilon$’s sufficiently close $0$ with
$E_{A,B,C}(0)=E_{1,1,1}$. For $\epsilon>0$, inequalities $(\ref{A})$,
$(\ref{B})$ and $(\ref{C})$ from the above section implies that

(7) $0\geq\alpha>\beta>\gamma>-1/\epsilon.$

Using inequalities derived in the above section and some calculus, we will
compute sufficient condition on $\alpha,\beta,$ $\gamma$ and $\epsilon$ for
the Gaussian curvature to be greater than equal to one at all points in
$E_{A,B,C}(\epsilon)$. We will write $K_{min}(E_{A,B,C})=:K_{min}(\epsilon)$.

####  2.3.1. Biaxial ellipsoids of type $E_{A,B,B}(\epsilon)$.

Inequality $(\ref{A})$ and $(\ref{ABC})$ implies
$0\geq\alpha>\beta>-1/\epsilon$. Since
$K_{min}(\epsilon)=\frac{1}{(1+\alpha\epsilon)^{2}}$ and $\epsilon>0$,
therefore

(8) $K_{min}(\epsilon)\geq
1\;\text{and}\;A>B>0\iff\quad\begin{cases}\alpha=0>\beta\;\text{and}\;0<\epsilon<-1/\beta,\\\
0>\alpha>\beta\;\text{and}\;\alpha/\beta<\epsilon<-1/\beta.\end{cases}$

####  2.3.2. Biaxial ellipsoids of type $E_{A,A,B}(\epsilon)$.

Since
$K_{min}(\epsilon)=\frac{(1+\epsilon\beta)^{2}}{(1+\epsilon\alpha)^{4}}$,
therefore

(9) $K_{min}(\epsilon)\geq
1\;\text{and}\;A>B>0\iff\quad\alpha<0\;\text{and}\;2\alpha<\beta<\alpha\;\text{and}\;0<\epsilon\leq\frac{\beta-2\alpha}{\alpha^{2}}.$

####  2.3.3. Triaxial ellipsoids

$K_{min}(\epsilon)=\frac{(1+\gamma\epsilon)^{2}}{(1+\beta\epsilon)^{2}(1+\gamma\epsilon)^{2}}$.
Therefore,

(10) $K_{min}(\epsilon)\geq 1\;\text{and}\;A>B>C>0\iff
0>\alpha>\beta>\gamma>\alpha+\beta\;\text{and}\;0<\epsilon\leq\frac{\gamma-\alpha-\beta}{\alpha\beta},$

where $A=1+\alpha\epsilon$, $B=1+\beta\epsilon$ and $C=1+\gamma\epsilon$.

##  3\. Comparison of the Eigenvalues

In this section, we will state and prove the main theorem using results from
[3].

###  3.1. Biaxial Ellipsoids

Assume $A,B>0$ and $E_{A,A,B}$ be the biaxial ellipsoid with no-ordering in
$A$ and $B$. Also, assume that $A=1+\alpha\epsilon$ and $B=1+\beta\epsilon$
and $-\Delta_{g}$ is the positive Laplace Beltrami operator on $E_{A,A,B}$.
Let $g_{\epsilon}$ be the Riemannian metric obtained by restricting the
Euclidean metric from $\mathbb{R}^{3}$ to $E_{A,A,B}$. With these notations,
Theorem 2 of [3] is restated as follows.

######  Theorem 3.1.

Given $L\in\mathbb{N}$ and $\alpha$, $\beta\in\mathbb{R}$. There exists
$\epsilon_{0}(\alpha,\beta,L)$ such that for all $0<\epsilon<\epsilon_{0}$ and
$\Lambda(\epsilon)\in spec(-\Delta_{g_{\epsilon}})\cap[0,L(L+1)]$, we have

(11) $\Lambda(\epsilon)=l(l+1)+\lambda_{1}\epsilon+O(\epsilon^{2}),$

for $l=0,1,2,\dots,L$ and $m=-l,\dots,l$ with $\lambda_{1}$ given by the
formula

(12) $\lambda_{1}=-2\alpha
l(l+1)+(\alpha-\beta)\frac{2l(l+1)}{(2l+3)(2l-1)}(2l^{2}-2m^{2}+2l-1).$

Moreover, each $\Lambda(\epsilon)$ has multiplicity two except for those where
$m=0$, which in this case corresponds to multiplicity one.

We define a new notation for the eigenvalues of the ellipsoid
$E_{A,A,B}(\epsilon)$ with above theorem in mind. For $L\in\mathbb{N}$ and
$\alpha,\beta\in\mathbb{R}$, write $\Lambda_{l}^{m}(\epsilon)$ for each
eigenvalue corresponding to $l\in\\{0,1,\dots,L\\}$ and
$m\in\\{-l,\dots,0,\dots,l\\}$ and the corresponding first order term
$\lambda_{1}$ by $\lambda_{1}(l,m)$.

######  Remark 1.

In Theorem $(\ref{Perturbegvalue})$, the semi-axes lengths are not ordered.
Therefore, for ellipsoids of the type $E_{A,B,B}$, by interchanging $\alpha$
and $\beta$, $\lambda_{1}(l,m)$ is given by

(13) $\lambda_{1}(l,m)=-2\beta
l(l+1)+(\beta-\alpha)\frac{2l(l+1)}{(2l+3)(2l-1)}(2l^{2}-2m^{2}+2l-1).$

We prove the following lemmas to be used later in proving the main theorem.

######  Lemma 3.1.1.

Let $E_{A,A,B}(\epsilon)$ be a family of biaxial ellipsoids with the Gaussian
curvature greater than equal to one at all points. Given any $L\in\mathbb{N}$,
there exists an $\epsilon_{1}(A,B,L)>0$, such that for all
$0<\epsilon<\epsilon_{1}$

$\lambda_{1}(l,m)\geq
0,\qquad\text{for}\;\;\text{all}\;\;l=0,1,\dots,L,\qquad\text{and}\quad
m-l,\dots,l.$

######  Proof.

Using $(\ref{Bi1})$, we have $0\geq\alpha>\beta$ as $K_{min}(\epsilon)\geq 1$.
Therefore

$\lambda_{1}(l,m)=-2\alpha
l(l+1)+(\alpha-\beta)\frac{2l(l+1)}{(2l+3)(2l-1)}(2l^{2}-2m^{2}+2l-1)\geq 0,$

for all $l=0,1,\dots,L$ and $m=-l,\dots,0,\dots,l$.

######  Lemma 3.1.2.

Let $E_{A,B,B}(\epsilon)$ be a family of biaxial ellipsoids with the Gaussian
curvature greater than equal to one at all points. Given any $L\in\mathbb{N}$,
there exists an $\epsilon_{1}(A,B,L)>0,$ such that for all
$\epsilon<\epsilon_{1}$

$\lambda_{1}(l,m)\geq
0,\qquad\text{for}\;\;\text{all}\;\;l=0,1,\dots,L,\qquad\text{and}\quad
m=-l,\dots,l.$

######  Proof.

We have $0>\alpha>\beta$ by $(\ref{Bi2})$ as $K_{min}(\epsilon)\geq 1$.
Writing $\beta=\alpha-\delta$ where $\delta>0$. Using $(\ref{evalue2})$ and
replacing $\beta$ with $\alpha-\delta$, we have

$\lambda_{1}(l,m)=-2\alpha l(l+1)+2\delta
l(l+1)\frac{(2l^{2}+2l-2+2m^{2})}{(2l+3)(2l-1)}\geq 0.$

###  3.2. Triaxial ellipsoids

We will restate Theorem $4$ of [3]. Assume $\epsilon>0$, $A=1+\alpha\epsilon$,
$B=1+\beta\epsilon$ and $C=1+\gamma\epsilon$ such that $A,B$ and $C>0$.

######  Theorem 3.2.

Given $L\in\mathbb{N}$ and $\alpha$, $\beta$, $\gamma\in\mathbb{R}$ with
atleast one being non-zero and $g_{\epsilon}$ the metric from $\mathbb{R}^{3}$
restricted to $E_{A,B,C}$. Then there exist
$\epsilon_{0}(\alpha,\beta,\gamma,L)$ such that for all
$0<\epsilon<\epsilon_{0}$ and $\Lambda(\epsilon)\in$
spec($-\Delta_{g}$)$\cap[l(l+1)-2l,l(l+1)+2l)]$, we have

(14) $\Lambda(\epsilon)=l(l+1)+\lambda_{1}\epsilon+\mathcal{O}(\epsilon^{2}),$

where $\lambda_{1}$ is an eigenvalue of $2l+1$ eigenvalues of the following
symmetric tridiagonal matrices denoted by $M_{\cos,e}$, $M_{\cos,o}$,
$M_{\sin,e}$ and $M_{\sin,o}$.

We need some notations to define these matrices. Define

$l_{e}:=\begin{cases}l&,\text{if $l$ is even},\\\ l-1&,\text{if $l$ is
odd}.\end{cases};\qquad l_{o}:=\begin{cases}l&,\text{if $l$ is odd},\\\
l-1&,\text{if $l$ is even}.\end{cases}$

Then $M_{\cos,e}$ is given by

$M_{\cos,e}:=\begin{bmatrix}a_{0}&&b_{0}&&0&&\dots&&0\\\
b_{0}&&a_{1}&&b_{1}&&&&0\\\ 0&&b_{1}&&a_{2}&&&&\vdots\\\
\vdots&&&&&&\ddots&&\vdots\\\ &&&&&&&&b_{k-1}\\\
0&&\dots&&&&b_{k-1}&&a_{k}\end{bmatrix}_{(k+1)\times(k+1)},$

where $k$ is such that $l_{e}=2k$ i.e; $l=2k$ or $2k+1$ and $M_{\sin,e}$ is
given as

$M_{\sin,e}:=\begin{bmatrix}a_{1}&&b_{1}&&0&&\dots&&0\\\
b_{1}&&a_{2}&&b_{2}&&&&0\\\ 0&&b_{2}&&a_{3}&&&&\vdots\\\
\vdots&&&&&&\ddots&&\vdots\\\ &&&&&&&&b_{k-1}\\\
0&&\dots&&&&b_{k-1}&&a_{k}\end{bmatrix}_{k\times k},$

where $k$ is such that $l_{e}=2k$ i.e; $l=2k$ or $2k+1$ and

(15) $\displaystyle-a_{p}$ $\displaystyle=2\gamma
l(l+1)+(\alpha+\beta-2\gamma)\frac{2l(l+1)}{(2l-1)(2l+3)}(l^{2}+4p^{2}+l-1),\quad
p=0,1,\dots,k,$ (16) $\displaystyle-b_{0}$
$\displaystyle=\sqrt{2}(\beta-\alpha)\frac{l(l+1)}{(2l+3)(2l-1)}\sqrt{(l-1)l(l+1)(l+2)},$
(17) $\displaystyle-b_{p}$
$\displaystyle=(\beta-\alpha)\frac{l(l+1)}{(2l+3)(2l-1)}\sqrt{(l-2p-1)(l-2p)(l+2p+1)(l+2p+2)},\quad
p=1,\dots,k-1.$

When $l=2k+1$, $M_{\cos,o}$ is given by

$M_{\cos,o}:=\begin{bmatrix}{\phi}_{0}&&{\psi}_{0}&&0&&\dots&&0\\\
{\psi}_{0}&&{\phi}_{1}&&{\psi}_{1}&&&&0\\\
0&&{\psi}_{1}&&{\phi}_{2}&&&&\vdots\\\ \vdots&&&&&&\ddots&&\vdots\\\
&&&&&&&&{\psi}_{k-1}\\\
0&&\dots&&&&{\psi}_{k-1}&&{\phi}_{k}\end{bmatrix}_{(k+1)\times(k+1)},$

and when $l=2k$, $M_{\cos,o}$ is given by

$M_{\cos,o}:=\begin{bmatrix}{\phi}_{0}&&{\psi}_{0}&&0&&\dots&&0\\\
{\psi}_{0}&&{\phi}_{1}&&{\psi}_{1}&&&&0\\\
0&&{\psi}_{1}&&{\phi}_{2}&&&&\vdots\\\ \vdots&&&&&&\ddots&&\vdots\\\
&&&&&&&&{\psi}_{k-2}\\\
0&&\dots&&&&{\psi}_{k-2}&&{\phi}_{k-1}\end{bmatrix}_{k\times k}.$

$M_{\sin,o}$ is given by

$M_{\sin,o}:=\begin{bmatrix}\tilde{{\phi}_{0}}&&{\psi}_{0}&&0&&\dots&&0\\\
{\psi}_{0}&&{\phi}_{1}&&{\psi}_{1}&&&&0\\\
0&&{\psi}_{1}&&{\phi}_{2}&&&&\vdots\\\ \vdots&&&&&&\ddots&&\vdots\\\
&&&&&&&&{\psi}_{k-1}\\\
0&&\dots&&&&{\psi}_{k-1}&&{\phi}_{k}\end{bmatrix}_{(k+1)\times(k+1)},$

when $l=2k+1$ and

$M_{\sin,o}:=\begin{bmatrix}\tilde{{\phi}_{0}}&&{\psi}_{0}&&0&&\dots&&0\\\
{\psi}_{0}&&{\phi}_{1}&&{\psi}_{1}&&&&0\\\
0&&{\psi}_{1}&&{\phi}_{2}&&&&\vdots\\\ \vdots&&&&&&\ddots&&\vdots\\\
&&&&&&&&{\psi}_{k-2}\\\
0&&\dots&&&&{\psi}_{k-2}&&{\phi}_{k-1}\end{bmatrix}_{k\times k},$

when $l=2k$, where

(18) $\displaystyle-\phi_{0}$
$\displaystyle=(\frac{3\alpha}{2}+\frac{\beta}{2}-2\gamma)\frac{2l^{2}(l+1)^{2}}{(2l-1)(2l+3)}+2l(l+1)\gamma,$
(19) $\displaystyle-\tilde{\phi_{0}}$
$\displaystyle=(\frac{3\beta}{2}+\frac{\alpha}{2}-2\gamma)\frac{2l^{2}(l+1)^{2}}{(2l-1)(2l+3)}+2l(l+1)\gamma,$
(20)
$\displaystyle-\phi_{p}=(\alpha+\beta)\frac{2l(l+1)}{(2l+3)(2l-1)}(l^{2}+4p^{2}+4p+l)$
$+2\gamma\frac{l(l+1)}{(2l-1)(2l+3)}(2l^{2}-8p^{2}-8p+2l-3),\qquad
p=1,\dots,k,$ (21) $\displaystyle-\psi_{p}$
$\displaystyle=\frac{(\beta-\alpha)l(l+1)}{(2l+3)(2l-1)}\sqrt{(l-2p-2)(l-2p-1)(l+2p+2)(l+2p+3)},\quad\;\;p=1,\dots,k-1.$

####  3.2.1. Positive Definiteness of Tridiagonal Matrices.

A real symmetric matrix is positive definite if determinant of all principal
minors is positive. Milica Andelie and C.M. da Fonseca in [1] has stated the
following result for positive definiteness of real symmetric matrices with
positive diagonal entries.

######  Theorem 3.3.

Let $A_{n}$ be a real symmetric tridiagonal matrix with positive diagonal
entries defined as:

$A_{n}:=\begin{bmatrix}r_{1}&&s_{1}&&0&&\dots&&0\\\
s_{1}&&r_{2}&&s_{2}&&&&0\\\ 0&&s_{2}&&r_{3}&&&&\vdots\\\
\vdots&&&&&&\ddots&&\vdots\\\ &&&&&&&&s_{n-1}\\\
0&&\dots&&&&s_{n-1}&&r_{n}\end{bmatrix}_{n\times n}.$

If

(22)
${s_{i}}^{2}<\frac{1}{4}r_{i}r_{i+1}\frac{1}{\cos^{2}{(\frac{\pi}{n+1})}},\quad\text{for}\;\;i=1,\dots,n-1,$

then $A_{n}$ is positive definite.

We will show that the matrices $M_{\cos,e}$, $M_{\cos,o}$, $M_{\sin,e}$ and
$M_{\sin,o}$ have positive diagonal entries given the curvature conditions are
satisfied and then prove the positive definiteness using the above theorem
along with curvature conditions.

######  Lemma 3.3.1.

Given $\alpha,\beta$ and $\gamma<0$, $\tilde{\phi_{0}}$, $a_{p}$ and
$\phi_{p}$ are positive for all possible entries of $p$ in above matrices.

######  Proof.

Rewriting $a_{p}$ and $\phi_{p}$ we have,

(23)
$-a_{p}=(\alpha+\beta)\frac{2l(l+1)}{(2l-1)(2l+3)}(l^{2}+4p^{2}+l-1)+2\gamma\frac{l(l+1)}{(2l-1)(2l+3)}(2l^{2}-8p^{2}+2l-1),$

for $p=0,1,\dots,k$, and

(24)
$-\phi_{p}=(\alpha+\beta)\frac{2l(l+1)}{(2l+3)(2l-1)}(l^{2}+4p^{2}+4p+l)+2\gamma\frac{l(l+1)}{(2l-1)(2l+3)}(2l^{2}-8p^{2}-8p+2l-3),$

for $p=1,\dots,k.$ Since $l=2k$ or $2k+1$ and $\alpha,\beta,\gamma<0$,
therefore

$a_{p}>0,\quad\text{for}\quad p=0,\dots,k,$

and

$\phi_{p}>0,\quad\text{for}\quad p=1,\dots,k.$

Rewriting $\tilde{\phi_{0}}$ and $\phi_{0}$, we have

(25)
$-\tilde{\phi_{0}}=(\frac{3\beta}{2}+\frac{\alpha}{2})\frac{2l^{2}(l+1)^{2}}{(2l-1)(2l+3)}+2\gamma\frac{2l^{2}(l+1)^{2}}{(2l-1)(2l+3)}(2l^{2}+2l-3),$

(26)
$-\phi_{0}=(\frac{3\alpha}{2}+\frac{\beta}{2})\frac{2l^{2}(l+1)^{2}}{(2l-1)(2l+3)}+2\gamma\frac{2l^{2}(l+1)^{2}}{(2l-1)(2l+3)}(2l^{2}+2l-3).$

Here $l=2k$ or $2k+1$ and $\alpha,\beta,\gamma<0$. Therefore,

$\phi_{0}>0\quad\text{and}\quad\tilde{\phi_{0}}>0.$

The order of these matrices depend on $l$ and is given as

$l$ | $M_{\cos,e}$ | $M_{\sin,e}$ | $M_{\cos,o}$ | $M_{\sin,o}$
---|---|---|---|---
$2k+1$ | $(k+1)\times(k+1)$ | $k\times k$ | $(k+1)\times(k+1)$ | $(k+1)\times(k+1)$
$2k$ | $(k+1)\times(k+1)$ | $k\times k$ | $k\times k$ | $k\times k$

We will write the order of these matrices for some values of $l$.

$l$ | $M_{\cos,e}$ | $M_{\sin,e}$ | $M_{\cos,o}$ | $M_{\sin,o}$
---|---|---|---|---
$1$ | $1\times 1$ | - | $1\times 1$ | $1\times 1$
$2$ | $2\times 2$ | $1\times 1$ | $1\times 1$ | $1\times 1$
$3$ | $2\times 2$ | $1\times 1$ | $2\times 2$ | $2\times 2$
$4$ | $3\times 3$ | $2\times 2$ | $2\times 2$ | $2\times 2$
$5$ | $3\times 3$ | $2\times 2$ | $3\times 3$ | $3\times 3$
$6$ | $4\times 4$ | $3\times 3$ | $3\times 3$ | $3\times 3$

Since all the diagonal entries are positive, matrices of order $1\times 1$ are
positive definite. Therefore, $M_{\cos,e}$ is positive definite for $l=1$,
$M_{\sin,e}$ is positive definite for $l=2$ and $3$, $M_{\cos,o}$ and
$M_{\sin,o}$ are positive definite for $l=1$ and $2$. For matrices of order
$2\times 2$, we will show that the determinant is positive and hence the
matrices are positive definite. And for matrices of order greater than
$2\times 2$, we will use Theorem $(\ref{PDM})$ to prove the positive
definiteness as diagonal entries are positive.

######  Lemma 3.3.2.

$M_{\cos,e}$ is positive definite for all $l=1,2,\dots,L$.

######  Proof.

Using $(\ref{Me})$, we have

(27)
$\frac{4b_{0}^{2}}{a_{0}a_{1}}=\frac{2(\beta-\alpha)^{2}(l-1)l(l+1)(l+2)}{\\{(\alpha+\beta)(l^{2}+l-1)+\gamma(2l^{2}+2l-1)\\}*\\{(\alpha+\beta)(l^{2}+l+3)+\gamma(2l^{2}+2l-9)\\}}.$

But $0>\alpha>\beta>\gamma\geq\alpha+\beta$, therefore

(28)
$\begin{split}\frac{4b_{0}^{2}}{a_{0}a_{1}}&\leq\frac{2(\beta-\alpha)^{2}}{\gamma^{2}}\frac{(l-1)l(l+1)(l+2)}{(3l^{2}+3l-2)(3l^{2}+3l-6)}\\\
&\leq\frac{2(\beta-\alpha)^{2}}{9\gamma^{2}}\frac{l(l+1)}{(l-1)(l+2)},\quad\text{for}\quad
l\geq 2\\\ &\leq\frac{4}{9}\frac{(\beta-\alpha)^{2}}{\gamma^{2}}.\\\
\end{split}$

Therefore,

(29)
$\frac{4b_{0}^{2}}{a_{0}a_{1}}\cos^{2}{(\frac{\pi}{k+2})}\leq\frac{4}{9}\frac{(\beta-\alpha)^{2}}{\gamma^{2}}<1.$

For $p>0$, we have

(30)
$\frac{4b_{p}^{2}}{a_{p}a_{p+1}}=\frac{(\beta-\alpha)^{2}(l-2p-1)(l-2p)(l+2p+1)(l+2p+2)}{\\{(\alpha+\beta)(l^{2}+4p^{2}+l-1)+\gamma(2l^{2}-8p^{2}+2l-1)\\}}$
$\times\frac{1}{\\{(\alpha+\beta)(l^{2}+4p^{2}+8p+l+3)+\gamma(2l^{2}-8p^{2}-16p+2l-9)\\}}.$

But $0>\alpha>\beta>\gamma\geq\alpha+\beta$, therefore

(31)
$\begin{split}\frac{4b_{p}^{2}}{a_{p}a_{p+1}}&\leq\frac{(\beta-\alpha)^{2}}{\gamma^{2}}\frac{(l+2p+1)(l-2p)}{9(l-2p-1)(l+2p+2)}\quad\text{for}\quad
l\geq 4,\\\ &\leq\frac{2(\beta-\alpha)^{2}}{9\gamma^{2}}.\end{split}$

Therefore, we have

(32)
$\frac{4b_{p}^{2}}{a_{p}a_{p+1}}\cos^{2}{(\frac{\pi}{k+2})}\leq\frac{2(\beta-\alpha)^{2}}{9\gamma^{2}}<1\quad\text{where}\quad
k\geq 2,$

for $p=1,\dots,k-1$. Therefore, by Theorem $(\ref{PDM})$, $M_{\cos,e}$ is
positive definite for all $l\geq 4$. When $l=3$ or $2$, $M_{\cos,e}$ will be
of order $2\times 2$. We will show that $a_{0}a_{1}-b_{0}^{2}>0$ for $l=2$ and
$l=3$. Using $(\ref{Me})$, we have

$\displaystyle a_{0}a_{1}$ $\displaystyle=$
$\displaystyle\frac{4l^{2}(l+1)^{2}}{(2l-1)^{2}(2l+3)^{2}}\\{(\alpha+\beta)(l^{2}+l-1)+\gamma(2l^{2}+2l-1)\\}\\{(\alpha+\beta)(l^{2}+l+3)+\gamma(2l^{2}+2l-9)\\}$
$\displaystyle>$ $\displaystyle 0.$ $\displaystyle b_{0}^{2}$ $\displaystyle=$
$\displaystyle
2(\beta-\alpha)^{2}\frac{l^{2}(l+1)^{2}}{(2l-1)^{2}(2l+3)^{2}}(l-1)l(l+1)(l+2)>0.$

The determinant $a_{0}a_{1}-b_{0}^{2}$ will be positive if the coefficient of
$\alpha^{2}$ in $a_{0}a_{1}$ is greater than coefficient of $\alpha^{2}$ in
$b_{0}^{2}$. Writing the difference of coefficients, we have

$\frac{l^{2}(l+1)^{2}}{(2l-1)^{2}(2l+3)^{2}}\bigg{(}4(l^{2}+l-1)(l^{2}+l+3)-(l-1)l(l+1)(l+2)\bigg{)}>0\quad\text{for}\quad
l=3\;\text{or}\;4.$

Therefore, $M_{\cos,e}$ is positive definite for all $l\geq 1$.

######  Lemma 3.3.3.

$M_{\sin,e}$ is positive definite for all $l=2,\dots,L$.

######  Proof.

By Theorem $(\ref{PDM})$, $M_{\sin,e}$ will be positive definite if

$4\frac{b_{p}^{2}}{a_{p}a_{p+1}}\cos^{2}{(\frac{\pi}{k+1})}<1\quad\text{for}\quad
p=1,\dots,k-1.$

But $(\ref{apap1})$ implies that

$\frac{4b_{p}^{2}}{a_{p}a_{p+1}}\leq\frac{(\beta-\alpha)^{2}}{\gamma^{2}}\quad\text{for}\quad
l\geq 4.$

So, we have

$4\frac{b_{p}^{2}}{a_{p}a_{p+1}}\cos^{2}{(\frac{\pi}{k+1})}\leq\frac{(\beta-\alpha)^{2}}{\gamma^{2}}<1.$

Therefore, $M_{\sin,e}$ positive definite for all $l\geq 4$. For $l=1$,
$M_{\sin,e}$ does not exist and for $l=2$ and $3$, $M_{\sin,e}$ is of order
$1\times 1$ and hence positive definite.

######  Lemma 3.3.4.

$M_{\cos,o}$ and $M_{\sin,o}$ are positive definite matrices for all
$l=1,2,\dots,L$.

######  Proof.

Using $(\ref{phip1})$, we have

$\frac{4\psi_{0}^{2}}{\phi_{0}\phi_{1}}=\frac{(\beta-\alpha)^{2}(l-2)(l-1)(l+2)(l+3)}{\\{(3\alpha+\beta)l(l+1)/2+\gamma(2l^{2}+2l-3)\\}\times\\{(\alpha+\beta)(l^{2}++8)+\gamma(2l^{2}+2l-19)\\}}.$

But $0>\alpha>\beta>\gamma\geq\alpha+\beta$, therefore

$\begin{split}\frac{4\psi_{0}^{2}}{\phi_{0}\phi_{1}}&\leq\frac{2(\beta-\alpha)^{2}}{15\gamma^{2}}\frac{(l-1)(l+2)}{(l-2)(l+3)}\quad\text{for}\quad
l\geq 4,\\\ &\leq\frac{(\beta-\alpha)^{2}}{\gamma^{2}}.\end{split}$

Therefore,

$\frac{4\psi_{0}^{2}}{\phi_{0}\phi_{1}}\cos^{2}{(\frac{\pi}{n+1})}\leq\frac{(\beta-\alpha)^{2}}{\gamma^{2}}<1,\quad\text{where}\quad
n=\begin{cases}k+1,&l=2k+1,\\\ k,&l=2k.\end{cases}$

Now, for $p>0$,

(33)
$\frac{4\psi_{p}^{2}}{\phi_{p}\phi_{p+1}}=\frac{(\beta-\alpha)^{2}(l-2p-2)(l-2p-1)(l+2p+2)(l+2p+3)}{\\{(3\alpha+\beta)(l^{2}+4p^{2}+4p+l)+\gamma(2l^{2}-8p^{2}-8p+2l-3)\\}}$
$\times\frac{1}{\\{(\alpha+\beta)(l^{2}+4p^{2}+12p+l+8)+\gamma(2l^{2}-8p^{2}-24p+2l-19)\\}}.$

Therefore,

(34)
$\begin{split}\frac{4b_{\psi}^{2}}{\phi_{p}\phi_{p+1}}&\leq\frac{(\beta-\alpha)^{2}}{9\gamma^{2}}\frac{(l+2p+2)(l-2p-1)}{(l-2p-2)(l+2p+3)},\quad\text{for}\quad
l\geq 5,\\\ &\leq\frac{2(\beta-\alpha)^{2}}{9\gamma^{2}}.\end{split}$

Hence,

$\frac{4\psi_{p}^{2}}{\phi_{p}\phi_{p+1}}\cos^{2}{(\frac{\pi}{n+1})}\leq\frac{(\beta-\alpha)^{2}}{\gamma^{2}}<1,\quad
where\quad n=\begin{cases}k+1,&l=2k+1,\\\ k,&l=2k.\end{cases}$

Therefore, $M_{\cos,o}$ is positive definite for $l\geq 5$ by Theorem
$(\ref{PDM})$. Also,

$\frac{4\psi_{0}^{2}}{\tilde{\phi_{0}}\phi_{1}}=\frac{(\beta-\alpha)^{2}(l-2)(l-1)(l+2)(l+3)}{\\{(\alpha+3\beta)l(l+1)/2+\gamma(2l^{2}+2l-3)\\}\times\\{(\alpha+\beta)(l^{2}++8)+\gamma(2l^{2}+2l-19)\\}}.$

But $0>\alpha>\beta>\gamma\geq\alpha+\beta$, therefore

$\begin{split}\frac{4\psi_{0}^{2}}{\tilde{\phi_{0}}\phi_{1}}&\leq\frac{4(\beta-\alpha)^{2}}{15\gamma^{2}},\quad\text{for}\quad
l\geq 4.\end{split}$

Therefore,

$\frac{4\psi_{0}^{2}}{\tilde{\phi_{0}}\phi_{1}}\cos^{2}{(\frac{\pi}{n+1})}\leq\frac{(\beta-\alpha)^{2}}{\gamma^{2}}<1,\quad
where\quad n=\begin{cases}k+1,&l=2k+1,\\\ k,&l=2k.\end{cases}$

Therefore, $M_{\sin,o}$ is positive definite for all $l\geq 5$ by Theorem
$(\ref{PDM})$. When $l=3$ or $4$, we will show that the determinant is
positive and hence the matrices are positive definite. The determinant will be
positive iff

$\frac{3l(l+1)(l^{2}+l+8)}{2}-(l-2)(l-2)(l+2)(l+3)>0$

But for $l=3$ and $4$, above quantity is positive. Hence, $M_{\cos,o}$ and
$M_{\sin,o}$ are positive definite for $l=3$ and $4$.

Using lemmas (3.3.2), (3.3.3) and (3.3.4), we have the following theorem.

######  Theorem 3.4.

$M_{\cos,e}$, $M_{\sin,e}$, $M_{\cos,o}$ and $M_{\sin,o}$ are positive
definite for $\alpha$, $\beta$, $\gamma$ and $\epsilon\in(0,\epsilon_{1})$
whenever the corresponding triaxial ellipsoids $E_{A,B,C}(\epsilon)$ has the
Gaussian curvature greater than equal to one at all points.

######  Theorem 3.5.

Let $\lambda_{i}(g_{\epsilon})$ be the $i^{th}$ eigenvalue of the Laplace
operator $\Delta_{g_{\epsilon}}$ on the ellipsoid $E_{A,B,C}(\epsilon)$.
Consider an analytic perturbation of the standard Euclidean sphere through
ellipsoids, $E_{A,B,C}(\epsilon)$, with the Gaussian curvature bounded below
by one. Then given $L\in\mathbb{N}$, there exists $j$ depending on $L$ such
that

$\lambda_{i}(g_{\epsilon})\geq\lambda_{i}(g_{0}),\;\;\;\text{for all}\;\;i\leq
j,$

for $\epsilon$’s sufficiently close to $0$.

######  Proof.

The proof follows directly from lemmas $(\ref{Eaab})$, $(\ref{Eabb})$ and
Theorem $(\ref{TriM})$.

## References

  * [1] Milica Anđelić and C.. Fonseca  “Sufficient conditions for positive definiteness of tridiagonal matrices revisited”, 2011, pp. 155–159
  * [2] Harold Donnelly  “Harmonic functions on manifolds of nonnegative Ricci curvature”  In _International Mathematics Research Notices_ 2001.8, 2001, pp. 429–434
  * [3] Suresh Eswarathasan and Theodore Kolokolnikov  “Laplace–Beltrami spectrum of ellipsoids that are close to spheres and analytic perturbation theory”  In _IMA Journal of Applied Mathematics_ , 2021  DOI: 10.1093/imamat/hxab045
  * [4] Anandateertha Mangasuli  “On the Eigenvalues of the Laplacian for Certain Perturbations of the Standard Euclidean Metric on $S^{2}$”  In _Asian Journal of Mathematics_ 13.2, 2009, pp. 271–282  DOI: 10.4310/AJM.2009.v13.n2.a6
  * [5] Anandateertha Mangasuli  “On the eigenvalues of the Laplacian for left-invariant Riemannaian metrics on $S^{3}$”  In _International Journal of Mathematics_ 18.08, 2007, pp. 895–901  DOI: 10.1142/S0129167X07004382


# Joint Learning of Feature Extraction and Cost Aggregation for Semantic
Correspondence

###### Abstract

Establishing dense correspondences across semantically similar images is one
of the challenging tasks due to the significant intra-class variations and
background clutters. To solve these problems, numerous methods have been
proposed, focused on learning feature extractor or cost aggregation
independently, which yields sub-optimal performance. In this paper, we propose
a novel framework for jointly learning feature extraction and cost aggregation
for semantic correspondence. By exploiting the pseudo labels from each module,
the networks consisting of feature extraction and cost aggregation modules are
simultaneously learned in a boosting fashion. Moreover, to ignore unreliable
pseudo labels, we present a confidence-aware contrastive loss function for
learning the networks in a weakly-supervised manner. We demonstrate our
competitive results on standard benchmarks for semantic correspondence.

Index Terms—  semantic correspondence, feature extraction, cost aggregation,
contrastive learning

##  1 Introduction

Semantic correspondence is one of the essential tasks on various Computer
Vision applications [1, 2, 3], which generally aims to establish pixel-wise,
but locally-consistent correspondences across semantically similar images. It
is an extremely challenging task because finding semantic correspondences can
be easily distracted by non-rigid deformations and large variations on the
appearance within the same class [4].

Recent methods [5, 3] solved the task by designing deep Convolutional Neural
Networks (CNNs). The networks often consist of feature extraction and cost
aggregation steps. For the feature extraction step, instead of relying on
hand-crafted descriptors as in conventional methods [6], recently, there has
been an increasing interest in leveraging the representation power of CNNs [7,
8, 4]. However, they can struggle with determining the correct matches from
the cost volume because ambiguous matching pairs are often generated by
repetitive patterns and occlusions. For cost aggregation step, methods [9, 10,
5, 11, 12, 13, 14, 3] attempted to determine the correct matches between the
great majority of dense information and non-distinctive matching pairs. Unlike
previous strategies [6, 15], recent methods proposed a trainable matching cost
aggregation in the overall network [9, 10, 5, 11, 12, 13, 14, 3].

Learning semantic correspondence networks consisting of feature extraction and
cost aggregation modules in a supervised manner requires a large-scale ground-
truth which is notoriously hard to build. To alleviate this, several methods
leveraged pseudo-labels, extracted from networks’ prediction itself by Winner-
Take-All (WTA), and train the networks in an unsupervised111They are often
also called weak-supervised methods since they require the image pairs. manner
[10, 5]. Although they are appealing alternatives, they are sensitive to
uncertain pseudo labels. Furthermore, jointly using the pseudo labels from
feature extraction and cost aggregation modules may boost the performance, but
there was no study for this approach.

Some recent self-supervised methods [16, 17] use dense contrastive loss for
pixel-wise prediction tasks instead of using image-level contrastive loss.
However, they do not consider unconfident matches generated from repetitive
fields or occlusions and semantic appearance variations.

(a) Learning feature extraction only (b) Learning cost aggregation only (c)
Ours

Fig. 1: Intuition of our method: (a) methods for solely training a feature
extraction network [7, 8, 4], (b) methods for solely training a cost
aggregation network for filtering the ambiguous matching cost [9, 10, 5, 11,
14], and (c) Ours, which jointly training feature extraction and cost
aggregation networks with the proposed loss function.

In this work, we present a novel framework that jointly learns feature
extraction and cost aggregation for semantic correspondence. Motivated by the
observation that the pseudo labels from feature extraction and cost
aggregation steps can complement each other, we encourage the feature
extraction and cost aggregation modules to be jointly trained by exploiting
the pseudo-labels from each other module. In addition, to filter out
unreliable pseudo-labels, we present a confidence-aware contrastive loss
function that exploits a forward-backward consistency to remove incorrect
matches. We demonstrate experiments on several benchmarks [18, 19], proving
the robustness of the proposed method over the latest methods for semantic
correspondence.

##  2 Methodology

###  2.1 Preliminaries

Given a pair of images, i.e., source $I_{s}$ and target $I_{t}$, which
represent semantically similar images, the objective of semantic
correspondence is to predict dense correspondence fields $P$ between the two
images at each pixel. To achieve this, most predominant methods consist of two
steps, namely feature extraction and cost aggregation [8, 5]. In specific, the
first step is to apply feature extraction networks to obtain a 3D tensor
$D\in\mathbb{R}^{h\times w\times d}$, where $h\times w$ is the spatial
resolution of the feature maps and $d$ denotes the number of channels. To
estimate correspondences, the similarities between feature maps $D_{s}$ and
$D_{t}$ from source and target images, respectively, are measured, which
outputs a 4D cost volume $\mathcal{C}(i,j)$, where ${i,j\in\\{1,...,h\times
w\\}}$, through a cosine similarity score such that
$\mathcal{C}(i,j)=D_{s}(i)^{T}D_{t}(j)$. Estimating the correspondence with
sole reliance on matching similarities is sensitive to matching outliers, and
thus the cost aggregation steps are used to refine the initial matching
similarities to achieve the aggregated cost $\mathcal{A}(i,j)$ through cost
aggregation networks.

Learning such networks, i.e., feature extraction and cost aggregation modules,
in a supervised manner requires manually annotated ground-truth
correspondences $P^{*}$, which is extremely labor-intensive and subjective to
build [19, 18, 5]. To overcome this challenge, an alternative way is to
leverage Winner-Take-All (WTA) matching point, which is the most likely match
by an argmax function on $\mathcal{C}(i,j)$ or $\mathcal{A}(i,j)$, as a pseudo
correspondence label $F$. For instance, NCNet [5] (and its variants [13, 12,
14]) and DenseCL [17] utilized such correspondences $F$ to learn the cost
aggregation networks and feature extraction networks, respectively, in an
unsupervised fashion, as exemplified in Fig. 1. Although they are definitely
appealing alternatives, these frameworks are highly sensitive to uncertain
pseudo labels. Moreover, there exists no study to jointly train the feature
extraction networks and cost aggregation networks in a complementary and
boosting manner.

(a) $I_{s}$

(b) $I_{t}$

(c) $F^{\mathcal{C},0}_{s\rightarrow t}$

(d) $F^{\mathcal{A},0}_{s\rightarrow t}$

(e) $F^{\mathcal{C},500}_{s\rightarrow t}$

(f) $F^{\mathcal{A},500}_{s\rightarrow t}$

(g) $F^{\mathcal{C},1000}_{s\rightarrow t}$

(h) $F^{\mathcal{A},1000}_{s\rightarrow t}$

Fig. 2: Comparison on PF-Pascal [18] applying masking and warping by the
predicted correspondence map. At later iteration (1000) with
$F^{\mathcal{A},1000}_{s\rightarrow t}$ have more confident correspondences
than with $F^{\mathcal{C},1000}_{s\rightarrow t}$ and at earlier iteration
(500) vice versa.

###  2.2 Confidence-aware Contrastive Learning

In this section, we first study how to achieve better pseudo labels for dense
correspondence, and then present a confidence-aware contrastive learning.

We start from classic uncertainty measurement based on forward-backward
consistency checking as proposed in [20, 21, 22], where an argmax operator was
applied twice for forward and backward directions, respectively. Specifically,
the pseudo matching map $F^{\mathcal{C}}_{s\rightarrow t}$ from the matching
cost $\mathcal{C}$, warping $I_{s}$ toward $I_{t}$, is defined as follows:

$F^{\mathcal{C}}_{s\rightarrow
t}(i)=\mathrm{argmax}_{j^{\prime}}\,\mathcal{C}\left(i,j^{\prime}\right)-i$
(1)

where $j^{\prime}$ is defined for all the points in the target. Similarly,
$F^{\mathcal{C}}_{t\rightarrow s}$ can be computed, warping $I_{t}$ toward
$I_{s}$. In an non-occlusion region, we get a backward flow vector
$F^{\mathcal{C}}_{t\rightarrow s}$ in the inverse direction as the forward
flow vector $F^{\mathcal{C}}_{s\rightarrow t}$. If this consistency constraint
is not satisfied, the points in the target are occluded at the matches in the
source, or the estimated flow vector is incorrect. These constraints can be
defined such that

$\begin{split}\|F^{\mathcal{C}}_{s\rightarrow
t}(i)+&F^{\mathcal{C}}_{t\rightarrow s}(i+F^{\mathcal{C}}_{s\rightarrow
t}(i))\|^{2}\\\ <&{\alpha}_{1}(\|F^{\mathcal{C}}_{s\rightarrow
t}(i)\|^{2}+\|F^{\mathcal{C}}_{t\rightarrow s}(i+F^{\mathcal{C}}_{s\rightarrow
t}(i))\|^{2})+{\alpha}_{2}.\end{split}$ (2)

Since there may be some estimation errors in the flows, we grant a tolerance
interval by setting hyper-parameters ${\alpha}_{1}$ and ${\alpha}_{2}$. A
binary mask $M^{\mathcal{C}}$ is then obtained by such forward-backward
consistency checking, representing a non-occluded region as 1 and an occluded
region as 0.

Based on the estimated mask $M^{\mathcal{C}}$, we present a confidence-aware
contrastive loss function, aiming to maximize the similarities at the reliably
matched points while minimizing for the others, defined such that

$\mathcal{L}^{\mathcal{C}}_{\mathrm{ccl}}=-\frac{1}{N^{\mathcal{C}}}\sum_{i}M^{\mathcal{C}}(i)\mathrm{log}\left(\frac{\mathrm{exp}(\mathcal{C}(i,i+F^{\mathcal{C}}_{s\rightarrow
t}(i))/\gamma)}{\sum_{j}\mathrm{exp}(\mathcal{C}(i,j)/\gamma)}\right),$ (3)

where $N^{\mathcal{C}}$ is the number of non-occluded pixels, and $\gamma$ is
a temperature hyper-parameter. Our loss function enables rejecting ambiguous
matches with a thresholding while accepting the confident matches. While
$\mathcal{L}^{\mathcal{C}}_{\mathrm{ccl}}$ is formulated to train the feature
extraction network itself, this loss function can also be defined for
aggregated cost $\mathcal{A}$ as $\mathcal{L}^{\mathcal{A}}_{\mathrm{ccl}}$,
which can be used to train the cost aggregation networks.

Fig. 3: Overall framework.

###  2.3 Joint Learning

The proposed confidence-aware contrastive loss function can be independently
used for learning feature extraction and cost aggregation networks through
$\mathcal{L}^{\mathcal{C}}_{\mathrm{ccl}}$ and
$\mathcal{L}^{\mathcal{A}}_{\mathrm{ccl}}$, respectively. However, since two
pseudo labels from feature extraction networks and cost aggregation networks
may have complementary information, using two pseudo labels in a joint manner
can enable further boosting the performance. For instance, at early stages of
training, the pseudo label by pre-trained feature extractor provides more
reliable cues than ones by randomly-initialized cost aggregation networks,
which may help the cost aggregation networks converge much faster, as
exemplified in Fig. 2. In addition, as the training progresses, the well-
trained cost aggregation networks produce superior correspondences than the
pseudo label by feature extractor, as exemplified in Fig. 2.

Method | Base | Trainable Comp. | Joint | PF-PSCAL ($\alpha$ = 0.1) | PF-Willow ($\alpha$ = 0.1) | TSS ($\alpha$ = 0.05)
---|---|---|---|---|---|---
Feature | Aggreg. | FG3D | JODS | PASCAL | Avg
WTA | ResNet-101 | - | - | - | 53.3 | 46.9 | - | - | - | -
WeakAlign [10] | ResNet-101 | ✓ | ✓ | - | 75.8 | 84.3 | 90.3 | 76.4 | 56.5 | 74.4
RTNs [23] | ResNet-101 | - | ✓ | - | 75.9 | 71.9 | 90.1 | 78.2 | 63.3 | 72.2
NCNet [5] | ResNet-101 | - | ✓ | - | 78.9 | 84.3 | 94.5 | 81.4 | 57.1 | 77.7
DCCNet [24] | ResNet-101 | ✓ | ✓ | - | 82.3 | 73.8 | 93.5 | 82.6 | 57.6 | 77.9
ANCNet [13] | ResNet-101 | ✓ | ✓ | - | 86.1 | - | - | - | - | -
PMNC [14] | ResNet-101 | - | ✓ | - | 90.6 | - | - | - | - | -
CATs [3] | ResNet-101 | ✓ | ✓ | - | 87.3 | 76.9 | 85.3 | 73.7 | 55.4 | 73.6
Ours w/NCNet | ResNet-101 | ✓ | ✓ | ✓ | 80.0 | 86.6 | 95.0 | 82.3 | 55.8 | 78.4
Ours w/CATs | ResNet-101 | ✓ | ✓ | ✓ | 92.5 | 79.8 | 91.7 | 81.2 |  60.9 |  80.0
|  |  |  |  |  |  |  |  |  |

Table 1: Comparison with state-of-the-art methods on standard benchmarks [18,
19].

To leverage complementary information during training, we use a pseudo label
output of each module, namely feature extraction and cost aggregation modules,
defined such that

$\begin{split}&\mathcal{L}^{\mathcal{A}\rightarrow\mathcal{C}}_{\mathrm{ccl}}=\\\
&-\frac{1}{N^{\mathcal{A}}}\sum_{i}M^{\mathcal{A}}(i)\mathrm{log}\left(\frac{\mathrm{exp}(\mathcal{C}(i,i+F^{\mathcal{A}}_{s\rightarrow
t}(i))/\gamma)}{\sum_{j}\mathrm{exp}(\mathcal{C}(i,j)/\gamma)}\right),\end{split}$
(4)

where $F^{\mathcal{A}}_{s\rightarrow
t}(i)={\mathrm{argmax}_{j^{\prime}}}\,\mathcal{A}\left(i,j^{\prime}\right)$.
$\mathcal{L}^{\mathcal{C}\rightarrow\mathcal{A}}_{\mathrm{ccl}}$ is similarly
defined. Our final loss function is thus defined such that

$\mathcal{L}=\lambda^{\mathcal{C}}(\mathcal{L}^{\mathcal{C}}_{\mathrm{ccl}}+\mathcal{L}^{\mathcal{A}\rightarrow\mathcal{C}}_{\mathrm{ccl}})+\lambda^{\mathcal{A}}(\mathcal{L}^{\mathcal{A}}_{\mathrm{ccl}}+\mathcal{L}^{\mathcal{C}\rightarrow\mathcal{A}}_{\mathrm{ccl}}),$
(5)

where $\lambda^{\mathcal{C}}$ and $\lambda^{\mathcal{A}}$ represent hyper-
parameters. Fig. 3 illustrates the overall architecture of the proposed
methods.

##  3 Experiments

###  3.1 Implementation Details

In our framework, we used the ResNet-101 [25] pretrained on ImageNet [26]
benchmark. We added additional layers followed by this to transform features
to be highly discriminative w.r.t. both appearance and spatial context [4]. In
addition, we used two types of cost aggregation modules like 4D CNN [5],
denoted Ours w/NCNet, and transformer-based architecture [3], denoted Ours
w/CATs. We used 256x256 size for the input image and 16x16 size for the
feature map. The learning rate is adjusted, starting from differently 3e-5 and
3e-6 for feature extraction and cost aggregation, respectively, and adjusted
using AdamW optimizer. We set ${\alpha}_{1}={0.1}$, ${\alpha}_{2}={0.05}$,
$\lambda^{\mathcal{C}}=0.5$, and $\lambda^{\mathcal{A}}=0.5$.

###  3.2 Experimental Settings

In this section, we demonstrate that our method is effective through
comparison with others; WeakAlign [10], RTNs [23], NCNet [5], DCCNet [24],
ANCNet [13], PMNC [14], and CATs [3]. We also conduct an analysis of each
component in our framework in the ablation study. To evaluate semantic
matching, Proposal Flow [18] and TSS [19] benchmarks were used. The Proposal
Flow benchmark contains PF-Pascal and PF-Willow [18]. TSS dataset is split
into three subgroups: FG3D, JODS and PASCAL [19]. A percentage of correct
keypoints (PCK) is employed for evaluation.

###  3.3 Experimental Results

Table 1 shows the quantitative results. We conduct experiments on various
benchmarks, such as PF-Pascal [18], PF-Willow [18], and TSS [19]. Our method
records higher accuracy than both baselines, NCNet and CATs, by 1.1 and 5.2 on
PF-Pascal, 2.3 and 2.9 on PF-Willow, respectively. Specifically, ours w/CATs
outperforms other methods over all benchmarks and show the biggest performance
improvement about 2 on PF-Pascal dataset compared to 90.6, which is the state-
of-the-art result [14] among the similar network architectures and algorithms.
Ours w/NCNet shows the highest performance on the FG3D as 95.0, and records
average PCK of 78.4 on TSS benchmark. The qualitative results of semantic
matching on the PF dataset are shown in Fig. 4. (c), (d), (e) are warped
images from (a) to (b) by WTA, NCNet [5], and ours, respectively. Through (c),
we could observe that errors of matches produced from feature extraction
network affect the final output. Compared to (d), (e) shows accurate matching
results even in difficult examples with occlusion, background clutter, and
repetitive textures.

Components | Accuracy
---|---
(a) | Ours | 80.0
(b) | (-) Joint learning | 78.4
(c) | (-) Confidence-aware loss | 77.7
|  |
Table 2: Ablation study on our modules.

Loss component | PCK ($\alpha$ = 0.1)
---|---
| $\mathcal{L}^{\mathcal{C}}_{\mathrm{ccl}}$ | $\mathcal{L}^{\mathcal{A}\rightarrow\mathcal{C}}_{\mathrm{ccl}}$ | $\mathcal{L}^{\mathcal{A}}_{\mathrm{ccl}}$ | $\mathcal{L}^{\mathcal{C}\rightarrow\mathcal{A}}_{\mathrm{ccl}}$ |
(a) | - | - | ✓ | - | 70.0
(b) | - | - | ✓ | ✓ | 71.7
(c) | ✓ | - | ✓ | - | 78.4
(d) | ✓ | ✓ | ✓ | ✓ | 80.0
|  |  |  |  |

Table 3: Ablation study of our loss formulation.

###  3.4 Ablation Study

In this section, we analyze the main components in our method, confidence-
aware contrastive loss and joint learning, with NCNet baseline [5] on PF-
Pascal. First, in Table 2 we validate the effectiveness of joint learning and
confidence-aware contrastive loss by the lower performance of (b) and (c)
compared to (a) which has both of these components. This proves that training
two networks in a complementary manner boosts the performance and confidence-
aware contrastive loss leads the unreliable pseudo labels to be filtered out
during the training process. We also verify the effectiveness of each
confidence-aware contrastive loss component through possible combinations of
components displayed in Table 3. Compared to (a) and (b), both of which train
two networks only with the loss from the aggregation module, (c) and (d) show
better PCK results by using separate losses that come from feature extraction
and cost aggregation respectively. From this, we can confirm that the direct
loss from each module works as a sufficient supervision signal for training
which is free from gradient vanishing. Based on the performance improvements
observed between (b) and (a), and between (d) and (c), we can also confirm
that the reliable sample from one module helps training the other module, as
it supports the formulation of better loss signals.

(a) $I_{s}$

(b) $I_{t}$

(c) WTA

(d) NCNet [5]

(e) Ours

Fig. 4: Qualitative results on PF-Pascal dataset [18].

##  4 Conclusion

We address the limitations of the existing methods by jointly training feature
extraction networks and aggregation networks in an end-to-end manner with the
proposed confidence-aware contrastive loss. By jointly learning the networks
with a novel loss function, our model outperforms the baseline and shows
competitive results on standard benchmarks.

Acknowledgements. This research was supported by the National Research
Foundation of Korea (NRF-2021R1C1C1006897).

## References

  * [1] Hajime Taira, Ignacio Rocco, Jiri Sedlar, Masatoshi Okutomi, Josef Sivic, Tomas Pajdla, Torsten Sattler, and Akihiko Torii,  “Is this the right place? geometric-semantic pose verification for indoor visual localization,”  in ICCV, 2019.
  * [2] Qianqian Wang, Xiaowei Zhou, Bharath Hariharan, and Noah Snavely,  “Learning feature descriptors using camera pose supervision,”  in ECCV, 2020.
  * [3] Seokju Cho, Sunghwan Hong, Sangryul Jeon, Yunsung Lee, Kwanghoon Sohn, and Seungryong Kim,  “Semantic correspondence with transformers,”  arXiv:2106.02520, 2021.
  * [4] Junghyup Lee, Dohyung Kim, Jean Ponce, and Bumsub Ham,  “Sfnet: Learning object-aware semantic correspondence,”  in CVPR, 2019.
  * [5] Ignacio Rocco, Mircea Cimpoi, Relja Arandjelović, Akihiko Torii, Tomas Pajdla, and Josef Sivic,  “Neighbourhood consensus networks,”  arXiv:1810.10510, 2018.
  * [6] David G Lowe,  “Distinctive image features from scale-invariant keypoints,”  in IJCV, 2004.
  * [7] Christopher B Choy, JunYoung Gwak, Silvio Savarese, and Manmohan Chandraker,  “Universal correspondence network,”  arXiv:1606.03558, 2016.
  * [8] Seungryong Kim, Dongbo Min, Bumsub Ham, Sangryul Jeon, Stephen Lin, and Kwanghoon Sohn,  “Fcss: Fully convolutional self-similarity for dense semantic correspondence,”  in CVPR, 2017.
  * [9] Ignacio Rocco, Relja Arandjelovic, and Josef Sivic,  “Convolutional neural network architecture for geometric matching,”  in CVPR, 2017.
  * [10] Ignacio Rocco, Relja Arandjelović, and Josef Sivic,  “End-to-end weakly-supervised semantic alignment,”  in CVPR, 2018.
  * [11] Iaroslav Melekhov, Aleksei Tiulpin, Torsten Sattler, Marc Pollefeys, Esa Rahtu, and Juho Kannala,  “Dgc-net: Dense geometric correspondence network,”  in WACV, 2019.
  * [12] Ignacio Rocco, Relja Arandjelović, and Josef Sivic,  “Efficient neighbourhood consensus networks via submanifold sparse convolutions,”  in ECCV, 2020.
  * [13] Shuda Li, Kai Han, Theo W Costain, Henry Howard-Jenkins, and Victor Prisacariu,  “Correspondence networks with adaptive neighbourhood consensus,”  in CVPR, 2020.
  * [14] Jae Yong Lee, Joseph DeGol, Victor Fragoso, and Sudipta N Sinha,  “Patchmatch-based neighborhood consensus for semantic correspondence,”  in CVPR, 2021.
  * [15] Bastian Leibe, Aleš Leonardis, and Bernt Schiele,  “Robust object detection with interleaved categorization and segmentation,”  in IJCV, 2008.
  * [16] Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu,  “Contrastive learning for unpaired image-to-image translation,”  in ECCV, 2020.
  * [17] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li,  “Dense contrastive learning for self-supervised visual pre-training,”  arXiv:2011.09157, 2020.
  * [18] Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce,  “Proposal flow: Semantic correspondences from object proposals,”  in TPAMI, 2017.
  * [19] Tatsunori Taniai, Sudipta N Sinha, and Yoichi Sato,  “Joint recovery of dense correspondence and cosegmentation in two images,”  in CVPR, 2016.
  * [20] Narayanan Sundaram, Thomas Brox, and Kurt Keutzer,  “Dense point trajectories by gpu-accelerated large displacement optical flow,”  in ECCV, 2010.
  * [21] Simon Meister, Junhwa Hur, and Stefan Roth,  “Unflow: Unsupervised learning of optical flow with a bidirectional census loss,”  in AAAI, 2018.
  * [22] Pengpeng Liu, Irwin King, Michael R Lyu, and Jia Xu,  “Ddflow: Learning optical flow with unlabeled data distillation,”  in AAAI, 2019.
  * [23] Seungryong Kim, Stephen Lin, Sang Ryul Jeon, Dongbo Min, and Kwanghoon Sohn,  “Recurrent transformer networks for semantic correspondence,”  in NeurIPS, 2018.
  * [24] Shuaiyi Huang, Qiuyue Wang, Songyang Zhang, Shipeng Yan, and Xuming He,  “Dynamic context correspondence network for semantic alignment,”  in ICCV, 2019.
  * [25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in CVPR, 2016.
  * [26] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei,  “Imagenet: A large-scale hierarchical image database,”  in CVPR, 2009.