# Spherically Symmetric Gravitational Collapse Of Inhomogeneous Dust Cloud In
The Background Of Dark Energy

Anjali Pandey<EMAIL_ADDRESS>,Rajesh Kumar<EMAIL_ADDRESS>,Sudhir Kumar Srivastava<EMAIL_ADDRESS>
Department of Mathematics $\&$ Statistics,
Deen Dayal Upadhyaya Gorakhpur University, Gorakhpur, INDIA.

Abstract

The present paper deals with the gravitational collapse of an inhomogeneous
spherical star consisting of dust fluid ($\rho$) in the background of dark
energy components ($p_{{}_{DE}},\rho_{{}_{DE}}$) with equation of state
$p_{{}_{DE}}=\omega\rho_{{}_{DE}}$ ($\omega$ being a non zero constant
parameter describing dark energy component). We discussed the development of
apparent horizon to investigate the black- hole formation in gravitational
collapsing process. The collapsing process is examined first separately for
dust cloud and dark energy and then under the combined effect of dust
interactig with dark energy. It is obtained that when only dust cloud or dark
energy is present the collapse leads to the formation of black-hole under
certian conditions. When both of them are present, collapsing star does not
form black-hole. However when dark energy is considered as cosmological
constant $(\omega=-1)$, the collapse leads to black hole formation.

Keywords Gravitational collapse, Black hole, Dust cloud, Dark energy, Naked-
Singularity, Apparent Horizon

##  1 INTRODUCTION

Gravitational collapse is one of the most significant phenomenon in general
theory of relativity. A massive star undergoes a continual gravitational
collapse when the pressures inside it become insufficient to balance the pull
of its own gravity. The problem of general relativistic gravitational collapse
of massive stars has attracted attention of researchers for many years,
starting with the seminal paper by Oppenheimer and Snyder[1]. The motivation
for such interest is easily understood-the gravitational collapse of massive
stars represents one of the few observable phenomena, where general relativity
is expected to play a relevant role and the physics of it being crucial and
interesting amongst astrophysicists. The studies concentrated on the
singularity formation (Black hole or Naked singularity) in gravitational
collapse within the framework of general relativity and other gravitational
theories are the interesting problems for theoretical astrophysicists[2]-[8].
The astrophysical observation suggested that present universe consists of
approximately 71 % of dark energy and 15% dark matter. The nature of dark
energy as well as dark matter is unknown and many radically different models
have been proposed, such as cosmological constant, quintessence[9, 10], DGP
branes [11, 12], Gauss- Bonnet [13] and dark energy in brane worlds[14]-[17]
for their studies. Such kinds of exotic fluids do not appear to interact with
the usual standard model particles and therefore it is also very difficult to
detect them by the modern detectors [18, 19].
For the stars made of baryonic matter, besides gravitational interaction, they
can also interact with each other by means of strong, weak and electromagnetic
forces. Dark energy is supposed to interact with ordinary matter only through
gravity. Therefore, only by its gravitational effects, one can gather
information, such as its positions, mass, density profiles, etc. A natural
question is how dark energy affects the process of the gravitational collapse
of a star. It is known that dark energy exerts a repulsive force on its
surrounding and this repulsive force may prevent the star from collapse.
Indeed, there are speculations that a massive stars do not simply collapse to
form a black hole, instead to the formation of stars that contain dark energy.
Some recent works have considered the spherically symmetric star consists of
non-baryonic (DE and DM) matter and discussed its nature of singularity
formation[20]-[24]. The dark energy, by virtue of its repulsive gravitational
nature, is interesting to study gravitational collapse and formation of Black-
hole. As all massive stars do not form black holes (may be neutron stars or
white dwarfs), so it is generally seen [23, 24, 25] that the dark energy may
play an important role in the collapsing stars. Gravitational collapse and
formation of black holes in the presence of dark energy have been considered
in several works [24, 26, 27]. Cai and Wang [24] investigated the black hole
formation from homogeneous collapsing dust in the background of dark energy,
and showed that the dark energy itself never collapses to form black holes but
when both the dark energy and the dust are present, Black-Holes, can be formed
due to the condensation of the dust. It has been shown that the mass of a
Black-Hole decreases due to phantom energy accretion and tends to zero when
the Big Rip approaches [28]. Thus with the notion of such exotic content of
universe, it is necessary to introduce this new kinds of matter distribution
in our studies. Motivated from above works, we considered the inhomogeneous
dust collapse in background of dark energy components and discuss the
singularity formations. The paper is organised as- in section-2 we considered
a class of shear-free spherically symmetric metric consisting of inhomogeneous
dust fluid with dark energy. In sec.3 and 4 we discussed the gravitational
collapse of an inhomogeneous dust cloud and dark energy separately. In
section-5 we considered the combined effect of dust cloud interacting with
dark energy and find that such interaction does not form the apparent horizon,
that is, no formation of black hole. The last section-6 contains the
conclusion part of the paper.

##  2 Fluid distribution and The field equations

Consider the collapsing spherically symmetric inhomogeneous star consists of
dust cloud in the background of dark energy. The spherical system is divide in
three different regions $\Sigma$, $V^{+}$ and $V^{-}$, where $\Sigma$ denotes
the surface of the star, $V^{+}$ and $V^{-}$ are exterior and the interior of
the star.
The spherically symmetric inhomogeneous space time inside the star ( $V^{-}$)
is described by the metric

$ds^{2}_{-}=A^{2}(t,r)dt^{2}-C^{2}(t,r)(dr^{2}+d\Omega^{2})$ (1)

where, $d\Omega^{2}=d\theta^{2}+sin^{2}\theta d\phi^{2}$. The coordinate is
taken as $x^{-i}=(t,r,\theta,\phi),i=0,1,2,3$

The energy momentum tensor $T^{-}_{ij}$ of matter field inside the star is
given by

$T^{-}_{ij}=(\rho+\rho_{{}_{DE}}+p_{{}_{DE}})u_{i}u_{j}-p_{{}_{DE}}g_{ij}$ (2)

where $\rho$ is energy density of inhomogeneous dust cloud, $\rho_{{}_{DE}}$
and $p_{{}_{DE}}$ are the energy density and pressure of dark energy
respectively, while $u^{-}_{i}$ is four velocity vector given by
$u_{i}=A(t,r)\delta^{0}_{i}$
The explicit form of the energy conservation equation $T^{-i}_{j;i}=0$ are

$3(\rho+\rho_{{}_{DE}}+p_{{}_{DE}})\frac{\dot{C}}{C}+(\dot{\rho}+\dot{\rho}_{{}_{DE}})=0$
(3)
$(\rho+\rho_{{}_{DE}}+p_{{}_{DE}})\frac{A^{\prime}}{A}+p_{{}_{DE}}^{\prime}=0$
(4)

where a dot (.) represents differentiation with respect to time $t$ and prime
$(^{\prime})$ represents differentiation with respect to $r$.
The non vanishing component of Einstein field equation
$(G_{ij}^{-}=-kT^{-}_{ij})$ for the metric (1) and the matter field of the
form of equ.(2) are given as,

$G^{-}_{00}\equiv\frac{1}{C^{2}}+\frac{C^{\prime
2}}{C^{4}}-2\frac{C^{\prime\prime}}{C^{3}}+3\frac{\dot{C}^{2}}{A^{2}C^{2}}=-k(\rho+\rho_{{}_{DE}})$
(5) $G^{-}_{11}\equiv\frac{2}{AC^{3}}A^{\prime}C^{\prime}+\frac{C^{\prime
2}}{C^{4}}+\frac{2}{A^{3}C}\dot{A}\dot{C}-\frac{\dot{C}^{2}}{A^{2}C^{2}}-\frac{2}{A^{2}C}\ddot{C}-\frac{1}{C^{2}}=-kp_{{}_{DE}}$
(6) $G^{-}_{22}\equiv\frac{A^{\prime\prime}}{AC^{2}}-\frac{C^{\prime
2}}{C^{4}}+\frac{C^{\prime\prime}}{C^{3}}++\frac{2}{A^{3}C}\dot{A}\dot{C}-\frac{\dot{C}^{2}}{A^{2}C^{2}}-2\frac{\ddot{C}}{A^{2}C}=-kp_{{}_{DE}}$
(7) $G^{-}_{33}=\textrm{sin}\theta G_{22}$ (8)
$G^{-}_{01}=G^{-}_{10}\equiv\frac{A^{\prime}}{A}+\frac{C^{\prime}}{C}-\frac{\dot{C^{\prime}}}{\dot{C}}=0$
(9)

where $k=\frac{8\pi G}{c^{4}}$. An apparent horizon is described as the
boundary of all trapped surfaces and the black hole formation is identified by
the development of apparent horizon on which we have

$C,_{\alpha}C,_{\beta}g^{-\alpha\beta}=\frac{\dot{C^{2}}}{A^{2}}-\frac{C^{\prime
2}}{C^{2}}=0$ (10)

where$(),_{x}\equiv\partial()/\partial x$
Another important quantity to describe the collapse is the mass function
$m(t,r)$, given by

$m(t,r)=\frac{1}{2}C\left(1+\frac{\dot{C^{2}}}{A^{2}}-\frac{C^{\prime
2}}{C^{2}}\right)$ (11)

which can be intepreted mass inside the collapsing star any instant $(t,r)$.
Assuming that $t_{AH}$ be the time when the whole star is collapses inside the
apparent horizon surface $r=r_{{}_{AH}}$, then equ.(10) gives

$\frac{\dot{C}^{2}(t_{{}_{AH}},r_{{}_{AH}})}{A^{2}(t_{{}_{AH}},r_{{}_{AH}})}=\frac{C^{\prime
2}(t_{{}_{AH}},r_{{}_{AH}})}{C^{2}(t_{{}_{AH}}r_{{}_{AH}})}$ (12)

The total mass contributed by the collapsing star to the mass of black hole is

$M_{{}_{BH}}=M(t_{{}_{AH}})$ (13)

If no matter continously falls into the black hole from outside of the star
after the formation of apparent horizon (i.e after the moment $t_{{}_{AH}}$),
we can calculate the total mass of the collapsing body (as the total mass of
the black hole) by above equ.(13). We also assume that the collapsing star is
not initially trapped then we should have

$\frac{\dot{C^{2}}}{A^{2}}-\frac{C^{\prime 2}}{C^{2}}<0$ (14)

##  3 Gravitational Collapse of Dust Cloud

In this case we assume that

$\rho\neq 0,\rho_{{}_{DE}}=0=p_{{}_{DE}}$ (15)

that is, the collapsing star consist of a only dust cloud. Historically,
gravitational collapse of dust cloud studied by Openheimer and Synder[1],
which leads to the formation of black holes. In this section we adapted the
different approach to discuss the singularity formation of collapsing dust
cloud.
In view of equs.(3) and (15) we find that

$\rho=\frac{\rho_{0}}{C^{3}}$ (16)

where $\rho_{0}\equiv\rho_{0}(r)$ is constant of integration. It follows from
equs.(4) and (15) that $A(t,r)\equiv A(t)$. For the suitable coordinate
transformation $A(\tilde{t})d\tilde{t}\rightarrow dt$ one may take $A\sim 1$.
Therefore equ.(9) gives

$C=\mu(t)\beta(r)$ (17)

where $\mu(t)$ and $\beta(r)$ arbitarary functions choosen as integrating
constants.
Since for collapsing configuration $\frac{\dot{C}}{C}<0$ which implies
$\dot{\mu}<0$
Thus we have from equs.(16) and (17)

$\rho=\frac{\rho_{0}}{\beta^{3}\mu^{3}}$ (18)
$\frac{\dot{C}}{C}=\frac{\dot{\mu}}{\mu}$ (19)

Let us assume the star starts to collapse from $t=t_{0}$ , $r=r_{\Sigma}$
(since the star is supposed not initially trapped ie., satisfying equ(14)) and
end at $t=t_{e}$ and $r=r_{e}$ (singularity formation). It can be seen from
equs.(18-19) that collapse of star end when $\mu(t_{e})$= constant= 0 where a
spacetime singularity is finally formed
$(\rho\rightarrow\infty,\dot{C}\rightarrow 0)$
From equ.(10), the development of apparent horizon in this case can be given
by

$\dot{\mu}=\frac{\beta^{\prime}}{\beta^{2}}$ (20)

Let apparent horizon is formed at the epoch $(t_{AH},r_{AH})$, then from
equ.(20) we have

$-K(t_{e}-t_{{}_{AH}})=\mu(t_{{}_{AH}})>0$ (21)

and

$K(r_{e}-r_{{}_{AH}})=\frac{\beta(r_{e})-\beta(r_{{}_{AH}})}{\beta(r_{e})\beta(r_{{}_{AH}})}$
(22)

where, $K=\frac{\beta^{\prime}}{\beta^{2}}<0$. Thus, in this case the horizon
is formed before the end-state $(t_{{}_{AH}}<t_{e})$ and therefore the
collapse will form black-hole.
It should be noted that if the collapse end as black-hole, then
$r_{e}<r_{{}_{AH}}$. Therefore from equ.(22)

$\beta(r_{{}_{AH}})-\beta(r_{e})<0$

Thus in the present case the arbitrary function $\beta(r)$ may be choosen as
$\frac{1}{r^{n}}$, n is a positive integer. Then the mass of the black hole
from equs.(11) and (13) is

$M_{BH}=\frac{\mu(t_{{}_{AH}})}{2r_{{}_{AH}}^{3n}}\left[\dot{\mu}^{2}(t_{{}_{AH}})+r^{2n}_{{}_{AH}}(1-\frac{n^{2}}{r_{{}_{AH}}^{2}})\right]$
(23)

##  4 Gravitational Collapse of Dark energy

In this section we discuss the gravitational collapse in the background of
dark energy component only,

$\rho=0,\quad p_{{}_{DE}}=\omega\rho_{{}_{DE}}\neq 0$ (24)

where the parameter $\omega(\neq-1)$ is a non zero constant.
Then equ.(9) yields

$A=\nu(t)\frac{\dot{C}}{C}$ (25)

where $\nu(t)$ is arbitrary. In this case equ.(3) yields

$\rho_{DE}=\frac{1}{C^{3(1+\omega)}}$ (26)

In view of equs.(25) and (26) we obtain from equ.(4)

$\frac{\dot{C}^{\frac{1+\omega}{\omega}}}{C^{\frac{1+4\omega+3\omega^{2}}{\omega}}}=\lambda(t)$
(27)

where $\lambda$ is arbitrary
Let us assume that the collapse of star end at $(t_{e},r_{e})$ where a
spacetime singularity is formed. It can also be seen from equ.(27) that
collapse end when $\lambda(t_{e})\rightarrow 0$.
Consider the star is not initially trapped and let apparent horizon is formed
at $(t,r)=(t_{{}_{AH}},r_{{}_{AH}})$. Then, from equ(15) development of
apparent horizon gives

$\frac{C(r_{{}_{AH}},r_{{}_{AH}})-C(t_{e},r_{e})}{C(t_{{}_{AH}},r_{{}_{AH}})C(t_{e},r_{e})}=\frac{\nu(t_{e})r_{{}_{AH}}-\nu(t_{{}_{AH}})r_{e}}{\nu(t_{{}_{AH}})\nu(t_{e})}$
(28)

Further the effect of dark energy on collapsing star will be studied by
parameterizing $\omega$.
Case (i) $\omega=-\frac{2}{3}$

Soma et al.[26] obtained the singularity formation for
$-1\leq\omega\leq-\frac{2}{3}$ and showed that when $\omega<-\frac{2}{3}$
collapse do not form Black-hole. However for $\omega=-\frac{2}{3}$ the end
state of collapse may be either Black-hole or Nacked singularity depending on
initial density.
In this case, it follows from equ.(27) that

$C\dot{C}=\lambda^{2}(t)$ (29)

which gives by integrating

$C=\sqrt{\lambda_{1}(t)+\beta(r)}$ (30)

where $\beta(r)$ and $\lambda_{1}(t)=2\int\lambda^{2}(t)dt$ are arbitrary.
Also from equ.(26) that

$\rho_{DE}=\frac{1}{\sqrt{\lambda_{1}(t)+\beta(r))}}$ (31)

Now, in new of equs.(28) and (30) we observe an apparent horizon form (black
hole formation) when

$\lambda_{1}(t_{{}_{AH}})>\beta(r_{e})-\beta(r_{{}_{AH}})$

The contribution of collapsing star to the mass of black-hole,

$M_{{}_{BH}}=\frac{1}{2}[\lambda_{1}(t_{{}_{AH}})+\beta(r_{{}_{AH}})]^{3/2}\biggl{[}\frac{1}{\mu^{2}(t_{{}_{AH}})}+\\\
\frac{1}{4(\lambda_{1}(t_{{}_{AH}})+\beta(r_{{}_{AH}}))^{3}}\\{4(\lambda_{1}(t_{{}_{AH}})+\beta(r_{{}_{AH}}))^{2}-\beta^{\prime
2}(r_{{}_{AH}})\\}\biggr{]}$ (32)

Case (ii) Phantom fluid
The equation of state with $\omega<-1$ is named as " Phantom fluid" has
recieved attention amongest researchers[29][30]. For example
$\omega=-\frac{2}{3}$, the phantom density increases with time which voilates
the dominant energy condition.
Consider $\omega=-\frac{2}{3}$, we obtain from equ.(27) that

$C=[\lambda_{2}(t)+\alpha(r)]^{\frac{2}{9}}$ (33)

where $\alpha(r)$ and $\lambda_{2}(t)=\int\frac{9}{2}\lambda^{3}(t)dt$ are
arbitrary. From equ.(26)

$\rho_{{}_{DE}}=[\lambda_{2}(t)+\alpha(r)]^{\frac{1}{3}}$ (34)

It follows from equs.(28) and (33) that the collapse end as black hole
(formation of an apparent horizon) when

$\lambda_{2}(t_{{}_{AH}})>\alpha(r_{e})-\alpha(r_{{}_{AH}})$

and the mass of the formed black hole is

$\begin{split}M_{{}_{BH}}=\frac{1}{2}[\lambda_{2}(t_{{}_{AH}})+\alpha(r_{{}_{AH}})]^{\frac{-16}{9}}[\\{1+(\lambda_{2}(t_{{}_{AH}})+\alpha(r_{{}_{AH}}))^{\frac{4}{9}}\\}(\lambda_{2}(t_{{}_{AH}})+\alpha(r_{{}_{AH}}))^{2}-\\\
\frac{4}{81}\alpha^{\prime 2}]\end{split}$ (35)

##  5 Gravitational Collapse with combined effect of dust cloud and dark
energy

Recently, number of authors have been considered the interaction of dust cloud
and dark energy by assuming([21, 24, 27, 38] and references therein)

$\frac{\rho_{{}_{DE}}}{\rho}=\gamma C^{3m}$ (36)

where $\lambda$(>0) and $m$ are arbitrary constants and $C(t,r)$ be the
geometric radius of star. The relation (36) shows that the energy flow between
dust fluid and dark energy. Assuming that the dark energy satisfy the equation
of state $p_{{}_{DE}}=\omega\rho_{{}_{DE}}$, $\omega$ being a constant
parameter.
From equ.(3),(9) and (36) we obtain

$\rho_{{}_{DE}}=C^{-3(1-m)}(1+\gamma C^{3m})^{-\frac{m+\omega}{m}}$ (37)

$\rho=\frac{C^{-3}}{\gamma}(1+\gamma C^{3m})^{-\frac{m+\omega}{m}}$ (38)

It follows from above equations that the spacetime has central singularity at
$C(t,r)=0$. Taking use of eqns. $(\ref{eqn:37})-(\ref{eqn:38})$ into equ.(4),
we obtain

$\begin{split}\dot{C^{\prime}}(1+\l\gamma
C^{3m})[1+\gamma(1+\omega)C^{3m}]-\dot{C}C^{\prime}[1+\gamma^{2}(1+3\omega^{2}+4\omega)C^{3m}\\\
-\gamma\\{-2+(3m-4)\omega\\}]C^{3m-1}=0\end{split}$ (39)

The equ.(39) is highly non-linear in $C(t,r)$ therefore for further discussion
in the following sections we consider different value of $m$ and $\omega$ to
investigate the collapsing phenomenon and apparent horizon formation of dust
cloud interacting with dark energy.

(A). m = $\frac{1}{2}$

when $m=\frac{1}{2}$, it follows from equ.(39) that

$\frac{\dot{C}}{C}=\begin{cases}\frac{\beta_{1}(t)}{(1+\gamma
C^{\frac{3}{2}})},&\text{for $\omega=-1$}\\\ \frac{\beta_{2}(t)}{(1+\gamma
C^{\frac{3}{2}})(3+\gamma C^{\frac{3}{2}})},&\text{for $\omega=-\frac{2}{3}$
}\\\ \frac{\beta_{3}(t)}{(1+\gamma C^{\frac{3}{2}})^{2}(2-\gamma
C^{\frac{3}{2}})},&\text{for $\omega=-\frac{3}{2}$}\end{cases}$ (40)

where $\beta_{1},\beta_{2}$ and $\beta_{3}$ are arbitrary functions of $t$ and

$Ce^{\frac{2}{3}\gamma C^{\frac{3}{2}}}=\alpha_{1}(t)K_{1}(r)\quad\textrm{for
$\omega$= -1}$ (41)

$\begin{split}13+14\gamma C^{\frac{3}{2}}+\gamma^{2}C^{3}-\frac{6}{\gamma
c^{\frac{3}{2}}}(1+\gamma
C^{\frac{3}{2}})^{\frac{2}{3}}\textrm{Hypergeometric}2F1\left[\frac{2}{3},\frac{2}{3},\frac{5}{3},-\frac{1}{\gamma
C^{\frac{3}{2}}}\right]\\\ =2[\alpha_{2}(t)+K_{2}(r)](1+\gamma
C^{\frac{3}{2}})^{\frac{2}{3}}\quad\textrm{for
$\omega=-\frac{2}{3}$}\end{split}$ (42)

and

$Ce^{\gamma
C^{\frac{3}{2}}}-\frac{1}{9}\gamma^{3}C^{\frac{9}{2}}=K_{3}(r)\alpha_{3}(t)\quad\textrm{for
$\omega=-\frac{3}{2}$}$ (43)

where $\alpha_{1}(t),\alpha_{2}(t),\alpha_{3}(t),K_{1}(r),K_{2}(r)$, and
$K_{3}(r)$ are arbitrary constants of integration. We can observe from
equ.(40) the collapsing process will not end untill the
$\beta_{1},\beta_{2},\beta_{3}$ are zero.
Also from equs.(37) and (38) we have

$\rho_{{}_{DE}}=\begin{cases}\gamma+\frac{1}{C^{\frac{3}{2}}},&\text{for
$\omega=-1$}.\\\ \frac{(1+\gamma
C^{\frac{3}{2}})^{\frac{1}{3}}}{C^{\frac{3}{2}}},&\text{for
$\omega=-\frac{2}{3}$}\\\ \frac{(1+\gamma
C^{\frac{3}{2}})^{2}}{C^{\frac{3}{2}}},&\text{for
$\omega=-\frac{3}{2}$}\end{cases}$ (44)

$\rho=\begin{cases}\frac{(1+\gamma C^{\frac{3}{2}})}{C^{3}},&\text{for $\omega
--1$}\\\ \frac{(1+\gamma C^{\frac{3}{2}})^{\frac{1}{3}}}{C^{3}},&\text{for
$\omega=-\frac{2}{3}$}.\\\ \frac{(1+\gamma
C^{\frac{3}{2}})^{2}}{C^{3}},&\text{for $\omega=-\frac{3}{2}$}\par\end{cases}$
(45)

The development of apparent horizon in this case are,

$C_{{}_{AH}}(1+\gamma
C_{{}_{AH}}^{\frac{3}{2}})=K_{1}(r_{{}_{AH}})\mu(t_{{}_{AH}})\quad\textrm{for
$\omega$= -1}$ (46)

$C_{{}_{AH}}(1+\gamma C_{{}_{AH}}^{\frac{3}{2}})^{\frac{1}{3}}(3+\gamma
C_{{}_{AH}}^{\frac{3}{2}})=K_{2}(r_{{}_{AH}})\mu(t_{{}_{AH}})\quad\textrm{for
$\omega=-\frac{2}{3}$}$ (47)

and

$C_{{}_{AH}}(1+\gamma C_{{}_{AH}}^{\frac{3}{2}})^{2}(2-\gamma
C_{{}_{AH}}^{\frac{3}{2}})=K_{3}(r_{{}_{AH}})\mu(t_{{}_{AH}})\quad\textrm{for
$\omega=-\frac{3}{2}$}$ (48)

where $C_{{}_{AH}}$ being surface radius of apparent horizon at instant
$(t_{{}_{AH}},r_{{}_{AH}})$. The equations (46)-(48) have no real solution for
$C_{{}_{AH}}$, therefore the collapse denied to the formation of apparent
horizon surface. It also follows from equs.(44)-(45) that the collapsing star
has only central singularity when $C(t,0)=0$.

(B). m=1

In this case For $\omega=-1$, eqn.(39) yields
when $\omega=-1$

$C=\eta_{1}(t)\xi_{1}(r)$ (49)
$\frac{\dot{C}}{C}=\frac{\dot{\eta_{1}}}{\eta_{1}}$ (50)
$\rho=\frac{1}{\eta_{1}^{3}\xi_{1}^{3}},\quad\rho_{{}_{DE}}=1$ (51)

where $\eta_{1}(t)$ and $\xi_{1}(r)$ are arbitrary constants of integration.
The development of apparent horizon in this case is

$C_{{}_{AH}}=\xi_{1}(r_{{}_{AH}})\mu(t_{{}_{AH}})$ (52)

From equs.(50)-(51) we see that the singularity occured when either
$\eta_{1}(t)=0$ or $\xi_{1}(r)=0$ or both zero and equ.(52) reveals the
formation of apparent horizon.
When $\omega=-\frac{2}{3}$ and $\omega=-\frac{3}{2}$, the equ.(39) yields

$\frac{\dot{C}}{C}=\begin{cases}\frac{\eta_{2}(t)(1+\gamma
C^{3})^{\frac{1}{3}}}{(3+\gamma C^{3})},&\text{for $\omega=-\frac{2}{3}$}.\\\
\frac{\eta_{3}(t)(1+\gamma C^{3})^{\frac{1}{2}}}{(2+\gamma C^{3})},&\text{for
$\omega=-\frac{3}{2}$}\end{cases}$ (53)

where $\eta_{2}(t),\eta_{3}(t)$ are arbitrary and

$\begin{split}1+\gamma C^{3}-\frac{6(1+\gamma C^{3})^{\frac{1}{3}}}{(\gamma
c^{3})^{\frac{1}{3}}}\textrm{Hypergeometric}2F1\left[\frac{1}{3},\frac{1}{3},\frac{4}{3},-\frac{1}{\gamma
c^{3}}\right]-\\\ 2(1+\gamma
C^{3})^{\frac{1}{3}}\eta_{4}(t)=0\quad\textrm{for}\quad\omega=-\frac{2}{3}\end{split}$
(54)

and

$2\textrm{tanh}^{-1}(1+\gamma C^{3})^{\frac{1}{2}}-(1+\gamma
C^{3})^{\frac{1}{2}}+\frac{3}{2}\eta_{5}(t)=0\quad\textrm{for
$\omega=-\frac{3}{2}$}$ (55)

The development of of apparent horizon

$C_{{}_{AH}}(3+\gamma
C_{{}_{AH}}^{3})-\xi_{2}(r_{{}_{AH}})\mu(t_{{}_{AH}})(1+\gamma
C_{{}_{AH}}^{3})^{\frac{1}{3}}=0\quad\textrm{for $\omega=-\frac{2}{3}$}$ (56)

$C_{{}_{AH}}(2+\gamma
C_{{}_{AH}}^{3})-\xi_{3}(r_{{}_{AH}})\mu(t_{{}_{AH}})(1+\gamma
C_{{}_{AH}}^{3})^{\frac{1}{2}}=0\quad\textrm{for $\omega=-\frac{3}{2}$}$ (57)

here $C_{{}_{AH}}$ is the surface radius of apparent horizon at moment
$(t_{{}_{AH}},r_{{}_{AH}})$ and $\xi_{2},\xi_{3}$ are arbitrary. Since
equs.(56) and (57) have no real solution, therefore collapsing star do not
form apparent horizon for $\omega=-\frac{2}{3}$ and$-\frac{3}{2}$. It can also
be seen from equ.(53) that collapsing process will not end untill $\eta_{2}$
and $\eta_{3}$ are zero.

##  6 Conclusing Remarks

Owing to the perplexing nature of dark energy, its study got much interest in
the last two decades([31, 32, 33] and references therein). Due to negative
pressure, there are speculation that collapse with dark energy component
should not form black hole. However, some recent works has been consider the
gravitational collapse of spherically symmetric homogeneous star in background
of dark energy and dark matter and discuss their singularity formation
([21],[24],[34]-[37]). In this paper, we have considered the gravitational
collapse of an inhomogeneous spherical star consisted of dust cloud and dark
enegry and discuss the problem of black hole formation.We considered three
distinct cases of gravitational collapse seperately- (i) collapse with dust
cloud only (ii) collapse with dark energy only (iii) gravitational collapse
with combined effect of dust and dark energy. We have discussed the formation
of black hole by the development of apparent horizon in collapsing process.
The obtained results in gravitational collapse of dust cloud case manifest the
formation of black hole which consistent with the oppenheimer and Synder
result[1]. For dark energy in the form of perfect fluid has considered with
the two candidate of dark energy components $\omega<-\frac{1}{3}$ and
$\omega<-1$ (phantom). When $\omega=-\frac{2}{3}$, we observed the formation
of apparent horizon (black hole) under the condition
$\beta(r_{e})-\beta(r_{{}_{AH}})<\lambda_{1}(t_{{}_{AH}})$. For
$\omega=-\frac{3}{2}$, it is observed that due to presence of phantom fluid,
the collapse end as black hole when
$\lambda_{2}(t_{{}_{AH}})>\alpha(r_{e})-\alpha(r_{{}_{AH}})$. Thus we see that
the collapsing stars which consist only dark energy fluid, may also leads to
the Black-hole formation.
In section(5) we consider the combined effect of dust cloud and dark energy on
the gravitational collapse. To study the effect of interaction between dust
cloud and dark energy we consider the gravitational collapse by assuming that
$\rho_{{}_{DE}}=\gamma\rho C^{3m}$, where $\gamma$ and $m$ are arbitrary
constants[24, 38] and equation of state $p_{{}_{DE}}=\omega\rho_{{}_{DE}}$,
$\omega$ being non-zero constant parameters. By considering several specific
models for different value of $m=\frac{1}{2},1$ and
$\omega=-1,-\frac{2}{3},-\frac{3}{2}$ we obtained the analogous conclusions
that gravitational collapse denied the formation of apparent horizon, that is,
black- holes do not form due to the collapse of dust cloud interacting with
dark energy. Although when $m=1$, $\omega=-1$ (cosmological constant) the
collapse may interpret the development of apparent horizon. In ref.[24][39]
several collapsing models on homogeneous and isotropic fluids have been
discussed and found that black hole may be formed by the gravitational
collapse even when considering the interaction between dust cloud and dark
energy. Since the collapsing stars that consist of homogeneous and isotropic
fluid are very ideal case, and in more realistic cases are the internal region
of the collapsing star should be inhomogeneous. Therefore we believe that the
results of the present paper will be in more realistic cases. However our
results obtained in this paper do not seemingly support the speulations that
black holes do not exists due to presence of dark energy.

## References

  * [1] J Robert Oppenheimer and Hartland Snyder. On continued gravitational contraction. Physical Review, 56(5):455, 1939.
  * [2] SR Maiti. Fluid with heat flux in a conformally flat space-time. Physical Review D, 25(10):2518, 1982.
  * [3] Bijan Modak. Cosmological solution with an energy flux. Journal of Astrophysics and Astronomy, 5(3):317–322, 1984.
  * [4] A Banerjee, SB Dutta Choudhury, and Bidyut K Bhui. Conformally flat solution with heat flux. Physical Review D, 40(2):670, 1989.
  * [5] Dirk Schafer and Hubert F Goenner. The gravitational field of a radiating and contracting spherically-symmetric body with heat flow. General Relativity and Gravitation, 32(11):2119–2130, 2000.
  * [6] BVIvanov. Collapsing shear-free perfect fluid spheres with heat flow. General Relativity and Gravitation, 44(7):1835–1855, 2012.
  * [7] L Herrera, G Le Denmat, NO Santos, and A Wang. Shear-free radiating collapse and conformal flatness. International Journal of Modern Physics D, 13(04):583–592, 2004.
  * [8] Soumya Chakrabarti and Narayan Banerjee. Scalar field collapse in a conformally flat spacetime. The European Physical Journal C, 77(3):1–9, 2017.
  * [9] Andrew R Liddle and Robert J Scherrer. Classification of scalar field potentials with cosmological scaling solutions. Physical Review D, 59(2):023509, 1998.
  * [10] Paul J Steinhardt, Limin Wang, and Ivaylo Zlatev. Cosmological tracking solutions. Physical Review D, 59(12):123504, 1999.
  * [11] Gia Dvali, Gregory Gabadadze, and Massimo Porrati. 4d gravity on a brane in 5d minkowski space. Physics Letters B, 485(1-3):208–214, 2000.
  * [12] Cedric Deffayet. Cosmology on a brane in minkowski bulk. Physics Letters B, 502(1-4):199–208, 2001.
  * [13] Daniele Malafarina, Bobir Toshmatov, and Naresh Dadhich. Dust collapse in 4d Einstein–Gauss–Bonnet gravity. Physics of the Dark Universe, 30:100598, 2020.
  * [14] James M Cline and Jeremie Vinet. Problems with time-varying extra dimensions or “cardassian expansion” as alternatives to dark energy. Physical Review D, 68(2):025015, 2003.
  * [15] Yungui Gong, Anzhong Wang, and Qiang Wu. Cosmological constant and late transient acceleration of the universe in the horava–witten heterotic m theory on s1/z2. Physics Letters B, 663(3):147–151, 2008.
  * [16] PR Pereira, MFA Da Silva, and R Chan. Anisotropic self-similar cosmological model with dark energy. International Journal of Modern Physics D, 15(07):991–999, 2006.
  * [17] CFC Brandt, R Chan, MAF Da Silva, and Jaime F Villas Da Rocha. Inhomogeneous dark energy and cosmological acceleration. General Relativity and Gravitation, 39(10):1675–1687, 2007.
  * [18] Vera C Rubin, W Kent Ford Jr, and Norbert Thonnard, The Astrophysical Journal, 225:L107–L111, 1978.
  * [19] Vera C Rubin,WKent Ford Jr, and Norbert Thonnard. Rotational properties of 21 sc galaxies with a large range of luminosities and radii, from ngc 4605/r= 4kpc/to ugc 2885/r= 122 kpc. The Astrophysical Journal, 238:471– 487, 1980.
  * [20] Prabir Rudra, Ritabrata Biswas, and Ujjal Debnath. Presence of dark energy and dark matter: does cosmic acceleration signifies a weak gravitational collapse? Astrophysics and Space Science, 342(2):557–574, 2012.
  * [21] Syed Zaheer Abbas, Hasrat Hussain Shah, Huafei Sun, Farook Rahaman, and Faizuddin Ahmed. Gravitational collapse of dust fluid and dark energy in the presence of curvature: Black hole formation. Modern Physics Letters A, 34(29):1950240, 2019.
  * [22] Pawel O Mazur and Emil Mottola. Gravitational vacuum condensate stars. Proceedings of the National Academy of Sciences, 101(26):9545–9550, 2004.
  * [23] David F Mota. On the spherical collapse model in dark energy cosmologies, Astronomy & Astrophysics, 421(1):71–81, 2004.
  * [24] Rong-Gen Cai and Anzhong Wang. Black hole formation from collapsing dust fluid in a background of dark energy. Physical Review D, 73(6):063005, 2006.
  * [25] Pedro G Ferreira and Michael Joyce. Structure formation with a self-tuning scalar field. Physical Review Letters, 79(24):4740, 1997.
  * [26] Soma Nath, Subenoy Chakraborty, and Ujjal Debnath. Gravitational collapse due to dark matter and dark energy in the braneworld scenario. International Journal of Modern Physics D, 15(08):1225–1236, 2006
  * [27] . Hasrat Hussain Shah, Farook Rahaman, Amna Ali, and Sabirudin Molla. Gravitational collapse of an interacting vacuum energy density with an anisotropic fluid. Physics of the Dark Universe, 24:100291, 2019.
  * [28] Eugeny Babichev, Vyacheslav Dokuchaev, and Yu Eroshenko. Black hole mass decreasing due to phantom energy accretion. Physical Review Letters, 93(2):021102, 2004.
  * [29] Caldwell, R.R. , Kamionkowski, M. , and Weinberg, N.N. (2003). Phantum Energy: Dark Energy with $w<-1$ Causes a cosmic Doomsday. Physical Review Letters, 91(7). doi.1103/physervlett91.071301
  * [30] Caldwell, R. . (2002). A phantom menace? Cosmological consequences of a dark energy component with super-negative equation of state. Physics Letters B, 545(1-2), 23–29. doi:10.1016/s0370-2693(02)02589-3
  * [31] V.Sahni, "Dark Matter and Dark Energy," arXiv:astro-ph/0403324(2004).
  * [32] V. Sahni, "Cosmological Surprises from Braneworld models of Dark Energy," arXiv:astro-ph/0502032(2005).
  * [33] Ma, C.-P., Caldwell, R. R., Bode, P., & Wang, L. (1999). The Mass Power Spectrum in Quintessence Cosmological Models. The Astrophysical Journal, 521(1), L1–L4. doi:10.1086/312183 .
  * [34] Chakraborty, S., & Bandyopadhayaya, T. (2008). Collapsing Inhomogeneous Dust In The Background Of Perfect (Or Anisotropic) Fluid. International Journal of Modern Physics D, 17(08), 1271–1281. doi:10.1142/s0218271808012814
  * [35] LI, Z.-H., & WANG, A. (2007). Existence Of Black Holes In Friedmann–Robertson–Walker Universe Dominated By Dark Energy. Modern Physics Letters A, 22(22), 1663–1676. doi:10.1142/s0217732307024048
  * [36] H.H. Shah, F.Rahaman. A.Ali and S. Molla, Gravitational Collapse of an interacting vacumm energy density with an anisotropic fluid, Physics of the dark Universe, 24:100241(2019).
  * [37] Chan, R., da Silva, M. F. A.,& Villas da Rocha, J. F. (2009). Star models with dark energy. General Relativity and Gravitation, 41(8), 1835–1851. doi:10.1007/s10714-008-0755-9 .
  * [38] Cai, R.-G., & Wang, A. (2005). Cosmology with interaction between phantom dark energy and dark matter and the coincidence problem. Journal of Cosmology and Astroparticle Physics, 2005(03), 002–002. doi:10.1088/1475-7516/2005/03/002
  * [39] Shah, H. H., & Iqbal, Q. (2017). Gravitational collapse of dark matter interacting with dark energy: Black hole formation. International Journal of Modern Physics D, 26(13), 1750142. doi:10.1142/s0218271817501425 .


# Ego2Hands: A Dataset for Egocentric Two-hand Segmentation and Detection

Fanqing Lin
Brigham Young University
<EMAIL_ADDRESS>Brian Price
Adobe Research
<EMAIL_ADDRESS>Tony Martinez
Brigham Young University
<EMAIL_ADDRESS>
###### Abstract

Hand segmentation and detection in truly unconstrained RGB-based settings is
important for many applications. However, existing datasets are far from
sufficient in terms of size and variety due to the infeasibility of manual
annotation of large amounts of segmentation and detection data. As a result,
current methods are limited by many underlying assumptions such as constrained
environment, consistent skin color and lighting. In this work, we present
Ego2Hands, a large-scale RGB-based egocentric hand segmentation/detection
dataset that is semi-automatically annotated and a color-invariant
compositing-based data generation technique capable of creating training data
with large quantity and variety. For quantitative analysis, we manually
annotated an evaluation set that significantly exceeds existing benchmarks in
quantity, diversity and annotation accuracy. We provide cross-dataset
evaluation as well as thorough analysis on the performance of state-of-the-art
models on Ego2Hands to show that our dataset and data generation technique can
produce models that generalize to unseen environments without domain
adaptation.

Figure 1: Our proposed dataset and training scheme enables domain
generalization for two-hand segmentation and detection. Given an image with a
new environment and hands not present in the training data (top), the trained
models can provide accurate segmentation and detection results for both hands
regardless of inter-hand occlusion (bottom).

##  1 Introduction

With the rapid growing usage of wearable technologies generating massive
volumes of egocentric image data [4, 3, 2, 1], the ability for machines to
understand the human hands becomes crucial for applications such as human-
computer interaction (HCI), activity logging, gesture/sign language
recognition and VR/AR. Consequently, hand detection and segmentation are
fundamental in areas such as 2D/3D hand pose estimation [24, 19, 16] and
gesture recognition [6, 12]. However, hand segmentation on images in the wild
is extremely challenging due to numerous factors: vastness of the color space,
different skin color/texture, inter-hand occlusion, complex background noise,
motion blur, lighting type/color, shadow features, speed and model size
requirement, etc. As a result, existing color-based approaches can only
perform in constrained environments with proper lighting and skin color
consistent with the training data. These limitations are largely due to the
lack of annotated segmentation data, a common limiting factor for segmentation
tasks because manual annotation is oftentimes required but infeasible for
large-scale data generation.
In this work, we aim to push the boundary for the task of real-time egocentric
two-hand segmentation and detection on images in the wild (Fig. 1). Since hand
segmentation and detection are highly correlated and both imperative for
subsequent applications, we find it natural to tackle both tasks
simultaneously.
We first address the issue of the lack of annotated data. In general, real-
world RGB data with segmentation ground truth is very labor-intensive to
annotate. For this reason, existing hand segmentation datasets [15, 13, 6, 12,
14, 10] lack the quantity and sufficient variety necessary for learning-based
approaches. Although synthetic data [16] with perfect ground truth can be
generated with little cost, it is more challenging for methods trained on
synthetic data to generalize to real-world data as CNNs are sensitive to
textural differences between domains. We propose to collect massive amounts of
segmentation data for the egocentric right hands that can be automatically
annotated in a green screen setting, and a novel data generation technique
that composites training instances by combining a pair of randomly selected
right hands with one horizontally flipped as the left hand. In addition, we
introduce a novel semi-automated annotation method for the heatmap energy of
the hand. Unlike hand segmentation, the hand energy excludes the arm and can
be used for hand detection. In order to develop a color-invariant approach, we
explore the grayscale image space coupled with an edge map as the input space
and show successful generalization to unseen environments. This data
generation method can push segmentation models beyond the limitation of a
fixed-sized training set and evaluation set and enable models to produce
accurate segmentation and detection results in unseen environments without
domain adaptation, which can also be easily applied to further improve model
accuracy for specific environments.
We introduce Ego2Hands which includes a training set with $\sim$180,000 unique
right-hand instances and an evaluation set with 2,000 manually annotated
frames from diverse video sequences. In-depth comparison and cross-dataset
evaluation between Ego2Hands and previous datasets show the superiority of our
dataset in quantity, diversity, annotation accuracy and generalization
ability. For quantitative analysis, we provide comprehensive comparison
between the state-of-the-art approaches on our dataset.
To demonstrate the benefits of having fast and accurate two-hand segmentation
and detection results, we show real-time gesture control in the supplementary
video. Our work opens up promising directions for two-hand gesture control
systems using only low-cost RGB input devices.

##  2 Related Works

Depth-based datasets. Early works [25, 23] have proposed depth based datasets
for single-hand segmentation. [7] introduced a dataset that enabled two-hand
segmentation from a third-person viewpoint. Additionally, [24, 19] presented
datasets with two-hand segmentation data for egocentric pose reconstruction
and hand tracking, which indicates the significance of two-hand segmentation
for other gesture-related tasks as it provides information not only on the
location and the shape of the hands, but also on how interacting hands are
occluded by each other or other objects.
In this work, we focus on a RGB-based setting as depth cameras have additional
setup overhead and indoor requirements with higher power consumption and cost.
This leads to limited applicable applications compared to the ubiquitous RGB
cameras. We point out that existing depth-based datasets obtain the
segmentation ground truth using color thresholding and require subjects to
wear thin colored gloves. As a result, depth-based hand segmentation datasets
are not suitable for training RGB-based approaches.
Color-based datasets. Pioneering work [13] contributed three egocentric videos
(EDSH1, EDSH2 and EDSH-kitchen) with varying illumination for binary hand
segmentation. For activity recognition, [15] proposed the Georgia Tech
Egocentric Activity Dataset (GTEA) with 625 frames consisting of two-hand
labeling and 38 frames with binary-labeling. [14] later published an extended
version (EGTEA) with 12,799 frames with binary-labeling and 1,048 frames with
two-hand labeling. To enable hand segmentation in more unconstrained settings,
[6] introduced EgoHands as the first large-scale hand segmentation dataset
with 4,800 annotated frames consisting of a maximum of 4 interacting hands.
For the same purpose, [12] additionally introduced EgoYouTubeHands (EYTH) with
$\sim$1290 binary-labeled frames from three Youtube videos and HandOverFace
(HoF) with 300 annotated frames from third-person Web images. To demonstrate
cross-dataset adaptation performance, [10] provided binary-labelling for 855
and 488 frames from human grasping datasets UTG [9] and YHG [8] respectively.
To address the issue of data scarcity, [16] introduced a large-scale synthetic
dataset (Ego3DHands) with a total of over 100,000 annotated frames on two
hands.

##  3 Ego2Hands

There are several major factors that prevent existing datasets from enabling
generalization of two-hand segmentation in unseen domains. First, many
datasets [13, 12, 10] only contain binary-label for hand segmentation, which
does not distinguish between the left and right hand. For datasets that
contain frames with two-hand labels, the quantity of annotated data is very
limited due to the need for manual segmentation annotation. The annotated
frames also possess limited diversity in terms of background environment,
lighting, skin tone and range of motion. The aforementioned properties are
obtainable for synthetic data, however, models trained on synthetic data
generally generalize poorly to real-world data due to the domain gap.
To enable generalization to real-world domains, we introduce a large-scale
dataset Ego2Hands that consists of 188,362 annotated frames in the training
set for the right hand. Segmentation masks are obtained by automatically
removing the background in a green screen setting. 22 participants with
diverse skin colors and hand features are instructed to perform free hand
motion that covers a wide range of locations/poses while recording using a
head-mount webcam (Logitech C922) at 30 fps. This process allows simple and
fast data collection for segmentation data. We obtained consent from all
participants and the collected egocentric hand data does not contain
personally identifiable information.

(a) Original output channel

(b) Modified output channels

Figure 2: Addition of the convolutional output layer to enable estimation of
both two-hand segmentation and energy.

###  3.1 Hand Energy Annotation

There are two major types of representation for hand segmentation. Some
datasets [15, 13, 14, 10] include the arm in segmentation and neglect the task
of hand detection, while others [6, 12] contain hand segmentation that
excludes the arm, therefore combining the task of segmentation and detection
into one. The disadvantage of the second representation is the loss of
detection information during inter-hand occlusion. To enable both hand
segmentation and detection, we choose to provide annotation for segmentation
as well as detection data in the form of heatmap energy.
Although segmentation annotation can be automatically obtained by green screen
removal, obtaining the hand energy annotation for over 180k frames that
excludes the arm is a nontrivial task. Our annotation process can be divided
into 1) transfer learning using synthetic and real-world data and 2) semi-
automated ”human-in-the-loop” annotation.
In stage 1, we develop a novel transfer learning technique to leverage
segmentation data from two domains and transfer the knowledge of energy
estimation from one to the other. To this end, we propose to add an additional
output layer to the segmentation network for energy estimation as shown in
Fig. 2. We theorize that the additional task of energy estimation should not
conflict with segmentation as they share similar features. We apply sigmoid
activation to the energy output and compute the energy loss
$\mathcal{L}_{energy}$ using Mean Squared Error (MSE). The combined loss for
network optimization can then be formulated as:

$\mathcal{L}_{total}=\mathcal{L}_{seg}+\mathcal{L}_{energy}$ (1)

where $\mathcal{L}_{seg}$ is computed using the standard Cross Entropy Loss.
We show in Section 4 that the modified models can achieve high accuracy on
both tasks simultaneously.
To prepare data from the synthetic domain, we follow [16] by generating an
internal synthetic dataset $\text{Ego3DHands}_{R}$ with only the right hand,
which contains perfect annotation for hand segmentation as well as energy. For
data in the real-world domain, we use the $\sim$180k frames from Ego2Hands
that contain the annotation only for segmentation. To simplify the input
space, all training images have background removed using the hand segmentation
masks. Finally, we jointly train the proposed HandSegNet [16] on both
$\text{Ego3DHands}_{R}$ and Ego2Hands using the following loss:

$\begin{gathered}\mathcal{L}_{combined}=\mathcal{L}^{synth}_{seg}+\mathcal{L}^{real}_{seg}+\mathcal{L}^{synth}_{energy},\end{gathered}$
(2)

where $\mathcal{L}^{synth}$ and $\mathcal{L}^{real}$ are losses computed using
synthetic and real-world data respectively. By learning to estimate hand
segmentation from both domains, the model is pushed to learn shared features
between domains. As a result, knowledge learned using
$\mathcal{L}^{synth}_{energy}$ can be transferred to estimate hand energy on
real-world images from Ego2Hands with promising accuracy.
In stage 2, inspired by interactive video object segmentation methods [17], we
build an annotation tool that uses the trained HandSegNet to automatically
estimate the hand energy given the input images of Ego2Hands. In cases where
the estimation becomes inaccurate, the human annotator manually corrects the
error to generate the ground truth. HandSegNet is immediately finetuned on the
corrected data for more accurate future predictions. This semi-automated
annotation method allows us to generate accurate hand energy for Ego2Hands in
a feasible manner. Energy annotation samples are included in the supplementary
document.

Figure 3: Pipeline for training data composition. Ground truth data includes
segmentation and energy annotation for the left and right hand.

###  3.2 Data Composition for Training

We use randomly selected pairs of right hand images (with one flipped as the
left hand) in the training set of Ego2Hands for data composition. Although
actual images of the two hands are more realistic than the composited version,
we point out that obtaining two-hand segmentation annotation is infeasible at
large-scale if both hands are present in the collected images since green
screen removal creates a binary segmentation label. For the background images,
we use the 19,216 images provided by [16] with the additional 14,997 high-
quality images in the DAVIS datasets [20, 21]. This results in approximately
$1.21\text{\times}{10}^{15}$ unique hand-scene combinations prior to data
augmentation.
With the obtained hand energy, we are able to composite more realistic
training images by selecting the proper overlaying order. After random
selection of the left and right hand, the hand with the larger energy sum is
selected to be overlaid on top of the other hand. To obtain a smooth
compositing boundary, we apply erosion and gaussian blur on the alpha-channel.
For each composited image, we further data augment by applying 1) random
horizontal and downward vertical translation within reasonable ranges on each
hand, 2) random smoothing with various kernel sizes to simulate blur from
motion or auto-focus, 3) color augmentation on the hands and background
images, 4) random horizontal flips and cropping on background images, and 5)
10% drop rate for each hand to accommodate single-hand scenarios. To enable
domain adaptation on specific environments, we can simply use the background
images collected from that scene for compositing training images. Unlike
conventional datasets with fixed sizes, we composite images at training time
for much higher quantity and more diversity, which is essential for domain
generalization. Fig. 3 illustrates our data composition process.

###  3.3 Evaluation Set

To support quantitative evaluation, we introduce an evaluation set that
includes 8 videos each with 250 annotated frames. We select 4 additional
participants with diverse skin tones to perform free two-hand motion in 8
different scenes under various lighting conditions. We manually annotate the
segmentation as well as the energy for the left and right hand. Fig. 4 shows a
comparison of annotation quality between Ego2Hands and other datasets. We show
additional qualitative examples of all 8 sequences in the supplementary
document to demonstrate the annotation quality and diversity in our sequences.

###  3.4 Comparison with Existing Datasets

We show in Table 1 a detailed comparison between Ego2Hands and existing
benchmark datasets. For real-world datasets with two-hand labeling (EgoHands,
EGTEA and GTEA), the largest existing training set is EgoHands that contains
3,600 annotated frames (36/12 split) with 3 scenes in total.
$\text{Ego2Hands}_{train}$ consists of 188,362 annotated frames for
compositing two-hand training data that can generalize to real-world data. For
a comparison between existing test sets, we notice a critical issue that the
three aforementioned datasets have test sets that share the same subjects and
scenes with their training sets. On top of the small quantity and diversity,
this setting further weakens evaluation of the methods’ generalization
ability. Our $\text{Ego2Hands}_{test}$ not only contains data with diverse
subjects and scenes unseen in the training set, but also provides 2,000 frames
for comprehensive evaluation, which is more than the test sets of EgoHands
(1,200 frames), GTEA (63 frames) and EGTEA (105 frames) combined.

Figure 4: Illustration of the difference in annotation accuracy between datasets. Existing datasets contain false positive labeling for gaps and holes and potentially inaccurate boundaries (Best viewed in magnification. Annotated masks are overlaid in colors.) Datasets |  Type |  | #Annotated
---
Frames
| #Hand
---
Instances
#Subjects |  #Scenes |  Objects |  #Classes |  Resolution
GTEA [15] |  Real |  663 |  1231 |  4 |  1 |  Yes |  2 & 3 |  $720\times 405$
EDSH [13] |  Real |  743 |  - |  1 |  3 |  Yes |  2 |  $1280\times 720$
EgoHands [6] |  Real |  4800 |  15053 |  4 |  3 |  Yes |  5 |  $1280\times 720$
EYTH [12] |  Real |  1290 |  2600 |  - |  - |  Yes |  2 |  $384\times 216$
HoF [12] |  Real |  300 |  507 |  - |  - |  No |  3 |  $384\times 216$
EGTEA [14] |  Real |  13847 |  - |  32 |  1 |  Yes |  2 & 3 |  $960\times 720$
UTG [10] |  Real |  855 |  - |  5 |  2 |  Yes |  2 |  $480\times 360$
YHG [10] |  Real |  488 |  - |  4 |  - |  Yes |  2 |  $480\times 360$
Ego3DHands [16] |  Synth |  110,000 |  $\sim$214,500 |  1 |  - |  No |  3 |  $960\times 540$
Ego2Hands (Ours) |  Real |  | 188,362 (train)
---
2,000 (test)
|  $\infty$ (train)
---
4,000 (test)
| 22 (train)
---
4 (test)
| \- (train)
---
8 (test)
No |  3 |  $800\times 448$
Table 1: Statistics of available hand segmentation datasets. Datasets with
#Classes = 2 only support binary segmentation.

Figure 5: Visualization of heatmap for spatial hand occurrence, histogram for
hand area relative to the image size and the color distribution of the hands
for each dataset. Color distribution for the training set of Ego2Hands is not
shown since color augmentation is applied.

We compare other dataset attributes such as diversity in hand locations, hand
sizes and skin colors in Fig. 5. We show that Ego2Hands contains hand
instances evenly distributed in a wide range of locations while other datasets
have hand spatial occurrence very concentrated in specific regions. Ego2Hands
also demonstrates the best hand size distribution while other datasets provide
hand size distribution that is heavily biased towards small sizes (the
computed hand area of GTEA and EGTEA includes the arm). Hand color
distribution is a very important topic rarely discussed in previous work. We
see that EgoHands, GTEA and EGTEA all contain very limited variation in skin
color, which is largely attributed to the limited number of subjects and
scenes. On the other hand, $\text{Ego2Hands}_{test}$ provides skin color that
spans a broader spectrum, covering color range that reaches close to $(0,0,0)$
and $(255,255,255)$.
In summary, it is more difficult for existing datasets to support
generalization on real-world data given their limitations in quantity and
diversity. In comparison, both $\text{Ego2Hands}_{train}$ and
$\text{Ego2Hands}_{test}$ provide significant improvements in all dataset
attributes while additionally enabling two-hand detection.

###  3.5 Color-invariant Input Domain

Despite the increased quantity in training instances, it is still challenging
for deep networks to learn the complete RGB space. For instance, for hands
under a particular-colored lighting (e.g. blue), learning-based models would
need sufficient training data with hands in that specific color. This issue is
oftentimes overlooked by previous works as their proposed datasets contain
skin tone and lighting with limited variation. Consequently, we explore the
grayscale image space coupled with an image edge map as input for a color-
invariant approach. In the grayscale domain, we find two major factors crucial
for generalization in the real-world domains: brightness and shadow features.
For diversity in brightness, we replace color augmentation with brightness
augmentation and scale the pixel values of both hands to shift the means to a
randomly selected value $\beta\in[15,240]$ while clipping pixel values within
[0, 255]. Variation in the brightness of the hands also contributes
significantly to diversity in skin tones. For different shadow features, we
include light sources from various directions during data collection (Fig. 6).
For quantitative comparison, we also include results obtained using the normal
RGB color domain to support this design choice in a later section.

Figure 6: Hand images (grayscale) with different visual shadow features from different lighting directions. Dataset |  | Ego2Hands
---
S = 8, F = 2,000
| Ego3DHands
---
S = NA, F = 5,000
| EgoHands
---
S = 3, F = 4,800
| EGTEA
---
S = 1, F = 1,048
| GTEA
---
S = 1, F = 625
Ego2Hands (Ours) | 0.832 | 0.749 | 0.330 | 0.638 | 0.602
Ego3DHands [16] | 0.502 | 0.953 | 0.399 | 0.284 | 0.086
EgoHands [6] | 0.262 | 0.160 | 0.824 | 0.541 | 0.376
EGTEA [14] | 0.557 | 0.591 | 0.411 | 0.910 | 0.698
GTEA [15] | 0.113 | 0.069 | 0.248 | 0.376 | 0.930
Table 2: Cross-dataset evaluation computed using mIoU on hand segmentation
datasets with two-hand annotation. Each row and column represent the training
and evaluation set respectively. The number of different scenes and frames
included in each dataset are shown for each column. The top-3 scores on each
evaluation dataset are marked as first, second and third.

###  3.6 Cross-dataset Generalization

To demonstrate that the composited data using $\text{Ego2Hands}_{train}$
enables generalization to the real-world domains, we first perform cross-
dataset evaluation on 5 existing datasets with two-hand labeling. For the base
architecture, we select RefineNet [18] for its high generalization ability
across various datasets [12]. The proposed compositing-based data augmentation
on grayscale images coupled with edge map is used for training using
Ego2Hands. For training using other datasets, we apply color/smoothness
augmentation on the original RGB images. For the training of each dataset, the
author-defined or a 90/10 split is used. For evaluation, we use the test sets
of Ego2Hands and Ego3DHands which contain 2k and 5k instances respectively.
Since the other datasets contain fewer instances, we evaluate on the entire
dataset for more accurate results. Note that EgoHands contains 4-hand
annotation and only the 2 egocentric hands are used for consistency. Averaged
mean Intersection over Union (mIoU) computed on the segmentation of two hands
from 3 trained model instances is reported.
It is generally expected for a model to achieve the best score on the dataset
it was trained on, for this reason, we focus on the generalization scores
obtained by evaluating on other datasets. Table 2 shows that training on
Ego2Hands achieves the best generalization results on Ego3DHands and EGTEA.
Ego2Hands also achieves high scores on GTEA despite performing slightly worse
compared to EGTEA. However, both EGTEA and GTEA are collected in 1 kitchen
scene that contains very similar features. We find that training on Ego2Hands
gives lower scores on EgoHands, which contains lower-quality images that can
lead to a domain gap in the edge map space.
Unlike other datasets that contain small-scale training/test sets that share
the same subjects and scenes with similar features, Ego2Hands has test
sequences and subjects not present in the training data. As a result, the best
evaluation score of a mIoU = 0.832 obtained on $\text{Ego2Hands}_{test}$ using
models trained on $\text{Ego2Hands}_{train}$ also indicates the strong
generalization ability of our dataset. We argue that evaluating on Ego2Hands
which contains scenes and subjects with diverse features produces more
comprehensive results compared to other datasets. For example, we show in the
supplementary document that models trained on other datasets all fail to
generalize to sequence 2, 5, 6 and 8 of $\text{Ego2Hands}_{test}$, which
contain skin color and illumination different from the data distribution of
existing datasets. In addition, since only Ego2Hands and Ego3DHands contain
instances with extensive inter-hand occlusion, models trained on other
datasets perform poorly when hands occlude each other or cross over.
Interestingly, Ego3DHands with synthetic data achieves decent results on
Ego2Hands and EgoHands, indicating that pretraining or mixed training on
synthetic data can be beneficial for real-world estimation. Surprisingly,
EGTEA achieves promising results across various datasets with only 1,048
training instances. However, the large performance gap between EGTEA and
$\text{Ego2Hands}_{train}$ evaluating on $\text{Ego2Hands}_{test}$ suggests
that small-scale datasets are insufficient for high-quality generalization.

##  4 Quantitative benchmarking

We evaluate existing state-of-the-art methods on the proposed evaluation set
of Ego2Hands (8 sequences each with 250 annotated frames) and compare the two-
hand segmentation and detection accuracy as well as the corresponding model
sizes and inference speed for benchmarking. We use the mIoU as the metric for
the segmentation task. For hand detection, we use the conventional metric of
Average Precision that classifies a detection bounding box as correct if its
IoU between the ground truth bounding box exceeds 50% ($AP_{0.5}$). The
predicted bounding boxes are obtained using the output energy thresholded at
0.5. The closing operation with a kernel size of 7 is performed on the energy
for noise removal. The ground truth bounding boxes are obtained using the
manually annotated hand energy heatmaps.
We compare the models’ performance in the RGB domain, with/without the edge
map in the color invariant domain and with/without the additional output
energy channel as an ablation study to justify our design choices. As it is
impossible for static pretrained models to produce highly accurate results in
all scenes, we also perform experiments to study the impact of domain
adaptation. To support scene-specific adaptation, we include a collected
background sequence ($\sim$30 seconds) for each evaluation sequence where the
scene-specific background images can be used to composite training instances.
The background collection process simulates an environment scanning procedure
using prospective egocentric color-based hand tracking devices.
The following state-of-the-art architectures are selected for evaluation: UNet
and $\text{UNet}_{1/8}$ (1/8 of the original network width) [22], RecUNet
[26], SegNet [5], ICNet [27], DeepLab V3+ [11] and RefineNet [18].

Model | #Params |  | Inference
---
time (ms)
| Pretrained
---
|  RGB energy ✗ |  edge ✗ energy ✗ |  edge ✓ energy ✗ |  w/ edge & energy mIoU $AP_{0.5}$
---|---|---|---
| Adapted
---
|  w/ edge & energy mIoU $AP_{0.5}$
---
$\text{UNet}_{1/8}$ [22] | 0.2M | 9.5 |  |  0.405 |  0.722 |  0.749 |  0.754 |  0.633
---|---|---|---|---
|  0.844 |  0.739
---|---
RecUNet [26] | 1.1M | 78.1 |  |  0.615 |  0.812 |  0.834 |  0.844 |  0.805
---|---|---|---|---
|  0.874 |  0.839
---|---
UNet [22] | 13.4M | 10.3 |  |  0.464 |  0.652 |  0.631 |  0.651 |  0.655
---|---|---|---|---
|  0.775 |  0.737
---|---
ICNet [27] | 28.3M | 43.1 |  |  0.792 |  0.828 |  0.824 |  0.823 |  0.886
---|---|---|---|---
|  0.885 |  0.945
---|---
SegNet [5] | 29.4M | 12.3 |  |  0.536 |  0.687 |  0.668 |  0.645 |  0.670
---|---|---|---|---
|  0.789 |  0.787
---|---
DeepLabV3+* [11] | 59.3M | 42.2 |  |  0.617 |  0.729 |  0.777 |  0.777 |  0.821
---|---|---|---|---
|  0.866 |  0.906
---|---
RefineNet* [18] | 113.9M | 50.5 |  |  0.650 |  0.825 |  0.847 |  0.836 |  0.874
---|---|---|---|---
|  0.884 |  0.903
---|---
Table 3: Evaluation of state-of-the-art models on Ego2Hands. Only mIoU is
reported for experiments without the energy output channel (denoted as ”energy
✗”). Models with * are trained using pretrained Resnet encoders.

###  4.1 Training Details

Experiments are divided into the pretraining and the adaptation phase. In the
pretraining phase, models are trained using random background images for 100k
iterations with a batch size of 4 and an initial learning rate of
$1.0\text{\times}{10}^{-4}$ decreased with a ratio of 0.5 every 20k
iterations. In the adaptation phase, we finetune the pretrained models (with
input edge map and output energy channel) for 10k iterations using scene-
specific background images and an initial learning rate of
$1.0\text{\times}{10}^{-5}$ decaying with the same ratio every 5k iterations.
We use the Adam optimizer and find general convergence from all models using
this setup. Averaged results from 3 trained model instances are reported for
each experiment. GeForce RTX 2080 Ti is used as the GPU in our experiments.
As illumination (not skin tone) is the dominant factor for the brightness of
the hands in input images and is known for specific environments, we perform
brightness augmentation within ranges specific to the scenes in the adaptation
phase. The mean brightness value $\beta$ of the composited hands is scaled to
be in the range [0, 55], [55,200], [55, 255] for scenes with dark (seq5),
normal (seq1, 3, 4, 6, 7) and bright (seq2, 8) illumination respectively.
Bright scenes have a wider brightness range due to the possible presence of
shadow. $\beta$ for background images is jittered by $\pm 50$.

###  4.2 Quantitative Analysis

Table 3 provides detailed quantitative results for the selected models on
different settings. We point out that a comprehensive comparison involves
various factors including model size, inference speed, segmentation/detection
accuracy, the ability to generalize and adapt.
First, we perform experiments using the RGB input domain and compare with our
proposed color-invariant domain. To generate more realistic instances in the
color domain, we generate a version of $\text{Ego2Hands}_{train}$ with the
green screen color spill problem resolved using the Keylight feature of Adobe
After Effects. Note that this issue is largely circumvented in our color-
invariant domain. During training, we specifically color-augment the
composited data according to the color distribution of
$\text{Ego2Hands}_{test}$ shown in Fig. 5. Results show that training in the
RGB space does not outperform the color-invariant domain on
$\text{Ego2Hands}_{test}$. In addition, we justify our design choice of the
additional input edge map and energy output channel. Models with an input edge
map and energy output show overall improvement in segmentation while
additionally estimating the output energy, which provides hand detection
information essential for many applications.
In our experiments, UNet and SegNet underperform with lower generalization and
adaptation accuracy. With a small number of parameters, $\text{UNet}_{1/8}$
and RecUNet achieve good segmentation accuracy but suffer from lower detection
accuracy. Additionally, the recursion on internal network states improves
accuracy while notably increasing the inference time, making RecUNet the
slowest models in our analysis. It is worth mentioning that inference speed is
crucial in real-world applications as there are oftentimes subsequent pose
estimation/gesture recognition modules. Heavy models (DeepLabV3+, RefineNet)
generally achieve high pretrained and adapted accuracy in both segmentation
and detection. We find that heavy models are dependent on pretrained encoders
for optimal performance.

Figure 7: Qualitative results obtained using scene-adapted ICNet. Odd rows
show sample images in evaluation sequences with various skin tones and
illumination. Even rows show the output visualization. Best viewed in
magnification.

Our experiments provide valuable insights on the trade-off between
architectures. In general, ICNet produces excellent pretrained and adapted
accuracy for both segmentation and detection with medium model size and
inference speed. In cases where the model size is not a limiting factor,
RefineNet achieves high generalization accuracy, which we leverage for cross-
dataset evaluation. In memory-constrained settings, $\text{UNet}_{1/8}$ has
extremely compact model size and achieves promising accuracy, which RecUNet
further improves by sacrificing inference speed. In time-sensitive
applications, shallow models such as $\text{UNet}_{1/8}$, UNet and SegNet have
faster inference speed with lower accuracy. Interestingly, by simply reducing
the network width, $\text{UNet}_{1/8}$ outperforms UNet in nearly every aspect
for the task of two-hand segmentation and detection.
We reemphasize that the evaluation sequences cover various ranges of
illumination and include hands (various skin tones) and scenes not present in
the training set of Ego2Hands. Our quantitative results show that the proposed
dataset and data composition technique enable models to generalize to the
real-world image domain. To provide a proper perspective for our significantly
increased level of generalization, as [10] recently tried to address the
problem of domain adaptation in a specific unseen environment for binary-label
hand segmentation, we enable models to achieve high accuracy on two-hand
segmentation and detection in a domain-invariant setting with the option to
further improve using scene-specific adaptation. We show qualitative results
in Fig. 2 and demonstrate real-time gesture control using fast and accurate
two-hand segmentation/detection estimation in the supplementary video.

##  5 Future Work

Hand segmentation with objects can be useful in applications that involve
object handling. However, the introduction of objects in green screen data
collection for hand segmentation requires an additional step to segment the
objects, making automatic large-scale annotation of segmentation data
challenging. Although a naive composition of objects in hand images is
achievable, realistic handling of the objects requires additional hand pose
information. Since many applications such as hand tracking in VR/AR,
gesture/sign language recognition focus more on bare hand scenarios, we leave
object interaction as future work.
Although our dataset is the first real-world hand segmentation dataset to
address inter-hand occlusion, it does not contain two-hand segmentation data
for close hand interactions involving interlaced fingers. This is extremely
challenging due to the difficulty in obtaining the corresponding segmentation
ground truth. In general, accurate two-hand segmentation will be very
beneficial for two-hand pose estimation and gesture recognition applications.

##  6 Conclusion

In this work, we introduce a large-scale RGB hand segmentation dataset with a
compositing-based data generation approach for two-hand segmentation and
detection in unconstrained settings. Cross-dataset comparisons show that our
dataset exceeds existing datasets significantly in quantity, diversity,
annotation quality and contains two-hand labeling that addresses inter-hand
occlusion. Validation and analysis of state-of-the-art models on our benchmark
dataset show that training on Ego2Hands enables high-quality generalization to
unseen scenes and scene-specific adaptation can achieve further improvements.

## References

  * [1] Narrative Clip.  http://getnarrative.com/, 2015.
  * [2] GoPro Camera Series.  https://gopro.com/en/us/, 2019.
  * [3] HTC Vive.  https://developer.vive.com/resources/vive-sense/sdk/vive-hand-tracking-sdk/, 2019\.
  * [4] Oculus Quest.  https://www.oculus.com/quest-2/, 2020.
  * [5] V. Badrinarayanan, A. Kendall, and R. Cipolla.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 39:2481–2495, 1 2017.
  * [6] S. Bambach, S. Lee, D. J. Crandall, and C. Yu.  Lending A Hand: Detecting Hands and Recognizing Activities in Complex Egocentric Interactions.  In ICCV, 2015.
  * [7] A. K. Bojja, F. Mueller, S. R. Malireddi, M. Oberweger, V. Lepetit, C. Theobalt, K. M. Yi, and A. Tagliasacchi.  HandSeg: An Automatically Labeled Dataset for Hand Segmentation from Depth Images.  In CRV, 2019.
  * [8] I. M. Bullock, T. Feix, and A. M. Dollar.  The Yale human grasping dataset: Grasp, object, and task data in household and machine shop environments.  The International Journal of Robotics Research, 34, 2015.
  * [9] M. Cai, K. Kitani, and Y. Sato.  An Ego-vision System for Hand Grasp Analysis.  IEEE Transactions on Human-Machine Systems, 47:524–535, 8 2017\.
  * [10] M. Cai, F. Lu, and Y. Sato.  Generalizing Hand Segmentation in Egocentric Videos with Uncertainty-Guided Model Adaptation.  In CVPR, 2020.
  * [11] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam.  Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation.  In ECCV, 2018.
  * [12] A. U. Khan and A. Borji.  Analysis of Hand Segmentation in the Wild.  In CVPR, 2018.
  * [13] C. Li and K. M. Kitani.  Pixel-level Hand Detection in Ego-Centric Videos.  In CVPR, 2013.
  * [14] Y. Li, M. Liu, and J. M. Rehg.  In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video.  In ECCV, 2018.
  * [15] Y. Li, Z. Ye, and J. M. Rehg.  Delving into Egocentric Actions.  In CVPR, 2015.
  * [16] F. Lin, C. Wilhelm, and T. Martinez.  Two-hand Global 3D Pose Estimation Using Monocular RGB.  arXiv preprint arXiv:2006.01320, 2020.
  * [17] F. Lin, C. Yao, and T. Martinez.  Flow Adaptive Video Object Segmentation.  Image and Vision Computing, 2020.
  * [18] G. Lin, A. Milan, C. Shen, and I. Reid.  RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation.  In CVPR, 2017.
  * [19] F. Mueller, M. Davis, F. Bernard, O. Sotnychenko, M. Verschoor, M. A. Otaduy, D. Casas, and C. Theobalt.  Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera.  ACM Transactions on Graphics (TOG), 2019.
  * [20] F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung.  A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation.  In CVPR, 2016.
  * [21] J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbeláez, A. Sorkine-Hornung, and L. Van Gool.  The 2017 DAVIS Challenge on Video Object Segmentation.  arXiv:1704.00675, 2017.
  * [22] O. Ronneberger, P. Fischer, and T. Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  In MICCAI, 2015.
  * [23] T. Sharp, C. Keskin, D. Robertson, J. Taylor, J. Shotton, D. Kim, C. Rhemann, I. Leichter, A. Vinnikov, Y. Wei, D. Freedman, P. Kohli, E. Krupka, A. Fitzgibbon, and S. Izadi.  Accurate, Robust, and Flexible Real-time Hand Tracking.  In CHI, 2015.
  * [24] J. Taylor, V. Tankovich, D. Tang, C. Keskin, D. Kim, P. Davidson, A. Kowdle, and S. Izadi.  Articulated Distance Fields for Ultra-Fast Tracking of Hands Interacting.  In SIGGRAPH Asia, 2017.
  * [25] J. Tompson, M. Stein, Y. Lecun, and K. Perlin.  Real-Time Continuous Pose Recovery of Human Hands Using Convolutional Networks.  ACM Transactions on Graphics (TOG), 2014.
  * [26] W. Wang, K. Yu, J. Hugonot, P. Fua, and M. Salzmann.  Recurrent U-Net for Resource-Constrained Segmentation.  In ICCV, 2019.
  * [27] H. Zhao, X. Qi, X. Shen, J. Shi, and J. Jia.  ICNet for Real-Time Semantic Segmentation on High-Resolution Images.  In ECCV, 2018.

Supplementary Document:
Ego2Hands: A Dataset for Egocentric Two-hand Segmentation and Detection

##  1 Ego2Hands Qualitative Examples

###  1.1 Training set

We show sample collected images with the corresponding hand heatmap energy for
the detection task. The energy map annotates the hand (without the arm) as the
foreground and all other regions as the background. Our training data covers a
wide range of hand locations, hand poses, hand sizes, skin tones and
illuminations.

Figure 1: Sample images from the training set of Ego2Hands (odd columns).
Energy for hand detection is annotated for the hand region (even columns). The
rows contain annotated instances from subject 0 to 8 respectively.

###  1.2 Evaluation set

We show sample images from all 8 sequences below to demonstrate the diversity
and annotation accuracy of our evaluation set. The evaluation sequences
contain free two-hand motion with various skin tones and illuminations,
possible heavy occlusion and motion blur. All annotations are provided with
the original image resolution of $800\times 448$.

t = 0

t = 49

t = 99

t = 149

t = 199

t = 249

Figure 2: Sample annotated images from the 8 sequences of the evaluation set
of Ego2Hands. Segmentation annotation is overlaid in colors. Detection is
annotated as energy and visualized as bounding boxes. Best viewed in
magnification.

##  2 Cross-dataset Evaluation Comparison

In this section, we show additional qualitative samples of RefineNet
evaluating on the $\text{Ego2Hands}_{test}$ for cross-dataset evaluation. In
this experiment, models are only trained on the training set of each dataset
and are not trained for domain adaptation. To demonstrate that models trained
on other datasets with limited quantity and diversity have difficulty
generalizing to the unseen domains of Ego2Hands, we show results evaluating on
sequence 2, 5, 6, 8 of $\text{Ego2Hands}_{test}$, which contain skin color or
illumination different from the standard distribution of the other datasets.
We first show results of Refinenet trained on $\text{Ego2Hands}_{train}$ in
Fig. 3. We emphasize that scenes and subjects in the test set of Ego2Hands are
not present in the corresponding training set. As a result, Fig. 3 indicates
that training on the composited data of $\text{Ego2Hands}_{train}$ achieves
high generalization accuracy.
Fig. 4-7 show results achieved from models trained on Ego3DHands, EgoHands,
GTEA and EGTEA respectively. In Fig. 4, we see that training on synthetic data
does have some generalization ability on real-world data. Despite EgoHands
containing segmentation masks that exclude the arm, fig. 5 shows that
RefineNet trained on EgoHands has low accuracy for distinguishing the left and
right hand. It also has very unstable and inaccurate predictions for both
hands in general. Fig. 6 shows that training on GTEA has very low
generalization accuracy and can completely fail in cases of bright
illumination. In Fig. 7, we see that training on EGTEA can enable limited
generalization for both hands in diverse scenes. However, its accuracy for
two-hand segmentation still significantly falls behind RefineNet trained on
our dataset. Note that in the case of inter-hand occlusion and cross-over
(e.g. t = 99 in row 1), the model trained on EGTEA mistakenly predicts the
right hand as the left hand. This problem is caused by the lack of instances
with inter-hand occlusion from datasets such as EgoHands, GTEA and EGTEA.
We point out that training on our dataset not only provides high
generalization accuracy in two-hand segmentation, but also enables two-hand
detection despite inter-hand occlusion, an important feature unavailable from
training using other hand segmentation datasets.

t = 0

t = 49

t = 99

t = 149

t = 199

t = 249

Figure 3: Qualitative samples of model trained on
$\textbf{Ego2Hands}_{\textbf{{train}}}$ on sequence 2, 5, 6, 8 of
$\text{Ego2Hands}_{test}$.

t = 0

t = 49

t = 99

t = 149

t = 199

t = 249

Figure 4: Qualitative samples of model trained on Ego3DHands on sequence 2, 5,
6, 8 of $\text{Ego2Hands}_{test}$.

t = 0

t = 49

t = 99

t = 149

t = 199

t = 249

Figure 5: Qualitative samples of model trained on EgoHands on sequence 2, 5,
6, 8 of $\text{Ego2Hands}_{test}$.

t = 0

t = 49

t = 99

t = 149

t = 199

t = 249

Figure 6: Qualitative samples of model trained on GTEA on sequence 2, 5, 6, 8
of $\text{Ego2Hands}_{test}$.

t = 0

t = 49

t = 99

t = 149

t = 199

t = 249

Figure 7: Qualitative samples of model trained on EGTEA on sequence 2, 5, 6, 8
of $\text{Ego2Hands}_{test}$.