# Dynamical neighbors:
phase and jamming transitions in hard-spheres fluids

L. Aramis de Icaza Astiz<EMAIL_ADDRESS>Atahualpa S. Kraemer
<EMAIL_ADDRESS>Physics Departament, Faculty of Science,
Universidad Nacional Autónoma de México, Ciudad Universitaria, México D.F.
04510, Mexico s

###### Abstract

We introduce the concept of dynamic neighbors. This concept defines an order
parameter with which phase transitions and jamming can be detected in fluids
of hard-spheres without considering orientational or translational symmetry.
It also gives a method to estimate the configurational entropy of the system.
Using molecular dynamics simulations, we measure the number of dynamic
neighbors. With this, it is possible to detect three phases in $2$-dimensional
system and two phases in the $3$-dimensional version. In the hard disk system,
these regions correspond to the fluid, hexatic and solid phases, while in the
$3$-dimensional case, they correspond to the solid and fluid phase. We observe
a continuous transition in the $2$-dimensional system and a first-order phase
transition in the $3$-dimensional case. We also observe only two “states” in
the case of binary mixtures and for hard-spheres with a fast compression
speed. These two states correspond to fluid and jammed states.

††preprint: AIP/123-QED

The study of fluids of particles with hardcore potential has led to an
understanding of the mechanisms by which phase transitions occur Alder and
Wainwright (1962), including transitions to quasicrystalline systems Haji-
Akbari et al. (2009); Wang et al. (2021), but it has also served to better
understand the glass transition Berthier and Biroli (2011) via the jamming
transition. As simple as hard-sphere models may seem, there is still debate
about the kind of transition they present, and their study opened the doors to
the Kosterlitz-Thouless-Halperling-Nelson-Young (KTHNY) theory which allowed
the recognition of topological phases Kosterlitz and Thouless (1973); Halperin
and Nelson (1978); Young (1979). Unfortunately, until now, this theory has not
been generalized for higher dimensional systems; although there have been some
efforts Bauerschmidt et al. (2019).

This theory predicted the existence of a phase transition before the
crystalline phase in hard disk fluids, known as the _hexatic phase_ Nelson and
Halperin (1979). However, there is still discussion on the type of transition
between the fluid phase and the hexatic phase Ruiz et al. (2019). The KTHNY
theory predicts that this transition should be continuous Kosterlitz and
Thouless (1973); Halperin and Nelson (1978); Nelson and Halperin (1979); Young
(1979). In contrast, recent numerical experiments conclude that the transition
is of first-order Binder et al. (2002); Engel et al. (2013); Bernard and
Krauth (2011); Mak (2006), and this was measured experimentally Thorneywork et
al. (2017a); Huang et al. (2020), with no theoretical explanation.

On the other hand, the jamming transition in hard-sphere fluids is still a
matter of discussion due to the lack of agreement on the type of transition
that occurs Biroli (2007). Two types of systems are usually used to study this
kind of transition: (i) 3D or higher-dimensional monodisperse systems Kraemer
and Naumis (2008); Estrada and Robles (2011); Torquato and Stillinger (2010a);
Wang (2005); Ustinov (2017); Thorneywork et al. (2017b) and (ii) 2D mixtures
particles of two different sizes Donev et al. (2006a); Callaham and Machta
(2017); Torquato and Stillinger (2010a). However, sometimes it has been used
in 2D monodisperse systems Torquato and Jiao (2010). In the 2-dimensional
case, it has been shown that the system does not crystallize when the ratio in
the area between the large and small particles is approximately 2:1, and the
ratio of particles is 1:2 Donev et al. (2006a). While for the 3-dimensional
case, crystallization depends on the rate of compression Torquato and
Stillinger (2010a).

The first step in studying a phase transition is to define an order parameter
that determines when the system is in one phase or another Zinn-Justin (2007).
In the case of hard-sphere systems, the orientational order parameter is
usually used Torquato and Stillinger (2010b). This parameter locally compares
the geometry of the fluid with that of a periodic array. For example, in $2$
dimensions, it is usually compared with a triangular arrangement with 6-fold
symmetry Binder et al. (2002); Engel et al. (2013); Bernard and Krauth (2011);
Jaster (2004, 1999); Basurto et al. (2018), while in 3-dimensions, it is
usually compared with an FCC arrangement Torquato et al. (2000); Steinhardt et
al. (1983); Odriozola (2009). In both cases, the geometry structures with
which it is compared are those of maximum density Wang (2005); Hales et al.
(2017). This order parameter requires defining the close neighbors. As far as
we have been able to review, there are two ways in which close neighbors are
usually defined in hard-sphere systems: (i) using Voronoi cells Jaster (1999);
Ruiz et al. (2019) and (ii) as a function of distance Jaster (2004); Engel et
al. (2013); Bernard and Krauth (2011); Odriozola (2009); Basurto et al.
(2018). Although the system has no apparent rotational or translational
symmetry in a jammed state, locally, the orientational order parameters are
relatively high, so they are also used to characterize the jamming transition
Donev et al. (2006a).

In this work, we propose to define _dynamical neighbors_ as those where there
is an interaction between the particles, i.e., two particles are neighbors if
they collide with each other. We measure the number of dynamical neighbors as
a function of the density. With this, we are available to compute an order
parameter independent of the geometry of the densest array and the
configurational entropy of the system in a more straightforward way.

##  I Model

Although most of the ideas presented here can be used for hard particles of
any form, we did our simulations on hard-spheres fluids. The systems we study
in this paper are: (i) Single size discs in 2D; (ii) binary mixture of disks
width radii $1$ and $\sqrt{(}2)\sim 1.4$, with a proportion of 2:1; (iii) and
one-size spheres in 3D. In all three systems, periodic boundary conditions
were applied to reduce the size effects. In the case of the $2$-dimensional
systems $10,044$ disks were used, while in the $3$-dimensional system $2,048$
spheres were used. We use the _packing fraction_ $\phi$ as the independent
variable.

To perform the simulations, we first used the Lubachevsky–Stillinger algorithm
Lubachevsky and Stillinger (1990) to obtain systems with a certain packing
fraction $\phi$ numerically. Then we used a molecular dynamics algorithm with
hard-spheres Sigurgeirsson et al. (2001) to evolve the system, both written in
Julia Bezanson et al. (2017).

It is common to define an order parameter $\Psi_{n}$ that compares the
orientational symmetry of the system to one with $n$-fold symmetry. In the
$2$-dimensional case $n=6$ is typically used to compare the system with a
perfect triangular arrangement and observe at what density there is a jump of
this parameter. In the general case, the order parameter $n$ is defined as:

$\Psi_{n}=\frac{1}{N}\sum_{i=1}^{N}\left\lVert\frac{1}{Z_{i}}\sum_{j=1}^{Z_{i}}e^{in\theta_{ij}}\right\rVert,$
(1)

where $Z_{i}$ is the number of neighbors of the particle $i$, and
$\theta_{ij}$ is the angle formed by the segment that joins the particles $i$
and $j$ with the horizontal axis. A similar definition is used for the
$3$-dimensional systems Torquato and Stillinger (2010a). This definition of
the order parameter does not restrict what is considered a neighbor of the
particle $i$.

##  II Dynamical Neighbors

One of the simplest ways of defining neighbors is to use distances between
particles. In this case, $\Psi_{n}$ is a function of $r$. If $r$ is too small,
$\Psi_{n}(r)$ is not well defined because the number of neighbors $Z(r)$ is
zero, but if it is too long, $\Psi_{n}(r)\rightarrow 0$.

In the case of a crystalline system, it is more or less clear which the close
neighbors are, but for systems that do not have translational symmetry, it is
more difficult to recognize a distance at which the close neighbors can
interact.

Alternatively, we can fix the number of neighbors such that the first $n$
closest particles are considered the neighbors Häggström and Meester (1996).
Choosing the number $n$ of neighbors is usually compared to an arbitrary
structure, such as the array of maximum density, where the neighbors are
defined by contact. In the case of monodispersive fluids hard spheres, this
structure is the FCC lattice or the triangular lattice for 3 and 2 dimensions,
respectively. In these two examples, $n$ is the same for every sphere;
however, the number of neighbors in the maximum density lattice may vary from
disk to disk for systems such as mixtures of spheres Yi et al. (2012) or even
for other shapes such as fluids of tetrahedra Haji-Akbari et al. (2009). When
the number of neighbors in the array of maximum density is not constant (as
for example when there is a binary mixture of particles), it is unclear which
is the number of neighbors to consider.

Another alternative is to use Voronoi cells Sastry et al. (1997a, b), defining
neighboring particles as those that share one face of their Voronoi cells.
This definition has been used to study the topological properties of random
close packing Coxeter et al. (1958); Sastry et al. (1997a). In particular, it
can be shown that there is no regular honeycomb with the topological
properties of the Voronoi tessellation produced by a random close packing
Coxeter et al. (1958), since the cells would have to have faces with
approximately $5.12$ sides, and each sphere should have $13.6$ neighbors.
Although a regular tessellation with these characteristics is meaningless, it
is possible to have a non-regular tessellation that, on average, has that
number of neighbors with that average number of sides per face in each Voronoi
cell.

Inspired by this result, Finney Finney (1970) obtained the distribution of
faces (neighbors) and edges of each face in a random close packing, obtaining
on average $14.251$ faces per cell and $5.158$ edges per face. In addition to
the usefulness of studying the topological properties of jammed states,
Voronoi cells have also been used to estimate the free volume and thus obtain
a heuristic deduction of the Vogel-Fulcher-Tamman law Elliott (1983); Wales et
al. (2003). Moreover, this has laid the foundation for the cage theory used to
estimate the densities at which the phase transition occurs Kraemer and Naumis
(2008), and has been used directly to measure the transition between a jamming
state and the crystalline phase in hard-spheres when checking the probability
of having a certain volume of the Voronoi cell Jin and Makse (2010).

Despite its usefulness in the description of phase transitions, the definition
of a neighbor via a Voronoi cell does not reflect the interactions between
particles, and it is expensive computationally Engel et al. (2013), especially
when the shape of the particles is not spherical Schaller et al. (2013). The
computational complexity grows fast when we increase the dimension of the
system Boissonnat et al. (1998).

On the other hand, the number of collisions has been related with pressure via
$\frac{pV}{k_{B}T}=1+\frac{N_{coll}\pi^{1/2}}{9\delta t}$ Woodcock (1997),
where $N_{coll}$ is the number of collisions per particle, and $\delta t$ is
the time window when the collisions occur. In figure 1 we show the average
number of collisions as a function of the packing fraction $\phi$. The inset
plots correspond to the zoom close to the phase transition in the case of $2$
and $3$-dimensional systems. We can see that the behavior of the average
number of collisions as a function of $\phi$ is similar to the observed
pressure Engel et al. (2013) close to the phase transition. This observation
motivated us to consider the dynamical information that collisions give to
define neighbors instead of using geometric information as the distance or the
faces of Voronoi cells.

Figure 1: (Color online) The number of average collisions per particle as a
function of $\phi$ for a system of hard discs of the same size (dark blue), a
binary mixture (light blue), hard spheres with a slow compression speed (dark
red), a fast compression speed with a short relaxation time (orange), and fast
compression speed with a long relaxation time (pink). The insets are the zooms
to the phase transition region. The vertical dashed lines correspond to
$\phi=0.54,0.59,0.702,0.724,0.8,0.82$.

We propose to define the _dynamical neighbors_ of a particle $i$ as the
particles that collide with $i$ in a given time window $\delta t$. This
implies that the number of dynamical neighbors $Z_{d}$ is a function of
$\delta t$. If the time window is large enough, each particle will have
collided with all the others. On the other hand, if the time considered is too
short, most of the particles will not collide. This is why it is necessary to
define an _adequate_ time window, one large enough so that all the particles
have collided several times (not necessarily with different particles) but not
so long that most have collided with all the particles. As the system is
compressed, the energy increases, which means the velocity of particles
increases. This means that the distance traveled by each particle in a fixed
time window increases as a function of energy. To avoid this problem, we
propose that the time window is a function of the kinetic energy $E$ of the
particles $\delta t(E)=\frac{c}{\sqrt{E}}$, where $c$ is a constant.

To choose the proper value of $c$, let us first note that unlike neighbors
defined using Voronoi cells or as a function of the distance, $Z_{d}$ can be
very large. We can use this difference in our favor for two reasons: (i) we
can produce much wider distributions of neighbors for low densities (specially
if $c$ is large), and extract information from the shape of those
distributions, and (ii) we can define a _binding parameter_ :
$\mu_{C/Z_{d}}=\frac{N_{coll}}{\langle Z_{d}\rangle\sqrt{E}}$ where collisions
$N_{coll}$ is the number of total collision per particle, and $E$ is the
energy of the system. Because this binding parameter is the average number of
redundant collisions, it measures a kind of linking intensity between
neighbors.

We will use (i) to give a method to choose $c$. The distribution of $Z_{d}$
depends on $c$. Figure 2 shows how the distribution of neighbors changes as a
function of $c$ for $\phi=0.2$. In general, for low packing fractions, the
higher the value of $c$, the farther the expected value is from 0, and the
wider the distribution is. Once the system solidifies, the distribution keeps
the same expected value for every $c$, but the width of the distribution
reduces as $c$ increases.

Figure 2: Distribution of dynamic neighbors $Z_{d}$ for a system with 1024
particles at $\phi=0.2$ with various values of $c$. The labels correspond to
the value of $c$.

Because $Z_{d}$ is bounded by the minimum number of neighbors of one particle
$n_{m}$ in the solid phase and by the number of particles $N$, the choice of
$c$ should consider that these boundaries $N$ and $n_{m}$ have little effect.
So, we need that the value of $c$ is large enough such that $|n_{m}-\langle
Z_{d}\rangle|>>\sigma$, where $\sigma$ is the standard deviation of the
distribution of $Z_{d}$, but small enough such that $|N-\langle
Z_{d}\rangle|>>\sigma$.

In general, a larger value of $c$ is better because this means more statistics
to obtain the distribution of $Z_{d}$, so, if we are not close to the limit
$\langle Z_{d}\rangle=N$, it is better to have a large value of $c$. If $N$ is
a large number, it typically is computationally challenging to reach a value
of $c$ such that $|N-\langle Z_{d}\rangle|<\sigma$, so the larger the value of
$c$, the better.

Because the neighbors distributions are mostly symmetric, unimodal
distributions, the position of the maximum, the most probable dynamical
neighbor $Z_{d}$, and the average number of neighbors $\langle Z_{d}\rangle$
are close related quantities, but they are not identical. Close to a phase
transition, the distributions are not perfectly symmetric. In practice
$\langle Z_{d}\rangle(\phi)$ is smoother than the most probable dynamical
neighbor $Z_{d}(\phi)$.

To qualitatively describe the shape of $Z_{d}(\phi)$ as a function of $\phi$,
first note that for the ideal gas $Z_{d}(0)=0$, since there are no collisions.
As the packing fraction increases, $Z_{d}$ increases almost linearly (see
figure 3). The latter is due to the fact that for low packing fractions most
interactions between two given particles occur on a single occasion, then, the
mean free path $\langle l\rangle\propto t_{av}=\delta tN/(2N_{coll})$ , where
$t_{av}$ is the mean time between collisions and $N$ is the number of
particles and also $\langle l\rangle\propto V/(Nv_{1}^{\prime}(r))\sim r/\phi$
Serway and Jewett (2018). The last relation follows since $v_{1}(r)\sim
v_{1}^{\prime}(r)r$. This implies $Z_{d}\sim\langle Z_{d}\rangle\propto
N_{coll}\propto\phi$. However, if the packing fraction continues to grow,
collisions with the same particle (redundant collisions) become common, so the
number of dynamic neighbors decreases. So, $Z_{d}$ is $0$ for $\phi=0$, then
grows linearly until redundant collisions become dominant where a maximum is
reached and then decreases. When the system crystallizes $Z_{d}$ must become
$6$ or $12$ for $2$ and $3$ dimensions respectively, so we expect an abrupt
change in $Z_{d}$ as shows figure 3 (solid lines). It can also occur that the
system has a jamming transition; in such a case, $Z_{d}$ continues to decrease
(dashed lines).

Figure 3: (Color online) Most probable $Z_{d}$ as a function of the packing
fraction $\phi$ for a system of hard discs of the same size (dark blue), a
binary mixture (light blue), hard-spheres with a slow compression speed (dark
red) a fast compression speed with a short relaxation time (orange), and fast
compression speed with a long relaxation time (pink). The insets are the same
plot but with no logarithmic scale. The horizontal dashed lines represent $6$
dynamic neighbors and $12$ dynamic neighbors. The vertical dashed lines
correspond to $\phi=0.54,0.59,0.702,0.724,0.8,0.82$.

###  II.1 Diffusion coefficients

Now we would like to focus on the relationship between $\langle Z_{d}\rangle$
and the diffusion coefficient. For low densities, $\langle Z_{d}\rangle\approx
N_{coll}/N$, since each collision generally occurs with a different particle.
However, when the packing fraction grows, this is no longer true. Suppose that
only the particle $i$ can move. The rest have their position fixed. If we tile
the space using Voronoi cells omitting the particle $i$, for medium and high
packing fractions, practically every time the particle $i$ visits a Voronoi
cell, it will collide with the particle in that cell. Thus $\langle
Z_{d}\rangle$ approximates the number of different Voronoi cells that were
visited. Since the average volume of the Voronoi cells is $V/N$, $\langle
Z_{d}\rangle V/N$ is approximately the volume covered by a particle in time
$\delta t$, that is, the mean square displacement $\langle x^{2}(\delta
t)\rangle\sim D\delta t\approx(\langle Z_{d}\rangle V/N)^{2/d}$, which implies
that $D\approx(\langle Z_{d}\rangle V/N)^{2/d}1/\delta t$, where $d$ is the
dimension of the system.

###  II.2 Orientational order parameters

As we have mentioned, another important quantity is $\mu_{C/Z_{d}}$, which can
be used as an order parameter without the need to compare with any particular
structure, which is a usual problem as discussed in Zachary and Torquato
(2009).

Before showing that $\mu_{C/Z_{d}}$ can be used as an order parameter, we will
discuss the typical orientational order parameters using the dynamical
neighbors.

We obtained the orientational $\Psi_{n}$ for different values of $n$ both in
the monodispersive system of hard disks and in the mixture for $\phi=0.88$ for
the monodisperse case and $\phi=0.84$ for the binary mixture. Figure 4 shows
the order parameter $\Psi_{n}$, varying $n$ from $0$ to $15$ for the
monodisperse (continuous line) and binary mixture (dashed line) systems. As
expected, this parameter has peaks around $n=6k$, with $k$ an integer for the
monodisperse case. The other peaks correspond to the cases $n=k+1/2$. These
peaks also meet the expectations of equation (1) for a triangular array.

Figure 4: $\Psi_{n}$ as a function of the orientational order $n$ for the
densest arrangement obtained in the monodisperse system (continuous line) and
the binary mixture (dashed line).The vertical dashed lines correspond to
$\phi=0.54,0.59,0.702,0.724,0.8,0.82$.

For the binary mixture, it could be expected that the orientation order
parameter would depend on the reached configuration. Still, after performing
several simulations, almost the same graph was always obtained. In this case,
it is notable that $\Psi_{n}$ has a maximum for $n=6$, so this parameter can
be used to measure the jamming transition, just like in the crystalline case.
However, unlike the crystalline case, $\Psi_{5}$ and $\Psi_{7}$ are not
negligible and neither $\Psi_{11}$. Therefore, using $\Psi_{6}$, one can
expect to see a jump in both cases, while using $\Psi_{5}$ and $\Psi_{11}$,
one should only see a jump when the transition is a jamming transition.

The fact that the orientational order parameter depends on which structure it
is compared with has the consequence that there may be an order that we cannot
observe for some systems. For example, the maximum density array would appear
disordered when measuring the orientational order parameter $\Psi_{5}$ as well
as a quasicrystal with pentagonal symmetry using $\Psi_{6}$. Instead, we can
use an order parameter that is independent of the geometry but depends on the
dynamical properties, as is the case of $\mu_{C/Z_{d}}$. If the number of
degrees of freedom of the system is reduced, as, during the break of
rotational or translational symmetry, we would expect that the number of
redundant collisions must change, so $\mu_{C/Z_{d}}$ must change its behavior
as well.

Figure 5 shows the orientational order parameter $\Psi_{6}$ ($2$-dimensional
systems), $Q_{6}$ ($3$-dimensional systems) defined as in Torquato and
Stillinger (2010a) and $\mu_{C/Z_{d}}$ as a function of $\phi$ for all the
systems we study in this paper. We observe that $\Psi_{6}$ and $\mu_{C/Z_{d}}$
are very close for the monodisperse hard disk fluid, showing a transition
around $\phi\approx 0.7$. This result agrees with what was observed using
other definitions of close neighbors Huerta and Naumis (2003). For the binary
mixture $\mu_{C/Z_{d}}$ and $\Psi_{6}$ are also similar showing that the
jamming transition is around $\phi\approx 0.75$.

In the case of $3$-dimensional systems $Q_{6}$ and $\mu_{C/Z_{d}}$ are very
different for low packing fractions, but for high packing fractions both
parameters are correlated, showing a transition at $\phi\approx 0.54$. For the
jammed system, $Q_{6}$ is very low. It has a small jump before the jamming
transition for which we believe is in a region of meta-stability of the fluid
and then it goes to 0 while $\mu_{C/Z_{d}}$ is always growing, with only a
change in the behavior around $\phi=0.63$, which is close to the maximal
random jamming.

Figure 5: (color online) Orientational order parameters $\Psi_{6}$ and $Q_{6}$
(solid lines) and binding parameter $\mu_{C/Z_{d}}$ (dashed lines) as a
function of the packing fraction for $3$-dimensional systems (red-orange
lines), and $2$-dimensional systems (blue lines). Red and dark blue lines
correspond to the systems with a phase transition, while orange and light blue
lines represent the systems with a jamming transition. Pink lines correspond
to the systems where the compression rate is fast, but the relaxation time is
long.

We can observe that $\mu_{C/Z_{d}}$ behaves very close to the orientational
order parameter when the orientational order parameter captures a phase or
jamming transition but differ when the orientational order parameter does not
recognize such transition. So we have an order parameter independent of the
final structure we obtain that can be useful if the final structure is
unknown, as in a mixture of different size particles or non-spherical
particles like tetrahedra Haji-Akbari et al. (2009).

###  II.3 Configurational Entropy

The understanding glass transition is still one of the most critical
challenges in material sciences. A possible scenario proposed by Gibbs and
Adams Adam and Gibbs (1965) relates the increases in viscosity with a dramatic
reduction of the number of configurations available to the fluid. An ideal
glass transition occurs if the liquid is trapped in one of the remaining
configurations.

Configurational entropy $s_{c}$ measures the number of possible distinct
states accessible to a system, and it is believed that $s_{c}\approx
s_{L}-s_{g}$ Donev et al. (2007a); Berthier et al. (2019), where $s_{L}$ is
the entropy in the liquid state and $s_{g}$ is the entropy of the glassy
state. Because in the jammed state, particles are trapped, it is possible to
estimate $s_{g}$ in single-occupancy cells (SOC) using the Bonding Cell
Molecular Dynamics algorithm Donev et al. (2007b) to measure the average
reduced pressure on the walls of each cell $p_{c}=P_{c}V_{c}/k_{B}T$ and then
obtain the change in free energy as the work of compressing the cells by
integrating $p_{c}$. This gives the free energy $f_{g}=-s_{g}$ of the SOC
system Donev et al. (2007b). $s_{L}$ can be obtained simply by thermodynamic
integration of the equation of state of the fluid (see Luding (2001)) from the
ideal-gas limit. This algorithm has proved to be accurate Donev et al.
(2006b), but it is admitted to be computational expensive even if it is
already an improvement to other similar algorithms Senthil Kumar and Kumaran
(2005). The reason is that reducing the volume of the cells must be very slow
because when the size of the cell is close to the radius of the particles, the
number of collisions increases exponentially.

There are at least two other approaches to estimate $s_{c}$, by approximating
it with the entropy of only two-particle interactions Baranyai and Evans
(1989) and using the Shannon entropy of information Walraven and Leermakers
(2020). The first of these approaches uses the expansion of the entropy in a
series of n-particle correlation functions Nettleton and Green (1958). The
correlation function for three or more particles is difficult to calculate,
but the pair correlation function is not difficult to estimate via
computational simulations. The second approach is based on data compression
algorithms. Walraven et al. proposed to store the information of a hard-sphere
system in a file, as the positions of the particles, then apply a compression
algorithm to the data to obtain a smaller file. The compression ratio (the
ratio of file sizes before and after compression) is related to the
information entropy of the file, which must be related to the configurational
entropy of the system. Applying this idea to a 3-dimensional hard-sphere
system of 256 particles, they obtained an almost constant entropy for packing
fractions between 0 and 0.5. Then they observe an abrupt reduction in the
entropy similar to what has been observed in Donev et al. (2007a).

The problem with the approach using the two-particle approximation is that no
phase transition is observed, so higher terms are necessary. In the case of
the data compression algorithms, they depend on the used algorithm, and the
behavior for very high packing fraction seems to move away from the true
entropy.

Here we propose an alternative method to estimate the configurational entropy
that can be easily obtained for any hard-potential particles.

If the packing fraction is high enough, practically all visits to a Voronoi
cell involve a collision with the particle in that cell. Then, the probability
that a particle visits $Z_{d}$ of neighbors in a time $\delta t$ depends on
the local configuration of particles. So, the standard deviation $\sigma$ of
the distribution $\rho_{\phi}(Z_{d})$ must be proportional to the number of
local configurations, i.e., the number of different local arrays of particles
in the system. We assume that this distribution of local arrays is
proportional to the number of accessible configurations.

Then the number of configurations $\Omega(\phi)$, must be proportional to the
standard deviation $\sigma$ of the probability distribution
$\rho_{\phi}(Z_{d})$. If the probability distribution of $Z_{d}$ is a normal
distribution, then, to estimate $\sigma$, we just need to calculate the value
of the maximum $\max(\rho_{Z_{d}})$, since $\sigma\sim 1/\max(\rho_{Z_{d}})$.
Now, using the Boltzmann equation for the entropy we can be calculated $s_{c}$
as $s_{c}=K_{B}\log(\Omega)$, then $s_{c}\sim-\log(\max(\rho_{Z_{d}}))$.

Figure 6 (top-left) shows the probability distributions of the number of
dynamic neighbors for various packing fractions for monodisperse hard disk
fluid. In general increasing $\phi$ increases $\max(\rho_{Z_{d}}))$; however,
for $\phi\in[0.7,0.72]$ there is a drop in $\max(\rho_{Z_{d}}))$. The range of
values for which there is this decrease in $\max(\rho_{Z_{d}}))$ corresponds
to the packing fractions of the hexatic phase Engel et al. (2013); Bernard and
Krauth (2011); Jaster (2004).

A similar effect is observed in figure 6 (bottom-left), where the
crystallization process of the fluid of $3$-dimensional hard-spheres is shown.
However, in this case, there is a jump in the average number of neighbors (it
passes from 60 to 12).

Figure 6: (Color online) Distributions of the number of dynamic neighbors for
various packing fractions for systems: (top-left) monodispersive, and (top-
right) binary mixtures of hard discs (bottom-left) slow compression speed and,
(bottom-right) fast compression speed of hard spheres. Some labels with the
packing fraction were added for easier reading.

In the case of the binary mixture and the fast compression speed,
crystallization is frustrated and no fall appears in the maximum of the
probability distribution for medium densities. This is shown in figure 6 (top-
right), and (bottom-right).

With the height of these distributions, we computed the entropy of the system
using $s_{c}\approx-\log(\max(\rho_{Z_{d}}))$. This is shown in figure 7. The
blue lines correspond to the $2$-dimensional systems. The red line correspond
to a $3$-dimensional system with a slow rate of compression, while the orange
line and pink line correspond to a fast rate. The difference between the
orange and solid pink is that in one case (orange) the time for relaxation of
the system was short (pink) while the other was a long time of relaxation. We
can see that the paths with slow and fast relaxation times differ notoriously
for $\phi\in[0.54,0.59]$ (vertical lines). This shows that the states
corresponding to the short relaxation times are meta-stable. However, for
$\phi>0.59$ both paths almost match. We interpret this to mean that it is at
$\phi=0.59$ where the jamming transition occurs.

The dark blue curve shows an increment of the entropy just before the phase
transition to the crystal phase occurs (similarly the orange curve at the
beginning of the meta-stable phase). This increment occurs just before the
hexatic phase, which correspond to the lowering of the maximum in the
distribution of figure 6 (top-left) at $\phi=0.702$.

Figure 7: (color online) Entropy for the $2$ and $3$-dimensional systems. Blue
lines correspond to the $2$-dimensional systems, while red and orange lines
correspond to the $3$-dimensional cases. We added a plot of the
configurational entropy minus the entropy of the binary mixture for the
$2$-dimensional cases, using the data from Donev et al. (2007a) (green squares
for monodispersive systems and blue triangles for binary mixture). The
vertical dashed lines correspond to $\phi=0.54,0.59,0.702,0.724,0.8,0.82$.

We want to highlight that in the $3$-dimensional case, the change in entropy
when passing from one phase to the other appears discontinuous; that is, it is
a first-order transition. However, for the $2$-dimensional case, we do not
observe that there is an abrupt change in entropy, the hexatic-solid
transition, but for the liquid-hexatic transition, there is a jump increasing
the entropy, which could be a discontinuity in entropy. This coincides with
the abrupt change in the number of dynamic neighbors for the $3$-dimensional
system in contrast to what happens in the $2$-dimensional case (see inset of
figure 3) and would also be in agreement with Engel et al. (2013).

About the metastability region $\phi\in[0.54,59]$, it has been suggested that
it includes any $\phi>0.54$ Rintoul and Torquato (1996), in such case, there
is no glass transition; however, we did not observe that all the systems
crystallize.

##  III Soft particles and Monte-Carlo generalizations

With hard-core potentials, the deviation angle of the particles after a
collision is independent of the speed of the particles. Thus, if we consider,
for example, only the collisions that produce a minimum deviation, this would
be independent of the speed of the particles that collide, and all the
measurements would be proportional. Thus, the number of neighbors can be
decoupled from temperature independent of any imposition to consider a minimum
deviation (for example, at least angles of $45$ degrees).

In the case of soft particles, the deviation depends on the speed of the
particles. Then, to take this into account, any generalization of the
dynamical neighbors to soft potential must rely on the speed of particles that
collide. Then the distribution of dynamical neighbors would be temperature-
dependent, as well as the entropy or the binding parameter.

On the other hand, in Monte-Carlo simulations, a particle’s movement is
rejected with some probability if the energy is increased. The rejection can
be interpreted as a collision. So, all the particles involved in rejecting a
movement can be considered dynamical neighbors. For hard particles, this means
every time there is an overlap between particles, but in the case of soft
particles, this depends on the temperature.

## Conclusions and remarks

We proposed to define dynamical neighbors in systems with hard-core potential
as those that have a collision in a given time window. We found that the
average number of dynamical neighbors must be proportional to the diffusion
coefficient. Also, we proposed defining an order parameter using the number of
collisions and the number of dynamical neighbors. We found that this order
parameter, which is independent of the system’s geometry, can be used to
detect phase and jamming transitions in $2$ and $3-$dimensions. In
$2-$dimensions, this parameter was available to see fluid, hexatic, and
crystal phases. Finally, we propose a method to estimate the configurational
entropy using the distribution of dynamic neighbors. We computed it for four
systems: hard disks of a single size of particles, a binary mixture of hard
disks, and hard spheres with slow and fast compression rates. The measurements
of this entropy were in agreement with other measurements reported in the
literature. Since the simplicity of the method, it was possible to measure the
entropy in many more packing fractions and with larger systems. This allowed
us to observe an increase in the entropy in the transition from fluid to the
hexatic phase and the phase transition from fluid to solid in $3-$dimensions.
We also observe a meta-stable phase in $3-$ dimensions and a drop in the
entropy during the transition from hexatic to the solid phase. We also
observed that the metastable phase becomes stable at a packing fraction
corresponding to the jamming transition in 3D.

###### Acknowledgements.

We thank David P. Sanders for useful discussions. The authors appreciate the
computing platform provided by the Laboratorio de Cómputo de Alto Rendimiento,
under coordination of Departamento de Matemáticas of Facultad de Ciencias,
UNAM.

## References

  * Alder and Wainwright (1962) B. Alder and T. Wainwright, Phys. Rev. 127, 359 (1962).
  * Haji-Akbari et al. (2009) A. Haji-Akbari, M. Engel, A. S. Keys, X. Zheng, R. G. Petschek, P. Palffy-Muhoray, and S. C. Glotzer, Nature 462, 773 (2009).
  * Wang et al. (2021) D. Wang, T. Dasgupta, E. B. van der Wee, D. Zanaga, T. Altantzis, Y. Wu, G. M. Coli, C. B. Murray, S. Bals, M. Dijkstra, et al., Nature Phys. 17, 128 (2021).
  * Berthier and Biroli (2011) L. Berthier and G. Biroli, Rev. Mod. Phys. 83, 587 (2011).
  * Kosterlitz and Thouless (1973) J. M. Kosterlitz and D. J. Thouless, J. Phys. Condens. Matter 6, 1181 (1973).
  * Halperin and Nelson (1978) B. Halperin and D. R. Nelson, Phys. Rev. Lett. 41, 121 (1978).
  * Young (1979) A. Young, Phys. Rev. B 19, 1855 (1979).
  * Bauerschmidt et al. (2019) R. Bauerschmidt, D. Conache, M. Heydenreich, F. Merkl, and S. W. Rolles, in _Ann. Henri Poincaré_ (2019), vol. 20 Issue 9, pp. 3019–3057.
  * Nelson and Halperin (1979) D. R. Nelson and B. Halperin, Phys. Rev. B 19, 2457 (1979).
  * Ruiz et al. (2019) P. S. Ruiz, Q.-l. Lei, and R. Ni, Comm. Phys. 2, 1 (2019).
  * Binder et al. (2002) K. Binder, S. Sengupta, and P. Nielaba, J. Phys. Condens. Matter 14, 2323 (2002).
  * Engel et al. (2013) M. Engel, J. A. Anderson, S. C. Glotzer, M. Isobe, E. P. Bernard, and W. Krauth, Phys. Rev. E 87, 042134 (2013).
  * Bernard and Krauth (2011) E. P. Bernard and W. Krauth, Phys. Rev. Lett. 107, 155704 (2011), URL https://link.aps.org/doi/10.1103/PhysRevLett.107.155704.
  * Mak (2006) C. H. Mak, Phys. Rev. E 73, 065104 (2006), URL https://link.aps.org/doi/10.1103/PhysRevE.73.065104.
  * Thorneywork et al. (2017a) A. L. Thorneywork, J. L. Abbott, D. G. A. L. Aarts, and R. P. A. Dullens, Phys. Rev. Lett. 118, 158001 (2017a), URL https://link.aps.org/doi/10.1103/PhysRevLett.118.158001.
  * Huang et al. (2020) P. Huang, T. Schönenberger, M. Cantoni, L. Heinen, A. Magrez, A. Rosch, F. Carbone, and H. M. Rønnow, Nature Nano. 15, 761 (2020).
  * Biroli (2007) G. Biroli, Nature Phys. 3, 222 (2007).
  * Kraemer and Naumis (2008) A. S. Kraemer and G. G. Naumis, J. of Chem. Phys. 128, 134516 (2008).
  * Estrada and Robles (2011) C. Estrada and M. Robles, J. of Chem. Phys. 134, 044115 (2011).
  * Torquato and Stillinger (2010a) S. Torquato and F. H. Stillinger, Rev. of Mod. Phys. 82, 2633 (2010a).
  * Wang (2005) X.-Z. Wang, J. Chem. Phys. 122, 044515 (2005).
  * Ustinov (2017) E. Ustinov, J. of Chem. Phys. 146, 034110 (2017).
  * Thorneywork et al. (2017b) A. L. Thorneywork, D. G. Aarts, J. Horbach, and R. P. Dullens, Phys. Rev. E 95, 012614 (2017b).
  * Donev et al. (2006a) A. Donev, F. H. Stillinger, and S. Torquato, Phys. Rev. Lett. 96, 225502 (2006a).
  * Callaham and Machta (2017) J. Callaham and J. Machta, Phys. Rev. E 95, 063315 (2017).
  * Torquato and Jiao (2010) S. Torquato and Y. Jiao, Phys. Rev. E 82, 061302 (2010).
  * Zinn-Justin (2007) J. Zinn-Justin, _Phase Transitions and Renormalization Group_ (Oxford University Press, 2007).
  * Torquato and Stillinger (2010b) S. Torquato and F. H. Stillinger, Rev. Mod. Phys. 82, 2633 (2010b).
  * Jaster (2004) A. Jaster, Phys. Lett. A 330, 120 (2004).
  * Jaster (1999) A. Jaster, Phys. Rev. E 59, 2594 (1999).
  * Basurto et al. (2018) E. Basurto, C. Haro-Pérez, C. A. Vargas, and G. Odriozola, Phys. Chem. Chem. Phys. 20, 27490 (2018).
  * Torquato et al. (2000) S. Torquato, T. M. Truskett, and P. G. Debenedetti, Phys. Rev. Lett. 84, 2064 (2000).
  * Steinhardt et al. (1983) P. J. Steinhardt, D. R. Nelson, and M. Ronchetti, Phys. Rev. B 28, 784 (1983).
  * Odriozola (2009) G. Odriozola, J. Chem. Phys. 131, 144107 (2009).
  * Hales et al. (2017) T. Hales, M. Adams, G. Bauer, T. D. Dang, J. Harrison, H. Le Truong, C. Kaliszyk, V. Magron, S. McLaughlin, T. T. Nguyen, et al., in _Forum of Mathematics, Pi_ (Cambridge University Press, 2017), vol. 5.
  * Lubachevsky and Stillinger (1990) B. D. Lubachevsky and F. H. Stillinger, J. Stat. Phys. 60, 561 (1990).
  * Sigurgeirsson et al. (2001) H. Sigurgeirsson, A. Stuart, and W.-L. Wan, J. Comp. Phys. 172, 766 (2001).
  * Bezanson et al. (2017) J. Bezanson, A. Edelman, S. Karpinski, and V. B. Shah, SIAM Rev. 59, 65 (2017).
  * Häggström and Meester (1996) O. Häggström and R. Meester, Random Struct Algorithms 9, 295 (1996).
  * Yi et al. (2012) L. Yi, K. J. Dong, R. Zou, and A. Yu, Powder Technol. 224, 129 (2012).
  * Sastry et al. (1997a) S. Sastry, D. S. Corti, P. G. Debenedetti, and F. H. Stillinger, Phys. Rev. E 56, 5524 (1997a).
  * Sastry et al. (1997b) S. Sastry, P. G. Debenedetti, and F. H. Stillinger, Phys. Rev. E 56, 5533 (1997b).
  * Coxeter et al. (1958) H. Coxeter et al., Ill. J. Math. 2, 746 (1958).
  * Finney (1970) J. Finney, Proc. R. Soc. A 319, 479 (1970).
  * Elliott (1983) S. R. Elliott, _Physics of amorphous materials_ (Longman Group, Longman House, Burnt Mill, Harlow, Essex CM 20 2 JE, England, 1983., 1983).
  * Wales et al. (2003) D. Wales et al., _Energy landscapes: Applications to clusters, biomolecules and glasses_ (Cambridge University Press, 2003).
  * Jin and Makse (2010) Y. Jin and H. A. Makse, Physica A 389, 5362 (2010).
  * Schaller et al. (2013) F. M. Schaller, S. C. Kapfer, M. E. Evans, M. J. Hoffmann, T. Aste, M. Saadatfar, K. Mecke, G. W. Delaney, and G. E. Schröder-Turk, Philos Mag 93, 3993 (2013).
  * Boissonnat et al. (1998) J.-D. Boissonnat, M. Sharir, B. Tagansky, and M. Yvinec, Discrete Comput Geom 19, 485 (1998).
  * Woodcock (1997) L. Woodcock, Faraday Discuss. 106, 325 (1997).
  * Serway and Jewett (2018) R. A. Serway and J. W. Jewett, _Physics for scientists and engineers_ (Cengage learning, 2018).
  * Zachary and Torquato (2009) C. E. Zachary and S. Torquato, J. Stat. Mech. Theory Exp. 2009, P12015 (2009).
  * Huerta and Naumis (2003) A. Huerta and G. G. Naumis, Phys. Rev. Lett. 90, 145701 (2003).
  * Adam and Gibbs (1965) G. Adam and J. H. Gibbs, J. of Chem. Phys. 43, 139 (1965).
  * Donev et al. (2007a) A. Donev, F. H. Stillinger, and S. Torquato, J. of Chem. Phys. 127, 124509 (2007a).
  * Berthier et al. (2019) L. Berthier, M. Ozawa, and C. Scalliet, J. of Chem. Phys. 150, 160902 (2019).
  * Donev et al. (2007b) A. Donev, F. H. Stillinger, and S. Torquato, J. Comp. Phys. 225, 509 (2007b).
  * Luding (2001) S. Luding, Phys. Rev. E 63, 042201 (2001).
  * Donev et al. (2006b) A. Donev, F. H. Stillinger, and S. Torquato, Phys. Rev. Lett. 96, 225502 (2006b).
  * Senthil Kumar and Kumaran (2005) V. Senthil Kumar and V. Kumaran, J. of Chem. Phys. 123, 114501 (2005).
  * Baranyai and Evans (1989) A. Baranyai and D. J. Evans, Phys. Rev. A 40, 3817 (1989).
  * Walraven and Leermakers (2020) E. Walraven and F. Leermakers, Soft Matter 16, 3740 (2020).
  * Nettleton and Green (1958) R. Nettleton and M. Green, J. of Chem. Phys. 29, 1365 (1958).
  * Rintoul and Torquato (1996) M. Rintoul and S. Torquato, Phys. rev. lett. 77, 4198 (1996).


Erik Ostrowski<EMAIL_ADDRESS>
Bharath Srinivas Prabakaran<EMAIL_ADDRESS>
Institute of Computer Engineering, Technische Universität Wien (TU Wien),
Austria
Muhammad Shafique<EMAIL_ADDRESS>
Division of Engineering, New York University Abu Dhabi, United Arab Emirates

# Exploring Weakly Supervised Semantic Segmentation Ensembles for Medical
Imaging Systems

###### Abstract

Reliable classification and detection of certain medical conditions, in
images, with state-of-the-art semantic segmentation networks, require vast
amounts of pixel-wise annotation. However, the public availability of such
datasets is minimal. Therefore, semantic segmentation with image-level labels
presents a promising alternative to this problem. Nevertheless, very few works
have focused on evaluating this technique and its applicability to the medical
sector. Due to their complexity and the small number of training examples in
medical datasets, classifier-based weakly supervised networks like class
activation maps (CAMs) struggle to extract useful information from them.
However, most state-of-the-art approaches rely on them to achieve their
improvements. Therefore, we propose a framework that can still utilize the
low-quality CAM predictions of complicated datasets to improve the accuracy of
our results. Our framework achieves that by first utilizing lower threshold
CAMs to cover the target object with high certainty; second, by combining
multiple low-threshold CAMs that even out their errors while highlighting the
target object. We performed exhaustive experiments on the popular multi-modal
BRATS and prostate DECATHLON segmentation challenge datasets. Using the
proposed framework, we have demonstrated an improved dice score of up to 8% on
BRATS and 6% on DECATHLON datasets compared to the previous state-of-the-art.

###### keywords:

Semantic Segmentation, CAMs, Deep Learning, Image-level Labels.

##  1 Introduction

Semantic segmentation describes classifying objects in a given image and
returning their pixel-wise locations. Semantic segmentation offers, in
contrast to simple classification, a particular region of detection within the
image, therefore offering a much higher degree of explainability of the
network’s decision. Therefore, their use in medicine would help a doctor or
clinician conduct further investigations based on the assistant’s decision.

State-of-the-art models for semantic segmentation are achieving new top
performances at incredible speed, enabling this approach in the field of
computer-aided diagnostics (CAD) and other related to medicine. For example,
there are a variety of different applications, like colon crypt segmentation
[Cohen et al.(2015)Cohen, Rivlin, Shimshoni, and Sabo], brain tumor detection
[Tek et al.(2002)Tek, Bergtholdt, Comaniciu, and Williams, Rehman et
al.(2020)Rehman, Naz, Razzak, Akram, and Imran], detection of gastric cancer
[An et al.(2020)An, Yang, Wang, Wu, Zhou, Zeng, Huang, Xiao, Hu, Chen, et al.,
Song et al.(2020)Song, Zou, Zhou, Huang, Shao, Yuan, Gou, Jin, Wang, Chen, et
al.], discovering and tracking medical devices in surgery[Wei et al.(1997)Wei,
Arbter, and Hirzinger, Zhao et al.(2017)Zhao, Voros, Weng, Chang, and Li],
etc.. However, one significant factor limiting the quality of semantic
segmentation results in medicine is that state-of-the-art methods are fully
supervised. Fully supervised means that to achieve high-quality prediction
results, the models require vast training data with pixel-wise annotated
ground truth masks. Such datasets almost do not exist in the medical field
because generating those masks in a normal context is tedious and time-
consuming [Cordts et al.(2016)Cordts, Omran, Ramos, Rehfeld, Enzweiler,
Benenson, Franke, Roth, and Schiele]. However, in the medical sector, this
task gets exponentially more difficult. Hence, other solutions are needed if
we want to leverage the power of semantic segmentation networks in the medical
sector on a broader level. One such approach is to create semantic
segmentation with only image-level annotations. blackWe focus only on semantic
segmentation using image-level annotations in this work, because image-level
annotations are the fastest and easiest to acquire, compared to any other
weaker form of supervision

Several past works in this field have already shown that this approach can
lead to promising results [Du et al.(2022)Du, Fu, Liu, and Wang, Jiang et
al.(2022)Jiang, Yang, Hou, and Wei, Chang et al.(2020)Chang, Wang, Hung,
Piramuthu, Tsai, and Yang]. However, their applications in the field of
medicine are still very niche. Patel et al. [Patel and Dolz(2022)] have tried
to apply those methods specifically in a medical context. Their WSS-CMER
approach uses a class activation map-based approach (CAM) that extends the
ideas proposed by [Wang et al.(2020)Wang, Zhang, Kan, Shan, and Chen]. CAMs
are the most popular method for predicting semantic segmentation using image-
level labels. Furthermore, Guo et al.[Guo et al.(2020)Guo, Xu, Chi, Zhang, and
Hua] used image-level labels to localize organs in 3D images. [Chen et
al.(2022)Chen, Tian, Zhu, Li, and Du] uses an adapted CAM approach for image-
level segmentation on image in medical context. Nevertheless, CAMs have been
shown to be reliable for the localization of target objects, but their masks
lack detail. For example, with a brain tumor, they will highlight the center
of the tumor but become more uncertain towards the edges, either including too
many background pixels or not covering the whole tumor. Therefore research
concentrates on using CAMs as a base prediction and then aims to construct
methods to improve their predictions.

In this light, the state-of-the-art approaches mostly tackle the
classification loss of the CAM by adding regularizations that aim to guide the
network to predict finer and more complete masks. For example, additional
regulations for the loss function were used in [Patel and Dolz(2022)] and
[Wang et al.(2020)Wang, Zhang, Kan, Shan, and Chen], like matching the
predictions of affine transformations to the original predictions.
Alternatively, many approaches refine the CAM prediction after the fact, using
methods like pixel-similarity, for example, as in [Ahn and Kwak(2018)].
Nevertheless, those approaches do not work as well in the medical sector. We
observed that we were unable to generate reliable classifiers on our target
datasets due to size and complexity. Therefore, using the adapted loss
functions, like in [Wang et al.(2020)Wang, Zhang, Kan, Shan, and Chen], on top
of the classification loss does not improve the result. Moreover, we also
observed that refining the CAM masks after the fact was not easy since their
quality was too low. Based on our initial experiments, we have made the
following observations: First, CAM predictions with a low enough threshold
cover the target object with very high confidence. Second, when examining the
output of different CAMs, be it different base models or altogether different
methods, we noticed that they vary, especially when applying low thresholds.
Therefore, we propose a framework that combines the predictions of multiple
Grad-CAMs [Selvaraju et al.(2017)Selvaraju, Cogswell, Das, Vedantam, Parikh,
and Batra] built using different base models by determining the most optimal
ensemble of thresholds in the training set. We conducted an exhaustive
analysis of our framework on the popular BRATS 2022 [Menze et al.(2014)Menze,
Jakab, Bauer, Kalpathy-Cramer, Farahani, Kirby, Burren, Porz, Slotboom, Wiest,
et al., Bakas et al.(2017)Bakas, Akbari, Sotiras, Bilello, Rozycki, Kirby,
Freymann, Farahani, and Davatzikos, Bakas et al.(2018)Bakas, Reyes, Jakab,
Bauer, Rempfler, Crimi, Shinohara, Berger, Ha, Rozycki, et al.] and DECATHLON
[Antonelli et al.(2022)Antonelli, Reinke, Bakas, Farahani, Kopp-Schneider,
Landman, Litjens, Menze, Ronneberger, Summers, et al.] datasets and compared
it with the relevant state-of-the-art approaches in this context. Our main
contributions can be summarized as follows:

  1. 1.

A novel semantic segmentation framework using just medical imaging labels by
evaluating ensemble network configurations.

  2. 2.

Evaluations on the BRATS and DECATHLON datasets have achieved performance
improvements of 6% and 8%, respectively, over the current state-of-the-art.

  3. 3.

Our framework is completely open-source and accessible
online111https://github.com/ErikOstrowski/Automated_Ensemble.

##  2 Our Automated Ensemble Search

fig:framework

Figure 1: Overview of our Framework. (1) Stages in the classifier training,
which will be used in the Grad-CAM; (2) The Grad-CAM uses trained models and
input images to generate CAMs; (3) The Ensemble will test out multiple methods
of how to combine the CAMS;

Fig. LABEL:fig:framework presents an overview of our framework. First, we
start by training a classifier model on the target dataset. In our case, the
ResNet-34 and ResNet-50 [He et al.(2016)He, Zhang, Ren, and Sun] models proved
to be the most successful, but the framework accommodates the use of any other
classifier instance. Second, we use Grad-CAM to create the first masksfor the
different classifiers. In our case, the standard Grad-CAM worked the best.
Next, we test ensemble methods to combine two or more prediction sets. For our
final version, we choose the ensemble version that offers the best possible
results, followed by the calibration step to determine the best ensemble of
thresholds for the highest detection score.

###  2.1 Training and Exploration of Classifier Model Instances

Our framework aims to create an ensemble of different CAM methods because
their ensemble evens out their shortcomings and therefore results in more
accurate predictions than their singular pieces. Following the observation
that the best ensembles are generated from high-quality CAMs, and high-quality
CAMs are generated from high-quality classifiers, we have to investigate
classifiers for our target datasets. We limited ourselves to multiple ResNets
classifiers in our experiments. Note that in our framework, any classifier can
be used. blackAdditional information about the ResNets can be found in
Appendix A.

Instead of striving to test more complex networks, we took some inspiration
from other methods like the self-supervised Swav [Caron et al.(2020)Caron,
Misra, Mairal, Goyal, Bojanowski, and Joulin]. Swav tries to learn to
differentiate pictures without guidance instead of using annotations. For this
purpose Swav uses a contrastive loss function that compares pairs of images.
The goal of the loss function is to push away images that are different in the
feature space while pulling together those from transformations, or views, of
the same image in the feature space. In particular, our interest in the Swav
approach relies on two reasons: first, using a pre-trained unsupervised model
for the medical datasets may improve the quality of the Grad-CAM results. This
may be the case since unsupervised models are guided more toward
distinguishing between shapes rather than just guessing the correct class.
Second, many state-of-the-art approaches add additional regularization to the
classification loss. Those regularizations, e.g., the affine transformation in
SEAM, are very much in the same spirit as Swav’s contrastive learning loss.
Therefore, we assume that the activations of an unsupervised trained model may
recognize the objects more completely. We have evaluated multiple trained Swav
models but observed that their contrastive loss approach did not result in
higher-quality CAMs compared to the conventionally trained classifiers.

###  2.2 Evaluating Grad-CAMs for the trained models

After creating our candidate classifiers, we can focus on generating CAM
predictions. For that purpose, we will apply Grad-CAMs. Gradient-weighted
Class Activation Mapping (Grad-CAM) takes a network and image as input and
returns a rough mask.

However, it was shown that Grad-CAM fails to properly localize objects in an
image if it contains multiple occurrences of the same class. Moreover, it was
also found that due to not weighing the average of the partial derivatives,
the localization often does not correspond to the entire object but only parts
of it. Hence, Grad-CAM++ [Chattopadhay et al.(2018)Chattopadhay, Sarkar,
Howlader, and Balasubramanian] was introduced, which solves those problems by
using a more sophisticated formula for the importance score. As a further
optimization, SmoothGrad-CAM++ [Omeiza et al.(2019)Omeiza, Speakman, Cintas,
and Weldermariam] was introduced. blackAdditional information about GradCAMs
can be found in Appendix B. However, these improvements aim to increase the
sharpness of the boundaries of the detected object and alleviate the issues of
the original method with multiple objects of the same class within the same
image. But, many of those problems do not occur in the observed medical
datasets. All images in the BraTS and DECATHLON datasets never contain
multiple instances of the target object. Furthermore, those target objects
often have a round-ish shape, whose borders are ambiguous even for experts.
Nevertheless, we also tested SmoothGrad-CAM++ as it is a more recent iteration
of this approach. For CAM generation, we will run our trained candidate models
and the images through the candidate Grad-CAM, creating masks for all images.

###  2.3 Ensemble methods

fig:ensemble

Figure 2: Visualization of the ensemble methods: (a) ‘or’, (b) ‘and’, (c)
‘$<$’, (d) ‘$>$’

Once we have the collected masks of our candidate models and Grad-CAMs, we aim
to combine them for higher-quality results.

We considered four simple and intuitive approaches for the ensemble methods,
Fig. 2 presents a visualization of those methods. First, we have the ‘or’
ensemble, which sums up the predictions of candidates. This approach works
best when both masks have high true positive rates, generating the biggest
possible activation area between the combined masks. Second is the ‘and’
approach, which multiplies the predictions of candidates. The ‘and’ approach
minimizes the possible detection area of both masks in contrast to the ‘or’
approach. This approach works best when both models have a high true negative
rate. Next, the ‘min’ and ‘max’ approaches only take the masks with the least
or most positive classified pixels, respectively. In this way, we address the
problem of models tending to predict the size of the target object as too big
or too small while the complete prediction of one candidate is still better
than the ‘and’ or ‘or’ of all candidates.

Since the Grad-CAM methods return the predictions in logits ranging from $0$
to $1$, we can determine the threshold of the value at which we consider the
pixel a positive or negative classification. This hyperparameter gives us a
massive leeway about any given prediction’s false positive and false negative
rate. Using very high thresholds drastically reduces the area classified as
positive, leading to a high false negative rate. And vice-versa, using very
low thresholds drastically increases the area classified as positive, leading
to a high false positive rate. The most optimal threshold value varies from
candidate model to candidate model. Therefore, we decided to test the chosen
ensemble approach with all combinations of thresholds from $0$ to $1$ in steps
of $0.1$. We conduct these tests on the training set to determine the
thresholds we would use on the validation set. Our experiments have shown that
the combination of thresholds most optimal for the training set is also one of
the most optimal combinations for the validation set. A more comprehensive
ablation study of all the threshold combinations used can be found in Appendix
E.

##  3 Results and Discussion

We compare the results of our framework to relevant prior literature, namely
WSS-CMER and SEAM, which was a baseline used in the WSS-CMER work. To the best
of our knowledge, WSS-CMER is the only method in this domain specifically
targeted for medical images. Moreover, we include the results of SEAM because
it represents a current approach to weakly supervised segmentation that is not
optimized for data from the medical sector. The goal of our framework was to
extend current applications of semantic segmentation with image-level labels
to the medical sector. Primarily because we notice a massive demand for
dealing with data scarcity in the medical sector, especially with respect to
the availability of pixel-wise annotated medical images.

First, we discuss the experimental setup used for the experiments discussed in
this section. We ran the experiments on a CentOS 7.9 Operating System
executing on an Intel Core i7-8700 CPU with 16GB RAM and 2 Nvidia GeForce GTX
1080 Ti GPUs. We executed our scripts with the following software versions:
CUDA 11.5, Pytorch 1.13.0, and torchvision 0.14.0. We tested our framework on
the BraTS 2020 and DECATHLON datasets for evaluation and comparison.
Additional information about the evaluation metrics used can be found in
Appendix C.

Moreover, we trained and evaluated the datasets in a three-fold cross-
validation scheme. For BraTS, we divided the randomly shuffled dataset into
three equally sized batches and trained and tested three models, each trained
on two of the three samples and evaluated on the remaining sample. For
DECATHLON, we downloaded the three-fold cross-validation splits of WSSS-CMER.
We reported the average results over the three runs. Note that we could not
reproduce the results of WSS-CMER and SEAM. Therefore we used the numbers
published in [Patel and Dolz(2022)], blackwhen stating the results from WSS-
CMER and SEAM. We ended up using mostly the ensemble of ResNet-34 and
ResNet-50 as those two base models achieved the be results on their own.
Adding more components or different models to the ensemble resulted in lower
prediction quality. A visualization of the results on example images can be
found in Appendix D for both datasets.

BraTS: The BraTS 2020 dataset is a multi-modal brain tumor segmentation in
Magnetresonance images. Three hundred sixty-nine multi-modal scans with their
corresponding expert segmentation masks are available for the tumor
segmentation task. For comparability with WSS-CMER and SEAM, we will exclude
the additional 24 scans of BraTS 2020. These datasets use the same naming
convention and direct filename mapping between them. The scans are composed of
four modalities, which include T1, T1c, T2, and T2 Fluid Attenuated Inversion
Recovery (FLAIR). We use T1c, T2, and FLAIR for classifier training, and for
mask generation with Grad-CAM, we only use FLAIR. We achieved the best results
in our experiments just using FLAIR . We separated each scan into its set of
slides, resulting in 47,232 slides, cropped them to the size $128\text{x}128$,
and applied a threshold of $0.8$ to the normalized frame, which generated the
best results.

For this work, we followed the procedure of WSS-CMER, which only considered a
binary segmentation class, i.e., healthy vs. non-healthy targets. Therefore,
we merge the different tumor classes into one single ‘positive’ class. Our
framework is not limited to binary scenarios. However, since the used datasets
are quite complex and the work on this domain in the medical sector is still
in the early stages, we are constrained to this simplified scenario.
Nevertheless, Grad-CAM predictions have been shown to be class-specific in
natural images, as widely observed in the computer vision literature. But, in
natural images, we do not find the case that certain classes are only present
in combination with other classes, which enables the classifier to
discriminate between those classes. In both of the used datasets, most classes
only appear in combination with the other classes, so our classifier cannot
distinguish between them. Since we observed that our classifiers are already
struggling to distinguish between images with or without the presence of a
tumor, we saw no point in adding different tumor classes additionally. For
instance, [Wu et al.(2019)Wu, Du, Luo, Wen, Shen, and Feng] reported that
adding those classes results in worse predictions. In table 1, we compare the
results of the BraTS dataset. blackWe added the “Empty” mask, where we just
predicted a background mask for every image. Note that [Patel and Dolz(2022)]
do not achieve an improvement over “Empty” masks, which stresses the necessity
to improve predictions in this use case. We also added the performance of a
fully supervised semantic segmentation (FSSS) network for both datasets as an
upper bound. When we tried to reproduce [Patel and Dolz(2022)]’s results, we
experienced that the classifiers had an accuracy of around $50\%$ for both
datasets, which is in line with all other classifiers that we tested. This
shows, that adding a more sophisticated loss to the training process, like in
WSS-CMER or SEAM, does not lead to better CAMs, if the baseline classifier is
not able to work with the respective dataset. Our baseline Grad-CAM with
ResNet-34 is around $7\%$ better than WSS-CMER. Moreover, Grad-CAM ResNet-50
is $1\%$ better than its ResNet-34 counterpart. Using the bigger ResNet-101
did achieve worse results, as anticipated, due to its larger number of
parameters. We did not observe a difference in CAM quality when using
SmoothGrad-CAM++ for both datasets. Finally, by combining the two ResNets via
the ‘and’ scheme, we generate our best result, which is $1.5\%$ better than
the previous high score. blackNote that SEAM and WSS-CMER showed a variation
of over $11\%$ between samples, while our Ensemble varied less than $2\%$.
Additional information about the variations between the sample evaluations can
be found in Appendix F in table3.

DECATHLON: The DECATHLON segmentation challenge is a multi-modal prostate
dataset, with the task to localize and detect the prostate peripheral zone and
the transition zone. The dataset consists of 32 volumetric scans containing
MRI-T2 and apparent diffusion coefficient (ADC) maps with their corresponding
segmentation masks. We only used the ADC maps for classifier training and CAM
generation, as they generated the best results. We followed a similar scheme
as with BraTS, where the prostate peripheral zone and the transition zone are
combined into a single ‘positive’ class. In table 2, we compare the results of
the DECATHLON datasets. Our baseline Grad-CAM with ResNet-34 is around $7\%$
better than WSS-CMER. However, Grad-CAM ResNet-50 is $2\%$ worse than its
ResNet-34 counterpart. The smaller ResNet-18 achieved worse results as it
could not learn the relevant information. Finally, by combining the two
ResNets via the ‘and’ scheme, we generate our best result, which is $1.4\%$
better than the previous high score. blackNote that SEAM and WSS-CMER showed a
variation of over $13\%$ between samples, while our Ensemble varied less than
$3.5\%$. Additional information about the variations between the sample
evaluations can be found in Appendix F in table4.

Method | Best AVG DSC | Best AVG mIoU
---|---|---
SEAM | 56.1 | 39.0
WSS-CMER | 59.7 | 42.6
Empty | 64.6 | 47.7
ResNet-34 | 67.3 | 50.7
ResNet-50 | 68.5 | 52.1
Ensemble ResNet-34x50 ‘or’ | 67.8 | 51.3
Ensemble ResNet-34x50 ‘$<$’ | 68.3 | 51.9
Ensemble ResNet-34x50 ‘$>$’ | 68.6 | 52.2
Ensemble ResNet-34x50 ‘and’ | 70.3 | 54.2
FSSS | 81.8 | 69.2
Table 1: Comparison of EnsembleCAM with state-of-the-art techniques on the BraTS dataset. Method | Best AVG DSC | Best AVG mIoU
---|---|---
VGG19 | 63.6 | 46.6
Empty | 65.5 | 48.7
ResNet101 | 65.9 | 49.1
SEAM | 65.9 | 49.1
ResNet-18 | 67.0 | 50.3
Swav | 70.2 | 54.1
WSS-CMER | 71.3 | 55.4
ResNet-50 | 76.1 | 61.4
ResNet-34 | 78.0 | 63.8
Ensemble ResNet-34x50 ‘or’ | 77.3 | 63.0
Ensemble ResNet-34x50 ‘$<$’ | 77.7 | 63.5
Ensemble ResNet-34x50 ‘$>$’ | 78.4 | 64.4
Ensemble ResNet-34xSwav ‘and’ | 78.7 | 64.9
Ensemble ResNet-34x50 ‘and’ | 79.3 | 65.7
FSSS | 86.8 | 76.7
Table 2: Comparison of EnsembleCAM with state-of-the-art techniques on the
DECATHLON dataset.

##  4 Conclusion

In this paper, we have proposed our novel framework, which illustrates the
approach for finding an ensemble of CAM methods for semantic segmentations
with image-level labels for data from the medical sector to address the lack
of research specified for this application. Our framework proposes a scheme to
generate useful prediction masks despite the lack of quality prediction of
base models due to the complexity and size of the used datasets. Therefore,
this framework can also be applied in different contexts where the standard
approaches do not generate high-quality results. We showed that the
predictions generated by our framework achieve state-of-the-art performance on
the BraTS 2020 and DECATHLON datasets, which proves its effectiveness compared
to other approaches. Our framework is open-source and accessible at
https://github.com/ErikOstrowski/Automated_Ensemble.

This work is part of the Moore4Medical project funded by the ECSEL Joint
Undertaking under grant number H2020-ECSEL-2019-IA-876190.

## References

  * [Ahn and Kwak(2018)] Jiwoon Ahn and Suha Kwak.  Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4981–4990, 2018.
  * [An et al.(2020)An, Yang, Wang, Wu, Zhou, Zeng, Huang, Xiao, Hu, Chen, et al.] Ping An, Dongmei Yang, Jing Wang, Lianlian Wu, Jie Zhou, Zhi Zeng, Xu Huang, Yong Xiao, Shan Hu, Yiyun Chen, et al.  A deep learning method for delineating early gastric cancer resection margin under chromoendoscopy and white light endoscopy.  _Gastric Cancer_ , 23(5):884–892, 2020.
  * [Antonelli et al.(2022)Antonelli, Reinke, Bakas, Farahani, Kopp-Schneider, Landman, Litjens, Menze, Ronneberger, Summers, et al.] Michela Antonelli, Annika Reinke, Spyridon Bakas, Keyvan Farahani, Annette Kopp-Schneider, Bennett A Landman, Geert Litjens, Bjoern Menze, Olaf Ronneberger, Ronald M Summers, et al.  The medical segmentation decathlon.  _Nature communications_ , 13(1):1–13, 2022.
  * [Bakas et al.(2017)Bakas, Akbari, Sotiras, Bilello, Rozycki, Kirby, Freymann, Farahani, and Davatzikos] Spyridon Bakas, Hamed Akbari, Aristeidis Sotiras, Michel Bilello, Martin Rozycki, Justin S Kirby, John B Freymann, Keyvan Farahani, and Christos Davatzikos.  Advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features.  _Scientific data_ , 4(1):1–13, 2017.
  * [Bakas et al.(2018)Bakas, Reyes, Jakab, Bauer, Rempfler, Crimi, Shinohara, Berger, Ha, Rozycki, et al.] Spyridon Bakas, Mauricio Reyes, Andras Jakab, Stefan Bauer, Markus Rempfler, Alessandro Crimi, Russell Takeshi Shinohara, Christoph Berger, Sung Min Ha, Martin Rozycki, et al.  Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge.  _arXiv preprint arXiv:1811.02629_ , 2018.
  * [Caron et al.(2020)Caron, Misra, Mairal, Goyal, Bojanowski, and Joulin] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  _Advances in Neural Information Processing Systems_ , 33:9912–9924, 2020.
  * [Chang et al.(2020)Chang, Wang, Hung, Piramuthu, Tsai, and Yang] Yu-Ting Chang, Qiaosong Wang, Wei-Chih Hung, Robinson Piramuthu, Yi-Hsuan Tsai, and Ming-Hsuan Yang.  Weakly-supervised semantic segmentation via sub-category exploration.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 8991–9000, 2020.
  * [Chattopadhay et al.(2018)Chattopadhay, Sarkar, Howlader, and Balasubramanian] Aditya Chattopadhay, Anirban Sarkar, Prantik Howlader, and Vineeth N Balasubramanian.  Grad-cam++: Generalized gradient-based visual explanations for deep convolutional networks.  In _2018 IEEE winter conference on applications of computer vision (WACV)_ , pages 839–847. IEEE, 2018.
  * [Chen et al.(2022)Chen, Tian, Zhu, Li, and Du] Zhang Chen, Zhiqiang Tian, Jihua Zhu, Ce Li, and Shaoyi Du.  C-cam: Causal cam for weakly supervised semantic segmentation on medical image.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11676–11685, 2022.
  * [Cohen et al.(2015)Cohen, Rivlin, Shimshoni, and Sabo] Assaf Cohen, Ehud Rivlin, Ilan Shimshoni, and Edmond Sabo.  Memory based active contour algorithm using pixel-level classified images for colon crypt segmentation.  _Computerized Medical Imaging and Graphics_ , 43:150–164, 2015.  ISSN 0895-6111.  https://doi.org/10.1016/j.compmedimag.2014.12.006.  URL https://www.sciencedirect.com/science/article/pii/S0895611114002031.
  * [Cordts et al.(2016)Cordts, Omran, Ramos, Rehfeld, Enzweiler, Benenson, Franke, Roth, and Schiele] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3213–3223, 2016.
  * [Du et al.(2022)Du, Fu, Liu, and Wang] Ye Du, Zehua Fu, Qingjie Liu, and Yunhong Wang.  Weakly supervised semantic segmentation by pixel-to-prototype contrast.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2022.
  * [Guo et al.(2020)Guo, Xu, Chi, Zhang, and Hua] Heng Guo, Minfeng Xu, Ying Chi, Lei Zhang, and Xian-Sheng Hua.  Weakly supervised organ localization with attention maps regularized by local area reconstruction.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2020: 23rd International Conference, Lima, Peru, October 4–8, 2020, Proceedings, Part I 23_ , pages 243–252. Springer, 2020.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * [Jiang et al.(2022)Jiang, Yang, Hou, and Wei] Peng-Tao Jiang, Yuqi Yang, Qibin Hou, and Yunchao Wei.  L2g: A simple local-to-global knowledge transfer framework for weakly supervised semantic segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022.
  * [Menze et al.(2014)Menze, Jakab, Bauer, Kalpathy-Cramer, Farahani, Kirby, Burren, Porz, Slotboom, Wiest, et al.] Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al.  The multimodal brain tumor image segmentation benchmark (brats).  _IEEE transactions on medical imaging_ , 34(10):1993–2024, 2014.
  * [Omeiza et al.(2019)Omeiza, Speakman, Cintas, and Weldermariam] Daniel Omeiza, Skyler Speakman, Celia Cintas, and Komminist Weldermariam.  Smooth grad-cam++: An enhanced inference level visualization technique for deep convolutional neural network models.  _arXiv preprint arXiv:1908.01224_ , 2019.
  * [Patel and Dolz(2022)] Gaurav Patel and Jose Dolz.  Weakly supervised segmentation with cross-modality equivariant constraints.  _Medical Image Analysis_ , 77:102374, 2022.  ISSN 1361-8415.  https://doi.org/10.1016/j.media.2022.102374.  URL https://www.sciencedirect.com/science/article/pii/S1361841522000275.
  * [Rehman et al.(2020)Rehman, Naz, Razzak, Akram, and Imran] Arshia Rehman, Saeeda Naz, Muhammad Imran Razzak, Faiza Akram, and Muhammad Imran.  A deep learning-based framework for automatic brain tumors classification using transfer learning.  _Circuits, Systems, and Signal Processing_ , 39(2):757–775, 2020.
  * [Selvaraju et al.(2017)Selvaraju, Cogswell, Das, Vedantam, Parikh, and Batra] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , pages 618–626, 2017.
  * [Simonyan and Zisserman(2014)] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014.
  * [Smilkov et al.(2017)Smilkov, Thorat, Kim, Viégas, and Wattenberg] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg.  Smoothgrad: removing noise by adding noise.  _arXiv preprint arXiv:1706.03825_ , 2017.
  * [Song et al.(2020)Song, Zou, Zhou, Huang, Shao, Yuan, Gou, Jin, Wang, Chen, et al.] Zhigang Song, Shuangmei Zou, Weixun Zhou, Yong Huang, Liwei Shao, Jing Yuan, Xiangnan Gou, Wei Jin, Zhanbo Wang, Xin Chen, et al.  Clinically applicable histopathological diagnosis system for gastric cancer detection using deep learning.  _Nature communications_ , 11(1):1–9, 2020.
  * [Tek et al.(2002)Tek, Bergtholdt, Comaniciu, and Williams] Hüseyin Tek, Martin Bergtholdt, Dorin Comaniciu, and James Williams.  Segmentation of 3d medical structures using robust ray propagation.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 572–579. Springer, 2002.
  * [Wang et al.(2020)Wang, Zhang, Kan, Shan, and Chen] Yude Wang, Jie Zhang, Meina Kan, Shiguang Shan, and Xilin Chen.  Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12275–12284, 2020.
  * [Wei et al.(1997)Wei, Arbter, and Hirzinger] Guo-Qing Wei, Klaus Arbter, and Gerd Hirzinger.  Automatic tracking of laparoscopic instruments by color coding.  In _CVRMed-MRCAS’97_ , pages 357–366. Springer, 1997.
  * [Wu et al.(2019)Wu, Du, Luo, Wen, Shen, and Feng] Kai Wu, Bowen Du, Man Luo, Hongkai Wen, Yiran Shen, and Jianfeng Feng.  Weakly supervised brain lesion segmentation via attentional representation learning.  In _International Conference on Medical Image Computing and Computer-Assisted Intervention_ , pages 211–219. Springer, 2019.
  * [Zhao et al.(2017)Zhao, Voros, Weng, Chang, and Li] Zijian Zhao, Sandrine Voros, Ying Weng, Faliang Chang, and Ruijian Li.  Tracking-by-detection of surgical instruments in minimally invasive surgery via the convolutional neural network deep learning-based method.  _Computer Assisted Surgery_ , 22(sup1):26–35, 2017.

black

##  Appendix A ResNets

When we aim to use deeper architectures than VGG, we must resort to networks
that employ techniques like ResNet’s skip connections to avoid the vanishing
gradients problem. Additionally, ResNet introduced batch normalization as a
regularization strategy. In contrast to VGG’s [Simonyan and Zisserman(2014)]
19 layers, ResNet can utilize up to 200. Hence, ResNet can retrieve more
complex information and is better at handling larger datasets. Nevertheless,
in our experiments, we encountered worse results when employing bigger
ResNets, and therefore, we refrained from employing deeper and more complex
networks.

black

##  Appendix B GradCAMs

GradCAM runs an input image through a model and takes the $K$ outputs
$A^{k}\in\mathbb{R}^{u\times v}$ of the final convolutional layer. Next, we
calculate the gradient score for each class $c$, of the logits $y^{c}$ for the
feature map activations $A^{k}$, i.e., $\frac{\partial y^{c}}{\partial
A^{k}}$. Then the gradients are global average pooled across each feature map
to give us the importance score $\alpha^{c}_{k}$:

$\alpha^{c}_{k}=\frac{1}{uv}\sum_{i}\sum_{j}\frac{\partial y^{c}}{\partial
A^{k}_{i,j}}$

where $k$ denotes the index of the $Z$ activation maps, and $i,j$ are the
feature map coordinates. The importance score weights the relevance of each
feature map $k$ for each class $c$. Then, we multiply each activation map
$A^{k}$ by its importance score $\alpha^{c}_{k}$ for each class and take their
sum to gain the prediction mask $M^{c}$:

$M^{c}=ReLU(\sum_{k}\alpha^{c}_{k}A^{k})$

We apply ReLU to the summation to only consider the pixels that positively
influence the score of the class of interest. The mask $M^{c}$ highlights
areas of the image that were activated for the respective class. Finally, we
resize and rescale $M^{c}$ so that the mask fits the original image
dimensions. GradCAM++’s formula for the importance score
$\alpha_{i,j}^{c_{k}}$:

$\alpha_{i,j}^{c_{k}}=\frac{\frac{\partial^{2}y^{c}}{(\partial
A^{k}_{i,j})^{2}}}{2\frac{\partial^{2}y^{c}}{(\partial
A^{k}_{i,j})^{2}}+\sum_{a}\sum_{b}A_{ab}^{k}(\frac{\partial^{3}y^{c}}{(\partial
A^{k}_{i,j})^{3}})}$

SmoothGrad-CAM++ combined Grad-CAM++ with SMOOTHGRAD [Smilkov et
al.(2017)Smilkov, Thorat, Kim, Viégas, and Wattenberg], which improves the
original method by sharpening the Grad-CAM output by taking random samples in
a neighborhood of an input $x$ and averaging the resulting outputs.

##  Appendix C Evaluation metrics

To assess the performance of the proposed approach, we employ the common Dice
Similarity coefficient (DSC) and mean Intersection over Union (mIoU):

$DSC(GT,Pred)=\frac{2|GT\cdot Pred|}{|GT|+|Pred|}$

Where $GT$ and $Pred$ are binary matrices (with values of 1 for elements
inside a group and 0 otherwise), $GT$ signifies the ground truth, and $Pred$
signifies the classification result.
The mean Intersection-over-Union (mIoU):

$mIoU=\frac{1}{N}\sum_{i=1}^{N}\frac{p_{i,i}}{\sum_{j=1}^{N}p_{i,j}+\sum_{j=1}^{N}p_{j,i}-p_{i,i}}$

where $N$ is the total number of classes, $p_{i,i}$ the number of pixels
classified as class $i$ when labelled as class $i$. $p_{i,j}$ and $p_{j,i}$
are the number of pixels classified as class $i$ that were labelled as class
$j$ and vice-versa, respectively.

##  Appendix D Analyzing results of the framework

Fig. LABEL:fig:example presents an overview of our experimental results. The
first two rows show examples from the DECATHLON dataset, and the second two
rows show examples from the BraTS dataset. We notice that the (f) with the
‘and’ combination achieves the biggest overlap with the ground-truth row (b)
in terms of visual and metric-based evaluations, as shown in Section 3.
Whereas in contrast, the ‘or’ ensemble used in the (e) row creates the results
with the least amount of overlap when considering the false positives. This
shows our observation that motivated the framework. Both ResNet-34 and
ResNet-50 cover a lot of ground-truth but also too many background pixels.
However, we notice that the activated background pixels differ between the two
models, resulting in the best predictions when using the ‘and’ ensemble and
the worst when using the ‘or’ ensemble.

fig:example

Figure 3: Example results of our Framework on BraTS and DECATHLON: (a) Source
image, (b) ground-truth, (c) ResNet34 Grad-CAM, (d) ResNet50 Grad-CAM, (e)
‘or’ ensemble, (f) ‘and’ ensemble

##  Appendix E Threshold testing

Fig. LABEL:fig:heat presents an overview of the mIoU achieved when different
thresholds are used for the two models under consideration. These values are
obtained by using the training dataset results averaged over all three cross-
validation iterations. The left graphic shows the results of BraTS, and the
right shows the DECATHLON results. We notice that the left graphic is lighter
in color than the right. The reason for that is that overall mIoU results on
BraTS are lower than the results on DECATHLON. We notice that both datasets
reach their best results at thresholds $70\%$ for both models. Both graphics
show results achieved with the ‘and’ ensemble; the optimal thresholds appear
to vary based on the ensemble method under consideration.

fig:heat

Figure 4: All threshold combinations of ResNet-34 and ResNet-50 ‘and’
combination on the train set. Y-axis describes the threshold used for
ResNet-34, X-axis is the threshold used for ResNet-50. BraTS on the left,
DECATHLON on the right.

##  Appendix F Analyzing Sample Variations

Method | Best AVG DSC | Best AVG mIoU
---|---|---
SEAM | 56.1 $\pm$ 11.9 | 39.0 $\pm$ 6.3
WSS-CMER | 59.7 $\pm$ 11.8 | 42.6 $\pm$ 6.3
Ensemble ResNet-34x50 ‘and’ |  70.3 $\pm$ 1.5 |  54.2 $\pm$ 0.8
Table 3: Comparison of EnsembleCAM with state-of-the-art techniques on the BraTS dataset, including the variations observed between the samples. Method | Best AVG DSC | Best AVG mIoU
---|---|---
SEAM | 65.9 $\pm$ 13.35 | 49.1 $\pm$ 7.2
WSS-CMER | 71.3 $\pm$ 14.6 | 55.4 $\pm$ 7.9
Ensemble ResNet-34x50 ‘and’ |  79.3 $\pm$ 3.1 |  65.7 $\pm$ 1.6
Table 4: Comparison of EnsembleCAM with state-of-the-art techniques on the
DECATHLON dataset, including the variations observed between the samples.

Table. 3 and table 4 give an overview of the DSC and mIoU achieved with state-
of-the-art techniques on both datasets, including the variations between the
three different validation samples. Note that we cannot give a more
comprehensive analysis regarding what exact score was achieved on each
validation set, because the WSS-CMER did only provide the average variations.
Nevertheless, we notice that for both datasets, the variation in the
evaluation results is much lower for EnsembleCAM compared to both SEAM and
WSS-CMER. Showing that EnsembleCAM is much more consistent when being
confronted with new data.