# NUCLEAR PAIRING AT FINITE TEMPERATURE AND
ANGULAR MOMENTUM 111To appear in the Proceedings of the First Workshop on
State of the Art in Nuclear Cluster Physics, Strasbourg 13 - 16 May, 2008.

N. DINH DANG  Heavy-Ion Nuclear Physics Laboratory, Nishina Center for
Accelerator-Based Science,
RIKEN 2-1 Hirosawa, Wako city, 351-0198 Saitama, Japan
and
Institute for Nuclear Science and Technique, Hanoi, Vietnam
<EMAIL_ADDRESS>N. QUANG HUNG 222On leave of absent from the Institute of
Physics and Electronics, Hanoi, Vietnam. Heavy-Ion Nuclear Physics Laboratory,
Nishina Center for Accelerator-Based Science,
RIKEN 2-1 Hirosawa, Wako city, 351-0198 Saitama, Japan
<EMAIL_ADDRESS>
###### Abstract

An approach is proposed to nuclear pairing at finite temperature and angular
momentum, which includes the effects of the quasiparticle-number fluctuation
and dynamic coupling to pair vibrations within the self-consistent
quasiparticle random-phase approximation. The numerical calculations of
pairing gaps, total energies, and heat capacities are carried out within a
doubly folded multilevel model as well as several realistic nuclei. The
results obtained show that, in the region of moderate and strong couplings,
the sharp transition between the superconducting and normal phases is smoothed
out, causing a thermal pairing gap, which does not collapse at a critical
temperature predicted by the conventional Bardeen-Cooper-Schrieffer’s (BCS)
theory, but has a tail extended to high temperatures. The theory also predicts
the appearance of a thermally assisted pairing in hot rotating nuclei.

##  1 Introduction

The effect of temperature and angular momentum on pairing properties is an
interesting subject in the study of nuclear structure. Because of its
simplicity, the BCS theory is often used, which offers a good description of
pairing correlation in the macroscopic systems such as metallic
superconductors. It predicts a collapse of the pairing gap at $T_{\rm c}$,
which signals the sharp superfluid-normal (SN) phase transition at finite
temperature. The BCS theory, however, ignores quantal and thermal
fluctuations, which are significant in finite small systems. Therefore, it
needs to be corrected for the application to finite nuclei. Various
theoretical approaches have been proposed to study the effects of fluctuations
on nuclear pairing [2, 3, 4]. Their results show that, at zero angular
momentum, thermal fluctuations smear out the sharp SN phase transition,
resulting in a pairing gap, which does not collapse at finite temperature. In
rotating nuclei, a phenomenon of temperature induced pair correlations, which
reflects the strong fluctuations of the order parameter in small systems, has
also been predicted [5]. The recent microscopic approach, called the modified
BCS (MBCS) theory [6] has shown, for the fist time, that the microscopic
source causing the non-collapsing pairing gap is the quasiparticle-number
fluctuation (QNF).

Recently, we proposed the self-consistent quasiparticle random-phase
approximation (SCQRPA) [7], which includes the QNF as well as the quantal
fluctuations due to dynamic coupling to pair vibrations. The purpose of
present work is to extend this approach to finite temperature and finite
angular momentum.

##  2 Formalism

The pairing Hamiltonian is considered, which describes a system of $N$
particles interacting via a pairing force with the parameter $G$ and rotating
with angular velocity $\gamma$ and a fixed angular momentum projection $M$ on
the laboratory (or body) fixed $z-$ axis:

$H=\sum_{k}\epsilon_{k}(N_{k}+N_{-k})-G\sum_{k,k^{\prime}}P_{k}^{\dagger}P_{k^{\prime}}-\lambda\hat{N}-\gamma\hat{M}~{},\hskip
14.22636ptN_{\pm k}=a_{\pm k}^{\dagger}a_{\pm k}~{},\hskip
14.22636ptP_{k}^{\dagger}=a_{k}^{\dagger}a_{-k}^{\dagger}~{},$ (1)

where $a_{\pm k}^{\dagger}$ ($a_{\pm k}$) is the operator that creates
(annihilates) a particle with angular momentum $k$, spin projection $m_{k}$ or
$-m_{k}$, and energy $\epsilon_{k}$. For simplicity, the subscripts $k$ label
the single-particle states $|k,m_{k}\rangle$ with $m_{k}>$ 0, whereas $-k$
denote the time-reversal states $|k,-m_{k}\rangle$. The particle number
operator $\hat{N}$ is defined as
$\hat{N}=\sum_{k}(a_{k}^{\dagger}a_{k}+a_{-k}^{\dagger}a_{-k})$, whereas
$\hat{M}=\sum_{k}m_{k}(a_{k}^{\dagger}a_{k}-a_{-k}^{\dagger}a_{-k})$ is the
$z$-projection of total angular momentum. The variational procedure is applied
to minimize the expectation value of this Hamiltonian in the grand canonical
ensemble. The result yields the final equations for the pairing gap, particle
number and total angular momentum, which include the effect of QNF in the form

$\Delta_{k}=\Delta+\delta\Delta_{k}=G\sum_{k^{\prime}}u_{k^{\prime}}v_{k^{\prime}}\langle{\cal
D}_{k^{\prime}}\rangle+G\frac{\delta{\cal N}_{k}^{2}}{\langle{\cal
D}_{k}\rangle}u_{k}v_{k}~{},$ (2) $N=2\sum_{k}\bigg{[}v_{k}^{2}\langle{\cal
D}_{k}\rangle+\frac{1}{2}\big{(}1-\langle{\cal
D}_{k}\rangle\big{)}\bigg{]}~{},\hskip
14.22636ptM=\sum_{k}m_{k}(n_{k}^{+}-n_{k}^{-})~{},$ (3)

where the quasiparticle energy $E_{k}$ and renormalized single-particle energy
$\epsilon_{k}^{\prime}$ are given as

$E_{k}=\sqrt{(\epsilon^{\prime}_{k}-Gv_{k}^{2}-\lambda)^{2}+\Delta_{k}^{2}}~{},$
(4) $\epsilon_{k}^{\prime}=\epsilon_{k}+\frac{G}{\langle{\cal
D}_{k}\rangle}\sum_{k^{\prime}}(u_{k^{\prime}}^{2}-v_{k^{\prime}}^{2})\bigg{(}\langle{\cal
A}_{k}^{\dagger}{\cal A}_{k^{\prime}}\rangle+\langle{\cal
A}_{k}^{\dagger}{\cal A}_{k^{\prime}}^{\dagger}\rangle_{k\neq
k^{\prime}}\bigg{)}~{},$ (5)

with $\langle{\cal D}_{k}\rangle=1-n_{k}^{+}-n_{k}^{-}$, and ${\cal
A}^{\dagger}_{k}\equiv\alpha_{k}^{\dagger}\alpha_{-k}^{\dagger}$. The
expectation values $\langle{\cal A}^{\dagger}_{k}{\cal A}_{k^{\prime}}\rangle$
and $\langle{\cal A}^{\dagger}_{k}{\cal A}^{\dagger}_{k^{\prime}}\rangle$ are
evaluated by solving a set of coupled equations, which contain the SCQRPA $X$
and $Y$ amplitudes. The QNF is given as $\delta{\cal
N}_{k}^{2}=n_{k}^{+}(1-n_{k}^{+})+n_{k}^{-}(1-n_{k}^{-})$, where the
quasiparticle occupation numbers $n_{\pm k}$ are found from the integral
equations

$n_{k}^{\pm}=\frac{1}{\pi}\int_{-\infty}^{\infty}\frac{\gamma_{k}^{\pm}(\omega)(e^{\beta\omega}+1)^{-1}}{[\omega-
E_{k}\pm\gamma
m_{k}-M_{k}^{\pm}(\omega)]^{2}+[\gamma_{k}^{\pm}(\omega)]^{2}}d\omega~{},$ (6)

with the mass operators $M_{k}^{\pm}(\omega)$ obtained by solving the set of
equations for double-time quasiparticle Green’s functions and those of a
quasiparticle coupled with SCQRPA pair vibrations. The quasiparticle dampings
are given as $\gamma_{k}^{\pm}(\omega)=\Im m[M_{k}^{\pm}(\omega\pm
i\varepsilon)]$.

The proposed approach is called the FTBCS1+SCQRPA theory. Neglecting the
coupling to SCQRPA, i.e. the factors $\langle{\cal A}^{\dagger}_{k}{\cal
A}_{k^{\prime}}\rangle$ and $\langle{\cal A}^{\dagger}_{k}{\cal
A}^{\dagger}_{k^{\prime}}\rangle$, it becomes the FTBCS1 theory, which is
different from the conventional FTBCS theory by the presence of the QNF. The
violation of particle number at zero angular momentum is approximately removed
by applying the Lipkin-Nogami (LN) method. The corresponding approaches are
called the FTLN1+SCQRPA and FTLN1.

##  3 Results

The numerical calculations are carried out within the $\Omega$ doubly
degenerate equidistant model with the number $\Omega$ of levels equal to that
of particles, $N$, as well as for 20O, 44Ca, 56Fe, and 120Sn. The results
obtained show that, at zero angular momentum, under the effect of QNF within
the FTBCS1 (FTLN1), the sharp SN phase transition predicted by the FTBCS
theory is smoothed out. As the result, the pairing gap does not collapse at
$T=T_{\rm c}$, but has a tail, which extends to high $T$. The dynamic coupling
to the SCQRPA vibrations significantly improves the agreement with the exact
results for the total energies and heat capacities obtained for $N=10$ as well
as those obtained 56Fe within the finite-temperature quantum Monte Carlo
method [8] [Figs. 1 (a) – 1 (c)]. However, for heavy nuclei such as 120Sn, the
SCQRPA corrections are found to be negligible in comparison with the FTBCS1
(FTLN1) results.

For 20O and 44Ca, the FTBCS1 pairing gaps, obtained at different $M$,
decreases as $T$ increases and do not collapses at high $T$. At $M$ higher
than the critical value $M_{\rm c}$, where the FTBCS gap for $T=0$ disappears,
there appear thermally assisted pairing correlations, in which the FTBCS1 gap
reappears at a given $T_{1}>0$, and remains finite at $T>T_{1}$ [Fig. 1 (d)].
This phenomenon is caused by the QNF within the FTBCS1 theory. At $T=0$, the
QNF is zero, so the FTBCS and FTBCS1 gaps are the same as functions of $M$ (or
$\gamma$), and both collapse at $M=M_{\rm c}$. However, with increasing $T$,
the FTBCS1 gaps, which are obtained at different $T$, collapse at $M>M_{\rm
c}$, and remain finite even at very high $T$, whereas those given by the
conventional FTBCS theory vanish at $M\geq M_{\rm c}$ and $T\geq T_{\rm c}$
[Figs. 1. (e) and 1 (f)].

file=fig.eps,width=8.5cm

Figure 1: Left panels: Pairing gaps (a), total energies (b), and heat
capacities (c) obtained within the FTBCS (dotted lines), FTBCS1 (thin solid
lines), FTLN1 (thin dashed lines), FTBCS1+SCQRPA (thick solid lines) and
FTLN1+SCQRPA (thick dashed lines) for neutrons in 56Fe. Boxes and crosses with
error bars connected by dash-dotted lines are results of Ref. 8. Right panels:
pairing gaps as functions of $T$ at different $M$ (d), and as functions of $M$
(e) and $\gamma$ (f) at various $T$ obtained within the FTBCS1 theory for
neutrons in 20O.

## References

  * [1] L. G. Moretto, Phys. Lett. B 35, 397 (1971); Nucl. Phys. A 185, 145 (1971).
  * [2] L. G. Moretto, Phys. Lett. B 40, 1 (1972).
  * [3] A. L. Goodman, Nucl. Phys. A 352, 30 (1981); Phys. Rev. C 29, 1887 (1984).
  * [4] R. Rossignoli, P. Ring, and N.D. Dang, Phys. Lett. B 297, 9 (1992).
  * [5] S. Frauendorf, _et. al._ , Phys. Rev. B 68, 024518 (2003).
  * [6] N.D. Dang and V. Zelevinsky, Phys. Rev. C 64, 064319 (2001); N.D. Dang and A. Arima, Phys. Rev. C 67, 014304 (2003); N. D. Dang, Nucl. Phys. A 784, 147 (2007).
  * [7] N.Q. Hung and N.D. Dang, Phys. Rev. C 76, 054302 (2007), Ibid. 77, 029905(E) (2008).
  * [8] S. Rombouts, K. Heyde, and N. Jachowicz, Phys. Rev. C 58, 3295 (1998).


# Detailed retinal vessel segmentation without human annotations using
simulated optical coherence tomography angiographs

Linus Kreitner  Johannes C. Paetzold  Nikolaus Rauch  Chen Chen  Ahmed M.
Hagag  Alaa E. Fayed  Sobha Sivaprasad  Sebastian Rausch  Julian Weichsel
Bjoern H. Menze  Matthias Harders  Benjamin Knier  Daniel Rueckert  and Martin
J. Menten  L. K., D. R. and M. J. M. are affiliated with the Lab for AI in
Medicine, Klinikum rechts der Isar, Technical University of Munich, Germany.
J. C. P., C. C., D. R. and M. J. M are with BioMedIA, Imperial College London,
United Kingdom (UK). J. C. P is also affiliated with ITERM Institute Helmholtz
Zentrum Muenchen, Germany. N. R. and M. H. are with the Interactive Graphics
and Simulation Group, University of Innsbruck, Austria. C. C. is also with the
Oxford BioMedIA group, University of Oxford, UK. A. M. H. and S. S. are with
the NIHR Moorfields Biomedical Research Centre, Moorfields Eye Hospital NHS
Foundation Trust, London, UK. A. M. H. is also with Boehringer Ingelheim
Limited, UK. S. S. is with University College London, UK. A. E. F. is with the
Department of Ophthalmology, Kasr Al-Ainy School of Medicine, Cairo
University, Egypt and the Watany Eye Hospital, Cairo, Egypt. S. R. and J. W.
are with Heidelberg Engineering GmbH, Heidelberg, Germany. B. H. M. is with
the Department of Quantitative Biomedicine, University of Zurich, Switzerland.
B.K̇. is with the Department of Neurology, Klinikum rechts der Isar, Technical
University of Munich, Germany.

###### Abstract

Optical coherence tomography angiography (OCTA) is a non-invasive imaging
modality that can acquire high-resolution volumes of the retinal vasculature
and aid the diagnosis of ocular, neurological and cardiac diseases.
Segmentation of the visible blood vessels is a common first step when
extracting quantitative biomarkers from these images. Classical segmentation
algorithms based on thresholding are strongly affected by image artifacts and
limited signal-to-noise ratio. The use of modern, deep learning-based
segmentation methods has been inhibited by a lack of large datasets with
detailed annotations of the blood vessels. To address this issue, recent work
has employed transfer learning, where a segmentation network is trained on
synthetic OCTA images and is then applied to real data. However, the
previously proposed simulation models are incapable of faithfully modeling the
retinal vasculature and do not provide effective domain adaptation. Because of
this, current methods are not able to fully segment the retinal vasculature,
in particular the smallest capillaries. In this work, we present a lightweight
simulation of the retinal vascular network based on space colonization for
faster and more realistic OCTA synthesis. Moreover, we introduce three
contrast adaptation pipelines to decrease the domain gap between real and
artificial images. We demonstrate the superior performance of our approach in
extensive quantitative and qualitative experiments on three public datasets
that compare our method to traditional computer vision algorithms and
supervised training using human annotations. Finally, we make our entire
pipeline publicly available, including the source code, pretrained models, and
a large dataset of synthetic OCTA images.

Blood vessels, Deep learning, Image segmentation, OCTA, Transfer learning

##  1 Introduction

Figure 1:  In our proposed pipeline, a CNN is trained for blood vessel
segmentation on synthetic OCTA images. When applied to real data, the trained
network is able to produce highly detailed segmentation maps. We evaluate our
method on three public datasets and publish our codebase, together with
pretrained models and a large synthetic dataset.

Optical coherence tomography angiography (OCTA) is a non-invasive imaging
modality that can acquire high-resolution volumes of the retinal vasculature.
The technology can aid the diagnosis of ocular, neurological, and cardiac
diseases [1, 2, 3]. Recent work has investigated the feasibility of automated
disease classification and grading based on OCTA images in an end-to-end
approach using convolutional neural networks (CNN) [4]. However, these systems
only offer limited explainability and might be subject to unknown training
biases. For this reason, others first extract a vessel segmentation map and
then compute disease-relevant biomarkers, such as vessel density, radius, or
tortuosity. Currently, most segmentation methods are based on thresholding
algorithms [5]. The most reliable choices are found to be adaptive
thresholding, Frangi filters with binarization, Gabor filters with
binarization, and optimally oriented flux [6]. However, all methods require
manual tuning and suffer from poor robustness towards the diverse set of image
artifacts in real OCTA data. Furthermore, small vessels such as the
capillaries from the deep vascular complex (DVC) are hard to detect using
thresholding but crucial for early disease detection [5]. The adoption of
machine learning-based vessel segmentation algorithms is hindered by a lack of
sufficient training labels. Manual annotation is not only time-consuming, but
also difficult due to complex vessel branching, low resolution, and low-
contrast areas. Under great effort, a limited number of OCTA datasets with
vessel annotations have been published. However, they mostly do not contain
labels for smaller vessels and sometimes exhibit labeling inconsistencies [7,
8, 6].

In settings without a sufficient amount of labeled training data, transfer
learning can be used. Thereby, a network is trained in a related data domain,
in which a large amount of annotations is available, before being applied to
the target domain. Costa et al. propose generating synthetic training samples
that can be used to train a network for fundus vessel segmentation [9]. An
autoencoder network generates vessel graphs following a data distribution
given by an annotated set of fundus images. In a second step, a generative
adversarial network (GAN) transforms the vessel graph into a realistic looking
fundus images. The authors are able to generate fairly realistic data pairs,
but breaks in generated vessels and limited realism of the images inhibit the
downstream segmentation performance on real data. Fu et al. propose to train a
U-net for image denoising on paired fundus images and apply it on OCTA en-face
images for vessel enhancement [10]. Their transfer learning method surpasses
the segmentation performance of supervised methods on a small evaluation
dataset. Recently, Menten et al. presented a novel approach for OCTA data that
does not require any labeled data [11]. They leverage a physiology-based
simulation model to generate artificial vessel maps of the retinal vasculature
with inherently matching labels. A CNN is then trained on these synthetic OCTA
images in a supervised fashion (see figure 1). Several image augmentations
simulate OCTA artifacts and make the network more robust against the domain
shift when the pre-trained network is applied to real data. However, their
work has several limitations: First, The rigid nature of the simulation model
together with image synthesis taking up to eight hours per sample prevents the
simulation of smaller capillaries in the retinal vasculature. It further
complicates hyperparameter tuning to increase the image realism. Second, the
manually tuned image augmentations are not able to fully bridge the domain gap
between synthetic and real images. Because of these shortcomings, the method
proposed by Menten et al. is not able to fully segment real OCTA images,
partially ignoring the smallest capillaries.

In this work, we introduce a pipeline to train a CNN for vessel segmentation
of 2D macular OCTA images using synthetic OCTA images that overcomes the
aforementioned disadvantages. To this end, we make the following key
contributions:

  1. 1.

We replace the computationally costly physiology-based simulation with a
statistical angiogenesis model based on space colonization from Rauch and
Harders [12] (see figure 2). The new algorithm substantially speeds up the
image generation and enables us to quickly test new configurations. With our
method, we can control the look and dynamics of the vessel growth, leading to
a more complex and realistic vasculature of the retina.

  2. 2.

We employ and compare several advanced contrast adaptation strategies to
bridge the domain gap between real and synthetic images. We show that this
step is crucial for the segmentation robustness on real data.

  3. 3.

We perform extensive quantitative and qualitative benchmarking on three public
datasets. We compare our methods with traditional computer vision algorithms
and models trained directly on the dataset labels.

  4. 4.

We published our entire pipeline as an open source tool, including the
synthetic dataset and the pretrained models at https://github.com/TUM-
AIMED/OCTA-seg.

Figure 2:  Comparison of Menten et al.’s image synthesis and ours. Synthetic
images are generated by 1) growing a vascular network via an angiogenesis
driven simulation model, and 2) enhancing vessel map realism with data
augmentations. Our simulation model enables faster synthesis of complex vessel
structures and models artifacts more realistically.

##  2 Methods

###  2.1 Statistical simulation of the retinal vasculature

A central component of our pipeline is the realistic simulation of retinal
vasculature. Originating from the optical nerve, blood vessels traverse the
entire retina as two vascular complexes. The superficial vascular complex
(SVC) mostly contains larger vessels, while the deep vascular complex (DVC)
consists primarily of capillaries with a radius as small as 2.5$\,\mu$m [13].
The retina also contains a circular shaped avascular zone around the fovea
(FAZ) of about 0.35 mm in diameter. Similar to other works, we simulate vessel
development using a forest of 3D rooted binary trees [12, 14, 15, 16]. Growth
of these graphs is governed by a set of predefined rules. Each tree is
initialized by a root node with a single child node. An edge encodes the
length and radius of a vessel segment. There are three types of nodes:

  * •

Leaf-node: The node does not have any children and is only connected to its
parent.

  * •

Inter-node: The node has exactly one child.

  * •

Bounded node: The node has reached its maximum of two children and is not
considered for proliferation.

In clinical practice, OCTA images are mostly viewed as 2D en-face projection
representation instead of the full 3D volume. We therefore relax the realism
in depth, as we are only interested in the resulting 2D maximum intensity
projection (MIP) along the z-axis. To simulate the geometrical shape of a
3$\times$3 mm2 central crop of the retina, we define our simulation space as a
slab with dimensions of $1\times 1\times\frac{1}{76}$. Since the optical nerve
is located outside the FOV for macular images, we randomly place 16 root
stumps at the lateral faces of the simulation space cuboid.

Menten et al. base their growth model on an angiogenesis simulation by
Schneider et al. [14], which has been frequently used to create synthetic
blood vessels [17, 18]. Inspired by angiogenesis in biological organisms, they
model the diffusion of oxygen (O2) and the related vascular endothelial growth
factor concentration iteratively for the entire volume to control the
sprouting of new vessels from leaf- or inter-nodes. The repeated calculation
of the oxygenation for each vessel sprout is computationally very expensive
and leads to an excessive synthesis duration of about eight hours per image
for Menten et al. This prevents the generation of more complex vessel graphs
and complicates further development of the algorithm. Additionally, the
complexity of the simulation leads to a reduced configurability and inhibits
the generation of vessel trees with specific growth pattern. Rauch and Harders
present a simplified angiogenesis model that completely avoids the explicit
calculation of the oxygen concentration by using a statistical approach based
on space colonization [12]. In this work, we adapt this approach and will
discuss the algorithm in the following. A list of all mentioned variables is
provided in table 1.

Table 1: The parameters used by our angiogenesis model. The growth process of the OCTA volume is divided into a phase to simulate the SVC, and the second phase to generate the smaller and more dense vessels of the DVC. Var | Description | SVC | DVC
---|---|---|---
r${}_{\textnormal{faz}}$ | Radius of the FAZ [mm] |  $\sim\mathcal{N}$(0.45,0.021)
r${}_{\textnormal{rot}}$ | Rotation effect radius around the FAZ [mm] | 1.05 | 1.05
I | Number of iterations | 100 | 100
N | Number of added OSs per iteration | 1000 | 2000
d | Terminal segment length [mm] | 0.1 | 0.04
r | Terminal vessel radius | 0.0025 | 0.0025
$\epsilon_{n}$ | Min. distance between nodes and OSs [mm] | See equation 1
$\epsilon_{s}$ | Min. distance between OSs [mm] | 0.135 | 0.045
$\epsilon_{k}$ | Satisfaction rangeof nodes [mm] | 0.135 | 0.045
$\delta$ | Distance of perception volume [mm] | 0.2925 | 0.0975
$\gamma$ | Angle of perception cone w.r.t. parent | 50° | 90°
$\phi$ | Min. std of attr. vector angles for sym. bif. | 15° | 15°
$\omega$ | Weighting factor optimal growth direction | 0.3 | 0
$\kappa$ | Bifurcation exponent | 2.55 | 2.9
$\Delta\sigma$ | Linear growth rate of sim. space per iter | 0.02 | 0.02

####  2.1.1 Oxygen sink placement

Rauch and Harders model the circulatory system with oxygen-providing arterial
trees and CO2-removing venous trees [12]. Instead of explicitly computing the
oxygen distribution, the authors propose to randomly place $N$ oxygen sinks
(OSs) in the simulation space every iteration. Each OS acts as an attraction
point for its closest arterial node within range $\delta$. Each OS must have a
minimal distance $\epsilon_{s}$ to existing OSs and a minimal distance
$\epsilon_{n}$ to nearby arterial nodes. In reality, smaller vessels supply
more oxygen to surrounding tissue than larger ones. We model this by allowing
new OSs to be placed closer to arterial nodes as the vessels’ diameter
increase. We use the oxygen concentration heuristic defined by Schneider et
al. [14], and set $\epsilon_{n}$ relative to a vessel’s radius:

$\epsilon_{n}=0.02\cdot
203.9\,\frac{\mu\textnormal{l}}{\textnormal{ml}}\cdot\frac{r_{\textnormal{node}}}{3.5\,\mu\textnormal{m}}\exp\left(1-\frac{r_{\textnormal{node}}}{3.5\,\mu\textnormal{m}}\right).$
(1)

Once a new vessel node is placed within range $\epsilon_{k}$ of an attraction
point, the attraction point is considered saturated and is turned into a CO2
source. These CO2 emitters now act as attraction points for venous trees,
whose growth process follows analogue to arterial trees. Once CO2 sources are
satisfied, they are removed.

####  2.1.2 Leaf-node proliferation

Every leaf node has a perception cone with angle $\gamma$ and distance
$\delta$ (see figure 3). A given attraction point at position
$\vec{p}_{\textnormal{att}}$ is considered by the node for proliferation if:

$\left\|\vec{p}_{\textnormal{node}}-\vec{p}_{\textnormal{att}}\right\|_{2}\leq\delta\quad\mathrm{and}\quad\angle\vec{p}_{\textnormal{parent}}\vec{p}_{\textnormal{node}}\vec{p}_{\textnormal{att}}\leq\frac{\gamma}{2}.$
(2)

Let $S_{\textnormal{att}}$ be the set of considered attraction points and
$\vec{v}_{\textnormal{opt}}$ the optimal branching unit vector. For leaf-
nodes, $\vec{v}_{\textnormal{opt}}$ denotes the vector of the parent to the
current node. A new child node is placed in the direction of the elongation
vector $\vec{g}$ at position $\vec{p}_{\textnormal{new}}$, with

$\displaystyle\vec{g}=\textnormal{norm}\left(\omega\vec{v}_{\textnormal{opt}}+(1-\omega)\sum_{\vec{p}_{\textnormal{att}}\in
S_{\textnormal{att}}}\textnormal{norm}(\vec{p}_{\textnormal{att}}-\vec{p}_{\textnormal{node}})\right)$
(3)
$\displaystyle\vec{p}_{\textnormal{new}}=\vec{p}_{\textnormal{node}}+d\vec{g},$

where norm($\cdot$) normalizes the vector by its L2 norm and $\omega$ is a
weighting factor to control the allowed deviation of the average attraction
vector $\vec{a}$ to the hypothetical optimal branching vector
$\vec{v}_{\textnormal{opt}}$. $d$ is the fixed length of the segment (see
section 2.1.4). The radius of the new segment is set to the fixed terminal
vessel radius $r$. This process is called elongation. If the angle of all
attraction vectors is larger than a threshold $\phi$, a bifurcation is
initiated instead and two child nodes are added. We set the radii $r_{c1}$ and
$r_{c2}$ of the two child nodes to the terminal vessel radius $r$. The angles
$\alpha$ and $\beta$ from $\vec{a}$ to the child segments are calculated
following Murray’s law of minimum work [19].

$\alpha=\cos^{-1}\left(\frac{r^{4}_{\textnormal{node}}}{2r^{2}_{\textnormal{node}}r^{2}}\right)=-\beta.$
(4)

The parent vessel segment radius $r_{\textnormal{node}}$ is updated to satisfy

$r^{\kappa}_{\textnormal{node}}=r^{\kappa}_{\textnormal{c1}}+r^{\kappa}_{\textnormal{c2}},$
(5)

where $\kappa$ denotes the bifurcation exponent. This radius adjustment is
recursively repeated up to the root node. The child nodes are placed in the
plane spanned by the leaf node and the line that cuts through the mean of all
attraction points while minimizing the orthogonal distance to them.

Figure 3: Perception volumes for inter-node sprouting (left) and elongation
(right) in 2D. The growth direction for the new node is given by a combination
of the mean attraction vector and the optimal branching vector.

####  2.1.3 Inter-node sprouting

Attraction points are considered by an inter-node if they lay within the
frustum of a sphere with radius $\delta$,

$\displaystyle\phi_{1}+\phi_{2}-\frac{\gamma}{2}\leq$
$\displaystyle\angle\vec{p}_{\textnormal{node}}\vec{p}_{\textnormal{child}}\vec{p}_{\textnormal{att}}\leq\phi_{1}+\phi_{2}+\frac{\gamma}{2},$
(6) and
$\displaystyle\angle\vec{p}_{\textnormal{parent}}\vec{p}_{\textnormal{node}}\vec{p}_{\textnormal{att}}\leq\frac{\gamma}{2}+\phi_{2}.$

Here we extend Rauch and Harders’ definition of the perception volume to
prevent unrealistic branching angles. To find the optimal growth direction, we
look at the set $S_{\vec{v}_{\textnormal{opt}}}$ of hypothetical optimal
branching vectors towards the existing child segment following Murray’s law.
The vectors can be thought of as freely rotating around the child segment at
an angle $\phi_{2}$. The closest optimal branching vector
$\vec{v}_{\textnormal{opt}}\in S_{\vec{v}_{\textnormal{opt}}}$ to the average
attraction vector $\vec{a}$ is used for proliferation. The position of the new
child node is computed following equation 3.

####  2.1.4 Simulation space expansion

To ensure that the vasculature network grows homogeneously in the entire
simulation space while not being limited by its size, we expand the simulation
space as the networks grows. For this, all distance related parameters are
linearly reduced every iteration. Specifically, a parameter $p^{(t)}$ at time
step $t$ is given by dividing the initial value $p^{(0)}$ with the scaling
factor $\sigma^{(t)}=1+t\times\Delta\sigma$. We keep the terminal vessel
radius $r$ fixed to the minimal vessel size, and only increase a vessel’s
diameter following Murray’s law. This prevents the placement of abnormally
large vessel stumps. Additionally, we shrink the segment length parameter $d$
only to a minimal value of 0.04 mm.

####  2.1.5 Simulation of the foveal avascular zone

To recreate the shape of the FAZ, we extend the algorithm with several rules.
To prevent vessel growth inside the FAZ, we avoid the placement of OSs within
a central circular region with radius $r_{\textnormal{faz}}$. As the retinal
vasculature features fewer bifurcations close to the FAZ, we decrease the
chance of bifurcations and inter-node sprouting in regions closer to the
center of the FAZ, and instead perform a simple elongation for leaf nodes. To
achieve a more circular look of the FAZ, we add a rotation term to the
elongation vector $\vec{g}$. We calculate the orthogonal rotation vector
$\vec{v}_{rot}$ by a $None$ rotation of
$\vec{c}=\vec{p}_{\textnormal{center}}-\vec{p}_{\textnormal{node}}$ towards
the direction of the average attraction vector $\vec{a}$. To prevent a
degenerate fully circular growth pattern, we also add a weighted term
$\vec{v}_{\textnormal{out}}$ to the elongation vector that gradually motivates
growth away from the center. The vector is given by the negative center vector
$\vec{c}$. Let $w=\sqrt{r_{\textnormal{rot}}-\left\|\vec{c}\right\|_{2}}$ be
the distance-dependent weighting factor. The position for a new node after
elongation is given by

$\vec{p}_{\textnormal{new}}=\vec{p}_{\textnormal{node}}+d\cdot\textnormal{norm}\left((1-w)\vec{g}+\frac{2w}{3}\vec{v}_{\textnormal{rot}}+\frac{w}{3}\vec{v}_{\textnormal{out}}\right).$
(7)

In reality, we observe varying FAZ shapes and radii. We therefore randomly
choose $r_{\textnormal{faz}}$ from a normal distribution.

Figure 4: Adding a rotation vector $\vec{v}_{\textnormal{rot}}$ and an out
vector $\vec{v}_{\textnormal{out}}$ to the elongation vector leads to a more
realistic growth pattern around the FAZ.

###  2.2 Image and label generation from vascular graphs

After generating the vessel graphs, we voxelize the simulation space, take the
maximum intensity projection along the z-dimension, and compute the anti-
aliased grayscale images. The synthetic vessel annotations $\psi(x)$ are given
by binarizing the images. We note that segmentation maps necessitate a
specific resolution in order to accurately distinguish individual small blood
vessels. We therefore upsample all OCTA images via bilinear interpolation
$\phi(\cdot)$ to a pixel size equal to the minimal vessel radius of
approximately 2.5$\,\mu$m. For instance, 3$\times$3 mm2 images with a
dimension of 304$\times$304 pixels are upsampled to 1216$\times$1216 pixels.
The synthetic segmentation labels are directly generated in the upsampled
simulation space to ensure correct vessel diameter representation.

Figure 5: Structure of our proposed handcrafted noise model to simulate
artifacts and contrast variations of real OCTA images. Given a synthetic
vessel map $I$ and a background vessel map $I_{D}$, the module successively
performs 1) background noise addition, 2) brightness augmentation, and 3)
contrast adaptation. In each block, we use a sparse control point matrix to
generate a field for locally varying contrast.

###  2.3 Domain adaptation of synthetic images

Using the “raw” artificial OCTA images to train a segmentation network results
in poor performance on real data. Although we realistically simulate the
vessel geometry, the contrast and SNR of the images are different. In
practice, OCTA images exhibit artifacts such as noise, low-contrast regions,
and blurry edges. Menten et al. apply shearing, stretching, and binomial noise
artifacts on their training corpus. However, they find that these
augmentations do not significantly boost their segmentation performance. In
the following, we introduce three advanced strategies to adapt the contrast of
training images with the ultimate goal of improving the segmentation
performance on real images.

####  2.3.1 Handcrafted noise model

Our first approach is a handcrafted multistep noise model designed to recreate
the artifacts observed in real data (see figure 5). First, we model the
structured background noise that is caused by sub-resolution capillaries in
the image. We use our vascular simulation to generate a dense background noise
image $I_{D}\in[0,1]^{H\times W}$ of capillaries. We multiply this layer with
an additional Beta distribution noise layer $\Delta$ to locally modulate the
SNR. Depending on the parameters of the Beta distribution, the noise can
exhibit properties of a uniform, normal, or even binary distribution. Instead
of a single noise distribution for the entire image, we use a sparse grid of
control points that govern the noise in the area surrounding them [20]. Let
$A_{\Delta}\in\mathbb{R}_{+}^{9\times 9}$ and
$B_{\Delta}\in\mathbb{R}_{+}^{9\times 9}$ be the control points for the
parameters $\alpha$ and $\beta$ of the beta distribution. We use bicubic
interpolation to compute the pixelwise parameters $\widetilde{A}_{\Delta}$ and
$\widetilde{B}_{\Delta}$. We then sample the modulation factor for each pixel
as
$\Delta^{(i)}\sim\textnormal{Beta}(\widetilde{A}^{(i)}_{\Delta},\widetilde{B}^{(i)}_{\Delta})$.

Second, we use the same procedure to generate a Beta noise matrix $N$ from a
set of control points to model speckle noise and local brightness adjustments.
Third, we generate a field for locally varying contrast. Let
$\Gamma\in[-1,1]^{9\times 9}$ denote the contrast control points and
$\widetilde{\Gamma}=\textnormal{Bicubic}(\Gamma)$ the resulting matrix after
interpolation.

Each transformation step is parameterized by a weighting factor $\lambda$. The
final output $\widetilde{I}$ of the noise model is given by:

$\displaystyle I_{\Delta}$
$\displaystyle=\textnormal{MAX}[I,I_{D}\cdot\lambda_{\Delta}\Delta]$ (8)
$\displaystyle I_{N}$
$\displaystyle=I_{\Delta}\cdot\left((1-\lambda_{N})N+\lambda_{N}\right)$
$\displaystyle\widetilde{I}$
$\displaystyle=(I_{N})^{(\lambda_{\Gamma}\widetilde{\Gamma}+1)}.$

To simulate blurry edges and lower quality images, we apply random down-
sampling by a factor $s\in[0.25,1]$ and a successive up-sampling to the
original size.

####  2.3.2 Noise modeling via adversarial training

The contrast adaptation of synthetic training images aims to ultimately
improve the performance of the segmentation network. The random selection of
control point values in the previously described approach does not prioritize
difficult augmentations and is inefficient. To this end, we explore an
adversarial training (AT) approach, where an intensity transformation is
computed for each image that maximizes the segmentation loss given the
training label. To prevent pixelwise attacks that could alter the alignment of
images and labels, we adopt a strategy proposed by Chen et al. [20]. In order
to segment MR images in a scarce data environment, their work focuses on
augmenting training samples by applying adversarially tuned magnetic resonance
bias fields. They propose to only tune a small set of control points via AT
that are then used to compute the transform. Integrating this into our noise
model, we turn the control point values of each layer into tunable parameters.
Since sampling from a Beta noise distribution is not differentiable, we make
use of a reparameterization trick to facilitate the use of iterative projected
gradient ascent to optimize the $\alpha$ and $\beta$ values.

####  2.3.3 Generative adversarial training

Figure 6: Our proposed framework uses a generator network to transform
synthetic vessel maps into realistic OCTA images. An adversarial discriminator
network is trained to differentiate real images from generated ones. A
segmentation network is trained to predict the vessel segmentation from the
generated image.  Figure 7: Our OCTA image synthesis pipeline using GAN
augmentation is able to produce highly realistic training samples that closely
match the geometry of the retinal vasculature and contrast of real OCTA
images.

Our third strategy for overcoming the domain gap is to replace the
heuristically designed noise model with a learned transformation by a neural
network. We formulate this task as an image-to-image style translation
problem, where a generator network transforms a synthetic image from our
simulation model to match the distribution and style of real images. An
adversarial discriminator network judges the realism of the produced
translation. Since the generator is unconstrained in how much it transforms
the image, there is no guarantee that the original image-to-label alignment is
preserved. To this end, Zhu et al. propose the CycleGAN framework to enable
unpaired image-to-image translation [21]. A second pair of generator and
discriminator is used to transform the generated image back to source domain.
A cycle-consistency loss term computes the difference between the input and
the recovered image, encouraging the preservation of image content. CycleGAN
has shown remarkable results in domain adaptation and has been used widely in
medical settings [22, 23, 24].

In our approach, we replace the reverse generator with a segmentation network.
The cycle consistency loss is directly computed using the segmentation maps
that are obtained from the vascular graphs (see figure 6). We find that this
loss function is a strong regularizer for the generator and that we do not
require a second discriminator. As commonly used in image-to-image
translation, we use a ResNet generator with 9 residual blocks and a $70\times
70$ PatchGAN discriminator [21, 25]. To further constrain the generator
network, and to enable the segmentation network to handle real data, an
identity loss term is added. To this end, the segmentation network predicts
the vessel map of real images after being processed by the generator. We
compare this output with the predicted segmentation of the original real image
and penalize differences. To save computation time, we perform the style
transfer step of the generator on the original image size and only upsample
the images before segmentation. Furthermore, non-healthy samples in real data
sometimes exhibit low- or non-perfusion areas. To model these, we randomly
remove edges and all their descendants with probability $U(0,0.02)$ from the
source-label pair. Integrating these samples into our training improves the
segmentation performance on OCTA images depicting diseased eyes.

##  3 Experiments

The ultimate aim of our work is to generate synthetic OCTA training images
that can be used to train neural networks for segmentation of blood vessels in
real OCTA images without human annotations. We now describe the evaluation
procedure to compare the segmentation performance of our method with state-of-
the-art baseline methods on multiple OCTA datasets.

###  3.1 OCTA datasets

We use three public datasets containing OCTA images and matching blood vessel
annotations (see table 2). For all datasets, we only consider the en-face
projections of macular OCTA images. We exclusively work with images of
3$\times$3mm2 and 304$\times$304 pixels, as they cover a sufficiently large
area for analysis while maintaining a high image resolution.

Table 2: Summary of all three evaluation datasets: OCTA-500 focuses on major blood vessels, while ROSE-1 provides more detailed annotations. ROSE-1, however, includes small and faint vessels inconsistently and may ignore larger ones. In some low-contrast regions, no vessels are segmented at all. Giarratano et al.’s dataset offers the most detailed annotations. Name | Dimensions | No. of samples | Label detail
---|---|---|---
OCTA-500 [8] | 304$\times$304 | 200 | coarse
ROSE-1 [7] | 304$\times$304 | 40 | medium
Giarratano et al. [6] | 91$\times$91 | 55 | detailed

###  3.2 Alignment of synthetic and provided vessel labels

The level of detail in the segmentation labels varies across the presented
datasets, which presents a challenge. This is because any segmentation
predictions that surpass the level of detail provided in the human annotations
would be penalized during quantitative evaluation. To correctly evaluate the
performance of a model on each dataset, we need to condition the model to
produce a prediction matching the required level of detail. We observe that
the datasets’ segmentation labels roughly correlate with the vessel diameter.
We therefore filter our synthetic segmentation labels by vessel radius to
match the individual dataset labels. For instance, for the OCTA-500 dataset,
we only include vessels with $r>10\,\mu$m in the segmentation map. We estimate
the thresholds by manually analyzing a few samples from each set.

###  3.3 Tested segmentation methods

We first benchmark a neural network trained directly on the small set of
annotated real images and two established computer vision algorithms for
vessel segmentation. We then compare these baselines with the transfer
learning approach of training a network on synthetic OCTA images using the
synthesis procedure proposed by Menten et al., and training a network on our
images. We evaluate four variants of our method, to assess the performance
gains of using each of our three proposed contrast adaptation strategies.

####  3.3.1 Supervised machine learning baselines

For all of our machine learning experiments, we choose a variation of the
widely successful U-Net architecture as our network [26]. We follow the nnU-
Net guidelines proposed by Isensee et al. to select the image preprocessing
steps, network architecture, and network training strategy [27].

We first train a supervised baseline using the provided labels for each
dataset. We apply random rotations by $k\times$$\pm$$,k\in\\{0,1,2,3\\}$,
flipping, and elastic deformation as data augmentations. The network is
trained for 60 epochs with a batch size of 4 and the latest checkpoint is
selected for testing.

####  3.3.2 Traditional computer vision baselines

Frangi filters are a commonly used method to enhance the visibility of tubular
structures in images. We follow the recommendations by Giarratano et al. and
select the filter scales $\sigma\in[0.5,1,1.5,2]$, the Frangi correction
constant for plate-like structures $\alpha=1$ and for blob-like structures
$\beta=15$. The enhanced image is binarized using thresholding while small
objects are removed from the segmentation map. We adjust the threshold to
align the prediction with the label map of each dataset. We use the Bayesian
Optimization HyperBand (BOHB) search algorithm with 20 concurrent actors and
five iterations to find the optimal threshold and minimal object size for each
training set [28].

We also test a 2D implementation of the optimal oriented flux (OOF) filter,
which seems to be especially well suited for dense segmentation tasks [29]. We
again use BOHB to find the optimal threshold and minimal object size.

####  3.3.3 Training with synthetic OCTA images by Menten et al.

We also train a neural network on 250 synthetic OCTA images generated using
the method by Menten et al.. We apply random rotation and flipping as
augmentations, train for 30 epochs, and select the latest checkpoint for
testing. To prevent any performance drops on the dataset by Giarratano et al.
caused by differences of image size during testing, we crop the training
samples from the models trained on synthetic images to the same number of
pixels.

####  3.3.4 Training with our synthetic OCTA images

We now describe the training procedure for our proposed methods. We first
train a baseline model using our synthetic images without any of the proposed
augmentation strategies (S). The network is trained on 500 synthetic images,
and we use the same training procedure as for training on the images by Menten
et al. In a second setup, we augment the training samples using our proposed
handcrafted noise model with random control point values within a
heuristically chosen interval (S+RA). We set $\lambda_{\Delta}=1$,
$\lambda_{N}=0.7$, and $\lambda_{\Gamma}=0.3$. Next, we test our adversarial
training approach to optimize the control points (S+AT). We set the
adversarial learning rate to $\alpha=0.2$ and use three adversarial
optimization steps. Finally, we test our image noise model based on GAN
augmentation (S+GAN). We first train the generator network with complete label
maps for 50 epochs and then use the trained generator to augment synthetic
images. Afterward, we train a separate U-Net on these transformed images and
segmentation maps with the respective level of detail.

###  3.4 Evaluation

We perform a 5-fold cross-validation for all methods. For Frangi and OOF, we
use the training set of each fold to find the best threshold and minimal
object size. For U-Nets trained on synthetic data, we train five separate
models and validate them on the test split. The obtained predictions are
quantitatively compared to the provided annotations using six different
metrics. We calculate the mean Dice similarity coefficient (DSC),
centerlineDice (clDice) [30], area under the receiver operating characteristic
curve (AUC), accuracy, recall, and precision with their respective standard
errors. However, we note that while the quantitative comparison does provide
some insights into a network’s predictions, the results need to be taken with
a grain of salt. The provided annotations are often imperfect and therefore do
not allow to accurately assess the quality of more detailed segmentation maps
obtained by the automated methods.

##  4 Results

Figure 7 shows a comparison of synthetic images generated by Menten et al.,
real images, and our method. The new framework allows intuitive tuning of
parameters and straightforward formulation of additional constraints to create
more realistic images. GAN augmentation further reduces the domain gap between
synthetic images and the target dataset. Using a Python implementation with
k-d trees for efficient queries based on coordinates, we are able to generate
images with more than 10,000 nodes in less than a single minute, compared to
the eight hours required by Menten et al.’s simulation.

Figure 8: Qualitative comparison of all eight tested segmentation methods on a
representative example from each of the three considered datasets. Our methods
with additional contrast adaptations are reliable in extracting realistic
segmentation maps across all datasets. Figure 9: By controlling the minimal
vessel radius in the synthetic labels, the network can be conditioned to
create segmentations with varying levels of detail. This enables benchmarks on
datasets with different label predictions. Figure 10: Performance of our and
the supervised method on a DR patient. Our approach is superior in extracting
detailed vessel maps in low contrast areas, such as around the FAZ. Table 3:
5-fold cross-validation segmentation performance of all methods on the three
evaluation datasets. The best (underline + bold) and the second best (bold)
results are marked. Our methods outperform traditional computer vision
algorithms.

OCTA-500 Model Trained on Augmentation DSC clDice AUC ACC Recall Precision
Supervised U-Net Real Random 0.912$\pm$0.001 0.940$\pm$0.002 0.950$\pm$0.001
0.984$\pm$0.000 0.910$\pm$0.003 0.916$\pm$0.003 Traditional Frangi - -
0.807$\pm$0.003 0.848$\pm$0.004 0.895$\pm$0.002 0.975$\pm$0.000
0.802$\pm$0.005 0.820$\pm$0.003 OOF 0.734$\pm$0.004 0.785$\pm$0.003
0.851$\pm$0.004 0.966$\pm$0.001 0.719$\pm$0.009 0.760$\pm$0.006 Menten et al.
U-Net Synthetic Random 0.594$\pm$0.011 0.536$\pm$0.014 0.936$\pm$0.002
0.913$\pm$0.004 0.963$\pm$0.001 0.434$\pm$0.011 Ours U-Net Synthetic -
0.734$\pm$0.006 0.731$\pm$0.009 0.827$\pm$0.005 0.968$\pm$0.001
0.663$\pm$0.010 0.827$\pm$0.003 Random 0.849$\pm$0.001 0.892$\pm$0.002
0.926$\pm$0.002 0.971$\pm$0.000 0.871$\pm$0.005 0.831$\pm$0.004 Adversarial
0.840$\pm$0.003 0.881$\pm$0.002 0.915$\pm$0.003 0.970$\pm$0.000
0.848$\pm$0.006 0.837$\pm$0.004 GAN 0.840$\pm$0.005 0.876$\pm$0.006
0.907$\pm$0.002 0.971$\pm$0.001 0.829$\pm$0.005 0.855$\pm$0.010 ROSE-1 Model
Trained on Augmentation DSC clDice AUC ACC Recall Precision Supervised U-Net
Real Random 0.717$\pm$0.004 0.700$\pm$0.006 0.799$\pm$0.003 0.843$\pm$0.003
0.678$\pm$0.006 0.774$\pm$0.005 Traditional Frangi - - 0.670$\pm$0.003
0.601$\pm$0.007 0.789$\pm$0.003 0.875$\pm$0.002 0.644$\pm$0.009
0.704$\pm$0.012 OOF 0.594$\pm$0.003 0.559$\pm$0.003 0.752$\pm$0.004
0.831$\pm$0.006 0.623$\pm$0.016 0.578$\pm$0.018 Menten et al. U-Net Synthetic
Random 0.606$\pm$0.004 0.534$\pm$0.005 0.746$\pm$0.003 0.861$\pm$0.001
0.556$\pm$0.008 0.673$\pm$0.004 Ours U-Net Synthetic - 0.613$\pm$0.004
0.546$\pm$0.004 0.743$\pm$0.002 0.870$\pm$0.002 0.532$\pm$0.003
0.731$\pm$0.006 Random 0.649$\pm$0.001 0.653$\pm$0.002 0.754$\pm$0.001
0.806$\pm$0.001 0.617$\pm$0.002 0.699$\pm$0.002 Adversarial 0.665$\pm$0.004
0.658$\pm$0.005 0.756$\pm$0.003 0.816$\pm$0.003 0.601$\pm$0.006
0.734$\pm$0.002 GAN 0.637$\pm$0.005 0.639$\pm$0.007 0.745$\pm$0.003
0.804$\pm$0.004 0.591$\pm$0.005 0.705$\pm$0.005 Giarratano et al. Model
Trained on Augmentation DSC clDice AUC ACC Recall Precision Supervised U-Net
Real Random 0.907$\pm$0.002 0.954$\pm$0.003 0.887$\pm$0.004 0.895$\pm$0.002
0.925$\pm$0.003 0.981$\pm$0.006 Traditional Frangi - - 0.769$\pm$0.009
0.833$\pm$0.007 0.815$\pm$0.002 0.797$\pm$0.003 0.895$\pm$0.010
0.683$\pm$0.012 OOF 0.812$\pm$0.005 0.848$\pm$0.006 0.851$\pm$0.003
0.854$\pm$0.002 0.808$\pm$0.017 0.827$\pm$0.019 Menten et al. U-Net Synthetic
Random 0.471$\pm$0.014 0.496$\pm$0.012 0.651$\pm$0.017 0.710$\pm$0.011
0.318$\pm$0.014 0.937$\pm$0.004 Ours U-Net Synthetic - 0.781$\pm$0.006
0.808$\pm$0.008 0.827$\pm$0.009 0.812$\pm$0.003 0.887$\pm$0.003
0.710$\pm$0.012 Random 0.850$\pm$0.005 0.933$\pm$0.002 0.846$\pm$0.001
0.839$\pm$0.001 0.829$\pm$0.003 0.879$\pm$0.003 Adversarial 0.842$\pm$0.005
0.931$\pm$0.008 0.840$\pm$0.004 0.833$\pm$0.006 0.814$\pm$0.004
0.881$\pm$0.010 GAN 0.834$\pm$0.002 0.902$\pm$0.004 0.829$\pm$0.002
0.822$\pm$0.002 0.828$\pm$0.004 0.853$\pm$0.006

Table 3 lists the segmentation performance of all algorithms on the three OCTA
datasets. The supervised model performs best quantitatively on all datasets
given the manually annotated labels. Notably, the performance on the ROSE-1
dataset is relatively low for all methods. The predicted label maps often
contain too many or too few annotations. All models struggle to decide which
vessels need to be segmented and which can be ignored. Of the traditional
computer vision methods, the Frangi filter performs best on larger vessel
sizes, while OOF excels on dense segmentation. The training samples from
Menten et al. only offer a single level of segmentation detail, which does not
align well with the evaluation datasets. Limited complexity in the simulated
vessel graphs and inferior contrast adaptation prevent a detailed segmentation
on the dataset by Giarratano et al.. While the method does achieve a high
recall for large vessel and a high precision on the dataset by Giarratano et
al., it generally underperforms in all experiments with regard to the Dice
score.

Training on our synthetic training samples without contrast adaptations
performs similar to Frangi and OOF on all datasets. Figure 9 shows that the
pipeline can be adjusted to align the level of detail with the respective
dataset labels. Adding any of our proposed contrast adaptation strategies
boosts the performance notably and enables the network to outperform
traditional computer vision algorithms. We find that our proposed noise model
makes the segmentation network more robust against common OCTA artifacts. The
trained network is able to identify large and medium-sized vessels in noisy
and low-contrast regions, and smaller vessels in high-contrast areas. We did
not measure a quantitative benefit of the adversarial or the GAN-based method
on the considered OCTA datasets.

A qualitative comparison for detailed segmentation in figure 10 shows that
existing methods are not able to extract realistic label maps. For clinical
utility it is especially important to be able to reliably segment vessels for
out of distribution cases, such as patients with disease. The supervised model
trained on the dataset by Giarratano et al. does not recognize vessels with
lower visibility around the FAZ. In contrast, our proposed method is much more
robust, even for progressed DR patients.

We hypothesize that this discrepancy between quantitative and qualitative
performance stems from the quality of the available annotations that are used
as ground truth. Any deviation from the label map is penalized, even if the
predicted segmentation map includes more details.

##  5 Discussion and Conclusion

In this work, we proposed a method to generate highly realistic synthetic OCTA
images to train a CNN for vessel segmentation in real images. We substantially
surpass the realism of the generated vascular networks proposed by Menten et
al. Our new simulation based on space colonization creates more detailed
vascular graphs in less than one minute. We proposed three contrast adaptation
strategies to minimize the domain gap between synthetic and real images, with
the goal of improving the robustness of the trained network. Using these, we
are able to extract detailed vessel segmentation maps from OCTA en-face images
without any human annotations. In experiments on three public datasets, we
outperform established computer vision algorithms.

Differences in the degree of detail for provided annotations in existing
validation datasets require aligning the level of segmentation detail of the
training labels with those of the respective validation dataset. To this end,
we proposed to filter vessels in the label map by diameter. However, the
manual annotations do not always correlate with vessel diameter and
inconsistencies in labeling impede the alignment of synthetic annotations.
Poorly visible vessels, even if they were relatively large in diameter, were
often ignored by the annotators. Therefore, quantitative comparisons with
existing datasets labels do not allow to accurately assess the quality of
detailed segmentation maps. Any deviations from the human annotations, even if
correct, are penalized.

Because of this limited expressiveness, we draw qualitative comparisons on
dense segmentation for unseen data. While the OOF filter achieves satisfying
performance for areas with good image quality, it struggles to connect vessel
segments in low-contrast regions. The low amount of training data limits the
performance of the supervised model to recognize vessel with low visibility.
This becomes especially pronounced around the FAZ and in progressed DR
patients. In contrast, our approach is especially well suited to extract
realistically detailed segmentations and surpasses every other method on
unseen samples from the OCTA-500 dataset.

Future work should concentrate on exploring the usefulness of extracted
biomarkers for downstream tasks. Taking the example of DR classification, one
could compare the performance between direct classification from the image, or
based on classification using extracted biomarkers. By taking the extra step
of determining biomarkers, we provide clinicians with more interpretability of
the machine learning pipeline. It is also possible to extend our framework to
3D OCTA volumes by adjusting the simulation model to realistically recreate
the retinal anatomy in 3D. By controlling the placement of the OSs and vessel
growth hyperparameters in the simulation model, it is possible to simulate
different vascular layers. This would provide segmentation maps with
additional depth information, enabling more accurate analysis.

We make our entire pipeline, pretrained models, as well as a large dataset of
synthetic OCTA images publicly available at https://github.com/TUM-AIMED/OCTA-
seg. Our tool allows the automated extraction of detailed segmentation maps
and enables the computation of quantitative biomarkers. By this, we hope to
make a valuable contribution to the ophthalmology community and motivate more
research in automated OCTA analysis.

## References

  * [1] J. Chua et al., “Optical coherence tomography angiography in diabetes and diabetic retinopathy,” _Journal of Clinical Medicine_ , vol. 9, no. 6, 2020.
  * [2] J. Wang, J. Jiang, Y. Zhang, Y. W. Qian, J. F. Zhang, and Z. L. Wang, “Retinal and choroidal vascular changes in coronary heart disease: an optical coherence tomography angiography study,” _Biomedical Optics Express_ , vol. 10, no. 4, p. 1532, 2019.
  * [3] L. Aly  et al., “Dynamics of retinal vessel loss after acute optic neuritis in patients with relapsing multiple sclerosis,” _Neurology - Neuroimmunology Neuroinflammation_ , vol. 9, no. 3, 2022.
  * [4] I. Yasser, F. Khalifa, H. Abdeltawab, M. Ghazal, H. S. Sandhu, and A. El-Baz, “Automated diagnosis of optical coherence tomography angiography (OCTA) based on machine learning techniques,” _Sensors_ , vol. 22, no. 6, 2022.
  * [5] K. M. Meiburger, M. Salvi, G. Rotunno, W. Drexler, and M. Liu, “Automatic segmentation and classification methods using optical coherence tomography angiography (OCTA): A review and handbook,” _Applied Sciences_ , vol. 11, no. 20, p. 9734, 2021.
  * [6] Y. Giarratano et al., “Automated segmentation of optical coherence tomography angiography images: Benchmark data and clinically relevant metrics,” _Translational vision science & technology_, vol. 9, no. 13, p. 5, 2020.
  * [7] Y. Ma et al., “Rose: A retinal oct-angiography vessel segmentation dataset and new model,” _IEEE transactions on medical imaging_ , vol. 40, no. 3, pp. 928–939, 2021.
  * [8] M. Li et al., “IPN-V2 and OCTA-500: Methodology and dataset for retinal image segmentation,” 2020, arXiv:2012.07261v3.
  * [9] P. Costa et al., “End-to-end adversarial retinal image synthesis,” _IEEE transactions on medical imaging_ , vol. 37, no. 3, pp. 781–791, 2018\.
  * [10] W. Fu et al., “Form follows function: Smart network design enables zero-shot network reuse,” in _Bildverarbeitung für die Medizin 2022_. Springer Fachmedien Wiesbaden, 2022, pp. 121–126.
  * [11] M. J. Menten, J. C. Paetzold, A. Dima, B. H. Menze, B. Knier, and D. Rueckert, “Physiology-based simulation of the retinal vasculature enables annotation-free segmentation of oct angiographs,” in _MICCAI 2022_ , ser. Lecture Notes in Computer Science. Springer, 2022, vol. 13438, pp. 330–340.
  * [12] N. Rauch and M. Harders, “Interactive Synthesis of 3D Geometries of Blood Vessels,” in _Eurographics 2021 - Short Papers_. The Eurographics Association, 2021.
  * [13] J. Campbell et al., “Detailed vascular anatomy of the human retina by projection-resolved optical coherence tomography angiography,” _Scientific reports_ , vol. 7, p. 42201, 2017.
  * [14] M. Schneider, J. Reichold, B. Weber, G. Székely, and S. Hirsch, “Tissue metabolism driven arterial tree generation,” _Medical image analysis_ , vol. 16, no. 7, pp. 1397–1414, 2012.
  * [15] F. Nekka, S. Kyriacos, C. Kerrigan, and L. Cartilier, “A model of growing vascular structures,” _Bulletin of mathematical biology_ , vol. 58, no. 3, pp. 409–424, 1996.
  * [16] G. D. M. Talou, S. Safaei, P. J. Hunter, and P. J. Blanco, “Adaptive constrained constructive optimisation for complex vascularisation processes,” _Scientific Reports_ , vol. 11, no. 1, p. 6180, 2021.
  * [17] S. Gerl et al., “A distance-based loss for smooth and continuous skin layer segmentation in optoacoustic images,” in _MICCAI 2020_. Springer, 2020, pp. 309–319.
  * [18] M. Todorov et al., “Machine learning analysis of whole mouse brain vasculature,” _Nature methods_ , vol. 17, no. 4, pp. 442–449, 2020.
  * [19] C. D. Murray, “The physiological principle of minimum work: I. the vascular system and the cost of blood volume,” _Proceedings of the National Academy of Sciences of the United States of America_ , vol. 12, no. 3, pp. 207–214, 1926.
  * [20] C. Chen et al., “Realistic adversarial data augmentation for MR image segmentation,” in _MICCAI 2020_. Springer, 2020, pp. 667–677.
  * [21] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired image-to-image translation using cycle-consistent adversarial networks,” in _2017 IEEE International Conference on Computer Vision (ICCV)_. IEEE, 2017, pp. 2242–2251.
  * [22] M. Hammami, D. Friboulet, and R. Kechichian, “Cycle GAN-based data augmentation for multi-organ detection in CT images via Yolo,” in _2020 IEEE International Conference on Image Processing (ICIP)_ , 2020, pp. 390–393.
  * [23] S.-W. Huang, C.-T. Lin, S.-P. Chen, Y.-Y. Wu, P.-H. Hsu, and S.-H. Lai, “AugGAN: Cross domain adaptation with GAN-based data augmentation,” in _Computer Vision - ECCV 2018_. Springer International Publishing, 2018, pp. 731–744.
  * [24] V. Sandfort, K. Yan, P. J. Pickhardt, and R. M. Summers, “Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks,” _Scientific Reports_ , vol. 9, no. 1, p. 16884, 2019.
  * [25] T. Park, A. A. Efros, R. Zhang, and J.-Y. Zhu, “Contrastive learning for unpaired image-to-image translation,” in _Computer Vision - ECCV 2020_ , ser. Lecture Notes in Computer Science. Springer, 2020, vol. 12354, pp. 319–345.
  * [26] O. Ronneberger, P. Fischer, and T. Brox, “U-Net: Convolutional networks for biomedical image segmentation,” in _MICCAI 2015_. Springer International Publishing, 2015, pp. 234–241.
  * [27] F. Isensee, P. F. Jaeger, S. A. A. Kohl, J. Petersen, and K. H. Maier-Hein, “nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation,” _Nature methods_ , vol. 18, no. 2, pp. 203–211, 2021\.
  * [28] S. Falkner, A. Klein, and F. Hutter, “BOHB: Robust and efficient hyperparameter optimization at scale,” in _Proceedings of the 35th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, vol. 80. PMLR, 2018, pp. 1437–1446.
  * [29] M. W. K. Law and A. C. S. Chung, “Three dimensional curvilinear structure detection using optimally oriented flux,” in _Computer Vision – ECCV 2008_ , ser. Lecture Notes in Computer Science. Springer, 2008, vol. 5305, pp. 368–382.
  * [30] S. Shit et al., “clDice - A novel topology-preserving loss function for tubular structure segmentation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2021, pp. 16 560–16 569.


follows that $((W,N^{\prime}),w)$ is a model of that diagram too. Hence
$i_{P,((W,M),w)}((W,N^{\prime}),w))$ is a homomorphism
$((W,M),w)\rightarrow((W,N),w)$ where $((W,N),w)$ is the $P$-reduct of
$((W,N^{\prime}),w)$. Let us denote it by $h$. Then by the homomorphism
property of $h$ we have that $M^{v}\subseteq N^{h(v)}$. But $M^{v}=P$ and
$N^{w}=N^{v}=\emptyset$, hence $M^{v}\not\subseteq N^{h(v)}$. This
contradiction invalidates our supposition of the existence of diagrams.

∎

##  6\. Conclusions

In this paper we have introduced a new technique for representing stratified
institutions by a decomposition at the level of the models. Then we applied
this decomposition technique for developing general results on the existence
of model amalgamation and of diagrams in stratified institutions. In the
latter case it has emerged that some nominals infrastructure is needed. This
is hardly surprising because fundamentally diagrams reflect a fine balance
between syntax and semantics (i.e. model homomorphisms represented as models
of theories) and the presence of nominals restore such a balance for Kripke
semnantics.

### Future work

The potential of our decomposition technique should be further explored along
the following directions:

  1. (1)

Quasi-varieties and initial semantics in stratified institutions. These have
been studied for the particular half-abstract case of hybridised institutions
in [8]. However by the decomposition technique we should be able to do those
at the higher level of generality of abstract stratified institutions, one of
the consequences being a wider class of concrete applications.

  2. (2)

Develop some general results supporting the existence of important structures
in stratified institutional model theory that in the current literature have
an “assumed” status. An example is that of filtered products of models [9].

  3. (3)

Generate new interesting examples of stratified institutions that break from
the modal logics tradition. In this respect pragmatic motivations may come
from computing science which has many areas whose foundations involve some
form of models with states. In those situations there is usually an almost
automatic reliance on modal logics in their more or less conventional
acceptations, although those have not been developed for those specific
purposes, but rather for pure logic interests. Stratified institutions and
their decomposition technique has the potential to offer a powerful
theoretical tool for going beyond modal logics by defining model theoretic
frameworks that are finer tuned to respective concrete applications.

### Acknowledgement

This work was supported by a grant of the Romanian Ministry of Education and
Research, CNCS – UEFISCDI, project number PN-III-P4-ID-PCE-2020-0446, within
PNCDI III.

## References

  * [1] Marc Aiguier and Isabelle Bloch.  Logical dual concepts based on mathematical morphology in stratified institutions: applications to spatial reasoning.  Journal of Applied Non-Classical Logics, 29(4):392–429, 2019.
  * [2] Marc Aiguier and Răzvan Diaconescu.  Stratified institutions and elementary homomorphisms.  Information Processing Letters, 103(1):5–13, 2007.
  * [3] M. Arrais and José L. Fiadeiro.  Unifying theories in different institutions.  In Magne Haveraaen, Olaf Owe, and Ole-Johan Dahl, editors, Recent Trends in Data Type Specification, volume 1130 of Lecture Notes in Computer Science, pages 81–101. Springer, 1996.
  * [4] Patrick Blackburn.  Representation, reasoning, and relational structures: a hybrid logic manifesto.  Logic Journal of IGPL, 8(3):339–365, 2000.
  * [5] Patrick Blackburn, Maarten de Rijke, and Yde Venema.  Modal Logic.  Cambridge University Press, 2001.
  * [6] Răzvan Diaconescu.  Elementary diagrams in institutions.  Journal of Logic and Computation, 14(5):651–674, 2004.
  * [7] Răzvan Diaconescu.  Institution-independent Model Theory.  Birkhäuser, 2008.
  * [8] Răzvan Diaconescu.  Quasi-varieties and initial semantics in hybridized institutions.  Journal of Logic and Computation, 26(3):855–891, 2016.
  * [9] Răzvan Diaconescu.  Implicit Kripke semantics and ultraproducts in stratified institutions.  Journal of Logic and Computation, 27(5):1577–1606, 2017.
  * [10] Răzvan Diaconescu.  Introducing H, an institution-based formal specification and verification language.  Logica Universalis, 14(2):259–277, 2020.
  * [11] Răzvan Diaconescu and Ionuţ Ţuţu.  On the algebra of structured specifications.  Theoretical Computer Science, 412(28):3145–3174, 2011.
  * [12] Răzvan Diaconescu, Joseph Goguen, and Petros Stefaneas.  Logical support for modularisation.  In Gerard Huet and Gordon Plotkin, editors, Logical Environments, pages 83–130. Cambridge, 1993.  Proceedings of a Workshop held in Edinburgh, Scotland, May 1991.
  * [13] Răzvan Diaconescu and Alexandre Madeira.  Encoding hybridized institutions into first order logic.  Mathematical Structures in Computer Science, 26:745–788, 2016.
  * [14] Răzvan Diaconescu and Petros Stefaneas.  Modality in open institutions with concrete syntax.  Bulletin of the Greek Mathematical Societ, 49:91–101, 2004.  Previously published as JAIST Tech Report IS-RR-97-0046, 1997.
  * [15] Răzvan Diaconescu and Petros Stefaneas.  Ultraproducts and possible worlds semantics in institutions.  Theoretical Computer Science, 379(1):210–230, 2007.
  * [16] Melvin Fitting and Richard L. Mendelsohn.  First-order Modal Logic.  Kluwer/Springer, 1998.
  * [17] Dov M. Gabbay, A. Kurucz, F. Wolter, and M. Zakharyaschev.  Many-dimensional modal logics: theory and applications. Elsevier, 2003.
  * [18] Joseph Goguen and Rod Burstall.  Institutions: Abstract model theory for specification and programming.  Journal of the Association for Computing Machinery, 39(1):95–146, 1992.
  * [19] Joseph Goguen and Grigore Roşu.  Institution morphisms.  Formal Aspects of Computing, 13:274–307, 2002.
  * [20] Valentin Goranko.  Hierarchies of modal and temporal logics with reference pointers.  Journal of Logic, Language and Information, 5(1):1–24, 1996.
  * [21] Daniel Găină.  Foundations of logic programming in hybridised logics.  In Codescu M., Diaconescu R., and Ţuţu I., editors, Recent Trends in Algebraic Development Techniques. WADT 2015, volume 9463, pages 69–89. Springer, Cham, 2015.
  * [22] Daniel Găină.  Forcing and calculi for hybrid logics.  Journal of ACM, 67(4):25:1–25:55, 2020.
  * [23] Saunders Mac Lane.  Categories for the Working Mathematician.  Springer, second edition, 1998.
  * [24] Alexandre Madeira.  Foundations and techniques for software reconfigurability.  PhD thesis, Universidades do Minho, Aveiro and Porto (Joint MAP-i Doctoral Programme), 2014.
  * [25] Manuel-Antonio Martins, Alexandre Madeira, Răzvan Diaconescu, and Luis Barbosa.  Hybridization of institutions.  In Andrea Corradini, Bartek Klin, and Corina Cîrstea, editors, Algebra and Coalgebra in Computer Science, volume 6859 of Lecture Notes in Computer Science, pages 283–297. Springer, 2011.
  * [26] José Meseguer.  General logics.  In H.-D. Ebbinghaus et al., editors, Proceedings, Logic Colloquium, 1987, pages 275–329. North-Holland, 1989.
  * [27] Till Mossakowski.  Different types of arrow between logical frameworks.  In F. Meyer auf der Heide and B. Monien, editors, Proc. ICALP 96, volume 1099 of Lecture Notes in Computer Science, pages 158–169. Springer Verlag, 1996.
  * [28] Arthur N. Prior.  Past, Present and Future.  Oxford University Press, 1967.
  * [29] Donald Sannella and Andrzej Tarlecki.  Specifications in an arbitrary institution.  Information and Control, 76:165–210, 1988.
  * [30] Donald Sannella and Andrzej Tarlecki.  Foundations of Algebraic Specifications and Formal Software Development.  Springer, 2012.
  * [31] Andrzej Tarlecki.  On the existence of free models in abstract algebraic institutions.  Theoretical Computer Science, 37:269–304, 1986.
  * [32] Andrzej Tarlecki.  Quasi-varieties in abstract algebraic institutions.  Journal of Computer and System Sciences, 33(3):333–360, 1986.
  * [33] Andrzej Tarlecki.  Moving between logical systems.  In Magne Haveraaen, Olaf Owe, and Ole-Johan Dahl, editors, Recent Trends in Data Type Specification, volume 1130 of Lecture Notes in Computer Science, pages 478–502. Springer, 1996.
  * [34] Andrzej Tarlecki.  Towards heterogeneous specifications.  In D. Gabbay and M. van Rijke, editors, Proceedings, International Conference on Frontiers of Combining Systems (FroCoS’98), pages 337–360. Research Studies Press, 2000.


# Improving astroBERT using Semantic Textual Similarity

Felix Grezes1, Thomas Allen, Sergi Blanco-Cuaresma, Alberto Accomazzi, Michael
J. Kurtz, Golnaz Shapurian, Edwin Henneken, Carolyn S. Grant, Donna M.
Thompson, Timothy W. Hostetler, Matthew R. Templeton, Kelly E. Lockhart,
Shinyi Chen, Jennifer Koch, Taylor Jacovich, and Pavlos Protopapas

###### Abstract

The NASA Astrophysics Data System (ADS) is an essential tool for researchers
that allows them to explore the astronomy and astrophysics scientific
literature, but it has yet to exploit recent advances in natural language
processing. At ADASS 2021, we introduced astroBERT, a machine learning
language model tailored to the text used in astronomy papers in ADS.
In this work we:
1. announce the first public release of the astroBERT language model; 2. show how astroBERT improves over existing public language models on astrophysics specific tasks; 3. and detail how ADS plans to harness the unique structure of scientific papers, the citation graph and citation context, to further improve astroBERT.

1Harvard-Smithsonian Center for Astrophysics, Cambridge, MA, USA;
<EMAIL_ADDRESS>
##  1 Introduction

The NASA Astrophysics Data System111https://ui.adsabs.harvard.edu (ADS) has
been an essential tool for astrophysicists since it was announced at ADASS II
by Kurtz et al. (1993). With a bibliographic collection of over 15 million
records, discoverability (of authors, missions, telescopes, etc …) can be
challenging even for the most seasoned ADS user, despite the help of the
existing search tools within ADS (filters, Boolean operators, citation graph
and more). A particular difficulty stems from the ambiguity of language e.g.
tagging papers with results from the "Planck mission" currently requires
manually differentiation from the person, the institute, the constant with the
same name. Furthermore new entities are constantly being named, and need to be
searchable. Introduced at ADASS XXXI by Grezes et al. (2021), we publicly
release an improved version of astroBERT, a language model tailored to
astrophysics. We show that astroBERT outperforms the existing public languages
BERT (Devlin et al. 2018) and SciBERT (Beltagy et al. 2019) on the task of
detecting entities of interest in astrophysics literature. Finally we detail
our plans to improve astroBERT using information unique to scientific
literature, the citation graph.

##  2 astroBERT Pretraining

We trained the base astroBERT language model on $\sim$400K recent astrophysics
papers from ADS comprised in total of $\sim$4B tokens. The pretraining tasks
were Masked Language Modeling (MLM) and Next Sentence Prediction (NSP).
Pretraining lasted for 40 epochs over $\sim$50 days. All computation was done
with two NVIDIA V100 GPUs.

##  3 Detecting Entities

To properly compare astroBERT to the existing language models BERT and
SciBERT, we finetuned all three models on the Detecting Entities in the
Astrophysics Literature
(DEAL)222https://ui.adsabs.harvard.edu/WIESP/2022/SharedTasks task (Grezes et
al. 2022), which consists of identifying and labeling Named Entities in a
dataset composed by full-text fragments and acknowledgements from the
astrophysics literature. The 32 types of entities are of particular interest
to astrophysicists: celestial objects, observational techniques, organizations
and persons, etc…Full list and definitions with examples available in footnote
2. Finetuning took $\sim$12 hours.

##  4 Public Release

We release the astroBERT model to the public via the popular Huggingface hub,
along with its associated WordPiece tokenizer and tutorials. The astrophysics
research community can start using astroBERT in minutes, on predefined tasks
such as detecting named entities in astrophysics literature or finetuning for
their own custom tasks.

https://huggingface.co/adsabs/astroBERT

Figure 1.: All you need to get you started with astroBERT.

##  5 Results: Comparison to BERT and SciBERT

Table 1 shows how the current astroBERT outperforms BERT, sciBERT and the
version of astroBERT that was used as the baseline for DEAL. The models were
compared using Matthews correlation coefficient (MCC), which takes into
account every value in the confusion matrix and is generally regarded a
balanced measure, along with the standard micro-averaged F-1, precision, and
recall scores. Additionally, we included a model making random predictions
based on label frequency for comparison.

Table 1.: astroBERT outperforms other language models on astrophysics data.

| Metric $\Longrightarrow$ | MCC | overall | overall | overall | overall
---|---|---|---|---|---|---
Model | Split $\Downarrow$ |  | F1 score | precision | recall | accuracy
| train | 0.1037 | 0.0170 | 0.0122 | 0.0278 | 0.7146
Random | val | 0.1083 | 0.0166 | 0.0119 | 0.0273 | 0.7059
| test | 0.1057 | 0.0162 | 0.0116 | 0.0269 | 0.6876
| train | 0.7542 | 0.4920 | 0.4995 | 0.4848 | 0.9256
BERT | val | 0.7405 | 0.4739 | 0.4780 | 0.4698 | 0.9188
| test | 0.7229 | 0.4513 | 0.4622 | 0.4409 | 0.9094
| train | 0.8159 | 0.5867 | 0.5753 | 0.5986 | 0.9430
SciBERT | val | 0.8019 | 0.5601 | 0.5463 | 0.5745 | 0.9366
| test | 0.7844 | 0.5355 | 0.5313 | 0.5398 | 0.9280
| train | 0.8296 | 0.6138 | 0.5889 | 0.6409 | 0.9468
astroBERT | val | 0.8104 | 0.5779 | 0.5508 | 0.6077 | 0.9389
(WIESP) | test | 0.7939 | 0.5561 | 0.5387 | 0.5746 | 0.9308
| train | 0.8250 | 0.5995 | 0.5701 | 0.6319 | 0.9442
astroBERT | val | 0.8194 | 0.5907 | 0.5575 | 0.6282 | 0.9405
(public release) | test | 0.8302 | 0.6093 | 0.5846 | 0.6362 | 0.9418

Figure 2 compares the performances of astroBERT to that of SciBERT over each
label along both precision and recall. Larger circles indicate a higher number
of occurrences of that label in the data, while the color indicates if the
label is mostly present in the fulltext (red) or the acknowledgment (blue)
section of the paper. This tells us where astroBERT does well, for example on
Observational Techniques (ObT) and Database (DaB), and which areas of to focus
on to improve astroBERT further, URLs which can be captured using regular
expressions, and which labels that will benefit the most from additional data,
for example Event (Eve) and Identifier (Ide) (see footnote 2 the meaning of
the abbreviations).

Figure 2.: Detailed comparison between astroBERT and SciBERT.

##  6 Harnessing Semantic Similarity

While the current version of astroBERT uses MLM and NSP as unsupervised
training tasks, it does not exploit the information unique to scientific
papers, i.e. the citation graph and the context of any given citation. Based
on the works of Reimers & Gurevych (2019) on Sentence-BERT, we will train
astroBERT on semantically similar sentences extracted from the context of
citations i.e. abstracts from the cited paper paired with the paragraph
preceding the citation. This pairing should be produce sentences embeddings
more useful to astrophysics tasks than NSP or the weakly labeled sentence
pairings used by Reimers et al. from Agirre et al. (2013).

## References

  * Agirre et al. (2013) Agirre, E., Cer, D., Diab, M., Gonzalez-Agirre, A., & Guo, W. 2013, in Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity (Atlanta, Georgia, USA: Association for Computational Linguistics), 32. URL https://aclanthology.org/S13-1004
  * Beltagy et al. (2019) Beltagy, I., Lo, K., & Cohan, A. 2019, arXiv e-prints, arXiv:1903.10676. 1903.10676
  * Devlin et al. (2018) Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. 2018, arXiv e-prints, arXiv:1810.04805. 1810.04805
  * Grezes et al. (2022) Grezes, F., Allen, T., Ghosal, T., & Blanco-Cuaresma, S. 2022, in Proceedings of the 1st Workshop on Information Extraction from Scientific Publications (Taipei, Taiwan: Association for Computational Linguistics)
  * Grezes et al. (2021) Grezes, F., Blanco-Cuaresma, S., Accomazzi, A., Kurtz, M. J., Shapurian, G., Henneken, E., Grant, C. S., Thompson, D. M., Chyla, R., McDonald, S., Hostetler, T. W., Templeton, M. R., Lockhart, K. E., Martinovic, N., Chen, S., Tanner, C., & Protopapas, P. 2021, arXiv e-prints, arXiv:2112.00590. 2112.00590
  * Kurtz et al. (1993) Kurtz, M. J., Karakashian, T., Grant, C. S., Eichhorn, G., Murray, S. S., Watson, J. M., Ossorio, P. G., & Stoner, J. L. 1993, in Astronomical Data Analysis Software and Systems II, edited by R. J. Hanisch, R. J. V. Brissenden, & J. Barnes, vol. 52 of Astronomical Society of the Pacific Conference Series, 132
  * Reimers & Gurevych (2019) Reimers, N., & Gurevych, I. 2019, arXiv e-prints, arXiv:1908.10084. 1908.10084