# Local-Adaptive Face Recognition via Graph-based Meta-Clustering and
Regularized Adaptation

Wenbin Zhu1 Chien-Yi Wang2††footnotemark:  Kuan-Lun Tseng2 Shang-Hong Lai2
Baoyuan Wang3
1Microsoft Cloud and AI 2Microsoft AI R&D Center, Taiwan 3Xiaobing.AI
{wenzh, chiwa, kutseng<EMAIL_ADDRESS><EMAIL_ADDRESS>indicates equal
contribution.

###### Abstract

Due to the rising concern of data privacy, it’s reasonable to assume the local
client data can’t be transferred to a centralized server, nor their associated
identity label is provided. To support continuous learning and fill the last-
mile quality gap, we introduce a new problem setup called “local-adaptive face
recognition (LaFR)”. Leveraging the environment-specific local data after the
deployment of the initial global model, LaFR aims at getting optimal
performance by training local-adapted models automatically and un-supervisely,
as opposed to fixing their initial global model. We achieve this by a newly
proposed embedding cluster model based on Graph Convolution Network (GCN),
which is trained via meta-optimization procedure. Compared with previous
works, our meta-clustering model can generalize well in unseen local
environments. With the pseudo identity labels from the clustering results, we
further introduce novel regularization techniques to improve the model
adaptation performance. Extensive experiments on racial and internal sensor
adaptation demonstrate that our proposed solution is more effective for
adapting face recognition models in each specific environment. Meanwhile, we
show that LaFR can further improve the global model by a simple federated
aggregation over the updated local models.

Figure 1: Local-Adaptive Face Recognition (LaFR): For each local environment,
a specialized model is produced by the adapter module with only the pre-
trained model and images from the environment. Note that there is no real
identity label associated with the images, the meta-cluster model generates
pseudo labels for robust model adaptation.

##  1 Introduction

Face recognition [30] has been commercialized widely for a variety of
applications, such as FaceID, surveillance monitoring. The COVID-19 pandemic
even accelerates the biometric technologies for touch-less solutions, such as
face recognition enabled payment and access control. Although remarkable
progress has been achieved lately, one has to admit that face recognition
still hasn’t been fully solved. Among many other remaining challenges (i.e.,
vulnerability for adversarial attack [4]), how to scale up the representation
learning to reduce the risk of fairness and bias to support various local
environments becomes a more urgent challenge. As studied in previous works
[39, 44, 14], such fairness and bias issues come from both algorithmic design
and under-represented data distributions. For example, when the model is
predominantly trained on RGB images, it generalizes poorly for images captured
by Infrared cameras. Likewise, for a model pre-trained on Caucasian only, it
performs substantially worse for African and Indian.

While it is worthwhile to push the domain-invariant face recognition [10, 25]
with the hope of generalizing to everywhere without adaptation, it is arguably
that the challenges for real-world scenarios could be more than we expected.
So the question is: given an imperfect pre-trained model, how can we improve
and fill the last-mile performance gap for each local environment and
thereafter scale up the process? In this paper, we are interested in studying
how to properly adapt the pre-trained model to a “specialized” one that
tailors for the specific environment in an automatic and unsupervised manner.
Here, the “environment” could be defined broadly, including a specific new
camera sensor (i.e. an infrared camera with particular wavelength), a unique
identity distribution with racial bias, or a physical environment that has
unique camera placement and lighting condition, etc. We call such a problem
setup as “Local-Adaptive Face Recognition (LaFR)”, whose workflow (see Fig. 1)
starts from an imperfect pre-trained global model deployed to a specific
environment, where it accumulates some amount of new data. It then applies
unsupervised adaptation technique to adapt the initial model locally, hence no
data is transferred to server. Finally, after the adaptation, the new model is
expected to perform much better than the initial global model as it is trained
to tailor that environment. As an optional step, Federated Learning [24] is
further employed to aggregate many such local models in a secure manner.
Therefore, “LaFR” essentially provides a way to scale up the representation
learning and model generalization via such “dual-loop” paradigm.

Although unsupervised domain adaptation (UDA) [37] has been widely studied in
person re-identification (re-ID) [28, 7, 19, 47, 20, 18, 43], it is much less
explored in face recognition except [39, 33]. Most of those works either
designed special for person re-ID [7], or their setups require both source and
target dataset to be available during the adaptation stage, or they only aim
at closed-set problem. Moreover, person re-ID works heavily depend on variants
of triplet loss, as we show in prior works, there are more robust losses
(i.e., CircleLoss [29]) that proved to work better for face recognition.

To overcome the challenges, we first introduce a graph-based meta-clustering
algorithm designed to predict pseudo labels for any unlabelled dataset. To do
this, we collect a set of labeled datasets from multiple domains and extract
their face embeddings from the given pre-trained model, we then apply Graph
Convolution Network (GCN) to model the non-convex structure relationship for
face embeddings within each set, which is trained efficiently through meta-
learning [6] to make the cluster prediction more generalizable for the unseen
dataset. Secondly, to better facilitate the transfer learning, we introduce a
new technique by transferring the representation of class (pseudo label)
center from the pre-trained model to the classifier of the new model and keep
it fixed while only fine-tuning the feature representation in the context of
margin-based training objectives such as [34, 3, 29]. Moreover, instead of
regularizing the feature distance (commonly used in knowledge distillation [8,
13, 27]), we regularize the network weights to ensure a small deviation
between the pre-trained and the new local model.

To summarize, we make the following major contributions: (1) we introduce a
novel unsupervised model adaptation problem setup for face recognition, we
argue that it’s practical yet scalable motivated from both continuous learning
and data privacy concerns; (2) we use graph convolution network (GCN) to model
the dataset structure and predict the clustering labels, through a meta
learning framework; (3) Our novel regularized center transfer (RCT) technique
can significantly reduce the risk of overfitting and improve transfer learning
performance for even smaller datasets; (4) Experiments show that our entire
solution not only outperform other strong baselines for local adaption but
also enable the federated learning to further improve the global model.

Figure 2: Overview of our Local-Adaptive face recognition (LaFR) framework.
Meta-Cluster Model Training: A Graph Convolution Network (GCN) based embedding
cluster model is trained with the meta-optimization procedure to generalize
well in unseen local environments. Test-time Local Adaptation: In the local
adaptation phase, we obtain the pseudo identity label of local face images
from the meta-cluster model. Given the pre-trained face recognition model,
local images, and their corresponding pseudo labels, the proposed Regularized
Center transfer (RCT) technique can adapt the model more robustly to produce a
specialized face recognition model.

##  2 Related Works

#### Generalized Face Recognition

Most of the recent works focus on generalizing the representation power
through novel loss functions, such as CircleLoss [29], Arcface [3],
Regularface [5] and AM-SoftMax [34], with both solid theoretical foundation
and remarkable empirical results are reported in SOTA models. A set of other
new works target to improve the performance from the data distribution
perspective, including data augmentation [44] for under-represented classes,
improved training strategy for unbalanced and long-tailed data [45], and
uncertainty modeling for noisy training set [2]. More recently, generalized
representation learning has received more and more attention. [26] learns
universal representations to tackle pose, resolution, and occlusion variation,
while [10] uses meta-learning to make it easily generalized to the unseen
domain without requiring adaptation. [35] proposes to use the adversarial
decorrelation technique to make the identity representation invariant to age
information. Causal relationship modeling through Invariant Risk Minimization
[1] is a promising direction to address the out-of-distribution (OOD)
challenge, yet, no impressive results have been reported so far for face
recognition. Our method is both compatible and complementary with all those
latest works. Improving the generalization capability of the pre-trained model
will provide a better initialization for subsequent adaptation.

#### Unsupervised Domain Adaptation (UDA)

There is a large body of research works in unsupervised domain adaption [37]
for person re-identification [28, 7, 19, 47, 20, 18, 43], semantic
segmentation [31], image classifications [36, 33, 22] and attribute
recognition [15]. However, most of those approaches either target for closed-
set setting or require both source and target to be available during the
adaptation to align the feature distribution through maximum mean discrepancy
(MMD) [32]. Conceptually, a clustering technique is first needed to assign a
pseudo label to each image before the training. Prior person re-ID works have
explored various heuristic methods such as bottom-up cluster [19] and
tracklet-based clustering [18]. In contrast, our clustering is achieved
through a graph convolution network (GCN) to learn the intrinsic structure for
each unseen data-set via meta-learning. To our best knowledge, there is little
work of applying UDA for face recognition [23, 27]. Perhaps the most closely
related work is [39] in which a traditional MMD loss is required to be
employed. However, as we argued before, in the context of “LaFR”, the source
data is not available during the adaptation due to the data privacy and
security concern.

#### Graph Convolution Network based Face Clustering

Face clustering is another fundamental problem in computer vision, which is
also quite related to our work. A recent series of studies [42, 41, 46] show
that supervised clustering performs significantly better than the conventional
unsupervised ones such as K-Means [21] and Spectral-Clustering [16]. The
reason behind is that these Graph Convolution Network (GCN) [17] based methods
are more capable of finding the local non-convex structures. Inspired by such
remarkable progress, we move one step further and marry GCN-based clustering
with meta-learning [6] to make the resulting clustering model more adaptive to
the unseen dataset.

##  3 Overview

Given an imperfect pre-trained model $\Theta_{0}$, our whole system requires a
labeled dataset $\mathbb{S}_{C}$ in the global-end, which consists of a few
subsets ($\mathbb{S}_{{C}_{1}},...,\mathbb{S}_{{C}_{k}}$) from multiple
domains to train a novel graph-based meta-clustering model $\Phi$. Once
trained, given any unlabelled new dataset $\mathbb{S}_{\mathcal{A}}$ on a
specific local environment, we will feed it into $\Phi$ to obtain the pseudo
label for each image. Then, we apply our proposed regularized center transfer
(RCT) technique to perform the adaptation and produce a specialized local
model $\Theta_{A}$ tailors for that environment. Figure 2 illustrates the
framework of our system.

##  4 Graph-based Meta-Clustering Learning

###  4.1 Preliminary of Graph Convolution Network

Given a face embedding set $\mathbb{S}=\\{{f_{i},y_{i}}\\}^{N}_{i=1}$, which
$y_{i}$ is the corresponding identity label of each embedding. We first define
a graph $\mathcal{G}(\mathcal{V},\mathcal{E})$ that connects each image with
its K-NN neighbor, and the affinity matrix $\mathcal{A}$ is computed by cosine
similarity between $f_{i}$ and its neighbor $f_{j}(j\in\mathcal{N}_{i})$, so
$\mathcal{A}_{ij}=a_{ij}=\cos(f_{i},f_{j})$ and $\mathcal{N}_{i}$ represents
the neighbors of $f_{i}$. Let us use $F_{0}\in\mathbb{R}^{N\times d_{in}}$ to
represent the input feature embedding for a Graph Convolution Network(GCN),
then after $l$ layers of convolutions on the graph, the feature embedding
$F_{l}\in\mathbb{R}^{N\times d_{out}}$ becomes

$F_{l}=\sigma(g(\bar{A},F_{l-1})W^{\mathcal{G}}_{l-1})$ (1)

where $\bar{A}=\bar{D}^{-1}(A+I)$ and $\bar{D}_{ii}=\sum_{j}(A+I)_{j}$ is a
diagonal degree matrix, $W^{\mathcal{G}}_{l-1}\in\mathbb{R}^{2\times
d_{in}\times d_{out}}$ is a trainable matrix to transfer the input embedding
into a new space, $\sigma$ is the nonlinear activation function(i.e., ReLU),
and $g(\cdot,\cdot)$ is a concatenation function following previous works [42,
41, 46], which is defined as

$g(\bar{A},F_{l-1})=[(F_{l-1})^{T},(\bar{A}F_{l-1})^{T}]^{T}$ (2)

In the context of this work, as we want to learn how to predict the face
clustering for a given dataset, we let the last layer output a 1-D confidence
value for each vertex. Assume $c^{\prime}$ represents the vector of confidence
values for each vertex, then

$c^{\prime}=F_{l}W^{\mathcal{G}}+b$ (3)

where $W^{\mathcal{G}}$ is the projection matrix and $b$ is the bias, both are
learnable. Again, following the design of “GCN-V” [41], we define the ground-
truth confidence $c_{i}$ as

$c_{i}=\frac{1}{|\mathcal{N}_{i}|}\sum_{j\in\mathcal{N}_{i}}(\mathbbm{1}(y_{j}=y_{i})-(\mathbbm{1}(y_{j}\neq
y_{i})))\cdot a_{ij}$ (4)

where $\mathbbm{1}()$ is an indicator function. Therefore, we minimize the
following loss to train GCN:

$\mathcal{L}^{\mathcal{G}}(\Phi)=\frac{1}{N}\sum_{i}^{N}|c_{i}-c^{\prime}_{i}|$
(5)

where $c^{\prime}_{i}$ is the confidence value from $c^{\prime}$ for the
corresponding vertex $i$, and $\Phi$ represents the parameters of the GCN
model. Intuitively, a lower $c_{i}$ means this vertex could lie near the
boundary between two different clusters according to the local graph
structure.

###  4.2 Meta-Clustering with GCN

By no means we are the first to apply GCN for face clustering given the
impressive works done recently [42, 41, 46]. Nevertheless, our work is
compatible yet builds on top of them by changing the training strategy into a
meta-learning paradigm. We are interested in solving the aforementioned Local-
Adaptive face recognition scenarios, where the pre-trained model is already
deployed and their source data is not available anymore. Therefore, an
automatic and unsupervised adaptation from a pre-trained model becomes a
necessary and urgent problem, in order to massively scale-up the adaptation.

To train a meta-clustering model on GCN, we first collect a few labeled
datasets $\mathbb{S}_{C_{1}},...,\mathbb{S}_{C_{k}}$ from $C_{K}$ different
domains and then perform domain-level sampling for outer-loop iteration, where
we sample $C_{K}-1$ domains data for the meta-train while the data from the
remaining domain for the meta-test. Our goal is to let this GCN be more
generalizable for any dataset from unseen domains. We describe the detailed
training procedure in Algorithm 1. Specifically, We start with a randomly
initialized GCN parameter setting $\Phi_{0}$, and for each meta-train
iteration, we perform the conventional GCN training via the following
equation:

$\Phi^{{}^{\prime}}=\Phi-\alpha\nabla_{\Phi}\mathcal{L}_{mtr}^{\mathcal{G}}(\Phi)$
(6)

where $\mathcal{L}_{mtr}^{\mathcal{G}}(\Phi)$ denotes the loss on the meta-
train dataset. The model is then tested on the meta-test dataset. Similarly,
we compute the corresponding loss as
$\mathcal{L}_{mte}^{\mathcal{G}}(\Phi^{{}^{\prime}})$ with the updated model
$\Phi^{{}^{\prime}}$. The meta-model will then be updated jointly from both
gradients (Line $\\#$8). This process is iterated until it reaches the
maximum.

Algorithm 1 Graph-based Meta-Clustering

0: Initialize $\Phi$ as $\Phi_{0}$; Datasets
$\mathbb{S}$[$\mathbb{S}_{C_{1}},...,\mathbb{S}_{C_{k}}$]; Hyper-parameters
$\alpha,\beta,\xi$

0: $\Phi$

1: for iter $<$ MaxIter do

2: Split: $\mathbb{S}$ into $\mathbb{S}_{mtr}$ and $\mathbb{S}_{mte}$ randomly

3: Meta-Train:

4: Train on $\mathbb{S}_{mtr}:$
$\Phi^{{}^{\prime}}=\Phi-\alpha\nabla_{\Phi}\mathcal{L}_{mtr}^{\mathcal{G}}(\Phi)$

5: Meta-Test:

6: Compute loss on
$\mathbb{S}_{mte}:\mathcal{L}_{mte}^{\mathcal{G}}(\Phi^{{}^{\prime}})$

7: Meta-optimization: Update $\Phi$

8:
$\Phi=\Phi-\beta(\nabla_{\Phi}\mathcal{L}_{mtr}^{\mathcal{G}}(\Phi)+\xi\nabla_{\Phi}\mathcal{L}_{mte}^{\mathcal{G}}(\Phi^{{}^{\prime}}))$

9: end for

##  5 Unsupervised Model Adaptation

As discussed before, our pipeline should only take the pre-trained model
$\Theta_{0}$ and an unlabeled dataset $\mathbb{S}_{A}$
($\\{{x_{i}}\\}^{N}_{i=1}$) accumulated from the specific local environment as
input. We will then run the above graph-based meta-clustering module to obtain
the estimated number of pseudo “identity”(or class) labels as well as their
belonging images. Let us denote $(x_{i},y_{i})$ as one training image
embedding with its associated pseudo ID: $y_{i}$, one can then employ any of
the recent loss functions such as ArcFace [3], AM-Softmax [34] and CircleLoss
[29] to train the adapted model $\Theta_{A}$ from $\Theta_{0}$.

###  5.1 Preliminary of Face Recognition Training

The early idea behind face recognition is to treat it as a classification
problem and apply standard softmax loss to train the deep face representation:
$\mathcal{L}=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{e^{W^{T}_{{y_{i}}}f_{i}+b_{y_{i}}}}{\sum^{\mathcal{C}}_{j=1}e^{W^{T}_{j}f_{i}+b_{i}}}$
,where $f_{i}\in\mathbb{R}^{d}$ denotes the deep feature learned through the
network for each image; $W_{i}\in\mathbb{R}^{d}$ represents the corresponding
classifier vector from the last fully connected layer for label ${y_{i}}$, and
$b_{i}$ is the bias term. By normalizing both classifier vector $\parallel
W_{j}\parallel=1$ and $\parallel f_{i}\parallel=1$, while adding a scale
factor $\gamma$, we get $W_{j}^{T}f_{i}=\parallel w_{j}\parallel\parallel
f_{i}\parallel\cos\theta_{j}=\cos\theta_{j}$, where $\theta_{j}$ is the angle
between $w_{j}$ and $f_{i}$, for simplicity, we also fix $b_{j}=0$. To improve
the generalization, recent study shows that adding different types of
margins[29][34][3] can bring significant gain in many places. For example,
ArcFace[3] adds angular margin while AM-softmax [34] adds cosine margin.
Without loss of generality, we take AM-softmax to as the training objective:
$\mathcal{L}(\Theta)=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{e^{\gamma(\cos({\theta_{y_{i}})-m})}}{e^{\gamma(\cos({\theta_{y_{i}})-m})}+\sum^{\mathcal{C}}_{j\neq
y_{i}}e^{\gamma\cos\theta_{j}}}$, where $\mathcal{C}$ is the total number of
classes, and $m$ is the margin that needs empirically determined.

###  5.2 Regularized Center transfer (RCT)

We assume pre-trained representation (from $\Theta_{0}$) already has strong
discriminative power. After training, images belonging to the same identity
tend to be clustered in a small region on the hyper-sphere, while still
maintaining a cosine margin (defined as $m$) with images from different
identities. If the pre-trained model was trained with a large number of
different identities, the cluster area for each identity should have been
squeezed on the hyper-sphere to reduce the intra-class variation. Let us
denote $C^{\Theta_{0}}_{y_{i}}$ as the center of the face embedding for
$y_{i}$ on the pre-trained model, then
$C^{\Theta_{0}}_{y_{i}}=\frac{1}{M_{i}}\sum^{M_{i}}_{k=1}\mathbbm{1}(y_{k}=y_{i})f^{\Theta_{0}}_{k}$,
where $M_{i}$ is the total number of images belonging to class $y_{i}$.
Inspired by center loss[40], instead of learning $f_{i}$ and its corresponding
$W_{y_{i}}$ from the new dataset from scratch, we propose to transfer this
pre-trained class center as prior knowledge to the adapted model $\Theta_{A}$,
so basically we want to have ${C_{y_{i}}^{\Theta_{A}}}$ to be as close to
$C^{\Theta_{0}}_{y_{i}}$ as possible during the adaptation. In the context of
AM-softmax[34] or other latest training objectives such as CircleLoss [29], we
notice that classifier vector $W_{y_{i}}$ is also close to
$C_{y_{i}}^{\Theta_{0}}$ even though they are not exactly the same. Therefore,
in practice, to simplify the learning process, we directly use
$C_{y_{i}}^{\Theta_{0}}$ to initialize each corresponding $W_{y_{i}}$, rather
than learning it from scratch. To further reduce the overfitting risk when
adapting to a small dataset, we add another model regularization term to let
$\Theta_{A}$ not deviate too much from $\Theta_{0}$. Our final loss function
is defined as follows:

$\displaystyle\begin{split}\mathcal{L}(\Theta_{A})={}&-\frac{1}{N}\sum_{i=1}^{N}\log\frac{e^{\gamma(\cos({\theta_{y_{i}})-m})}}{e^{\gamma(\cos({\theta_{y_{i}})-m})}+\sum^{\mathcal{C}}_{j\neq
y_{i}}e^{\gamma\cos\theta_{j}}}\\\
&+\lambda\parallel\Theta_{A}-\Theta_{0}\parallel^{2}_{2}\end{split}$ (7)

subject to

$\displaystyle\begin{split}{}&W={W^{*}}/{\parallel W^{*}\parallel},\\\
&f={f^{*}}/{\parallel f^{*}\parallel},\\\ &\cos\theta_{j}=W^{T}_{j}f_{i},\\\
&W_{y_{i}}=C^{\Theta_{0}}_{y_{i}}\end{split}$ (8)

where $W$ is the normalized classifier matrix, $x$ is the normalized feature
vector for each image, $\lambda$ is a hyper-parameter to trade-off between the
loss on the new dataset and the model regularization term. During the model
adaptation training, we initialize each $W_{y_{i}}$ with
$C^{\Theta_{0}}_{y_{i}}$ and keep it fixed and only fine-tune the feature
representation. Compared with standard transfer learning without RCT, our
regularized center transfer can better preserve the pre-trained
representation, especially for small datasets, therefore reducing the risk of
overfitting.

###  5.3 Federated Aggregation

As we argued in Sec. 1, with the rising concern of data privacy, it is
reasonable to assume that training data would be highly de-centralized across
different clients in the future, where model adaptation is conducted first
locally at each client and later on being aggregated via federated learning
[24] in a secure way. We argue that our proposed unsupervised modal adaptation
is designed to further facilitate such fully automated “dual-loop” learning
paradigm. To reduce the risk of losing privacy and adversarial attack, we
remove the top classification layer and only transfer the backbone model
parameters between each client and server and conduct simple model averaging,
we denote such partial averaging setup as “FedPav” in our experiments [48].

##  6 Experimental Settings

###  6.1 Datasets and Protocols

In each Local-Adaptive face recognition (LaFR) protocol, the pre-trained face
recognition model is trained with base dataset
$\mathbb{S}_{B}=\\{{x^{B}_{i}},{y^{B}_{i}}\\}$, and the meta-cluster GCN model
is trained with multiple non-overlapped labeled datasets
$\mathbb{S}_{C}=\\{{x^{{C}_{1}}_{i}},{y^{{C}_{1}}_{i}}\\}\cup\\{{x^{{C}_{2}}_{i}},{y^{{C}_{2}}_{i}}\\}\cup...\\{{x^{{C}_{k}}_{i}},{y^{{C}_{k}}_{i}}\\}$.
While deploying the model in the specific scenario, the model is adapted with
unlabeled dataset $\mathbb{S}_{A}=\\{{x^{A}_{i}}\\}$ from the environment. The
final face recognition performance is evaluated on the testing dataset
$\mathbb{S}_{T}=\\{{x^{T}_{i}},{y^{T}_{i}}\\}$. Here, $x_{i}$ and $y_{i}$
represent the i-th face image and the corresponding identity label,
respectively. Note that the base dataset $\mathbb{S}_{B}$ is not available
during clustering and adaptation.

#### Adapt to Different Local Races

The existence of face image distribution shift between different races has
been proved in related works [39][38], so we re-organized the racial datasets
collected by [39] to build the LaFR protocols. We leverage the Caucasian
dataset from BUPT-Transferface [39] as $\mathbb{S}_{B}$ for base face
recognition model training, and racial faces in-the-wild (RFW) dataset [39] is
used as $\mathbb{S}_{T}$ for testing after adaptation. BUPT-Balancedface [39],
which has non-overlapped 7k subjects for each race, is leveraged for meta-
cluster model training and further adaptation. The dataset details are
summarized in Table 1. We build three leave-one-out protocols, which one of
the races from {African, Asian, Indian} is selected as the local target, and
dataset from other races can be used for meta-cluster GCN model training.

#### Adapt to Different Local Infrared Sensors

Different wavelengths have different penetration rates into human skins,
leading to different contrasts on the IR face images. Most cameras are
sensitive to 850nm infrared light and generally considered to have higher
contrast than 940nm infrared light. Meanwhile, different cameras also adapt
different Image Signal Processors (ISP), so the way to handle tone-mapping,
noise reduction, and blurriness are different too, which causes another type
of significant appearance distribution bias across different sensors. In this
paper, we are interested in studying how to adapt a pre-trained RGB face
recognition model to different local environments, each of which mimics a
specific infrared camera enabled face recognition scenario. As there is no
public infrared camera dataset with different ISP or wavelength, we collected
an internal dataset with four different infrared camera sensors that capture
infrared wavelength ranging from 850 nm to 940 nm. Figure 3 shows a few
typical examples sampled from our four datasets, whose details are summarized
in Table 2. We partition the collected dataset by identity to form
$\mathbb{S}_{C}$, $\mathbb{S}_{A}$, and $\mathbb{S}_{T}$, and we leverage
commonly used RGB dataset MS-1M [12] as $\mathbb{S}_{B}$. Similarly, we build
four leave-one-out protocols, where we use datasets from 3 sensors to train
our proposed meta-clustering while the remaining one for both adaptation and
final testing.

Dataset | Race | # Subjects | # Images
---|---|---|---
$\mathbb{S}_{B}$ | Caucasian | 10000 | 468139
$\mathbb{S}_{C},\mathbb{S}_{A}$ | Indian | 7000 | 275095
Asian | 7000 | 325475
African | 7000 | 324376
$\mathbb{S}_{T}$ | Indian | 2984 | 10308
Asian | 2492 | 9688
African | 2995 | 10415

Table 1: Statistics of the datasets for races.

Dataset | Sensor | # Subjects | # Images
---|---|---|---
$\mathbb{S}_{B}$ | RGB | 94,430 | 5,179,510
$\mathbb{S}_{C},\mathbb{S}_{A}$ | IR-A | 430 | 57,818
IR-B | 250 | 34,592
IR-C | 220 | 30,689
IR-D | 400 | 55,312
$\mathbb{S}_{T}$ | IR-A | 210 | 28,916
IR-B | 3,372 | 27,155
IR-C | 200 | 14,035
IR-D | 235 | 32,388

Table 2: Statistics of the datasets for sensors.

###  6.2 Evaluation Metrics

#### Face Embedding Clustering

Given the ground truth label from adaptation datasets $\mathbb{S}_{A}$, the
intermediate face clustering result from the GCN model can be evaluated to
indicate the accuracy of identity label assignment during adaptation. Two
common clustering metrics [42, 41]: Pairwise F-score ($F_{P}$) and B-Cubed
F-score ($F_{B}$) are used, which calculate the harmonic mean of precision and
recall. The metric $F_{p}$ puts relatively more emphasis on large face
clusters, while $F_{B}$ weights clusters linearly based on their size.

#### Local Adaptive Face Recognition

The face recognition performance in the adaptation target is evaluated with
standard face recognition metrics. In race adaptation protocols, we calculate
the verification accuracy on 6000 difficult pairs selected by [39]. In sensor
adaptation protocols, we report the False Non-Match Rates where the False
Match Rate is 1e-6 (FNMR@FMR=1e-6).

Figure 3: Top: Infrared Images captured by 4 different infrared cameras (the
first two capture 940nm wavelength, the other two 850nm); Bottom: Their
corresponding RGB images captured by another 4 RGB cameras.

###  6.3 Implementation Details

#### Meta-Clustering with GCN Training

For GCN model training, we set the meta learning rate $\alpha$ to 0.1, the
outer loop learning rate $\beta$ to 0.1, and the meta loss weight $\xi$ to 1.0
to conduct the meta optimization. In all experiments, we set the momentum to
0.9 and train 30000 iterations with the SGD optimizer. In each iteration, we
randomly select one dataset from $\mathbb{S}_{C}$ as the meta-test set and the
other datasets as the meta-train set. It takes around 7 hours to train on one
TITAN X GPU.

#### Model Adaptation

After clustering, we follow [41] to set a threshold $\tau$ to 0.8 to cut off
the edges with small similarities, and obtain the pseudo identity labels by
simply connecting each vertex to their nearest neighbors. During model
adaptation with Regularized Center Transfer (RCT), we follow Equ. 7 to
calculate the face embeddings center of each identity according to the pseudo
labels. We set the model regularization $\lambda$ to 0.1 and employ CircleLoss
[29] as the classification loss. In all adaptation experiments, we set the
learning rate to 0.001 and train for 50 epochs. It takes around 5 hours to
train on one TITAN X GPU.

##  7 Experimental Results

###  7.1 Adapt to Different Races

#### Face Embedding Clustering

With the ground truth identity label (not available during clustering and
adaptation) from the dataset $\mathbb{S}_{A}$, we can evaluate the
intermediate face clustering results from different methods using the F-score
metrics described in Sec. 6.2. We compare the clustering performance with the
original GCN [41] method, which uses all the datasets in $\mathbb{S}_{C}$ to
build a large graph and learn the embedding confidence prediction through the
supervised training scheme described in [41]. Besides, we also compare with
another distance-based clustering method [39], which connects all the
embedding pairs whose cosine distance between them is less than a fixed
threshold. We report the result with the threshold 0.3, which has the best
performance across all sampled thresholds. The clustering performance of
racial adaptation protocols is shown in Table 3, and we denote our proposed
meta-clustering method as “meta-GCN”. From the results, we can observe that
the simple distance-based clustering method [39] has a very low Pairwise
F-score ($F_{P}$) but high Bcubed F-score ($F_{B}$) in some adaptation
protocols, which means that it cannot handle large embedding clusters. Our
proposed “meta-GCN” outperforms the original GCN [41] in most of the
benchmarks. It indicates that the proposed meta-learning scheme can learn more
generalized GCN parameters and cluster the face embeddings better in unseen
local environments. The clustering output with higher F-scores produces
cleaner pseudo identity labels, which benefit more to the model adaptation
process.

Methods | African | Asian | Indian
---|---|---|---
Distance-based [39] | 0.0086 / 0.7282 | 0.0035 / 0.6267 | 0.6891 / 0.7481
GCN [41] | 0.6115 / 0.8132 | 0.3492 / 0.6432 | 0.8559 / 0.8725
Meta-GCN |  0.8129 / 0.8535 |  0.3768 / 0.6876 |  0.8849 / 0.8551

Table 3: Comparison of face embedding clustering performance on three racial
adaptation protocols. Two common clustering metrics: Pairwise / Bcubed F-score
($F_{P}/F_{B}$) pairs are reported.

#### RCT Technique for Model Adaptation

To evaluate the effectiveness of our proposed Regularized Center Transfer
(RCT) technique, we leverage the ground truth (GT) identity labels from the
adaptation dataset $\mathbb{S}_{A}$ to perform model adaptation. We compare
the result with standard transfer learning, which is adapting the face
recognition model without regularizing the classifier during the adaptation
process. From Table 4, we have the following observations. First, the pre-
trained model cannot perform well on three local targets without adaptation
due to the race gap. Second, standard fine-tuning could possibly harm the face
recognition model (ex. Indian protocol) due to the risk of overfitting on
small dataset, which is illustrated in Figure 4) where it shows RCT is
superior to standard fine-tuning across different size of identities. Third,
our regularization techniques can effectively prevent the local model from
overfitting and optimize the representation better than fine-tuning by a large
margin in all protocols. Furthermore, we conduct ablation experiments to study
the effect of different loss functions and the regularization term used in
RCT. Results from Table 5 demonstrate that: 1) Our RCT can work with other
margin-based classification loss like AM-softmax [34], but perform better with
the SOTA CircleLoss [29]. 2) The weight regularization term (controlled by
$\lambda$) does contribute a lot to the final adaptation performance.

Methods | Caucasian | African | Asian | Indian
---|---|---|---|---
Pre-trained | 0.9512 | 0.8537 | 0.8633 | 0.9047
$\mathbb{S}_{A}$-GT + Fine-tune | - | 0.8758 | 0.8705 | 0.8528
$\mathbb{S}_{A}$-GT + RCT | - | 0.9227 | 0.9212 | 0.9323

Table 4: Deployment model performance comparison while using the ground-truth
(GT) labels from adaptation dataset ($\mathbb{S}_{A}$) on three racial
protocols. Verification accuracy on 6000 pairs of the testing dataset
($\mathbb{S}_{T}$) is reported.

Methods | African | Asian | Indian
---|---|---|---
Pre-trained | 0.8537 | 0.8633 | 0.9047
RCT (AM-Softmax, $\lambda$=0.0) | 0.9125 | 0.9050 | 0.9048
RCT (CircleLoss, $\lambda$=0.0) | 0.9145 | 0.9033 | 0.9120
RCT (AM-Softmax, $\lambda$=0.1) | 0.9158 | 0.9153 | 0.9278
RCT (CircleLoss, $\lambda$=0.1) | 0.9227 | 0.9212 | 0.9323

Table 5: The influence of loss function choices and weight regularization
$\lambda$ on racial adaptation protocols. All experiments are conducted with
“$\mathbb{S}_{A}$-GT + RCT” setting.

#### End-to-end Unsupervised LaFR

We further conduct end-to-end Local-Adaptive face recognition (LaFR)
experiments, which combine face embedding clustering methods (from Table 3)
with the proposed RCT adaptation technique. The end-to-end LaFR performance on
three race protocols are shown in Table 6. The result in the last row which
adopts ground truth (GT) labels from $\mathbb{S}_{A}$ serves as the upper
bound of end-to-end unsupervised LaFR methods. We also re-implement and
compare with the recent generalized face recognition method MFR [11] in our
protocol. The dataset in $\mathbb{S}_{C}$ are leveraged into the meta-
optimization process to obtain a better universal model. Although MFR works
better than standard pretraining, we still observe the performance gap between
MFR and our adaptation results, indicating the value of local adaptation for
optimized performance. Meanwhile, Table 6 also shows that our proposed “meta-
GCN” outperforms both distance-based[39] and regular GCN[41] based clustering
methods in our protocols, which proves the effectiveness of the graph
representation as well as the meta learning on top of the GCN.

Methods | African | Asian | Indian
---|---|---|---
Pre-trained | 0.8537 | 0.8633 | 0.9047
MFR [11] | 0.8882 | 0.8768 | 0.9118
Distance-based [39] \+ RCT | 0.7613 | 0.7488 | 0.8862
GCN [41] \+ RCT | 0.8692 | 0.8717 | 0.8940
Meta-GCN + RCT | 0.8912 | 0.8773 | 0.9258
$\mathbb{S}_{A}$-GT + RCT | 0.9227 | 0.9212 | 0.9323

Table 6: Deployment model performance comparison while using pseudo identity
labels from different clustering methods combined with RCT on three racial
protocols. Verification accuracy on 6000 pairs of the testing dataset
($\mathbb{S}_{T}$) is reported. Figure 4: Performance comparison between
$\mathbb{S}_{A}$-GT + Fine-tune and $\mathbb{S}_{A}$-GT + RCT under different
percent of identities on the “African” adaptation protocol. Figure 5: The ROC
curves of deployed models evaluated on all pairs of (a) IR-A, (b) IR-B, (c)
IR-C, (d) IR-D protocols.

Label | Loss | FedPav | IR-A | IR-B | IR-C | IR-D
---|---|---|---|---|---|---
$\mathbb{S}_{B}$-GT | (Pre-trained) |  | 0.0344 | 0.0330 | 0.0633 | 0.0626
SpC [9] | UCL [9] |  | 0.0332 | 0.0298 | 0.0579 | 0.0623
SpC [9] | RCT |  | 0.0256 | 0.0256 | 0.0443 | 0.0425
D-based [39] | RCT |  | 0.0267 | 0.0254 | 0.0433 | 0.0384
GCN [41] | RCT |  | 0.0245 | 0.0223 | 0.0358 | 0.0331
Meta-GCN | RCT |  | 0.0212 | 0.0198 | 0.0298 | 0.0299
Meta-GCN | RCT | ✓ | 0.0207 | 0.0188 | 0.0274 | 0.0267
$\mathbb{S}_{A}$-GT | RCT |  | 0.0175 | 0.0158 | 0.0192 | 0.0251
$\mathbb{S}_{A}$-GT | RCT | ✓ | 0.0175 | 0.0164 | 0.0187 | 0.0228

Table 7: Comparison between different LaFR methods on sensor adaptation
protocols. The performance is measured by FNMR@FMR=1e-6, the lower the better.

###  7.2 Adapt to Different Sensors

We conduct similar end-to-end LaFR experiments on four infrared sensor
adaptation protocols, note that those sensors are from real industry face
recognition systems. From Table 7 and Figure 5, we have the following
observations. First, as the upper bound for tested end-to-end LaFR methods,
“$\mathbb{S}_{A}$-GT + RCT” can reduce the adaptation non-match error up to
70% compared with the strong RGB pre-trained model, which demonstrate the
effectiveness of our adaptation regularization techniques for face
recognition. Second, with limited number of identities collected in the
environment (ex. 220 in IR-C), standard fine-tuning (“$\mathbb{S}_{A}$-GT +
fine-tune”) is prone to overfitting and cannot achieve better performance
while sensor adaptation compared with the pre-trained model. Third, our
proposed “Meta-GCN + RCT” achieves the best performance in all protocols among
different clustering methods, which shows the generalization ability and
robustness of our meta-clustering GCN model in different scenarios.

#### Compared with SOTA Domain-adaptive ReID

To show the necessity of our proposed solution, we further compare with the
SOTA object Re-ID method [9]. We applied both their Self-paced Clustering(SpC)
and the Unified Contrastive Loss(UCL) in our system. Even though SpC performs
impressively on person reID task, their techniques are still inferior to our
proposed Meta-GCN with RCT on the task of sensor adaptation for face
recognition, which is likely caused by the unique challenging nature of face
recognition task.

#### Experiments on Federated Learning

We conduct experiments to verify the effectiveness of our LaFR in federated
learning setting. Assuming there are $K$ clients ($i=1,2,...,K$), each
associates with unlabeled private dataset $D_{i}$, and the face recognition
model $\Theta_{i}^{t}$ at the optimization dual-loop step $t$. The continual
model adaptation process within federated learning setup is shown in Algorithm
2. The training procedure contains 20 rounds of dual-loop, and each dual-loop
consists of a local adaptation and a partial federated aggregation (FedPav)
which averages the backbone parameters from the local clients. The results
shown in Tab. 7 verify the effectiveness of the federated learning pipeline to
iteratively improve the local face recognition models without breaching user’s
privacy on each client.

Algorithm 2 Model Adaptation in Federated Learning

0: Pretrained model on server $\Theta_{s}^{0}$; Client models
$\Theta_{c_{i}}^{0}$; Client datasets $D_{i}$ (i=1,2,…,K);

0: Client models $\Theta_{c_{i}}^{T}$;

1: for each step t = 0 to T-1 do

2: for each model i = 1 to K do

3: $\Theta_{c_{i}}^{t+1}$ $\leftarrow$ RCT($\Theta_{s}^{t}$,
$\Theta_{c_{i}}^{t}$, $D_{i}$)

4: end for

5: $\Theta_{s}^{t+1}$ $\leftarrow$ $\frac{1}{K}\sum_{i=1}^{K}$
$\Theta_{c_{i}}^{t}$

6: end for

##  8 Conclusions

We introduce a new problem setup called “Local-Adaptive Face Recognition
(LaFR)”, which aims to produce specialized face recognition models tailored
for each local environment. Our proposed graph-based meta-clustering model can
better cluster the face embeddings for unseen environments, which provides
cleaner identity labels in an unsupervised manner during adaptation. Combined
with the proposed RCT module, our framework can robustly produce LaFR models
adapted from imperfect pre-trained face recognition models. The effectiveness
of our framework is proven on various protocols, including racial and sensor
adaptation, with or without federated aggregation. We hope these efforts can
open up future directions towards specialized face recognition models at
scale.

## References

  * [1] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz.  Invariant risk minimization.  arXiv preprint arXiv:1907.02893, 2019.
  * [2] Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen Wei.  Data uncertainty learning in face recognition.  In CVPR, 2020.
  * [3] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In CVPR, 2019.
  * [4] Yinpeng Dong, Hang Su, Baoyuan Wu, Zhifeng Li, Wei Liu, Tong Zhang, and Jun Zhu.  Efficient decision-based black-box adversarial attacks on face recognition.  In CVPR, 2019.
  * [5] Yueqi Duan, Jiwen Lu, and Jie Zhou.  Uniformface: Learning deep equidistributed representation for face recognition.  In CVPR, 2019.
  * [6] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In ICML, 2017.
  * [7] Y. Fu, Y. Wei, G. Wang, Y. Zhou, H. Shi, U. Uiuc, and T. Huang.  Self-similarity grouping: A simple unsupervised cross domain adaptation approach for person re-identification.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6111–6120, 2019.
  * [8] S. Ge, S. Zhao, C. Li, and J. Li.  Low-resolution face recognition in the wild via selective knowledge distillation.  IEEE Transactions on Image Processing, 28(4):2051–2062, 2019.
  * [9] Yixiao Ge, Feng Zhu, Dapeng Chen, Rui Zhao, and Hongsheng Li.  Self-paced contrastive learning with hybrid memory for domain adaptive object re-id.  In NeurIPS, 2020.
  * [10] Jianzhu Guo, Xiangyu Zhu, Chenxu Zhao, Dong Cao, Zhen Lei, and Stan Z. Li.  Learning meta face recognition in unseen domains.  In CVPR, 2020.
  * [11] Jianzhu Guo, Xiangyu Zhu, Chenxu Zhao, Dong Cao, Zhen Lei, and Stan Z Li.  Learning meta face recognition in unseen domains.  In CVPR, 2020.
  * [12] Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao.  Ms-celeb-1m: A dataset and benchmark for large-scale face recognition.  In ECCV, 2016.
  * [13] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network, 2015.
  * [14] Grant Van Horn and Pietro Perona.  The devil is in the tails: Fine-grained classification in the wild.  CoRR, abs/1709.01450, 2017.
  * [15] Wen Ji, Kelei He, Jing Huo, Zheng Gu, and Yang Gao.  Unsupervised domain attention adaptation network for caricature attribute recognition, 2020.
  * [16] Jianbo Shi and J. Malik.  Normalized cuts and image segmentation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000\.
  * [17] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks, 2017\.
  * [18] Devinder Kumar, Parthipan Siva, Paul Marchwica, and Alexander Wong.  Unsupervised domain adaptation in person re-id via k-reciprocal clustering and large-scale heterogeneous environment synthesis, 2020.
  * [19] Yutian Lin, Xuanyi Dong, Liang Zheng, Yan Yan, and Yi Yang.  A bottom-up clustering approach to unsupervised person re-identification.  Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):8738–8745, Jul. 2019.
  * [20] Y. Lin, Y. Wu, C. Yan, M. Xu, and Y. Yang.  Unsupervised person re-identification via cross-camera similarity exploration.  IEEE Transactions on Image Processing, 29:5481–5490, 2020.
  * [21] Stuart Lloyd.  Least squares quantization in pcm.  In IEEE transactions on information theory, pages 28(2):129–137, 1982.
  * [22] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael I. Jordan.  Learning transferable features with deep adaptation networks.  In ICML, 2015.
  * [23] Z. Luo, J. Hu, W. Deng, and H. Shen.  Deep unsupervised domain adaptation for face recognition.  In IEEE International Conference on Automatic Face Gesture Recognition (FG), 2018.
  * [24] H. Brendan McMahan, Eider Moore, Daniel Ramage, and Blaise Agüera y Arcas.  Federated learning of deep networks using model averaging.  CoRR, abs/1602.05629, 2016.
  * [25] Yichun Shi, Xiang Yu, Kihyuk Sohn, Manmohan Chandraker, and Anil K. Jain.  Towards universal representation learning for deep face recognition.  In CVPR, 2020.
  * [26] Yichun Shi, Xiang Yu, Kihyuk Sohn, Manmohan Chandraker, and Anil K Jain.  Towards universal representation learning for deep face recognition.  In CVPR, 2020.
  * [27] Kihyuk Sohn, Sifei Liu, Guangyu Zhong, Xiang Yu, Ming-Hsuan Yang, and Manmohan Chandraker.  Unsupervised domain adaptation for face recognition in unlabeled videos.  In ICCV, 2017.
  * [28] Liangchen Song, Cheng Wang, Lefei Zhang, Bo Du, Qian Zhang, Chang Huang, and Xinggang Wang.  Unsupervised domain adaptive re-identification: Theory and practice, 2018\.
  * [29] Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei.  Circle loss: A unified perspective of pair similarity optimization.  In CVPR, 2020.
  * [30] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf.  Deepface: Closing the gap to human-level performance in face verification.  In CVPR, 2014.
  * [31] O. Tasar, S. L. Happy, Y. Tarabalka, and P. Alliez.  Colormapgan: Unsupervised domain adaptation for semantic segmentation using color mapping generative adversarial networks.  IEEE Transactions on Geoscience and Remote Sensing, 58(10):7178–7193, 2020.
  * [32] Ilya O Tolstikhin, Bharath K Sriperumbudur, and Bernhard Schölkopf.  Minimax estimation of maximum mean discrepancy with radial kernels.  In Advances in Neural Information Processing Systems, 2016.
  * [33] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.  Adversarial discriminative domain adaptation.  In CVPR, 2017.
  * [34] Feng Wang, Weiyang Liu, Haijun Liu, and Jian Cheng.  Additive margin softmax for face verification.  arXiv preprint arXiv:1801.05599, 2018.
  * [35] Hao Wang, Dihong Gong, Zhifeng Li, and Wei Liu.  Decorrelated adversarial learning for age-invariant face recognition.  In CVPR, 2019.
  * [36] Hui Wang, Jian Tian, Songyuan Li, Hanbin Zhao, Qi Tian, Fei Wu, and Xi Li.  Unsupervised domain adaptation for image classification via structure-conditioned adversarial learning, 2021.
  * [37] Mei Wang and Weihong Deng.  Deep visual domain adaptation: A survey.  arXiv preprint arXiv:1802.03601, 2018.
  * [38] Mei Wang and Weihong Deng.  Mitigating bias in face recognition using skewness-aware reinforcement learning.  In CVPR, 2020.
  * [39] Mei Wang, Weihong Deng, Jiani Hu, Xunqiang Tao, and Yaohai Huang.  Racial faces in the wild: Reducing racial bias by information maximization adaptation network.  In ICCV, 2019.
  * [40] Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao.  A discriminative feature learning approach for deep face recognition.  In ECCV, 2016.
  * [41] Lei Yang, Dapeng Chen, Xiaohang Zhan, Rui Zhao, Chen Change Loy, and Dahua Lin.  Learning to cluster faces via confidence and connectivity estimation.  In CVPR, 2020.
  * [42] Lei Yang, Xiaohang Zhan, Dapeng Chen, Junjie Yan, Chen Change Loy, and Dahua Lin.  Learning to cluster faces on an affinity graph.  In CVPR, 2019.
  * [43] Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven C. H. Hoi.  Deep learning for person re-identification: A survey and outlook, 2021\.
  * [44] Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker.  Feature transfer learning for face recognition with under-represented data.  In CVPR, 2019.
  * [45] Yaoyao Zhong, Weihong Deng, Mei Wang, Jiani Hu, Jianteng Peng, Xunqiang Tao, and Yaohai Huang.  Unequal-training for deep face recognition with long-tailed noisy data.  In CVPR, 2019.
  * [46] Yali Li Zhongdao Wang, Liang Zheng and Shengjin Wang.  Linkage-based face clustering via graph convolution network.  In CVPR, 2019.
  * [47] Weiming Zhuang, Yonggang Wen, Xuesen Zhang, Xin Gan, Daiying Yin, Dongzhan Zhou, Shuai Zhang, and Shuai Yi.  Performance optimization of federated person re-identification via benchmark analysis.  Proceedings of the 28th ACM International Conference on Multimedia, Oct 2020.
  * [48] Weiming Zhuang, Yonggang Wen, Xuesen Zhang, Xin Gan, Daiying Yin, Dongzhan Zhou, Shuai Zhang, and Shuai Yi.  Performance optimization of federated person re-identification via benchmark analysis.  Proceedings of the 28th ACM International Conference on Multimedia, Oct 2020.


# The Magellanic Stream at 20 kpc:
A New Orbital History for the Magellanic Clouds

Scott Lucchini Department of Physics, University of Wisconsin - Madison,
Madison, WI, USA  Elena D’Onghia Department of Physics, University of
Wisconsin - Madison, Madison, WI, USA  Department of Astronomy, University of
Wisconsin - Madison, Madison, WI, USA  Andrew J. Fox AURA for ESA, Space
Telescope Science Institute, 3700 San Martin Drive, Baltimore, MD, USA

###### Abstract

We present new simulations of the formation of the Magellanic Stream based on
an updated first-passage interaction history for the Magellanic Clouds,
including both the Galactic and Magellanic Coronae and a live dark matter halo
for the Milky Way. This new interaction history is needed because previously
successful orbits need updating to account for the Magellanic Corona and the
loosely bound nature of the Magellanic Group. These orbits involve two tidal
interactions over the last 3.5 Gyrs and reproduce the Stream’s position and
appearance on the sky, mass distribution, and velocity profile. Most
importantly, our simulated Stream is only $\sim$20 kpc away from the Sun at
its closest point, whereas previous first-infall models predicted a distance
of $100-200$ kpc. This dramatic paradigm shift in the Stream’s 3D position
would have several important implications. First, estimates of the observed
neutral and ionized masses would be reduced by a factor of $\sim$5\. Second,
the stellar component of the Stream is also predicted to be $<$20 kpc away.
Third, the enhanced interactions with the MW’s hot corona at this small
distance would substantially shorten the Stream’s lifetime. Finally, the MW’s
UV radiation field would be much stronger, potentially explaining the
H$\alpha$ emission observed along most of the Stream. Our prediction of a 20
kpc Stream could be tested by searching for UV absorption lines towards
distant MW halo stars projected onto the Stream.

Galaxy physics (612) — Galaxy dynamics (591) — Magellanic Clouds (990) —
Magellanic Stream (991)

††journal: ApJL††software: Astropy (https://www.astropy.org/; Astropy
Collaboration et al. 2018), gizmo
(http://www.tapir.caltech.edu/~phopkins/Site/GIZMO.html; Hopkins 2015;
Springel 2005), Matplotlib (https://matplotlib.org/), Plotly
(https://plotly.com/), pygad (https://bitbucket.org/broett/pygad/src/master/;
Röttgers et al. 2020)

##  1 Introduction

The Magellanic System is essential to our understanding of the ongoing
formation and evolution of the Local Group. It consists of the two closest
massive dwarf galaxies to the Milky Way (MW), the Large and Small Magellanic
Clouds (LMC and SMC), and the Magellanic Stream, a massive network of gaseous
filaments trailing behind the Clouds (see D’Onghia & Fox 2016 for a review).
Significant theoretical and computational effort has gone into furthering our
understanding of the origin and formation of the Magellanic Stream, beginning
with simple analytical models (e.g. Fujimoto & Sofue, 1977), eventually
including simple hydrodynamics and self-gravity (e.g. Moore & Davis, 1994;
Gardiner & Noguchi, 1996). Modern high-resolution simulations include live
N-body models for both the LMC and the SMC as well as self-consistent
hydrodynamics with radiative cooling and star formation (Besla et al. 2012,
hereafter B12; Hammer et al. 2015; Pardy et al. 2018, hereafter P18). Most
recently, Lucchini et al. (2020) (hereafter L20) showed that including a
Magellanic Corona of warm gas surrounding the LMC and SMC can explain the
ionized gas component of the Magellanic Stream (Fox et al., 2014).

However, as these models have improved, one piece of the puzzle has remained
unconstrained: the exact past orbits of the LMC and SMC. Proper motion (PM)
measurements for the Magellanic Clouds have become very precise (Kallivayalil
et al., 2013; Zivick et al., 2018), but the total mass of the MW and the LMC
are still imprecisely known (D’Onghia & Fox, 2016; Bland-Hawthorn & Gerhard,
2016). The PM measurements favor a first-infall scenario (Besla et al., 2007),
which is supported by the LMC’s wake in the dark matter distribution of the MW
halo (Conroy et al., 2021). The largest uncertainties in the orbits of the
Clouds come from hydrodynamical effects including ram-pressure and tidal
energy losses, which are difficult to include in analytical integrators. Given
the recent indications for the Magellanic Corona (see L20, ), as well as the
need to include the MW’s hot circumgalactic medium (CGM), these hydrodynamical
effects will play a significant role in the orbital history of the Clouds.
Upon the inclusion of the gaseous halos, we find that the Clouds can survive
fewer recent interactions than previously thought (B12) if they are to remain
separated at the present day. Additionally, evidence for a Magellanic Group
(e.g. D’Onghia & Lake, 2008; Nichols et al., 2011) suggests that, being
loosely bound, its two largest members should expect only a couple of direct
interactions within the past $\sim$5$-$7 Gyrs.

In this paper we explore the large-scale structure and location of the Stream
resulting from an alternate first-passage interaction history between the
Clouds as motivated by the existence of the Magellanic Corona. Our new
simulations are consistent with the observed PMs of the Clouds and have
dramatic implications on the 3D location of the Magellanic Stream. In Section
2 we outline the methods and initial conditions used in our simulations. In
Section 3 we discuss our main results, and in Section 4 we dissect the
significant outcomes and implications of the model.

##  2 Methods

For this work, we used the gizmo massively-parallel, multi-physics code
(Hopkins, 2015; Springel, 2005). gizmo employs a Lagrangian meshless finite-
mass (MFM) hydrodynamics scheme that is ideal for simulations with large bulk
velocities and large dynamic ranges in density. The MFM scheme provides the
ability to track individual fluid elements while still capturing Kelvin-
Helmholtz instabilities and shocks (Hopkins, 2015). We used adaptive
gravitational softening lengths for gas particles, and softening lengths of
350 pc and 100 pc for the DM and stellar components respectively.
Additionally, the default cooling (see Appendix B of Hopkins et al., 2018) and
star formation (Springel & Hernquist, 2003) routines in gizmo were included.

We do not include metal-line cooling, time-variable ionizing radiation from
the MW disk, or UV background radiation in our model, because although these
complex mechanisms would influence the Stream’s thermal state and ionization
level, we expect they would not affect its location, which is the focus of
this paper. Following L20, we assume the cold material (the H I Stream)
originates from the disks of the LMC and SMC (the Magellanic ISM) whereas the
warm ionized material originates from the Magellanic Corona.

Table 1: Initial and Final Properties of the Galaxies in the Simulation.

| MW | LMC | SMC
---|---|---|---
$v_{200}$ (km s-1) | 166.1 | 92.72 | 45.22
DM Concentrationa | 12 | 9 | 15
DM Mass ($\mathrm{M_{\odot}}$) | $10^{12}$ | $1.8\times 10^{11}$ | $1.9\times 10^{10}$
Stellar Mass ($\mathrm{M_{\odot}}$) | $4.8\times 10^{10}$ | $5\times 10^{9}$ | $2.6\times 10^{8}$
Stellar Scale Length (kpc) | 2.4 | 0.9 | 0.8
Disk Gas Mass ($\mathrm{M_{\odot}}$) | $10^{10}$ | $5\times 10^{9}$ | $1.6\times 10^{9}$
Gas Scale Length (kpc) | 7.0 | 2.8 | 2.0
Corona Mass ($\mathrm{M_{\odot}}$) | $10^{11}$ | $8.3\times 10^{9}$ | $-$
Corona Temp (K) | $2.4\times 10^{6}$ | $2.4\times 10^{5}$ | $-$
N Particles | $7.8\times 10^{6}$ | $2.6\times 10^{6}$ | $5.3\times 10^{5}$
Initial Position (kpc) | (0, 0, 0) | (47.36, 546.38, 150.52) | ($-$19.79, 412.29, 183.75)
Initial Velocity (km s-1) | (0, 0, 0) | (1.71, $-$99.02, $-$63.73) | (13.43, $-$77.21, $-$80.33)
Sim Position (kpc) | $-$ | (2.0, $-$40.8, $-$31.0) | (13.3, $-$39.3, $-$45.4)
Observed Position (kpc) | $-$ | ($-$1.0, $-$40.9, $-$27.7)e | (14.9, $-$38.1, $-$44.2)e
Sim Velocity (km s-2) | $-$ | ($-$101.1, $-$275.8, 229.2) | ($-$89.8, $-$300.1, 168.6)
Observed Velocity (km s-1) | $-$ | ($-57\pm 13$, $-226\pm 15$, $221\pm 19$)e | ($18\pm 6$, $-179\pm 16$, $174\pm 13$)f
Sim PMb,c (mas yr-1) | $-$ | ($-2.18\\!\pm\\!0.02\\!\pm\\!0.23$, $0.20\\!\pm\\!0.01\\!\pm\\!0.28$) | ($-1.10\\!\pm\\!0.02\\!\pm\\!0.45$, $-0.93\\!\pm\\!0.02\\!\pm\\!0.49$)
Observed PMb (mas yr-1) | $-$ | ($-1.91\pm 0.02$, $0.23\pm 0.05$)e | ($-0.83\pm 0.12$, $-1.21\pm 0.04$)f
Sim RVd (km s-1) | $-$ | $262.2\pm 6.2\pm 12.5$ | $165.8\pm 5.0\pm 32.0$
Observed RV (km s-1) | $-$ | $262.2\pm 3.4^{g}$ | $145.6\pm 0.6^{h}$

Notes. The resultant galaxies have rotation curve peaks of $\sim$240,
$\sim$120, and $\sim$65 km s-1 at 12, 8.5, and 5 kpc for the MW, LMC, and SMC
respectively.
a The DM concentration parameter, $c$, is defined as the ratio of the virial
radius, $R_{\mathrm{vir}}$, to the NFW scale radius, $R_{s}$:
$R_{\mathrm{vir}}=cR_{s}$. The NFW scale radius is then converted into a
Hernquist scale radius with $a=R_{s}\sqrt{2\left(\log(1+c)-c/(1+c)\right)}$.
b Proper motions given as ($\mu_{W}$, $\mu_{N}$) following the convention in
Kallivayalil et al. (2013).
c Simulation PM errors are given as ($\mu_{W,N}\pm$err${}_{\odot}\pm$errres),
where err⊙ is the error due to the propagation of the uncertainties in the
observed solar velocity and location via the bootstrapping method (solar
values and errors from Kallivayalil et al. 2013, section 5). errres is an
approximation of the variability in kinematics of the Clouds due to small-
scale power effects, which change with the numerical resolution. It is
computed by measuring the standard deviation of the resultant PMs and RVs in
simulations of three different resolutions.
d Simulation radial velocity errors are given as
($v_{\mathrm{rad}}\pm$err${}_{\odot}\pm$errres) with err⊙ and errres as
defined above in $c$.
e Kallivayalil et al. (2013); f Zivick et al. (2018); g van der Marel et al.
(2002); h Harris & Zaritsky (2006)

###  2.1 Initial Conditions

The simulation presented here contains the same components as those used in
L20, except we have added a live dark matter (DM) halo for the MW. In brief,
those components are galaxies with stellar and gaseous exponential disks
embedded in live Hernquist profile DM halos following the methods outlined in
Springel et al. (2005), and gaseous coronae around the LMC and MW. Table 1
outlines the parameters used to generate the initial setup of the simulation.

Based on recent measurements (Peñarrubia et al., 2016; Erkal et al., 2019),
and the mounting evidence for a Magellanic Group (e.g. D’Onghia & Lake, 2008;
Nichols et al., 2011), we include a high-mass LMC in our model. It has a total
DM mass of $1.8\times 10^{11}~{}\mathrm{M_{\odot}}$. Our SMC is consistent
with previous works (; ) with a DM mass of $1.9\times
10^{10}~{}\mathrm{M_{\odot}}$. We use the MW model of D’Onghia & Aguerri
(2020) with a gaseous component added to the disk. The total number of
particles used for each galaxy is given in row 10 of Table 1, and leads to
masses per particle ranging from $1.8\times 10^{5}$ to $7.0\times
10^{5}~{}\mathrm{M_{\odot}}$ for DM, $4.2\times 10^{3}$ to $1.6\times
10^{4}~{}\mathrm{M_{\odot}}$ for stars, and $4.3\times 10^{3}$ to $1.8\times
10^{4}~{}\mathrm{M_{\odot}}$ for gas.

Following L20, we included a Magellanic Corona around the LMC and SMC by
extracting the radial density profile from an LMC analogue in the Auriga
cosmological simulations (Grand et al., 2017), although we used slightly
different selection criteria. Specifically, we selected all gas with
$T>2.5\times 10^{4}$ K, the lowest temperature at which the galaxy’s disk gas
was excluded from the selection. The corona was then added around our LMC with
its total mass scaled by the ratio between the Auriga LMC analogue’s total
mass and our LMC’s total mass, leading to a mass of $8.3\times
10^{9}~{}\mathrm{M_{\odot}}$ in the Magellanic Corona and a density of
$10^{-4}$ cm-3 at 50 kpc from the LMC. Finally the corona’s temperature was
set to $2.4\times 10^{5}$ K, the expected virial temperature of the LMC. These
values are also consistent with other recent cosmological simulations (Hafen
et al., 2019; Jahn et al., 2021). Run in isolation for 4 Gyrs, the LMC and
Magellanic Corona remain stable.

Around the MW, we initially included a gaseous corona following the “fiducial”
density profile in Salem et al. (2015): a $\beta$-profile with $n_{0}=0.46$
cm-3, $r_{c}=0.35$ kpc, and $\beta=0.559$. The maximum density was capped
within 13 kpc from the Galactic Center where the corona overlapped with the
gaseous disk. Additionally, the profile exponentially declines for
$r>r_{\mathrm{vir}}=166$ kpc. This initial simulation was unable to reproduce
the velocity profile of the Stream (see Section 3), so in the final simulation
presented here, the total mass was increased by a factor of two to $4\times
10^{10}~{}\mathrm{M_{\odot}}$, which solved the kinematic discrepancy. This
increased the MW halo density at 50 kpc from $1.1\times 10^{-4}$ cm-3 to
$2.0\times 10^{-4}$ cm-3, still consistent with current data (e.g. Anderson &
Bregman 2010; Li & Bregman 2017). We acknowledge that the Galactic and
Magellanic Coronae included in these simulations are simplified when compared
with the complex multiphase intricacies known to exist in circumgalactic media
(van de Voort et al., 2019).

###  2.2 Orbits

The orbits used in previous simulations of Magellanic Stream formation were
established without the Galactic or Magellanic Coronae included (; ). While
the Cloud orbits will clearly require modification due to the changes in total
masses of the galaxies due to these added components, the increased friction
and ram pressure that the Clouds experience as they move through these media
also play a significant role. Therefore, to match the observed positions and
velocities of the Magellanic Clouds while including the Galactic and
Magellanic Coronae in our simulations, we need to determine an alternate
orbital history with fewer recent interactions between the Clouds.

To do this, we analytically integrated the orbits of the LMC, SMC, and MW
backwards in time starting from their present day observed positions and
velocities. We included radially-extended Hernquist DM halos including
dynamical friction and represented the stellar and gaseous disks as point
particles. By varying the present-day velocities within the observed errors,
we obtained a suite of 1,458 possible orbits. We filtered these to select
first-passage orbits with multiple interactions between the Clouds. These
orbits are generally consistent with the results found with previous analytic
integrations after accounting for differences in LMC mass and live vs static
MW DM halos (Besla et al., 2007; Kallivayalil et al., 2013; Garavito-Camargo
et al., 2019). We then chose ten of these orbits sampling the parameter space
with varying morphologies to run in full N-body hydrodynamical simulations.
Due to the effects of tidal stripping, ram pressure, and friction, the initial
conditions required modification to match the present-day observations of the
Clouds. For nine of the ten chosen orbits, we were unable to modify them such
that they reproduced the present-day observations. However, after a few
iterations on one of the orbits, we found a solution that could match the
Clouds’ observed kinematic properties, which we present in this Letter. The
initial positions and velocities used for the orbits in this simulation are
given in rows 11 and 12 of Table 1.

Given the backwards-integrated orbits, we had to choose when to begin the
N-body simulation. We chose the apocenter between the Clouds after their
second interaction (3.5 Gyrs ago). While the Clouds would have another
encounter if we integrate these orbits further back in time, this interaction
would occur 7 Gyrs ago (5.5 Gyrs before their next interaction) at a distance
of 0.9 Mpc away from the MW. Any gas stripped in this interaction would be
tidally thrown out to great distances from the Clouds and become too diffuse
and distant to contribute to the Stream today (see material stripped from the
first interaction in Figure 2 of P18).

####  2.2.1 Comparison with Previous Orbits

In the previous orbital model of B12 and P18 (B12’s “Model 2”), the LMC and
the SMC experience three interactions in isolation (without any MW influence)
over $\sim$6 Gyrs. They are then rotated into the correct orientation, placed
at the virial radius of the MW, and allowed to continue evolving until they
reach their present day positions ($\sim$1 Gyr). During this infall they
experience an additional direct collision which forms the Magellanic Bridge.

In the model presented here, all the interactions between the Clouds and the
MW occur in a single, self-consistent simulation lasting 3.5 Gyrs. The LMC and
the SMC have two interactions, the second of which has very low impact
parameter and forms the Magellanic Bridge. The main differences between these
two orbital models are (see Figure 1b):

  * •

the number of interactions (4 vs. 2),

  * •

the length of the simulation (7 Gyrs vs. 3.5 Gyrs),

  * •

the maximum separation between the Clouds (100 kpc vs. 150 kpc), and

  * •

the sense of the SMC’s orbit around the LMC (see Section 4 and Figure 4).

While there have been many other proposed orbital models of the interactions
between the Clouds (e.g. Diaz & Bekki, 2011; Růžička et al., 2010; Guglielmo
et al., 2014), none have looked at this explicit combination especially when
considering a first-passage scenario with the inclusion of a MW CGM and the
Magellanic Corona.

Figure 1: The orbital history of the Magellanic Clouds. Panel (a) shows the
Magellanic Stream in cartesian coordinates at three different times during the
Clouds’ infall. The present day Stream is shown in color. The MW disk is at
the origin and denoted by the gray shaded oval (the sun is located at
$(x,y,z)=(-8.3,0,0.027)$ kpc). The solid and dashed lines represent the past
orbital trajectories of the LMC and SMC respectively. A zoomed inset of the
present day Stream can be seen in the panel on the left with the locations of
the Clouds in the simulation marked with circles. The color of the gas
represents plane-projected density on an arbitrary scale with higher densities
represented as lighter colors. Panel (b) shows the distance between the center
of masses of the LMC and SMC in kpc as they orbit around each other and fall
in towards the MW. The model presented here is compared with previous works
(shown as a dotted line) (; ). Present day is on the right of the plot
($t=0$), and the Clouds’ initial state is on the left ($t=-3.5$ Gyrs for the
present work). Vertical dashed lines denote the times of the two past images
in panel (a) (shown in greyscale). An animation of this figure is available
online. It is 9 seconds long and shows the evolution of the Clouds and the
Stream over the past 3.5 Gyrs.

##  3 Results

The simulation ran for 3.46 Gyrs, when the positions and velocities of the
Clouds matched current observations just after their first pericentric
passage. When the LMC and SMC are at their present-day sky positions in our
model, they are at distances of $52.1\pm 1.7$ kpc and $78.0\pm 7.8$ kpc
respectively (with errors calculated as $\mathrm{err}_{\mathrm{res}}$
discussed in footnote $c$ of Table 1; $\mathrm{err}_{\odot}$ are negligible),
with proper motions and radial velocities listed in Table 1. The kinematics of
the Clouds are fully consistent with the observed values: within 1 sigma for
the proper motions and radial velocity for both the LMC and the SMC (with the
exception of $\mu_{W}$ for the LMC at 1.18$\sigma$). Additionally, the
relative velocity between the Clouds (66 km s-1) matches within 2$\sigma$.

The present day disk gas masses111These masses were calculated in physical 3D
space by summing the particle masses within spheres centered on each galaxy
with diameters 13.5 kpc for the LMC, and 5.5 kpc for the SMC. for the LMC and
SMC are $4.4\times 10^{8}$ and $3.3\times 10^{7}~{}\mathrm{M_{\odot}}$
respectively (with peak column densities of $10^{21.8}$ and $10^{21.4}$ cm-2).
A full exploration of the structure of the Clouds themselves will be performed
in an upcoming paper with more complete star formation and feedback routines
(Lucchini et al., in prep).

Throughout their 3.5 Gyr interaction history, a trailing Stream is formed
through tidal interactions between the Clouds. The orbital paths of the LMC
and SMC along with the resultant Stream at the present day are shown projected
onto the $y$–$z$ plane relative to the MW disk in Figure 1a. The LMC and SMC
experience two close encounters shown as minima in Figure 1b. Their first
interaction, 1.4 Gyrs ago with an impact parameter of 9.9 kpc, provides the
tidal forces necessary to strip material from the SMC to create the bulk of
the H I Stream. Their second interaction, 295 Myrs ago, has a significantly
lower impact parameter of 3.0 kpc, and this direct collision forms the
Magellanic Bridge. The present-day position of the Magellanic System can also
be seen in 3D in Figure 2.

Figure 2: A 3D model of the Magellanic Stream. The orbits of the LMC and SMC
are shown as solid and dashed lines respectively. The MW disk is shown in gray
with the location of the Sun marked with a yellow sphere. An interactive
version of this figure is accessible online as an interactive HTML document.

To test the dominant stripping mechanism in our model, we ran two additional
simulations: one consisting of just the MW, its hot corona, and the SMC
(without the LMC), and another with the MW, its hot corona, and the LMC with
the Magellanic Corona (without the SMC). When the SMC alone passes through the
MW’s CGM, we find only 8% of the total gas mass stripped from the full model.
When the LMC alone passes through, we find negligible stripping of gas from
the LMC, but the Magellanic Corona still sees 95% of the stripping of the full
model. Therefore, we conclude that the neutral Stream is stripped through
tidal forces (consistent with previous findings, e.g. Salem et al. 2015),
whereas the ionized component of the Stream (originating in the Magellanic
Corona) is stripped mostly through ram-pressure against the MW hot corona,
even during the first passage.

Figure 3: Properties of the Magellanic Stream produced in our simulations.
Panels (a) and (b) show the observed and simulated Stream respectively in
zenithal equal-area coordinates with line-of-sight velocity indicated by the
color scale and the relative gas column density indicated by the brightness.
The H I data in Panel (a) is from the GASS survey (McClure-Griffiths et al.,
2009) with the points showing sightlines with UV-absorption-line observations
from the _Hubble Space Telescope_ (Fox et al., 2014) colored by their line-of-
sight velocity. Panel (c) shows the column density of the total gas in the
Stream (including the ionized Magellanic Corona and neutral Magellanic disk
components) in Magellanic Coordinates ($l_{\mathrm{MS}}$ and
$b_{\mathrm{MS}}$). Panel (d) only shows the neutral gas originating in the
disks of the LMC and SMC compared to the observed data from Nidever et al.
(2010) shown in contours (black, grey, and white correspond to $10^{21}$,
$10^{20}$, and $10^{19}$ cm-2). The centroids of the LMC and SMC stellar disks
are marked by white circles. Panel (e) shows the local standard of rest (LSR)
velocity gradient along the Stream with data shown as contours (Nidever et
al., 2010). Panel (f) shows the line-of-sight distance to the gas in the
Stream along its length with the centroids of the Clouds marked with circles.
Note that the bulk of the simulated Stream is significantly closer to us than
the Magellanic Clouds are ($\sim$20 kpc vs $\sim$60 kpc).

As in L20, the bulk of the mass of the Magellanic Stream is composed of
ionized gas originating in the Magellanic Corona. However, the distribution of
the ionized Magellanic Corona gas on the sky is substantially different in our
new model (compare Figure 3c with L20 Figure 2a), but the neutral Stream’s
appearance on the sky in our model is generally consistent with previous
models and observations (Figure 3a,b,d). While the simulated Stream is longer
than observed and slightly offset spatially, the morphology of the Stream in
this model more accurately reproduces the turbulent, filamentary nature of the
data. Due to its interactions with the Magellanic Corona and the MW’s hot CGM,
instabilities fragment and distort the Stream, leading to a significant
improvement its appearance and morphology when compared to models that do not
include these gaseous components (e.g. P18, ).

The velocity profile of our simulated Stream also matches observations (see
Figure 3e). Previous models found a velocity gradient too shallow when
compared to the data (L20), but this is resolved in our new model. As stated
in Section 2.1, the density profile and total mass of the MW hot corona was
increased by a factor of two over the Salem et al. (2015) values to better
match the velocity gradient along the Stream. If the MW’s hot halo is not
massive enough, the stripped gas from the Clouds is accelerated towards the MW
and in some cases ends up with greater velocities than the LMC and SMC
themselves. A higher gas density around the MW provides the ram-pressure
forces to slow down the trailing Stream to match the observed velocity
gradient. This is in contrast to previous works that placed upper limits on
the MW coronal mass such that a Leading Arm can form (Tepper-García et al.
2019; L20).

Additionally, as seen in Figure 3d, this model does not self-consistently
reproduce the Leading Arm gas. This is because the Clouds only have two close
encounters so there is not enough time for gas to be tidally thrown ahead of
the Clouds in their orbits. This is in contrast to previous studies (Tepper-
García et al. 2019; L20) where the lack of a Leading Arm was due to the MW hot
coronal density being too high. In this new model, even without a MW hot
corona, a Leading Arm is not formed. The true nature of the Leading Arm is one
of the biggest outstanding questions in the Magellanic System (D’Onghia & Fox,
2016). While a Magellanic origin is supported by the kinematics (Putman et
al., 1998) and the metallicities (although they vary with position; Fox et
al., 2018), several works have proposed alternative, non-Magellanic sources.
Hammer et al. (2015) and Tepper-García et al. (2015) suggested the Leading Arm
structures could be remnants from dwarf spheroidal satellites of the MW whose
gas has been stripped from the MW hot corona, and a non-Magellanic explanation
for the Leading Arm remains a possibility.

The most notable implication of this new model is that the Stream is
significantly closer to us than previously thought (Figure 3f). While some
past models have predicted a close Stream (via multiple passages around the MW
e.g. Moore & Davis 1994; Diaz & Bekki 2012), all previous first-passage orbits
have resulted in the Stream flowing behind the Clouds out to distances of
$100-200$ kpc or greater (see Figure 4e; ; ; ). The first-passage model
presented here forms a tidal Stream that reaches as close as $\sim$20 kpc away
from the Sun with a column density-weighted average distance of 24.7 kpc
(between Magellanic longitude of $-25^{\circ}$ and $-150^{\circ}$). See
Section 4 for an in-depth discussion of this finding.

We emphasize that in all our simulations that formed a Stream (seven of the
ten selected from the backwards integration, see Section 2.2), that Stream was
$<$50 kpc away from us. While the distances and kinematics of the Clouds, and
the Stream morphology, vary greatly between these individual simulation runs,
the finding that the Stream remains nearby is a robust prediction of our
first-infall models. Additionally, we have run several convergence tests at
various numerical resolutions, and despite minor differences on small scales,
all runs produced accurate positions and velocities for the Clouds, and all
predicted a nearby Stream with total ionized and neutral masses consistent
with observations.

Figure 4: Orbital schematics and distance to the Magellanic Stream compared
for two different models. The left column (panels (a), (c) and (e)) shows the
clockwise orbit of previous works (; ; ), while the right column (panels (b),
(d) and (f)) shows the counter-clockwise orbit used in the model presented
here. In panels (a) and (b), the orbital path of the SMC (red) around the LMC
(blue) is shown at three different times in the $y$–$z$ plane: before the
interaction (top), at apoapsis when the Stream material is stripped out of the
SMC (middle), and at their present day orientation (bottom). Panels (c) and
(d) show the present day positions of the Clouds and the Stream in the two
models with respect to the MW, again in the $y$–$z$ plane. The arrow shows the
direction of motion of the Clouds around the MW. Panels (e) and (f) both show
line-of-sight distance to the gas in the simulated Stream (grey) and the stars
in the Stream (orange). In previous orbital models (left column), the Stream
stretches out and away from the MW leading to distances of 100–200 kpc.
Whereas in the new orbital history presented here (right column), the Stream
is stripped during the SMC’s counter-clockwise motion around the LMC and then
pushed into place through the ram pressure and frictional forces of the MW’s
CGM leading to distances of as little as $\sim$20 kpc.

##  4 Discussion & Conclusions

The new first-passage interaction history of the Magellanic Clouds presented
here leads to a dramatically different 3D spatial positioning of the Stream
than previous models. Previous first-passage, tidal simulations led to a
Stream extending away from the Clouds out to distances upwards of 200 kpc,
whereas our new model results in the Stream angling up towards the MW reaching
as close as 20 kpc to the Sun. While there are many differences between
previous models and the model presented here (see Section 2.2.1), two
differences in particular lead to this dramatic shift in positioning of the
Stream:

  1. 1.

a qualitative difference in the SMC’s orbit around the LMC, and

  2. 2.

the inclusion of the Galactic and Magellanic Coronae.

First, the orientation of the SMC’s orbit around the LMC is qualitatively
different than the most recent previous models (; ; ). To compare these
different orbital histories, we need to analyze the orbits in a consistent
coordinate system, so we will discuss the relative motion of the SMC around
the LMC when viewed projected onto the $y$–$z$ plane as defined in Figure 1
(relative to the MW disk which is in the $x$–$y$ plane with the Sun located at
$(x,y,z)=(-8.3,0,0.027)$ kpc)222Note that this is the same perspective as Fig.
3 in B12 but is not consistent with the coordinate systems in B12’s Fig. 2,
L20’s Extended Data Fig. 2, or P18’s Fig. 2. Because of rotations performed
before the Clouds fall into the MW potential in these models, the $y$–$z$
perspective discussed above is approximately equivalent to mirroring these
figures across the $y$-axis.. When viewed from this perspective, our model has
the SMC on a counter-clockwise orbit around the LMC, whereas in the B12 model,
the SMC rotates around the LMC clockwise. In the clockwise orbit, the Stream
is tidally thrown out in the $+y$ direction with a velocity in the $-z$
direction, leading to it stretching away from the MW disk. Whereas in the
counter-clockwise orbit, the Stream is still tidally stripped in the $+y$
direction, but its velocity is in the $+z$ direction. This leads to a Stream
angled up in the $+y$ and $+z$ directions, resulting in low line-of-sight
distances to the Sun. A schematic of these two orbital orientations is shown
in Figure 4a-d.

Second, the inclusion of the Galactic and Magellanic Coronae is crucial. While
previous parameter space searches of orbits for the Clouds would possibly have
explored this orbital configuration in the past (e.g. Růžička et al., 2010;
Guglielmo et al., 2014), none included the Galactic and Magellanic Coronae.
These are key elements as the ram pressure and friction from the MW’s CGM is
able to “push” the Stream into its present day position and velocity. Without
the MW’s gaseous halo, the Stream would collide with the MW disk before the
Clouds reach their present day positions. Moreover, the Magellanic Corona is
required to shield the neutral Stream from the intense forces and pressures as
it moves through high density regions of the Galactic corona L20.

A Stream reaching $\sim$20 kpc has a number of implications. First, the total
observed mass (neutral and ionized) of the Stream would be reduced. Total mass
estimates of the Stream have previously assumed the Stream has a similar
distance as the Clouds (55 kpc), but they include a scale factor of
$(d/55~{}\mathrm{kpc})^{2}$ which equals 0.2 for $d=24.7$ kpc (the column
density-weighted average distance in the simulation). This leads to new values
of $9.7\times 10^{7}~{}\mathrm{M_{\odot}}$ of neutral gas (Brüns et al., 2005)
and $4\times 10^{8}~{}\mathrm{M_{\odot}}$ of ionized gas (Fox et al., 2014),
although this ionized gas mass should be considered as a lower limit as the
spatial extent of the Stream’s ionized phase may be significantly greater than
the area on the sky that has currently been explored, and the ionized gas may
be multi-phase (Fox et al., 2014). Summing the masses of all gravitationally
unbound particles that fall in the region of the trailing Stream in our
simulation gives values of $2.0\times 10^{8}~{}\mathrm{M_{\odot}}$ (neutral)
and $3.2\times 10^{9}~{}\mathrm{M_{\odot}}$ (ionized) for our model of the
Stream, in good agreement with the observations. Second, as shown in Figure
4f, the stellar component of the Stream is also closeby, with stars predicted
at $d\lesssim 20$ kpc (with a mean surface brightness of 31 mag arcsec-2).
Previous works have predicted that any stars associated with the Magellanic
System are at large distances (; ), but our new predictions suggest that
continued searches for the Stream’s stellar component are worthwhile (Zaritsky
et al., 2020). Third, the interaction between the Stream and the MW CGM will
be enhanced due to its proximity, because of the higher MW corona density. At
a closer distance, the Stream would be closer to pressure equilibrium with the
MW CGM, which could help explain its multi-phase nature (Wolfire et al., 1995)
and the high number of “head-tail” clouds seen in high-resolution H I
observations (For et al., 2014). It could also lead to dramatically shorter
lifetimes for the Stream in the future (Murali, 2000; Bland-Hawthorn et al.,
2007). Indeed, due to its angle of approach with respect to the MW, the Stream
may even directly collide with the MW disk within the next $\sim$50 Myrs. This
moves the timescale of gas accretion onto the disk up by a factor of $\sim$10
from previous predictions. The new distance to the Stream therefore implies a
factor of $\sim$8 less gas accreting onto the MW $\sim$10 times earlier than
previously thought, resulting in approximately the same infall rate of
$\sim$4–7 $\mathrm{M_{\odot}}$ yr-1 as derived before (Fox et al., 2014).
Fourth, a closer Stream should be subject to a more intense UV radiation field
from the MW and hence significantly brighter in H$\alpha$. This could explain
the high observed H$\alpha$ emission from Stream (Bland-Hawthorn et al., 2013,
2019; Barger et al., 2017); at a distance of 20 kpc, the Galactic UV
background could lead to H$\alpha$ emission as high as $150-300$ mR (Tepper-
García et al., 2015), in agreement with observed levels along most of the
Stream. However, this increased radiation still cannot explain the extremely
high H$\alpha$ emission seen in the region of the Stream under the South
Galactic Pole, which may indicate a recent Seyfert flare from the Galactic
Center (Bland-Hawthorn et al., 2019). The enhanced radiation field intensity
at $d$=20 kpc would also affect the ionization level inferred from UV metal-
line studies of the Stream (Fox et al., 2014, 2020); however, the
ionized/neutral ratio of $\sim$3:1 is distance-independent since both the
neutral and ionized masses scale as $d^{2}$.

To test our prediction of a 20 kpc Stream, UV or optical spectroscopic studies
could be performed to look for absorption at Magellanic velocities towards
distant MW halo stars, such as Blue Horizontal Branch stars with distances
from _Gaia_. Lehner & Howk (2011) found no UV absorption at Magellanic
velocities when looking at 28 halo stars out to $z$-distances of 12.6 kpc, but
searches to larger distances using new stellar catalogs are needed. This
offers a pathway for confirming our prediction of a nearby Stream.

We thank the anonymous referee for their insightful comments. We thank Bart
Wakker for useful discussions and Jay Gallagher and Snez̆ana Stanimirović for
comments on the manuscript. This work was partially supported by HST Grant
HST-AR-16363.001-A. Additional support was provided by NASA under Award No.
RFP21_4.0 issued through Wisconsin Space Grant Consortium, and the the
University of Wisconsin - Madison Office of the Vice Chancellor for Research
and Graduate Education with funding from the Wisconsin Alumni Research
Foundation. This research was performed using the compute resources and
assistance of the UW-Madison Center For High Throughput Computing (CHTC) in
the Department of Computer Sciences. The CHTC is supported by UW-Madison, the
Advanced Computing Initiative, the Wisconsin Alumni Research Foundation, the
Wisconsin Institutes for Discovery, and the National Science Foundation, and
is an active member of the Open Science Grid, which is supported by the
National Science Foundation and the U.S. Department of Energy’s Office of
Science.

## References

  * Anderson & Bregman (2010) Anderson, M. E., & Bregman, J. N. 2010, ApJ, 714, 320, doi: 10.1088/0004-637X/714/1/320
  * Astropy Collaboration et al. (2018) Astropy Collaboration, Price-Whelan, A. M., Sipőcz, B. M., et al. 2018, AJ, 156, 123, doi: 10.3847/1538-3881/aabc4f
  * Barger et al. (2017) Barger, K. A., Madsen, G. J., Fox, A. J., et al. 2017, ApJ, 851, 110, doi: 10.3847/1538-4357/aa992a
  * Besla et al. (2007) Besla, G., Kallivayalil, N., Hernquist, L., et al. 2007, ApJ, 668, 949, doi: 10.1086/521385
  * Besla et al. (2012) —. 2012, MNRAS, 421, 2109, doi: 10.1111/j.1365-2966.2012.20466.x
  * Bland-Hawthorn & Gerhard (2016) Bland-Hawthorn, J., & Gerhard, O. 2016, ARA&A, 54, 529, doi: 10.1146/annurev-astro-081915-023441
  * Bland-Hawthorn et al. (2013) Bland-Hawthorn, J., Maloney, P. R., Sutherland, R. S., & Madsen, G. J. 2013, ApJ, 778, 58, doi: 10.1088/0004-637X/778/1/58
  * Bland-Hawthorn et al. (2007) Bland-Hawthorn, J., Sutherland, R., Agertz, O., & Moore, B. 2007, ApJ, 670, L109, doi: 10.1086/524657
  * Bland-Hawthorn et al. (2019) Bland-Hawthorn, J., Maloney, P. R., Sutherland, R., et al. 2019, ApJ, 886, 45, doi: 10.3847/1538-4357/ab44c8
  * Brüns et al. (2005) Brüns, C., Kerp, J., Staveley-Smith, L., et al. 2005, A&A, 432, 45, doi: 10.1051/0004-6361:20040321
  * Conroy et al. (2021) Conroy, C., Naidu, R. P., Garavito-Camargo, N., et al. 2021, Nature, 592, 534–536, doi: 10.1038/s41586-021-03385-7
  * Diaz & Bekki (2011) Diaz, J., & Bekki, K. 2011, MNRAS, 413, 2015, doi: 10.1111/j.1365-2966.2011.18289.x
  * Diaz & Bekki (2012) Diaz, J. D., & Bekki, K. 2012, ApJ, 750, 36, doi: 10.1088/0004-637X/750/1/36
  * D’Onghia & Aguerri (2020) D’Onghia, E., & Aguerri, J. A. L. 2020, ApJ, 890, 117, doi: 10.3847/1538-4357/ab6bd6
  * D’Onghia & Fox (2016) D’Onghia, E., & Fox, A. J. 2016, ARA&A, 54, 363, doi: 10.1146/annurev-astro-081915-023251
  * D’Onghia & Lake (2008) D’Onghia, E., & Lake, G. 2008, ApJ, 686, L61, doi: 10.1086/592995
  * Erkal et al. (2019) Erkal, D., Belokurov, V., Laporte, C. F. P., et al. 2019, MNRAS, 487, 2685, doi: 10.1093/mnras/stz1371
  * For et al. (2014) For, B. Q., Staveley-Smith, L., Matthews, D., & McClure-Griffiths, N. M. 2014, ApJ, 792, 43, doi: 10.1088/0004-637X/792/1/43
  * Fox et al. (2020) Fox, A. J., Frazer, E. M., Bland-Hawthorn, J., et al. 2020, ApJ, 897, 23, doi: 10.3847/1538-4357/ab92a3
  * Fox et al. (2014) Fox, A. J., Wakker, B. P., Barger, K. A., et al. 2014, ApJ, 787, 147, doi: 10.1088/0004-637X/787/2/147
  * Fox et al. (2018) Fox, A. J., Barger, K. A., Wakker, B. P., et al. 2018, ApJ, 854, 142, doi: 10.3847/1538-4357/aaa9bb
  * Fujimoto & Sofue (1977) Fujimoto, M., & Sofue, Y. 1977, A&A, 61, 199
  * Garavito-Camargo et al. (2019) Garavito-Camargo, N., Besla, G., Laporte, C. F. P., et al. 2019, ApJ, 884, 51, doi: 10.3847/1538-4357/ab32eb
  * Gardiner & Noguchi (1996) Gardiner, L. T., & Noguchi, M. 1996, MNRAS, 278, 191, doi: 10.1093/mnras/278.1.191
  * Grand et al. (2017) Grand, R. J. J., Gómez, F. A., Marinacci, F., et al. 2017, MNRAS, 467, 179, doi: 10.1093/mnras/stx071
  * Guglielmo et al. (2014) Guglielmo, M., Lewis, G. F., & Bland-Hawthorn, J. 2014, MNRAS, 444, 1759, doi: 10.1093/mnras/stu1549
  * Hafen et al. (2019) Hafen, Z., Faucher-Giguère, C.-A., Anglés-Alcázar, D., et al. 2019, MNRAS, 488, 1248, doi: 10.1093/mnras/stz1773
  * Hammer et al. (2015) Hammer, F., Yang, Y. B., Flores, H., Puech, M., & Fouquet, S. 2015, ApJ, 813, 110, doi: 10.1088/0004-637X/813/2/110
  * Harris & Zaritsky (2006) Harris, J., & Zaritsky, D. 2006, AJ, 131, 2514, doi: 10.1086/500974
  * Hopkins (2015) Hopkins, P. F. 2015, MNRAS, 450, 53, doi: 10.1093/mnras/stv195
  * Hopkins et al. (2018) Hopkins, P. F., Wetzel, A., Kereš, D., et al. 2018, MNRAS, 480, 800, doi: 10.1093/mnras/sty1690
  * Jahn et al. (2021) Jahn, E. D., Sales, L. V., Wetzel, A., et al. 2021, arXiv e-prints, arXiv:2106.03861.  https://arxiv.org/abs/2106.03861
  * Kallivayalil et al. (2013) Kallivayalil, N., van der Marel, R. P., Besla, G., Anderson, J., & Alcock, C. 2013, ApJ, 764, 161, doi: 10.1088/0004-637X/764/2/161
  * Lehner & Howk (2011) Lehner, N., & Howk, J. C. 2011, Science, 334, 955, doi: 10.1126/science.1209069
  * Li & Bregman (2017) Li, Y., & Bregman, J. 2017, ApJ, 849, 105, doi: 10.3847/1538-4357/aa92c6
  * Lucchini et al. (2020) Lucchini, S., D’Onghia, E., Fox, A. J., et al. 2020, Nature, 585, 203, doi: 10.1038/s41586-020-2663-4
  * McClure-Griffiths et al. (2009) McClure-Griffiths, N. M., Pisano, D. J., Calabretta, M. R., et al. 2009, ApJS, 181, 398, doi: 10.1088/0067-0049/181/2/398
  * Moore & Davis (1994) Moore, B., & Davis, M. 1994, MNRAS, 270, 209, doi: 10.1093/mnras/270.2.209
  * Murali (2000) Murali, C. 2000, ApJL, 529, L81, doi: 10.1086/312462
  * Nichols et al. (2011) Nichols, M., Colless, J., Colless, M., & Bland-Hawthorn, J. 2011, ApJ, 742, 110, doi: 10.1088/0004-637X/742/2/110
  * Nidever et al. (2010) Nidever, D. L., Majewski, S. R., Butler Burton, W., & Nigra, L. 2010, ApJ, 723, 1618, doi: 10.1088/0004-637X/723/2/1618
  * Pardy et al. (2018) Pardy, S. A., D’Onghia, E., & Fox, A. J. 2018, ApJ, 857, 101, doi: 10.3847/1538-4357/aab95b
  * Peñarrubia et al. (2016) Peñarrubia, J., Gómez, F. A., Besla, G., Erkal, D., & Ma, Y.-Z. 2016, MNRAS, 456, L54, doi: 10.1093/mnrasl/slv160
  * Putman et al. (1998) Putman, M. E., Gibson, B. K., Staveley-Smith, L., et al. 1998, Nature, 394, 752, doi: 10.1038/29466
  * Röttgers et al. (2020) Röttgers, B., Naab, T., Cernetic, M., et al. 2020, MNRAS, 496, 152, doi: 10.1093/mnras/staa1490
  * Růžička et al. (2010) Růžička, A., Theis, C., & Palouš, J. 2010, ApJ, 725, 369, doi: 10.1088/0004-637X/725/1/369
  * Salem et al. (2015) Salem, M., Besla, G., Bryan, G., et al. 2015, ApJ, 815, 77, doi: 10.1088/0004-637X/815/1/77
  * Springel (2005) Springel, V. 2005, MNRAS, 364, 1105, doi: 10.1111/j.1365-2966.2005.09655.x
  * Springel et al. (2005) Springel, V., Di Matteo, T., & Hernquist, L. 2005, MNRAS, 361, 776, doi: 10.1111/j.1365-2966.2005.09238.x
  * Springel & Hernquist (2003) Springel, V., & Hernquist, L. 2003, MNRAS, 339, 289, doi: 10.1046/j.1365-8711.2003.06206.x
  * Tepper-García et al. (2019) Tepper-García, T., Bland-Hawthorn, J., Pawlowski, M. S., & Fritz, T. K. 2019, arXiv e-prints, arXiv:1901.05636.  https://arxiv.org/abs/1901.05636
  * Tepper-García et al. (2015) Tepper-García, T., Bland-Hawthorn, J., & Sutherland , R. S. 2015, ApJ, 813, 94, doi: 10.1088/0004-637X/813/2/94
  * van de Voort et al. (2019) van de Voort, F., Springel, V., Mandelker, N., van den Bosch, F. C., & Pakmor, R. 2019, MNRAS, 482, L85, doi: 10.1093/mnrasl/sly190
  * van der Marel et al. (2002) van der Marel, R. P., Alves, D. R., Hardy, E., & Suntzeff, N. B. 2002, AJ, 124, 2639, doi: 10.1086/343775
  * Wolfire et al. (1995) Wolfire, M. G., McKee, C. F., Hollenbach, D., & Tielens, A. G. G. M. 1995, ApJ, 453, 673, doi: 10.1086/176429
  * Zaritsky et al. (2020) Zaritsky, D., Conroy, C., Naidu, R. P., et al. 2020, ApJL, 905, L3, doi: 10.3847/2041-8213/abcb83
  * Zivick et al. (2018) Zivick, P., Kallivayalil, N., van der Marel, R. P., et al. 2018, ApJ, 864, 55, doi: 10.3847/1538-4357/aad4b0