# Leveraging Advantages of Interactive and Non-Interactive Models for Vector-
Based Cross-Lingual Information Retrieval

Linlong Xu, Baosong Yang, Xiaoyu Lv, Tianchi Bi, Dayiheng Liu, Haibo Zhang

###### Abstract

Interactive and non-interactive model are the two de-facto standard frameworks
in vector-based cross-lingual information retrieval (V-CLIR), which embed
queries and documents in synchronous and asynchronous fashions, respectively.
From the retrieval accuracy and computational efficiency perspectives, each
model has its own superiority and shortcoming. In this paper, we propose a
novel framework to leverage the advantages of these two paradigms. Concretely,
we introduce semi-interactive mechanism, which builds our model upon non-
interactive architecture but encodes each document together with its
associated multilingual queries. Accordingly, cross-lingual features can be
better learned like an interactive model. Besides, we further transfer
knowledge from a well-trained interactive model to ours by reusing its word
embeddings and adopting knowledge distillation. Our model is initialized from
a multilingual pre-trained language model M-BERT, and evaluated on two open-
resource CLIR datasets derived from Wikipedia and an in-house dataset
collected from a real-world search engine. Extensive analyses reveal that our
methods significantly boost the retrieval accuracy while maintaining the
computational efficiency. 111Our codes are released on
https://github.com/wutangA/Semi-Interactive-CLIR.

##  1 Introduction

Conventional cross-lingual information retrieval (CLIR) system mainly
separates two stages, i.e., query translation and monolingual information
retrieval (Sun, Sia, and Duh 2020; Zhou et al. 2012; Sabet et al. 2019; Li et
al. 2020; Yao et al. 2020a). On the contrary, vector-based CLIR (V-CLIR)
(Vulic and Moens 2015; Yarmohammadi et al. 2019) employs neural networks to
encode input texts and score their associated similarity with end-to-end
training, thus avoiding the problem of error propagation in translation-based
systems (Wu and He 2010; Bosca and Dini 2010).

A widely used neural-based retrieval framework is deep relevance matching
model (DRMM, Guo et al. 2016), which is built upon a unified semantic matching
model to generate embeddings for queries and documents. In this paradigm,
features of query and document are directly interacted at the representation
learning time (being called as interactive mechanism, as shown in Figure 1
(a)), leading to promising results in a variety of information retrieval tasks
(Mikolov et al. 2013; Bojanowski et al. 2017). However, in a real-world search
engine, a user query has to match with millions of documents in database. When
performing the interactive framework, for each query, massive real-time
computations have to be carried out for obtaining joint representations.

Towards approaching this problem, existing V-CLIR systems are mainly built
with a non-interactive architecture (Huang et al. 2013; Reimers and Gurevych
2019; Lu, Jiao, and Zhang 2020), as shown in Figure 1 (b). This model encodes
queries and documents asynchronously, making the offline learning of document
representations available. In this way, each query only requires to be encoded
once during online inference. Despite its superiority on computational
efficiency, such kind of architecture lacks in feature fusion among queries
and documents, further raising the difficulty of cross-lingual semantic
matching (Reimers and Gurevych 2019; Lu, Jiao, and Zhang 2020). It can be said
that how to build a fast yet high-quality deep semantic matching model to
enhance V-CLIR still remains a great challenge.

(a) Interactive Mechanism

(b) Non-Interactive Mechanism

(c) Ours: Semi-Interactive Mechanism

Figure 1: Illustration of a) interactive mechanism which encodes inputs
jointly, b) non-interactive architecture that learns representations of query
and document separately, and 3) the proposed semi-interactive mechanism in
which the relevant multilingual queries are incorporated into document
representation thus narrowing the cross-lingual semantic latent spaces.

A natural question arises: Is it possible to leverage the advantages of
interactive and non-interactive mechanisms simultaneously? In this paper, we
propose a novel model named semi-interactive mechanism, as visualized in
Figure 1 (c). Concretely, our model is built upon a non-interactive framework
to maintain the computational efficiency. In order to narrow the distance
between the latent spaces of cross-lingual queries and documents, we fuse the
information of a document with its associated multilingual queries, which can
be either ready-made or collected from the search log. The representation of
document therefore contains rich multilingual contexts. To further leverage
the advantage of interactive mechanism, we set a well-trained interactive
model as teacher and transfer its knowledge to our model by reusing word
embeddings and adopting knowledge distillation at the training time (Sanh et
al. 2019; Sun et al. 2019). With these enhancements, the proposed model is
able to preserve the online inference speed like a non-interactive model, in
the meanwhile, fill the quality gap to its interactive counterpart.

Following the common settings (Yilmaz et al. 2019; Jiang et al. 2020), we
build our approach upon a multilingual pre-trained language model – M-BERT
(Devlin et al. 2019). The training set is constructed from a large-scale CLIR
dataset – WikiCLIR (Sasaki et al. 2018), for Russian-to-English, Spanish-to-
English, Portuguese-to-English, French-to-English and Arabic-to-English
document retrieval. Our model is examined on both semantic similarity tasks
collected from WikiCLIR and document search tasks provided in MULTI-8 (Sun and
Duh 2020). Empirical results demonstrate that the proposed semi-interactive
mechanism and knowledge transferring progressively improve the retrieval
accuracy over non-interactive baseline and marginally drop the inference
speed. In addition, we conduct experiments on a real-world search engine. Our
model still significantly outperforms non-interactive based approach. Our
visualization of representation distribution reveal that the proposed model
can approximately transform both query and document representations to a
standard orthogonal basis (Su et al. 2021), leading to more discriminative
representations for similarity computation.

Major contributions of our work are three-fold:

  * •

We introduce a novel semi-interactive mechanism, offering V-CLIR model ability
to combine the advantages of interactive and non-interactive paradigms.

  * •

We propose to transfer knowledge from a well-trained interactive model, thus
boosting the retrieval quality of a non-interactive model.

  * •

Extensive analyses indicate the universal-effectiveness of our work and verify
that our model is able to learn better representation for multilingual texts.

##  2 Preliminary

###  2.1 Cross-Lingual Information Retrieval

CLIR has attracted increasing interests over the past decades. The main
challenge is to bridge the language discrepancy between user queries and
documents (Nie 2010). Most of existing approaches use translation based
technique, which first calls a machine translation (MT) model to translate
either queries or documents into the same language (Levow, Oard, and Resnik
2005; Martino et al. 2017), the translation results are then passed to a
monolingual IR system. However, such kind of framework requires huge amounts
of parallel data, which is scarce for many language pairs and domains in low-
resource. In addition, the search results are directly affected by translation
accuracy.

With this point, researches start to explore vector-based IR methods (Nigam et
al. 2019; Mitra and Craswell 2018; Tai, Ren, and Kita 2002; Mihalcea, Corley,
and Strapparava 2006; Bi et al. 2020; Yao et al. 2020b) without the need to
build MT systems. In this context, queries and documents are represented as
vectors for similarity computation.

###  2.2 Vector-Based Retrieval Models

Vector-based IR model can be divided into two directions: interactive and non-
interactive models.

#### Interactive Mechanism

Deep relevance matching model (DRMM, Guo et al. 2016) and kernel-based neural
ranking model (Xiong et al. 2017, KNRM,) are two representatives under
interactive framework. These approaches concatenate and feed query-document
pairs into a unified deep neural encoder to learn their representations for
semantic matching, as shown in Figure 1(a). Accordingly, features in query and
document are interacted and fused, resulting in promising quality on a variety
of IR tasks (McDonald, Brokos, and Androutsopoulos 2018). Recent studies
further initialize the encoder with large-scale pre-trained language model,
such as BERT (Devlin et al. 2019), which brings better model abilities on
representation learning and relevance scoring (Jiang et al. 2020).
Specifically, given the input token embeddings of a query Q and a document D,
the interactive model first encodes them into hidden states:

$\textbf{H}=\mathrm{Encoder}([\textbf{Q},\textbf{D}])$ (1)

where $\mathrm{Encoder}(\cdot)$ indicates the BERT model, $[\cdot]$ represents
the concatenation of inputs. Then, a non-linear active function is adopted to
calculate the relevance score s over the mean of the output hidden states
$\overline{\textbf{H}}$:

$\textbf{s}=\mathrm{sigmoid}(\overline{\textbf{H}})$ (2)

#### Non-Interactive Mechanism

Contrary to the synchronous encoding procedure in interactive architecture,
the representations of queries and documents in non-interactive architecture
(Figure 1(b)) are encoded asynchronously (Huang et al. 2013; Shen et al. 2014;
Reimers and Gurevych 2019; Lu, Jiao, and Zhang 2020). It employs two encoders
for query and document representation learning, respectively. Formally, the
representations of query and document are calculated in a separate manner:

$\textbf{H}_{Q}=\mathrm{Encoder}_{Q}(\textbf{Q}),~{}~{}~{}~{}\textbf{H}_{D}=\mathrm{Encoder}_{D}(\textbf{D})$
(3)

A cosine function is assigned for the similarity calculation:

$\textbf{s}=\cos(\overline{\textbf{H}}_{Q},\overline{\textbf{H}}_{D})=\frac{\overline{\textbf{H}}_{Q}\cdot\overline{\textbf{H}}_{D}}{||\overline{\textbf{H}}_{Q}||\cdot||\overline{\textbf{H}}_{D}||}$
(4)

#### Training

The parameters of both interactive and non-interactive models are optimized by
minimizing the cross-entropy loss over the training set:

$\textbf{L}_{cross}=-\sum_{\textbf{Q},\textbf{D}}\textbf{y}\mathrm{log}(\textbf{s})+(1-\textbf{y})\mathrm{log}(1-\textbf{s})$
(5)

where $\textbf{y}\in\\{0,1\\}$ denotes the groundtruth relevance label of Q
and D.

##  3 Methodology

###  3.1 Motivation

To find the most similar document of Q in a collection of millions documents,
interactive model has to recompute the joint representation
$\overline{\textbf{H}}$ millions times for all the documents. This massive
computational overhead makes it unsuitable for real-word search engine (Zhang
and Zhai 2015). The superiority of non-interactive architecture lies in the
fast online inference. Nevertheless, the lack of interaction between Q and D
may cause biases between latent vectors, raising the difficulty on semantic
matching. In particular, Q and D in different languages further deteriorate
this problem, resulting in unsatisfied retrieval quality.

Accordingly, our goal is to leverage the advantages of interactive and non-
interactive models. In this work, we handle this problem from two aspects:

  * •

From the model perspective, we propose semi-interactive module (Figure 1(c))
to encode the representation of a document with its relevant multilingual
queries. In this way, multilingual features can be supplemented into the
document representation.

  * •

From the optimization perspective, we introduce knowledge transfer methods, in
which a well-trained interactive model is regarded as teacher. We reuse its
word embeddings and distill its knowledge to our model.

###  3.2 Semi-Interactive Mechanism

One principle of our model is to maintain the inference speed, in the
meanwhile, improve retrieval quality. To this end, we build the proposed model
upon a non-interactive architecture. A natural solution to alleviate the
discrepancy between latent spaces of query and document is to enhance the
document representations with cross-lingual features. Specifically, we propose
to provide multilingual contexts for the document encoder. In this way, cross-
lingual properties are incorporated at the representation learning time, thus
producing more informative representations. More importantly, since each
document and its associated contexts can be encoded offline, the computational
efficiency is still preserved. We serve the multilingual keywords or queries
that is relevant to the document as its multilingual contexts.

As shown in Figure 2, given $N$ relevant queries
$\\{\textbf{Q}^{r}_{1},…,\textbf{Q}^{r}_{N}\\}$ for the document D, where
$\textbf{Q}^{r}_{i}$ ($i\in\\{1,...,N\\}$) denotes the $i$-th relevant query,
the document encoder produces joint representations $\textbf{H}_{D}$ by
modifying Equation 3 as following:

$\textbf{H}_{D}=\mathrm{Encoder}_{D}([\textbf{D},\textbf{Q}^{r}_{1},…,\textbf{Q}^{r}_{N}])$
(6)

In this paper, we set $N=3$ as default according to ablation study as Section
5.2 described.

#### Collecting Relevant Queries

Several potential manners can collect relevant multilingual queries:

  * •

Ready-Made Queries. Several documents have already provided multilingual
summarization or keywords. For example, the dataset WikiCLIR (Sasaki et al.
2018) collects large amount of pages of Wikipedia, each of which links with
multilingual relevant keywords. For simplification, we can directly collect
$N$ ready-made relevant queries labeled as “relevant” for each document.

  * •

Mining-Based Queries. For an existing cross-lingual search engine (no matter
translation-based or vector-based), the relevant queries for each document can
be extracted according to its clickthrough data. The click behavior reflects
users’ interests and the latent semantic relationships between the input
queries and the clicked documents (Radlinski, Kurup, and Joachims 2008; Ma et
al. 2008). We can collect Top $N$ queries according to the click-through rate
(CTR).

A special circumstances is that there is neither off-the-rack multilingual
relevant queries, nor search log. In this paper, we do not take attention into
this case, because a vector-based CLIR system even can not be well-trained
without large-scale cross-lingual annotated data, letting alone considering
the trade-off between efficiency and quality. Despite of that, several few-
shot learning approaches in IR contexts can be carried out to handle this
problem, such as generating pseudo data using MT (Chidambaram et al. 2019;
Yang et al. 2020) and searching associated queries using a preliminary
retrieval system (Litschko et al. 2018).

Figure 2: Illustration of our model architecture. We cast the proposed semi-
interactive approach as a student model in which each document is supplemented
with its relevant multilingual queries. We further transfer knowledge from the
well-trained interactive model to semi-interactive one by: 1) reusing word
embeddings, and 2) distilling knowledge.

###  3.3 Knowledge Transfer

Another promising direction is to transfer knowledge from the well-trained
interactive model to the non-interactive one. As interactive architecture
benefits to the representation learning and relevance scoring, we treat it as
a teacher to assist the training of non-interactive models. Our model profits
from the teacher model in two aspects: 1) initializing word embeddings of our
model with that of teacher model and 2) distilling prediction distribution of
teacher model to ours.

#### Reusing Word Embeddings

The distribution of word embeddings immediately affects the final
representations, therefore plays a critical role in neural-based natural
language understanding tasks (Levy and Goldberg 2014; Kusner et al. 2015). As
cross-lingual queries and documents are jointly encoded by the interactive
model, word embeddings have relatively less discrepancy caused by distinct
languages. As a result, we reuse the well-trained word embedding layer of the
teacher model to initialize that of non-interactive models. We expect this can
carry a better initialization for model parameters, thus contribute to the
subsequent training procedure.

#### Knowledge Distillation

Knowledge distillation approach (Ba and Caruana 2014; Hinton, Vinyals, and
Dean 2015) enables the transfer of knowledge from a teacher model to a student
model, which is improved in the process. Here, we exploit knowledge
distillation to guide both non-interactive and semi-interactive model to learn
the probability distribution output by a well-trained interactive model.
Specifically, given the relevance probability $\textbf{s}{{}^{\prime}}$
predicted by the teacher model, the goal of distillation is to force our model
to fit the probability distribution of the teacher:

$\textbf{L}_{distill}=-\sum_{\textbf{Q},\textbf{D}}\textbf{s}{{}^{\prime}}\mathrm{log}(\textbf{s})$
(7)

The final loss can be formally expressed as:

$\textbf{L}=\alpha\textbf{L}_{cross}+(1-\alpha)\textbf{L}_{distill}$ (8)

where $\textbf{L}_{cross}$ is the conventional cross-entropy loss as defined
in Equation 5. $\alpha$ denotes a factor to balance the two loss, which is set
to 0.7 as default according to ablation study as Section 5.2 described.

Recent studies (Tang et al. 2019; Lu, Jiao, and Zhang 2020; Izacard and Grave
2020) has successfully adopted such kind of methods to the area of monolingual
IR. For example, Tang et al. (2019) and Lu, Jiao, and Zhang (2020) distill
knowledge from the pre-trained language model BERT to a BiLSTM-based model and
a non-interactive model, respectively. Contrast with these researches, we are
the first to distill knowledge from a interactive model to the semi-
interactive one, and examine the effectiveness of knowledge distillation under
the cross-lingual scenario. Besides, the input of teacher model and student
model are same in prior studies, while ours are different.

##  4 Experiment Setting

We examine our methods on both semantic similarity task and document search
task. In this section, we describe the experimental setting in detail.

Direction | Training | Validation | Testing
---|---|---|---
WikiCLIR | WikiCLIR |  | Semantic
---
Similarity
| Document
---
Search
Ru$\Rightarrow$En | 2.2M | 30.0K | 60.0K | 100.0K
Es$\Rightarrow$En | 2.8M | 30.0K | 60.0K | 100.0K
Fr$\Rightarrow$En | 5.0M | 30.0K | 60.0K | 100.0K
Pt$\Rightarrow$En | 1.6M | 30.0K | 60.0K | -
Ar$\Rightarrow$En | 0.4M | 30.0K | 60.0K | 100.0K
De$\Rightarrow$En | - | - | - | 100.0K
Ja$\Rightarrow$En | - | - | - | 100.0K
Zh$\Rightarrow$En | - | - | - | 100.0K
Total | 12.0M | 150.0K | 300.0K | 700.0K

Table 1: Statistics of samples used in our study.

###  4.1 Training and Validation Set

WikiCLIR (Sasaki et al. 2018) is a large-scale CLIR dataset collected from
Wikipedia which has provided positive and negative samples. We construct our
training and validation set on English (En) document and five multilingual
queries in Russian (Ru), Spanish (Es), French (Fr), Portuguese (Pt) as well as
Arabic (Ar). For each language pair, we randomly select 0.4M to 5.0M (M =
million) samples as the training set, covering low-resource and high-resource
language directions. And 60K (K = thousand) samples are extracted as the
validation set for each language direction. The statistics of the used datset
is concluded in Table 1. Each document in WikiCLIR has been linked to multiple
multilingual queries, we randomly collect $N$ ready-made relevant queries
labeled as “relevant” for each document in the proposed semi-interactive
mechanism. Finally, each sample is formed as:
${<query,document,relevant\\_queries,label>}$. To cope with the pre-trained
language model M-BERT, all the queries and documents are preprocessed
following Devlin et al. (2019), the first token is a special classification
token ([CLS]) and the document and relevant queries are concatenated with a
special token ([SEP]).

###  4.2 Evaluation Tasks and Metrics

#### Semantic Similarity

Semantic similarity tasks aim to predict whether query and document are
semantically relevant or not. We evaluate our methods on WikiCLIR in five
languages: Russian (Ru), Spanish (Es), French (Fr), Portuguese (Pt) as well as
Arabic (Ar). For each language pair, we randomly extract 30K samples as the
test set. To evaluate the performance, we use the area under the curve (AUC,
Lydick, Epstein, and White 1995). Moreover, in order to verify the
effectiveness of mining queries from search log, we also conduct a group of
experiments on a dataset collected from a real-world search engine, as
described in Section 5.4.

Model | Ru$\Rightarrow$En | Es$\Rightarrow$En | Pt$\Rightarrow$En | Fr$\Rightarrow$En | Ar$\Rightarrow$En | AVG | Inf.Time(ms)
---|---|---|---|---|---|---|---
Interact | 90.71 | 96.57 | 96.56 | 96.37 | 90.86 | 94.86 | 45.8
Non-Interact | 77.83 | 80.91 | 79.94 | 80.70 | 80.20 | 78.92 | 8.4
\+ KT | 79.57 | 83.96 | 82.60 | 83.82 | 78.66 | 81.71 | 8.4
Semi-Interact | $82.31^{\ddagger}$ | $85.81^{\ddagger}$ | $84.98^{\ddagger}$ | $85.68^{\ddagger}$ | $84.44^{\ddagger}$ | $84.04^{\ddagger}$ | 8.9
\+ KT | $\textbf{86.55}^{\ddagger}$ | $\textbf{90.75}^{\ddagger}$ | $\textbf{90.69}^{\ddagger}$ | $\textbf{91.03}^{\ddagger}$ | $\textbf{87.79}^{\ddagger}$ | $\textbf{89.26}^{\ddagger}$ | 8.9

Table 2:  AUC of different models on WikiCLIR semantic similarity task. “Inf.
Time” means the average inference time for each sample. $\ddagger$/$\dagger$
represents our model is significantly better than Non-Interact (p<0.01/0.05),
tested by bootstrap resampling (Koehn 2004).

Model | Supervised | Unsupervised
---|---|---
Ru$\Rightarrow$En | Es$\Rightarrow$En | Fr$\Rightarrow$En | Ar$\Rightarrow$En | De$\Rightarrow$En | Zh$\Rightarrow$En | Ja$\Rightarrow$En
(Sun and Duh 2020) | 0.7100 | 0.7600 | 0.7600 | 0.6000 | 0.7500 | 0.6300 | 0.7100
Interact | 0.7717 | 0.7954 | 0.8050 | 0.7458 | 0.7579 | 0.7478 | 0.7504
Non-Interact | 0.7052 | 0.6957 | 0.6965 | 0.7135 | 0.6669 | 0.7001 | 0.7062
\+ KT | 0.7101 | 0.6989 | 0.6982 | 0.7203 | 0.6736 | 0.7024 | 0.7100
Semi-Interact | $0.7129^{\ddagger}$ | $0.7131^{\ddagger}$ | $0.7100^{\ddagger}$ | $0.7315^{\ddagger}$ | $0.6845^{\ddagger}$ | $0.7049^{\dagger}$ | $0.7218^{\ddagger}$
\+ KT | $\textbf{0.7259}^{\ddagger}$ | $\textbf{0.7181}^{\ddagger}$ | $\textbf{0.7255}^{\ddagger}$ | $\textbf{0.7328}^{\ddagger}$ | $\textbf{0.7041}^{\ddagger}$ | $\textbf{0.7252}^{\ddagger}$ | $\textbf{0.7419}^{\ddagger}$

Table 3:  NDCG@10 of different models on MULTI-8 retrieval task. “Supervised”
means the training data covers these retrieval directions, while
“Unsupervised” does not.

#### Document Search

This task is to rank candidate documents by relevance with respect to a
multilingual query. Here, we use the test data of MULTI-8 (Sun and Duh 2020)
to further evaluate our proposed methods. MULTI-8 is also constructed from
Wikipedia but allows for more finergrained levels of relevance. We evaluate
our methods on seven multilingual queries (Arabic (Ar), German (De), Spanish
(Es), French (Fr), Japanese (Ja), Russian (Ru), and Chinese (Zh)) with each
language pair contains 100K samples. Following the default setting in Sun and
Duh (2020), we calculate normalized discounted cumulative gain (NDCG, Järvelin
and Kekäläinen 2002) to evaluate the search performance.

###  4.3 Implementation Details

We implement our model and the associated baselines upon Tensorflow. In order
to preliminarily map different languages into the same latent space, all the
encoders are initialized by the pre-trained cross-lingual language model
M-Bert (Devlin et al. 2019).222https://github.com/google-research/bert We
follow Lu, Jiao, and Zhang (2020) to build each model with 6 layers in
encoders. For the model training, we use Adam optimizer (Kingma and Ba 2015)
and set learning rate to 3e-5 with 8K warmup steps. Besides, dropout rate is
set to a constant of 0.1 to enhance the model robustness. We train our model
on a single Tesla P100 GPU with a mini-batch consisting of 1,024 samples. The
training of each model is early-stopped to maximize AUC on the validation set.
Other hyper-parameters are assigned with the common finetune configurations
described in Devlin et al. (2019). We compare following models:

  * •

Interact: We first build an interactive model as baseline and teacher model.
Since there are few studies explore V-CLIR, we build the model following Jiang
et al. (2020). Contrary to previous study, we initialize the encoder using
multilingual BERT rather than the monolingual version (Devlin et al. 2019).

  * •

Non-Interact: We borrow the advanced monolingual vector-based IR model
(Reimers and Gurevych 2019) into the CLIR task as our non-interactive
baseline.

  * •

Ours: We examine the proposed semi-interactive mechanism (Semi-Interact) and
knowledge transfer method (KT). Considering the former, we randomly select
$N=3$ relevant queries for each document. For the latter, we set $\alpha$ to
0.7 as default.

##  5 Results

###  5.1 Main Results

#### Results on Semantic Similarity

As shown in Table 2, our model significantly outperforms the non-interactive
baseline across different language pairs, demonstrating the universal-
effectiveness of the proposed method. In the meanwhile, our model yields over
4 times faster than its interactive counterpart at the inference time.

Concretely, although interactive model outperforms non-interactive model, it
significantly increases the inference time and fails to be employed in a real-
world IR system. We regard this model as an upper limit of retrieval quality.
The proposed semi-interactive mechanism narrows the accuracy gap, which
confirms our hypothesis that supplementing the document representation with
its relevant multilingual contexts is conductive to V-CLIR. Moreover, the
interactions among multilingual queries and documents of our approach are able
to be conducted offline, maintaining the superiority of non-interactive model
on processing speed.

We assess knowledge transferring on non- and semi-interactive models. Both the
models boost the retrieval performance, indicating that transferring knowledge
from a well-trained interactive model to non-interactive models is able to
narrow the quality gaps among them. Besides, our results demonstrate that the
Semi-Interact and KT are complementary to each other, progressively benefiting
to the cross-lingual semantic matching.

#### Results on Document Search

Table 3 shows the NDCG@10 of different models on MULTI-8 document search
tasks. Similar to semantic similarity tasks, our method yields consistently
better results across language pairs. As a superiority of pre-trained
multilingual language model lies in its ability on knowledge transferring, we
can transfer knowledge from high-resource languages to low-resource ones in
the downstream tasks. Accordingly, we conduct three retrieval tasks that are
not considered at the training time. As seen, the proposed methods yield
considerable performance on both the three tasks. It is encouraging to see
that, although we do not enhance documents with relevant queries in these
three languages, the improvements are still preserved on these tasks.

###  5.2 Ablation Study

In this section, we conduct ablation study of different components on WikiCLIR
semantic similarity tasks. We report the average score for simplification.

#### Effects of the Number of Relevant Queries

We plot Figure 3 (a) to show the effects of the number of relevant queries
added to document representation. Specifically, via incorporating queries (N
$>$ 0), the relevance accuracy increases, confirming that the associated
queries are helpful for the document representation learning. We observe that
the number with 3 is superior to other settings. When the number of queries
goes up (N $>$ 3), the classification performance inversely drops. One
possible reason is that the conventional semantic meaning of the document may
be potentially overlooked when complementing superfluous multilingual queries.
This results in biases in document representations and raises the difficulty
on relevance prediction.

(a) Number $N$ of Queries

(b) Factor $\alpha$ in Loss

Figure 3: Effects of (a) the number of relevant queries being used in document
encoding, and (b) the value of weights $\alpha$ in loss. Experiments are
conducted based on Semi-Interact.

Model | Training
---|---
✓ | ✗
Testing | ✓ | 84.04 | 78.76
✗ | 79.97 | 78.92

Table 4:  Ablation studies on whether relevant queries are incorporated during
training and testing.

#### Usage of Semi-Interactive Mechanism

We verify the effect of semi-interactive on different stages, i.e. training
and inference. As shown in Table 4, we found that exploiting relevant queries
at the training time can consistently improve the performance, no matter
relevant queries are utilized or not during inference. When adapting semi-
interactive mechanism only in inference, the performance marginally changes.
We attribute these to the fact that complementing multilingual contexts during
training benefits the representation learning. Our results suggest that
carrying out our method on both training and inference stages performs best.

#### Effects of Factor in Distillation

Figure 3 (b) visuals the effects of the factor $\alpha$ used in loss function
at knowledge distilling time. Our results reveal that the weight significantly
affect the performance. A small $\alpha$ makes the teacher model plays a
dominate role during training time, while too large values result in less
affects from the distillation procedure. Weight with 0.7 is the best
configuration to balance the learning from the groundtruth and the teacher
model.

#### Effectiveness of Knowledge Transferring Strategies

We further conduct experiments to examine knowledge transferring strategies.
As shown in Table 5, reusing teacher model word embeddings (Initialize) and
knowledge distillation (Distillation) improve the relevance quality separately
and are complementary to each other. These results verify again that
transferring knowledge from interactive model is facilitate to the
representation learning and relevance scoring of non-interactive model.

Model | AUC
---|---
Semi-Interact | 84.04
\+ Initialize | 88.33
\+ Distillation | 87.54
\+ Initialize + Distillation | 89.26

Table 5:  Ablation studies on knowledge transferring.

###  5.3 Representation Visualization

A natural question is why and how the proposed mechanism boosts the model
performance. We pay our attention into the representation learning of queries
and documents. Su et al. (2021) and Gao et al. (2019) pointed out that
sentence embeddings not in a standard orthogonal basis cause poor performance
of semantic similarity tasks, especially for the BERT-like models.

For a more intuitive comparison, we randomly select 10K query and document
representations from our training set, and project them into a 3-dimensional
space using principal component analysis (PCA, Abdi and Williams 2010). As
shown in Figure 4, the representations learnt from interactive model (Figure
4(a)) are diversely distributed around the origin using PCA projection. In
contrast, the representations in non-interactive model (Figure 4(b))
degenerated into a narrow boundary, which means that the representations do
not have enough capacity to model the diverse semantics in natural languages
(Yang et al. 2017; McCann et al. 2017). Furthermore, we find the
representations of semi-interactive (Figure 4(c)) are gradually distributed to
the origin, which is more obviously while applying transferring knowledge
methods (Figure 4(d)).

(a) Interactive

(b) Non-Interactive

(c) Semi-Interactive

(d) Semi-Interactive + KT

Figure 4: 3D visualization of query and document representations distribution.
Each point in the figure represents a query or document representation.
Different languages are distinguished by different colors. Compared with non-
interactive, the representations of semi-interactive are gradually distributed
to more uniform representation space.

###  5.4 Mining Relevant Queries from Search Log

All the above experiments are based on ready-made queries provided in Wiki
pages. We are interested at the effectiveness of the proposed semi-interactive
mechanism in a real-world search engine where off-the-rack multilingual
relevant queries for each document are not available.

To this end, we examine our method on our in-house dataset extracted from a
real-world search engine Alibaba.com. 333Note that, we cannot release the
data, we report this results to illustrate the effectiveness of our approach
in a real-world scenario. Readers can re-produce our results using the open-
released dataset described in Section 4.1. Specifically, we conduct
experiments on Russian-to-English and Spanish-to-English semantic similarity
tasks. Query-title training pairs are collected from the search logs,
resulting in 1 million training samples for each task. For each query, we
randomly sample documents from the document pool as negatives, while using the
one user clicked as positive. The test set is extracted from training data and
manually checked by bilingual experts, resulting in 30K samples for each
directions. We select top 3 multi-lingual queries for each document according
to clickthrough data (as described in Section 3.2)).

As shown in Table 6, our model significantly outperforms non-interactive based
approach over 7 AUC in average. The results confirm the effectiveness of our
method when mining relevant queries from search log.

Model | Ru$\Rightarrow$En | Es$\Rightarrow$En | AVG
---|---|---|---
Interact | 84.96 | 85.67 | 85.38
Non-Interact | 68.54 | 69.83 | 69.65
\+ KT | 69.90 | 70.64 | 70.39
Semi-Interact | 72.97 | 73.42 | 73.21
\+ KT | 75.66 | 77.63 | 76.90
Table 6:  AUC of models when mining relevant queries from a real-world search
log.

##  6 Conclusion

In this paper, we provide insights on how to leverage the advantages of
interactive model and non-interactive model for vector-based CLIR. Via
supplementing multilingual features into document representation and
transferring knowledge from a well-trained interactive model, the proposed
model yields significant improvements over the conventional non-interactive
model, while preserving its advantage with respect to computational
efficiency. Our in-depth analyses further suggest that 1) the proposed
strategies can tackle the representation degeneration problem in non-
interactive models, thus obtaining discriminative representations; and 2) both
of ready-made queries and mined queries are effect for supplementing
multilingual features to documents.

Several directions are worth to be explored in future. An interesting problem
is to explore more effective approaches to collect multilingual queries,
especially for few-shot tasks, e.g. generating associated multilingual queries
using text summarization (Gambhir and Gupta 2017; Liu and Lapata 2019).
Another potential direction is to select queries with a more flexible manner
such as using a dynamic routing layer (Sabour, Frosst, and Hinton 2017; Li et
al. 2019).

## References

  * Abdi and Williams (2010) Abdi, H.; and Williams, L. J. 2010.  Principal component analysis.  _Wiley interdisciplinary reviews: computational statistics_.
  * Ba and Caruana (2014) Ba, J.; and Caruana, R. 2014.  Do Deep Nets Really Need to be Deep?  In _NIPS_.
  * Bi et al. (2020) Bi, T.; Yao, L.; Yang, B.; Zhang, H.; Luo, W.; and Chen, B. 2020.  Constraint Translation Candidates: A Bridge between Neural Query Translation and Cross-lingual Information Retrieval.  _CoRR_.
  * Bojanowski et al. (2017) Bojanowski, P.; Grave, E.; Joulin, A.; and Mikolov, T. 2017.  Enriching Word Vectors with Subword Information.  In _TACL_.
  * Bosca and Dini (2010) Bosca, A.; and Dini, L. 2010.  Language Identification Strategies for Cross Language Information Retrieval.  In _CLEF_.
  * Chidambaram et al. (2019) Chidambaram, M.; Yang, Y.; Cer, D.; Yuan, S.; Sung, Y.; Strope, B.; and Kurzweil, R. 2019.  Learning Cross-Lingual Sentence Representations via a Multi-task Dual-Encoder Model.  In _ACL_.
  * Devlin et al. (2019) Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _NAACL-HLT_.
  * Gambhir and Gupta (2017) Gambhir, M.; and Gupta, V. 2017.  Recent automatic text summarization techniques: a survey.  _Artif. Intell. Rev._
  * Gao et al. (2019) Gao, J.; He, D.; Tan, X.; Qin, T.; Wang, L.; and Liu, T. 2019.  Representation Degeneration Problem in Training Natural Language Generation Models.  In _ICLR_.
  * Guo et al. (2016) Guo, J.; Fan, Y.; Ai, Q.; and Croft, W. B. 2016.  A deep relevance matching model for ad-hoc retrieval.  In _CLKM_.
  * Hinton, Vinyals, and Dean (2015) Hinton, G. E.; Vinyals, O.; and Dean, J. 2015.  Distilling the Knowledge in a Neural Network.  _CoRR_.
  * Huang et al. (2013) Huang, P.; He, X.; Gao, J.; Deng, L.; Acero, A.; and Heck, L. P. 2013.  Learning deep structured semantic models for web search using clickthrough data.  In _CIKM_.
  * Izacard and Grave (2020) Izacard, G.; and Grave, E. 2020.  Distilling Knowledge from Reader to Retriever for Question Answering.  _arXiv_.
  * Järvelin and Kekäläinen (2002) Järvelin, K.; and Kekäläinen, J. 2002.  Cumulated gain-based evaluation of IR techniques.  _ACM_.
  * Jiang et al. (2020) Jiang, Z.; El-Jaroudi, A.; Hartmann, W.; Karakos, D. G.; and Zhao, L. 2020.  Cross-lingual Information Retrieval with BERT.  In _LREC_.
  * Kingma and Ba (2015) Kingma, D. P.; and Ba, J. 2015.  Adam: A Method for Stochastic Optimization.  In _ICLR_.
  * Koehn (2004) Koehn, P. 2004.  Statistical Significance Tests for Machine Translation Evaluation.  In _EMNLP_.
  * Kusner et al. (2015) Kusner, M.; Sun, Y.; Kolkin, N.; and Weinberger, K. 2015.  From word embeddings to document distances.  In _ICML_.
  * Levow, Oard, and Resnik (2005) Levow, G.; Oard, D. W.; and Resnik, P. 2005.  Dictionary-based techniques for cross-language information retrieval.  _Inf. Process. Manag._
  * Levy and Goldberg (2014) Levy, O.; and Goldberg, Y. 2014.  Dependency-based word embeddings.  In _ACL_.
  * Li et al. (2020) Li, J.; Liu, C.; Wang, J.; Bing, L.; Li, H.; Liu, X.; Zhao, D.; and Yan, R. 2020\.  Cross-Lingual Low-Resource Set-to-Description Retrieval for Global E-Commerce.  In _AAAI_.
  * Li et al. (2019) Li, J.; Yang, B.; Dou, Z.-Y.; Wang, X.; Lyu, M. R.; and Tu, Z. 2019.  Information Aggregation for Multi-Head Attention with Routing-by-Agreement.  In _NAACL_.
  * Litschko et al. (2018) Litschko, R.; Glavas, G.; Ponzetto, S. P.; and Vulic, I. 2018.  Unsupervised Cross-Lingual Information Retrieval Using Monolingual Data Only.  In _SIGIR_.
  * Liu and Lapata (2019) Liu, Y.; and Lapata, M. 2019.  Text Summarization with Pretrained Encoders.  In _EMNLP-IJCNLP_.
  * Lu, Jiao, and Zhang (2020) Lu, W.; Jiao, J.; and Zhang, R. 2020.  TwinBERT: Distilling Knowledge to Twin-Structured Compressed BERT Models for Large-Scale Retrieval.  In _CIKM_.
  * Lydick, Epstein, and White (1995) Lydick, E.; Epstein, R. S.; and White, D. H. J. 1995.  Area under the curve: A metric for patient subjective responses in episodic diseases.  _Quality of Life Research_.
  * Ma et al. (2008) Ma, H.; Yang, H.; King, I.; and Lyu, M. R. 2008.  Learning latent semantic relations from clickthrough data for query suggestion.  In _CIKM_.
  * Martino et al. (2017) Martino, G. D. S.; Romeo, S.; Barrón-Cedeño, A.; Joty, S. R.; Màrquez, L.; Moschitti, A.; and Nakov, P. 2017.  Cross-Language Question Re-Ranking.  In _SIGIR_.
  * McCann et al. (2017) McCann, B.; Bradbury, J.; Xiong, C.; and Socher, R. 2017.  Learned in translation: Contextualized word vectors.  _arXiv_.
  * McDonald, Brokos, and Androutsopoulos (2018) McDonald, R. T.; Brokos, G.; and Androutsopoulos, I. 2018.  Deep Relevance Ranking using Enhanced Document-Query Interactions.  In _EMNLP_.
  * Mihalcea, Corley, and Strapparava (2006) Mihalcea, R.; Corley, C. D.; and Strapparava, C. 2006.  Corpus-based and Knowledge-based Measures of Text Semantic Similarity.  In _AAAI_.
  * Mikolov et al. (2013) Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G. S.; and Dean, J. 2013.  Distributed representations of words and phrases and their compositionality.  In _NIPS_.
  * Mitra and Craswell (2018) Mitra, B.; and Craswell, N. 2018.  An Introduction to Neural Information Retrieval.  _FTIR_.
  * Nie (2010) Nie, J.-Y. 2010.  Cross-language information retrieval.  _Synthesis Lectures on Human Language Technologies_.
  * Nigam et al. (2019) Nigam, P.; Song, Y.; Mohan, V.; Lakshman, V.; Ding, W. A.; Shingavi, A.; Teo, C. H.; Gu, H.; and Yin, B. 2019.  Semantic Product Search.  In _SIGKDD_.
  * Radlinski, Kurup, and Joachims (2008) Radlinski, F.; Kurup, M.; and Joachims, T. 2008.  How does clickthrough data reflect retrieval quality?  In _CIKM_.
  * Reimers and Gurevych (2019) Reimers, N.; and Gurevych, I. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  In _EMNLP-IJCNLP_.
  * Sabet et al. (2019) Sabet, A.; Gupta, P.; Cordonnier, J.; West, R.; and Jaggi, M. 2019.  Robust Cross-lingual Embeddings from Parallel Sentences.  _CoRR_.
  * Sabour, Frosst, and Hinton (2017) Sabour, S.; Frosst, N.; and Hinton, G. E. 2017.  Dynamic routing between capsules.  _NIPS_.
  * Sanh et al. (2019) Sanh, V.; Debut, L.; Chaumond, J.; and Wolf, T. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv_.
  * Sasaki et al. (2018) Sasaki, S.; Sun, S.; Schamoni, S.; Duh, K.; and Inui, K. 2018.  Cross-lingual learning-to-rank with shared representations.  In _NAACL-HLT_.
  * Shen et al. (2014) Shen, Y.; He, X.; Gao, J.; Deng, L.; and Mesnil, G. 2014.  Learning semantic representations using convolutional neural networks for web search.  In _WWW_.
  * Su et al. (2021) Su, J.; Cao, J.; Liu, W.; and Ou, Y. 2021.  Whitening Sentence Representations for Better Semantics and Faster Retrieval.  _CoRR_.
  * Sun et al. (2019) Sun, S.; Cheng, Y.; Gan, Z.; and Liu, J. 2019.  Patient Knowledge Distillation for BERT Model Compression.  In _EMNLP-IJCNLP_.
  * Sun and Duh (2020) Sun, S.; and Duh, K. 2020.  CLIRMatrix: A massively large collection of bilingual and multilingual datasets for Cross-Lingual Information Retrieval.  In _EMNLP_.
  * Sun, Sia, and Duh (2020) Sun, S.; Sia, S.; and Duh, K. 2020.  CLIReval: Evaluating Machine Translation as a Cross-Lingual Information Retrieval Task.  In _ACL_.
  * Tai, Ren, and Kita (2002) Tai, X.; Ren, F.; and Kita, K. 2002.  An information retrieval model based on vector space method by supervised learning.  _Inf. Process. Manag._
  * Tang et al. (2019) Tang, R.; Lu, Y.; Liu, L.; Mou, L.; Vechtomova, O.; and Lin, J. 2019.  Distilling task-specific knowledge from bert into simple neural networks.  _arXiv_.
  * Vulic and Moens (2015) Vulic, I.; and Moens, M. 2015.  Monolingual and Cross-Lingual Information Retrieval Models Based on (Bilingual) Word Embeddings.  In _SIGIR_.
  * Wu and He (2010) Wu, D.; and He, D. 2010.  A study of query translation using google machine translation system.  In _IEEE_.
  * Xiong et al. (2017) Xiong, C.; Dai, Z.; Callan, J.; Liu, Z.; and Power, R. 2017.  End-to-end neural ad-hoc ranking with kernel pooling.  In _SIGIR_.
  * Yang et al. (2020) Yang, Y.; Cer, D.; Ahmad, A.; Guo, M.; Law, J.; Constant, N.; Ábrego, G. H.; Yuan, S.; Tar, C.; Sung, Y.; Strope, B.; and Kurzweil, R. 2020.  Multilingual Universal Sentence Encoder for Semantic Retrieval.  In _ACL_.
  * Yang et al. (2017) Yang, Z.; Dai, Z.; Salakhutdinov, R.; and Cohen, W. W. 2017.  Breaking the softmax bottleneck: A high-rank RNN language model.  _arXiv_.
  * Yao et al. (2020a) Yao, L.; Yang, B.; Zhang, H.; Chen, B.; and Luo, W. 2020a.  Domain Transfer based Data Augmentation for Neural Query Translation.  In Scott, D.; Bel, N.; and Zong, C., eds., _COLING_.
  * Yao et al. (2020b) Yao, L.; Yang, B.; Zhang, H.; Luo, W.; and Chen, B. 2020b.  Exploiting Neural Query Translation into Cross Lingual Information Retrieval.  _CoRR_.
  * Yarmohammadi et al. (2019) Yarmohammadi, M.; Ma, X.; Hisamoto, S.; Rahman, M.; Wang, Y.; Xu, H.; Povey, D.; Koehn, P.; and Duh, K. 2019.  Robust Document Representations for Cross-Lingual Information Retrieval in Low-Resource Settings.  In _MT_.
  * Yilmaz et al. (2019) Yilmaz, Z. A.; Wang, S.; Yang, W.; Zhang, H.; and Lin, J. 2019.  Applying BERT to document retrieval with birch.  In _EMNLP-IJCNLP_.
  * Zhang and Zhai (2015) Zhang, Y.; and Zhai, C. 2015.  Information retrieval as card playing: A formal model for optimizing interactive retrieval interface.  In _SIGIR_.
  * Zhou et al. (2012) Zhou, D.; Truran, M.; Brailsford, T. J.; Wade, V.; and Ashman, H. 2012.  Translation techniques in cross-language information retrieval.  _ACM Comput. Surv._


# BALANCING DOMAIN EXPERTS FOR LONG-TAILED CAMERA-TRAP RECOGNITION

###### Abstract

Label distributions in camera-trap images are highly imbalanced and long-
tailed, resulting in neural networks tending to be biased towards head-classes
that appear frequently. Although long-tail learning has been extremely
explored to address data imbalances, few studies have been conducted to
consider camera-trap characteristics, such as multi-domain and multi-frame
setup. Here, we propose a unified framework and introduce two datasets for
long-tailed camera-trap recognition. We first design domain experts, where
each expert learns to balance imperfect decision boundaries caused by data
imbalances and complement each other to generate domain-balanced decision
boundaries. Also, we propose a flow consistency loss to focus on moving
objects, expecting class activation maps of multi-frame matches the flow with
optical flow maps for input images. Moreover, two long-tailed camera-trap
datasets, WCS-LT and DMZ-LT, are introduced to validate our methods.
Experimental results show the effectiveness of our framework, and proposed
methods outperform previous methods on recessive domain samples.

Index Terms—  Long-tailed recognition, Multi-domain and multi-frame camera-
trap dataset, Flow consistency

##  1 Introduction

Biologists and ethologists often use camera-traps to capture animals
inconspicuously to study the population biology and dynamics [1]. While these
cameras automatically collect massive data, identifying species by humans is
time-consuming and labor-intensive, limiting research productivity. Therefore,
deep neural networks [2, 3] have recently received attention for their ability
to automate the identification process, making camera-trap studies scalable
[4, 5]. Nevertheless, neural networks tend to be biased towards the species
that frequently appear, limiting studies that require diverse animal species,
specifically on endangered species.

Early camera-trap recognition methods focus on long-tailed recognition to make
the neural network more tail-sensitive, and prevailing methods are summarized
as follows: Re-weighting the loss [6, 7]; Re-sampling the data for minor
classes [8]; Transfer learning [9]. Recently, multi-expert networks have
achieved considerable successes in long-tailed recognition by forcing experts
to learn each classifier for different sub-groups in parallel [10, 11].

Fig. 1: Two classes are highly imbalanced between two domains. Existing long-
tailed recognition methods resolve the imbalanced label distribution (black
dotted line). However, the discriminability of recessive domain features is
not sufficiently improved due to data imbalances, resulting in samples being
located near the boundary. Consequently, IR boundaries (blue dotted line) and
RGB boundaries (RGB dotted line) are still biased towards the head-class of
each domain. Thus, we propose that domain experts balanced by a simple re-
weighting method give a margin for tail-classes of each domain, and experts
complement each other to generate domain-balanced decision boundaries (blue-
black-red solid line). Fig. 2: Network architecture for training and
inference. In the training scheme, batches are processed to estimate the past
flow $f_{2\to 1}$ and the future flow $f_{2\to 3}$, and also processed to
feature extractor $W_{\theta}$. The output feature is then processed into each
expert $\mbox{EXP}_{i}$ (i.e., $\psi_{i}$) and generate class activation map
$M^{i}_{y}$. The classification loss $L^{i}_{cls}$ is applied for each expert,
and the flow consistency loss $L^{i}_{fc}$ is applied on each expert with
$f_{2\to 1}$ and $f_{2\to 3}$. In the inference time, a single input image is
processed into $W_{\theta}$, and the full-domain expert and the corresponding
expert complement each other to output the label.

Despite these efforts to prefer tail-classes, limited efforts have been made
to address the data imbalance between domains when images are acquired from
multiple domains with different label distributions. Especially in camera-trap
images, the samples of diurnal (e.g., _marten_) and nocturnal (e.g.,
_raccoon_) animals are biased in the corresponding domain, respectively,
resulting in the previous methods being often biased towards the dominant
domain. Therefore, the boundary of the samples in the recessive domain may
have the potential to shrink, degrading the classification performance.

In this paper, we propose domain experts that mitigate the bias by combining
decision boundaries, where domain experts are separately learned from each
domain. There are two types of experts, the sub-domain expert and the full-
domain expert. Exclusive sub-domain experts, one is for the night (i.e., IR)
and the other is for the day (i.e., RGB), are individually specialized in that
domain, and the focal loss [12] is applied to balance the imperfect decision
boundary caused by the data imbalance. The full-domain expert learns from all
input images since IR and RGB images are essential for learning object
boundaries and contextual information, respectively. The full-domain expert
and two sub-domain experts complement each other to create better domain-
balanced decision boundaries, and details are shown in Fig. 1.

While previous methods treat successive images taken by the camera-trap as
independent images, we further propose a flow consistency loss for each expert
to leverage the multi-frame information. We regulate the class activation map
of multi-frames following the optical flow map estimated from pre-trained PWC-
Net [13]. Thus, the flow consistency loss enhances experts to pay more
attention to moving objects.

To validate our method, we introduce two camera-trap datasets, WCS-LT and DMZ-
LT, which are multi-domain and multi-frame with long-tailed distributions. In
addition, we evaluate the accuracy on these datasets and show that our method
outperforms the previous methods for samples from the recessive domain as well
as the dominant domain.

##  2 METHOD

###  2.1 Network Architecture

The architecture of the proposed network is shown in Fig. 2, and multiple
experts are trained in parallel with a shared backbone. We use three
consecutive frames as an input sequence, and details are described in Section
3. Existing classifiers tend to perform better on the dominant domain samples
than on the recessive domain samples; however, domain experts can mitigate the
bias. Therefore, we design domain experts consisting of the full-domain expert
and the $K$ sub-domain experts. We fix $K=2$ to treat _day_ and _night_ domain
in the camera-trap setup. These two types of experts are complemented each
other from two aspects: (1) The full-domain expert learns valuable information
from both domains and makes robust predictions but is biased towards dominant
domain samples; (2) Sub-domain experts support the full-domain expert to
predict without prejudices and give confidence to the prediction.

###  2.2 Training Scheme

In this section, we briefly illustrate the training scheme for a input
sequence $\mathcal{S}=\left\\{I_{1},I_{2},I_{3}\right\\}$ and experts
$\Psi=\left\\{\psi_{full},\psi_{day},\psi_{night}\right\\}$. Here, the domain
set and the class set are defined as
$\mathbb{D}=\left\\{\mbox{day},\mbox{night}\right\\}$ and
$\mathbb{C}=\left\\{1,2,\cdots,C\right\\}$, respectively. The domain
$z\in\mathbb{D}$ of $S$ is determined by ensuring that the input values of
each channel are identical, as IR images are gray-scale. We denote
$y\in\mathbb{C}$ as the class label of $S$, and determine whether the inputs
are majority samples (MJs) or minority samples (MNs) depending on whether $z$
is the dominant domain of $y$.

While $\psi_{full}$ uses all input sequences, each sub-domain expert uses the
sequences of the corresponding domain. With this data split mechanism,
$\psi_{day}$ and $\psi_{night}$ learn the domain-specific decision boundaries
without being hindered by data imbalances between domains. Following [11, 14],
we use ResNet-50 [3] as a backbone and define each expert $\psi_{i}\in\Psi$ as
a residual block followed by a global average pooling layer and a learnable
weight scaling classifier. Consequently, output logits before SoftMax
operation of $\psi_{i}$ are
$\textbf{x}_{i,1},\textbf{x}_{i,2},\textbf{x}_{i,3}\in\mathbb{R}^{1\times C}$.
To avoid interfering with each other’s learning, loss functions are applied to
the experts separately. First, we use the focal loss [12] for $\psi_{i}$ as
the classification loss as

$L^{i}_{cls}=-\sum^{3}_{j=1}{(1-\sigma(\textbf{x}_{i,j})_{y})^{\gamma}\log(\sigma(\textbf{x}_{i,j})_{y})},$
(1)

where $\sigma(\textbf{x}_{i,j})_{y}\in\mathbb{R}$ is the output logit of the
class $y$ after the SoftMax operation for the input logit $\textbf{x}_{i,j}$,
and we fix $\gamma=5$.

To further increase the discriminability of each expert, a flow consistency
loss is proposed to make flow-consistent experts expect to pay more attention
to moving objects. We apply the flow consistency loss for the class activation
map of each expert, where the class activation map of multiple frames to have
a flow-consistent with the optical flow map estimated in the pre-trained PWC-
Net [13].

We first extract the class activation map $M^{ij}_{y}$ for the class label $y$
with the $j$-th frame and $\psi_{i}$ as

$M^{ij}_{y}=\sum_{k}{w^{i}_{k,y}A^{ij}_{k}},$ (2)

where $w^{i}_{k,y}\in\mathbb{R}$ is the fully-connected layer’s weight of
$\psi_{i}$ at the $k$-th row and the $y$-th column, and $A^{ij}_{k}$ is the
$k$-th channel of the feature map at the last convolution layer of $\psi_{i}$
for the $j$-th frame. In the context of [14], we freeze the feature map to
allow the gradient back-propagates only to the fully-connected layer.

With two flow maps estimated from the pre-trained PWC-Net, a past flow map
$f_{2\to 1}$ and a future flow map $f_{2\to 3}$, we generate warped maps
$\hat{M}^{i,1}_{y}$ and $\hat{M}^{i,3}_{y}$ from $M^{i,1}_{y}$ and
$M^{i,3}_{y}$, respectively. Then, the flow consistency loss is applied for
$\psi_{i}$ to match the warped maps with $M^{i,2}_{y}$ as

$L^{i}_{fc}=L_{ph}(M^{i,2}_{y},\hat{M}^{i,1}_{y})+L_{ph}(M^{i,2}_{y},\hat{M}^{i,3}_{y}),$
(3)

where $L_{ph}$ is a photometric consistency loss which is commonly used for
self-supervised optical flow and depth estimation tasks [15, 16] as

$L_{ph}(a,b)=\frac{\alpha}{2}({1-SSIM_{a,b}})+(1-\alpha){||a-b||}_{1}.$ (4)

Here, we fix $\alpha=0.85$, and $SSIM_{a,b}$ is the structure similarity [17]
between $a$ and $b$. With a weight for the flow consistency loss $\beta=0.02$,
the overall loss function for $\psi_{i}$ is defined as

$L^{i}=L^{i}_{cls}+\beta L^{i}_{fc}.$ (5)

###  2.3 Inference Scheme

Different from the training phase, inferences are made on one image,
considering the camera-trap only captures a single image. Depending on the
domain $z$ of the input data, we use a full-domain expert $\psi_{full}$ and a
sub-domain expert $\psi_{z}$. Similar to [11], the output logit
$\textbf{x}_{z}\in\mathbb{R}^{1\times C}$ of $\psi_{z}$ is modified to be
${\tilde{\textbf{x}}}_{z}$ by the l2-norm of the fully-connected layer’s
weights as

${\tilde{\textbf{x}}}_{z}=\frac{\sqrt{\sum_{k}{\sum_{c\in\mathbb{C}}{({w^{z}_{k,c}})^{2}}}}}{\sqrt{\sum_{k}{\sum_{c\in\mathbb{C}}{({w^{full}_{k,c}})^{2}}}}}\cdot\textbf{x}_{z}.$
(6)

Then, the modified output logit is averaged over two experts as

$\textbf{x}=\frac{\textbf{x}_{full}+\tilde{\textbf{x}}_{z}}{2},$ (7)

and the estimated category is defined as

$y_{pred}=\operatorname*{argmax}_{c\in\mathbb{C}}{(\sigma(\textbf{x})_{c})}.$
(8)

##  3 Camera-Trap Datasets

(a)

(b)

(c)

Fig. 3: iWildCAM2020 [18] statistics of (a) The number of frames per a
sequence, (b) Accuracy according to the number of frames used. (c) Long-tailed
label distributions of WCS-LT (left) and DMZ-LT (right).

Dataset | Method | Top-1 Accuracy (%)
---|---|---
Many | Medium | Few | Major | Minor | All
Balance | Imbalance | Total | Balance | Imbalance | Total
WCS-LT | baseline (ResNet-50) | 88.0 | 60.6 | 36.9 | 79.3 | 66.5 | 77.9 | 81.3 | 59.8 | 78.8 | 78.4
Focal loss [12] | 89.7 | 62.2 | 39.9 | 82.2 | 67.9 | 80.6 | 82.4 | 57.6 | 79.6 | 80.1
CB loss [6] | 89.2 | 58.9 | 36.9 | 80.0 | 70.5 | 79.0 | 81.0 | 58.9 | 78.5 | 78.7
LDAM+DRW [7] | 88.9 | 62.1 | 44.4 | 80.1 | 67.9 | 78.7 | 83.1 | 61.2 | 80.7 | 79.7
ACE (3 experts) [11] | 80.4 | 59.9 | 62.6 | 74.8 | 69.6 | 74.2 | 77.0 | 52.7 | 74.2 | 74.2
Ours | 89.8 | 66.6 | 52.0 | 82.2 | 75.5 | 81.4 | 84.6 | 64.7 | 82.4 | 81.9
DMZ-LT | baseline (ResNet-50) | 50.0 | 59.6 | - | 50.6 | 89.9 | 51.7 | 51.8 | 37.7 | 51.4 | 51.5
Focal loss [12] | 48.8 | 59.8 | - | 49.4 | 88.4 | 50.4 | 51.1 | 39.1 | 50.7 | 50.6
CB loss [6] | 51.1 | 45.1 | - | 51.0 | 78.3 | 51.7 | 49.7 | 14.5 | 48.8 | 50.2
LDAM+DRW [7] | 52.2 | 65.1 | - | 57.1 | 91.3 | 58.0 | 50.6 | 46.4 | 50.5 | 54.2
ACE (3 experts) [11] | 64.9 | 42.7 | - | 54.9 | 87.0 | 55.7 | 50.5 | 31.9 | 50.0 | 52.9
Ours | 56.6 | 62.9 | - | 61.4 | 81.2 | 61.9 | 53.0 | 65.2 | 53.3 | 57.6
|  |  |  |  |  |  |  |  |  |  |

Table 1: Comparison on long-tailed camera-trap datasets, WCS-LT and DMZ-LT. We
show that our proposed method outperforms the previous methods in most
evaluation metrics. Best results in each metric are in bold. Note that we
except the accuracy for the few-shot split in the DMZ-LT dataset since there
is one category that belongs to the few-shot split.

We explore the relationship between the number of frames in a sequence and the
classification accuracy since iWildCAM2020 [18] provides the frame
information. Given that camera-traps capture images during the object moves,
Fig. 3(a) shows that most sequences consisting of up to three frames capture
dynamic objects, and the rest of the sequences capture barely moving objects.
Also, we observe that neural networks overfit to these redundant frames.
Figure 3(b) shows the classification performance increases as more images are
used rather than one image per sequence, and the best performance is when the
first three frames are used for training while the performance deteriorates
when all frames are used.

While existing camera-trap datasets [18, 19] consider consecutive frames as
independent images, and also disregard prior knowledge for each domain, we
introduce two benchmarks to cover general camera-trap settings with three
characteristics: (1) Training on multi-frame sequences and testing with a
single image; (2) Multi-domain with different long-tailed label distributions;
(3) Domain-Balanced test dataset.

WCS-LT Dataset is provided by the Wildlife Conservation Society (WCS), and
Beery _et al_. [18] split the data by camera location, focusing on predicting
unseen camera-trap images. We use the annotated train split of [18], which
contains 217,959 images from 22,111 sequences where only 8,563 sequences
include animal species. Here, we use sequences with at least three frames and
then select the first three frames according to our observation. Furthermore,
we filter out dominant domain samples to fit the number of recessive domain
samples to create a domain-balanced test dataset.

We use 60% of filtered sequences as the training set and 40% as the test set,
and select only categories with at least one data in each domain and each
split (i.e., _train_ and _test_). The training and test set contains 7,416 and
3,990 images, respectively, collected from 211 locations and 34 species
represented in the dataset.

DMZ-LT Dataset is collected from the Korean Demilitarized Zone (DMZ), which is
currently inaccessible due to the ceasefire. The 4,772 sequences consisting of
three consecutive frames contain 10 species captured in 99 locations. The two
species (i.e., _elk_ and _wild boar_) account for 70% of the entire dataset,
resulting in the highly imbalanced label distribution that makes the task
challenging. We also filter out dominant domain samples to create a domain-
balanced test dataset. Then, we split half of the entire sequences into the
training set and the other half into the test set. The training set contains
7,146 images, and the test set contains 5,148 images.

##  4 Experiments

###  4.1 Settings

Implementation Details. During training, we set the base learning rate
$\eta_{full}$ of the SGD optimizer to 0.01 for WCS-LT and 0.001 for DMZ-LT for
100 epochs, and batch size is set to 48. The $\psi_{full}$ uses $\eta_{full}$,
while the learning rate $\eta_{i}$ of each sub-domain expert $\psi_{i}$
follows the Linear Scaling Rule [20] as

$\eta_{i}=\eta_{full}\cdot\frac{\sum_{c\in\mathbb{C}}{n^{c}_{i}}}{\sum_{z\in\mathbb{D}}{\sum_{c\in\mathbb{C}}{n^{c}_{z}}}},$
(9)

where $n^{c}_{z}$ is the number of samples for domain $z$ and label $c$. Input
images are resized to 256 $\times$ 256, flipped horizontally with a
probability of $\frac{1}{2}$. Moreover, $L^{full}$ updates the backbone and
parameters of $\psi_{full}$, and each $L^{i}$ only updates $\psi_{i}$ to
alleviate the learning conflict.

Evaluation Metrics. We first evaluate the accuracy on many-shot (more than 100
samples), medium-shot (20 $\sim$ 100 samples), and few-shot (less than 20
samples) splits, which are generally evaluated for long-tailed recognition
tasks. To better understand the performance of different methods for multiple
long-tailed distributions, we calculate the accuracy for major samples (MJs)
and minor samples (MNs) separately. We further split $\mathbb{C}$ into a
balanced class set and an imbalanced class set according to the ratio of the
number of samples in the domain, i.e., the imbalanced class set is defined as

$\mathbb{C}_{imbal}=\left\\{c\in\mathbb{C}\ |\
\frac{\max_{z\in\mathbb{D}}n^{c}_{z}}{\min_{z\in\mathbb{D}}n^{c}_{z}}\geq
3\right\\}.$ (10)

Then, we define the remaining class set as a balanced class set
$\mathbb{C}_{bal}$. This results in 32.3% categories of WCS-LT and 30%
categories of DMZ-LT being in $\mathbb{C}_{imbal}$. Taken together, we
evaluate the accuracy for MJs and MNs of $\mathbb{C}_{bal}$ and
$\mathbb{C}_{imbal}$. Note that the average accuracy of MJs and MNs is equal
to the total accuracy since we use the domain-balanced test dataset.

###  4.2 Experimental Results

In this section, we validate our proposed method with comparison to previous
long-tailed recognition algorithms [6, 7, 11, 12], and experimental results
are represented in Table 1.

For the WCS-LT dataset, our method outperforms other methods for all
evaluation metrics except for the few-shot split. Although ACE [11] achieves
the best performance on the few-shot split, total accuracy is much lower than
baseline since the network is biased toward MJs. We achieve remarkable
improvement on MNs, exceeding the baseline by a margin of 3.3%p for
$\mathbb{C}_{bal}$ and 4.9%p for $\mathbb{C}_{imbal}$. Interestingly, our
method improves even for MJs on $\mathbb{C}_{imbal}$ by 9%p, which means that
domain experts complement each other to potentiate the classification
confidence for MJs.

For the DMZ-LT dataset, the difference in accuracy between MJs and MNs of
$\mathbb{C}_{bal}$ is about 1%p, even MNs are more accurate, whereas
$\mathbb{C}_{imbal}$ has a difference of more than 50%p. These biases towards
the dominant domain attenuate the recessive domain prediction, leading to the
shrunken decision boundary for the recessive domain. Our method exceeds the
baseline on MNs for $\mathbb{C}_{imbal}$ by a margin of 28%p, meaning that the
proposed framework resolves the bias even with severe data imbalances.

###  4.3 Ablation Study

We also conduct an ablation study to confirm that each part of our unified
framework significantly improves the performance of camera-trap recognition.

|  |  |  |
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
(a) Input Images |  (b) Focal [12] |  (c) Focal w/ FC |  (d) Ours (full) |  (e) Flow Maps

Fig. 4: Visualizations of class activation maps and optical flow maps for
_”cephalophus silvicultor”_. The upper three rows are for RGB sequences, and
the lower three rows are for IR sequences. Also, two consecutive flow maps for
each domain represent $f_{2\to 1}$ and $f_{2\to 3}$, respectively.

Baseline |  DE |  FC | Top-1 Accuracy (%)
---|---|---|---
Major | Minor | All
Focal Loss [12] |  |  | 80.6 | 79.6 | 80.1
✓ |  | 81.2 | 80.2 | 80.7
| ✓ | 79.4 | 79.5 | 79.4
✓ | ✓ | 81.4 | 82.4 | 81.9
|  |  |  |  |

Table 2: Ablation study for our proposed method on the WCS-LT dataset. We use
a simple long-tailed recognition method [12] as the baseline. DE means the
domain-experts and FC means the flow consistency loss. Major and Minor metrics
are the total accuracy for entire major and minor samples, respectively. Best
results in each metric are in bold.

Table 2 shows the quantitative result to verify the effect of domain experts
and the flow consistency loss. Domain experts improve the classification
performance for all evaluation metrics, and further improvements are achieved
when the flow consistency loss is applied to experts. Surprisingly, the flow
consistency loss does not address the data imbalances in baseline, but
synergizes with domain experts to considerably improve the performance.
Collectively, our unified framework improves 1.8%p for MJs and 2.8%p for MNs
compared to the focal loss [12].

Qualitative results in Fig. 4 indicate the flow consistency loss regulates the
classifier to pay more attention to moving objects. Specifically, our method
focuses on the general contextual information of the animal in RGB images, and
the class activation map of our method mainly highlights the moving animal
while the baseline focuses on the brightest object in a situation with low
light conditions. In this regard, qualitative and quantitative results
strengthen the position that complementary domain experts have better
discriminability and are superior to the baseline.

##  5 Conclusion

In this work, we have proposed a unified framework for long-tailed camera-trap
recognition and introduced two benchmark datasets, WCS-LT and DMZ-LT. The main
contribution is that domain experts are balanced through the loss re-weighting
and complement each other to provide the domain-balanced decision boundaries.
We also design the flow consistency loss that experts pay more attention to
moving objects in camera-trap images. We believe that our datasets will
contribute to camera-trap studies. In the future, we plan to extend our
framework for domain generalization tasks, considering long-tailed
distributions for diverse domains.

##  6 Acknowledgement

This work was supported by the National Research Foundation of Korea (NRF)
grand founded by the Korea Government (MSIT) (NRF-2018R1A5A7025409)

## References

  * [1] A Cole Burton, Eric Neilson, Dario Moreira, Andrew Ladle, Robin Steenweg, Jason T Fisher, Erin Bayne, and Stan Boutin,  “Wildlife camera trapping: a review and recommendations for linking surveys to ecological processes,”  Journal of Applied Ecology, vol. 52, no. 3, pp. 675–685, 2015.
  * [2] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  arXiv preprint arXiv:1409.1556, 2014.
  * [3] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,  “Deep residual learning for image recognition,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [4] Hyojun Go, Junyoung Byun, Byeongjun Park, Myung-Ae Choi, Seunghwa Yoo, and Changick Kim,  “Fine-grained multi-class object counting,”  in Proceedings of the 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2021, pp. 509–513.
  * [5] Mohammad Sadegh Norouzzadeh, Anh Nguyen, Margaret Kosmala, Alexandra Swanson, Meredith S Palmer, Craig Packer, and Jeff Clune,  “Automatically identifying, counting, and describing wild animals in camera-trap images with deep learning,”  Proceedings of the National Academy of Sciences, vol. 115, no. 25, pp. E5716–E5725, 2018.
  * [6] Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie,  “Class-balanced loss based on effective number of samples,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 9268–9277.
  * [7] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma,  “Learning imbalanced datasets with label-distribution-aware margin loss,”  in Proceedings of the Advances in Neural Information Processing Systems, 2019.
  * [8] Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang,  “Unsupervised domain adaptation for semantic segmentation via class-balanced self-training,”  in Proceedings of the European conference on computer vision, 2018, pp. 289–305.
  * [9] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu,  “Large-scale long-tailed recognition in an open world,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 2537–2546.
  * [10] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu,  “Long-tailed recognition by routing diverse distribution-aware experts,”  in Proceedings of the International Conference on Learning Representations, 2021.
  * [11] Jiarui Cai, Yizhou Wang, and Jenq-Neng Hwang,  “Ace: Ally complementary experts for solving long-tailed recognition in one-shot,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 112–121.
  * [12] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár,  “Focal loss for dense object detection,”  in Proceedings of the IEEE international conference on computer vision, 2017, pp. 2980–2988.
  * [13] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz,  “Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8934–8943.
  * [14] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis,  “Decoupling representation and classifier for long-tailed recognition,”  in Proceedings of the Eighth International Conference on Learning Representations (ICLR), 2020.
  * [15] Rico Jonschkowski, Austin Stone, Jonathan T Barron, Ariel Gordon, Kurt Konolige, and Anelia Angelova,  “What matters in unsupervised optical flow,”  in Proceedings of the European Conference on Computer Vision. Springer, 2020, pp. 557–572.
  * [16] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J Brostow,  “Digging into self-supervised monocular depth estimation,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 3828–3838.
  * [17] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli,  “Image quality assessment: from error visibility to structural similarity,”  IEEE transactions on image processing, vol. 13, no. 4, pp. 600–612, 2004.
  * [18] Sara Beery, Elijah Cole, and Arvi Gjoka,  “The iwildcam 2020 competition dataset,”  arXiv preprint arXiv:2004.10340, 2020.
  * [19] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie,  “The inaturalist species classification and detection dataset,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8769–8778.
  * [20] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He,  “Accurate, large minibatch sgd: Training imagenet in 1 hour,”  arXiv preprint arXiv:1706.02677, 2017.


* Brady (2013) Edwin Brady. 2013\.  Idris, a general-purpose dependently typed programming language: Design and implementation.  _Journal of Functional Programming_ 23, 5 (2013), 552–593.
  * Broadbent and Kobayashi (2013) Christopher Broadbent and Naoki Kobayashi. 2013.  Saturation-Based Model Checking of Higher-Order Recursion Schemes. In _Computer Science Logic 2013 (CSL 2013)_ _(Leibniz International Proceedings in Informatics (LIPIcs), Vol. 23)_ , Simona Ronchi Della Rocca (Ed.). 129–148.
  * Broadbent et al. (2010) Christopher H. Broadbent, Arnaud Carayol, C.-H. Luke Ong, and Olivier Serre. 2010.  Recursion Schemes and Logical Reflection. In _2010 25th Annual IEEE Symposium on Logic in Computer Science_. 120–129.
  * Carayol and Serre (2012) Arnaud Carayol and Olivier Serre. 2012.  Collapsible Pushdown Automata and Labeled Recursion Schemes: Equivalence, Safety and Effective Selection. In _2012 27th Annual IEEE Symposium on Logic in Computer Science_. 165–174.
  * Clairambault and Murawski (2013) Pierre Clairambault and Andrzej S. Murawski. 2013.  Böhm Trees as Higher-Order Recursive Schemes. In _IARCS Annual Conference on Foundations of Software Technology and Theoretical Computer Science (FSTTCS 2013)_ _(Leibniz International Proceedings in Informatics (LIPIcs), Vol. 24)_ , Anil Seth and Nisheeth K. Vishnoi (Eds.). 91–102.
  * Clarke (1997) Edmund M. Clarke. 1997\.  Model checking. In _Foundations of Software Technology and Theoretical Computer Science_ , S. Ramesh and G. Sivakumar (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 54–56.
  * Clarke et al. (2018) Edmund M Clarke, Thomas A Henzinger, Helmut Veith, Roderick Bloem, et al. 2018\.  _Handbook of model checking_. Vol. 10.  Springer.
  * Courcelle (1995) Bruno Courcelle. 1995\.  The monadic second-order logic of graphs IX: Machines and their behaviours.  _Theoretical Computer Science_ 151, 1 (1995), 125–162.  Selected Papers of the Workshop on Topology and Completion in Semantics.
  * Dal Lago et al. (2017) Ugo Dal Lago, Francesco Gavazzo, and Paul Levy. 2017\.  Effectful applicative bisimilarity: Monads, relators, and Howe’s method. In _Logic in Computer Science (LICS 2017)_. 1–12.
  * Davies and Pfenning (2000) Rowan Davies and Frank Pfenning. 2000.  Intersection Types and Computational Effects. In _Proceedings of the Fifth ACM SIGPLAN International Conference on Functional Programming_ _(ICFP ’00)_. Association for Computing Machinery, New York, NY, USA, 198–208.
  * de Groote (1994) Philippe de Groote. 1994\.  A CPS-translation of the $\lambda$$\mu$-calculus. In _Trees in Algebra and Programming—CAAP’94: 19th International Colloquium Edinburgh, UK, April 11–13, 1994 Proceedings 19_. Springer, 85–99.
  * Emerson and Jutla (1991) E.A. Emerson and C.S. Jutla. 1991.  Tree automata, mu-calculus and determinacy. In _[1991] Proceedings 32nd Annual Symposium of Foundations of Computer Science_. 368–377.
  * Freeman and Pfenning (1991) Tim Freeman and Frank Pfenning. 1991.  Refinement Types for ML. In _PLDI91_. 268–277.
  * Grädel et al. (2003) Erich Grädel, Wolfgang Thomas, and Thomas Wilke. 2003\.  _Automata, logics, and infinite games: a guide to current research_. Vol. 2500.  Springer.
  * Hague et al. (2008) M. Hague, A.S. Murawski, C.-H.L. Ong, and O. Serre. 2008\.  Collapsible Pushdown Automata and Recursion Schemes. In _2008 23rd Annual IEEE Symposium on Logic in Computer Science_. 452–461.
  * Hillerström and Lindley (2018) Daniel Hillerström and Sam Lindley. 2018.  Shallow Effect Handlers. In _Programming Languages and Systems_ , Sukyoung Ryu (Ed.). Springer International Publishing, Cham, 415–435.
  * Hillerström et al. (2017) Daniel Hillerström, Sam Lindley, Robert Atkey, and K. C. Sivaramakrishnan. 2017.  Continuation Passing Style for Effect Handlers. In _2nd International Conference on Formal Structures for Computation and Deduction (FSCD 2017)_ _(Leibniz International Proceedings in Informatics (LIPIcs), Vol. 84)_ , Dale Miller (Ed.). Schloss Dagstuhl–Leibniz-Zentrum fuer Informatik, Dagstuhl, Germany, 18:1–18:19.
  * Hughes et al. (1996) John Hughes, Lars Pareto, and Amr Sabry. 1996.  Proving the correctness of reactive systems using sized types. In _Proceedings of the 23rd ACM SIGPLAN-SIGACT symposium on Principles of programming languages_. ACM, 410–423.
  * Johann et al. (2010) Patricia Johann, Alex Simpson, and Janis Voigtländer. 2010\.  A Generic Operational Metatheory for Algebraic Effects. In _2010 25th Annual IEEE Symposium on Logic in Computer Science_. 209–218.
  * Jung et al. (2018) Ralf Jung, Robbert Krebbers, Jacques-Henri Jourdan, Ales Bizjak, Lars Birkedal, and Derek Dreyer. 2018.  Iris from the ground up: A modular foundation for higher-order concurrent separation logic.  _Journal of Functional Programming_ 28 (2018), e20.
  * Kammar et al. (2013) Ohad Kammar, Sam Lindley, and Nicolas Oury. 2013.  Handlers in Action. In _Proceedings of the 18th ACM SIGPLAN International Conference on Functional Programming_ (Boston, Massachusetts, USA) _(ICFP ’13)_. Association for Computing Machinery, New York, NY, USA, 145–158.
  * King (1976) James C King. 1976\.  Symbolic execution and program testing.  _Commun. ACM_ 19, 7 (1976), 385–394.
  * Knapik et al. (2001) Teodor Knapik, Damian Niwiński, and Paweł Urzyczyn. 2001\.  Deciding Monadic Theories of Hyperalgebraic Trees. In _Typed Lambda Calculi and Applications_ , Samson Abramsky (Ed.). Springer Berlin Heidelberg, 253–267.
  * Knapik et al. (2002) Teodor Knapik, Damian Niwiński, and Paweł Urzyczyn. 2002\.  Higher-Order Pushdown Trees Are Easy. In _Foundations of Software Science and Computation Structures_ , Mogens Nielsen and Uffe Engberg (Eds.). Springer Berlin Heidelberg, 205–222.
  * Kobayashi (2009) Naoki Kobayashi. 2009\.  Types and higher-order recursion schemes for verification of higher-order programs. In _Proceedings of the 36th annual ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages_. 416–428.
  * Kobayashi (2011) Naoki Kobayashi. 2011\.  A Practical Linear Time Algorithm for Trivial Automata Model Checking of Higher-Order Recursion Schemes. In _Foundations of Software Science and Computational Structures_ , Martin Hofmann (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 260–274.
  * Kobayashi et al. (2020) Naoki Kobayashi, Ugo Dal Lago, and Charles Grellois. 2020\.  On the termination problem for probabilistic higher-order recursive programs.  _Logical Methods in Computer Science_ 16 (2020).
  * Kobayashi and Igarashi (2013) Naoki Kobayashi and Atsushi Igarashi. 2013.  Model-Checking Higher-Order Programs with Recursive Types. In _Programming Languages and Systems_ , Matthias Felleisen and Philippa Gardner (Eds.). Springer Berlin Heidelberg, 431–450.
  * Kobayashi and Ong (2009) Naoki Kobayashi and C-H Luke Ong. 2009.  A type system equivalent to the modal mu-calculus model checking of higher-order recursion schemes. In _2009 24th Annual IEEE Symposium on Logic In Computer Science_. IEEE, 179–188.
  * Leroy (2009) Xavier Leroy. 2009\.  Formal Verification of a Realistic Compiler.  _Commun. ACM_ 52, 7 (jul 2009), 107–115.
  * Matache (2018) Cristina Matache. 2018\.  Program equivalence for algebraic effects via modalities.. In _Master’s thesis_. University of Oxford, Department of Computer Science.
  * Matache and Staton (2019) Cristina Matache and Sam Staton. 2019.  A Sound and Complete Logic for Algebraic Effects. In _Foundations of Software Science and Computation Structures - 22nd International Conference, FOSSACS 2019_ _(Lecture Notes in Computer Science, Vol. 11425)_. Springer, 382–399.
  * Moggi (1988) Eugenio Moggi. 1988\.  _Computational Lambda-calculus and Monads_.  University of Edinburgh, Department of Computer Science.
  * Nakamura et al. (2020) Yoshiki Nakamura, Kazuyuki Asada, Naoki Kobayashi, Ryoma Sin’ya, and Takeshi Tsukada. 2020.  On Average-Case Hardness of Higher-Order Model Checking. In _5th International Conference on Formal Structures for Computation and Deduction (FSCD 2020)_ _(Leibniz International Proceedings in Informatics (LIPIcs), Vol. 167)_. 21:1–21:23.
  * Neatherway et al. (2012) Robin P. Neatherway, Steven J. Ramsay, and Chih-Hao Luke Ong. 2012.  A Traversal-Based Algorithm for Higher-Order Model Checking.  _SIGPLAN Not._ 47, 9 (2012), 353–364.
  * Ong (2006) C-HL Ong. 2006.  On model-checking trees generated by higher-order recursion schemes. In _21st Annual IEEE Symposium on Logic in Computer Science (LICS’06)_. IEEE, 81–90.
  * Ong (2015) Luke Ong. 2015.  Higher-order model checking: An overview. In _2015 30th Annual ACM/IEEE Symposium on Logic in Computer Science_. IEEE, 1–15.
  * Plotkin (1975) G.D. Plotkin. 1975\.  Call-by-name, call-by-value and the $\lambda$-calculus.  _Theoretical Computer Science_ 1, 2 (1975), 125–159.
  * Plotkin and Power (2003) G. Plotkin and J. Power. 2003.  Algebraic Operations and Generic Effects.  _Applied Categorical Structures_ 11 (2003), 69–94.  https://doi.org/10.1023/A:1023064908962.
  * Plotkin and Pretnar (2008) Gordon Plotkin and Matija Pretnar. 2008.  A logic for algebraic effects. In _2008 23rd Annual IEEE Symposium on Logic in Computer Science_. IEEE, 118–129.
  * Plotkin and Pretnar (2009) Gordon Plotkin and Matija Pretnar. 2009.  Handlers of Algebraic Effects. In _Programming Languages and Systems_ , Giuseppe Castagna (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 80–94.
  * Pretnar (2015) Matija Pretnar. 2015\.  An introduction to algebraic effects and handlers. Invited tutorial paper.  _Electronic notes in theoretical computer science_ 319 (2015), 19–35.
  * Ramsay et al. (2014) Steven J. Ramsay, Robin P. Neatherway, and C.-H. Luke Ong. 2014\.  A Type-Directed Abstraction Refinement Approach to Higher-Order Model Checking.  _SIGPLAN Not._ 49, 1 (2014).
  * Salvati and Walukiewicz (2011) Sylvain Salvati and Igor Walukiewicz. 2011.  Krivine Machines and Higher-Order Schemes, In Automata, Languages and Programming.  _Information and Computation_ 239\.
  * Sekiyama et al. (2020) Taro Sekiyama, Takeshi Tsukada, and Atsushi Igarashi. 2020\.  Signature Restriction for Polymorphic Algebraic Effects.  _Proc. ACM Program. Lang._ 4, ICFP, Article 117 (2020), 30 pages.
  * Simpson and Voorneveld (2019) Alex Simpson and Niels Voorneveld. 2019.  Behavioural Equivalence via Modalities for Algebraic Effects.  _ACM Trans. Program. Lang. Syst._ 42, 1, Article 4 (2019), 45 pages.
  * Tobin-Hochstadt and Van Horn (2012) Sam Tobin-Hochstadt and David Van Horn. 2012.  Higher-order symbolic execution via contracts. In _Proceedings of the ACM international conference on Object oriented programming systems languages and applications_. 537–554.
  * Tsukada and Kobayashi (2014) Takeshi Tsukada and Naoki Kobayashi. 2014.  Complexity of Model-Checking Call-by-Value Programs. In _Foundations of Software Science and Computation Structures_ , Anca Muscholl (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 180–194.
  * Walukiewicz (1993) Igor Walukiewicz. 1993\.  On completeness of the mu-calculus. In _[1993] Proceedings Eighth Annual IEEE Symposium on Logic in Computer Science_. IEEE, 136–146.
  * Walukiewicz (2016) Igor Walukiewicz. 2016\.  Automata Theory and Higher-Order Model-Checking.  _ACM SIGLOG News_ 3, 4 (2016), 13–31.