#  ${\rm\textbf{R}^{\textbf{2}}\textbf{F}}$: A General Retrieval, Reading and
Fusion Framework
for Document-level Natural Language Inference

Hao Wang§, Yixin Cao†∗, Yangguang Li‡, Zhen Huang§
Kun Wang‡, Jing Shao‡
§ National University of Defense Technology
† Singapore Management University ‡ SenseTime

###### Abstract

Document-level natural language inference (DOCNLI) is a new challenging task
in natural language processing, aiming at judging the entailment relationship
between a pair of hypothesis and premise documents. Current datasets and
baselines largely follow sentence-level settings, but fail to address the
issues raised by longer documents. In this paper, we establish a general
solution, named Retrieval, Reading and Fusion (R2F) framework, and a new
setting, by analyzing the main challenges of DOCNLI: interpretability, long-
range dependency, and cross-sentence inference. The basic idea of the
framework is to simplify document-level task into a set of sentence-level
tasks, and improve both performance and interpretability with the power of
evidence. For each hypothesis sentence, the framework retrieves evidence
sentences from the premise, and reads to estimate its credibility. Then the
sentence-level results are fused to judge the relationship between the
documents. For the setting, we contribute complementary evidence and
entailment label annotation on hypothesis sentences, for interpretability
study. Our experimental results show that R2F framework can obtain state-of-
the-art performance and is robust for diverse evidence retrieval methods.
Moreover, it can give more interpretable prediction results. Our model and
code are released at https://github.com/phoenixsecularbird/R2F.

${\rm\textbf{R}^{\textbf{2}}\textbf{F}}$: A General Retrieval, Reading and
Fusion Framework
for Document-level Natural Language Inference

Hao Wang§, Yixin Cao†∗, Yangguang Li‡, Zhen Huang§ Kun Wang‡, Jing Shao‡ §
National University of Defense Technology † Singapore Management University ‡
SenseTime

11footnotetext: Corresponding Author.

##  1 Introduction

Natural Language Inference (NLI) is the task of determining whether a
hypothesis is entailed or not in a premise. While earlier works Bowman et al.
(2015); Williams et al. (2018); Wang et al. (2019); Nie et al. (2020) assume
that both hypothesis and premise are single sentences, recent research pays
more attention on document-level task, namely Document-level NLI (DOCNLI) Yin
et al. (2021). The task can enlarge the task scope to judge the variability of
semantic expression for many Natural Language Processing (NLP) tasks, e.g.,
exposure bias Bengio et al. (2015); Arora et al. (2022) alleviation for text
summarization Sandhaus (2008); Narayan et al. (2018), and human-manipulated
news articles recognition for automatic fake news detection Jawahar et al.
(2022); Huang et al. (2022).

Label: not_entailment Hypothesis: ①US cities along the Gulf of Mexico from
Alabama to eastern Texas were on alert last night as Hurricane Andrew headed
west after hitting southern Florida leaving at least eight dead, causing
severe property damage, and leaving 1.2 million homes without electricity.
②Gusts of up to 165 mph were recorded. ③It is the fiercest hurricane to hit
the US in 1989. ④As Andrew moved across the Gulf there was concern that it
might hit New Orleans, which would be particularly susceptible to flooding, or
smash into the concentrated offshore oil facilities. ⑤President Bush
authorized federal disaster assistance for the affected areas. Premise: ①US
CITIES along the Gulf of Mexico from Alabama to eastern Texas were on storm
watch last night as Hurricane Andrew headed west after sweeping across
southern Florida, causing at least eight deaths and severe property damage.
③The hurricane was one of the fiercest in the US in decades and the first to
hit Miami directly in a quarter of a century. $\cdots$ ④However, Hurricane
Andrew gathered fresh strength as it moved across the Gulf of Mexico and there
was concern last night that it might head towards New Orleans, which is
especially low lying and could suffer severe flood damage. $\cdots$ ④It could
threaten the large concentration of offshore oil production facilities in the
Gulf of Mexico. $\cdots$ ②Andrew, the first Caribbean hurricane of the season,
hit the eastern coast of Florida early yesterday, gusting up to 165mph.
$\cdots$ ①The Florida Power and Light company said that about 1.2m of its
customers, or 32 per cent, were without power. $\cdots$ ⑤President Bush
authorized federal disaster assistance for the affected areas and made plans
for an inspection tour of the state. $\cdots$ Figure 1: A sample of DOCNLI
dataset. For each sample, only the entailment label between the documents is
annotated. For display, we mark each hypothesis sentence and its corresponding
premise sentences (namely the evidences, not annotated in the original
dataset) with the same number and color. The sample is annotated as not
entailment due to the disinformation of “in 1989” in the third hypothesis
sentence. The premise is partly omitted.

Compared with sentence-level NLI, DOCNLI poses many new challenges, while
there are only a few datasets and models. In terms of datasets, Yin et al.
(2021) reformat some mainstream NLP tasks, e.g., text summarization and
question answering, and build the first large scale dataset DOCNLI with over 1
million document pairs111Yin et al. (2021) propose the task and annotate the
dataset with the same name DOCNLI.. However, the dataset does not provide
evidence annotation about how the labels are inferred, i.e., which hypothesis
sentences lead to semantic inconsistency, or which premise sentences help to
decide the entailment relationship. As shown in Figure 1, although the sample
is annotated as not entailment, most of the hypothesis sentences are actually
entailed. By contrast, the detailed disinformation of “in 1989” in the third
hypothesis sentence eventually decides the entailment relationship between the
documents. For each hypothesis sentence, only several premise sentences are
enough to serve as the exact evidence to judge its own sentence-level
entailment label.

In this paper, we argue that evidence discovery is important and challenging
for DOCNLI. Our pilot experiments in Section 4.3 and 4.5 show that randomly
selected evidences can still contribute to comparable performance. Thus, only
the black-box models may be not so convincing. However, to annotate evidence
for evaluation is non-trivial. For each hypothesis sentence, on one hand, it
may refer to multiple premise sentences. On the other hand, there may be
several evidence groups, where each group can independently serve the label
prediction. We highlight this as interpretability challenge.

In term of models, current baselines Yin et al. (2021); Zhong et al. (2020)
still largely follow sentence-level NLI. They either concatenate two documents
for mutual information interaction for classification, or encode them
separately for semantic match with document-level representations. However,
except for the interpretability issue, they will leave the following
challenges unexplored:

$\bullet$ Long-range Dependency Modeling The task requests to handle a pair of
long documents at the same time, where we observe that 29.81% samples of
DOCNLI dataset Yin et al. (2021) contain more than 500 words222A word may
correspond to multiple tokens for PLMs., while 10.47% samples contain more
than 1000 words. This will not only exceed the input limit of Pre-trained
Language Models (PLMs), but also make it far more difficult to capture long-
range dependency. Necessary information interaction between the hypothesis and
some key premise sentences may not be guaranteed. Besides, most contexts are
uninformative for entailment inference and will only serve as noise.

$\bullet$ Cross-sentence Inference To judge the relationship between the
documents, it is supposed to consider all hypothesis sentences, where the
detailed disinformation issue still remains unsolved. Besides, the
verification of one hypothesis sentence may request to combine multiple and
distant premise sentences, different from the sentence pair mode in sentence-
level NLI. As shown in Figure 1, to process the first one, it needs to take
both the first and sixth premise sentences (all in red fonts).

Figure 2: $\rm{R^{2}F}$ framework. For each hypothesis sentence, the framework
firstly retrieves evidence sentences from the premise, and then reads to
estimate the credibility upon the evidences. Finally it fuses the sentence-
level results, and judges the entailment relationship between these two
documents. $\hat{y}_{i}$ and $\hat{y}_{HP}$ are the credibility score of the
i-th hypothesis sentence and the sample, while $Evi_{ij}$ is the j-th evidence
of the i-th hypothesis sentence.

In this paper, we establish a general solution, named Retrieval, Reading and
Fusion (R2F) framework, and a new setting for the task. The basic idea of the
framework is to simplify document-level classification task into a set of
sentence-level tasks, and then improve both performance and interpretability
with the power of evidence. Specifically, the framework splits the hypothesis
document into sentences. Then for each hypothesis sentence, it retrieves
evidence sentences from the premise, and reads to estimate its credibility
score upon the evidences. Finally, it fuses the sentence-level results and
judge the entailment relationship between the two documents. For the setting,
we contribute complementary fine-grained annotations for interpretability
study. For each hypothesis sentence, we manually annotate entailment label and
several evidence groups, where each group is enough to independently infer the
label. In summary, our contributions are as follows:

$\bullet$ We propose a Retrieval, Read and Fusion framework as a general
solution for DOCNLI task.

$\bullet$ We contribute complementary evidence and entailment label annotation
for each hypothesis sentence on a subset of DOCNLI dataset for
interpretability study.

$\bullet$ Our experimental results on DOCNLI dataset indicate that the
framework obtains state-of-the-art performance. Besides, it is robust for
diverse retrieval methods. Moreover, the framework can give more interpretable
prediction results.

##  2 ${\rm\textbf{R}^{\textbf{2}}\textbf{F}}$ Framework

Our $\rm{R^{2}F}$ framework aims at a general solution for DOCNLI task with
interpretability, i.e., to obtain corresponding evidence and predict
entailment label for each hypothesis sentence. As shown in Figure 2, the
framework consists of 3 components, namely evidence retrieval, reading for
credibility estimation, and credibility fusion. For efficiency, the retrieval
component is an independent unit to provide evidence input for the other two
components, which are optimized jointly.

###  2.1 Task Formulation

Similar to previous sentence-level NLI tasks, for each sample in DOCNLI task,
given a hypothesis document H and a premise document P, it is requested to
judge the entailment relationship R between these two documents. Here, R $\in$
{“entailment”, “not_entailment”} for DOCNLI dataset, but may not be restricted
to binary classification.

###  2.2 Evidence Retrieval

Given each sample, we split the hypothesis into sentences and retrieve
evidence sentences from the premise. Formally, we split the hypothesis H and
the premise P into single sentences {$H_{1}$, $H_{2}$, $\cdots$, $H_{m}$} and
{$P_{1}$, $P_{2}$, $\cdots$, $P_{n}$}, through NLTK
tool333https://www.nltk.org/. Here, m and n are the sentence numbers.

For each hypothesis sentence $H_{i}$, we respectively utilize the following
retrieval methods to calculate the relevance score with all premise sentences.
Then according to the scores, we remain top K sentences as the corresponding
evidence. The value of K is a trade-off between evidence recall and precision.
A lower value pursues higher evidence precision, but may lead to evidence
missing, while a higher value guarantees higher evidence recall, but may
introduce too many uninformative sentences as noise. Moreover, to keep and
utilize contextual information, for each hypothesis sentence, we reorder the
evidence sentences according to their original order in the premise.

To calculate the relevance score, we take several sparse and dense retrieval
methods into consideration:

$\bullet$ ROUGE-1 Inspired by Mao et al. (2022) and Zhang et al. (2022), we
adopt ROUGE-1 retrieval. For a pair of sentences, this sparse retrieval method
focuses on n-gram match of the pair to calculate ROUGE-1 score as the
relevance metric. We take it as the main retrieval method.

$\bullet$ BM25444We adopt the implementation from
https://github.com/dorianbrown/rank_bm25. BM25 is one of the most advanced
sparse retrieval methods. We take all premise sentences as the corpus. For a
pair of sentences, BM25 involves not only the pair itself but also the whole
corpus, to count term frequency and inverse-document frequency to obtain the
relevance score.

$\bullet$ SimCSE555We adopt unsupervised and supervised version of
RoBERTabase. Inspired by Gao et al. (2021), we utilize SimCSE Gao et al.
(2021), a strong sentence embedding model, as dense retrieval method for
semantic match. For a pair of sentences, we take the cosine similarity of the
sentence embeddings as the relevance score.

Except for above retrieval methods, we also adopt another simple but effective
strategy. If a hypothesis sentence is a substring of the premise, then it is
naturally entailed in the premise through string match and does not need
further study.

###  2.3 Reading & Credibility Estimation

For each hypothesis sentence, we concentrate informative premise sentences and
filter out most noisy ones through evidence retrieval. Then this component
aims to estimate its credibility, which involves the hypothesis sentence
itself and several evidence sentences. This is different from conventional
sentence-level NLI, which studies the relationship between a pair of
sentences. Hence we adopt reading models. Our general $\rm{R^{2}F}$ framework
is compatible to any arbitrary reading models, which may be enhanced by
several advanced learning technologies, e.g., graph neural network Kipf and
Welling (2017); Velickovic et al. (2018), commonsense knowledge injection
Zhang et al. (2019); Wang et al. (2021), and syntactic structure analysis
Kitaev et al. (2022). Herein, we adopt a simple and straightforward one
without loss of generality.

Specifically, for a hypothesis sentence $H_{i}$, and its corresponding
evidence {$Evi_{i1}$, $Evi_{i2}$, $\cdots$, $Evi_{iK}$}, we concatenate them
all as the input:

$[CLS]\;H_{i}\;[SEP]\;Evi_{i1}\;Evi_{i2}\;\cdots\;Evi_{iK}\;[SEP]$ (1)

Then we leverage transformer-based pre-trained language model, i.e., RoBERTa
Liu et al. (2019), to encode the input sequence. Through the multi-head self-
attention mechanism Vaswani et al. (2017), token-level information interaction
among the hypothesis sentence and all its evidences is conducted. Besides,
since evidence are concentrated, it is much easier to handle the multiple
evidence combination issue for cross-sentence inference.

Then the credibility score is calculated through a Multi Layer Perceptron
(MLP) with sigmoid activation function:

$\hat{y}_{i}={\rm Sigmoid}({\rm MLP}(h_{i}))$ (2)

where $h_{i}$ is the hidden state of the special [CLS] token, and is taken as
the inference vector of $H_{i}$. Besides, $\hat{y}_{i}$ $\in$ [0.0, 1.0] is
the credibility score of $H_{i}$, and a higher score means that the sentence
is more likely to be entailed by the premise.

###  2.4 Credibility Fusion

After reading, the inference vector $h_{i}$ and credibility score
$\hat{y}_{i}$ of each hypothesis sentence $H_{i}$ is obtained. Nevertheless,
the reading model cannot be trained directly since the detailed entailment
label of $H_{i}$ is not available. To this end, we fuse the sentence-level
results to judge the entailment relationship between the documents, and
indirectly train the model through document-level entailment label. Besides,
the fusion process is also expected to solve the detailed disinformation issue
for cross-sentence inference, and expand the interpretability of the
framework. Herein, we design three fusion methods for comparison.

$\bullet$ Credibility Score Minimum Pooling The logic basis for this method is
that if the premise entails the hypothesis, then it will entail all the
hypothesis sentences, even the one with the lowest credibility score. By
contrast, if the premise does not entail the hypothesis, then it will conflict
to at least one hypothesis sentence. This one is expected to be assigned with
the lowest credibility score.

Formally, for a pair of documents H and P, the credibility scores of the
hypothesis sentences are {$\hat{y}_{1}$, $\hat{y}_{2}$, $\cdots$,
$\hat{y}_{m}$}. Then the credibility score of the sample is calculated as:

$\hat{y}_{HP}=min(\\{\hat{y}_{1},\hat{y}_{2},\cdots,\hat{y}_{m}\\})$ (3)

For this fusion method, the sample prediction result comes from that of the
least credible hypothesis sentence, i.e., with the lowest credibility score.
The framework is requested to conduct internal contrast among the hypothesis
sentences to decide the least credible one. Thus, it is forced to understand
the entailment relationship between each hypothesis sentence and its
corresponding evidences although without direct entailment label. During
prediction, the credibility score $\hat{y}_{i}$ is utilized to predict the
entailment label of hypothesis sentence $H_{i}$. We take this as the main
fusion method.

$\bullet$ Inference Vector Minimum Pooling For this method, we conduct minimum
pooling on the inference vector $h_{i}$ rather than the credibility score
$\hat{y}_{i}$. For the inference vector of the sample $h_{HP}$, the j-th
dimension is:

$h^{j}_{HP}=min(\\{h^{j}_{1},h^{j}_{2},\cdots,h^{j}_{m}\\})$ (4)

Then the credibility score of the sample is:

$\hat{y}_{HP}={\rm Sigmoid}({\rm MLP}(h_{HP}))$ (5)

Many conventional neural models tend to adopt this fusion method for better
performance, but suffer from low interpretability, since the practical meaning
of each dimension of the inference vector can hardly be probed.

$\bullet$ Gaussian Kernel Pooling To further explore the influence of fusion
methods, we adopt Gaussian Kernel Pooling Xiong et al. (2017); Liu et al.
(2020); Sheng et al. (2022). Specifically, we utilize C Gaussian kernels
$\\{K_{j}\\}_{j=1}^{C}$. For a credibility score $\hat{y}_{i}$, the output of
the j-th kernel is:

$V_{i}^{j}=\exp({-\frac{(\hat{y}_{i}-\mu_{j})^{2}}{2{\sigma}^{2}_{j}}})$ (6)

where $\mu_{j}$ and $\sigma_{j}$ are respectively the mean and width of the
j-th kernel. In this way, the score $\hat{y}_{i}$ is projected to a kernel
vector $V_{i}$ $\in$ $\mathcal{R}^{C}$. The kernel vector of the sample is:

$V_{HP}=\frac{1}{m}\sum_{i=1}^{m}V_{i}\in\mathcal{R}^{C}$ (7)

And the credibility score of the sample is:

$\hat{y}_{HP}={\rm Sigmoid}({\rm MLP}(V_{HP}))$ (8)

This is a mean pooling method, which conducts mean pooling on the
corresponding kernel vectors, rather than the credibility scores.

During training, the loss function of the sample is set as binary cross
entropy loss:

$\mathcal{L}=y_{HP}log(\hat{y}_{HP})+(1-y_{HP})log(1-\hat{y}_{HP})$ (9)

where $y_{HP}$ is the sample entailment label, 1 for entailment samples while
0 for not entailment ones. During prediction, we set a threshold on the sample
credibility score $\hat{y}_{HP}$ to obtain the result.

##  3 Experiment

###  3.1 Dataset

We conduct our experiments on DOCNLI dataset Yin et al. (2021), which is a
newly proposed and the only large scale dataset in the field. The detailed
statistic information is shown in Table 1. The training set comes with
balanced label distribution, while the development and test sets come with
pretty unbalanced ones.

Set | Entailment | Not_Entailment | Total
---|---|---|---
train | 466,653 | 475,661 | 942,314
dev | 28,890 | 205,368 | 234,258
test | 33,128 | 233,958 | 267,086
Table 1: Statistic information of DOCNLI dataset.

###  3.2 Complementary Annotation

For interpretability study, we contribute complementary annotation666The
annotation is also released.. The hypothesis sentences may involve cross-
sentence inference and request multiple evidences. Besides, they may also
correspond to several evidence groups, where each group itself is enough to
independently infer the entailment label. Thus the annotation process needs
heavy workload and comes with great complexity. Herein, we adopt a proposal
and correction annotation strategy. Specifically, for each hypothesis
sentence, we firstly retrieve candidate evidences through the diverse methods
in Section 2.2. Then we manually check, remove repeated or unrelated ones and
add missing ones, and combine several evidence groups. Finally, we decide the
entailment label according to the evidences.

Label: entailment Hypothesis Sentence: Tony Abbott will withdraw Australia’s
ambassador to Indonesia. Evidence Group 1: Prime Minister Tony Abbott said
Australia will withdraw its ambassador to Indonesia in an unprecedented
diplomatic response to the executions of Myuran Sukumaran and Andrew Chan.
Evidence Group 2: Mr Prasetyo shrugged off diplomatic backlash from Australia
after Prime Minister Tony Abbott slammed the executions as “cruel and
unnecessary” and announced he would withdraw Australia’s ambassador to
Indonesia Paul Grigson. Figure 3: Annotation example for hypothesis sentence.
The hypothesis sentence is annotated as entailment with two evidence groups.

Due to the heavy workload and great complexity, we manually annotate 100
longer samples (all over 800 words) randomly selected from the test set, which
contain more than 350 hypothesis sentences in total. An annotation example is
shown in Figure 3. The hypothesis sentence is annotated as entailment with two
evidence groups. More detailed statistic information is summarized in Appendix
A.

Model | Entailment | Not_Entailment | Total
---|---|---|---
P | R | F1 | P | R | F1 | Micro F1 | Macro F1
Concatenation-Longformer${}_{base}\clubsuit$ | - | - | 44.42 | - | - | - | - | -
Concatenation-Longformer${}_{base}\spadesuit$ | 29.73 | 80.32 | 43.39 | 96.33 | 73.11 | 83.13 | 74.01 | 63.26
Concatenation-RoBERTa${}_{base}\spadesuit$ | 39.73 | 91.57 | 55.41 | 98.54 | 80.33 | 88.51 | 81.72 | 71.96
Semantic Match-RoBERTa${}_{base}\triangle$ | 31.60 | 85.49 | 46.15 | 97.29 | 73.80 | 83.93 | 75.25 | 65.04
ours $\rm R^{2}F$-RoBERTabase | 43.52 | 87.04 | 58.02 | 97.86 | 84.01 | 90.41 | 84.38 | 74.22
Concatenation-RoBERTa${}_{large}\clubsuit$ | 45.15 | 94.99 | 61.21 | 99.16 | 83.66 | 90.75 | 85.06 | 75.98
Concatenation-RoBERTa${}_{large}\spadesuit$ | 44.70 | 94.74 | 60.74 | 99.11 | 83.40 | 90.58 | 84.81 | 75.66
ours $\rm R^{2}F$-RoBERTalarge | 50.61 | 88.07 | 64.28 | 98.11 | 87.82 | 92.68 | 87.86 | 78.48
Table 2: Model performance on the test set. $\clubsuit$ and $\spadesuit$
denote the original and reproduced results of the model from Yin et al.
(2021). $\triangle$ denotes the reproduced results of the model modified from
Zhong et al. (2020).

###  3.3 Evaluation Metric

For DOCNLI task, we adopt micro and macro F1 scores, and attach more
importance to the latter. Due to the unbalanced label distribution, even
majority guess model will obtain high micro F1 score, but pretty low macro F1
score. For sentence-level evaluation, we adopt evidence precision, recall and
F1 for retrieval, while micro and macro F1 scores for label prediction.
Moreover, inspired by Thorne et al. (2018), we propose a more strict metric
full accuracy. Herein, evidence recall requests to find at least one complete
evidence group, and full accuracy further requests correct label prediction.

###  3.4 Experiment Setup

Our $\rm{R^{2}F}$ framework is implemented through Pytorch 1.8.0. We adopt
AdamW optimizer, keep a random number seed of 42, set max input length as 256,
and set mini batch size as 8 with gradient accumulation step as 4. For base
encoder, we choose initial learning rate of 1e-5, while for large encoder, we
choose 5e-6. For evidence retrieval, we set K as 5. During prediction, we
adopt a threshold of 0.5. More setup is shown in Appendix B.

###  3.5 Baseline

Since DOCNLI is still a new task, we adopt the concatenation model from Yin et
al. (2021), and modify the semantic match model from Zhong et al. (2020) for
comparison. Please refer to the original papers and Appendix C for detailed
information.

##  4 Results

###  4.1 Main Results

Main results of our framework on the test set are displayed in Table 2. The
framework obtains the best performance with the highest micro and macro F1
scores, with both base and large encoders, indicating its strength. The
situation is similar on the development set in Appendix D.

All models show far better performance (higher F1 score) on not entailment
samples than entailment samples, while entailment samples still come with even
higher recall. This may be due to the label distribution difference among
different sets (in Table 1). Among the baselines, semantic match model
conducts coarse-grained information interaction between the hypothesis and
premise through document-level vector representations. Thus it shows pretty
low performance although it can avoid possible key information missing caused
by the truncation of overlong samples. This indicates that fine-grained
information interaction is essential for this task. Besides, concatenation
model with Longformer encoder, although it is able to handle much longer
inputs, shows much lower performance than that with RoBERTa encoder, which
truncates the inputs. The simplified multi-head self-attention mechanism in
Longformer encoder Beltagy et al. (2020) seems not competent for fine-grained
information interaction in the task.

###  4.2 Performance vs Sample Length

To examine the ability of our framework on processing long inputs, the
performance with varying sample length on the test set is shown in Figure 4.
Here, the length of a sample is counted in the number of words. Both models
obtain far better performance on shorter samples than longer ones (more than
10% absolute difference on both micro F1 and macro F1 scores). Hence it is
still a relatively difficult problem to process longer samples. Moreover, our
R2F framework consistently and greatly outperforms the strongest concatenation
baseline on samples with varying length. Especially, it shows much higher
performance on longer samples. These indicate the framework is able to handle
longer ones efficiently through breaking the document-level task into
sentence-level task with the retrieval, reading and fusion process. On the
development set, the tendency is similar, and detailed results are displayed
in Appendix D.

Figure 4: Model performance with varying sample length on the test set.
Concatenation baseline and R2F framework with RoBERTabase encoder are
compared. The horizontal axis is the sample length in the number of words. The
vertical axis is the performance.

###  4.3 Influence of Evidence Retrieval

To investigate the influence of evidence retrieval, we focus on different
retrieval methods, and take those mentioned in Section 2.2 for comparison.
Besides, we also adopt a random baseline, which adopts K randomly selected
premise sentences as the evidences.

Retrieval Method | Dev Set | Test Set
---|---|---
Mi F1 | Ma F1 | Mi F1 | Ma F1
Random | 83.27 | 71.77 | 82.45 | 70.98
ROUGE-1 | 85.58 | 75.44 | 84.38 | 74.22
BM25 | 85.55 | 75.36 | 83.98 | 73.71
SimCSE∗ | 86.13 | 75.57 | 85.34 | 74.74
SimCSE# | 84.87 | 74.33 | 83.65 | 73.28
Table 3: DOCNLI performance with different evidence retrieval methods. ∗ and #
respectively denote unsupervised and supervised version, and the same below.

As shown in Table 3, the random baseline can still obtain relatively high
performance although it cannot outperform the concatenation baseline. This may
be due to the evidence dependency bias discussed in Section 4.5. Besides, for
shorter samples with only a few premise sentences, evidence retrieval will
show less importance. All other retrieval methods can contribute to higher
performance than the concatenation baseline, indicating the strength of the
framework on the task and its robustness for diverse retrieval methods.
However, supervised SimCSE, although trained on human-annotated NLI
benchmarks, shows the lowest performance, which may suffer transfer issue
caused by the domain difference between its own training data and DOCNLI
dataset.

###  4.4 Influence of Fusion Method

Performance of different fusion methods are compared in Figure 5. Gaussian
kernel pooling tends to incorrectly predicts all samples as not entailment and
can hardly recognize entailment samples. Thus it comes with much higher micro
F1 score but far lower macro F1 score under the pretty unbalanced label
distribution (in Table 1). This mean pooling fusion method seems not feasible
for the task. On both sets, credibility score minimum pooling outperforms all
other fusion methods. This fusion method also comes with clear logic basis to
expand the interpretability of the framework.

Figure 5: DOCNLI performance with different fusion methods.

###  4.5 Interpretability Study

Figure 6: Case study on the sample in Figure 1. For each hypothesis sentence,
the elements in the bracket denote whether at least one complete evidence
group is obtained, T for True while F for False, and entailment label
prediction result, E for entailment while NE for not entailment. Wrong
predictions are in red. The number below is credibility score. The Golden
branch denotes the groundtruth.

To investigate the interpretability of our framework, we conduct sentence-
level evaluation on the subset annotated by ourselves. As shown in Table 4, we
focus on evidence retrieval and entailment label prediction for hypothesis
sentences.

$\bullet$ Evidence Retrieval For these longer samples (all over 800 words),
the random baseline can hardly obtain the evidence, with extremely low
evidence recall. All other retrieval methods can find at least one complete
evidence group for most of the hypothesis sentences, with relatively high
evidence recall around 85%. However, with pretty low evidence precision and F1
score, all these methods will introduce plenty of uninformative sentences as
noise. Thus, the evidence retrieval component may need further improvement.

Retrieval Method | Evi. Retri. | Label Pre. | FA
---|---|---|---
P | R | F1 | Mi F1 | Ma F1
Random | 6.83 | 17.21 | 9.78 | 66.94 | 63.32 | 13.11
ROUGE-1 | 31.53 | 87.43 | 46.35 | 76.23 | 73.92 | 68.85
BM25 | 31.53 | 87.16 | 46.31 | 73.22 | 70.46 | 66.12
SimCSE∗ | 30.38 | 84.97 | 44.76 | 76.50 | 74.69 | 67.49
SimCSE# | 31.31 | 84.97 | 45.76 | 73.50 | 70.48 | 65.03
Table 4: Sentence-level evaluation results. Evi. Retri., Label Pre., and FA
respectively denote Evidence Retrieval, Label Prediction and Full Accuracy.

$\bullet$ Entailment Label Prediction Regardless of poor retrieval
performance, the random baseline still shows relatively high performance on
entailment label prediction. This is due to evidence dependency bias.
Entailment samples strictly request at least one complete evidence group.
However, not entailment samples are insensitive to evidence retrieval. With
complete evidence group obtained, they are taken as conflict to the premise,
while with evidence missing, they will be taken as not mentioned. Both
situations are considered as not entailment. This kind of bias will also
contribute to the high document-level performance of the random baseline in
Table 3. All other retrieval methods obtain much higher performance with the
power of evidence. However, it seems that high evidence recall is not
promising for more accurate entailment label prediction for hypothesis
sentences. This may be also due to evidence dependency bias. Besides, with
evidence precision at only around 30%, evidence noise is also an important
issue, the influence of which is difficult to estimate. Moreover, the high
full accuracy score means for more than 65% hypothesis sentences, our
framework can find at least one complete evidence group and correctly predict
their entailment label. Therefore, taking that sentence-level annotation is
not available during training, our R2F framework is able to give more
interpretable prediction results and help to locate the semantic
inconsistency.

###  4.6 Case Study

To further display the interpretability of our framework, we conduct case
study on the sample in Figure 1. As shown in Figure 6, for this long sample
(about 667 words in total), the random baseline is pretty weak and cannot find
complete evidence group for any hypothesis sentence, although it can obtain
correct prediction for the sample. This also demonstrates the importance of
evidence discovery for interpretability study. By contrast, all other
retrieval methods can obtain complete evidence groups, and contribute to
correctly predict sentence-level entailment label for most of the hypothesis
sentences. Thus the framework can give more interpretable prediction results,
and accurately locate the semantic inconsistency in the third hypothesis
sentence. Furthermore, with the two sparse retrieval methods, our framework
can even successfully handle all the hypothesis sentences. However, the two
dense methods fail to obtain complete evidence group for the first hypothesis
sentence. As shown in Figure 1, this one requests two evidences, whose most
words are related to the first evidence while only a few words are related to
the second one. Moreover, it rephrases the second one with totally different
expressions. Therefore, it is difficult to find the second one. This may be
the exact situation that leads to the lower evidence recall of dense retrieval
methods than sparse ones in Table 4.

##  5 Related Work

Natural language inference is a fundamental yet important task in natural
language processing. For sentence-level NLI, several benchmarks Bowman et al.
(2015); Williams et al. (2018); Wang et al. (2019); Nie et al. (2020) have
been proposed and they have been attracting research attention. Moreover,
Koreeda and Manning (2021) propose Contract NLI targeting the legal and
business domain, which is a small scale benchmark. The premises are long
contract documents while the hypotheses are actually single sentences.
Besides, Tian et al. (2022) suggest to debias natural language inference and
understanding models through causal intervention and counterfactual reasoning.
Lin et al. (2022) adopt commonsense inference to enhance future event
generation. For document-level NLI, Yin et al. (2021) propose the first large
scale benchmark DOCNLI based on a set of early benchmarks. Current models for
the task still largely follow sentence-level NLI. Differently, in this paper,
we emphasize the importance of evidence discovery and aim at a general
solution for the task.

##  6 Conclusion

In this paper, we propose R2F framework as a general solution for DOCNLI task
and contribute complementary annotation on DOCNLI dataset. Our experimental
results show that our framework can obtain state-of-the-art performance.
Besides, the framework is robust for diverse retrieval methods, and
consistently obtains higher performance on samples with varying length,
especially longer samples. Moreover, the framework can give more interpretable
prediction results and help to locate the semantic inconsistency. In the
future, we will explore an end-to-end framework for the task.

## Limitations

The main limitation of our R2F framework is that the framework is a pipeline
one rather than an end-to-end one. For efficiency, the evidence retrieval
component is an independent unit, which provides evidence input for the
jointly trained reading and fusion components. However, the evidence retrieval
component in our pipeline framework will not bring additional heavy
computation. First, it can be conducted offline efficiently. The sparse
retrieval methods only involve item frequency counting, and the dense ones are
based on pretrained sentence embeddings without further fine-tuning. Second,
there are many acceleration techniques for retrieval in industrial field.

Furthermore, we will try to improve the evidence retrieval component in the
future. On one hand, we will try to utilize document-level entailment label to
improve sentence-level evidence retrieval. On the other hand, we will also
explore an efficient end-to-end model, which may benefit from reinforcement
learning Williams (1992); Lei et al. (2016), reparameterization trick Maddison
et al. (2017); Jang et al. (2017), or Expectation-Maximum algorithm Dempster
et al. (1977). However, the evidence dependency bias issue, discussed in
Section 4.5, will pose a great challenge. That is, entailment samples strictly
request complete evidence groups, while not entailment samples are insensitive
to evidence retrieval.

## Acknowledgments

This research is supported by the Singapore Ministry of Education (MOE)
Academic Research Fund (AcRF) Tier 1 grant, under the RIE2020 Industry
Alignment Fund – Industry Collaboration Projects (IAF-ICP) Funding Initiative,
as well as cash and in-kind contribution from the industry partner(s).

## References

  * Arora et al. (2022) Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Chi Kit Cheung. 2022\.  Why exposure bias matters: An imitation learning perspective of error accumulation in language generation.  In _Findings of 2022 the Association for Computational Linguistics_ , pages 700–710. Association for Computational Linguistics.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _CoRR_ , abs/2004.05150.
  * Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015.  Scheduled sampling for sequence prediction with recurrent neural networks.  In _Proceedings of the 2015 Advances in Neural Information Processing System_ , pages 1171–1179.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642. The Association for Computational Linguistics.
  * Chen et al. (2017) Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017\.  Enhanced lstm for natural language inference.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_ , pages 1657–1668. Association for Computational Linguistics.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdinov. 2019.  Transformer-xl: Attentive language models beyond a fixed-length context.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics_ , pages 2978–2988. Association for Computational Linguistics.
  * Dempster et al. (1977) A. P. Dempster, N. M. Laird, and D. B. Rubin. 1977.  Maximum likelihood from incomplete data via the em algorithm.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 39(1):1–22.
  * Ding et al. (2020) Ming Ding, Chang Zhou, Hongxia Yang, and Jie Tang. 2020.  Cogltx: Applying BERT to long texts.  In _Proceedings of the 2020 Annual Conference on Neural Information Processing Systems_.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  Simcse: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910. Association for Computational Linguistics.
  * Huang et al. (2022) Kung Hsiang Huang, Kathleen R. McKeown, Preslav Nakov, Yejin Choi, and Heng Ji. 2022\.  Faking fake news for real fake news detection: Propaganda-loaded training data generation.  _CoRR_ , abs/2203.05386.
  * Jang et al. (2017) Eric Jang, Shixiang Gu, and Ben Poole. 2017.  Categorical reparameterization with gumbel-softmax.  In _5th International Conference on Learning Representations_. OpenReview.net.
  * Jawahar et al. (2022) Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks V. S. Lakshmanan. 2022.  Automatic detection of entity-manipulated text using factual knowledge.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_ , pages 86–93. Association for Computational Linguistics.
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-supervised classification with graph convolutional networks.  In _Proceedings of 5th International Conference on Learning Representations_. OpenReview.net.
  * Kitaev et al. (2022) Nikita Kitaev, Thomas Lu, and Dan Klein. 2022.  Learned incremental representations for parsing.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_ , pages 3086–3095. Association for Computational Linguistics.
  * Koreeda and Manning (2021) Yuta Koreeda and Christopher D. Manning. 2021.  Contractnli: A dataset for document-level natural language inference for contracts.  In _Findings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1907–1919. Association for Computational Linguistics.
  * Lei et al. (2016) Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2016.  Rationalizing neural predictions.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 107–117. The Association for Computational Linguistics.
  * Lin et al. (2022) Li Lin, Yixin Cao, Lifu Huang, Shuang Li, Xuming Hu, Lijie Wen, and Jianmin Wang. 2022.  What makes the story forward?: Inferring commonsense explanations as prompts for future event generation.  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 1098–1109. ACM.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _CoRR_ , abs/1907.11692.
  * Liu et al. (2020) Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020.  Fine-grained fact verification with kernel graph attention network.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7342–7351. Association for Computational Linguistics.
  * Maddison et al. (2017) Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017.  The concrete distribution: A continuous relaxation of discrete random variables.  In _5th International Conference on Learning Representations_. OpenReview.net.
  * Mao et al. (2022) Ziming Mao, Chen Henry Wu, Ansong Ni, Yusen Zhang, Rui Zhang, Tao Yu, Budhaditya Deb, Chenguang Zhu, Ahmed Hassan Awadallah, and Dragomir R. Radev. 2022\.  Dyle: Dynamic latent extraction for abstractive long-input summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_ , pages 1687–1698. Association for Computational Linguistics.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807. Association for Computational Linguistics.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial nli: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4885–4901. Association for Computational Linguistics.
  * Park et al. (2022) Hyunji Hayley Park, Yogarshi Vyas, and Kashif Shah. 2022.  Efficient classification of long documents using transformers.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_ , pages 702–709. Association for Computational Linguistics.
  * Sandhaus (2008) Evan Sandhaus. 2008.  The new york times annotated corpus.  In _LDC corpora_. Linguistic Data Consortium.
  * Sheng et al. (2022) Qiang Sheng, Juan Cao, Xueyao Zhang, Rundong Li, Danding Wang, and Yongchun Zhu. 2022.  Zoom out and observe: News environment perception for fake news detection.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_ , pages 4543–4556. Association for Computational Linguistics.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  Fever: a large-scale dataset for fact extraction and verification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies_ , pages 809–819. Association for Computational Linguistics.
  * Tian et al. (2022) Bing Tian, Yixin Cao, Yong Zhang, and Chunxiao Xing. 2022.  Debiasing nlu models via causal intervention and counterfactual reasoning.  In _Proceedings of the 36th AAAI Conference on Artificial Intelligence_ , pages 11376–11384. AAAI Press.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of the 2017 Advances in Neural Information Processing Systems_ , pages 5998–6008.
  * Velickovic et al. (2018) Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018.  Graph attention networks.  In _Proceedings of 6th International Conference on Learning Representations_. OpenReview.net.
  * Wang et al. (2019) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the7th International Conference on Learning Representations_. OpenReview.net.
  * Wang et al. (2021) Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021.  K-adapter: Infusing knowledge into pre-trained models with adapters.  In _Findings of the 2021 Association for Computational Linguistics_ , pages 1405–1418. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,_ , pages 1112–1122. Association for Computational Linguistics.
  * Williams (1992) Ronald J. Williams. 1992.  Simple statistical gradient-following algorithms for connectionist reinforcement learning.  _Machine Learning_ , 8:229–256.
  * Wolpert and Macready (1997) David H. Wolpert and William G. Macready. 1997.  No free lunch theorems for optimization.  _IEEE Transactions on Evolutionary Computation_ , 1(1):67–82.
  * Wu et al. (2021) Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2021.  Hi-transformer: Hierarchical interactive transformer for efficient and effective long document modeling.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics_ , pages 848–853. Association for Computational Linguistics.
  * Xiong et al. (2017) Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017.  End-to-end neural ad-hoc ranking with kernel pooling.  In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 55–64. ACM.
  * Yin et al. (2021) Wenpeng Yin, Dragomir R. Radev, and Caiming Xiong. 2021.  Docnli: A large-scale dataset for document-level natural language inference.  In _Findings of the 2021 Association for Computational Linguistic_ , pages 4913–4922. Association for Computational Linguistics.
  * Zhang et al. (2022) Yusen Zhang, Ansong Ni, Ziming Mao, Chen Henry Wu, Chenguang Zhu, Budhaditya Deb, Ahmed Hassan Awadallah, Dragomir R. Radev, and Rui Zhang. 2022.  $summ^{n}$: A multi-stage summarization framework for long input dialogues and documents.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics_ , pages 1592–1604. Association for Computational Linguistics.
  * Zhang et al. (2019) Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019.  Ernie: Enhanced language representation with informative entities.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics_ , pages 1441–1451. Association for Computational Linguistics.
  * Zhong et al. (2020) Ming Zhong, Pengfei Liu, Yiran Chen, Danqing Wang, Xipeng Qiu, and Xuanjing Huang. 2020.  Extractive summarization as text matching.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6197–6208. Association for Computational Linguistics.

##  Appendix A Statistic Information of Complementary Annotation

Model | Entailment | Not_Entailment | Total
---|---|---|---
P | R | F1 | P | R | F1 | Micro F1 | Macro F1
Concatenation-Longformer${}_{base}\clubsuit$ | - | - | 46.18 | - | - | - | - | -
Concatenation-Longformer${}_{base}\spadesuit$ | 31.09 | 81.33 | 44.99 | 96.60 | 74.64 | 84.22 | 75.47 | 64.60
Concatenation-RoBERTa${}_{base}\spadesuit$ | 42.27 | 90.82 | 57.69 | 98.45 | 82.55 | 89.81 | 83.57 | 73.75
Semantic Match-RoBERTa${}_{base}\triangle$ | 31.01 | 84.32 | 45.34 | 97.09 | 73.61 | 83.74 | 74.93 | 64.54
ours $\rm R^{2}F$-RoBERTabase | 45.54 | 86.43 | 59.66 | 97.82 | 85.46 | 91.22 | 85.58 | 75.44
Concatenation-RoBERTa${}_{large}\clubsuit$ | 47.15 | 95.09 | 63.04 | 99.19 | 85.01 | 91.55 | 86.25 | 77.30
Concatenation-RoBERTa${}_{large}\spadesuit$ | 47.04 | 94.82 | 62.89 | 99.15 | 84.99 | 91.52 | 86.20 | 77.21
ours $\rm R^{2}F$-RoBERTalarge | 53.70 | 87.10 | 66.43 | 98.01 | 89.43 | 93.53 | 89.15 | 79.98
Table 5: Model performance on the dev set. $\clubsuit$ and $\spadesuit$ denote
the original and reproduced results of the model from Yin et al. (2021).
$\triangle$ denotes the reproduced results of the model modified from Zhong et
al. (2020).

Due to heavy workload and great complexity, we manually annotate 100 longer
samples (all over 800 words) randomly selected from the test set, which
contain more than 350 hypothesis sentences in total. Among the samples, 84%
are annotated as not entailment, while 16% are annotated as entailment.
However, among the hypothesis sentences, about 43% are annotated as not
entailment, while about 57% are annotated as entailment. The great label
distribution difference is natural. For entailment samples, all hypothesis
sentences are entailed by the premise. However, not entailment samples can
still contain entailed hypothesis sentences. Besides, about 47% hypothesis
sentences involve cross-sentence inference and request multiple evidence
sentences. Moreover, about 17% hypothesis sentences are annotated with more
than one evidence groups. These also indicate the great complexity of fine-
grained sentence-level annotation.

##  Appendix B Detailed Experiment Setup

Our $\rm{R^{2}F}$ framework is implemented through Pytorch 1.8.0 and hugging
face transformers777https://huggingface.co/. All experiments are conducted on
a computation node with Nvidia 40G A100 GPUs. For evidence retrieval, we set K
as 5 to remain top 5 sentences as evidences during retrieval. We adopt RoBERTa
Liu et al. (2019) as encoder, including base and large version. For all
experiments, we adopt AdamW optimizer, keep a random number seed of 42, set
max input length as 256, and set mini batch size as 8 with gradient
accumulation step as 4. For base encoder, we choose initial learning rate as
1e-5, while for large encoder, we choose initial learning rate as 5e-6. We
train 5 epochs, evaluate each 3750 steps, and choose the model parameters with
the highest performance on the development set. For Gaussian Kernel Pooling,
we keep 11 kernels with the same width of 0.01. However, the mean values of
the kernels come from a uniform distribution within the interval of [0.0,
1.0]. During prediction, we adopt a threshold of 0.5.

##  Appendix C Detailed Baseline

Since DOCNLI is still a new task, we adopt the concatenation model from Yin et
al. (2021), and modify the semantic match model from Zhong et al. (2020) for
comparison.

$\bullet$ Concatenation We concatenate the hypothesis and the premise
documents into a sequence as input. Overlong samples will be truncated to max
input length. Then we adopt the hidden state of the special [CLS] token as the
sample representation for binary classification.

$\bullet$ Semantic Match We respectively encode the hypothesis and premise
documents to obtain their own document-level vector representation. For
documents exceeding the max input length, we split them into chunks with
sliding window. Then inspired by Chen et al. (2017), we enhance the vector
representations of these two for classification.

##  Appendix D Performance on Development Set

The detailed model performance on the development set are shown in Table 5 and
Figure 7. The situations are similar to those on the text set. Our framework
obtains the highest performance on the development set. Furthermore, except
for the slight performance decrease on samples no longer than 150 words, it
greatly and consistently outperforms the strongest concatenation baseline on
samples with varying length, especially on longer samples. These also show the
strength of our R2F framework on the task.

Figure 7: Model performance with varying sample length on the dev set.
Concatenation baseline and R2F framework with RoBERTabase encoder are
compared. The horizontal axis is the sample length in the number of words. The
vertical axis is the performance.

##  Appendix E Influence of K Value

To investigate the influence of evidence retrieval, we also pay attention to
the value of K, which is the hyperparameter about how many sentences to remain
during retrieval. As shown in Table 6, the framework is pretty sensitive to
the value. Specifically, the highest performance is obtained with K as 5.
Besides, with K as 3, 6 or 7, the framework can also obtain promoted
performance than the concatenation baseline. However, with K as 4, it shows
similar performance with the baseline. As discussed in Section 2.2, K is to
balance evidence precision and recall. Lower values pursue higher evidence
precision, but may lead to evidence missing, while higher values guarantee
higher evidence recall, but may introduce too much noise. Besides, the choice
of the value is also closely related to data distribution. The high
sensitivity to the value indicates that the evidence retrieval process will
need further improvement, where a possible way is to utilize the document-
level entailment label to improve it.

K | Dev | Test
---|---|---
Mi F1 | Ma F1 | Mi F1 | Ma F1
3 | 84.79 | 74.50 | 83.05 | 72.65
4 | 83.68 | 73.14 | 81.68 | 71.30
5 | 85.58 | 75.44 | 84.38 | 74.22
6 | 84.64 | 74.50 | 83.50 | 73.39
7 | 84.34 | 73.95 | 82.50 | 72.31
Table 6: DOCNLI performance with varying values of K.

##  Appendix F Related Work on Long Document Processing

For long document processing, two common methods are to truncate the input
sequence to the maximum length, or cut into several independent segments with
sliding window. Dai et al. (2019) propose segment-level recurrence mechanism
for information interaction among segments. Beltagy et al. (2020) simplify the
self-attention mechanism Vaswani et al. (2017) to reduce memory overhead to
handle longer input sequence. Moreover, Ding et al. (2020) suggest to
introduce two independent iteratively trained models, respectively for neural
evidence retrieval and semantic inference. Wu et al. (2021) propose
hierarchical interactive transformer structure with stacked transformers
respectively for sentence and document encoding. However, a recent study Park
et al. (2022) on text classification task indicates that none of these models
will consistently outperform other models across datasets. The situation is
similar to the famous No Free Lunch principle Wolpert and Macready (1997).
Therefore, long document processing may need further exploration.


# Fast and Scalded: Capillary Leidenfrost Droplets in micro-Ratchets

Álvaro<EMAIL_ADDRESS>Daniel del Cerro2, Gert-Willem Römer2,
Detlef Lohse3
_1 Institute of Fluid Mechanics, Universität der Bundeswehr München, Germany_
_2 Applied Laser Technology, University of Twente, The Netherlands_
_3 Physics of Fluids Group, University of Twente, The Netherlands_

In the Fluid Dynamics Video included in the ancillary files (a different
version is also available on http://youtu.be/CS0c05WQ_js), we illustrate the
special dynamics of Capillary self-propelled Leidenfrost droplets [1][2] and
confirm the so-called “viscous mechanism” model [3] by testing it in
micrometric ratchets with capillary droplets. In order to be able to propel
water droplets of sizes of the order of 1 mm, micro-ratchets were produced by
direct material removal using a picosecond pulsed laser source. Surface micro-
patterning with picosecond laser pulses allows creating a well controlled
topography on a variety of substrates, with a resolution typically in the
micron range[4]. The experiments yielded the surprising result that capillary
drops can be much faster, and be propelled as much as bigger droplets. Based
on the viscous mechanism model by D. Quéré and C. Clanet [3] and adapting
their scaling laws to capillary drops we obtain good agreement with the
experimental results. More information can be found in reference [5] and [6].

## References

  * [1] H. Linke, B. Alemán, L. Melling, M. Taormina, M. Francis, C. Dow-Hygelund, V. Narayanan, R. Taylor, and A. Stout.  Self-Propelled Leidenfrost Droplets.  Physical Review Letters, 96(15), April 2006.
  * [2] G. Lagubeau, M. Le Merrer, C. Clanet, and D. Quéré.  Leidenfrost on a ratchet.  Nature Physics, 7(5):395–398, 2011.
  * [3] G. Dupeux, M. Le Merrer, G. Lagubeau, C. Clanet, S. Hardt, and D. Quéré.  Viscous mechanism for Leidenfrost propulsion on a ratchet.  Europhysics Letters, 96:1–7, November 2011.
  * [4] D. Arnaldo del Cerro, G. Römer, and A. J. Huis In’t Veld.  Erosion resistant anti-ice surfaces generated by ultra short laser pulses.  Physics Procedia, 5:231–235, 2010.
  * [5] A. G. Marin, Arnaldo del Cerro, D., G. W. Römer, B. Pathiraj, A. Huis in ’t Veld, and D. Lohse.  Capillary droplets on leidenfrost micro-ratchets.  Physics of fluids, 24(12):1–10, 2012.
  * [6] A. G. Marin, Arnaldo del Cerro, D., G. W. Römer, B. Pathiraj, A. Huis in ’t Veld, and D. Lohse.  Capillary droplets on leidenfrost micro-ratchets.  arXiv preprint arXiv:1210.4978, 2012.


[1,2]Kargaltseva N. S.

# Influence of the magnetic field on the formation of protostellar disks

Khaibrakhmanov S. A  Dudorov A. E. Zamozdra S. N  Zhilkin A. G

###### Abstract

We numerically model the collapse of magnetic rotating protostellar clouds
with mass of 10 $M_{\odot}$. The simulations are carried out with the help of
2D MHD code Enlil. The structure of the cloud at the isothermal stage of the
collapse is investigated for the cases of weak, moderate, and strong initial
magnetic field. Simulations reveal the universal hierarchical structure of
collapsing protostellar clouds, consisting of flattened envelope with the
qausi-magnetostatc disk inside and the first core in its center. The size of
the primary disk increases with the initial magnetic energy of the cloud. The
magnetic braking efficiently transports the angular momentum from the primary
disk into the envelope in the case, when initial magnetic energy of the cloud
is more than 20 % of its gravitational energy. Intensity of the outflows
launched from the region near the boundary of the first core increases with
initial magnetic energy. The `dead' zone with small ionization fraction,
$x<10^{-11}$, forms inside the first hydrostatic core and at the base of the
outflow. Ohmic dissipation and ambipolar diffusion determine conditions for
further formation of the protostellar disk in this region.

00footnotetext: Corresponding Author: Kargaltseva N. S.: Chelyabinsk state
university, 129 Br. Kashirinykh str., Chelyabinsk 454001, Russia; Ural Federal
University, 51 Lenina str., Ekaterinburg 620000, Russia; Email:
<EMAIL_ADDRESS>Khaibrakhmanov S. A.: Chelyabinsk state
university, 129 Br. Kashirinykh str., Chelyabinsk 454001, Russia; Ural Federal
University, 51 Lenina str., Ekaterinburg 620000, Russia00footnotetext: Dudorov
A. E.: Chelyabinsk state university, 129 Br. Kashirinykh str., Chelyabinsk
454001, Russia; Ural Federal University, 51 Lenina str., Ekaterinburg 620000,
Russia00footnotetext: Zamozdra S. N.: Chelyabinsk state university, 129 Br.
Kashirinykh str., Chelyabinsk 454001, Russia00footnotetext: Zhilkin A. G.:
Institute of Astronomy of the Russian Academy of Sciences (INASAN), Moscow,
119017, Russia

##  1 Introduction

Observations show that stars form in cold dense cores of filamentary
interstellar molecular clouds. This protostellar clouds (PSCs hereafter) have
typical density $n=10^{3}-10^{6}$ cm-3, temperature $T=10-20$ K, size
$R=0.03-0.3$ pc, mass $M=0.1-30\,M_{\rm\odot}$, velocity dispersion
$\sigma\leq 3$ km s-1, and angular velocity $\omega=10^{-12}-10^{-14}$ s-1
(see, e.g., Carey et al. (1998); Bergin and Tafalla (2007); Morii et al.
(2021) and also review by Dudorov and Khaibrahmanov (2017)).

The distribution of the specific angular momentum inside PSCs was studied only
on large scales of $0.1$ pc. The energy of rotation of PSCs is of the order of
several percent of the gravitational energy (Goodman et al., 1993; Caselli et
al., 2002).

Measurements of the Zeeman effect in the OH lines indicates that PSCs have
magnetic field with strength of $B=10^{-5}-10^{-4}$ G. According to
polarization mapping of PSCs, the magnetic field has hourglass morphology
(Girart et. al.,, 2006; Crutcher, 2012; Li, 2021).

It is believed that gravitationally unstable PSCs undergo gravitational
collapse and evolve into class 0 young stellar objects (YSOs hereafter)
observed as the IR sources and interpreted as protostars surrounded by dense
flattened envelope of gas and dust (Andre et al., 1993). Bipolar outflows are
the characteristic feature of class 0 YSOs (Myers et al., 1988; Andre, 1995;
Galametz et al., 2020).

Studies in the submillimeter range show that the envelopes of class 0 YSOs
have size of $500-12000$ au. The envelopes are flattened along the axis of
rotation and/or the magnetic field direction (Wiseman et al., 2001; Maureira
et al., 2020). High angular resolution studies of the central regions around
protostars in recent years revealed small disks with signs of Keplerian
rotation and sizes of $2-300$ au (Ohashi et al., 2014; Dunham et al., 2014;
Persson et al., 2016; Pineda et al., 2019; Tobin et al., 2020). These disks
can be associated with protostellar accretion disks.

The geometry of the magnetic field is quasi-radial and partially quasi-
toroidal inside the envelopes of class 0 YSOs (Lee, et al., 2019; Hull and
Zhang, 2019). The angular momentum distribution changes across disk to
envelope transition at radii from 1000 to 10000 au (Goodman et al., 1993;
Ohashi et al., 1997; Caselli et al., 2002; Belloche, 2013).

The transition state between the PSC and class 0 YSO is still not possible to
observe. In order to determine the conditions for the formation of
protostellar disks, it is important to perform numerical modelling of the
initial stages of the collapse of PSCs and investigate the formation and
evolution of the internal structure of the PSC.

In the simulations of the collapse of magnetic rotating PSCs in the ideal MHD
approximation, disks are not formed or very small geometrically thick disks
are formed, which does not agree with the observations of protostellar disks
(Mellon and Li, 2008). This is due to the fact that the frozen-in magnetic
field transports the specific angular momentum too quickly from the central
part of the cloud. This so-called catastrophic magnetic braking can be
weakened due to the action of magnetic ambipolar diffusion, Ohmic dissipation,
and/or turbulence allowing for the formation of the protostellar disks (Black
and Scott, 1982; Mouschovias, 1991; Hennebelle and Ciardi, 2009; Tsukamoto et
al., 2017; Zhao et al., 2020). However, the exact conditions for the formation
of protostellar disks are not determined because of large difference between
the initial conditions and numerical models of the PSC. Modern simulations are
mainly concentrated on the accretion stages of the collapse of solar mass PSC
(see, e.g. Hennebelle and Fromang, 2008; Zhao et al., 2020). Investigation of
the initial stages of collapse will make it possible to more accurately
determine the conditions for the formation of protostellar disks and come
closer to solution of the problem of magnetic braking catastrophe.

Previously, we numerically simulated the isothermal collapse of rotating
magnetic PSCs and found that the hierarchical structure of the cloud forms at
this stage (Khaibrakhmanov et al., 2021; Kargaltseva et al., 2021). In this
paper, we further develop our approach and analyse the efficiency of the
magnetic braking at the isothermal stage of the collapse by simulating the
collapse for various initial magnetic energies of the cloud. In Section 2, we
describe the problem setup and the code Enlil used for simulations. General
picture of the hierarchical structure of the collapsing PSC is outlined in
Section 3.1. Dynamics of the collapse in clouds with different initial
magnetic energies is investigated in Section 3.2. Section 3.3 analyzes the
role of the Ohmic dissipation and magnetic ambipolar diffusion in the
evolution of the magnetic flux of the cloud. Section 4 contains main
conclusions and discussion of the results.

##  2 Problem statement and numerical code

We consider a homogeneous spherically symmetric rotating magnetic PSC with
mass $M_{0}=10\,M_{\odot}$ and radius $R_{0}=0.1$ pc. The rotation axis
coincides with the initial magnetic field direction. We adopt axial symmetry
approximation and use the cylindrical coordinates, $(r,\,0,z)$, where $r$ is
the radial distance from the cloud's center, $z$ is the height above its
equatorial plane.

Initial state of the cloud is determined by the ratios of thermal, rotational,
and magnetic energies of the cloud to the modulus of its gravitational energy:
$\varepsilon_{\rm t}$, $\varepsilon_{\rm w}$, and $\varepsilon_{\rm m}$,
respectively.

The simulations are carried out using the two-dimensional MHD code Enlil
(Dudorov et al., 1999a, b). To solve the equations of ideal MHD, the code
utilizes the quasi-monotone TVD scheme of the third order of approximation in
the spatial variable and the first order of approximation in time. Poison's
equation is solved with an implicit alternating direction method. The
generalized Lagrange multiplier method is used to clean up the divergence of
the magnetic field. An adaptive moving mesh is used with $150\times 150$
cells. Ohmic dissipation and ambipolar diffusion are taken into account in the
model (see Zhilkin et al., 2009), and the ionization fraction is calculated
following Dudorov and Sazonov (1987). Thermal evolution of the cloud is
modelled using a variable effective adiabatic index, $\gamma_{\rm eff}$, in
the same manner as in Kargaltseva et al. (2021). In the process of isothermal
collapse, $\gamma_{\rm eff}=1.001$ is used in the equation of state of the
ideal gas. When the density reaches $\rho_{\rm c}=10^{-13}$ g cm-3, the
effective adiabatic index switches to the value of $5/3$. This corresponds to
the moment of the formation of the first hydrostatic core (Larson, 1969).
Similar approach was used, for example, by Masunaga and Inutsuka (2000).

##  3 Results

In order to investigate the influence of the magnetic field on the collapse,
we set the thermal and rotational parameters to $\varepsilon_{\rm t}=0.3$ and
$\varepsilon_{\rm w}=0.01$, and varied the magnetic parameter in simulations.
Three simulation runs were performed with $\varepsilon_{\rm m}=0.01$ (run I),
$0.2$ (run II) and $0.6$ (run III). The first case corresponds to highly non-
equilibrium cloud, $\varepsilon_{\rm t}+\varepsilon_{\rm w}+\varepsilon_{\rm
m}\ll 1$ with weak magnetic field. The cloud with moderate magnetic field
strength is considered in the second run. The third run studies the nearly
equilibrium cloud, $\varepsilon_{\rm t}+\varepsilon_{\rm w}+\varepsilon_{\rm
m}\lesssim 1$, with strong magnetic field. Adopted values of the magnetic
parameter correspond to the following values of non-dimensional mass-to-flux
ratio: $\lambda=10.5$ (Run I), $2.4$ (Run II) and $1.4$ (Run III).

In the following, we use the characteristic time of the collapse of the
magnetic rotating PSC $t_{\rm fmw}=t_{\rm ff}\left(1-\varepsilon_{\rm
m}-\varepsilon_{\rm w}\right)^{-1/2}$ as a unit of time, where $t_{\rm ff}$ is
the free fall time (see Khaibrakhmanov et al., 2021).

###  3.1 Hierarchical structure of the collapsing protostellar cloud

First of all, let us discuss general picture of the isothermal collapse of the
magnetic rotating PSC using run II as a reference. This run with moderate
magnetic field has been analysed in detail in our previous paper (Kargaltseva
et al., 2021).

Fig. 1: Distribution of density (color filling), velocity field (arrows) and
poloidal magnetic field (white lines) in the simulation of the collapse of the
magnetic rotating PSC of mass $10\,M_{\odot}$ with initial radius $R_{0}=0.1$
pc, and non-dimensional parameters $\varepsilon_{t}=0.3$,
$\varepsilon_{m}=0.2$, and $\varepsilon_{w}=0.01$ (run II) at $t\approx
1.0\,t_{\rm fmw}$. Panel (a): Region with $r\leq 0.8\,R_{0}$ and $z\leq
0.4\,R_{0}$ containing the entire cloud. Blue line corresponds to the border
of the cloud. Panel (b): Region near the primary disk. Panel (c): Central part
of the cloud. Green line shows the boundary of the quasi-magnetostatic primary
disk and the boundary of the first hydrostatic core.

In Figure 1, we present the structure of the collapsing PSC at the end of the
dynamical collapse, $t\sim 1\,t_{\rm fwm}$, when typical picture of the
collapse is established. Figure 1 shows that hierarchical structure of the
cloud is formed. The hierarchy consists of an optically thin geometrically
thick flattened envelope, a geometrically and optically thin quasi-
magnetostatic, $v_{\rm z}\ll v_{\rm r}$, primary disk inside the envelope, and
optically thick first core in the center of the primary disk. The boundary of
the quasi-magnetostatic primary disk is characterized by a sharp jump in the
velocity profile $v_{z}(z)$, when almost free fall of gas along the
$z$-direction turns to magnetostatic equilibrium, $v_{z}\approx 0$.
Specifically, we determine the primary disk as a region near the equatorial
plane characterized by $v_{z}<\left(c_{\rm s}^{2}+v_{\rm A}^{2}\right)^{1/2}$,
where $c_{\rm s}$ is the sound speed, and $v_{\rm A}$ is the Alfven speed. The
first core is a hydrostatic region, at the boundary of which both $v_{r}$ and
$v_{z}$ abruptly decrease below the value of local sound speed $c_{\rm s}$.

After the formation of the primary disk, a fast MHD shock wave moves from its
boundary nearly upwards (see Figure 1(b)). Magnetic field lines are bent and
strong toroidal magnetic field is generated behind the shock wave front.

Soon after the formation of the first core, quasi-magnetostatic equilibrium is
violated near its boundary, and an outflow arises in this region, propagating
along the lines of the magnetic field (see Figure 1(c)).

According to our simulations, the magnetic field geometry changes through the
internal hierarchy of the cloud. The magnetic field has a quasi-radial
geometry, $B_{r}\sim B_{z}$, in the envelope. It remains quasi-uniform,
$(B_{r},\,B_{\varphi})\ll B_{z}$, inside the primary disk. The magnetic field
acquires a quasi-toroidal geometry, $B_{\varphi}\sim B_{z}$, in the region
between the boundary of the primary disk and the fast MHD shock wave front. As
Kargaltseva et al. (2021) have shown, the specific angular momentum is
accumulated near the boundary of the primary disk and then transferred to the
envelope by the magnetic braking. The region of efficient magnetic braking
lies between the boundary of the primary disk and fast MHD shock wave front.

The picture of the collapse discussed above demonstrates the leading role of
the primary disks in the evolution of the collapsing PSC at the initial stages
of collapse. The primary disk is the main reservoir of mass, angular momentum
and magnetic flux feeding the first core and thus determining the
characteristics of the protostellar disk forming during further evolution of
the system.

###  3.2 Influence of the magnetic field on the dynamics of the collapse

Let us analyze the effect of the magnetic field on the dynamics of the
collapse of rotating magnetic PSCs, paying special attention to the primary
disks.

In Figure 2, we demonstrate structure of the PSC in runs I (left panels), II
(middle panels), and III (right panels) at four different time moments after
the formation of the primary disk. We plot two-dimensional distribution of the
specific angular momentum in order to investigate the efficiency of magnetic
braking in each run.

Fig. 2: Distribution of the specific angular momentum (color filling),
velocity field (arrows) and poloidal magnetic field (white lines) in the
simulations with $\varepsilon_{m}=0.01$ (run I, left panels),
$\varepsilon_{m}=0.2$ (run II, middle panels), $\varepsilon_{m}=0.6$ (run III,
right panels) at different time moments after the primary disk formation
(panels from top to bottom). The region near the primary disk is considered.
Green line shows the border of the primary disk, blue line is the border of
the cloud.

Figure 2 shows that the hierarchical structure of the collapsing PSC discussed
in Section 3.1 forms in all considered runs. During the collapse, primary
disks grow in size. For example, the radius of the primary disk in run I
increases from $R_{\rm pd}\approx 0.01\,R_{0}\approx 200$ au to $\approx
0.025\,R_{0}\approx 500$ au (see Figures 2(a, b, c, d)). In all cases the
angular momentum firstly accumulates near the surface of the primary disk and
then it is transferred to the envelope due to the magnetic braking in the
region behind the fast MHD shock wave front. Outflow launching from the region
near the first core is also a general feature of considered cases. Typical
speed of the outflow is of $2$ km s-1.

The characteristics of primary disks, dynamics of the outflow and efficiency
of the magnetic braking changes with $\varepsilon_{\rm m}$. The radius and
typical half-thickness of the primary disk increase with $\varepsilon_{\rm
m}$. This reflects the increasing role of the electromagnetic force in
establishing the magnetostatic equilibrium inside the central part of the
cloud. For example, final radius of the primary disk increases from $\approx
0.025\,R_{0}$ in run I up to $\approx 0.6\,R_{0}$ in run III (compare bottom
panels in Figure 2). Therefore, the cloud with strong magnetic field evolves
into a state of magnetostatic equilibrium practically as a whole.

The size of the region of efficient magnetic braking bounded by the front of
the fast MHD shock wave increases with $\varepsilon_{\rm m}$. Figure 2(d)
shows that the shock wave travels small distance of $2\times 10^{-3}\,R_{0}$
along the $z$-direction within the typical dynamical time $t\sim 1\,t_{\rm
fmw}$ in run I, while the shock wave in run III travels a distance of
$0.3-0.4\,R_{0}$ and comes out from the envelope by the time $t\sim
0.83\,t_{\rm fmw}$ (see Figure 2(l)). This is explained by the fact that the
speed of fast MHD shock wave increases with $\varepsilon_{\rm m}$, i. e. with
initial magnetic field strength of the cloud.

The size of the outflow region by the end of the dynamical collapse also
increases with increasing role of the magnetic field, since the outflow is
driven by the electromagnetic force.

Fig. 3: Profiles of radial velocity ($-v_{\rm r}$) and azimuthal velocity
$v_{\varphi}$ along the equatorial plane for runs with weak (Run I, the first
column), moderate (Run II, the second column), and strong (Run III, the third
column) magnetic fields at different moments of time. Line 1 (red color):
start of the collapse. Line 2 (pink color): the moment of primary disc
formation corresponding to the figures 2 (a, e, i). Line 3 (blue color): the
moment of the first core formation corresponding to the figures 2 (c, f, k).
Line 4 (black color): the end of the simulation corresponding to the figures 2
(d, h, l). Dashed lines with numbers show characteristic slopes. Vertical
lines show the boundaries of the first core and primary disk at the end of the
simulation.

In order to analyze the dynamics of the collapse in more detail, we plot the
profiles of the radial and azimuthal velocities along the equatorial plane for
runs I, II, and III in Figure 3.

At the initial moment of time (lines 1), the cloud rotates rigidly, so that
the azimuthal velocity increases with distance as $v_{\varphi}\propto r$. The
cloud is at rest in the radial direction initially, $v_{r}=0$. Consider run I
with weak magnetic field (Figures 3a and d). By the time of the formation of
the primary disk, differential rotation is established in the region from
disk’s boundary, $r\approx 0.02R_{0}$, to the periphery of the cloud (line 2).
The region of differential rotation increases further in time (line 3). The
radial velocity is practically zero inside the first core, $v_{\rm r}\approx
0$, and the entire core rotates rigidly, $v_{\varphi}\propto r$, by the end of
the simulation (line 4). The envelope and the primary disk rotate
differentially. The azimuthal velocity profile is close to the Keplerian law
$v_{\varphi}\propto r^{-1/2}$ in both primary disk and envelope. There is
prominent boundary between the primary disk and the envelope seen in
$v_{\varphi}(r)$ profile at $r\approx 0.02\,R_{0}$. Keplerian rotation profile
implies that the first core becomes dominant source of the gravity in the
system.

The azimuthal velocity becomes comparable to the radial one at the boundary of
the first core, $v_{\rm r}\approx v_{\varphi}\approx 1.9$ km s-1.

In runs with moderate magnetic field (Figures 3b and e) and strong magnetic
field (Figures 3d and f), the velocity profiles are different from the
velocity profiles in the case of weak magnetic field.

In run II , the first core almost does not rotate by the end of the
simulation. The region between the first core and the primary disk,
$10^{-3}\lesssim r\lesssim 7\times 10^{-3}\,R_{0}$, has transient reversed
rotation, which is caused by the excitation of the torsional Alfvén wave in
this region. As in the case of weak magnetic field, the primary disk rotates
with Keplerian speed, although the speed is smaller than in Run I. On the
contrary, the radial speed is larger with maximum value of $3.5$ km s-1 near
the surface of the first core, $r\approx 0.02\,R_{0}$.

In run III, the first core rotates differentially, but not in solid-state by
the end of the simulation. Inner part of the primary disk rotates with
Keplerian speed, while the outer part has almost constant azimuthal velocity.
The rotation speed is minimum as compared to Runs I and II, which implies very
efficient magnetic braking of the disk.

In both runs II and III $v_{\rm r}>v_{\varphi}$, that is, the centrifugal
barrier has not yet formed. Thus, as the magnetic parameter $\varepsilon_{\rm
m}$ increases, magnetic braking becomes more efficient.

###  3.3 Role of dissipative MHD effects

In this section, we analyze the influence of ambipolar diffusion and Ohmic
dissipation on the magnetic field strength in the collapsing PSC. We consider
run II as a reference one. In order to investigate the role of the dissipative
MHD effects in the evolution of the cloud, we performed runs within the ideal
MHD limit, as well as taking into account Ohmic dissipation, magnetic
ambipolar diffusion and both types of diffusion.

The efficiency of magnetic diffusion depends on the level of ionization. In
Figure 4, we plot the vertical profiles of the ionization fraction, $x$, along
the $z$-direction at $r=100$ au for run II at the same times, as in panels (e,
f, g, h) of Figure 2. Considered radial distance $r=100$ au corresponds to the
region inside the first core close to its boundary. Figure 4 shows that the
ionization fraction increases with $z$ up to a maximum value of $10^{-5}$ at
the cloud periphery. This is due to the decrease in gas density and
corresponding increase in the intensity of the interstellar cosmic and X-rays
ionization rate. Minimum value of $x$ corresponds to the maximum density in
the center of the cloud. Before the formation of the first core at
$t=0.9081\,t_{\rm fmw}$ (red line), $x\sim 10^{-8}$ in the central part of the
cloud. After the formation of the first core, the degree of ionization drops
rapidly down to nearly constant value $x\lesssim 10^{-11}$ inside the core.
Therefore, the first core becomes a `dead' zone with small ionization
fraction, in which ambipolar diffusion and Ohmic dissipation weaken the
magnetic field. The jumps in the profiles of the degree of ionization in the
region $z=10^{-3}-2\times 10^{-2}\,R_{0}$, correspond to the region of the
outflow. Our simulations show that the center of the first core is
characterized by smaller ionization fraction of $10^{-13}$.

Fig. 4: Profiles of the ionization fraction, $x$, along the $z$-direction at
$r=100$ au in run II at several times corresponding to the Figures 2(e, f, g,
h).

In Figure 5, we plot corresponding vertical profiles of the radial component
of the magnetic field $B_{r}$ at $r=100$ au inside the first core for runs
with different MHD effects included. Figure 5 shows that, in all four cases,
$B_{r}$ increases from zero value at the equatorial plane, $z=0$, up to a
maximum value of $6\cdot 10^{-2}$ G at $z\sim 100$ au and then decreases
further with $z$. Maximum in the $B_{r}$ profiles corresponds to the surface
of the first hydrostatic core, $z\approx 100$ au.

Inside the first core, the strength of $B_{r}$ is larger in the case of ideal
MHD comparing to the runs with non-ideal MHD effects. This demonstrates that
ambipolar diffusion and Ohmic dissipation weaken the magnetic field in this
`dead' zone. This difference will increase with time as the ionization
fraction inside the `dead' zone will drop down during further evolution of the
system. Above the first core, the profiles $B_{r}(z)$ are the same in all runs
meaning that the magnetic field is frozen into gas in the envelope.

Fig. 5: Vertical profiles of the radial component of the magnetic field
$B_{r}$ at 100 au in run with $\varepsilon_{t}=0.3$, $\varepsilon_{m}=0.2$ and
$\varepsilon_{w}=0.01$. Different MHD effects are considered: ideal MHD (red
line), Ohmic dissipation (pink line), magnetic ambipolar diffusion (blue line)
and both types of diffusion (black line).

##  4 Conclusions and discussion

We performed numerical simulations of the collapse of rotating magnetic PSCs
with mass of $10\,M_{\odot}$ up to the formation of the first hydrostatic
core. The simulations were carried out for the cases with weak, moderate and
strong magnetic field to analyse influence of the magnetic field on the
dynamics of the collapse. The effect of Ohmic dissipation and magnetic
ambipolar diffusion on the collapse was analyzed.

The simulations have shown that the formation of the hierarchical structure of
the PSC found in our previous works (Khaibrakhmanov et al., 2021; Kargaltseva
et al., 2021) is the universal property of the collapsing rotating magnetic
PSCs. The hierarchy consists of flattened cloud's envelope with the
magnetostatic primary disk inside. The first core forms in the central part of
the primary disk. The primary disk acts as a reservoir of mass, angular
momentum and magnetic flux for the protostar and further forming protostellar
disk. Therefore, the characteristics of primary disks are of great importance
from the point of view of protostellar disks formation.

Properties and evolution of the structures at each level of the hierarchy
strongly depends on the initial magnetic energy of the cloud. The radius of
the primary disk increases with $\varepsilon_{\rm m}$ from 500 au at
$\varepsilon_{\rm m}=0.01$ to 14000 au at $\varepsilon_{\rm m}=0.6$. The
thickness of the primary disk also increases with $\varepsilon_{\rm m}$,
although the primary disk always remain geometrically thin under considered
parameters. The radii of the primary disks are consistent with the observed
sizes of flattened envelopes of class 0 YSO (Ohashi et al., 2014; Dunham et
al., 2014; Tobin et al., 2020; Maureira et al., 2020).

The efficiency of the magnetic braking increases with the initial magnetic
energy of the cloud. The region of magnetic braking lies behind the front of
the fast MHD shock wave propagating out of the primary disk's surface along
the initial magnetic field direction. The magnetic field lines are bent behind
the shock front and strong toroidal magnetic field is generated, which drives
the magnetic braking. This region grows in time as the shock wave travels into
the envelope. In the case of weak initial magnetic field, $\varepsilon_{\rm
m}=0.01$, the fast MHD shock wave travels only small distance of
$10^{-3}\,R_{0}$ above the primary disk within the dynamical time scale of the
collapse $t_{\rm fmw}$, i. e. the angular momentum practically does not
transferred from the central part of the cloud to its envelope. In the case of
moderate initial magnetic field, $\varepsilon_{\rm m}=0.2$, the fast MHD shock
wave passes the distance of $0.1\,R_{0}$ and transfers part of the angular
momentum from the primary disk to the envelope within the dynamical time
scale. The speed of the fast MHD shock wave is so large in the case of strong
magnetic field, $\varepsilon_{\rm m}=0.6$, that its front comes out of the
cloud's envelope at $t\sim 0.8\,t_{\rm fmw}$, i. e. earlier than the end of
the dynamical collapse ($t\sim 1\,t_{\rm fmw}$). Significant part of the
angular momentum is transported out from the cloud in this case.

Rotational evolution of the cloud depends significantly on the
$\varepsilon_{\rm m}$. In the case of weak magnetic field, $\varepsilon_{\rm
m}=0.01$ (non-dimensional mass-to-flux ratio $10.5$), following picture
establishes after the first core formation. The first core rotates rigidly.
Centrifugal barrier appears at its boundary with $v_{\varphi}\sim v_{r}\approx
1.9$ km s-1. The primary disk rotates with the Keplerian velocity, as well as
the envelope does, which implies that the first core is the dominant source of
gravity in the system. In the case of moderate magnetic field,
$\varepsilon_{\rm m}=0.2$ (mass-to-flux ratio $2.4$), the first core is
practically non-rotating. The region between the first core and the primary
disk is characterized by transient reversed rotation because of torsional
Alfvén wave excitation. The primary disk rotates with Keplerian speed, but
slower than in the case of weak magnetic field. In the case of strong magnetic
field, $\varepsilon_{\rm m}=0.6$ (mass-to-flux ratio $1.4$), the first core
rotates differentially, but it is not in solid-state rotation. Inner part of
the primary disk slowly rotates with Keplerian speed, while its periphery
rotates with almost constant azimuthal speed. This is a consequence of
efficient magnetic braking of the disk's rotation.

According to our simulations, the size of the outflow region at the end of the
dynamical collapse increases with $\varepsilon_{\rm m}$. In the weak field
case, the outflow region occupies only small volume near the core, while in
the strong field case the outflow front propagates practically through the
whole cloud by the end of the dynamical collapse. This means that the
characteristic time of the outflow development and propagation from the
cloud's center decreases with $\varepsilon_{\rm m}$. This is due to the
increasing role of the electromagnetic force driving the outflow.

The `dead' zone with the low ionization fraction, $x\leq 10^{-13}-10^{-11}$,
forms inside the first core and at the base of the outflow region. In this
region, Ohmic dissipation and magnetic ambipolar diffusion weaken the magnetic
field comparing to the ideal MHD case, This effect is most pronounced for the
radial component of the magnetic field. This result is consistent with early
studies of the influence of non-ideal MHD effects at the initial stages of
collapse (e.g., Black and Scott, 1982; Dudorov and Sazonov, 1987; Mouschovias,
1991). Evolution of the first core and primary disk, as well as properties of
the further forming protostellar disks in clouds with different initial
magnetic energies, will depend on the efficiency of the Ohmic dissipation and
magnetic ambipolar diffusion in this region.

In future we plan to develop our approach and investigate the hierarchy of the
collapsing rotating magnetic PSCs in application to more realistic initial
configurations of the cloud. Construction of the synthetic continuum emission
maps and polarization maps of collapsing PSCs on the basis of our simulations
will allow to interpret observations of class 0 YSOs and analyze the
conditions for the formation of protostellar disks.

### Acknowledgements

This work is financially supported by the Russian Science Foundation (project
19-72-10012). The authors thank anonymous referee for useful comments.

## References

  * Andre et al. (1993) Andre P, Ward-Thompson D, Barsony M. 1993. Submillimeter Continuum Observations of rho Ophiuchi A: The Candidate Protostar VLA 1623 and Prestellar Clumps. Astrophys J. 406:122.
  * Andre (1995) Andre P. 1995. Low-Mass Protostars and Protostellar Stages. Astrophys Space Sci. 224(1-2):29.
  * Belloche (2013) Belloche A. 2013. Observation of rotation in star forming regions: clouds, cores, disks, and jets. Role and Mechanisms of Angular Momentum Transport During the Formation and Early Evolution of Stars, Evry Schatzman School 2012. Edited by P. Hennebelle and C. Charbonnel. EAS Publications Series. 62:25.
  * Bergin and Tafalla (2007) Bergin EA, Tafalla M. 2007. Cold Dark Clouds: The Initial Conditions for Star Formation. Annu Rev Astron Astr. 45(1):339-396.
  * Black and Scott (1982) Black DC, Scott EH. 1982. A numerical study of the effects of ambipolar diffusion on the collapse of magnetic gas clouds. Astrophys J. 263:696-715.
  * Carey et al. (1998) Carey SJ, Clark FO, Egan MP, Price SD, Shipman RF, Kuchar TA. 1998. The Physical Properties of the Midcourse Space Experiment Galactic Infrared-dark Clouds. Astrophys J. 508:721-728.
  * Caselli et al. (2002) Caselli P, Benson PJ, Myers PC, Tafalla M. 2002. Dense Cores in Dark Clouds. XIV. N2H+ (1-0) Maps of Dense Cloud Cores. Astrophys J. 572(1):238-263.
  * Crutcher (2012) Crutcher RM. 2012. Magnetic Fields in Molecular Clouds. Annu Rev Astron Astr. 50:29-63.
  * Dudorov and Sazonov (1987) Dudorov AE, Sazonov YuV. 1987. Hydrodynamical collapse of interstellar clouds. IV. The ionization fraction and ambipolar diffusion. Nauchnye Informatsii. 63:68.
  * Dudorov et al. (1999a) Dudorov AE, Zhilkin AG, Kuznetsov OA. 1999. Numerical Simulations of the Astrophysical MHD Flows. Numerical Astrophysics. Astrophysics and Space Science Library. 240:389.
  * Dudorov et al. (1999b) Dudorov AE, Zhilkin AG. 1999. MHD-Collapse of protostellar clouds. Astronomical and Astrophysical Transactions. 19:91-100.
  * Dudorov and Khaibrahmanov (2017) Dudorov AE, Khaibrahmanov SA. 2017. Hierarchical structure of the interstellar molecular clouds and star formation. Open Astronomy. 26(1):285-292.
  * Dunham et al. (2014) Dunham MM, Stutz AM, Allen LE, Evans NJ, Fischer W.J, Megeath ST et al. 2014. The Evolution of Protostars: Insights from Ten Years of Infrared Surveys with Spitzer and Herschel. Protostars and Planets VI, Henrik Beuther, Ralf S. Klessen, Cornelis P. Dullemond, and Thomas Henning (eds.), University of Arizona Press, Tucson. 914:195-218.
  * Galametz et al. (2020) Galametz M, Maury A, Girart JM, Rao R, Zhang Q, Gaudel M et al. 2020. An observational correlation between magnetic field, angular momentum and fragmentation in the envelopes of Class 0 protostars?. Astron Astrophys. 644(A47):19.
  * Girart et. al., (2006) Girart JM, Rao R, Marrone DP. 2006. Magnetic Fields in the Formation of Sun-Like Stars. Science. 313(5788):812-814.
  * Goodman et al. (1993) Goodman AA, Benson PJ, Fuller GA, Myers PC. 1993. Dense Cores in Dark Clouds. VIII. Velocity Gradients. Astrophys J. 406:528-547.
  * Hennebelle and Fromang (2008) Hennebelle P, Fromang S. 2008. Astron Astrophys. 477:9–24.
  * Hennebelle and Ciardi (2009) Hennebelle P, Ciardi A. 2009. Disk formation during collapse of magnetized protostellar cores. Astron Astrophys. 506(2):L29-32.
  * Hull and Zhang (2019) Hull CLH, Zhang Q. 2019. Interferometric observations of magnetic fields in forming stars. Frontiers in Astronomy and Space Sciences. 6:3.
  * Kargaltseva et al. (2021) Kargaltseva NS, Khaibrakhmanov SA, Dudorov AE, Zhilkin AG. 2021. Primary Disks and Their Observational Appearance in Collapsing Magnetic Rotating Protostellar Clouds. B Lebedev Phys Inst+. 48(9):268.
  * Khaibrakhmanov et al. (2021) Khaibrakhmanov SA, Kargaltseva NS, Dudorov AE, Zhilkin AG. 2021. Simulations of the Isothermal Collapse of Magnetic Rotating Protostellar Clouds. Astron Rep+. 65(8):693-704.
  * Larson (1969) Larson RB. 1969. Numerical calculations of the dynamics of collapsing proto-star. Monthly Notices Roy. Astron. Soc. 145:271.
  * Li (2021) Li H-B. 2021. Magnetic Fields in Molecular Clouds—Observation and Interpretation. Galaxies. 9(2):41.
  * Lee, et al. (2019) Lee C-F, Kwon W, Jhan K-S, Hirano N, Hwang H-C, Lai S-P et al. 2019. A Pseudodisk Threaded with a Toroidal and Pinched Poloidal Magnetic Field Morphology in the HH 211 Protostellar System. Astrophys J. 879(2):101.
  * Maureira et al. (2020) Maureira MJ, Arce HG, Dunham MM, Mardones D, Guzman AE, Pineda JE et al. 2020. ALMA observations of envelopes around first hydrostatic core candidates. Mon Not R Astron Soc. 499(3):4394.
  * Masunaga and Inutsuka (2000) Masunaga H, Inutsuka Sh-i. 2000. A Radiation Hydrodynamic Model for Protostellar Collapse. II. The Second Collapse and the Birth of a Protostar. Astrophys J. 531:350-365.
  * Mellon and Li (2008) Mellon RR, Li Z-Y. 2008. Magnetic Braking and Protostellar Disk Formation: The Ideal MHD Limit. Astrophys J. 681:1356.
  * Morii et al. (2021) Morii K, Sanhueza P, Nakamura F, Jackson JM, Li S, Beuther H et al. 2021. The ALMA Survey of 70 $\mu$m Dark High-mass Clumps in Early Stages (ASHES). IV. Star formation signatures in G023.477. eprint arXiv:2109.01231.
  * Mouschovias (1991) Mouschovias TCh. 1991. Magnetic Braking, Ambipolar Diffusion, Cloud Cores, and Star Formation: Natural Length Scales and Protostellar Masses. Astrophys J. 373:169.
  * Myers et al. (1988) Myers PC, Heyer M, Snell RL, Goldsmith PF. Dense Cores in Dark Clouds. V. CO Outflow. Astrophys J. 324:907-919.
  * Ohashi et al. (1997) Ohashi N, Hayashi M, Ho PTP, Momose M. 1997. Interferometric Imaging of IRAS 04368+2557 in the L1527 Molecular Cloud Core: A Dynamically Infalling Envelope with Rotation. Astrophys J. 475:211.
  * Ohashi et al. (2014) Ohashi N, Saigo K, Aso Y, Aikawa Y, Koyamatsu S, Machida MN et al. 2014. Formation of a Keplerian disk in the infalling envelope around L1527 IRS: transformation from infalling motions to Kepler motions. Astrophys J. 796(2):131.
  * Persson et al. (2016) Persson MV, Harsono D, Tobin JJ, van Dishoeck EF, Jorgensen JK, Murillo N et al. 2016. Constraining the physical structure of the inner few 100 AU scales of deeply embedded low-mass protostars. Astron Astrophys. 590(A33):15.
  * Pineda et al. (2019) Pineda JE, Zhao B, Schmiedeke A, Segura-Cox DM, Caselli P, Myers PC et al. 2019. The Specific Angular Momentum Radial Profile in Dense Cores: Improved Initial Conditions for Disk Formation. Astrophys J. 882(2):103.
  * Tobin et al. (2020) Tobin JJ, Sheehan PD, Megeath ST, Diaz-Rodriguez AK, Offner SSR, Murillo NM et al. 2020. The VLA/ALMA Nascent Disk and Multiplicity (VANDAM) Survey of Orion Protostars. II. A Statistical Characterization of Class 0 and Class I Protostellar Disks. Astrophys J. 890(2):130.
  * Tsukamoto et al. (2017) Tsukamoto Y, Okuzumi S, Iwasaki K, Machida MN, Inutsuka S. 2017. The impact of the Hall effect during cloud core collapse: Implications for circumstellar disk evolution. Publ Astron Soc Jpn. 69(6):95.
  * Wiseman et al. (2001) Wiseman J, Wootten A, Zinnecker H, McCaughrean M. 2001. The Flattened, Rotating Molecular Gas Core of Protostellar Jet HH 212. Astrophys J. 550(1):L87-L90.
  * Zhao et al. (2020) Zhao B, Tomida K, Hennebelle P, Tobin JJ, Maury A, Hirota T et al. 2020. Formation and Evolution of Disks around Young Stellar Objects. Space Sci Rev. 216(3):43.
  * Zhilkin et al. (2009) Zhilkin AG, Pavlyuchenkov YaN, Zamozdra SN. 2009. Modeling of protostellar clouds and their observational properties. Astron Rep+. 53(7):590-604.