# AdaptSum: Towards Low-Resource Domain Adaptation for
Abstractive Summarization

Tiezheng Yu , Zihan Liu∗, Pascale Fung
Center for Artificial Intelligence Research (CAiRE)
Department of Electronic and Computer Engineering
The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong
<EMAIL_ADDRESS><EMAIL_ADDRESS>∗ Equal contributions.
Listing order is random.

###### Abstract

State-of-the-art abstractive summarization models generally rely on extensive
labeled data, which lowers their generalization ability on domains where such
data are not available. In this paper, we present a study of domain adaptation
for the abstractive summarization task across six diverse target domains in a
low-resource setting. Specifically, we investigate the second phase of pre-
training on large-scale generative models under three different settings: 1)
source domain pre-training; 2) domain-adaptive pre-training; and 3) task-
adaptive pre-training. Experiments show that the effectiveness of pre-training
is correlated with the similarity between the pre-training data and the target
domain task. Moreover, we find that continuing pre-training could lead to the
pre-trained model’s catastrophic forgetting, and a learning method with less
forgetting can alleviate this issue. Furthermore, results illustrate that a
huge gap still exists between the low-resource and high-resource settings,
which highlights the need for more advanced domain adaptation methods for the
abstractive summarization task.111The code and data are released at:
https://github.com/TysonYu/AdaptSum

##  1 Introduction

Abstractive summarization models aim to extract essential information from
long documents and to generate short, concise and readable text. Recently,
neural abstractive summarization models have achieved remarkable performance
Gehrmann et al. (2018); Paulus et al. (2018), and large-scale generative pre-
training Lewis et al. (2019); Raffel et al. (2019) has shown itself to be
surprisingly effective at generation tasks, including abstractive
summarization. However, these models generally require large numbers of human-
annotated summaries to achieve state-of-the-art performance, which makes them
not scalable to low-resource domains where only a few labeled data are
available.

Domain adaptation methods have naturally arisen to tackle the low-resource
issue and enable models to quickly adapt to target domain tasks. Yet, despite
their practicality, very few studies have used domain adaptation methods on
the low-resource scenario for the abstractive summarization task. To address
this research gap, we present AdaptSum, the first benchmark to simulate the
low-resource domain Adaptation setting for abstractive Summarization systems
with a combination of existing datasets across six diverse domains (dialog
Gliwa et al. (2019), email Zhang and Tetreault (2019), movie review Wang and
Ling (2016), debate Wang and Ling (2016), social media Kim et al. (2019), and
science Yasunaga et al. (2019)), and for each domain, we reduce the number of
training samples to a small quantity so as to create a low-resource scenario.

Recently, conducting a second pre-training step on large-scale language models
(e.g., BERT Devlin et al. (2019), RoBERTa Liu et al. (2019a)) has proven to be
effective for domain adaptation tasks Lee et al. (2020); Gururangan et al.
(2020). However, the current methods incorporating such a step are mainly
focused on classification or classification-based (e.g., named entity
recognition) tasks, leaving a research gap in exploring their use for
generation tasks. In this paper, we systematically investigate adding a second
phase of pre-training on large-scale generative models under three settings:
1) source domain pre-training (SDPT) based on a labeled source domain
summarization dataset; 2) domain-adaptive pre-training (DAPT) based on an
unlabeled substantial domain-related corpus; and 3) task-adaptive pre-training
(TAPT) based on an unlabeled small-scale task-related corpus. The second phase
of pre-training could cause the catastrophic forgetting in the pre-trained
model. Thus, we propose to apply RecAdam Chen et al. (2020) into the pre-
training process to alleviate this issue and further improve the adaptation
performance.

Experimental results show that SDPT and TAPT can generally improve on the
performance of the fine-tuning method, while the effectiveness of DAPT is
correlated to the similarity between the pre-training data and the target
domain task data. Different from previous insights into adaptive pre-training
on classification tasks Gururangan et al. (2020), we find that in the
summarization task, DAPT could make the adaptation performance worse, even
though the pre-training corpus is collected from domain-related sources.
Furthermore, we show that RecAdam can further boost the performance of the
second pre-training step by effectively maintaining the pre-trained model’s
knowledge gained in the first phase of pre-training.

Our contributions are summarized as follows:

  * •

We introduce a low-resource domain adaptation scenario for the abstractive
summarization task to move towards the fast adaptation of summarization
systems.

  * •

To the best of our knowledge, we are the first to systematically study the
domain- and task-adaptative pre-training for a low-resource generation task.

  * •

Our work highlights the research questions and challenges in the low-resource
abstractive summarization task, which we hope will catalyze research in this
area.

##  2 Related Work

###  2.1 Abstractive Summarization

Abstractive summarization aims to generate short, concise and readable text
that captures the core meaning of the input documents. Neural networks have
achieved remarkable results for the abstractive summarization due to the
emergence of Seq2Seq models Sutskever et al. (2014) and attention mechanisms
Bahdanau et al. (2014). See et al. (2017), Paulus et al. (2017) and Gehrmann
et al. (2018) applied a pointer network to solve the out-of-vocabulary issue.
Further, See et al. (2017) used a coverage mechanism Tu et al. (2016) to keep
track of the already summarized content, which discourages repetition, while
Paulus et al. (2017) and Chen and Bansal (2018) combined reinforcement
learning into an end2end setting. Recently, pre-trained language models Peters
et al. (2018); Radford et al. (2018); Devlin et al. (2019); Dong et al.
(2019); Lewis et al. (2019) have achieved impressive gains in a wide variety
of natural language tasks. Many studies on the use of pre-trained language
models in the abstractive summarization task Liu and Lapata (2019); Yan et al.
(2020); Su et al. (2020); Yu et al. (2020) have been undertaken and have
achieved the state-of-the-art performance.

###  2.2 Domain Adaptation

Domain adaption for natural language processing and computer vision tasks is
widely studied Blitzer et al. (2007); Mansour et al. (2008); Daumé III (2009);
Sandu et al. (2010); Foster et al. (2010); Wang and Cardie (2013); Sun et al.
(2016); Liu et al. (2019b, 2020b); Gururangan et al. (2020); Winata et al.
(2020); Jadon (2020); Yin (2020); Liu et al. (2020a, d); Dai et al. (2020).
However, little has been done to investigate domain adaption for the
abstractive summarization task. Hua and Wang (2017) first studied the
adaptation of neural summarization models and showed that the models were able
to select salient information from the source domain data. Wang et al. (2019)
investigated the domain shift problem for the extractive summarization task.
Recently, Magooda and Litman (2020) studied cross-domain transfer between two
entirely different domains and introduced data synthesis methods. To the best
of our knowledge, we are the first to systematically study the domain- and
task-adaptative pre-training based on the pre-trained generative model in the
low-resource abstractive summarization task across multiple diverse domains.

Domain Unlabeled Corpus Labeled data # Tokens Size Train Valid Test Dialog
44.96M 212MB 300 818 819 Email 117.54M 705MB 300 1960 1906 Movie R. 11.36M
62MB 300 500 2931 Debate 122.99M 693MB 300 956 1003 Social M. 153.30M 786MB
300 1000 1000 Science 41.73M 291MB 100 350 497

Table 1: Data statistics of AdaptSum for the unlabeled corpus and labeled
summarization data across the six domains (“R.” and “M.” are the abbreviations
for Review and Media, respectively).

##  3 AdaptSum

The goal of AdaptSum is to provide an accessible benchmark for the evaluation
of low-resource domain adaptation for abstractive summarization on a diverse
set of domains. The vocabulary overlaps between domains are shown in Figure 1.
AdaptSum consists of six diverse target domains and the corresponding
unlabeled domain-related corpora for DAPT. We provide the data statistics of
all domains in Table 1, and the details are as follows.

#### Dialog

Gliwa et al. (2019) introduced a human-annotated abstractive chat dialog
summarization dataset. The unlabeled dialog corpus from different sources,
namely, Reddit conversations,222https://github.com/PolyAI-LDN/conversational-
datasets/tree/master/reddit personalized dialogs Zhang et al. (2018),
empathetic dialogs Rashkin et al. (2019), and Wizard of Wikipedia dialogs
Dinan et al. (2019).

#### Email

Zhang and Tetreault (2019) introduced an abstractive business and personal
email summarization dataset which consists of email and subject pairs. We
collect the unlabeled email corpus from the Enron Email
Dataset.333https://www.cs.cmu.edu/~./enron/

#### Movie Review

Wang and Ling (2016) introduced a human-annotated abstractive movie review
summarization dataset. We collect the unlabeled corpus for this domain from
IDMB Movie Review Maas et al. (2011).

#### Debate

Wang and Ling (2016) introduced an abstractive debate summarization dataset
which consists of arguments and the debate topic pairs. The unlabeled corpus
is from Ajjour et al. (2019).

#### Social Media

Kim et al. (2019) introduced an abstractive summarization dataset of Reddit
TIFU posts, where the summary for each post come from its title. We collect
the unlabeled corpus directly from Reddit
TIFU.444https://convokit.cornell.edu/documentation/subreddit.html

#### Science

Yasunaga et al. (2019) introduced a human-annotated abstractive summarization
dataset on computational linguistics. We collect the unlabeled domain corpus
from the ACL anthology Bird et al. (2008).

Figure 1: Vocabulary overlaps of the summarization validation set between
domains. The News domain is the source domain and the other six domains are
low-resource target domains. Vocabularies for each domain are created by
considering the top 10K most frequent words (excluding stopwords). We observe
that the vocabulary overlaps between domains are generally small, which
illustrates that the overlaps between domains are comparably small and the
chosen domains are diverse.

##  4 Methodology

In this section, we will first introduce the three different settings that we
investigate for a second pre-training step. Then, we will discuss how we cope
with the catastrophic forgetting issue in the second phase of pre-training.

###  4.1 A Second Phase of Pre-Training

We conduct a second pre-training phase based on a pre-trained generative
model, BART Lewis et al. (2019), on three different settings. Then, we fine-
tune it to the summarization task in the target domains. The three settings
are described as follows.

#### Source Domain Pre-Training (SDPT)

Inspired by the cross-domain setting Jia et al. (2019); Liu et al. (2020c, d),
we leverage substantial training samples from a source (News) domain (XSum
Narayan et al. (2018)), to aid in the fast adaptation in target domains. We
choose the News domain as the source domain because it is a rich-resource
domain in the summarization task, and from Figure 1, the similarity between
this domain and target domains is generally low which increases the challenge
of the domain adaptation.

Our method to conduct SDPT is straightforward. We continue pre-training BART
using the source domain summarization data. The objective function for this
pre-training is not the sentence reconstruction, as in the original pre-
training of BART. Instead, we utilize the supervisions from the source domain
summarization data to train BART on the summarization task. The purpose of
this pre-training is to inject the task knowledge into the pre-trained
language model so that the model can quickly adapt to the same task in target
domains.

#### Domain-Adaptive Pre-Training (DAPT)

We leverage an unlabeled domain-related corpus to continue pre-training BART
using its original pre-training objective function (corrupting documents and
then optimizing a reconstruction loss—the cross-entropy between the decoder’s
output and the original document). The intuition behind this method is to
introduce the domain knowledge into the pre-trained language model so as to
enable its fast adaptation to the target domains.

#### Task-Adaptive Pre-Training (TAPT)

The size of the domain-related corpus for DAPT is usually enormous, which
results in two potential drawbacks. First, such a large corpus might not be
always available, especially for the low-resource domains. Second, pre-
training on such a large corpus is time-consuming and requires excessive
computational resources. Therefore, investigating pre-training on a smaller
unlabeled corpus is a practical and beneficial research direction. TAPT refers
to pre-training on a set of the unlabeled documents in the target domain’s
summarization task. Compared to DAPT, TAPT uses a much smaller but far more
task-relevant pre-training corpus since it directly uses the input documents
from summarization task. This setting makes TAPT much less expensive to run
and independent of the collection of the large domain-related corpus.

Models Dialog Email Movie R. Debate Social M. Science Average BART Fine-tuning
39.95 24.71 25.13 24.48 21.76 72.76 34.80 SDPT 42.84 25.16 25.45 25.61 22.43
73.09 35.76 w/ RecAdam 45.23 26.97 26.06 25.17 23.25 72.60 36.55 DAPT 41.22
26.50 24.25 26.71 22.95 71.88 35.59 w/ RecAdam 40.05 25.66 25.78 25.01 21.51
72.23 35.04 TAPT 40.15 25.30 25.27 24.59 22.81 73.08 35.20 w/ RecAdam 41.34
25.73 25.65 24.70 23.01 72.80 35.54

Table 2: ROUGE-1 scores on different pre-training methods compared to the
baseline BART over all domains.

Corpus Dialog Email Movie R. Debate Social M. Science Average DAPT 212MB 705MB
62MB 693MB 786MB 291MB 458.2MB TAPT 7.9MB 14MB 3.3MB 2.4MB 74MB 384KB 17.0MB

Table 3: Corpus size comparisons between DAPT and TAPT.

###  4.2 Recall and Learn

Although the second pre-training step allows the pre-trained model to learn
the task or domain knowledge, it might lead to the catastrophic forgetting
issue and cause the pre-trained model to partly lose the language
understanding ability that it gains in the first pre-training step. To
alleviate this issue, we expect the pre-trained model to recall the previously
learned knowledge during the process of learning new knowledge. A
straightforward way to achieve this goal is to borrow the idea of continual
learning methods Kirkpatrick et al. (2017); Lopez-Paz and Ranzato (2017); Chen
et al. (2020). In this paper, we adopt RecAdam from Chen et al. (2020) for the
second phase of pre-training to weaken the catastrophic forgetting issue. The
reason for choosing RecAdam is twofold: 1) it does not require the first step
pre-training data from the pre-trained model, which is usually not available;
2) it is the most recent approach that is being successfully applied to
natural language processing tasks. The RecAdam is introduced as follows.

Based on the Adam optimizer Kingma and Ba (2015), RecAdam reconstructs the
objective function to allow it to gradually shift to the target task:

$Loss=\lambda(t)\cdot Loss_{T}+(1-\lambda(t))\cdot Loss_{S},$ (1)
$\lambda(t)=\frac{1}{1+\textnormal{exp}(-k\cdot(t-t_{0}))},$ (2)

where $k$ and $t_{0}$ are the hyper-parameters controlling the annealing rate
and time steps, $Loss_{T}$ represents the target task objective function, and
$Loss_{S}$ is used to simulate the first pre-training step of the pre-trained
model. $Loss_{S}$ can be simplified as:

$Loss_{S}=\frac{1}{2}\gamma\sum_{i}(\theta_{i}-\theta^{*}_{i})^{2},$ (3)

where $\frac{1}{2}\gamma$ is the coefficient of the quadratic penalty,
$\theta$ is the parameters of the model, and $\theta^{*}$ (fixed) is the
original parameters of the pre-trained model.

Although RecAdam has shown its effectiveness in fine-tuning BERT-like models
(e.g., BERT Devlin et al. (2019) and ALBERT Lan et al. (2020)) to the GLUE
benchmark Wang et al. (2018), exploring the effectiveness of RecAdam in the
second phase of pre-training for generative pre-trained models is not trivial.
First, the second pre-training step of a language model is a completely
different task compared to fine-tuning to downstream tasks. Second, a
generative model (e.g., BART) is structurally different from BERT-like models.
Third, the corpus sizes for SDPT and DAPT are generally much larger than the
sizes of GLUE tasks, which could affect the learning process.

##  5 Experimental Setup

#### Training Details

We evaluate all of our models on AdaptSum. For the dialog and email domains,
we use the standard splits of Gliwa et al. (2019); Zhang and Tetreault (2019),
while for movie review, debate, social media and science domains, we split the
whole dataset into training, validation and test sets by ourselves since the
original works do not specify how to split these datasets or the published
datasets do not contain the split training, validation and test sets. Since
the dataset sizes are limited for science, movie review and dialog domains,
the maximum training samples for these domains are 100, 300, and 300,
respectively, while for dialog, email, and social media domains, the maximum
training samples for them are 14732, 14436, and 60354, respectively, and we
select 300 samples for each domain to construct a low-resource setting. We
truncate the input documents into 1024 tokens due to the limitation of the
maximum input length for BART. For all the experiments, we use the BART-base
version to implement our models. We use a mini-batch size of 4 with a gradient
accumulation for 10 iterations. We use Adam optimizer with momentum
$\beta_{1}=0.9$, $\beta_{2}=0.998$ and noam decay with warm up steps of 1000.
In the decoding stage, we use beam search with a beam size of 4. The decoding
process will not stop until an end-of-sequence (EOS) token is emitted or the
length of the generated summary reaches to 256 tokens. As for the
hyperparameters of RecAdam, we select the best $t0$ and $k$ in
$\\{500,600,700,800,900,1,000\\}$ and $\\{1e-2,1e-3,1e-4,1e-5,1e-6\\}$,
respectively, for the annealing coefficient $\lambda(t)$ (Eq. 2).

#### Baseline

As our baseline, we use an off-the-shelf BART model Lewis et al. (2019) and
perform supervised fine-tuning of its parameters for the summarization task in
each domain. BART serves as a good baseline since it provides the state-of-
the-art performance in the summarization task. And, as a single generative
language model, it can be easily adapted to different target domains.

#### Evaluation Metrics

We use ROUGE Lin and Hovy (2003) to measure the quality of the summary
produced in our experiments. Following the previous work Nema et al. (2017),
we report ROUGE F1 (ROUGE-1) on the AdaptSum dataset.555We use pyrouge to
compute all ROUGE scores, with parameters “-c 95 -2 4 -U -r 1000 -n 4 -w 1.2
-a”. The full results of all the models with ROUGE-2 and ROUGE-L are reported
in the Appendix.

##  6 Results & Analysis

###  6.1 Main Results

From Table 2, we can see that SDPT is able to generally improve the
summarization performance of the fine-tuning method for all domains. This is
because SDPT teaches the model how to do the task using large numbers of
annotated examples, which enables the model to adapt to target domains faster
than the fine-tuning method, and SDPT is able to outperform both DAPT and TAPT
in terms of the averaged ROUGE-1 score. The enormous unlabeled corpus makes
DAPT quite effective in certain domains, such as email, debate and social
media, with close to or more than 2 ROUGE-1 scores improvements over the fine-
tuning baseline. As we can see from Table 3, although TAPT uses a far smaller
pre-training corpus than DAPT, the performance of TAPT is on par with that of
DAPT, which accords with the results in Gururangan et al. (2020), where the
experiments are conducted for domain adaptation in classification tasks.
Additionally, adding RecAdam into the second phase of pre-training can
generally further boost the adaptation performance for SDPT and TAPT, while it
only boost the performance on the movie review and science domains for DAPT.
We conjecture that a relatively large corpus can potentially weaken the
effectiveness of RecAdam, and we observe that the corpus used for DAPT is
comparably small for movie review and science domains, and the number of data
samples for XSum (204k) is also much smaller than those of DAPT corpora in
many domains (e.g., email), which have more than 1M sentences. According to
Eq. 1, extensive training data could result in a comparatively large
$Loss_{s}$ (the model’s parameters tend to be greatly modified) which lead to
an unstable loss and a negative effect to the pre-training process. In
addition, we find that RecAdam is originally shown to be effective at fine-
tuning to the downstream GLUE tasks Chen et al. (2020), the sizes of which are
much smaller than the datasets used for SDPT and DAPT.

Domains DAPT Corpus TAPT Corpus Dialog 37.56 (-7.04) 44.60 Email 51.87 (-5.93)
57.80 Movie R. 46.63 (-14.59) 61.22 Debate 53.49 (-8.99) 62.48 Social M. 48.10
(-3.82) 51.92 Science 36.94 (-20.90) 57.84 Average 45.44 (-10.54) 55.98

Table 4: Vocabulary overlaps (%) between the pre-training corpus (for DAPT or
TAPT) and the validation set of the summarization task for each domain. The
numbers in the brackets denote the vocabulary overlap differences between the
two pre-training corpora, and the bold numbers denote the large discrepancies.

###  6.2 How Pre-training Data Affects DAPT

According to prior experiments on domain adaptation for classification or
classification-based tasks Beltagy et al. (2019); Lee et al. (2020);
Gururangan et al. (2020), DAPT improves the performance for all domains on the
fine-tuning baseline. However, as we can see from Table 2, DAPT causes the
performance to drop for the movie review and science domains in the
summarization task, while TAPT boosts the performance for all the domains. To
further investigate the reasons, we aim to analyze the similarity (e.g.,
vocabulary overlap) between the pre-training corpus for DAPT and the
summarization task in the target domain, which we represent with the target
domain validation set of the summarization task to represent. We notice that
it is difficult to justify how much overlap is large enough for DAPT to be
considered as effective. Hence, we add the TAPT corpus, which is directly
related to the target domain’s summarization task, as an upper bound for the
comparison.

Table 4 illustrates the vocabulary overlaps for DAPT and TAPT for each
domain.666To ensure the comparison between DAPT and TAPT is fair, we sample
partial data from the DAPT corpus to make its size comparable to the TAPT
corpus and create vocabularies for each based on the top 5K most frequent
words (excluding stopwords). The vocabulary for the validation set of the
summarization task is also created in the same way. We find large
discrepancies between the DAPT corpus and TAPT corpus on the movie review and
science domains, which indicates that the domain-related corpora in these two
domains are not quite related to the task domains, and pre-training on a
domain-unrelated or less related corpus can lead to a performance drop
compared to the fine-tuning method. Given that the corpus construction is done
by looking for the domain-related sources (as mentioned in Section 3), the
experimental results point out that collecting a domain-related corpus for
DAPT in the summarization task is not straightforward. Thus, we leave
exploring how to construct an effective corpus for DAPT for future work.

Figure 2: TAPT performance over different pre-training epoch numbers in the
email domain in terms of using and not using RecAdam.

Source Domains Target Domains XSum CNN/DM Dialog Email Movie R. Debate Social
M. Science Document 354.16 676.03 91.64 124.47 2112.97 196.69 229.31 633.03
Summary 21.13 57.91 20.28 4.10 21.28 11.07 6.31 150.01

Table 5: Averaged length of the input documents and output summaries for the
source and target domains.

###  6.3 Catastrophic Forgetting Issue

We speculate that the second phase of pre-training will result in the
catastrophic forgetting for the pre-trained model, which could hurt the
adaptation performance. Figure 2 illustrates that the performance of TAPT
without RecAdam keeps dropping as the pre-training continues, and it starts to
perform worse than the fine-tuning method after three epochs’ pre-training,
while the performance of TAPT with RecAdam remains stable at around a 25.5
ROUGE-1 score. We conjecture that excessive pre-training makes the pre-trained
model overfit to the pre-training data and partially lose its language
understanding and generation ability. However, the model is required to
possess both language ability and domain knowledge for better performance in
the domain adaptation task. RecAdam helps the pre-trained model preserve its
original language ability while continuing pre-training on a new corpus, which
boosts the effectiveness of pre-training. However, as we can see from Table 2,
RecAdam fails to improve the performance on DAPT using large corpora. We
speculate that the catastrophic forgetting issue does not do much harm to the
performance of DAPT because pre-training on the large corpus enables the pre-
trained model to possess a good language understanding ability in the target
domain even though it could lead to partial forgetting in previous domains,
and RecAdam makes DAPT stay somewhere in the middle (not forgetting much the
previous learned knowledge, but not learning well in the target domain,
either). It indicates that more advanced learning methods are needed for
coping with the second pre-training phase on a large corpus.

Domains BART SDPT DAPT SDPT+DAPT Dialog 39.95 42.84 41.22 42.27 Email 24.71
25.16 26.50 23.71 Movie R. 25.13 25.45 24.25 22.20 Debate 24.48 25.61 26.71
25.16 Social M. 21.76 22.43 22.95 22.03 Science 72.76 73.09 71.88 71.56
Average 34.80 35.76 35.59 34.49

Table 6: ROUGE-1 results for SDPT+DAPT compared to the SDPT, DAPT and BART
fine-tuning.

###  6.4 Incorporating SDPT and DAPT

Intuitively, incorporating both the summarization task and target domain
knowledge into the pre-trained model could further boost the domain adaptation
performance in the summarization task. Therefore, we propose to combine SDPT
and DAPT in the second pre-training step. Since SDPT and DAPT use different
objective functions, jointly learning these two tasks will make BART confused
about what to generate (summarization or sentence reconstruction) given the
input sequences. To cope with this issue, we use two BART models (one for SDPT
and one for DAPT) and share their encoders in this joint pre-training process
to learn the knowledge from both the task and domain. Then, we use the BART
model for SDPT to fine-tune to the summarization task in the target domain.

As shown in Table 6, the experimental results are contradictory to the
intuition. We find that SDPT+DAPT can not further improve upon the performance
of SDPT and DAPT. For the dialog and social media domains, the performances of
SDPT+DAPT stay between those of SDPT and DAPT, while for the science, movie
review and email domains, the performances of SDPT+DAPT are even lower than
that of the BART fine-tuning. We conjecture that SDPT and DAPT are two
completely different tasks, and jointly pre-training based on them could
confuse the model about the knowledge that it learns. However, integrating the
task and domain knowledge is still a promising direction for domain
adaptation. We leave how to incorporate SDPT and DAPT for future work.

Domains BART SDPT (XSum) SDPT (CNN) Dialog 39.95 42.84 43.13 Email 24.71 25.16
23.81 Movie R. 25.13 25.45 24.51 Debate 24.48 25.61 23.98 Social M. 21.76
22.43 22.56 Science 72.76 73.09 72.41 Average 34.80 35.76 35.07

Table 7: ROUGE-1 results for SDPT based on the XSum and CNN/DM (denoted as CNN
in the table) datasets.

Figure 3: ROUGE-1 results of BART fine-tuning, DAPT and SDPT over different
numbers of training data for email (left) and dialog (right) domains. We
consider both low-resource settings (50, 100, 200 and 300 ($\sim$2%) samples),
medium-resource settings (25% and 50% samples), and high-resource settings
(75% and 100% samples).

###  6.5 Different Source Domain Data for SDPT

To explore how different source domain data can affect the performance of
SDPT, we use another News domain dataset, CNN/Daily Mail (DM) dataset Hermann
et al. (2015); Nallapati et al. (2016), as the labeled summarization data for
SDPT. As we can see from Table 7, SDPT based on CNN/DM only achieves marginal
improvements upon the BART fine-tuning baseline in terms of the averaged
score, and for all the domains, it generally performs worse or similar
compared to SDPT based on XSum. Since both of them are from the News domain
but the number of training samples in CNN/DM (287k) is higher than that in
XSum (204k), pre-training on CNN/DM should have achieved better performance
than pre-training on XSum. To further analyze the reason, we calculate the
averaged length of input documents and output summaries for the source and
target domains. From Table 5, we find that the averaged length of XSum is much
shorter than that of CNN/DM in terms of both document and summary, and
surprisingly, SDPT based on XSum can outperform SDPT based on CNN/DM in
domains with short length document and summary (e.g., debate and email) as
well as the domains with long length document or summary (e.g., movie review
and science). Hence, we conjecture that pre-training with relatively short
document and summary is more effective for SDPT. Another reason can be
attributed to the fact that the summaries of the CNN/DM tend to copy the
content in the input documents, while XSum has larger amounts of novel tokens
in the summaries. Therefore, we conjecture that XSum enables model learn a
more powerful summarization ability, which helps it to better adapt to low-
resource target domains. We leave investigating the effectiveness of different
source domain datasets in SDPT for future work.

###  6.6 Performance vs. Training Sample Size

We investigate how well models perform in an extremely low-resource scenario
(e.g., 50 training samples) and the performance discrepancies among different
levels of resources. The performance over different numbers of training
samples is illustrated in Figure 3. We find that BART fine-tuning with the 25%
data samples significantly outperforms that with $\sim$2% data samples in the
dialog domain, but such improvements are not remarkable in the email domain.
We conjecture that the input and output lengths for the email domain are
relatively short compared to the dialog domain (according to Table 5), making
the domain adaptation easier.

Interestingly, DAPT outperforms other models in the medium-resource and high-
resource settings in the email domain but not in the dialog domain. We
speculate the reasons are twofold. First, based on the vocabulary overlaps
from Table 4, the email corpus is more effective for DAPT than the dialog
domain. Second, email corpus is much larger than the dialog corpus from Table
3. However, the performance of DAPT using a high-quality corpus will be still
limited by the low-resource scenario, and it needs large enough training
samples to achieve remarkable improvements. Moreover, the performance of TAPT
is better than BART fine-tuning in the low-resource setting, while it becomes
worse in the medium-resource and high-resource settings. We conjecture that
training with more data will aggravate the catastrophic forgetting caused by
TAPT, which leads to the worse performance.

Surprisingly, the performance of DAPT with medium-resource is close to that
with high-resource, which can be attributed to the combination of the powerful
adaptation ability of the large pre-trained generative model and the
effectiveness of the second phase of pre-training. However, there is still a
large performance gap for all the models between the low-resource and high-
resource settings and all the models perform badly when there is only 50
training samples, which highlights the needs for more advanced domain
adaptation models for the summarization task.

##  7 Conclusion and Future Work

In this paper, we present AdaptSum, the first benchmark to simulate the low-
resource setting for the abstractive summarization task with a combination of
existing datasets across six diverse domains. We systematically study three
different methods for a second phase of pre-training (i.e., SDPT, DAPT and
TAPT), and propose to leverage RecAdam to alleviate the catastrophic
forgetting issue caused by the continuing pre-training. Experiments show that
SDPT and TAPT can generally improve on the performance of the fine-tuning
method, while the effectiveness of DAPT depends on the similarity between the
pre-training data and the target domain task data, which is different from the
insights into DAPT for classification tasks. Further analysis illustrates that
RecAdam successfully alleviates the catastrophic forgetting issue for TAPT and
further boost its performance.

Finally, our work highlights several research challenges in low-resource
domain adaptation for the abstractive summarization task: (1) How to construct
an effective corpus for DAPT; (2) How to better cope with the catastrophic
forgetting issue for the second pre-training phase on a large corpus; (3) How
to effectively integrate the task and domain knowledge (i.e., incorporate SDPT
and DAPT); (4) How to choose better source domain datasets for conducting
SDPT; (5) How to build a more powerful domain adaptation models for the
extremely low-resource summarization task. We hope that the proposed dataset
and the highlighted research directions will accelerate the studies in this
area.

## Acknowledgments

We want to thank the anonymous reviewers for their constructive feedback. This
work is partially funded by ITF/319/16FP, ITS/353/19FP and MRP/055/18 of the
Innovation Technology Commission, the Hong Kong SAR Government.

## References

  * Ajjour et al. (2019) Yamen Ajjour, Henning Wachsmuth, Johannes Kiesel, Martin Potthast, Matthias Hagen, and Benno Stein. 2019.  Data acquisition for argument search: The args. me corpus.  In _Joint German/Austrian Conference on Artificial Intelligence (Künstliche Intelligenz)_ , pages 48–59. Springer.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_.
  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  Scibert: A pretrained language model for scientific text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3606–3611.
  * Bird et al. (2008) Steven Bird, Robert Dale, Bonnie J. Dorr, Bryan Gibson, Mark T. Joseph, Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir R. Radev, and Yee Fan Tan. 2008.  The ACL Anthology Reference Corpus: A Reference Dataset for Bibliographic Research in Computational Linguistics.  In _Proc. of the 6th International Conference on Language Resources and Evaluation Conference (LREC’08)_ , pages 1755–1759.
  * Blitzer et al. (2007) John Blitzer, Mark Dredze, and Fernando Pereira. 2007.  Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification.  In _Proceedings of the 45th annual meeting of the association of computational linguistics_ , pages 440–447.
  * Chen et al. (2020) Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020\.  Recall and learn: Fine-tuning deep pretrained language models with less forgetting.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7870–7881, Online. Association for Computational Linguistics.
  * Chen and Bansal (2018) Yen-Chun Chen and Mohit Bansal. 2018.  Fast abstractive summarization with reinforce-selected sentence rewriting.  _arXiv preprint arXiv:1805.11080_.
  * Dai et al. (2020) Wenliang Dai, Zihan Liu, Tiezheng Yu, and Pascale Fung. 2020.  Modality-transferable emotion embeddings for low-resource multimodal emotion recognition.  _arXiv preprint arXiv:2009.09629_.
  * Daumé III (2009) Hal Daumé III. 2009.  Frustratingly easy domain adaptation.  _arXiv preprint arXiv:0907.1815_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Dinan et al. (2019) Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019.  Wizard of wikipedia: Knowledge-powered conversational agents.  In _International Conference on Learning Representations_.
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified language model pre-training for natural language understanding and generation.  In _Advances in Neural Information Processing Systems_ , pages 13063–13075.
  * Foster et al. (2010) George Foster, Cyril Goutte, and Roland Kuhn. 2010.  Discriminative instance weighting for domain adaptation in statistical machine translation.  In _Proceedings of the 2010 conference on empirical methods in natural language processing_ , pages 451–459.
  * Gehrmann et al. (2018) Sebastian Gehrmann, Yuntian Deng, and Alexander M Rush. 2018.  Bottom-up abstractive summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4098–4109.
  * Gliwa et al. (2019) Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019.  Samsum corpus: A human-annotated dialogue dataset for abstractive summarization.  In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pages 70–79.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  In _Proceedings of ACL_.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _Advances in neural information processing systems_ , pages 1693–1701.
  * Hua and Wang (2017) Xinyu Hua and Lu Wang. 2017.  A pilot study of domain adaptation effect for neural abstractive summarization.  _arXiv preprint arXiv:1707.07062_.
  * Jadon (2020) Shruti Jadon. 2020.  An overview of deep learning architectures in few-shot learning domain.  _arXiv preprint arXiv:2008.06365_.
  * Jia et al. (2019) Chen Jia, Xiaobo Liang, and Yue Zhang. 2019.  Cross-domain ner using cross-domain language modeling.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2464–2474.
  * Kim et al. (2019) Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2019.  Abstractive summarization of reddit posts with multi-level memory networks.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2519–2531.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _ICLR (Poster)_.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  Albert: A lite bert for self-supervised learning of language representations.  In _International Conference on Learning Representations_.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Lin and Hovy (2003) Chin-Yew Lin and Eduard Hovy. 2003.  Automatic evaluation of summaries using n-gram co-occurrence statistics.  In _Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 150–157.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text summarization with pretrained encoders.  _arXiv preprint arXiv:1908.08345_.
  * Liu et al. (2019a) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019a.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Liu et al. (2019b) Zihan Liu, Jamin Shin, Yan Xu, Genta Indra Winata, Peng Xu, Andrea Madotto, and Pascale Fung. 2019b.  Zero-shot cross-lingual dialogue systems with transferable latent variables.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1297–1303.
  * Liu et al. (2020a) Zihan Liu, Genta Indra Winata, and Pascale Fung. 2020a.  Zero-resource cross-domain named entity recognition.  In _Proceedings of the 5th Workshop on Representation Learning for NLP_ , pages 1–6.
  * Liu et al. (2020b) Zihan Liu, Genta Indra Winata, Zhaojiang Lin, Peng Xu, and Pascale Fung. 2020b.  Attention-informed mixed-language training for zero-shot cross-lingual task-oriented dialogue systems.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8433–8440.
  * Liu et al. (2020c) Zihan Liu, Genta Indra Winata, Peng Xu, and Pascale Fung. 2020c.  Coach: A coarse-to-fine approach for cross-domain slot filling.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 19–25, Online. Association for Computational Linguistics.
  * Liu et al. (2020d) Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. 2020d.  Crossner: Evaluating cross-domain named entity recognition.  _arXiv preprint arXiv:2012.04373_.
  * Lopez-Paz and Ranzato (2017) David Lopez-Paz and Marc’Aurelio Ranzato. 2017.  Gradient episodic memory for continual learning.  In _Advances in neural information processing systems_ , pages 6467–6476.
  * Maas et al. (2011) Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies_ , pages 142–150.
  * Magooda and Litman (2020) Ahmed Magooda and Diane Litman. 2020.  Abstractive summarization for low resource data using domain transfer and data synthesis.  _arXiv preprint arXiv:2002.03407_.
  * Mansour et al. (2008) Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. 2008.  Domain adaptation with multiple sources.  _Advances in neural information processing systems_ , 21:1041–1048.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016.  Abstractive text summarization using sequence-to-sequence rnns and beyond.  _arXiv preprint arXiv:1602.06023_.
  * Narayan et al. (2018) Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807.
  * Nema et al. (2017) Preksha Nema, Mitesh M Khapra, Anirban Laha, and Balaraman Ravindran. 2017.  Diversity driven attention model for query-based abstractive summarization.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1063–1072.
  * Paulus et al. (2017) Romain Paulus, Caiming Xiong, and Richard Socher. 2017.  A deep reinforced model for abstractive summarization.  _arXiv preprint arXiv:1705.04304_.
  * Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. 2018.  A deep reinforced model for abstractive summarization.  In _International Conference on Learning Representations_.
  * Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  _arXiv preprint arXiv:1802.05365_.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving language understanding by generative pre-training.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Rashkin et al. (2019) Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019.  Towards empathetic open-domain conversation models: A new benchmark and dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5370–5381.
  * Sandu et al. (2010) Oana Sandu, Giuseppe Carenini, Gabriel Murray, and Raymond Ng. 2010.  Domain adaptation to summarize human conversations.  In _Proceedings of the 2010 Workshop on Domain Adaptation for Natural Language Processing_ , pages 16–22.
  * See et al. (2017) Abigail See, Peter J Liu, and Christopher D Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  _arXiv preprint arXiv:1704.04368_.
  * Su et al. (2020) Dan Su, Yan Xu, Tiezheng Yu, Farhad Bin Siddique, Elham J Barezi, and Pascale Fung. 2020.  Caire-covid: A question answering and multi-document summarization system for covid-19 research.  _arXiv preprint arXiv:2005.03975_.
  * Sun et al. (2016) Baochen Sun, Jiashi Feng, and Kate Saenko. 2016.  Return of frustratingly easy domain adaptation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 30.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  In _Advances in neural information processing systems_ , pages 3104–3112.
  * Tu et al. (2016) Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016.  Modeling coverage for neural machine translation.  _arXiv preprint arXiv:1601.04811_.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355.
  * Wang et al. (2019) Danqing Wang, Pengfei Liu, Ming Zhong, Jie Fu, Xipeng Qiu, and Xuanjing Huang. 2019\.  Exploring domain shift in extractive text summarization.  _arXiv preprint arXiv:1908.11664_.
  * Wang and Cardie (2013) Lu Wang and Claire Cardie. 2013.  Domain-independent abstract generation for focused meeting summarization.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1395–1405.
  * Wang and Ling (2016) Lu Wang and Wang Ling. 2016.  Neural network-based abstract generation for opinions and arguments.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 47–57.
  * Winata et al. (2020) Genta Indra Winata, Samuel Cahyawijaya, Zihan Liu, Zhaojiang Lin, Andrea Madotto, Peng Xu, and Pascale Fung. 2020.  Learning fast adaptation on cross-accented speech recognition.  _arXiv preprint arXiv:2003.01901_.
  * Yan et al. (2020) Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020.  Prophetnet: Predicting future n-gram for sequence-to-sequence pre-training.  _arXiv preprint arXiv:2001.04063_.
  * Yasunaga et al. (2019) Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander R Fabbri, Irene Li, Dan Friedman, and Dragomir R Radev. 2019.  Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 7386–7393.
  * Yin (2020) Wenpeng Yin. 2020.  Meta-learning for few-shot natural language processing: A survey.  _arXiv preprint arXiv:2007.09604_.
  * Yu et al. (2020) Tiezheng Yu, Dan Su, Wenliang Dai, and Pascale Fung. 2020.  Dimsum@ laysumm 20: Bart-based approach for scientific document summarization.  _arXiv preprint arXiv:2010.09252_.
  * Zhang and Tetreault (2019) Rui Zhang and Joel Tetreault. 2019.  This email could save your life: Introducing the task of email subject line generation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 446–456.
  * Zhang et al. (2018) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018.  Personalizing dialogue agents: I have a dog, do you have pets too?  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2204–2213.

##  Appendix A Full Results of All Models

The full results of all models are shown in Table 8.

Domains ROUGE Scores BART Fine-tuning SDPT SDPT w/ RecAdam DAPT DAPT w/
RecAdam TAPT TAPT w/ RecAdam Dialog ROUGE-1 F1 39.95 42.84 45.23 41.22 40.05
40.15 41.34 ROUGE-2 F1 17.50 17.51 19.43 17.88 17.62 16.99 17.88 ROUGE-L F1
31.64 33.79 35.37 32.40 31.36 31.21 32.31 Email ROUGE-1 F1 24.71 25.16 26.97
26.50 25.66 25.30 25.73 ROUGE-2 F1 11.71 12.2 13.44 13.14 12.89 12.03 12.69
ROUGE-L F1 24.15 24.28 25.98 25.61 25.14 24.63 25.32 Movie R. ROUGE-1 F1 25.13
25.45 26.06 24.25 25.78 25.27 25.65 ROUGE-2 F1 9.22 9.49 10.27 9.06 9.84 9.24
9.13 ROUGE-L F1 20.04 20.11 20.91 19.56 20.69 20.09 20.45 Debate ROUGE-1 F1
24.48 25.61 25.17 26.71 25.01 24.59 24.70 ROUGE-2 F1 8.21 8.48 8.38 9.14 8.42
8.13 8.43 ROUGE-L F1 21.96 22.86 22.39 23.64 22.17 22.04 22.25 Social M.
ROUGE-1 F1 21.76 22.43 23.25 22.95 21.51 22.81 23.01 ROUGE-2 F1 8.11 9.06 9.01
9.66 8.25 8.96 8.49 ROUGE-L F1 21.03 21.03 22.18 21.93 20.69 22.06 21.95
Science ROUGE-1 F1 72.76 73.09 72.60 71.88 72.23 73.08 72.80 ROUGE-2 F1 64.66
65.15 63.79 63.73 63.32 65.04 64.26 ROUGE-L F1 68.40 68.62 68.06 67.34 67.62
68.81 68.41

Table 8: Full results of all models.

##  Appendix B Training Details

Our model contains $\sim$139.4 million parameters and we train all models on
one GTX 1080 Ti. We train all the models for 50 epochs in around three hours.
We manually tune the hyperparameter values.


$\displaystyle\max_{1\leq\ell\leq n}{\rm
E}\left|\widehat{\chi}^{(n)}_{\ell\bm{\varsigma}}-\chi_{\ell\bm{\varsigma}}\right|\leq\
$ $\displaystyle
C\max(n^{-1/2},\alpha_{\bar{\bm{\varsigma}}}^{1/2})M_{S_{1}}M_{S_{2}}M_{T}+C^{*}(1+\varepsilon_{1})^{-\kappa^{*}_{1}(s_{1})}(1+\varepsilon_{2})^{-\kappa^{*}_{2}(s_{2})}(1+\varepsilon_{3})^{-\kappa^{*}_{3}(t)}.$

This proves part (i).

Furthermore, we have that

$(1+\varepsilon_{1})^{-\kappa^{*}_{1}(s_{1})}(1+\varepsilon_{2})^{-\kappa^{*}_{2}(s_{2})}(1+\varepsilon_{3})^{-\kappa^{*}_{3}(t)}=o\left\\{(1+\varepsilon_{1})^{-M_{S_{1}}}(1+\varepsilon_{2})^{-M_{S_{2}}}(1+\varepsilon_{3})^{-M_{T}}\right\\},$

which by Assumption 6.5 is dominated by the first term of part (i). This
proves part (ii) and completes the proof.

##  Appendix F Proofs of Results of of Section 7

In order to prove the theorem, we need the following result

######  Lemma F.1.

Let Assumptions 3.1, 3.2, 6.1, 6.2, 6.3, 6.4, 6.5, and 7.1 hold. Then, for all
$\epsilon>0$ there exist $\delta_{\epsilon}$, $S_{1\epsilon}$, $S_{2\epsilon}$
and $T_{\epsilon}$ such that for any fixed $q_{\max}$, $n$,
$S_{1}>S_{1\epsilon}$, $S_{2}>S_{2\epsilon}$, and $T>T_{\epsilon}$,

$\displaystyle\max_{1\leq k\leq q_{\max}}\sup_{\bm{\theta}\in\bm{\Theta}}{\rm
P}\left\\{\min\left[\frac{1}{\log B_{S_{1}}\log B_{S_{2}}\log
B_{T}}\sqrt{\frac{S_{1}S_{2}T}{B_{S_{1}}B_{S_{2}}B_{T}}},B_{S_{1}}^{\vartheta_{1}},B_{S_{2}}^{\vartheta_{2}},B_{T}^{\vartheta_{3}}\right]\frac{|\widehat{\lambda}^{x}_{nk}(\bm{\theta})-{\lambda}^{x}_{nk}(\bm{\theta})|}{n}>\delta_{\epsilon}\right\\}<\epsilon.$

###### Proof.

The result follows immediately from Lemma A.1 in Hallin and Liška (2007) and
Lemma E.5(i). ∎

It then suffices to prove that for all $k\neq q$, $0\leq k\leq q_{\max}$, as
$n,S_{1},S_{2},T\to\infty$,

${\rm P}\left[\widehat{\rm IC}^{(n)}(k)-\widehat{\rm
IC}^{(n)}(q)>0\right]\rightarrow 1.$ (F.44)

Start with the case that $k<q$. Letting

$\widehat{D}_{nk}=\frac{1}{n}\sum_{j=k+1}^{n}\frac{1}{8\pi^{3}}\int_{\bm{\Theta}}\widehat{\lambda}^{x}_{nj}(\bm{\theta})\mathrm{d}\bm{\theta}\quad\text{
and
}\quad{D}_{nk}=\frac{1}{n}\sum_{j=k+1}^{n}\frac{1}{8\pi^{3}}\int_{\bm{\Theta}}{\lambda}^{x}_{nj}(\bm{\theta})\mathrm{d}\bm{\theta},$

we have

$\displaystyle\widehat{\rm IC}^{(n)}(k)-\widehat{\rm
IC}^{(n)}(q)=\log\left[\frac{(\widehat{D}_{nk}-\widehat{D}_{nq})}{\widehat{D}_{nq}}+1\right]+(k-q)p(n,S_{1},S_{2},T).$
(F.45)

Now Lemma E.7, implies that
$\inf_{\bm{\theta}\in\bm{\Theta}}n^{-1}\lambda^{x}_{nq}>0$ and this yields
that, as $n\rightarrow\infty$

$\log\left[\frac{({D}_{nk}-{D}_{nq})}{{D}_{nq}}+1\right]>0.$ (F.46)

The desired result (F.44) follows by using (F.46), Lemma F.1, and Assumption
7.1 (specifically, the assumption $p(n,S_{1},S_{2},T\rightarrow 0$)) in
(F.45).

Let us then consider the case $k>q$. Note that

${D}_{nq}-{D}_{nk}=\frac{1}{n}\sum_{j=q+1}^{k}\frac{1}{8\pi^{3}}\int_{\bm{\Theta}}{\lambda}^{x}_{nj}(\bm{\theta})\mathrm{d}\bm{\theta}\leq\frac{(k-q)C}{n},$
(F.47)

for some finite $C>0$ independent of $n$. Indeed, by Weyl’s inequality and
Proposition E.1,

$\sup_{\bm{\theta}\in\bm{\Theta}}\sup_{n\in\mathbb{N}}\lambda^{x}_{n,q+1}(\bm{\theta})\leq\sup_{\bm{\theta}\in\bm{\Theta}}\sup_{n\in\mathbb{N}}\left\\{\lambda^{\chi}_{n,q+1}(\bm{\theta})+\lambda^{\xi}_{n,q+1}(\bm{\theta})\right\\}=\sup_{\bm{\theta}\in\bm{\Theta}}\sup_{n\in\mathbb{N}}\lambda^{\chi}_{n,q+1}(\bm{\theta})\leq
C.$

From (F.47) and Lemma F.1 we have

$\widehat{D}_{nq}-\widehat{D}_{nk}=O_{\rm P}\left\\{\max\left[\frac{1}{n},\log
B_{S_{1}}\log B_{S_{2}}\log
B_{T}\sqrt{\frac{B_{S_{1}}B_{S_{2}}B_{T}}{S_{1}S_{2}T}},\frac{1}{B_{S_{1}}^{\vartheta_{1}}},\frac{1}{B_{S_{2}}^{\vartheta_{2}}},\frac{1}{B_{T}^{\vartheta_{3}}}\right]\right\\},$

which implies also

$\displaystyle\log\left[\frac{(\widehat{D}_{nq}-\widehat{D}_{nk})}{\widehat{D}_{nk}}+1\right]=O_{\rm
P}\left\\{\max\left[\frac{1}{n},\log B_{S_{1}}\log B_{S_{2}}\log
B_{T}\sqrt{\frac{B_{S_{1}}B_{S_{2}}B_{T}}{S_{1}S_{2}T}},\frac{1}{B_{S_{1}}^{\vartheta_{1}}},\frac{1}{B_{S_{2}}^{\vartheta_{2}}},\frac{1}{B_{T}^{\vartheta_{3}}}\right]\right\\}.$

Now, given that

$\displaystyle\widehat{\rm IC}_{n}(q)-\widehat{\rm
IC}_{n}(k)=\log\left[\frac{(\widehat{D}_{nk}-\widehat{D}_{nq})}{\widehat{D}_{nq}}+1\right]+(q-k)p(n,S_{1},S_{2},T),$

(F.44) follows from Assumption 7.1, which yields

${\rm
P}\left((k-q)p(n,S_{1},S_{2},T)>\log\left[\frac{(\widehat{D}_{nq}-\widehat{D}_{nk})}{\widehat{D}_{nk}}+1\right]\right)\to
1\quad\text{as}\quad n,S_{1},S_{2},T\to\infty.$

This completes the proof.

##  Appendix G Selection of the number of factors in practice

Clearly, $c\mapsto\widehat{q}_{c}^{(n)}$ is a non-increasing map: a small
(large) value of $c$ corresponds to underpenalization (overpenalization).
Therefore, the correct identification of $q$ should be based on a sequence of
$c$, starting from a small value until appropriate penalization is reached. A
thorough discussion and numerical analysis on this aspect in the context of
GDFM is available in Hallin and Liška (2007). Adopting it to our setting of
spatio-temporal setting, we propose the following procedure for the choice of
$c$. For a given sample of dimension $n$ and a fixed $c>0$, consider a
sequence of estimator $\widehat{q}_{c}^{(n_{j})},j=1,\ldots,J$, where
$0<n_{1}<\ldots<n_{J}=n$, and define a measure of variability by

$S_{c}=\frac{1}{J}\sum_{j=1}^{J}\left(\widehat{q}_{c}^{(n_{j})}-\frac{1}{J}\sum_{j=1}^{J}\widehat{q}_{c}^{(n_{j})}\right)^{2}.$
(G.48)

Notice that for $c$ close to zero, due to underpenalization issue, one always
obtains $\widehat{q}_{c}^{(n)}=q_{\max}$, so that this yields the first
“stability interval" of the map $c\mapsto S_{c}$, where $S_{c}=0$ for any $c$
in this interval. On the contrary, for a large $c$, overpenalization leads to
a stability interval of the map $c\mapsto S_{c}$, where $S_{c}=0$ and
$\widehat{q}_{c}^{(n)}=0$ for any $c$ in this interval. Numerical studies in
Hallin and Liška (2007) suggest choosing $c$ and the corresponding
$\widehat{q}_{c}^{(n)}$ that belongs to the second stability interval of the
map $c\mapsto S_{c}$.

Input: data $\\{x_{\ell\bm{\varsigma}},\
\ell=1,\ldots,n,\bm{\varsigma}=(s_{1}\ s_{2}\
t)^{\top},s_{1}=1,\ldots,S_{1},s_{2}=1,\ldots,S_{2},t=1,\ldots,T\\}$; upper
bound $q_{\max}$;a sequence $n_{1},\ldots,n_{J}$ integers for subsample
dimensions; a sequence values $c_{1},\ldots,c_{L}$ of reals; kernel functions
$K_{1}(\cdot)$, $K_{2}(\cdot)$, and $K_{3}(\cdot)$;bandwidths integers
$B_{S_{1}},B_{S_{2}}$, and $B_{T}$; penalty $p(n,S_{1},S_{2},T)$.

Output: $\widehat{q}_{\widehat{c}}^{(n)}$.

1

2Choose a random permutation of the $n$ cross-sectional items.

3for  _$\ell\leftarrow 1$ to $L$_ do

4 for  _$j\leftarrow 1$ to $J$_ do

5 Compute $\widehat{\bm{\Sigma}}_{n_{j}}^{x}(\bm{\theta}_{\bm{h}})$ as in
(17), with $\bm{\theta}_{\bm{h}}$ as in Remark 6.1.

6 Compute the $q_{\max}$ largest eigenvalues of
$\widehat{\bm{\Sigma}}_{n_{j}}^{x}(\bm{\theta})$.

7 Compute the information criterion $\widehat{\rm IC}^{(n_{j})}(k)$ as in
(22), with the penalty $kc_{\ell}p(n_{j},,S_{1},S_{2},T)$.

8 Obtain $\widehat{q}_{c_{\ell}}^{(n_{j})}$.

9 end for

10

11 Compute $S_{c_{\ell}}$ as in (G.48).

12 end for

13

14Plot $c\mapsto S_{c}$ and choose a $\widehat{c}$ that belongs to the second
stability interval of the plot.

15return _$\widehat{q}_{\widehat{c}}^{(n)}$._

Algorithm 2 Algorithm for selection of $q$

In Figure 3, we plot $c\mapsto 4S_{c}$ (in blue) and
$c\mapsto\widehat{q}_{{c}}^{(n)}$ (in red) for Model (a) in (23) and Model (b)
in (24). The first stability interval, where $S_{c}=0$ and $c$ is close to 0,
corresponds to $\widehat{q}_{c}^{(n)}=q_{\max}$. For $c$ in the second
stability interval, we have $\widehat{q}_{\widehat{c}}^{(n)}=q$ for both
models, as expected. These plots and the results in Table 3 are obtained with
$n=100$, $(S_{1},S_{2},T)=(25,25,25)$, $q=3$, $n_{j}=n-5j,j=1,2,\ldots,16$,
$c_{\ell}=\ell/2000,\ell=0,1,\ldots,6000$, $q_{\max}=10$ and penalty

$p(n,S_{1},S_{2},T)=(n^{-1}+B_{S_{1}}^{-\vartheta_{1}}+B_{S_{2}}^{-\vartheta_{2}}+B_{T}^{-\vartheta_{3}}+V_{\bar{\bm{\varsigma}}}^{-1})\log[\min(n,B_{S_{1}}^{\vartheta_{1}},B_{S_{2}}^{\vartheta_{2}},B_{T}^{\vartheta_{3}},V_{\bar{\bm{\varsigma}}})],$

where

$V_{\bar{\bm{\varsigma}}}=\frac{(S_{1}S_{2}T)^{1/2}}{(B_{S_{1}}B_{S_{2}}B_{T})^{1/2}\log(B_{S_{1}})\log(B_{S_{2}})\log(B_{T})}.$

Figure 3: Plots of $c\mapsto 4S_{c}$ (blue) and
$c\mapsto\widehat{q}_{{c}}^{(n)}$ (red) for model (24) (left) and model (23)
(right), with $n=100$, $(S_{1},S_{2},T)=(25,25,25)$, and $q=3$

##  Appendix H Real data example

###  H.1 Data preparation

We apply the proposed method to model a resting-state cerebral functional
magnetic resonance imaging (rs-fMRI) dataset obtained from the Alzhimer’s
Disease Neuroimaging Initiative (ADNI) research project
(https://adni.loni.usc.edu). The primary goal of ANDI is to measure the
progression of mild cognitive impairment and early Alzheimer’s disease. The
subjects in our rs-fMRI dataset consists of 139 aged 55-90 years old from 59
research centers in the U.S. and Canada. Their first brain scans after
enrolled in ANDI are examined in our dataset. Out of the 139 subjects, we have
41 participants that are cognitively normal (CN), and 98 participants with
different conditions or diseases, which can be further divided into several
sub-groups: 32 Alzheimer’s disease (AD) patients, 40 mild cognitive impairment
(MCI) patients and 26 significant memory concern (SMC) patients.

Following the protocol pre-processing steps as described in Appendix H.5, we
obtain numerical brain activity measurements for each subject at 130 time
points and at 116 spatial locations, which are corresponding to the Anatomical
Automatic Labeling brain atlases template (Tzourio-Mazoyer et al., 2002).
These locations in human brain are characterised by irregular coordinates. To
apply our methodology we need to transfer the data into a regular spatial
lattice. This can be performed resorting on optimal transportation (OT)
theory, which provides a transportation map from irregular data in
$\mathbb{R}^{d}$ to regular data in $\mathbb{R}^{d^{\prime}}$, with
$d,d^{\prime}\geq 1$. Here we simply say that thanks to the use of OT, we
obtain an optimal coupling that allows to map irregular spatial 3D data to
regular data on a 2D lattice, while preserving the spatial structure to the
maximum extent. Thanks this procedure, we obtain a $10\times 11$ grid over a
2D spatial network. Moreover, as it is customary in the statistical analysis
of fMRI data, for each subject we smooth the time available series by applying
moving average filters—with window length $(2,2,4)$.

###  H.2 Spatio-temporal correlations

We first analyse the strength of spatial-temporal dependence in each group of
patients. To do so, consider the estimator
$\widehat{\bm{\Gamma}}_{n}^{x}(\bm{h})$ and let
$\widehat{\gamma}_{n,ij}^{x}(\bm{h})$, $i,j=1,\ldots,n,$ denote the $(i,j)$-th
entry of $\widehat{\bm{\Gamma}}_{n}^{x}(\bm{h})$, and denote by
$\widehat{\rho}_{ni}^{x}(\bm{h})=\widehat{\gamma}_{n,ii}^{x}(\bm{h})/\widehat{\gamma}_{n,ii}^{x}(\bm{0})$
the sample spatio-temporal autocorrelation for the rf $x_{i}$.

To investigate the spatial correlations, Figure 4 shows the heatmap of
$n^{-1}\sum_{i=1}^{n}\widehat{\rho}_{ni}^{x}(\bm{h})$ (that is, the average,
over patients, of $\widehat{\rho}_{ni}^{x}(\bm{h})$) for each subgroup, where
we set $h_{1},h_{2}=0,\ldots,4$ and $h_{3}=0$. Clearly, for all subgroups,
strong spatial dependence exists along both spatial directions. For the
temporal correlation, in Figure 5, we display the heatmap of
$n^{-1}\sum_{i=1}^{n}\widehat{\rho}_{ni}^{x}(\bm{h})$ for each subgroup, where
we set $h_{1},h_{2}=0$ and $h_{3}=0,\ldots,4$. Hence, to model these spatio-
temporal dynamics, we can apply a GSTFM.

Figure 4: Spatial correlations. Heatmaps of the
$n^{-1}\sum_{i=1}^{n}\widehat{\rho}_{ni}^{x}(\bm{h})$ for the subgroups (CN,
AD, MCI, and SMC) ($h_{1},h_{2}=0,\ldots,4$ and $h_{3}=0$). Figure 5: Temporal
correlations. Heatmaps of the
$n^{-1}\sum_{i=1}^{n}\widehat{\rho}_{ni}^{x}(\bm{h})$ for the subgroups (CN,
AD, MCI, and SMC) ($h_{1},h_{2}=0$ and $h_{3}=0,\ldots,4$).

###  H.3 Number of factors

For all the subgroups, we plot the largest 20 eigenvalues in Figure 6 for
frequency $(\pi,\pi,\pi)$ (top panel) and for the averaged values over
$8\times 8\times 8$ frequencies on the regular grid of $[-\pi,\pi]^{3}$
(bottom panel). A rapid inspection of both figures reveals difference between
the CN and the other subgroups: the gaps between the first and second
eigenvalues of the CN are significantly larger than the other subgroups.

Figure 6: Plot of the largest 20 dynamic spatio-temporal eigenvalues at
frequency $(\pi,\pi,\pi)$ (top) and averaged over $8\times 8\times 8$
frequencies on the regular grid of $[-\pi,\pi]^{3}$ (bottom), for the
subgroups (CN, AD, MCI, and SMC). Figure 7: Plots of $c\mapsto S_{c}$ (blue)
and $c\mapsto\widehat{q}_{\widehat{c}}^{(n)}$ (red) for the subgroups (CN, AD,
MCI, and SMC).

To gain further understanding, we then apply Algorithm 2 to select the number
of latent factors $q$. In Figure 7 we display the plots of $c\mapsto S_{c}$
(blue line) and $c\mapsto\widehat{q}_{{c}}^{(n)}$ (red line). The plots
suggest that $\widehat{q}^{(n)}_{\widehat{c}}=1$ for the CN group, while
$\widehat{q}^{(n)}_{\widehat{c}}=0$ for the other subgroups. Computing all of
the averaged dynamic eigenvalues of the CN subgroup, we note that the
explained variance of the common component is $20\%$.

###  H.4 Estimation of the common component

Next, we analyse the common component ${\chi}_{\ell\bm{\varsigma}}$,
$\ell=1,\ldots,n,$ of the CN subgroup based on the estimator
$\widehat{\chi}^{(n)}_{\ell\bm{\varsigma}}$. In order to construct a quantity
that represents strength, over time, of the common component for all subjects
in the CN subgroup, we define a temporal coincident indicator based on the
weighted average of the estimated common component (averaged over space in
advance). More specifically, the temporal coincident indicator is defined as

${\rm CI}_{t}^{(n)}=\sum_{\ell=1}^{n}w_{\ell}\bar{\chi}^{(n)}_{\ell t},\quad
t=1,\ldots,T,$

where $\bar{\chi}^{(n)}_{\ell
t}=S_{1}^{-1}S_{2}^{-1}\sum_{s_{1}=1}^{S_{1}}\sum_{s_{1}=1}^{S_{2}}\widehat{\chi}^{(n)}_{\ell\bm{\varsigma}}$
is the average of $\widehat{\chi}^{(n)}_{\ell\bm{\varsigma}}$ taken over
space, and the weight

$w_{\ell}=\frac{\sum_{t=1}^{T}(\bar{\chi}^{(n)}_{\ell
t})^{2}}{\sum_{\ell=1}^{n}\sum_{t=1}^{T}(\bar{\chi}^{(n)}_{\ell t})^{2}}$

is defined according to the level (in time domain) of the common component of
subject $\ell$. Heuristically, by this construction of weights, the subjects
that are the main drivers of the common factor can gain more weights in ${\rm
CI}_{t}^{(n)}$.

Figure 8 shows plots of ${\rm CI}_{t}^{(n)}$, $t=1,\ldots,T$, and of the
observed spatio-temporal rf (averaged over space) for four randomly selected
subjects in the CN subgroup. Obvious co-movements of the CI and rf can be
observed and the CI seems to capture the magnitude of fluctuations of the rf
for all subjects. To investigate further the behaviour of ${\rm CI}_{t}^{(n)}$
and rf in frequency domain, we plot their periodograms in Figure 9 (to make a
fair comparison, the periodograms, smoothed with Hamming window, of the rf are
rescaled through dividing the ratios of their integral over frequency with
respect to that of ${\rm CI}_{t}^{(n)}$). In general, we observe that the
periodogram of ${\rm CI}_{t}^{(n)}$ and the subject specific periodogram have
similar shapes: they both have several peaks between $0$ and $0.4\pi$, with
the former showing a slight phase shift to higher frequencies compared to the
latter.

Figure 8: Plots of the temporal coincident indicator $\rm{CI}_{t}^{(n)}$ and
rf (averaged over space) for four randomly selected subjects in the CN
subgroup. Figure 9: Plots of the periodograms of the temporal coincident
indicator and rf (averaged over space) over frequency $[0,\pi]$ for four
randomly selected subjects in the CN subgroup.

In a similar fashion, we can analyse the common component in space domain by
defining a spatial coincident indicator based on the weighted average of the
estimated common component (averaged over time). More precisely, the spatial
coincident indicator in space domain is defined as

${\rm
CI}_{{\bm{s}}}^{(n)}=\sum_{\ell=1}^{n}\widetilde{w}_{\ell}\widetilde{\chi}^{(n)}_{\ell{\bm{s}}},\quad{\bm{s}}=(s_{1}\
s_{2})^{\top},\quad s_{1}=1,\ldots,S_{1},\ s_{2}=1,\ldots,S_{2},$

where
$\widetilde{\chi}^{(n)}_{\ell{\bm{s}}}=T^{-1}\sum_{t=1}^{T}\widehat{\chi}^{(n)}_{\ell\bm{\varsigma}}$
is the average of $\widehat{\chi}^{(n)}_{\ell\bm{\varsigma}}$ taken over time,
and the weight

$\widetilde{w}_{\ell}=\frac{\sum_{s_{1}=1}^{S_{1}}\sum_{s_{1}=1}^{S_{2}}(\widetilde{\chi}^{(n)}_{\ell
t})^{2}}{\sum_{\ell=1}^{n}\sum_{s_{1}=1}^{S_{1}}\sum_{s_{1}=1}^{S_{2}}(\widetilde{\chi}^{(n)}_{\ell
t})^{2}}$

is defined according to the level (in space domain) of the common component of
subject $\ell$. To elaborate more on the ${\rm CI}_{{\bm{s}}}^{(n)}$, we now
turn to the analysis of its spatial periodogram. Following (Mandrekar and
Redett, 2017, Ch. 4), the spatial periodogram of rf $\\{y_{s_{1},s_{2}};1\leq
s_{1}\leq S_{1},1\leq s_{2}\leq S_{2}\\}$ is

$I_{S_{1},S_{2}}({\bm{h}})=\frac{1}{(2\pi)^{2}S_{1}S_{2}}\left\|\sum_{s_{1}=1}^{S_{1}}\sum_{s_{1}=1}^{S_{2}}y_{s_{1},s_{2}}e^{-i2\pi\langle{\bf
s}\cdot{\bm{h}}\rangle}\right\|^{2},\quad{\bm{h}}\in\mathbb{R}^{2}.$

Figure 10: Plots of the periodograms of coincident indicator and rf (averaged
over time) for four randomly selected subjects in the CN subgroup. The spatial
CI and rf are averaged over one spatial direction first.

To evaluate the spatial periodograms over two marginal spatial directions, we
process ${\rm CI}_{{\bm{s}}}^{(n)}$ and the rf (averaged over time) in the
following way: (i) take the average over one spatial direction of the data and
(ii) compute the periodogram as in the time domain for the other spatial
direction. The periodograms of ${\rm CI}_{{\bm{s}}}^{(n)}$ and rf along
$s_{1}$ and $s_{2}$ directions are available in the left and right panels of
Figure 10, respectively (coherently with the previous analysis for the time
component, the periodograms of the rf are rescaled). We notice that for the
second subject, the periodograms, along $s_{1}$ direction, of the rf have two
peaks, where the first one has the same frequency as the peak of the
periodograms of ${\rm CI}_{{\bm{s}}}^{(n)}$. For all the other cases, the
periodograms of ${\rm CI}_{{\bm{s}}}^{(n)}$ and rf have a single peak that
occurs at almost the same frequency. In general, similarly to what we
concluded looking at the time component only, we observe that the periodogram
of ${\rm CI}_{S}^{(n)}$ and the subject specific spatial periodogram have
similar shapes.

###  H.5 fMRI data pre-processing

In the ADNI research, each subject’s brain was scanned by a 3.0T Philips MR
Ingenia Elition scanner and produced a 3D map of the voxels measuring the
brain activity. The rs-fMRI ouput is firstly preprocessed using the software
Data Processing Assistant for Resting-State fMRI (DPARSF) based on Statistical
Parametric Mapping 12 (SPM12) on MATLAB. Next, for each participant we discard
the first 10 time points to avoid the instability of the initial MRI signals.
Moreover, we introduced an extra correction for the acquisition time delay and
head motion in the images. The inclusion criteria is below 3 mm translation
and below 3∘ rotation for the head movements during the fMRI scan. After these
corrections, the images were normalized to the standard Montreal Neurological
Institute (MNI) template at a 3 mm $\times$ 3 mm $\times$ 3 mm resolution. The
final resultant data were filtered through a temporal band-pass (0.01–0.1 Hz)
to avoid the interferences of low-frequency drift and physiological noises.

##  Appendix I Additional simulation results

We show that the proposed estimator
$\widehat{\bm{\Sigma}}_{n}^{x}(\bm{\theta})$ yields estimated spatio-temporal
dynamic eigenvalues having an eigen-gap under GSTFM (see Theorem 4.1). To this
end, we set $n=100$, $(S_{1},S_{2},T)=(15,15,15)$, and, for each MC run, we
estimate the spectral density, as in (17), and the spatio-temporal dynamic
eigenvalues at selected frequencies. For the ease-of-computation, we set the
frequencies on a $5\times 5\times 5$ equally spaced grid over $[0,\pi]^{3}$.
To see how the eigen-gap changes with the dimension of the rf, we simulate
different rf ${\bm{x}}_{m}$, with increasing dimension $m=1,\ldots,n$. For
each frequency over the grid, we treat each estimated eigenvalue as a function
of $m$ and obtain $100$ curves (one curve for each MC run). To summarize the
behavior of these curves, we consider the largest $q+1$ eigenvalues averaged
over the $125$ discrete frequencies and we plot the resulting average curves
against $m$. To complete the picture, we repeat this analysis for different
number of factors: $q=2,3,4$.

In Figure 12 and Figure 12 we display the related functional boxplots (Sun and
Genton (2011)). The figures illustrate that, even for small values of $m$, an
eigen-gap is clearly detectable: the first $q$ eigenvalues seem to diverge
with $m$ almost linearly, while the $(q+1)$-th eigenvalue remains close to
zero.

|  |
---|---|---

Figure 11: Functional boxplots of the largest $q+1$ spatio-temporal dynamic
eigenvalues (averaged over all frequencies) for Model (a) in (23), with
$n=100$, $(S_{1},S_{2},T)=(15,15,15)$, and $q=2$ (left), $q=3$ (middle), and
$q=4$ (right).

|  |
---|---|---

Figure 12: Functional boxplots of the largest $q+1$ spatio-temporal dynamic
eigenvalues (averaged over all frequencies) for Model (b) in (24), with
$n=100$, $(S_{1},S_{2},T)=(15,15,15)$, and $q=2$ (left), $q=3$ (middle), and
$q=4$ (right).

|  |
---|---|---

Figure 13: Functional boxplots, over $100$ replications, of the largest $q+1$
spatio-temporal dynamic eigenvalues at frequency $0$ for Model (a) in (23),
with $n=100,(S_{1},S_{2},T)=(15,15,15)$, and $q=2$ (left), $q=3$ (middle), and
$q=4$ (right).

|  |
---|---|---

Figure 14: Functional boxplots, over $100$ replications, of the largest $q+1$
spatio-temporal dynamic eigenvalues at frequency $0$ for Model (b) in (24),
with $n=100,(S_{1},S_{2},T)=(15,15,15)$, and $q=2$ (left), $q=3$ (middle), and
$q=4$ (right).

Figures 16 and 14 display the functional boxplots of the largest $q+1$
eigenvalues at $0$ frequency for Model (a) in (23) and Model (b) in (24),
respectively. The left, middle and right panels in each figure are for
$q=2,3,4$, respectively. In each plot, the $50\%$ central regions of different
eigenvalues are shown in different colours. The black curves in the central
regions represent the sample median functions. The blue curves represent the
envelope (i.e., 1.5 times the $50\%$ central region). The red dashed curves
are the outliers outside the envelope.

|  |
---|---|---

Figure 15: Functional boxplots, over $100$ replications, of the largest $q+1$
spatio-temporal dynamic eigenvalues at frequency $0$ for Model (a) in (23)
with the idiosyncratic components generated as in (I.49), and with
$n=100,(S_{1},S_{2},T)=(15,15,15)$, and $q=2$ (left), $q=3$ (middle), and
$q=4$ (right).

|  |
---|---|---

Figure 16: Functional boxplots, over $100$ replications, of the largest $q+1$
spatio-temporal dynamic eigenvalues at frequency $0$ for Model (b) in (24)
with the idiosyncratic components generated as in (I.49), and with
$n=100,(S_{1},S_{2},T)=(15,15,15)$, and $q=2$ (left), $q=3$ (middle), and
$q=4$ (right).

Finally, since the GSTFM is able to accommodate also the presence of mildly
cross-sectionally correlated idiosyncratic component, we illustrate the
presence of the eigen-gap also in a novel setting. Specifically, we assume
that in Model (a) and Model (b), the idiosyncratic component
$\xi_{\ell\bm{\varsigma}}$ is

$\xi_{\ell\bm{\varsigma}}=\sum_{\bm{\kappa}=(-1\ -1\ 0)}^{(1\ 1\
1)}\sum_{j=0}^{4}0.5^{|\kappa_{1}|+|\kappa_{2}|+|\kappa_{3}|+j}c_{\ell
j\bm{\kappa}}L^{\bm{\kappa}}{v}_{\ell+j,\bm{\varsigma}},$ (I.49)

where ${v}_{\ell,\bm{\varsigma}}$ from i.i.d. standard normal distribution and
$c_{\ell i\bm{\kappa}}$ from i.i.d. uniform distributions on $[0.5,0.8]$ are
independent. By design, the $\xi_{\ell\bm{\varsigma}}$’s are autocorrelated in
cross-section, space and time. In Figures 16 and 16, we show that also in this
case the eigen-gap is clearly detectable.


# All countable subsets of pseudocompact quasitopological Korovin groups are
closed, discrete and $C^{\ast}$-embedded

Evgenii Reznichenko<EMAIL_ADDRESS>Department of General Topology and
Geometry, Mechanics and Mathematics Faculty, M. V. Lomonosov Moscow State
University, Leninskie Gory 1, Moscow, 199991 Russia  Mikhail Tkachenko111The
second listed author was supported by grant number CAR-64356 of the Program
‘‘Ciencia de Frontera 2019’’ of the CONACyT, Mexico<EMAIL_ADDRESS>Departamento de Matemáticas, Universidad Autónoma Metropolitana, Av. San
Rafael Atlixco 186, Col. Vicentina, C.P. 09340, Del. Iztapalapa, Mexico City,
Mexico

(May 15, 2023)

###### Abstract

We show that all countable subsets of any pseudocompact quasitopological group
in the form of a Korovin orbit are closed, discrete, and $C^{\ast}$-embedded.
Consequently, any infinite pseudocompact Korovin orbit is not homeomorphic to
a topological group. Moreover, infinite pseudocompact Korovin orbits are not
homeomorphic to any Mal’tsev space.

###### keywords:

Pseudocompact groups , Quasitopological groups , Korovin orbit , Mal’tsev
space

##  1 Introduction

A group with a topology is called semitopological if multiplication in the
group is separately continuous. Ellis [8] proved that a locally compact
semitopological group is a topological group, that is, multiplication and
inverse are continuous. In [15], it is proved that a completely regular
countably compact semitopological group is a topological group.

A semitopological group is called quasitopological if inversion in the group
is continuous. In [12], A. Korovin presented a method for the construction of
pseudocompact semitopological groups that are neither topological groups nor
homeomorphic to any topological group (the latter is shown in our Corollary
1). An analysis of Korovin’s method led to the notions of _Korovin map_ and
_Korovin orbit_ [3, 11] (see also Theorem 2.4.13 in [4]). Korovin orbits are
also used to construct pseudocompact quasitopological groups with various
additional properties [3, 11, 5, 19].

Our main result states that all countable subsets of a pseudocompact Korovin
orbit are closed, discrete, and $C^{\ast}$-embedded (Theorem 2). Consequently,
pseudocompact Korovin’s orbits are not homeomorphic to topological groups or
even paratopological groups. Moreover, pseudocompact Korovin’s orbits are not
homeomorphic to any Mal’tsev space (Corollary 2).

### Around Grothendieck’s theorem

We say that a Tychonoff space $X$ is a Korovin space if for any continuous
mapping $f\colon X\to C_{p}(X)$, the image $f(X)$ has compact closure. Here
$C_{p}(X)$ denotes the space of continuous real-valued functions on $X$ with
the topology of pointwise convergence. Since $C_{p}(X)$ contains a closed copy
of the real line, any Korovin space is pseudocompact. In [15], it was proved
that if a semitopological group $G$ is a Korovin space, then it is a
topological group.

A Grothendieck’s theorem in [10] states that the closure of a countably
compact subspace of $C_{p}(X)$, for a countably compact $X$, is compact. It
follows from this result that countably compact spaces are Korovin spaces,
which implies the aforementioned result about countably compact
semitopological groups. Numerous generalizations of Grothendieck’s theorem
show that the class of Korovin spaces is quite wide, for example, it includes
both separable and Fréchet–Urysohn pseudocompact spaces [12, 15, 2].

As shown in Corollary 1, any infinite pseudocompact Korovin orbit is not
homeomorphic to a topological group. According to [15] and the above-mentioned
result, every infinite pseudocompact Korovin orbit is not a Korovin space. We
conclude, therefore, that Grothendieck’s theorem cannot be extended to
pseudocompact spaces. Using D. Shakhmatov’s example of an infinite
pseudocompact space $X$ in which every countable subset is closed, discrete,
and $C^{\ast}$-embedded may provide another example of this type, as pointed
out by V. Tkachuk. The subspace $Y=C_{p}(X,[0,1])$ of $C_{p}(X)$ is closed in
$C_{p}(X)$, pseudocompact, and non-compact. Theorem 2 shows that in the above
argument, pseudocompact Korovin orbits can be used in place of Shakhmatov’s
space from [18].

### Mal’tsev spaces

In a pseudocompact topological group, any infinite subgroup is not discrete
[11]. Hence, an infinite pseudocompact Korovin orbit is not homeomorphic to a
topological group. A ternary operation $M\colon X^{3}\to X$ is called a
Mal’tsev operation if the equality

$M(y,y,x)=M(x,y,y)=x$

holds for all $x,y\in X$. A space with a continuous Mal’tsev operation is
called Mal’tsev space. There exists a natural Mal’tsev operation on every
group $X$ defined by $M(x,y,z)=xy^{-1}z$. If $X$ is a topological group, then
$M$ is continuous. If $X$ is a quasitopological group, then $M$ is separately
continuous. Therefore, every pseudocompact quasitopological Korovin orbit
admits a separately continuous Mal’tsev operation, but it does not admit a
continuous Mal’tsev operation.

It is worth mentioning that if $X$ is a pseudocompact Mal’tsev space or a
Korovin space with a separately continuous Mal’tsev operation, then the
Stone–Čech compactification $\beta X$ of $X$ is a Dugundji compactum [17, 16].

##  2 Korovin orbits

Let $(G,+)$ be an Abelian group and $X$ a space with at least two elements.
The group $G$ acts on $X^{G}$ as follows:

$s:G\times X^{G}\to X^{G},\ s(g,f)(h)=f(g+h),$

where $f\in X^{G}$ and $g,h\in G$. An element $f\in X^{G}$ is called a Korovin
mapping if for any $M\subset G$ with $|M|\leq\omega$ and $h\in X^{M}$ there
exists an element $g\in G$ such that $h=\left.s(g,f)\right|_{M}$.

######  Theorem 1 ([12] and for $|G|=2^{\omega}$, Theorem 2.4.13 in [4]).

If $G$ is an Abelian group, $X$ is a space, and
$2\leq|X|\leq|G|=|G^{\omega}|$, then there exists a Korovin mapping $f\colon
G\to X$.

The subspace $G_{f}=\\{s(g,f)\,:\,g\in G\\}$ of $X^{G}$, for a Korovin mapping
$f$, is called a Korovin orbit. Let $\pi_{M}\colon X^{G}\to X^{M}$ be the
projection, where $M\subset G$.

######  Proposition 1 ([12] and Proposition 2.4.14 in [4]).

If $G$ is an Abelian group, $X$ is a space, and $f\in X^{G}$ is a Korovin
mapping, then $\pi_{M}(G_{f})=X^{M}$, for any countable set $M\subset G$.

Proposition 1 implies the following result.

######  Proposition 2 ([12, 3]).

A Korovin orbit $G_{f}\subset X^{G}$ is pseudocompact if and only if
$X^{\omega}$ is pseudocompact. In this case, $\beta G_{f}=(\beta X)^{G}$.

Let $G_{f}\subset X^{G}$ be a Korovin orbit. The mapping ${\mathfrak{i}}\colon
G\to G_{f}$ defined by ${\mathfrak{i}}(g)=s(g,f)$ for all $g\in G$ is
bijective [3]. We consider the algebraic structure on $G_{f}$ induced by $G$.
This means that we define the sum in $G_{f}$ be letting
${\mathfrak{i}}(g)+{\mathfrak{i}}(h)={\mathfrak{i}}(g+h)$ for all $g,h\in G$.
If $e$ is the identity element of $G$, then ${\mathfrak{i}}(e)=s(e,f)$ is the
identity of the group $G_{f}$. Also, $G_{f}$ is a semitopological group
provided it carries the topology inherited from $X^{G}$ and if $G$ is a
Boolean group, then $G_{f}$ is a quasitopological group.

######  Proposition 3.

Let $G$ be an Abelian group, $X$ a space, and $f\in X^{G}$ a Korovin mapping.
The function $h\circ{\mathfrak{i}}^{-1}\colon G_{f}\to X$ is continuous, for
each $h\in G_{f}$.

###### Proof.

Given an element $h\in G_{f}$, there is $\alpha\in G$ such that
$h=s(\alpha,f)$.

Claim. _$(h\circ{\mathfrak{i}}^{-1})(x)=x(\alpha)$ for each $x\in G_{f}$._

Indeed, take $\beta\in G$ such that $x=s(\beta,f)$. Then

$\displaystyle(h\circ{\mathfrak{i}}^{-1})(x)$
$\displaystyle=h({\mathfrak{i}}^{-1}(x))=h({\mathfrak{i}}^{-1}(s(\beta,f)))=h(\beta)=s(\alpha,f)(\beta)$
$\displaystyle=f(\alpha+\beta)=f(\beta+\alpha)=s(\beta,f)(\alpha)=x(\alpha).$

It follows from our Claim that
$h\circ{\mathfrak{i}}^{-1}=\left.\pi_{\alpha}\right|_{G_{f}}$, where

$\pi_{\alpha}:G^{X}\to X,\ h\mapsto h(\alpha)$

is the projection. Since $\pi_{\alpha}$ is continuous, the mapping
$h\circ{\mathfrak{i}}^{-1}$ is continuous as well. ∎

Recall that a subset $Y$ of a space $X$ is called $C$-embedded
($C^{\ast}$-embedded) in $X$ if any (bounded) continuous real-valued function
on $Y$ extends to a continuous real-valued function on $X$. Two nonempty
subsets $A,B\subset X$ are called _completely separated_ in $X$ if there
exists a continuous function $f\colon X\to\mathbb{R}$ such that $f(A)=\\{0\\}$
and $f(B)=\\{1\\}$. According to [9, 1.17], a subset $Y$ of a space $X$ is
$C^{\ast}$-embedded in $X$ if and only if every two completely separated
subsets of $Y$ are completely separated in $X$. It is essential to note that
the space $X$ is not assumed to satisfy any separation restriction in this
fact (known as _Urysohn’s extension theorem_).

The following proposition is immediate from Urysohn’s extension theorem.

######  Proposition 4.

Let $X$ be a space and $D\subset X$. Then $D$ is discrete and
$C^{\ast}$-embedded in $X$ if and only if any two nonempty disjoint subsets of
$D$ are completely separated in $X$.

The following theorem is the article’s main result. In it, we do not impose
any separation restriction on $X$.

######  Theorem 2.

Let $G$ be an Abelian group, $X$ be a space with $|X|\geq 2$, and $f\colon
G\to X$ a Korovin mapping.

  1. (1)

If $X$ is not anti-discrete, then then every countable subset of the Korovin
orbit $G_{f}$ is closed and discrete in $G_{f}$.

  2. (2)

If there exists a non-constant continuous real-valued function on $X$, then
every countable subset in the Korovin orbit $G_{f}$ is $C^{\ast}$-embedded.

  3. (3)

If there is an unbounded continuous real-valued function on $X$, then any
countable subset of the Korovin orbit $G_{f}$ is $C$-embedded.

###### Proof.

(1) Let $A$ be a countable subset of $G_{f}$ and $g\in G_{f}\setminus A$. It
suffices to check that $g\notin\overline{A}$. Let $U$ be a proper nonempty
open subset of $X$. Take elements $x\in U$ and $y\in X\setminus U$. Since $f$
is a Korovin mapping, there exists $h\in G_{f}$ such that
$h({\mathfrak{i}}^{-1}(g))=x$ and $h({\mathfrak{i}}^{-1}(A))=\\{y\\}$.
Proposition 3 implies that the mapping $h\circ{\mathfrak{i}}^{-1}$ is
continuous, so

$V=(h\circ{\mathfrak{i}}^{-1})^{-1}(U)\ni g$

is open in $G_{f}$ and $A\subset G_{f}\setminus V$. Hence
$g\notin\overline{A}$.

(2) According to Proposition 4 it suffices to prove that any countable
disjoint subsets $A,B$ of $G_{f}$ are completely separated in $G_{f}$. Let
$A^{\prime}={\mathfrak{i}}^{-1}(A)$ and $B^{\prime}={\mathfrak{i}}^{-1}(B)$.
Let $q\colon X\to\mathbb{R}$ be a non-constant continuous function,
$x_{1},x_{2}\in X$ and $q(x_{1})\neq q(x_{2})$. Let $M=A^{\prime}\cup
B^{\prime}$ and $h\colon M\to X$ be a function such that
$h(A^{\prime})=\\{x_{1}\\}$ and $h(B^{\prime})=\\{x_{2}\\}$. Since $f$ is a
Korovin mapping, there exists $g\in G$ such that $h=\left.s(g,f)\right|_{M}$.
Proposition 3 implies that the mapping
$\varphi=s(g,f)\circ{\mathfrak{i}}^{-1}:G_{f}\to X$ is continuous. Then
$\varphi(A)=\\{x_{1}\\}$ and $\varphi(B)=\\{x_{2}\\}$. The function
$\psi=q\circ\varphi$ is continuous, $\psi(A)=\\{q(x_{1})\\}$,
$\psi(B)=\\{q(x_{2})\\}$, and $q(x_{1})\neq q(x_{2})$.

(3) Let $A=\\{a_{n}\,:\,n\in\omega\\}\subset G_{f}$ be a countable set and
$v\colon A\to\mathbb{R}$ be a function. Let also $q\colon X\to\mathbb{R}$ be
an unbounded continuous function. There is a sequence
$(x_{n})_{n\in\omega}\subset X$ such that $Q=\\{q(x_{n}):n\in\omega\\}$ is a
closed discrete subset of $\mathbb{R}$ and $q(x_{n})\neq q(x_{m})$ if $n\neq
m$. Since $Q$ is $C$-embedded in $\mathbb{R}$, there exists a continuous
function $r\colon\mathbb{R}\to\mathbb{R}$ such that $r(q(x_{n}))=v(a_{n})$ for
each $n\in\omega$. Let $M={\mathfrak{i}}^{-1}(A)$ and $h\colon M\to X$ be a
function such that $h({\mathfrak{i}}^{-1}(a_{n}))=x_{n}$ for each
$n\in\omega$. Since $f$ is a Korovin mapping, there exists $g\in G$ such that
$h=\left.s(g,f)\right|_{M}$. Proposition 3 implies that the mapping
$\varphi=s(g,f)\circ{\mathfrak{i}}^{-1}\colon G_{f}\to X$ is continuous. Then
the mapping $\psi=r\circ q\circ\varphi$ is continuous and
$v=\left.\psi\right|_{A}$. ∎

For the reader convenience, we present a short proof of the following well-
known result (see e.g. [4, Corollary 1.4.24]).

######  Proposition 5.

An infinite pseudocompact topological group has a countable non-closed subset.

###### Proof.

A pseudocompact topological group is precompact [6, Theorem 1.1], a subgroup
of a precompact topological group is precompact, and a discrete precompact
group is finite. Hence, in an infinite pseudocompact topological group $G$,
each infinite subgroup is not discrete, and $G$ has a countable non-discrete
subspace. ∎

In fact, the above argument shows that Proposition 5 remains valid for
infinite precompact topological groups.

Proposition 5 and Theorem 2 imply the following assertion announced in the
abstract.

######  Corollary 1.

Any infinite pseudocompact Korovin orbit is not homeomorphic to a topological
group.

##  3 Discrete subsets of Mal’tsev spaces

Our aim in this short section is to extend Corollary 1 to pseudocompact
Mal’tsev spaces.

A subset $Y$ of a Mal’tsev space $X$ with a Mal’tsev operation $M$ is called a
Mal’tsev subalgebra if $Y$ is closed under the operation $M$, that is
$M(Y^{3})=Y$.

######  Theorem 3.

Let $X$ be a pseudocompact Mal’tsev space and $Y$ be an infinite discrete
Mal’tsev subalgebra of $X$. Then the closure of $Y$ in the Stone–Čech
compactification $\beta X$ of $X$ is a metrizable compact set and $Y$ is not
$C^{\ast}$-embedded in $X$.

###### Proof.

Let $M$ be a continuous Mal’tsev operation on $X$. Then $M$ extends to a
continuous Mal’tsev operation $\widehat{M}\colon(\beta X)^{3}\to\beta X$ on
the Stone–Čech compactification $\beta X$ of $X$ [17]. Let $K$ be the closure
of $Y$ in $\beta X$. Then $K$ is a Mal’tsev subalgebra of $\beta X$ and $K$ is
a Mal’tsev space with the dense discrete subspace $Y$. Compact Mal’tsev spaces
are Dugundji compact spaces [20, 17]. Dugundji compacta are dyadic compacta,
and every dyadic compact space with a dense set of isolated points is
metrizable (see [7] or [1, Corollary 12]). Hence, $K$ is a metrizable compact
space. If $Y$ is $C^{\ast}$-embedded in $X$, then $K$ is homeomorphic to
$\beta Y$ and $\beta Y$ is not metrizable, which is a contradiction. ∎

Every countable subset of a Mal’tsev space $X$ is contained in a countable
Mal’tsev subalgebra of $X$. Therefore, Theorem 3 implies the following fact.

######  Proposition 6.

An infinite pseudocompact Mal’tsev space contains either a countable non-
closed subset or a countable subset that fails to be $C^{\ast}$-embedded.

Combining Proposition 6 and Theorem 2 we deduce the following assertion that
improves upon Corollary 1.

######  Corollary 2.

An infinite pseudocompact Korovin orbit is not homeomorphic to any
pseudocompact Mal’tsev space.

We do not know whether every infinite pseudocompact Mal’tsev space contains a
countable non-closed subset (see Problem 1 below). In the following
proposition we establish that pseudocompact Mal’tsev spaces have a weaker
property.

######  Proposition 7.

An infinite pseudocompact Mal’tsev space contains a non-closed subset of
cardinality less than or equal to $\omega_{1}$.

###### Proof.

Let $X$ be an infinite pseudocompact Mal’tsev space. If $X$ is countable, then
it is compact and metrizable, so the conclusion of the proposition is evident.
We assume therefore that $|X|\geq\omega_{1}$. Let $Y$ be a Mal’tsev subalgebra
of $X$ of cardinality $\omega_{1}$. Then $Y$ is not discrete — otherwise
Theorem 3 implies that the closure of $Y$ in the Stone–Čech compactification
$\beta X$ of $X$ is a metrizable compact space with an uncountable discrete
subspace. Consequently, $Y$ is not discrete and the set $Y\setminus\\{y\\}$ is
not closed for some non-isolated point $y$ in $Y$. ∎

##  4 Questions

In connection with Propositions 5–7 it is natural to pose the following
problem.

######  Problem 1.

Does there exist an infinite pseudocompact Mal’tsev space in which every
countable subset is closed and discrete?

We mention the following properties of the pseudocompact space $X$ constructed
in [14] in relation with Problem 1 and Theorem 3: (1) $X$ is a pseudocompact
space of cardinality continuum , and each subset of $X$ whose cardinality is
less than the continuum is closed and discrete; (2) $\beta X$ is homeomorphic
to a Tychonoff cube of uncountable weight and is a Dugundji compactum; (3) the
closure of every countable subset of $X$ in $\beta X$ is a metrizable compact
space; (4) $X$ is a hereditarily metalindelöf space [13]. Proposition 7
implies that, assuming the negation of the Continuum Hypothesis, the space $X$
cannot be a Mal’tsev space. If we assume the Continuum Hypothesis, then it is
not clear whether the space $X$ is Mal’tsev or not.

Considering Corollaries 1 and 2, the following problem arises.

######  Problem 2.

Can an infinite Korovin orbit $G_{f}\subset X^{G}$ be homeomorphic to a
topological group or a Mal’tsev space? What if $X$ is a (countable) discrete
space?

A space $X$ is called weakly pc-Grothendieck if the closure of every
pseudocompact subspace $C_{p}(X)$ is compact [2]. A pseudocompact weakly pc-
Grothendieck space is a Korovin space.

The only instances of pseudocompact semitopological groups that are not
topological groups are Korovin orbits. Also, Shakhmatov’s example in [18] is a
pseudocompact space that is not weakly pc-Grothendieck because the closed
pseudocompact subspace $Y=\\{f\in C_{p}(X):f(X)\subset[-1,1]\\}$ of $C_{p}(X)$
is not compact.

Let us call a pseudocompact space $Z$ Shakhmatov space if any countable subset
of $Z$ is discrete, closed, and $C^{\ast}$-embedded. The space in [18] and
pseudocompact Korovin orbits are Shakhmatov spaces.

######  Problem 3.

Let ${\mathcal{P}}$ be one of the classes listed below.

  1. (${\mathcal{P}}_{1}$)

Pseudocompact spaces with countable extend.

  2. (${\mathcal{P}}_{2}$)

Pseudocompact spaces that have a dense subspace with countable extend.

  3. (${\mathcal{P}}_{3}$)

Pseudocompact spaces that have a dense Lindelöf subspace.

Find out which of the following statements are true:

  1. (1)

There is a Korovin orbit from ${\mathcal{P}}$.

  2. (2)

There is a Shakhmatov space from ${\mathcal{P}}$.

  3. (3)

There is a not weakly pc-Grothendieck space from ${\mathcal{P}}$.

  4. (4)

There is a semitopological group from ${\mathcal{P}}$ which is not a
topological group.

It is easy to see that $(1)\rightarrow(2)\rightarrow(3)$ and
$(1)\rightarrow(4)\rightarrow(3)$.

######  Problem 4.

Let ${\mathcal{P}}$ be one of the classes listed below.

  1. (${\mathcal{P}}_{4}$)

Pseudocompact spaces that have a dense subspace with countable tightness.

  2. (${\mathcal{P}}_{5}$)

Pseudocompact spaces with a dense subspace which is $k$-space.

Find out which of the following statements are true:

  1. (1)

There is a space from ${\mathcal{P}}$ that fails to be weakly pc-Grothendieck.

  2. (2)

There is a semitopological group from ${\mathcal{P}}$ which is not a
topological group.

Note that in Problem 4, (2) implies (1). We also mention that Shakhmatov’s
space in [18] does not help to answer (1) of Problem 4, while Korovin orbits
do not help answering (2) of the problem. These conclusions follow from the
fact that both Shakhmatov spaces of countable tightness and Shakhmatov
$k$-spaces are discrete.

## References

  * [1] A. V. Arkhangel’skij, Compactness, in: General topology II. Compactness. Homologies of general spaces. Transl. from the Russian by J. M. Lysko, Berlin: Springer-Verlag, 1989, p. 1.
  * [2] A. Arhangel’skii, On a theorem of Grothendieck in $C_{p}$-theory, Topology and its Applications 80 (1) (1997) 21–41, memory of P.S. Alexandroff.  doi:10.1016/S0166-8641(96)00167-8.
URL https://www.sciencedirect.com/science/article/pii/S0166864196001678

  * [3] A. V. Arhangel’skii, M. Hušek, Extensions of topological and semitopological groups and the product operation, Commentationes Mathematicae Universitatis Carolinae 42 (1) (2001) 173–186.
  * [4] A. Arhangel’skii, M. Tkachenko, Topological Groups and Related Structures, Atlantis Press, 2008.  doi:10.2991/978-94-91216-35-0.
URL https://doi.org/10.2991/978-94-91216-35-0

  * [5] B. Batíková, Completion of quasi-topological groups, Topology and its Applications 156 (12) (2009) 2123–2128. http://dx.doi.org/10.1016/j.topol.2009.03.035
  * [6] W.W. Comfort, K. A. Ross,  Pseudocompactness and uniform continuity in topological groups,  Pacific J. Math. 16 (1966) 483–496.
  * [7] B. A. Efimov, Dyadic bicompacta, Trudy Moskovskogo Matematicheskogo Obshchestva 14 (1965) 211–247.
  * [8] R. Ellis, Locally compact transformation groups, Duke Math. J. 24 (2) (1957) 119–125.
  * [9] L. Gillman, M. Jerison, Rings of Continuous Functions, Springer New York, 1960.  doi:10.1007/978-1-4615-7819-2.
URL https://doi.org/10.1007/978-1-4615-7819-2

  * [10] A. Grothendieck, Criteres de compacite dans les espaces fonctionnels generaux, Amer. J. Math. 74 (1952) 168.
  * [11] C. Hernández, M. Tkachenko, Three examples of pseudocompact quasitopological groups, Topol. Appl. 153 (18) (2006) 3615–3620.
  * [12] A. V. Korovin, Continuous actions of pseudocompact groups and axioms of topological group, Comment. Math. Univ. Carolinae 33 (2) (1992) 335–343.
URL http://eudml.org/doc/247414

  * [13] O. I. Pavlov, Reznichenko’s example is metalindelof, Vestnik Samarskogo Universiteta. Estestvenno-Nauchnaya Seriya 74 (8) (2009) 61–66.
  * [14] E. Reznichenko, A pseudocompact space in which only sets of complete cardinality are not closed and not discrete, Moscow Univ. Math. Bull. 6 (1989) 69–70.
  * [15] E. Reznichenko, Extension of functions defined on products of pseudocompact spaces and continuity of the inverse in pseudocompact groups, Topol. Appl. 59 (3) (1994) 233–244.  doi:10.1016/0166-8641(94)90021-3.
URL https://www.sciencedirect.com/science/article/pii/0166864194900213

  * [16] E. Reznichenko, Extension of mappings from the product of pseudocompact spaces, Topol. Appl. 322 (2022) 108329.  doi:10.1016/j.topol.2022.108329.
  * [17] E. Reznichenko, V. Uspenskij, Pseudocompact Mal’tsev spaces, Topol. Appl. 86 (1) (1998) 83–104, topological Groups.  doi:10.1016/S0166-8641(97)00124-7.
URL https://www.sciencedirect.com/science/article/pii/S0166864197001247

  * [18] D. Shakhmatov, A pseudocompact Tychonoff space all countable subsets of which are closed and $C^{*}$-embedded, Topol. Appl. 22 (2) (1986) 139–144.  doi:10.1016/0166-8641(86)90004-0.
URL https://www.sciencedirect.com/science/article/pii/0166864186900040

  * [19] Z. Tang, S. Lin, W.-F. Xuan, On some classes of quasitopological groups, Topol. Appl. 295 (2021) 107671.
  * [20] V. Uspenskij, Topological groups and Dugundji compacta, Mat. Sb. 180 (8) (1989) 1092–1118.