# Privacy-preserving machine learning for healthcare: open challenges and
future perspectives

Alejandro Guerra-Manzanares, L. Julian Lechuga Lopez∗,
Michail Maniatakos and Farah E. Shamout
Department of Computer Engineering, New York University Abu Dhabi
{ag9454, ljl5178, mm6446<EMAIL_ADDRESS>Equal contributions.

###### Abstract

Machine Learning (ML) has recently shown tremendous success in modeling
various healthcare prediction tasks, ranging from disease diagnosis and
prognosis to patient treatment. Due to the sensitive nature of medical data,
privacy must be considered along the entire ML pipeline, from model training
to inference. In this paper, we conduct a review of recent literature
concerning Privacy-Preserving Machine Learning (PPML) for healthcare. We
primarily focus on privacy-preserving training and inference-as-a-service, and
perform a comprehensive review of existing trends, identify challenges, and
discuss opportunities for future research directions. The aim of this review
is to guide the development of private and efficient ML models in healthcare,
with the prospects of translating research efforts into real-world settings.

##  1 Introduction

Machine Learning (ML) and Deep Learning (DL) have shown great promise in many
domains, leveraging the use of large datasets. Some notable contributions
include _AlphaFold_ (Jumper et al., 2021) for the prediction of protein
structures and _Transformers_ (Vaswani et al., 2017) for natural language
processing. Healthcare is one of the domains in which ML is expected to
provide substantial improvements in the delivery of patient care worldwide
(WHO, 2021). Given the rapid growth in the number of models over the last
couple of years (Ravì et al., 2016; Miotto et al., 2018; Kaul et al., 2022;
Javaid et al., 2022), healthcare applications deserve special consideration
considering the sensitive nature of the data that is required to train the
models and the safety-critical nature of medical decision-making.

In this regard, real-world implementation of such models is still hampered by
ethical and legal constraints. Legal frameworks have been developed and
enforced to guarantee the transparency and privacy of ML-based healthcare
solutions, such as the _Health Insurance Portability and Accountability Act
(HIPAA)_ in the United States (Gostin et al., 2009) and the _General Data
Protection Regulation (GDPR)_ in Europe (Voigt & Von dem Bussche, 2017).
Therefore, there is a crucial need for Privacy-Preserving Machine Learning
(PPML) in healthcare to enable the implementation of trustworthy systems in
the future. The main goal of this review is to provide a comprehensive
overview of state-of-the-art PPML in healthcare and encourage the development
of new methodologies that tackle specific challenges relevant to the nature of
the domain.

Motivation. There exist several related literature reviews that focus on a
specific subset of PPML for healthcare. Several highlight recent advancements
in federated learning (Xu et al., 2021; Ali et al., 2022; Joshi et al., 2022;
Nguyen et al., 2022), cryptographic techniques (Zalonis et al., 2022), or
security aspects of ML models, such as adversarial attacks (Liu et al., 2021).
Existing review articles cover a wide range of applications related to health
and input data modalities, ranging from IoT sensors to medical images (Qayyum
et al., 2020). Compared to existing work, our review has three main
contributions with the intent of bridging between research pertaining to ML
for healthcare and cybersecurity. First, we distinguish between PPML for
training and inference, i.e., _ML-as-a-service_. Second, we focus on state-of-
the-art (SOTA) literature published in the last three years, considering the
high proliferation of ML in healthcare and recent methodological advancements
in ML and DL (e.g., network architectures, model pre-training, etc.). Third,
we consider studies that develop or apply methodologies using two popular
modalities based on publicly available datasets and state-of-the-art in ML for
healthcare, namely medical images and data extracted from Electronic Health
Records (EHR) (Kaul et al., 2022).Despite the use of other input modalities in
medical applications, such as video (Ouyang et al., 2020) or text (Srivastava
et al., 2019), our review exclusively focuses on medical images and EHR as
they are the most prevalent input modalities in diagnostic and prognostic
settings (Shehab et al., 2022). Lastly, although we acknowledge the importance
of security for ML models, it is out of the scope of this paper since we
primarily focus on privacy.

To this end, we review papers that meet the following inclusion criteria:

  1. 1.

We include recently published work i.e., publication year $\geq 2020$.

  2. 2.

We include articles that focus on the application or development of PPML
either for model training and/or inference, including but not restricted to
homomorphic encryption, differential privacy, federated learning, and multi-
party secure computation.

  3. 3.

We include articles that consider clinical tasks involving medical images
and/or EHR data.

In Section 2, we provide background knowledge about concepts and terminology
concerning PPML. In Section 3, we provide an overview of the state-of-the-art
pertaining to PPML for training (Section 3.1) and for inference (Section 3.2).
Later in Section 4, we discuss open challenges and derive future directions.
Finally, we provide concluding remarks in Section 5.

##  2 Privacy-preserving machine learning: background & terminology

###  2.1 Federated Learning

Since medical data is highly sensitive, data sharing is difficult, and subject
to ethical restrictions and legal constraints if at all possible. Federated
learning (FL) (McMahan et al., 2017) aims to overcome the challenges of data
sharing by enabling collaborative training, which does not require that the
involved parties share their training data. Therefore, the data remains
private to each local node within the FL network, such that only the model
updates are shared and integrated in a centralized model.

Federated averaging (McMahan et al., 2017) is the most common form of FL. In
this setting, a centralized server is connected to $N$ entities, which have
their own training data. The central server orchestrates the collaborative
training process as follows: (1) the initial model is distributed amongst all
entities, (2) each entity performs a training iteration on their local model
using their own training data, typically one epoch, and shares its resulting
model parameters with the central server, (3) the server averages the model
parameters shared by all entities and distributes the resulting (averaged)
model amongst all entities, and (4) steps (2) and (3) are repeated
sequentially until a performance threshold or a specific number of training
iterations is achieved. FL has proven to be very efficient in training models
with strong performance, while avoiding the need for data sharing (McMahan et
al., 2017). However, FL might be vulnerable to privacy issues such as
reconstruction attacks (Liu et al., 2022), thus requiring that it is combined
with other privacy-preserving methods to ensure robust privacy guarantees
(Nguyen et al., 2022).

###  2.2 Differential Privacy

Differential Privacy (DP) has its origins in statistical analysis of
databases. Its main aim is to address the paradox of learning nothing about
specific individuals, while learning useful information about the general
population (Dwork et al., 2014). In the FL context, it is usually incorporated
in the form of additive noise to model updates, either artificially or using a
differentiable private optimizer, prior to transferring the updates from the
entities to the central server (Abadi et al., 2016). The amount of artificial
noise added is directly proportional to the degree of privacy desired (i.e.,
privacy budget) (Zhang et al., 2021c). DP can successfully make privacy
attacks fail, such as reconstruction attacks, as the added noise hinders the
inference of actual knowledge about the training data by the attacker.
However, adding too much noise (i.e., high privacy budget) can hamper learning
and negatively impact the model accuracy (Chilukoti et al., 2022).

###  2.3 Homomorphic Encryption

In mathematics, the term _homomorphic_ refers to the transformation of a given
set into another while preserving the relation between the elements in both
sets. Thus, Homomorphic Encryption (HE) refers to the conversion of plaintext
into ciphertext while preserving the structure of the data. Consequently,
specific operations applied to the ciphertext will provide the same results as
if they were applied to the plaintext but without compromising the encryption
(Acar et al., 2018). That is, the plaintext data is never accessed nor
decrypted as the operations are directly applied to the encrypted data. The
result of the transformations on the ciphertext can only be decrypted back to
plaintext by the encryption key owner.

Despite the benefit of provable privacy guarantees, the range of operations
available in HE is restricted to addition and multiplication i.e., _fully_
homomorphic encryption. This limits the set and number of transformations
applicable to the data and requires the use of approximations for more complex
operations (e.g., HE-ReLU is the polynomial approximation of the ReLU function
(Yue et al., 2021b)). This also significantly increases the computational time
needed to process encrypted text compared to plaintext by several orders of
magnitude (Popescu et al., 2021).

###  2.4 Secure Multi-party Computation

Secure Multi-Party Computation (SMPC) (Goldreich, 1998) provides a framework
in which two or more parties jointly compute a public function with their data
while keeping the inputs private and hidden from other parties using
cryptographic protocols. Most protocols used for SMPC with more than two
parties are based on Secret Sharing (SS). In SS, a portion of the secret input
is shared among a number of other parties. Most ML methods use Shamir’s SS and
additive SS (Singh & Shukla, 2021b). Although these methods are considered
information-theoretic secure cryptosystems, recent studies show that leakage
of global data properties can occur in some scenarios (Zhang et al., 2021a).
While both FL and SMPC rely on collaborative training via knowledge sharing
and keep the end-point data private, their implementation differs
significantly. SMPC involves cryptography and can be used for training and
inference, whereas FL does not involve cryptography nor provides strong
privacy guarantees, and is only used for model training.

##  3 Overview of state-of-the-art

Following the inclusion criteria described in Section 1, we summarize existing
work on PPML for healthcare based on whether the work focuses on model
training (Table 1) or model inference (Table 2). For each study (row) we
describe several attributes. _Use case_ provides a succinct summary of the
objective of the study. _Model_ reports the ML or DL architecture that was
employed to model the task. _Medical datasets_ summarizes the datasets that
were used for model training and evaluation. Additionally, we use the * symbol
to indicate the use of a private dataset. _ML task_ describes the nature of
the prediction task (e.g., binary or multi-class classification). _Input
modality_ reports the nature of the model’s input data, which could either be
_I_ for medical images or _E_ for EHR data. In the _Validation_ column, we
report whether the trained model was internally and/or externally evaluated,
with indicating the use of internal validation i.e., test set from the same
distribution of the training data, and indicating the assessment of the
generalization of the model on an external test dataset. Lastly, _Metrics_
lists the evaluation metrics used to describe the performance of the proposed
model.

###  3.1 Privacy-preserving training for healthcare

Table 1: Summary of PPML in healthcare for model training. We summarize
studies that focus on developing PPML in the context of model training. We
group them based on the methodology considered, i.e. federated learning,
homomorphic encryption, and differential privacy.

Reference | Use case | Model |  Medical dataset/s | ML task |  Input modality | Validation | Metrics
---|---|---|---|---|---|---|---
FEDERATED LEARNING
Dou et al. (2021) |  COVID-19 Computed Tomography (CT) analysis | RetinaNet |  Multi-institution lung CT data* |  Object detection | I |  |  mAP, Specifity, Recall, AUROC
Field et al. (2022) |  Cardiovascular admission after lung cancer treatment | Logistic regression |  Multi-institution lung CT data* | Risk prediction | I+E |  | AUROC, C-index
Lee & Shin (2020) |  FL benchmarking and reliability in healthcare |  Neural Network, LSTM, CNN |  MIMIC-III, PhysioNet ECG |  Mortality prediction, Multi-class classification | I+E |  |  AUROC, AUPRC, F1-score
Yang et al. (2022) |  FL benchmarking and monetary cost in healthcare |  Transformer, EfficientNet-B0, ResNet-NC-SE |  eICU, ISIC19, HAM10000, PhysioNet ECG |  Mortality prediction, Length of stay, Discharge time, Acuity prediction | I+E |  |  AUROC, AUPRC
Sadilek et al. (2021) |  FL benchmarking vs. centralized learning in healthcare |  Logistic regression, Neural Network, Generalized linear model |  UCI Heart failure, MIMIC-III, Malignancy in SARS- CoV-2 infection | Risk prediction | E |  | AUROC
Loftus et al. (2022) |  COVID-19 detection | DenseNet |  Multi-institution COVID-19 X-ray* |  Binary classification | I |  |  AUROC, AUPRC
Wolff et al. (2022) |  Coronary artery calcification (CAC) forecast |  Random Forest | CAC risk factors* | Risk prediction | E |  |  Recall, Specificity
Wang & Zhou (2022) |  Cancer inference via gene expression |  Gradient Boosting Decision Tree | iDASH 2020 |  Multi-class classification | E |  |  Accuracy, AUC, Recall, Precision, F1-score
Islam et al. (2022a) |  Diabetic kidney risk prediction |  Logistic regression, MLP |  CERNER Health Facts | Risk prediction | E |  | F1-score
Deist et al. (2020) |  Lung cancer post- treatment 2-year survival |  Logistic regression |  Multi-institution lung cancer EHR* | Mortality prediction | E |  |  RMSE, Accuracy, AUROC
Park et al. (2021) |  COVID-19 detection |  Transformer with DenseNet, TransUNet and RetinaNet |  Multi-institution COVID-19 X-ray (public and private datasets) |  Multi-task: classification, segmentation, object detection | I |  |  AUC, mAP, Dice coefficient
Yan et al. (2023) |  Multiple medical prediction tasks |  Self-supervised vision transformer |  COVID-19 X-ray, Kaggle Diabetic Retinopathy, Dermatology ISIC |  Binary/multi-class classification, Object detection | I |  |  Accuracy, F1-score
HOMOMORPHIC ENCRYPTION
Boulila et al. (2022) |  COVID-19 detection | MobileNet-V2 | COVID-19 X-ray |  Multi-class classification | I |  |  Accuracy, Recall, Precision, F1-score
Ma et al. (2020) |  Heart and thyroid disease classification | XGBoost |  UCI Heart Disease, Kaggle Hypothyroid |  Binary classification | E |  | Accuracy
Paul et al. (2021) |  Intensive Care Unit patient outcome | LSTM | MIMIC-III |  Binary classification | E |  |  Recall, AUROC, Precision
Chen et al. (2022) |  Dermatology diagnostics | SVM | UCI Dermatology |  Multi-class classification | E |  | Accuracy
Baruch et al. (2022) |  COVID-19 detection |  AlexNet, SqueezeNet |  COVID-19 X-ray, COVID-19 CT |  Multi-class classification | I |  |  Accuracy, F1-score
DIFFERENTIAL PRIVACY
Zhang et al. (2021b) |  Thoracic pathology detection | DenseNet-121 | CheXpert |  Multi-class classification | I+E |  |  AUROC, Accuracy
Chilukoti et al. (2022) |  COVID-19 detection | EfficientNet-B2 | COVID-19 X-ray |  Binary classification | I |  | Accuracy
Suriyakumar et al. (2021) |  Multiple medical prediction tasks |  CNN, DenseNet-121, Logistic regression, GRU-D |  MNIST NIH Chest X-ray, MIMIC-III |  Binary, Multi-class classification | I+E |  | AUROC

Table 1 Continued: Continued summary of PPML in healthcare for model training.
We summarize here studies that use a combination of federated learning and
other privacy-preserving techniques, blockchain, Secure Multi-Party
Computation (SMPC), image encryption, and image modification.

Reference | Use case | Model |  Medical dataset/s | ML task |  Input modality | Validation | Metrics
---|---|---|---|---|---|---|---
FEDERATED LEARNING + DIFFERENTIAL PRIVACY
Islam et al. (2022b) |  Cardiomyopathy risk prediction |  Random Forest, Naive Bayes |  iDASH 2021, Breast Cancer TCGA |  Risk prediction | E |  | AUROC
Kerkouche et al. (2021) |  In-hospital mortality prediction | CNN |  Premier Healthcare Database* |  Mortality prediction | E |  |  AUROC, Overhead
Dayan et al. (2021) |  COVID-19 patient triage |  ResNet-34 DeepCrossNet |  Multi-institution chest x-ray and EHR* |  Risk prediction | I+E |  |  AUROC, Recall, Specificity
BLOCKCHAIN
Zerka et al. (2020) |  Distributed training | ResNet-18 | NSCLC-Radiomics |  Binary classification | I |  | AUROC
Warnat-Herresthal et al. (2020) |  Disease classification | Neural Network | Blood transcriptomes* |  Binary classification | E |  |  Accuracy
FEDERATED LEARNING+HOMOMORPHIC ENCRYPTION
Wibawa et al. (2022) |  COVID-19 detection | CNN | COVID-19 X-ray |  Binary classification | I |  |  Accuracy, Recall Precision, F1 score, Execution time
FEDERATED LEARNING+HOMOMORPHIC ENCRYPTION+SMPC
Zhang et al. (2022) |  Skin cancer classification | CNN | HAM10000 |  Multi-class classification | I |  |  Accuracy, Overhead
SMPC
Hong et al. (2020) |  Tumor detection |  Logistic regression | iDASH 2019 |  Binary classification | E |  |  Accuracy, Overhead
IMAGE ENCRYPTION
Huang et al. (2022) |  Brain tumor, COVID-19 |  DenseNet-121, XceptionNet |  MRI Brain Tumor, COVID-19 X-ray |  Multi-class classification | I |  | F1-score
IMAGE MODIFICATION
Montenegro et al. (2021) |  Glaucoma recognition |  VGAN-based CNN |  Warsaw-BioBase Disease-Iris v2.1 |  Binary classification | I |  |  F1-score, Accuracy

As observed in Table 1, the most commonly used privacy-preserving approach for
model training is FL, either independently or in combination with DP. DP is
added to increase the privacy of the FL training updates i.e., adding noise to
the shared weights, thus making the system more robust to privacy threats,
such as reconstruction attacks by an external actor intercepting the
communication channel or an _honest-but-curious_ central server (Nguyen et
al., 2022).

The second most commonly investigated approach for private training is HE,
which leverages encryption schemes to provide privacy with provable
mathematical guarantees. However, as described in the previous section,
training ML models on encrypted data significantly increases the computational
complexity and the processing overhead by several orders of magnitude (Wibawa
et al., 2022; Zhang et al., 2022). It also adds noise to the training process
due to the approximations of activation functions, especially in large models.

The third most common approach is standalone DP, which is less computationally
demanding and provides strong privacy guarantees. However, the increase in
privacy guarantees is negatively correlated with model accuracy, as it is
associated with an increase in the quantity of noise applied. Therefore, the
trade-off between privacy (i.e., privacy budget) and model accuracy is a
relevant factor to take into account for the inclusion of DP in any ML
solution. There are other PPML approaches for model training that have been
evaluated in related work, including the addition of a blockchain ledger to
avoid the centralization of training (i.e., fully distributed learning), image
modification to increase data privacy in the context of model explainability,
and SMPC as an alternative encryption scheme to HE.

Most of the reviewed studies use a single source of input data i.e., image or
EHR and only one medical dataset. Although some studies train their models on
several datasets, including popular computer vision benchmarks, the vast
majority restrict their evaluation to one input modality from the same
dataset.

This limits the generalization of the results and neglects the potential
improvement in predictive performance that could result from combining
different data sources in multi-modal learning settings (Ramachandram &
Taylor, 2017). Furthermore, most studies perform internal validation, such
that the test sets are from the same distribution as the training dataset.
This is generally a common challenge in healthcare applications considering
distribution shifts across different hospitals, for example due to differences
in patient demographics. Finally, most existing work focuses on convolutional
neural networks to handle computer vision tasks. However, validation schemes
and metrics reported are not consistent, making the comparison among them very
difficult. Due to these reasons and the lack of medical benchmark datasets, a
fair comparison of the approaches is difficult, and therefore we do not assess
performance metrics results in this review and defer it to future work.

###  3.2 Privacy-preserving inference for healthcare

We now focus on the literature employing PPML methods for inference, as
summarized in Table 2. We frame PPML for inference as providing private
_machine-learning-as-a-service_ (MLaaS) or _inference-as-a-service_ (IaaS)
(Lins et al., 2021). In this scenario, a model with strong performance is
controlled by a single party (i.e., model owner), and other external parties
(i.e., clients) would like the model to perform inference on their own data.
The external parties can share data samples with the model owner and their
predictions are sent back. Due to legal and/or ethical constraints related to
privacy, clients cannot disclose their data with the model owner, thus
requiring the use of PPML to maintain the privacy of the data they wish to
share.

Table 2: Summary of PPML in healthcare for model inference. We summarize
studies that focus on developing PPML in the context of model inference. We
group them based on the methodology considered, i.e. homomorphic encryption,
combination of federated learning and Secure Multi-Party Computation (SMPC),
differential privacy and SMPC, federated learning with blockchain and SMPC,
and finally federated learning with differential privacy and homomorphic
encryption.

Reference | Use case | Model |  Medical dataset/s | ML task |  Input modality | Validation | Metrics
---|---|---|---|---|---|---|---
HOMOMORPHIC ENCRYPTION
Yue et al. (2021a) |  Breast and cervical cancer classification |  Convolutional LSTM |  Cervigram Image, BreaKHis |  Binary, Multi-class classification | I |  | AUROC
T’Jonck et al. (2022) |  Breast cancer classification |  Neural Network, SVM |  UCI IRIS, UCI Breast Cancer |  Binary, Multi-class classification | E |  |  Accuracy, Privacy budget, Overhead
Sarkar et al. (2022) |  Cancer inference via gene expression |  SVM, Logistic regression, Neural Network | iDASH 2020 |  Multi-class classification | E |  | Accuracy, AUROC
Vizitiu et al. (2020) |  Coronary angiography view classification | CNN |  X-ray coronary angiography* |  Binary, Multi-class classification | I |  | Accuracy
FEDERATED LEARNING + SMPC
Ziller et al. (2020), Kaissis et al. (2021)+ |  Paediatric chest X-ray classification | ResNet-18 | Chest X-ray* |  Multi-class classification | I |  | AUROC, Latency
DIFFERENTIAL PRIVACY + SMPC
Singh & Shukla (2021a) |  Pneumonia detection |  CNN, VGG-16 |  Kaggle X-ray Pneumonia |  Binary classification | I |  | Accuracy
Jarin & Eshete (2021) |  Accuracy-privacy trade-off analysis | Neural Network |  Kaggle IDC, MIMIC-III |  Binary, Multi-class classification | I |  |  Accuracy, Recall Precision, Privacy
FEDERATED LEARNING + BLOCKCHAIN + SMPC
Kasyap & Tripathy (2021) |  Multiple medical image datasets classification | CNN |  MedMNIST (CXR, Breast, Hand, ChestCT, Abdomen, HeadCT) |  Multi-class classification | I |  | Accuracy
FEDERATED LEARNING + DIFFERENTIAL PRIVACY + HOMOMORPHIC ENCRYPTION
Gopalakrishnan et al. (2021) |  Multiple medical image datasets classification | CNN |  MedMNIST (Pneumonia Breast, Retina, Blood) |  Multi-class classification | I |  |  Accuracy, Execution time, Bandwidth
\+ Kaissis et al. (2021) is an extension of Ziller et al. (2020).

Compared to the number of studies addressing PPML for training, a relatively
fewer number have explored PPML for inference. Most studies within the theme
of PPML for inference, focus on the deployment of the trained model as a
service and its use by third parties. The most common approach for delivering
PPML IaaS is HE, which ensures with provable mathematical guarantees that
neither the model owner nor any intermediate party are able to inspect the
original data nor the detection result i.e., both are encrypted and can only
be decrypted by the data owner. Another common approach is SMPC, which also
leverages encryption schemes, being used in combination with other privacy-
preserving collaborative approaches such as FL, DP and blockchain.

Similar to PPML for training, most studies here use a single source of input
data (i.e., images in most cases), neglecting many other diverse medical
modalities of varying characteristics. The lack of use of benchmark medical
datasets and inconsistent validation schemes and metrics hinders the
generalization of the proposed approaches.

###  3.3 Open challenges

There is no _one-size-fits-all_ PPML approach for model training or inference
by design. We observe that previous work pick and choose PPML approaches based
on the intended clinical use case. Currently, there is no consensus on what
different “privacy models” look like in healthcare. Since the methodology
depends on the use case, we also observe a clear trade-off between privacy and
accuracy, based on the availability of computational resources. For instance,
standalone FL is computationally faster than HE, but it does not provide
strong privacy guarantees. On the other hand, HE and DP can provide strong
privacy guarantees but they add noise to the model both for private training
and private inference resulting in less accurate solutions. For HE, this is
especially critical for model training where successive layers of
approximations are needed to perform operations that are not supported, such
as _softmax_ , or that are computationally inefficient, such as max pooling.
In general, encryption-based options are provably secure but computationally
inefficient, since they increase the processing overhead of training using
cyphertext compared to plaintext data.

Additionally, the availability of computational resources is a decisive factor
in choosing a particular PPML methodology. For instance, in the HE scenario,
sending data over a communication channel does not require infrastructure for
model training but still requires handling the encryption/decryption process
appropriately. In the FL context, it requires that the entity has allocated
resources for model training.

The centralization of model training in FL poses an additional security
threat. Relying on a single central server entails a single point of failure
that is highly susceptible to security attacks such as Denial-of-Service.
Although blockchain has been proposed to achieve fully distributed training
and mitigate this threat, it increases the complexity of the information
technology infrastructure significantly, requiring dedicated resources for the
implementation of the distributed ledger and modeling framework.

Most existing work use a single dataset and do not conduct external
validation, thus arising concerns about the generalization of the results. We
observe that existing work focus on a limited set of medical datasets.
Additionally, some work only evaluate their solutions on computer vision
benchmark datasets (e.g., _MNIST_ or _CIFAR-10_) inferring that good
performance on these datasets will provide similar results on medical image
data (Festag & Spreckelsen, 2020; Onesimu & Karthikeyan, 2020). However, this
assumption is not empirically supported by work that uses both medical and
non-medical datasets (Suriyakumar et al., 2021; Zhang et al., 2021b;
Gopalakrishnan et al., 2021; Jarin & Eshete, 2021; Vizitiu et al., 2020) and
must, therefore, be avoided.

MLaaS for healthcare has not been explored thoroughly. As demonstrated by the
limited literature on this topic, we observe that the literature is highly
skewed towards PPML for training. Considering disparities in technical
capabilities and expertise, information technology resources, and availability
of data across medical institutions, the case in which an entity does not have
enough resources to perform model training independently is highly likely.
Thus, the usage of third-party models as inference systems that can run on
proprietary data is a prominent scenario that has not been thoroughly explored
and should be considered in future research. MLaaS can provide access to
models with strong performance, enabling full preservation of data privacy
using PPML methods. This makes it a more efficient solution for small-scale or
low-resource medical entities to access and leverage third-party knowledge.

##  4 Future research directions

###  4.1 Comprehensive evaluation on diverse medical datasets

For the sake of comparison and generalization of results, studies should
complement their internal dataset evaluation with additional extensive
evaluation on benchmark medical datasets. This is due to the fact that most of
the existing work use a single dataset and do not perform external validation.
The number of studies that use external datasets for validation is marginal.
Only 9 out of the 40 studies considered validated their results with an
independent test set. This hampers model generalization and hinders
performance comparison among approaches built for the same medical task. For
benchmarking, we suggest _MedMNIST_ (Yang et al., 2023), which contains
curated datasets for different medical tasks and modalities. Therefore,
similar to _MNIST_ or _CIFAR-10_ for computer vision models, this medical
dataset could be employed as a common benchmark for medical applications.

###  4.2 Multi-modal models

Current advances in ML for healthcare are moving towards multi-modal learning,
where several sources of information are combined to improve performance
(Ramachandram & Taylor, 2017; Soenksen et al., 2022). This approach not only
tends to provide better performance but also ensures a comprehensive
understanding of the different physiological variables involved in studying
and modeling the development of human biology and pathology. As observed in
Sections 3.1 and 3.2, most work is restricted to a single modality. To develop
robust and strong ML models, the use of different data sources to develop
multi-modal systems is paramount. Notwithstanding that, the use of more
clinical data entails more privacy concerns (e.g., individuals may be
identified using correlated data) and requires more training resources due to
increased model complexity. Therefore, additional privacy and computational
constraints must be considered in the design of these algorithms.

###  4.3 Machine learning as a service (MLaaS)

The deployment of PPML within MLaaS is a very promising opportunity to access
strong proprietary models by less resourceful institutions. Indeed, one of the
main objectives of ML in healthcare is to develop efficient and scalable
solutions that improve healthcare delivery. In addition to lack of resources,
the deployment of these systems in medical settings can also be highly
challenging (Kreuzberger et al., 2022; Wiesenfeld et al., 2022). The
development of MLaaS is significantly less investigated than PPML for model
training. Therefore, further research on this topic is required to provide
secure, private and efficient data sharing between third-party model providers
and client institutions. Reducing obstacles for clinical institutions to
access powerful inference systems could lead to a major improvement in
healthcare delivery across regions, bypassing physical barriers. It can also
lead to an increase in the confidence and widespread adoption of ML in
healthcare. It is important to note that the success of MLaaS is dependent on
improvements in model generalizability and fairness in external datasets.

###  4.4 Integration of SOTA and advances in deep learning

Future work should also investigate the integration of recent advances in DL
and ML models in healthcare, considering that most of the current PPML work
focuses on convolutional neural networks. For instance, the _Transformer_
architecture and its variants (Dosovitskiy et al., 2020), which are considered
the current SOTA for many computer vision or natural language processing
tasks, are only adopted by Park et al. (2021), Yang et al. (2022), and Yan et
al. (2023) in the current related literature. Adopting SOTA architectures can
take advantage of the latest advances in research, both in terms of optimizing
hardware and software, to maintain performance improvements in clinical
prediction tasks.

###  4.5 Global and local explainability

Transparency and model explainability are essential for trustworthy artificial
intelligence (OECD, 2023b). However, PPML methods, such as data encryption or
noise addition, hinder global model and local prediction explainability. The
collision between two key principles for trustworthy artificial intelligence,
secure and PPML (OECD, 2023a) and explainability, highlights an important
research problem that is currently under-investigated. Only Montenegro et al.
(2021) attempt to address this problem, which should encourage future work in
this research direction.

##  5 Conclusion

In this paper, we introduce and summarize recent literature concerning PPML
for model training and inference in the healthcare domain. We highlight
trends, challenges and promising future research directions. In conclusion, we
recognize the lack of consensus when it comes to defining the requirements of
privacy-preserving frameworks in healthcare. This requires collaboration
between machine learning scientists, healthcare practitioners, and privacy and
security experts. From the perspective of advancing ML approaches, we
encourage researchers to perform comprehensive evaluation of proposed
algorithms on diverse medical datasets to increase generalization, to
investigate the constraints of PPML in multi-modal learning settings, to
further consider the promise of MLaaS in healthcare as a catalyst for improved
healthcare delivery, and to adopt state-of-the-art advances in deep learning
architectures to enhance model performance. Our suggestions aim to address
research gaps and guide future research in PPML to facilitate the future
adoption of trustworthy and private ML for healthcare.

## References

  * Abadi et al. (2016) Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang.  Deep learning with differential privacy.  In _Proceedings of the 2016 ACM SIGSAC conference on computer and communications security_ , pp. 308–318, 2016.
  * Acar et al. (2018) Abbas Acar, Hidayet Aksu, A Selcuk Uluagac, and Mauro Conti.  A survey on homomorphic encryption schemes: Theory and implementation.  _ACM Computing Surveys (Csur)_ , 51(4):1–35, 2018\.
  * Ali et al. (2022) Mansoor Ali, Faisal Naeem, Muhammad Tariq, and Geroges Kaddoum.  Federated learning for privacy preservation in smart healthcare systems: A comprehensive survey.  _IEEE journal of biomedical and health informatics_ , 2022.
  * Baruch et al. (2022) Moran Baruch, Nir Drucker, Lev Greenberg, and Guy Moshkowich.  A methodology for training homomorphic encryption friendly neural networks.  In _Applied Cryptography and Network Security Workshops: ACNS 2022 Satellite Workshops, AIBlock, AIHWS, AIoTS, CIMSS, Cloud S &P, SCI, SecMT, SiMLA, Rome, Italy, June 20–23, 2022, Proceedings_, pp. 536–553. Springer, 2022.
  * Boulila et al. (2022) Wadii Boulila, Adel Ammar, Bilel Benjdira, and Anis Koubaa.  Securing the classification of covid-19 in chest x-ray images: a privacy-preserving deep learning approach.  In _2022 2nd International Conference of Smart Systems and Emerging Technologies (SMARTTECH)_ , pp. 220–225. IEEE, 2022.
  * Chen et al. (2022) Yange Chen, Qinyu Mao, Baocang Wang, Pu Duan, Benyu Zhang, and Zhiyong Hong.  Privacy-preserving multi-class support vector machine model on medical diagnosis.  _IEEE Journal of Biomedical and Health Informatics_ , 26(7):3342–3353, 2022.
  * Chilukoti et al. (2022) Vijay Srinivas Tida Sai Venkatesh Chilukoti, Sonya Hsu, and Xiali Hei.  Privacy-preserving deep learning model for covid-19 disease detection.  _arXiv preprint arXiv:2209.04445_ , 2022.
  * Dayan et al. (2021) Ittai Dayan, Holger R Roth, Aoxiao Zhong, Ahmed Harouni, Amilcare Gentili, Anas Z Abidin, Andrew Liu, Anthony Beardsworth Costa, Bradford J Wood, Chien-Sung Tsai, et al.  Federated learning for predicting clinical outcomes in patients with covid-19.  _Nature medicine_ , 27(10):1735–1743, 2021.
  * Deist et al. (2020) Timo M Deist, Frank JWM Dankers, Priyanka Ojha, M Scott Marshall, Tomas Janssen, Corinne Faivre-Finn, Carlotta Masciocchi, Vincenzo Valentini, Jiazhou Wang, Jiayan Chen, et al.  Distributed learning on 20 000+ lung cancer patients–the personal health train.  _Radiotherapy and Oncology_ , 144:189–200, 2020.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Dou et al. (2021) Qi Dou, Tiffany Y So, Meirui Jiang, Quande Liu, Varut Vardhanabhuti, Georgios Kaissis, Zeju Li, Weixin Si, Heather HC Lee, Kevin Yu, et al.  Federated deep learning for detecting covid-19 lung abnormalities in ct: a privacy-preserving multinational validation study.  _NPJ digital medicine_ , 4(1):60, 2021.
  * Dwork et al. (2014) Cynthia Dwork, Aaron Roth, et al.  The algorithmic foundations of differential privacy.  _Foundations and Trends® in Theoretical Computer Science_ , 9(3–4):211–407, 2014.
  * Festag & Spreckelsen (2020) Sven Festag and Cord Spreckelsen.  Privacy-preserving deep learning for the detection of protected health information in real-world data: Comparative evaluation.  _JMIR Formative Research_ , 4(5):e14064, 2020\.
  * Field et al. (2022) Matthew Field, David I Thwaites, Martin Carolan, Geoff P Delaney, Joerg Lehmann, Jonathan Sykes, Shalini Vinod, and Lois Holloway.  Infrastructure platform for privacy-preserving distributed machine learning development of computer-assisted theragnostics in cancer.  _Journal of Biomedical Informatics_ , 134:104181, 2022.
  * Goldreich (1998) Oded Goldreich.  Secure multi-party computation.  _Manuscript. Preliminary version_ , 78(110), 1998.
  * Gopalakrishnan et al. (2021) Aparna Gopalakrishnan, Narayan P Kulkarni, Chethan Raghavendra, Raghavendra Manjappa, Prasad B Honnavalli, and Sivaraman Eswaran.  Primed: Private federated training and encrypted inference on medical images in healthcare.  _Available at SSRN 4196696_ , 2021.
  * Gostin et al. (2009) Lawrence O Gostin, Laura A Levit, Sharyl J Nass, et al.  Beyond the hipaa privacy rule: enhancing privacy, improving health through research.  2009\.
  * Hong et al. (2020) Cheng Hong, Zhicong Huang, Wen-jie Lu, Hunter Qu, Li Ma, Morten Dahl, and Jason Mancuso.  Privacy-preserving collaborative machine learning on genomic data using tensorflow.  In _Proceedings of the ACM Turing Celebration Conference-China_ , pp. 39–44, 2020.
  * Huang et al. (2022) Qi-Xian Huang, Wai Leong Yap, Min-Yi Chiu, and Hung-Min Sun.  Privacy-preserving deep learning with learnable image encryption on medical images.  _IEEE Access_ , 10:66345–66355, 2022.
  * Islam et al. (2022a) Humayera Islam, Khuder Alaboud, Tanmoy Paul, Md Kamruz Zaman Rana, and Abu Mosa.  A privacy-preserved transfer learning concept to predict diabetic kidney disease at out-of-network siloed sites using an in-network federated model on real-world data.  In _AMIA Annual Symposium Proceedings_ , volume 2022, pp. 264. American Medical Informatics Association, 2022a.
  * Islam et al. (2022b) Tanzir Ul Islam, Reza Ghasemi, and Noman Mohammed.  Privacy-preserving federated learning model for healthcare data.  In _2022 IEEE 12th Annual Computing and Communication Workshop and Conference (CCWC)_ , pp. 0281–0287. IEEE, 2022b.
  * Jarin & Eshete (2021) Ismat Jarin and Birhanu Eshete.  Pricure: privacy-preserving collaborative inference in a multi-party setting.  In _Proceedings of the 2021 ACM Workshop on Security and Privacy Analytics_ , pp. 25–35, 2021.
  * Javaid et al. (2022) Mohd Javaid, Abid Haleem, Ravi Pratap Singh, Rajiv Suman, and Shanay Rab.  Significance of machine learning in healthcare: Features, pillars and applications.  _International Journal of Intelligent Networks_ , 3:58–73, 2022.
  * Joshi et al. (2022) Madhura Joshi, Ankit Pal, and Malaikannan Sankarasubbu.  Federated learning for healthcare domain-pipeline, applications and challenges.  _ACM Transactions on Computing for Healthcare_ , 3(4):1–36, 2022.
  * Jumper et al. (2021) John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, Anna Potapenko, et al.  Highly accurate protein structure prediction with alphafold.  _Nature_ , 596(7873):583–589, 2021.
  * Kaissis et al. (2021) Georgios Kaissis, Alexander Ziller, Jonathan Passerat-Palmbach, Théo Ryffel, Dmitrii Usynin, Andrew Trask, Ionésio Lima Jr, Jason Mancuso, Friederike Jungmann, Marc-Matthias Steinborn, et al.  End-to-end privacy preserving deep learning on multi-institutional medical imaging.  _Nature Machine Intelligence_ , 3(6):473–484, 2021.
  * Kasyap & Tripathy (2021) Harsh Kasyap and Somanath Tripathy.  Privacy-preserving decentralized learning framework for healthcare system.  _ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)_ , 17(2s):1–24, 2021.
  * Kaul et al. (2022) Deeksha Kaul, Harika Raju, and BK Tripathy.  Deep learning in healthcare.  _Deep Learning in Data Analytics: Recent Techniques, Practices and Applications_ , pp. 97–115, 2022.
  * Kerkouche et al. (2021) Raouf Kerkouche, Gergely Acs, Claude Castelluccia, and Pierre Genevès.  Privacy-preserving and bandwidth-efficient federated learning: An application to in-hospital mortality prediction.  In _Proceedings of the Conference on Health, Inference, and Learning_ , pp. 25–35, 2021.
  * Kreuzberger et al. (2022) Dominik Kreuzberger, Niklas Kühl, and Sebastian Hirschl.  Machine learning operations (mlops): Overview, definition, and architecture.  _arXiv preprint arXiv:2205.02302_ , 2022.
  * Lee & Shin (2020) Geun Hyeong Lee and Soo-Yong Shin.  Federated learning on clinical benchmark data: performance assessment.  _Journal of medical Internet research_ , 22(10):e20891, 2020.
  * Lins et al. (2021) Sebastian Lins, Konstantin D Pandl, Heiner Teigeler, Scott Thiebes, Calvin Bayer, and Ali Sunyaev.  Artificial intelligence as a service: Classification and research directions.  _Business & Information Systems Engineering_, 63:441–456, 2021.
  * Liu et al. (2021) Bo Liu, Ming Ding, Sina Shaham, Wenny Rahayu, Farhad Farokhi, and Zihuai Lin.  When machine learning meets privacy: A survey and outlook.  _ACM Computing Surveys (CSUR)_ , 54(2):1–36, 2021\.
  * Liu et al. (2022) Pengrui Liu, Xiangrui Xu, and Wei Wang.  Threats, attacks and defenses to federated learning: issues, taxonomy and perspectives.  _Cybersecurity_ , 5(1):1–19, 2022.
  * Loftus et al. (2022) Tyler J Loftus, Matthew M Ruppert, Benjamin Shickel, Tezcan Ozrazgat-Baslanti, Jeremy A Balch, Philip A Efron, Gilbert R Upchurch Jr, Parisa Rashidi, Christopher Tignanelli, Jiang Bian, et al.  Federated learning for preserving data privacy in collaborative healthcare research.  _Digital Health_ , 8:20552076221134455, 2022.
  * Ma et al. (2020) Zhuoran Ma, Jianfeng Ma, Yinbin Miao, Ximeng Liu, Kim-Kwang Raymond Choo, Ruikang Yang, and Xiangyu Wang.  Lightweight privacy-preserving medical diagnosis in edge computing.  _IEEE Transactions on Services Computing_ , 15(3):1606–1618, 2020.
  * McMahan et al. (2017) Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas.  Communication-efficient learning of deep networks from decentralized data.  In _Artificial intelligence and statistics_ , pp. 1273–1282. PMLR, 2017.
  * Miotto et al. (2018) Riccardo Miotto, Fei Wang, Shuang Wang, Xiaoqian Jiang, and Joel T Dudley.  Deep learning for healthcare: review, opportunities and challenges.  _Briefings in bioinformatics_ , 19(6):1236–1246, 2018.
  * Montenegro et al. (2021) Helena Montenegro, Wilson Silva, and Jaime S Cardoso.  Privacy-preserving generative adversarial network for case-based explainability in medical image analysis.  _IEEE Access_ , 9:148037–148047, 2021.
  * Nguyen et al. (2022) Dinh C Nguyen, Quoc-Viet Pham, Pubudu N Pathirana, Ming Ding, Aruna Seneviratne, Zihuai Lin, Octavia Dobre, and Won-Joo Hwang.  Federated learning for smart healthcare: A survey.  _ACM Computing Surveys (CSUR)_ , 55(3):1–37, 2022\.
  * OECD (2023a) OECD.  Robustness, security and safety (principle 1.4), 2023a.  URL https://oecd.ai/en/dashboards/ai-principles/P8.
  * OECD (2023b) OECD.  Transparency and explainability (principle 1.3), 2023b.  URL https://oecd.ai/en/dashboards/ai-principles/P7.
  * Onesimu & Karthikeyan (2020) J A Onesimu and J Karthikeyan.  An efficient privacy-preserving deep learning scheme for medical image analysis.  _Journal of Information Technology Management_ , 12(Special Issue: The Importance of Human Computer Interaction: Challenges, Methods and Applications.):50–67, 2020.
  * Ouyang et al. (2020) David Ouyang, Bryan He, Amirata Ghorbani, Neal Yuan, Joseph Ebinger, Curtis P Langlotz, Paul A Heidenreich, Robert A Harrington, David H Liang, Euan A Ashley, et al.  Video-based ai for beat-to-beat assessment of cardiac function.  _Nature_ , 580(7802):252–256, 2020.
  * Park et al. (2021) Sangjoon Park, Gwanghyun Kim, Jeongsol Kim, Boah Kim, and Jong Chul Ye.  Federated split vision transformer for covid-19 cxr diagnosis using task-agnostic training.  _arXiv preprint arXiv:2111.01338_ , 2021.
  * Paul et al. (2021) Jestine Paul, Meenatchi Sundaram Muthu Selva Annamalai, William Ming, Ahmad Al Badawi, Bharadwaj Veeravalli, and Khin Mi Mi Aung.  Privacy-preserving collective learning with homomorphic encryption.  _IEEE Access_ , 9:132084–132096, 2021.
  * Popescu et al. (2021) Andreea Bianca Popescu, Ioana Antonia Taca, Cosmin Ioan Nita, Anamaria Vizitiu, Robert Demeter, Constantin Suciu, and Lucian Mihai Itu.  Privacy preserving classification of eeg data using machine learning and homomorphic encryption.  _Applied Sciences_ , 11(16):7360, 2021.
  * Qayyum et al. (2020) Adnan Qayyum, Junaid Qadir, Muhammad Bilal, and Ala Al-Fuqaha.  Secure and robust machine learning for healthcare: A survey.  _IEEE Reviews in Biomedical Engineering_ , 14:156–180, 2020\.
  * Ramachandram & Taylor (2017) Dhanesh Ramachandram and Graham W Taylor.  Deep multimodal learning: A survey on recent advances and trends.  _IEEE signal processing magazine_ , 34(6):96–108, 2017.
  * Ravì et al. (2016) Daniele Ravì, Charence Wong, Fani Deligianni, Melissa Berthelot, Javier Andreu-Perez, Benny Lo, and Guang-Zhong Yang.  Deep learning for health informatics.  _IEEE journal of biomedical and health informatics_ , 21(1):4–21, 2016.
  * Sadilek et al. (2021) Adam Sadilek, Luyang Liu, Dung Nguyen, Methun Kamruzzaman, Stylianos Serghiou, Benjamin Rader, Alex Ingerman, Stefan Mellem, Peter Kairouz, Elaine O Nsoesie, et al.  Privacy-first health research with federated learning.  _NPJ digital medicine_ , 4(1):132, 2021.
  * Sarkar et al. (2022) Esha Sarkar, Eduardo Chielle, Gamze Gursoy, Leo Chen, Mark Gerstein, and Michail Maniatakos.  Scalable privacy-preserving cancer type prediction with homomorphic encryption.  _arXiv preprint arXiv:2204.05496_ , 2022.
  * Shehab et al. (2022) Mohammad Shehab, Laith Abualigah, Qusai Shambour, Muhannad A Abu-Hashem, Mohd Khaled Yousef Shambour, Ahmed Izzat Alsalibi, and Amir H Gandomi.  Machine learning in medical applications: A review of state-of-the-art methods.  _Computers in Biology and Medicine_ , 145:105458, 2022.
  * Singh & Shukla (2021a) Shreyansh Singh and KK Shukla.  Privacy-preserving machine learning for medical image classification.  _arXiv preprint arXiv:2108.12816_ , 2021a.
  * Singh & Shukla (2021b) Shreyansh Singh and KK Shukla.  Privacy-preserving machine learning for medical image classification.  _arXiv preprint arXiv:2108.12816_ , 2021b.
  * Soenksen et al. (2022) Luis R Soenksen, Yu Ma, Cynthia Zeng, Leonard Boussioux, Kimberly Villalobos Carballo, Liangyuan Na, Holly M Wiberg, Michael L Li, Ignacio Fuentes, and Dimitris Bertsimas.  Integrated multimodal artificial intelligence framework for healthcare applications.  _NPJ Digital Medicine_ , 5(1):149, 2022.
  * Srivastava et al. (2019) Saurabh Kumar Srivastava, Sandeep Kumar Singh, and Jasjit S Suri.  Effect of incremental feature enrichment on healthcare text classification system: A machine learning paradigm.  _Computer methods and programs in biomedicine_ , 172:35–51, 2019.
  * Suriyakumar et al. (2021) Vinith M Suriyakumar, Nicolas Papernot, Anna Goldenberg, and Marzyeh Ghassemi.  Chasing your long tails: Differentially private prediction in health care settings.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , pp. 723–734, 2021.
  * T’Jonck et al. (2022) Kristof T’Jonck, Chandrakanth R Kancharla, Bozheng Pang, Hans Hallez, and Jeroen Boydens.  Privacy preserving classification via machine learning model inference on homomorphic encrypted medical data.  In _2022 XXXI International Scientific Conference Electronics (ET)_ , pp. 1–6. IEEE, 2022.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Vizitiu et al. (2020) Anamaria Vizitiu, Cosmin Ioan Niţă, Andrei Puiu, Constantin Suciu, and Lucian Mihai Itu.  Towards privacy-preserving deep learning based medical imaging applications.  In _2019 IEEE international symposium on medical measurements and applications (MeMeA)_ , pp. 1–6. IEEE, 2020.
  * Voigt & Von dem Bussche (2017) Paul Voigt and Axel Von dem Bussche.  The eu general data protection regulation (gdpr).  _A Practical Guide, 1st Ed., Cham: Springer International Publishing_ , 10(3152676):10–5555, 2017.
  * Wang & Zhou (2022) Qingyong Wang and Yun Zhou.  Fedspl: federated self-paced learning for privacy-preserving disease diagnosis.  _Briefings in Bioinformatics_ , 23(1):bbab498, 2022.
  * Warnat-Herresthal et al. (2020) Stefanie Warnat-Herresthal, Hartmut Schultze, Krishnaprasad Lingadahalli Shastry, Sathyanarayanan Manamohan, Saikat Mukherjee, Vishesh Garg, Ravi Sarveswara, Kristian Händler, Peter Pickkers, N Ahmad Aziz, et al.  Swarm learning as a privacy-preserving machine learning approach for disease classification.  _BioRxiv_ , pp. 2020–06, 2020.
  * WHO (2021) WHO.  Who issues first global report on artificial intelligence (ai) in health and six guiding principles for its design and use, 2021.  URL https://www.who.int/news/item/28-06-2021-who-issues-first-global-report-on-ai-in-health-and-six-guiding-principles-for-its-design-and-use.
  * Wibawa et al. (2022) Febrianti Wibawa, Ferhat Ozgur Catak, Murat Kuzlu, Salih Sarp, and Umit Cali.  Homomorphic encryption and federated learning based privacy-preserving cnn training: Covid-19 detection use-case.  In _Proceedings of the 2022 European Interdisciplinary Cybersecurity Conference_ , pp. 85–90, 2022.
  * Wiesenfeld et al. (2022) Batia Mishan Wiesenfeld, Yin Aphinyanaphongs, and Oded Nov.  Ai model transferability in healthcare: a sociotechnical perspective.  _Nature Machine Intelligence_ , 4(10):807–809, 2022.
  * Wolff et al. (2022) Justus Wolff, Julian Matschinske, Dietrich Baumgart, Anne Pytlik, Andreas Keck, Arunakiry Natarajan, Claudio E von Schacky, Josch K Pauling, and Jan Baumbach.  Federated machine learning for a facilitated implementation of artificial intelligence in healthcare–a proof of concept study for the prediction of coronary artery calcification scores.  _Journal of Integrative Bioinformatics_ , 19(4), 2022.
  * Xu et al. (2021) Jie Xu, Benjamin S Glicksberg, Chang Su, Peter Walker, Jiang Bian, and Fei Wang.  Federated learning for healthcare informatics.  _Journal of Healthcare Informatics Research_ , 5:1–19, 2021\.
  * Yan et al. (2023) Rui Yan, Liangqiong Qu, Qingyue Wei, Shih-Cheng Huang, Liyue Shen, Daniel Rubin, Lei Xing, and Yuyin Zhou.  Label-efficient self-supervised federated learning for tackling data heterogeneity in medical imaging.  _IEEE Transactions on Medical Imaging_ , 2023.
  * Yang et al. (2023) Jiancheng Yang, Rui Shi, Donglai Wei, Zequan Liu, Lin Zhao, Bilian Ke, Hanspeter Pfister, and Bingbing Ni.  Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification.  _Scientific Data_ , 10(1):41, 2023.
  * Yang et al. (2022) Seongjun Yang, Hyeonji Hwang, Daeyoung Kim, Radhika Dua, Jong-Yeup Kim, Eunho Yang, and Edward Choi.  Towards the practical utility of federated learning in the medical domain.  _arXiv preprint arXiv:2207.03075_ , 2022.
  * Yue et al. (2021a) Zijie Yue, Shuai Ding, Lei Zhao, Youtao Zhang, Zehong Cao, Mohammad Tanveer, Alireza Jolfaei, and Xi Zheng.  Privacy-preserving time-series medical images analysis using a hybrid deep learning framework.  _ACM Transactions on Internet Technology (TOIT)_ , 21(3):1–21, 2021a.
  * Yue et al. (2021b) Zijie Yue, Shuai Ding, Lei Zhao, Youtao Zhang, Zehong Cao, Mohammad Tanveer, Alireza Jolfaei, and Xi Zheng.  Privacy-preserving time-series medical images analysis using a hybrid deep learning framework.  _ACM Transactions on Internet Technology (TOIT)_ , 21(3):1–21, 2021b.
  * Zalonis et al. (2022) Jasmin Zalonis, Frederik Armknecht, Björn Grohmann, and Manuel Koch.  Report: State of the art solutions for privacy preserving machine learning in the medical context.  _arXiv preprint arXiv:2201.11406_ , 2022.
  * Zerka et al. (2020) Fadila Zerka, Visara Urovi, Akshayaa Vaidyanathan, Samir Barakat, Ralph TH Leijenaar, Sean Walsh, Hanif Gabrani-Juma, Benjamin Miraglio, Henry C Woodruff, Michel Dumontier, et al.  Blockchain for privacy preserving and trustworthy distributed machine learning in multicentric medical imaging (c-distrim).  _Ieee Access_ , 8:183939–183951, 2020.
  * Zhang et al. (2022) Li Zhang, Jianbo Xu, Pandi Vijayakumar, Pradip Kumar Sharma, and Uttam Ghosh.  Homomorphic encryption-based privacy-preserving federated learning in iot-enabled healthcare system.  _IEEE Transactions on Network Science and Engineering_ , 2022.
  * Zhang et al. (2021a) Wanrong Zhang, Shruti Tople, and Olga Ohrimenko.  Leakage of dataset properties in multi-party machine learning.  In _USENIX Security Symposium_ , pp. 2687–2704, 2021a.
  * Zhang et al. (2021b) Xinyue Zhang, Jiahao Ding, Maoqiang Wu, Stephen TC Wong, Hien Van Nguyen, and Miao Pan.  Adaptive privacy preserving deep learning algorithms for medical data.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 1169–1178, 2021b.
  * Zhang et al. (2021c) Xinyue Zhang, Jiahao Ding, Maoqiang Wu, Stephen TC Wong, Hien Van Nguyen, and Miao Pan.  Adaptive privacy preserving deep learning algorithms for medical data.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pp. 1169–1178, 2021c.
  * Ziller et al. (2020) Alexander Ziller, Jonathan Passerat-Palmbach, Théo Ryffel, Dmitrii Usynin, Andrew Trask, Ionésio Da Lima Costa Junior, Jason Mancuso, Marcus Makowski, Daniel Rueckert, Rickmer Braren, et al.  Privacy-preserving medical image analysis.  _arXiv preprint arXiv:2012.06354_ , 2020.


* Gopalakrishnan et al. (2019) Karthik Gopalakrishnan, Behnam Hedayatnia, Qinlang Chen, Anna Gottardi, Sanjeev Kwatra, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tür. 2019\.  Topical-Chat: Towards Knowledge-Grounded Open-Domain Conversations. In _Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019_ , Gernot Kubin and Zdravko Kacic (Eds.). ISCA, 1891–1895.  https://doi.org/10.21437/INTERSPEECH.2019-3079
  * Gu et al. (2021) Jing Gu, Mostafa Mirshekari, Zhou Yu, and Aaron Sisto. 2021\.  ChainCQG: Flow-Aware Conversational Question Generation. In _EACL_. 2061–2070.
  * Gupta et al. (2024) Aman Gupta, Anup Shirgaonkar, Angels de Luis Balaguer, Bruno Silva, Daniel Holstein, Dawei Li, Jennifer Marsman, Leonardo O Nunes, Mahsa Rouzbahman, Morris Sharp, et al. 2024\.  RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Case Study on Agriculture.  _arXiv preprint arXiv:2401.08406_ (2024).
  * Gur et al. (2018) Izzeddin Gur, Dilek Hakkani-Tür, Gökhan Tür, and Pararth Shah. 2018.  User Modeling for Task Oriented Dialogues. In _2018 IEEE Spoken Language Technology Workshop, SLT 2018, Athens, Greece, December 18-21, 2018_. IEEE, 900–906.  https://doi.org/10.1109/SLT.2018.8639652
  * He et al. (2018) He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018\.  Decoupling Strategy and Generation in Negotiation Dialogues. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018_ , Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (Eds.). Association for Computational Linguistics, 2333–2343.  https://doi.org/10.18653/V1/D18-1256
  * Henderson et al. (2014) Matthew Henderson, Blaise Thomson, and Jason D. Williams. 2014\.  The Second Dialog State Tracking Challenge. In _Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)_ , Kallirroi Georgila, Matthew Stone, Helen Hastie, and Ani Nenkova (Eds.). Association for Computational Linguistics, Philadelphia, PA, U.S.A., 263–272.  https://doi.org/10.3115/v1/W14-4337
  * Hwang et al. (2022a) Seonjeong Hwang, Yunsu Kim, and Gary Geunbae Lee. 2022a.  Multi-Type Conversational Question-Answer Generation with Closed-ended and Unanswerable Questions. In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing, AACL/IJCNLP 2022 - Volume 2: Short Papers, Online only, November 20-23, 2022_. Association for Computational Linguistics, 169–177.  https://aclanthology.org/2022.aacl-short.22
  * Hwang et al. (2022b) Seonjeong Hwang, Yunsu Kim, and Gary Geunbae Lee. 2022b.  Multi-Type Conversational Question-Answer Generation with Closed-ended and Unanswerable Questions. In _AACL_. 169–177.
  * Hwang and Lee (2022) Seonjeong Hwang and Gary Geunbae Lee. 2022.  Conversational QA Dataset Generation with Answer Revision. In _Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022_. International Committee on Computational Linguistics, 1636–1644.  https://aclanthology.org/2022.coling-1.140
  * Jaderberg et al. (2017) Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, and Koray Kavukcuoglu. 2017.  Reinforcement Learning with Unsupervised Auxiliary Tasks. In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.  https://openreview.net/forum?id=SJ6yPD5xg
  * Jandaghi et al. (2023) Pegah Jandaghi, XiangHai Sheng, Xinyi Bai, Jay Pujara, and Hakim Sidahmed. 2023.  Faithful Persona-based Conversational Dataset Generation with Large Language Models.  _CoRR_ abs/2312.10007 (2023).  https://doi.org/10.48550/ARXIV.2312.10007 arXiv:2312.10007
  * Joko et al. (2024) Hideaki Joko, Shubham Chatterjee, Andrew Ramsay, Arjen P. de Vries, Jeff Dalton, Faegheh Hasibi, Krisztian Balog, and Arjen P. de Vries. 2024.  Doing Personal LAPS: LLM-Augmented Dialogue Construction for Personalized Multi-Session Conversational Search. In _Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’24)_.
  * Joko and Hasibi (2022) Hideaki Joko and Faegheh Hasibi. 2022.  Personal Entity, Concept, and Named Entity Linking in Conversations. In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_ _(CIKM ’22)_. 4099–4103.  https://doi.org/10.1145/3511808.3557667
  * Joko et al. (2021) Hideaki Joko, Faegheh Hasibi, Krisztian Balog, and Arjen P de Vries. 2021.  Conversational Entity Linking: Problem Definition and Datasets. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’21)_. 2390–2397.  https://doi.org/10.1145/3404835.3463258
  * Kaiser et al. (2021) Magdalena Kaiser, Rishiraj Saha Roy, and Gerhard Weikum. 2021\.  Reinforcement Learning from Reformulations in Conversational Question Answering over Knowledge Graphs. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’21)_. 459–469.  https://doi.org/10.1145/3404835.3462859
  * Kim et al. (2022b) Gangwoo Kim, Sungdong Kim, Kang Min Yoo, and Jaewoo Kang. 2022b.  Generating Information-Seeking Conversations from Unlabeled Documents. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, 2362–2378.  https://doi.org/10.18653/V1/2022.EMNLP-MAIN.151
  * Kim et al. (2023) Hyunwoo Kim, Jack Hessel, Liwei Jiang, Peter West, Ximing Lu, Youngjae Yu, Pei Zhou, Ronan Le Bras, Malihe Alikhani, Gunhee Kim, Maarten Sap, and Yejin Choi. 2023\.  SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_ , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 12930–12949.  https://aclanthology.org/2023.emnlp-main.799
  * Kim et al. (2022a) Minju Kim, Chaehyeong Kim, Yong Ho Song, Seung-won Hwang, and Jinyoung Yeo. 2022a.  BotsTalk: Machine-sourced Framework for Automatic Curation of Large-scale Multi-skill Dialogue Datasets. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 5149–5170.  https://doi.org/10.18653/v1/2022.emnlp-main.344
  * Kim et al. (2021) Sungdong Kim, Minsuk Chang, and Sang-Woo Lee. 2021\.  NeuralWOZ: Learning to Collect Task-Oriented Dialogue via Model-Based Simulation. In _ACL_. 3704–3717.
  * Kreyssig et al. (2018) Florian Kreyssig, Iñigo Casanueva, Paweł Budzianowski, and Milica Gašić. 2018\.  Neural User Simulation for Corpus-based Policy Optimisation of Spoken Dialogue Systems. In _Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue_ , Kazunori Komatani, Diane Litman, Kai Yu, Alex Papangelis, Lawrence Cavedon, and Mikio Nakano (Eds.). Association for Computational Linguistics, Melbourne, Australia, 60–69.  https://doi.org/10.18653/v1/W18-5007
  * Lee et al. (2022) Young-Jun Lee, Chae-Gyun Lim, Yunsu Choi, Ji-Hui Lm, and Ho-Jin Choi. 2022.  PERSONACHATGEN: Generating Personalized Dialogues using GPT-3. In _Proceedings of the 1st Workshop on Customized Chat Grounding Persona and Knowledge_. Association for Computational Linguistics, Gyeongju, Republic of Korea, 29–48.  https://aclanthology.org/2022.ccgpk-1.4
  * Leszczynski et al. (2023) Megan Leszczynski, Shu Zhang, Ravi Ganti, Krisztian Balog, Filip Radlinski, Fernando Pereira, and Arun Tejasvi Chaganty. 2023.  Talk the Walk: Synthetic Data Generation for Conversational Music Recommendation.  _arXiv preprint arXiv:2301.11489_ (2023).
  * Leszczynski et al. (2022) Megan Eileen Leszczynski, Ravi Ganti, Shu Zhang, Krisztian Balog, Filip Radlinski, Fernando Pereira, and Arun Tejasvi Chaganty. 2022.  Conversational Music Retrieval with Synthetic Data. In _Second Workshop on Interactive Learning for Natural Language Processing at NeurIPS_ , Vol. 2022.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (Eds.). 7871–7880.  https://doi.org/10.18653/v1/2020.acl-main.703
  * Li et al. (2016a) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a.  A Diversity-Promoting Objective Function for Neural Conversation Models. In _NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016_. The Association for Computational Linguistics, 110–119.  https://doi.org/10.18653/V1/N16-1014
  * Li et al. (2023) Siheng Li, Yichun Yin, Cheng Yang, Wangjie Jiang, Yiwei Li, Zesen Cheng, Lifeng Shang, Xin Jiang, Qun Liu, and Yujiu Yang. 2023\.  NewsDialogues: Towards Proactive News Grounded Conversation. In _Findings of the Association for Computational Linguistics: ACL 2023_ , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 3634–3649.  https://doi.org/10.18653/v1/2023.findings-acl.224
  * Li et al. (2016b) Xiujun Li, Zachary C. Lipton, Bhuwan Dhingra, Lihong Li, Jianfeng Gao, and Yun-Nung Chen. 2016b.  A User Simulator for Task-Completion Dialogues.  _CoRR_ abs/1612.05688 (2016).  arXiv:1612.05688 http://arxiv.org/abs/1612.05688
  * Li et al. (2017) Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017\.  DailyDialog: A Manually Labelled Multi-turn Dialogue Dataset. In _Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , Greg Kondrak and Taro Watanabe (Eds.). Asian Federation of Natural Language Processing, Taipei, Taiwan, 986–995.  https://aclanthology.org/I17-1099
  * Li and Zhang (2024) Yiming Li and Zhao Zhang. 2024.  The First Place Solution of WSDM Cup 2024: Leveraging Large Language Models for Conversational Multi-Doc QA.  _CoRR_ abs/2402.18385 (2024).  https://doi.org/10.48550/ARXIV.2402.18385 arXiv:2402.18385
  * Li et al. (2022) Zekun Li, Wenhu Chen, Shiyang Li, Hong Wang, Jing Qian, and Xifeng Yan. 2022\.  Controllable Dialogue Simulation with In-context Learning. In _EMNLP_. 4330–4347.
  * Lin et al. (2021) Hsien-chin Lin, Nurul Lubis, Songbo Hu, Carel van Niekerk, Christian Geishauser, Michael Heck, Shutong Feng, and Milica Gasic. 2021\.  Domain-independent User Simulation with Transformers for Task-oriented Dialogue Systems. In _SIGDIAL_. 445–456.
  * Lin et al. (2022) Yen Ting Lin, Alexandros Papangelis, Seokhwan Kim, and Dilek Hakkani-Tur. 2022.  Knowledge-Grounded Conversational Data Augmentation with Generative Conversational Networks. In _Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGDIAL 2022, Edinburgh, UK, 07-09 September 2022_ , Oliver Lemon, Dilek Hakkani-Tür, Junyi Jessy Li, Arash Ashrafzadeh, Daniel Hernández García, Malihe Alikhani, David Vandyke, and Ondrej Dusek (Eds.). Association for Computational Linguistics, 26–38.  https://aclanthology.org/2022.sigdial-1.3
  * Liu et al. (2023b) Anqi Liu, Bo Wang, Yue Tan, Dongming Zhao, Kun Huang, Ruifang He, and Yuexian Hou. 2023b.  MTGP: Multi-turn Target-oriented Dialogue Guided by Generative Global Path with Flexible Turns. In _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_ , Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 259–271.  https://doi.org/10.18653/V1/2023.FINDINGS-ACL.18
  * Liu et al. (2021b) Siyang Liu, Chujie Zheng, Orianna Demasi, Sahand Sabour, Yu Li, Zhou Yu, Yong Jiang, and Minlie Huang. 2021b.  Towards Emotional Support Dialog Systems. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, 3469–3483.  https://doi.org/10.18653/V1/2021.ACL-LONG.269
  * Liu et al. (2023a) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023a.  G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_. Association for Computational Linguistics, 2511–2522.  https://doi.org/10.18653/V1/2023.EMNLP-MAIN.153
  * Liu et al. (2021a) Zeming Liu, Haifeng Wang, Zheng-Yu Niu, Hua Wu, and Wanxiang Che. 2021a.  DuRecDial 2.0: A Bilingual Parallel Corpus for Conversational Recommendation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 4335–4347.  https://doi.org/10.18653/v1/2021.emnlp-main.356
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023.  Self-Refine: Iterative Refinement with Self-Feedback. In _Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023_.  http://papers.nips.cc/paper_files/paper/2023/hash/91edff07232fb1b55a505a9e9f6c0ff3-Abstract-Conference.html
  * Mallen et al. (2023) Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023.  When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Toronto, Canada, 9802–9822.  https://doi.org/10.18653/v1/2023.acl-long.546
  * Mehri and Eskénazi (2020) Shikib Mehri and Maxine Eskénazi. 2020.  USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (Eds.). Association for Computational Linguistics, 681–707.  https://doi.org/10.18653/V1/2020.ACL-MAIN.64
  * Meng et al. (2003) Helen M. Meng, Carmen Wai, and Roberto Pieraccini. 2003\.  The use of belief networks for mixed-initiative dialog modeling.  _IEEE Trans. Speech Audio Process._ 11, 6 (2003), 757–773.  https://doi.org/10.1109/TSA.2003.814380
  * Mohapatra et al. (2021) Biswesh Mohapatra, Gaurav Pandey, Danish Contractor, and Sachindra Joshi. 2021.  Simulated Chats for Building Dialog Systems: Learning to Generate Conversations from Instructions. In _EMNLP_. 1190–1203.
  * Narayanan and McIlraith (2002) Srini Narayanan and Sheila A. McIlraith. 2002.  Simulation, verification and automated composition of web services. In _Proceedings of the Eleventh International World Wide Web Conference, WWW 2002, May 7-11, 2002, Honolulu, Hawaii, USA_ , David Lassner, David De Roure, and Arun Iyengar (Eds.). ACM, 77–88.  https://doi.org/10.1145/511446.511457
  * Ni et al. (2023a) Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, and Erik Cambria. 2023a.  Recent advances in deep learning based dialogue systems: a systematic survey.  _Artif. Intell. Rev._ 56, 4 (2023), 3055–3155.  https://doi.org/10.1007/s10462-022-10248-8
  * Ni et al. (2023b) Jinjie Ni, Tom Young, Vlad Pandelea, Fuzhao Xue, and Erik Cambria. 2023b.  Recent advances in deep learning based dialogue systems: a systematic survey.  _Artif. Intell. Rev._ , 3055–3155.
  * Olteanu et al. (2021) Alexandra Olteanu, Jean Garcia-Gathright, Maarten de Rijke, Michael D. Ekstrand, Adam Roegiest, Aldo Lipani, Alex Beutel, Alexandra Olteanu, Ana Lucic, Ana-Andreea Stoica, Anubrata Das, Asia Biega, Bart Voorn, Claudia Hauff, Damiano Spina, David Lewis, Douglas W. Oard, Emine Yilmaz, Faegheh Hasibi, Gabriella Kazai, Graham McDonald, Hinda Haned, Iadh Ounis, Ilse van der Linden, Jean Garcia-Gathright, Joris Baan, Kamuela N. Lau, Krisztian Balog, Maarten de Rijke, Mahmoud Sayed, Maria Panteli, Mark Sanderson, Matthew Lease, Michael D. Ekstrand, Preethi Lahoti, and Toshihiro Kamishima. 2021\.  FACTS-IR: fairness, accountability, confidentiality, transparency, and safety in information retrieval.  _SIGIR Forum_ 53, 2 (mar 2021), 20–43.  https://doi.org/10.1145/3458553.3458556
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022\.  Training language models to follow instructions with human feedback. In _Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022_ , Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.).  http://papers.nips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html
  * Ovadia et al. (2023a) Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. 2023a.  Fine-tuning or retrieval? comparing knowledge injection in llms.  _arXiv preprint arXiv:2312.05934_ (2023).
  * Ovadia et al. (2023b) Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. 2023b.  Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs.  _CoRR_ abs/2312.05934 (2023).  https://doi.org/10.48550/ARXIV.2312.05934 arXiv:2312.05934
  * Pan et al. (2019) Boyuan Pan, Hao Li, Ziyu Yao, Deng Cai, and Huan Sun. 2019.  Reinforced Dynamic Reasoning for Conversational Question Generation. In _ACL_.
  * Park et al. (2018) Yookoon Park, Jaemin Cho, and Gunhee Kim. 2018.  A Hierarchical Latent Structure for Variational Conversation Modeling. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers)_ , Marilyn A. Walker, Heng Ji, and Amanda Stent (Eds.). Association for Computational Linguistics, 1792–1801.  https://doi.org/10.18653/V1/N18-1162
  * Penha et al. (2019) Gustavo Penha, Alexandru Balan, and Claudia Hauff. 2019\.  Introducing MANtIS: a novel Multi-Domain Information Seeking Dialogues Dataset.  _CoRR_ abs/1912.04639 (2019).  arXiv:1912.04639 http://arxiv.org/abs/1912.04639
  * Qi et al. (2023) Peng Qi, Nina Du, Christopher Manning, and Jing Huang. 2023\.  PragmatiCQA: A Dataset for Pragmatic Question Answering in Conversations. In _Findings of the Association for Computational Linguistics: ACL 2023_ , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 6175–6191.  https://doi.org/10.18653/v1/2023.findings-acl.385
  * Qu et al. (2018a) C. Qu, L. Yang, W. B. Croft, J. Trippas, Y. Zhang, and M. Qiu. 2018a.  Analyzing and Characterizing User Intent in Information-seeking Conversations.. In _SIGIR ’18_.
  * Qu et al. (2018b) Chen Qu, Liu Yang, W. Bruce Croft, Johanne R. Trippas, Yongfeng Zhang, and Minghui Qiu. 2018b.  Analyzing and Characterizing User Intent in Information-seeking Conversations. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018_, Kevyn Collins-Thompson, Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz (Eds.). ACM, 989–992.  https://doi.org/10.1145/3209978.3210124
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _J. Mach. Learn. Res._ 21 (2020), 140:1–140:67.  http://jmlr.org/papers/v21/20-074.html
  * Rashkin et al. (2019) Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019.  Towards Empathetic Open-domain Conversation Models: A New Benchmark and Dataset. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , Anna Korhonen, David Traum, and Lluís Màrquez (Eds.). Association for Computational Linguistics, Florence, Italy, 5370–5381.  https://doi.org/10.18653/v1/P19-1534
  * Rastogi et al. (2020) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020.  Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue Dataset. In _AAAI_. 8689–8696.
  * Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019\.  CoQA: A Conversational Question Answering Challenge.  _Trans. Assoc. Comput. Linguistics_ 7 (2019), 249–266.  https://doi.org/10.1162/tacl_a_00266
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_. Association for Computational Linguistics, 3980–3990.  https://doi.org/10.18653/V1/D19-1410
  * Ruggeri et al. (2023) Federico Ruggeri, Mohsen Mesgar, and Iryna Gurevych. 2023\.  A Dataset of Argumentative Dialogues on Scientific Papers. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, Toronto, Canada, 7684–7699.  https://doi.org/10.18653/v1/2023.acl-long.425
  * Samarinas et al. (2024) Chris Samarinas, Pracha Promthaw, Atharva Nijasure, Hansi Zeng, Julian Killingback, and Hamed Zamani. 2024\.  Simulating Task-Oriented Dialogues with State Transition Graphs and Large Language Models.  _arXiv preprint arXiv:2404.14772_ (2024).
  * Schatzmann et al. (2007) Jost Schatzmann, Blaise Thomson, and Steve J. Young. 2007\.  Statistical User Simulation with a Hidden Agenda. In _Proceedings of the 8th SIGdial Workshop on Discourse and Dialogue, SIGdial 2007, Antwerp, Belgium, September 1-2, 2007_ , Harry Bunt, Simon Keizer, and Tim Paek (Eds.). Association for Computational Linguistics, 273–282.  https://aclanthology.org/2007.sigdial-1.48/
  * Schuster et al. (2019) Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019\.  Cross-lingual Transfer Learning for Multilingual Task Oriented Dialog. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 3795–3805.  https://doi.org/10.18653/V1/N19-1380
  * Sevegnani et al. (2021) Karin Sevegnani, David M. Howcroft, Ioannis Konstas, and Verena Rieser. 2021.  OTTers: One-turn Topic Transitions for Open-Domain Dialogue. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Eds.). Association for Computational Linguistics, Online, 2492–2504.  https://doi.org/10.18653/v1/2021.acl-long.194
  * Shah et al. (2018) Pararth Shah, Dilek Hakkani-Tür, Gökhan Tür, Abhinav Rastogi, Ankur Bapna, Neha Nayak, and Larry P. Heck. 2018\.  Building a Conversational Agent Overnight with Dialogue Self-Play.  _Arxiv_.
  * Smith et al. (2022) Eric Michael Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, and Jason Weston. 2022\.  Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents. In _Proceedings of the 4th Workshop on NLP for Conversational AI, ConvAI@ACL 2022, Dublin, Ireland, May 27, 2022_. Association for Computational Linguistics, 77–97.  https://doi.org/10.18653/V1/2022.NLP4CONVAI-1.8
  * Soudani et al. (2023) Heydar Soudani, Evangelos Kanoulas, and Faegheh Hasibi. 2023\.  Data Augmentation for Conversational AI. In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21-25, 2023_ , Ingo Frommholz, Frank Hopfgartner, Mark Lee, Michael Oakes, Mounia Lalmas, Min Zhang, and Rodrygo L. T. Santos (Eds.). ACM, 5220–5223.  https://doi.org/10.1145/3583780.3615291
  * Soudani et al. (2024) Heydar Soudani, Evangelos Kanoulas, and Faegheh Hasibi. 2024\.  Fine Tuning vs. Retrieval Augmented Generation for Less Popular Knowledge.  _CoRR_ abs/2403.01432 (2024).  https://doi.org/10.48550/ARXIV.2403.01432 arXiv:2403.01432
  * Tang et al. (2019) Jianheng Tang, Tiancheng Zhao, Chenyan Xiong, Xiaodan Liang, Eric Xing, and Zhiting Hu. 2019\.  Target-Guided Open-Domain Conversation. In _ACL_. 5624–5634.
  * Terragni et al. (2023) Silvia Terragni, Modestas Filipavicius, Nghia Khau, Bruna Guedes, André Manso, and Roland Mathis. 2023\.  In-Context Learning User Simulators for Task-Oriented Dialog Systems.  _arXiv_.
  * Tseng et al. (2021) Bo-Hsiang Tseng, Yinpei Dai, Florian Kreyssig, and Bill Byrne. 2021.  Transferable Dialogue Systems and User Simulators. In _ACL, IJCNLP_. 152–166.
  * Wan et al. (2022) Dazhen Wan, Zheng Zhang, Qi Zhu, Lizi Liao, and Minlie Huang. 2022.  A Unified Dialogue User Simulator for Few-shot Data Augmentation. In _EMNLP_. 3788–3799.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wang et al. (2023) Jian Wang, Yi Cheng, Dongding Lin, Chak Tou Leong, and Wenjie Li. 2023.  Target-oriented Proactive Dialogue Systems with Personalization: Problem Formulation and Dataset Curation. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_ , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 1132–1143.  https://aclanthology.org/2023.emnlp-main.72
  * Wang et al. (2017) Weiyan Wang, Yuxiang Wu, Yu Zhang, Zhongqi Lu, Kaixiang Mo, and Qiang Yang. 2017\.  Integrating User and Agent Models: A Deep Task-Oriented Dialogue System.  _CoRR_ abs/1711.03697 (2017).  arXiv:1711.03697 http://arxiv.org/abs/1711.03697
  * Weizenbaum (1966) Joseph Weizenbaum. 1966\.  ELIZA—a computer program for the study of natural language communication between man and machine.  _Commun. ACM_ 9, 1 (1966), 36–45.
  * Wen et al. (2017a) Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina Maria Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve J. Young. 2017a.  A Network-based End-to-End Trainable Task-oriented Dialogue System. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 1: Long Papers_ , Mirella Lapata, Phil Blunsom, and Alexander Koller (Eds.). Association for Computational Linguistics, 438–449.  https://doi.org/10.18653/V1/E17-1042
  * Wen et al. (2017b) Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina Maria Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve J. Young. 2017b.  A Network-based End-to-End Trainable Task-oriented Dialogue System. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 1: Long Papers_ , Mirella Lapata, Phil Blunsom, and Alexander Koller (Eds.). Association for Computational Linguistics, 438–449.  https://doi.org/10.18653/V1/E17-1042
  * West et al. (2022) Peter West, Chandra Bhagavatula, Jack Hessel, Jena D. Hwang, Liwei Jiang, Ronan Le Bras, Ximing Lu, Sean Welleck, and Yejin Choi. 2022.  Symbolic Knowledge Distillation: from General Language Models to Commonsense Models. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022_ , Marine Carpuat, Marie-Catherine de Marneffe, and Iván Vladimir Meza Ruíz (Eds.). Association for Computational Linguistics, 4602–4625.  https://doi.org/10.18653/V1/2022.NAACL-MAIN.341
  * Wu et al. (2022a) Qingyang Wu, Song Feng, Derek Chen, Sachindra Joshi, Luis Lastras, and Zhou Yu. 2022a.  DG2: Data Augmentation Through Document Grounded Dialogue Generation. In _SIGDIAL_. 204–216.
  * Wu et al. (2022b) Zeqiu Wu, Ryu Parish, Hao Cheng, Sewon Min, Prithviraj Ammanabrolu, Mari Ostendorf, and Hannaneh Hajishirzi. 2022b.  INSCIT: Information-Seeking Conversations with Mixed-Initiative Interactions.  _CoRR_ abs/2207.00746 (2022).  https://doi.org/10.48550/ARXIV.2207.00746 arXiv:2207.00746
  * Xu et al. (2023a) Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2023a.  Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, Singapore, 6268–6278.  https://doi.org/10.18653/v1/2023.emnlp-main.385
  * Xu et al. (2023b) Jing Xu, Megan Ung, Mojtaba Komeili, Kushal Arora, Y-Lan Boureau, and Jason Weston. 2023b.  Learning New Skills after Deployment: Improving open-domain internet-driven dialogue with human feedback. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023_ , Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 13557–13572.  https://doi.org/10.18653/V1/2023.ACL-LONG.758
  * Yang et al. (2018) Liu Yang, Minghui Qiu, Chen Qu, Jiafeng Guo, Yongfeng Zhang, W. Bruce Croft, Jun Huang, and Haiqing Chen. 2018\.  Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018_, Kevyn Collins-Thompson, Qiaozhu Mei, Brian D. Davison, Yiqun Liu, and Emine Yilmaz (Eds.). ACM, 245–254.  https://doi.org/10.1145/3209978.3210011
  * Yang et al. (2020b) Shiquan Yang, Rui Zhang, and Sarah M. Erfani. 2020b.  GraphDialog: Integrating Graph Knowledge into End-to-End Task-Oriented Dialogue Systems. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 1878–1888.  https://doi.org/10.18653/V1/2020.EMNLP-MAIN.147
  * Yang et al. (2020a) Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020a.  G-DAug: Generative Data Augmentation for Commonsense Reasoning. In _EMNLP_. 1008–1025.
  * Yang et al. (2022) Zhitong Yang, Bo Wang, Jinfeng Zhou, Yue Tan, Dongming Zhao, Kun Huang, Ruifang He, and Yuexian Hou. 2022\.  TopKG: Target-oriented Dialog via Global Planning on Knowledge Graph. In _COLING_. 745–755.
  * Yi et al. (2024) Zihao Yi, Jiarui Ouyang, Yuwen Liu, Tianhao Liao, Zhe Xu, and Ying Shen. 2024\.  A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems.  _arXiv preprint arXiv:2402.18013_ (2024).
  * Yoo et al. (2020) Kang Min Yoo, Hanbit Lee, Franck Dernoncourt, Trung Bui, Walter Chang, and Sang-goo Lee. 2020\.  Variational Hierarchical Dialog Autoencoder for Dialog State Tracking Data Augmentation. In _EMNLP_. 3406–3425.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021\.  BARTScore: Evaluating Generated Text as Text Generation. In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (Eds.). 27263–27277.  https://proceedings.neurips.cc/paper/2021/hash/e4d2b6e6fdeca3e60e0f1a62fee3d9dd-Abstract.html
  * Zamani et al. (2022) Hamed Zamani, Johanne R. Trippas, Jeff Dalton, and Filip Radlinski. 2022.  Conversational Information Seeking.  _Arxiv_.
  * Zhang et al. (2023) Jiahao Zhang, Haiyang Zhang, Dongmei Zhang, Yong Liu, and Shen Huang. 2023.  Beam Retrieval: General End-to-End Retrieval for Multi-Hop Question Answering.  _CoRR_ abs/2308.08973 (2023).  https://doi.org/10.48550/ARXIV.2308.08973 arXiv:2308.08973
  * Zhang et al. (2018a) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018a.  Personalizing Dialogue Agents: I have a dog, do you have pets too?. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Melbourne, Australia, 2204–2213.  https://doi.org/10.18653/v1/P18-1205
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020.  BERTScore: Evaluating Text Generation with BERT. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.  https://openreview.net/forum?id=SkeHuCVFDr
  * Zhang et al. (2018b) Yizhe Zhang, Michel Galley, Jianfeng Gao, Zhe Gan, Xiujun Li, Chris Brockett, and Bill Dolan. 2018b.  Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_. 1815–1825.  https://proceedings.neurips.cc/paper/2018/hash/23ce1851341ec1fa9e0c259de10bf87c-Abstract.html
  * Zhao et al. (2019) Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. 2019\.  Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models. In _NAACL-HLT_. 1208–1218.
  * Zheng et al. (2023) Chujie Zheng, Sahand Sabour, Jiaxin Wen, Zheng Zhang, and Minlie Huang. 2023.  AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation. In _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_ , Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 1552–1568.  https://doi.org/10.18653/V1/2023.FINDINGS-ACL.99
  * Zhong et al. (2022) Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022.  Towards a Unified Multi-Dimensional Evaluator for Text Generation. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2023–2038.  https://doi.org/10.18653/v1/2022.emnlp-main.131
  * Zhu et al. (2018) Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018.  Texygen: A Benchmarking Platform for Text Generation Models. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018_. ACM, 1097–1100.  https://doi.org/10.1145/3209978.3210080