# Overcoming Catastrophic Forgetting by Exemplar Selection in
Task-oriented Dialogue System

Chen Chen1, Ruizhe Li2, Yuchen Hu1, Yuanyuan Chen1, Chengwei Qin1, Qiang
Zhang3
1Nanyang Technological University, Singapore 2University of Aberdeen, UK
3 Zhejiang University, China

###### Abstract

Intelligent task-oriented dialogue systems (ToDs) are expected to continuously
acquire new knowledge, also known as Continual Learning (CL), which is crucial
to fit ever-changing user needs. However, catastrophic forgetting dramatically
degrades the model performance in face of a long streamed curriculum. In this
paper, we aim to overcome the forgetting problem in ToDs and propose a method
(HESIT) with hyper-gradient-based exemplar strategy, which samples influential
exemplars for periodic retraining. Instead of unilaterally observing data or
models, HESIT adopts a profound exemplar selection strategy that considers the
general performance of the trained model when selecting exemplars for each
task domain. Specifically, HESIT analyzes the training data influence by
tracing their hyper-gradient in the optimization process. Furthermore, HESIT
avoids estimating Hessian to make it compatible for ToDs with a large pre-
trained model. Experimental results show that HESIT effectively alleviates
catastrophic forgetting by exemplar selection, and achieves state-of-the-art
performance on the largest CL benchmark of ToDs in terms of all metrics.

##  1 Introduction

Serving as a core technique of smart assistants, task-oriented dialogue
systems (ToDs) are expected to continuously acquire new knowledge through time
regarding user needs Madotto et al. (2020), e.g., adding fresh slot-value
pairs or handling dissimilar tasks. This ability is also known as Continual
Learning (CL) Mundt et al. (2020), which has recently attracted a surge of
interest in NLP community Biesialska et al. (2020), as well as other machine
learning techniques Qu et al. (2021).

In this setting, the central problem of CL is _catastrophic forgetting_
Kirkpatrick et al. (2017), as the data is streamed and the neural model
inevitably forgets previously learned knowledge when fitting new training data
in the sequential order. Such a phenomenon is particularly conspicuous in ToDs
tasks due to the obvious distributional shift between task scenarios, e.g.,
setting an alarm clock and booking a flight are completely irrelevant tasks.
As a result, it is challenging for a single model that is learned from a
unidirectional curriculum to simultaneously handle multi-domain tasks.

Figure 1: (a) Rehearsal-based CL in Task-oriented dialogue system. Exemplars
$E_{t}$ are sampled from $t$-domain training data for episodic rehearsal. (b)
Exemplar selection in terms of influence chain “ _Data_ (I) – _Model_ (II) –
_Performance_ (III)”. Our method penetrates into the performance perspective.

To mitigate the problem of forgetting, existing efforts De Lange et al. (2021)
follow the three lines: (i) adding regularization terms to consolidate learned
knowledge, (ii) developing dynamic architectures for the task-specific domain,
and (iii) applying historical data rehearsal. Previous works Madotto et al.
(2020) have demonstrated that regularization methods gradually lose
effectiveness if faced with a long curriculum, and we confirm this belief with
a relative experiment. Dynamic architecture methods (e.g. adapter tuning) are
usually considered as an unfair baseline Rusu et al. (2016); Maltoni and
Lomonaco (2019); Mi et al. (2020) due to multiple sets of parameters, and it
requires a further step during inference to determine the correct parameters
resulting in high latency for ToDs. Therefore, this work focuses on rehearsal
methods (Fig. 1 (a)) Rolnick et al. (2019) that store a small number of past
samples (a.k.a. exemplars) in episodic memory and replay them periodically,
which is simple but effective. Then we raise our research question: _which
utterances are suitable to be selected as rehearsal exemplars for ToDs?_

Although statistic methods (e.g., reservoir algorithm Chaudhry et al. (2019))
conduce to select better exemplars than random sampling, according to the
accredited influence chain “ _Data-Model-Performance_ ” Sun et al. (2022), an
exemplar selection strategy should further consider the general performance of
models during testing. In other words, we should select the exemplars that
positively influence the _performance_ of the trained network on unseen data,
as shown in Fig. 1 (b).

To explore the influence of training data from the performance perspective,
one pioneering work uses the Influence Functions (IFs) based on the derivation
chain rule Koh and Liang (2017) to quantify the contribution of an individual
training sample, by observing its impact on the test loss when removing it.
Though IFs have shown effectiveness in image classification tasks, this method
requires estimating the inverse Hessian matrix, leading to high computing
costs Guo et al. (2020) and unstable results for large pre-trained models that
are widely used in ToDs. Furthermore, IFs only measure the contribution around
the model parameters at the final training epoch, which fails to trace the
data contribution in the dynamic optimization process Chen et al. (2021).

In this paper, we propose a rehearsal-based CL method, named HESIT (Hyper-
gradient-based Exemplar Selection by Influence Tracing) to effectively
overcome the catastrophic forgetting in ToDs. Specifically, instead of only
focusing on the model parameters of the final epoch, HESIT traces and analyzes
the hyper-gradient of training examples in the complicated optimization
process. More importantly, its exemplar selection strategy roots in model
_performance_ on unseen data to measure the influence of these traced samples.
In this way, influential and representative exemplars can be determined and
stored, and then utilized to remind the model of learned knowledge. Moreover,
HESIT is Hessian-free to avoid the instability when estimating Hessian matrix
of a large pre-trained models, which has been widely used in ToDs.
Experimental results demonstrate that HESIT achieves state-of-the-art
performance on the largest CL benchmark of ToDs Madotto et al. (2020)
including 37 domain tasks in terms of all metrics. Furthermore, comparative
experiments show that the exemplar selection in HESIT surpasses all other
selection strategies from other perspectives.

The main contributions of this paper can be summarized as follows:

  * •

We propose HESIT – a rehearsal-based CL method with a novel exemplar selection
strategy that can effectively overcome catastrophic forgetting in ToDs.

  * •

HESIT dynamically traces the hyper-gradient of candidate data in the training
process and selects exemplars in terms of data influence to model performance.

  * •

Compared with other influence function-based methods, HESIT avoids estimating
the Hessian matrix without degrading effect, which is compatible for ToDs with
a large-scale pre-trained model.

  * •

Extensive experiments show that HESIT can achieve state-of-the-art performance
on the largest CL benchmark of ToDs (37 domain tasks).

##  2 Related Work

Continual learning. In the past few years, CL has achieved remarkable progress
in mitigating catastrophic forgetting McCloskey and Cohen (1989), which can be
classified into three categories: (1) _regularization-based_ methods
Kirkpatrick et al. (2017); Serra et al. (2018); Huang et al. (2021); Qin and
Joty (2022a); Qin et al. (2024), which focus on regularizing the parameters
corresponding to be inherited from the old tasks and penalizing the feature
drift. (2) _architecture-based_ methods Xu and Zhu (2018); Li et al. (2019);
Ermis et al. (2022); Qin et al. (2023), which develop dynamic parameter
isolation or expansion during CL training, where each task domain learns a
specific architecture. (3) _rehearsal-based_ methods Cui et al. (2019);
Chaudhry et al. (2019); Verwimp et al. (2021); Qin and Joty (2022b), which
utilize a small replay buffer to retain a fraction of learned training data
and utilize them to retain the task knowledge. One shortcoming of
architecture-based and rehearsal-based methods is that they require extra
memory space for storing parameters and old data samples De Lange et al.
(2021), while regularization-based methods might be overwhelmed when handling
many task domains Mai et al. (2022).

Continual learning in ToDs. Early work on CL for ToDs was introduced by Lee
(2017), where the elastic weight consolidation (EWC) method was utilized to
alleviate catastrophic forgetting on 3 sequential domains. Wu et al. (2019)
studied CL for DST sub-task on the MultiWOZ dataset, where several baselines
have been compared. Mi et al. (2020) combined data rehearsal and EWC on NLG
sub-task and expanded the learned knowledge to 13 task domains. Furthermore,
Madotto et al. (2020) developed a large benchmark for all INTENT, DST, and NLG
tasks, where mainstream CL methods (e.g., L2, AGEM, and Adapter) were compared
on a total of 37 domains. However, it shows that regularization-based methods
lose effectiveness when ToDs encounter dozens of tasks Madotto et al. (2020).
Meanwhile, architecture-based methods require a further step to determine
which set of parameters to use Wortsman et al. (2020), thus blocking the real-
time response from ToDs. In this work, we focus on rehearsal-based CL methods
as it is simple and effective, with an acceptable cost of extra memory.

Exemplar selection. To sample representative or informative exemplars from a
large dataset, reservoir sampling has been introduced in Isele and Cosgun
(2018); Chaudhry et al. (2019) so that the data distribution in the buffer
follows the data distribution that has already been seen. Rebuffi et al.
(2017) proposed a herding-based strategy to maintain an online coreset.
Similarly, gradient-based methods have also been proposed in Broderick et al.
(2013); Aljundi et al. (2019) to maximize the variance in the buffer. For ToDs
with clear task boundaries, Mi et al. (2020) defined representative exemplars
as a small set of utterances that fulfill a loss-based criterion on the
current domain. The above-mentioned methods leverage models to choose high-
quality data points, but fail to provide an insightful perspective on data
contribution analysis in terms of model performance on unseen data.

##  3 Methodology

###  3.1 Background: End-to-end ToDs

We formulate task-oriented dialogue systems as a sequence-to-sequence
generation problem that generates both API-calls and system responses. The
API-call consists of the user intent and the current dialogue state, which can
be empty or the system speech-act, to generate the system response. Thanks to
the recent advance in ToDs, all ground-truth information at each turn is
provided by existing annotated dialogue datasets. In this setting, the data
format of API-call $\mathcal{C}$ is shown as follows:

$\mathcal{C}(\mathbf{H})=\underbrace{\mathbf{I}}_{\text{Intent}}\underbrace{(s_{1}=v_{1},\dots,s_{p}=v_{p})}_{\text{Slot-
value \ pairs}},$ (1)

where $\mathbf{H}$ is the dialogue history, $\mathbf{I}$ denotes the user
intent, and ($s_{i}$, $v_{i}$) stands for a slot-value pair from dialogue
state tracking.

ToDs can be typically decomposed into different modules, including intent
recognition (INTENT), history state tracking (DST), and natural language
generation (NLG). In this paper, besides the sub-modules, we define the end-
to-end (E2E) manner that directly generates the system response $\mathbf{R}$:

$\displaystyle\mathbf{H}+\mathcal{C}(\mathbf{H})\rightarrow\mathbf{R}$ (2)

where $\mathcal{C}(\mathbf{H})$ is often empty, and thus the model can
directly generate a response according to dialogue history
($\mathbf{H}\rightarrow\mathbf{R}$). In addition, our method HESIT can work
for both E2E and modular ToDs. More details are attached in Appendix B.

###  3.2 Rehearsal-based CL in ToDs

Given the training data $\mathcal{D}$ consisting of $T$ ToDs task domains
$\mathcal{D}=\\{\mathcal{D}_{1}\cdots\mathcal{D}_{T}$}, continual learning
aims to train a neural model $f_{\theta}$ on $\mathcal{D}$ in a sequence of
tasks. In each task $t$, new data $\mathcal{D}_{t}$ is used to update model
$f_{\theta_{t-1}}$, while the updated model $f_{\theta_{t}}$ needs to perform
well on all tasks so far. In general, the crux of learning $f$ is to overcome
the catastrophic forgetting problem where learned knowledge is continually
overwritten by streamed data.

To suppress it, the rehearsal-based CL methods construct a small size of
memory buffer $\mathcal{M}_{t}$ from each task domain $\mathcal{D}_{t}$. When
training task $t$, the data in $\mathcal{M}_{1:{t-1}}$ will be retrained
together with $\mathcal{D}_{t}$. Accordingly, a specific training step of task
$t$ is written as:

$\begin{split}&\hat{\theta}=\arg\min_{\theta}\frac{1}{N}\sum_{(x_{i},y_{i})\in\mathcal{D}_{t}^{\textrm{trn}}}\ell((x_{i},y_{i}),\theta)\\\
\textrm{s.t.}\quad&(x_{i},y_{i})\in\mathcal{M}_{1:{t-1}}\cup\mathcal{D}_{t}^{\textrm{trn}}\end{split}$
(3)

where $N$ is the total number of training data points from $T$ tasks, $\ell$
denotes the empirical loss and $(x_{i},y_{i})$ is a training data point of
task $t$. Considering the storage burden of $\mathcal{M}$, the retained
samples, called exemplars, should be informative or influential for
$\mathcal{D}_{t}^{\textrm{trn}}$.

###  3.3 HESIT: Exemplar Selection

In this subsection, we present the details of the exemplar selection strategy
to find influential utterances in $\mathcal{M}_{t}$, which also serves as the
core technique in HESIT.

Intuitively, given a training sample
$z_{i}\coloneqq(x_{i},y_{i})\in\mathcal{D}_{t}^{\textrm{trn}}$, we want to
quantify its influence on the validation loss
$\mathcal{L}(\mathcal{D}_{t}^{\textrm{val}},\hat{\theta})$ via the resulting
model $\hat{\theta}$. One possible way is to find out the loss difference
$\mathcal{I}(z_{i},\mathcal{D}_{t}^{\textrm{val}})\coloneqq\mathcal{L}(\mathcal{D}_{t}^{\textrm{val}},\hat{\theta}_{\bar{z}_{i}})-\mathcal{L}(\mathcal{D}_{t}^{\textrm{val}},\hat{\theta})$
where $\hat{\theta}_{\bar{z}_{i}}$ would be the resulting model if $z_{i}$
were not present in $\mathcal{D}_{t}^{\textrm{trn}}$. For this purpose, we
introduce a weight variable $\epsilon_{i}$ and consider the new full-set
training loss:

$\mathcal{L}_{\textrm{trn}}(\theta)=\frac{1}{N}\sum_{(x_{i},y_{i})\in\mathcal{D}_{t}^{\textrm{trn}}}(\ell((x_{i},y_{i}),\theta)+\epsilon_{i}\ell(z_{i},\theta)).$
(4)

Now $\hat{\theta}$ and $\mathcal{I}(z_{i},\mathcal{D}_{t}^{\textrm{val}})$ are
taken as functions of $\epsilon_{i}$ and
$\mathcal{I}(z_{i},\mathcal{D}_{t}^{\textrm{val}})$ can be approximated by the
first order Taylor expansion:

$\begin{split}\mathcal{I}(z_{i},\mathcal{D}_{t}^{\textrm{val}})&\approx-\frac{1}{N}\frac{\mathrm{d}\mathcal{L}(\mathcal{D}_{t}^{\textrm{val}},\hat{\theta})}{\mathrm{d}\epsilon_{i}}{|}_{\epsilon_{i}=0}\\\
&=-\frac{1}{N}\frac{\partial\mathcal{L}(\mathcal{D}_{t}^{\textrm{val}},\hat{\theta})}{\partial\hat{\theta}}\frac{\mathrm{d}\hat{\theta}}{\mathrm{d}\epsilon_{i}}{|}_{\epsilon_{i}=0}.\end{split}$
(5)

While
$\frac{\partial\mathcal{L}(\mathcal{D}_{t}^{\textrm{val}},\hat{\theta})}{\partial\hat{\theta}}$
can be exactly calculated using gradient back-propagation on the validation
data, computing
$\nabla_{i}\coloneqq\frac{\mathrm{d}\hat{\theta}}{\mathrm{d}\epsilon_{i}}$,
which is also named hyper-gradient Bengio (2000), is not straightforward
because the parameter $\epsilon_{i}$ is involved in the entire optimization
process.

In gradient descent based optimization, for each iteration step $r$, we have
$\theta_{r}=\theta_{r-1}-\gamma g_{r-1}$ where $\gamma$ is the learning rate
and $g_{r-1}$ is the batch gradient. Then $\nabla_{i}$ can be computed
recursively as:

$\nabla_{i,r}=\nabla_{i,r-1}-\gamma(H_{r-1}\nabla_{i,r-1}+\frac{\partial
g_{r-1}}{\partial\epsilon_{i}})$ (6)

where $H_{r-1}$ denotes the Hessian of the batch loss with respect to
$\theta_{r-1}$. Most related works impose some assumptions on the model to
ensure that $H$ is invertible and attempt to approach the inverse Hessian
vector product by numerical methods Koh and Liang (2017). However, the
assumptions are too strong to be feasible in practice, and with increasing
model and dataset size, the numerical methods are time-consuming and may lead
to diverging results. Chen et al. (2021) propose a faster scheme by discarding
the Hessian computations, which have a bounded error. Therefore, HESIT can
recurrently update the $\nabla_{i,r}$ through the optimization trajectory with
an acceptable time cost. We further design a validation experiment in Section
5.3 to demonstrate that such a manner can successfully estimate data influence
without the estimation of Hessian.

Algorithm 1 Influential Example Tracing

1:Traced example set $\mathcal{Z}_{t}=\\{z_{i}\\}_{i=1}^{I}$ of task $t$,
training dataset $\mathcal{D}^{\textrm{trn}}_{t}$, validation dataset
$\mathcal{D}^{\textrm{val}}_{t}$, batch size $B$, training dataset size $N$,
total iteration steps $R$, learning rate $\gamma(t)$

2:Model $\hat{\theta}$, influence values
$\mathcal{I}(\mathcal{Z},\mathcal{D}_{t}^{\textrm{val}})$

3:Setup reproducible training environment

4:Train model with $\mathcal{D}^{\textrm{trn}}_{t}$ and obtain $\hat{\theta}$

5:$\boldsymbol{v}\leftarrow\frac{\partial\mathcal{L}(\mathcal{D}_{t}^{\textrm{val}},\hat{\theta})}{\partial\hat{\theta}}$

6:Reset training environment and retrain model with
$\mathcal{D}^{\textrm{trn}}_{t}$ for second identical training

7:for $i=1$ to $I$ do

8: $\boldsymbol{v}\nabla_{i,0}\leftarrow 0$

9:end for

10:for $r=1$ to $R$ do

11: if $\mathrm{current\ batch\ contains}\ z_{i}$ then

12:
$\boldsymbol{v}\nabla_{i,r}\leftarrow\gamma(t)\boldsymbol{v}\nabla_{i,r-1}-\frac{N}{B}\boldsymbol{v}g_{i,r-1}$

13: else

14:
$\boldsymbol{v}\nabla_{i,r}\leftarrow\gamma(t)\boldsymbol{v}\nabla_{i,r-1}$

15: end if

16:end for

17:for $i=1$ to $I$ do

18:
$\mathcal{I}(z_{i},\mathcal{D}^{\textrm{val}}_{t})\leftarrow-\frac{1}{N}\boldsymbol{v}\nabla_{i,R}$

19:end for

Algorithm 2 HESIT Training

1:Sequential $T$-domain training data
$\mathcal{D}=\\{(\mathcal{D}_{1}^{\textrm{trn}},\mathcal{D}_{1}^{\textrm{val}}),\ldots,(\mathcal{D}_{T}^{\textrm{trn}},\mathcal{D}_{T}^{\textrm{val}})\\}$

2:Parameter $\theta_{T}$ that handles T-domain tasks

3:Initialize the model parameter $\theta_{0}$ and replay buffer
$\mathcal{M}_{0}$

4:for $t=1$ to $T$ do

5: Sample $\mathcal{Z}_{t}\sim\mathcal{D}_{t}^{\textrm{trn}}$

6: Calculate $\mathcal{I}(\mathcal{Z}_{t},\mathcal{D}_{t}^{\textrm{val}})$ and
$\theta_{t-1}$ by Algo.1

7: Select $E^{K}_{t}\subset\mathcal{Z}_{t}$ by top
$\left\lVert\mathcal{I}(\mathcal{Z}_{t},\mathcal{D}_{t}^{\textrm{val}})\right\rVert$

8: while  _not converge_ do

9: Update $\theta_{t-1}$ using
{$\mathcal{D}_{t}^{\textrm{trn}}\cup\mathcal{M}_{t-1}\\}$

10: end while

11: $\mathcal{M}_{t}\leftarrow E^{K}_{t}\cup\mathcal{M}_{t-1}$

12:end for

As illustrated in Algorithm 1, the influence scores of training examples are
computed via reproducible retraining to reduce the space complexity. In this
way, the time complexity to trace hyper-gradient is $\mathcal{O}(I\cdot
R\cdot\omega)$, where $\omega$ is the time spent in computing parameter
gradient, but the space complexity has been reduced to $\mathcal{O}(I)$.
Finally, a set of exemplars $E^{K}_{t}$ are picked out according to the
influence scores, with top-$K$ strategy.

###  3.4 HESIT: Training Schedule

We present the training schedule of HESIT in Algorithm 2, which integrates the
exemplar selection strategy and rehearsal-based CL into ToDs. Specifically,
for each domain $t$, we select $E_{t}^{K}$ from $\mathcal{D}_{t}$ using a
hyper-gradient based strategy and feed them into the replay buffer
$\mathcal{M}$. Then $\mathcal{M}$ will be involved in the subsequent training
process for retraining the neural model.

To further accelerate training, we adopt two simplifications. Firstly, instead
of tracing all training examples, we sample and trace a subset
$\mathcal{Z}_{t}\gg E_{t}^{K}$ from each $\mathcal{D}^{\textrm{trn}}_{t}$ to
reduce the cost. Secondly, tracing is performed only in the first $R$
iterations rather than the entire training process, as the main benefits of
optimization have been obtained in these iterations. In addition, the
subsequent training (line 6-8) is depend on the size of domain $t$.

##  4 Experiment

###  4.1 Dataset

To evaluate the performance of our method, we employ the largest CL benchmark
of the ToDs task developed in Madotto et al. (2020). Specifically, four ToD
datasets are merged: TaskMaster 2019 (TM19) Byrne et al. (2019), TaskMaster
2020 (TM20) Byrne et al. (2019), Schema Guided Dialogue (SGD) Lin et al.
(2021), and MultiWoZ Budzianowski et al. (2018). Consequently, a curriculum of
37 domains is to be learned continuously in ToDs, where three modules (INTENT,
DST, and NLG) are well annotated after pre-processing. We consider two
settings which are Modularized setting that learns three modules separately,
and E2E setting that learns these modules in a unified manner, as illustrated
in 3.1.

In addition, we summarize the main statistics and detailed sample numbers for
each domain in Appendix D. It is noted that the amount of examples is highly
imbalanced across different domains, which ranges from a few hundred to more
than 36k. This distribution is more proximate to reality, as some task domains
lack training data in practice.

###  4.2 Evaluation Metrics

Based on three modules, we employ the following well-defined metrics to
evaluate system performance:

  * •

INTENT recognition is directly measured in terms of accuracy between the
predicted intent and ground-truth intent.

  * •

DST is evaluated by Joint Goal Accuracy (JGA) Wu et al. (2019) over the
ground-truth dialogue.

  * •

NLG is evaluated by two metrics: 1) BLEU score Papineni et al. (2002) which is
calculated by the distance between generated response and reference sentence,
and 2) slot error rate (EER) Wen et al. (2015) which is computed as the ratio
between the total number of slots and the values not appearing in the
response.

For all metrics except EER, a higher value denotes better performance. In
addition, same as Kale and Rastogi (2020), we do not calculate EER count for
the SGD dataset, since some slots of the SGD dataset have only binary values,
_e.g.,_ yes or no, which is unfair to calculate average with others.

###  4.3 Baselines

In order to compare the effects of different CL methods, we investigate both
regularization-based and rehearsal-based baselines in the ToDs benchmark.
Architecture-based methods are excluded as they require a further decision
step to determine the agnostic domain of the test set, resulting in a slow
system response.

Regularization baseline. We consider two kinds of regularization terms to
constrain the model parameter update, which is identity function (L2) and
Fisher information matrix (EWC) proposed in Kirkpatrick et al. (2017).

Rehearsal baseline. We first employ 2 kinds of mainstream rehearsal-based CL
methods, which are A-GEM Chaudhry et al. (2018), and LAMOL Sun et al. (2019).
Since our contribution focuses on exemplar selection, we further reproduce 4
coreset selection strategies in the ToDs benchmark, where 2 baselines are from
data-perspective: (1) Random denotes that exemplars are randomly sampled from
the train set, and (2) UNIFORM denotes that we uniformly select exemplars to
ensure all user intents would be contained in the buffer. Meanwhile, the other
two baselines are from the model perspective: (3) GSS Aljundi et al. (2019)
selects the examples that maximize the gradient variance in the replay buffer.
(4) ARPER Mi et al. (2020) selects exemplars from the training set that
obtains minimum loss-based criterion. (1)$\sim$(4) share the same training
schedule with the proposed HESIT but they adopt different exemplar selection
strategies respectively.

Additionally, we attach a lower-bound baseline, called VANILLA, which is
trained on each task continuously without any anti-forgetting mechanism. We
also provide the MULTI baseline which trains models with all data in an
integrated curriculum simultaneously, which is widely viewed as the upper
bound of CL methods.

###  4.4 Experimental Setup

For all experiments, we leverage the pre-trained GPT-2 Radford et al. (2019)
as the ToDs backbone. Each domain is trained for 10 epochs with early stopping
over the validation set. The learning rate is set as 0.001 with a warm-up
schedule. In L2 and EWC, the regularization weight is set as 0.001.
Considering the buffer memory, all data replay-based methods sample 50
exemplars for each task domain. In HESIT, the $\mathcal{Z}_{t}$ is set as 1000
for each domain, and the interested training examples are traced for the first
5 epochs. Task order in the curriculum usually has a slight impact on the
final performance. To avoid contingency, all experiments are repeated three
times and report the average results.

Method | Mem. | INTENT | DST | NLG
---|---|---|---|---
_Accuracy_ $\uparrow$ |  _JGA_ $\uparrow$ |  _EER_ $\downarrow$ |  _BLEU_ $\uparrow$
_VANILLA_ | $\emptyset$ | 2.65 | 9.33 | 50.91 | 4.49
Regularization-based methods
_L2_ | $\emptyset$ | 2.33 | 6.85 | 56.42 | 5.08
_EWC_ | $\emptyset$ | 2.46 | 8.98 | 52.61 | 4.70
Rehearsal-based methods
_A-GEM_ | $t|\mathcal{M}|$ | 31.02 | 11.23 | 60.98 | 4.53
_LAMOL_ | $\emptyset$ | 2.68 | 9.42 | 66.31 | 3.82
|  | Data-perspective
_RANDOM_ | $t|\mathcal{M}|$ | 78.22 | 29.47 | 19.36 | 16.92
_UNIFORM_ | $t|\mathcal{M}|$ | 80.67 | 28.94 | 19.63 | 17.84
|  | Model-perspective
_GSS_ | $t|\mathcal{M}|$ | 81.42 | 30.33 | 17.48 | 17.97
_ARPER_ | $t|\mathcal{M}|$ | 77.60 | 27.82 | 20.52 | 16.44
|  | Performance-perspective (ours)
_HESIT_ | $t|\mathcal{M}|$ | 83.46 | 31.22 | 16.78 | 18.25
_MULTI_ | - | 95.45 | 48.90 | 12.56 | 23.61

Table 1: E2E results on the test set in terms of INTENT accuracy, JGA, EER,
and BLEU. “Mem.” denotes the memory size of the buffer, where $t=37$ and
$|\mathcal{M}|=50$.

Method | Mem. | INTENT | DST | NLG
---|---|---|---|---
_Accuracy_ $\uparrow$ |  _JGA_ $\uparrow$ |  _EER_ $\downarrow$ |  _BLEU_ $\uparrow$
_VANILLA_ | $\emptyset$ | 3.06 | 10.28 | 18.09 | 10.42
Regularization-based methods
_L2_ | $\emptyset$ | 3.59 | 9.94 | 18.16 | 11.13
_EWC_ | $\emptyset$ | 3.72 | 10.06 | 18.12 | 11.70
Rehearsal-based methods
_A-GEM_ | $t|\mathcal{M}|$ | 10.57 | 9.86 | 36.22 | 6.40
_LAMOL_ | $\emptyset$ | 2.72 | 9.44 | 35.83 | 4.43
|  | Data-perspective
_RANDOM_ | $t|\mathcal{M}|$ | 79.92 | 39.54 | 5.82 | 21.33
_UNIFORM_ | $t|\mathcal{M}|$ | 81.10 | 39.92 | 5.42 | 21.17
|  | Model-perspective
_GSS_ | $t|\mathcal{M}|$ | 81.36 | 39.80 | 5.17 | 21.62
_ARPER_ | $t|\mathcal{M}|$ | 79.63 | 39.21 | 6.08 | 21.12
|  | Performance-perspective (ours)
_HESIT_ | $t|\mathcal{M}|$ | 82.71 | 40.04 | 5.11 | 21.48
_MULTI_ | - | 87.50 | 50.03 | 3.42 | 26.15

Table 2: Modularized results on the test set in terms of INTENT accuracy, JGA,
EER, and BLEU.

##  5 Result and Analysis

###  5.1 Main Comparison Results

E2E setting. We report the main E2E results on the full test set in Table 1,
where all metrics are evaluated at the end of the curriculum (37 task
domains). We observe that regularization baselines (L2 and EWC) can not
alleviate catastrophic forgetting well, and even achieve worse performance
than VANILLA in terms of most metrics. The learning curve of EWC is visualized
in Fig. 4. The reason is that the regularization item gradually constrains the
optimization space of the neural model, making it difficult to handle
subsequent tasks in a long curriculum. Similarly, due to clear task boundaries
of ToDs, A-GEM and LAMOL baselines also lose effectiveness as their learning
schedules depend on correlations across different domains. In this case,
regardless of the selection strategy, storing some raw data and retraining the
model periodically can obtain remarkable performance gain. As a performance-
based selection strategy, our proposed HESIT performs significantly better
than the RANDOM baseline, which respectively achieves 6.70%, 5.94%, 13.33%,
and 7.86% relative improvement in terms of INTENT accuracy, JGA, EER, and
BLEU. In addition, compared with the model perspective GSS baseline, HESIT
respectively achieves 2.04%, 2.93%, 4.0%, and 1.48% relative improvement.

Modularized setting. We summarize modularized results in Table 2 that INTENT,
DST, and NLG modules are evaluated separately. Compared with the E2E setting,
we observe some result fluctuations in different evaluation metrics. On the
one hand, each neural model focuses on a single module leading to performance
gains. Especially in the NLG sub-task, the modularized model surpasses the E2E
model by a large margin, where the EER of MULTI reduces nearly fourfold
(12.56%$\rightarrow$3.42%). On the other hand, the modularized model ignores
the relevance of each sub-task, resulting in performance degradation according
to INTENT accuracy (95.45%$\rightarrow$87.50%). Regardless of E2E or
modularized settings, HESIT shows superiority in alleviating catastrophic
forgetting in most metrics.

Figure 2: Learning curve for INTENT accuracy in E2E setting. Each test point
is evaluated on the already learned task in the curriculum.

Forgetting curve. We visualize the learning curve of INTENT accuracy in the
37-domain curriculum to observe how HESIT overcomes catastrophic forgetting.
To avoid the overlapping of the chart, partial baselines are selected for
comparison and shown in Fig. 2. It is worth noting that we fix the task order
in the curriculum, and each test point on the x-axis evaluates the model
performance on the test set of learned tasks.

From the forgetting curve, we observe that 1) Catastrophic forgetting happened
in the VANILLA baseline, and the model performance obviously degrades over
time. 2) EWC, A-GEM, and LAMOL can alleviate catastrophic forgetting in the
first few tasks, but gradually fail to handle learned tasks with an increase
in task amounts. In addition, LAMOL achieves outstanding accuracy at the
beginning of the curriculum and performs better than HESIT. 3) For HESIT,
there are some fluctuations in performance, but then stabilize at a high
performance in terms of accuracy.

Method | # Er. | INTENT | DST | NLG
---|---|---|---|---
_Accuracy_ $\uparrow$ |  _JGA_ $\uparrow$ |  _EER_ $\downarrow$ |  _BLEU_ $\uparrow$
_VANILLA_ | - | 2.65 | 9.33 | 50.91 | 4.49
| Data-perspective
_UNIFORM_ | 20 | 65.15 | 24.08 | 23.31 | 16.47
30 | 72.22 | 25.98 | 22.01 | 17.19
40 | 77.56 | 27.80 | 19.36 | 17.44
50 | 80.67 | 28.94 | 19.63 | 17.84
| Model-perspective
_GSS_ | 20 | 66.78 | 23.85 | 18.99 | 17.04
30 | 74.32 | 26.68 | 21.02 | 17.55
40 | 78.09 | 29.76 | 18.61 | 17.70
50 | 81.42 | 30.33 | 17.48 | 17.97
| Performance-perspective
_HESIT_ | 20 | 70.32 | 24.97 | 19.94 | 16.81
30 | 72.85 | 27.85 | 18.40 | 17.45
40 | 80.01 | 30.29 | 18.52 | 17.93
50 | 83.46 | 31.22 | 16.78 | 18.25

Table 3: E2E results using different exemplar selection strategies. “# Er.”
denotes the amount of exemplars for each task domain.

ID | Domain | Module | Type | Utterance | Score
---|---|---|---|---|---
1 | Train | INTENT | $\mathbf{H}$ $\rightarrow$ $\mathcal{C}(\mathbf{H})$ |  USER: "Yes, I would like a taxi to the Town Centre. From the train station." API: "taxi_inform() train_inform() [eos]" | $-0.64$
2 | Taxi |  INTENT & DST | $\mathbf{H}$ $\rightarrow$ $\mathcal{C}(\mathbf{H})$ |  USER: "I need the phone number and location of the nearest Red Lobster in the downtown Cambridge area?" API: "taxi_request(phone="?") [eos]” | $-0.77$
3 | Hotel | NLG | $\mathbf{H}$ $\rightarrow$ $\mathbf{R}$ |  USER: "Yes, I am attending a physician’s conference and need to locate a room for tonight. API-OUT:" SYSTEM: "Do you have an area or price preference? [eos]" | $-0.30$

Table 4: Detrimental examples with low score influences selected by HESIT from
the Multi-WoZ dataset. Scores of traced examples are re-normalized to [-1,1].

###  5.2 Analysis for Exemplar Selection

Amount of exemplar. Using different selection strategies, we vary the size of
the memory buffer containing [20,30,40,50] exemplars for each domain. The
results are shown in Table 3. We observe that 20 exemplars can significantly
alleviate cartographic forgetting, benefiting all metrics compared with the
VANILLA baseline. As the buffer size increases, all systems achieve better
performance including each ToDs module. Specifically, GSS achieves the best
NLG performance when the exemplar amount is 20. In other cases, HESIT performs
the best regardless of the exemplar amount, demonstrating its superiority over
the performance perspective methods.

Case study. As each training sample is assessed using the Algo. 1’s Influence
Functions to evaluate the impact of unseen data, we conduct a case study on
several detrimental examples with low influence scores to examine how HESIT
eliminates these negative examples. To this end, we simultaneously trace 1k
training points of each domain and randomly sample some low-score examples in
Table 4. The first two examples adhere to the format of API-call, and the
third example is a system response. In the first two cases, the HESIT
successfully identifies the incorrect train_inform() intent and the missing
value-slot pairs with extremely negative scores. The third case is likewise
eliminated by the HESIT with a negative score because the model is confused by
the empty API-OUT from the history.

###  5.3 Effect of HESIT

In this part, we conduct experiments to demonstrate that HESIT can
successfully measure the training data influence without estimating Hessian.
To this end, we construct two Hessian-involved IFs baselines Conjugate
gradients (CG) and Stochastic estimation (LISSA) Agarwal et al. (2017) and one
Hessian-free Hydra Chen et al. (2021) for comparison. As the Hessian of a
large pre-trained model is likely irreversible, we further employ a CNN
backbone and CIFAR-10 dataset Krizhevsky et al. (2009) to evaluate both
computation time and performance. The details of the dataset, model and
baselines are attached in Appendix E.

We vary the amount of training and validation data and report the time spent
in influence analysis in Table 5. “Hes.” indicates whether the method needs to
estimate the Hessian of the model. “Max_Iter.” denotes CG iteration in Martens
et al. (2010), and is set according to $\mathbf{T}$. In LISSA, “depth” and
“repeat” are two hyper-parameters that influence the computation time, and
generally, their products should be equal to $\mathbf{T}$. By comparing the
baselines and HESIT on ($\mathbf{T}$, $\mathbf{V}$) training-validation
dataset pairs of varying sizes, we find that HESIT, Hydra and LISSA require
comparable calculation time, whereas baseline CG is significantly slower.

Method | Hes. | # ($\mathbf{T}$, $\mathbf{V}$) | Setting |  Time (Sec.)
---|---|---|---|---
_CG_ | ✓ | ($10^{2},10$) | _Max_Iter._ = $\mathbf{T}$ | 23.1
($10^{3},10^{2}$) | 665.8
($10^{4},10^{3}$) | -
_LISSA_ | ✓ | ($10^{2},10$) |  _Depth_ = $\mathbf{T}$ / $10$ _Repeat_ = $10$ | 3.6
($10^{3},10^{2}$) | 13.2
($10^{4},10^{3}$) | 136.7
_Hydra_ | ✗ | ($10^{2},10$) | _Trace_ID_ = $\mathbf{T}$ | 2.4
($10^{3},10^{2}$) | 6.9
($10^{4},10^{3}$) | 161.1
_HESIT_ | ✗ | ($10^{2},10$) | _Trace_ID_ = $\mathbf{T}$ | 2.2
($10^{3},10^{2}$) | 6.3
($10^{4},10^{3}$) | 132.9

Table 5: Computation time for data analysis methods. # ($\mathbf{T}$,
$\mathbf{V}$) denotes that measure the influence of $\mathbf{T}$ training data
to the performance on $\mathbf{V}$ validation data. The computing devices are
AMD-EPYC 7763 (CPU) and NVIDIA A100 (GPU). Figure 3: Inter-class and Intra-
class (diagonal elements) contributions for CIFAR-10 dataset, which are
measured by (a) LISSA and (b) HESIT.

To further demonstrate that HESIT can still properly qualify data contribution
even without estimating inverse Hessian, we then visualize the inter-class and
intra-class contributions in Fig 3, where diagonal elements denote the
contribution of the intra-class training data to the test class. It is
observed that without estimating inverse Hessian, HESIT can still successfully
measure the intra-class data influence by hyper-gradient tracing. In addition,
it is observed that compared with LISSA, HESIT provides a higher estimation of
intra-class contribution due to iteratively tracing.

##  6 Conclusion

In this paper, we propose HESIT to address the catastrophic forgetting in
ToDs. It selects exemplar for rehearsal-based CL methods that examines
training examples in each domain from a performance perspective. Furthermore,
HESIT tracks the hyper-gradient of training examples in an optimization method
that is Hessian-free and compatible with large pre-trained models. Experiments
show that HESIT effectively overcomes catastrophic forgetting and delivers
state-of-the-art performance on the largest ToDs benchmark for CL.

## Limitations

When selecting exemplars for the current domain, HESIT only evaluates the
performance on the self-domain validation set. However, we suggest that cross-
domain influences should be taken into account, which means that the
validation set of the later domain in the curriculum might be utilised to
examine the picked example of the earlier domain. We would leave it as future
work. In addition, we can only compare our model to the state-of-the-art work
on ToDs from 2020, as there has been little work on ToDs employing CL in
recent years compared to the extensive explorations of CL in computer vision,
especially classification task.

## Ethical Consideration

In developing and evaluating HESIT, our hyper-gradient-based exemplar strategy
for Intelligent task-oriented dialogue systems (ToDs), we considered a number
of significant ethical implications to ensure that our research conforms to
generally accepted standards for the development of responsible AI.

Privacy and Data Security: While HESIT requires the selection of exemplars
from the training data, we are mindful of user privacy and data security. The
data used for training and testing HESIT are either publicly accessible or
anonymized and used with the proper permissions, ensuring that no personally
identifiable user information is exploited.

Bias and Fairness: Despite the optimistic results demonstrated by HESIT, it is
essential to recognise the inherent possibility of bias in any AI system. We
have attempted to mitigate this risk by assuring a diverse and representative
training dataset, thereby preventing the model from favouring certain task
domains disproportionately over others. Future work will include continued
evaluations of potential biases with the objective of developing additional
bias mitigation strategies.

Transparency and Accountability: The performance of HESIT is primarily
dependent on the exemplars chosen from each task domain. This strategy is
effective in terms of model performance, but if not managed carefully, it
could contribute to a lack of transparency. We intend to maintain detailed
documentation regarding the exemplar selection procedure and to make our
algorithms, methods, and results accessible and interpretable to both
practitioners and users.

Impact on Employment: The development of ToDs with the capacity for continuous
learning may raise concerns regarding potential employment displacement. While
our research contributes to the advancement of AI, it is not intended to
supplant humans but rather to enhance their capabilities and productivity. To
ensure a responsible transition to this technology, we encourage ongoing
discussions regarding this issue.

Potential Misuse: As with all AI technologies, there is the potential for
misuse with our method. If HESIT is used with nefarious intent or if the model
is fed inappropriate data, there is a risk of abuse. We strongly advise
establishing appropriate precautions and usage policies.

Long-term effects: We recognise that the long-term effects of HESIT and
analogous technologies are indeterminate and require constant monitoring. As
researchers, we are committed to continuous monitoring of our technology, its
applications, and their societal implications.

In conclusion, we believe that the advantages of developing Continual Learning
systems such as HESIT must be weighed against these ethical considerations. As
a result, we do not perceive this work as a destination, but rather as a
stepping stone on the path towards responsible and beneficial AI.

## References

  * Agarwal et al. (2017) Naman Agarwal, Brian Bullins, and Elad Hazan. 2017.  Second-order stochastic optimization for machine learning in linear time.  _The Journal of Machine Learning Research_ , 18(1):4148–4187.
  * Aljundi et al. (2019) Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. 2019.  Gradient based sample selection for online continual learning.  _Advances in neural information processing systems_ , 32.
  * Bae et al. (2022) Sanghwan Bae, Donghyun Kwak, Sungdong Kim, Donghoon Ham, Soyoung Kang, Sang-Woo Lee, and Woomyoung Park. 2022.  Building a role specified open-domain dialogue system leveraging large-scale language models.  _arXiv preprint arXiv:2205.00176_.
  * Bengio (2000) Yoshua Bengio. 2000.  Continuous optimization of hyper-parameters.  In _Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium_ , volume 1, pages 305–310. IEEE.
  * Biesialska et al. (2020) Magdalena Biesialska, Katarzyna Biesialska, and Marta R Costa-Jussa. 2020.  Continual lifelong learning in natural language processing: A survey.  _arXiv preprint arXiv:2012.09823_.
  * Broderick et al. (2013) Tamara Broderick, Nicholas Boyd, Andre Wibisono, Ashia C Wilson, and Michael I Jordan. 2013.  Streaming variational bayes.  _Advances in neural information processing systems_ , 26.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  Multiwoz–a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.  _arXiv preprint arXiv:1810.00278_.
  * Byrne et al. (2019) Bill Byrne, Karthik Krishnamoorthi, Chinnadhurai Sankar, Arvind Neelakantan, Daniel Duckworth, Semih Yavuz, Ben Goodrich, Amit Dubey, Andy Cedilnik, and Kyu-Young Kim. 2019.  Taskmaster-1: Toward a realistic and diverse dialog dataset.  _arXiv preprint arXiv:1909.05358_.
  * Chaudhry et al. (2018) Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. 2018\.  Efficient lifelong learning with a-gem.  _arXiv preprint arXiv:1812.00420_.
  * Chaudhry et al. (2019) Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and M Ranzato. 2019.  Continual learning with tiny episodic memories.  _arXiv preprint arXiv:1902.10486_.
  * Chen et al. (2021) Yuanyuan Chen, Boyang Li, Han Yu, Pengcheng Wu, and Chunyan Miao. 2021.  Hydra: Hypergradient data relevance analysis for interpreting deep neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pages 7081–7089.
  * Cui et al. (2019) Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. 2019.  Class-balanced loss based on effective number of samples.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 9268–9277.
  * De Lange et al. (2021) Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. 2021.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE transactions on pattern analysis and machine intelligence_ , 44(7):3366–3385.
  * Ermis et al. (2022) Beyza Ermis, Giovanni Zappella, Martin Wistuba, Aditya Rawal, and Cédric Archambeau. 2022.  Continual learning with transformers for image classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3774–3781.
  * Guo et al. (2020) Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. 2020\.  Fastif: Scalable influence functions for efficient model interpretation and debugging.  _arXiv preprint arXiv:2012.15781_.
  * Huang et al. (2021) Yufan Huang, Yanzhe Zhang, Jiaao Chen, Xuezhi Wang, and Diyi Yang. 2021.  Continual learning for text classification with information disentanglement based regularization.  _arXiv preprint arXiv:2104.05489_.
  * Isele and Cosgun (2018) David Isele and Akansel Cosgun. 2018.  Selective experience replay for lifelong learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Kale and Rastogi (2020) Mihir Kale and Abhinav Rastogi. 2020.  Few-shot natural language generation by rewriting templates.  _arXiv preprint arXiv:2004.15006_.
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ , 114(13):3521–3526.
  * Koh and Liang (2017) Pang Wei Koh and Percy Liang. 2017.  Understanding black-box predictions via influence functions.  In _International conference on machine learning_ , pages 1885–1894. PMLR.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al. 2009.  Learning multiple layers of features from tiny images.
  * Lee (2017) Sungjin Lee. 2017.  Toward continual learning for conversational agents.  _arXiv preprint arXiv:1712.09943_.
  * Li et al. (2019) Xilai Li, Yingbo Zhou, Tianfu Wu, Richard Socher, and Caiming Xiong. 2019.  Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting.  In _International Conference on Machine Learning_ , pages 3925–3934. PMLR.
  * Lin et al. (2021) Zhaojiang Lin, Andrea Madotto, Genta Indra Winata, Peng Xu, Feijun Jiang, Yuxiang Hu, Chen Shi, and Pascale Fung. 2021.  Bitod: A bilingual multi-domain dataset for task-oriented dialogue modeling.  _arXiv preprint arXiv:2106.02787_.
  * Madotto et al. (2020) Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eunjoon Cho, and Zhiguang Wang. 2020.  Continual learning in task-oriented dialogue systems.  _arXiv preprint arXiv:2012.15504_.
  * Mai et al. (2022) Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. 2022.  Online continual learning in image classification: An empirical survey.  _Neurocomputing_ , 469:28–51.
  * Maltoni and Lomonaco (2019) Davide Maltoni and Vincenzo Lomonaco. 2019.  Continuous learning in single-incremental-task scenarios.  _Neural Networks_ , 116:56–73.
  * Martens et al. (2010) James Martens et al. 2010.  Deep learning via hessian-free optimization.  In _ICML_ , volume 27, pages 735–742.
  * McCloskey and Cohen (1989) Michael McCloskey and Neal J Cohen. 1989.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_ , volume 24, pages 109–165. Elsevier.
  * Mi et al. (2020) Fei Mi, Liangwei Chen, Mengjie Zhao, Minlie Huang, and Boi Faltings. 2020.  Continual learning for natural language generation in task-oriented dialog systems.  _arXiv preprint arXiv:2010.00910_.
  * Mundt et al. (2020) Martin Mundt, Yong Won Hong, Iuliia Pliushch, and Visvanathan Ramesh. 2020.  A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning.  _arXiv preprint arXiv:2009.01797_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Pruthi et al. (2020) Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. 2020.  Estimating training data influence by tracing gradient descent.  _Advances in Neural Information Processing Systems_ , 33:19920–19930.
  * Qin et al. (2023) Chengwei Qin, Chen Chen, and Shafiq Joty. 2023.  Lifelong sequence generation with dynamic module expansion and adaptation.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 6701–6714, Singapore. Association for Computational Linguistics.
  * Qin et al. (2024) Chengwei Qin, Ruirui Chen, Ruochen Zhao, Wenhan Xia, and Shafiq Joty. 2024.  Lifelong event detection with embedding space separation and compaction.  In _2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics_.
  * Qin and Joty (2022a) Chengwei Qin and Shafiq Joty. 2022a.  Continual few-shot relation learning via embedding space regularization and data augmentation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2776–2789, Dublin, Ireland. Association for Computational Linguistics.
  * Qin and Joty (2022b) Chengwei Qin and Shafiq Joty. 2022b.  LFPT5: A unified framework for lifelong few-shot language learning based on prompt tuning of t5.  In _International Conference on Learning Representations_.
  * Qu et al. (2021) Haoxuan Qu, Hossein Rahmani, Li Xu, Bryan Williams, and Jun Liu. 2021.  Recent advances of continual learning in computer vision: An overview.  _arXiv preprint arXiv:2109.11369_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rebuffi et al. (2017) Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. 2017.  icarl: Incremental classifier and representation learning.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pages 2001–2010.
  * Rolnick et al. (2019) David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. 2019.  Experience replay for continual learning.  _Advances in Neural Information Processing Systems_ , 32.
  * Rusu et al. (2016) Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. 2016.  Progressive neural networks.  _arXiv preprint arXiv:1606.04671_.
  * Serra et al. (2018) Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. 2018.  Overcoming catastrophic forgetting with hard attention to the task.  In _International Conference on Machine Learning_ , pages 4548–4557. PMLR.
  * Sun et al. (2019) Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. 2019.  Lamol: Language modeling for lifelong language learning.  _arXiv preprint arXiv:1909.03329_.
  * Sun et al. (2022) Qing Sun, Fan Lyu, Fanhua Shang, Wei Feng, and Liang Wan. 2022.  Exploring example influence in continual learning.  _arXiv preprint arXiv:2209.12241_.
  * Verwimp et al. (2021) Eli Verwimp, Matthias De Lange, and Tinne Tuytelaars. 2021.  Rehearsal revealed: The limits and merits of revisiting samples in continual learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9385–9394.
  * Wen et al. (2015) Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young. 2015.  Semantically conditioned lstm-based natural language generation for spoken dialogue systems.  _arXiv preprint arXiv:1508.01745_.
  * Wortsman et al. (2020) Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. 2020.  Supermasks in superposition.  _Advances in Neural Information Processing Systems_ , 33:15173–15184.
  * Wu et al. (2019) Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung. 2019.  Transferable multi-domain state generator for task-oriented dialogue systems.  _arXiv preprint arXiv:1905.08743_.
  * Xu and Zhu (2018) Ju Xu and Zhanxing Zhu. 2018.  Reinforced continual learning.  _Advances in Neural Information Processing Systems_ , 31.

##  Appendix A Large Language Model and ToDs

Large Language Models (LLMs) have emerged as an epistemic beacon in the field
of natural language processing (NLP), endowing text-based tasks with
substantial performance gains including ToDs Bae et al. (2022). Furthermore, a
foundation model with multiple adapters seems to be potential solution for
overcoming catastrophic forgetting. However, this combination can not fit
continual learning in ToDs, especially with increasing task domains. Firstly,
the task domain of inference data is agnostic that requires a extra procedure
to determine which of adapter should be activated for inference. Secondly, no
matter what procedure is utilized, all of 37 combinations need to be traversed
once, resulting in unacceptable delay for ToDs. Therefore, instead of using
LLMs with adapter, we utilize single model to adapt to all domains.

##  Appendix B Examples of E2E ToDs

Figure 4: Example of input-out pairs, for the four settings, INTENT, DST, NLG
and end-to-end (E2E).

We first give an example to elaborate INTENT, DST, NLG, and E2E. As shown in
Fig 4, the grey box denotes the API-call $\mathcal{C}(\mathbf{H})$ process,
where the user intents (INTENT) and slot-value pairs (DST) are both predicted
according to dialogue history $\mathbf{H}$. Then the system will generate a
response (NLG) based on both $\mathbf{H}$ and $\mathcal{C}(\mathbf{H})$:
$\mathbf{H}+\mathcal{C}(\mathbf{H})\rightarrow\mathbf{R}$. In addition, when
$\mathcal{C}(\mathbf{H})$ is empty, the system can directly generate responses
based on history: $\mathbf{H}\rightarrow\mathbf{R}$.

##  Appendix C Methodology Supplement

For each task domain, Let $\mathcal{D}_{t}$ consists of training set
$\mathcal{D}_{t}^{\textrm{trn}}$, validation set
$\mathcal{D}_{t}^{\textrm{val}}$, and test set $\mathcal{D}_{t}^{tst}$. Assume
$\mathcal{D}_{t}^{\textrm{trn}}$ contains $N$ training-points which is defined
as $z_{i}=(x_{i},y_{i})$. Our goal is to pick out those influential $z_{i}$s
that benefit model performance on unseen test-point
$z^{\prime}\in\mathcal{D}_{t}^{\textrm{val}}$, as $\mathcal{D}_{t}^{tst}$ is
unavailable during training. Leveraging derivation chain rule, the IFs Koh and
Liang (2017) define the influence of $z$ on $z^{\prime}$ as:

$\mathcal{I}(z,z^{\prime})=-\nabla_{\theta}\ell(z^{\prime},\hat{\theta})^{T}H^{-1}_{\hat{\theta}}\nabla_{\theta}\ell(z,\hat{\theta})$
(7)

where $H_{\hat{\theta}}=\nabla_{\theta}^{2}\ell(z,\hat{\theta})$ is the
Hessian and is positive definite (PD) by assumption. In practice,
$H_{\hat{\theta}}$ can not be materialized in memory, let alone be inverted by
standard linear algebra.

Mainstream methods employ LISSA Agarwal et al. (2017), which only samples a
single point per iteration as an estimator of inverse Hessian. Compared with
the standard transformation of matrix inversion Martens et al. (2010), LISSA
shows significant speedups in the following iteration $r$:

$H_{r}^{-1}v=v+(1-H)H^{-1}_{r-1}v$ (8)

where $H$ is approximated on random batch and
$v=\nabla_{\theta}\ell(z^{\prime},\hat{\theta})$ is a gradient. However, this
method faces two drawbacks. Firstly, only the final parameters are utilized to
calculate the gradient, while the data influence is involved in a dynamic
optimization process. Secondly, Hessian might not be positive definite for a
large pre-trained model, and the estimation is not accurate enough. In
practice, we found that using LISSA to estimate the inverse Hessian of GPT-2
is unstable and time-consuming.

To address the above defects, _TracIn_ Pruthi et al. (2020) traces the changes
in the loss across all gradient steps, and avoids the estimation of inverse
Hessian by the following definition:

$\mathcal{I}_{\emph{TracIn}}(z,z^{\prime})=\frac{1}{C}\sum_{i=1}^{C}\gamma_{i}\
\nabla_{\theta_{i}}\ell(z,\hat{\theta_{i}})\cdot\nabla_{\theta_{i}}\ell(z^{\prime},\hat{\theta_{i}})$
(9)

where $C$ is the number of the checkpoints during training. Despite
considering the optimization process, the computing complexity is $C$ times
slower than that of using exact gradient similarity and, as discussed in
Pruthi et al. (2020), care needs to be taken in selecting checkpoints. Another
practical obstacle of _TracIn_ in CL settings is that only several epochs are
trained for each domain, resulting in insufficient $C$ for influence analysis.

##  Appendix D Dataset Statistics

The main dataset statistics are shown in Table 6. It contains 37 domains, 280
types of intents, and more than 31K training dialogues. The detailed example
amounts of modules as well as training, validation, and test set are
illustrated in Table 7.

Name | Train | Valid | Test | Dom. | Intents | Turns
---|---|---|---|---|---|---
TM19 | 4,403 | 551 | 553 | 6 | 112 | 19.97
TM20 | 13,839 | 1,731 | 1,734 | 7 | 128 | 16.92
MWoZ | 7,906 | 1,000 | 1,000 | 5 | 15 | 13.93
SGD | 5,278 | 761 | 1,531 | 19 | 43 | 14.71
Total | 31,426 | 4,043 | 4,818 | 37 | 280 | 16.23

Table 6: Main datasets statistics.

Due to a serious imbalance of task domain, we observe that some CL methods
(e.g., EWC) are vulnerable to the task order in the curriculum. If the model
first learns on a small domain, then the regularization item would influence
the subsequent learning on large domains. Meanwhile, we found the data replay
methods are more insensitive to task order, as the data amount of each domain
stored for retraining is equal. In addition, we have also attempted to sample
different amounts of exemplars according to the size of the domain. However,
it achieves similar results compared with sampling an equal amount of
exemplars for each domain.

Domains | DST & INTENT | NLG | End-to-End
---|---|---|---
Train | Dev | Test | Train | Dev | Test | Train | Dev | Test
missingTM19 movie | 4733 | 584 | 500 | 3010 | 366 | 341 | 12766 | 1632 | 1481
missingTM19 auto | 3897 | 448 | 522 | 2128 | 223 | 283 | 10918 | 1248 | 1443
missingTM19 restaurant | 4434 | 568 | 561 | 2582 | 330 | 333 | 12862 | 1669 | 1630
missingTM19 pizza | 2883 | 381 | 359 | 1326 | 171 | 171 | 8720 | 1145 | 1083
missingTM19 uber | 4378 | 535 | 525 | 2418 | 290 | 278 | 11331 | 1362 | 1361
missingTM19 coffee | 2591 | 302 | 335 | 1381 | 151 | 184 | 7429 | 894 | 936
missingTM20 flight | 15868 | 1974 | 1940 | 10148 | 1272 | 1245 | 36778 | 4579 | 4569
missingTM20 food-ordering | 3404 | 411 | 431 | 2394 | 277 | 287 | 7838 | 941 | 986
missingTM20 hotel | 15029 | 1908 | 1960 | 6590 | 842 | 869 | 35022 | 4400 | 4532
missingTM20 music | 5917 | 764 | 769 | 4196 | 537 | 523 | 13723 | 1773 | 1787
missingTM20 restaurant | 13738 | 1761 | 1691 | 8356 | 1063 | 994 | 34560 | 4398 | 4297
missingTM20 sport | 13072 | 1668 | 1654 | 12044 | 1553 | 1542 | 29391 | 3765 | 3723
missingTM20 movie | 13221 | 1703 | 1567 | 9406 | 1203 | 1093 | 32423 | 4158 | 3881
missingMWOZ taxi | 1239 | 234 | 194 | 402 | 71 | 56 | 2478 | 468 | 388
missingMWOZ train | 1452 | 158 | 160 | 563 | 63 | 59 | 2905 | 316 | 320
missingMWOZ restaurant | 5227 | 243 | 281 | 3333 | 141 | 177 | 10461 | 486 | 563
missingMWOZ hotel | 2798 | 289 | 385 | 1924 | 194 | 258 | 5602 | 579 | 771
missingMWOZ attraction | 484 | 43 | 42 | 295 | 27 | 26 | 975 | 86 | 85
missingsgd restaurants | 2686 | 278 | 616 | 1720 | 166 | 386 | 5756 | 606 | 1354
missingsgd media | 1411 | 230 | 458 | 988 | 167 | 324 | 3114 | 502 | 1005
missingsgd events | 4881 | 598 | 989 | 3241 | 389 | 590 | 10555 | 1317 | 2197
missingsgd music | 1892 | 275 | 556 | 1506 | 224 | 464 | 4040 | 597 | 1215
missingsgd movies | 1665 | 181 | 52 | 996 | 114 | 44 | 3760 | 420 | 126
missingsgd flights | 4766 | 1041 | 1756 | 2571 | 627 | 982 | 10429 | 2244 | 3833
missingsgd ridesharing | 652 | 85 | 187 | 377 | 48 | 107 | 1448 | 188 | 418
missingsgd rentalcars | 1510 | 250 | 469 | 865 | 153 | 280 | 3277 | 538 | 1009
missingsgd buses | 1862 | 331 | 653 | 1102 | 218 | 412 | 4050 | 709 | 1393
missingsgd hotels | 3237 | 394 | 948 | 1997 | 243 | 597 | 6983 | 858 | 2053
missingsgd services | 3328 | 360 | 926 | 2225 | 230 | 611 | 7262 | 803 | 2016
missingsgd homes | 2098 | 170 | 533 | 1312 | 96 | 338 | 4519 | 394 | 1158
missingsgd banks | 1188 | 139 | 293 | 723 | 84 | 181 | 2599 | 319 | 667
missingsgd calendar | 592 | 115 | 236 | 397 | 65 | 133 | 1313 | 246 | 501
missingsgd alarm | 212 | 34 | 91 | 221 | 30 | 74 | 580 | 82 | 198
missingsgd weather | 196 | 32 | 80 | 123 | 23 | 59 | 433 | 70 | 169
missingsgd travel | 186 | 23 | 48 | 121 | 14 | 30 | 420 | 53 | 106
missingsgd payment | 227 | 21 | 51 | 143 | 14 | 32 | 497 | 44 | 113
missingsgd trains | 300 | 73 | 128 | 149 | 43 | 66 | 668 | 158 | 274
Total | 147254 | 18604 | 22946 | 93273 | 11722 | 14429 | 347885 | 44047 | 53641
Table 7: Detained data amounts for each domian and module

##  Appendix E Experimental Details

###  E.1 CIFAR-10 Dataset

The CIFAR-10 dataset Krizhevsky et al. (2009) consists of 60000 32x32 color
images in 10 classes, which can be split into 50000 training images and 10000
test images. There are 10 different classes in the CIFAR-10 dataset, including
cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. Each class
contains 6000 images.

###  E.2 CNN backbone

We employ a simple convolutional neural network, including 2$\times$2D CNN
layers with pooling layers, then a ReLU activation function and 2$\times$
linear layer are added to predict the labels.

In order to obtain Fig. 3, the CNN network is trained by an SGD optimizer with
a momentum of 0.9. The learning rate is $1\times e^{-3}$ and 50 training
epochs are repeated. Then we calculate the contribution of training data to
test data regarding different categories.


# A Lightweight Calibrated Simulation Enabling Efficient Offline Learning for
Optimal Control of Real Buildings

Judah Goldfeder  Google, Columbia UniversityUnited States<EMAIL_ADDRESS>and  John Sipple  GoogleUnited States<EMAIL_ADDRESS>
(2023)

###### Abstract.

Modern commercial Heating, Ventilation, and Air Conditioning (HVAC) devices
form a complex and interconnected thermodynamic system with the building and
outside weather conditions, and current setpoint control policies are not
fully optimized for minimizing energy use and carbon emission. Given a
suitable training environment, a Reinforcement Learning (RL) model is able to
improve upon these policies, but training such a model, especially in a way
that scales to thousands of buildings, presents many real world challenges. We
propose a novel simulation-based approach, where a customized simulator is
used to train the agent for each building. Our open-source
simulator111Available online: https://github.com/google/sbsim is lightweight
and calibrated via telemetry from the building to reach a higher level of
fidelity. On a two-story, 68,000 square foot building, with 127 devices, we
were able to calibrate our simulator to have just over half a degree of drift
from the real world over a six-hour interval. This approach is an important
step toward having a real-world RL control system that can be scaled to many
buildings, allowing for greater efficiency and resulting in reduced energy
consumption and carbon emissions.

HVAC Optimization, Simulation, Reinforcement Learning

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: RLEM ’23; November 11–12, 2023; Istanbul, Turkey

##  1\. Introduction

Energy optimization and management in commercial buildings is a very important
problem, whose importance is only growing with time. Buildings account for 39%
of all US carbon emissions (Lu and Lai, 2020). Reducing those emissions by
even a small percentage can have a significant effect, especially in more
extreme climates. Our contributions include a highly customizable and scalable
HVAC and building simulator, a rapid configuration method to customize the
simulator to a particular building, a calibration method to improve this
fidelity using real world data, and an evaluation method to measure the
simulator fidelity.

Overview of the HVAC problem Most office buildings are equipped with advanced
HVAC devices, like Variable Air Volume (VAV) devices, Hot Water Systems (HWS),
Air Conditioner (AC) and Air Handlers that are configured and tuned by the
engineers, manufacturers, installers, and operators to run efficiently with
the device’s local control loops (McQuiston et al., 2023). However,
integrating multiple HVAC devices from diverse vendors into a building
“system” requires technicians to program fixed operating conditions for these
units, which may not be optimal for every building and every potential weather
condition. If existing setpoint control policies are not optimal under all
conditions, the possibility exists that an ML model may be trained to
continuously tune a small number of setpoints to achieve greater energy
efficiency and reduced carbon emission (Wei et al., 2017).

RL for HVAC systems We define the state of the office building $S_{t}$ at time
$t$ as a fixed length vector of measurements from sensors on the building’s
devices, such as a specific VAV’s zone air temperature, gas meter’s flow rate,
etc. The action on the building $A_{t}$ is a fixed-length vector of device
setpoints selected by the agent at time $t$, such as the boiler’s supply water
temperature setpoint, etc. We also define a custom feedback signal, or reward,
that indicates the quality of taking an action in the current state, as a
weighted sum of negative cost functions for carbon emission, energy
consumption, and zone-level setpoint deviation. The goal of RL is to maximize
the expected long-term cumulative reward (Sutton and Barto, 2018). In order to
converge to the optimal policy, the agent requires many training iterations,
making training directly on the building from scratch inefficient and
impracticable. Therefore, it is necessary to train in an offline sandbox
environment that adequately emulates the dynamics of the building before being
deployed on the actual building.

##  2\. Related Work

Considerable attention has been paid to HVAC control in recent years, and a
growing portion of that has considered how RL and its various associated
algorithms can be leveraged (Gao and Wang, 2023; Mason and Grijalva, 2019;
Wang et al., 2023; Yu et al., 2021, 2020). A central requirement in RL is the
offline environment that trains the RL agent. Several methods for this have
been proposed, largely falling under three broad categories.

Data-driven Emulators: Some works attempt to learn a dynamics model as a
multivariate regression model from real world data (Zhang et al., 2019; Zou et
al., 2020), often using recurrent neural network architectures, such as LSTMs
(Sendra-Arranz and Gutiérrez, 2020; Velswamy et al., 2017; Zhuang et al.,
2023). The difficulty here is that data-driven models often do not generalize
well to circumstances outside the training distribution, especially since they
are not physics based.

Physics-based Simulation: EnergyPlus (Crawley et al., 2001), a high-fidelity
simulator developed by the Department of Energy, is commonly used (Azuatalam
et al., 2020; Wani et al., 2019; Wei et al., 2017; Zhao et al., 2015), but
suffers from the scalability issues outlined above.

Offline RL: The second approach is to train the agent directly from the
historical real world data, without ever producing an interactive environment
(Blad et al., 2022; Chen et al., 2020, 2023). While the real world data is
obviously of high accuracy and quality, this presents a major challenge, since
the agent cannot take actions in the real world and interact with any form of
an environment, severely limiting its ability to improve over the baseline
policy that produced the real world data (Levine et al., 2020).

To overcome the limitations of each of the above three methods, some work has
proposed a hybrid approach (Balali et al., 2023; Zhao et al., 2021), and this
is the category our work falls under. Our approach uses a physics-based
simulator that achieves an ideal balance between speed and fidelity, which is
sufficient to train an effective control agent off-line.

##  3\. A simple calibrated Simulation

Design Considerations A fundamental tradeoff when designing a simulator is
speed versus fidelity. Fidelity is the simulator’s ability to reproduce the
building’s true dynamics that affect the optimization process. Speed refers to
both simulator configuration time, and agent training time, i.e., the time
necessary for the agent to optimize its policy using the simulator. Every
building is unique, due to its physical layout, equipment, and location. Fully
customizing a high fidelity simulation to a specific target building requires
nearly exhaustive knowledge of the building structure, materials, location,
etc., some of which is unknowable. Also, the configuration time required for
high-fidelity simulations limits their utility for deploying RL-based
optimization to many buildings. Our goal is to develop a method for applying
RL at scale to commercial buildings. To this end, we must have a simulated
environment to train the agent, as real world training is not possible. In
order to scale to many buildings it must be easy for our simulator to be
configured to a new building, with enough fidelity to the real world to be
useful. We note that while RL is the control method proposed in this work, our
calibrated simulator is useful as well for other controller methods such as
Model Predictive Control (Kouvaritakis and Cannon, 2016). To meet these
requirements we designed a lightweight simulator based on Finite Differences
approximation of heat exchange, with a novel automated procedure to go from
building floor plans to a custom simulator with little manual effort. For
improved fidelity, we designed a calibration and evaluation pipeline based on
real telemetry.

Building Thermal Model We developed our simulation around our abstract thermal
model for office buildings, shown in Figure 1. A building consists of
conditioned zones, where the measured zone temperature, $T_{z}$, should be
within upper and lower setpoints, $\hat{T}_{z,max}$ and $\hat{T}_{z,min}$.
Thermal power for heating or cooling is supplied to each zone, $\dot{Q}_{s}$,
and recirculated from the zone, $\dot{Q}_{r}$ from the HVAC system, with
additional thermal exchange $\dot{Q}_{z}$ from walls, doors, etc. The Air
Handler supplies the building with air at supply air temperature setpoint
$\hat{T}_{s}$ drawing fresh air, $\dot{m}_{amb}$ at ambient temperatures,
$T_{amb}$ and returning exhaust air $\dot{m}_{exhaust}$ at temperature
$T_{exhaust}$ to the outside using intake and exhaust fans: $\dot{W}_{a,in}$
and $\dot{W}_{a,out}$. Some of the return air can be recirculated,
$\dot{m}_{recirc}$. Central air conditioning is achieved with a chiller and
pump that joins a refrigeration cycle to the supply air, consuming electrical
energy for the AC compressor $\dot{W}_{c}$ and coolant circulation,
$\dot{W}_{c,p}$. The hot water cycle consists of a boiler that maintains the
supply water temperature at $T_{b}$ heated by natural gas power $\dot{Q}_{b}$,
and a pump that circulates hot water through the building, with electrical
power $\dot{W}_{b,p}$. Supply air is delivered to the zones through VAV
devices.

We selected water supply temperature $\hat{T}_{b}$ and the air handler supply
temperature $\hat{T}_{s}$ as agent actions because they affect the balance of
electricity and natural gas consumption, they affect multiple device
interactions, and they affect occupant comfort.

Finite Differences Approximation The diffusion of thermal energy in time and
space of the building can be approximated using the method of Finite
Differences (FD)(Sparrow, 1993; Lomax et al., 2002), and applying an energy
balance. This method divides each floor of the building into a grid of three-
dimensional control volumes and applies thermal diffusion equations to
estimate the temperature of each control volume. By assuming each floor is
thermally isolated, (i.e., no heat is transferred between floors), we can
simplify the three-spatial dimensions into a spatial two-dimensional heat
transfer problem. Each control volume is a narrow volume bounded horizontally,
parameterized by $\Delta x^{2}$, and vertically by the height of the floor.
The energy balance, shown below, is applied to each discrete control volume in
the FD grid, and consists of the following components: (a) the thermal
exchange across each face of the four participating faces control volume via
conduction or convection $Q_{1}$, $Q_{2}$, $Q_{3}$, $Q_{4}$, (b) the change in
internal energy over time in the control volume $Mc\frac{\Delta T}{\Delta t}$,
and (c) an external energy source that enables applying local thermal energy
from the HVAC model only for those control volumes that include an airflow
diffuser, $Q_{ext}$. The equation is
$Q_{ext}+Q_{1}+Q_{2}+Q_{3}+Q_{4}=Mc\frac{\Delta T}{\Delta t}$, where $M$ is
the mass and $c$ is the heat capacity of the control volume, $\Delta T$ is the
temperature change from the prior timestep and $\Delta t$ is the timestep
interval.

Figure 1.  Building Thermal Model

The thermal exchange in (a) is calculated using Fourier’s law of steady
conduction in the interior control volumes (walls and interior air),
parameterized by the conductivity of the volume, and the exchange across the
exterior faces of control volumes is calculated using the forced convection
equation, parameterized by the convection coefficient, which approximates
winds and currents surrounding the building. The change in internal energy (b)
is parameterized by the density, and heat capacity of the control volume.
Finally, the thermal energy associated with the VAV (c) is equally distributed
to all associated control volumes that have a diffuser.

Thermal diffusion within the building is mainly accomplished via forced or
natural convection currents, which can be notoriously difficult to estimate
accurately. We note that heat transfer using air circulation is effectively
the exchange of air mass between control volumes, which we approximate by a
randomized shuffling of air within zones, parameterized by a shuffle
probability.

HVAC Model The HVAC system is modeled as an energy balance with a lossy
thermal exchange between the hot water and air circuits, evaluating
temperature differences, and air and water flow rates with efficiency
parameters. Aggregate heating and cooling demands at the AC and HWS are
governed by thermostats that create demand when its zone temperature is near
zone air temperature setpoints. Special control volumes associated with air
diffusers allow thermal energy exchange between the HVAC model and the FD
grid, $\dot{Q}_{s}$.

Simulator Configuration For RL to scale to many buildings, it is critical to
be able to easily and rapidly configure the simulator to any arbitrary
building. To configure the simulator, we require floorplans, and HVAC
metadata. We preprocess the detailed floorplan blueprints of the building, and
extract a grid that gives us an approximate placement of walls. We also employ
a user interface to label the location of each HVAC device on the floorplan
grid. We tested this pipeline on our pilot building, which consisted of two
floors with combined surface area of 68,000 $\text{ft}^{2}$, and 127 HVAC
devices. Given floorplans and HVAC layout information, a single technician was
able to generate a fully specified simulation in under 3 hours that matched
the real building in every device and room.

Simulator Calibration and Evaluation In order to calibrate the simulator to
the real world using data, we must have a metric with which to evaluate our
simulator, and an optimization method to improve our simulator on this metric.
We proposed a novel evaluation procedure, based on $N$-step prediction. Each
iteration step of our simulator was designed to represent a five minute
interval, and our real world data is also sampled in five minute intervals. To
evaluate the simulator, we take a chunk of real data of $N$ observations. We
then initialize the simulator so that its initial state matches that of the
starting observation, and run the simulator for $N$ steps, replaying the same
HVAC policy as was used in the real world observations. Next, we calculate our
simulation fidelity metric, which is the mean absolute error of the measured
zone air temperatures, at the $N$th timestep. More formally, we define the
spatial Mean Absolute Error (MAE) of $Z$ zones at timestep $t$ as
$\epsilon_{t}=\frac{1}{Z}\sum^{Z}_{z=1}|T_{real,t,z}-T_{sim,t,z}|$, where
$T_{real,t,z}$ is the measured zone air temperature for zone $z$ at timestamp
$t$, and $T_{sim,t,z}=\frac{1}{C_{z}}\sum^{C_{z}}_{c=1}T_{t,c}$ is the mean
temperature of all control volumes $C_{z}$ in zone $z$ at time $t$. Thus, to
evaluate the simulator on $N$-step prediction, we run the simulator for
timesteps $0$ to $N-1$, and calculate the above metric for $t=N-1$.

Simulator Calibration Once we defined our simulation fidelity metric, we can
minimize the error by searching over the physical parameters that affect the
simulation response. The variables tuned during the parameter search included
the following: (a) the forced convection coefficient quantifying outside wind
and air currents against the building exterior surfaces; (b) the thermal
conductivity, heat capacity, and density of exterior and interior walls; and
(c) the shuffle probability that controls how likely an air volume will be
exchanged with another within the zone to approximate the internal air
circulation and interior forced convection.

##  4\. Experiment Results

We now demonstrate the results of how our simulator, when tuned and
calibrated, is able to make real-world predictions.

Experiment Setup To test out our simulator, we obtained telemetry recordings
from our pilot building, a commercial office building located in Mountain
View, California. The building has two stories with a combined surface area of
68,000 square feet, and has 127 HVAC devices. We obtained floor plan
blueprints and used them to configure a customized simulator for the building,
a process that took a single human less than three hours to complete.

Calibration Data To tune and evaluate our simulator, we took three, non-
overlapping intervals of telemetry: (a) Thursday, July 6 2023, from 1:40 AM to
7:40 AM ($N$ = 72), (b) Friday, July 7 2023, from 1:40 AM to 7:40 AM ($N$ =
72), and (c) from Friday, July 7 2023, from 11:40 AM to 5:40 PM ($N$ = 72).
The first telemetry interval, (a), was used to tune the simulator for
calibration. The second and third telemetry intervals, (b) and (c), were used
for validation.

Table 1. Hyperparameter ranges and chosen values Hyperparameter | range | best
---|---|---
exterior convection coefficient ($W/m^{2}/K$) | 5 - 800 | 800
exterior wall conductivity ($W/m/K$) | 0.01 - 1 | 0.01
exterior wall density ($kg/m^{3}$) | 1 - 3000 | 2748
exterior wall heat capacity $(J/Kg/K)$ | 100 - 2500 | 2500
interior wall conductivity ($W/m/K$) | 5 - 800 | 780
interior wall density ($kg/m^{3}$) | 0.5 - 1500 | 0.5
interior wall heat capacity $(J/Kg/K)$ | 500 - 1500 | 500
interior air volume shuffle probability | 0 - 1 | 1

Calibration Procedure We tuned for 100 iterations on our simulator. The
parameters varied, the sweep range limits, and the values found that minimized
the calibration metric are shown in Table 1. Because of our focus on speed, we
can similate a five minute interval in about 15 seconds on one CPU, allowing
for very rapid tuning. We reviewed the parameters that yielded the lowest
simulation error from calibration. Densities, heat capacities, and
conductivities plausibly matched common interior and exterior building
materials. However, the external convection coefficient was higher than under
normal weather conditions, and likely is compensating for radiative losses and
gains, which were not directly simulated.

Calibration Results Table 2 shows the $N$-step prediction fidelity, as MAE,
over a six-hour prediction window ($N=72$). We calculated the spatial mean
absolute temperature error, as defined above. We also present a second metric,
the median spatial temperature error. This was not used in the tuning process,
but gives us some insight into how well the calibration process is performing.
As indicated in Table 2, our tuning procedure drifts only $0.6^{\circ}$ on
average over a 6 hour period on the tuning set, and we get good
generalization: almost identical error for the same time period of a different
day. It should be noted that an uncalibrated model had a much larger error of
$2.1^{\circ}$. Furthermore, the median temperature in these two cases is near
zero, indicating that even despite drift, our prediction remains a mostly
unbiased estimator. For the different time of day scenario, we observe a
larger drift of $1.2^{\circ}$, and the median is shifted as well, indicated
that tuning on one time of day is not as useful when making predictions on
another.

Table 2. Mean and Median Absolute Error on $N=72$ prediction window Metric | Tuning Data (a) | Validation (b) | Validation (c)
---|---|---|---
MAE | 0.64 ${}^{\circ}C$ | 0.63 ${}^{\circ}C$ | 1.18 ${}^{\circ}C$
Median | 0.01 ${}^{\circ}C$ | 0.18 ${}^{\circ}C$ | 0.98 ${}^{\circ}C$
Figure 2.  Temperature Drift Over 6 hours, from validation (b).

Visualizing Temperature Drift Over Time Figure 2 illustrates temperature drift
over time. At each time step, we plot the real world temperatures in a blue
boxplot and the simulator ones in orange.

Visualizing Spatial Errors Figure 3 is a heatmap of the spatial temperature
difference across both floors after six hours. Red regions indicate the
simulator was warmer than the validation data, blue regions indicate cooler,
and white regions had no temperature difference. The ring of blue around the
building indicates that our simulator is too cold on the perimeter, which
implies that the heat exchange with the outside is happening more rapidly than
it would in the real world. The inside of the building remains red, which
means that despite the simulator perimeter being cooler than the real world,
the inside is warmer. The white band region running along the perimeter is
where the temperature achieved a perfect fit with the validation data. The
interior red regions are due to slight errors in modeling interior heat
exchange, and the blue regions indicate errors due to high thermal losses with
the outside. We suspect these errors are largely due to assuming thermal
isolation between floors and assuming no radiative exchange with the outside.
Consequently, the calibration process compensates with a high convection
coefficient. Internal thermal exchange seems slower than the real world, and
the calibration compensates with more rapid outside thermal exchange.

Figure 3. Heatmap representing the temperature difference between the
simulator and the real world after 6 hours, with red indicating the simulator
is hotter, and blue colder.

##  5\. Discussion and Future Work

Developing a simulator is an important step toward achieving our objective of
training and deploying an RL agent on the actual building. We already have a
real building that we have the ability to control, and using our pipeline
produced a calibrated simulator that perfectly matches all of its devices,
setpoints and sensors. Our future goal, showing how an agent trained on this
simulator is able to produce a useful RL policy on the real building, will
require a few more components. In future live trials on the real building, we
will train an RL agent on our simulator, and measure its performance in the
real world. Care must be taken in this process, as RL agents are prone to
learning false patterns from the simulator (Arroyo et al., 2022; Liu and
Henze, 2006).

We also have outlined several approaches that we intend to use to further
improve our simulator and fine tuning process

  1. (1)

Tuning on data from multiple seasons, various times of day, and longer time
windows, to improve generalizability.

  2. (2)

Updating our simulation by adding in a radiative heat transfer model that
enables solar gains during the daytime and losses at night.

  3. (3)

Expanding the action space to include AC static pressure and HWS differential
pressure that govern the air and water flow rates.

  4. (4)

Adding a data-driven predictor-corrector to improve fidelity.

We are optimistic that our novel simulation tuning process will allow us to
develop a useful and transferable, HVAC control solution, and we hope our work
will inspire further effort in this field.

## References

  * (1)
  * Arroyo et al. (2022) Javier Arroyo, Carlo Manna, Fred Spiessens, and Lieve Helsen. 2022.  Reinforced model predictive control (RL-MPC) for building energy management.  _Applied Energy_ 309 (2022), 118346.
  * Azuatalam et al. (2020) Donald Azuatalam, Wee-Lih Lee, Frits de Nijs, and Ariel Liebman. 2020.  Reinforcement learning for whole-building HVAC control and demand response.  _Energy and AI_ 2 (2020), 100020.
  * Balali et al. (2023) Yasaman Balali, Adrian Chong, Andrew Busch, and Steven O’Keefe. 2023.  Energy modelling and control of building heating and cooling systems with data-driven and hybrid models—A review.  _Renewable and Sustainable Energy Reviews_ 183 (2023), 113496.
  * Blad et al. (2022) Christian Blad, Simon Bøgh, and Carsten Skovmose Kallesøe. 2022\.  Data-driven offline reinforcement learning for HVAC-systems.  _Energy_ 261 (2022), 125290.
  * Chen et al. (2020) Bingqing Chen, Zicheng Cai, and Mario Bergés. 2020\.  Gnu-rl: A practical and scalable reinforcement learning solution for building hvac control using a differentiable mpc policy.  _Frontiers in Built Environment_ 6 (2020), 562239.
  * Chen et al. (2023) Liangliang Chen, Fei Meng, and Ying Zhang. 2023.  Fast Human-in-the-loop Control for HVAC Systems via Meta-learning and Model-based Offline Reinforcement Learning.  _IEEE Transactions on Sustainable Computing_ (2023).
  * Crawley et al. (2001) Drury B Crawley, Linda K Lawrie, Frederick C Winkelmann, Walter F Buhl, Y Joe Huang, Curtis O Pedersen, Richard K Strand, Richard J Liesen, Daniel E Fisher, Michael J Witte, et al. 2001\.  EnergyPlus: creating a new-generation building energy simulation program.  _Energy and buildings_ 33, 4 (2001), 319–331.
  * Gao and Wang (2023) Cheng Gao and Dan Wang. 2023.  Comparative study of model-based and model-free reinforcement learning control performance in HVAC systems.  _Journal of Building Engineering_ 74 (2023), 106852.
  * Kouvaritakis and Cannon (2016) Basil Kouvaritakis and Mark Cannon. 2016.  Model predictive control.  _Switzerland: Springer International Publishing_ 38 (2016).
  * Levine et al. (2020) Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. 2020\.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ (2020).
  * Liu and Henze (2006) Simeng Liu and Gregor P Henze. 2006.  Experimental analysis of simulated reinforcement learning control for active and passive building thermal storage inventory: Part 2: Results and analysis.  _Energy and buildings_ 38, 2 (2006), 148–161.
  * Lomax et al. (2002) Harvard Lomax, Thomas H Pulliam, David W Zingg, and TA Kowalewski. 2002.  Fundamentals of computational fluid dynamics.  _Appl. Mech. Rev._ 55, 4 (2002), B61–B61.
  * Lu and Lai (2020) Mengxue Lu and Joseph Lai. 2020.  Review on carbon emissions of commercial buildings.  _Renewable and Sustainable Energy Reviews_ 119 (2020), 109545.
  * Mason and Grijalva (2019) Karl Mason and Santiago Grijalva. 2019.  A review of reinforcement learning for autonomous building energy management.  _Computers & Electrical Engineering_ 78 (2019), 300–312.
  * McQuiston et al. (2023) Faye C McQuiston, Jerald D Parker, Jeffrey D Spitler, and Hessam Taherian. 2023.  _Heating, ventilating, and air conditioning: analysis and design_.  John Wiley & Sons.
  * Sendra-Arranz and Gutiérrez (2020) R Sendra-Arranz and A Gutiérrez. 2020.  A long short-term memory artificial neural network to predict daily HVAC consumption in buildings.  _Energy and Buildings_ 216 (2020), 109952.
  * Sparrow (1993) E M Sparrow. 1993\.  Heat Transfer: Conduction [Lecture Notes].
  * Sutton and Barto (2018) Richard S Sutton and Andrew G Barto. 2018.  _Reinforcement learning: An introduction_.  MIT press.
  * Velswamy et al. (2017) Kirubakaran Velswamy, Biao Huang, et al. 2017\.  A Long-Short Term Memory Recurrent Neural Network Based Reinforcement Learning Controller for Office Heating Ventilation and Air Conditioning Systems.  (2017).
  * Wang et al. (2023) Marshall Wang, John Willes, Thomas Jiralerspong, and Matin Moezzi. 2023.  A Comparison of Classical and Deep Reinforcement Learning Methods for HVAC Control.  _arXiv preprint arXiv:2308.05711_ (2023).
  * Wani et al. (2019) Mubashir Wani, Akshya Swain, and Abhisek Ukil. 2019\.  Control strategies for energy optimization of HVAC systems in small office buildings using energyplus tm. In _2019 IEEE Innovative Smart Grid Technologies-Asia (ISGT Asia)_. IEEE, 2698–2703.
  * Wei et al. (2017) Tianshu Wei, Yanzhi Wang, and Qi Zhu. 2017.  Deep reinforcement learning for building HVAC control. In _Proceedings of the 54th annual design automation conference 2017_. 1–6.
  * Yu et al. (2021) Liang Yu, Shuqi Qin, Meng Zhang, Chao Shen, Tao Jiang, and Xiaohong Guan. 2021\.  A review of deep reinforcement learning for smart building energy management.  _IEEE Internet of Things Journal_ 8, 15 (2021), 12046–12063.
  * Yu et al. (2020) Liang Yu, Yi Sun, Zhanbo Xu, Chao Shen, Dong Yue, Tao Jiang, and Xiaohong Guan. 2020.  Multi-agent deep reinforcement learning for HVAC control in commercial buildings.  _IEEE Transactions on Smart Grid_ 12, 1 (2020), 407–419.
  * Zhang et al. (2019) Chi Zhang, Sanmukh R Kuppannagari, Rajgopal Kannan, and Viktor K Prasanna. 2019.  Building HVAC scheduling using reinforcement learning via neural network based model approximation. In _Proceedings of the 6th ACM international conference on systems for energy-efficient buildings, cities, and transportation_. 287–296.
  * Zhao et al. (2021) Huan Zhao, Junhua Zhao, Ting Shu, and Zibin Pan. 2021\.  Hybrid-model-based deep reinforcement learning for heating, ventilation, and air-conditioning control.  _Frontiers in Energy Research_ 8 (2021), 610518.
  * Zhao et al. (2015) Jie Zhao, Khee Poh Lam, B Erik Ydstie, and Omer T Karaguzel. 2015\.  EnergyPlus model-based predictive control within design–build–operate energy information modelling infrastructure.  _Journal of Building Performance Simulation_ 8, 3 (2015), 121–134.
  * Zhuang et al. (2023) Dian Zhuang, Vincent JL Gan, Zeynep Duygu Tekler, Adrian Chong, Shuai Tian, and Xing Shi. 2023\.  Data-driven predictive control for smart HVAC system in IoT-integrated buildings with time-series forecasting and reinforcement learning.  _Applied Energy_ 338 (2023), 120936.
  * Zou et al. (2020) Zhengbo Zou, Xinran Yu, and Semiha Ergan. 2020.  Towards optimal control of air handling units using deep reinforcement learning and recurrent neural network.  _Building and Environment_ 168 (2020), 106535.