# sDPO: Don’t Use Your Data All at Once

Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim
Chanjun Park†

Upstage AI, South Korea
{kdahyun, eddie, ynot, choco_9966, yoonsoo, limerobot,
<EMAIL_ADDRESS>
###### Abstract

††† Corresponding Author

As development of large language models (LLM) progresses, aligning them with
human preferences has become increasingly important. We propose stepwise DPO
(sDPO), an extension of the recently popularized direct preference
optimization (DPO) for alignment tuning. This approach involves dividing the
available preference datasets and utilizing them in a stepwise manner, rather
than employing it all at once. We demonstrate that this method facilitates the
use of more precisely aligned reference models within the DPO training
framework. Furthermore, sDPO trains the final model to be more performant,
even outperforming other popular LLMs with more parameters.

sDPO: Don’t Use Your Data All at Once

Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim
Chanjun Park† Upstage AI, South Korea {kdahyun, eddie, ynot, choco_9966,
yoonsoo, limerobot<EMAIL_ADDRESS>

##  1 Introduction

Large language models (LLMs) have revolutionized the field of natural language
processing (NLP) through a training process that includes pre-training,
supervised fine-tuning, and alignment tuning, with the latter ensuring the
safety and usefulness of the model. Thus, reinforcement learning techniques
Christiano et al. (2017); Bai et al. (2022), such as proximal policy
optimization (PPO) Schulman et al. (2017), are key in this alignment phase,
despite their complexity.

To address the complicated nature of reinforcement learning in LLM training,
direct preference optimization (DPO) Rafailov et al. (2023), among other
methods Yuan et al. (2023); Dong et al. (2023), have been popularized for its
simplicity and effectiveness. DPO involves curating preference datasets using
human or strong AI (e.g., GPT-4 OpenAI (2023)) judgement to select chosen and
rejected responses to questions. These datasets are used to train LLMs by
comparing log probabilities of chosen versus rejected answers. However,
obtaining these probabilities can be challenging with proprietary models like
GPT-4, since they do not offer log probabilities for inputs.

Model | Reference Model | H4
---|---|---
Mistral-7B-OpenOrca | N/A | 65.84
Mistral-7B-OpenOrca + DPO | SFT Base | 68.87
Mistral-7B-OpenOrca + DPO | SOLAR-0-70B | 67.86
Mistral-7B-OpenOrca + DPO | Intel-7B-DPO | 70.13
OpenHermes-2.5-Mistral-7B | N/A | 66.10
OpenHermes-2.5-Mistral-7B + DPO | SFT Base | 68.41
OpenHermes-2.5-Mistral-7B + DPO | SOLAR-0-70B | 68.90
OpenHermes-2.5-Mistral-7B + DPO | Intel-7B-DPO | 69.72

Table 1: DPO results in terms of H4 scores for Mistral-7B-OpenOrca and
OpenHermes-2.5-Mistral-7B with different reference models. The best results
for each SFT base model are shown in bold.

Thus, in most practical scenarios, the reference model is simply set as the
base SFT model Tunstall et al. (2023); Intel (2023b); Ivison et al. (2023),
which is a much weaker alternative with potentially misaligned preferences.
This reference model acts as a lower bound in DPO, i.e., the target model is
optimized to be at least as aligned as the reference model. Thus, we argue
that a reference model that is already more aligned will serve as a better
lower bound for DPO training, which would be beneficial for the alignment
tuning. One option would be to utilize the plethora of open source models
Tunstall et al. (2023); Ivison et al. (2023) that have already undergone
alignment tuning.

Figure 1: Overview of sDPO where preference datasets are divided to be used in
multiple steps. The aligned model from the previous step is used as the
reference and target models for the current step. The reference model is used
to calculate the log probabilities and the target model is trained using the
preference loss of DPO at each step.

Note that the above may not be feasible due to the absence of such aligned
models, or the fact that it renounces control over the reference model,
leading to safety concerns. Instead, we propose ‘stepwise DPO’, named sDPO,
where we use the preference datasets (or subsets of a preference dataset) in a
step-by-step manner when undergoing DPO training. The aligned model in the
previous step is used as the reference model for the current step, which
results in utilizing a more aligned reference model (i.e., a better lower
bound). Empirically, we show that using sDPO results in a more performant
final aligned model as well.

While concurrent works (Yuan et al., 2024) that focus on an iterative pipeline
of generating new preference data have been proposed, our method focuses on
utilizing the currently available preference datasets. Thus, our approach is
complementary as sDPO can be easily applied to any preference data and further
combination with concurrent works would be an exciting future direction.

##  2 Methodology

###  2.1 Preliminary Investigation on Reference Models

To gauge the importance of using a well-aligned reference model in DPO, we
perform preliminary experiments of DPO training with the Ultrafeedback dataset
Cui et al. (2023) on Mistral-7B-OpenOrca Lian et al. (2023) and
OpenHermes-2.5-Mistral-7B Teknium (2023) as the SFT base model, owing to their
excellent performance and small size. We compare the following reference
models: i) the SFT base model itself, same as the conventional DPO setup; ii)
SOLAR-0-70B Upstage (2023), a larger and much more performant model; and iii)
Intel-7B-DPO Intel (2023a), an already aligned reference model. The results
are summarized in Tab. 1.

As the table shows, using Intel-7B-DPO as the reference model results in the
best performance, even better than using SOLAR-0-70B, which is a much larger
model that was trained with more data. Thus, whether the reference model is
pre-aligned or not plays an important role in the resulting aligned model’s
performance. Unfortunately, it is not always possible to simply use a open
sourced pre-aligned model as the reference model due to technical and safety
concerns, i.e., such a model may not exist yet or can be susceptible to
various domain-specific harmfulness and fairness criteria. To remedy the
above, we propose sDPO, which uses more aligned reference models as a part of
the training framework.

###  2.2 Stepwise DPO

In sDPO, we propose to use the available preference datasets in a stepwise
manner instead of using them all at once. The comparison of the overall flow
of DPO and sDPO is presented in Fig. 1.

#### Reference model.

The reference model is used to calculate the log probabilities of the
preference dataset. For each step, only a subset of the total data is used and
the reference model is initialized as $M_{t-1}$, i.e, the aligned model from
the previous step. The initial reference model is set as $S$, the SFT base
model. This results in using a more aligned reference model than conventional
DPO.

#### Target model.

For $t>1$, the target model which is trained using the preference loss of DPO
in each step of sDPO is also initialized as $M_{t-1}$ instead of $S$. This
ensures that the final model trained with sDPO has been directly trained with
the same amount data as a model trained with DPO.

Model | Size | Type | H4 (Avg.) | ARC | HellaSwag | MMLU | TruthfulQA
---|---|---|---|---|---|---|---
SOLAR 10.7B + SFT + sDPO |  $\sim$ 11B | Alignment-tuned | 74.31 | 71.33 | 88.08 | 65.39 | 72.45
SOLAR 10.7B + SFT + DPO |  $\sim$ 11B | Alignment-tuned | 72.67 | 69.62 | 87.16 | 66.00 | 67.90
SOLAR 10.7B + SFT + sDPO Strat. |  $\sim$ 11B | Alignment-tuned | 72.56 | 69.20 | 87.27 | 65.96 | 67.81
Mixtral 8x7B-Instruct-v0.1 |  $\sim$ 47B | Alignment-tuned | 73.40 | 70.22 | 87.63 | 71.16 | 64.58
SOLAR-0-70B-16bit |  $\sim$ 70B | Instruction-tuned | 72.93 | 71.08 | 87.89 | 70.58 | 62.25
Qwen 72B |  $\sim$ 72B | Pretrained | 72.17 | 65.19 | 85.94 | 77.37 | 60.19
Yi 34B |  $\sim$ 34B | Pretrained | 70.72 | 64.59 | 85.69 | 76.35 | 56.23
SOLAR 10.7B + SFT |  $\sim$ 11B | Instruction-tuned | 69.51 | 67.32 | 85.96 | 65.95 | 58.80
Mistral 7B-Instruct-v0.2 |  $\sim$ 7B | Instruction-tuned | 69.27 | 63.14 | 84.88 | 60.78 | 68.26
Falcon 180B |  $\sim$ 180B | Pretrained | 68.57 | 69.45 | 88.86 | 70.50 | 45.47
Mixtral 8x7B-v0.1 |  $\sim$ 47B | Pretrained | 67.78 | 66.04 | 86.49 | 71.82 | 46.78
Llama 2 70B |  $\sim$ 70B | Pretrained | 67.35 | 67.32 | 87.33 | 69.83 | 44.92
Zephyr |  $\sim$ 7B | Alignment-tuned | 66.36 | 62.03 | 84.52 | 61.44 | 57.44
Qwen 14B |  $\sim$ 14B | Pretrained | 64.85 | 58.28 | 83.99 | 67.70 | 49.43
SOLAR 10.7B |  $\sim$ 11B | Pretrained | 64.27 | 61.95 | 84.60 | 65.48 | 45.04
Mistral 7B |  $\sim$ 7B | Pretrained | 62.40 | 59.98 | 83.31 | 64.16 | 42.15

Table 2: Performance comparison of applying sDPO (and ablated versions) to
SOLAR 10.7B + SFT against various top performing models. Size is shown in
units of billions of parameters and type is reported as one of {‘Pretrained’,
‘Instruction-tuned’, ‘Alignment-tuned’}. Models based on SOLAR 10.7B are shown
in purple color. The best scores in each column are shown in bold.

#### Intuitive explanation.

To gain a deeper understanding of sDPO, we rearrange the DPO loss from
Rafailov et al. (2023), as follows:

$\begin{aligned} &\mathcal{L}_{\text{DPO}}(\pi_{\theta},\pi_{ref})\\\
&=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\left[\log{\sigma\left(\beta\log{{\pi_{\theta}(y_{w}|x)\over\pi_{ref}(y_{w}|x)}}-\beta\log{{\pi_{\theta}(y_{l}|x)\over\pi_{ref}(y_{l}|x)}}\right)}\right]\\\
&=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\left[\log{\sigma\left(\beta\cdot(\gamma_{\pi_{\theta}}(x,y_{w},y_{l})-\gamma_{\pi_{ref}}(x,y_{w},y_{l})\right)}\right],\end{aligned}$

(1)

where $D$ is the preference dataset, $x$ is the question, $y_{w}$ and $y_{l}$
are the chosen and rejected answers respectively, $\theta$ is the learnable
parameters of the model, and
$\gamma_{\pi}(x,y_{w},y_{l})=\log{{\pi(y_{w}|x)\over\pi(y_{l}|x)}}$, i.e., the
logratio of the chosen and rejected samples w.r.t. the policy $\pi$. As
$\log\sigma(\cdot)$ is a monotonically increasing function and
$\gamma_{\pi_{ref}}$ is fixed before training, the minimization of
$\mathcal{L}_{\text{DPO}}(\pi_{\theta},\pi_{ref})$ leads to
$\gamma_{\pi_{\theta}}>\gamma_{\pi_{ref}}$ (on average). Thus,
$\gamma_{\pi_{ref}}$ can be understood as a lower bound defined by the
reference model, of which the target model is trained such that
$\gamma_{\pi_{\theta}}>\gamma_{\pi_{ref}}$. In sDPO, $\gamma_{\pi_{ref}}$
increases as the steps progress because the reference model that defines it is
more and more aligned. Hence, $\gamma_{\pi_{ref}}$ becomes a stricter lower
bound as the steps pass, inducing a curriculum learning from easy to hard
optimization tasks.

##  3 Experiments

###  3.1 Experimental Setup

#### Training details.

We use a supervised fine-tuned SOLAR 10.7B Kim et al. (2023) as our SFT base
model $S$ as it delivers excellent performance with its uncommon 10.7B size.
Further, the scarcity of 10.7B sized models leads to the absence of open
source models that can be adopted as reference models, making the usage of
sDPO more necessary. We use OpenOrca Mukherjee et al. (2023) ($\sim 12K$
samples) and Ultrafeedback Cleaned ($\sim 60K$ samples) Cui et al. (2023);
Ivison et al. (2023) as our preference datasets. The training hyper-parameters
closely follow that of Tunstall et al. (2023). We use two steps in sDPO, where
we use OpenOrca as dataset $D_{1}$ in the first step and Ultrafeedback Cleaned
as dataset $D_{2}$ in the second step.

#### Evaluation.

We utilize four of the six tasks in the HuggingFace Open LLM Leaderboard
Beeching et al. (2023): ARC Clark et al. (2018), HellaSWAG Zellers et al.
(2019), MMLU Hendrycks et al. (2020), TruthfulQA Lin et al. (2022). We also
report the average scores for the four tasks, which is denoted as H4.
Winogrande Sakaguchi et al. (2021) and GSM8K Cobbe et al. (2021) are excluded
to control the complexity of the experiments, i.e., we excluded generation
tasks in contrast to multiple choice tasks.

###  3.2 Main Results

Evaluation results for applying sDPO to the SFT base model, along with results
for other top-performing models are shown in Tab. 2. Comparing the pretrained-
only ‘SOLAR 10.7B’ to the instruction-tuned ‘SOLAR 10.7B + SFT’, we can see an
increase of $+5.24$ in terms of H4. Applying sDPO on SOLAR 10.7B + SFT further
increases the H4 score upto $74.31$, an improvement of $+4.80$. Notably,
‘SOLAR 10.7B + SFT + sDPO’ outperforms other larger models such as Mixtral
8x7B-Instruct-v0.1, despite the smaller number of parameters. This highlights
that effective alignment tuning could be the key to unlocking next level
performance for smaller LLMs. Further, applying sDPO results in substantially
higher score of $72.45$ for TruthfulQA, which shows the effectiveness of the
alignment tuning process.

###  3.3 Ablation Studies

We also report evaluation results for ablated models in Tab. 2. ‘SOLAR 10.7B +
SFT + DPO’ uses all the DPO data at once, i.e., $D_{1}+D_{2}$, same as the
conventional DPO training setup. ‘SOLAR 10.7B + SFT + sDPO Strat.’ uses
stratified sampling to sample $\sim 16.67\%$ of the data points from the union
of OpenOrca and Ultrafeedback Cleaned to form $D_{1}$ and use the remaining
$\sim 83.33\%$ as $D_{2}$ to mirror the dataset size of $D_{1}$ and $D_{2}$
used in SOLAR 10.7B + SFT + sDPO.

Comparing SOLAR 10.7B + SFT + DPO and SOLAR 10.7B + SFT + sDPO, we can see
that using sDPO over DPO results in a higher H4 score overall, with noticeable
improvements in ARC and TruthfulQA scores. Therefore, we believe sDPO could
function as a drop-in replacement for DPO training with better performance.
Looking at SOLAR 10.7B + SFT + sDPO and SOLAR 10.7B + SFT + sDPO Strat., we
see that the specific way of splitting the available DPO data into multiple
$D_{t}$ can also impact performance. We find that the natural split of using
different preference datasets as $D_{t}$ works best in our experiments. We
believe further exploration of how to define $D_{t}$ is an interesting
direction for future research.

###  3.4 Reference Models in sDPO

Figure 2: Mean $\gamma_{\pi_{ref}}$ on Ultrafeedback Cleaned dataset for
different reference models $S,M_{1},$ and $M_{2}$. Note that the x-axis is in
log scale.

#### Effectiveness of sDPO in terms of alignment tuning.

In Sec. 2.2, we explain that the reference models in sDPO are more aligned,
resulting in higher $\gamma_{\pi_{ref}}$, i.e., a stricter lower bound. We
verify the above empirically in Fig. 2 by comparing the mean
$\gamma_{\pi_{ref}}$ on the Ultrafeedback Cleaned dataset for the reference
models in steps 1 and 2 of sDPO, i.e., $S$ and $M_{1}$. Note that these two
models have not been trained on the aforementioned dataset. Using the SFT base
model $S$ as the reference model, the mean of $\gamma_{\pi_{ref}}$ is
$-38.60$. On the other hand, using the aligned model $M_{1}$ from step 1 of
sDPO as the reference model, the mean of $\gamma_{\pi_{ref}}$ is $-25.10$, an
increase of $13.50$ in log scale. Thus, a single step of sDPO greatly
increases $\gamma_{\pi_{ref}}$, which results in a more performant aligned
model as seen in Tab. 2.

#### Adopting open source models as reference models could be dangerous.

We also show mean $\gamma_{\pi_{ref}}$ of $M_{2}$, the aligned model from step
2 of sDPO. Unlike $S$ and $M_{1}$, $M_{2}$ is trained on the Ultrafeedback
Cleaned dataset, i.e., $M_{2}$ is used as a reference model on data that was
already used to train it. Note that such a case could happen commonly when
adopting various open source models as reference models. This is because the
datasets that were used in training those models are often unclear and could
overlap with the preference datasets unintentionally. Mean
$\gamma_{\pi_{ref}}$ of $M_{2}$ is $84.35$, which is staggeringly higher than
either $S$ or $M_{1}$. The strikingly high value for $M_{2}$ likely points to
overfitting of $M_{2}$ to the Ultrafeedback Cleaned dataset. This result
highlights the potential danger of merely adopting open source models as
reference models instead of using sDPO.

###  3.5 Target Model Initialization in sDPO

The target model in each step of sDPO is also initialized with $M_{t-1}$, the
aligned model from the last step. This ensures that the final model in sDPO
has undergone training with the same amount of data as the final model in DPO.
On the other hand, one concern of such design choice is that it may become
increasingly difficult to stabilize the training of the target model as the
steps progress, since it has already undergone training with a decreasing
learning rate schedule in the preceding steps. Thus, another option is to use
the initial SFT base model $S$ as the target model for all steps of sDPO.

However, as shown in Fig. 3, initializing the target model as $S$ results in a
much bigger initial loss than that of $M_{t-1}$, which could lead to an
unstable training. The main reason is that DPO training is usually done where
the reference and target models are the same. In contrast, initializing the
target model as $S$ creates a differential in the reference and target models,
which may be amplified as the steps progress. Thus, for stable training,
initializing the target model as $M_{t-1}$ was chosen for sDPO.

Figure 3: Loss curve comparison in step 2 of sDPO for different
initializations of the target model.

##  4 Conclusion

We propose sDPO where we use the preference data in a stepwise way instead of
all at once. We show that applying sDPO results in more performant models than
DPO in terms of H4 score. We also empirically exhibit that sDPO results in
more aligned reference models by comparing mean $\gamma_{\pi_{ref}}$.

## Limitations

While we have demonstrated the effectiveness of employing different datasets
in distinct stages of sDPO, identifying an optimal strategy for segmenting
more intricate DPO data collections remains an area for further exploration.
This task is particularly challenging due to the complexities within these
datasets. Our approach, while promising, necessitates a more deeper
understanding of dataset characteristics and their impact on the performance
of sDPO.

Furthermore, our experiments predominantly utilized SOLAR 10.7B models, driven
by the state-of-the-art performance at the time of experimentation along with
its unique 10.7 billion parameter size. The unique size of SOLAR 10.7B models
made the usage of sDPO more necessary as there are far fewer open source LLMs
that can be adopted as reference models.

Additionally, as with most research on LLMs, we operated within our
limitations in computational resources. Although this focus has yielded
significant insights, expanding our experimental framework to incorporate a
broader range of Large Language Models (LLMs) could potentially unveil more
comprehensive understanding of the strengths and limitations of sDPO. Such an
expansion would allow for a more robust comparison across different model
architectures and sizes, further enriching our findings.

Evaluating the efficacy of LLMs is an evolving challenge in the field. In our
study, we primarily employed tasks from the Huggingface Open LLM Leaderboard
as benchmarks for evaluation. While this provided comparative results, future
research could benefit from incorporating a wider array of tasks and
benchmarks. These could include tasks that judge actual human or strong AI
preference alignment. Such additional evaluation would not only enhance the
validity of our findings but also contribute to the broader discourse on LLM
assessment methodologies.

## Ethics Statement

In this study, we strictly adhered to ethical standards in the conduct of our
research. Our experiments were based entirely on open models and open
datasets, ensuring transparency and accessibility. We took meticulous care to
avoid any biases or data contamination, thereby maintaining the integrity of
our research process. The experimental environment was rigorously designed to
be objective, ensuring that all comparisons conducted were fair and impartial.
This approach reinforces the reliability and validity of our findings,
contributing positively to the field while upholding the highest ethical
standards. We confirmed that all the data used in our experiments were free of
licensing issues.

## References

  * Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_.
  * Bai et al. (2022) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022.  Constitutional ai: Harmlessness from ai feedback.  _arXiv preprint arXiv:2212.08073_.
  * Beeching et al. (2023) Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. 2023.  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * Cui et al. (2023) Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023.  Ultrafeedback: Boosting language models with high-quality feedback.  _arXiv preprint arXiv:2310.01377_.
  * Dong et al. (2023) Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023.  Raft: Reward ranked finetuning for generative foundation model alignment.  _arXiv preprint arXiv:2304.06767_.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations_.
  * Hernandez et al. (2021) Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. 2021.  Scaling laws for transfer.  _arXiv preprint arXiv:2102.01293_.
  * Intel (2023a) Intel. 2023a.  Intel/neural-chat-7b-v3-1.  https://huggingface.co/Intel/neural-chat-7b-v3-1.
  * Intel (2023b) Intel. 2023b.  Supervised fine-tuning and direct preference optimization on intel gaudi2.
  * Ivison et al. (2023) Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023.  Camels in a changing climate: Enhancing lm adaptation with tulu 2.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_.
  * Kim et al. (2023) Dahyun Kim, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim, Yungi Kim, Hyeonju Lee, Jihoo Kim, Changbae Ahn, Seonghoon Yang, Sukyung Lee, Hyunbyung Park, Gyoungjin Gim, Mikyoung Cha, Hwalsuk Lee, and Sunghun Kim. 2023.  Solar 10.7b: Scaling large language models with simple yet effective depth up-scaling.
  * Lian et al. (2023) Wing Lian, Bleys Goodson, Guan Wang, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023.  Mistralorca: Mistral-7b model instruct-tuned on filtered openorcav1 gpt-4 dataset.  https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca.
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.  Truthfulqa: Measuring how models mimic human falsehoods.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3214–3252.
  * Mukherjee et al. (2023) Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023.  Orca: Progressive learning from complex explanation traces of gpt-4.  _arXiv preprint arXiv:2306.02707_.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023.  Direct preference optimization: Your language model is secretly a reward model.  _arXiv preprint arXiv:2305.18290_.
  * Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021.  Winogrande: An adversarial winograd schema challenge at scale.  _Communications of the ACM_ , 64(9):99–106.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_.
  * Teknium (2023) Teknium. 2023.  teknium/openhermes-2.5-mistral-7b.  https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B.
  * Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. 2023.  Zephyr: Direct distillation of lm alignment.  _arXiv preprint arXiv:2310.16944_.
  * Upstage (2023) Upstage. 2023.  upstage/solar-0-70b-16bit.  https://huggingface.co/upstage/SOLAR-0-70b-16bit.
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_.
  * Yuan et al. (2024) Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024.  Self-rewarding language models.  _arXiv preprint arXiv:2401.10020_.
  * Yuan et al. (2023) Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023.  Rrhf: Rank responses to align language models with human feedback without tears.  _arXiv preprint arXiv:2304.05302_.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  Hellaswag: Can a machine really finish your sentence?  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4791–4800.
  * Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019.  Fine-tuning language models from human preferences.  _arXiv preprint arXiv:1909.08593_.

##  Appendix A Related Work

###  A.1 Large Language Models

Recent research has highlighted a "scaling law" in the field of context-based
language models Kaplan et al. (2020); Hernandez et al. (2021); Anil et al.
(2023), showing a proportional relationship between the size of the model plus
the training data and the resulting performance improvements. Consequently,
this has led to the advent of Large Language Models (LLMs). In contrast to
earlier models, LLMs can perform in-context learning, which includes abilities
such as zero-shot learning Radford et al. (2019) and few-shot learning Brown
et al. (2020), allowing them to adapt and perform tasks without the need for
weight adjustments. These emergent abilities of LLMs, absent in their smaller
counterparts, signal a significant evolution in language model capabilities
Wei et al. (2022).

###  A.2 Alignment Tuning

LLMs have been recognized to produce text that may seem linguistically
inconsistent to human interpreters because their pretraining is based not on
an understanding of human intentions but on a broad spectrum of domain-
specific knowledge, as indicated in Ziegler et al. (2019). In an effort to
rectify this issue and better mirror human intentions, prior research Ziegler
et al. (2019) has suggested the adoption of Reinforcement Learning with Human
Feedback (RLHF). RLHF seeks to refine the LLM’s output by constructing a
reward model that aligns with human preferences and applying reinforcement
learning to direct the LLM towards selections that garner the most favorable
reward metrics. This approach is intended to bolster the safety, decorum, and
general excellence of the responses produced by the LLM. Nonetheless, despite
showing promising results, RLHF is confronted with challenges, such as the
intricate handling of an extensive set of hyperparameters and the necessity to
amalgamate several models (policy, value, reward, and reference models).

To address these issues, there have been proposals for supervised fine-tuning
methodologies such as Rank Responses to align Human Feedback (RRHF) Yuan et
al. (2023), Reward rAnked FineTuning (RAFT) Dong et al. (2023), and Direct
Preference Optimization (DPO) Rafailov et al. (2023). These methods circumvent
the intricacies inherent in reinforcement learning and have been shown to
yield empirical results on par with RLHF. Notably, the DPO technique
straightforwardly encourages the LLM to favor positive responses and
discourage negative ones. DPO has been observed to yield performant learning
outcomes, in spite of its uncomplicated training procedure.

Concurrent to our work, Yuan et al. (2024) have developed an iterative
framework for generating new preference datasets and performing DPO training
on the resulting datasets. They empirically demonstrated the superiority of
their iterative framework in terms of AlpacaEval 2.0. In contrast, our work is
complementary to the above in the sense that we focus on utilizing the current
preference data and does not undergo new data generation. Thus, our method can
also be applied to Yuan et al. (2024) by changing the DPO training part to
using sDPO instead. We leave the above combination as an interesting future
work. Additionally, the evaluation used in Yuan et al. (2024) is also
different to ours as we utilize tasks from Open LLM Leaderboard whereas Yuan
et al. (2024) uses AlpacaEval 2.0.


# Learn to Focus: Hierarchical Dynamic Copy Network for Dialogue State
Tracking

Linhao Zhang, Houfeng Wang
MOE Key Lab of Computational Linguistics, Peking University, Beijing, 100871,
China
{zhanglinhao<EMAIL_ADDRESS>

###### Abstract

Recently, researchers have explored using the encoder-decoder framework to
tackle dialogue state tracking (DST), which is a key component of task-
oriented dialogue systems. However, they regard a multi-turn dialogue as a
flat sequence, failing to focus on useful information when the sequence is
long. In this paper, we propose a Hierarchical Dynamic Copy Network (HDCN) to
facilitate focusing on the most informative turn, making it easier to extract
slot values from the dialogue context. Based on the encoder-decoder framework,
we adopt a hierarchical copy approach that calculates two levels of attention
at the word- and turn-level, which are then re-normalized to obtain the final
copy distribution. A focus loss term is employed to encourage the model to
assign the highest turn-level attention weight to the most informative turn.
Experimental results show that our model achieves 46.76% joint accuracy on the
MultiWOZ 2.1 dataset.

##  1 Introduction

Much attention has recently been paid to dialogue state tracking (DST), which
aims to extract user goals at each turn of a dialogue and represents them
using slot-value pairs. These pairs are later used by the policy learning
module to decide the next system action.

With the development of deep learning, neural network-based methods have been
widely applied to DST. Many previous studies [Mrksic et al., 2016, Zhong et
al., 2018, Balaraman and Magnini, 2019] decompose DST into a series of
classification problems. They first score all possible slot-values pairs and
then choose the value with the highest probability as the predicted value for
that slot.

Some more recent work proposes generation-based DST [Wu et al., 2019]. Based
on the encoder-decoder framework [Sutskever et al., 2014], it directly
generates value for each slot. It is further enhanced with a soft copy
mechanism [See et al., 2017], which enables it to generate values using text
from the input source.

Figure 1: A DST example from MultiWOZ 2.1 dataset. Dialogue state is composed
of slot-value pairs. Note that throughout this paper, we use the term _slot_
to refer to the concatenation of _domain_ (e.g., hotel) and _slot_ (e.g.,
area), following the convention of MultiWOZ 2.1 dataset.

Despite its remarkable success, [Wu et al., 2019] simply concatenates
different dialogue turns and feeds them as a flat sequence to the model. For
example, it would organize the dialogue in Figure 1 in the form of
$M_{1}U_{1}M_{2}U_{2}M_{3}U_{3}...$. Obviously, when the dialogue contains
multiple turns, the sequence would become long, and copying the correct slot
values becomes especially difficult.

To this end, we propose a Hierarchical Dynamic Copy Network (HDCN) to
facilitate focusing on the most informative turns, making it easier to copy
slot values from the dialogue context. We define the _information turn_ of a
certain slot as the turn where its corresponding value appears, or can be
inferred. For example, in Figure 1, the _information turn_ for slot _hotel-
area_ is turn 3, where its value _east_ appears. Obviously, to generate value
for a certain slot, we should focus more on its _information turn_.

To achieve this, we use two levels of attention applied to the word- and turn-
level, which are then re-normalized and result in the final copy distribution.
We further add a focus loss term to encourage our model to assign higher turn-
level attention weight to the _information turn_. Notably, our turn
representation is calculated dynamically during decoding using the word-level
attention. This dynamic calculation leads to better turn representations for
two reasons: 1) it utilizes all the hidden states instead of only the last
one, and 2) it enables our model to form different turn representations for
different slots. As shown in Figure 1, when calculating word-level attention
over turn 4, the slot _hotel-book day_ would attend more to the word _Tuesday_
, and the slot _hotel-book stay_ would assign more attention to the word _2_.
In this way, the resulting turn-representations for these two slots also
become different.

We conduct experiments on the MultiWOZ 2.1 dataset. Results show that HDCN
brings over 1% (absolute) improvement in terms of joint accuracy, with better
interpretability. Note that our goal is not to beat the state-of-the-art
systems such as [Kim et al., 2020, Hosseini-Asl et al., 2020], as they are
often based on pretrained language model (e.g., BERT). Rather, we aim to offer
a lightweight improvement over the best system in the pre-BERT era [Wu et al.,
2019]. We believe although BERT-based models can achieve high accuracy, they
inevitably cost more inference time, making the DST component not practical in
real-word dialog systems.

##  2 Model

###  2.1 Task Definition

Given a set of machine and user utterance pairs in _T_ turns of dialogue
$X=\\{(M_{1},U_{1}),...,(M_{T},U_{T})\\}$, we aim to generate dialogue states
$B=\\{(S_{1},V_{1}),...,(S_{M},V_{M})\\}$ containing a series of slot-value
pairs $(S_{m},V_{m})$, where $M$ is the number of all the possible slots. In
the following subsection, we first give a brief description of generation-
based DST and introduce our work in detail.

###  2.2 Generation-Based DST

[Wu et al., 2019] first proposes to regard DST as a generation problem. Their
proposed model, _TRADE_ , achieves the best performance in the pre-BERT era.
It follows the encoder-decoder paradigm [Sutskever et al., 2014], and adopts a
soft-copy mechanism [See et al., 2017] to copy slot values from the input
source.

At encoding time, it uses a bi-directional gated recurrent units (GRU) [Cho et
al., 2014] to encode the dialogue history, which is organized as a flat
sequence $M_{1}U_{1}M_{2}U_{2}...M_{T}U_{T}$.

To generate value $V_{m}$ for slot $S_{m}$, it supplies the _summed embedding_
of $S_{m}$ as the first input to the decoder111More specifically, the _summed
embedding_ of the $m_{th}$ domain and slot. For example, if $S_{m}$ is _hotel-
area_ , then the summed embedding of $S_{m}$ is _Emb_(hotel) + _Emb_(area),
and decodes $M$ times independently, where $M$ is the number of all the
possible slots.

At decoding step $k$ , the decoder returns a hidden state $h_{k}$, which is
later mapped into the vocabulary space $P_{k}^{vocab}$ using embedding matrix
$E$. Formally,

$P_{k}^{vocab}=softmax\left(E\cdot(h_{k})^{\top}\right)$ (1)

At the same time, $h_{k}$ is used to calculate attention weights $a_{i}^{k}$
over the encoded dialogue history. $a_{i}^{k}$ and $P_{k}^{vocab}$ are then
weighted and summed to obtain the final word distribution [See et al., 2017]:

$P_{k}(w)=p_{\mathrm{gen}}P_{k}^{\mathrm{vocab}}(w)+\left(1-p_{\mathrm{gen}}\right)\sum_{i:w_{i}=w}a_{i}^{k}$
(2)
$p_{\mathrm{gen}}=\sigma\left(W_{p}\cdot\left[h_{k};e_{k};c_{k}\right]\right)$
(3)

where $W_{p}$ is a trainable matrix, $e_{k}$ is the input word embedding to
the decoder at time step $k$ and $c_{k}$ is the context vector [Bahdanau et
al., 2014]. Intuitively, $p_{gen}$ can be seen as a soft switch to choose
between generating a word from the vocabulary, and copying a word from the
dialogue history.

_TRADE_ also contains a slot gate, which is a simple three-way classifier that
maps the context vector to a probability distribution over _ptr_ , _none_ ,
and _dontcare_ classes. For more details, please refer to the original paper
[Wu et al., 2019].

###  2.3 HDCN

Although _TRADE_ takes an important first step towards generation-based DST,
it simply concatenates different dialogue turns and regards the multi-turn
dialogue as a flat sequence. Correspondingly, copying the correct slot values
becomes especially hard when the sequence is long. In this paper, we propose
the hierarchical dynamic copy network (HDCN) to address this limitation. The
overall structure of our model is shown in Figure 2.

Figure 2: The architecture of HDCN. It follows the encoder-decoder framework,
with the dialogue as input to the encoder and the target slot $S_{m}$ as the
first input to the decoder. It has two levels of attention applied at the
word- and turn-level, which are then re-normalized and result in the final
copy distribution. The model is further enhanced with a focus mechanism to
encourage focusing on the _information turn_ of the target slot. We pad a
sentry turn at the beginning of the dialogue, which acts as the _information
turn_ of slots never mentioned in the dialogue.

We first encode each turn with a bidirectional LSTM (BiLSTM) encoder and
summarize the information in the forward and backward BiLSTM hidden states,
namely $h_{i}^{j}=[\overrightarrow{h_{i}^{j}};\overleftarrow{h_{i}^{j}}]$.
During encoding, we compare two initialization strategies for $h_{0}^{j}$:

  1. 1.

_zero initialization_ : As in [Serban et al., 2015, Serban et al., 2016], we
initialize $h_{0}^{j}$ with zero vector.

  2. 2.

_last initialization_ : $h_{0}^{j}$ is initialized as the last hidden state of
previous turn $h^{j-1}$.

After encoding, we aggregate all the encoded information into a fix-length
vector using max-pooling, with which we initialize our LSTM decoder. As in
_TRADE_ , our model decodes $M$ times independently to generate the dialogue
state. For the slot $S_{m}$ at time step $k$:

$s_{k}^{m}=LSTM(e_{k}^{m},s_{k-1}^{m})$ (4)

where $e_{k}^{m}$ is the word embedding and $s_{k}^{m}$ is the decoder hidden
state. Following _TRADE_ , we also supply the _summed embedding_ of $S_{m}$ as
the first input, $e_{0}^{m}$, to the decoder.

After obtaining $s_{k}^{m}$, we adopt a hierarchical dynamic copy approach to
obtain copy distribution. Specifically, we first calculate word-level
attention distribution $\alpha_{ij}^{k}$ over the dialogue history [Luong et
al., 2015]:

$\alpha_{ij}^{k}=softmax(s_{k}^{m}W_{w}h_{i}^{j})\\\ $ (5)

where $W_{w}$ is trainable parameters.

Note that $\alpha_{ij}^{k}$ is dependent on the target slot $m$ (we skip $m$
in the notations $\alpha_{ij}^{k}$ for simplicity.). Intuitively, the decoder
hidden state $s_{k}^{m}$ can be seen as a high level representation of a fixed
query: _what is the informative words for the $m_{th}$ slot_?

We then aggregate the representations of those informative words to form turn
representations:

$g_{j}^{k}=\sum_{i}\alpha_{ij}^{k}h_{i}^{j}\\\ $ (6)

Traditional hierarchical LSTM methods regard the last hidden state of the
word-level LSTM as the turn representation. By contrast, our turn
representation considers all the hidden states $h_{i}^{j}$. More importantly,
this dynamic calculation procedure enables our model to form different turn
representations $g_{j}^{k}$ for different target slots, leading to more
flexible and robust representations. For example, in Figure 1, when
calculating $\alpha_{ij}^{k}$ over turn 4, the slot _hotel-book day_ would
attend more to the word _Tuesday_ , and the slot _hotel-book stay_ would
attend more to the word _2_. In this way, the turn representations $g_{j}^{k}$
for these two slots also differ.

Similarly, to identify informative turns for the $m_{th}$ slot, we calculate
turn-level attention weights using $s_{k}^{m}$.

$\beta_{j}^{k}=softmax(s_{k}^{m}W_{t}g_{j}^{k})$ (7)

where $W_{t}$ is model parameters. Intuitively, $\beta_{j}^{k}$ can be seen as
an indicator of _how informative turn j is for the $m_{th}$ slot_ (similarly,
we skip $m$ in the notation for simplicity).

Then, we re-normalize the two levels of attention to obtain the final copy
distribution. Formally,

$\gamma_{ij}^{k}=\beta_{j}^{k}*\alpha_{ij}^{k}$ (8)

In this way, words from more informative turns get rewarded, and words from
less informative turns are discouraged. A more detailed analysis can be found
in Subsection 4.3.

Finally, $\gamma_{ij}^{k}$ and the vocabulary distribution $P^{vocab}_{k}$ are
weighted and summed to obtain the final word distribution:

$P_{k}(w)=p_{\mathrm{gen}}P^{\mathrm{vocab}}_{k}(w)+\left(1-p_{\mathrm{gen}}\right)\sum_{ij:w_{ij}=w}\gamma_{ij}^{k}$
(9)

where $P_{k}^{vocab}(w)$ and $p_{gen}$ are calculated similar to Equation 1
and 3, respectively.

###  2.4 Turn Focus Mechanism

We define the _information turn_ for a certain slot as the turn where its
corresponding value appears, or can be inferred. It is essential to assign the
highest turn-level attention weight to the _information turn_ in order to
capture key information.

To achieve this, we sum and normalize all the turn-level attention weights
after decoding, and then calculate the turn focus loss as the negative log
probability of the information turn. Formally,

$\beta_{j}=softmax(\sum_{k}\beta_{j}^{k})$ (10)

$\mathcal{L}_{focus}^{m}=-\log(\hat{\beta_{j}})$ (11)

where $\hat{\beta_{j}}$ is the _information turn_ for the $m_{th}$ slot.

In this way, our model can learn to assign the highest turn-level attention
weight to the _information turn_. Combined with our hierarchical dynamic copy
mechanism, words from the _information turn_ are more likely to be copied (see
Equation 8).

Note that some slots are never mentioned in the dialogue, and they thus do not
have corresponding _information turns_. To handle these slots, we pad a sentry
turn at the beginning of the dialogue (see Figure 2). The sentry turn contains
only one word _none_ and acts as the _information turn_ for these slots.

Besides, we observe that slot values that contain multiple words are mostly
named entities mentioned in the dialogue, like _holy trinity church_ or
_london liverpool street_. Obviously, these words are almost always from the
same turn. Based on this observation, we further propose two methods to better
handle multi-words values: _HDCN-freeze_ and _HDCN-force_.

As the name suggests, _HDCN-freeze_ freezes turn-level attention after the
first decoding step. That is, we modify Equation 7 into the following:

$\beta_{j}^{k}=\left\\{\begin{array}[]{ll}{softmax(s_{k}^{m}W_{t}g_{j}^{k})}&{\text{
if k $=$ 0}}\\\ {\beta_{j}^{0}}&{\text{ if k $>$ 0}}\end{array}\right.$ (12)

However, we found that this approach is too rigid. Inspired by the coverage
mechanism [See et al., 2017], we further propose _HDCN-cover_ , a more
flexible approach. The basic idea is to _encourage_ rather than _force_ our
model to focus on the same turn during decoding. Specifically, we maintain a
turn coverage vector $c^{k}$ during decoding, which is the sum of turn-level
attention distribution over all previous decoding steps:

$c^{k}=\sum_{k=0}^{k-1}\beta^{k}$ (13)

$c^{k}$ is used as extra input to the calculation of turn-level attention,
changing equation 7 to:

$\beta_{j}^{k}=softmax(s_{k}W_{k}g_{j}+w_{c}c_{j}^{k})$ (14)

where $w_{c}$ is a learnable parameter randomly initialized. During training,
we find that $w_{c}$ steadily increases, suggesting that turns receiving
higher attention in previous decoding steps indeed get rewarded.

Note that our coverage vector serves the exact opposite purpose of the
original coverage mechanism [See et al., 2017]. Coverage mechanism was first
proposed for repetition reduction by _discouraging_ the model from attending
to the same place. By contrast, we here want to _encourage_ the model to
attend to the same turn during decoding.

###  2.5 Task Learning

During training, the cross-entropy loss for the $m_{th}$ slot is calculated
as:

$\mathcal{L}_{ce}^{m}=-\frac{1}{K}\sum_{k=1}^{K}\log
P\left(\hat{w}_{k}^{m}\right)$ (15)

where $\hat{w}_{k}^{m}$ is the ground truth at time step $k$ and $K$ is the
total decoding steps.

Combined with Equation 11, the final loss function is defined as:

$\mathcal{L}=\sum_{m=1}^{M}(\mathcal{L}_{ce}^{m}+\xi\mathcal{L}_{focus}^{m})$
(16)

where $\xi$ is the focus ratio and is used to adjust the relative importance
of each term. $M$ is the number of all the possible slots.

##  3 Experiments

###  3.1 Dataset

Following previous work [Wu et al., 2019, Ren et al., 2019, Gao et al., 2019],
we train and evaluate our model with MultiWOZ [Budzianowski et al., 2018]
dataset. It is currently the largest multi-domain conversational corpus
spanning over seven domains, containing 8438 multi-turn dialogues. The
training set, validation set and test set contain 8,438, 1,000 and 1,000
dialogues, respectively.

The original MultiWOZ dataset were detected to contain substantial errors in
the state annotation. These errors were later fixed in MultiWOZ 2.1 dataset
[Eric et al., 2019]. The correction process results in changes to over 32% of
state annotations across 40% of the dialogue turns. In this paper, we
conducted experiments with the corrected version dataset, MultiWOZ 2.1.

Note that MultiWOZ dataset does not contain _information turn_ labels, yet it
can be easily inferred using rule-based method. Specifically, if the dialogue
state changes from $B_{t-1}$ to $B_{t}$ after turn $t$, then $t$ is the
_information_ turn for those slots whose values differ in $B_{t-1}$ and
$B_{t}$. The _information turn_ labels are only used during training.

###  3.2 Training Procedure

Our model was implemented in PyTorch and trained on a single NVIDIA 1080 Ti
GPU. The LSTMs [Hochreiter and Schmidhuber, 1997] and word embeddings are 400
dimensional. Following previous work [Zhong et al., 2018, Wu et al., 2019], we
initialized the embeddings by concatenating Glove embeddings [Pennington et
al., 2014] and character embeddings [Hashimoto et al., 2016]. Dropout [Hinton
et al., 2012] is used with dropout rate set to 0.5. The number of layers of
LSTM encoder/decoder is set to 2. The batch size is set to 16. $\xi$ in
Equation 16 is set to 0.1.

We use Adam [Kingma and Ba, 2014] with learning rate = $10^{-3}$,
$\beta_{1}=0.9,\beta_{2}=0.98$ and $\epsilon=10^{-9}$. We choose our model
according to the results on the development set. The reported results are the
average of five test set results.

###  3.3 Systems for Comparison

In this paper, we compare our model against the following baselines:

FJST [Eric et al., 2019] is a classification-based model. It uses
bidirectional LSTM network to encode the full dialogue history.

HJST [Eric et al., 2019] differs from FJST in that it encodes the dialogue
using hierarchical LSTM [Serban et al., 2015].

DST Reader [Gao et al., 2019] first proposes to regard the problem of DST as a
reading comprehension problem. It uses a simple attention-based neural network
to point to the slot values within the dialogue.

HyST [Goel et al., 2019] combines a hierarchical encoder fixed vocabulary
system with an open vocabulary n-gram copy-based system

TRADE [Wu et al., 2019] is the first generation-based DST model and achieves
the state-of-the-art performance in the pre-BERT era. A detailed description
is in Subsection 2.2

Hier-LSTM based is our baseline to examine the effectiveness of our
hierarchical dynamic copy mechanism. It is based on the hierarchical LSTM
encoder, and the decoding process is the same as _TRADE_. For a fair
comparison, we also implement copy and focus mechanism to this baseline.

HDCN is our proposed model, and we compare four variants of HDCN in this
paper. The difference between zero- and last-initialization is elaborated in
Subsection 2.3 and the difference between _HDCN-freeze_ and _HDCN-cover_ is
detailed in Subsection 2.4.

###  3.4 Metrics

Joint accuracy - Joint accuracy is the most widely used evaluation metrics for
DST [Zhong et al., 2018, Wu et al., 2019, Ren et al., 2019]. It compares the
predicted dialogue state to the ground truth at each dialogue turn, and the
output is considered correct if and only if all the predicted values exactly
match the ground truth values. All the baseline results are taken from the
official paper of MultiWOZ 2.1 dataset [Eric et al., 2019].

Focus accuracy - In this work, we also consider focus accuracy to examine our
claim that the focus mechanism can encourage the model to focus on
_information turns_. For each slot, focus accuracy compares its summed turn-
level attention weights $\sum_{k=0}^{K}\beta^{k}_{j}$ with its _information
turn_. The result is considered correct if and only if the _information turn_
receives the highest attention weight.

Models | Joint Acc.
---|---
DST Reader [Gao et al., 2019] | 36.40
HyST [Goel et al., 2019] | 38.10
FJST [Eric et al., 2019] | 38.00
HJST [Eric et al., 2019] | 35.55
TRADE [Wu et al., 2019] | 45.60
Hier-LSTM based (baseline) | 39.39
HDCN-freeze, zero initialized | 45.48
HDCN-freeze, last initialized | 46.11
HDCN-cover, zero initialized | 45.62
HDCN-cover, last initialized | 46.76

Table 1:  Joint accuracy on MultiWOZ test set.

Settings | Focus Acc. | Joint Acc.
---|---|---
$\xi=0$ | 94.89 | 50.23
$\xi=0.01$ | 95.70 | 51.06
$\xi=0.05$ | 96.17 | 51.18
$\xi=0.1$ | 96.31 | 51.32
$\xi=0.2$ | 96.47 | 50.80
$\xi=1$ | 96.63 | 48.87

Table 2:  Development set results to examine the effectiveness of our focus
mechanism.

##  4 Results

###  4.1 Overall Performance

The joint accuracy results on the MultiWOZ dataset are shown in Table 1.

Our first observation is that the _Hier-LSTM based_ baseline gives poor
performance, with joint accuracy lower than the flat-structured _TRADE_.
Besides, we can see that _HJST_ also performs worse than _FJST_ , which
further proves that simply using two-levels of LSTM to model multi-turn
dialogue is not suited for DST. We suppose this is because the last hidden
state of word-level LSTM is not sufficient to capture all the key information
within the turn. By contrast, our proposed model uses the attention mechanism
to dynamically calculate turn representations and selectively uses all the
word representations within the turn. More importantly, our model allows for
different turn representations for different target slots.

Besides, we can see that _HDCN-cover_ generally performs better than _HDNC-
freeze_. Although they both aim for consistent focusing on the same turn
during decoding, the former approach allows for more flexibility. Another
observation is that the _last-initialization_ approach performs notably better
than _zero-initialization_ , with about 1% absolute improvement. This is
because the former can effectively utilize history information when encoding
each turn, while the latter encodes each turn independently and discards the
history information entirely.

Figure 3: The turn- and word-level attention weights.

###  4.2 Focus Mechanism

One of the main claims of this paper is that we can improve DST performance by
encouraging our model to focus on the _information turn_. To further examine
this claim, we conduct experiments with different focus ratio, i.e., $\xi$ in
Equation 16 on the development set. The results are shown in Table 2.

Firstly, we can see that the focus accuracy indeed increases with the focus
ratio. This is understandable because a higher focus ratio means we attach
greater importance to the focus loss term.

Besides, the results also buttress our claim that the focus mechanism can
improve DST performance. We can see that even with 0.01 focus ratio, the
improvement is notable. Our model reaches the best performance when $\xi=0.1$
and the reported test set results are based on this setting.

Furthermore, we note that the joint accuracy does not increase monotonously
with the focus ratio. After reaching its peak value, the joint accuracy
slightly decreases. We suppose this is because focusing too much on a single
turn may lead to the neglect of contextual information. Besides, from Equation
16 we can see that excessively high focus ratio could hinder the decoding
process.

###  4.3 Attention Weights Analysis

We show in Figure 3 the turn- and word-level attention weights for the example
in Figure 1.

As demonstrated in Subgraph (a), all the five slots successfully focus on
their corresponding _information slots_. For example, slot _hotel-book day_
assigns the highest attention weight to turn 4, and slot _hotel-book people_
assigns the highest attention weight to turn 5. Besides, since the slot
_hotel-parking_ is never mentioned in the dialogue, it assigns the highest
attention weight to the sentry turn, which only contains one word _none_.

Subgraph (b) gives the word-level attention over turn 5 before re-
normalization with Equation 8. A crucial observation is that both slot _hotel-
book stay_ and _hotel-book people_ assign the highest attention weight to the
word _3_. This is because both slots tend to receive a number as value, and
the number _3_ can hence become puzzling. In fact, we find that _TRADE_ ,
which adopts flat-copy and does not involve the re-normalization process,
wrongly assigns the value for slot _slot-book stay_ to be _3_.

Subgraph (c) gives the word-level attention over turn 5 after re-
normalization. We can see that since slot _hotel-book stay_ barely attends to
turn 5 (see Subgraph (a)), its attention on the word _3_ almost becomes 0
after re-normalization. By contrast, slot _hotel-book people_ still
(rightfully) assigns the highest attention to _3_ , because turn 5 is its
_information turn_ and receives the highest turn-level attention.

From the above analysis, we can see why the proposed model improves over
_TRADE_. _TRADE_ organizes the dialogue as a long sequence. It is hence hard
for it to attend to and copy the _gold value_ , as there can be many puzzling
words along that sequence, like _3_ for slot _hotel-book stay_ in the above
example. By contrast, our model adopts a hierarchical copy process and can
assign the highest turn-level attention weight to the _information turn_ ,
thanks to the focus mechanism.

##  5 Related Work

DST -  In recent years, neural network approaches have defined the state-of-
the-art in DST research. Most previous works decomposed this task as a series
of classification problems [Zhong et al., 2018, Mrksic et al., 2016, Ramadan
et al., 2018, Balaraman and Magnini, 2019, Lee et al., 2019]. They took each
of the slot-value pairs as input for scoring, and output the value with the
highest score as the predicted value for a slot. For such models, a predefined
ontology is required, which lists all the possible slot-value pairs. However,
such an ontology is not always available in practice. Besides, the number of
all the possible slot-values pairs can be large, and enumerating all the
possible slot-values pairs sometimes requires astronomical time and resources.
For example, in MultiWOZ dataset, there are over 4,500 possible slot values in
total, meaning that such models have to perform over 4,500 classifications to
determine the current dialogue state.

To address these limitations, some recent works discarded the classification
framework [Xu and Hu, 2018, Wu et al., 2019, Balaraman and Magnini, 2019, Gao
et al., 2019]. [Xu and Hu, 2018] is the first model that applies pointer
networks [Vinyals et al., 2015] to the single-domain DST problem, which
generates both start and end pointers to perform index-based copying. [Gao et
al., 2019] used a simple attention-based neural network to point to the slot
values within the conversation. _TRADE_ [Wu et al., 2019] first proposed to
regard this problem as a text generation problem. It encodes the dialogue
history as a long sequence with a GRU and uses another GRU as the decoder to
predict the value for each slot independently. [Ren et al., 2019] proposed to
sequentially decode domain, slot and value, hence remained a constant
computational complexity, regardless of the number of predefined slots.

Hierarchical Attention Network - Our hierarchical dynamic copy mechanism draws
inspiration from Hierarchical Attention Network (HAN) [Yang et al., 2016],
which was first proposed for text classification. Similar to HAN, our approach
also has two levels of attention mechanisms applied at the word- and sentence-
level. However, our approach is notably different from HAN in two ways: one is
that that when calculating attention scores (Equation 5 and 7), we use the
decoder hidden state $s_{t}^{m}$ while HAN uses randomly initialized vector.
In this way, our model can form different representations for different target
slots. The other difference is that we re-normalize the two levels of
attention to form the final copy distribution, while HAN only uses attention
distribution to form sentence and document representations.

Copy Mechanism There are three common methods to perform copying, namely,
index-based copy [Vinyals et al., 2015], hard-gated copy [Gulcehre et al.,
2016]and soft-gated copy [See et al., 2017]. [Wu et al., 2019] first
introduced soft-copy into DST area. We extend their idea from flat copy to
hierarchical dynamic copy.

##  6 Conclusion

In this work, we propose to tackle the DST problem with HDCN. Instead of
viewing the dialogue as a flat sequence, we propose the hierarchical dynamic
copy mechanism to facilitate focusing on the _information turns_ , making it
easier to copy slot values from the dialogue context. Besides, we further
propose two methods to better handle multi-words values. Extensive experiments
show that our model improve over TRADE over 1%.

## References

  * [Bahdanau et al., 2014]  Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  2014\.  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473.
  * [Balaraman and Magnini, 2019]  Vevake Balaraman and Bernardo Magnini.  2019\.  Scalable neural dialogue state tracking.  ArXiv, abs/1910.09942.
  * [Budzianowski et al., 2018]  Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic.  2018\.  Multiwoz - a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.  In EMNLP.
  * [Cho et al., 2014]  Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  2014\.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  arXiv preprint arXiv:1406.1078.
  * [Eric et al., 2019]  Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, and Dilek Z. Hakkani-Tür.  2019\.  Multiwoz 2.1: Multi-domain dialogue state corrections and state tracking baselines.  ArXiv, abs/1907.01669.
  * [Gao et al., 2019]  Shuyang Gao, Abhishek Sethi, Sanchit Agarwal, Tagyoung Chung, and Dilek Z. Hakkani-Tür.  2019\.  Dialog state tracking: A neural reading comprehension approach.  ArXiv, abs/1908.01946.
  * [Goel et al., 2019]  Rahul Goel, Shachi Paul, and Dilek Z. Hakkani-Tür.  2019\.  Hyst: A hybrid approach for flexible and accurate dialogue state tracking.  ArXiv, abs/1907.00883.
  * [Gulcehre et al., 2016]  Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio.  2016\.  Pointing the unknown words.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
  * [Hashimoto et al., 2016]  Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher.  2016\.  A joint many-task model: Growing a neural network for multiple nlp tasks.  In EMNLP.
  * [Hinton et al., 2012]  Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdinov.  2012\.  Improving neural networks by preventing co-adaptation of feature detectors.  arXiv preprint arXiv:1207.0580.
  * [Hochreiter and Schmidhuber, 1997]  Sepp Hochreiter and Jürgen Schmidhuber.  1997\.  Long short-term memory.  Neural computation, 9(8):1735–1780.
  * [Hosseini-Asl et al., 2020]  Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher.  2020\.  A simple language model for task-oriented dialogue.  arXiv preprint arXiv:2005.00796.
  * [Kim et al., 2020]  Sungdong Kim, Sohee Yang, Gyuwan Kim, and Sang-woo Lee.  2020\.  Efficient dialogue state tracking by selectively overwriting memory.  In ACL.
  * [Kingma and Ba, 2014]  Diederik P Kingma and Jimmy Ba.  2014\.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980.
  * [Lee et al., 2019]  Hwaran Lee, Jinsik Lee, and Tae-Yoon Kim.  2019\.  Sumbt: Slot-utterance matching for universal and scalable belief tracking.  In ACL.
  * [Luong et al., 2015]  Minh-Thang Luong, Hieu Pham, and Christopher D Manning.  2015\.  Effective approaches to attention-based neural machine translation.  arXiv preprint arXiv:1508.04025.
  * [Mrksic et al., 2016]  Nikola Mrksic, Diarmuid Ó Séaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve J. Young.  2016\.  Neural belief tracker: Data-driven dialogue state tracking.  In ACL.
  * [Pennington et al., 2014]  Jeffrey Pennington, Richard Socher, and Christopher D. Manning.  2014\.  Glove: Global vectors for word representation.  In EMNLP.
  * [Ramadan et al., 2018]  Osman Ramadan, Pawel Budzianowski, and Milica Gai.  2018\.  Large-scale multi-domain belief tracking with knowledge sharing.  In ACL.
  * [Ren et al., 2019]  Liliang Ren, Jianmo Ni, and Julian J. McAuley.  2019\.  Scalable and accurate dialogue state tracking via hierarchical sequence generation.  In EMNLP/IJCNLP.
  * [See et al., 2017]  Abigail See, Peter J Liu, and Christopher D Manning.  2017\.  Get to the point: Summarization with pointer-generator networks.  arXiv preprint arXiv:1704.04368.
  * [Serban et al., 2015]  Iulian Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C. Courville, and Joelle Pineau.  2015\.  Building end-to-end dialogue systems using generative hierarchical neural network models.  In AAAI.
  * [Serban et al., 2016]  Iulian Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio.  2016\.  A hierarchical latent variable encoder-decoder model for generating dialogues.  In AAAI.
  * [Sutskever et al., 2014]  Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  2014\.  Sequence to sequence learning with neural networks.  Advances in Neural Information Processing Systems (NIPS), pages 3104–3112.
  * [Vinyals et al., 2015]  Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly.  2015\.  Pointer networks.  In International Conference on Neural Information Processing Systems.
  * [Wu et al., 2019]  Chien-Sheng Wu, Andrea Madotto, Ehsan Hosseini-Asl, Caiming Xiong, Richard Socher, and Pascale Fung.  2019\.  Transferable multi-domain state generator for task-oriented dialogue systems.  In ACL.
  * [Xu and Hu, 2018]  Puyang Xu and Qi Hu.  2018\.  An end-to-end approach for handling unknown slot values in dialogue state tracking.  In ACL.
  * [Yang et al., 2016]  Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy.  2016\.  Hierarchical attention networks for document classification.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480–1489.
  * [Zhong et al., 2018]  Victor Zhong, Caiming Xiong, and Richard Socher.  2018\.  Global-locally self-attentive dialogue state tracker.  ArXiv, abs/1805.09655.