DART
Difficulty-Aware Rejection Tuning
# 🎯 DART-Math

> Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving

📝 [Paper@arXiv](https://arxiv.org/abs/2407.13690) | 🤗 [Datasets&Models@HF](https://huggingface.co/collections/hkust-nlp/dart-math-665704599b35de59f8fdf6c1) | 🐱 [Code@GitHub](https://github.com/hkust-nlp/dart-math)

🐦 [Thread@X(Twitter)](https://x.com/tongyx361/status/1811413243350454455) | 🐶 [中文博客@知乎](https://zhuanlan.zhihu.com/p/708371895) | 📊 [Leaderboard@PapersWithCode](https://paperswithcode.com/paper/dart-math-difficulty-aware-rejection-tuning#results) | 📑 [BibTeX](https://github.com/hkust-nlp/dart-math?tab=readme-ov-file#citation)

## Datasets: `DART-Math`

`DART-Math` datasets are the **state-of-the-art** and **data-efficient** **open-source** instruction tuning datasets for mathematical reasoning.

`DART-Math-Hard` contains \~585k mathematical QA pair samples constructed by applying `DARS-Prop2Diff` to the query set from MATH and GSK8K training sets, achieves **SOTA** on many challenging mathematical reasoning benchmarks. It introduces a **deliberate bias towards hard queries**, opposite to vanilla rejection sampling.

Performance produced by  `DART-Math-Hard` is usually but not necessarily **slightly better (\~1% absolutely)** than `DART-Math-Uniform`, which contains \~591k samples constructed by applying `DARS-Uniform`.

### Comparison between Mathematical Instruction Tuning Datasets

Most of previous datasets are **constructed with ChatGPT**, and many of them are **not open-source**, especially for ones of the best performance.

| Math SFT Dataset                                                                   | # of Samples | [MATH](https://huggingface.co/datasets/hendrycks/competition_math) | [GSM8K](https://huggingface.co/datasets/gsm8k) | [College](https://github.com/hkust-nlp/dart-math/tree/main/data/eval-dsets/mwpbench/college-math-test.jsonl) | Synthesis Agent(s)      |                                 Open-Source                                 |
| :--------------------------------------------------------------------------------- | -----------: | -----------------------------------------------------------------: | ---------------------------------------------: | -----------------------------------------------------------------------------------------------------------: | :---------------------- | :-------------------------------------------------------------------------: |
| [WizardMath](https://arxiv.org/abs/2308.09583)                                     |          96k |                                                               32.3 |                                           80.4 |                                                                                                         23.1 | GPT-4                   |                                      ✗                                      |
| [MetaMathQA](https://arxiv.org/abs/2309.12284)                                     |         395k |                                                               29.8 |                                           76.5 |                                                                                                         19.3 | GPT-3.5                 |          [✓](https://huggingface.co/datasets/meta-math/MetaMathQA)          |
| [MMIQC](https://arxiv.org/abs/2401.09003)                                          |    **2294k** |                                                               37.4 |                                           75.4 |                                                                                                       _28.5_ | **GPT-4+GPT-3.5+Human** |           [**✓**](https://huggingface.co/datasets/Vivacem/MMIQC)            |
| [Orca-Math](https://arxiv.org/abs/2402.14830)                                      |         200k |                                                                 -- |                                             -- |                                                                                                           -- | GPT-4                   | [✓](https://huggingface.co/datasets/microsoft/orca-math-word-problems-200k) |
| [Xwin-Math-V1.1](https://arxiv.org/abs/2403.04706)                                 |    **1440k** |                                                             _45.5_ |                                       **84.9** |                                                                                                         27.6 | **GPT-4**               |                                    **✗**                                    |
| [KPMath-Plus](https://arxiv.org/abs/2403.02333)                                    |    **1576k** |                                                           **46.8** |                                           82.1 |                                                                                                           -– | **GPT-4**               |                                    **✗**                                    |
| [MathScaleQA](https://arxiv.org/abs/2403.02884)                                    |        2021k |                                                               35.2 |                                           74.8 |                                                                                                         21.8 | GPT-3.5+Human           |                                      ✗                                      |
| [`DART-Math-Uniform`](https://huggingface.co/datasets/hkust-nlp/dart-math-uniform) |     **591k** |                                                               43.5 |                                         _82.6_ |                                                                                                         26.9 | **DeepSeekMath-7B-RL**  |    [**✓**](https://huggingface.co/datasets/hkust-nlp/dart-math-uniform)     |
| [`DART-Math-Hard`](https://huggingface.co/datasets/hkust-nlp/dart-math-hard)       |     **585k** |                                                             _45.5_ |                                           81.1 |                                                                                                     **29.4** | **DeepSeekMath-7B-RL**  |      [**✓**](https://huggingface.co/datasets/hkust-nlp/dart-math-hard)      |

<sup>MATH and GSM8K are **in-domain**, while College(Math) is **out-of-domain**. Performance here are of models fine-tuned from [Mistral-7B](https://huggingface.co/mistralai/Mistral-7B-v0.1), except for Xwin-Math-V1.1 based on [Llama2-7B](https://huggingface.co/meta-llama/Llama-2-7b-hf). **Bold**/_Italic_ means the best/second best score here.</sup>

## Dataset Construction: `DARS` - Difficulty-Aware Rejection Sampling

Previous works usually synthesize data from proprietary models to augment existing datasets, followed by instruction tuning to achieve top-tier results.
However, our analysis of these datasets reveals **severe biases towards easy queries, with frequent failures to generate any correct response for the most challenging queries**.

Motivated by the observation above, we propose to *Difficulty-Aware Rejection Sampling* (`DARS`), to collect more responses for more difficult queries.
Specifically, we introduce two strategies to increase the number of correct responses for difficult queries:

1) **Uniform**, which involves sampling responses for each query until **each query accumulates $k_u$ correct
responses**, where $k_u$ is a preset hyperparameter determined by the desired size of the synthetic dataset;
2) **Prop2Diff**, where we continue sampling responses until the number of correct responses for each
query is **proportional to its difficulty score**. The most challenging queries will receive $k_p$ responses
and kp is a hyperparameter. This method introduces a deliberate bias in the opposite direction to
vanilla rejection sampling, towards more difficult queries, inspired by previous works
that demonstrate **difficult samples can be more effective to enhance model capabilities** ([Sorscher et al.,
2022](https://proceedings.neurips.cc/paper_files/paper/2022/hash/7b75da9b61eda40fa35453ee5d077df6-Abstract-Conference.html); [Liu et al., 2024b](https://openreview.net/forum?id=BTKAeLqLMw)).

See [Figure 1 (Right)](https://tongyx361.github.io/assets/dart-math/main-nresp-vs-query.png) for examples of `DART-Math-Uniform` by `DARS-Uniform` and `DART-Math-Hard` by `DARS-Prop2Diff`.

## Citation

If you find our data, model or code useful for your work, please kindly cite [our paper](https://arxiv.org/abs/2407.13690):

```latex
@article{tong2024dartmath,
  title={DART-Math: Difficulty-Aware Rejection Tuning for Mathematical Problem-Solving},
  author={Yuxuan Tong and Xiwen Zhang and Rui Wang and Ruidong Wu and Junxian He},
  year={2024},
  eprint={2407.13690},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2407.13690},
}
```