# Measuring Sample Importance in Data Pruning for Training LLMs from a Data
Compression Perspective

Minsang Kim
Korea University
Dept. of Computer Science and Engr.
South Korea
<EMAIL_ADDRESS>
&Seungjun Baek
Korea University
Dept. of Computer Science and Engr.
South Korea
<EMAIL_ADDRESS>

###### Abstract

Compute-efficient training of large language models (LLMs) has become an
important research problem. In this work, we consider data pruning as a method
of data-efficient training of LLMs, where we take a data compression view on
data pruning. We argue that the amount of information of a sample, or the
achievable compression on its description length, represents its sample
importance. The key idea is that, less informative samples are likely to
contain redundant information, and thus should be pruned first. We leverage
log-likelihood function of trained models as a surrogate to measure
information content of samples. Experiments reveal a surprising insight that
information-based pruning can enhance the generalization capability of the
model, improves upon language modeling and downstream tasks as compared to the
model trained on the entire dataset.

##  1 Introduction & Related Work

Training large language models (LLM) requires high computational costs due to
their immense sizes and commensurately large datasets Kaplan et al. [2020],
Hoffmann et al. [2022]. Recently, computationally efficient methods to train
deep models based on _data pruning_ have gained interest. Data pruning
concerns deciding which samples are important for training, and remove
unimportant samples from training datasets. Sorscher et al. [2022] proposed
that neural scaling law Kaplan et al. [2020] may be overcome with data
pruning. Their method uses distances between samples and their cluster
centroids in the embedding space to measure the sample importance. Tan et al.
[2024] proposed to assess the sample importance by efficiently measuring the
change in empirical risk when the sample is removed from the training set. The
aforementioned works apply data pruning to labeled data for supervised
learning or focus on visual understanding tasks. By contrast, data pruning
methods for LLMs have been largely underexplored. Instead, text deduplication
methods have been proposed Raffel et al. [2020], Abbas et al. [2023] to remove
redundant data based on semantic similarity. However, they do not quantify the
importance of samples for data pruning.

Meanwhile, connections between LLMs and data compression have been recently
explored. Ge et al. [2023] proposed In-Context Autoencoder for context
compression. Deletang et al. [2023] extensively study the view of LLM as a
model for data compression. The predictive power of LLMs can be used for an
optimally efficient expression of data based on the prediction probability, so
as to compress various types of multi-modal data.

In this paper, we propose to leverage the compression capability of LLMs for
data pruning. If a model learns the distribution of corpus, it can estimate
the amount of information of a sample, i.e., the optimal number of bits
required to compress the sample. Importantly, if the amount of information of
a sample is low, the sample likely contains information which is redundant or
appear frequently in dataset. Less informative samples are regarded as less
important and thus are removed from dataset. To estimate the information of
samples, we first train called _data probe model_ which is a small model
trained with a subset of corpus. We use the log-likelihood output of data
probe model as a surrogate to measure the sample information. Next, we prune
the dataset based on the estimates by data probe model, and train a target
model with the pruned dataset. Experimental results are quite surprising: in
many cases, the proposed pruning actually _improves_ the model performance. In
some cases, the performance of generation and/or downstream tasks are
maintained up to pruning 50% of the pretraining corpus. The key insight is
that, the proposed pruning based on sample information helps removing
irrelevant data, which promotes the generalization capability of language
models.

##  2 Method

###  2.1 Compression View to Data Pruning

We take a data compression view to data pruning for language models as
follows. Let $p(\cdot)$ denote the true distribution of words from a corpus.
Given a sample $W=(w_{1}w_{2}\ldots w_{n})$ which is a sequence of words
$\\{w_{i}\\}$, consider

$H(W,p)=\frac{1}{n}\sum_{i=1}^{n}\log\frac{1}{p(w_{i}|w_{<i})}$

We view $H(W,p)$ as related to the minimum description length required to
compress words in $W$. Indeed, the optimal compression based on entropy, e.g,
Huffman coding Huffman [1952] or arithmetic coding Rissanen [1976], uses
approximately $\lceil\log_{2}1/p(w_{i}|w_{<i})\rceil$ bits to encode $w_{i}$
given previous words. Alternatively, $H(W,p)$ represents the amount of
information per word in passage $W$. Consider a next-word predictor $q$, i.e.,
a language model. The negative log-likelihood of $W$ on model $q$ averaged
over sequence length, which is our main metric for sample importance, is given
by

$\displaystyle H(W,q)=\frac{1}{n}\sum_{i=1}^{n}\log\frac{1}{q(w_{i}|w_{<i})}$
(1)

From an information theoretic view, (1) represents the description length of
words in $W$ using sub-optimal distribution $q$. Meanwhile, if we average
$nH(W,q)$ over $W$, it is the log-loss used for training language models.
Thus, training a language model is equivalent to minimizing the description
length of, i.e., compressing, corpus data, as observed in Deletang et al.
[2023]. In practice, the samples in batches have a fixed sequence length when
training Transformer-based causal language models such as GPT-3 Brown et al.
[2020] or Llama-2 Touvron et al. [2023], where the sequence length $n$ is
fixed to 2048 and 4096 respectively. For example, if the lengths of documents
are larger than $n$, they are divided into multiple sequences (samples); if it
is shorter than $n$, multiple documents are packed into a single sequence
Raffel et al. [2020]. Thus, from the implementation point of view, we can
regard sequence length $n$ as fixed, and (1) can be regarded as being
proportional to the negative log-likelihood or log-loss.

We hypothesize that, if $q$ is well-trained to approximate $p$, we may use
$H(W,q)$ as a surrogate for $H(W,p)$. In particular, $H(W,q)$ can provide an
upper bound of $H(W,p)$ in an average sense. Indeed, we have

$\displaystyle\mathcal{H}(p)$ $\displaystyle=\mathbf{E}_{W\sim
p}\left[H(W,p)\right]$ $\displaystyle\mathcal{H}(p,q)$
$\displaystyle=\mathbf{E}_{W\sim p}\left[H(W,q)\right]$

where $\mathcal{H}(p)$ and $\mathcal{H}(p,q)$ denotes the per-word entropy and
cross-entropy with respect to $q$, respectively. The cross-entropy is an upper
bound of entropy Cover [1999], i.e.,

$\mathcal{H}(p)\leq\mathcal{H}(p,q)$

That is, we always need extra bits to encode words with distribution $p$ when
$q$ is used for encoding.

Thus, if $q$ is sufficiently trained to achieve low log-loss (cross-entropy)
for next-word prediction, $H(W,q)$ is likely to provide a good upper bound on
$H(W,p)$ for sufficiently long $n$. In turn, we hypothesize that sample $W$
with low $H(W,q)$ (implying low $H(W,p)$) is of low informational value,
because it has short description length. That is, $W$ likely contains
word/phrases which appear frequently in the corpus, i.e., contains redundant
information. Thus, it makes sense to remove samples with small $H(\cdot,q)$
values from the dataset.

Another view is that, one can show that the _perplexity_ of $W$ for model $q$
is given by $e^{H(W,q)}.$ Thus, low $H(W,q)$ results in low perplexity or
“surprisal”, where the surprisal of a sample is a term used for the
information content of the sample in an information theoretic context.

###  2.2 Pruning Method

Overview. We first train reference model $q$, called _data probe model_. The
goal is to economically measure $H(\cdot,q)$ of the samples. To that end, we
only use a subsample of the pretraining corpus to train $q$. Next, we prune
the samples with low $H(\cdot,q)$ from the dataset. The pruned dataset is used
for training the target LLM.

Data probe model. We train a data probe model denoted by $q$. To reduce
computational costs, we train $q$ on a subsample of the pre-training corpus.
Still, the subsample size should be sufficient so that $H(\cdot,q)$ provide a
good bound on the description lengths. We heuristically determine the
subsample size by hypothesizing that the model is sufficiently trained if the
rate of decrease in the training loss saturates, similar to the early stopping
method.

As a result, the subsample size is set to be about 12% of training dataset:
see Fig. 1. Later experiments show that the proposed level of training is
sufficient for a good pruning performance.

Pruning. Next, we perform pruning on the dataset $\mathcal{D}$. We first
measure $H(W,q)$ for all $W\in\mathcal{D}$. In our setup, $W$ is a fixed-
length sequence. Denote the target pruning ratio by $\eta$. Prune bottom
$\eta$% of samples in terms of $H(\cdot,q)$ values from $\mathcal{D}$. Use the
pruned dataset to train the LLM.

Figure 1: Loss curve of GPT-125M over 1 sweep of the training dataset. For
data efficiency, we stopped training the probe model at the point where the
decrease in loss saturates, i.e., at about 12% of the entire dataset.

##  3 Experiment

###  3.1 Experimental Settings

As pretraining corpus, we use a subset of c4 Raffel et al. [2020] via random
sampling, which is a total of about 3 billion tokens from GPT-2 tokenizer
Radford et al. [2019]. We use decoder-only transformers Vaswani et al. [2017]
using GPT-2 tokenizer as the language model. As the data probe model, we use a
125M-parameter model trained on 0.3 billion tokens randomly sampled from the
total corpus. As the target model, we use both 125M- and 345M-parameter
models. We evaluate the language modeling of target models on One billion
words Chelba et al. [2014] and wikitext-103 Merity et al. [2016] where the
models are pre-trained on different datasets. The downstream tasks for the
target models are evaluated using glue benchmark Wang et al. [2018]. Detailed
hyperparameters are in Appendix A.1.

###  3.2 Pretraining Loss with Pruning

Fig 2 shows the test loss of the 345M target model during pretraining.
Interestingly, pruning 10% achieves a _lower_ and pruning 20% obtains a
similar test loss, as compared to no-pruning case. As we will see later, the
lower loss indeed leads to improved performances in language modeling and
downstream tasks with pruning.

Figure 2: Test loss of 345M GPT per training token.

Pruning % Proposed Random High Entropy 0 90.23 10 89.32 90.95 98.47 20 86.88
93.40 100.06 30 87.09 99.89 109.51 40 84.19 93.73 107.48 50 91.11 97.08 104.39
60 91.24 101.32 103.30 70 102.93 104.65 152.94 80 101.70 110.39 181.16 90
147.18 165.39 284.59

Table 1: Perplexity of 125M per pruning ratio on One Billion Words corpus.
Bold denotes the best performance for each pruning ratio, and the underline
indicates better performances than no-pruning case.

Pruning % Proposed Random High Entropy 0 52.23 10 50.59 51.56 54.71 20 51.71
52.25 57.99 30 53.46 54.97 61.35 40 55.06 56.91 71.03 50 58.26 59.63 74.50 60
61.97 64.29 85.91 70 62.66 70.22 97.87 80 71.46 85.42 126.69 90 115.03 135.34
212.67

Table 2: Perplexity of 125M per pruning ratio on wikitext-103 corpus. Bold
denotes the best performance for each pruning ratio, and the underline
indicates better performances than no-pruning case.

###  3.3 Language Modeling

We evaluate language modeling of target models using two test corpus. Table 1
shows the perplexity of each model per pruning ratio on One Billion Words
corpus Chelba et al. [2014]. The proposed pruning significantly outperforms
the random pruning. Surprisingly, the proposed pruning achieves _better_
performance up to 40% pruning ratio, and on-par performance up to 60% ratio,
as compared to no-pruning case. We observe a similar trend with different
corpus: Table 2 shows the perplexity results of language modeling on
wikitext-103 corpus Merity et al. [2016]. The results show that, our pruning
can actually improve the generalization capability of the target model, which
we explain as follows. Our pruning removes samples that are less informative
or contain highly redundant information. Such redundancy in the dataset may
cause overfitting such samples similar to duplication, i.e., the model may get
biased towards such information, hampering generalization. Thus, pruning based
on estimates of information content of samples may alleviate overfitting
problems. For comparison, we tested removing samples with high $H(\cdot,q)$
first: see “High Entropy” columns in Table 1 and 2. Interestingly, it performs
much worse than random pruning, which conversely shows the effectiveness of
the proposed importance metric of samples.

###  3.4 Downstream tasks

Figure 3: Text classification results on CoLA dataset in glue benchmark. Left:
125M, Right: 345M. The green line indicates the performance without pruning.

Figure 4: Textual similarity results on SST2 dataset in glue benchmark. Left:
125M, Right: 345M. The green line indicates the performance without pruning.

We finetune and evaluate each model in downstream tasks such as text
classification and textual similarity tasks from glue Wang et al. [2018]. The
left of Fig. 3 shows the text classification results of 125M models on CoLA
dataset, which compare between the proposed method and random pruning. The
results show that, the accuracy of our method is always above that of the
random pruning. Next, we scaled up the target model to 345M, and the right of
Fig. 3 shows that the proposed data pruning method achieves significant
performance gains over random pruning. Notably, the proposed pruning
outperforms no-pruning case with almost up to 50% pruning ratio, i.e., pruning
helps improving the performance of language models.

Fig. 4 illustrates the results of the textual similarity task on SST2 dataset
for 125M and 345M models. For both models, while random pruning continually
decreases performances, the proposed method achieves similar or even better
accuracy than no-pruning case up to 50% pruning ratio.

Overall, we make a similar observation to pretraining case: the proposed
pruning not only reduces the training cost of language models, but also
_improves_ the performance of downstream tasks. We conclude that pruning based
on the information content of samples can improve the generalization
capabilities of the models on downstream tasks.

##  4 Conclusion

In this paper, we took an information-theoretic view of compression for data
pruning in training language models. We use the log-likelihood function of the
model to estimate the compressed description length of a sample, or its
informativeness. Pruning based on the proposed estimate of sample importance
enables removing samples with redundant information, which not only reduces
computational costs, but also, surprisingly, enhances the generalization
capability of language models. Experiments with various corpus and tasks
validated the effectiveness of the proposed pruning.

## References

  * Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_ , 2022.
  * Sorscher et al. [2022] Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos.  Beyond neural scaling laws: beating power law scaling via data pruning.  _Advances in Neural Information Processing Systems_ , 35:19523–19536, 2022.
  * Tan et al. [2024] Haoru Tan, Sitong Wu, Fei Du, Yukang Chen, Zhibin Wang, Fan Wang, and Xiaojuan Qi.  Data pruning via moving-one-sample-out.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of machine learning research_ , 21(140):1–67, 2020.
  * Abbas et al. [2023] Amro Kamal Mohamed Abbas, Kushal Tirumala, Daniel Simig, Surya Ganguli, and Ari S Morcos.  Semdedup: Data-efficient learning at web-scale through semantic deduplication.  In _ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models_ , 2023.
  * Ge et al. [2023] Tao Ge, Hu Jing, Lei Wang, Xun Wang, Si-Qing Chen, and Furu Wei.  In-context autoencoder for context compression in a large language model.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Deletang et al. [2023] Gregoire Deletang, Anian Ruoss, Paul-Ambroise Duquenne, Elliot Catt, Tim Genewein, Christopher Mattern, Jordi Grau-Moya, Li Kevin Wenliang, Matthew Aitchison, Laurent Orseau, et al.  Language modeling is compression.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Huffman [1952] David A Huffman.  A method for the construction of minimum-redundancy codes.  _Proceedings of the IRE_ , 40(9):1098–1101, 1952.
  * Rissanen [1976] Jorma J Rissanen.  Generalized kraft inequality and arithmetic coding.  _IBM Journal of research and development_ , 20(3):198–203, 1976.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Cover [1999] Thomas M Cover.  _Elements of information theory_.  John Wiley & Sons, 1999.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Chelba et al. [2014] Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson.  One billion word benchmark for measuring progress in statistical language modeling.  _Interspeech 2014_ , 2014.
  * Merity et al. [2016] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models.  In _International Conference on Learning Representations_ , 2016.
  * Wang et al. [2018] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, 2018.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.

##  Appendix A Appendix

###  A.1 Hyperparameters

Hyperparameter Value Model size 125M, 345M Pretrain learning rate 5e-4
Pretrain Learning rate scheduler Cosine Pretrain warmup ratio 0.01 % Pretrain
weight decay 0.1 Pretrain batch size 64 Sequence length 1024 Finetuning
learning rate 2e-5 Finetuning weight decay 0.01 Finetuning batch size 32
Optimizer Adam Kingma and Ba [2014]

Table 3: Detailed hyperparameters.