# Using Seq2SeqTrainer for decoders?

**URL:** <https://discuss.huggingface.co/t/using-seq2seqtrainer-for-decoders/132982>\
**Category:** 🤗Transformers\
**Created:** [December 25, 2024, 8:10am UTC](https://discuss.huggingface.co/t/using-seq2seqtrainer-for-decoders/132982 "2024-12-25T08:10:37Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![guymorlan](https://avatars.discourse-cdn.com/v4/letter/g/3da27b/32.png) [@guymorlan](https://discuss.huggingface.co/u/guymorlan)\
**Post date:** [December 25, 2024, 8:10am UTC](https://discuss.huggingface.co/t/using-seq2seqtrainer-for-decoders/132982/1 "2024-12-25T08:10:37Z")

</div>

I have a case where I can’t use SFTTrainer because:

1. My data is pretokenized
2. I need to use predict\_with\_generate to get validation set evals during training

The regular Trainer can handle pretokenized data but doesn’t have predict\_with\_generate.  
Can I use Seq2SeqTrainer or does that only make sense for encoder-decoder models?

I’m fine tuning a 7B LLM with Qlora.

Thanks!