# TFT5ForConditionalGeneration with custom loss

**URL:** <https://discuss.huggingface.co/t/tft5forconditionalgeneration-with-custom-loss/16457>\
**Category:** Beginners\
**Created:** [April 4, 2022, 3:19pm UTC](https://discuss.huggingface.co/t/tft5forconditionalgeneration-with-custom-loss/16457 "2022-04-04T15:19:07Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![BenM61](https://avatars.discourse-cdn.com/v4/letter/b/b19c9b/32.png) [@BenM61](https://discuss.huggingface.co/u/BenM61)\
**Post date:** [April 4, 2022, 3:19pm UTC](https://discuss.huggingface.co/t/tft5forconditionalgeneration-with-custom-loss/16457/1 "2022-04-04T15:19:07Z")

</div>

Hey, I want to fine-tune `TFT5ForConditionalGeneration`. Very similar to [this thread](https://discuss.huggingface.co/t/t5-user-defined-loss-function/566/4), but I want to use the teacher forcing (i.e. use the `forward`, rather than `generate` method).

One thing that crossed my mind is to inherit from `TFT5ForConditionalGeneration`, override the forward with a forward exactly like the original (from [the source code](https://github.com/huggingface/transformers/blob/77321481247787c97568c3b9f64b19e22351bab8/src/transformers/models/t5/modeling_t5.py)), but change the loss to my loss. Is that a good idea?

I will add more context below, but my question is how to do something else than `CrossEntropyLoss` for the t5 model loss.

**Context:**  
My task is- given tokenized song lyrics, classify the genres of that song (multilabel classification).

The output I want from t5 is a string containing all the genres predicted, followed by eos token, and than padding tokens to keep all samples with equal length (some songs has more than one genre). For example

```auto
'FUNK,POP,,,</s> <pad> <pad> <pad> <pad> <pad>',

```

When I used the default t5 loss (training via forward), I decreased the loss during training but the eval loss is bad and it predicts nonsense. I think the model can just learn where to place the paddings to get a loss improvement and that’s not what I want.

I’m not sure how to implement differential loss while ignore the pads, but for now I only ask how to change the loss in general.

I can add the code if you want, Thanks