Verification of script to train a LLM on supervised data

John6666
Tue, 24 Jun 2025 05:18:00 +0000

df['train'] = df['test']

This makes the dataset smaller, so it seems prone to overfitting, but it’s probably just set up this way for testing purposes.

I don’t see any obvious issues with the data preprocessing code. I’m not sure if padding tokens should be included in the labels, and if that differs between Seq2Seq and CausalLM… I’m not sure about that part.

The LR is slightly high (though I don’t think it’s a problem), and the lack of LoRA Dropout or Weight Decay settings might be related to overfitting.

huggingface.co

Supervised Fine-tuning Trainer

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

docs.unsloth.ai

LoRA Hyperparameters Guide | Unsloth Documentation

Welcome! This guide covers the best practices for LoRA hyperparameters and how they affect the fine-tuning process.