Verification of script to train a LLM on supervised data

John6666
Wed, 25 Jun 2025 15:19:43 +0000

I see, that makes sense for research and optimization purposes.

the tokenization part

One is about special token masking. I’m not sure if this needs to be passed when using a non-embedding model. I just wondered.
https://stackoverflow.com/questions/61707371/about-get-special-tokens-mask-in-huggingface-transformers

github.com/huggingface/tokenizers

"return_special_tokens_mask" does not mask new tokens when added via "add_special_tokens"

opened 02:41PM - 31 Aug 21 UTC

closed 01:49AM - 24 Mar 24 UTC

vgoklani

Stale

Consider the following example: ```python from transformers.models.roberta.t…okenization_roberta_fast import RobertaTokenizerFast pretrained_model_name_or_path = "distilroberta-base" tokenizer = RobertaTokenizerFast.from_pretrained(pretrained_model_name_or_path, use_fast=True) special_tokens_dict = {'additional_special_tokens': ['BASEBALL_CARDS']} tokenizer.add_special_tokens(special_tokens_dict) text = "Let's go buy some BASEBALL_CARDS at Yankee stadium" encoding = tokenizer.encode_plus(text, return_special_tokens_mask=True) " ".join(tokenizer.convert_ids_to_tokens(input_id) for input_id in encoding['input_ids']) # " Let 's Ġgo Ġbuy Ġsome Ġ BASEBALL_CARDS Ġat ĠYankee Ġstadium " encoding['special_tokens_mask'] # [1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1] ``` The special_tokens_mask masks both the \<s\> and the \</s\> tokens, but skips the newly created "BASEBALL_CARDS" special token. Is this the intended behavior, or a bug? Also, when does one use "add_special_tokens" vs "add_tokens"? Thanks!

The other is about chat templates. If using an instruct model, it would be desirable to pass tokens that have been converted from appropriately processed strings for each LLM. I guess this probably isn’t an issue with the base model?

huggingface.co

Templates