VoxSign/Speech-to-Gloss

VoxSign — English → Ugandan Sign Language (USL) Gloss

flan-t5-base v10. Fully fine-tuned (no LoRA, no adapters). Converts English sentences into a flat uppercase USL gloss approximation.

Read this before using the model

This gloss is not Ugandan Sign Language. Real USL carries spatial agreement, classifier predicates, non-manual grammatical marking (brow position, negation headshake), verb directionality and role shift. None of that survives as a string of tokens. What this model produces is a consistent, broad-vocabulary gloss approximation — a foundation for a signing system, not a translation.

The gloss conventions are not authoritative. They are a best reading of salvaged clean data plus general sign-language gloss practice, revised once (2026-09-05) by a single signer. Before output reaches Deaf users, the specification and a sample of the test set need review by native USL signers, ideally through UNAD (Uganda National Association of the Deaf).

This model is not what serves production. The VoxSign endpoint runs a deterministic rule engine, because the rule engine beats this model on every metric in every length bucket. See below. The model is published for reproducibility, rollback and further research.

Intended use

Research and development on English→USL gloss. Not suitable for unsupervised deployment to Deaf users, and not a substitute for a human interpreter in any setting where accuracy matters — medical, legal, educational or emergency.

Evaluation

308-row hand-authored gold test set, verified to share no source sentence with train or validation (0 leaked sources, 0 leaked targets). Targets were written by hand, not generated by the rule engine — a reference produced by the system it measures would measure nothing.

Scored with sacreBLEU through the same post-processor the endpoint applies.

	BLEU	chrF++	exact	validity	hallucination
rule engine (production)	90.61	96.90	83.8%	100%	0.0%
flan-t5-base v10 (this model)	85.77	94.93	70.5%	100%	0.0%
flan-t5-base v9	85.07	94.37	69.8%	100%	0.0%

By input length — report this, not the aggregate. The gold set is 80% short sentences, so the overall figure describes short declaratives rather than conversation:

bucket	n	BLEU	chrF++	exact	hallucination
≤8 words	247	85.63	95.71	78.5%	0.0%
9–18 words	37	87.50	95.40	48.6%	0.0%
19+ words	24	84.25	92.96	20.8%	0.0%

• validity — share of outputs satisfying the gloss specification.
• hallucination — share of outputs containing a token not derivable from the input.

Both are 100% / 0.0% in every bucket. Note what that does not cover: these checks catch invented tokens, not omitted ones. A model that silently drops a negation scores 0.0% hallucination while inverting the meaning of the sentence.

Why the rule engine is primary

The training targets are generated by that same rule engine, and the engine reproduces 90.6% of them exactly. The model is therefore fitting the engine's own output and can only approach it from below. This has been measured across six training runs; v10 is the best of them and still trails by 4.84 BLEU and 13.3 points of exact match.

Raising the ceiling requires human-authored targets, not hyperparameter search.

Training

base	google/flan-t5-base, fully fine-tuned
corpus	23,597 train / 5,900 validation, rebuilt 2026-09-10
epochs	10
batch	32 × 2 grad accumulation (effective 64)
learning rate	1e-4, linear decay
label smoothing	0.1
precision	bfloat16
seed	20260826
hardware	1× H100, 9.7 min

Corpus sources: Tatoeba English sentences (CC-BY 2.0 FR), filtered for grammaticality and length-stratified; synthesised multi-clause sentences for the long tail; 530 salvaged human-authored rows. Gloss targets generated by the VoxSign rule engine and gated by an automated QA suite (no templated rows, no duplicate inputs, no train/test leakage).

Two constraints that are not optional

Never load or train this model in float16. T5 was trained in bfloat16 and overflows fp16's 65504 ceiling in T5DenseReluDense.wo, producing inf and then NaN through T5LayerNorm — a silent, total failure. Use bfloat16 on Ampere or newer, float32 otherwise. A T4 is Turing and has no bfloat16, so a T4 means float32.

Targets are trained in lowercase. Uppercase is a presentation convention applied at serving time. T5's sentencepiece vocabulary was learned from mixed-case text, so uppercase gloss costs 2.08 tokens per word against 1.15 lowercase — 44.5% longer — and smears each sign across subword pieces that carry no meaning alone. A controlled run (identical in every parameter except target casing) measured 82.14 BLEU / 1.0% hallucination lowercase against 68.57 / 27.6% uppercase.

Usage

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch

REPO = "VoxSign/Speech-to-Gloss"
REVISION = "53366efa5832c737fffa4d2eff6bcf854fe246d6"          # pin a SHA, never "main"

tokenizer = AutoTokenizer.from_pretrained(REPO, revision=REVISION)
model = AutoModelForSeq2SeqLM.from_pretrained(
    REPO, revision=REVISION, torch_dtype=torch.bfloat16   # never float16
)

text = "I am going to the hospital tomorrow"
batch = tokenizer(f"translate English to USL gloss: {text}",
                  return_tensors="pt", max_length=256, truncation=True)
out = model.generate(**batch, max_length=160, num_beams=4, early_stopping=True)
print(tokenizer.decode(out[0], skip_special_tokens=True).upper())

The translate English to USL gloss: prefix is required — it was present on every training example, and omitting it degrades output.

Always pin a commit SHA rather than main. A deploy that cannot be reproduced cannot be rolled back, and a model trained against a superseded convention degrades invisibly: an earlier revision (v8) emits a token the current convention forbids, which would fail validation on roughly a third of clauses while still reporting 0% hallucination.

Long input should be segmented into clauses before generation and validated per clause, as the production service does. Generating a 30-word sentence as one unit is what the first version of this system did, and it collapsed.

Limitations

• Flat gloss cannot express USL's spatial or non-manual grammar.
• Conventions are provisional and await native-signer review.
• Accuracy falls sharply with input length: 78.5% exact at ≤8 words, 20.8% at 19+.
• Validation catches invented tokens, not omitted ones.
• Trained on Tatoeba-derived English, which is not Ugandan conversational English; named entities and local vocabulary outside the lexicon are passed through as-is.
• Gloss for indefinite articles is one dimension the test set does not encode, so it is unmeasured rather than verified.

Citation

English source sentences from the Tatoeba Project, CC-BY 2.0 FR.