Viveka-34M (विवेक)
Viveka is Sanskrit for discernment: telling what is right from what only looks right.
A small English language model trained from scratch (its own tokenizer, random start) on free Colab T4 GPUs. Its strength is judging: given a few versions of a sentence, it tells which one makes the most sense.
Model
| Design | Llama-style (rotary positions, SwiGLU, RMSNorm), 8 layers, width 512, 33.9M numbers |
| Tokenizer | its own byte-level BPE, 16,000 word pieces |
| Context | 256 word pieces |
| Training | 45 minutes from scratch, then 150 more minutes at a lower learning rate (one T4 each) |
| Stored as | half precision, 68 MB |
Training text
| Source | Licence |
|---|---|
| FineWeb-Edu (educational web pages) | ODC-BY |
| Project Gutenberg books | public domain |
| English Wikipedia | CC BY-SA |
| IAM handwriting texts (first 45 minutes only, 10%) | non-commercial research |
Because of the IAM texts, this model is non-commercial (CC BY-NC 4.0).
Results
On 20 fresh everyday sentences that no version trained on (lower is better):
| Perplexity per word | Sense test | |
|---|---|---|
| First Viveka-34M | 233.8 | 12 / 12 |
| This version | 176.9 | 12 / 12 |
The sense test asks it to pick the sensible sentence ("She drank a glass of cold water." over "...cold window.").
Use
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained('vrishv/Viveka-34M')
lm = AutoModelForCausalLM.from_pretrained('vrishv/Viveka-34M', dtype='float32')
x = tok('The cat sat on the', return_tensors='pt')
print(tok.decode(lm.generate(**x, max_new_tokens=20, do_sample=True, top_k=40)[0]))
Limitations
Very small: it knows grammar and common phrases but little about the world. It is meant for judging sentences, not chatting or facts. English only; it may repeat biases or old-fashioned language from its training text.
About the maker
Viveka was built by Vrishvardhan Sharma (vrishv), a 9-year-old self-taught builder from India. His other work:
- GitaVardhan: a free, offline Bhagavad Gita app in 13 Indian languages, and a pocket-sized voice-controlled Gita device he designed, wired and coded himself.
- Sanskrit Vardhan: an app for learning Sanskrit through the Bhagavad Gita, with word-by-word meanings and step-by-step Pāṇinian derivations. Launched on 20 October 2026 by Prof. Shrinivasa Varakhedi, Vice-Chancellor of Central Sanskrit University.
- Kannada, Tamil and Telugu Vardhan: apps that teach everyday spoken Kannada, Tamil and Telugu.
- Hindi Vardhan (hindivardhan.in): a dictionary of pure, Sanskrit-rooted Hindi. Over 6,000 borrowed words in everyday Hindi, each with native alternatives checked against five classical Sanskrit dictionaries.
- AksharaVardhan: a handwriting reader that runs on an ordinary computer.
- graphics-studio: a free Space that makes and edits images, videos, music and narration.
- Shodhana-34M: Viveka trained further to correct lines misread by a handwriting reader.
- Jigyasa Simple: 473 everyday science questions with short, plain-text answers.
- Jigyasa Medium and Jigyasa Big: 969 and 970 deeper science questions, built from his own questions, with answers of up to 7 and 10 sentences.
He has memorised the Bhagavad Gita and Pāṇini's Aṣṭādhyāyī, won national Sanskrit recitation competitions, and speaks six languages. More at vrishvardhan.in.
CC BY-NC 4.0 © 2026 Vrishvardhan Sharma.
- Downloads last month
- 95