Amharic Skip-gram Word Embeddings (baseline_001)
A small Skip-gram word embedding model for Amharic (α ααα), implemented from scratch in pure Python (lists, loops, math, random; no NumPy, no autograd, no pretrained vectors) as part of a university/internship assignment.
This is an educational baseline, not a production model. It has 665 words and 16 dimensions. Please read the Limitations section.
- Code & tests: https://github.com/YibeltalMarie/ethiopian-word-embeddings
- Developed by: Yibeltal (AASTU / iCog Labs)
Model details
| Architecture | Skip-gram, full softmax, plain SGD (one pair per update) |
| Parameters | E (VΓd) input embeddings, U (dΓV) output weights |
| Vocabulary size (V) | 665 |
| Embedding dimension (d) | 16 |
| Context window | 2 |
| Learning rate | 0.1 |
| Epochs | 8 |
| Initialization | uniform(-0.1, 0.1), random.Random(1) |
| Pair shuffle | reshuffled every epoch, seed 5 |
| Training time | ~6,070 s (about 101 min), pure Python |
The embedding for a word is its row of E.
Training data
- Source: Amharic Wikipedia dump
amwiki-20260901(pages-articles-multistream.xml.bz2), SHA-25661deab0e1f33c56d5fd1212a9e5e8d4258211b01e181f747e3a7106285fe9706. - Preprocessing: wiki-markup cleaning, sentence splitting, tokenization, exact-duplicate sentence removal.
- Slice: 10,000 sentences sampled with seed 42 from 138,271; vocabulary = words with frequency β₯ 20 that contain an Ethiopic letter. Out-of-vocabulary tokens are deleted; sentences left with fewer than 2 tokens are dropped.
- Result: 7,274 sentences, 43,837 training tokens kept (of 126,470).
- The raw corpus is not included in this repository.
Files
| File | Content |
|---|---|
E.json |
input embeddings, 665 Γ 16 |
U.json |
output weights, 16 Γ 665 |
vocabulary.json |
list of 665 words; index = row of E |
config.json, metadata.json |
hyperparameters, seeds, corpus provenance |
training_results.json |
per-epoch loss, epoch times, nearest-neighbour outputs |
example.py |
dependency-free loader + nearest-neighbour query |
Usage
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("Yibeltal1/amharic-skipgram-baseline")
sys.path.insert(0, path)
from example import load, nearest
E, vocab = load(path)
print(nearest("α¨α°α", E, vocab, k=3))
# [('α¨α°αα', 0.8275), ('α¨ααα΅', 0.7906), ('ααα΅', 0.7618)]
Evaluation
Optimization (does it learn the training objective?)
Mean training cross-entropy per epoch (ln V = 6.4998 is the loss of a uniform guess):
| Epoch | 0 (before) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|---|
| Loss | 6.4999 | 5.4891 | 5.4464 | 5.4201 | 5.4109 | 5.4297 | 5.4224 | 5.4293 | 5.4233 |
Loss falls from 6.50 to 5.42 (about 17%), mostly in the first epoch, then plateaus around 5.41β5.43. The model learned the objective, with diminishing returns after about 4 epochs at this learning rate.
Embedding quality (do vectors show useful relationships?)
Qualitative only: top-3 cosine nearest neighbours (query excluded) for five words.
| Query | Neighbours (cosine) |
|---|---|
| α¨α°α (city) | α¨α°αα 0.828 (the city), α¨ααα΅ 0.791 (of trade), ααα΅ 0.762 (state) |
| α°αα«α (blue) | ααα 0.880 (color), αα₯α 0.819 (number), αα 0.808 (red) |
| ααα (world) | αααα₯α³α΅ 0.793 (nations), α αα 0.792 (-wide/international), α³αα 0.763 (famous) |
| αα (day) | ααα΅ 0.761 (to mean), αα 0.737 (month), α α½α³ 0.732 (disease) |
| α¨α ααα (of Amharic) | α¨αα 0.766, αα³α 0.757 (example), αα 0.757 (who) |
Some neighbours are sensible (α¨α°α β α¨α°αα, α°αα«α β ααα/αα, αα β αα), and some are noise (α°αα«α β αα₯α, αα β α α½α³). Training loss alone does not establish semantic quality, and no intrinsic benchmark (e.g. word similarity or analogy test) has been run, because none exists for this vocabulary.
Limitations
- Tiny scale: 665 words, ~44k training tokens, 16 dimensions. Rare words, most inflected forms and most of the Amharic lexicon are missing.
- No subword information: Amharic is morphologically rich, so inflected forms of one word are separate (or absent) entries.
- Out-of-vocabulary words cannot be embedded.
- Frequent function words (e.g. α , αα) carry large norms and can dominate neighbours.
- Domain bias: Wikipedia text only; reflects its topics and any biases.
- Single run, single seed; no hyperparameter sweeps. Cosine scores are not calibrated measures of similarity.
- Not evaluated for any downstream task. Do not use for decisions affecting people.
Intended use
Teaching, reproducing Skip-gram mathematics by hand, and as a baseline for experiments. Not intended for production NLP.
License and attribution
The vectors are derived from Amharic Wikipedia text and released under CC BY-SA 4.0, with attribution to Wikipedia contributors (https://am.wikipedia.org). Derivatives must be shared under the same license. Source code in the GitHub repository is licensed separately.
Citation
Yibeltal. Amharic Skip-gram Word Embeddings (baseline_001), 2026. https://huggingface.co/Yibeltal1/amharic-skipgram-baseline
- Downloads last month
- 17