Update card for v12 (evaluation, limitations, windowed inference)

README.md

@@ -1,290 +1,58 @@
 ---
-language:
-- ar
-license: mit
-tags:
-- arabic
-- punctuation-restoration
-- token-classification
-- xlm-roberta
-- natural-language-processing
+language: ar
+license: apache-2.0
 pipeline_tag: token-classification
-base_model: xlm-roberta-large
-model-index:
-- name: Naqta
-  results:
-  - task:
-      type: token-classification
-      name: Arabic Punctuation Restoration
-    dataset:
-      name: Mixed Arabic punctuation restoration corpus
-      type: custom
-    metrics:
-    - type: f1
-      value: 0.8960
-      name: Validation Macro F1
-    - type: accuracy
-      value: 0.9714
-      name: Validation Accuracy
+tags: [arabic, punctuation-restoration, xlm-roberta, token-classification]
 ---
 
-<div align="center">
+# Naqta - Arabic Punctuation Restoration
 
-# 🔤 Naqta — نقطة
+Restores punctuation in Modern Standard Arabic. Two components in one repo:
 
-### Arabic Punctuation Restoration
+| component | path | task |
+|---|---|---|
+| point-mark model | repo root | 8 classes (XLM-RoBERTa-large) |
+| span tagger | `span-tagger/` | paired interjection dashes (BIO, XLM-RoBERTa-base) |
 
-[![Model](https://img.shields.io/badge/🤗%20Model-MostafaMaroof%2FNaqta-blue)](https://huggingface.co/MostafaMaroof/Naqta)
-[![Language](https://img.shields.io/badge/Language-Arabic-green)](https://huggingface.co/MostafaMaroof/Naqta)
-[![Task](https://img.shields.io/badge/Task-Token%20Classification-orange)](https://huggingface.co/MostafaMaroof/Naqta)
-[![License](https://img.shields.io/badge/License-MIT-yellow)](https://opensource.org/licenses/MIT)
-[![Macro F1](https://img.shields.io/badge/Macro%20F1-89.6%25-brightgreen)](https://huggingface.co/MostafaMaroof/Naqta)
-
-</div>
-
----
-
-**Naqta** (Arabic: نقطة, "dot/period") is a high-performance Arabic punctuation restoration model. Given plain unpunctuated Arabic text, it predicts the correct punctuation mark after each word using token-level sequence classification on top of **XLM-RoBERTa Large**.
-
-> 💡 **Try it live** on the [Hugging Face Space](https://huggingface.co/spaces/MostafaMaroof/Naqta)
-
----
-
-## ✨ What Does It Restore?
-
-| Symbol | Name | Example |
-|:---:|---|---|
-| `.` | Period | نهاية الجملة |
-| `،` | Arabic comma | فاصلة عربية |
-| `؟` | Arabic question mark | علامة استفهام |
-| `!` | Exclamation mark | علامة تعجب |
-| `:` | Colon | نقطتان |
-| `؛` | Arabic semicolon | فاصلة منقوطة |
-| `-` | Dash | شرطة |
-
----
-
-## 🏆 Results
-
-### Validation Metrics (v11d — Final)
-
-| Metric | Score |
-|---|---:|
-| 🎯 **Macro F1** | **0.8960** |
-| ✅ Accuracy | 0.9714 |
-
-### Per-Class F1 Score
-
-| Class | Symbol | F1 | Performance |
-|---|:---:|---:|---|
-| Exclamation | `!` | 0.8897 | 🟢 Excellent |
-| Arabic semicolon | `؛` | 0.8042 | 🟢 Excellent |
-| Question mark | `؟` | 0.9665 | 🟢 Excellent |
-| Dash | `-` | 0.9007 | 🟢 Excellent |
-| Arabic comma | `،` | 0.8100 | 🟢 Excellent |
-| Period | `.` | 0.8968 | 🟢 Excellent |
-
----
-
-## 🗂️ Training Data
-
-The model was trained on a large multi-source Arabic corpus totaling over **1.4 million paragraphs** from six diverse sources, covering a broad range of Arabic writing styles and domains.
-
-### Corpus Sources
-
-| Source | Rows | Domain |
-|---|---:|---|
-| **ABC / UNPC** | ~1,020,000 | News & formal Arabic (United Nations Parallel Corpus) |
-| **HF Tashkeel** | ~151,000 | Vocalized Arabic text (diacritized corpus) |
-| **Hindawi E-Books** | ~100,000 | Literary Arabic prose (novels & non-fiction) |
-| **Wikipedia (AR)** | ~98,500 | Encyclopedia articles |
-| **CBT** | ~69,000 | Classical Arabic books & religious texts |
-| **ARCD + XQuAD** | ~2,050 | Arabic QA pairs (rich in question marks `؟`) |
-| **Total (raw)** | **~1,441,000** | — |
-
-> All paragraphs were filtered to contain at least one Arabic letter and one target punctuation mark. Non-target punctuation (e.g., `«»`, `…`, parentheses) was removed before training.
-
-### Punctuation Coverage (raw corpus)
-
-| Mark | Name | Paragraphs | Coverage |
-|:---:|---|---:|---:|
-| `،` | Arabic comma | 922,721 | 64.0% |
-| `:` | Colon | 230,150 | 16.0% |
-| `؛` | Arabic semicolon | 128,744 | 8.9% |
-| `؟` | Question mark | 50,282 | 3.5% |
-| `!` | Exclamation | 15,976 | 1.1% |
-| `-` | Dash | ~1 | <0.1% |
-
-### Data Balance Strategy
-
-To prevent the model from ignoring rare punctuation marks, a targeted oversampling strategy was applied:
-
-| Strategy | Marks | Multiplier | Cap |
-|---|:---:|:---:|---:|
-| Strong oversampling | `؟` `!` | ×8 | 80,000 rows |
-| Light oversampling | `؛` `-` | ×6 | 80,000 rows |
-
-After oversampling, the combined training pool grew to **~2.4 million paragraphs**.
-
-### Dataset Splits
-
-| Split | Sequences | Share |
-|---|---:|---:|
-| Train (capped) | 1,000,000 | 85% |
-| Validation | 40,000 | 10% |
-| Test | — | 5% |
-
-- **Sliding-window context** (window=3 sentences, stride=2) was applied to training data only
-- Validation and test sets remain un-windowed for clean, unbiased evaluation
-- Splits were stratified by the rarest punctuation mark in each sequence
-
-### Preprocessing
-
-- Arabic normalization: alef variants → `ا`, ya variants → `ي`, diacritics stripped
-- Label assigned per word = punctuation mark **following** that word
-- Multi-subword words: only the first subword receives the label; others are masked (`-100`)
-
----
-
-## ⚙️ Model Architecture & Training
-
-| Setting | Value |
-|---|---|
-| Base model | `xlm-roberta-large` (~560M params) |
-| Task | Token classification (8 labels) |
-| Max sequence length | 384 tokens |
-| Training examples | 1,000,000 (capped) |
-| Validation examples | 40,000 |
-
-### Two-Phase Training
-
-| Phase | Epochs | LR | Loss | Notes |
-|---|:---:|---|---|---|
-| Phase 1 | 2 | 2e-5 | Cross-entropy + label smoothing | Full model fine-tuning |
-| Phase 2 | 1 | 6e-6 | Focal loss (γ=2.0) + class weights | Bottom 12 layers frozen |
-
-### Class Weights
-
-Rare class weights were additionally boosted:
-
-| Class | Boost |
-|:---:|---|
-| `؟` | ×1.2 |
-| `!` | ×3.0 |
-| `؛` | ×2.0 |
-| `-` | ×1.3 |
-
----
-
-## 🚀 Quick Start
+## Usage
 
 ```python
-from transformers import AutoTokenizer, AutoModelForTokenClassification
-import torch
-
-repo_id = "MostafaMaroof/Naqta"
-
-tokenizer = AutoTokenizer.from_pretrained(repo_id)
-model = AutoModelForTokenClassification.from_pretrained(repo_id)
-model.eval()
-
-id2label = model.config.id2label
+from transformers import AutoModelForTokenClassification, AutoTokenizer
 
-text = "بلغت نسبة النمو الاقتصادي 4.7 بالمئة خلال الربع الثالث من عام 2024 وهو اعلى مستوى منذ خمس سنوات"
-words = text.split()
+tok = AutoTokenizer.from_pretrained("MostafaMaroof/Naqta")
+model = AutoModelForTokenClassification.from_pretrained("MostafaMaroof/Naqta")
 
-inputs = tokenizer(
-    words,
-    is_split_into_words=True,
-    return_tensors="pt",
-    truncation=True,
-    max_length=384,
-)
-
-with torch.no_grad():
-    logits = model(**inputs).logits
-
-pred_ids = logits.argmax(dim=-1)[0].tolist()
-word_ids = inputs.word_ids(batch_index=0)
-
-restored_words = []
-previous_word_id = None
-for token_id, word_id in zip(pred_ids, word_ids):
-    if word_id is None or word_id == previous_word_id:
-        continue
-    word = words[word_id]
-    label = id2label[token_id]
-    if label != "O":
-        word = word + label
-    restored_words.append(word)
-    previous_word_id = word_id
-
-restored_text = " ".join(restored_words)
-print(restored_text)
-# → بلغت نسبة النمو الاقتصادي 4.7 بالمئة خلال الربع الثالث من عام 2024، وهو اعلى مستوى منذ خمس سنوات.
+span_tok = AutoTokenizer.from_pretrained("MostafaMaroof/Naqta", subfolder="span-tagger")
+span_model = AutoModelForTokenClassification.from_pretrained(
+    "MostafaMaroof/Naqta", subfolder="span-tagger")
 ```
 
----
+**Use sliding-window inference for long text.** The model was fine-tuned on short
+windows; a single truncated pass leaves everything past roughly 230 words
+unpunctuated. See `app.py` in the Space for a reference implementation.
 
-## 📖 Example
+## Evaluation
 
-**Input** (unpunctuated):
-```
-اذا اردت ان تنجح في حياتك فعليك ان تحدد اهدافك واضحة وان تعمل بجد واستمرارية ولا تيأس عند اول عقبة تواجهها
-```
+Held-out set of 91 Modern Standard Arabic passages annotated to a published
+punctuation guideline (about 27k tokens):
 
-**Output** (restored):
-```
-اذا اردت ان تنجح في حياتك، فعليك ان تحدد اهدافك واضحة، وان تعمل بجد واستمرارية، ولا تيأس عند اول عقبة تواجهها.
-```
+| metric | previous | current |
+|---|---|---|
+| Macro F1 | 54.64 | **61.60** |
+| Weighted F1 | 89.90 | **91.54** |
+| Token accuracy | 90.96 | **91.33** |
 
-**Question example:**
-```
-من اخترع الهاتف وفي اي سنة تم ذلك وما هي اهمية هذا الاختراع
-```
-```
-من اخترع الهاتف، وفي اي سنة تم ذلك، وما هي اهمية هذا الاختراع؟
-```
-
----
-
-## 🎯 Intended Use
-
-Naqta is well-suited for:
-
-- 🎙️ **ASR post-processing** — restoring punctuation in Arabic speech transcripts
-- 📄 **Readability enhancement** — making raw Arabic text easier to read
-- 🔧 **NLP preprocessing** — improving text quality for downstream Arabic NLP tasks
-- 🔬 **Research** — Arabic punctuation restoration benchmark evaluation
-
----
+## Limitations
 
-## ⚠️ Limitations
+- Exclamation and semicolon are over-predicted (precision 20.4 and 36.2): the
+  model finds these marks but does not withhold them reliably.
+- Brackets and quotation marks are not produced. The span tagger covers them in
+  training but their precision is too low to enable in production.
+- Punctuation conventions vary across Arabic sources; performance is highest on
+  text punctuated to a consistent editorial guideline.
 
-- Punctuation restoration is partly stylistic — multiple valid outputs may exist for a single input.
-- Performance may degrade on highly dialectal, technical, or domain-specific text.
-- The model does not predict quotation marks or dialogue markers (`«»`).
-- Very short or fragmented text (< 5 words) may produce less reliable results.
-- The model predicts punctuation position only and does not perform grammar correction.
+## Training
 
----
-
-## 📜 License
-
-This model is released under the **MIT License**.
-
----
-
-## 🔗 Citation
-
-If you use Naqta in your work, please reference:
-
-```bibtex
-@misc{naqta2025,
-  title     = {Naqta: Arabic Punctuation Restoration with XLM-RoBERTa},
-  author    = {MostafaMaroof},
-  year      = {2025},
-  publisher = {Hugging Face},
-  url       = {https://huggingface.co/MostafaMaroof/Naqta}
-}
-```
+XLM-RoBERTa-large fine-tuned on about 1.3M paragraphs (classical and modern
+books, Wikipedia, news, QA corpora) with weighted focal loss, oversampling of
+rare marks, and a final adaptation stage on guideline-conforming data.