Instructions to use jhu-clsp/mmBERT-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use jhu-clsp/mmBERT-small with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="jhu-clsp/mmBERT-small")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("jhu-clsp/mmBERT-small") model = AutoModelForMaskedLM.from_pretrained("jhu-clsp/mmBERT-small", device_map="auto") - Inference
- Notebooks
- Google Colab
- Kaggle
Update pipeline tag to feature-extraction
Browse filesThis PR updates the `pipeline_tag` for the `mmBERT-base` model from `fill-mask` to `feature-extraction`.
As an encoder-only model primarily used for tasks like classification, embedding, and retrieval, `feature-extraction` more accurately reflects its core functionality and common usage on the Hugging Face Hub. This change will help users discover the model more effectively for embedding and general feature extraction tasks.
The existing content, including the detailed usage examples and links, remains unchanged.
README.md
CHANGED
|
@@ -1,13 +1,13 @@
|
|
| 1 |
---
|
| 2 |
-
license: mit
|
| 3 |
datasets:
|
| 4 |
- jhu-clsp/mmbert-decay
|
| 5 |
- jhu-clsp/mmbert-midtraining
|
| 6 |
- jhu-clsp/mmbert-pretrain-p1-fineweb2-langs
|
| 7 |
- jhu-clsp/mmbert-pretrain-p2-fineweb2-remaining
|
| 8 |
- jhu-clsp/mmbert-pretrain-p3-others
|
| 9 |
-
pipeline_tag: fill-mask
|
| 10 |
library_name: transformers
|
|
|
|
|
|
|
| 11 |
---
|
| 12 |
|
| 13 |
# mmBERT: A Modern Multilingual Encoder
|
|
@@ -59,11 +59,11 @@ outputs = model(**inputs)
|
|
| 59 |
|
| 60 |
mmBERT represents the first significant advancement over XLM-R for massively multilingual encoder models. Key features include:
|
| 61 |
|
| 62 |
-
1.
|
| 63 |
-
2.
|
| 64 |
-
3.
|
| 65 |
-
4.
|
| 66 |
-
5.
|
| 67 |
|
| 68 |
The model uses bidirectional attention with masked language modeling objectives, optimized specifically for multilingual understanding and cross-lingual transfer.
|
| 69 |
|
|
@@ -92,7 +92,7 @@ mmBERT training data is publicly available across different phases:
|
|
| 92 |
|:------|:--------|:-------|:------------|
|
| 93 |
| Pre-training P1 | [mmbert-pretrain-p1](https://huggingface.co/datasets/jhu-clsp/mmbert-pretrain-p1-fineweb2-langs) | 2.3T | 60 languages, foundational training |
|
| 94 |
| Pre-training P2 | [mmbert-pretrain-p2](https://huggingface.co/datasets/jhu-clsp/mmBERT-pretrain-p2-fineweb2-remaining) | - | Extension data for pre-training phase |
|
| 95 |
-
| Pre-training P3 | [mmbert-pretrain-p3](https://huggingface.co/datasets/jhu-clsp/
|
| 96 |
| Mid-training | [mmbert-midtraining](https://huggingface.co/datasets/jhu-clsp/mmbert-midtraining-data) | 600B | 110 languages, context extension to 8K |
|
| 97 |
| Decay Phase | [mmbert-decay](https://huggingface.co/datasets/jhu-clsp/mmbert-decay-data) | 100B | 1833 languages, premium quality |
|
| 98 |
|
|
@@ -479,5 +479,4 @@ If you use mmBERT in your research, please cite our work:
|
|
| 479 |
primaryClass={cs.CL},
|
| 480 |
url={https://arxiv.org/abs/2509.06888},
|
| 481 |
}
|
| 482 |
-
```
|
| 483 |
-
"""
|
|
|
|
| 1 |
---
|
|
|
|
| 2 |
datasets:
|
| 3 |
- jhu-clsp/mmbert-decay
|
| 4 |
- jhu-clsp/mmbert-midtraining
|
| 5 |
- jhu-clsp/mmbert-pretrain-p1-fineweb2-langs
|
| 6 |
- jhu-clsp/mmbert-pretrain-p2-fineweb2-remaining
|
| 7 |
- jhu-clsp/mmbert-pretrain-p3-others
|
|
|
|
| 8 |
library_name: transformers
|
| 9 |
+
license: mit
|
| 10 |
+
pipeline_tag: feature-extraction
|
| 11 |
---
|
| 12 |
|
| 13 |
# mmBERT: A Modern Multilingual Encoder
|
|
|
|
| 59 |
|
| 60 |
mmBERT represents the first significant advancement over XLM-R for massively multilingual encoder models. Key features include:
|
| 61 |
|
| 62 |
+
1. **Massive Language Coverage** - Trained on over 1800 languages with progressive inclusion strategy
|
| 63 |
+
2. **Modern Architecture** - Built on ModernBERT foundation with Flash Attention 2 and unpadding techniques
|
| 64 |
+
3. **Novel Training Recipe** - Introduces inverse mask scheduling and temperature sampling
|
| 65 |
+
4. **Open Training Data** - Complete 3T+ token dataset publicly available
|
| 66 |
+
5. **Decay Phase Innovation** - Demonstrates effective learning of low-resource languages in final training phase
|
| 67 |
|
| 68 |
The model uses bidirectional attention with masked language modeling objectives, optimized specifically for multilingual understanding and cross-lingual transfer.
|
| 69 |
|
|
|
|
| 92 |
|:------|:--------|:-------|:------------|
|
| 93 |
| Pre-training P1 | [mmbert-pretrain-p1](https://huggingface.co/datasets/jhu-clsp/mmbert-pretrain-p1-fineweb2-langs) | 2.3T | 60 languages, foundational training |
|
| 94 |
| Pre-training P2 | [mmbert-pretrain-p2](https://huggingface.co/datasets/jhu-clsp/mmBERT-pretrain-p2-fineweb2-remaining) | - | Extension data for pre-training phase |
|
| 95 |
+
| Pre-training P3 | [mmbert-pretrain-p3](https://huggingface.co/datasets/jhu-clsp/mmbert-pretrain-p3-others) | - | Final pre-training data |
|
| 96 |
| Mid-training | [mmbert-midtraining](https://huggingface.co/datasets/jhu-clsp/mmbert-midtraining-data) | 600B | 110 languages, context extension to 8K |
|
| 97 |
| Decay Phase | [mmbert-decay](https://huggingface.co/datasets/jhu-clsp/mmbert-decay-data) | 100B | 1833 languages, premium quality |
|
| 98 |
|
|
|
|
| 479 |
primaryClass={cs.CL},
|
| 480 |
url={https://arxiv.org/abs/2509.06888},
|
| 481 |
}
|
| 482 |
+
```
|
|
|