Fill-Mask
Transformers
PyTorch
modernbert
nielsr HF Staff commited on
Commit
759db26
·
verified ·
1 Parent(s): 13f0442

Update pipeline tag to feature-extraction

Browse files

This PR updates the `pipeline_tag` for the `mmBERT-base` model from `fill-mask` to `feature-extraction`.

As an encoder-only model primarily used for tasks like classification, embedding, and retrieval, `feature-extraction` more accurately reflects its core functionality and common usage on the Hugging Face Hub. This change will help users discover the model more effectively for embedding and general feature extraction tasks.

The existing content, including the detailed usage examples and links, remains unchanged.

Files changed (1) hide show
  1. README.md +9 -10
README.md CHANGED
@@ -1,13 +1,13 @@
1
  ---
2
- license: mit
3
  datasets:
4
  - jhu-clsp/mmbert-decay
5
  - jhu-clsp/mmbert-midtraining
6
  - jhu-clsp/mmbert-pretrain-p1-fineweb2-langs
7
  - jhu-clsp/mmbert-pretrain-p2-fineweb2-remaining
8
  - jhu-clsp/mmbert-pretrain-p3-others
9
- pipeline_tag: fill-mask
10
  library_name: transformers
 
 
11
  ---
12
 
13
  # mmBERT: A Modern Multilingual Encoder
@@ -59,11 +59,11 @@ outputs = model(**inputs)
59
 
60
  mmBERT represents the first significant advancement over XLM-R for massively multilingual encoder models. Key features include:
61
 
62
- 1. **Massive Language Coverage** - Trained on over 1800 languages with progressive inclusion strategy
63
- 2. **Modern Architecture** - Built on ModernBERT foundation with Flash Attention 2 and unpadding techniques
64
- 3. **Novel Training Recipe** - Introduces inverse mask scheduling and temperature sampling
65
- 4. **Open Training Data** - Complete 3T+ token dataset publicly available
66
- 5. **Decay Phase Innovation** - Demonstrates effective learning of low-resource languages in final training phase
67
 
68
  The model uses bidirectional attention with masked language modeling objectives, optimized specifically for multilingual understanding and cross-lingual transfer.
69
 
@@ -92,7 +92,7 @@ mmBERT training data is publicly available across different phases:
92
  |:------|:--------|:-------|:------------|
93
  | Pre-training P1 | [mmbert-pretrain-p1](https://huggingface.co/datasets/jhu-clsp/mmbert-pretrain-p1-fineweb2-langs) | 2.3T | 60 languages, foundational training |
94
  | Pre-training P2 | [mmbert-pretrain-p2](https://huggingface.co/datasets/jhu-clsp/mmBERT-pretrain-p2-fineweb2-remaining) | - | Extension data for pre-training phase |
95
- | Pre-training P3 | [mmbert-pretrain-p3](https://huggingface.co/datasets/jhu-clsp/mmBERT-pretrain-p3-others) | - | Final pre-training data |
96
  | Mid-training | [mmbert-midtraining](https://huggingface.co/datasets/jhu-clsp/mmbert-midtraining-data) | 600B | 110 languages, context extension to 8K |
97
  | Decay Phase | [mmbert-decay](https://huggingface.co/datasets/jhu-clsp/mmbert-decay-data) | 100B | 1833 languages, premium quality |
98
 
@@ -479,5 +479,4 @@ If you use mmBERT in your research, please cite our work:
479
  primaryClass={cs.CL},
480
  url={https://arxiv.org/abs/2509.06888},
481
  }
482
- ```
483
- """
 
1
  ---
 
2
  datasets:
3
  - jhu-clsp/mmbert-decay
4
  - jhu-clsp/mmbert-midtraining
5
  - jhu-clsp/mmbert-pretrain-p1-fineweb2-langs
6
  - jhu-clsp/mmbert-pretrain-p2-fineweb2-remaining
7
  - jhu-clsp/mmbert-pretrain-p3-others
 
8
  library_name: transformers
9
+ license: mit
10
+ pipeline_tag: feature-extraction
11
  ---
12
 
13
  # mmBERT: A Modern Multilingual Encoder
 
59
 
60
  mmBERT represents the first significant advancement over XLM-R for massively multilingual encoder models. Key features include:
61
 
62
+ 1. **Massive Language Coverage** - Trained on over 1800 languages with progressive inclusion strategy
63
+ 2. **Modern Architecture** - Built on ModernBERT foundation with Flash Attention 2 and unpadding techniques
64
+ 3. **Novel Training Recipe** - Introduces inverse mask scheduling and temperature sampling
65
+ 4. **Open Training Data** - Complete 3T+ token dataset publicly available
66
+ 5. **Decay Phase Innovation** - Demonstrates effective learning of low-resource languages in final training phase
67
 
68
  The model uses bidirectional attention with masked language modeling objectives, optimized specifically for multilingual understanding and cross-lingual transfer.
69
 
 
92
  |:------|:--------|:-------|:------------|
93
  | Pre-training P1 | [mmbert-pretrain-p1](https://huggingface.co/datasets/jhu-clsp/mmbert-pretrain-p1-fineweb2-langs) | 2.3T | 60 languages, foundational training |
94
  | Pre-training P2 | [mmbert-pretrain-p2](https://huggingface.co/datasets/jhu-clsp/mmBERT-pretrain-p2-fineweb2-remaining) | - | Extension data for pre-training phase |
95
+ | Pre-training P3 | [mmbert-pretrain-p3](https://huggingface.co/datasets/jhu-clsp/mmbert-pretrain-p3-others) | - | Final pre-training data |
96
  | Mid-training | [mmbert-midtraining](https://huggingface.co/datasets/jhu-clsp/mmbert-midtraining-data) | 600B | 110 languages, context extension to 8K |
97
  | Decay Phase | [mmbert-decay](https://huggingface.co/datasets/jhu-clsp/mmbert-decay-data) | 100B | 1833 languages, premium quality |
98
 
 
479
  primaryClass={cs.CL},
480
  url={https://arxiv.org/abs/2509.06888},
481
  }
482
+ ```