Remove supported encoders table and license info
Browse files
README.md
CHANGED
|
@@ -37,18 +37,8 @@ let tokens = tokenizer.encode("Hello, world!", true);
|
|
| 37 |
|
| 38 |
`from_pretrained()` tries `.tkz` first, then falls back to `tokenizer.json` — so these repos are fully compatible with the standard HuggingFace loading flow.
|
| 39 |
|
| 40 |
-
## Supported encoders
|
| 41 |
-
|
| 42 |
-
| Encoder | Models |
|
| 43 |
-
|---------|--------|
|
| 44 |
-
| **BPE (Backtracking)** | GPT-2, cl100k, o200k, Llama 3/4, Mistral, Phi, Qwen |
|
| 45 |
-
| **BPE (SentencePiece)** | T5, XLM-RoBERTa, CodeLlama |
|
| 46 |
-
| **WordPiece** | BERT, MiniLM, BGE, GTE, E5, ModernBERT |
|
| 47 |
-
| **Unigram** | Coming soon |
|
| 48 |
-
|
| 49 |
## Links
|
| 50 |
|
| 51 |
- **GitHub**: [chonkie-inc/tokie](https://github.com/chonkie-inc/tokie)
|
| 52 |
- **crates.io**: [tokie](https://crates.io/crates/tokie)
|
| 53 |
-
- **License**: MIT OR Apache-2.0
|
| 54 |
- **Built by**: [chonkie-inc](https://github.com/chonkie-inc)
|
|
|
|
| 37 |
|
| 38 |
`from_pretrained()` tries `.tkz` first, then falls back to `tokenizer.json` — so these repos are fully compatible with the standard HuggingFace loading flow.
|
| 39 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
## Links
|
| 41 |
|
| 42 |
- **GitHub**: [chonkie-inc/tokie](https://github.com/chonkie-inc/tokie)
|
| 43 |
- **crates.io**: [tokie](https://crates.io/crates/tokie)
|
|
|
|
| 44 |
- **Built by**: [chonkie-inc](https://github.com/chonkie-inc)
|