# KorByte-128K public tokenizer comparison

**First-place gate: PASS**
**Coverage gate: PASS**

First place means rank 1 by both fertility and effective bits per byte among the successfully loaded, revision-pinned public comparison set.

## [Multilingual Tokenizer Benchmark — Korean](https://huggingface.co/datasets/eduagarcia/multilingual_tokenizer_benchmark)

- Dataset revision: `74fa5907af0265f42c09e21379bf21847e066990`
- Role: development evaluation

| Fert. rank | EBPB rank | System | Revision | Vocab | Fertility ↓ | Bytes/token ↑ | EBPB ↓ | Exact docs |
|---:|---:|---|---|---:|---:|---:|---:|---:|
| 1 | 1 | [korbyte-128k](https://huggingface.co/dawncr0w/KorByte-128K) | release artifact | 128,256 | 1.4463 | 5.4064 | 3.0710 | 100.0000% |
| 4 | 2 | [kanana-2](https://huggingface.co/kakaocorp/kanana-2-3b-base) | [9fa2a678c379](https://huggingface.co/kakaocorp/kanana-2-3b-base/tree/9fa2a678c379700868d72685a22ea8ed0263ef33) | 128,256 | 1.5491 | 5.0477 | 3.1226 | 100.0000% |
| 2 | 3 | [k-exaone](https://huggingface.co/LGAI-EXAONE/K-EXAONE-236B-A23B) | [61e6d578eb10](https://huggingface.co/LGAI-EXAONE/K-EXAONE-236B-A23B/tree/61e6d578eb102b578e5704e2916ac841df9eca0a) | 153,600 | 1.5283 | 5.1164 | 3.1269 | 99.8737% |
| 3 | 4 | [solar-pro3](https://huggingface.co/upstage/solar-pro3-tokenizer) | [c41b71f0519c](https://huggingface.co/upstage/solar-pro3-tokenizer/tree/c41b71f0519c580610cb0fd2af4ed2b23cad544f) | 196,608 | 1.5306 | 5.1088 | 3.1538 | 100.0000% |
| 5 | 5 | [ax-4.0](https://huggingface.co/skt/A.X-4.0) | [a3393fd67bf0](https://huggingface.co/skt/A.X-4.0/tree/a3393fd67bf09a34a85e66dfd2c2da61c4497b67) | 102,402 | 1.5582 | 5.0183 | 3.2153 | 99.8737% |
| 7 | 6 | [trillion-7b](https://huggingface.co/trillionlabs/Trillion-7B-preview) | [bc1650c5a954](https://huggingface.co/trillionlabs/Trillion-7B-preview/tree/bc1650c5a954c7de0509d28c8dbc5bea560856e1) | 128,128 | 1.6128 | 4.8484 | 3.2318 | 100.0000% |
| 6 | 7 | [midm-2.0](https://huggingface.co/K-intelligence/Midm-2.0-Base-Instruct) | [35479c5fc9a1](https://huggingface.co/K-intelligence/Midm-2.0-Base-Instruct/tree/35479c5fc9a18a5db7cc6dbadcf1db68db7beab0) | 131,384 | 1.6035 | 4.8764 | 3.2963 | 100.0000% |
| 8 | 8 | [hyperclovax](https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Think-32B) | [a6cdfd3464d1](https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Think-32B/tree/a6cdfd3464d1b767259cad23e164eaf39d3e3960) | 128,256 | 1.8149 | 4.3085 | 3.6430 | 100.0000% |

Unavailable pinned artifacts:

- [`thunder`](https://huggingface.co/thunder-research-group/Llama-Thunder-LLM-8B/tree/ecfd6c909944939ae42982f3d87cf36898da308b) at `ecfd6c909944939ae42982f3d87cf36898da308b`: `Exception: data did not match any variant of untagged enum ModelUntagged at line 770599 column 1`

## [KMMLU test — frozen post-selection audit](https://huggingface.co/datasets/HAERAE-HUB/KMMLU)

- Dataset revision: `d61b3f19e552c576bf5960dd24289763edc36a88`
- Role: frozen post-selection audit; no algorithm changes followed

| Fert. rank | EBPB rank | System | Revision | Vocab | Fertility ↓ | Bytes/token ↑ | EBPB ↓ | Exact docs |
|---:|---:|---|---|---:|---:|---:|---:|---:|
| 1 | 1 | [korbyte-128k](https://huggingface.co/dawncr0w/KorByte-128K) | release artifact | 128,256 | 1.9364 | 4.8039 | 3.2942 | 100.0000% |
| 2 | 2 | [solar-pro3](https://huggingface.co/upstage/solar-pro3-tokenizer) | [c41b71f0519c](https://huggingface.co/upstage/solar-pro3-tokenizer/tree/c41b71f0519c580610cb0fd2af4ed2b23cad544f) | 196,608 | 2.0985 | 4.4326 | 3.5143 | 100.0000% |
| 4 | 3 | [kanana-2](https://huggingface.co/kakaocorp/kanana-2-3b-base) | [9fa2a678c379](https://huggingface.co/kakaocorp/kanana-2-3b-base/tree/9fa2a678c379700868d72685a22ea8ed0263ef33) | 128,256 | 2.1625 | 4.3015 | 3.5157 | 100.0000% |
| 3 | 4 | [k-exaone](https://huggingface.co/LGAI-EXAONE/K-EXAONE-236B-A23B) | [61e6d578eb10](https://huggingface.co/LGAI-EXAONE/K-EXAONE-236B-A23B/tree/61e6d578eb102b578e5704e2916ac841df9eca0a) | 153,600 | 2.1458 | 4.3351 | 3.5167 | 99.5575% |
| 7 | 5 | [trillion-7b](https://huggingface.co/trillionlabs/Trillion-7B-preview) | [bc1650c5a954](https://huggingface.co/trillionlabs/Trillion-7B-preview/tree/bc1650c5a954c7de0509d28c8dbc5bea560856e1) | 128,128 | 2.2263 | 4.1783 | 3.6054 | 100.0000% |
| 5 | 6 | [midm-2.0](https://huggingface.co/K-intelligence/Midm-2.0-Base-Instruct) | [35479c5fc9a1](https://huggingface.co/K-intelligence/Midm-2.0-Base-Instruct/tree/35479c5fc9a18a5db7cc6dbadcf1db68db7beab0) | 131,384 | 2.2209 | 4.1885 | 3.6669 | 100.0000% |
| 6 | 7 | [ax-4.0](https://huggingface.co/skt/A.X-4.0) | [a3393fd67bf0](https://huggingface.co/skt/A.X-4.0/tree/a3393fd67bf09a34a85e66dfd2c2da61c4497b67) | 102,402 | 2.2210 | 4.1883 | 3.6793 | 99.5575% |
| 8 | 8 | [hyperclovax](https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Think-32B) | [a6cdfd3464d1](https://huggingface.co/naver-hyperclovax/HyperCLOVAX-SEED-Think-32B/tree/a6cdfd3464d1b767259cad23e164eaf39d3e3960) | 128,256 | 2.5157 | 3.6976 | 4.0426 | 100.0000% |

Unavailable pinned artifacts:

- [`thunder`](https://huggingface.co/thunder-research-group/Llama-Thunder-LLM-8B/tree/ecfd6c909944939ae42982f3d87cf36898da308b) at `ecfd6c909944939ae42982f3d87cf36898da308b`: `Exception: data did not match any variant of untagged enum ModelUntagged at line 770599 column 1`

## Interpretation limits

- Intrinsic tokenization metrics do not establish downstream model quality.
- The comparison set is broad and current but cannot prove a universal ranking.
- Thunder's public tokenizer uses an unsupported custom `Beta` model and is recorded as unavailable.