reports/comparison.md

KorByte-128K public tokenizer comparison

First-place gate: PASS Coverage gate: PASS

First place means rank 1 by both fertility and effective bits per byte among the successfully loaded, revision-pinned public comparison set.

Multilingual Tokenizer Benchmark — Korean

• Dataset revision: 74fa5907af0265f42c09e21379bf21847e066990
• Role: development evaluation

Fert. rank	EBPB rank	System	Revision	Vocab	Fertility ↓	Bytes/token ↑	EBPB ↓	Exact docs
1	1	korbyte-128k	release artifact	128,256	1.4463	5.4064	3.0710	100.0000%
4	2	kanana-2	9fa2a678c379	128,256	1.5491	5.0477	3.1226	100.0000%
2	3	k-exaone	61e6d578eb10	153,600	1.5283	5.1164	3.1269	99.8737%
3	4	solar-pro3	c41b71f0519c	196,608	1.5306	5.1088	3.1538	100.0000%
5	5	ax-4.0	a3393fd67bf0	102,402	1.5582	5.0183	3.2153	99.8737%
7	6	trillion-7b	bc1650c5a954	128,128	1.6128	4.8484	3.2318	100.0000%
6	7	midm-2.0	35479c5fc9a1	131,384	1.6035	4.8764	3.2963	100.0000%
8	8	hyperclovax	a6cdfd3464d1	128,256	1.8149	4.3085	3.6430	100.0000%

Unavailable pinned artifacts:

• thunder at ecfd6c909944939ae42982f3d87cf36898da308b: Exception: data did not match any variant of untagged enum ModelUntagged at line 770599 column 1

KMMLU test — frozen post-selection audit

• Dataset revision: d61b3f19e552c576bf5960dd24289763edc36a88
• Role: frozen post-selection audit; no algorithm changes followed

Fert. rank	EBPB rank	System	Revision	Vocab	Fertility ↓	Bytes/token ↑	EBPB ↓	Exact docs
1	1	korbyte-128k	release artifact	128,256	1.9364	4.8039	3.2942	100.0000%
2	2	solar-pro3	c41b71f0519c	196,608	2.0985	4.4326	3.5143	100.0000%
4	3	kanana-2	9fa2a678c379	128,256	2.1625	4.3015	3.5157	100.0000%
3	4	k-exaone	61e6d578eb10	153,600	2.1458	4.3351	3.5167	99.5575%
7	5	trillion-7b	bc1650c5a954	128,128	2.2263	4.1783	3.6054	100.0000%
5	6	midm-2.0	35479c5fc9a1	131,384	2.2209	4.1885	3.6669	100.0000%
6	7	ax-4.0	a3393fd67bf0	102,402	2.2210	4.1883	3.6793	99.5575%
8	8	hyperclovax	a6cdfd3464d1	128,256	2.5157	3.6976	4.0426	100.0000%

Unavailable pinned artifacts:

• thunder at ecfd6c909944939ae42982f3d87cf36898da308b: Exception: data did not match any variant of untagged enum ModelUntagged at line 770599 column 1

Interpretation limits

• Intrinsic tokenization metrics do not establish downstream model quality.
• The comparison set is broad and current but cannot prove a universal ranking.
• Thunder's public tokenizer uses an unsupported custom Beta model and is recorded as unavailable.