ModernBERT pre-trained from scratch on various obfuscated representations of the same 10B tokens.
🔄 In a Training Loop
Daniel Gallagher
DanielGallagherIRE
AI & ML interests
NLP Researcher at InfAI in Leipzig, Germany.
Recent Activity
updated a dataset about 23 hours ago
DanielGallagherIRE/FineWeb-Edu-GA-300-to-1000 published a dataset about 23 hours ago
DanielGallagherIRE/FineWeb-Edu-GA-300-to-1000 updated a model 1 day ago
crawka/Crawka-Mini-2EPOrganizations
A collection of obfuscated version of a 20B-token sample of the FineWeb Edu dataset.
-
DanielGallagherIRE/FineWeb-Edu-10B-Obfuscation
Viewer • Updated • 19.4M • 1.01k -
DanielGallagherIRE/FineWeb-Edu-10B-Nouns-Only
Viewer • Updated • 9.71M • 786 -
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled
Viewer • Updated • 9.68M • 581 -
DanielGallagherIRE/FineWeb-Edu-10B-PMI-Filtered
Viewer • Updated • 9.71M • 505
ModernBERT pre-trained from scratch on various obfuscated representations of the same 10B tokens.
A collection of obfuscated version of a 20B-token sample of the FineWeb Edu dataset.
-
DanielGallagherIRE/FineWeb-Edu-10B-Obfuscation
Viewer • Updated • 19.4M • 1.01k -
DanielGallagherIRE/FineWeb-Edu-10B-Nouns-Only
Viewer • Updated • 9.71M • 786 -
DanielGallagherIRE/FineWeb-Edu-10B-Shuffled
Viewer • Updated • 9.68M • 581 -
DanielGallagherIRE/FineWeb-Edu-10B-PMI-Filtered
Viewer • Updated • 9.71M • 505
Models pre-trained on various obfuscated versions of a sample of 10B tokens to analyse whether such obfuscated models are still performant.