occiglot/tokenizer-wiki-bench download history
occiglot/tokenizer-wiki-bench is a dataset on the Hugging Face Hub. In the last 30 days it was downloaded 49,490 times (5,411 in the last 7 days), and 489,902 times in total. It ranks #507 among datasets by monthly downloads.
Multilingual Tokenizer Benchmark This dataset includes pre-processed wikipedia data for tokenizer evaluation in 45 languages. We provide more information on the evaluation task in general this blogpost. Usage The dataset allows us to easily calculate tokenizer fertility and the
Spaces using tokenizer-wiki-bench
- Model Pulse 8 likes