huggingface/tokbench-corpora download history

huggingface/tokbench-corpora is a text generation dataset on the Hugging Face Hub. In the last 30 days it was downloaded 342 times (165 in the last 7 days), and 342 times in total. It ranks #44,323 among datasets by monthly downloads.

tokbench corpora The input corpora for tokbench, a benchmark that measures tokenizer implementations against each other on the same bytes. Each config is one corpus: ~5 MB of real text chosen to stress a different part of a tokenizer. Nothing here is new text. It is a fixed, pinned, r

Open huggingface/tokbench-corpora on Hugging Face

Sister project: Paper Pulse, the upvote history of every Hugging Face Daily Paper.