pcuenq/tokenizer-conformance download history

pcuenq/tokenizer-conformance is an other dataset on the Hugging Face Hub. In the last 30 days it was downloaded 54 times (20 in the last 7 days), and 179 times in total. It ranks #247,950 among datasets by monthly downloads.

Tokenizer conformance fixtures Reference inputs and Python fast-tokenizer outputs for tokenizer implementations. The initial corpus contains 83 inputs in 30 categories, with 498 reference encodings across six tokenizers. This is a regression dataset, not a model-quality benchmark.

Open pcuenq/tokenizer-conformance on Hugging Face

Sister project: Paper Pulse, the upvote history of every Hugging Face Daily Paper.