brendanlong/retok-noncanonical-tokenization download history

brendanlong/retok-noncanonical-tokenization is a text generation dataset on the Hugging Face Hub. In the last 30 days it was downloaded 53 times (26 in the last 7 days), and 1,055 times in total. It ranks #177,952 among datasets by monthly downloads.

Non-canonical tokenization in LLM generations Per-generation records from seven language models, capturing the token IDs each model actually emitted alongside the canonical re-encoding of its own output — plus the trained toy-model checkpoints from the accompanying controlled experime

Open brendanlong/retok-noncanonical-tokenization on Hugging Face

Sister project: Paper Pulse, the upvote history of every Hugging Face Daily Paper.