gaunernst/tokenized-datasets download history

gaunernst/tokenized-datasets is a dataset on the Hugging Face Hub. In the last 30 days it was downloaded 14 times (4 in the last 7 days), and 581 times in total. It ranks #577,463 among datasets by monthly downloads.

Tokenized datasets Containing some small datasets pre-tokenized for pre-training. Using Llama2 tokenizer Datasets under llama2 folder are tokenized with Llama2 tokenizer. Since Llama2's vocab size is 32,000, using uint16 (0-65,535) to represent each token is sufficient.

Open gaunernst/tokenized-datasets on Hugging Face

Sister project: Paper Pulse, the upvote history of every Hugging Face Daily Paper.