ZhuofengLi/fineweb-edu-pretokenized-10b download history

ZhuofengLi/fineweb-edu-pretokenized-10b is a dataset on the Hugging Face Hub. In the last 30 days it was downloaded 161 times (58 in the last 7 days), and 377 times in total. It ranks #79,814 among datasets by monthly downloads.

FineWeb-Edu Pretokenized 10B Megatron indexed-dataset (.bin/.idx) versions of HuggingFaceFW/fineweb-edu sample/10BT. Each Hugging Face subset contains a small metadata.parquet index. The actual Megatron files are under <subset>/files/; pass each prefix without the .bin/.idx suffix to

Open ZhuofengLi/fineweb-edu-pretokenized-10b on Hugging Face

Sister project: Paper Pulse, the upvote history of every Hugging Face Daily Paper.