minhnguyent546/fineweb-edu-10BT-for-modern-gpt download history

minhnguyent546/fineweb-edu-10BT-for-modern-gpt is a text generation dataset on the Hugging Face Hub. In the last 30 days it was downloaded 202 times (22 in the last 7 days), and 1,039 times in total. It ranks #67,481 among datasets by monthly downloads.

Dataset Summary This is a tokenized Fineweb-Edu (10BT subset) using GPT2 tokenizer (via tiktoken) for pre-training GPT2 model. Data is divided into shards (.npy files) for easier to load with PyTorch IterableDataset. Each .npy file can be loaded with numpy.load('file_name.npy'). Spl

Open minhnguyent546/fineweb-edu-10BT-for-modern-gpt on Hugging Face

Sister project: Paper Pulse, the upvote history of every Hugging Face Daily Paper.