seonglae/wikipedia-256-token download history

seonglae/wikipedia-256-token is a dataset on the Hugging Face Hub. In the last 30 days it was downloaded 321 times (38 in the last 7 days), and 2,624 times in total. It ranks #46,573 among datasets by monthly downloads.

This is Wikidedia passages dataset for ODQA retriever. Each passages have 256~ tokens splitteed by gpt-4 tokenizer using tiktoken. Token count {'~128': 1415068, '128~256': 1290011, '256~512': 18756476, '512~1024': 667, '1024~2048': 12, '2048~4096': 0, '4096~8192': 0, '8192~16384': 0, '16384~32768':

Open seonglae/wikipedia-256-token on Hugging Face