datablations/oscar-dedup-expanded download history
datablations/oscar-dedup-expanded is a dataset on the Hugging Face Hub. In the last 30 days it was downloaded 2,144 times (1,467 in the last 7 days), and 307,311 times in total. It ranks #10,706 among datasets by monthly downloads.
Use the 25% suffix array to deduplicate the full Oscar, i.e. remove any document which has an at least 100-char span overlapping with the 25% chunk we selected in the previous bullet. This is more permissive and leaves us with 136 million documents or 31% of the original dataset. Also for reasons th