OpenTransformer/turbo-crawl download history

OpenTransformer/turbo-crawl is a dataset on the Hugging Face Hub. In the last 30 days it was downloaded 5 times (2 in the last 7 days), and 350 times in total. It ranks #779,199 among datasets by monthly downloads.

Turbo Crawl v1 Fast web scraper in Rust. ~10 MB/s sustained. Build cargo build --release Run ./target/release/turbo_crawl Output: /workspace/crawl.jsonl (JSONL format with url, text, tokens_est) Utils watchdog.sh - Auto-kills scraper at 145GB disk usage

Open OpenTransformer/turbo-crawl on Hugging Face

Sister project: Paper Pulse, the upvote history of every Hugging Face Daily Paper.