pathwren/ai-crawler-index download history
pathwren/ai-crawler-index is a dataset on the Hugging Face Hub. In the last 30 days it was downloaded 744 times (185 in the last 7 days), and 886 times in total. It ranks #24,223 among datasets by monthly downloads.
AI Crawler Index 150 web crawlers and AI user agents from 74 operators — what each one is for, what blocking it costs you, and the IP ranges its operator publishes. Plus a compiled user-agent regex and the union of 1997 IPv4 and 1062 IPv6 prefixes from 15 operator-published range file