desearch/ai-crawlers-robots-txt-study download history
desearch/ai-crawlers-robots-txt-study is a dataset on the Hugging Face Hub. In the last 30 days it was downloaded 64 times (24 in the last 7 days), and 112 times in total. It ranks #214,797 among datasets by monthly downloads.
AI Crawler robots.txt Study (Tranco top 10,000, October 2026) Published by Desearch. Full write-up and method: https://www.desearch.ai/blog/ai-crawlers-robots-txt-study What is in it One row per domain for all 10,000 domains of the Tranco standard list (list ID 64VKX, d
Open desearch/ai-crawlers-robots-txt-study on Hugging Face
Sister project: Paper Pulse, the upvote history of every Hugging Face Daily Paper.