NbAiLab/aurora-robots-txt download history

NbAiLab/aurora-robots-txt is a dataset on the Hugging Face Hub. In the last 30 days it was downloaded 11 times (4 in the last 7 days), and 149 times in total. It ranks #540,266 among datasets by monthly downloads.

Fetched robots.txt for creation of the Aurora dataset. Columns are url: The URL of the robots.txt file fetched content: The content of the response. Pages that responded with 404 Not Found or 410 Gone were assumed to allow all. Pages that responded with 403 Forbidden were assumed to disallow all.

Open NbAiLab/aurora-robots-txt on Hugging Face

Sister project: Paper Pulse, the upvote history of every Hugging Face Daily Paper.