🔒 Audit AI and search bot access using robots.txt checks and Gmail reports
⚡ 0 views · 🔒 SecOps & Security Automation
Description
Quick overview
This workflow runs monthly to audit whether popular AI and search crawlers can access a specific page by comparing robots.txt rules with real HTTP responses, then emails a verdict report via Gmail.
How it works
- Runs on a monthly schedule trigger.
- Sets the target URL, the bot user-agent list, a browser user-agent control, and the delay between requests.
- Fetches the site’s robots.txt and evaluates, for each bot, whether the target page is allowed based on longest-match and wildcard fallback rules.
- Requests the target page once per bot using that bot’s User-Agent, pacing requests to reduce rate limiting.
- Makes one additional request as a normal browser and compares each bot’s HTTP status and response size against the control response.
- Classifies each bot as ALLOWED, BLOCKED, PARTIAL, CONTRADICTION, RATE_LIMITED, or UNKNOWN and aggregates the results into a summary.
- Sends the full audit report to the configured recipient using Gmail.
Setup
- Add a Gmail OAuth2 credential and set the recipient address in the email step.
- Update the testUrl value to the page you want to audit.
- Adjust the schedule timing and the secondsBetweenRequests delay to match your site’s rate limiting behavior.
- Optionally edit the bot list and user-agent strings to add or remove specific crawlers you want to test.
🔗 Nodes Used
HTTP Request, Gmail, Schedule Trigger
📥 Import
Download workflow.json and import into n8n:
Workflow menu → Import from File