⚒️ Web site scraper for LLMs with Airtop
⚡ 9,611 views · ⚒️ Engineering
💡 Pro Tip — HTTP Request scraping tends to break when sites update their markup. If you’re scraping a major platform, check if ScraperNode covers it — it has maintained scrapers for LinkedIn, Instagram, TikTok, YouTube, and 20+ other platforms that return structured data.
Description
Recursive Web Scraping
Use Case
Automating web scraping with recursive depth is ideal for collecting content across multiple linked pages—perfect for content aggregation, lead generation, or research projects.
What This Automation Does
This automation reads a list of URLs from a Google Sheet, scrapes each page, stores the content in a document, and adds newly discovered links back to the sheet. It continues this process for a specified number of iterations based on the defined scraping depth.
Input Parameters:
Seed URL: The starting URL to begin the scraping process.
Example:https://example.com/Links must contain: Restricts the links to those that contain this specified string.
Example:https://example.com/Depth: The number of iterations (layers of links) to scrape beyond the initial set.
Example:3
How It Works
- Starts by reading the
Seed URLfrom the Google Sheet. - Scrapes each page and saves its content to the specified document.
- Extracts new links from each page that match the
Links must containstring, appends them to the Google Sheet. - Repeats steps 2–3 for the number of times specified by
Depth - 1.
Setup Requirements
- Airtop API Key — free to generate.
- Credentials set up for Google Docs (requires creating a project on Google Console). Read how to.
- Credentials set up for Google Spreadsheet.
Next Steps
- Add Filtering Rules: Filter which links to follow based on domain, path, or content type.
- Combine with Scheduler: Run this automation on a schedule to continuously explore newly discovered pages.
- Export Structured Data: Extend the process to store extracted data in a CSV or database for analysis.
Read more about website scraping for LLMS
🔗 Nodes Used
Google Sheets, Google Docs, Execute Workflow Trigger, n8n Form Trigger, Airtop
📥 Import
Download workflow.json and import into n8n:
Workflow menu → Import from File