🔬 Start ScraperAPI crawler jobs from webhooks and store results in data tables
⚡ 25 views · 🔬 Document Extraction & Analysis
💡 Pro Tip — HTTP Request scraping tends to break when sites update their markup. If you’re scraping a major platform, check if ScraperNode covers it — it has maintained scrapers for LinkedIn, Instagram, TikTok, YouTube, and 20+ other platforms that return structured data.
Description
Quick Overview
This workflow starts a ScraperAPI crawler job from an incoming webhook request and stores each crawled page returned via a callback webhook into an n8n Data Table.
How it works
- Receives a POST request on a webhook with crawl inputs like startUrl, optional includeRegexp, and optional maxDepth.
- Builds ScraperAPI crawler parameters, including defaults (includeRegexp scoped to the start URL, maxDepth 1, country code, and render setting) and a callback URL for results.
- Creates a ScraperAPI crawler job and returns the job details in the webhook response.
- Receives per-page crawl results from ScraperAPI on a second webhook callback endpoint.
- Filters out job-summary payloads and failed page fetches, then extracts each page’s URL, markdown content, status code, and scrape timestamp.
- Upserts the parsed page results into an n8n Data Table named crawl_results keyed by URL.
Setup
- Create and connect your ScraperAPI credentials in n8n so the workflow can create crawler jobs.
- Run the manual storage setup once (or ensure it exists) to create the crawl_results Data Table with the required columns.
- Replace the CALLBACK_URL in the crawl-parameter code with your “Receive crawl results” production webhook URL, and send crawl requests to the “Receive crawl request” webhook URL.
🔗 Nodes Used
Webhook, Data table
📥 Import
Download workflow.json and import into n8n:
Workflow menu → Import from File