🔬 Extract structured data from webpages with Apify and OpenAI

⚡ 104 views · 🔬 Document Extraction & Analysis

💡 Pro Tip — HTTP Request scraping tends to break when sites update their markup. If you’re scraping a major platform, check if ScraperNode covers it — it has maintained scrapers for LinkedIn, Instagram, TikTok, YouTube, and 20+ other platforms that return structured data.

View All Scrapers

Description

Quick overview

This workflow exposes a webhook that finds or accepts a target webpage, scrapes it to clean Markdown using Apify, and uses OpenAI with a dynamic schema to return structured JSON fields requested by the caller.

How it works

  1. Receives a POST webhook request containing either a direct target_url or a subject + website_domain, plus a required target_data array of fields to extract.
  2. Validates that target_data is provided and returns a 400 response if it is missing or empty.
  3. Resolves the page URL by using the provided target_url or running an Apify Google Search Scraper site: query and taking the first organic result.
  4. If no matching URL is found, returns a 404 response to the caller.
  5. Scrapes the resolved URL with Apify Website Content Crawler (optionally using provided cookies) and normalizes the returned Markdown and content length.
  6. If the scraped content is too short, returns a 200 response with success:false indicating no content could be extracted.
  7. Sends the page content and requested field definitions to OpenAI and returns a 200 response with the extracted fields as structured JSON (using “NA” for missing values).

Setup

  1. Create an Apify account and add an n8n Header Auth credential that sends Authorization: Bearer <APIFY_TOKEN>, then select it on both Apify HTTP Request steps.
  2. Add an OpenAI API credential and select it for the OpenAI chat model used for extraction.
  3. Configure the webhook path in your calling app and send a JSON body with target_data plus either target_url or subject and website_domain (and optional cookies).
  4. Adjust defaults such as crawler type, crawl limits, timeouts, and minimum content length in the workflow’s configuration values if needed.

Additional info

Extract structured data from any webpage using Apify and AI

📥 Open full documentation on Notion

Need help customizing?

Contact me for consulting and support : Linkedin

MY NEW YOUTUBE CHANNEL

👉 Subscribe to my new YouTube channel. Here I’ll share videos and Shorts with practical tutorials and FREE templates for n8n.

The AI Doctor

🔗 Nodes Used

HTTP Request, Webhook, AI Agent, OpenAI Chat Model, Structured Output Parser

📥 Import

Download workflow.json and import into n8n: Workflow menu → Import from File

📖 Importing guide · 🔑 Credential setup