📊 Scrape, search and browse the web with a Firecrawl AI agent webhook

⚡ 4,096 views · 📊 Market Research & Insights

💡 Pro Tip — HTTP Request scraping tends to break when sites update their markup. If you’re scraping a major platform, check if ScraperNode covers it — it has maintained scrapers for LinkedIn, Instagram, TikTok, YouTube, and 20+ other platforms that return structured data.

View All Scrapers

Description

Turn any prompt into structured web data. Send a POST request with a natural language prompt and an optional JSON schema, and get back clean, structured results scraped from the web by an AI agent powered by Firecrawl.

Use Cases

How It Works

POST /webhook/scrape-agent

  1. Receive Scrape Request receives a POST request with prompt and an optional output_schema.

  2. Validate Output Schema checks the schema. If none is provided, it falls back to a permissive default. If the schema is malformed, it returns a clear error via Return Schema Error.

  3. Research & Extract Web Data takes the prompt and uses the full Firecrawl toolkit to research the web:

    • Search (/search): Finds relevant pages and sources across the web.
    • Scrape (/scrape): Extracts clean, structured content from any URL.
    • Interact (interactContext, interact, interactStop): Lets the agent interact with scraped pages in a live session. After scraping a page, the agent can click buttons, fill forms, navigate dynamic content, and extract data that static scraping cannot reach, all without managing sessions manually.

    This combination gives the AI agent complete web navigation capabilities. It can discover sources, read pages, and interact with dynamic content autonomously.

  4. Format Response to Schema (Structured Output Parser) formats the agent’s response to match the provided (or default) schema.

  5. Return Structured Results sends the structured JSON back to the caller.

Setup Requirements

API Reference

Endpoint

POST https://your-n8n-instance/webhook/scrape-agent

Request Body

FieldTypeRequiredDescription
promptstringYesNatural language instruction for the agent
output_schemaobjectNoJSON Schema defining the desired output structure

Response

Returns a JSON object matching the provided schema, or a flexible object if no schema was given.

Testing Examples

1. Basic Request (No Schema)

The agent decides the output structure on its own.

curl -X POST "https://your-n8n-instance/webhook/scrape-agent" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Find the latest pricing for Firecrawl"
  }' | jq

Expected output: A JSON object with whatever structure the agent finds most appropriate for the data. Since no schema was provided, the internal default ({ "type": "object", "additionalProperties": true }) is used.

2. Request With a Custom Schema

You define exactly the shape of data you want back.

curl -X POST "https://your-n8n-instance/webhook/scrape-agent" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Find the latest pricing for Firecrawl",
    "output_schema": {
      "type": "object",
      "properties": {
        "source": { "type": "string" },
        "plans": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name": { "type": "string" },
              "price": { "type": "string" },
              "credits": { "type": "string" },
              "highlights": {
                "type": "array",
                "items": { "type": "string" }
              }
            }
          }
        }
      }
    }
  }' | jq

Expected output:

{
  "output": {
    "source": "https://www.firecrawl.dev/pricing",
    "plans": [
      {
        "name": "Free",
        "price": "$0 (one-time)",
        "credits": "500 credits (one-time)",
        "highlights": [
          "Scrape up to 500 pages",
          "2 concurrent requests",
          "Low rate limits",
          "No credit card required"
        ]
      },
      {
        "name": "Hobby",
        "price": "$16/month (billed yearly, save $38)",
        "credits": "3,000 credits / month",
        "highlights": [
          "Scrape up to 3,000 pages",
          "5 concurrent requests",
          "Basic support",
          "$9 per extra 1k credits"
        ]
      }
    ]
  }
}

3. Invalid Schema (String Instead of Object)

curl -X POST "https://your-n8n-instance/webhook/scrape-agent" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Find the latest pricing for Firecrawl",
    "output_schema": "not a valid schema"
  }' | jq

Expected output:

{
  "error": true,
  "message": "Invalid output_schema: must be a JSON object with a valid 'type' property (object, array, string, number, boolean)",
  "example_schema": {
    "type": "object",
    "properties": {
      "name": { "type": "string" },
      "price": { "type": "number" }
    }
  }
}

4. Invalid Schema (Array Instead of Object)

curl -X POST "https://your-n8n-instance/webhook/scrape-agent" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Find the latest pricing for Firecrawl",
    "output_schema": [1, 2, 3]
  }' | jq

Expected output: Same error response as above.

5. Invalid Schema (Missing type Property)

curl -X POST "https://your-n8n-instance/webhook/scrape-agent" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Find the latest pricing for Firecrawl",
    "output_schema": {
      "properties": {
        "name": { "type": "string" }
      }
    }
  }' | jq

Expected output: Same error response as above.

6. Invalid Schema (Invalid type Value)

curl -X POST "https://your-n8n-instance/webhook/scrape-agent" \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "Find the latest pricing for Firecrawl",
    "output_schema": {
      "type": "banana"
    }
  }' | jq

Expected output: Same error response as above.

Workflow Architecture

Receive Scrape Request (POST)
  |
  v
Validate Output Schema
  |--- Error --> Return Schema Error (error JSON)
  |--- Success --> Research & Extract Web Data (AI Agent)
                     |
                     |--- Primary Chat Model
                     |--- Fallback Chat Model
                     |--- Search & Scrape:
                     |      - /search with Firecrawl
                     |      - /scrape with Firecrawl
                     |--- Interact Tool:
                     |      - Interact context with Firecrawl
                     |      - Execute interaction with Firecrawl
                     |      - Stop interaction with Firecrawl
                     |
                     v
                   Return Structured Results
                     |
                     |--- Format Response to Schema (Output Parser)
                            |
                            |--- Parser Chat Model

Schema Validation Logic

The Validate Output Schema node runs this validation before passing data to the agent:

Notes

🔗 Nodes Used

Webhook, AI Agent, Structured Output Parser, OpenRouter Chat Model

📥 Import

Download workflow.json and import into n8n: Workflow menu → Import from File

📖 Importing guide · 🔑 Credential setup