📊 Scrape NSR specialist data into Google Sheets with pagination and detail pages
⚡ 2 views · 📊 Market Research & Insights
💡 Pro Tip — HTTP Request scraping tends to break when sites update their markup. If you’re scraping a major platform, check if ScraperNode covers it — it has maintained scrapers for LinkedIn, Instagram, TikTok, YouTube, and 20+ other platforms that return structured data.
Description
Quick overview
This workflow manually scrapes Malaysia’s National Specialist Register (nsr.org.my) for a selected specialty (configured for Cardiology), paginates through all result pages, enriches each doctor by fetching their detail page, and appends or updates the structured results in Google Sheets.
How it works
- Starts when you run the workflow manually.
- Requests nsr.org.my to obtain a session cookie and submits a specialty search (configured with code 302 for Cardiology).
- Iterates through each results page on nsr.org.my, parses the HTML table, and accumulates each doctor’s NSR number, name, gender, and account ID.
- After the last page is reached, loops through the full doctor list and opens each doctor’s detail page on nsr.org.my.
- Extracts the graduation year (from the Basic Degree section) and splits the listed specialty field into Specialty 1 and Specialty 2.
- Waits briefly to rate-limit requests, then appends or updates the doctor record in Google Sheets using the License (NSR number) as the matching key.
Setup
- Create a Google Sheets OAuth2 connection in n8n.
- Set the target Google Sheets document and sheet tab, and ensure it has columns for Name, License, Specialty 1, Specialty 2, Gender, and Grad Year.
- (Optional) To scrape a different specialty, update the allSpecialty_ForSearch value in the NSR search request and adjust the specialty label used during parsing if you rely on it downstream.
Requirements
- n8n instance (self-hosted or cloud)
- Google Sheets account with a spreadsheet ready (headers: Name, License, Specialty 1, Specialty 2, Gender, Grad Year)
- Google Sheets credential connected in n8n
- Internet access to nsr.org.my (no auth required, public register)
Customization
- Target specialty (currently Cardiology, code 302) — swap via numeric specialty code from NSR’s search form
- Output spreadsheet/sheet destination
- Rate-limit delay between requests (default 0.7s)
- Dedupe key for re-runs (default: License / NSR No)
Additional info
Scrapes Malaysia’s National Specialist Register (nsr.org.my) for a chosen medical specialty. Runs in 2 phases: paginates all result pages to collect the doctor list, then fetches each doctor’s detail page for Gender/Specialty/Grad Year. Handles session-cookie-based filtering (NSR has no URL-based filter). ~10-15 min runtime for ~460 records.
🔗 Nodes Used
Google Sheets, HTTP Request
📥 Import
Download workflow.json and import into n8n:
Workflow menu → Import from File