๐ Automate news article scraping with ScrapegraphAI and store in Google Sheets
โก 2,051 views ยท ๐ Market Research & Insights
๐ก Pro Tip โ HTTP Request scraping tends to break when sites update their markup. If youโre scraping a major platform, check if ScraperNode covers it โ it has maintained scrapers for LinkedIn, Instagram, TikTok, YouTube, and 20+ other platforms that return structured data.
Description
This workflow contains community nodes that are only compatible with the self-hosted version of n8n.
News Article Scraping and Analysis with AI and Google Sheets Integration
๐ฏ Target Audience
- News aggregators and content curators
- Media monitoring professionals
- Market researchers tracking industry news
- PR professionals monitoring brand mentions
- Journalists and content creators
- Business analysts tracking competitor news
- Academic researchers collecting news data
๐ Problem Statement
Manual news monitoring is time-consuming and often misses important articles. This template solves the challenge of automatically collecting, structuring, and storing news articles from any website for comprehensive analysis and tracking.
๐ง How it Works
This workflow automatically scrapes news articles from websites using AI-powered extraction and stores them in Google Sheets for analysis and tracking.
Key Components
- Scheduled Trigger: Runs automatically at specified intervals to collect fresh content
- AI-Powered Scraping: Uses ScrapeGraphAI to intelligently extract article titles, URLs, and categories from any news website
- Data Processing: Formats extracted data for optimal spreadsheet compatibility
- Automated Storage: Saves all articles to Google Sheets with metadata for easy filtering and analysis
๐ Google Sheets Column Specifications
The template creates the following columns in your Google Sheets:
| Column | Data Type | Description | Example |
|---|---|---|---|
| title | String | Article headline and title | โโMy friend died right in front of meโ - Student describes moment air force jet crashed into schoolโ |
| url | URL | Direct link to the article | โhttps://www.bbc.com/news/articles/cglzw8y5wy5oโ |
| category | String | Article category or section | โAsiaโ |
๐ ๏ธ Setup Instructions
Estimated setup time: 10-15 minutes
Prerequisites
- n8n instance with community nodes enabled
- ScrapeGraphAI API account and credentials
- Google Sheets account with API access
Step-by-Step Configuration
1. Install Community Nodes
# Install ScrapeGraphAI community node
npm install n8n-nodes-scrapegraphai2. Configure ScrapeGraphAI Credentials
- Navigate to Credentials in your n8n instance
- Add new ScrapeGraphAI API credentials
- Enter your API key from ScrapeGraphAI dashboard
- Test the connection to ensure itโs working
3. Set up Google Sheets Connection
- Add Google Sheets OAuth2 credentials
- Grant necessary permissions for spreadsheet access
- Select or create a target spreadsheet for data storage
- Configure the sheet name (default: โSheet1โ)
4. Customize News Source Parameters
- Update the
websiteUrlparameter in the ScrapeGraphAI node - Modify the target news website URL as needed
- Adjust the user prompt to extract additional fields if required
- Test with a small website first before scaling to larger news sites
5. Configure Schedule Trigger
- Set your preferred execution frequency (daily, hourly, etc.)
- Choose appropriate time zones for your business hours
- Consider the news websiteโs update frequency when setting intervals
6. Test and Validate
- Run the workflow manually to verify all connections
- Check Google Sheets for proper data formatting
- Validate that all required fields are being captured
๐ Workflow Customization Options
Modify News Sources
- Change the website URL to target different news sources
- Add multiple news websites for comprehensive coverage
- Implement filters for specific topics or categories
Extend Data Collection
- Modify the user prompt to extract additional fields (author, date, summary)
- Add sentiment analysis for article content
- Integrate with other data sources for comprehensive analysis
Output Customization
- Change Google Sheets operation from โappendโ to โupsertโ for deduplication
- Add data validation and cleaning steps
- Implement error handling and retry logic
๐ Use Cases
- Media Monitoring: Track mentions of your brand, competitors, or industry keywords
- Content Curation: Automatically collect articles for newsletters or content aggregation
- Market Research: Monitor industry trends and competitor activities
- News Aggregation: Build custom news feeds for specific topics or sources
- Academic Research: Collect news data for research projects and analysis
- Crisis Management: Monitor breaking news and emerging stories
๏ฟฝ๏ฟฝ Important Notes
- Respect the target websiteโs terms of service and robots.txt
- Consider implementing delays between requests for large datasets
- Regularly review and update your scraping parameters
- Monitor API usage to manage costs effectively
- Keep your credentials secure and rotate them regularly
๐ง Troubleshooting
Common Issues:
- ScrapeGraphAI connection errors: Verify API key and account status
- Google Sheets permission errors: Check OAuth2 scope and permissions
- Data formatting issues: Review the Code nodeโs JavaScript logic
- Rate limiting: Adjust schedule frequency and implement delays
Pro Tips:
- Keep detailed configuration notes in the sticky notes within the workflow
- Test with a small website first before scaling to larger news sites
- Consider adding filters in the Code node to exclude certain article types or categories
- Monitor execution logs for any issues and adjust parameters accordingly
Support Resources:
- ScrapeGraphAI documentation and API reference
- n8n community forums for workflow assistance
- Google Sheets API documentation for advanced configurations
๐ Nodes Used
Google Sheets, Schedule Trigger
๐ฅ Import
Download workflow.json and import into n8n:
Workflow menu โ Import from File