ScrapingDog MCP Server

A Model Context Protocol (MCP) server for ScrapingDog API that provides comprehensive web scraping, search engine data extraction, social media scraping, and e-commerce data collection capabilities.

Features

🌐 General Web Scraping

• HTML Extraction: Scrape any webpage with automatic proxy rotation and CAPTCHA solving
• JavaScript Rendering: Support for dynamic content with headless Chrome browsers
• Premium Proxies: Access to 40+ million rotating residential proxies
• Geo-targeting: Country-specific scraping for localized content

🔍 Search Engine APIs

• Google Search: Organic results, ads, and related search data
• Google AI Search: AI-powered search with overview and insights
• Bing Search: Comprehensive Bing search results
• Google Maps: Business listings and location data
• Google News: Real-time news articles and updates

🛒 E-commerce Data Extraction

• Amazon Products: Product search, details, pricing, and availability
• Amazon Reviews: Customer reviews and ratings extraction
• Walmart Products: Product information and pricing data

📱 Social Media Scraping

• LinkedIn: Profile and company information extraction
• LinkedIn Jobs: Job listings by location and keywords
• Twitter/X: Posts, likes, and bookmarks extraction
• Instagram: Profile and posts data
• Facebook: Page and profile information

💼 Job Board Integration

• Indeed: Job listings with advanced filtering options

Installation

Prerequisites

• Node.js 18+
• ScrapingDog API key (get one at scrapingdog.com)

Install from npm

npm install -g scrapingdog-mcp-server

Install from source

git clone https://github.com/nvrunx/scrapingdog-mcp-server.git
cd scrapingdog-mcp-server
npm install
npm run build

Usage

With Claude Desktop

Add to your Claude Desktop configuration file:

macOS: ~/Library/Application Support/Claude/claude_desktop_config.json Windows: %APPDATA%/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "scrapingdog": {
      "command": "npx",
      "args": ["scrapingdog-mcp-server"]
    }
  }
}

With MCP Client

import { Client } from '@modelcontextprotocol/sdk/client/index.js';
import { StdioClientTransport } from '@modelcontextprotocol/sdk/client/stdio.js';

const transport = new StdioClientTransport({
  command: 'scrapingdog-mcp-server',
});

const client = new Client({
  name: 'scrapingdog-client',
  version: '1.0.0',
}, {
  capabilities: {},
});

await client.connect(transport);

Available Tools

General Web Scraping

#### scrape_webpage Scrape any webpage with advanced features.

{
  "url": "https://example.com",
  "api_key": "your-api-key",
  "dynamic": true,
  "premium": false,
  "country": "US",
  "wait": 3,
  "format": "html"
}

Parameters:

• url (required): Target URL to scrape
• api_key (required): Your ScrapingDog API key
• dynamic: Enable JavaScript rendering (5 credits)
• premium: Use premium residential proxies (25 credits with dynamic)
• country: Country code for geo-targeting (US, UK, CA, etc.)
• wait: Wait time in seconds (1-10)
• format: Response format (html, json)

Search Engine Tools

#### google_search Search Google and get comprehensive results (5 credits).

{
  "query": "artificial intelligence trends 2024",
  "api_key": "your-api-key",
  "country": "US",
  "language": "en",
  "page": 1,
  "num": 10
}

#### google_ai_search Get AI-powered search insights (10 credits).

{
  "query": "machine learning applications",
  "api_key": "your-api-key",
  "country": "US",
  "language": "en"
}

#### bing_search Search Bing for organic results (5 credits).

{
  "query": "web scraping best practices",
  "api_key": "your-api-key",
  "country": "US",
  "page": 1
}

#### google_maps_search Find business listings and locations (5 credits).

{
  "query": "restaurants in New York",
  "api_key": "your-api-key",
  "country": "US",
  "language": "en"
}

#### google_news_search Get latest news articles (5 credits).

{
  "query": "technology news",
  "api_key": "your-api-key",
  "country": "US",
  "language": "en",
  "time_range": "d"
}

E-commerce Tools

#### amazon_product_search Search Amazon products (1 credit).

{
  "query": "wireless headphones",
  "api_key": "your-api-key",
  "country": "US",
  "page": 1
}

#### amazon_reviews Get product reviews (100 credits).

{
  "asin": "B08N5WRWNW",
  "api_key": "your-api-key",
  "country": "US",
  "page": 1
}

#### walmart_product_search Search Walmart products (5 credits).

{
  "query": "laptop computers",
  "api_key": "your-api-key",
  "page": 1
}

Social Media Tools

#### linkedin_profile_scraper Extract LinkedIn profile data (50-100 credits).

{
  "profile_url": "https://www.linkedin.com/in/username",
  "api_key": "your-api-key"
}

#### linkedin_company_scraper Extract LinkedIn company information (50-100 credits).

{
  "company_url": "https://www.linkedin.com/company/company-name",
  "api_key": "your-api-key"
}

#### linkedin_jobs_search Search LinkedIn job listings (5 credits).

{
  "query": "software engineer",
  "location": "San Francisco, CA",
  "api_key": "your-api-key",
  "page": 1
}

#### twitter_post_scraper Extract Twitter/X posts (5 credits).

{
  "username": "elonmusk",
  "api_key": "your-api-key",
  "count": 20
}

#### instagram_profile_scraper Get Instagram profile information (15 credits).

{
  "username": "instagram",
  "api_key": "your-api-key"
}

#### instagram_posts_scraper Extract Instagram posts (15 credits).

{
  "username": "natgeo",
  "api_key": "your-api-key",
  "count": 10
}

#### facebook_scraper Extract Facebook page data (5 credits).

{
  "url": "https://www.facebook.com/page-name",
  "api_key": "your-api-key"
}

Job Board Tools

#### indeed_jobs_search Search Indeed job listings.

{
  "query": "data scientist",
  "location": "Remote",
  "api_key": "your-api-key",
  "country": "US",
  "page": 1
}

Credit System

ScrapingDog uses a credit-based pricing system:

Error Handling

The server includes comprehensive error handling:

• Validation Errors: Input parameter validation using Zod schemas
• API Errors: ScrapingDog API error responses with status codes
• Network Errors: Timeout and connection error handling
• Rate Limiting: Automatic retry logic for rate-limited requests

Development

Setup Development Environment

git clone https://github.com/nvrunx/scrapingdog-mcp-server.git
cd scrapingdog-mcp-server
npm install

Build and Run

# Build TypeScript
npm run build

# Run in development mode
npm run dev

# Run production build
npm start

Testing

# Install MCP Inspector for testing
npx @modelcontextprotocol/inspector scrapingdog-mcp-server

Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

Development Guidelines

1. Follow TypeScript best practices
2. Add proper error handling for new tools
3. Include JSDoc comments for all functions
4. Update README for new features
5. Test with MCP Inspector before submitting

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

• ScrapingDog API Documentation: docs.scrapingdog.com
• Issues: GitHub Issues
• MCP Documentation: modelcontextprotocol.io

Disclaimer

This is an unofficial MCP server for ScrapingDog API. Please ensure you comply with the terms of service of both ScrapingDog and the websites you're scraping. Always respect robots.txt and rate limiting guidelines.

Built with ❤️ by [Guddu Kumar](https://github.com/nvrunx)