mcp-server-spider

3.2

If you are the rightful owner of mcp-server-spider and would like to certify it and/or have it hosted online, please leave a comment on the right or send an email to henry@mcphub.com.

A Model Context Protocol server for Spider crawler interaction and automation.

mcp-server-spider: A spider MCP server

Overview

A Model Context Protocol server for Spider crawler interaction and automation. This server provides tools to crawl and scrape web pages.

Please note that mcp-server-spider is currently in early develpoment. There might be bugs and features added in the future.

Tools

crawl
- Crawls the given url and returns the list of URLs that were found
- Input:
  - url: The url to crawl
  - headers: Additional headers passed along with crawl requests
  - user_agent: User agent to use for the crawl requests
  - depth: The depth of link traversal
  - blacklist: A list of regural expression to blacklist URLs from the crawling process
  - whitelist: A list of regular expression to whitelist URLS from the crawling process
  - respect_robots_txt: Whether to respect robots.txt file
  - accept_invalid_certs: Whether to accept invalid certifcates or not
- Returns: List of URLs found
scrape
- Scrapes the given url and returns a list of JSON objects that contain the url, links and content of each page discovered
- Input: Same as crawl
- Returns: A list of JSON objects (as a string) that contain the url, links and content of each page discovered

Installation

Using uv (recommended)

When using uv no specific installation is needed. We will use uvx to directly run mcp-server-spider.

Using PIP

Alternatively you can install mcp-server-spider via pip:

pip install mcp-server-spider

After installation, you can run it as a script using:

python -m mcp_server_spider

Related MCP Servers

View all browser_automation servers →

firecrawl-mcp-server

4.7

by mendableai

Firecrawl MCP Server is a Model Context Protocol server implementation that integrates with Firecrawl for web scraping capabilities.

browser_automation

deepwiki-mcp

4.6

by regenrek

This is an unofficial Deepwiki MCP Server that processes Deepwiki URLs, crawls pages, converts them to Markdown, and returns documents or lists by page.

browser_automation

tavily-mcp

4.3

by tavily-ai

The Tavily MCP server is a Model Context Protocol server that integrates with AI systems to provide real-time web search and data extraction capabilities.

browser_automation

fetcher-mcp

4.3

by jae-jae

Fetcher MCP is a server designed to fetch web page content using the Playwright headless browser.

browser_automation

web-eval-agent

3.8

by Operative-Sh

operative.sh's MCP Server is a tool for autonomous debugging of web applications directly from your code editor.

browser_automation

mcp-crawl4ai-rag

3.8

by coleam00

Crawl4AI RAG MCP Server is a powerful implementation of the Model Context Protocol (MCP) integrated with Crawl4AI and Supabase, providing AI agents and AI coding assistants with advanced web crawling and RAG capabilities.

browser_automation

playwright-mcp

3.8

by microsoft

Playwright MCP is a Model Context Protocol server that provides browser automation capabilities using Playwright, enabling LLMs to interact with web pages through structured accessibility snapshots.

browser_automation

browser-use-mcp-server

3.8

by co-browser

An MCP server that enables AI agents to control web browsers using browser-use.

browser_automation

Notte Browser

3.7

by nottelabs

The Notte MCP Server is a Model Context Protocol server designed to manage sessions, run agents, and perform actions such as observing, scraping, and authenticating within the agentic ecosystem.

cloud_platforms

mcp-browser-use

3.7

by Saik0s

AI-driven browser automation server implementing the Model Context Protocol (MCP) for natural language browser control and web research.

browser_automation

RedNote-MCP

3.7

by iFurySt

RedNote MCP is a Model Context Protocol server for accessing content on Xiaohongshu.

browser_automation

duckduckgo-mcp-server

3.7

by nickclyde

A Model Context Protocol (MCP) server that provides web search capabilities through DuckDuckGo, with additional features for content fetching and parsing.

browser_automation

MCP-Server-Playwright

3.6

by Automata-Labs-team

A Model Context Protocol server that provides browser automation capabilities using Playwright.

browser_automation

moling

3.6

by gojue

MoLing is a dependency-free local office automation assistant that interacts with the system through operating system APIs, enabling file system operations and command execution.

os_automation

google-search

3.6

by web-agent-master

A Playwright-based Node.js tool that bypasses search engine anti-scraping mechanisms to execute Google searches and extract results.

browser_automation

webscraper

3.6

by saishridhar

Webscraper MCP is an MCP server designed for Claude desktop, enabling it to scrape text from websites, YouTube transcripts, and PDFs.

browser_automation

crawl4ai-mcp-server

3.6

by weidwonder

Crawl4AI MCP Server is an intelligent information retrieval server based on the Model Context Protocol (MCP), designed to enhance AI assistant systems with powerful search capabilities and web content understanding optimized for LLMs.

browser_automation

kagimcp

3.6

by kagisearch

Kagi MCP server is a model context protocol server designed to integrate with various platforms, providing search and summarization capabilities through the Kagi API.

browser_automation

website-downloader

3.6

by pskill9

The Website Downloader MCP Server is designed to download entire websites using wget, preserving the website structure and converting links for local use.

browser_automation

browser-mcp

3.6

by djyde

A browser extension and MCP server that allows you to interact with the browser you are using.

browser_automation

jcrawl4ai-mcp-server

3.5

by rainyheart

jcrawl4ai-mcp-server is a Java-based MCP server for interacting with the Crawl4ai API, enabling web crawling with specified parameters.

browser_automation

mcp-ui

3.5

by machaojin1917939763

MCP聊天应用是一个基于Vue.js构建的现代化聊天界面，支持通过Model Context Protocol (MCP)与各种服务和工具进行交互。

ai_chatbot

mcp-server-serper

3.5

by marcopesani

A TypeScript-based MCP server that provides web search and webpage scraping capabilities using the Serper API.

browser_automation

editor-mcp-server

3.5

by playcanvas

An MCP Server for automating the PlayCanvas Editor using an LLM.

browser_automation

scrapeless-mcp-server

3.5

by scrapeless-ai

Seamlessly integrate real-time Google SERP results into your LLM applications using the Scrapeless MCP server.

browser_automation

Sketch-Context-MCP

3.5

by jshmllr

A Model Context Protocol (MCP) server for integrating Sketch designs with IDEs such as Cursor, Cline, or Windsurf.

browser_automation

xhs-mcp-server

3.5

by aicu-icu

小红书MCP服务器是基于Electron和小红书Web API的开发工具，旨在提供快速、便捷的开发体验。

browser_automation