web-crawler-mcp-server
If you are the rightful owner of web-crawler-mcp-server and would like to certify it and/or have it hosted online, please leave a comment on the right or send an email to henry@mcphub.com.
A Model Context Protocol (MCP) server that provides a web crawling and content extraction tool for AI assistants such as Claude Desktop, Cursor, and other MCP-compatible clients.
The Web Crawler MCP Server is designed to facilitate web content extraction for AI assistants by leveraging the Model Context Protocol. It utilizes Puppeteer with a stealth plugin to navigate and extract content from public web pages while bypassing anti-bot measures. The server processes the extracted data to produce clean, whitespace-normalized text, making it ideal for consumption by language models. This server is easily integrable with MCP-compatible clients like Claude Desktop, providing a seamless experience for users looking to enhance their AI's web interaction capabilities.
Features
- Extracts and cleans main text content from any public web page.
- Uses Puppeteer with stealth plugin to bypass anti-bot protections.
- Returns readable, whitespace-normalized text for LLM consumption.
- Easy integration with Claude Desktop and other MCP clients.
Tools
web-crawler
Extracts and returns the cleaned text content from a specified URL.
