WEB-SCRAPING-MCP
If you are the rightful owner of WEB-SCRAPING-MCP and would like to certify it and/or have it hosted online, please leave a comment on the right or send an email to henry@mcphub.com.
Crawl4AI Web Scraper MCP Server is a project that provides a Model Context Protocol server using the crawl4ai library for web scraping and intelligent content extraction tasks.
The Crawl4AI Web Scraper MCP Server is designed to facilitate web scraping and content extraction using the crawl4ai library. It allows AI agents to interact with web pages, retrieve content, and perform LLM-based extraction based on natural language instructions. The server is built using FastMCP for creating the MCP server endpoint and dotenv for managing API keys. It supports containerized deployment using Docker, making it easy to deploy and manage. The server communicates over Server-Sent Events (SSE) and is configurable via environment variables. It exposes several MCP tools for web interaction, including scraping URLs, extracting text by query, and smart extraction using an LLM.
Features
- Exposes MCP tools for web interaction, including scrape_url, extract_text_by_query, and smart_extract.
- Configurable via environment variables for API keys.
- Includes Docker configuration for easy deployment.
- Communicates over Server-Sent Events (SSE) on port 8002 by default.
Tools
scrape_url
Crawl the web page and return the content in Markdown format
extract_text_by_query
Extract text fragments from web pages based on query
smart_extract
Use LLM to intelligently extract information based on instructions
