elchika-inc/open-crawler-mcp-server
3.1
If you are the rightful owner of open-crawler-mcp-server and would like to certify it and/or have it hosted online, please leave a comment on the right or send an email to henry@mcphub.com.
Open Crawler MCP Server is a versatile tool for web crawling and content extraction, supporting multiple output formats and ensuring compliance with web standards.
Tools
Functions exposed to the LLM to take actions
crawl_page
Extracts content from a web page in multiple formats with automatic robots.txt compliance checking.
check_robots
Validates if a URL is allowed to be crawled according to the site's robots.txt file.
Prompts
Interactive templates invoked by user choice
No prompts
Resources
Contextual data attached and managed by the client