web-search-mcp-server by slinusc - MCP Server

Web Search MCP Server

A production-ready Model Context Protocol (MCP) server that provides web search capabilities through SearXNG. Designed for seamless integration with LLM agents, featuring HTTP/SSE transport and LLM-optimized response formatting.

Features

Multi-Engine Search - Aggregates results from Brave, DuckDuckGo, Startpage, and more
LLM-Optimized - Three specialized tools with formatted responses for agent consumption
Knowledge Extraction - Automatic extraction of structured facts from Wikidata
Real-Time Streaming - MCP over HTTP/SSE for live agent integration
Docker Ready - Fully containerized with docker-compose
Production Features - Health checks, metrics, structured logging, request tracking

Architecture

┌─────────────────────┐
│   LLM Agent         │
│  (MCP Client)       │
└──────────┬──────────┘
           │ MCP over HTTP/SSE
           ↓
┌─────────────────────┐
│  Web Search MCP     │  Port 8003
│     Server          │
│  - 3 Search Tools   │
│  - REST API         │
│  - Metrics          │
└──────────┬──────────┘
           │ HTTP
           ↓
┌─────────────────────┐
│     SearXNG         │  Port 8888
│  (Meta-Search)      │
│  - Brave            │
│  - DuckDuckGo       │
│  - Startpage        │
│  - Wikipedia        │
└─────────────────────┘

Quick Start

Prerequisites

Docker & Docker Compose
(Optional) Python 3.11+ for local development

1. Clone and Start

git clone https://github.com/slinusc/web-search-mcp-server.git
cd web-search-mcp-server
docker compose up -d

This starts:

SearXNG on port 8888 (search backend)
MCP Server on port 8003 (agent interface)

2. Test the API

# Quick search (LLM-optimized)
curl -X POST http://localhost:8003/api/quick_search \
  -H "Content-Type: application/json" \
  -d '{"query": "latest developments in AI"}'

# Structured search (with facts)
curl -X POST http://localhost:8003/api/structured_search \
  -H "Content-Type: application/json" \
  -d '{"query": "climate change 2024"}'

# Health check
curl http://localhost:8003/healthz

3. Connect Your MCP Client

{
  "mcpServers": {
    "websearch": {
      "url": "http://localhost:8003/sse"
    }
  }
}

MCP Tools

1. `web_search`

Comprehensive search with full metadata

{
  "query": "machine learning frameworks",
  "categories": "general",  # or "news", "images", "videos"
  "pageno": 1
}

Returns: Full SearXNG response with all engines, suggestions, and infoboxes.

2. `quick_search`

Fast, LLM-optimized results

{
  "query": "who invented the internet",
  "categories": "general"
}

Returns: Top 10 ranked results with knowledge base, relevance scores, and source attribution.

3. `structured_search`

Facts-focused with context

{
  "query": "Python programming language",
  "categories": "general"
}

Returns: Ranked results + Wikidata attributes + context snippets + suggestions.

Response Format

Quick Search Example

{
  "ok": true,
  "request_id": "a1b2c3d4e5f6",
  "query": "Python programming",
  "total_results": 28,
  "knowledge_base": {
    "entity": "Python (programming language)",
    "description": "Python is a high-level, interpreted programming language...",
    "attributes": {
      "Developer": "Python Software Foundation",
      "First appeared": "1991",
      "Influenced by": "ABC, Modula-3, C"
    },
    "image": "https://upload.wikimedia.org/..."
  },
  "top_results": [
    {
      "rank": 1,
      "title": "Python.org",
      "url": "https://www.python.org/",
      "snippet": "Python is a programming language that lets you work quickly...",
      "score": 9.0,
      "engines": ["brave", "duckduckgo", "startpage"]
    }
  ],
  "engines_used": ["brave", "duckduckgo", "startpage", "wikipedia"]
}

REST API Reference

Endpoint	Method	Description
`/sse`	GET	MCP SSE stream (for agents)
`/messages`	POST	MCP client messages
`/api/search`	POST	General web search
`/api/quick_search`	POST	Quick LLM-optimized search
`/api/structured_search`	POST	Structured facts search
`/healthz`	GET	Health check
`/metrics`	GET	Performance metrics
`/tools`	GET	List available MCP tools

Configuration

Environment Variables

# docker-compose.yml or .env
SEARXNG_URL=http://searxng:8080  # SearXNG backend URL
SEARXNG_TIMEOUT=10.0              # Request timeout (seconds)

SearXNG Configuration

Edit config/settings.yml to:

Enable/disable search engines
Configure result formats
Set rate limiting
Customize search behavior

Key settings:

search:
  formats:
    - html
    - json  # Required for MCP server

server:
  limiter: false  # Adjust for production

Development

Local Setup

# Create virtual environment
python -m venv venv
source venv/bin/activate  # or `venv\Scripts\activate` on Windows

# Install dependencies
pip install -r requirements.txt

# Run server
python server.py

Server runs on http://localhost:8003

Testing

# Test quick search
curl -X POST http://localhost:8003/api/quick_search \
  -H "Content-Type: application/json" \
  -d '{"query": "test query"}'

# Check metrics
curl http://localhost:8003/metrics

# List available tools
curl http://localhost:8003/tools | jq

Docker

Build

docker build -t mcp-server-websearch .

Run Standalone

docker run -d \
  --name mcp-websearch \
  -p 8003:8003 \
  -e SEARXNG_URL=http://searxng:8080 \
  mcp-server-websearch

Compose (Recommended)

# Start all services
docker compose up -d

# View logs
docker compose logs -f mcp-websearch

# Stop services
docker compose down

Monitoring

Health Check

curl http://localhost:8003/healthz
# Response: ok

Metrics

curl http://localhost:8003/metrics

Response:

{
  "ok": true,
  "metrics": {
    "api_search_calls": 42,
    "api_quick_search_calls": 156,
    "api_structured_search_calls": 23,
    "mcp_call_tool_calls": 221,
    "searxng_requests": 221,
    "errors_total": 2
  }
}

Agent Integration

Generic MCP Client

The server exposes three MCP tools that are automatically available to any MCP-compatible agent:

web_search - For comprehensive search with full details
quick_search - For fast answers with top results
structured_search - For research with structured facts

LangChain Integration

from langchain.tools import Tool

# Wrap the MCP tools
tools = [
    Tool(
        name="web_search",
        func=call_mcp_tool,  # Your MCP client wrapper
        description="Search the web for current information"
    )
]

Use Cases

Question Answering - Quick answers with source attribution
Research Assistants - Deep research with structured facts
RAG Systems - Retrieve current information for context
News Monitoring - Track recent events and developments
Fact Checking - Verify information across multiple sources

Security Notes

Rate Limiting: Configure SearXNG's limiter.toml for production
API Keys: Not required (SearXNG is self-hosted)
CORS: Currently allows all origins - restrict in production
Network: Use Docker networks to isolate services

Troubleshooting

SearXNG returns 403 Forbidden

# Check if JSON format is enabled in config/settings.yml
docker exec searxng grep -A 3 "formats:" /etc/searxng/settings.yml

# Should show:
# formats:
#   - html
#   - json

Connection refused errors

# Ensure services are running
docker compose ps

# Check logs
docker compose logs searxng
docker compose logs mcp-websearch

No search results

# Test SearXNG directly
curl -X POST http://localhost:8888/search \
  -H "X-Forwarded-For: 127.0.0.1" \
  --data "q=test&format=json"

License

MIT License - see file for details.

Acknowledgments

SearXNG - Privacy-respecting metasearch engine
Model Context Protocol - Anthropic's MCP specification
Starlette - Lightweight ASGI framework

Contributing

Contributions welcome! Please:

Fork the repository
Create a feature branch
Commit your changes
Push to the branch
Open a Pull Request

Support

Issues: GitHub Issues
Discussions: GitHub Discussions

slinusc/web-search-mcp-server