croakingtoad/mcp-jina-supabase-rag

MCP Jina Supabase RAG

A lean, focused MCP server for crawling documentation websites and indexing them to Supabase for RAG (Retrieval-Augmented Generation).

Features

• Smart URL Discovery: Tries sitemap.xml first, falls back to Crawl4AI recursive discovery
• Hybrid Content Extraction: Uses Jina AI for fast content extraction, Crawl4AI as fallback
• Multi-Project Support: Index multiple documentation sites to separate Supabase projects
• Efficient Chunking: Intelligent text chunking with configurable size and overlap
• Vector Embeddings: OpenAI embeddings stored in Supabase pgvector

Architecture

┌─────────────────────────────────────────────────────────────┐
│                    MCP Server Tools                         │
├─────────────────────────────────────────────────────────────┤
│  1. crawl_and_index(url_pattern, project_name)             │
│  2. list_projects()                                         │
│  3. search_documents(query, project_name, limit)           │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                   Discovery Layer                           │
├─────────────────────────────────────────────────────────────┤
│  • Try sitemap.xml (fast)                                   │
│  • Try common doc patterns                                  │
│  • Crawl4AI recursive discovery (fallback)                  │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                  Extraction Layer                           │
├─────────────────────────────────────────────────────────────┤
│  • Jina AI Reader API (primary, fast)                       │
│  • Crawl4AI (fallback for complex pages)                    │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│              Chunking & Embedding Layer                     │
├─────────────────────────────────────────────────────────────┤
│  • Smart text chunking                                      │
│  • OpenAI embeddings (text-embedding-3-small)               │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                   Supabase Storage                          │
├─────────────────────────────────────────────────────────────┤
│  • pgvector for similarity search                           │
│  • Project isolation via source column                      │
└─────────────────────────────────────────────────────────────┘

Installation

Prerequisites

• Python 3.12+
• Supabase account
• OpenAI API key
• Jina AI API key (optional, recommended)

Setup

1. Clone the repository:

git clone https://github.com/yourusername/mcp-jina-supabase-rag.git
cd mcp-jina-supabase-rag

2. Install dependencies:

# Using uv (recommended)
uv venv
source .venv/bin/activate  # or .venv\Scripts\activate on Windows
uv pip install -e .

# Or using pip
pip install -e .

3. Set up Supabase database:

# Run the SQL in supabase_schema.sql in your Supabase SQL Editor

4. Configure environment:

cp .env.example .env
# Edit .env with your credentials

Usage

Running the MCP Server

# SSE transport (recommended for remote connections)
python src/main.py

# The server will start on http://localhost:8052/sse

Configure MCP Client

Claude Code

claude mcp add --transport sse jina-supabase http://localhost:8052/sse

Cursor / Claude Desktop

{
  "mcpServers": {
    "jina-supabase": {
      "transport": "sse",
      "url": "http://localhost:8052/sse"
    }
  }
}

Slash Command

Create /home/marty/.claude/commands/jina.md:

---
allowed-tools: mcp__jina-supabase
argument-hint: <url_pattern> <project_name>
description: Crawl documentation and index to Supabase RAG
---

# Index Documentation to Supabase

Use the jina-supabase MCP server to crawl and index documentation.

Arguments:
- $1: URL pattern (e.g., https://docs.example.com/*)
- $2: Project name for isolation

Example:
/jina https://docs.anthropic.com/claude/* anthropic-docs

Tools

crawl_and_index

Crawl a documentation site and index to Supabase.

Parameters:

• url_pattern (string): URL or pattern to crawl
• project_name (string): Project identifier for isolation
• discovery_method (string, optional): auto, sitemap, or crawl
• extraction_method (string, optional): auto, jina, or crawl4ai

Example:

await crawl_and_index(
    url_pattern="https://docs.supabase.com/docs/*",
    project_name="supabase-docs",
    discovery_method="auto",
    extraction_method="jina"
)

list_projects

List all indexed projects.

Returns: List of project names with document counts

search_documents

Search indexed documents using vector similarity.

Parameters:

• query (string): Search query
• project_name (string, optional): Filter by project
• limit (int, optional): Max results (default: 5)

Example:

results = await search_documents(
    query="How do I set up authentication?",
    project_name="supabase-docs",
    limit=10
)

Configuration

See .env.example for all configuration options.

Discovery Methods

• auto: Try sitemap first, fallback to crawl
• sitemap: Only use sitemap.xml (fast, fails if no sitemap)
• crawl: Only use Crawl4AI recursive discovery (slow, comprehensive)

Extraction Methods

• auto: Use Jina for bulk extraction (>10 URLs), Crawl4AI otherwise
• jina: Use Jina AI Reader API (fast, requires API key)
• crawl4ai: Use Crawl4AI browser automation (slow, no API key needed)

Development

# Install dev dependencies
uv pip install -e ".[dev]"

# Run tests
pytest

# Format code
black src/

# Lint
ruff check src/

Differences from mcp-crawl4ai-rag

License

MIT

Contributing

Contributions welcome! Please open an issue first to discuss changes.