rodcar/pnda-mcp

📋 Table of Contents

• 🎯 Overview
• 🎬 Demo
• 🔧 Tools
• 💬 Prompts
• 🚀 How to Use
• 💡 Examples
• 🏛️ Architecture Diagram
• ⚙️ ETL Pipeline
• 📝 License

🎯 Overview

PNDA-MCP is a Model Context Protocol (MCP) server for Peru's National Open Data Platform (Plataforma Nacional de Datos Abiertos). Although Peru's open data platform datosabiertos.gob.pe hosts valuable datasets, it can be a challenging for AI agents to find and retrieve the most relevant data for a specific data analysis question. PNDA-MCP simplifies this by providing tools and prompts that let AI agents or any MCP client (such as VS Code or Claude Desktop) easily search for and access datasets metadata, and associated data files. The goal is to enable data scientist agents or code agents to automatically discover and analyze public datasets.

This repository includes the ETL pipeline used to extract, transform, and index dataset titles (see etl folder).

🎬 Demo

Demo (Spanish):

🔧 Tools

💬 Prompts

🚀 How to Use

VS Code (Remote Server)

Note: Requires npx which comes bundled with npm. If you don't have npm installed, install Node.js which includes npm.

The fastest and easiest way to try this MCP is to use the 1-click installation button:

Note: If the MCP tools and prompts do not load immediately, please try restarting VS Code.

Manual installation:

1. Open the Command Palette: View > Command Palette (or Cmd+Shift+P on Mac / Ctrl+Shift+P on Windows/Linux)
2. Type and select: MCP: Add Server...
3. Choose "Command (stdio)" as the server type
4. For "Command to run (with optional arguments)", enter: npx mcp-remote https://pnda-mcp.onrender.com/mcp
5. Set the name for the MCP server: pnda-mcp
6. Select where to save the configuration: User Settings saves the config globally for all projects. Workspace Settings saves it locally for just the current one.
7. Save the configuration
8. Restart VS Code for the MCP server to become available.

VS Code (Local Server)

Important: Before running the MCP server locally, you need to:

1. Have an OpenAI API key. Get your OpenAI API key from platform.openai.com.
2. Have a Pinecone account. If you don't have an account, you can sign up at pinecone.io.
3. Configure your OpenAI API key and Pinecone API key in the .env configuration file.
4. Run the ETL pipeline to index the datasets metadata from PNDA to Pinecone (see the ETL Pipeline section below)

1. Open the Command Palette: View > Command Palette (or Cmd+Shift+P on Mac / Ctrl+Shift+P on Windows/Linux)
2. Type and select: MCP: Add Server...
3. Choose "Command (stdio)" as the server type

Note: Replace /path/to/pnda-mcp with the actual path where you cloned the repository.

4. For "Command to run (with optional arguments)", enter: uv --directory /path/to/pnda-mcp run main.py
5. Set the name for the MCP server: pnda-mcp
6. Select where to save the configuration: User Settings saves the config globally for all projects. Workspace Settings saves it locally for just the current one.
7. Save the configuration
8. Restart VS Code for the MCP server to become available.

MCP Inspector (Alternative)

Important: Before running the MCP server locally, you need to:

1. Have an OpenAI API key. Get your OpenAI API key from platform.openai.com.
2. Have a Pinecone account. If you don't have an account, you can sign up at pinecone.io.
3. Configure your OpenAI API key and Pinecone API key in the .env configuration file.
4. Run the ETL pipeline to index the datasets metadata from PNDA to Pinecone (see the ETL Pipeline section below)

Note: Requires npx which comes bundled with npm. If you don't have npm installed, install Node.js which includes npm.

Note: Replace /path/to/pnda-mcp with the actual path where you cloned the repository.

Run:

npx @modelcontextprotocol/inspector \
  uv \
  --directory /path/to/pnda-mcp \                     
  run \
  main.py

Open MCP Inspector (URL displayed in the console) and configure the MCP client with the following settings:

• Transport Type: STDIO
• Command: python
• Arguments: main.py

💡 Examples

🏛️ Architecture Diagram

PNDA-MCP follows the Model Context Protocol specification and provides a clean abstraction layer for PNDA.

graph LR
    CLIENT[MCP Client<br/>VS Code, Cursor, etc.] --> MCP_SERVER[PNDA-MCP Server]
    
    subgraph TOOLS ["🔧 Tools"]
        DATASET_SEARCH[dataset_search]
        DATASET_DETAILS[dataset_details]
    end
    
    subgraph "💬 Prompts"
        QUESTION_GEN[question_generation]
        ANALYSIS_QUICK[analysis_quick]
        ANALYSIS_FULL[analysis_full]
    end
    
    MCP_SERVER --> DATASET_SEARCH
    MCP_SERVER --> DATASET_DETAILS
    MCP_SERVER --> QUESTION_GEN
    MCP_SERVER --> ANALYSIS_QUICK
    MCP_SERVER --> ANALYSIS_FULL
    
    DATASET_SEARCH -->|semantic search| PINECONE[Pinecone Vector Database]
    DATASET_SEARCH --> OPENAI[OpenAI Text Embeddings API]
    DATASET_DETAILS --> CACHE[Cache Layer]
    CACHE --> |fallback source| PNDA_API[PNDA API]
    CACHE --> |secondary fallback| PINECONE
    
    style CLIENT fill:#e3f2fd
    style MCP_SERVER fill:#f3e5f5
    style PNDA_API fill:#fff3e0
    style PINECONE fill:#fff3e0
    style OPENAI fill:#fff3e0

⚙️ ETL Pipeline

Important: The following ETL documentation is only needed if you want to run the MCP locally or deploy your own MCP service. You can use the remote MCP service without running the ETL.

To search datasets using natural language, semantic search with text vector embeddings is used. The ETL pipeline handles the initial indexing and ongoing synchronization of the vector database containing dataset metadata from Peru's National Open Data Platform. It can be run manually or automatically via cron jobs to ensure the dataset information stays up to date.

Requirements

• Docker & Redis: Runs Redis server locally which serves as a message broker and result backend to coordinate tasks during ETL pipeline execution with Celery workers.
• OpenAI API key: The OpenAI Text Embeddings API converts dataset titles into vectors using the text-embedding-3-small model. Get your OpenAI API key from platform.openai.com.
• Pinecone account: Dataset titles are indexed in Pinecone cloud vector database for semantic search. If you don't have an account, you can sign up at pinecone.io.

Setup and Usage

Note: Make sure you have uv installed. If not, install it from uv.tool.

1. Clone and install:

   git clone https://github.com/rodcar/pnda-mcp.git
   cd pnda-mcp
   uv sync
   

2. Create .env file

   MacOS/Linux:

   cp .env.example .env
   

   Windows:

   copy .env.example .env
   

3. Set your OPENAI_API_KEY and PINECONE_API_KEY values in the .env file.

   Note: Get your OpenAI API key from platform.openai.com and your Pinecone API key from app.pinecone.io.

4. Run Redis with Docker

   Note: Celery also supports other broker and backend options. See Celery documentation for more details.

   docker run -d -p 6379:6379 redis
   

5. Start Celery worker

   MacOS/Linux:

   ./etl/celery_worker.sh
   

   Windows:

   uv run celery -A etl.tasks.app worker --loglevel=info
   

   Note: The Celery worker processes ETL tasks asynchronously. Keep this terminal window open, you'll see task execution logs here when the pipeline runs.

6. Run the ETL pipeline

   The pipeline can be executed manually (on-demand) or automated using a cron job for daily execution. It is recommended to perform the initial indexing manually, then use the cron job to maintain data synchronization.

   Manual Execution:

   python -m etl.pipeline
   

   Note: The execution might take several minutes. You can see the logs in the etl/logs/etl.log file, and the output files of intermediate ETL tasks in the etl/results folder.

   Note: You can remove all pending the tasks from the Celery task queue with the following command: celery -A etl.tasks.app purge -f.

   Scheduled with Cron Job (MacOS/Linux):

   a. Make the script executable:

   Note: Replace /path/to/pnda-mcp/etl/cron.sh with the actual path to the cron.sh file.

   chmod +x /path/to/pnda-mcp/etl/cron.sh
   

   b. Edit crontab:

   crontab -e
   

   c. Add this line (runs daily at 2 AM):

   Note: Replace /path/to/pnda-mcp/etl/cron.sh with the actual path to the cron.sh file.

   Note: If you are using vim, press i to enter insert mode and paste the cron job; press Esc to return to normal mode. Use :wq to save and exit.

   Note: To change the hour replacing the 2 (which means 2 AM) with your desired hour in 24-hour format (e.g., 14 for 2 PM).

   0 2 * * * /path/to/pnda-mcp/etl/cron.sh
   

   d. Verify the cron job was added to the crontab:

   crontab -l
   

   The pipeline will execute daily at the time specified in the crontab configuration.

   Note: You can see the logs in the etl/logs/etl.log file, and the output files of intermediate ETL tasks in the etl/results folder.

ETL Diagram

The following diagram shows the three-stage ETL pipeline that processes dataset metadata from Peru's National Open Data Platform.

flowchart LR
    subgraph EXTRACT_WRAPPER["<b>Extract</b>"]
        EXTRACT["Fetch complete dataset list from PNDA API"] --> PNDA_API["For each dataset, fetch metadata from PNDA API"]
    end
    
    subgraph TRANSFORM_WRAPPER["<b>Transform</b>"]
        FILTER["Filter active datasets"] --> STRUCTURE["Format dataset metadata for indexing"]
    end
    
    subgraph LOAD_WRAPPER["<b>Load</b>"]
        FILTER_CHANGED["Filter datasets with changes*"] --> EMBEDDINGS["Generate embeddings using OpenAI Text Embeddings API"] --> UPSERT["Upsert embeddings to the vector database (Pinecone)"]
    end
    
    EXTRACT_WRAPPER e1@==> TRANSFORM_WRAPPER
    TRANSFORM_WRAPPER e2@==> LOAD_WRAPPER
    
    e1@{ animate: true }
    e2@{ animate: true }
    
    style EXTRACT fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
    style PNDA_API fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
    style FILTER fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
    style STRUCTURE fill:#e8f5e9,stroke:#388e3c,stroke-width:2px
    style FILTER_CHANGED fill:#fff3e0,stroke:#f57c00,stroke-width:2px
    style EMBEDDINGS fill:#fff3e0,stroke:#f57c00,stroke-width:2px
    style UPSERT fill:#fff3e0,stroke:#f57c00,stroke-width:2px

*Filters datasets where metadata_modified has changed since the last local version (etl/results/processing_results.json). This means the metadata must be updated in the vector database.

📝 License

This project is licensed under the .