mcp-metaculus by chestnutforty - MCP Server

MCP Metaculus Server

MCP server providing historical prediction data from the Metaculus forecasting platform with backtesting support.

Overview

This server retrieves question metadata, community predictions, and user comments from Metaculus questions, with strict filtering to prevent future information leakage. Perfect for forecasting applications that need to analyze historical prediction markets.

Features

Backtesting Compliant: All predictions and comments filtered by cutoff date
Comprehensive Data: Question details, background, resolution criteria, and fine print
Historical Predictions: Community forecast history with timestamps and forecaster counts
User Comments: Optional comment scraping via Firecrawl API with date filtering
Multi-Format Support: Binary, numeric/date, and multiple-choice questions

Tools

`get_metaculus_question_info`

Get Metaculus question information with historical predictions filtered by cutoff date.

Parameters:

question_url (str): The full Metaculus question URL (e.g., 'https://www.metaculus.com/questions/39771/')
cutoff_date (str): ISO format date (YYYY-MM-DD) - only return predictions made before this date

Returns:

Formatted string containing:
- Question title, ID, post ID, type
- Background information
- Resolution criteria
- Fine print (if any)
- User comments (if Firecrawl API available, filtered by date)
- Community prediction history (filtered by cutoff_date)
- Last 15 historical entries with timestamps, predictions, and forecaster counts
- Total number of historical entries

Example:

result = await get_metaculus_question_info(
    question_url="https://www.metaculus.com/questions/39771/will-there-be-a-stronger-hurricane/",
    cutoff_date="2024-06-01"
)
# Returns only predictions made before June 1, 2024

Environment Variables

Optional:

FIRECRAWL_API_KEY: API key for Firecrawl comment scraping (gracefully degrades if not provided)

Get your API key at: https://www.firecrawl.dev/

Installation

cd mcp-metaculus
uv sync

Usage

Testing Locally

mcp run -t sse metaculus_server.py:mcp

As Git Submodule

git submodule add <repo-url> mcp-servers/mcp-metaculus

Backtesting Compliance

This server is designed for strict backtesting requirements:

Prediction History Filtering

Timestamp-Based: Filters predictions where end_time <= cutoff_timestamp
No Future Data: Only includes predictions made before the cutoff date
Unix Timestamps: Uses precise timestamp comparisons for accuracy

Comment Filtering

Date Parsing: Parses time_posted field from Firecrawl results
Conservative Approach: If date can't be parsed, includes the comment (safe for backtesting)
Optional Feature: Comments require Firecrawl API, gracefully degrades without it

Question Type Support

Binary Questions:

Returns probability as percentage (e.g., "72.5%")
Shows most recent community forecast

Numeric/Date Questions:

Returns median value
Includes confidence interval range (lower and upper bounds)
Handles both numeric ranges and date predictions

Multiple Choice Questions:

Returns probabilities for all options
Format: options: ['45.0%', '30.0%', '25.0%']

API Details

Metaculus API

Endpoint: https://www.metaculus.com/api/posts/{post_id}/
Authentication: None required (public API)
Data: Question metadata and prediction history with timestamps

Firecrawl API (Optional)

Endpoint: https://api.firecrawl.dev/v2/scrape
Authentication: Bearer token via FIRECRAWL_API_KEY
Structured Extraction: Uses JSON schema to extract comment data
Caching: 48-hour cache (maxAge: 172800000)

Error Handling

The tool returns user-friendly error messages for common issues:

Invalid date format (not YYYY-MM-DD)
Malformed question URL (can't extract post ID)
API request failures with status codes
No prediction history found
No predictions before cutoff date

All errors are returned as strings rather than raising exceptions.

Data Structure

Prediction History Entry

{
    "start_time": 1234567890,       # Unix timestamp
    "end_time": 1234567890,         # Unix timestamp
    "centers": [0.725],             # Prediction values (binary: 0-1, numeric: actual values)
    "forecaster_count": 42,         # Number of forecasters
    "interval_lower_bounds": [...], # For numeric questions
    "interval_upper_bounds": [...], # For numeric questions
}

Comment Structure (from Firecrawl)

{
    "content": "Comment text...",
    "time_posted": "2024-05-15T10:30:00Z",
    "upvotes": 5,
    "downvotes": 1,
    "changed_my_mind_votes": 2,
    "author": "username"
}

Limitations

Comment Date Filtering: Relies on Firecrawl's structured extraction accuracy
API Rate Limits: Subject to Metaculus API rate limits (typically generous)
Firecrawl Costs: Comment scraping requires paid Firecrawl subscription
Historical Data: Only available for questions with prediction history
URL Format: Requires standard Metaculus question URL format

Testing

Setup

Install test dependencies:

uv pip install -e ".[test]"

(Optional) Configure Firecrawl API key by copying .env.example to .env:

cp .env.example .env

(Optional) Add your Firecrawl API key to .env:
- FIRECRAWL_API_KEY - Optional from https://firecrawl.dev/
- Tests work without this key, but comments won't be fetched

Running Tests

Run all tests:

pytest

Run with verbose output:

pytest -v

Run specific test file:

pytest tests/test_metaculus_server.py

Run specific test:

pytest tests/test_metaculus_server.py::test_get_metaculus_question_basic -v

Test Coverage

The test suite covers:

Basic functionality: Question retrieval with various cutoff dates
Output format: Validation of all expected sections (title, background, predictions, etc.)
Prediction history: Historical data filtering by cutoff date
URL handling: Valid URLs, invalid formats, nonexistent questions
Date validation: Valid dates, invalid formats, malformed dates, very early dates
Prediction details: Forecaster counts, most recent markers, total entry counts
Post ID extraction: Correct parsing of question URLs

Note: Tests make real API calls to Metaculus (no API key required). The Metaculus API is public and free. Firecrawl API key is optional - tests work without it.

Dependencies

fastmcp: MCP server framework
httpx: Async HTTP client for API requests
python-dotenv: Environment variable management