NolanRobbins/LLM-MCP

AI Gateway MCP Server 🚀

Production-ready AI Gateway with intelligent routing across multiple providers

Plug & Play AI Infrastructure - Single integration point for GPT-5, Claude Opus 4.1, Gemini 2.5, and Grok 4 with built-in cost optimization, semantic caching, and intelligent routing.

🌟 Why This Exists

Businesses want to use the latest AI models but face challenges:

• Multiple APIs to integrate and maintain
• Cost spirals without optimization
• Reliability issues when providers go down
• Complexity in choosing the right model for each task

AI Gateway MCP Server solves this by providing a single, intelligent interface to all major AI providers with automatic cost optimization and failover.

✨ Features

🧠 Intelligent Routing

• Task Classification: Automatically identifies request type (code, creative, reasoning, math)
• Multi-Factor Scoring: Routes based on cost, latency, quality, and capabilities
• Requirement Matching: Honors business requirements (low_cost, high_quality, low_latency)

💰 Cost Optimization

• Semantic Caching: 30-50% cost reduction through FAISS similarity search
• Smart Model Selection: Always uses cheapest model that meets requirements
• Real-time Tracking: Monitor spending across all providers
• Usage Analytics: Detailed cost breakdowns and optimization recommendations

🛡️ Reliability

• Health Monitoring: Real-time provider status checking
• Automatic Failover: Seamless switching when providers are down
• Rate Limiting: Adaptive throttling to prevent API limit violations
• Circuit Breaking: Prevents cascade failures

🚀 Production Ready

• Google Cloud Run: Auto-scaling serverless deployment
• Secret Manager: Secure API key management
• Monitoring: Built-in metrics and logging
• MCP Protocol: Standard interface for AI tool integration

🤖 Supported Models (2025)

🚀 Quick Start

1. Clone & Setup

git clone https://github.com/NolanRobbins/LLM-MCP.git
cd LLM-MCP

# Install dependencies using Makefile
make install

# Or manually:
pip install uv
uv venv
source .venv/bin/activate  # Linux/Mac
# .venv\Scripts\activate     # Windows
uv pip install -e ".[dev]"

2. Configure Environment

# Load environment variables (edit setup/set_env.sh first)
source setup/set_env.sh

# Or copy environment template for local development
cp .env.example .env
# Edit .env with your API keys
nano .env

Required API keys:

• OPENAI_API_KEY - OpenAI API key
• ANTHROPIC_API_KEY - Anthropic Claude API key
• GOOGLE_API_KEY - Google Gemini API key
• XAI_API_KEY - xAI Grok API key

3. Test Locally

# Start the MCP server
python main.py
# Or using Makefile
make run

# Test in another terminal
python scripts/test_server.py

# Run tests
make test

4. Deploy to Google Cloud

# First-time setup (creates project, enables APIs)
make setup

# Or manually:
cd setup/
./setup_gcp.sh
./create_project.sh
source set_env.sh  # Edit this file first with your details
./enable_apis.sh

# Deploy to Cloud Run
make deploy
# Or manually:
cd deployment/
./deploy.sh

📡 Usage

MCP Client Integration

The server exposes these MCP tools:

unified_completion - Smart AI Routing

response = mcp_client.call_tool("unified_completion", {
    "prompt": "Write a Python function to sort a list",
    "requirements": {"low_cost": True}  # Routes to O4-mini
})

get_provider_status - Health Monitoring

status = mcp_client.call_tool("get_provider_status")
# Returns real-time provider health and latency

get_usage_metrics - Cost Analytics

metrics = mcp_client.call_tool("get_usage_metrics", {
    "time_range": "24h"
})
# Returns detailed cost and performance metrics

optimize_prompt - AI-Powered Improvement

optimized = mcp_client.call_tool("optimize_prompt", {
    "prompt": "Tell me about AI",
    "optimization_goal": "clarity"
})

run_ab_test - Prompt Comparison

results = mcp_client.call_tool("run_ab_test", {
    "prompt_a": "Explain quantum computing",
    "prompt_b": "Describe quantum computing simply",
    "iterations": 3
})

Claude Code Integration

Add to your MCP client configuration:

{
  "servers": {
    "ai-gateway": {
      "command": "python",
      "args": ["path/to/main.py"]
    }
  }
}

HTTP API (Alternative)

# Direct API calls with authentication
curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
     -H "Content-Type: application/json" \
     -d '{"prompt": "Hello AI", "requirements": {"low_cost": true}}' \
     https://your-gateway.run.app/unified_completion

🏗️ Architecture

Client Apps → AI Gateway MCP Server → Provider APIs
                     ↓
              [Intelligent Router]
                     ↓
           ┌─────────┼─────────┐
           ▼         ▼         ▼
      [OpenAI]  [Anthropic] [Google] [xAI]

Project Structure:

LLM-MCP/
├── src/llm_mcp/           # Main application
│   ├── gateway/           # Core gateway components
│   ├── agents/            # AI agents
│   ├── server/            # MCP server
│   └── utils/             # Utilities
├── setup/                 # Environment & GCP setup
├── deployment/            # Deployment scripts
├── scripts/               # Test & demo scripts
├── config/                # Configuration files
├── tests/                 # Test suite
└── main.py               # Entry point

Core Components:

• Router (src/llm_mcp/gateway/router.py) - Intelligent model selection
• Cache (src/llm_mcp/gateway/cache.py) - Semantic similarity caching
• Rate Limiter (src/llm_mcp/gateway/rate_limiter.py) - Adaptive throttling
• Cost Tracker (src/llm_mcp/gateway/cost_tracker.py) - Real-time cost monitoring
• Metrics (src/llm_mcp/gateway/metrics.py) - Performance analytics

🔧 Configuration

Environment Variables

Model Configuration

Edit config/models.yaml to:

• Add new models
• Update pricing
• Modify capabilities
• Adjust scoring weights

🛠️ Development Commands

The project includes a comprehensive Makefile for easy development:

# Setup and Installation
make install          # Install package and dependencies
make dev             # Install in development mode

# Development
make run             # Run the server
make run-dev         # Run with development dependencies

# Code Quality
make test            # Run test suite with coverage
make lint            # Run type checking and linting
make format          # Format code with black and ruff
make clean           # Clean cache and build artifacts

# Deployment
make setup           # Setup GCP environment (first time)
make deploy          # Deploy to Google Cloud Run

# Environment
make env             # Show environment setup command

📊 Performance

Cost Savings

• 30-50% reduction through semantic caching
• Optimal routing always uses cheapest suitable model
• Real-time tracking prevents budget overruns

Reliability

• 99.9% uptime with automatic failover
• <100ms routing overhead
• 1000+ req/min throughput

Developer Experience

• Single integration instead of 4+ APIs
• 5-minute setup with deployment script
• Zero maintenance with Cloud Run auto-scaling

🧪 Testing

# Unit tests
make test
# Or manually:
pytest tests/

# Load testing
python scripts/load_test.py --requests=100

# Integration testing
python scripts/test_server.py

# Simple test
python scripts/simple_test.py

# Health check
curl http://localhost:8080/health

🚀 Deployment

Google Cloud Run (Recommended)

# First time setup
make setup

# Deploy
make deploy
# Or manually:
cd deployment/
./deploy.sh

Docker

docker build -t ai-gateway .
docker run -p 8080:8080 ai-gateway

Local Development

# Using entry point
python main.py

# Using Makefile
make run

# Development mode with auto-reload
make run-dev

🔐 Security

• API Keys: Stored in Google Secret Manager
• Authentication: Google Cloud IAM integration
• Network: HTTPS encryption and VPC controls
• Audit: Complete request/response logging
• Secrets: Never committed to git (see .gitignore)

📈 Monitoring

Built-in Metrics

• Request latency and throughput
• Cost per request and provider
• Cache hit rates and savings
• Provider health and availability

Google Cloud Integration

• Cloud Logging for request traces
• Cloud Monitoring for alerts
• Error Reporting for exceptions
• Cloud Run metrics for scaling

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests
5. Submit a pull request

📄 License

MIT License - see file for details.

🆘 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: Check docs/ directory

🙏 Acknowledgments

• Model Context Protocol (MCP) - Standard protocol
• FastMCP - MCP server framework
• Google Cloud Run - Serverless deployment
• AI Providers: OpenAI, Anthropic, Google, xAI

Built with ❤️ for the AI community

Star ⭐ this repo if it helps you build better AI applications!