news-mcp

CytrexSGR/news-mcp

3.2

If you are the rightful owner of news-mcp and would like to certify it and/or have it hosted online, please leave a comment on the right or send an email to henry@mcphub.com.

News MCP is a Model Context Protocol server designed for intelligent RSS aggregation and AI-powered news analysis, enabling seamless control of news feeds through MCP-enabled LLM clients.

Tools
5
Resources
0
Prompts
0

News MCP - MCP-Native RSS Intelligence Platform

Python 3.9+ FastAPI PostgreSQL MCP License: GPL-3.0

Model Context Protocol (MCP) server for intelligent RSS aggregation and AI-powered news analysis. Control your news feeds directly from Claude Desktop or any MCP-enabled LLM client with 20+ native tools. Enterprise-grade system with automatic feed management, sentiment analysis, and real-time monitoring.

🚀 Features

Core Functionality

  • RSS Feed Management: Automatic collection and processing of RSS feeds
  • Auto-Analysis System: Automatic AI analysis of new feed items (Phase 2 ✅)
  • Auto-Scraping: Full-text article extraction with trafilatura
    • Automatic scraping for feeds with enable_scraping=true
    • Triggered during sentiment analysis workflow
    • Stores scraped content in items.scraped_content (JSONB)
    • Error tracking and safety mechanisms (3-retry limit, timeout protection)
  • AI-Powered Analysis: Multi-dimensional sentiment analysis with OpenAI GPT
    • Sentiment Scoring: Overall, market, and thematic sentiment (-1.0 to +1.0)
    • Geopolitical Analysis: 13-field assessment including stability, security, diplomatic impact
    • Impact & Urgency: Quantified metrics for prioritization (0.0 to 1.0)
    • Market Indicators: Bullish/bearish/neutral sentiment, volatility assessment
  • Perplexity Research: Advanced web research with curated domain filtering
    • Dynamic parameter forms for research templates
    • Time-filtered research with Top-20 domain curation
    • Enhanced error handling and retry logic
    • Real-time test functionality in template editor
  • Real-time Dashboard: Live monitoring of feed status and analysis runs
  • Advanced Analytics: Detailed statistics and performance metrics
  • Template System: Dynamic feed templates for flexible configuration

MCP Integration (Model Context Protocol)

  • 🔌 20+ MCP Tools: Complete system control via LLM-native interface
  • Feed Management: List, add, update, delete feeds via Claude Desktop/LLM clients
  • Analytics Tools: Dashboard stats, trending topics, article search
  • Database Access: Safe read-only SQL queries with predefined templates
  • Health Monitoring: System diagnostics, error analysis, scheduler status
  • LAN Access Ready: HTTP bridge for remote Claude Desktop integration
  • See for full details

Technical Features

  • Async Processing: High-performance asynchronous processing
  • Queue Management: Intelligent job queue with rate limiting
  • Centralized Caching: In-memory selection cache for optimized performance
  • Modular Architecture: Clean separation of API, services, and views
  • Dark Mode UI: Modern, responsive web interface
  • Feature Flags: Gradual rollout with circuit breaker and A/B testing

📸 Screenshots

Get a visual overview of the system's capabilities:

  • - RSS feed monitoring with health status
  • - Real-time article feed with AI analysis
  • - Manual analysis interface
  • - Automated AI processing
  • - Performance metrics and charts
  • - Query interface with templates

📋 Table of Contents

🏗️ Architecture

The system follows a modern, modular architecture with clear separation of concerns:

  • Repository Pattern: Type-safe data access with SQLModel
  • Service Layer: Business logic and orchestration
  • API Layer: RESTful endpoints with FastAPI
  • Worker System: Background processing with rate limiting
  • UI Layer: HTMX + Alpine.js for progressive enhancement

See for detailed documentation.

Recent Refactoring (v4.0.0)

  • Modularized 765-line monolithic repository into focused modules
  • Improved error handling and recovery mechanisms
  • Enhanced performance with skip tracking
  • Fixed scope limit handling for proper item selection

🛠 Installation

Prerequisites

# System Requirements
- Python 3.9+
- PostgreSQL 15+
- Redis (optional, for extended caching features)
- Git

# Ubuntu/Debian
sudo apt update
sudo apt install python3 python3-pip postgresql postgresql-contrib git

# macOS
brew install python postgresql git

Setup

  1. Clone Repository
git clone https://github.com/CytrexSGR/news-mcp.git
cd news-mcp
  1. Create Virtual Environment
python3 -m venv venv
source venv/bin/activate  # Linux/macOS
# or: venv\Scripts\activate  # Windows
  1. Install Dependencies
pip install -r requirements.txt
  1. Database Setup
# Create PostgreSQL user and database
sudo -u postgres psql
CREATE USER news_user WITH PASSWORD 'news_password';
CREATE DATABASE news_db OWNER news_user;
GRANT ALL PRIVILEGES ON DATABASE news_db TO news_user;
\q

# Set environment variables
export PGPASSWORD=news_password
export DATABASE_URL="postgresql://news_user:news_password@localhost/news_db"
  1. Initialize Database
# Alembic migrations
alembic upgrade head

# Or direct schema creation
python -c "from app.database import create_tables; create_tables()"
  1. Configure Environment
# Create .env file (if not exists)
cp .env.example .env

# Edit .env and set required values:
# - DATABASE_URL (PostgreSQL connection)
# - OPENAI_API_KEY (for AI analysis)
# - API_HOST (default: 0.0.0.0)
# - API_PORT (default: 8000)
  1. Start Server
# Start all services (API + Worker + Scheduler)
./scripts/start-all.sh

# Or start individual services
./scripts/start-api.sh       # Web server (Port 8000)
./scripts/start-worker.sh    # Analysis worker
./scripts/start-scheduler.sh # Feed scheduler

# Check service status
./scripts/status.sh

# Stop all services
./scripts/stop-all.sh

Service Architecture:

  • API Server: Web UI + REST API (Port 8000)
  • Analysis Worker: Background AI processing (Port 9090 metrics)
  • Feed Scheduler: Automatic RSS fetching

Access the application at: http://localhost:8000 (or your configured API_HOST:API_PORT)

⚙️ Configuration

Environment Variables

# Create .env file
cp .env.example .env

# Important configurations
DATABASE_URL=postgresql://news_user:news_password@localhost/news_db
OPENAI_API_KEY=your_openai_api_key_here
ENVIRONMENT=development
LOG_LEVEL=INFO

# Auto-Analysis Configuration (Production Values)
MAX_CONCURRENT_RUNS=6
MAX_DAILY_RUNS=300
MAX_DAILY_AUTO_RUNS=1000
MAX_HOURLY_RUNS=50
AUTO_ANALYSIS_RATE_PER_SECOND=3.0
ANALYSIS_BATCH_LIMIT=200
ANALYSIS_RPS=1.5

Configuration Files

  • app/core/config.py - Main configuration
  • alembic.ini - Database migration configuration
  • scripts/ - Deployment and management scripts

📊 Usage

Web Interface

# Main dashboard
http://localhost:8000/

# Analysis cockpit
http://localhost:8000/admin/analysis

# Feed management
http://localhost:8000/admin/feeds

# Auto-analysis dashboard
http://localhost:8000/admin/auto-analysis

# Manager control center
http://localhost:8000/admin/manager

CLI Tools

# Service Management
./scripts/start-all.sh               # Start all services (API + Worker + Scheduler)
./scripts/start-api.sh               # Start API server only (Port 8000)
./scripts/start-worker.sh            # Start analysis worker only
./scripts/start-scheduler.sh         # Start feed scheduler only
./scripts/stop-all.sh                # Stop all services (graceful shutdown with timeout)
./scripts/status.sh                  # Check service status and PIDs

# Service Architecture
# - API Server: Web UI + REST API (logs/api.log)
# - Analysis Worker: Background AI processing (logs/worker.log)
# - Feed Scheduler: Automatic RSS fetching (logs/scheduler.log)
# - All services use PID files in /tmp/news-mcp-*.pid

🔌 MCP Integration (Model Context Protocol)

News MCP provides a complete Model Context Protocol implementation, allowing LLMs like Claude to directly interact with your RSS system.

Quick Start with Claude Desktop

Note: MCP Server integration is available but requires separate setup. The main application (API + Web UI) works standalone without MCP.

  1. Configure Claude Desktop

Add to your Claude Desktop config (~/Library/Application Support/Claude/claude_desktop_config.json on macOS):

{
  "mcpServers": {
    "news-mcp": {
      "command": "node",
      "args": ["/path/to/news-mcp/mcp-http-bridge.js"],
      "env": {
        "NEWS_MCP_SERVER_URL": "http://localhost:8001"
      }
    }
  }
}
  1. Restart Claude Desktop - The MCP tools will appear automatically

Available MCP Tools (20+)

Feed Management
- list_feeds          # View all RSS feeds with status
- add_feed            # Add new RSS feed
- update_feed         # Update feed configuration
- delete_feed         # Remove feed
- test_feed           # Test feed URL
- refresh_feed        # Manually trigger feed refresh
Analytics & Insights
- get_dashboard       # System overview with statistics
- latest_articles     # Get recent articles with filters
- search_articles     # Full-text article search
- trending_topics     # Analyze trending keywords
- feed_performance    # Performance metrics per feed
- export_data         # Export articles as JSON/CSV
Health & Monitoring
- system_health       # Overall system status
- feed_diagnostics    # Detailed feed health check
- error_analysis      # System error patterns
- scheduler_status    # Feed scheduler status
Database Access
- execute_query       # Safe read-only SQL queries
- table_info          # Database schema information
- quick_queries       # Predefined useful queries
Template Management
- list_templates      # View dynamic feed templates
- template_performance # Template usage statistics
- assign_template     # Assign template to feed

Example Claude Interactions

You: "Show me all my RSS feeds"
Claude: *uses list_feeds tool*
        "You have 37 active feeds. Here are the details..."

You: "What are the trending topics today?"
Claude: *uses trending_topics tool*
        "Based on 450 articles today, the top topics are..."

You: "Add feed https://techcrunch.com/feed/"
Claude: *uses add_feed tool*
        "Added TechCrunch feed successfully..."

You: "Which feeds are having problems?"
Claude: *uses feed_diagnostics tool*
        "2 feeds have issues: Feed #12 has 5 consecutive failures..."

LAN Access (Remote Claude Desktop)

For using Claude Desktop on a different machine:

  1. Configure Bridge for LAN
export NEWS_MCP_SERVER_URL=http://192.168.1.100:8001
node mcp-http-bridge.js
  1. Update Claude Desktop Config
{
  "mcpServers": {
    "news-mcp": {
      "command": "node",
      "args": ["/path/to/mcp-http-bridge.js"],
      "env": {
        "NEWS_MCP_SERVER_URL": "http://192.168.1.100:8001"
      }
    }
  }
}

MCP Architecture

Claude Desktop
    │
    ├─> mcp-http-bridge.js (stdio → HTTP)
    │
    └─> HTTP MCP Server (Port 8001)
            │
            └─> News MCP API (Port 8000)
                    │
                    └─> PostgreSQL Database

Full MCP Documentation: See for complete setup guide, tool reference, and advanced configuration.

📡 API Documentation

Core Endpoints

Feed Management
GET    /api/feeds              # List all feeds
POST   /api/feeds              # Create new feed
PUT    /api/feeds/{id}         # Update feed
DELETE /api/feeds/{id}         # Delete feed
Article Management
GET    /api/items              # List articles
GET    /api/items/{id}         # Get single article
POST   /api/items/search       # Search articles
Analysis API
POST   /api/analysis/selection        # Create article selection
GET    /api/analysis/runs             # List analysis runs
POST   /api/analysis/runs             # Start new analysis
GET    /api/analysis/runs/{id}        # Get run status
POST   /api/analysis/runs/{id}/cancel # Cancel run
Auto-Analysis API (Phase 2)
POST   /api/feeds/{id}/toggle-auto-analysis  # Enable/disable auto-analysis
GET    /api/feeds/{id}/auto-analysis-status  # Auto-analysis status
GET    /api/auto-analysis/queue              # Queue status
GET    /api/auto-analysis/history            # Auto-analysis history
System Management
GET    /api/analysis/manager/status          # System status
POST   /api/analysis/manager/emergency-stop  # Emergency stop
POST   /api/analysis/manager/resume          # Resume operations
GET    /api/analysis/manager/queue           # Queue status

Interactive API Documentation

🗄 Database Schema

Core Tables

  • feeds - RSS feed configurations
  • items - News articles/items
  • sources - News sources
  • categories - Content categories
  • analysis_runs - Analysis execution records
  • item_analysis - Analysis results per item
  • pending_auto_analysis - Auto-analysis queue

Relationships

feeds (1) ─────< (N) items
feeds (N) ─────> (M) categories
items (1) ─────< (1) item_analysis
feeds (1) ─────< (N) pending_auto_analysis

🚀 Deployment

Production Setup

  1. Environment Configuration
# Production .env
ENVIRONMENT=production
DEBUG=false
LOG_LEVEL=INFO
DATABASE_URL=postgresql://user:pass@prod-host:5432/news_db
OPENAI_API_KEY=prod_api_key
  1. Start Services
# All services in background
./scripts/start-all-background.sh

# Individual services
./scripts/start-web-server.sh
./scripts/start-worker.sh
./scripts/start-scheduler.sh
  1. Service Management
# Check status
./scripts/status.sh

# Stop all
./scripts/stop-all.sh

Systemd Services

# Copy service files
sudo cp systemd/*.service /etc/systemd/system/

# Enable and start
sudo systemctl enable news-mcp-web news-mcp-worker news-mcp-scheduler
sudo systemctl start news-mcp-web news-mcp-worker news-mcp-scheduler

# Check status
sudo systemctl status news-mcp-*

🧪 Development

Running Tests

# All tests
pytest

# Specific test file
pytest tests/test_feeds.py

# With coverage
pytest --cov=app tests/

# Integration tests
pytest tests/integration/

Code Quality

# Linting
ruff check app/

# Type checking
mypy app/

# Format code
ruff format app/

Database Migrations

# Create new migration
alembic revision --autogenerate -m "description"

# Apply migrations
alembic upgrade head

# Rollback
alembic downgrade -1

# Show current version
alembic current

🏗 Architecture

System Components

┌─────────────────────────────────────────────────┐
│                  Web Interface                   │
│         (FastAPI + HTMX + Bootstrap)            │
└─────────────────────────────────────────────────┘
                       │
┌─────────────────────────────────────────────────┐
│                  API Layer                       │
│  /api/feeds  /api/items  /api/analysis          │
└─────────────────────────────────────────────────┘
                       │
┌─────────────────────────────────────────────────┐
│                 Service Layer                    │
│  Feed Service  │  Analysis Service  │  Auto-    │
│                │  Run Manager       │  Analysis │
└─────────────────────────────────────────────────┘
                       │
┌─────────────────────────────────────────────────┐
│              Background Workers                  │
│  Analysis Worker  │  Feed Scheduler             │
└─────────────────────────────────────────────────┘
                       │
┌─────────────────────────────────────────────────┐
│               Database Layer                     │
│            PostgreSQL + SQLModel                 │
└─────────────────────────────────────────────────┘

Key Patterns

  • Repository Pattern: Data access abstraction
  • Service Layer: Business logic encapsulation
  • Feature Flags: Gradual rollout with circuit breaker
  • Queue Management: Rate-limited processing with priorities
  • HTMX Components: Progressive enhancement for UI

📈 Performance

Current Metrics

  • Feeds: 41 total (34 active, 7 error)
  • Articles: 21,339 items
  • Analysis Runs: 1,523 completed
  • Analyzed Items: 8,591 items processed
  • Concurrent Processing: 6 simultaneous runs
  • OpenAI Rate: 3.0 requests/second
  • Auto-Analysis: 12 feeds with automatic analysis enabled
  • Success Rate: >95% (production-tested)

Limits

MAX_CONCURRENT_RUNS=6       # Simultaneous analysis runs
MAX_DAILY_RUNS=300          # Manual analysis runs per day
MAX_DAILY_AUTO_RUNS=1000    # Auto-analysis runs per day
MAX_HOURLY_RUNS=50          # Runs per hour (all types)
ANALYSIS_BATCH_LIMIT=200    # Articles per batch
ANALYSIS_RPS=1.5            # OpenAI requests per second per run

🔧 Troubleshooting

Common Issues

Database connection errors:

# Check PostgreSQL status
sudo systemctl status postgresql

# Test connection
psql -h localhost -U news_user -d news_db

Worker not processing:

# Check worker logs
tail -f logs/analysis-worker.log

# Restart worker
./scripts/stop-all.sh
./scripts/start-worker.sh

Feed fetch failures:

# Check scheduler logs
tail -f logs/scheduler.log

# Manual feed fetch
curl -X POST http://localhost:8000/api/feeds/{feed_id}/fetch

📚 Documentation

🤝 Contributing

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

See for detailed guidelines.

📄 License

This project is licensed under the GNU General Public License v3.0 - see the file for details.

🙏 Acknowledgments

  • FastAPI for the excellent web framework
  • OpenAI for GPT API integration
  • PostgreSQL for robust data storage
  • The open-source community

📞 Support

Current Version: v4.2.0 - Auto-Scraping & Research Enhancements

Status: ✅ Production Ready - Stable Multi-Service Architecture (API + Worker + Scheduler)

Recent Updates:

  • ✅ Auto-scraping integration with sentiment analysis workflow
  • ✅ Perplexity Research with curated domain filtering
  • ✅ Enhanced error handling and retry logic
  • ✅ Dynamic parameter forms for research templates
  • ✅ Comprehensive testing infrastructure (Playwright E2E + pytest)