rashaideh93/Ultimate-QA-MCP-Server-AI-Driven-Automated-Testing

Ultimate QA MCP Server - AI-Driven Automated Testing

An advanced MCP (Model Context Protocol) server that converts natural language QA scenarios into fully automated test execution. This server supports comprehensive testing across UI (Playwright), APIs (HTTP), Databases (MySQL/PostgreSQL/MongoDB), and Performance monitoring.

🎯 Overview

The Ultimate QA MCP Server is a production-ready intelligent testing framework that:

• ✅ Converts plain English QA scenarios to automated tests
• ✅ Executes tests across UI, API, and Database layers
• ✅ Supports multiple LLM providers (OpenAI, Anthropic, Custom)
• ✅ Manages test context, variables, and extracted data
• ✅ Provides intelligent error recovery with retry logic
• ✅ Generates comprehensive test reports

Key Capabilities

• Natural Language QA - Describe tests in plain English
• Multi-Executor Architecture - UI, API, DB, Performance testing
• Multi-LLM Support - OpenAI GPT-4, Anthropic Claude, Custom models
• Context Management - Session tracking, variables, data extraction
• Error Recovery - Exponential backoff retry with heuristics
• Cross-Browser Testing - Chromium, Firefox, WebKit

✨ Core Features

7 MCP Tools

1. parse_scenario - Convert English QA descriptions to structured test steps
2. execute_scenario - Parse and execute complete test scenarios
3. launch_browser - Start Playwright browser instance (chromium/firefox/webkit)
4. close_browser - Clean up browser resources
5. get_context - Retrieve session variables and extracted data
6. initialize_llm - Setup LLM provider (OpenAI, Anthropic, Custom)
7. get_status - Server status and active browsers

11 Core Modules

• LLM Integration - Multi-model abstraction (OpenAI, Anthropic, Custom)
• Natural Language Parser - English to JSON step conversion
• UI Executor - Playwright-based browser automation (17 action types)
• API Executor - HTTP testing with auth, validation, extraction
• Database Executor - MySQL, PostgreSQL, MongoDB support
• Context Manager - Session tracking, variables, data extraction
• Action Router - Orchestrates multi-executor workflows
• Error Handler - Retry logic with exponential backoff
• Configuration - Environment-based setup
• Type System - Full TypeScript type safety
• MCP Server - Protocol implementation

📦 Installation

Prerequisites

• Node.js 17 or higher
• npm 8 or higher
• TypeScript 5.3.3+
• Playwright 1.48.0+

Quick Start

1. Clone the repository:

   git clone https://github.com/rashaideh93/Ultimate-QA-MCP-Server-AI-Driven-Automated-Testing.git
   cd test
   

2. Install dependencies:

   npm install
   

3. Configure environment:

   cp .env.example .env
   # Edit .env with your settings:
   # LLM_PROVIDER=openai (or anthropic, custom)
   # LLM_MODEL_NAME=gpt-4
   # LLM_API_KEY=your-api-key
   

4. Build the project:

   npm run build
   

5. Start the server:

   npm start
   

🚀 Usage

Natural Language Example

Simply describe your test in plain English:

"Navigate to https://example.com/login, 
fill email with admin@test.com, 
fill password with Pass123!, 
click the Login button, 
wait 2 seconds for page load, 
verify Welcome message appears, 
take a screenshot"

The server will automatically:

1. ✅ Parse the scenario into structured steps
2. ✅ Launch a Playwright browser
3. ✅ Execute each action in sequence
4. ✅ Capture screenshots and logs
5. ✅ Return detailed results

Running Tests

# Build the project
npm run build

# Start the server
npm start

# Run tests (in another terminal)
npm test

# Watch mode for development
npm run dev

Integration with Claude/Copilot

Claude Desktop

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "ultimate-qa": {
      "command": "node",
      "args": ["/path/to/test/build/index.js"],
      "env": {
        "LLM_PROVIDER": "openai",
        "LLM_MODEL_NAME": "gpt-4",
        "LLM_API_KEY": "sk-..."
      }
    }
  }
}

VS Code Copilot

1. Press Cmd+Shift+P (Mac) or Ctrl+Shift+P (Windows/Linux)
2. Select MCP: Add server...
3. Configure with node ./build/index.js

Other MCP Clients

Configure stdio transport connecting to node ./build/index.js

📝 Usage Examples

Example 1: Simple Login Test

Natural Language Input:

"Navigate to example.com/login, 
fill email with test@example.com, 
fill password with TestPass123, 
click Login button, 
verify Dashboard page appears"

Automatic Execution:

• ✅ Parse scenario
• ✅ Launch browser
• ✅ Fill form fields
• ✅ Click button
• ✅ Verify page state
• ✅ Generate report

Example 2: API + UI Combined Test

Scenario:

"Create user via POST /api/users with name: John, email: john@test.com
Extract userId from response
Navigate to /users/dashboard
Search for user by email
Verify user appears in list"

Result:

• ✅ API call execution
• ✅ Data extraction
• ✅ UI navigation
• ✅ Element verification
• ✅ Test report

Example 3: Database Validation

Scenario:

"Add product to cart via UI
Complete purchase
Verify order status in database
Check inventory was updated"

Result:

• ✅ UI actions
• ✅ API calls
• ✅ Database queries
• ✅ Cross-layer validation

Full Examples

See for:

• 7+ detailed real-world scenarios
• Multi-tab testing
• Data-driven testing
• Conditional flows
• Performance monitoring
• CI/CD integration examples

🏗️ Architecture

Layered Design

┌─────────────────────────────────────────┐
│  MCP Server (7 Tools)                   │
├─────────────────────────────────────────┤
│  Application Layer                      │
│  - Action Router                        │
│  - Context Manager                      │
│  - Error Handler                        │
├─────────────────────────────────────────┤
│  Execution Layer (Multi-Executor)       │
│  - UI Executor (Playwright)             │
│  - API Executor (HTTP)                  │
│  - DB Executor (MySQL/Postgres/MongoDB) │
│  - Performance Monitor                  │
├─────────────────────────────────────────┤
│  Intelligence Layer                     │
│  - LLM Provider (OpenAI/Anthropic)     │
│  - Natural Language Parser              │
├─────────────────────────────────────────┤
│  Infrastructure                         │
│  - Playwright                           │
│  - Axios HTTP Client                    │
│  - Database Drivers                     │
└─────────────────────────────────────────┘

Module Breakdown

Error Handling & Retry Logic

• Exponential Backoff - Automatic retry with increasing delays
• Selector Heuristics - Multiple fallback strategies
• Error Recovery - Graceful degradation
• Comprehensive Logging - Full execution trace

Testing

npm test                 # Run all tests
npm run test:watch      # Watch mode
npm run test:coverage   # Coverage report
npm run build           # Build TypeScript

For detailed testing guide, see the test files in src/__tests__/

📁 Project Structure

ultimate-qa-mcp-server/
├── src/                                  # Source code
│   ├── index.ts                         # MCP server entry
│   ├── config.ts                        # Configuration
│   ├── types.ts                         # Type definitions
│   ├── llm.ts                           # LLM abstraction
│   ├── nlParser.ts                      # NL parser
│   ├── contextManager.ts                # Session manager
│   ├── uiExecutor.ts                    # Playwright
│   ├── apiExecutor.ts                   # HTTP testing
│   ├── dbExecutor.ts                    # Database
│   ├── actionRouter.ts                  # Orchestration
│   ├── errorHandler.ts                  # Error handling
│   └── __tests__/                       # Test suites
│
├── build/                                # Compiled JS (auto-generated)
│
├── 📚 Documentation
│   ├── README.md                        # This file
│   ├── ULTIMATE_QA_GUIDE.md             # Complete user guide (2000+ lines)
│   ├── USAGE_EXAMPLES.md                # Real-world examples
│   └── API_REFERENCE.md                 # MCP tool reference
│
├── ⚙️ Configuration
│   ├── package.json                     # Dependencies
│   ├── tsconfig.json                    # TypeScript config
│   ├── jest.config.cjs                  # Test config
│   ├── .env.example                     # Environment template
│   └── qodana.yaml                      # Code analysis
│
└── 📋 Version Control
    ├── .git/                            # Git repository
    ├── .gitignore                       # Git exclusions
    └── .npmignore                       # NPM exclusions

Building

npm run build    # Compile TypeScript to JavaScript
npm run dev      # Development mode with watch
npm start        # Start the server

Testing

# Run all tests
npm test

# Test specific files
npm test -- actionRouter

# Use MCP Inspector for interactive testing
npx @modelcontextprotocol/inspector node ./build/index.js

The MCP Inspector provides:

• Interactive tool testing
• Schema inspection
• Output visualization
• Debugging interface

🤖 AI Integration

Claude Desktop

Configure in claude_desktop_config.json:

{
  "mcpServers": {
    "ultimate-qa": {
      "command": "node",
      "args": ["/path/to/test/build/index.js"],
      "env": {
        "LLM_PROVIDER": "openai",
        "LLM_MODEL_NAME": "gpt-4",
        "LLM_API_KEY": "sk-..."
      }
    }
  }
}

GitHub Copilot in VS Code

1. Install MCP extension for VS Code
2. Configure with node ./build/index.js
3. Use natural language for test scenarios

Other MCP Clients

Compatible with:

• LM Studio - Local LLM deployment
• Cursor - Code editor with AI
• Windsurf - AI-powered IDE
• Cline - Command-line interface
• Custom clients - Any MCP-compatible application

Supported LLM Models

OpenAI:

• GPT-4 (recommended)
• GPT-3.5-turbo

Anthropic:

• Claude 3 Opus
• Claude 3 Sonnet
• Claude 2

Custom:

• On-premise OpenAI-compatible API
• Self-hosted models

⚙️ Configuration

Environment Variables

Create .env file with:

# LLM Configuration
LLM_PROVIDER=openai                    # openai | anthropic | custom
LLM_MODEL_NAME=gpt-4                   # Model to use
LLM_API_KEY=sk-...                     # Your API key
LLM_BASE_URL=                          # For custom models
LLM_TIMEOUT=30000                      # Request timeout (ms)

# Database Configuration (Optional)
DB_TYPE=mysql                          # mysql | postgres | mongodb
DB_HOST=localhost
DB_PORT=3306
DB_USERNAME=user
DB_PASSWORD=password
DB_NAME=testdb

# Server Configuration
SCREENSHOTS_ENABLED=true               # Capture screenshots
SCREENSHOTS_PATH=./screenshots
REPORTS_ENABLED=true
REPORTS_PATH=./qa-reports
REPORT_FORMATS=json,markdown,html

# Performance
PERFORMANCE_CHECKS=true
PAGE_LOAD_THRESHOLD_MS=3000
API_RESPONSE_THRESHOLD_MS=1000

# Retries
MAX_RETRY_ATTEMPTS=3
RETRY_DELAY_MS=1000
RETRY_BACKOFF_MULTIPLIER=1.5

Browser Options

• browserType - "chromium" (default), "firefox", "webkit"
• headless - true (default) or false for headed mode
• url - Initial URL to navigate to

Database Support

• MySQL - Version 5.7+, Driver: mysql2
• PostgreSQL - Version 10+, Driver: pg
• MongoDB - Version 4.0+, Driver: mongodb

🔧 Troubleshooting

Server Won't Start

• ✅ Ensure Node.js 17+ installed: node --version
• ✅ Check dependencies: npm install
• ✅ Verify build: npm run build
• ✅ Check .env file exists with LLM_API_KEY

Connection Issues

• ✅ Verify absolute path in MCP config is correct
• ✅ Confirm build/index.js exists
• ✅ Check stderr output: npm start 2>&1
• ✅ Test MCP connection: npx @modelcontextprotocol/inspector node ./build/index.js

Test Failures

• ✅ Verify element selectors are correct (CSS or XPath)
• ✅ Check elements exist before interaction
• ✅ Review Playwright docs for specific elements
• ✅ Enable screenshots for debugging: SCREENSHOTS_ENABLED=true
• ✅ Check logs in ./qa-reports directory

Performance Issues

• ✅ Use headless mode (default): faster execution
• ✅ Close browsers when done: close_browser
• ✅ Use appropriate waits: avoid excessive delays
• ✅ Enable connection pooling for databases
• ✅ Monitor network requests in DevTools

Database Errors

• ✅ Verify DB connection string in .env
• ✅ Check database is running and accessible
• ✅ Ensure user has appropriate permissions
• ✅ Test connection: mysql -u user -p -h host

For more help, see

📖 MCP Tools Reference

1. initialize_llm

Initialize LLM provider for natural language parsing

{
  "provider": "openai|anthropic|custom",
  "modelName": "gpt-4|claude-3-opus|...",
  "apiKey": "your-api-key",
  "baseUrl": "optional-for-custom"
}

2. parse_scenario

Convert English QA scenario to structured steps

{
  "scenarioText": "Navigate to login, fill email with test@test.com, click Login",
  "extractVariables": true
}

3. execute_scenario

Parse and execute complete scenario

{
  "browserId": "browser-1",
  "scenarioText": "Your test scenario...",
  "generateReport": true
}

4. launch_browser

Start Playwright browser instance

{
  "browserType": "chromium|firefox|webkit",
  "headless": true,
  "url": "https://example.com"
}

5. close_browser

Clean up browser resources

{
  "browserId": "browser-1"
}

6. get_context

Retrieve session variables and data

{
  "browserId": "browser-1"
}

7. get_status

Server status and active browsers

{}

For detailed tool documentation, see

📦 Dependencies

Core

• @modelcontextprotocol/sdk (2.0+) - MCP protocol
• playwright (1.48+) - Browser automation
• axios (1.6+) - HTTP client
• zod (3.22+) - Type validation
• dotenv (16.3+) - Environment config

Databases

• mysql2 (3.6+) - MySQL driver
• pg (8.11+) - PostgreSQL driver
• mongodb (6.3+) - MongoDB driver

Development

• typescript (5.3+) - Language support
• jest (29+) - Testing framework
• @types/node - Type definitions

📊 Project Statistics

📄 Documentation

Complete documentation available:

1. (2000+ lines)

   • Complete user guide
   • Detailed examples
   • Best practices
   • Advanced features

2. • 7+ real-world scenarios
   • Multi-system testing
   • Performance monitoring
   • CI/CD integration

3. • MCP tool documentation
   • Parameter details
   • Response formats

🔐 Security & Best Practices

✅ 100% TypeScript - Full type safety
✅ Input Validation - Zod schema validation
✅ Error Handling - Comprehensive error recovery
✅ Secrets Management - Environment variable based
✅ Connection Pooling - Database optimization
✅ Retry Logic - Exponential backoff

📜 License

MIT License - See LICENSE file for details

🤝 Contributing

Contributions welcome! Please:

1. Fork the repository
2. Create a feature branch
3. Submit a pull request
4. Include test coverage

📚 Resources

• MCP Documentation
• Playwright Documentation
• OpenAI API
• Anthropic Claude

💬 Support

For issues and questions:

1. Check Documentation - See guides above
2. Review Examples - See USAGE_EXAMPLES.md
3. Check Logs - Enable debug logging
4. Run Tests - npm test
5. Open Issue - GitHub repository

Version: 2.0.0