taskcrew/cua-mcp-server

CUA MCP Server

An agentic Model Context Protocol (MCP) server for CUA Cloud - delegate desktop automation tasks to an autonomous vision-based agent. Images never leave the server; only text summaries are returned.

Production URL: https://cua-mcp-server.vercel.app/mcp

What is CUA?

CUA (Computer Use Agent) provides cloud-based virtual machine sandboxes that AI agents can control. This MCP server exposes CUA's capabilities through a clean task-delegation API:

• Create and manage VMs (Linux, Windows, macOS)
• Delegate tasks - "Open Chrome and navigate to google.com"
• Get text summaries - No images in your context window
• Query screen state - Vision-based descriptions without taking action

Architecture

Claude Code (Orchestrator)
    │
    │ run_task("Open Chrome and go to google.com")
    ▼
┌─────────────────────────────────────────────────────────────┐
│  CUA MCP Server (Agentic)                                   │
│  ┌───────────────────────────────────────────────────────┐  │
│  │  Internal Agent Loop                                  │  │
│  │  1. screenshot() → CUA sandbox                        │  │
│  │  2. screenshot → Claude API (computer_use tool)       │  │
│  │  3. Claude returns: click(x,y) / type("text") / done  │  │
│  │  4. Execute action on sandbox                         │  │
│  │  5. Loop until complete                               │  │
│  └───────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────┘
    │
    ▼
{ success: true, summary: "Opened Chrome...", steps_taken: 5 }
(TEXT ONLY - no images)

Project Structure

api/mcp.ts                     # MCP protocol handler
lib/
├── agent/                     # Modular agent architecture
│   ├── index.ts               # Public exports
│   ├── types.ts               # Type definitions
│   ├── config.ts              # Model configurations
│   ├── validation.ts          # Coordinate validation helpers
│   ├── execute.ts             # Main agent loop
│   ├── describe.ts            # Screen description
│   ├── progress.ts            # Progress tracking
│   ├── utils.ts               # Utilities (sleep, generateTaskId)
│   └── actions/               # Action handler registry (16 handlers)
├── cua-client.ts              # CUA Cloud API client
└── tool-schemas.ts            # MCP tool definitions

Available Tools (9 total)

Sandbox Management (5 tools)

Note: Create and delete sandboxes via the CUA Dashboard - the Cloud API doesn't expose these operations.

Agentic Tools (4 tools)

Quick Start

1. Get a CUA API Key

1. Go to cua.ai/signin
2. Navigate to Dashboard > API Keys > New API Key
3. Copy your API key (starts with sk_cua-api01_...)

2. Configure Claude Code

Add to your ~/.claude.json:

{
  "mcpServers": {
    "cua": {
      "command": "npx",
      "args": ["-y", "mcp-remote", "https://cua-mcp-server.vercel.app/mcp"]
    }
  }
}

3. Use with Claude Code

You: "List my CUA sandboxes"
Claude: [Uses list_sandboxes tool]

You: "Start my-sandbox"
Claude: [Uses start_sandbox tool]

You: "Open Firefox and go to google.com on my-sandbox"
Claude: [Uses run_task with task="Open Firefox and navigate to google.com"]
→ Returns: { success: true, summary: "Opened Firefox, navigated to google.com", steps_taken: 4 }

You: "What's currently on the screen?"
Claude: [Uses describe_screen tool]
→ Returns: { description: "Firefox browser showing Google homepage with search box..." }

Usage Examples

Automate a Web Task

You: "On my-sandbox, open Chrome, go to github.com, and search for 'mcp server'"

Claude uses run_task:
- task: "Open Chrome browser, navigate to github.com, find the search box, type 'mcp server' and press Enter"
- Returns summary of what happened (no screenshots in your context)

Check Screen State

You: "What's on the screen right now?"

Claude uses describe_screen:
- focus: "ui" (or "text" or "full")
- Returns text description of UI elements, buttons, text content

Ask Specific Questions

You: "Is there a login button visible?"

Claude uses describe_screen:
- question: "Is there a login button visible?"
- Returns: "Yes, there is a blue 'Sign In' button in the top right corner..."

Self-Hosting

Prerequisites

• Vercel account with Pro plan (for 800s function timeout)
• Vercel Blob storage
• Anthropic API key

Deploy Your Own Instance

# Clone the repository
git clone https://github.com/anthropics/cua-mcp-server.git
cd cua-mcp-server

# Install dependencies
npm install

# Deploy to Vercel
vercel --prod

Environment Variables

Setting Up Vercel Blob

1. Go to your Vercel project dashboard
2. Navigate to Storage → Create → Blob
3. The BLOB_READ_WRITE_TOKEN will be automatically added

Pass API Key Per-Request

If you don't want to store the CUA API key on the server:

{
  "mcpServers": {
    "cua": {
      "command": "npx",
      "args": [
        "-y", "mcp-remote",
        "https://your-deployment.vercel.app/mcp",
        "--header", "X-CUA-API-Key: sk_cua-api01_your-key-here"
      ]
    }
  }
}

API Reference

MCP Endpoint

URL: POST /mcp

Content-Type: application/json

Example: Run Task

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "id": 1,
  "params": {
    "name": "run_task",
    "arguments": {
      "sandbox_name": "s-linux-abc123",
      "task": "Open Firefox and navigate to google.com",
      "max_steps": 30,
      "timeout_seconds": 120
    }
  }
}

Response:

{
  "jsonrpc": "2.0",
  "id": 1,
  "result": {
    "content": [{
      "type": "text",
      "text": "{\"task_id\":\"task_123...\",\"success\":true,\"summary\":\"Opened Firefox, navigated to google.com\",\"steps_taken\":4,\"duration_ms\":8500}"
    }]
  }
}

Example: Describe Screen

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "id": 2,
  "params": {
    "name": "describe_screen",
    "arguments": {
      "sandbox_name": "s-linux-abc123",
      "focus": "ui",
      "question": "Is there a search box visible?"
    }
  }
}

Model Support

Supported Computer Actions

The agent can perform the following actions autonomously:

UI Actions:

• screenshot - Capture current screen
• left_click, right_click, double_click, triple_click, middle_click - Mouse clicks at coordinates
• mouse_move - Move cursor to coordinates
• left_click_drag - Click and drag from start to end coordinates
• left_mouse_down, left_mouse_up - Press/release mouse button
• scroll - Scroll up/down/left/right
• wait - Pause execution
• zoom - View specific screen region at full resolution (Opus 4.5 only, defaults to center if no coordinate)

Keyboard:

• type - Type text
• key - Press key or key combination (e.g., "ctrl+c")
• hold_key - Hold a modifier key down (auto-releases after next action)

Constraints

Sandbox Types

Regions

• north-america - US East (lowest latency for US users)
• europe - EU West
• asia - Asia Pacific

Troubleshooting

"CUA API key required"

Set CUA_API_KEY environment variable in Vercel or pass via X-CUA-API-Key header.

"ANTHROPIC_API_KEY not configured"

The server needs an Anthropic API key for vision processing. Add it to your Vercel environment variables.

Task times out

• Default timeout is 750 seconds
• Reduce task complexity or break into smaller steps
• Check if sandbox is responsive with describe_screen

Task exceeds max steps

• Default is 100 steps (max 100)
• Break complex tasks into smaller subtasks
• Use more specific task descriptions

Resources

• CUA Documentation
• CUA Cloud Dashboard
• Anthropic Computer Use
• MCP Protocol Specification

License

MIT