rajeshpenki/databricks-jobs-mcp-server

Databricks Jobs MCP Server

A Model Context Protocol (MCP) server for interacting with the Databricks Jobs API. This server provides tools to manage and monitor Databricks jobs through MCP.

Features

• List jobs with pagination and filtering
• Get detailed job information
• Run jobs with custom parameters
• List job runs with various filters
• Get run details and output
• Cancel active runs
• Delete jobs

Available Transports

This server supports two transport methods:

1. Stdio Transport (index.ts)

Standard MCP server using stdio transport - suitable for Claude Desktop integration.

2. Streamable HTTP Transport (index-http.ts)

HTTP-based server with streaming capabilities - suitable for web applications and HTTP clients.

Setup

Prerequisites

1. Node.js 18+
2. Databricks workspace and personal access token

Environment Variables

DATABRICKS_HOST=https://your-workspace.azuredatabricks.net
DATABRICKS_TOKEN=your-personal-access-token
PORT=3000  # Only for HTTP transport

Installation

npm install

Build

# Build both versions
npm run build

# Or build individually
npm run build:http  # HTTP version only

Usage

Stdio Transport (for Claude Desktop)

1. Build and run:

   npm run build
   npm start
   

2. Claude Desktop Configuration: Add to ~/.config/claude/claude_desktop_config.json:

   {
     "mcpServers": {
       "databricks-jobs": {
         "command": "node",
         "args": ["/path/to/your/dist/index.js"],
         "env": {
           "DATABRICKS_HOST": "https://your-workspace.azuredatabricks.net",
           "DATABRICKS_TOKEN": "your-token-here"
         }
       }
     }
   }
   

HTTP Transport (for web apps)

1. Run the HTTP server:

   npm run dev:http  # Development
   # or
   npm run build && npm run start:http  # Production
   

2. Endpoints:

   • POST http://localhost:3000/mcp - Main MCP endpoint
   • GET http://localhost:3000/sse - Server-Sent Events endpoint
   • GET http://localhost:3000/health - Health check
   • DELETE http://localhost:3000/mcp/:sessionId - Close session

3. Features:

   • Session management with UUIDs
   • CORS enabled
   • Streaming support via SSE
   • Health monitoring

Available Tools

list_jobs

List all jobs in the workspace with optional filtering and pagination.

Parameters:

• limit (number): Maximum jobs to return (default: 25, max: 25)
• offset (number): Pagination offset (default: 0)
• expand_tasks (boolean): Include task details (default: false)
• name (string): Filter by job name

get_job

Get detailed information about a specific job.

Parameters:

• job_id (number, required): Job identifier

run_job_now

Trigger a new run of an existing job with optional parameter overrides.

Parameters:

• job_id (number, required): Job identifier
• jar_params (array): JAR task parameters
• notebook_params (object): Notebook task parameters
• python_params (array): Python task parameters
• spark_submit_params (array): Spark submit parameters

list_runs

List job runs with filtering and pagination options.

Parameters:

• job_id (number): Filter by job ID
• active_only (boolean): Show only active runs
• completed_only (boolean): Show only completed runs
• limit (number): Maximum runs to return
• offset (number): Pagination offset
• start_time_from (number): Filter by start time (Unix timestamp)
• start_time_to (number): Filter by end time (Unix timestamp)

get_run

Get detailed information about a specific job run.

Parameters:

• run_id (number, required): Run identifier
• include_history (boolean): Include repair history

get_run_output

Get the output of a completed job run.

Parameters:

• run_id (number, required): Run identifier

cancel_run

Cancel an active job run.

Parameters:

• run_id (number, required): Run identifier

delete_job

Delete a job (cannot be undone).

Parameters:

• job_id (number, required): Job identifier

Docker Deployment

Quick Start with Docker

1. Setup environment:

   cp .env.example .env
   # Edit .env with your Databricks credentials
   

2. Build and run:

   ./docker-manage.sh build
   ./docker-manage.sh up
   

3. Access the server:

   • Main server: http://localhost:3000
   • Health check: http://localhost:3000/health

Docker Management Script

The docker-manage.sh script provides easy container management:

# Build the Docker image
./docker-manage.sh build

# Start services (development mode)
./docker-manage.sh up

# Start with nginx proxy (production mode)
./docker-manage.sh up-prod

# Stop services
./docker-manage.sh down

# View logs
./docker-manage.sh logs -f

# Check health
./docker-manage.sh health

# Open shell in container
./docker-manage.sh shell

# Clean up everything
./docker-manage.sh clean

Production Deployment

For production deployment with nginx reverse proxy:

# Start with production profile
./docker-manage.sh up-prod

This includes:

• Nginx reverse proxy with SSL support
• Rate limiting
• Security headers
• Proper SSE handling
• Health checks

Development

Run in development mode:

# Stdio version
npm run dev

# HTTP version  
npm run dev:http

# Or with Docker
./docker-manage.sh up

Test the HTTP server:

# Health check
curl http://localhost:3000/health

# Example MCP request
curl -X POST http://localhost:3000/mcp \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "tools/list",
    "params": {}
  }'

Architecture

• Transport Layer: Supports both stdio and HTTP transports
• Session Management: HTTP version includes session tracking
• Error Handling: Comprehensive error handling with Databricks API error details
• Type Safety: Full TypeScript implementation with strict typing

Dependencies

• @modelcontextprotocol/sdk: MCP SDK for server implementation
• axios: HTTP client for Databricks API calls
• express: Web framework (HTTP version only)

License

MIT# databricks-jobs-mcp-server