kinjal-1007/confluent-mcp-server

Kafka MCP Server

A Model Context Protocol (MCP) server that provides tools to interact with Apache Kafka clusters. This server allows Claude to manage topics, produce messages, and consume messages from your Kafka infrastructure.

Features

• Topic Management: List, describe, and create Kafka topics
• Message Production: Send messages to any Kafka topic
• Message Consumption: Read messages from topics with configurable consumer groups
• Cluster Inspection: Get detailed information about topic partitions and replication

Prerequisites

• Python 3.9+
• uv (The ultra-fast Python package and project manager)
• Claude Desktop App
• Access to a Kafka cluster (Confluent Cloud or self-hosted)
• Kafka connection credentials

Installation & Setup

1. Navigate to the project directory:

   cd /path/to/this/folder
   

2. Initialize the project and create a virtual environment:

   uv init kafka-mcp
   uv venv
   

3. Install the dependencies from the provided requirements.txt:

   uv add -r requirements.txt
   

4. Configure Kafka connection: Edit the main.py file and replace the Kafka configuration with your actual credentials:

   # Load Kafka configuration (use your client.properties values)
   KAFKA_CONFIG = {
       "bootstrap.servers": "your-bootstrap-server:9092",
       "security.protocol": "SASL_SSL",
       "sasl.mechanisms": "PLAIN",
       "sasl.username": "your-username",
       "sasl.password": "your-password",
       "client.id": "mcp-server",
       "session.timeout.ms": 180000,
   }
   

   Get these values from your Confluent Cloud dashboard or Kafka cluster configuration.

Running the Server

To test and run the MCP server locally, use:

uv run --with "mcp[cli]" mcp run main.py

If it runs without errors, you are ready to connect it to Claude.

Connecting to Claude Desktop

1. Open Claude Desktop.
2. Go to Settings -> Developer -> Edit MCP Server Configuration. This will open the claude_desktop_config.json file.
3. Add a new configuration for this server. Replace the paths with the absolute paths on your system.

{
  "mcpServers": {
    "kafka-mcp": {
      "command": "/path/to/your/uv",
      "args": [
        "run",
        "--directory",
        "/path/to/your/kafka-mcp",
        "python",
        "main.py"
      ]
    }
  }
}

• command: The absolute path to your uv installation. Find it by running which uv in your terminal.
• args[3] (--directory): The absolute path to this project folder.

4. Save the file and restart Claude Desktop.

Usage Examples

Once configured, you can ask Claude to interact with your Kafka cluster:

• "Check my Kafka cluster and describe the topics."
• "Create a new topic called mcp-test-topic."
• "Produce a message to mcp-test-topic with the content 'test message'."
• "Consume all messages from the mcp-test-topic."
• "Describe the user-database-topic and show its partition information."

Available Tools

list_topics()

Lists all topics in the Kafka cluster with their configuration details.

describe_topic(topic: str)

Provides detailed information about a specific topic including partition distribution and replica placement.

create_topic(topic: str, num_partitions: int = 1, replication_factor: int = 3)

Creates a new Kafka topic with specified partition count and replication factor.

produce_message(topic: str, key: str = None, value: str = None)

Produces a message to the specified Kafka topic with optional key.

consume_messages(topic: str, group_id: str = "mcp-consumer", max_messages: int = 5)

Consumes messages from a topic using the specified consumer group.

Example Workflow

1. Cluster Inspection: Check what topics exist in your cluster
2. Topic Creation: Create new topics for testing or production use
3. Message Production: Send test messages or production data
4. Message Consumption: Verify messages are being processed correctly
5. Topic Management: Monitor and manage topic configurations

Troubleshooting

• Connection Issues: Verify your Kafka credentials and network connectivity
• Topic Errors: Ensure you have proper permissions to create/manage topics
• Consumer Issues: Check that consumer groups are properly configured
• Timeout Errors: Increase timeout values in the configuration if needed

Security Notes

• Keep your Kafka credentials secure and never commit them to version control
• Use appropriate ACLs (Access Control Lists) in your Kafka cluster
• Consider using environment variables for sensitive configuration data
• Regularly rotate credentials for production environments

Example Output

When you ask to list topics, Claude will return:

• Complete list of all topics in the cluster
• Partition counts and replication factors
• Topic organization and naming patterns
• Health status based on leader distribution

Note: This tool provides direct access to your Kafka infrastructure. Use with caution in production environments and ensure proper access controls are in place.