BERDataLakehouse/datalake-mcp-server

BERDL Datalake MCP Server

A FastAPI-based service that enables AI assistants to interact with Delta Lake tables stored in MinIO through Spark, implementing the Model Context Protocol (MCP) for natural language data operations.

⚠️ Important Warning:

This service allows arbitrary read-oriented queries to be executed against Delta Lake tables. Query results will be sent to the model host server, unless you are hosting your model locally.

❌ Additionally, this service is NOT approved for deployment to any production environment, including CI, until explicit approval is granted by KBase leadership. Use strictly for local development or evaluation purposes only.

Documentation

Guides

- Complete documentation for setting up and using the MCP server:

• - Bring the local service up and running
• - Set up local test data
• - Direct API usage examples
• - Configure and use with MCP Host tools

- Using the BERDL MCP server with Claude Code CLI:

• Connect Claude Code to the BERDL data lake
• Query your Delta Lake tables using natural language
• Step-by-step setup and troubleshooting
• Best practices and example queries

Quick Start

Option A: Using Pre-Built Images (Recommended)

1. Clone the repository:

   git clone https://github.com/BERDataLakehouse/datalake-mcp-server.git
   cd datalake-mcp-server
   

2. Edit docker-compose.yaml:

   • Uncomment all image: and platform: lines
   • Comment out all build: sections

3. Start the services:

   docker compose up -d
   

Option B: Building from Source (For Developers)

1. Clone required repositories:

   # Clone at the same directory level as datalake-mcp-server
   cd ..
   git clone https://github.com/BERDataLakehouse/spark_notebook_base.git
   git clone https://github.com/BERDataLakehouse/kube_spark_manager_image.git
   git clone https://github.com/BERDataLakehouse/hive_metastore.git
   cd datalake-mcp-server
   

2. Build base images:

   cd ../spark_notebook_base
   docker build -t spark_notebook_base:local .
   cd ../datalake-mcp-server
   

3. Ensure docker-compose.yaml has build: sections uncommented (default)

4. Start the services:

   docker compose up -d --build
   

Access the Services

• MCP Server API: http://localhost:8000/apis/mcp/docs
• MCP Server Root: http://localhost:8000/docs
• MinIO Console: http://localhost:9003 (credentials: minio/minio123)
• Spark Master UI: http://localhost:8090
• PostgreSQL: localhost:5432 (credentials: hive/hivepassword)

Note: The MCP server is mounted at /apis/mcp by default. Set SERVICE_ROOT_PATH="" environment variable to serve at root.

Authentication

The service uses KBase authentication with role-based access control:

Environment Variables

Testing

# Install dependencies (only required on first run or when the uv.lock file changes)
uv sync --locked

# Run tests
PYTHONPATH=. uv run pytest tests

# Run with coverage
PYTHONPATH=. uv run pytest --cov=src tests/