erichowens/metal-shader-mcp

Metal Shader MCP - Claude's Shader Development Playground

A complete system for AI-assisted Metal shader development where Claude can write, modify, and visually iterate on shaders in real-time.

Project overview

New: Optional Core ML post-processing on exported frames. See section below for configuration.

Note: The legacy file-bridge (writing commands to Resources/communication) is being deprecated in favor of a strict MCP client. See Deprecation notice below. Metal Shader MCP is a macOS SwiftUI + Metal playground with a disciplined workflow for shader iteration, visual evidence, and CI. An MCP layer is planned to let AI assistants interact with the app (compile, preview, snapshot), but today the primary entry point is the macOS app you can compile and run locally.

Current state (as of 2025-10-02)

• macOS app (SwiftUI + Metal) builds and runs locally and in CI.
• Live shader editing with a simple renderer and screenshot/export helpers.
• Session Browser (History tab) captures per-session snapshots (code + image + meta).
• CI enforces: build, tests, visual-evidence capture, WARP workflow compliance, UI smoke (tab selection/status), docs checks.
• Branch protection & merge policy: All CI checks must pass before merging to main. Direct pushes to main are rejected. Changes must go through PRs with green checks. Overriding requires explicit user decision and is discouraged.
• Single-flight PR policy: One active (non-draft) core PR at a time; others remain draft until the active one merges.

Roadmap

• MCP server integration to drive the macOS app (compile/preview/snapshot) from tools.
• Visual regression tests with baselines across multiple resolutions.
• Shader library with names, descriptions, and persistent metadata (see WARP/metadata rules).
• Export pipelines (PNG/video) with parameter sweeps and performance profiling.
• Xcode project or Swift Package targets for automatic file discovery in CI.

Quick start (macOS app)

• Prereqs: macOS with Xcode (latest stable), Metal-capable device.
• Build and run locally:

swiftc -o MetalShaderStudio \
  ShaderPlayground.swift AppShellView.swift HistoryTabView.swift SessionRecorder.swift \
  Sources/MetalShaderCore/MCPClient.swift \
  -framework SwiftUI -framework MetalKit -framework AppKit -framework UniformTypeIdentifiers \
  -parse-as-library

./MetalShaderStudio --tab history

Features

• Live Compilation: Compile Metal shaders in real-time with error reporting
• Hot Reload: Automatic recompilation on file changes
• Performance Profiling: Measure FPS, GPU/CPU time, and memory usage
• Preview Engine: Real-time shader preview with WebSocket updates
• Shader Templates: Built-in shader examples and effects
• MCP Integration: Seamless integration with AI assistants

Installation

• macOS app (current): see Quick start above.
• MCP server (planned): the existing npm scripts are placeholders for the future MCP server. They are not required to run the macOS app today.

# Planned (MCP server), not required for the macOS app today
npm install
npm run build

Headless renderer (for dataset and CI)

Render a shader to PNG without launching the app:

swift run ShaderRenderCLI --shader-file shaders/plasma_fractal.metal --out data/sample.png --width 256 --height 256 --time 0.0

Core ML post-processing (optional)

• Configure at Resources/communication/coreml_config.json (template provided).
• Provide a model at Resources/models/YourModel.mlmodelc or .mlmodel.
• When present and valid, exports will be passed through the model before saving.

Config fields:

• modelPath: path to model (e.g., Resources/models/StyleTransfer.mlmodelc)
• inputName: image input feature name (e.g., image)
• outputName: output image feature name (e.g., stylizedImage)
• width, height: model input dimensions

Note: UI remains unchanged; this affects exported frames and session snapshots only.

Usage

Start MCP Server

npm start

Development Mode

npm run dev

Available MCP Tools

1. compile_shader: Compile Metal shader code

   • Supports AIR, metallib, and SPIRV targets
   • Optimization options
   • Error and warning reporting

2. preview_shader: Generate preview images

   • Real-time rendering
   • Customizable resolution
   • Touch interaction support

3. update_uniforms: Update shader parameters

   • Time, touch position, resolution
   • Custom uniform values

4. profile_performance: Performance metrics

   • FPS measurement
   • GPU/CPU time tracking
   • Memory usage monitoring

5. hot_reload: File watching

   • Automatic recompilation
   • WebSocket notifications
   • Development dashboard

6. validate_shader: Syntax validation

   • Error detection
   • Performance suggestions
   • Code analysis

Shader Development

Example: Kaleidoscope Effect

#include <metal_stdlib>
using namespace metal;

fragment float4 kaleidoscopeFragment(
    VertexOut in [[stage_in]],
    texture2d<float> texture [[texture(0)]],
    constant Uniforms& uniforms [[buffer(0)]]
) {
    // Kaleidoscope transformation
    float2 uv = kaleidoscope(in.texCoord, 6, uniforms.time);
    
    // Generate RGBY color blocks
    float4 color = generateColorBlock(uv, uniforms.blockSize);
    
    return color;
}

Performance Requirements

• Target: 60fps on modern devices
• Memory: Efficient memory usage
• Compilation: <500ms

Shader metadata conventions

Shaders should include a docstring that declares name and description at the top. This is parsed by ShaderMetadata.from(code:path:) and will power Library search, thumbnails, and metadata views.

Example:

/**
 * Kaleidoscope Blocks
 * Geometric color blocks with animation controls.
 */
#include <metal_stdlib>
using namespace metal;
fragment float4 fragmentShader() { return float4(0,0,0,1); }

• First non-empty doc line → name
• Following non-empty lines (until blank) → description
• Source path is recorded when available for provenance

Library tab and thumbnails

• Library entries derive their title/description from the shader docstring.
• Thumbnails can be sourced from session snapshots or generated via the headless renderer for consistency.
• As Library UX expands, expect search/filtering by tags and quick-compare overlays.

File-bridge communication contract (transitional)

While the strict MCP client is being integrated, the app uses a simple file-bridge in Resources/communication/:

• commands.json (input, optional in local workflows): queued actions (e.g., set_shader, export_frame)
• status.json (output): last action status and diagnostics
• current_shader_meta.json: active shader’s parsed name/description/path
• library_index.json: indexed library metadata
• uniforms.json: current uniform values
• compilation_errors.json: last compile diagnostics

Keys and exact shapes are subject to stabilization as the MCP transport replaces the bridge, but these files are the current source of truth for local tooling integrations.

Architecture

metal-shader-mcp/
├── src/
│   ├── index.ts         # MCP server
│   ├── compiler.ts      # Metal compilation
│   ├── preview.ts       # Preview engine
│   ├── hotReload.ts     # File watching
│   ├── profiler.ts      # Performance profiling
│   └── parameters.ts    # Uniform management
├── shaders/
│   └── kaleidoscope.metal  # Example shader
└── dist/                # Compiled output

Hot Reload Dashboard

Access the development dashboard at:

http://localhost:3000/dashboard

Features:

• Real-time compilation status
• Error and warning display
• Performance metrics
• File watching status

Shader Examples

The included kaleidoscope shader demonstrates:

• Real-time geometric transformations
• Perlin noise generation
• Color block effects
• Interactive animations
• Performance optimization techniques

Performance Optimization

The profiler measures:

• Average frame time
• Frames per second
• GPU processing time
• CPU overhead
• Memory usage
• Power consumption estimate

Development

Important: Stable paths regardless of launch directory

This project previously resolved Resources paths using the current working directory, which caused mismatched behavior when launching the MCP from different folders (e.g., animation works but library/history don’t, or vice‑versa). Paths are now resolved relative to the project root, detected as follows:

1. If METAL_SHADER_MCP_ROOT is set, that directory is used (expects a Resources subfolder)
2. Otherwise, we walk up from the MCP source location until we find a Resources directory
3. Fallback: process.cwd() (only if the above fail)

If you need to override, set an environment variable before launching:

• macOS/zsh export METAL_SHADER_MCP_ROOT="/Users/erichowens/coding/metal-shader-mcp"

This ensures the MCP and app read/write the same Resources/communication and Resources/screenshots folders regardless of where you run the CLI.

Service keys / secrets

This repository does not require external service keys to run the core ShaderPlayground + MCP flow. If you add integrations that require secrets, store them in one of the following, in order of preference:

• Local development: .env.local (gitignored)
• Shared dev on the same machine: .env (gitignored)
• CI: Configure secrets in the CI provider’s encrypted secret store
• Production: Use your deployment platform’s secret manager (e.g., GitHub Actions Secrets, 1Password, AWS/GCP secret managers)

Never commit secrets to the repo. Use environment variables in code (e.g., process.env.MY_KEY) and document any required variables here.

After-Action Requirements

Every significant development action must complete these steps:

1. Update BUGS.md - Document any issues discovered
2. Update CHANGELOG.md - Record what was accomplished
3. Capture Visual Evidence - Screenshots for UI/shader changes
4. Run Tests - Ensure no regressions introduced
5. Git Operations - Commit with descriptive messages

See WARP.md for detailed workflow documentation.

Visual Testing

This project includes a visual regression harness with shader fixtures and golden images.

# Capture screenshots of current state (optional manual evidence)
./scripts/screenshot_app.sh "feature_description"

# Run visual tests
swift test --filter VisualRegressionTests

# If you intentionally changed visuals, regenerate goldens
make regen-goldens

• On failure, artifacts are saved to Resources/screenshots/tests/ (e.g., actual_*.png, diff_*.png, and a summary JSON).
• Goldens live in Tests/MetalShaderTests/Fixtures/ and are bundled via SPM resources.

Documentation Files

• WARP.md - Agent workflow requirements
• CLAUDE.md - Creative vision and AI interaction patterns
• VISUAL_TESTING.md - Visual testing framework
• BUGS.md - Current issues and solutions
• CHANGELOG.md - Project evolution history

Deprecation notice (file-bridge)

• The file-bridge (commands.json polling) is transitional and will be removed once the strict MCP client (MCPLiveClient) is integrated into the Swift app via MCPBridge.
• Target: replace bridge with proper MCP transport (stdio/websocket), remove polling, add robust error handling.
• Timeline: as soon as PR #29 merges, we’ll proceed with PR #32 to complete this migration.

Contributing

Contributions are welcome! Please follow the workflow requirements in WARP.md:

1. Create feature branch from main
2. Implement changes with visual evidence
3. Update relevant documentation
4. Run visual regression tests
5. Submit pull request with screenshots

License

MIT