Virtual Screen & Speakers MCP Companion

This directory implements a python-based Model Context Protocol (MCP) server that hosts a premium, futuristic web dashboard. You can open this web interface on any local network device (like a phone, tablet, or secondary monitor) to act as a virtual screen and speakers for the LLM.

┌─────────────────────────────────┐
│     Local LLM Client / Cursor   │
└────────────────┬────────────────┘
                 │ (JSON-RPC over Stdio)
                 ▼
┌─────────────────────────────────┐
│           server.py             │ <── (Maintains PIL canvas state in memory)
└────────────────┬────────────────┘
                 │ (JSON-RPC over WebSockets)
                 ▼
┌─────────────────────────────────┐
│       Browser Web Client        │ (Render canvas, play Audio API tones/WAV,
│       (index.html UI)           │  capture mic and upload WAV)
└─────────────────────────────────┘

1. Prerequisites

Ensure you have Python 3.10+ installed along with the required libraries:

pip install pillow tornado

2. Launching the Server

Start the companion server from this folder:

python3 server.py --port 8080

Upon launching, it will print out the connection URLs, such as:

--------------------------------------------------
Virtual Screen & Speaker MCP Server Initialized.
Connect local devices in your browser to:
==> http://192.168.1.15:8080/
==> http://localhost:8080/
--------------------------------------------------

Open either URL on your laptop or enter the local IP version (http://192.168.1.15:8080/) into the browser of your phone or tablet.

3. Claude Desktop Integration

To register this server in Claude Desktop, open your configuration file:

macOS: /Users/<username>/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

Add the server to the list:

{
  "mcpServers": {
    "virtual-companion": {
      "command": "python3",
      "args": [
        "/Users/adolforeyna/Projects/MicroPython/test1/Screen/virtual_screen_mcp/server.py",
        "--port",
        "8080"
      ]
    }
  }
}

Restart Claude Desktop to load the companion tools.

4. Features & Tools

The server exposes the following MCP tools to the LLM:

Tool Name	Parameters	Description
`clear_screen`	`color` (optional CSS color, e.g., `#1e1e2e` or legacy 0/1)	Clears the web canvas.
`draw_text`	`text`, `x`, `y`, `size`, `color`	Draws text with custom fonts and colors on the display.
`draw_shape`	`shape` (rect/circle/line), `x`, `y`, `width`, `height`, `color`, `fill`	Draws vector shapes on the canvas.
`draw_image`	`image_base64`, `x`, `y`, `width`, `height`	Draws base64 encoded images.
`get_screenshot`	(None)	Captures the virtual display buffer as a PNG image for the LLM.
`set_led`	`r`, `g`, `b`, `mode` (static/breath/rainbow/off)	Controls the CSS WS2812 NeoPixel ring.
`get_sensors`	(None)	Reads telemetry values adjusted by the dashboard sliders.
`play_tone`	`frequency`, `duration_ms`, `volume`	Synthesizes pure tones on the browser speakers.
`play_audio`	`audio_base64_or_url`, `volume`	Plays an audio track on the browser speakers.
`record_voice`	`duration_sec`	Captures microphone input from the browser, encodes it as a mono 16-bit WAV, and saves it.

5. Web Client Telemetry

The dashboard provides interactive elements:

Virtual Sensor Sliders: Move the sliders for Temperature, Humidity, and Light to feed custom telemetry data to the LLM. When the LLM calls get_sensors, it receives these live values.
Microphone Recorder: Shows active microphone status, handles media constraints, and encodes 16-bit WAV files locally on the browser side.
LED NeoPixel ring: Visualizes breath, rainbow, and static LED modes in full glowing CSS.

4.3 KiB Raw Permalink Blame History