Adolfo Reyna
102 lines
4.3 KiB
Markdown
102 lines
4.3 KiB
Markdown
# Virtual Screen & Speakers MCP Companion
|
|
|
|
This directory implements a python-based Model Context Protocol (MCP) server that hosts a premium, futuristic web dashboard. You can open this web interface on **any local network device** (like a phone, tablet, or secondary monitor) to act as a virtual screen and speakers for the LLM.
|
|
|
|
```
|
|
┌─────────────────────────────────┐
|
|
│ Local LLM Client / Cursor │
|
|
└────────────────┬────────────────┘
|
|
│ (JSON-RPC over Stdio)
|
|
▼
|
|
┌─────────────────────────────────┐
|
|
│ server.py │ <── (Maintains PIL canvas state in memory)
|
|
└────────────────┬────────────────┘
|
|
│ (JSON-RPC over WebSockets)
|
|
▼
|
|
┌─────────────────────────────────┐
|
|
│ Browser Web Client │ (Render canvas, play Audio API tones/WAV,
|
|
│ (index.html UI) │ capture mic and upload WAV)
|
|
└─────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## 1. Prerequisites
|
|
|
|
Ensure you have Python 3.10+ installed along with the required libraries:
|
|
```bash
|
|
pip install pillow tornado
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Launching the Server
|
|
|
|
Start the companion server from this folder:
|
|
```bash
|
|
python3 server.py --port 8080
|
|
```
|
|
Upon launching, it will print out the connection URLs, such as:
|
|
```text
|
|
--------------------------------------------------
|
|
Virtual Screen & Speaker MCP Server Initialized.
|
|
Connect local devices in your browser to:
|
|
==> http://192.168.1.15:8080/
|
|
==> http://localhost:8080/
|
|
--------------------------------------------------
|
|
```
|
|
Open either URL on your laptop or enter the local IP version (`http://192.168.1.15:8080/`) into the browser of your phone or tablet.
|
|
|
|
---
|
|
|
|
## 3. Claude Desktop Integration
|
|
|
|
To register this server in Claude Desktop, open your configuration file:
|
|
* **macOS**: `/Users/<username>/Library/Application Support/Claude/claude_desktop_config.json`
|
|
* **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
|
|
|
|
Add the server to the list:
|
|
```json
|
|
{
|
|
"mcpServers": {
|
|
"virtual-companion": {
|
|
"command": "python3",
|
|
"args": [
|
|
"/Users/adolforeyna/Projects/MicroPython/test1/Screen/virtual_screen_mcp/server.py",
|
|
"--port",
|
|
"8080"
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
Restart Claude Desktop to load the companion tools.
|
|
|
|
---
|
|
|
|
## 4. Features & Tools
|
|
|
|
The server exposes the following MCP tools to the LLM:
|
|
|
|
| Tool Name | Parameters | Description |
|
|
|---|---|---|
|
|
| `clear_screen` | `color` (optional CSS color, e.g., `#1e1e2e` or legacy 0/1) | Clears the web canvas. |
|
|
| `draw_text` | `text`, `x`, `y`, `size`, `color` | Draws text with custom fonts and colors on the display. |
|
|
| `draw_shape` | `shape` (rect/circle/line), `x`, `y`, `width`, `height`, `color`, `fill` | Draws vector shapes on the canvas. |
|
|
| `draw_image` | `image_base64`, `x`, `y`, `width`, `height` | Draws base64 encoded images. |
|
|
| `get_screenshot` | *(None)* | Captures the virtual display buffer as a PNG image for the LLM. |
|
|
| `set_led` | `r`, `g`, `b`, `mode` (static/breath/rainbow/off) | Controls the CSS WS2812 NeoPixel ring. |
|
|
| `get_sensors` | *(None)* | Reads telemetry values adjusted by the dashboard sliders. |
|
|
| `play_tone` | `frequency`, `duration_ms`, `volume` | Synthesizes pure tones on the browser speakers. |
|
|
| `play_audio` | `audio_base64_or_url`, `volume` | Plays an audio track on the browser speakers. |
|
|
| `record_voice` | `duration_sec` | Captures microphone input from the browser, encodes it as a mono 16-bit WAV, and saves it. |
|
|
|
|
---
|
|
|
|
## 5. Web Client Telemetry
|
|
|
|
The dashboard provides interactive elements:
|
|
* **Virtual Sensor Sliders**: Move the sliders for Temperature, Humidity, and Light to feed custom telemetry data to the LLM. When the LLM calls `get_sensors`, it receives these live values.
|
|
* **Microphone Recorder**: Shows active microphone status, handles media constraints, and encodes 16-bit WAV files locally on the browser side.
|
|
* **LED NeoPixel ring**: Visualizes breath, rainbow, and static LED modes in full glowing CSS.
|