# ESP32-S3-RLCD-4.2 Model Context Protocol (MCP) Companion

This repository contains the MicroPython firmware and host-side bridge script to expose the **Waveshare ESP32-S3-RLCD-4.2** development board as a physical, Model Context Protocol (MCP)-compliant agentic companion. 

With this setup, local AI harnesses (such as Claude Desktop, OpenClaude, Cursor, or Hermes) can directly control the board's screen, NeoPixels, audio codec, microphone array, sensors, and remote filesystem.

---

## 1. System Architecture

To bypass the memory and protocol constraints of the microcontroller, we use a hybrid **Stdio-to-HTTP LAN Bridge**:

```
┌──────────────────────────────┐
│  Local LLM Client / Harness  │
└──────────────┬───────────────┘
               │ (JSON-RPC over Stdio)
               ▼
┌──────────────────────────────┐
│       mcp_bridge.py          │  <── (Converts PNG/PBM images on-the-fly)
└──────────────┬───────────────┘
               │ (JSON-RPC over HTTP POST)
               ▼
┌──────────────────────────────┐
│   ESP32-S3 HTTP Server       │  (Running on Port 80)
└──────────────┬───────────────┘
               │ (MicroPython calls)
               ▼
┌──────────────────────────────┐
│   Hardware Peripherals       │  (Display, Led, Mic, Speaker, Sensors)
└──────────────────────────────┘
```

1.  **MicroPython HTTP Server (`mcp_server.py`)**: Runs directly on the ESP32-S3, accepting JSON-RPC 2.0 requests at `POST /api/mcp`.
2.  **Host Stdio Bridge (`mcp_bridge.py`)**: Runs on your host PC, listening for stdio JSON-RPC calls, forwarding them to the board over Wi-Fi, and returning the output.
3.  **On-the-Fly Image Conversion**: The board returns raw dithered PBM screen buffers to save bandwidth. The host bridge automatically converts these into standard PNG images (using macOS `sips` or `Pillow`) so the LLM can "see" what is currently rendered on the screen.

---

## 2. Directory Structure & File Manifest

### MicroPython Device Files
*   **[boot.py](boot.py)**: Automatically connects to Wi-Fi using credentials in `wifi_config.py` and renders connection logs on the screen.
*   **[main.py](main.py)**: Background execution loop. Handlers for buttons (manual dashboard refresh, LED state cycle), NeoPixel animation loop, and non-blocking MCP server requests.
*   **[mcp_server.py](mcp_server.py)**: The lightweight JSON-RPC server implementing the MCP tools list and call handlers.
*   **[audio_util.py](audio_util.py)**: Audio drivers for the ES7210 microphone array and the ES8311 speaker DAC. Implements sine-wave tone generation (`play_tone`), voice recording (`record_audio`), and file-based audio playing (`play_wav`).
*   **[rlcd.py](rlcd.py)**: Low-level FrameBuffer driver for the 4.2" Reflective LCD, including PBM loaders and screenshot exporters.
*   **[sd_util.py](sd_util.py)**: Mounting and filesystem management utility for the onboard microSD card slot.
*   **Hardware Drivers**: 
    *   [shtc3_util.py](shtc3_util.py) (Temperature & Humidity)
    *   [rtc_util.py](rtc_util.py) (Hardware Real-Time Clock)
    *   [battery_util.py](battery_util.py) (Voltage & Capacity reader)
    *   [rgb_led_util.py](rgb_led_util.py) (WS2812 NeoPixel animations)
    *   [button_util.py](button_util.py) (Key & Boot button debouncer)
    *   [ble_util.py](ble_util.py) (Passive BLE scanning & UART)
*   **[wifi_config.py](wifi_config.py)**: Wi-Fi credentials. **(Do not commit real credentials to Git)**.

### Host-Side files
*   **[mcp_bridge.py](mcp_bridge.py)**: The stdio-to-HTTP LAN bridge connecting the LLM client to the board. Handles image resizing, dithering, and formatting.
*   **[voice_assistant.py](voice_assistant.py)**: An end-to-end local voice assistant loop script (requires OpenAI/local Whisper for STT and Mac TTS commands).
*   **[blog_post.md](blog_post.md)**: A draft technical write-up detailing this project's architecture and capabilities.

---

## 3. Quickstart Guide

### Step 1: Configure Wi-Fi
Edit `wifi_config.py` on your computer and set your local SSID and Password:
```python
WIFI_SSID = "Your_Wi-Fi_Name"
WIFI_PASS = "Your_Password"
```

### Step 2: Upload Files to the Board
Connect the board to your computer via USB (making sure to use the active ESP32 USB port).
Upload all Python files to the device flash memory using `mpremote`:
```bash
mpremote connect /dev/cu.usbmodem101 cp *.py :
```
*(If your serial port differs, change `/dev/cu.usbmodem101` accordingly).*

### Step 3: Boot the Board
Reset the board to apply changes:
```bash
mpremote connect /dev/cu.usbmodem101 reset
```
Once booted, the reflective screen will display connection logs, followed by the local IP address (e.g. `192.168.68.122`) once connected to Wi-Fi.

### Step 4: Register in Claude Desktop
Open your local Claude Desktop config file:
*   **macOS**: `/Users/<username>/Library/Application Support/Claude/claude_desktop_config.json`
*   **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`

Add the bridge script under the `mcpServers` list:

```json
{
  "mcpServers": {
    "esp32-rlcd": {
      "command": "python3",
      "args": [
        "/absolute/path/to/this/repository/mcp_bridge.py",
        "--ip",
        "192.168.68.122"
      ]
    }
  }
}
```

Restart Claude Desktop, and the assistant will have direct, real-time control over your physical device!

---

## 4. MCP Tools Reference

Here is a summary of the MCP tools exposed by the bridge:

| Tool Name | Parameters | Description |
|---|---|---|
| `clear_screen` | `color` (0=White, 1=Black) | Clears the RLCD display. |
| `draw_text` | `text`, `x`, `y`, `size` | Draws normal/large text on the display. |
| `draw_image` | `image_base64`, `x`, `y`, `dither` | Uploads and dithers an image to the display. |
| `get_screenshot` | *(None)* | Captures the screen buffer as a PNG image for the LLM. |
| `get_sensors` | *(None)* | Returns temperature & humidity from the SHTC3. |
| `get_battery` | *(None)* | Returns voltage and capacity percentage. |
| `scan_ble` | `duration_ms` | Scans for nearby BLE beacons / smart tags. |
| `play_tone` | `frequency`, `duration_ms`, `volume` | Plays a pure sine wave tone. |
| `play_audio` | `filename`, `volume` | Plays a WAV file from the flash storage or SD. |
| `play_audio_base64` | `wav_base64`, `volume` | Plays a base64 WAV stream (automatic caching). |
| `record_voice` | `duration_sec`, `filename` | Records audio from the microphones to a PCM file. |
| `download_file` | `url`, `filename`, `use_sd` | Memory-efficient download stream to flash or microSD. |
| `write_file` | `path`, `content` | Writes text file to board flash. |
| `read_file` | `path` | Reads text file from board flash. |
| `execute_python` | `code` | Executes arbitrary Python code dynamically. |