Adolfo Reyna
148 lines
8.2 KiB
Markdown
148 lines
8.2 KiB
Markdown
# ESP32-S3-RLCD-4.2 Model Context Protocol (MCP) Companion
|
|
|
|
This repository contains the MicroPython firmware and host-side bridge script to expose the **Waveshare ESP32-S3-RLCD-4.2** development board as a physical, Model Context Protocol (MCP)-compliant agentic companion.
|
|
|
|
It also includes a native jailbroken iPhone 6 target in [`iphone_app/`](iphone_app/README.md). The iPhone implementation exposes the same MCP-style screen, audio, camera, file, and monochrome streaming features, plus RGB565 color streaming and Hermes voice turns.
|
|
|
|
With this setup, local AI harnesses (such as Claude Desktop, OpenClaude, Cursor, or Hermes) can directly control the board's screen, NeoPixels, audio codec, microphone array, sensors, and remote filesystem.
|
|
|
|
---
|
|
|
|
## 1. System Architecture
|
|
|
|
To bypass the memory and protocol constraints of the microcontroller, we use a hybrid **Stdio-to-HTTP LAN Bridge**:
|
|
|
|
```
|
|
┌──────────────────────────────┐
|
|
│ Local LLM Client / Harness │
|
|
└──────────────┬───────────────┘
|
|
│ (JSON-RPC over Stdio)
|
|
▼
|
|
┌──────────────────────────────┐
|
|
│ mcp_bridge.py │ <── (Converts PNG/PBM images on-the-fly)
|
|
└──────────────┬───────────────┘
|
|
│ (JSON-RPC over HTTP POST)
|
|
▼
|
|
┌──────────────────────────────┐
|
|
│ ESP32-S3 HTTP Server │ (Running on Port 80)
|
|
└──────────────┬───────────────┘
|
|
│ (MicroPython calls)
|
|
▼
|
|
┌──────────────────────────────┐
|
|
│ Hardware Peripherals │ (Display, Led, Mic, Speaker, Sensors)
|
|
└──────────────────────────────┘
|
|
```
|
|
|
|
1. **MicroPython HTTP Server (`mcp_server.py`)**: Runs directly on the ESP32-S3, accepting JSON-RPC 2.0 requests at `POST /api/mcp`.
|
|
2. **Host Stdio Bridge (`mcp_bridge.py`)**: Runs on your host PC, listening for stdio JSON-RPC calls, forwarding them to the board over Wi-Fi, and returning the output.
|
|
3. **On-the-Fly Image Conversion**: The board returns raw dithered PBM screen buffers to save bandwidth. The host bridge automatically converts these into standard PNG images (using macOS `sips` or `Pillow`) so the LLM can "see" what is currently rendered on the screen.
|
|
|
|
---
|
|
|
|
## 2. Directory Structure & File Manifest
|
|
|
|
### MicroPython Device Files
|
|
* **[boot.py](boot.py)**: Automatically connects to Wi-Fi using credentials in `wifi_config.py` and renders connection logs on the screen.
|
|
* **[main.py](main.py)**: Background execution loop. Handlers for buttons (manual dashboard refresh, LED state cycle), NeoPixel animation loop, and non-blocking MCP server requests.
|
|
* **[mcp_server.py](mcp_server.py)**: The lightweight JSON-RPC server implementing the MCP tools list and call handlers.
|
|
* **[wifi_config.py](wifi_config.py)**: Wi-Fi credentials. **(Do not commit real credentials to Git)**.
|
|
* **`lib/`**: Hardware drivers and utility modules automatically searched by MicroPython:
|
|
* **[audio_util.py](lib/audio_util.py)**: Audio drivers for the ES7210 microphone array and the ES8311 speaker DAC. Implements sine-wave tone generation (`play_tone`), voice recording (`record_audio`), and file-based audio playing (`play_wav`).
|
|
* **[rlcd.py](lib/rlcd.py)**: Low-level FrameBuffer driver for the 4.2" Reflective LCD, including PBM loaders and screenshot exporters.
|
|
* **[sd_util.py](lib/sd_util.py)**: Mounting and filesystem management utility for the onboard microSD card slot.
|
|
* **[shtc3_util.py](lib/shtc3_util.py)**: Temperature & Humidity sensor utility.
|
|
* **[rtc_util.py](lib/rtc_util.py)**: PCF85063 Hardware Real-Time Clock driver.
|
|
* **[battery_util.py](lib/battery_util.py)**: Battery voltage and capacity reader.
|
|
* **[rgb_led_util.py](lib/rgb_led_util.py)**: WS2812 NeoPixel animation manager.
|
|
* **[button_util.py](lib/button_util.py)**: Boot/Key button debouncer utility.
|
|
* **[ble_util.py](lib/ble_util.py)**: Passive BLE scanning and BLE UART interface.
|
|
* **[ft6336u.py](lib/ft6336u.py)**: I2C Capacitive touchscreen controller driver.
|
|
* **[ili9341.py](lib/ili9341.py)**: ILI9341 LCD driver.
|
|
* **[st7796.py](lib/st7796.py)**: ST7796 LCD driver.
|
|
* **[download_util.py](lib/download_util.py)**: Helper for streaming downloads to the local filesystem.
|
|
|
|
### Host-Side files
|
|
* **[mcp_bridge.py](mcp_bridge.py)**: The stdio-to-HTTP LAN bridge connecting the LLM client to the board. Handles image resizing, dithering, and formatting.
|
|
* **[voice_assistant.py](voice_assistant.py)**: An end-to-end local voice assistant loop script (requires OpenAI/local Whisper for STT and Mac TTS commands).
|
|
* **[blog_post.md](blog_post.md)**: A draft technical write-up detailing this project's architecture and capabilities.
|
|
|
|
---
|
|
|
|
## 3. Quickstart Guide
|
|
|
|
### Step 1: Configure Wi-Fi
|
|
Edit `wifi_config.py` on your computer and set your local SSID and Password:
|
|
```python
|
|
WIFI_SSID = "Your_Wi-Fi_Name"
|
|
WIFI_PASS = "Your_Password"
|
|
```
|
|
|
|
### Step 2: Upload Files to the Board
|
|
Connect the board to your computer via USB (making sure to use the active ESP32 USB port).
|
|
Upload all Python files to the device flash memory using `mpremote`:
|
|
```bash
|
|
mpremote connect /dev/cu.usbmodem101 cp *.py :
|
|
```
|
|
*(If your serial port differs, change `/dev/cu.usbmodem101` accordingly).*
|
|
|
|
### Step 3: Boot the Board
|
|
Reset the board to apply changes:
|
|
```bash
|
|
mpremote connect /dev/cu.usbmodem101 reset
|
|
```
|
|
Once booted, the reflective screen will display connection logs, followed by the local IP address (e.g. `192.168.68.122`) once connected to Wi-Fi.
|
|
|
|
### Step 4: Register in Claude Desktop
|
|
Open your local Claude Desktop config file:
|
|
* **macOS**: `/Users/<username>/Library/Application Support/Claude/claude_desktop_config.json`
|
|
* **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
|
|
|
|
Add the bridge script under the `mcpServers` list:
|
|
|
|
```json
|
|
{
|
|
"mcpServers": {
|
|
"esp32-rlcd": {
|
|
"command": "python3",
|
|
"args": [
|
|
"/absolute/path/to/this/repository/mcp_bridge.py",
|
|
"--ip",
|
|
"192.168.68.122"
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
Restart Claude Desktop, and the assistant will have direct, real-time control over your physical device!
|
|
|
|
---
|
|
|
|
## 4. MCP Tools Reference
|
|
|
|
Here is a summary of the MCP tools exposed by the bridge:
|
|
|
|
| Tool Name | Parameters | Description |
|
|
|---|---|---|
|
|
| `clear_screen` | `color` (0=White, 1=Black) | Clears the RLCD display. |
|
|
| `draw_text` | `text`, `x`, `y`, `size` | Draws normal/large text on the display. |
|
|
| `draw_image` | `image_base64`, `x`, `y`, `dither` | Uploads and dithers an image to the display. |
|
|
| `get_screenshot` | *(None)* | Captures the screen buffer as a PNG image for the LLM. |
|
|
| `get_sensors` | *(None)* | Returns temperature & humidity from the SHTC3. |
|
|
| `get_battery` | *(None)* | Returns voltage and capacity percentage. |
|
|
| `scan_ble` | `duration_ms` | Scans for nearby BLE beacons / smart tags. |
|
|
| `play_tone` | `frequency`, `duration_ms`, `volume` | Plays a pure sine wave tone. |
|
|
| `play_audio` | `filename`, `volume` | Plays a WAV file from the flash storage or SD. |
|
|
| `play_audio_base64` | `wav_base64`, `volume` | Plays a base64 WAV stream (automatic caching). |
|
|
| `record_voice` | `duration_sec`, `filename` | Records audio from the microphones to a PCM file. |
|
|
| `download_file` | `url`, `filename`, `use_sd` | Memory-efficient download stream to flash or microSD. |
|
|
| `write_file` | `path`, `content` | Writes text file to board flash. |
|
|
| `read_file` | `path` | Reads text file from board flash. |
|
|
| `execute_python` | `code` | Executes arbitrary Python code dynamically. |
|
|
|
|
---
|
|
|
|
## 5. Hardware & Troubleshooting Findings
|
|
|
|
For detailed technical findings regarding the board's hardware (e.g. I2S bit-depth configuration, I2C bus lockup recovery, pinout configuration, and ES7210 microphone clock/OSR registers to resolve popping sound issues), refer to [hardware_findings.md](hardware_findings.md).
|