Add blog_post.md draft and project README.md
This commit is contained in:
Adolfo Reyna
@@ -0,0 +1,135 @@
|
||||
# ESP32-S3-RLCD-4.2 Model Context Protocol (MCP) Companion
|
||||
|
||||
This repository contains the MicroPython firmware and host-side bridge script to expose the **Waveshare ESP32-S3-RLCD-4.2** development board as a physical, Model Context Protocol (MCP)-compliant agentic companion.
|
||||
|
||||
With this setup, local AI harnesses (such as Claude Desktop, OpenClaude, Cursor, or Hermes) can directly control the board's screen, NeoPixels, audio codec, microphone array, sensors, and remote filesystem.
|
||||
|
||||
---
|
||||
|
||||
## 1. System Architecture
|
||||
|
||||
To bypass the memory and protocol constraints of the microcontroller, we use a hybrid **Stdio-to-HTTP LAN Bridge**:
|
||||
|
||||
```
|
||||
┌──────────────────────────────┐
|
||||
│ Local LLM Client / Harness │
|
||||
└──────────────┬───────────────┘
|
||||
│ (JSON-RPC over Stdio)
|
||||
▼
|
||||
┌──────────────────────────────┐
|
||||
│ mcp_bridge.py │ <── (Converts PNG/PBM images on-the-fly)
|
||||
└──────────────┬───────────────┘
|
||||
│ (JSON-RPC over HTTP POST)
|
||||
▼
|
||||
┌──────────────────────────────┐
|
||||
│ ESP32-S3 HTTP Server │ (Running on Port 80)
|
||||
└──────────────┬───────────────┘
|
||||
│ (MicroPython calls)
|
||||
▼
|
||||
┌──────────────────────────────┐
|
||||
│ Hardware Peripherals │ (Display, Led, Mic, Speaker, Sensors)
|
||||
└──────────────────────────────┘
|
||||
```
|
||||
|
||||
1. **MicroPython HTTP Server (`mcp_server.py`)**: Runs directly on the ESP32-S3, accepting JSON-RPC 2.0 requests at `POST /api/mcp`.
|
||||
2. **Host Stdio Bridge (`mcp_bridge.py`)**: Runs on your host PC, listening for stdio JSON-RPC calls, forwarding them to the board over Wi-Fi, and returning the output.
|
||||
3. **On-the-Fly Image Conversion**: The board returns raw dithered PBM screen buffers to save bandwidth. The host bridge automatically converts these into standard PNG images (using macOS `sips` or `Pillow`) so the LLM can "see" what is currently rendered on the screen.
|
||||
|
||||
---
|
||||
|
||||
## 2. Directory Structure & File Manifest
|
||||
|
||||
### MicroPython Device Files
|
||||
* **[boot.py](boot.py)**: Automatically connects to Wi-Fi using credentials in `wifi_config.py` and renders connection logs on the screen.
|
||||
* **[main.py](main.py)**: Background execution loop. Handlers for buttons (manual dashboard refresh, LED state cycle), NeoPixel animation loop, and non-blocking MCP server requests.
|
||||
* **[mcp_server.py](mcp_server.py)**: The lightweight JSON-RPC server implementing the MCP tools list and call handlers.
|
||||
* **[audio_util.py](audio_util.py)**: Audio drivers for the ES7210 microphone array and the ES8311 speaker DAC. Implements sine-wave tone generation (`play_tone`), voice recording (`record_audio`), and file-based audio playing (`play_wav`).
|
||||
* **[rlcd.py](rlcd.py)**: Low-level FrameBuffer driver for the 4.2" Reflective LCD, including PBM loaders and screenshot exporters.
|
||||
* **[sd_util.py](sd_util.py)**: Mounting and filesystem management utility for the onboard microSD card slot.
|
||||
* **Hardware Drivers**:
|
||||
* [shtc3_util.py](shtc3_util.py) (Temperature & Humidity)
|
||||
* [rtc_util.py](rtc_util.py) (Hardware Real-Time Clock)
|
||||
* [battery_util.py](battery_util.py) (Voltage & Capacity reader)
|
||||
* [rgb_led_util.py](rgb_led_util.py) (WS2812 NeoPixel animations)
|
||||
* [button_util.py](button_util.py) (Key & Boot button debouncer)
|
||||
* [ble_util.py](ble_util.py) (Passive BLE scanning & UART)
|
||||
* **[wifi_config.py](wifi_config.py)**: Wi-Fi credentials. **(Do not commit real credentials to Git)**.
|
||||
|
||||
### Host-Side files
|
||||
* **[mcp_bridge.py](mcp_bridge.py)**: The stdio-to-HTTP LAN bridge connecting the LLM client to the board. Handles image resizing, dithering, and formatting.
|
||||
* **[voice_assistant.py](voice_assistant.py)**: An end-to-end local voice assistant loop script (requires OpenAI/local Whisper for STT and Mac TTS commands).
|
||||
* **[blog_post.md](blog_post.md)**: A draft technical write-up detailing this project's architecture and capabilities.
|
||||
|
||||
---
|
||||
|
||||
## 3. Quickstart Guide
|
||||
|
||||
### Step 1: Configure Wi-Fi
|
||||
Edit `wifi_config.py` on your computer and set your local SSID and Password:
|
||||
```python
|
||||
WIFI_SSID = "Your_Wi-Fi_Name"
|
||||
WIFI_PASS = "Your_Password"
|
||||
```
|
||||
|
||||
### Step 2: Upload Files to the Board
|
||||
Connect the board to your computer via USB (making sure to use the active ESP32 USB port).
|
||||
Upload all Python files to the device flash memory using `mpremote`:
|
||||
```bash
|
||||
mpremote connect /dev/cu.usbmodem101 cp *.py :
|
||||
```
|
||||
*(If your serial port differs, change `/dev/cu.usbmodem101` accordingly).*
|
||||
|
||||
### Step 3: Boot the Board
|
||||
Reset the board to apply changes:
|
||||
```bash
|
||||
mpremote connect /dev/cu.usbmodem101 reset
|
||||
```
|
||||
Once booted, the reflective screen will display connection logs, followed by the local IP address (e.g. `192.168.68.122`) once connected to Wi-Fi.
|
||||
|
||||
### Step 4: Register in Claude Desktop
|
||||
Open your local Claude Desktop config file:
|
||||
* **macOS**: `/Users/<username>/Library/Application Support/Claude/claude_desktop_config.json`
|
||||
* **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
|
||||
|
||||
Add the bridge script under the `mcpServers` list:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"esp32-rlcd": {
|
||||
"command": "python3",
|
||||
"args": [
|
||||
"/absolute/path/to/this/repository/mcp_bridge.py",
|
||||
"--ip",
|
||||
"192.168.68.122"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Restart Claude Desktop, and the assistant will have direct, real-time control over your physical device!
|
||||
|
||||
---
|
||||
|
||||
## 4. MCP Tools Reference
|
||||
|
||||
Here is a summary of the MCP tools exposed by the bridge:
|
||||
|
||||
| Tool Name | Parameters | Description |
|
||||
|---|---|---|
|
||||
| `clear_screen` | `color` (0=White, 1=Black) | Clears the RLCD display. |
|
||||
| `draw_text` | `text`, `x`, `y`, `size` | Draws normal/large text on the display. |
|
||||
| `draw_image` | `image_base64`, `x`, `y`, `dither` | Uploads and dithers an image to the display. |
|
||||
| `get_screenshot` | *(None)* | Captures the screen buffer as a PNG image for the LLM. |
|
||||
| `get_sensors` | *(None)* | Returns temperature & humidity from the SHTC3. |
|
||||
| `get_battery` | *(None)* | Returns voltage and capacity percentage. |
|
||||
| `scan_ble` | `duration_ms` | Scans for nearby BLE beacons / smart tags. |
|
||||
| `play_tone` | `frequency`, `duration_ms`, `volume` | Plays a pure sine wave tone. |
|
||||
| `play_audio` | `filename`, `volume` | Plays a WAV file from the flash storage or SD. |
|
||||
| `play_audio_base64` | `wav_base64`, `volume` | Plays a base64 WAV stream (automatic caching). |
|
||||
| `record_voice` | `duration_sec`, `filename` | Records audio from the microphones to a PCM file. |
|
||||
| `download_file` | `url`, `filename`, `use_sd` | Memory-efficient download stream to flash or microSD. |
|
||||
| `write_file` | `path`, `content` | Writes text file to board flash. |
|
||||
| `read_file` | `path` | Reads text file from board flash. |
|
||||
| `execute_python` | `code` | Executes arbitrary Python code dynamically. |
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
# Bridging the Physical & Virtual: Building a MicroPython MCP Server for Local AI Harnesses
|
||||
|
||||
Large Language Models (LLMs) are incredibly capable, but they are traditionally trapped inside a digital sandbox—unable to perceive or interact with the physical world.
|
||||
|
||||
Enter the **Model Context Protocol (MCP)**, an open standard developed by Anthropic that provides a unified way for LLMs to query external databases, run code, and invoke APIs. While most developers use MCP to hook LLMs to spreadsheets or GitHub repositories, we decided to take it a step further: **giving a local AI harness direct control over physical hardware using MicroPython.**
|
||||
|
||||
In this post, we’ll walk through how we transformed a specialized ESP32-S3 development board into an interactive, local AI desktop companion capable of rendering graphics, playing audio streams, monitoring ambient environments, and executing remote python scripts—all driven natively by local LLM harnesses like **OpenClaude**, **Hermes**, or **Claude Desktop**.
|
||||
|
||||
---
|
||||
|
||||
## 1. The Hardware Platform: Waveshare ESP32-S3-RLCD-4.2
|
||||
|
||||
To build a true desktop companion, we needed a board that combined low power, rich displays, and audio capabilities. We selected the **Waveshare ESP32-S3-RLCD-4.2** development kit.
|
||||
|
||||
Here is what makes this hardware special:
|
||||
* **Core SoC**: ESP32-S3 Dual-core Xtensa processor with Wi-Fi and Bluetooth.
|
||||
* **The Screen**: A 4.2" Reflective LCD (400x300 resolution). Similar to E-paper, it has extremely high sunlight readability, requires zero backlighting (it reflects ambient room light), and retains static images indefinitely with minimal power draw.
|
||||
* **Environmental Telemetry**: Onboard SHTC3 temperature and relative humidity sensor.
|
||||
* **Time & Battery**: PCF85063 hardware RTC for accurate offline timekeeping, and a MAX1704x-equivalent battery monitor.
|
||||
* **Interactive Input**: Onboard Key/Boot buttons and a passive Bluetooth Low Energy (BLE) radio for nearby device tracking.
|
||||
* **RGB LED**: A multi-mode WS2812 NeoPixel for state animations.
|
||||
* **The Audio Pipeline**: An **ES7210 4-channel microphone array** for capturing voice commands, paired with an **ES8311 audio decoder/speaker driver** and built-in amplifier.
|
||||
* **Storage Expansion**: A microSD card slot to cache large media files like podcasts.
|
||||
|
||||
---
|
||||
|
||||
## 2. MicroPython MCP Architecture: Stdio-to-HTTP Bridge
|
||||
|
||||
Typical desktop LLMs interact with MCP servers locally over standard input/output (stdio) pipelines. However, running a full node environment or complex stdio protocol parser directly on a resource-constrained microcontroller is impractical.
|
||||
|
||||
To bridge this gap, we designed a **hybrid stdio-to-network bridge**:
|
||||
|
||||
```
|
||||
[ Local LLM Client / Harness ]
|
||||
│ (JSON-RPC over Stdio)
|
||||
▼
|
||||
[ host_bridge.py ] <─── (Pillow converts PNGs/PBMs on-the-fly)
|
||||
│ (JSON-RPC over HTTP POST)
|
||||
▼
|
||||
[ ESP32-S3 Web Server ]
|
||||
│ (MicroPython execution)
|
||||
▼
|
||||
[ Hardware Peripherals ]
|
||||
```
|
||||
|
||||
1. **On-Board HTTP Server**: A lightweight, non-blocking TCP socket server written in MicroPython (`mcp_server.py`) runs directly on the ESP32, listening on port `80`. It handles `POST /api/mcp` requests, parsing JSON-RPC 2.0 structures.
|
||||
2. **Host-Side Stdio Bridge**: A zero-dependency Python script (`mcp_bridge.py`) runs on the host PC. It reads stdio packets from the LLM client, forwards them over the local Wi-Fi to the ESP32’s IP address, and relays the board's responses back.
|
||||
3. **On-the-Fly Media Conversion**: Since the ESP32 has limited RAM, it returns screenshots as raw binary 1-bit PBM data. The host bridge intercepts this data, converts it into standard PNG format on your PC using native image tools (`sips` or `Pillow`), and serves it to the LLM. It does the reverse for image uploads (`draw_image`), rendering them locally as 1-bit dithered PBMs before sending them to the microcontroller.
|
||||
|
||||
---
|
||||
|
||||
## 3. The Tool Belt: 15 Exposed Hardware Utilities
|
||||
|
||||
By connecting the board to the MCP server, we exposed 15 specialized tools to the local AI. The LLM can choose to invoke any of these actions in response to your prompts:
|
||||
|
||||
### Display & Graphics
|
||||
1. **`clear_screen(color)`**: Clear the 400x300 reflective display to pure white or black.
|
||||
2. **`draw_text(text, x, y, size)`**: Draw custom labels or telemetry readouts at specified coordinates.
|
||||
3. **`draw_image(image_base64, x, y, dither)`**: Upload any standard image format (JPEG/PNG/GIF). The host bridge automatically resizes and dithers it, and the board displays it.
|
||||
4. **`get_screenshot()`**: Captures the exact reflective LCD frame buffer and returns a PNG. **This allows the LLM to inspect the screen and confirm if its renders are correct!**
|
||||
|
||||
### Environment & System Telemetry
|
||||
5. **`get_sensors()`**: Returns live temperature (°C) and humidity (%) from the SHTC3 sensor.
|
||||
6. **`get_battery()`**: Reads the current battery voltage and estimated capacity percentage.
|
||||
7. **`scan_ble(duration_ms)`**: Performs a BLE scan to log nearby smart tags or trackers.
|
||||
|
||||
### Audio & Sound Control
|
||||
8. **`play_tone(frequency, duration_ms, volume)`**: Play a clean, synthesized sine wave tone.
|
||||
9. **`play_audio(filename, volume)`**: Stream and play standard WAV files (e.g. system alerts, sound effects) from the board's storage.
|
||||
10. **`play_audio_base64(wav_base64, volume)`**: Allows the LLM to synthesize speech locally on your computer and stream it to the board's speaker in a single tool call (auto-caching and auto-cleaning temporary flash memory).
|
||||
11. **`record_voice(duration_sec, filename)`**: Activates the microphone array to record your voice to flash memory for voice command loops.
|
||||
|
||||
### Remote File & Code Execution
|
||||
12. **`write_file(path, content)`**: Write configuration text or script scripts to flash.
|
||||
13. **`read_file(path)`**: Read any file on the board's local filesystem.
|
||||
14. **`execute_python(code)`**: Run arbitrary Python code dynamically on the board via `exec()`. Standard output and tracebacks are captured and returned.
|
||||
15. **`download_file(url, filename, use_sd)`**: Downloads large media files (like music or podcasts) from the web directly to the microSD card in memory-efficient stream chunks.
|
||||
|
||||
---
|
||||
|
||||
## 4. How to Register the MCP Server on Local AI Harnesses
|
||||
|
||||
Adding this ESP32-S3 companion to your local AI environment is simple.
|
||||
|
||||
### Step 1: Discover the Device IP
|
||||
Ensure your ESP32 board is connected to the same local Wi-Fi router. Upon boot, the board's screen will display its assigned local IP address (e.g., `192.168.68.122`).
|
||||
|
||||
### Step 2: Register in Claude Desktop or Hermes/OpenClaude
|
||||
Open your local harness config file. For Claude Desktop, this is located at:
|
||||
* **macOS**: `/Users/<username>/Library/Application Support/Claude/claude_desktop_config.json`
|
||||
* **Windows**: `%APPDATA%\Claude\claude_desktop_config.json`
|
||||
|
||||
Add the bridge script under the `mcpServers` entry, specifying the board's IP address:
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"esp32-rlcd-companion": {
|
||||
"command": "python3",
|
||||
"args": [
|
||||
"/absolute/path/to/your/workspace/mcp_bridge.py",
|
||||
"--ip",
|
||||
"192.168.68.122"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Step 3: Launch and Interact
|
||||
Restart your local AI harness. You will see a small plug icon or tools notification indicating that the board's functions are registered.
|
||||
|
||||
You can now prompt the AI using natural language commands:
|
||||
* *"Check the room temperature. If it's over 25 degrees, set the LED to breathing red and show a warning message on the screen."*
|
||||
* *"Draw a cute dithered pixel-art cat at the center of the display, then take a screenshot so you can check how it looks."*
|
||||
* *"Download the podcast at this WAV URL to the SD card, and then play it on the speaker at 60% volume."*
|
||||
|
||||
The LLM will translate these requests into tool calls, control your hardware over the Wi-Fi bridge, and report back!
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
By mapping hardware control to standard MCP tool schemas, we've demonstrated how easily microcontrollers can be integrated into the modern agentic LLM ecosystem. The ESP32-S3 is no longer just a standalone sensor node—it is a physical avatar for your local AI.
|
||||
|
||||
Whether you want to build a smart e-paper calendar, an interactive virtual desk pet, or an offline voice assistant, MicroPython combined with MCP provides the ultimate foundation.
|
||||
Reference in New Issue
Block a user