ESP32-S3-RLCD-4.2 Model Context Protocol (MCP) Companion

This repository contains the MicroPython firmware and host-side bridge script to expose the Waveshare ESP32-S3-RLCD-4.2 development board as a physical, Model Context Protocol (MCP)-compliant agentic companion.

With this setup, local AI harnesses (such as Claude Desktop, OpenClaude, Cursor, or Hermes) can directly control the board's screen, NeoPixels, audio codec, microphone array, sensors, and remote filesystem.

1. System Architecture

To bypass the memory and protocol constraints of the microcontroller, we use a hybrid Stdio-to-HTTP LAN Bridge:

┌──────────────────────────────┐
│  Local LLM Client / Harness  │
└──────────────┬───────────────┘
               │ (JSON-RPC over Stdio)
               ▼
┌──────────────────────────────┐
│       mcp_bridge.py          │  <── (Converts PNG/PBM images on-the-fly)
└──────────────┬───────────────┘
               │ (JSON-RPC over HTTP POST)
               ▼
┌──────────────────────────────┐
│   ESP32-S3 HTTP Server       │  (Running on Port 80)
└──────────────┬───────────────┘
               │ (MicroPython calls)
               ▼
┌──────────────────────────────┐
│   Hardware Peripherals       │  (Display, Led, Mic, Speaker, Sensors)
└──────────────────────────────┘

MicroPython HTTP Server (mcp_server.py): Runs directly on the ESP32-S3, accepting JSON-RPC 2.0 requests at POST /api/mcp.
Host Stdio Bridge (mcp_bridge.py): Runs on your host PC, listening for stdio JSON-RPC calls, forwarding them to the board over Wi-Fi, and returning the output.
On-the-Fly Image Conversion: The board returns raw dithered PBM screen buffers to save bandwidth. The host bridge automatically converts these into standard PNG images (using macOS sips or Pillow) so the LLM can "see" what is currently rendered on the screen.

2. Directory Structure & File Manifest

MicroPython Device Files

boot.py: Automatically connects to Wi-Fi using credentials in wifi_config.py and renders connection logs on the screen.
main.py: Background execution loop. Handlers for buttons (manual dashboard refresh, LED state cycle), NeoPixel animation loop, and non-blocking MCP server requests.
mcp_server.py: The lightweight JSON-RPC server implementing the MCP tools list and call handlers.
wifi_config.py: Wi-Fi credentials. (Do not commit real credentials to Git).
lib/: Hardware drivers and utility modules automatically searched by MicroPython:
- audio_util.py: Audio drivers for the ES7210 microphone array and the ES8311 speaker DAC. Implements sine-wave tone generation (play_tone), voice recording (record_audio), and file-based audio playing (play_wav).
- rlcd.py: Low-level FrameBuffer driver for the 4.2" Reflective LCD, including PBM loaders and screenshot exporters.
- sd_util.py: Mounting and filesystem management utility for the onboard microSD card slot.
- shtc3_util.py: Temperature & Humidity sensor utility.
- rtc_util.py: PCF85063 Hardware Real-Time Clock driver.
- battery_util.py: Battery voltage and capacity reader.
- rgb_led_util.py: WS2812 NeoPixel animation manager.
- button_util.py: Boot/Key button debouncer utility.
- ble_util.py: Passive BLE scanning and BLE UART interface.
- ft6336u.py: I2C Capacitive touchscreen controller driver.
- ili9341.py: ILI9341 LCD driver.
- st7796.py: ST7796 LCD driver.
- download_util.py: Helper for streaming downloads to the local filesystem.

Host-Side files

mcp_bridge.py: The stdio-to-HTTP LAN bridge connecting the LLM client to the board. Handles image resizing, dithering, and formatting.
voice_assistant.py: An end-to-end local voice assistant loop script (requires OpenAI/local Whisper for STT and Mac TTS commands).
blog_post.md: A draft technical write-up detailing this project's architecture and capabilities.

3. Quickstart Guide

Step 1: Configure Wi-Fi

Edit wifi_config.py on your computer and set your local SSID and Password:

WIFI_SSID = "Your_Wi-Fi_Name"
WIFI_PASS = "Your_Password"

Step 2: Upload Files to the Board

Connect the board to your computer via USB (making sure to use the active ESP32 USB port). Upload all Python files to the device flash memory using mpremote:

mpremote connect /dev/cu.usbmodem101 cp *.py :

(If your serial port differs, change /dev/cu.usbmodem101 accordingly).

Step 3: Boot the Board

Reset the board to apply changes:

mpremote connect /dev/cu.usbmodem101 reset

Once booted, the reflective screen will display connection logs, followed by the local IP address (e.g. 192.168.68.122) once connected to Wi-Fi.

Step 4: Register in Claude Desktop

Open your local Claude Desktop config file:

macOS: /Users/<username>/Library/Application Support/Claude/claude_desktop_config.json
Windows: %APPDATA%\Claude\claude_desktop_config.json

Add the bridge script under the mcpServers list:

{
  "mcpServers": {
    "esp32-rlcd": {
      "command": "python3",
      "args": [
        "/absolute/path/to/this/repository/mcp_bridge.py",
        "--ip",
        "192.168.68.122"
      ]
    }
  }
}

Restart Claude Desktop, and the assistant will have direct, real-time control over your physical device!

4. MCP Tools Reference

Here is a summary of the MCP tools exposed by the bridge:

Tool Name	Parameters	Description
`clear_screen`	`color` (0=White, 1=Black)	Clears the RLCD display.
`draw_text`	`text`, `x`, `y`, `size`	Draws normal/large text on the display.
`draw_image`	`image_base64`, `x`, `y`, `dither`	Uploads and dithers an image to the display.
`get_screenshot`	(None)	Captures the screen buffer as a PNG image for the LLM.
`get_sensors`	(None)	Returns temperature & humidity from the SHTC3.
`get_battery`	(None)	Returns voltage and capacity percentage.
`scan_ble`	`duration_ms`	Scans for nearby BLE beacons / smart tags.
`play_tone`	`frequency`, `duration_ms`, `volume`	Plays a pure sine wave tone.
`play_audio`	`filename`, `volume`	Plays a WAV file from the flash storage or SD.
`play_audio_base64`	`wav_base64`, `volume`	Plays a base64 WAV stream (automatic caching).
`record_voice`	`duration_sec`, `filename`	Records audio from the microphones to a PCM file.
`download_file`	`url`, `filename`, `use_sd`	Memory-efficient download stream to flash or microSD.
`write_file`	`path`, `content`	Writes text file to board flash.
`read_file`	`path`	Reads text file from board flash.
`execute_python`	`code`	Executes arbitrary Python code dynamically.

7.6 KiB Raw Blame History