ESP32-S3-RLCD-4.2 Model Context Protocol (MCP) Companion

This repository contains the MicroPython firmware and host-side bridge script to expose the Waveshare ESP32-S3-RLCD-4.2 development board as a physical, Model Context Protocol (MCP)-compliant agentic companion.

With this setup, local AI harnesses (such as Claude Desktop, OpenClaude, Cursor, or Hermes) can directly control the board's screen, NeoPixels, audio codec, microphone array, sensors, and remote filesystem.

---

1. System Architecture

To bypass the memory and protocol constraints of the microcontroller, we use a hybrid Stdio-to-HTTP LAN Bridge:

┌──────────────────────────────┐
│  Local LLM Client / Harness  │
└──────────────┬───────────────┘
               │ (JSON-RPC over Stdio)
               ▼
┌──────────────────────────────┐
│       mcp_bridge.py          │  <── (Converts PNG/PBM images on-the-fly)
└──────────────┬───────────────┘
               │ (JSON-RPC over HTTP POST)
               ▼
┌──────────────────────────────┐
│   ESP32-S3 HTTP Server       │  (Running on Port 80)
└──────────────┬───────────────┘
               │ (MicroPython calls)
               ▼
┌──────────────────────────────┐
│   Hardware Peripherals       │  (Display, Led, Mic, Speaker, Sensors)
└──────────────────────────────┘

1. MicroPython HTTP Server (mcp_server.py): Runs directly on the ESP32-S3, accepting JSON-RPC 2.0 requests at POST /api/mcp.
2. Host Stdio Bridge (mcp_bridge.py): Runs on your host PC, listening for stdio JSON-RPC calls, forwarding them to the board over Wi-Fi, and returning the output.
3. On-the-Fly Image Conversion: The board returns raw dithered PBM screen buffers to save bandwidth. The host bridge automatically converts these into standard PNG images (using macOS sips or Pillow) so the LLM can "see" what is currently rendered on the screen.

---

2. Directory Structure & File Manifest

MicroPython Device Files

• boot.py: Automatically connects to Wi-Fi using credentials in wifi_config.py and renders connection logs on the screen.
• main.py: Background execution loop. Handlers for buttons (manual dashboard refresh, LED state cycle), NeoPixel animation loop, and non-blocking MCP server requests.
• mcp_server.py: The lightweight JSON-RPC server implementing the MCP tools list and call handlers.
• audio_util.py: Audio drivers for the ES7210 microphone array and the ES8311 speaker DAC. Implements sine-wave tone generation (play_tone), voice recording (record_audio), and file-based audio playing (play_wav).
• rlcd.py: Low-level FrameBuffer driver for the 4.2" Reflective LCD, including PBM loaders and screenshot exporters.
• sd_util.py: Mounting and filesystem management utility for the onboard microSD card slot.
• Hardware Drivers:
  • shtc3_util.py (Temperature & Humidity)
  • rtc_util.py (Hardware Real-Time Clock)
  • battery_util.py (Voltage & Capacity reader)
  • rgb_led_util.py (WS2812 NeoPixel animations)
  • button_util.py (Key & Boot button debouncer)
  • ble_util.py (Passive BLE scanning & UART)
• wifi_config.py: Wi-Fi credentials. (Do not commit real credentials to Git).

Host-Side files

• mcp_bridge.py: The stdio-to-HTTP LAN bridge connecting the LLM client to the board. Handles image resizing, dithering, and formatting.
• voice_assistant.py: An end-to-end local voice assistant loop script (requires OpenAI/local Whisper for STT and Mac TTS commands).
• blog_post.md: A draft technical write-up detailing this project's architecture and capabilities.

---

3. Quickstart Guide

Step 1: Configure Wi-Fi

Edit wifi_config.py on your computer and set your local SSID and Password:

WIFI_SSID = "Your_Wi-Fi_Name"
WIFI_PASS = "Your_Password"

Step 2: Upload Files to the Board

Connect the board to your computer via USB (making sure to use the active ESP32 USB port). Upload all Python files to the device flash memory using mpremote:

mpremote connect /dev/cu.usbmodem101 cp *.py :

(If your serial port differs, change /dev/cu.usbmodem101 accordingly).

Step 3: Boot the Board

Reset the board to apply changes:

mpremote connect /dev/cu.usbmodem101 reset

Once booted, the reflective screen will display connection logs, followed by the local IP address (e.g. 192.168.68.122) once connected to Wi-Fi.

Step 4: Register in Claude Desktop

Open your local Claude Desktop config file:

• macOS: /Users/<username>/Library/Application Support/Claude/claude_desktop_config.json
• Windows: %APPDATA%\Claude\claude_desktop_config.json

Add the bridge script under the mcpServers list:

{
  "mcpServers": {
    "esp32-rlcd": {
      "command": "python3",
      "args": [
        "/absolute/path/to/this/repository/mcp_bridge.py",
        "--ip",
        "192.168.68.122"
      ]
    }
  }
}

Restart Claude Desktop, and the assistant will have direct, real-time control over your physical device!

---

4. MCP Tools Reference

Here is a summary of the MCP tools exposed by the bridge:

Tool Name	Parameters	Description
clear_screen	color (0=White, 1=Black)	Clears the RLCD display.
draw_text	text, x, y, size	Draws normal/large text on the display.
draw_image	image_base64, x, y, dither	Uploads and dithers an image to the display.
get_screenshot	(None)	Captures the screen buffer as a PNG image for the LLM.
get_sensors	(None)	Returns temperature & humidity from the SHTC3.
get_battery	(None)	Returns voltage and capacity percentage.
scan_ble	duration_ms	Scans for nearby BLE beacons / smart tags.
play_tone	frequency, duration_ms, volume	Plays a pure sine wave tone.
play_audio	filename, volume	Plays a WAV file from the flash storage or SD.
play_audio_base64	wav_base64, volume	Plays a base64 WAV stream (automatic caching).
record_voice	duration_sec, filename	Records audio from the microphones to a PCM file.
download_file	url, filename, use_sd	Memory-efficient download stream to flash or microSD.
write_file	path, content	Writes text file to board flash.
read_file	path	Reads text file from board flash.
execute_python	code	Executes arbitrary Python code dynamically.