# Bridging the Physical & Virtual: Building a MicroPython MCP Server for Local AI Harnesses Large Language Models (LLMs) are incredibly capable, but they are traditionally trapped inside a digital sandbox—unable to perceive or interact with the physical world. Enter the **Model Context Protocol (MCP)**, an open standard developed by Anthropic that provides a unified way for LLMs to query external databases, run code, and invoke APIs. While most developers use MCP to hook LLMs to spreadsheets or GitHub repositories, we decided to take it a step further: **giving a local AI harness direct control over physical hardware using MicroPython.** In this post, we’ll walk through how we transformed a specialized ESP32-S3 development board into an interactive, local AI desktop companion capable of rendering graphics, playing audio streams, monitoring ambient environments, and executing remote python scripts—all driven natively by local LLM harnesses like **OpenClaude**, **Hermes**, or **Claude Desktop**. --- ## 1. The Hardware Platform: Waveshare ESP32-S3-RLCD-4.2 To build a true desktop companion, we needed a board that combined low power, rich displays, and audio capabilities. We selected the **Waveshare ESP32-S3-RLCD-4.2** development kit. Here is what makes this hardware special: * **Core SoC**: ESP32-S3 Dual-core Xtensa processor with Wi-Fi and Bluetooth. * **The Screen**: A 4.2" Reflective LCD (400x300 resolution). Similar to E-paper, it has extremely high sunlight readability, requires zero backlighting (it reflects ambient room light), and retains static images indefinitely with minimal power draw. * **Environmental Telemetry**: Onboard SHTC3 temperature and relative humidity sensor. * **Time & Battery**: PCF85063 hardware RTC for accurate offline timekeeping, and a MAX1704x-equivalent battery monitor. * **Interactive Input**: Onboard Key/Boot buttons and a passive Bluetooth Low Energy (BLE) radio for nearby device tracking. * **RGB LED**: A multi-mode WS2812 NeoPixel for state animations. * **The Audio Pipeline**: An **ES7210 4-channel microphone array** for capturing voice commands, paired with an **ES8311 audio decoder/speaker driver** and built-in amplifier. * **Storage Expansion**: A microSD card slot to cache large media files like podcasts. --- ## 2. MicroPython MCP Architecture: Stdio-to-HTTP Bridge Typical desktop LLMs interact with MCP servers locally over standard input/output (stdio) pipelines. However, running a full node environment or complex stdio protocol parser directly on a resource-constrained microcontroller is impractical. To bridge this gap, we designed a **hybrid stdio-to-network bridge**: ``` [ Local LLM Client / Harness ] │ (JSON-RPC over Stdio) ▼ [ host_bridge.py ] <─── (Pillow converts PNGs/PBMs on-the-fly) │ (JSON-RPC over HTTP POST) ▼ [ ESP32-S3 Web Server ] │ (MicroPython execution) ▼ [ Hardware Peripherals ] ``` 1. **On-Board HTTP Server**: A lightweight, non-blocking TCP socket server written in MicroPython (`mcp_server.py`) runs directly on the ESP32, listening on port `80`. It handles `POST /api/mcp` requests, parsing JSON-RPC 2.0 structures. 2. **Host-Side Stdio Bridge**: A zero-dependency Python script (`mcp_bridge.py`) runs on the host PC. It reads stdio packets from the LLM client, forwards them over the local Wi-Fi to the ESP32’s IP address, and relays the board's responses back. 3. **On-the-Fly Media Conversion**: Since the ESP32 has limited RAM, it returns screenshots as raw binary 1-bit PBM data. The host bridge intercepts this data, converts it into standard PNG format on your PC using native image tools (`sips` or `Pillow`), and serves it to the LLM. It does the reverse for image uploads (`draw_image`), rendering them locally as 1-bit dithered PBMs before sending them to the microcontroller. --- ## 3. The Tool Belt: 15 Exposed Hardware Utilities By connecting the board to the MCP server, we exposed 15 specialized tools to the local AI. The LLM can choose to invoke any of these actions in response to your prompts: ### Display & Graphics 1. **`clear_screen(color)`**: Clear the 400x300 reflective display to pure white or black. 2. **`draw_text(text, x, y, size)`**: Draw custom labels or telemetry readouts at specified coordinates. 3. **`draw_image(image_base64, x, y, dither)`**: Upload any standard image format (JPEG/PNG/GIF). The host bridge automatically resizes and dithers it, and the board displays it. 4. **`get_screenshot()`**: Captures the exact reflective LCD frame buffer and returns a PNG. **This allows the LLM to inspect the screen and confirm if its renders are correct!** ### Environment & System Telemetry 5. **`get_sensors()`**: Returns live temperature (°C) and humidity (%) from the SHTC3 sensor. 6. **`get_battery()`**: Reads the current battery voltage and estimated capacity percentage. 7. **`scan_ble(duration_ms)`**: Performs a BLE scan to log nearby smart tags or trackers. ### Audio & Sound Control 8. **`play_tone(frequency, duration_ms, volume)`**: Play a clean, synthesized sine wave tone. 9. **`play_audio(filename, volume)`**: Stream and play standard WAV files (e.g. system alerts, sound effects) from the board's storage. 10. **`play_audio_base64(wav_base64, volume)`**: Allows the LLM to synthesize speech locally on your computer and stream it to the board's speaker in a single tool call (auto-caching and auto-cleaning temporary flash memory). 11. **`record_voice(duration_sec, filename)`**: Activates the microphone array to record your voice to flash memory for voice command loops. ### Remote File & Code Execution 12. **`write_file(path, content)`**: Write configuration text or script scripts to flash. 13. **`read_file(path)`**: Read any file on the board's local filesystem. 14. **`execute_python(code)`**: Run arbitrary Python code dynamically on the board via `exec()`. Standard output and tracebacks are captured and returned. 15. **`download_file(url, filename, use_sd)`**: Downloads large media files (like music or podcasts) from the web directly to the microSD card in memory-efficient stream chunks. --- ## 4. How to Register the MCP Server on Local AI Harnesses Adding this ESP32-S3 companion to your local AI environment is simple. ### Step 1: Discover the Device IP Ensure your ESP32 board is connected to the same local Wi-Fi router. Upon boot, the board's screen will display its assigned local IP address (e.g., `192.168.68.122`). ### Step 2: Register in Claude Desktop or Hermes/OpenClaude Open your local harness config file. For Claude Desktop, this is located at: * **macOS**: `/Users//Library/Application Support/Claude/claude_desktop_config.json` * **Windows**: `%APPDATA%\Claude\claude_desktop_config.json` Add the bridge script under the `mcpServers` entry, specifying the board's IP address: ```json { "mcpServers": { "esp32-rlcd-companion": { "command": "python3", "args": [ "/absolute/path/to/your/workspace/mcp_bridge.py", "--ip", "192.168.68.122" ] } } } ``` ### Step 3: Launch and Interact Restart your local AI harness. You will see a small plug icon or tools notification indicating that the board's functions are registered. You can now prompt the AI using natural language commands: * *"Check the room temperature. If it's over 25 degrees, set the LED to breathing red and show a warning message on the screen."* * *"Draw a cute dithered pixel-art cat at the center of the display, then take a screenshot so you can check how it looks."* * *"Download the podcast at this WAV URL to the SD card, and then play it on the speaker at 60% volume."* The LLM will translate these requests into tool calls, control your hardware over the Wi-Fi bridge, and report back! --- ## Conclusion By mapping hardware control to standard MCP tool schemas, we've demonstrated how easily microcontrollers can be integrated into the modern agentic LLM ecosystem. The ESP32-S3 is no longer just a standalone sensor node—it is a physical avatar for your local AI. Whether you want to build a smart e-paper calendar, an interactive virtual desk pet, or an offline voice assistant, MicroPython combined with MCP provides the ultimate foundation.