tactility_apps/Apps/McpScreen/IMPLEMENTATION_PLAN.md

McpScreen for Tactility — Implementation Plan

Goal

Port the hardware-agnostic MCP Screen design from:

/Users/adolforeyna/Projects/MicroPython/test1/Screen

into a native Tactility application while preserving compatibility with the existing host-side MCP bridge and Hermes voice service.

The original reference documents and implementations are:

• system_design_specs.md
• mcp_server.py
• mcp_bridge.py
• main.py
• video_stream.py
• voice_assistant.py
• lib/audio_util.py

Target application

• Source folder: Apps/McpScreen
• App ID: one.tactility.mcpscreen
• Initial platform: ESP32-S3
• Initial Tactility SDK: 0.7.0-dev
• Language: C/C++ with ESP-IDF, FreeRTOS, Tactility SDK, and LVGL

Current implementation status

Phase 1 is implemented, builds successfully for ESP32-S3, and has been tested on a 320x240 Tactility device at runtime.

Completed:

• Tactility app lifecycle and app-owned drawing area
• HTTP JSON-RPC endpoint on port 80
• tools/list
• get_capabilities
• clear_screen
• draw_text
• request-size limits and JSON-RPC error responses
• live tools/list and get_capabilities calls
• live clear_screen and draw_text calls
• live malformed-JSON error handling
• crash-safe HTTP worker shutdown during app destruction
• three forced stop/start cycles with the management and MCP endpoints still responsive afterward

Blocked by the current external-app ABI:

• UDP discovery requires lwip_recvfrom and lwip_sendto, which the running Tactility 0.7.0-dev firmware does not export to ELF apps. The existing bridge falls back to HTTP subnet scanning, so MCP connectivity remains available.

Pending verification:

• verify service behavior while the app is genuinely hidden behind another running application. A remote HelloWorld launch did not trigger onHide, so this needs a manual launcher/device interaction test.

Shutdown implementation note

Tactility unloads external-app ELF memory immediately after onDestroy returns. An app task blocked in accept() therefore cannot be allowed to survive destruction. The HTTP listener is non-blocking and client reads have short timeouts. onHide only clears the run flag and returns immediately so it does not block Tactility's GUI lifecycle or close an lwIP descriptor from the wrong task. The worker owns the socket shutdown, then suspends itself asynchronously. The next onShow reaps it before starting a new worker, while onDestroy waits for and deletes it before returning.

Design principles

1. Preserve the existing MCP tool names and JSON-RPC contract wherever the underlying Tactility hardware supports them.
2. Keep network, audio, and long-running work outside the LVGL task.
3. Perform all LVGL operations under the Tactility LVGL lock or dispatch them onto the LVGL task.
4. Use Tactility device and HAL interfaces instead of board-specific GPIO assumptions wherever possible.
5. Return structured unsupported-capability errors instead of silently emulating unavailable hardware.
6. Treat files supplied over MCP as untrusted input and constrain file access to the application's user-data directory.
7. Do not port execute_python literally. Native firmware must not expose arbitrary code execution.

Proposed application structure

Apps/McpScreen/
├── CMakeLists.txt
├── manifest.properties
├── IMPLEMENTATION_PLAN.md
└── main/
    ├── CMakeLists.txt
    └── Source/
        ├── main.cpp
        ├── McpScreenApp.cpp
        ├── McpScreenApp.h
        ├── McpServer.cpp
        ├── McpServer.h
        ├── DiscoveryService.cpp
        ├── DiscoveryService.h
        ├── ToolRegistry.cpp
        ├── ToolRegistry.h
        ├── ToolDispatcher.cpp
        ├── ToolDispatcher.h
        ├── DisplayTools.cpp
        ├── DisplayTools.h
        ├── AudioTools.cpp
        ├── AudioTools.h
        ├── SystemTools.cpp
        ├── SystemTools.h
        ├── DashboardView.cpp
        └── DashboardView.h

The file boundaries may be consolidated during the first slice if a smaller implementation is easier to validate.

Runtime architecture

Application lifecycle

• onCreate
  • Allocate application state.
• onShow
  • Build the dashboard and MCP-controlled display canvas.
  • Attach the current UI objects to the shared application state.
  • Start the HTTP MCP service.
• onHide
  • Signal the HTTP MCP service to stop without blocking the GUI lifecycle.
  • Detach and invalidate UI object pointers.
• onDestroy
  • Join/delete any remaining worker and close sockets as a safety fallback.
  • Release buffers, devices, and application state.

The MCP server is deliberately foreground-only. Tactility does not need to keep the listener or display-control task active while another app owns the screen.

FreeRTOS responsibilities

• MCP HTTP task
  • Listen for HTTP requests.
  • Parse JSON-RPC requests.
  • Dispatch tools.
  • Serialize JSON-RPC responses.
• UDP discovery task
  • Bind UDP port 5000.
  • Reply to discovery packets.
• Display work
  • Marshal changes through the LVGL lock or an LVGL async callback.
• Audio tasks
  • Record and play I2S data without blocking the UI or MCP listener.
• Future streaming tasks
  • Own TCP/UDP frame sockets and preallocated frame buffers.

Network compatibility

UDP discovery

• Port: 5000
• Request: exact bytes DISCOVER_SCREEN
• Response: exact bytes SCREEN_IP_80

MCP HTTP endpoint

• Port: 80
• Method and path: POST /api/mcp
• Content type: application/json
• Protocol: JSON-RPC 2.0
• Required methods:
  • tools/list
  • tools/call

Raw display endpoint

Added after the basic MCP slice:

• Method and path: POST /api/screen/raw
• Query parameters: x, y, w, h
• Body: raw big-endian RGB565 bytes

MCP response behavior

Successful tool calls retain the existing shape:

{
  "jsonrpc": "2.0",
  "result": {
    "content": [
      {
        "type": "text",
        "text": "..."
      }
    ]
  },
  "id": 1
}

Errors use JSON-RPC error objects:

{
  "jsonrpc": "2.0",
  "error": {
    "code": -32000,
    "message": "..."
  },
  "id": 1
}

Phase 1 — MCP foundation

Scope

1. Scaffold Apps/McpScreen.
2. Add a status/dashboard view showing:
   • Wi-Fi state
   • server state
   • port
   • last tool
   • last error
3. Start the HTTP server on port 80.
4. Implement POST /api/mcp.
5. Implement JSON-RPC request validation and error handling.
6. Implement tools/list.
7. Implement UDP discovery on port 5000.
8. Implement the first three tools:
   • get_capabilities
   • clear_screen
   • draw_text
9. Test with the existing mcp_bridge.py.

Initial tool semantics

get_capabilities

Return live display information:

{
  "color": true,
  "width": 320,
  "height": 240,
  "formats": ["rgb565_base64", "bmp_base64"],
  "platform": "tactility",
  "appVersion": "0.1.0"
}

Width, height, and color format must be queried at runtime.

clear_screen

• Accept color values compatible with the original API:
  • 0: white
  • 1: black
• Mark MCP display override active.
• Clear the app-owned canvas/display area.

draw_text

• Accept text, x, y, and optional size.
• Mark MCP display override active.
• Render inside the app-owned canvas.
• Clamp coordinates to the drawable area.

Phase 1 acceptance criteria

• The app builds and packages for ESP32-S3.
• Opening the app displays its server status.
• UDP discovery returns SCREEN_IP_80.
• The existing bridge can discover the device.
• tools/list returns valid schemas for the three initial tools.
• get_capabilities reports the actual Tactility display dimensions.
• clear_screen and draw_text visibly update the app.
• Malformed JSON and unknown methods return JSON-RPC errors without crashing.
• Repeated requests do not leak tasks, sockets, or request buffers.
• App hiding/showing behavior is documented from a device test.

Phase 2 — Display tools

Add:

• draw_raw_rgb565
• POST /api/screen/raw
• draw_color_bmp
• draw_image
• get_screenshot
• set_backlight
• set_screen_power

Display commands should target an app-owned LVGL canvas or image buffer. Direct panel access should only be used when the Tactility display HAL guarantees safe ownership and synchronization.

Image preprocessing may remain in mcp_bridge.py initially. That avoids large PNG/JPEG decoders and unnecessary memory pressure on the ESP32-S3.

Phase 3 — Audio tools

Port the working implementation from Apps/AudioTest:

• play_tone
• record_voice
• play_audio
• play_audio_base64

Audio baseline:

• 16 kHz
• 16-bit signed PCM
• mono
• I2S device discovered through device_find_by_name("i2s0")

Recordings should be streamed to the application's user-data directory instead of requiring one large fixed-duration RAM allocation.

Phase 4 — Hardware and system tools

Add where supported:

• get_touch
• get_battery
• set_led
• get_sensors
• scan_ble
• sync_time
• read_file
• write_file
• download_file

File operations must:

• resolve paths under the app user-data directory;
• reject absolute paths and traversal such as ../;
• enforce practical request and file-size limits.

execute_python is intentionally excluded. A future allow-listed diagnostic command tool may replace it if needed.

Phase 5 — Video streaming

Add:

• TCP frame server
• UDP chunked frame server
• stream activity timeout
• get_video_streaming_instructions
• get_stream_stats

The new protocol should advertise the actual Tactility display dimensions and native color format. The old RLCD-specific 15,000-byte monochrome mapping may be offered only as an optional compatibility mode.

Use preallocated buffers and avoid per-frame heap allocation.

Phase 6 — Dashboard and Hermes voice assistant

Add:

• five-second dashboard refresh;
• MCP override mode;
• an explicit local action to leave override mode;
• Hermes WebSocket client;
• touch-to-talk;
• 16 kHz, 16-bit mono microphone streaming;
• incoming PCM/WAV playback;
• transcript, thinking, listening, and speaking UI states.

Compatibility notes

• Keep the existing host bridge usable throughout development.
• Preserve tool names and argument names unless a compatibility defect is documented.
• draw_image can continue to be host-preprocessed into PBM or RGB565.
• Screenshot output may change from PBM to PNG/RGB565 if the bridge is updated to understand both.
• Hardware-specific tools should report support through get_capabilities.

Security and robustness limits

The first implementation should define conservative limits for:

• HTTP header size
• JSON body size
• base64 payload size
• socket read timeout
• simultaneous clients
• filename length
• file size
• text length
• drawing bounds
• audio duration

Only one display mutation should run at a time. Audio operations should also be serialized around the shared I2S device.

Build workflow

From the TactilityApps repository:

. /Users/adolforeyna/esp/esp-idf/export.sh
export TACTILITY_SDK_PATH=/Users/adolforeyna/.gemini/antigravity/scratch/tactility/release/TactilitySDK
python3 tactility.py Apps/McpScreen build esp32s3 --local-sdk

Immediate next task

Implement Phase 1 as a minimal vertical slice:

1. Scaffold the app and lifecycle.
2. Add a small server-status UI.
3. Add UDP discovery.
4. Add HTTP and JSON-RPC parsing.
5. Add tools/list.
6. Add get_capabilities, clear_screen, and draw_text.
7. Build.
8. Run a host smoke test through the existing bridge.