docs: add McpScreen implementation plan

2026-06-24 23:19:03 -04:00
parent e5233efdf3
commit fb800bb5bd
1 changed files with 385 additions and 0 deletions
@@ -0,0 +1,385 @@
 # McpScreen for Tactility — Implementation Plan
 ## Goal
 Port the hardware-agnostic MCP Screen design from:
 `/Users/adolforeyna/Projects/MicroPython/test1/Screen`
 into a native Tactility application while preserving compatibility with the
 existing host-side MCP bridge and Hermes voice service.
 The original reference documents and implementations are:
 - `system_design_specs.md`
 - `mcp_server.py`
 - `mcp_bridge.py`
 - `main.py`
 - `video_stream.py`
 - `voice_assistant.py`
 - `lib/audio_util.py`
 ## Target application
 - Source folder: `Apps/McpScreen`
 - App ID: `one.tactility.mcpscreen`
 - Initial platform: ESP32-S3
 - Initial Tactility SDK: `0.7.0-dev`
 - Language: C/C++ with ESP-IDF, FreeRTOS, Tactility SDK, and LVGL
 ## Design principles
 1. Preserve the existing MCP tool names and JSON-RPC contract wherever the
   underlying Tactility hardware supports them.
 2. Keep network, audio, and long-running work outside the LVGL task.
 3. Perform all LVGL operations under the Tactility LVGL lock or dispatch them
   onto the LVGL task.
 4. Use Tactility device and HAL interfaces instead of board-specific GPIO
   assumptions wherever possible.
 5. Return structured unsupported-capability errors instead of silently
   emulating unavailable hardware.
 6. Treat files supplied over MCP as untrusted input and constrain file access
   to the application's user-data directory.
 7. Do not port `execute_python` literally. Native firmware must not expose
   arbitrary code execution.
 ## Proposed application structure
 ```text
 Apps/McpScreen/
 ├── CMakeLists.txt
 ├── manifest.properties
 ├── IMPLEMENTATION_PLAN.md
 └── main/
    ├── CMakeLists.txt
    └── Source/
        ├── main.cpp
        ├── McpScreenApp.cpp
        ├── McpScreenApp.h
        ├── McpServer.cpp
        ├── McpServer.h
        ├── DiscoveryService.cpp
        ├── DiscoveryService.h
        ├── ToolRegistry.cpp
        ├── ToolRegistry.h
        ├── ToolDispatcher.cpp
        ├── ToolDispatcher.h
        ├── DisplayTools.cpp
        ├── DisplayTools.h
        ├── AudioTools.cpp
        ├── AudioTools.h
        ├── SystemTools.cpp
        ├── SystemTools.h
        ├── DashboardView.cpp
        └── DashboardView.h
 ```
 The file boundaries may be consolidated during the first slice if a smaller
 implementation is easier to validate.
 ## Runtime architecture
 ### Application lifecycle
 - `onCreate`
  - Allocate application state.
  - Start UDP discovery and HTTP MCP service tasks.
 - `onShow`
  - Build the dashboard and MCP-controlled display canvas.
  - Attach the current UI objects to the shared application state.
 - `onHide`
  - Detach and invalidate UI object pointers.
  - Keep network services running while the application is merely hidden, if
    Tactility preserves the app instance.
 - `onDestroy`
  - Stop tasks and sockets.
  - Release buffers, devices, and application state.
 The first implementation slice must explicitly verify whether Tactility keeps
 the app process and `onCreate` state alive while another app is visible.
 ### FreeRTOS responsibilities
 - MCP HTTP task
  - Listen for HTTP requests.
  - Parse JSON-RPC requests.
  - Dispatch tools.
  - Serialize JSON-RPC responses.
 - UDP discovery task
  - Bind UDP port 5000.
  - Reply to discovery packets.
 - Display work
  - Marshal changes through the LVGL lock or an LVGL async callback.
 - Audio tasks
  - Record and play I2S data without blocking the UI or MCP listener.
 - Future streaming tasks
  - Own TCP/UDP frame sockets and preallocated frame buffers.
 ## Network compatibility
 ### UDP discovery
 - Port: `5000`
 - Request: exact bytes `DISCOVER_SCREEN`
 - Response: exact bytes `SCREEN_IP_80`
 ### MCP HTTP endpoint
 - Port: `80`
 - Method and path: `POST /api/mcp`
 - Content type: `application/json`
 - Protocol: JSON-RPC 2.0
 - Required methods:
  - `tools/list`
  - `tools/call`
 ### Raw display endpoint
 Added after the basic MCP slice:
 - Method and path: `POST /api/screen/raw`
 - Query parameters: `x`, `y`, `w`, `h`
 - Body: raw big-endian RGB565 bytes
 ## MCP response behavior
 Successful tool calls retain the existing shape:
 ```json
 {
  "jsonrpc": "2.0",
  "result": {
    "content": [
      {
        "type": "text",
        "text": "..."
      }
    ]
  },
  "id": 1
 }
 ```
 Errors use JSON-RPC error objects:
 ```json
 {
  "jsonrpc": "2.0",
  "error": {
    "code": -32000,
    "message": "..."
  },
  "id": 1
 }
 ```
 ## Phase 1 — MCP foundation
 ### Scope
 1. Scaffold `Apps/McpScreen`.
 2. Add a status/dashboard view showing:
   - Wi-Fi state
   - server state
   - port
   - last tool
   - last error
 3. Start the HTTP server on port 80.
 4. Implement `POST /api/mcp`.
 5. Implement JSON-RPC request validation and error handling.
 6. Implement `tools/list`.
 7. Implement UDP discovery on port 5000.
 8. Implement the first three tools:
   - `get_capabilities`
   - `clear_screen`
   - `draw_text`
 9. Test with the existing `mcp_bridge.py`.
 ### Initial tool semantics
 #### `get_capabilities`
 Return live display information:
 ```json
 {
  "color": true,
  "width": 320,
  "height": 240,
  "formats": ["rgb565_base64", "bmp_base64"],
  "platform": "tactility",
  "appVersion": "0.1.0"
 }
 ```
 Width, height, and color format must be queried at runtime.
 #### `clear_screen`
 - Accept `color` values compatible with the original API:
  - `0`: white
  - `1`: black
 - Mark MCP display override active.
 - Clear the app-owned canvas/display area.
 #### `draw_text`
 - Accept `text`, `x`, `y`, and optional `size`.
 - Mark MCP display override active.
 - Render inside the app-owned canvas.
 - Clamp coordinates to the drawable area.
 ### Phase 1 acceptance criteria
 - The app builds and packages for ESP32-S3.
 - Opening the app displays its server status.
 - UDP discovery returns `SCREEN_IP_80`.
 - The existing bridge can discover the device.
 - `tools/list` returns valid schemas for the three initial tools.
 - `get_capabilities` reports the actual Tactility display dimensions.
 - `clear_screen` and `draw_text` visibly update the app.
 - Malformed JSON and unknown methods return JSON-RPC errors without crashing.
 - Repeated requests do not leak tasks, sockets, or request buffers.
 - App hiding/showing behavior is documented from a device test.
 ## Phase 2 — Display tools
 Add:
 - `draw_raw_rgb565`
 - `POST /api/screen/raw`
 - `draw_color_bmp`
 - `draw_image`
 - `get_screenshot`
 - `set_backlight`
 - `set_screen_power`
 Display commands should target an app-owned LVGL canvas or image buffer. Direct
 panel access should only be used when the Tactility display HAL guarantees safe
 ownership and synchronization.
 Image preprocessing may remain in `mcp_bridge.py` initially. That avoids large
 PNG/JPEG decoders and unnecessary memory pressure on the ESP32-S3.
 ## Phase 3 — Audio tools
 Port the working implementation from `Apps/AudioTest`:
 - `play_tone`
 - `record_voice`
 - `play_audio`
 - `play_audio_base64`
 Audio baseline:
 - 16 kHz
 - 16-bit signed PCM
 - mono
 - I2S device discovered through `device_find_by_name("i2s0")`
 Recordings should be streamed to the application's user-data directory instead
 of requiring one large fixed-duration RAM allocation.
 ## Phase 4 — Hardware and system tools
 Add where supported:
 - `get_touch`
 - `get_battery`
 - `set_led`
 - `get_sensors`
 - `scan_ble`
 - `sync_time`
 - `read_file`
 - `write_file`
 - `download_file`
 File operations must:
 - resolve paths under the app user-data directory;
 - reject absolute paths and traversal such as `../`;
 - enforce practical request and file-size limits.
 `execute_python` is intentionally excluded. A future allow-listed diagnostic
 command tool may replace it if needed.
 ## Phase 5 — Video streaming
 Add:
 - TCP frame server
 - UDP chunked frame server
 - stream activity timeout
 - `get_video_streaming_instructions`
 - `get_stream_stats`
 The new protocol should advertise the actual Tactility display dimensions and
 native color format. The old RLCD-specific 15,000-byte monochrome mapping may
 be offered only as an optional compatibility mode.
 Use preallocated buffers and avoid per-frame heap allocation.
 ## Phase 6 — Dashboard and Hermes voice assistant
 Add:
 - five-second dashboard refresh;
 - MCP override mode;
 - an explicit local action to leave override mode;
 - Hermes WebSocket client;
 - touch-to-talk;
 - 16 kHz, 16-bit mono microphone streaming;
 - incoming PCM/WAV playback;
 - transcript, thinking, listening, and speaking UI states.
 ## Compatibility notes
 - Keep the existing host bridge usable throughout development.
 - Preserve tool names and argument names unless a compatibility defect is
  documented.
 - `draw_image` can continue to be host-preprocessed into PBM or RGB565.
 - Screenshot output may change from PBM to PNG/RGB565 if the bridge is updated
  to understand both.
 - Hardware-specific tools should report support through `get_capabilities`.
 ## Security and robustness limits
 The first implementation should define conservative limits for:
 - HTTP header size
 - JSON body size
 - base64 payload size
 - socket read timeout
 - simultaneous clients
 - filename length
 - file size
 - text length
 - drawing bounds
 - audio duration
 Only one display mutation should run at a time. Audio operations should also be
 serialized around the shared I2S device.
 ## Build workflow
 From the TactilityApps repository:
 ```bash
 . /Users/adolforeyna/esp/esp-idf/export.sh
 export TACTILITY_SDK_PATH=/Users/adolforeyna/.gemini/antigravity/scratch/tactility/release/TactilitySDK
 python3 tactility.py Apps/McpScreen build esp32s3 --local-sdk
 ```
 ## Immediate next task
 Implement Phase 1 as a minimal vertical slice:
 1. Scaffold the app and lifecycle.
 2. Add a small server-status UI.
 3. Add UDP discovery.
 4. Add HTTP and JSON-RPC parsing.
 5. Add `tools/list`.
 6. Add `get_capabilities`, `clear_screen`, and `draw_text`.
 7. Build.
 8. Run a host smoke test through the existing bridge.