diff --git a/Apps/McpScreen/IMPLEMENTATION_PLAN.md b/Apps/McpScreen/IMPLEMENTATION_PLAN.md new file mode 100644 index 0000000..3da76f7 --- /dev/null +++ b/Apps/McpScreen/IMPLEMENTATION_PLAN.md @@ -0,0 +1,385 @@ +# McpScreen for Tactility — Implementation Plan + +## Goal + +Port the hardware-agnostic MCP Screen design from: + +`/Users/adolforeyna/Projects/MicroPython/test1/Screen` + +into a native Tactility application while preserving compatibility with the +existing host-side MCP bridge and Hermes voice service. + +The original reference documents and implementations are: + +- `system_design_specs.md` +- `mcp_server.py` +- `mcp_bridge.py` +- `main.py` +- `video_stream.py` +- `voice_assistant.py` +- `lib/audio_util.py` + +## Target application + +- Source folder: `Apps/McpScreen` +- App ID: `one.tactility.mcpscreen` +- Initial platform: ESP32-S3 +- Initial Tactility SDK: `0.7.0-dev` +- Language: C/C++ with ESP-IDF, FreeRTOS, Tactility SDK, and LVGL + +## Design principles + +1. Preserve the existing MCP tool names and JSON-RPC contract wherever the + underlying Tactility hardware supports them. +2. Keep network, audio, and long-running work outside the LVGL task. +3. Perform all LVGL operations under the Tactility LVGL lock or dispatch them + onto the LVGL task. +4. Use Tactility device and HAL interfaces instead of board-specific GPIO + assumptions wherever possible. +5. Return structured unsupported-capability errors instead of silently + emulating unavailable hardware. +6. Treat files supplied over MCP as untrusted input and constrain file access + to the application's user-data directory. +7. Do not port `execute_python` literally. Native firmware must not expose + arbitrary code execution. + +## Proposed application structure + +```text +Apps/McpScreen/ +├── CMakeLists.txt +├── manifest.properties +├── IMPLEMENTATION_PLAN.md +└── main/ + ├── CMakeLists.txt + └── Source/ + ├── main.cpp + ├── McpScreenApp.cpp + ├── McpScreenApp.h + ├── McpServer.cpp + ├── McpServer.h + ├── DiscoveryService.cpp + ├── DiscoveryService.h + ├── ToolRegistry.cpp + ├── ToolRegistry.h + ├── ToolDispatcher.cpp + ├── ToolDispatcher.h + ├── DisplayTools.cpp + ├── DisplayTools.h + ├── AudioTools.cpp + ├── AudioTools.h + ├── SystemTools.cpp + ├── SystemTools.h + ├── DashboardView.cpp + └── DashboardView.h +``` + +The file boundaries may be consolidated during the first slice if a smaller +implementation is easier to validate. + +## Runtime architecture + +### Application lifecycle + +- `onCreate` + - Allocate application state. + - Start UDP discovery and HTTP MCP service tasks. +- `onShow` + - Build the dashboard and MCP-controlled display canvas. + - Attach the current UI objects to the shared application state. +- `onHide` + - Detach and invalidate UI object pointers. + - Keep network services running while the application is merely hidden, if + Tactility preserves the app instance. +- `onDestroy` + - Stop tasks and sockets. + - Release buffers, devices, and application state. + +The first implementation slice must explicitly verify whether Tactility keeps +the app process and `onCreate` state alive while another app is visible. + +### FreeRTOS responsibilities + +- MCP HTTP task + - Listen for HTTP requests. + - Parse JSON-RPC requests. + - Dispatch tools. + - Serialize JSON-RPC responses. +- UDP discovery task + - Bind UDP port 5000. + - Reply to discovery packets. +- Display work + - Marshal changes through the LVGL lock or an LVGL async callback. +- Audio tasks + - Record and play I2S data without blocking the UI or MCP listener. +- Future streaming tasks + - Own TCP/UDP frame sockets and preallocated frame buffers. + +## Network compatibility + +### UDP discovery + +- Port: `5000` +- Request: exact bytes `DISCOVER_SCREEN` +- Response: exact bytes `SCREEN_IP_80` + +### MCP HTTP endpoint + +- Port: `80` +- Method and path: `POST /api/mcp` +- Content type: `application/json` +- Protocol: JSON-RPC 2.0 +- Required methods: + - `tools/list` + - `tools/call` + +### Raw display endpoint + +Added after the basic MCP slice: + +- Method and path: `POST /api/screen/raw` +- Query parameters: `x`, `y`, `w`, `h` +- Body: raw big-endian RGB565 bytes + +## MCP response behavior + +Successful tool calls retain the existing shape: + +```json +{ + "jsonrpc": "2.0", + "result": { + "content": [ + { + "type": "text", + "text": "..." + } + ] + }, + "id": 1 +} +``` + +Errors use JSON-RPC error objects: + +```json +{ + "jsonrpc": "2.0", + "error": { + "code": -32000, + "message": "..." + }, + "id": 1 +} +``` + +## Phase 1 — MCP foundation + +### Scope + +1. Scaffold `Apps/McpScreen`. +2. Add a status/dashboard view showing: + - Wi-Fi state + - server state + - port + - last tool + - last error +3. Start the HTTP server on port 80. +4. Implement `POST /api/mcp`. +5. Implement JSON-RPC request validation and error handling. +6. Implement `tools/list`. +7. Implement UDP discovery on port 5000. +8. Implement the first three tools: + - `get_capabilities` + - `clear_screen` + - `draw_text` +9. Test with the existing `mcp_bridge.py`. + +### Initial tool semantics + +#### `get_capabilities` + +Return live display information: + +```json +{ + "color": true, + "width": 320, + "height": 240, + "formats": ["rgb565_base64", "bmp_base64"], + "platform": "tactility", + "appVersion": "0.1.0" +} +``` + +Width, height, and color format must be queried at runtime. + +#### `clear_screen` + +- Accept `color` values compatible with the original API: + - `0`: white + - `1`: black +- Mark MCP display override active. +- Clear the app-owned canvas/display area. + +#### `draw_text` + +- Accept `text`, `x`, `y`, and optional `size`. +- Mark MCP display override active. +- Render inside the app-owned canvas. +- Clamp coordinates to the drawable area. + +### Phase 1 acceptance criteria + +- The app builds and packages for ESP32-S3. +- Opening the app displays its server status. +- UDP discovery returns `SCREEN_IP_80`. +- The existing bridge can discover the device. +- `tools/list` returns valid schemas for the three initial tools. +- `get_capabilities` reports the actual Tactility display dimensions. +- `clear_screen` and `draw_text` visibly update the app. +- Malformed JSON and unknown methods return JSON-RPC errors without crashing. +- Repeated requests do not leak tasks, sockets, or request buffers. +- App hiding/showing behavior is documented from a device test. + +## Phase 2 — Display tools + +Add: + +- `draw_raw_rgb565` +- `POST /api/screen/raw` +- `draw_color_bmp` +- `draw_image` +- `get_screenshot` +- `set_backlight` +- `set_screen_power` + +Display commands should target an app-owned LVGL canvas or image buffer. Direct +panel access should only be used when the Tactility display HAL guarantees safe +ownership and synchronization. + +Image preprocessing may remain in `mcp_bridge.py` initially. That avoids large +PNG/JPEG decoders and unnecessary memory pressure on the ESP32-S3. + +## Phase 3 — Audio tools + +Port the working implementation from `Apps/AudioTest`: + +- `play_tone` +- `record_voice` +- `play_audio` +- `play_audio_base64` + +Audio baseline: + +- 16 kHz +- 16-bit signed PCM +- mono +- I2S device discovered through `device_find_by_name("i2s0")` + +Recordings should be streamed to the application's user-data directory instead +of requiring one large fixed-duration RAM allocation. + +## Phase 4 — Hardware and system tools + +Add where supported: + +- `get_touch` +- `get_battery` +- `set_led` +- `get_sensors` +- `scan_ble` +- `sync_time` +- `read_file` +- `write_file` +- `download_file` + +File operations must: + +- resolve paths under the app user-data directory; +- reject absolute paths and traversal such as `../`; +- enforce practical request and file-size limits. + +`execute_python` is intentionally excluded. A future allow-listed diagnostic +command tool may replace it if needed. + +## Phase 5 — Video streaming + +Add: + +- TCP frame server +- UDP chunked frame server +- stream activity timeout +- `get_video_streaming_instructions` +- `get_stream_stats` + +The new protocol should advertise the actual Tactility display dimensions and +native color format. The old RLCD-specific 15,000-byte monochrome mapping may +be offered only as an optional compatibility mode. + +Use preallocated buffers and avoid per-frame heap allocation. + +## Phase 6 — Dashboard and Hermes voice assistant + +Add: + +- five-second dashboard refresh; +- MCP override mode; +- an explicit local action to leave override mode; +- Hermes WebSocket client; +- touch-to-talk; +- 16 kHz, 16-bit mono microphone streaming; +- incoming PCM/WAV playback; +- transcript, thinking, listening, and speaking UI states. + +## Compatibility notes + +- Keep the existing host bridge usable throughout development. +- Preserve tool names and argument names unless a compatibility defect is + documented. +- `draw_image` can continue to be host-preprocessed into PBM or RGB565. +- Screenshot output may change from PBM to PNG/RGB565 if the bridge is updated + to understand both. +- Hardware-specific tools should report support through `get_capabilities`. + +## Security and robustness limits + +The first implementation should define conservative limits for: + +- HTTP header size +- JSON body size +- base64 payload size +- socket read timeout +- simultaneous clients +- filename length +- file size +- text length +- drawing bounds +- audio duration + +Only one display mutation should run at a time. Audio operations should also be +serialized around the shared I2S device. + +## Build workflow + +From the TactilityApps repository: + +```bash +. /Users/adolforeyna/esp/esp-idf/export.sh +export TACTILITY_SDK_PATH=/Users/adolforeyna/.gemini/antigravity/scratch/tactility/release/TactilitySDK +python3 tactility.py Apps/McpScreen build esp32s3 --local-sdk +``` + +## Immediate next task + +Implement Phase 1 as a minimal vertical slice: + +1. Scaffold the app and lifecycle. +2. Add a small server-status UI. +3. Add UDP discovery. +4. Add HTTP and JSON-RPC parsing. +5. Add `tools/list`. +6. Add `get_capabilities`, `clear_screen`, and `draw_text`. +7. Build. +8. Run a host smoke test through the existing bridge. +