tactility_apps/Apps/McpScreen/IMPLEMENTATION_PLAN.md

# McpScreen for Tactility — Implementation Plan

## Goal

Port the hardware-agnostic MCP Screen design from:

`/Users/adolforeyna/Projects/MicroPython/test1/Screen`

into a native Tactility application while preserving compatibility with the
existing host-side MCP bridge and Hermes voice service.

The original reference documents and implementations are:

- `system_design_specs.md`
- `mcp_server.py`
- `mcp_bridge.py`
- `main.py`
- `video_stream.py`
- `voice_assistant.py`
- `lib/audio_util.py`

## Target application

- Source folder: `Apps/McpScreen`
- App ID: `one.tactility.mcpscreen`
- Initial platform: ESP32-S3
- Initial Tactility SDK: `0.7.0-dev`
- Language: C/C++ with ESP-IDF, FreeRTOS, Tactility SDK, and LVGL

## Current implementation status

Phase 1 is implemented, builds successfully for ESP32-S3, and has been tested
on a 320x240 Tactility device at runtime.

Completed:

- Tactility app lifecycle and app-owned drawing area
- HTTP JSON-RPC endpoint on port 80
- `tools/list`
- `get_capabilities`
- `clear_screen`
- `draw_text`
- request-size limits and JSON-RPC error responses
- live `tools/list` and `get_capabilities` calls
- live `clear_screen` and `draw_text` calls
- live malformed-JSON error handling
- crash-safe HTTP worker shutdown during app destruction
- three forced stop/start cycles with the management and MCP endpoints still
  responsive afterward

Blocked by the current external-app ABI:

- UDP discovery requires `lwip_recvfrom` and `lwip_sendto`, which the running
  Tactility 0.7.0-dev firmware does not export to ELF apps. The existing bridge
  falls back to HTTP subnet scanning, so MCP connectivity remains available.

Pending verification:

- verify service behavior while the app is genuinely hidden behind another
  running application. A remote HelloWorld launch did not trigger `onHide`, so
  this needs a manual launcher/device interaction test.

### Shutdown implementation note

Tactility unloads external-app ELF memory immediately after `onDestroy`
returns. An app task blocked in `accept()` therefore cannot be allowed to
survive destruction. The HTTP listener is non-blocking and client reads have
short timeouts. `onHide` only clears the run flag and returns immediately so
it does not block Tactility's GUI lifecycle or close an lwIP descriptor from
the wrong task. The worker owns the socket shutdown, then suspends itself
asynchronously. The next `onShow` reaps it before starting a new worker, while
`onDestroy` waits for and deletes it before returning.

## Design principles

1. Preserve the existing MCP tool names and JSON-RPC contract wherever the
   underlying Tactility hardware supports them.
2. Keep network, audio, and long-running work outside the LVGL task.
3. Perform all LVGL operations under the Tactility LVGL lock or dispatch them
   onto the LVGL task.
4. Use Tactility device and HAL interfaces instead of board-specific GPIO
   assumptions wherever possible.
5. Return structured unsupported-capability errors instead of silently
   emulating unavailable hardware.
6. Treat files supplied over MCP as untrusted input and constrain file access
   to the application's user-data directory.
7. Do not port `execute_python` literally. Native firmware must not expose
   arbitrary code execution.

## Proposed application structure

```text
Apps/McpScreen/
├── CMakeLists.txt
├── manifest.properties
├── IMPLEMENTATION_PLAN.md
└── main/
    ├── CMakeLists.txt
    └── Source/
        ├── main.cpp
        ├── McpScreenApp.cpp
        ├── McpScreenApp.h
        ├── McpServer.cpp
        ├── McpServer.h
        ├── DiscoveryService.cpp
        ├── DiscoveryService.h
        ├── ToolRegistry.cpp
        ├── ToolRegistry.h
        ├── ToolDispatcher.cpp
        ├── ToolDispatcher.h
        ├── DisplayTools.cpp
        ├── DisplayTools.h
        ├── AudioTools.cpp
        ├── AudioTools.h
        ├── SystemTools.cpp
        ├── SystemTools.h
        ├── DashboardView.cpp
        └── DashboardView.h
```

The file boundaries may be consolidated during the first slice if a smaller
implementation is easier to validate.

## Runtime architecture

### Application lifecycle

- `onCreate`
  - Allocate application state.
- `onShow`
  - Build the dashboard and MCP-controlled display canvas.
  - Attach the current UI objects to the shared application state.
  - Start the HTTP MCP service.
- `onHide`
  - Signal the HTTP MCP service to stop without blocking the GUI lifecycle.
  - Detach and invalidate UI object pointers.
- `onDestroy`
  - Join/delete any remaining worker and close sockets as a safety fallback.
  - Release buffers, devices, and application state.

The MCP server is deliberately foreground-only. Tactility does not need to
keep the listener or display-control task active while another app owns the
screen.

### FreeRTOS responsibilities

- MCP HTTP task
  - Listen for HTTP requests.
  - Parse JSON-RPC requests.
  - Dispatch tools.
  - Serialize JSON-RPC responses.
- UDP discovery task
  - Bind UDP port 5000.
  - Reply to discovery packets.
- Display work
  - Marshal changes through the LVGL lock or an LVGL async callback.
- Audio tasks
  - Record and play I2S data without blocking the UI or MCP listener.
- Future streaming tasks
  - Own TCP/UDP frame sockets and preallocated frame buffers.

## Network compatibility

### UDP discovery

- Port: `5000`
- Request: exact bytes `DISCOVER_SCREEN`
- Response: exact bytes `SCREEN_IP_80`

### MCP HTTP endpoint

- Port: `80`
- Method and path: `POST /api/mcp`
- Content type: `application/json`
- Protocol: JSON-RPC 2.0
- Required methods:
  - `tools/list`
  - `tools/call`

### Raw display endpoint

Added after the basic MCP slice:

- Method and path: `POST /api/screen/raw`
- Query parameters: `x`, `y`, `w`, `h`
- Body: raw big-endian RGB565 bytes

## MCP response behavior

Successful tool calls retain the existing shape:

```json
{
  "jsonrpc": "2.0",
  "result": {
    "content": [
      {
        "type": "text",
        "text": "..."
      }
    ]
  },
  "id": 1
}
```

Errors use JSON-RPC error objects:

```json
{
  "jsonrpc": "2.0",
  "error": {
    "code": -32000,
    "message": "..."
  },
  "id": 1
}
```

## Phase 1 — MCP foundation

### Scope

1. Scaffold `Apps/McpScreen`.
2. Add a status/dashboard view showing:
   - Wi-Fi state
   - server state
   - port
   - last tool
   - last error
3. Start the HTTP server on port 80.
4. Implement `POST /api/mcp`.
5. Implement JSON-RPC request validation and error handling.
6. Implement `tools/list`.
7. Implement UDP discovery on port 5000.
8. Implement the first three tools:
   - `get_capabilities`
   - `clear_screen`
   - `draw_text`
9. Test with the existing `mcp_bridge.py`.

### Initial tool semantics

#### `get_capabilities`

Return live display information:

```json
{
  "color": true,
  "width": 320,
  "height": 240,
  "formats": ["rgb565_base64", "bmp_base64"],
  "platform": "tactility",
  "appVersion": "0.1.0"
}
```

Width, height, and color format must be queried at runtime.

#### `clear_screen`

- Accept `color` values compatible with the original API:
  - `0`: white
  - `1`: black
- Mark MCP display override active.
- Clear the app-owned canvas/display area.

#### `draw_text`

- Accept `text`, `x`, `y`, and optional `size`.
- Mark MCP display override active.
- Render inside the app-owned canvas.
- Clamp coordinates to the drawable area.

### Phase 1 acceptance criteria

- The app builds and packages for ESP32-S3.
- Opening the app displays its server status.
- UDP discovery returns `SCREEN_IP_80`.
- The existing bridge can discover the device.
- `tools/list` returns valid schemas for the three initial tools.
- `get_capabilities` reports the actual Tactility display dimensions.
- `clear_screen` and `draw_text` visibly update the app.
- Malformed JSON and unknown methods return JSON-RPC errors without crashing.
- Repeated requests do not leak tasks, sockets, or request buffers.
- App hiding/showing behavior is documented from a device test.

## Phase 2 — Display tools

Add:

- `draw_raw_rgb565`
- `POST /api/screen/raw`
- `draw_color_bmp`
- `draw_image`
- `get_screenshot`
- `set_backlight`
- `set_screen_power`

Display commands should target an app-owned LVGL canvas or image buffer. Direct
panel access should only be used when the Tactility display HAL guarantees safe
ownership and synchronization.

Image preprocessing may remain in `mcp_bridge.py` initially. That avoids large
PNG/JPEG decoders and unnecessary memory pressure on the ESP32-S3.

## Phase 3 — Audio tools

Port the working implementation from `Apps/AudioTest`:

- `play_tone`
- `record_voice`
- `play_audio`
- `play_audio_base64`

Audio baseline:

- 16 kHz
- 16-bit signed PCM
- mono
- I2S device discovered through `device_find_by_name("i2s0")`

Recordings should be streamed to the application's user-data directory instead
of requiring one large fixed-duration RAM allocation.

## Phase 4 — Hardware and system tools

Add where supported:

- `get_touch`
- `get_battery`
- `set_led`
- `get_sensors`
- `scan_ble`
- `sync_time`
- `read_file`
- `write_file`
- `download_file`

File operations must:

- resolve paths under the app user-data directory;
- reject absolute paths and traversal such as `../`;
- enforce practical request and file-size limits.

`execute_python` is intentionally excluded. A future allow-listed diagnostic
command tool may replace it if needed.

## Phase 5 — Video streaming

Add:

- TCP frame server
- UDP chunked frame server
- stream activity timeout
- `get_video_streaming_instructions`
- `get_stream_stats`

The new protocol should advertise the actual Tactility display dimensions and
native color format. The old RLCD-specific 15,000-byte monochrome mapping may
be offered only as an optional compatibility mode.

Use preallocated buffers and avoid per-frame heap allocation.

## Phase 6 — Dashboard and Hermes voice assistant

Add:

- five-second dashboard refresh;
- MCP override mode;
- an explicit local action to leave override mode;
- Hermes WebSocket client;
- touch-to-talk;
- 16 kHz, 16-bit mono microphone streaming;
- incoming PCM/WAV playback;
- transcript, thinking, listening, and speaking UI states.

## Compatibility notes

- Keep the existing host bridge usable throughout development.
- Preserve tool names and argument names unless a compatibility defect is
  documented.
- `draw_image` can continue to be host-preprocessed into PBM or RGB565.
- Screenshot output may change from PBM to PNG/RGB565 if the bridge is updated
  to understand both.
- Hardware-specific tools should report support through `get_capabilities`.

## Security and robustness limits

The first implementation should define conservative limits for:

- HTTP header size
- JSON body size
- base64 payload size
- socket read timeout
- simultaneous clients
- filename length
- file size
- text length
- drawing bounds
- audio duration

Only one display mutation should run at a time. Audio operations should also be
serialized around the shared I2S device.

## Build workflow

From the TactilityApps repository:

```bash
. /Users/adolforeyna/esp/esp-idf/export.sh
export TACTILITY_SDK_PATH=/Users/adolforeyna/.gemini/antigravity/scratch/tactility/release/TactilitySDK
python3 tactility.py Apps/McpScreen build esp32s3 --local-sdk
```

## Immediate next task

Implement Phase 1 as a minimal vertical slice:

1. Scaffold the app and lifecycle.
2. Add a small server-status UI.
3. Add UDP discovery.
4. Add HTTP and JSON-RPC parsing.
5. Add `tools/list`.
6. Add `get_capabilities`, `clear_screen`, and `draw_text`.
7. Build.
8. Run a host smoke test through the existing bridge.