Adolfo Reyna
12035f2f39
- Migrates tactility-firmware and tactility-bluetooth from ~/.hermes/skills/ into .claude/skills/ for repo co-location - Adds AGENTS.md with local workstation context, board table, board-direct build rule, Hermes PYTHONPATH pitfall, common Tactility pitfalls, and in-repo skill index Refs: personal Gitea mirror
102 lines
7.0 KiB
Markdown
102 lines
7.0 KiB
Markdown
# MCP System — Screensaver + WebServer + Video Streaming
|
|
|
|
## File Map
|
|
- `Private/Tactility/mcp/McpSystem.h` — global `McpSystemState` singleton
|
|
- `Source/mcp/McpSystem.cpp` — drawing (clear/text/RGB565/BMP/PBM), screenshot PBM base64, audio, sensors/battery/ble/SD file ops, video stream task, app listing
|
|
- `Source/service/webserver/McpHandler.cpp` — TOOLS_JSON, handle_rpc, handleApiMcp, handleApiScreenRaw
|
|
- `Source/service/webserver/WebServerService.cpp` — wildcard `/api/*` dispatches to MCP before auth, screenshot handler `/api/screenshot` (500ms timeout after fix)
|
|
- `Source/service/screenshot/ScreenshotTask.cpp` — `makeScreenshot` uses lvgl::lock(300ms after fix)
|
|
- `Source/service/displayidle/McpScreensaver.{h,cpp}` — canvas full-screen overlay, framebuffer SPIRAM, registers in McpSystemState
|
|
- `Include/Tactility/settings/McpSettings.h` + `Source/settings/McpSettings.cpp` — persisted at `getUserDataPath()+"/settings/mcp.properties"` = `/sdcard/tactility/...` on SD boards, `/data/tactility/...` internal. Old hardcoded `/data/service/mcp/settings.properties` bug fixed 2026-07-13.
|
|
|
|
## State Flow
|
|
1. WebServer `onStart()` loads `webServerSettings` + `mcpSettings` under `g_settingsMutex`, `serverEnabled = webEnabled || mcpEnabled`
|
|
2. `setEnabled(true)` → `startServer()` → `HttpServer(80, handlers, 12288)` → `httpd_start`
|
|
3. If `mcpEnabled` after HTTP up → `startVideoStreamServer()` → `xTaskCreatePinnedToCore(mcp_video_stream, 8192, prio5, core1)` allocates mono 15000 + color 153600 buffers SPIRAM
|
|
4. LLM draw → `drawRgb565/clear/drawText` → `ensureOverrideScreen()` → `displayidle::findService()->startMcpScreensaver()` → LVGL lock 200ms → create overlay canvas → register `drawArea`+`framebuffer`
|
|
5. Video stream client connects 8081/8083 → raw recv loop → `ensureOverrideScreen()` + `put_pixel` + `lv_obj_invalidate`
|
|
|
|
## Screensaver Integration (RLCD + MCP Persistence + Deadlock + Wake 50d86de4)
|
|
|
|
- Previously `DisplayIdle::onStart()` early returned on `!supportsBacklightDuty()` (ST7305) — killed timer + settings → MCP manual activation broken
|
|
- Now timer always starts, settings always loaded, `supportsBacklightDuty()` check only gates auto-backlight logic and duty calls in tick() and `startMcpScreensaver()`
|
|
- **Wake fix 50d86de4:** tick() timeout/wake logic must run for ALL displays, not gated by `supportsBacklight`. Only `setBacklightDuty()` guarded. Prevents RLCD locked state after dim.
|
|
- **Critical guard 6235fa05 — MCP flash-then-back <1s:** `tick()` every 50ms after `startMcpScreensaver()` sets `displayDimmed=true`, if `inactive_ms < 100ms` → `stopScreensaver()` deletes overlay. Fix: `bool isMcpActive = (drawArea != nullptr || overrideActive)` under mutex, guard all auto-stop `if (displayDimmed && !isMcpActive)`. Count `!isMcpActive>=3`.
|
|
- **Deadlock fix 3629ffef:** MCP video task locks `state.mutex -> lvgl`, tick must NOT do `lvgl -> mutex`. Read `isMcpActive` OUTSIDE LVGL lock.
|
|
- See `display-idle.md` for full tick flow final.
|
|
|
|
## RLCD 2-Button UX
|
|
|
|
User: "if i press the select button first it gets lock in there, so i need to press the change button first".
|
|
|
|
- Change GPIO18 KEY = LV_KEY_NEXT cycles focus
|
|
- Select GPIO0 BOOT = LV_KEY_ENTER activates
|
|
- No focus yet → Select does nothing → appears locked. Mitigation: auto-focus first btn on onShow.
|
|
|
|
## Serial Log Debugging — RLCD Crash Triage Recipe
|
|
|
|
```python
|
|
import serial
|
|
port="/dev/cu.usbmodem101"
|
|
ser=serial.Serial(port, 115200, timeout=1)
|
|
ser.dtr=False; ser.rts=False
|
|
buf=""
|
|
start=time.time()
|
|
while time.time()-start < 15:
|
|
data=ser.read(2048)
|
|
if data:
|
|
txt=data.decode('utf-8', errors='replace')
|
|
buf+=txt
|
|
print(txt, end='')
|
|
ser.close()
|
|
open("/tmp/serial.log","w").write(buf)
|
|
```
|
|
|
|
Look for:
|
|
- ESP-ROM + rst:0x15 = normal hard reset after flash
|
|
- Tactility v0.8.0-dev on WaveShare ESP32-S3-RLCD-4.2 = correct target
|
|
- Monochrome/RLCD detected = RLCD timer fix applied
|
|
- Mono TCP Video stream server started 8081 + Color 8083 = video OK
|
|
- Guru Meditation / Backtrace / Stack canary / abort() = crash, decode via `xtensa-esp32s3-elf-addr2line -pfiaC -e Tactility.elf <addr>`
|
|
- `i2c: CONFLICT! driver_ng` = legacy i2c vs i2c_master both registered in platform-esp32, benign after migration to `esp32-i2c-master` DTS but still warns; ES8311 now initializes after fix
|
|
- File lock function not set = benign
|
|
- No crash 15s + AP started + launcher → freeze likely deadlock not crash, check lock order (3629ffef)
|
|
- `WaveshareRLCD: ES8311 codec initialized` = I2C master fix 3770ac89 works
|
|
- `Hal: ST7305 started` time: should be ~2160ms with 150ms reset delay, vs 1828ms before (no delay) — freeze on fast boot fixed
|
|
|
|
## Screenshot Lock Contention — "could not acquire LVGL lock" (fixed 2026-07-14 6f095081)
|
|
|
|
**API:** `GET /api/screenshot` in `WebServerService.cpp:1522` — `httpd_resp_send_err(..., "could not acquire LVGL lock")` — this is what MCP reports as "Screenshot failed: could not acquire LVGL lock"
|
|
|
|
**Root causes:**
|
|
- Handler used `lvgl::lock(100)` — RLCD ST7305 SPI 10MHz full_refresh 120k pixels ~100ms/copy, 100ms too short
|
|
- `ScreenshotTask::makeScreenshot` used 50ms
|
|
- `mcp_video_stream_task` loop 5ms streaming hogs LVGL lock → UI frozen appearance "screen is still showing the freezed image"
|
|
|
|
**Fixes 6f095081 (build 0x2b79e0):**
|
|
- `WebServerService.cpp` 100→500ms: `lvgl::lock(pdMS_TO_TICKS(500))` + error log "within 500ms"
|
|
- `ScreenshotTask.cpp` 50→300ms
|
|
- `McpSystem.cpp` video stream 50ms idle→100ms, 5ms streaming→30ms with comment "RLCD ST7305 SPI is slow and shared"
|
|
- DisplayIdle: for monochrome don't auto-enter screensaver (BouncingBalls heavy)
|
|
- ST7305 RAM retains frozen image until power cycle — RTS reset not enough, need USB unplug 3s
|
|
- Verification 14 checks PASS hermes-verify-jw_55oxm.py
|
|
- See also `references/rlcd-initialization.md` + `references/display-idle.md` 6f095081
|
|
|
|
## Video Stream Protocol
|
|
- Mono: TCP 8081, 15000-byte frames, each byte 8 monochrome pixels RLCD packing, unpacked black/white with x_off/y_off centering
|
|
- Color: TCP 8083, packet: 16-byte header `magic=RAW\x01 (4) + x BE u16 + y BE u16 + w BE u16 + h BE u16 + payload_len BE u32`, payload BE RGB565
|
|
- Stats: `framesDrawn`, `tcpBytesReceived`, `lastDrawMs`, `lastFps`, `overrideActive`
|
|
|
|
## Adding New MCP Tool
|
|
1. Add function in `McpSystem.h/cpp`
|
|
2. Add JSON schema entry in `TOOLS_JSON` in `McpHandler.cpp`
|
|
3. Add `if (strcmp(name,"my_tool")==0)` branch
|
|
4. PSRAM malloc handling — use `heap_caps_malloc(SPIRAM)` + fallback `malloc`, free with `heap_caps_free`
|
|
|
|
## Known Gotchas
|
|
- `psram_malloc` fallback: `heap_caps_malloc(..., SPIRAM)` → if null `malloc`, free must be `heap_caps_free`
|
|
- `base64_decode` uses `malloc` not PSRAM — limit checked via `MCP_JSON_BODY_LIMIT 256KB` and `MCP_RAW_BODY_LIMIT 512KB`
|
|
- `ensureOverrideScreen()` blocking `vTaskDelay(50)*10=500ms` poll waiting for canvas — must not be called from LVGL task or deadlock
|
|
- Audio `audioBusy` spin flag prevents concurrent play/record
|
|
- Screenshot API `/api/screenshot` requires `TT_FEATURE_SCREENSHOT_ENABLED` — disabled saves build size but breaks MCP screenshot tool? Check sdkconfig
|