tactility/.claude/skills/tactility-firmware/references/mcp-system.md

MCP System — Screensaver + WebServer + Video Streaming

File Map

• Private/Tactility/mcp/McpSystem.h — global McpSystemState singleton
• Source/mcp/McpSystem.cpp — drawing (clear/text/RGB565/BMP/PBM), screenshot PBM base64, audio, sensors/battery/ble/SD file ops, video stream task, app listing
• Source/service/webserver/McpHandler.cpp — TOOLS_JSON, handle_rpc, handleApiMcp, handleApiScreenRaw
• Source/service/webserver/WebServerService.cpp — wildcard /api/* dispatches to MCP before auth, screenshot handler /api/screenshot (500ms timeout after fix)
• Source/service/screenshot/ScreenshotTask.cpp — makeScreenshot uses lvgl::lock(300ms after fix)
• Source/service/displayidle/McpScreensaver.{h,cpp} — canvas full-screen overlay, framebuffer SPIRAM, registers in McpSystemState
• Include/Tactility/settings/McpSettings.h + Source/settings/McpSettings.cpp — persisted at getUserDataPath()+"/settings/mcp.properties" = /sdcard/tactility/... on SD boards, /data/tactility/... internal. Old hardcoded /data/service/mcp/settings.properties bug fixed 2026-07-13.

State Flow

1. WebServer onStart() loads webServerSettings + mcpSettings under g_settingsMutex, serverEnabled = webEnabled || mcpEnabled
2. setEnabled(true) → startServer() → HttpServer(80, handlers, 12288) → httpd_start
3. If mcpEnabled after HTTP up → startVideoStreamServer() → xTaskCreatePinnedToCore(mcp_video_stream, 8192, prio5, core1) allocates mono 15000 + color 153600 buffers SPIRAM
4. LLM draw → drawRgb565/clear/drawText → ensureOverrideScreen() → displayidle::findService()->startMcpScreensaver() → LVGL lock 200ms → create overlay canvas → register drawArea+framebuffer
5. Video stream client connects 8081/8083 → raw recv loop → ensureOverrideScreen() + put_pixel + lv_obj_invalidate

Screensaver Integration (RLCD + MCP Persistence + Deadlock + Wake 50d86de4)

• Previously DisplayIdle::onStart() early returned on !supportsBacklightDuty() (ST7305) — killed timer + settings → MCP manual activation broken
• Now timer always starts, settings always loaded, supportsBacklightDuty() check only gates auto-backlight logic and duty calls in tick() and startMcpScreensaver()
• Wake fix 50d86de4: tick() timeout/wake logic must run for ALL displays, not gated by supportsBacklight. Only setBacklightDuty() guarded. Prevents RLCD locked state after dim.
• Critical guard 6235fa05 — MCP flash-then-back <1s: tick() every 50ms after startMcpScreensaver() sets displayDimmed=true, if inactive_ms < 100ms → stopScreensaver() deletes overlay. Fix: bool isMcpActive = (drawArea != nullptr || overrideActive) under mutex, guard all auto-stop if (displayDimmed && !isMcpActive). Count !isMcpActive>=3.
• Deadlock fix 3629ffef: MCP video task locks state.mutex -> lvgl, tick must NOT do lvgl -> mutex. Read isMcpActive OUTSIDE LVGL lock.
• See display-idle.md for full tick flow final.

RLCD 2-Button UX

User: "if i press the select button first it gets lock in there, so i need to press the change button first".

• Change GPIO18 KEY = LV_KEY_NEXT cycles focus
• Select GPIO0 BOOT = LV_KEY_ENTER activates
• No focus yet → Select does nothing → appears locked. Mitigation: auto-focus first btn on onShow.

Serial Log Debugging — RLCD Crash Triage Recipe

import serial
port="/dev/cu.usbmodem101"
ser=serial.Serial(port, 115200, timeout=1)
ser.dtr=False; ser.rts=False
buf=""
start=time.time()
while time.time()-start < 15:
  data=ser.read(2048)
  if data:
    txt=data.decode('utf-8', errors='replace')
    buf+=txt
    print(txt, end='')
ser.close()
open("/tmp/serial.log","w").write(buf)

Look for:

• ESP-ROM + rst:0x15 = normal hard reset after flash
• Tactility v0.8.0-dev on WaveShare ESP32-S3-RLCD-4.2 = correct target
• Monochrome/RLCD detected = RLCD timer fix applied
• Mono TCP Video stream server started 8081 + Color 8083 = video OK
• Guru Meditation / Backtrace / Stack canary / abort() = crash, decode via xtensa-esp32s3-elf-addr2line -pfiaC -e Tactility.elf <addr>
• i2c: CONFLICT! driver_ng = legacy i2c vs i2c_master both registered in platform-esp32, benign after migration to esp32-i2c-master DTS but still warns; ES8311 now initializes after fix
• File lock function not set = benign
• No crash 15s + AP started + launcher → freeze likely deadlock not crash, check lock order (3629ffef)
• WaveshareRLCD: ES8311 codec initialized = I2C master fix 3770ac89 works
• Hal: ST7305 started time: should be ~2160ms with 150ms reset delay, vs 1828ms before (no delay) — freeze on fast boot fixed

Screenshot Lock Contention — "could not acquire LVGL lock" (fixed 2026-07-14 6f095081)

API: GET /api/screenshot in WebServerService.cpp:1522 — httpd_resp_send_err(..., "could not acquire LVGL lock") — this is what MCP reports as "Screenshot failed: could not acquire LVGL lock"

Root causes:

• Handler used lvgl::lock(100) — RLCD ST7305 SPI 10MHz full_refresh 120k pixels ~100ms/copy, 100ms too short
• ScreenshotTask::makeScreenshot used 50ms
• mcp_video_stream_task loop 5ms streaming hogs LVGL lock → UI frozen appearance "screen is still showing the freezed image"

Fixes 6f095081 (build 0x2b79e0):

• WebServerService.cpp 100→500ms: lvgl::lock(pdMS_TO_TICKS(500)) + error log "within 500ms"
• ScreenshotTask.cpp 50→300ms
• McpSystem.cpp video stream 50ms idle→100ms, 5ms streaming→30ms with comment "RLCD ST7305 SPI is slow and shared"
• DisplayIdle: for monochrome don't auto-enter screensaver (BouncingBalls heavy)
• ST7305 RAM retains frozen image until power cycle — RTS reset not enough, need USB unplug 3s
• Verification 14 checks PASS hermes-verify-jw_55oxm.py
• See also references/rlcd-initialization.md + references/display-idle.md 6f095081

Video Stream Protocol

• Mono: TCP 8081, 15000-byte frames, each byte 8 monochrome pixels RLCD packing, unpacked black/white with x_off/y_off centering
• Color: TCP 8083, packet: 16-byte header magic=RAW\x01 (4) + x BE u16 + y BE u16 + w BE u16 + h BE u16 + payload_len BE u32, payload BE RGB565
• Stats: framesDrawn, tcpBytesReceived, lastDrawMs, lastFps, overrideActive

Adding New MCP Tool

1. Add function in McpSystem.h/cpp
2. Add JSON schema entry in TOOLS_JSON in McpHandler.cpp
3. Add if (strcmp(name,"my_tool")==0) branch
4. PSRAM malloc handling — use heap_caps_malloc(SPIRAM) + fallback malloc, free with heap_caps_free

Known Gotchas

• psram_malloc fallback: heap_caps_malloc(..., SPIRAM) → if null malloc, free must be heap_caps_free
• base64_decode uses malloc not PSRAM — limit checked via MCP_JSON_BODY_LIMIT 256KB and MCP_RAW_BODY_LIMIT 512KB
• ensureOverrideScreen() blocking vTaskDelay(50)*10=500ms poll waiting for canvas — must not be called from LVGL task or deadlock
• Audio audioBusy spin flag prevents concurrent play/record
• Screenshot API /api/screenshot requires TT_FEATURE_SCREENSHOT_ENABLED — disabled saves build size but breaks MCP screenshot tool? Check sdkconfig