feat(sim): web viewer /sim, touch injection, fast screenshot, SIM_DISPLAY_W/H

2026-09-14 21:43:41 -04:00
parent 2060095028
commit 89e8baf517
3 changed files with 271 additions and 18 deletions
@@ -95,3 +95,104 @@ Inspect the resulting tar member path, install through port-80 dashboard API,
 launch it, and read serial. Required evidence is the loader's `Loading
 .../bin/...`, an ELF entry address, `Task started`, and app-specific startup
 logs. HTTP 200 or a package simply appearing in `/api/apps` is insufficient.
+
+## Host simulator (buildsim) + web viewer
+
+The POSIX simulator runs the real firmware (LVGL, services, web server) on
+macOS/Linux with an SDL backend. macOS has no visible window (upstream
+`.github/workflows/build-simulator.yml`: "macOS simulator currently fails due
+to main thread requirement for rendering" — AppKit menu init must happen on
+the process main thread, but FreeRTOS-POSIX parks main in `sigwait` and runs
+everything on pthreads). The supported loop is **headless + web viewer**:
+screenshots render server-side regardless of any window.
+
+Code locations:
+
+- `Devices/simulator/Source/module.cpp` — display resolution + `SIM_DISPLAY_W/H`
+- `Devices/simulator/Source/drivers/sdl_display.{h,cpp}` — SDL backend
+- `Devices/simulator/Source/drivers/sdl_input.{h,cpp}` — pointer/key state +
+  web touch-injection override
+- `Tactility/Source/service/webserver/WebServerService.cpp` — `/sim` viewer,
+  `/sim/api/*` aliases, `POST /api/sim/touch`, `GET /api/screenshot?fast=`
+- `Tactility/Private/Tactility/service/webserver/WebServerService.h` — handler decls
+
+### Build and run
+
+```zsh
+# one-time host deps (outside any ESP-IDF env)
+mkdir -p /tmp/simbin && ln -sf "$(which python3)" /tmp/simbin/python
+pip3 install --break-system-packages lark pyyaml   # devicetree compiler
+
+cd /path/to/tactility
+export PATH="/tmp/simbin:$PATH"
+env -u ESP_IDF_VERSION -u IDF_PATH cmake -S . -B buildsim -DCMAKE_BUILD_TYPE=Release
+env -u ESP_IDF_VERSION -u IDF_PATH cmake --build buildsim --target Tactility -j "$(sysctl -n hw.ncpu)"
+
+# POSIX SDK for host apps (arm64)
+env -u ESP_IDF_VERSION -u IDF_PATH cmake --build buildsim --target TactilityKernel lvgl minitar minmea \
+  app-module crypt-module gps-module http-module lvgl-module lvgl-window-manager-module service-module
+env -u ESP_IDF_VERSION -u IDF_PATH python3 Buildscripts/release-sdk-posix.py /tmp/sim-sdk
+
+# release + headless run (MUST run from the firmware root: release-simulator.sh
+# uses relative version.txt / Data paths)
+sh Buildscripts/release-simulator.sh buildsim /tmp/simrun
+(cd /tmp/simrun && SIM_DISPLAY_W=480 SIM_DISPLAY_H=320 SDL_VIDEODRIVER=dummy \
+  nohup ./Tactility > /tmp/sim_web.log 2>&1 &)
+curl -s --max-time 5 http://127.0.0.1/api/sysinfo | head -c 120
+```
+
+Display resolution: `SIM_DISPLAY_W/H` env (default **480x320 landscape**,
+matching on-device screenshots). ES3C35P panel is 320x480 portrait in DTS but
+presents 480x320 landscape; ES3C28P is 320x240. The chosen geometry is logged
+as `Simulator Sim display WxH`. `SDL_VIDEODRIVER=dummy` is expected to log one
+`SdlDisplay Failed to create SDL window: Couldn't find matching render
+driver` line — LVGL still renders and screenshots work.
+
+### Web viewer, touch, screenshots
+
+- `GET /sim` → 301 to `/sim/` (trailing slash required so the page's relative
+  `api/` URLs resolve under `/sim/`). Viewer polls `api/screenshot?fast=1`
+  every 500 ms, footer shows live `naturalWidth×naturalHeight`, click/tap
+  POSTs `api/sim/touch?x=&y=`.
+- `POST /api/sim/touch?x=123&y=456[&down=0|1]` (also `/sim/api/sim/touch` via
+  alias). Coordinates are LVGL logical pixels. `down=1` (default) presses and
+  **auto-releases after 1500 ms** (`SIM_TOUCH_HOLD_MS` in `sdl_input.cpp`),
+  long enough for LVGL indev polls to register a click. Simulator-only: 404 on
+  ESP32 (`#ifndef ESP_PLATFORM`).
+- `GET /api/screenshot?fast=1` (default): `lv_snapshot_take` (RGB888) → in-place
+  BGR→RGB swap → `lodepng_encode24` **to memory** → chunked HTTP. No filesystem
+  touch, ~8 ms/shot. `?fast=0` keeps the legacy `webscreenshotN.png` file path
+  (slot scan + accumulation — avoid for viewer loops).
+- lodepng include in `.cpp`: `#define LODEPNG_NO_COMPILE_CPP` before
+  `#include "src/libs/lodepng/lodepng.h"`, otherwise its C++ `std::vector`
+  overloads collide with the C declarations (`conflicting types for 'encode'`).
+- MCP includes and `settings::mcp` reads are `#ifdef ESP_PLATFORM`-gated; the
+  sim has no `McpSystem`.
+
+### Tailscale viewer
+
+```zsh
+tailscale serve --bg --set-path=/simagent http://127.0.0.1:80/
+# open: https://<node>/simagent/sim/
+```
+
+The serve target must be `/` (not `/sim`): the page resolves `api/` against
+its own directory, so at `/simagent/sim/` fetches go to `/simagent/sim/api/…`,
+which tailscale strips to `/sim/api/…` and the firmware's `/sim/api/*`
+aliases (GET+POST, registered in `startServer()`) handle. Absolute `/api/…`
+URLs would 404 at the edge (no `/api` mount there).
+
+### Simulator pitfalls
+
+- **Rebuild ≠ redeploy.** `cmake --build buildsim` updates `buildsim/` only.
+  Re-run `release-simulator.sh`, restart the process, then retest. A stale
+  `/tmp/simrun/Tactility` serves old handlers with new logs nowhere to be found.
+- **C array `sizeof` decay.** A helper like
+  `f(HttpServerRequest*, char uri[256])` sees `sizeof(uri) == 8`, truncating
+  `get_uri` output to 7 chars (`/api/sy`, `/sim/ap` 404s). Pass the size
+  explicitly: `f(request, buf, sizeof(buf))`.
+- **Log truncation.** `LOG_QUEUE_MESSAGE_MAX_LENGTH` is 256 (`TactilityKernel/
+  private/tactility/log_queue.h`), including color/timestamp prefix. Long URIs
+  and messages truncate — don't over-interpret a short path in the log.
+- **Auth.** The viewer and API handlers enforce `validateRequestAuth` like any
+  other endpoint; failures surface as 401/404, not viewer bugs.