feat(sim): web viewer /sim, touch injection, fast screenshot, SIM_DISPLAY_W/H

.claude/skills/tactility-firmware-development/SKILL.md

@@ -95,3 +95,104 @@ Inspect the resulting tar member path, install through port-80 dashboard API,
 launch it, and read serial. Required evidence is the loader's `Loading
 .../bin/...`, an ELF entry address, `Task started`, and app-specific startup
 logs. HTTP 200 or a package simply appearing in `/api/apps` is insufficient.
+
+## Host simulator (buildsim) + web viewer
+
+The POSIX simulator runs the real firmware (LVGL, services, web server) on
+macOS/Linux with an SDL backend. macOS has no visible window (upstream
+`.github/workflows/build-simulator.yml`: "macOS simulator currently fails due
+to main thread requirement for rendering" — AppKit menu init must happen on
+the process main thread, but FreeRTOS-POSIX parks main in `sigwait` and runs
+everything on pthreads). The supported loop is **headless + web viewer**:
+screenshots render server-side regardless of any window.
+
+Code locations:
+
+- `Devices/simulator/Source/module.cpp` — display resolution + `SIM_DISPLAY_W/H`
+- `Devices/simulator/Source/drivers/sdl_display.{h,cpp}` — SDL backend
+- `Devices/simulator/Source/drivers/sdl_input.{h,cpp}` — pointer/key state +
+  web touch-injection override
+- `Tactility/Source/service/webserver/WebServerService.cpp` — `/sim` viewer,
+  `/sim/api/*` aliases, `POST /api/sim/touch`, `GET /api/screenshot?fast=`
+- `Tactility/Private/Tactility/service/webserver/WebServerService.h` — handler decls
+
+### Build and run
+
+```zsh
+# one-time host deps (outside any ESP-IDF env)
+mkdir -p /tmp/simbin && ln -sf "$(which python3)" /tmp/simbin/python
+pip3 install --break-system-packages lark pyyaml   # devicetree compiler
+
+cd /path/to/tactility
+export PATH="/tmp/simbin:$PATH"
+env -u ESP_IDF_VERSION -u IDF_PATH cmake -S . -B buildsim -DCMAKE_BUILD_TYPE=Release
+env -u ESP_IDF_VERSION -u IDF_PATH cmake --build buildsim --target Tactility -j "$(sysctl -n hw.ncpu)"
+
+# POSIX SDK for host apps (arm64)
+env -u ESP_IDF_VERSION -u IDF_PATH cmake --build buildsim --target TactilityKernel lvgl minitar minmea \
+  app-module crypt-module gps-module http-module lvgl-module lvgl-window-manager-module service-module
+env -u ESP_IDF_VERSION -u IDF_PATH python3 Buildscripts/release-sdk-posix.py /tmp/sim-sdk
+
+# release + headless run (MUST run from the firmware root: release-simulator.sh
+# uses relative version.txt / Data paths)
+sh Buildscripts/release-simulator.sh buildsim /tmp/simrun
+(cd /tmp/simrun && SIM_DISPLAY_W=480 SIM_DISPLAY_H=320 SDL_VIDEODRIVER=dummy \
+  nohup ./Tactility > /tmp/sim_web.log 2>&1 &)
+curl -s --max-time 5 http://127.0.0.1/api/sysinfo | head -c 120
+```
+
+Display resolution: `SIM_DISPLAY_W/H` env (default **480x320 landscape**,
+matching on-device screenshots). ES3C35P panel is 320x480 portrait in DTS but
+presents 480x320 landscape; ES3C28P is 320x240. The chosen geometry is logged
+as `Simulator Sim display WxH`. `SDL_VIDEODRIVER=dummy` is expected to log one
+`SdlDisplay Failed to create SDL window: Couldn't find matching render
+driver` line — LVGL still renders and screenshots work.
+
+### Web viewer, touch, screenshots
+
+- `GET /sim` → 301 to `/sim/` (trailing slash required so the page's relative
+  `api/` URLs resolve under `/sim/`). Viewer polls `api/screenshot?fast=1`
+  every 500 ms, footer shows live `naturalWidth×naturalHeight`, click/tap
+  POSTs `api/sim/touch?x=&y=`.
+- `POST /api/sim/touch?x=123&y=456[&down=0|1]` (also `/sim/api/sim/touch` via
+  alias). Coordinates are LVGL logical pixels. `down=1` (default) presses and
+  **auto-releases after 1500 ms** (`SIM_TOUCH_HOLD_MS` in `sdl_input.cpp`),
+  long enough for LVGL indev polls to register a click. Simulator-only: 404 on
+  ESP32 (`#ifndef ESP_PLATFORM`).
+- `GET /api/screenshot?fast=1` (default): `lv_snapshot_take` (RGB888) → in-place
+  BGR→RGB swap → `lodepng_encode24` **to memory** → chunked HTTP. No filesystem
+  touch, ~8 ms/shot. `?fast=0` keeps the legacy `webscreenshotN.png` file path
+  (slot scan + accumulation — avoid for viewer loops).
+- lodepng include in `.cpp`: `#define LODEPNG_NO_COMPILE_CPP` before
+  `#include "src/libs/lodepng/lodepng.h"`, otherwise its C++ `std::vector`
+  overloads collide with the C declarations (`conflicting types for 'encode'`).
+- MCP includes and `settings::mcp` reads are `#ifdef ESP_PLATFORM`-gated; the
+  sim has no `McpSystem`.
+
+### Tailscale viewer
+
+```zsh
+tailscale serve --bg --set-path=/simagent http://127.0.0.1:80/
+# open: https://<node>/simagent/sim/
+```
+
+The serve target must be `/` (not `/sim`): the page resolves `api/` against
+its own directory, so at `/simagent/sim/` fetches go to `/simagent/sim/api/…`,
+which tailscale strips to `/sim/api/…` and the firmware's `/sim/api/*`
+aliases (GET+POST, registered in `startServer()`) handle. Absolute `/api/…`
+URLs would 404 at the edge (no `/api` mount there).
+
+### Simulator pitfalls
+
+- **Rebuild ≠ redeploy.** `cmake --build buildsim` updates `buildsim/` only.
+  Re-run `release-simulator.sh`, restart the process, then retest. A stale
+  `/tmp/simrun/Tactility` serves old handlers with new logs nowhere to be found.
+- **C array `sizeof` decay.** A helper like
+  `f(HttpServerRequest*, char uri[256])` sees `sizeof(uri) == 8`, truncating
+  `get_uri` output to 7 chars (`/api/sy`, `/sim/ap` 404s). Pass the size
+  explicitly: `f(request, buf, sizeof(buf))`.
+- **Log truncation.** `LOG_QUEUE_MESSAGE_MAX_LENGTH` is 256 (`TactilityKernel/
+  private/tactility/log_queue.h`), including color/timestamp prefix. Long URIs
+  and messages truncate — don't over-interpret a short path in the log.
+- **Auth.** The viewer and API handlers enforce `validateRequestAuth` like any
+  other endpoint; failures surface as 401/404, not viewer bugs.

Devices/simulator/Source/module.cpp

@@ -29,7 +29,28 @@ static Driver* const simulator_drivers[] = {
 // These devices have no real bus to attach to (SDL has no notion of one), but every non-root
 // device is still expected to have a parent (see Device::parent) - they're parented to root once
 // it's available below.
-static const SdlDisplayConfig sdl_display_config = { 640, 480 };
+// Display resolution is overridable at runtime via SIM_DISPLAY_W/H env vars
+// so the sim can match real hardware (e.g. ES3C35P 3.5" = 320x480 portrait,
+// ES3C28P 2.8" = 320x240 landscape). Defaults to 640x480.
+static uint16_t sim_display_width(void) {
+    const char* w = getenv("SIM_DISPLAY_W");
+    if (w != nullptr) {
+        long v = atol(w);
+        if (v >= 120 && v <= 2048) return (uint16_t)v;
+    }
+    return 480;
+}
+
+static uint16_t sim_display_height(void) {
+    const char* h = getenv("SIM_DISPLAY_H");
+    if (h != nullptr) {
+        long v = atol(h);
+        if (v >= 120 && v <= 2048) return (uint16_t)v;
+    }
+    return 320;
+}
+
+static SdlDisplayConfig sdl_display_config = { 480, 320 };
 static Device sdl_display_device {};
 static Device sdl_pointer_device {};
 static Device sdl_keyboard_device {};
@@ -81,6 +102,10 @@ static void on_root_started(Device* device, DeviceEvent event, void* context) {
         return;
     }
 
+    sdl_display_config.horizontal_resolution = sim_display_width();
+    sdl_display_config.vertical_resolution = sim_display_height();
+    LOG_I(TAG, "Sim display %dx%d (SIM_DISPLAY_W/H override, default 480x320 landscape)",
+        sdl_display_config.horizontal_resolution, sdl_display_config.vertical_resolution);
     construct_add_start(&sdl_display_device, device, "display0", &sdl_display_config, "tactility,sdl-display");
     construct_add_start(&sdl_pointer_device, device, "pointer0", nullptr, "tactility,sdl-pointer");
     construct_add_start(&sdl_keyboard_device, device, "keyboard0", nullptr, "tactility,sdl-keyboard");

Tactility/Source/service/webserver/WebServerService.cpp

@@ -26,6 +26,11 @@
 
 #if TT_FEATURE_SCREENSHOT_ENABLED
 #include <lv_screenshot.h>
+// lodepng.h enables its C++ overloads (std::vector/std::string) when compiled
+// as C++; save_png.c compiles it as C. WebServerService.cpp is C++, so opt out
+// of the C++ wrapper to get the same C API save_png.c uses.
+#define LODEPNG_NO_COMPILE_CPP
+#include "src/libs/lodepng/lodepng.h"
 #endif
 
 #include "app/install.h"
@@ -536,6 +541,23 @@ bool WebServerService::startServer() {
             .callback  = handleApiPut,
             .user_ctx  = ctx
         },
+#ifndef ESP_PLATFORM
+        // Simulator viewer aliases: the /sim page resolves api/ against its
+        // own directory (/sim/api/...), so mirror the viewer-needed handlers
+        // here. Same callbacks, auth enforced inside each handler.
+        {
+            .uri       = "/sim/api/*",
+            .method    = HTTP_METHOD_GET,
+            .callback  = handleApiGet,
+            .user_ctx  = ctx
+        },
+        {
+            .uri       = "/sim/api/*",
+            .method    = HTTP_METHOD_POST,
+            .callback  = handleApiPost,
+            .user_ctx  = ctx
+        },
+#endif
         {
             .uri       = "/*",  // Catch-all for dynamic assets
             .method    = HTTP_METHOD_GET,
@@ -1053,9 +1075,17 @@ error_t WebServerService::handleAdminPost(HttpServerRequest* request, void* user
 
 // API GET dispatcher - returns JSON system information
 // Note: /api/sysinfo is intentionally public for monitoring use cases
+// The /sim/api/* alias routes (sim viewer page) strip the /sim prefix here
+// so both /api/... and /sim/api/... share one dispatch table.
+static const char* api_path_suffix(HttpServerRequest* request, char* uri, size_t uri_size) {
+    http_server_request_get_uri(request, uri, uri_size);
+    if (strncmp(uri, "/sim/api/", 9) == 0) return uri + 4; // -> /api/...
+    return uri;
+}
+
 error_t WebServerService::handleApiGet(HttpServerRequest* request, void* user_ctx) {
-    char uri[256];
+    char uri_buf[256];
-    http_server_request_get_uri(request, uri, sizeof(uri));
+    const char* uri = api_path_suffix(request, uri_buf, sizeof(uri_buf));
 
     // Public endpoint: sysinfo (basic device info for monitoring)
     if (strncmp(uri, "/api/sysinfo", 12) == 0) {
@@ -1096,8 +1126,8 @@ error_t WebServerService::handleApiPost(HttpServerRequest* request, void* user_c
         return authResult;
     }
 
-    char uri[256];
+    char uri_buf[256];
-    http_server_request_get_uri(request, uri, sizeof(uri));
+    const char* uri = api_path_suffix(request, uri_buf, sizeof(uri_buf));
     if (strncmp(uri, "/api/apps/run", 13) == 0) {
         return handleApiAppsRun(request, user_ctx);
     }
@@ -1539,11 +1569,76 @@ error_t WebServerService::handleApiWifi(HttpServerRequest* request, void*) {
 }
 
 // GET /api/screenshot - Capture and return screenshot as PNG
-// Screenshots are saved to SD card root (if available) or /data with incrementing numbers
+// Fast path (no query or ?fast=1): snapshot LVGL to a memory buffer, PNG-encode
+// to memory with lodepng, stream directly. No filesystem touch, no slot scan.
+// Legacy path (?fast=0): previous save-to-webscreenshotN.png behavior.
 error_t WebServerService::handleApiScreenshot(HttpServerRequest* request, void*) {
     LOG_I(TAG, "GET /api/screenshot");
 
 #if TT_FEATURE_SCREENSHOT_ENABLED
+    std::string sfast;
+    bool fast = true;
+    if (getQueryParam(request, "fast", sfast)) fast = (sfast != "0");
+
+    if (fast) {
+        // Hold the LVGL lock across snapshot + encode: both must see a stable
+        // framebuffer. lv_snapshot_take renders synchronously under the lock.
+        if (!lvgl_try_lock(pdMS_TO_TICKS(500))) {
+            LOG_E(TAG, "Could not acquire LVGL lock within 500ms");
+            http_server_request_send_error(request, 500, "could not acquire LVGL lock");
+            return ERROR_UNDEFINED;
+        }
+        lv_draw_buf_t* snapshot = lv_snapshot_take(lv_scr_act(), LV_COLOR_FORMAT_RGB888);
+        if (snapshot == nullptr) {
+            lvgl_unlock();
+            LOG_E(TAG, "lv_snapshot_take failed");
+            http_server_request_send_error(request, 500, "snapshot failed");
+            return ERROR_UNDEFINED;
+        }
+        // lodepng wants RGB triplets; snapshot is RGB888 = 3 bytes/px already,
+        // but in BGR order on little-endian — swap R and B in place.
+        uint32_t px_count = snapshot->header.w * snapshot->header.h;
+        uint8_t* px = snapshot->data;
+        for (uint32_t i = 0; i < px_count; i++) {
+            uint8_t tmp = px[0];
+            px[0] = px[2];
+            px[2] = tmp;
+            px += 3;
+        }
+        unsigned char* png = nullptr;
+        size_t png_size = 0;
+        unsigned enc_err = lodepng_encode24(&png, &png_size, snapshot->data,
+            snapshot->header.w, snapshot->header.h);
+        uint32_t w = snapshot->header.w, h = snapshot->header.h;
+        lv_draw_buf_destroy(snapshot);
+        lvgl_unlock();
+        if (enc_err != 0 || png == nullptr) {
+            LOG_E(TAG, "lodepng_encode24 failed: %u", enc_err);
+            http_server_request_send_error(request, 500, "png encode failed");
+            return ERROR_UNDEFINED;
+        }
+        LOG_I(TAG, "Screenshot %lux%lu %d bytes (memory path)", (unsigned long)w, (unsigned long)h, (int)png_size);
+        http_server_request_set_content_type(request, "image/png");
+        error_t result = ERROR_NONE;
+        if (http_server_request_send_chunk_start(request) != ERROR_NONE) {
+            result = ERROR_UNDEFINED;
+        } else {
+            size_t sent = 0;
+            while (sent < png_size) {
+                size_t n = png_size - sent > 8192 ? 8192 : png_size - sent;
+                if (http_server_request_send_chunk(request, png + sent, n) != ERROR_NONE) {
+                    result = ERROR_UNDEFINED;
+                    break;
+                }
+                sent += n;
+            }
+            http_server_request_send_chunk_end(request);
+        }
+        free(png);
+        LOG_I(TAG, "[200] /api/screenshot fast %d bytes", (int)png_size);
+        return result;
+    }
+
     // Determine save location: prefer SD card root if mounted, otherwise /data
     std::string save_path = getDataPath();
 
@@ -1787,9 +1882,9 @@ error_t WebServerService::handleReboot(HttpServerRequest* request, void*) {
 
 // POST /api/sim/touch?x=123&y=456[&down=0|1] - inject touch into simulator.
 // Simulator-only: on ESP32 there is no sdl-pointer backend, so this 404s.
-// x/y are LVGL logical pixels (sim display is 640x480). down=1 (default)
+// x/y are LVGL logical pixels (see viewer footer for current WxH).
-// presses and auto-releases after ~1.5s (long enough for LVGL indev polls to
+// down=1 (default) presses and auto-releases after ~1.5s (long enough for LVGL
-// register a click); down=0 releases immediately.
+// indev polls to register a click); down=0 releases immediately.
 error_t WebServerService::handleApiSimTouch(HttpServerRequest* request, void*) {
 #ifdef ESP_PLATFORM
     http_server_request_send_error(request, 404, "simulator only");
@@ -1820,6 +1915,13 @@ error_t WebServerService::handleSimViewer(HttpServerRequest* request, void*) {
         return authResult;
     }
     http_server_request_set_content_type(request, "text/html");
+    // NOTE: the page resolves api/ against its own directory (new URL(u,base)).
+    // Locally that is /sim -> /sim/api/...; behind tailscale
+    // (--set-path=/simagent -> /) it is /simagent/sim -> /simagent/api/...
+    // Both need handler aliases, registered in startServer() below:
+    // /sim/api/* mirrors /api/* (screenshot, sim/touch). The page itself must
+    // therefore be served with a trailing-slash-insensitive /sim match so the
+    // browser treats /sim as a directory (see handleAssets /sim route).
     static const char* page =
         "<!doctype html><html><head><meta charset=utf-8><meta name=viewport "
         "content='width=device-width,initial-scale=1'>"
@@ -1834,18 +1936,22 @@ error_t WebServerService::handleSimViewer(HttpServerRequest* request, void*) {
         "<div class=bar><b>Tactility Sim</b><span id=st>live</span>"
         "<button onclick='snap()'>refresh</button></div>"
         "<img id=scr alt='sim screen'>"
-        "<div class=bar>click/tap the screen to touch &middot; 640x480 logical</div>"
+        "<div class=bar>click/tap the screen to touch &middot; <span id=res></span> logical</div>"
         "<script>"
-        "const img=document.getElementById('scr'),st=document.getElementById('st');"
+        "const img=document.getElementById('scr'),st=document.getElementById('st'),"
-        "function snap(){img.src='/api/screenshot?ts='+Date.now();}"
+        "res=document.getElementById('res');"
-        "setInterval(snap,2000);snap();"
+        "const base=new URL('.',location.href).href;"
+        "const api=u=>new URL(u,base).href;"
+        "function snap(){img.src=api('api/screenshot?fast=1')+'&ts='+Date.now();}"
+        "setInterval(snap,500);snap();"
         "img.addEventListener('error',()=>{st.textContent='reconnecting…';});"
-        "img.addEventListener('load',()=>{st.textContent='live';});"
+        "img.addEventListener('load',()=>{st.textContent='live';"
+        "res.textContent=img.naturalWidth+'x'+img.naturalHeight;});"
         "img.addEventListener('pointerdown',e=>{"
         "const r=img.getBoundingClientRect();"
-        "const x=Math.round((e.clientX-r.left)/r.width*640);"
+        "const x=Math.round((e.clientX-r.left)/r.width*img.naturalWidth);"
-        "const y=Math.round((e.clientY-r.top)/r.height*480);"
+        "const y=Math.round((e.clientY-r.top)/r.height*img.naturalHeight);"
-        "fetch('/api/sim/touch?x='+x+'&y='+y,{method:'POST'});});"
+        "fetch(api('api/sim/touch')+'?x='+x+'&y='+y,{method:'POST'});});"
         "</script></body></html>";
     http_server_request_send_string(request, page);
     return ERROR_NONE;
@@ -1863,8 +1969,29 @@ error_t WebServerService::handleAssets(HttpServerRequest* request, void*) {
     http_server_request_get_uri(request, uri, sizeof(uri));
     LOG_I(TAG, "GET %s", uri);
 
-    // Simulator live viewer (no auth bypass: checked inside handler)
+    // Simulator live viewer (no auth bypass: checked inside handler).
+    // Served with redirect-to-slash so relative api/ URLs resolve under /sim/
+    // (browsers treat /sim as a file, /sim/ as a directory for URL purposes).
+    // NOTE: /sim/api/* does NOT need special-casing here: the /sim/api/*
+    // handler aliases registered in startServer() route straight into
+    // handleApiGet/handleApiPost, which strip the /sim prefix via
+    // api_path_suffix(). This block only serves the viewer page itself.
     if (strncmp(uri, "/sim", 4) == 0 && (uri[4] == '\0' || uri[4] == '?' || uri[4] == '/')) {
+        // Redirect bare /sim -> /sim/ so relative api/ resolves correctly.
+        if (uri[4] == '\0' || uri[4] == '?') {
+            std::string loc = "/sim/";
+            const char* q = strchr(uri, '?');
+            if (q != nullptr) { loc += q; }
+            http_server_request_set_header(request, "Location", loc.c_str());
+            http_server_request_send_error(request, 301, "see /sim/");
+            return ERROR_NONE;
+        }
+        if (strncmp(uri, "/sim/api/", 9) == 0) {
+            // Should have matched the /sim/api/* alias in startServer(); if we
+            // get here the method has no alias (e.g. PUT) — 404 it.
+            http_server_request_send_error(request, 404, "not found");
+            return ERROR_UNDEFINED;
+        }
         return handleSimViewer(request, nullptr);
     }