diff --git a/.claude/skills/tactility-firmware-development/SKILL.md b/.claude/skills/tactility-firmware-development/SKILL.md index 64318b4c..e861f7b8 100644 --- a/.claude/skills/tactility-firmware-development/SKILL.md +++ b/.claude/skills/tactility-firmware-development/SKILL.md @@ -95,3 +95,104 @@ Inspect the resulting tar member path, install through port-80 dashboard API, launch it, and read serial. Required evidence is the loader's `Loading .../bin/...`, an ELF entry address, `Task started`, and app-specific startup logs. HTTP 200 or a package simply appearing in `/api/apps` is insufficient. + +## Host simulator (buildsim) + web viewer + +The POSIX simulator runs the real firmware (LVGL, services, web server) on +macOS/Linux with an SDL backend. macOS has no visible window (upstream +`.github/workflows/build-simulator.yml`: "macOS simulator currently fails due +to main thread requirement for rendering" — AppKit menu init must happen on +the process main thread, but FreeRTOS-POSIX parks main in `sigwait` and runs +everything on pthreads). The supported loop is **headless + web viewer**: +screenshots render server-side regardless of any window. + +Code locations: + +- `Devices/simulator/Source/module.cpp` — display resolution + `SIM_DISPLAY_W/H` +- `Devices/simulator/Source/drivers/sdl_display.{h,cpp}` — SDL backend +- `Devices/simulator/Source/drivers/sdl_input.{h,cpp}` — pointer/key state + + web touch-injection override +- `Tactility/Source/service/webserver/WebServerService.cpp` — `/sim` viewer, + `/sim/api/*` aliases, `POST /api/sim/touch`, `GET /api/screenshot?fast=` +- `Tactility/Private/Tactility/service/webserver/WebServerService.h` — handler decls + +### Build and run + +```zsh +# one-time host deps (outside any ESP-IDF env) +mkdir -p /tmp/simbin && ln -sf "$(which python3)" /tmp/simbin/python +pip3 install --break-system-packages lark pyyaml # devicetree compiler + +cd /path/to/tactility +export PATH="/tmp/simbin:$PATH" +env -u ESP_IDF_VERSION -u IDF_PATH cmake -S . -B buildsim -DCMAKE_BUILD_TYPE=Release +env -u ESP_IDF_VERSION -u IDF_PATH cmake --build buildsim --target Tactility -j "$(sysctl -n hw.ncpu)" + +# POSIX SDK for host apps (arm64) +env -u ESP_IDF_VERSION -u IDF_PATH cmake --build buildsim --target TactilityKernel lvgl minitar minmea \ + app-module crypt-module gps-module http-module lvgl-module lvgl-window-manager-module service-module +env -u ESP_IDF_VERSION -u IDF_PATH python3 Buildscripts/release-sdk-posix.py /tmp/sim-sdk + +# release + headless run (MUST run from the firmware root: release-simulator.sh +# uses relative version.txt / Data paths) +sh Buildscripts/release-simulator.sh buildsim /tmp/simrun +(cd /tmp/simrun && SIM_DISPLAY_W=480 SIM_DISPLAY_H=320 SDL_VIDEODRIVER=dummy \ + nohup ./Tactility > /tmp/sim_web.log 2>&1 &) +curl -s --max-time 5 http://127.0.0.1/api/sysinfo | head -c 120 +``` + +Display resolution: `SIM_DISPLAY_W/H` env (default **480x320 landscape**, +matching on-device screenshots). ES3C35P panel is 320x480 portrait in DTS but +presents 480x320 landscape; ES3C28P is 320x240. The chosen geometry is logged +as `Simulator Sim display WxH`. `SDL_VIDEODRIVER=dummy` is expected to log one +`SdlDisplay Failed to create SDL window: Couldn't find matching render +driver` line — LVGL still renders and screenshots work. + +### Web viewer, touch, screenshots + +- `GET /sim` → 301 to `/sim/` (trailing slash required so the page's relative + `api/` URLs resolve under `/sim/`). Viewer polls `api/screenshot?fast=1` + every 500 ms, footer shows live `naturalWidth×naturalHeight`, click/tap + POSTs `api/sim/touch?x=&y=`. +- `POST /api/sim/touch?x=123&y=456[&down=0|1]` (also `/sim/api/sim/touch` via + alias). Coordinates are LVGL logical pixels. `down=1` (default) presses and + **auto-releases after 1500 ms** (`SIM_TOUCH_HOLD_MS` in `sdl_input.cpp`), + long enough for LVGL indev polls to register a click. Simulator-only: 404 on + ESP32 (`#ifndef ESP_PLATFORM`). +- `GET /api/screenshot?fast=1` (default): `lv_snapshot_take` (RGB888) → in-place + BGR→RGB swap → `lodepng_encode24` **to memory** → chunked HTTP. No filesystem + touch, ~8 ms/shot. `?fast=0` keeps the legacy `webscreenshotN.png` file path + (slot scan + accumulation — avoid for viewer loops). +- lodepng include in `.cpp`: `#define LODEPNG_NO_COMPILE_CPP` before + `#include "src/libs/lodepng/lodepng.h"`, otherwise its C++ `std::vector` + overloads collide with the C declarations (`conflicting types for 'encode'`). +- MCP includes and `settings::mcp` reads are `#ifdef ESP_PLATFORM`-gated; the + sim has no `McpSystem`. + +### Tailscale viewer + +```zsh +tailscale serve --bg --set-path=/simagent http://127.0.0.1:80/ +# open: https:///simagent/sim/ +``` + +The serve target must be `/` (not `/sim`): the page resolves `api/` against +its own directory, so at `/simagent/sim/` fetches go to `/simagent/sim/api/…`, +which tailscale strips to `/sim/api/…` and the firmware's `/sim/api/*` +aliases (GET+POST, registered in `startServer()`) handle. Absolute `/api/…` +URLs would 404 at the edge (no `/api` mount there). + +### Simulator pitfalls + +- **Rebuild ≠ redeploy.** `cmake --build buildsim` updates `buildsim/` only. + Re-run `release-simulator.sh`, restart the process, then retest. A stale + `/tmp/simrun/Tactility` serves old handlers with new logs nowhere to be found. +- **C array `sizeof` decay.** A helper like + `f(HttpServerRequest*, char uri[256])` sees `sizeof(uri) == 8`, truncating + `get_uri` output to 7 chars (`/api/sy`, `/sim/ap` 404s). Pass the size + explicitly: `f(request, buf, sizeof(buf))`. +- **Log truncation.** `LOG_QUEUE_MESSAGE_MAX_LENGTH` is 256 (`TactilityKernel/ + private/tactility/log_queue.h`), including color/timestamp prefix. Long URIs + and messages truncate — don't over-interpret a short path in the log. +- **Auth.** The viewer and API handlers enforce `validateRequestAuth` like any + other endpoint; failures surface as 401/404, not viewer bugs. diff --git a/Devices/simulator/Source/module.cpp b/Devices/simulator/Source/module.cpp index 02319c92..447bad40 100644 --- a/Devices/simulator/Source/module.cpp +++ b/Devices/simulator/Source/module.cpp @@ -29,7 +29,28 @@ static Driver* const simulator_drivers[] = { // These devices have no real bus to attach to (SDL has no notion of one), but every non-root // device is still expected to have a parent (see Device::parent) - they're parented to root once // it's available below. -static const SdlDisplayConfig sdl_display_config = { 640, 480 }; +// Display resolution is overridable at runtime via SIM_DISPLAY_W/H env vars +// so the sim can match real hardware (e.g. ES3C35P 3.5" = 320x480 portrait, +// ES3C28P 2.8" = 320x240 landscape). Defaults to 640x480. +static uint16_t sim_display_width(void) { + const char* w = getenv("SIM_DISPLAY_W"); + if (w != nullptr) { + long v = atol(w); + if (v >= 120 && v <= 2048) return (uint16_t)v; + } + return 480; +} + +static uint16_t sim_display_height(void) { + const char* h = getenv("SIM_DISPLAY_H"); + if (h != nullptr) { + long v = atol(h); + if (v >= 120 && v <= 2048) return (uint16_t)v; + } + return 320; +} + +static SdlDisplayConfig sdl_display_config = { 480, 320 }; static Device sdl_display_device {}; static Device sdl_pointer_device {}; static Device sdl_keyboard_device {}; @@ -81,6 +102,10 @@ static void on_root_started(Device* device, DeviceEvent event, void* context) { return; } + sdl_display_config.horizontal_resolution = sim_display_width(); + sdl_display_config.vertical_resolution = sim_display_height(); + LOG_I(TAG, "Sim display %dx%d (SIM_DISPLAY_W/H override, default 480x320 landscape)", + sdl_display_config.horizontal_resolution, sdl_display_config.vertical_resolution); construct_add_start(&sdl_display_device, device, "display0", &sdl_display_config, "tactility,sdl-display"); construct_add_start(&sdl_pointer_device, device, "pointer0", nullptr, "tactility,sdl-pointer"); construct_add_start(&sdl_keyboard_device, device, "keyboard0", nullptr, "tactility,sdl-keyboard"); diff --git a/Tactility/Source/service/webserver/WebServerService.cpp b/Tactility/Source/service/webserver/WebServerService.cpp index b4052a3a..6b5e332e 100644 --- a/Tactility/Source/service/webserver/WebServerService.cpp +++ b/Tactility/Source/service/webserver/WebServerService.cpp @@ -26,6 +26,11 @@ #if TT_FEATURE_SCREENSHOT_ENABLED #include +// lodepng.h enables its C++ overloads (std::vector/std::string) when compiled +// as C++; save_png.c compiles it as C. WebServerService.cpp is C++, so opt out +// of the C++ wrapper to get the same C API save_png.c uses. +#define LODEPNG_NO_COMPILE_CPP +#include "src/libs/lodepng/lodepng.h" #endif #include "app/install.h" @@ -536,6 +541,23 @@ bool WebServerService::startServer() { .callback = handleApiPut, .user_ctx = ctx }, +#ifndef ESP_PLATFORM + // Simulator viewer aliases: the /sim page resolves api/ against its + // own directory (/sim/api/...), so mirror the viewer-needed handlers + // here. Same callbacks, auth enforced inside each handler. + { + .uri = "/sim/api/*", + .method = HTTP_METHOD_GET, + .callback = handleApiGet, + .user_ctx = ctx + }, + { + .uri = "/sim/api/*", + .method = HTTP_METHOD_POST, + .callback = handleApiPost, + .user_ctx = ctx + }, +#endif { .uri = "/*", // Catch-all for dynamic assets .method = HTTP_METHOD_GET, @@ -1053,9 +1075,17 @@ error_t WebServerService::handleAdminPost(HttpServerRequest* request, void* user // API GET dispatcher - returns JSON system information // Note: /api/sysinfo is intentionally public for monitoring use cases +// The /sim/api/* alias routes (sim viewer page) strip the /sim prefix here +// so both /api/... and /sim/api/... share one dispatch table. +static const char* api_path_suffix(HttpServerRequest* request, char* uri, size_t uri_size) { + http_server_request_get_uri(request, uri, uri_size); + if (strncmp(uri, "/sim/api/", 9) == 0) return uri + 4; // -> /api/... + return uri; +} + error_t WebServerService::handleApiGet(HttpServerRequest* request, void* user_ctx) { - char uri[256]; - http_server_request_get_uri(request, uri, sizeof(uri)); + char uri_buf[256]; + const char* uri = api_path_suffix(request, uri_buf, sizeof(uri_buf)); // Public endpoint: sysinfo (basic device info for monitoring) if (strncmp(uri, "/api/sysinfo", 12) == 0) { @@ -1096,8 +1126,8 @@ error_t WebServerService::handleApiPost(HttpServerRequest* request, void* user_c return authResult; } - char uri[256]; - http_server_request_get_uri(request, uri, sizeof(uri)); + char uri_buf[256]; + const char* uri = api_path_suffix(request, uri_buf, sizeof(uri_buf)); if (strncmp(uri, "/api/apps/run", 13) == 0) { return handleApiAppsRun(request, user_ctx); } @@ -1539,11 +1569,76 @@ error_t WebServerService::handleApiWifi(HttpServerRequest* request, void*) { } // GET /api/screenshot - Capture and return screenshot as PNG -// Screenshots are saved to SD card root (if available) or /data with incrementing numbers +// Fast path (no query or ?fast=1): snapshot LVGL to a memory buffer, PNG-encode +// to memory with lodepng, stream directly. No filesystem touch, no slot scan. +// Legacy path (?fast=0): previous save-to-webscreenshotN.png behavior. error_t WebServerService::handleApiScreenshot(HttpServerRequest* request, void*) { LOG_I(TAG, "GET /api/screenshot"); #if TT_FEATURE_SCREENSHOT_ENABLED + std::string sfast; + bool fast = true; + if (getQueryParam(request, "fast", sfast)) fast = (sfast != "0"); + + if (fast) { + // Hold the LVGL lock across snapshot + encode: both must see a stable + // framebuffer. lv_snapshot_take renders synchronously under the lock. + if (!lvgl_try_lock(pdMS_TO_TICKS(500))) { + LOG_E(TAG, "Could not acquire LVGL lock within 500ms"); + http_server_request_send_error(request, 500, "could not acquire LVGL lock"); + return ERROR_UNDEFINED; + } + lv_draw_buf_t* snapshot = lv_snapshot_take(lv_scr_act(), LV_COLOR_FORMAT_RGB888); + if (snapshot == nullptr) { + lvgl_unlock(); + LOG_E(TAG, "lv_snapshot_take failed"); + http_server_request_send_error(request, 500, "snapshot failed"); + return ERROR_UNDEFINED; + } + // lodepng wants RGB triplets; snapshot is RGB888 = 3 bytes/px already, + // but in BGR order on little-endian — swap R and B in place. + uint32_t px_count = snapshot->header.w * snapshot->header.h; + uint8_t* px = snapshot->data; + for (uint32_t i = 0; i < px_count; i++) { + uint8_t tmp = px[0]; + px[0] = px[2]; + px[2] = tmp; + px += 3; + } + unsigned char* png = nullptr; + size_t png_size = 0; + unsigned enc_err = lodepng_encode24(&png, &png_size, snapshot->data, + snapshot->header.w, snapshot->header.h); + uint32_t w = snapshot->header.w, h = snapshot->header.h; + lv_draw_buf_destroy(snapshot); + lvgl_unlock(); + if (enc_err != 0 || png == nullptr) { + LOG_E(TAG, "lodepng_encode24 failed: %u", enc_err); + http_server_request_send_error(request, 500, "png encode failed"); + return ERROR_UNDEFINED; + } + LOG_I(TAG, "Screenshot %lux%lu %d bytes (memory path)", (unsigned long)w, (unsigned long)h, (int)png_size); + http_server_request_set_content_type(request, "image/png"); + error_t result = ERROR_NONE; + if (http_server_request_send_chunk_start(request) != ERROR_NONE) { + result = ERROR_UNDEFINED; + } else { + size_t sent = 0; + while (sent < png_size) { + size_t n = png_size - sent > 8192 ? 8192 : png_size - sent; + if (http_server_request_send_chunk(request, png + sent, n) != ERROR_NONE) { + result = ERROR_UNDEFINED; + break; + } + sent += n; + } + http_server_request_send_chunk_end(request); + } + free(png); + LOG_I(TAG, "[200] /api/screenshot fast %d bytes", (int)png_size); + return result; + } + // Determine save location: prefer SD card root if mounted, otherwise /data std::string save_path = getDataPath(); @@ -1787,9 +1882,9 @@ error_t WebServerService::handleReboot(HttpServerRequest* request, void*) { // POST /api/sim/touch?x=123&y=456[&down=0|1] - inject touch into simulator. // Simulator-only: on ESP32 there is no sdl-pointer backend, so this 404s. -// x/y are LVGL logical pixels (sim display is 640x480). down=1 (default) -// presses and auto-releases after ~1.5s (long enough for LVGL indev polls to -// register a click); down=0 releases immediately. +// x/y are LVGL logical pixels (see viewer footer for current WxH). +// down=1 (default) presses and auto-releases after ~1.5s (long enough for LVGL +// indev polls to register a click); down=0 releases immediately. error_t WebServerService::handleApiSimTouch(HttpServerRequest* request, void*) { #ifdef ESP_PLATFORM http_server_request_send_error(request, 404, "simulator only"); @@ -1820,6 +1915,13 @@ error_t WebServerService::handleSimViewer(HttpServerRequest* request, void*) { return authResult; } http_server_request_set_content_type(request, "text/html"); + // NOTE: the page resolves api/ against its own directory (new URL(u,base)). + // Locally that is /sim -> /sim/api/...; behind tailscale + // (--set-path=/simagent -> /) it is /simagent/sim -> /simagent/api/... + // Both need handler aliases, registered in startServer() below: + // /sim/api/* mirrors /api/* (screenshot, sim/touch). The page itself must + // therefore be served with a trailing-slash-insensitive /sim match so the + // browser treats /sim as a directory (see handleAssets /sim route). static const char* page = "" @@ -1834,18 +1936,22 @@ error_t WebServerService::handleSimViewer(HttpServerRequest* request, void*) { "
Tactility Simlive" "
" "sim screen" - "
click/tap the screen to touch · 640x480 logical
" + "
click/tap the screen to touch · logical
" ""; http_server_request_send_string(request, page); return ERROR_NONE; @@ -1863,8 +1969,29 @@ error_t WebServerService::handleAssets(HttpServerRequest* request, void*) { http_server_request_get_uri(request, uri, sizeof(uri)); LOG_I(TAG, "GET %s", uri); - // Simulator live viewer (no auth bypass: checked inside handler) + // Simulator live viewer (no auth bypass: checked inside handler). + // Served with redirect-to-slash so relative api/ URLs resolve under /sim/ + // (browsers treat /sim as a file, /sim/ as a directory for URL purposes). + // NOTE: /sim/api/* does NOT need special-casing here: the /sim/api/* + // handler aliases registered in startServer() route straight into + // handleApiGet/handleApiPost, which strip the /sim prefix via + // api_path_suffix(). This block only serves the viewer page itself. if (strncmp(uri, "/sim", 4) == 0 && (uri[4] == '\0' || uri[4] == '?' || uri[4] == '/')) { + // Redirect bare /sim -> /sim/ so relative api/ resolves correctly. + if (uri[4] == '\0' || uri[4] == '?') { + std::string loc = "/sim/"; + const char* q = strchr(uri, '?'); + if (q != nullptr) { loc += q; } + http_server_request_set_header(request, "Location", loc.c_str()); + http_server_request_send_error(request, 301, "see /sim/"); + return ERROR_NONE; + } + if (strncmp(uri, "/sim/api/", 9) == 0) { + // Should have matched the /sim/api/* alias in startServer(); if we + // get here the method has no alias (e.g. PUT) — 404 it. + http_server_request_send_error(request, 404, "not found"); + return ERROR_UNDEFINED; + } return handleSimViewer(request, nullptr); }