diff --git a/projects/tactility_gameboy_firmware_exports_needed_2026.md b/projects/tactility_gameboy_firmware_exports_needed_2026.md new file mode 100644 index 0000000..930e632 --- /dev/null +++ b/projects/tactility_gameboy_firmware_exports_needed_2026.md @@ -0,0 +1,86 @@ +# Tactility GameBoy firmware export notes — 2026-07-17 + +Context: GameBoy side-loaded ELF prototype on ESP32-S3 color board `192.168.68.129` / ES3C28P, Tactility `0.8.0-dev`, firmware built with ESP-IDF `5.3.2`. + +## What happened + +The GameBoy app package built locally against the newer cached SDK, but the board firmware rejected startup with the generic dialog: + +```text +Application failed to start: missing symbol +``` + +USB serial revealed exact missing symbols/runtime behavior. + +## Confirmed runtime findings + +1. Initial app install/run failed because app was not installed after an interrupted reinstall: + +```text +Loader: App not found: one.tactility.gameboy +``` + +2. After reinstall, the loader failed on: + +```text +ELF: Can't find symbol esp_log +ElfApp: Application failed to load: missing symbol +``` + +3. Earlier local/source verifier was misleading because it checked the local/generated firmware source exports, not the actual firmware currently running on the board. + +## Temporary app-side workaround applied + +To make the app launch on current board firmware: + +- Removed/avoided LVGL canvas APIs from the GameBoy app: + - `lv_canvas_create` + - `lv_canvas_set_buffer` + - `lv_obj_invalidate` + - display sizing helpers used only for scaling +- Disabled `ESP_LOGI/W/E` macros in the app source so the side-loaded ELF does not require: + - `esp_log` + - `esp_log_timestamp` +- Rebuilt and uploaded the patched ELF directly via `/fs/upload` because `/app/install` rebooted/crashed mid-install and left stale ELF content. + +Result: GameBoy now launches to the ROM browser screen, but the emulator display is intentionally disabled and shows a placeholder until firmware exports canvas APIs. + +## Firmware exports needed later for full GameBoy display + +Add/export these symbols in the firmware’s side-loaded ELF resolver/export table (likely `TactilityC/Source/tt_init.cpp` or module export path) and rebuild/flash the board firmware: + +```text +esp_log +esp_log_timestamp +lv_canvas_create +lv_canvas_set_buffer +lv_obj_invalidate +lv_display_get_horizontal_resolution +lv_display_get_vertical_resolution +``` + +Optional if restoring scaled/pretty display: + +```text +lv_obj_set_style_transform_scale_x +lv_obj_set_style_transform_scale_y +lv_obj_set_style_layout +``` + +## Install/debug lessons + +- `/app/run` returning HTTP 200 only means the run request was accepted; it does not prove ELF relocation/start succeeded. +- Always use USB serial logs while debugging side-loaded ELF startup. +- Always verify installed ELF hash by downloading it back: + +```bash +curl 'http://192.168.68.129/fs/download?path=/sdcard/tactility/app/one.tactility.gameboy/elf/esp32s3.elf' -o /tmp/installed_gameboy.elf +sha256sum /tmp/installed_gameboy.elf Apps/GameBoy/build/cmake-build-esp32s3/GameBoy.app.elf +``` + +- If `/app/install` times out or the board reboots, it may leave stale ELF content. Direct upload of ELF and manifest worked: + +```bash +POST /fs/upload?path=/sdcard/tactility/app/one.tactility.gameboy/elf/esp32s3.elf +POST /fs/upload?path=/sdcard/tactility/app/one.tactility.gameboy/manifest.properties +``` diff --git a/projects/tactility_voice_gateway_elato_transition_2026.md b/projects/tactility_voice_gateway_elato_transition_2026.md new file mode 100644 index 0000000..38c59a4 --- /dev/null +++ b/projects/tactility_voice_gateway_elato_transition_2026.md @@ -0,0 +1,337 @@ +--- +Date: 2026-07-17 +Author: Hermes +Tags: [project, tactility, esp32, voice-gateway, realtime-voice, elatoai, moss-tts-nano, opus, streaming-tts] +Status: next-step-plan +Related: [[voice_agent_platform_research_2026]], [[grace_poppy_storybook_image_voice_app]], [[pocketbase_family_baas_2026]] +--- + +# Tactility Voice Gateway Transition Plan — ElatoAI + MOSS-TTS-Nano Pattern + +## Why this exists + +Current Reyna voice UX already has strong pieces: + +- Hermes voice gateway on `:8642` receives **incoming mic PCM chunks** over WebSocket. +- Mac mini MCP at `.102` provides fast Apple Speech transcription and Apple 3B quick replies. +- iPhone/ESP32 screen clients can play returned audio through `play_audio_base64`. +- Hermes brings memory, skills, family brain context, tools, and routing. + +The missing piece is **streamed speech back to the device**. Today, the gateway usually returns a full WAV/base64 blob, then the device starts playback only after the whole audio file exists. + +ElatoAI appears close to the desired device architecture: ESP32 WebSocket audio, stateful voice turns, Opus binary packets back to the device, an audio ring buffer, I2S playback, and realtime toy/companion UX. + +MOSS-TTS-Nano is interesting as a possible **streaming TTS engine**, especially because it advertises a 0.1B CPU/ONNX/MLX-friendly realtime path. + +## Target end-state + +```text +Tactility ESP32 app/device + -> streams mic audio chunks over secure WebSocket + -> Hermes voice gateway :8642 + -> Apple/Mac mini STT + Hermes/Muse/full agent answer + -> streaming TTS engine: MOSS-TTS-Nano, Kokoro, Voicebox, or Apple say fallback + -> Opus or PCM audio chunks over same WebSocket + -> Tactility app decodes/queues/plays via I2S while chunks are still arriving +``` + +Design principle: **keep Hermes as the brain; borrow ElatoAI's realtime transport/playback architecture.** + +## Current vs desired architecture + +| Layer | Current setup | Desired transition | +|---|---|---| +| Device input | PCM chunks over WS already working | Keep, optionally support Opus mic later | +| STT | Mac mini SpeechAnalyzer/live pipe, local fallback | Keep; add latency telemetry per turn | +| Fast reply | Apple 3B quick reply + Mac mini TTS | Keep as first audible ACK | +| Full answer | Hermes background agent/session | Keep | +| TTS output | whole WAV/base64 blob via `play_audio_base64` | streamed audio chunks: Opus preferred, PCM16 fallback | +| Device playback | receive full file then play | ring buffer + decoder + I2S streaming task | +| State protocol | ready/listening/draft/transcript/instant_response/done | add audio_start/audio_chunk/audio_end/speaking/interrupted | +| Barge-in | partial / ad hoc | explicit interrupt: stop TTS, flush output, return to listening | + +## Reference project: ElatoAI + +Repository: `https://github.com/akdeb/ElatoAI` + +Observed useful components: + +- `firmware-arduino/src/Audio.cpp` + - `WebSocketsClient` + - `OpusAudioDecoder` + - `BufferRTOS audioBuffer` + - `I2SStream i2s` + - `QueueStream` + - `StreamCopy` for continuous playback + - state transitions: `LISTENING`, `PROCESSING`, `SPEAKING`, `SLEEP` +- `server/fastapi/esp32_transport.py` + - `RawPCMFrameSerializer` + - `InputAudioRawFrame` + - `OutputAudioRawFrame` + - `OpusEncoder` + - binary WebSocket audio frames +- `server/cloudflare/README.md` + - protocol events: `auth`, `AUDIO.COMMITTED`, `RESPONSE.CREATED`, binary audio frames, `RESPONSE.COMPLETE`, `SESSION.END` + - Opus packetized audio streamed back to ESP32 + +ElatoAI should be treated as a **transport/playback reference**, not necessarily a replacement for Hermes. + +## Reference engine: MOSS-TTS-Nano + +Repository: `https://github.com/OpenMOSS/MOSS-TTS-Nano` + +Why test it: + +- 0.1B parameters +- realtime speech generation target +- CPU-friendly claim +- ONNX CPU version +- MLX support for Apple Silicon +- streaming inference +- multilingual +- voice cloning workflow + +Caveats: + +- Not for running directly on ESP32. +- Native output is 48 kHz stereo; we likely need server-side resampling to 24 kHz or 16 kHz mono. +- Need to benchmark first-audio latency vs current Mac mini `say`, Kokoro, and Voicebox. + +## Transition plan — Phase 0: baseline and compatibility + +1. Capture current gateway behavior: + - Record events emitted by `/api/esp32/voice/ws`. + - Measure: mic stop -> transcript, instant reply text, first audio playback, full answer delivery. + - Save sample turn logs in `~/.hermes/audio_cache` or a project scratch folder. +2. Inventory playback clients: + - iPhone `.150` supports `play_audio_base64`, not arbitrary execution. + - Tactility ESP32 app can be extended to support streaming WS + I2S playback. +3. Decide transport fallback order: + - **Preferred:** Opus binary frames for low bandwidth and robust streaming. + - **Fallback:** raw PCM16 chunks for simplest first implementation. + - Keep full WAV/base64 as compatibility fallback. + +## Transition plan — Phase 1: gateway streamed-output protocol + +Add streamed TTS events to Hermes voice gateway without removing current behavior. + +Proposed server -> device events: + +```json +{"event":"response_audio_start","format":"pcm_s16le","sample_rate":24000,"channels":1,"codec":"pcm","turn_id":"..."} +``` + +Then binary frames: + +```text + +``` + +Then: + +```json +{"event":"response_audio_end","turn_id":"...","duration_ms":1234} +``` + +For Opus mode: + +```json +{"event":"response_audio_start","codec":"opus","sample_rate":24000,"channels":1,"frame_duration_ms":120,"bitrate":24000} +``` + +Binary frames are Opus packets. + +Also add: + +```json +{"event":"speaking","turn_id":"..."} +{"event":"interrupted","turn_id":"...","reason":"barge_in"} +{"event":"playback_flush","turn_id":"..."} +``` + +Implementation notes: + +- Keep `instant_response` text behavior for UI stickiness. +- For current iPhone path, keep generating full base64 WAV. +- For Tactility app path, prefer stream mode if client advertises it during `start`: + +```json +{"event":"start","audio_out":["opus","pcm_s16le","wav_base64"]} +``` + +## Transition plan — Phase 2: server-side TTS streaming adapter + +Create a pluggable TTS output abstraction in the gateway: + +```text +TTSAdapter.synthesize_stream(text, voice, sample_rate) -> async iterator[AudioChunk] +``` + +Adapters: + +1. **PCM dummy adapter** — split an already generated WAV into PCM chunks. This validates the device protocol even before true streaming TTS. +2. **Kokoro adapter** — start with full generation then chunk output; useful quality baseline. +3. **MOSS-TTS-Nano adapter** — true streaming target; benchmark ONNX CPU and MLX paths. +4. **Voicebox adapter** — probably full generation first, then chunk output; useful for Aiden/character voices. +5. **Apple say adapter** — fast fallback, full generation then chunk. + +Required telemetry: + +- `tts_request_at` +- `first_audio_chunk_at` +- `last_audio_chunk_at` +- `playback_started_at` from device ack if available +- codec, sample rate, chunk size, underrun count + +## Transition plan — Phase 3: Tactility app streaming audio client + +Build a Tactility app, likely named one of: + +- `Apps/VoiceGateway` +- `Apps/ReynaVoice` +- `Apps/HermesVoice` + +Responsibilities: + +1. Connect to Hermes voice gateway WS: + - `ws://:8642/api/esp32/voice/ws?device_id=` or proxied HTTPS path. + - Send auth bearer/config safely. +2. Capture mic audio: + - 16 kHz mono PCM16 first. + - Use VAD or push-to-talk first; wakeword later. +3. Send mic chunks as binary frames. +4. Receive JSON control events and binary audio frames. +5. Maintain state machine: + - `IDLE` + - `LISTENING` + - `PROCESSING` + - `SPEAKING` + - `INTERRUPTING` + - `ERROR` +6. Playback: + - PCM mode first: binary chunk -> ring buffer -> I2S output task. + - Opus mode next: binary packet -> Opus decoder -> ring buffer -> I2S. +7. UI: + - big listening/speaking state text + - live transcript/draft + - quick reply text + - connection/auth status + - mute/volume buttons +8. Barge-in: + - if mic detects speech while `SPEAKING`, send `interrupt` event + - stop/flush output buffer + - transition back to listening + +## Tactility app implementation strategy + +Start with a minimal app rather than a full ElatoAI firmware fork. + +Reuse from existing local work: + +- `Apps/McpScreen` for screen/audio tool patterns. +- `Apps/AudioTest` / `BookPlayer` / any I2S examples for playback. +- `Apps/ReynaBot` if it already has network/client patterns. +- ElatoAI's audio task design for ring buffer + decoder + state transitions. + +Milestones: + +### Milestone A — playback-only streaming test + +- Device connects to a local test WS endpoint. +- Server sends generated PCM chunks from a known WAV. +- Device plays chunks smoothly. +- Measure underruns and latency. + +### Milestone B — gateway stream compatibility + +- Gateway sends `response_audio_start`, binary PCM chunks, `response_audio_end`. +- Tactility app plays the response. +- Keep mic input disabled or mocked. + +### Milestone C — full duplex-ish voice turn + +- Tactility app records PCM chunks to gateway. +- Gateway transcribes and replies. +- Gateway streams output chunks back. +- Device plays without waiting for full WAV. + +### Milestone D — Opus mode + +- Add Opus encoder on gateway side. +- Add Opus decoder on device side, borrowing from ElatoAI if licensing is acceptable. +- Compare bandwidth/CPU/latency vs PCM. + +### Milestone E — barge-in and polish + +- Device can interrupt speech. +- Gateway cancels TTS generation/background playback. +- UI shows listening/thinking/speaking states. +- Add volume control and fallback to full WAV/base64 where needed. + +## Gateway implementation strategy + +Do not rewrite the gateway first. Add a parallel streamed-output path. + +1. Extend existing `/api/esp32/voice/ws` with output capability negotiation. +2. Keep current instant ACK and full-answer background flow. +3. For streamed output, first chunk existing generated WAVs to the device. +4. Once playback is solid, swap in MOSS-TTS-Nano/Kokoro streaming adapter. +5. Add Opus encoder after PCM path works. + +Pseudo-flow: + +```text +receive stop +-> final transcript +-> instant quick reply +-> if client supports stream_out: + send response_audio_start + for chunk in tts_adapter.synthesize_stream(instant_reply): + send binary chunk + send response_audio_end + else: + generate full wav/base64 and call play_audio_base64 +-> run full Hermes answer in background +-> optionally stream full answer audio later +``` + +## Open decisions + +- Should Tactility voice app be a new side-loaded app or merged into existing ReynaBot/McpScreen? +- Should first audio codec be PCM16 for speed of implementation, or Opus from day one? +- Should the app use push-to-talk first, then wakeword later? +- Which device is the first target: current 320x240 Tactility board, RLCD, or larger CrowPanel/JC board? +- Should MOSS run on Mac mini `.102`, iMac `.124`, or FamReynaServer `.110`? + +Suggested defaults: + +- New app: `Apps/HermesVoice`. +- First codec: PCM16 chunks. +- First interaction mode: push-to-talk or hold-to-talk. +- First server: Mac mini `.102` for TTS/STT engines; gateway remains `:8642`. +- Add Opus after PCM streaming is proven. + +## Risks and mitigations + +| Risk | Mitigation | +|---|---| +| ESP32 underruns during playback | ring buffer, larger chunks, playback ACK/underrun telemetry | +| TTS engine not truly streaming | chunk full WAV first; still improves protocol and app path | +| Opus decode complexity | PCM first; borrow ElatoAI implementation later | +| Gateway route patch wiped by Hermes update | keep restore plugin and restart guard bypass documented | +| Device CPU/memory pressure | PSRAM buffers, mono 16/24 kHz, no UI redraw during playback | +| Barge-in race conditions | explicit `interrupt` and `playback_flush` events with turn IDs | + +## First actionable next step + +Build a **playback-only Tactility streaming audio proof**: + +1. Write a tiny local WS server that sends: + - `response_audio_start` + - PCM16 chunks from a known WAV + - `response_audio_end` +2. Write/extend a Tactility app to connect and play those chunks via I2S. +3. Once smooth, connect the same app to Hermes `:8642` and let the gateway stream generated WAV chunks. +4. Only then add MOSS-TTS-Nano true streaming and Opus. + +This keeps risk low and gives immediate proof that Tactility can speak progressively instead of waiting for full base64 WAVs.