adolforeyna/FamReynaBrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

daily auto-sync 2026-07-17

Browse Source
This commit is contained in:
aeroreyna
2026-07-17 23:00:01 -04:00
parent 36860c2bc5
commit 8eeabfb5d4
2 changed files with 423 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
projects/tactility_gameboy_firmware_exports_needed_2026.md
+86
View File
@@ -0,0 +1,86 @@
# Tactility GameBoy firmware export notes — 2026-07-17
Context: GameBoy side-loaded ELF prototype on ESP32-S3 color board `192.168.68.129` / ES3C28P, Tactility `0.8.0-dev`, firmware built with ESP-IDF `5.3.2`.
## What happened
The GameBoy app package built locally against the newer cached SDK, but the board firmware rejected startup with the generic dialog:
```text
Application failed to start: missing symbol
```
USB serial revealed exact missing symbols/runtime behavior.
## Confirmed runtime findings
1. Initial app install/run failed because app was not installed after an interrupted reinstall:
```text
Loader: App not found: one.tactility.gameboy
```
2. After reinstall, the loader failed on:
```text
ELF: Can't find symbol esp_log
ElfApp: Application failed to load: missing symbol
```
3. Earlier local/source verifier was misleading because it checked the local/generated firmware source exports, not the actual firmware currently running on the board.
## Temporary app-side workaround applied
To make the app launch on current board firmware:
- Removed/avoided LVGL canvas APIs from the GameBoy app:
- `lv_canvas_create`
- `lv_canvas_set_buffer`
- `lv_obj_invalidate`
- display sizing helpers used only for scaling
- Disabled `ESP_LOGI/W/E` macros in the app source so the side-loaded ELF does not require:
- `esp_log`
- `esp_log_timestamp`
- Rebuilt and uploaded the patched ELF directly via `/fs/upload` because `/app/install` rebooted/crashed mid-install and left stale ELF content.
Result: GameBoy now launches to the ROM browser screen, but the emulator display is intentionally disabled and shows a placeholder until firmware exports canvas APIs.
## Firmware exports needed later for full GameBoy display
Add/export these symbols in the firmware’s side-loaded ELF resolver/export table (likely `TactilityC/Source/tt_init.cpp` or module export path) and rebuild/flash the board firmware:
```text
esp_log
esp_log_timestamp
lv_canvas_create
lv_canvas_set_buffer
lv_obj_invalidate
lv_display_get_horizontal_resolution
lv_display_get_vertical_resolution
```
Optional if restoring scaled/pretty display:
```text
lv_obj_set_style_transform_scale_x
lv_obj_set_style_transform_scale_y
lv_obj_set_style_layout
```
## Install/debug lessons
- `/app/run` returning HTTP 200 only means the run request was accepted; it does not prove ELF relocation/start succeeded.
- Always use USB serial logs while debugging side-loaded ELF startup.
- Always verify installed ELF hash by downloading it back:
```bash
curl 'http://192.168.68.129/fs/download?path=/sdcard/tactility/app/one.tactility.gameboy/elf/esp32s3.elf' -o /tmp/installed_gameboy.elf
sha256sum /tmp/installed_gameboy.elf Apps/GameBoy/build/cmake-build-esp32s3/GameBoy.app.elf
```
- If `/app/install` times out or the board reboots, it may leave stale ELF content. Direct upload of ELF and manifest worked:
```bash
POST /fs/upload?path=/sdcard/tactility/app/one.tactility.gameboy/elf/esp32s3.elf
POST /fs/upload?path=/sdcard/tactility/app/one.tactility.gameboy/manifest.properties
```
projects/tactility_voice_gateway_elato_transition_2026.md
+337
View File
@@ -0,0 +1,337 @@
---
Date: 2026-07-17
Author: Hermes
Tags: [project, tactility, esp32, voice-gateway, realtime-voice, elatoai, moss-tts-nano, opus, streaming-tts]
Status: next-step-plan
Related: [[voice_agent_platform_research_2026]], [[grace_poppy_storybook_image_voice_app]], [[pocketbase_family_baas_2026]]
---
# Tactility Voice Gateway Transition Plan — ElatoAI + MOSS-TTS-Nano Pattern
## Why this exists
Current Reyna voice UX already has strong pieces:
- Hermes voice gateway on `:8642` receives **incoming mic PCM chunks** over WebSocket.
- Mac mini MCP at `.102` provides fast Apple Speech transcription and Apple 3B quick replies.
- iPhone/ESP32 screen clients can play returned audio through `play_audio_base64`.
- Hermes brings memory, skills, family brain context, tools, and routing.
The missing piece is **streamed speech back to the device**. Today, the gateway usually returns a full WAV/base64 blob, then the device starts playback only after the whole audio file exists.
ElatoAI appears close to the desired device architecture: ESP32 WebSocket audio, stateful voice turns, Opus binary packets back to the device, an audio ring buffer, I2S playback, and realtime toy/companion UX.
MOSS-TTS-Nano is interesting as a possible **streaming TTS engine**, especially because it advertises a 0.1B CPU/ONNX/MLX-friendly realtime path.
## Target end-state
```text
Tactility ESP32 app/device
-> streams mic audio chunks over secure WebSocket
-> Hermes voice gateway :8642
-> Apple/Mac mini STT + Hermes/Muse/full agent answer
-> streaming TTS engine: MOSS-TTS-Nano, Kokoro, Voicebox, or Apple say fallback
-> Opus or PCM audio chunks over same WebSocket
-> Tactility app decodes/queues/plays via I2S while chunks are still arriving
```
Design principle: **keep Hermes as the brain; borrow ElatoAI's realtime transport/playback architecture.**
## Current vs desired architecture
| Layer | Current setup | Desired transition |
|---|---|---|
| Device input | PCM chunks over WS already working | Keep, optionally support Opus mic later |
| STT | Mac mini SpeechAnalyzer/live pipe, local fallback | Keep; add latency telemetry per turn |
| Fast reply | Apple 3B quick reply + Mac mini TTS | Keep as first audible ACK |
| Full answer | Hermes background agent/session | Keep |
| TTS output | whole WAV/base64 blob via `play_audio_base64` | streamed audio chunks: Opus preferred, PCM16 fallback |
| Device playback | receive full file then play | ring buffer + decoder + I2S streaming task |
| State protocol | ready/listening/draft/transcript/instant_response/done | add audio_start/audio_chunk/audio_end/speaking/interrupted |
| Barge-in | partial / ad hoc | explicit interrupt: stop TTS, flush output, return to listening |
## Reference project: ElatoAI
Repository: `https://github.com/akdeb/ElatoAI`
Observed useful components:
- `firmware-arduino/src/Audio.cpp`
- `WebSocketsClient`
- `OpusAudioDecoder`
- `BufferRTOS<uint8_t> audioBuffer`
- `I2SStream i2s`
- `QueueStream<uint8_t>`
- `StreamCopy` for continuous playback
- state transitions: `LISTENING`, `PROCESSING`, `SPEAKING`, `SLEEP`
- `server/fastapi/esp32_transport.py`
- `RawPCMFrameSerializer`
- `InputAudioRawFrame`
- `OutputAudioRawFrame`
- `OpusEncoder`
- binary WebSocket audio frames
- `server/cloudflare/README.md`
- protocol events: `auth`, `AUDIO.COMMITTED`, `RESPONSE.CREATED`, binary audio frames, `RESPONSE.COMPLETE`, `SESSION.END`
- Opus packetized audio streamed back to ESP32
ElatoAI should be treated as a **transport/playback reference**, not necessarily a replacement for Hermes.
## Reference engine: MOSS-TTS-Nano
Repository: `https://github.com/OpenMOSS/MOSS-TTS-Nano`
Why test it:
- 0.1B parameters
- realtime speech generation target
- CPU-friendly claim
- ONNX CPU version
- MLX support for Apple Silicon
- streaming inference
- multilingual
- voice cloning workflow
Caveats:
- Not for running directly on ESP32.
- Native output is 48 kHz stereo; we likely need server-side resampling to 24 kHz or 16 kHz mono.
- Need to benchmark first-audio latency vs current Mac mini `say`, Kokoro, and Voicebox.
## Transition plan — Phase 0: baseline and compatibility
1. Capture current gateway behavior:
- Record events emitted by `/api/esp32/voice/ws`.
- Measure: mic stop -> transcript, instant reply text, first audio playback, full answer delivery.
- Save sample turn logs in `~/.hermes/audio_cache` or a project scratch folder.
2. Inventory playback clients:
- iPhone `.150` supports `play_audio_base64`, not arbitrary execution.
- Tactility ESP32 app can be extended to support streaming WS + I2S playback.
3. Decide transport fallback order:
- **Preferred:** Opus binary frames for low bandwidth and robust streaming.
- **Fallback:** raw PCM16 chunks for simplest first implementation.
- Keep full WAV/base64 as compatibility fallback.
## Transition plan — Phase 1: gateway streamed-output protocol
Add streamed TTS events to Hermes voice gateway without removing current behavior.
Proposed server -> device events:
```json
{"event":"response_audio_start","format":"pcm_s16le","sample_rate":24000,"channels":1,"codec":"pcm","turn_id":"..."}
```
Then binary frames:
```text
<raw PCM16 chunk bytes>
```
Then:
```json
{"event":"response_audio_end","turn_id":"...","duration_ms":1234}
```
For Opus mode:
```json
{"event":"response_audio_start","codec":"opus","sample_rate":24000,"channels":1,"frame_duration_ms":120,"bitrate":24000}
```
Binary frames are Opus packets.
Also add:
```json
{"event":"speaking","turn_id":"..."}
{"event":"interrupted","turn_id":"...","reason":"barge_in"}
{"event":"playback_flush","turn_id":"..."}
```
Implementation notes:
- Keep `instant_response` text behavior for UI stickiness.
- For current iPhone path, keep generating full base64 WAV.
- For Tactility app path, prefer stream mode if client advertises it during `start`:
```json
{"event":"start","audio_out":["opus","pcm_s16le","wav_base64"]}
```
## Transition plan — Phase 2: server-side TTS streaming adapter
Create a pluggable TTS output abstraction in the gateway:
```text
TTSAdapter.synthesize_stream(text, voice, sample_rate) -> async iterator[AudioChunk]
```
Adapters:
1. **PCM dummy adapter** — split an already generated WAV into PCM chunks. This validates the device protocol even before true streaming TTS.
2. **Kokoro adapter** — start with full generation then chunk output; useful quality baseline.
3. **MOSS-TTS-Nano adapter** — true streaming target; benchmark ONNX CPU and MLX paths.
4. **Voicebox adapter** — probably full generation first, then chunk output; useful for Aiden/character voices.
5. **Apple say adapter** — fast fallback, full generation then chunk.
Required telemetry:
- `tts_request_at`
- `first_audio_chunk_at`
- `last_audio_chunk_at`
- `playback_started_at` from device ack if available
- codec, sample rate, chunk size, underrun count
## Transition plan — Phase 3: Tactility app streaming audio client
Build a Tactility app, likely named one of:
- `Apps/VoiceGateway`
- `Apps/ReynaVoice`
- `Apps/HermesVoice`
Responsibilities:
1. Connect to Hermes voice gateway WS:
- `ws://<gateway>:8642/api/esp32/voice/ws?device_id=<id>` or proxied HTTPS path.
- Send auth bearer/config safely.
2. Capture mic audio:
- 16 kHz mono PCM16 first.
- Use VAD or push-to-talk first; wakeword later.
3. Send mic chunks as binary frames.
4. Receive JSON control events and binary audio frames.
5. Maintain state machine:
- `IDLE`
- `LISTENING`
- `PROCESSING`
- `SPEAKING`
- `INTERRUPTING`
- `ERROR`
6. Playback:
- PCM mode first: binary chunk -> ring buffer -> I2S output task.
- Opus mode next: binary packet -> Opus decoder -> ring buffer -> I2S.
7. UI:
- big listening/speaking state text
- live transcript/draft
- quick reply text
- connection/auth status
- mute/volume buttons
8. Barge-in:
- if mic detects speech while `SPEAKING`, send `interrupt` event
- stop/flush output buffer
- transition back to listening
## Tactility app implementation strategy
Start with a minimal app rather than a full ElatoAI firmware fork.
Reuse from existing local work:
- `Apps/McpScreen` for screen/audio tool patterns.
- `Apps/AudioTest` / `BookPlayer` / any I2S examples for playback.
- `Apps/ReynaBot` if it already has network/client patterns.
- ElatoAI's audio task design for ring buffer + decoder + state transitions.
Milestones:
### Milestone A — playback-only streaming test
- Device connects to a local test WS endpoint.
- Server sends generated PCM chunks from a known WAV.
- Device plays chunks smoothly.
- Measure underruns and latency.
### Milestone B — gateway stream compatibility
- Gateway sends `response_audio_start`, binary PCM chunks, `response_audio_end`.
- Tactility app plays the response.
- Keep mic input disabled or mocked.
### Milestone C — full duplex-ish voice turn
- Tactility app records PCM chunks to gateway.
- Gateway transcribes and replies.
- Gateway streams output chunks back.
- Device plays without waiting for full WAV.
### Milestone D — Opus mode
- Add Opus encoder on gateway side.
- Add Opus decoder on device side, borrowing from ElatoAI if licensing is acceptable.
- Compare bandwidth/CPU/latency vs PCM.
### Milestone E — barge-in and polish
- Device can interrupt speech.
- Gateway cancels TTS generation/background playback.
- UI shows listening/thinking/speaking states.
- Add volume control and fallback to full WAV/base64 where needed.
## Gateway implementation strategy
Do not rewrite the gateway first. Add a parallel streamed-output path.
1. Extend existing `/api/esp32/voice/ws` with output capability negotiation.
2. Keep current instant ACK and full-answer background flow.
3. For streamed output, first chunk existing generated WAVs to the device.
4. Once playback is solid, swap in MOSS-TTS-Nano/Kokoro streaming adapter.
5. Add Opus encoder after PCM path works.
Pseudo-flow:
```text
receive stop
-> final transcript
-> instant quick reply
-> if client supports stream_out:
send response_audio_start
for chunk in tts_adapter.synthesize_stream(instant_reply):
send binary chunk
send response_audio_end
else:
generate full wav/base64 and call play_audio_base64
-> run full Hermes answer in background
-> optionally stream full answer audio later
```
## Open decisions
- Should Tactility voice app be a new side-loaded app or merged into existing ReynaBot/McpScreen?
- Should first audio codec be PCM16 for speed of implementation, or Opus from day one?
- Should the app use push-to-talk first, then wakeword later?
- Which device is the first target: current 320x240 Tactility board, RLCD, or larger CrowPanel/JC board?
- Should MOSS run on Mac mini `.102`, iMac `.124`, or FamReynaServer `.110`?
Suggested defaults:
- New app: `Apps/HermesVoice`.
- First codec: PCM16 chunks.
- First interaction mode: push-to-talk or hold-to-talk.
- First server: Mac mini `.102` for TTS/STT engines; gateway remains `:8642`.
- Add Opus after PCM streaming is proven.
## Risks and mitigations
| Risk | Mitigation |
|---|---|
| ESP32 underruns during playback | ring buffer, larger chunks, playback ACK/underrun telemetry |
| TTS engine not truly streaming | chunk full WAV first; still improves protocol and app path |
| Opus decode complexity | PCM first; borrow ElatoAI implementation later |
| Gateway route patch wiped by Hermes update | keep restore plugin and restart guard bypass documented |
| Device CPU/memory pressure | PSRAM buffers, mono 16/24 kHz, no UI redraw during playback |
| Barge-in race conditions | explicit `interrupt` and `playback_flush` events with turn IDs |
## First actionable next step
Build a **playback-only Tactility streaming audio proof**:
1. Write a tiny local WS server that sends:
- `response_audio_start`
- PCM16 chunks from a known WAV
- `response_audio_end`
2. Write/extend a Tactility app to connect and play those chunks via I2S.
3. Once smooth, connect the same app to Hermes `:8642` and let the gateway stream generated WAV chunks.
4. Only then add MOSS-TTS-Nano true streaming and Opus.
This keeps risk low and gives immediate proof that Tactility can speak progressively instead of waiting for full base64 WAVs.