--- Date: 2026-07-17 Author: Hermes Tags: [project, tactility, esp32, voice-gateway, realtime-voice, elatoai, moss-tts-nano, opus, streaming-tts] Status: next-step-plan Related: [[voice_agent_platform_research_2026]], [[grace_poppy_storybook_image_voice_app]], [[pocketbase_family_baas_2026]] --- # Tactility Voice Gateway Transition Plan — ElatoAI + MOSS-TTS-Nano Pattern ## Why this exists Current Reyna voice UX already has strong pieces: - Hermes voice gateway on `:8642` receives **incoming mic PCM chunks** over WebSocket. - Mac mini MCP at `.102` provides fast Apple Speech transcription and Apple 3B quick replies. - iPhone/ESP32 screen clients can play returned audio through `play_audio_base64`. - Hermes brings memory, skills, family brain context, tools, and routing. The missing piece is **streamed speech back to the device**. Today, the gateway usually returns a full WAV/base64 blob, then the device starts playback only after the whole audio file exists. ElatoAI appears close to the desired device architecture: ESP32 WebSocket audio, stateful voice turns, Opus binary packets back to the device, an audio ring buffer, I2S playback, and realtime toy/companion UX. MOSS-TTS-Nano is interesting as a possible **streaming TTS engine**, especially because it advertises a 0.1B CPU/ONNX/MLX-friendly realtime path. ## Target end-state ```text Tactility ESP32 app/device -> streams mic audio chunks over secure WebSocket -> Hermes voice gateway :8642 -> Apple/Mac mini STT + Hermes/Muse/full agent answer -> streaming TTS engine: MOSS-TTS-Nano, Kokoro, Voicebox, or Apple say fallback -> Opus or PCM audio chunks over same WebSocket -> Tactility app decodes/queues/plays via I2S while chunks are still arriving ``` Design principle: **keep Hermes as the brain; borrow ElatoAI's realtime transport/playback architecture.** ## Current vs desired architecture | Layer | Current setup | Desired transition | |---|---|---| | Device input | PCM chunks over WS already working | Keep, optionally support Opus mic later | | STT | Mac mini SpeechAnalyzer/live pipe, local fallback | Keep; add latency telemetry per turn | | Fast reply | Apple 3B quick reply + Mac mini TTS | Keep as first audible ACK | | Full answer | Hermes background agent/session | Keep | | TTS output | whole WAV/base64 blob via `play_audio_base64` | streamed audio chunks: Opus preferred, PCM16 fallback | | Device playback | receive full file then play | ring buffer + decoder + I2S streaming task | | State protocol | ready/listening/draft/transcript/instant_response/done | add audio_start/audio_chunk/audio_end/speaking/interrupted | | Barge-in | partial / ad hoc | explicit interrupt: stop TTS, flush output, return to listening | ## Reference project: ElatoAI Repository: `https://github.com/akdeb/ElatoAI` Observed useful components: - `firmware-arduino/src/Audio.cpp` - `WebSocketsClient` - `OpusAudioDecoder` - `BufferRTOS audioBuffer` - `I2SStream i2s` - `QueueStream` - `StreamCopy` for continuous playback - state transitions: `LISTENING`, `PROCESSING`, `SPEAKING`, `SLEEP` - `server/fastapi/esp32_transport.py` - `RawPCMFrameSerializer` - `InputAudioRawFrame` - `OutputAudioRawFrame` - `OpusEncoder` - binary WebSocket audio frames - `server/cloudflare/README.md` - protocol events: `auth`, `AUDIO.COMMITTED`, `RESPONSE.CREATED`, binary audio frames, `RESPONSE.COMPLETE`, `SESSION.END` - Opus packetized audio streamed back to ESP32 ElatoAI should be treated as a **transport/playback reference**, not necessarily a replacement for Hermes. ## Reference engine: MOSS-TTS-Nano Repository: `https://github.com/OpenMOSS/MOSS-TTS-Nano` Why test it: - 0.1B parameters - realtime speech generation target - CPU-friendly claim - ONNX CPU version - MLX support for Apple Silicon - streaming inference - multilingual - voice cloning workflow Caveats: - Not for running directly on ESP32. - Native output is 48 kHz stereo; we likely need server-side resampling to 24 kHz or 16 kHz mono. - Need to benchmark first-audio latency vs current Mac mini `say`, Kokoro, and Voicebox. ## Transition plan — Phase 0: baseline and compatibility 1. Capture current gateway behavior: - Record events emitted by `/api/esp32/voice/ws`. - Measure: mic stop -> transcript, instant reply text, first audio playback, full answer delivery. - Save sample turn logs in `~/.hermes/audio_cache` or a project scratch folder. 2. Inventory playback clients: - iPhone `.150` supports `play_audio_base64`, not arbitrary execution. - Tactility ESP32 app can be extended to support streaming WS + I2S playback. 3. Decide transport fallback order: - **Preferred:** Opus binary frames for low bandwidth and robust streaming. - **Fallback:** raw PCM16 chunks for simplest first implementation. - Keep full WAV/base64 as compatibility fallback. ## Transition plan — Phase 1: gateway streamed-output protocol Add streamed TTS events to Hermes voice gateway without removing current behavior. Proposed server -> device events: ```json {"event":"response_audio_start","format":"pcm_s16le","sample_rate":24000,"channels":1,"codec":"pcm","turn_id":"..."} ``` Then binary frames: ```text ``` Then: ```json {"event":"response_audio_end","turn_id":"...","duration_ms":1234} ``` For Opus mode: ```json {"event":"response_audio_start","codec":"opus","sample_rate":24000,"channels":1,"frame_duration_ms":120,"bitrate":24000} ``` Binary frames are Opus packets. Also add: ```json {"event":"speaking","turn_id":"..."} {"event":"interrupted","turn_id":"...","reason":"barge_in"} {"event":"playback_flush","turn_id":"..."} ``` Implementation notes: - Keep `instant_response` text behavior for UI stickiness. - For current iPhone path, keep generating full base64 WAV. - For Tactility app path, prefer stream mode if client advertises it during `start`: ```json {"event":"start","audio_out":["opus","pcm_s16le","wav_base64"]} ``` ## Transition plan — Phase 2: server-side TTS streaming adapter Create a pluggable TTS output abstraction in the gateway: ```text TTSAdapter.synthesize_stream(text, voice, sample_rate) -> async iterator[AudioChunk] ``` Adapters: 1. **PCM dummy adapter** — split an already generated WAV into PCM chunks. This validates the device protocol even before true streaming TTS. 2. **Kokoro adapter** — start with full generation then chunk output; useful quality baseline. 3. **MOSS-TTS-Nano adapter** — true streaming target; benchmark ONNX CPU and MLX paths. 4. **Voicebox adapter** — probably full generation first, then chunk output; useful for Aiden/character voices. 5. **Apple say adapter** — fast fallback, full generation then chunk. Required telemetry: - `tts_request_at` - `first_audio_chunk_at` - `last_audio_chunk_at` - `playback_started_at` from device ack if available - codec, sample rate, chunk size, underrun count ## Transition plan — Phase 3: Tactility app streaming audio client Build a Tactility app, likely named one of: - `Apps/VoiceGateway` - `Apps/ReynaVoice` - `Apps/HermesVoice` Responsibilities: 1. Connect to Hermes voice gateway WS: - `ws://:8642/api/esp32/voice/ws?device_id=` or proxied HTTPS path. - Send auth bearer/config safely. 2. Capture mic audio: - 16 kHz mono PCM16 first. - Use VAD or push-to-talk first; wakeword later. 3. Send mic chunks as binary frames. 4. Receive JSON control events and binary audio frames. 5. Maintain state machine: - `IDLE` - `LISTENING` - `PROCESSING` - `SPEAKING` - `INTERRUPTING` - `ERROR` 6. Playback: - PCM mode first: binary chunk -> ring buffer -> I2S output task. - Opus mode next: binary packet -> Opus decoder -> ring buffer -> I2S. 7. UI: - big listening/speaking state text - live transcript/draft - quick reply text - connection/auth status - mute/volume buttons 8. Barge-in: - if mic detects speech while `SPEAKING`, send `interrupt` event - stop/flush output buffer - transition back to listening ## Tactility app implementation strategy Start with a minimal app rather than a full ElatoAI firmware fork. Reuse from existing local work: - `Apps/McpScreen` for screen/audio tool patterns. - `Apps/AudioTest` / `BookPlayer` / any I2S examples for playback. - `Apps/ReynaBot` if it already has network/client patterns. - ElatoAI's audio task design for ring buffer + decoder + state transitions. Milestones: ### Milestone A — playback-only streaming test - Device connects to a local test WS endpoint. - Server sends generated PCM chunks from a known WAV. - Device plays chunks smoothly. - Measure underruns and latency. ### Milestone B — gateway stream compatibility - Gateway sends `response_audio_start`, binary PCM chunks, `response_audio_end`. - Tactility app plays the response. - Keep mic input disabled or mocked. ### Milestone C — full duplex-ish voice turn - Tactility app records PCM chunks to gateway. - Gateway transcribes and replies. - Gateway streams output chunks back. - Device plays without waiting for full WAV. ### Milestone D — Opus mode - Add Opus encoder on gateway side. - Add Opus decoder on device side, borrowing from ElatoAI if licensing is acceptable. - Compare bandwidth/CPU/latency vs PCM. ### Milestone E — barge-in and polish - Device can interrupt speech. - Gateway cancels TTS generation/background playback. - UI shows listening/thinking/speaking states. - Add volume control and fallback to full WAV/base64 where needed. ## Gateway implementation strategy Do not rewrite the gateway first. Add a parallel streamed-output path. 1. Extend existing `/api/esp32/voice/ws` with output capability negotiation. 2. Keep current instant ACK and full-answer background flow. 3. For streamed output, first chunk existing generated WAVs to the device. 4. Once playback is solid, swap in MOSS-TTS-Nano/Kokoro streaming adapter. 5. Add Opus encoder after PCM path works. Pseudo-flow: ```text receive stop -> final transcript -> instant quick reply -> if client supports stream_out: send response_audio_start for chunk in tts_adapter.synthesize_stream(instant_reply): send binary chunk send response_audio_end else: generate full wav/base64 and call play_audio_base64 -> run full Hermes answer in background -> optionally stream full answer audio later ``` ## Open decisions - Should Tactility voice app be a new side-loaded app or merged into existing ReynaBot/McpScreen? - Should first audio codec be PCM16 for speed of implementation, or Opus from day one? - Should the app use push-to-talk first, then wakeword later? - Which device is the first target: current 320x240 Tactility board, RLCD, or larger CrowPanel/JC board? - Should MOSS run on Mac mini `.102`, iMac `.124`, or FamReynaServer `.110`? Suggested defaults: - New app: `Apps/HermesVoice`. - First codec: PCM16 chunks. - First interaction mode: push-to-talk or hold-to-talk. - First server: Mac mini `.102` for TTS/STT engines; gateway remains `:8642`. - Add Opus after PCM streaming is proven. ## Risks and mitigations | Risk | Mitigation | |---|---| | ESP32 underruns during playback | ring buffer, larger chunks, playback ACK/underrun telemetry | | TTS engine not truly streaming | chunk full WAV first; still improves protocol and app path | | Opus decode complexity | PCM first; borrow ElatoAI implementation later | | Gateway route patch wiped by Hermes update | keep restore plugin and restart guard bypass documented | | Device CPU/memory pressure | PSRAM buffers, mono 16/24 kHz, no UI redraw during playback | | Barge-in race conditions | explicit `interrupt` and `playback_flush` events with turn IDs | ## First actionable next step Build a **playback-only Tactility streaming audio proof**: 1. Write a tiny local WS server that sends: - `response_audio_start` - PCM16 chunks from a known WAV - `response_audio_end` 2. Write/extend a Tactility app to connect and play those chunks via I2S. 3. Once smooth, connect the same app to Hermes `:8642` and let the gateway stream generated WAV chunks. 4. Only then add MOSS-TTS-Nano true streaming and Opus. This keeps risk low and gives immediate proof that Tactility can speak progressively instead of waiting for full base64 WAVs.