adolforeyna/FamReynaBrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
master
FamReynaBrain/projects/tactility_voice_gateway_elato_transition_2026.md
T
aeroreyna 8eeabfb5d4 daily auto-sync 2026-07-17
2026-07-17 23:00:01 -04:00

12 KiB
Raw Permalink Blame History

Date: 2026-07-17 Author: Hermes Tags: [project, tactility, esp32, voice-gateway, realtime-voice, elatoai, moss-tts-nano, opus, streaming-tts] Status: next-step-plan Related: voice_agent_platform_research_2026, grace_poppy_storybook_image_voice_app, pocketbase_family_baas_2026

Tactility Voice Gateway Transition Plan — ElatoAI + MOSS-TTS-Nano Pattern

Why this exists

Current Reyna voice UX already has strong pieces:

  • Hermes voice gateway on :8642 receives incoming mic PCM chunks over WebSocket.
  • Mac mini MCP at .102 provides fast Apple Speech transcription and Apple 3B quick replies.
  • iPhone/ESP32 screen clients can play returned audio through play_audio_base64.
  • Hermes brings memory, skills, family brain context, tools, and routing.

The missing piece is streamed speech back to the device. Today, the gateway usually returns a full WAV/base64 blob, then the device starts playback only after the whole audio file exists.

ElatoAI appears close to the desired device architecture: ESP32 WebSocket audio, stateful voice turns, Opus binary packets back to the device, an audio ring buffer, I2S playback, and realtime toy/companion UX.

MOSS-TTS-Nano is interesting as a possible streaming TTS engine, especially because it advertises a 0.1B CPU/ONNX/MLX-friendly realtime path.

Target end-state

Tactility ESP32 app/device
  -> streams mic audio chunks over secure WebSocket
  -> Hermes voice gateway :8642
  -> Apple/Mac mini STT + Hermes/Muse/full agent answer
  -> streaming TTS engine: MOSS-TTS-Nano, Kokoro, Voicebox, or Apple say fallback
  -> Opus or PCM audio chunks over same WebSocket
  -> Tactility app decodes/queues/plays via I2S while chunks are still arriving

Design principle: keep Hermes as the brain; borrow ElatoAI's realtime transport/playback architecture.

Current vs desired architecture

Layer Current setup Desired transition
Device input PCM chunks over WS already working Keep, optionally support Opus mic later
STT Mac mini SpeechAnalyzer/live pipe, local fallback Keep; add latency telemetry per turn
Fast reply Apple 3B quick reply + Mac mini TTS Keep as first audible ACK
Full answer Hermes background agent/session Keep
TTS output whole WAV/base64 blob via play_audio_base64 streamed audio chunks: Opus preferred, PCM16 fallback
Device playback receive full file then play ring buffer + decoder + I2S streaming task
State protocol ready/listening/draft/transcript/instant_response/done add audio_start/audio_chunk/audio_end/speaking/interrupted
Barge-in partial / ad hoc explicit interrupt: stop TTS, flush output, return to listening

Reference project: ElatoAI

Repository: https://github.com/akdeb/ElatoAI

Observed useful components:

  • firmware-arduino/src/Audio.cpp
    • WebSocketsClient
    • OpusAudioDecoder
    • BufferRTOS<uint8_t> audioBuffer
    • I2SStream i2s
    • QueueStream<uint8_t>
    • StreamCopy for continuous playback
    • state transitions: LISTENING, PROCESSING, SPEAKING, SLEEP
  • server/fastapi/esp32_transport.py
    • RawPCMFrameSerializer
    • InputAudioRawFrame
    • OutputAudioRawFrame
    • OpusEncoder
    • binary WebSocket audio frames
  • server/cloudflare/README.md
    • protocol events: auth, AUDIO.COMMITTED, RESPONSE.CREATED, binary audio frames, RESPONSE.COMPLETE, SESSION.END
    • Opus packetized audio streamed back to ESP32

ElatoAI should be treated as a transport/playback reference, not necessarily a replacement for Hermes.

Reference engine: MOSS-TTS-Nano

Repository: https://github.com/OpenMOSS/MOSS-TTS-Nano

Why test it:

  • 0.1B parameters
  • realtime speech generation target
  • CPU-friendly claim
  • ONNX CPU version
  • MLX support for Apple Silicon
  • streaming inference
  • multilingual
  • voice cloning workflow

Caveats:

  • Not for running directly on ESP32.
  • Native output is 48 kHz stereo; we likely need server-side resampling to 24 kHz or 16 kHz mono.
  • Need to benchmark first-audio latency vs current Mac mini say, Kokoro, and Voicebox.

Transition plan — Phase 0: baseline and compatibility

  1. Capture current gateway behavior:
    • Record events emitted by /api/esp32/voice/ws.
    • Measure: mic stop -> transcript, instant reply text, first audio playback, full answer delivery.
    • Save sample turn logs in ~/.hermes/audio_cache or a project scratch folder.
  2. Inventory playback clients:
    • iPhone .150 supports play_audio_base64, not arbitrary execution.
    • Tactility ESP32 app can be extended to support streaming WS + I2S playback.
  3. Decide transport fallback order:
    • Preferred: Opus binary frames for low bandwidth and robust streaming.
    • Fallback: raw PCM16 chunks for simplest first implementation.
    • Keep full WAV/base64 as compatibility fallback.

Transition plan — Phase 1: gateway streamed-output protocol

Add streamed TTS events to Hermes voice gateway without removing current behavior.

Proposed server -> device events:

{"event":"response_audio_start","format":"pcm_s16le","sample_rate":24000,"channels":1,"codec":"pcm","turn_id":"..."}

Then binary frames:

<raw PCM16 chunk bytes>

Then:

{"event":"response_audio_end","turn_id":"...","duration_ms":1234}

For Opus mode:

{"event":"response_audio_start","codec":"opus","sample_rate":24000,"channels":1,"frame_duration_ms":120,"bitrate":24000}

Binary frames are Opus packets.

Also add:

{"event":"speaking","turn_id":"..."}
{"event":"interrupted","turn_id":"...","reason":"barge_in"}
{"event":"playback_flush","turn_id":"..."}

Implementation notes:

  • Keep instant_response text behavior for UI stickiness.
  • For current iPhone path, keep generating full base64 WAV.
  • For Tactility app path, prefer stream mode if client advertises it during start:
{"event":"start","audio_out":["opus","pcm_s16le","wav_base64"]}

Transition plan — Phase 2: server-side TTS streaming adapter

Create a pluggable TTS output abstraction in the gateway:

TTSAdapter.synthesize_stream(text, voice, sample_rate) -> async iterator[AudioChunk]

Adapters:

  1. PCM dummy adapter — split an already generated WAV into PCM chunks. This validates the device protocol even before true streaming TTS.
  2. Kokoro adapter — start with full generation then chunk output; useful quality baseline.
  3. MOSS-TTS-Nano adapter — true streaming target; benchmark ONNX CPU and MLX paths.
  4. Voicebox adapter — probably full generation first, then chunk output; useful for Aiden/character voices.
  5. Apple say adapter — fast fallback, full generation then chunk.

Required telemetry:

  • tts_request_at
  • first_audio_chunk_at
  • last_audio_chunk_at
  • playback_started_at from device ack if available
  • codec, sample rate, chunk size, underrun count

Transition plan — Phase 3: Tactility app streaming audio client

Build a Tactility app, likely named one of:

  • Apps/VoiceGateway
  • Apps/ReynaVoice
  • Apps/HermesVoice

Responsibilities:

  1. Connect to Hermes voice gateway WS:
    • ws://<gateway>:8642/api/esp32/voice/ws?device_id=<id> or proxied HTTPS path.
    • Send auth bearer/config safely.
  2. Capture mic audio:
    • 16 kHz mono PCM16 first.
    • Use VAD or push-to-talk first; wakeword later.
  3. Send mic chunks as binary frames.
  4. Receive JSON control events and binary audio frames.
  5. Maintain state machine:
    • IDLE
    • LISTENING
    • PROCESSING
    • SPEAKING
    • INTERRUPTING
    • ERROR
  6. Playback:
    • PCM mode first: binary chunk -> ring buffer -> I2S output task.
    • Opus mode next: binary packet -> Opus decoder -> ring buffer -> I2S.
  7. UI:
    • big listening/speaking state text
    • live transcript/draft
    • quick reply text
    • connection/auth status
    • mute/volume buttons
  8. Barge-in:
    • if mic detects speech while SPEAKING, send interrupt event
    • stop/flush output buffer
    • transition back to listening

Tactility app implementation strategy

Start with a minimal app rather than a full ElatoAI firmware fork.

Reuse from existing local work:

  • Apps/McpScreen for screen/audio tool patterns.
  • Apps/AudioTest / BookPlayer / any I2S examples for playback.
  • Apps/ReynaBot if it already has network/client patterns.
  • ElatoAI's audio task design for ring buffer + decoder + state transitions.

Milestones:

Milestone A — playback-only streaming test

  • Device connects to a local test WS endpoint.
  • Server sends generated PCM chunks from a known WAV.
  • Device plays chunks smoothly.
  • Measure underruns and latency.

Milestone B — gateway stream compatibility

  • Gateway sends response_audio_start, binary PCM chunks, response_audio_end.
  • Tactility app plays the response.
  • Keep mic input disabled or mocked.

Milestone C — full duplex-ish voice turn

  • Tactility app records PCM chunks to gateway.
  • Gateway transcribes and replies.
  • Gateway streams output chunks back.
  • Device plays without waiting for full WAV.

Milestone D — Opus mode

  • Add Opus encoder on gateway side.
  • Add Opus decoder on device side, borrowing from ElatoAI if licensing is acceptable.
  • Compare bandwidth/CPU/latency vs PCM.

Milestone E — barge-in and polish

  • Device can interrupt speech.
  • Gateway cancels TTS generation/background playback.
  • UI shows listening/thinking/speaking states.
  • Add volume control and fallback to full WAV/base64 where needed.

Gateway implementation strategy

Do not rewrite the gateway first. Add a parallel streamed-output path.

  1. Extend existing /api/esp32/voice/ws with output capability negotiation.
  2. Keep current instant ACK and full-answer background flow.
  3. For streamed output, first chunk existing generated WAVs to the device.
  4. Once playback is solid, swap in MOSS-TTS-Nano/Kokoro streaming adapter.
  5. Add Opus encoder after PCM path works.

Pseudo-flow:

receive stop
-> final transcript
-> instant quick reply
-> if client supports stream_out:
     send response_audio_start
     for chunk in tts_adapter.synthesize_stream(instant_reply):
         send binary chunk
     send response_audio_end
   else:
     generate full wav/base64 and call play_audio_base64
-> run full Hermes answer in background
-> optionally stream full answer audio later

Open decisions

  • Should Tactility voice app be a new side-loaded app or merged into existing ReynaBot/McpScreen?
  • Should first audio codec be PCM16 for speed of implementation, or Opus from day one?
  • Should the app use push-to-talk first, then wakeword later?
  • Which device is the first target: current 320x240 Tactility board, RLCD, or larger CrowPanel/JC board?
  • Should MOSS run on Mac mini .102, iMac .124, or FamReynaServer .110?

Suggested defaults:

  • New app: Apps/HermesVoice.
  • First codec: PCM16 chunks.
  • First interaction mode: push-to-talk or hold-to-talk.
  • First server: Mac mini .102 for TTS/STT engines; gateway remains :8642.
  • Add Opus after PCM streaming is proven.

Risks and mitigations

Risk Mitigation
ESP32 underruns during playback ring buffer, larger chunks, playback ACK/underrun telemetry
TTS engine not truly streaming chunk full WAV first; still improves protocol and app path
Opus decode complexity PCM first; borrow ElatoAI implementation later
Gateway route patch wiped by Hermes update keep restore plugin and restart guard bypass documented
Device CPU/memory pressure PSRAM buffers, mono 16/24 kHz, no UI redraw during playback
Barge-in race conditions explicit interrupt and playback_flush events with turn IDs

First actionable next step

Build a playback-only Tactility streaming audio proof:

  1. Write a tiny local WS server that sends:
    • response_audio_start
    • PCM16 chunks from a known WAV
    • response_audio_end
  2. Write/extend a Tactility app to connect and play those chunks via I2S.
  3. Once smooth, connect the same app to Hermes :8642 and let the gateway stream generated WAV chunks.
  4. Only then add MOSS-TTS-Nano true streaming and Opus.

This keeps risk low and gives immediate proof that Tactility can speak progressively instead of waiting for full base64 WAVs.

Reference in New Issue View Git Blame Copy Permalink