FamReynaBrain/projects/tactility_voice_gateway_elato_transition_2026.md

---

Date: 2026-07-17 Author: Hermes Tags: [project, tactility, esp32, voice-gateway, realtime-voice, elatoai, moss-tts-nano, opus, streaming-tts] Status: next-step-plan Related: voice_agent_platform_research_2026, grace_poppy_storybook_image_voice_app, pocketbase_family_baas_2026

Tactility Voice Gateway Transition Plan — ElatoAI + MOSS-TTS-Nano Pattern

Why this exists

Current Reyna voice UX already has strong pieces:

• Hermes voice gateway on :8642 receives incoming mic PCM chunks over WebSocket.
• Mac mini MCP at .102 provides fast Apple Speech transcription and Apple 3B quick replies.
• iPhone/ESP32 screen clients can play returned audio through play_audio_base64.
• Hermes brings memory, skills, family brain context, tools, and routing.

The missing piece is streamed speech back to the device. Today, the gateway usually returns a full WAV/base64 blob, then the device starts playback only after the whole audio file exists.

ElatoAI appears close to the desired device architecture: ESP32 WebSocket audio, stateful voice turns, Opus binary packets back to the device, an audio ring buffer, I2S playback, and realtime toy/companion UX.

MOSS-TTS-Nano is interesting as a possible streaming TTS engine, especially because it advertises a 0.1B CPU/ONNX/MLX-friendly realtime path.

Target end-state

Tactility ESP32 app/device
  -> streams mic audio chunks over secure WebSocket
  -> Hermes voice gateway :8642
  -> Apple/Mac mini STT + Hermes/Muse/full agent answer
  -> streaming TTS engine: MOSS-TTS-Nano, Kokoro, Voicebox, or Apple say fallback
  -> Opus or PCM audio chunks over same WebSocket
  -> Tactility app decodes/queues/plays via I2S while chunks are still arriving

Design principle: keep Hermes as the brain; borrow ElatoAI's realtime transport/playback architecture.

Current vs desired architecture

Layer	Current setup	Desired transition
Device input	PCM chunks over WS already working	Keep, optionally support Opus mic later
STT	Mac mini SpeechAnalyzer/live pipe, local fallback	Keep; add latency telemetry per turn
Fast reply	Apple 3B quick reply + Mac mini TTS	Keep as first audible ACK
Full answer	Hermes background agent/session	Keep
TTS output	whole WAV/base64 blob via play_audio_base64	streamed audio chunks: Opus preferred, PCM16 fallback
Device playback	receive full file then play	ring buffer + decoder + I2S streaming task
State protocol	ready/listening/draft/transcript/instant_response/done	add audio_start/audio_chunk/audio_end/speaking/interrupted
Barge-in	partial / ad hoc	explicit interrupt: stop TTS, flush output, return to listening

Reference project: ElatoAI

Repository: https://github.com/akdeb/ElatoAI

Observed useful components:

• firmware-arduino/src/Audio.cpp
  • WebSocketsClient
  • OpusAudioDecoder
  • BufferRTOS<uint8_t> audioBuffer
  • I2SStream i2s
  • QueueStream<uint8_t>
  • StreamCopy for continuous playback
  • state transitions: LISTENING, PROCESSING, SPEAKING, SLEEP
• server/fastapi/esp32_transport.py
  • RawPCMFrameSerializer
  • InputAudioRawFrame
  • OutputAudioRawFrame
  • OpusEncoder
  • binary WebSocket audio frames
• server/cloudflare/README.md
  • protocol events: auth, AUDIO.COMMITTED, RESPONSE.CREATED, binary audio frames, RESPONSE.COMPLETE, SESSION.END
  • Opus packetized audio streamed back to ESP32

ElatoAI should be treated as a transport/playback reference, not necessarily a replacement for Hermes.

Reference engine: MOSS-TTS-Nano

Repository: https://github.com/OpenMOSS/MOSS-TTS-Nano

Why test it:

• 0.1B parameters
• realtime speech generation target
• CPU-friendly claim
• ONNX CPU version
• MLX support for Apple Silicon
• streaming inference
• multilingual
• voice cloning workflow

Caveats:

• Not for running directly on ESP32.
• Native output is 48 kHz stereo; we likely need server-side resampling to 24 kHz or 16 kHz mono.
• Need to benchmark first-audio latency vs current Mac mini say, Kokoro, and Voicebox.

Transition plan — Phase 0: baseline and compatibility

1. Capture current gateway behavior:
   • Record events emitted by /api/esp32/voice/ws.
   • Measure: mic stop -> transcript, instant reply text, first audio playback, full answer delivery.
   • Save sample turn logs in ~/.hermes/audio_cache or a project scratch folder.
2. Inventory playback clients:
   • iPhone .150 supports play_audio_base64, not arbitrary execution.
   • Tactility ESP32 app can be extended to support streaming WS + I2S playback.
3. Decide transport fallback order:
   • Preferred: Opus binary frames for low bandwidth and robust streaming.
   • Fallback: raw PCM16 chunks for simplest first implementation.
   • Keep full WAV/base64 as compatibility fallback.

Transition plan — Phase 1: gateway streamed-output protocol

Add streamed TTS events to Hermes voice gateway without removing current behavior.

Proposed server -> device events:

{"event":"response_audio_start","format":"pcm_s16le","sample_rate":24000,"channels":1,"codec":"pcm","turn_id":"..."}

Then binary frames:

<raw PCM16 chunk bytes>

Then:

{"event":"response_audio_end","turn_id":"...","duration_ms":1234}

For Opus mode:

{"event":"response_audio_start","codec":"opus","sample_rate":24000,"channels":1,"frame_duration_ms":120,"bitrate":24000}

Binary frames are Opus packets.

Also add:

{"event":"speaking","turn_id":"..."}
{"event":"interrupted","turn_id":"...","reason":"barge_in"}
{"event":"playback_flush","turn_id":"..."}

Implementation notes:

• Keep instant_response text behavior for UI stickiness.
• For current iPhone path, keep generating full base64 WAV.
• For Tactility app path, prefer stream mode if client advertises it during start:

{"event":"start","audio_out":["opus","pcm_s16le","wav_base64"]}

Transition plan — Phase 2: server-side TTS streaming adapter

Create a pluggable TTS output abstraction in the gateway:

TTSAdapter.synthesize_stream(text, voice, sample_rate) -> async iterator[AudioChunk]

Adapters:

1. PCM dummy adapter — split an already generated WAV into PCM chunks. This validates the device protocol even before true streaming TTS.
2. Kokoro adapter — start with full generation then chunk output; useful quality baseline.
3. MOSS-TTS-Nano adapter — true streaming target; benchmark ONNX CPU and MLX paths.
4. Voicebox adapter — probably full generation first, then chunk output; useful for Aiden/character voices.
5. Apple say adapter — fast fallback, full generation then chunk.

Required telemetry:

• tts_request_at
• first_audio_chunk_at
• last_audio_chunk_at
• playback_started_at from device ack if available
• codec, sample rate, chunk size, underrun count

Transition plan — Phase 3: Tactility app streaming audio client

Build a Tactility app, likely named one of:

• Apps/VoiceGateway
• Apps/ReynaVoice
• Apps/HermesVoice

Responsibilities:

1. Connect to Hermes voice gateway WS:
   • ws://<gateway>:8642/api/esp32/voice/ws?device_id=<id> or proxied HTTPS path.
   • Send auth bearer/config safely.
2. Capture mic audio:
   • 16 kHz mono PCM16 first.
   • Use VAD or push-to-talk first; wakeword later.
3. Send mic chunks as binary frames.
4. Receive JSON control events and binary audio frames.
5. Maintain state machine:
   • IDLE
   • LISTENING
   • PROCESSING
   • SPEAKING
   • INTERRUPTING
   • ERROR
6. Playback:
   • PCM mode first: binary chunk -> ring buffer -> I2S output task.
   • Opus mode next: binary packet -> Opus decoder -> ring buffer -> I2S.
7. UI:
   • big listening/speaking state text
   • live transcript/draft
   • quick reply text
   • connection/auth status
   • mute/volume buttons
8. Barge-in:
   • if mic detects speech while SPEAKING, send interrupt event
   • stop/flush output buffer
   • transition back to listening

Tactility app implementation strategy

Start with a minimal app rather than a full ElatoAI firmware fork.

Reuse from existing local work:

• Apps/McpScreen for screen/audio tool patterns.
• Apps/AudioTest / BookPlayer / any I2S examples for playback.
• Apps/ReynaBot if it already has network/client patterns.
• ElatoAI's audio task design for ring buffer + decoder + state transitions.

Milestones:

Milestone A — playback-only streaming test

• Device connects to a local test WS endpoint.
• Server sends generated PCM chunks from a known WAV.
• Device plays chunks smoothly.
• Measure underruns and latency.

Milestone B — gateway stream compatibility

• Gateway sends response_audio_start, binary PCM chunks, response_audio_end.
• Tactility app plays the response.
• Keep mic input disabled or mocked.

Milestone C — full duplex-ish voice turn

• Tactility app records PCM chunks to gateway.
• Gateway transcribes and replies.
• Gateway streams output chunks back.
• Device plays without waiting for full WAV.

Milestone D — Opus mode

• Add Opus encoder on gateway side.
• Add Opus decoder on device side, borrowing from ElatoAI if licensing is acceptable.
• Compare bandwidth/CPU/latency vs PCM.

Milestone E — barge-in and polish

• Device can interrupt speech.
• Gateway cancels TTS generation/background playback.
• UI shows listening/thinking/speaking states.
• Add volume control and fallback to full WAV/base64 where needed.

Gateway implementation strategy

Do not rewrite the gateway first. Add a parallel streamed-output path.

1. Extend existing /api/esp32/voice/ws with output capability negotiation.
2. Keep current instant ACK and full-answer background flow.
3. For streamed output, first chunk existing generated WAVs to the device.
4. Once playback is solid, swap in MOSS-TTS-Nano/Kokoro streaming adapter.
5. Add Opus encoder after PCM path works.

Pseudo-flow:

receive stop
-> final transcript
-> instant quick reply
-> if client supports stream_out:
     send response_audio_start
     for chunk in tts_adapter.synthesize_stream(instant_reply):
         send binary chunk
     send response_audio_end
   else:
     generate full wav/base64 and call play_audio_base64
-> run full Hermes answer in background
-> optionally stream full answer audio later

Open decisions

• Should Tactility voice app be a new side-loaded app or merged into existing ReynaBot/McpScreen?
• Should first audio codec be PCM16 for speed of implementation, or Opus from day one?
• Should the app use push-to-talk first, then wakeword later?
• Which device is the first target: current 320x240 Tactility board, RLCD, or larger CrowPanel/JC board?
• Should MOSS run on Mac mini .102, iMac .124, or FamReynaServer .110?

Suggested defaults:

• New app: Apps/HermesVoice.
• First codec: PCM16 chunks.
• First interaction mode: push-to-talk or hold-to-talk.
• First server: Mac mini .102 for TTS/STT engines; gateway remains :8642.
• Add Opus after PCM streaming is proven.

Risks and mitigations

Risk	Mitigation
ESP32 underruns during playback	ring buffer, larger chunks, playback ACK/underrun telemetry
TTS engine not truly streaming	chunk full WAV first; still improves protocol and app path
Opus decode complexity	PCM first; borrow ElatoAI implementation later
Gateway route patch wiped by Hermes update	keep restore plugin and restart guard bypass documented
Device CPU/memory pressure	PSRAM buffers, mono 16/24 kHz, no UI redraw during playback
Barge-in race conditions	explicit interrupt and playback_flush events with turn IDs

First actionable next step

Build a playback-only Tactility streaming audio proof:

1. Write a tiny local WS server that sends:
   • response_audio_start
   • PCM16 chunks from a known WAV
   • response_audio_end
2. Write/extend a Tactility app to connect and play those chunks via I2S.
3. Once smooth, connect the same app to Hermes :8642 and let the gateway stream generated WAV chunks.
4. Only then add MOSS-TTS-Nano true streaming and Opus.

This keeps risk low and gives immediate proof that Tactility can speak progressively instead of waiting for full base64 WAVs.