---
Date: 2026-07-17
Author: Hermes
Tags: [project, tactility, esp32, voice-gateway, realtime-voice, elatoai, moss-tts-nano, opus, streaming-tts]
Status: next-step-plan
Related: [[voice_agent_platform_research_2026]], [[grace_poppy_storybook_image_voice_app]], [[pocketbase_family_baas_2026]]
---

# Tactility Voice Gateway Transition Plan — ElatoAI + MOSS-TTS-Nano Pattern

## Why this exists

Current Reyna voice UX already has strong pieces:

- Hermes voice gateway on `:8642` receives **incoming mic PCM chunks** over WebSocket.
- Mac mini MCP at `.102` provides fast Apple Speech transcription and Apple 3B quick replies.
- iPhone/ESP32 screen clients can play returned audio through `play_audio_base64`.
- Hermes brings memory, skills, family brain context, tools, and routing.

The missing piece is **streamed speech back to the device**. Today, the gateway usually returns a full WAV/base64 blob, then the device starts playback only after the whole audio file exists.

ElatoAI appears close to the desired device architecture: ESP32 WebSocket audio, stateful voice turns, Opus binary packets back to the device, an audio ring buffer, I2S playback, and realtime toy/companion UX.

MOSS-TTS-Nano is interesting as a possible **streaming TTS engine**, especially because it advertises a 0.1B CPU/ONNX/MLX-friendly realtime path.

## Target end-state

```text
Tactility ESP32 app/device
  -> streams mic audio chunks over secure WebSocket
  -> Hermes voice gateway :8642
  -> Apple/Mac mini STT + Hermes/Muse/full agent answer
  -> streaming TTS engine: MOSS-TTS-Nano, Kokoro, Voicebox, or Apple say fallback
  -> Opus or PCM audio chunks over same WebSocket
  -> Tactility app decodes/queues/plays via I2S while chunks are still arriving
```

Design principle: **keep Hermes as the brain; borrow ElatoAI's realtime transport/playback architecture.**

## Current vs desired architecture

| Layer | Current setup | Desired transition |
|---|---|---|
| Device input | PCM chunks over WS already working | Keep, optionally support Opus mic later |
| STT | Mac mini SpeechAnalyzer/live pipe, local fallback | Keep; add latency telemetry per turn |
| Fast reply | Apple 3B quick reply + Mac mini TTS | Keep as first audible ACK |
| Full answer | Hermes background agent/session | Keep |
| TTS output | whole WAV/base64 blob via `play_audio_base64` | streamed audio chunks: Opus preferred, PCM16 fallback |
| Device playback | receive full file then play | ring buffer + decoder + I2S streaming task |
| State protocol | ready/listening/draft/transcript/instant_response/done | add audio_start/audio_chunk/audio_end/speaking/interrupted |
| Barge-in | partial / ad hoc | explicit interrupt: stop TTS, flush output, return to listening |

## Reference project: ElatoAI

Repository: `https://github.com/akdeb/ElatoAI`

Observed useful components:

- `firmware-arduino/src/Audio.cpp`
  - `WebSocketsClient`
  - `OpusAudioDecoder`
  - `BufferRTOS<uint8_t> audioBuffer`
  - `I2SStream i2s`
  - `QueueStream<uint8_t>`
  - `StreamCopy` for continuous playback
  - state transitions: `LISTENING`, `PROCESSING`, `SPEAKING`, `SLEEP`
- `server/fastapi/esp32_transport.py`
  - `RawPCMFrameSerializer`
  - `InputAudioRawFrame`
  - `OutputAudioRawFrame`
  - `OpusEncoder`
  - binary WebSocket audio frames
- `server/cloudflare/README.md`
  - protocol events: `auth`, `AUDIO.COMMITTED`, `RESPONSE.CREATED`, binary audio frames, `RESPONSE.COMPLETE`, `SESSION.END`
  - Opus packetized audio streamed back to ESP32

ElatoAI should be treated as a **transport/playback reference**, not necessarily a replacement for Hermes.

## Reference engine: MOSS-TTS-Nano

Repository: `https://github.com/OpenMOSS/MOSS-TTS-Nano`

Why test it:

- 0.1B parameters
- realtime speech generation target
- CPU-friendly claim
- ONNX CPU version
- MLX support for Apple Silicon
- streaming inference
- multilingual
- voice cloning workflow

Caveats:

- Not for running directly on ESP32.
- Native output is 48 kHz stereo; we likely need server-side resampling to 24 kHz or 16 kHz mono.
- Need to benchmark first-audio latency vs current Mac mini `say`, Kokoro, and Voicebox.

## Transition plan — Phase 0: baseline and compatibility

1. Capture current gateway behavior:
   - Record events emitted by `/api/esp32/voice/ws`.
   - Measure: mic stop -> transcript, instant reply text, first audio playback, full answer delivery.
   - Save sample turn logs in `~/.hermes/audio_cache` or a project scratch folder.
2. Inventory playback clients:
   - iPhone `.150` supports `play_audio_base64`, not arbitrary execution.
   - Tactility ESP32 app can be extended to support streaming WS + I2S playback.
3. Decide transport fallback order:
   - **Preferred:** Opus binary frames for low bandwidth and robust streaming.
   - **Fallback:** raw PCM16 chunks for simplest first implementation.
   - Keep full WAV/base64 as compatibility fallback.

## Transition plan — Phase 1: gateway streamed-output protocol

Add streamed TTS events to Hermes voice gateway without removing current behavior.

Proposed server -> device events:

```json
{"event":"response_audio_start","format":"pcm_s16le","sample_rate":24000,"channels":1,"codec":"pcm","turn_id":"..."}
```

Then binary frames:

```text
<raw PCM16 chunk bytes>
```

Then:

```json
{"event":"response_audio_end","turn_id":"...","duration_ms":1234}
```

For Opus mode:

```json
{"event":"response_audio_start","codec":"opus","sample_rate":24000,"channels":1,"frame_duration_ms":120,"bitrate":24000}
```

Binary frames are Opus packets.

Also add:

```json
{"event":"speaking","turn_id":"..."}
{"event":"interrupted","turn_id":"...","reason":"barge_in"}
{"event":"playback_flush","turn_id":"..."}
```

Implementation notes:

- Keep `instant_response` text behavior for UI stickiness.
- For current iPhone path, keep generating full base64 WAV.
- For Tactility app path, prefer stream mode if client advertises it during `start`:

```json
{"event":"start","audio_out":["opus","pcm_s16le","wav_base64"]}
```

## Transition plan — Phase 2: server-side TTS streaming adapter

Create a pluggable TTS output abstraction in the gateway:

```text
TTSAdapter.synthesize_stream(text, voice, sample_rate) -> async iterator[AudioChunk]
```

Adapters:

1. **PCM dummy adapter** — split an already generated WAV into PCM chunks. This validates the device protocol even before true streaming TTS.
2. **Kokoro adapter** — start with full generation then chunk output; useful quality baseline.
3. **MOSS-TTS-Nano adapter** — true streaming target; benchmark ONNX CPU and MLX paths.
4. **Voicebox adapter** — probably full generation first, then chunk output; useful for Aiden/character voices.
5. **Apple say adapter** — fast fallback, full generation then chunk.

Required telemetry:

- `tts_request_at`
- `first_audio_chunk_at`
- `last_audio_chunk_at`
- `playback_started_at` from device ack if available
- codec, sample rate, chunk size, underrun count

## Transition plan — Phase 3: Tactility app streaming audio client

Build a Tactility app, likely named one of:

- `Apps/VoiceGateway`
- `Apps/ReynaVoice`
- `Apps/HermesVoice`

Responsibilities:

1. Connect to Hermes voice gateway WS:
   - `ws://<gateway>:8642/api/esp32/voice/ws?device_id=<id>` or proxied HTTPS path.
   - Send auth bearer/config safely.
2. Capture mic audio:
   - 16 kHz mono PCM16 first.
   - Use VAD or push-to-talk first; wakeword later.
3. Send mic chunks as binary frames.
4. Receive JSON control events and binary audio frames.
5. Maintain state machine:
   - `IDLE`
   - `LISTENING`
   - `PROCESSING`
   - `SPEAKING`
   - `INTERRUPTING`
   - `ERROR`
6. Playback:
   - PCM mode first: binary chunk -> ring buffer -> I2S output task.
   - Opus mode next: binary packet -> Opus decoder -> ring buffer -> I2S.
7. UI:
   - big listening/speaking state text
   - live transcript/draft
   - quick reply text
   - connection/auth status
   - mute/volume buttons
8. Barge-in:
   - if mic detects speech while `SPEAKING`, send `interrupt` event
   - stop/flush output buffer
   - transition back to listening

## Tactility app implementation strategy

Start with a minimal app rather than a full ElatoAI firmware fork.

Reuse from existing local work:

- `Apps/McpScreen` for screen/audio tool patterns.
- `Apps/AudioTest` / `BookPlayer` / any I2S examples for playback.
- `Apps/ReynaBot` if it already has network/client patterns.
- ElatoAI's audio task design for ring buffer + decoder + state transitions.

Milestones:

### Milestone A — playback-only streaming test

- Device connects to a local test WS endpoint.
- Server sends generated PCM chunks from a known WAV.
- Device plays chunks smoothly.
- Measure underruns and latency.

### Milestone B — gateway stream compatibility

- Gateway sends `response_audio_start`, binary PCM chunks, `response_audio_end`.
- Tactility app plays the response.
- Keep mic input disabled or mocked.

### Milestone C — full duplex-ish voice turn

- Tactility app records PCM chunks to gateway.
- Gateway transcribes and replies.
- Gateway streams output chunks back.
- Device plays without waiting for full WAV.

### Milestone D — Opus mode

- Add Opus encoder on gateway side.
- Add Opus decoder on device side, borrowing from ElatoAI if licensing is acceptable.
- Compare bandwidth/CPU/latency vs PCM.

### Milestone E — barge-in and polish

- Device can interrupt speech.
- Gateway cancels TTS generation/background playback.
- UI shows listening/thinking/speaking states.
- Add volume control and fallback to full WAV/base64 where needed.

## Gateway implementation strategy

Do not rewrite the gateway first. Add a parallel streamed-output path.

1. Extend existing `/api/esp32/voice/ws` with output capability negotiation.
2. Keep current instant ACK and full-answer background flow.
3. For streamed output, first chunk existing generated WAVs to the device.
4. Once playback is solid, swap in MOSS-TTS-Nano/Kokoro streaming adapter.
5. Add Opus encoder after PCM path works.

Pseudo-flow:

```text
receive stop
-> final transcript
-> instant quick reply
-> if client supports stream_out:
     send response_audio_start
     for chunk in tts_adapter.synthesize_stream(instant_reply):
         send binary chunk
     send response_audio_end
   else:
     generate full wav/base64 and call play_audio_base64
-> run full Hermes answer in background
-> optionally stream full answer audio later
```

## Open decisions

- Should Tactility voice app be a new side-loaded app or merged into existing ReynaBot/McpScreen?
- Should first audio codec be PCM16 for speed of implementation, or Opus from day one?
- Should the app use push-to-talk first, then wakeword later?
- Which device is the first target: current 320x240 Tactility board, RLCD, or larger CrowPanel/JC board?
- Should MOSS run on Mac mini `.102`, iMac `.124`, or FamReynaServer `.110`?

Suggested defaults:

- New app: `Apps/HermesVoice`.
- First codec: PCM16 chunks.
- First interaction mode: push-to-talk or hold-to-talk.
- First server: Mac mini `.102` for TTS/STT engines; gateway remains `:8642`.
- Add Opus after PCM streaming is proven.

## Risks and mitigations

| Risk | Mitigation |
|---|---|
| ESP32 underruns during playback | ring buffer, larger chunks, playback ACK/underrun telemetry |
| TTS engine not truly streaming | chunk full WAV first; still improves protocol and app path |
| Opus decode complexity | PCM first; borrow ElatoAI implementation later |
| Gateway route patch wiped by Hermes update | keep restore plugin and restart guard bypass documented |
| Device CPU/memory pressure | PSRAM buffers, mono 16/24 kHz, no UI redraw during playback |
| Barge-in race conditions | explicit `interrupt` and `playback_flush` events with turn IDs |

## First actionable next step

Build a **playback-only Tactility streaming audio proof**:

1. Write a tiny local WS server that sends:
   - `response_audio_start`
   - PCM16 chunks from a known WAV
   - `response_audio_end`
2. Write/extend a Tactility app to connect and play those chunks via I2S.
3. Once smooth, connect the same app to Hermes `:8642` and let the gateway stream generated WAV chunks.
4. Only then add MOSS-TTS-Nano true streaming and Opus.

This keeps risk low and gives immediate proof that Tactility can speak progressively instead of waiting for full base64 WAVs.
