# Future Optimizations — ESP32 Voice Gateway (M2 native path + FreeFlow learnings)

> Goal: WS `/api/voice/live` single websocket endpoint (compat with current `/api/esp32/voice/ws` framing binary PCM s16le 16k mono) — instant captioning feedback, fast local LLM initial voice response custom based on partial captioning, empty audio / stream stopped trigger, multi-partials timed to audio tail while big LLM works. Native M2 Air implementation, no Mac mini hop. Keep ESP32 compat by endpoint shape, but design doc is device-agnostic.

This doc captures all learnings from July 2026 audit of Pi5 `api_server.py` custom gateway (6949 vs 5124 stock, 1675-line patch), Mac mini ANE accelerator (`:7331/mcp` `speech_live_transcribe` ApplePipeTranscriber volatile 60ms, `apple_llm_quick_reply` Foundation 3B ANE 258-498ms, `speech_synthesize_base64` Alex <500ms / `voicebox_quick_reply` Aiden Qwen 1.7B), desktop hooks `use-voice-conversation` idle→listening→transcribing→thinking→speaking VAD 0.075/1250ms, 60s cap, 3-strike halt, and FreeFlow/macOS SpeechAnalyzer patterns.

---

## 0. Current perf baseline (Pi5) — reference

- Live draft: snapshot `pcm_buffer` → in-mem WAV → base64 → HTTP POST to Mac mini `speech_live_transcribe` — 800ms interval `ESP32_LIVE_DRAFT_INTERVAL_MS=800` + `MIN_BYTES=8000`, dedup via `last_draft_text`. Events `ready → listening → draft/interim_transcript → thinking{transcribing} → transcript/final → instant_response/quick_reply → instant_audio → response_text → done{background} + thinking{waiting}`
- Quick ACK: draft >30 chars → `apple_llm_quick_reply {draft, context, instructions}` else use final `transcript[:400]` — guard generic "Perfect." triggers regenerate from final
- Instant audio: `speech_synthesize_base64 {text, voice:"Alex"}` boy voice <500ms preferred, fallback `voicebox_quick_reply profile:"Aiden"` Qwen3-TTS 1.7B 2-4s, pushed via Screen MCP `play_audio_base64 wav_base64 volume80` to `http://<source_ip>/api/mcp` (resolved from header `X-Hermes-Screen-Url` > query > `X-Device-ID=iphone*` lookup `local_devices.yaml` > peer_ip)
- Final STT: same Mac mini pipe full buffer `macmini-live-final` 753ms vs local faster-whisper base 11-13s cold fallback
- Full turn: persistent `esp32:{device_id}` / `voice:{id}` conversation via `response_store`, tool-use capable (calendar, memory, kanban, ESP32 `draw_text` etc.)
- TTS chain: Edge AriaNeural (Pi default, free 4.3 MOS) mp3→ffmpeg `16k mono s16le WAV`
- Audit: `~/.hermes/cache/audio/esp32/ + voice_requests.jsonl`
- Failure mode: raw STT is ground truth — hears "boy" instead of "voice" for voice domain, model keeps wrong transcript, answers about boy mode. Mac mini down (it is now — :7331 refused) kills all live.
- Patch fragility: `api_server.py` 6949 LOC patch breaks on `hermes update` — this plugin repo fixes via `~/.hermes/plugins/` auto-restore.

---

## 1. Pain Points — ranked, with origin

| # | Pain | Impact | Where | OSS ref / inspiration |
|---|------|--------|-------|----------------------|
| P0 | Mac mini MCP single point failure `:7331/mcp` down now | No live drafts, no quick ack, no instant audio — full gateway degradation to slow base | Pi5 gateway uses RPC instead of local Swift pipe | FreeFlow alternative: on-device `SpeechAnalyzer` 26.5 + `FoundationModels` 3B direct ANE no hop |
| P0 | `api_server.py` patch 1675 lines wiped by `hermes update` | Must restore, flakey 3-strike | Pi5 `git stash --include-untracked` + pull | This repo `hermes_plugin/__init__.py` auto-restore — TODO true plugin injection without file touch |
| P1 | Ground-truth lock: `boy → voice` kept as truth | Hermes tool calls wrong thing | `f'[Voice input...]\n{transcript}'` implies perfect | **FreeFlow** never treats raw as final — 3-stage polish + skip heuristic + anti-truncation guard |
| P1 | Disk IO storm: `raw_path` .s16le temp file per chunk | 60 chunks/min file writes unneeded | Pi5 WS handler writes every frame | FreeFlow + new spec: `ByteArray` only, one WAV encode at stop — `engine_apple_transcribe.py` `ApplePipeTranscriber` |
| P1 | No streaming LLM→TTS sentence queue | Whole answer TTS at end, perceived latency +2s even after ACK | Desktop + ESP32 background | **Pipecat** sentence aggregator + Cartesia streaming; `voice-playback.ts` already has `takeSpeechChunk /.+?[.!?]/` + 15s stall guard |
| P2 | Single VAD (RMS 0.075) tuned once, not adaptive | near-field AirPods vs far-field built-in built-in speech 0.002-0.005 RMS same threshold → silent presses rejected as speech or vice versa | Pi5 `SILENCE_RMS_THRESHOLD 200` vs desktop `0.075` | FreeFlow `effectiveSilenceThreshold()` far-field 0.001, near-field `max(ambient*1.2,0.0005)` capped 0.01; **Silero VAD 1MB** unified |
| P2 | Empty audio trigger missing as intentional signal | Tap no speech should = "yes?" prompt not dropped | Pi5 `len < threshold` error path | User spec: <1600B = 50ms guard treat as immediate final / clarification trigger |
| P3 | PIL UDP spam for screen animations | Network spam per draft | `_send_screen_animation` per draft | Skip diff, throttle 200ms + only on state change `ready/listening/thinking/speaking` |

---

## 2. FreeFlow (mrinalwadhwa/freeflow) deep dive — what to steal

Source: https://github.com/mrinalwadhwa/freeflow 91★ Apache-2.0 macOS 14+, dictation-anywhere hotkey app. Not voice assistant, but polish pipeline is gold reusable.

**Why it matters for voice gateway:**
FreeFlow problem is same: Apple SpeechAnalyzer raw is 2.12% WER 196M EN / 174M ES but robotic, homophones, spurious commas. It never trusts STT. Median keyRelease→injection **0.55s**. Warm backup WS pre-opened — 91% zero handshake. 83% skip LLM polish entirely via local heuristic. When polish needed, `gpt-4.1-nano` 320-780ms (original) but macOS 26 path uses on-device `SpeechAnalyzerDictationProvider` + `FoundationModelChatClient` local — same pattern as our ANE 3B.

**Code audited July 2026:**
- `FreeFlowKit/Sources/FreeFlowKit/Services/PolishPipeline.swift` `polishModel = "gpt-5.4-nano"` — 43 punctuation rules, `<keep>` tags, `collapseAdjacentPunctuation`, `cleanSpuriousCommas`, `stripNoisePhrases`, `stripFillerSounds`, `buildCloudSystemPrompt(context)`, `knownTerms` (Redis, Postgres, macOS… domain terms capitalized), `guard_against_truncation` (commit `7f96ccd Preload models on startup and guard against aggressive truncation` 2mo ago), `word_overlap_ratio` concept, `stripKeepTags` → `\n\n`, `sanitizeContextField` 80 chars `focusedFieldContent`, `toneLabel(for: bundleID) casual`
- Prompts: `PolishPromptEnglish.swift` huge tested 80-line system prompt: removes throat-clearing (`let me think, hold on, how do I put this, let me recall` → keep actual content), self-correction `no wait, actually, sorry, I mean, let me rephrase, never mind, or rather, make that → keep final`, abandon restart `no, forget it, nah, that won't work → drop before`, stutter double `the the → the` but `wait wait wait → Wait, wait, wait,` comma keep same sentence, `2-item inline never list`, `3+ items always vertical list with - or 1. 2. 3. if ordered`, recapping meeting format `- Person: update`, numbers always digits `one → 1`, `minus ten → -10`, phone dashes, email `john at example dot com → john@example.com`, preserve `kinda/gonna/wanna` contractions, `<keep>` must preserve exactly
- `PolishPromptLocal.swift` `systemPromptLocal = "Clean up this dictated text. Return only the cleaned text."` — for FoundationModels 3B ANE short window 4096 tokens guardrails
- `DictationPipeline.swift` `DictationPipeline: PipelineProviding` actor: `activate() → recording` immediate HUD return, audio setup in Detached task (AVAudioEngine 500-900ms + BT SCO 3s timeout `detachedWithTimeout`), `pendingContext` parallel, adaptive `effectiveSilenceThreshold()` far-field 0.001 vs near-field ambient*1.2 capped 0.01 floor 0.0005 multiplier 1.2, early silence short-circuit peakRMS <= threshold after 200ms = 4 ticks 50ms, streaming session reuse adoption `freshConnection vs adoptedBackup vs adoptedStaleBackup maxBackupAge 180s`, chunk buffer `rawChunkBuffer` until `endsAtSentenceBoundary` then `polishChunk()` + inject, rolling commit chunker `TimeAndSilenceChunkingStrategy` speechDebounce 10s (only genuine extended silence triggers commit not thinking pauses), `ChunkInjectedFlag` thread-safe, `RecoveryBox` audio saved for retry, `minimumAudioDuration 0.1s`, `SessionTiming` diagnostic

**Borrow list for this gateway:**

- **Deterministic pipeline (0 ms, no key):** `substituteDictatedPunctuation` + `cleanSpuriousCommas` + `stripFillerSounds` + `collapseAdjacentPunctuation` — already ported partially in `~/Projects/pythonwhisper/freeflow_polish.py` (`deterministic_polish()`, `substitute_dictated_punctuation`, `strip_keep_tags`, `normalize_formatting`, `capitalize_known_terms`, `collapse_adjacent_punctuation`, `strip_noise_phrases`, `strip_filler_sounds`, `clean_spurious_commas`) — KNOWN_TERMS currently DB/infra only, add Hermes domain
- **Skip heuristic 83% rate:** `is_clean_enough_to_skip_llm()` — if ends sentence + no fillers regex + no repeat `(\w+)\s+\1` → skip expensive LLM — medians 0.55s — implement in WS before ANE LLM call
- **Guards:** `guard_against_truncation len(polished)/len(pre) <0.25 → fallback`, `word_overlap_ratio <0.45 → fallback`, `match_input_casing` leading lowercase preserve when continuing sentence (`precedingText` mid-sentence `endsAtSentenceBoundary` false)
- **Chunk buffer:** accumulate volatile intermediates `rawChunkBuffer` until sentence boundary, then polish batch — don't polish volatile `speecheech → speech` word-level jitter per draft, only on sentence boundary or final — reduces ANE calls 31 drafts → ~5 polishes per 6.3s audio
- **Adaptive silence:** replace fixed 200 / 0.075 with FreeFlow's proximity-aware adaptive — use same for client AudioWorklet level + server VAD — far-field built-in needs 0.001 floor, near-field AirPods ambient measured
- **Warm backup:** pre-open Swift pipe daemons (`apple-speech-transcribe --pipe volatile` + `apple-llm-polish` JSONL pipe keeping `LanguageModelSession` KV warm) — already validated `pythonwhisper` `ApplePipeTranscriber` keep-alive `stdin 4-byte BE len + wav` → `stdout JSONL draft/final 60ms word-level 31 drafts`, fix `exit(0)` hang by return naturally not Process exit, `readExact` nil on EOF, `PYTHONPATH=""` isolate PIL `_imaging` collision
- **Context injection:** `AppContext` 80 chars last `focusedFieldContent` + preceding lines + tone casual → use for polish; for voice gateway = last 2 finals `refined_context_history deque maxlen2` + conversation summary `esp32:{id}`

---

## 3. OSS V2V Landscape 2025-26 — where we sit

Family 1 — **Pipeline orchestrators** (STT → LLM → TTS, text middle = tool-use, modular):

- **Pipecat (Daily pipecat-ai/pipecat) 6.5k★**: Python frame DAG, Daily/WebRTC transport `VAD → STT → LLM → TTS`, context aggregator keeps partials + tool results, interruption via VAD confidence + cancel + preserve context, TurnTracker `user_start → vad_silence → user_stop → thinking → bot_start` enum with `heardSpeechRef`, token streaming + sentence aggregator + Cartesia/Eleven. Lat 800-1200ms e2e $0.10/min Deepgram+Cartesia. **Steal:** TurnTracker + aggregator + barge-in cancel TTS keep partial LLM context + parallel tool calls without dropping audio + e2e tester audio file → asserts caption+timeline
- **LiveKit Agents + TEN (Agora)**: same frame pipeline, WebRTC room, graph editor Rust core extensions vision. **Steal:** visual flow state JSON for debugging `ready/listening/captioning/thinking/speaking`, SFU recording infra mature but cloud

Family 2 — **End-to-end speech models** (no text middle, full-duplex UX magic, but poor tool-use, GPU-hungry):

- **Moshi — Kyutai Apache 2.0**: Mimi codec 12.5Hz, 7B full-duplex inner monologue + 2 audio streams simultaneous (user+bot), 160ms theoretical, 2.2GB quant, A100/CUDA needed, weak function calling hallucinates. **Steal:** dual-stream idea — keep STT open while TTS playing via `echoCancellation:true` + second AnalyserNode RMS >0.12 while speaking → half→full duplex for free
- **Qwen2.5-Omni / Mini-Omni2 / Ultravox / Freeze-Omni**: Whisper encoder + LLM + soundstorm codec audio in → thought text → audio out, raw audio encoder brief → no STT errors emotion speaker id, but 3-5s TTFT, large VRAM, needs fine-tune for tools. **Note:** text middle needed for Hermes because calendar/memory/kanban tool calling via Claude Spark > omni accuracy; skip STT path would need audio→LLM direct that can't yet call tools reliably

**Our position — hybrid unique 2026:**
- Only project with **Apple Speech 26 pipe volatile 60ms 2.12% WER 196M EN** + **Foundation 3B on-device ANE 258-498ms both local** — FreeFlow validated 27-38x RTF 0mW ANE 7W spike single-core XPC `localspeechrecognition`
- **Fast voice from partials contextual not generic** — no OSS does this; Moshi inner thought generic "yeah", Pipecat none by default, we have draft >30 chars → Foundation quick ack `"On it — checking your calendar for tomorrow 3 meetings…"` already measured
- **Timed multi-partials while tool runs** — Pipecat usually waits all tools then speaks; we want instant → progress "found 3 events" → final timed to audio tail queue **sentence buffer `/.+?[.!?]/`** + start next in last 300ms prev tail + crossfade 150ms
- **In-mem only, no ffmpeg, no temp wav per chunk** — most OSS still writes file per chunk; we already have `ByteArray` only path in spec
- **WS compat** `WS /api/voice/live` same binary PCM s16le 16k framing as Pi5 ESP32 WS — new design says "keep endpoint with websocket" compatible but diagrams show no ESP32/other devices focus on MacBook only — so web can send WebM/opus or PCM flag, backward compat for future hardware
- **Offline $0 7W vs cloud $0.10/min + net jitter**
- Latency budget native: **VAD 1250ms + fast LLM 300ms + Kokoro 200ms = 1750ms first audio**, caption 60ms instant feels <1s streaming, big LLM token from 200ms streaming sentence queue. vs Pipecat ~2200ms cloud, Moshi ~600ms magic but no tools, Qwen-Omni 2.8-4s

---

## 4. Ground-truth lock fix — 3-layer defense (key feature request)

User worry: Pi hears "boy instead of voice" and model keep it as ground truth. FreeFlow fix is not trust acoustic as truth but semantic context.

**Before (current):**
```python
user_message = f'[Voice input from ESP32 device "{device_id}"]\n{transcript}'
```
=> Hermes thinks typed ground truth = boy mode.

**After — 3 layers:**

**L1 deterministic 0ms:** `deterministic_polish()` from `freeflow_polish.py` — strips fillers/noise, collapses `,,,`, Apple spurious commas — boy still boy here

**L2 semantic ANE 3B with context (fixes boy→voice):**
- `precedingText = last 2 finals + voice:{id} summary` (FreeFlow `refined_context_history deque 2`)
- Prompt includes domain `KNOWN_TERMS += voice, gateway, hotkey, dictate, Kokoro, VAD, barge-in, ANE, SpeechAnalyzer, DictationPipeline, FreeFlowKit`
- FreeFlow `systemPromptEnglish` full (80 lines tested) — self-correction `I mean…` → keep final, etc.
- ANE 3B `polish_line(substituted, prev1, prev2)` → if context = "voice gateway setup", "boy" acoustically close → semantic mismatch → corrects "voice"
- Guards: truncate 0.25 / overlap 0.45 / casing / keep tags
- Alternatives: Apple `SpeechTranscriber` attribute options `[.alternativeTranscriptions]` returns list — already parsed in pipe `--pipe` JSONL `alternatives` — pass to LLM: `Alternatives: boy mode (0.4), voice mode (0.6)` — model picks semantic fit

**L3 Hermes LLM disclaimer (your ask) + multi-partial ask confirm:**

Build `user_message` with explicit uncertainty + polished + raw + alternatives + instruction:

```
[Voice input — speech-to-text, may contain errors]
This was spoken, not typed. Homophones like
boy/voice, four/for, new/knew, dictate/dictation 
are common STT errors. Don't treat transcription
as ground truth.

Polished transcript (best guess, 1 edit, overlap 0.83):
"Set voice mode"

Raw STT alternatives (low confidence word jitter):
• "set boy mode" (raw final)
• "set voice mode" (draft volatile alternative, 3x seen)
Draft volatile instability is normal (speecheech → 
speech final) — don't treat as rewrite.

Context: user was talking about voice gateway setup
for MacBook native, previous turn: "add optimization
doc with FreeFlow ref".

If request unclear or semantically off in voice gateway
context, paraphrase back what you understood and ask for
confirmation before acting. Don't hallucinate tool call
on shaky transcript. Example: "Just to confirm — you mean
voice mode?"

User originally said (as heard / best):
"Set voice mode"
```

Ephemeral system add for whole `voice:{id}` session: `"Voice session: STT imperfect 2.12% WER even polished ANE. If confidence low or term out-of-domain, clarify."` + Hermes domain bias list.

This makes big LLM say "Just to confirm — you mean voice mode?" if still ambiguous instead of silently wrong. Fast ACK uses same polished transcript so <1.7s audio also correct.

---

## 5. Native M2 Air target — single WS endpoint spec (user final ask)

**Endpoint: `WS /api/voice/live`** (keep shape compat with Pi5 `/api/esp32/voice/ws` for future ESP32, but current design MacBook only focus)

**Transport:**
- Client → Server: binary `PCM s16le 16k mono` chunks 20ms AudioWorklet (replaces `ScriptProcessor`), OR empty binary (<1600B = 50ms guard = trigger), JSON control `{start:{format:"pcm_s16le"|"webm", sample_rate:16000, locale:"en-US"}, stop, ping}`
- Server → Client: `{ready} → listening{threshold} → draft volatile{draft,isFinal:false,is_volatile, alternatives, level} every ~400ms → interim_transcript → thinking{transcribing} → transcript final{polished,true, overlap, raw, alts, source:"apple" } → instant_response{quick, draft_count} contextual fast ack → instant_audio{b64/wavBase64, voice, quick_ms,speak_ms} → response_text streaming token via sentence buffer → audio_chunk{audio_b64, text, seq} timed to audio tail (start N in last 300ms of N-1 + 150ms crossfade) → done{background, instant_ok, keep_instant} + thinking{stage:waiting}`

**Instant caption feedback:**
- Apple `SpeechAnalyzer --pipe volatile` keep-alive single process, stdin 4-byte BE len + wav bytes len 0 = EOF, stdout JSONL draft/final word-level per chunk (~60ms granularity, 31 drafts/6.3s validated, EN 196M ES 174M), `bestAvailableAudioFormat` + `AVAudioConverter` + `AudioBufferQueue` actor, report `[.volatileResults, .alternativeTranscriptions]` attribute `[.audioTimeRange, .transcriptionConfidence]`. Volatile jitter normal.
- In-mem only `ByteArray` accumulator — zero disk, dedup `last_draft_text` same as Pi5 `_do_live_draft` but no file. Composer desktop shows italic .55 live while listening, solid on final `transcript`

**Trigger:**
- VAD 1250ms dual `effectiveSilenceThreshold` FreeFlow adaptive (far-field built-in 0.001, near-field max(ambient*1.2,0.0005) capped 0.01) — same threshold client AudioWorklet RMS AnalyserNode + server — `heardSpeechRef` pattern from `use-mic-recorder` + Pi5 3-strike `CONTINUOUS_NO_SPEECH_LIMIT=3` halt
- Empty audio trigger: binary <1600B or `{start}` then immediate `{stop}` → freeze draft → final transcript immediately → if still <2 words → prompt "yes?" clarification via fast ACK, not error — user spec

**Fast local LLM initial voice response custom based on partial captioning:**
- Foundation 3B ANE (reuse `apple_speech/.build/release/apple-llm-polish` pipe JSONL `{"id","mode":"line"|"paragraph","text","prev1","prev2","context","prevSource","language"}` → `{"id","ok","text","ms"}` KV cache warm, 258-498ms line polish, temp 0.1 line / 0.2 para `permissiveContentTransformations` guardrails, asset `com_apple_MobileAsset_UAF_FM_GenerativeModels` + Overrides 2GB quantized, ANE+GPU) — contextual: draft+final+context → "On it — checking your calendar for tomorrow, 3 meetings…" not generic "one sec" — must be >30 chars else use final[:400] for context else regenerate if generic "Perfect."
- Spoken instantly: Kokoro ONNX 82M standalone `kokoro==0.9.4` `voice=` param required, model cache `~/.cache/huggingface/hub/hexgrad_Kokoro-82M` 326MB 4-5x RTF 0.36-0.71s inference load once 4-6s daemon resident, MPS 19x SLOWER LSTM must CPU, Edge AriaNeural cached 4.3 MOS fallback, `voicebox_quick_reply` Aiden Qwen 1.7B 4.4 MOS if kokoro down, `say -v Eddy` not Siri 30MB MOS 3.8 don't use. Timed: play fast ACK <500ms Kokoro while big LLM already started parallel — don't wait fast finish to start big

**Big LLM while fast response playing, multiple partials timed to playback:**
- `voice:{device_id}` persistent conversation, `response_store`, ephemeral system disclaimer
- Streaming: `POST /api/sessions/{id}/chat/stream` SSE `delta` existing — after final transcript, submit polished transcript + disclaimer + alternatives + preceding context to big LLM (Claude Spark etc) streaming tokens → sentence buffer `/.+?[.!?。！？]/` soft boundary `, ; :` after 180 chars buf>220 (same as Pi5 `takeSpeechChunk`) → `playSpeechText(chunk, {source:'voice-conversation', messageId})` queue `PLAYBACK_STALL_MS=15000` sequence guard `stopVoicePlayback()`
- Multi-partial behavior: intent ack already spoken (fast), big LLM can send progressive updates while task runs: `"On it — checking your calendar…"` (fast) → `"Found 3 events tomorrow"` (progress while tool runs) → full final. Timed to tail: schedule next audio start in last 300ms of previous Audio element tail — requires audio duration from WAV header or estimate, crossfade 150ms, no overlap
- Tools: calendar fetch iCloud, memory, kanban etc — Pipecat usually waits all tools then speaks; we stream partials as tool results arrive via `tool_progress_callback("tool.started"/completed)` → inject as contextual voice update "found 5 events" etc

**Barge-in:**
- Client `AnalyserNode` RMS `level>0.12` while `speaking` → `stopVoicePlayback()` + `startListening()` → server cancel queue via `ping cancel` or `{barge_in}` event — Pi5 solved via `_tts_playing` Event + 300ms gap + cancel recorder before `play_audio_file`, desktop pause analyser during speaking to avoid self-trigger but keep second node for barge
- Server also `AnalyserNode` echoCancellation true to allow dual-stream half→full duplex for free (steal from Moshi idea)

**Power/perf truth (measured M2 Air Mac14,2):**
- SpeechAnalyzer: 0mW ANE in 50 powermetrics samples, 33.5% CPU single-core `localspeechrecognition XPC` 442mW idle → 7.3W spike E+P, GPU 67-193mW, 27-38x RTF file bench 18.9s→0.6s RC 31x pipe, RTF 31 drafts +2 finals, asset per-locale 174-196M EN 196M total 1.4G 4 locales preinstalled path `/System/Library/AssetsV2/com_apple_MobileAsset_UAF_...` Whisper Base 140M single multi-lang but venv Torch 800M — system footprint win no venv but heavier per-locale
- Foundation 3B: ANE+GPU ~258-498ms measured LM, 2GB quant, 4K context offline no key
- LLM log `logs/llm_requests.jsonl` → `provider:apple model:apple-fm apple_ms 351-498` vs Ollama 4-8s
- Missing log bug cause Apple LLM not wired fixed `2e6d6d2`

---

## 6. Implementation plan — low-hanging wins ordered

**Phase 0 — this plugin repo already done:**
- Pi custom gateway persisted as plugin `~/.hermes/plugins/esp32-voice-gateway` auto-restore bundled patch, verifies 400 not 404

**Phase 1 — polish layer (today, 80 lines reuse existing code):**

1. `hermes_plugin/future/freeflow_polish_port.py` — copy `freeflow_polish.py` deterministic + skip + guards + `KNOWN_TERMS += Hermes domain voice/gateway/hotkey/Kokoro/VAD/ANE/SpeechAnalyzer`, unit test boy→voice case with precedingContext `voice gateway setup`
2. Swift pipe daemons local — already built: `apple_speech/.build/release/apple-speech-transcribe --pipe` + `apple-llm-polish` — M2 Air native, daemon pattern model resident 0.36s inference after load
3. Alternatives in prompt — enable `.alternativeTranscriptions` report option, pass `alternatives` array in draft event `{"event":"draft","text","alternatives":["boy mode","voice mode"]}`
4. Guard tuning — overlap 0.45 not 0.7 to allow boy→voice 1-word edit, truncate 0.25 fallback
5. Composer opacity .55 italic live draft — no changes to Pi5 — future desktop
6. Skip PIL UDP spam — throttle screen animation to state change only
7. In-mem pipe — remove `raw_path` s16le temp files, one WAV at STOP `_s16le_to_wav_bytes` same as Pi5 but no file per chunk, bytearray copy every 800ms `snap = buffer.copy()` to avoid race

**Phase 2 — native `/api/voice/live` plugin proper (replace patch):**
Create `hermes_plugin/plugin_new/` true standalone plugin injecting routes at runtime without touching file — hook `APIServerAdapter.connect` — mount `POST /api/voice + WS /api/voice/live` reusing helpers `_s16le_to_wav_bytes`, `_write_pcm_s16le_wav`, `_resolve_esp32_screen_url` verbatim + audit logger, but using native Apple pipes instead of Mac mini MCP URLs from `MACMINI_MCP_URL/TOKEN` env / `config.yaml mcp_servers.macmini`

**Phase 3 — streaming sentence TTS timed queue:**
Port `takeSpeechChunk` regex + 15s stall guard from `voice-playback.ts` to server `audio_chunk` events timed 300ms before prev tail end, sentence buffer while LLM token streams, `stopVoicePlayback()` barge-in `level>0.12`

**Phase 4 — Hermes disclaimer + confirmation behavior:**
Add `build_voice_user_message()` wrapper that injects STT imperfect disclaimer + raw + polished + alts + domain context + instruction paraphrase back & ask if ambiguous — apply to both ESP32 existing `POST /api/esp32/voice` + new `/api/voice/live` — minimal change reduces hallucination risk immediate win

**Phase 5 — desktop `use-voice-conversation-live` hook ChatGPT Live parity:**
`use-mic-recorder.ts` expose PCM chunks via AudioWorklet 20ms + AnalyserNode RMS, WS client connect `ws://localhost:8642/api/voice/live`, live draft composer, streaming SSE response_text → sentence TTS queue per sentence, barge-in via `stopVoicePlayback() + startListening()`, continuous loop 3-strike silence halt, waveform pill level*100% width, Press Space to send, header dot green listening yellow transcribing blue thinking purple speaking

**Phase 6 — testing harness:**
Pipecat/LiveKit e2e tester pattern: audio file → WS → asserts caption + audio timeline regression, plus `boy→voice` specific test case with context

---

## 7. Latency target vs OSS

- Hermes M2 Native target (this spec): **caption 60ms volatile ANE** instant confidence, first sound **1750ms** (1250 VAD + 300 ANE ack + 200 Kokoro) but feels <1s because caption at 60ms, streaming token from 200ms sentence N speaks while N+1 gen, barge-in 12ms RMS detect
- Pipecat Cloud Deepgram+Cartesia: ~800-1200ms e2e streaming but no contextual fast ack, $0.10/min, net jitter, 350ms partials
- Moshi 7B full-duplex: 160ms codec theoretical + 400ms tokens → 600ms magic UX but needs A100/CUDA 30W GPU, weak tool-use, no caption (needs separate ASR), not offline M2
- Qwen2.5-Omni: 800ms encoder + 1.2s prefill → 2.8-4s first audio, good quality not streaming sentence TTS, 12GB VRAM

Winner for Hermes tool-use: native pipeline with text middle auditable transcript + freeflow deterministic readable, fast contextual voice unique.

---

## 8. References — in this repo + external

### Inside this repo:
- `patch/api_server_esp32_live_draft_with_llm.patch` — 1675-line current running code Pi5 verified 2026-07-14, origin `~/.hermes/hermes-agent/gateway/platforms/api_server.py` +1500 custom
- `hermes_plugin/__init__.py` — auto-restore plugin logic
- `docs/*.patch` — evolution reference

### External open source:
- **mrinalwadhwa/freeflow** https://github.com/mrinalwadhwa/freeflow — Apache-2.0 91★ macOS dictation anywhere, 288 commits, last 7f96ccd guard truncation. `FreeFlowKit/Sources/FreeFlowKit/Services/PolishPipeline.swift` + `DictationPipeline.swift` + Prompts `PolishPromptEnglish.swift` local/casual/qwen/hindi/kannada/tamil, Proposals `OpenAIRealtimeProvider.swift` WS WSS streaming backup warm standby 180s `SessionTiming`, adaptive silence `effectiveSilenceThreshold()`. BENCHMARK.md median 0.55s keyRelease→inject 83% skip LLM. mic compatibility issue #2.
- **Pipecat** https://github.com/pipecat-ai/pipecat — 6.5k★ Python frame processors, frame DAG, context aggregator, TurnTracker, barge-in cancel preserve context, Daily transport — steal TurnTracker enum + aggregator + testing harness
- **LiveKit Agents + TEN** — WebRTC SFU agents graph editor — steal flow state JSON visualization
- **Moshi Kyutai** https://github.com/kyutai-labs/moshi — 7B full-duplex Mimi codec 12.5Hz 2.2GB quant 160ms — steal dual-stream while TTS playing keep STT open echoCancellation true half→full duplex free
- **Qwen2.5-Omni / Mini-Omni2 / Ultavox** — audio LLM raw encoder → LLM → codec, no STT errors but high TTFT weak tools

### Local prior art:
- `~/Projects/pythonwhisper/` — `freeflow_polish.py` `deterministic_polish()` port, `engine_apple_transcribe.py` `ApplePipeTranscriber` pipe volatile 31 drafts/6.3s 60ms granularity, `engine_apple_llm.py` `AppleLLM` reader thread + pending-id queues ANE 258-498ms, `engine_llm.py` rolling paragraph refine `deque maxlen2` `post_correct_line()` Apple path primary Ollama fallback + `logs/llm_requests.jsonl`, `engine_kokoro_tts.py`, SQLite `transcribe.py` 61K
- `~/Projects/pythonwhisper/apple_speech/` — Swift CLI `apple-speech-transcribe --pipe`, `apple-llm-polish` 120KB FoundationModels 3B pipe JSONL KV warm, build workaround `swiftc -O -parse-as-library -target arm64-apple-macosx26.0 -sdk .../MacOSX26.5.sdk Sources/... -framework Speech -framework AVFoundation -framework Foundation` (Package.swift v26 needs PD 6.2 but CLTools SPM 6.0), ANE power trace `ane_power 0mW 7.3W CPU 33% XPC`, assets EN 196M, benchmarks Inscribe 2.12% clean vs Whisper Small 3.74% — see `~/.hermes/skills/apple-speech-api/SKILL.md`
- `~/.hermes/skills/hermes-live-voice/` — full audit refs `pi5-voice-gateway-audit.md` `ws-protocol.md` `desktop-live-voice-plan.md` `native-m2-websocket-only.md` `oss-v2v-comparison-2026.md`
- HTML diagrams: `~/hermes-live-voice-native.html` (MacBook-only WS-only target) + `~/hermes-voice-oss-comparison.html` + `~/hermes-voice-gateway-design.html` (Pi+Mac mini pain points) + `~/hermes-voice-imperfect-fix.html` (boy→voice 3-layer)

### Protocol docs:
- Binary `pcm_s16le 16k mono`, WS control `start{format,sample_rate}/stop/ping`, Events order `ready → listening → draft/interim_transcript → thinking{transcribing} → transcript/final → instant_response/quick_reply → instant_audio → response_text → done{background} + thinking{waiting}`, Headers `X-Hermes-Screen-Url`, `X-Device-ID=iphone*` → `local_devices.yaml`, peer_ip `http://{ip}/api/mcp` aliases `little32/esp32_screen`, MCP tools `clear_screen,draw_text,play_audio_base64,play_mp3_base64`, animations `hermes_voice_animations.py ack/captioning/waiting` `ESP32_VOICE_ANIMATION_PATH`
- Audio tools: `tools/voice_mode.py` recorder RMS 200/3s hallucination filter, `tools/tts_tool.py` 300+ voices Edge/Eleven, `hermes_cli/voice.py` Ctrl+B push-to-talk continuous VAD 3-strike

---

## 9. Quick wins TODO checklist

- [ ] **P0** Add `docs/future-optimizations.md` (this file) — done
- [ ] **P0** Add `build_voice_user_message()` wrapper with disclaimer + alternatives + confirmation instruction — minimal code, biggest reliability gain
- [ ] **P0** Add Hermes domain `KNOWN_TERMS` voice/gateway/hotkey/Kokoro/VAD/ANE
- [ ] **P1** Enable `.alternativeTranscriptions` in Apple pipe + emit in WS draft/final so LLM sees homophone candidates
- [ ] **P1** Tune guard `overlap 0.45 truncate 0.25` to allow boy→voice single-word fix
- [ ] **P1** In-mem only ByteArray no file per chunk + snap copy every 800ms race fix
- [ ] **P1** Adaptive silence `effectiveSilenceThreshold()` far-field 0.001 near-field ambient*1.2 cap 0.01
- [ ] **P1** Empty audio <1600B trigger as clarification not error
- [ ] **P2** Sentence TTS queue timed to tail 300ms start before tail + 150ms crossfade stall 15s
- [ ] **P2** Warm backup Swift pipe daemons model resident 0.36s inference
- [ ] **P2** Barge-in `level>0.12 && speaking` stop TTS + startListening + server cancel
- [ ] **P2** Multi-partial while tool runs — instant ack → progress found N → final, stream via tool_progress_callback
- [ ] **P3** Skip PIL UDP spam throttle to state change
- [ ] **P3** Desktop hook `use-voice-conversation-live` ChatGPT parity — AudioWorklet 20ms + live draft italic .55 + waveform + header dots
- [ ] **P3** True standalone plugin without file touch (eliminate patch reapply) + integration test curl 400 not 404 + boy→voice e2e test

---

## 10. Open questions / decisions

1. **TTS choice M2 native:** Kokoro 82M standalone `kokoro==0.9.4` imperative 200ms CPU daemon vs Edge AriaNeural 4.3 MOS cached vs Eleven fallback? Start Kokoro daemon (copy 325MB models from Pi5 `~/kokoro-models/kokoro-v1.0.onnx` + voices.bin) but `mlx-audio` wrapper needs `misaki[en]+spacy+pydantic` conflicts — use standalone `kokoro` pip not `mlx_audio`. MPS is 19x SLOWER LSTM keep CPU.
2. **Foundation 3B prompt for fast ack:** should we reuse `systemPromptLocal = "Clean up… Return only…"` for polish but new quick reply prompt `draft + final + "quick ack under 20 words contextual on intent"` with low temp 0.1/0.2? Already measured Foundation general + contentTagging UseCases ~2GB quantized ANE+GPU.
3. **WS compat long-term:** keep `/api/esp32/voice/ws` alias for ESP32 firmware already flashed or only `/api/voice/live` + adapter route? Current plugin runs both by patch; future true plugin could mount both aliases.
4. **FreeFlow license reuse:** Apache-2.0 clean, prompts can be reused with attribution — ideal for private repo (Reyna family private) not selling.
5. **Ephemeral vs stored transcript:** Should voice gateway keep both raw + polished in audit for metrics, but only polished + disclaimer in `voice_requests.jsonl`? Recommended: log raw, polished, alts, overlap, guard decisions for future tuning.

---

*Generated 2026-07-15. Sources: Pi5 live gateway audit, Mac mini MCP 37 tools, pythonwhisper ANE traces 0mW ANE 7W CPU, FreeFlow 7f96ccd, Pipecat/Moshi/Qwen OSS landscape.*