diff --git a/docs/future-optimizations.md b/docs/future-optimizations.md
new file mode 100644
index 0000000..36a1e59
--- /dev/null
+++ b/docs/future-optimizations.md
@@ -0,0 +1,285 @@
+# Future Optimizations — ESP32 Voice Gateway (M2 native path + FreeFlow learnings)
+
+> Goal: WS `/api/voice/live` single websocket endpoint (compat with current `/api/esp32/voice/ws` framing binary PCM s16le 16k mono) — instant captioning feedback, fast local LLM initial voice response custom based on partial captioning, empty audio / stream stopped trigger, multi-partials timed to audio tail while big LLM works. Native M2 Air implementation, no Mac mini hop. Keep ESP32 compat by endpoint shape, but design doc is device-agnostic.
+
+This doc captures all learnings from July 2026 audit of Pi5 `api_server.py` custom gateway (6949 vs 5124 stock, 1675-line patch), Mac mini ANE accelerator (`:7331/mcp` `speech_live_transcribe` ApplePipeTranscriber volatile 60ms, `apple_llm_quick_reply` Foundation 3B ANE 258-498ms, `speech_synthesize_base64` Alex <500ms / `voicebox_quick_reply` Aiden Qwen 1.7B), desktop hooks `use-voice-conversation` idle→listening→transcribing→thinking→speaking VAD 0.075/1250ms, 60s cap, 3-strike halt, and FreeFlow/macOS SpeechAnalyzer patterns.
+
+---
+
+## 0. Current perf baseline (Pi5) — reference
+
+- Live draft: snapshot `pcm_buffer` → in-mem WAV → base64 → HTTP POST to Mac mini `speech_live_transcribe` — 800ms interval `ESP32_LIVE_DRAFT_INTERVAL_MS=800` + `MIN_BYTES=8000`, dedup via `last_draft_text`. Events `ready → listening → draft/interim_transcript → thinking{transcribing} → transcript/final → instant_response/quick_reply → instant_audio → response_text → done{background} + thinking{waiting}`
+- Quick ACK: draft >30 chars → `apple_llm_quick_reply {draft, context, instructions}` else use final `transcript[:400]` — guard generic "Perfect." triggers regenerate from final
+- Instant audio: `speech_synthesize_base64 {text, voice:"Alex"}` boy voice <500ms preferred, fallback `voicebox_quick_reply profile:"Aiden"` Qwen3-TTS 1.7B 2-4s, pushed via Screen MCP `play_audio_base64 wav_base64 volume80` to `http://<source_ip>/api/mcp` (resolved from header `X-Hermes-Screen-Url` > query > `X-Device-ID=iphone*` lookup `local_devices.yaml` > peer_ip)
+- Final STT: same Mac mini pipe full buffer `macmini-live-final` 753ms vs local faster-whisper base 11-13s cold fallback
+- Full turn: persistent `esp32:{device_id}` / `voice:{id}` conversation via `response_store`, tool-use capable (calendar, memory, kanban, ESP32 `draw_text` etc.)
+- TTS chain: Edge AriaNeural (Pi default, free 4.3 MOS) mp3→ffmpeg `16k mono s16le WAV`
+- Audit: `~/.hermes/cache/audio/esp32/ + voice_requests.jsonl`
+- Failure mode: raw STT is ground truth — hears "boy" instead of "voice" for voice domain, model keeps wrong transcript, answers about boy mode. Mac mini down (it is now — :7331 refused) kills all live.
+- Patch fragility: `api_server.py` 6949 LOC patch breaks on `hermes update` — this plugin repo fixes via `~/.hermes/plugins/` auto-restore.
+
+---
+
+## 1. Pain Points — ranked, with origin
+
+| # | Pain | Impact | Where | OSS ref / inspiration |
+|---|------|--------|-------|----------------------|
+| P0 | Mac mini MCP single point failure `:7331/mcp` down now | No live drafts, no quick ack, no instant audio — full gateway degradation to slow base | Pi5 gateway uses RPC instead of local Swift pipe | FreeFlow alternative: on-device `SpeechAnalyzer` 26.5 + `FoundationModels` 3B direct ANE no hop |
+| P0 | `api_server.py` patch 1675 lines wiped by `hermes update` | Must restore, flakey 3-strike | Pi5 `git stash --include-untracked` + pull | This repo `hermes_plugin/__init__.py` auto-restore — TODO true plugin injection without file touch |
+| P1 | Ground-truth lock: `boy → voice` kept as truth | Hermes tool calls wrong thing | `f'[Voice input...]\n{transcript}'` implies perfect | **FreeFlow** never treats raw as final — 3-stage polish + skip heuristic + anti-truncation guard |
+| P1 | Disk IO storm: `raw_path` .s16le temp file per chunk | 60 chunks/min file writes unneeded | Pi5 WS handler writes every frame | FreeFlow + new spec: `ByteArray` only, one WAV encode at stop — `engine_apple_transcribe.py` `ApplePipeTranscriber` |
+| P1 | No streaming LLM→TTS sentence queue | Whole answer TTS at end, perceived latency +2s even after ACK | Desktop + ESP32 background | **Pipecat** sentence aggregator + Cartesia streaming; `voice-playback.ts` already has `takeSpeechChunk /.+?[.!?]/` + 15s stall guard |
+| P2 | Single VAD (RMS 0.075) tuned once, not adaptive | near-field AirPods vs far-field built-in built-in speech 0.002-0.005 RMS same threshold → silent presses rejected as speech or vice versa | Pi5 `SILENCE_RMS_THRESHOLD 200` vs desktop `0.075` | FreeFlow `effectiveSilenceThreshold()` far-field 0.001, near-field `max(ambient*1.2,0.0005)` capped 0.01; **Silero VAD 1MB** unified |
+| P2 | Empty audio trigger missing as intentional signal | Tap no speech should = "yes?" prompt not dropped | Pi5 `len < threshold` error path | User spec: <1600B = 50ms guard treat as immediate final / clarification trigger |
+| P3 | PIL UDP spam for screen animations | Network spam per draft | `_send_screen_animation` per draft | Skip diff, throttle 200ms + only on state change `ready/listening/thinking/speaking` |
+
+---
+
+## 2. FreeFlow (mrinalwadhwa/freeflow) deep dive — what to steal
+
+Source: https://github.com/mrinalwadhwa/freeflow 91★ Apache-2.0 macOS 14+, dictation-anywhere hotkey app. Not voice assistant, but polish pipeline is gold reusable.
+
+**Why it matters for voice gateway:**
+FreeFlow problem is same: Apple SpeechAnalyzer raw is 2.12% WER 196M EN / 174M ES but robotic, homophones, spurious commas. It never trusts STT. Median keyRelease→injection **0.55s**. Warm backup WS pre-opened — 91% zero handshake. 83% skip LLM polish entirely via local heuristic. When polish needed, `gpt-4.1-nano` 320-780ms (original) but macOS 26 path uses on-device `SpeechAnalyzerDictationProvider` + `FoundationModelChatClient` local — same pattern as our ANE 3B.
+
+**Code audited July 2026:**
+- `FreeFlowKit/Sources/FreeFlowKit/Services/PolishPipeline.swift` `polishModel = "gpt-5.4-nano"` — 43 punctuation rules, `<keep>` tags, `collapseAdjacentPunctuation`, `cleanSpuriousCommas`, `stripNoisePhrases`, `stripFillerSounds`, `buildCloudSystemPrompt(context)`, `knownTerms` (Redis, Postgres, macOS… domain terms capitalized), `guard_against_truncation` (commit `7f96ccd Preload models on startup and guard against aggressive truncation` 2mo ago), `word_overlap_ratio` concept, `stripKeepTags` → `\n\n`, `sanitizeContextField` 80 chars `focusedFieldContent`, `toneLabel(for: bundleID) casual`
+- Prompts: `PolishPromptEnglish.swift` huge tested 80-line system prompt: removes throat-clearing (`let me think, hold on, how do I put this, let me recall` → keep actual content), self-correction `no wait, actually, sorry, I mean, let me rephrase, never mind, or rather, make that → keep final`, abandon restart `no, forget it, nah, that won't work → drop before`, stutter double `the the → the` but `wait wait wait → Wait, wait, wait,` comma keep same sentence, `2-item inline never list`, `3+ items always vertical list with - or 1. 2. 3. if ordered`, recapping meeting format `- Person: update`, numbers always digits `one → 1`, `minus ten → -10`, phone dashes, email `john at example dot com → john@example.com`, preserve `kinda/gonna/wanna` contractions, `<keep>` must preserve exactly
+- `PolishPromptLocal.swift` `systemPromptLocal = "Clean up this dictated text. Return only the cleaned text."` — for FoundationModels 3B ANE short window 4096 tokens guardrails
+- `DictationPipeline.swift` `DictationPipeline: PipelineProviding` actor: `activate() → recording` immediate HUD return, audio setup in Detached task (AVAudioEngine 500-900ms + BT SCO 3s timeout `detachedWithTimeout`), `pendingContext` parallel, adaptive `effectiveSilenceThreshold()` far-field 0.001 vs near-field ambient*1.2 capped 0.01 floor 0.0005 multiplier 1.2, early silence short-circuit peakRMS <= threshold after 200ms = 4 ticks 50ms, streaming session reuse adoption `freshConnection vs adoptedBackup vs adoptedStaleBackup maxBackupAge 180s`, chunk buffer `rawChunkBuffer` until `endsAtSentenceBoundary` then `polishChunk()` + inject, rolling commit chunker `TimeAndSilenceChunkingStrategy` speechDebounce 10s (only genuine extended silence triggers commit not thinking pauses), `ChunkInjectedFlag` thread-safe, `RecoveryBox` audio saved for retry, `minimumAudioDuration 0.1s`, `SessionTiming` diagnostic
+
+**Borrow list for this gateway:**
+
+- **Deterministic pipeline (0 ms, no key):** `substituteDictatedPunctuation` + `cleanSpuriousCommas` + `stripFillerSounds` + `collapseAdjacentPunctuation` — already ported partially in `~/Projects/pythonwhisper/freeflow_polish.py` (`deterministic_polish()`, `substitute_dictated_punctuation`, `strip_keep_tags`, `normalize_formatting`, `capitalize_known_terms`, `collapse_adjacent_punctuation`, `strip_noise_phrases`, `strip_filler_sounds`, `clean_spurious_commas`) — KNOWN_TERMS currently DB/infra only, add Hermes domain
+- **Skip heuristic 83% rate:** `is_clean_enough_to_skip_llm()` — if ends sentence + no fillers regex + no repeat `(\w+)\s+\1` → skip expensive LLM — medians 0.55s — implement in WS before ANE LLM call
+- **Guards:** `guard_against_truncation len(polished)/len(pre) <0.25 → fallback`, `word_overlap_ratio <0.45 → fallback`, `match_input_casing` leading lowercase preserve when continuing sentence (`precedingText` mid-sentence `endsAtSentenceBoundary` false)
+- **Chunk buffer:** accumulate volatile intermediates `rawChunkBuffer` until sentence boundary, then polish batch — don't polish volatile `speecheech → speech` word-level jitter per draft, only on sentence boundary or final — reduces ANE calls 31 drafts → ~5 polishes per 6.3s audio
+- **Adaptive silence:** replace fixed 200 / 0.075 with FreeFlow's proximity-aware adaptive — use same for client AudioWorklet level + server VAD — far-field built-in needs 0.001 floor, near-field AirPods ambient measured
+- **Warm backup:** pre-open Swift pipe daemons (`apple-speech-transcribe --pipe volatile` + `apple-llm-polish` JSONL pipe keeping `LanguageModelSession` KV warm) — already validated `pythonwhisper` `ApplePipeTranscriber` keep-alive `stdin 4-byte BE len + wav` → `stdout JSONL draft/final 60ms word-level 31 drafts`, fix `exit(0)` hang by return naturally not Process exit, `readExact` nil on EOF, `PYTHONPATH=""` isolate PIL `_imaging` collision
+- **Context injection:** `AppContext` 80 chars last `focusedFieldContent` + preceding lines + tone casual → use for polish; for voice gateway = last 2 finals `refined_context_history deque maxlen2` + conversation summary `esp32:{id}`
+
+---
+
+## 3. OSS V2V Landscape 2025-26 — where we sit
+
+Family 1 — **Pipeline orchestrators** (STT → LLM → TTS, text middle = tool-use, modular):
+
+- **Pipecat (Daily pipecat-ai/pipecat) 6.5k★**: Python frame DAG, Daily/WebRTC transport `VAD → STT → LLM → TTS`, context aggregator keeps partials + tool results, interruption via VAD confidence + cancel + preserve context, TurnTracker `user_start → vad_silence → user_stop → thinking → bot_start` enum with `heardSpeechRef`, token streaming + sentence aggregator + Cartesia/Eleven. Lat 800-1200ms e2e $0.10/min Deepgram+Cartesia. **Steal:** TurnTracker + aggregator + barge-in cancel TTS keep partial LLM context + parallel tool calls without dropping audio + e2e tester audio file → asserts caption+timeline
+- **LiveKit Agents + TEN (Agora)**: same frame pipeline, WebRTC room, graph editor Rust core extensions vision. **Steal:** visual flow state JSON for debugging `ready/listening/captioning/thinking/speaking`, SFU recording infra mature but cloud
+
+Family 2 — **End-to-end speech models** (no text middle, full-duplex UX magic, but poor tool-use, GPU-hungry):
+
+- **Moshi — Kyutai Apache 2.0**: Mimi codec 12.5Hz, 7B full-duplex inner monologue + 2 audio streams simultaneous (user+bot), 160ms theoretical, 2.2GB quant, A100/CUDA needed, weak function calling hallucinates. **Steal:** dual-stream idea — keep STT open while TTS playing via `echoCancellation:true` + second AnalyserNode RMS >0.12 while speaking → half→full duplex for free
+- **Qwen2.5-Omni / Mini-Omni2 / Ultravox / Freeze-Omni**: Whisper encoder + LLM + soundstorm codec audio in → thought text → audio out, raw audio encoder brief → no STT errors emotion speaker id, but 3-5s TTFT, large VRAM, needs fine-tune for tools. **Note:** text middle needed for Hermes because calendar/memory/kanban tool calling via Claude Spark > omni accuracy; skip STT path would need audio→LLM direct that can't yet call tools reliably
+
+**Our position — hybrid unique 2026:**
+- Only project with **Apple Speech 26 pipe volatile 60ms 2.12% WER 196M EN** + **Foundation 3B on-device ANE 258-498ms both local** — FreeFlow validated 27-38x RTF 0mW ANE 7W spike single-core XPC `localspeechrecognition`
+- **Fast voice from partials contextual not generic** — no OSS does this; Moshi inner thought generic "yeah", Pipecat none by default, we have draft >30 chars → Foundation quick ack `"On it — checking your calendar for tomorrow 3 meetings…"` already measured
+- **Timed multi-partials while tool runs** — Pipecat usually waits all tools then speaks; we want instant → progress "found 3 events" → final timed to audio tail queue **sentence buffer `/.+?[.!?]/`** + start next in last 300ms prev tail + crossfade 150ms
+- **In-mem only, no ffmpeg, no temp wav per chunk** — most OSS still writes file per chunk; we already have `ByteArray` only path in spec
+- **WS compat** `WS /api/voice/live` same binary PCM s16le 16k framing as Pi5 ESP32 WS — new design says "keep endpoint with websocket" compatible but diagrams show no ESP32/other devices focus on MacBook only — so web can send WebM/opus or PCM flag, backward compat for future hardware
+- **Offline $0 7W vs cloud $0.10/min + net jitter**
+- Latency budget native: **VAD 1250ms + fast LLM 300ms + Kokoro 200ms = 1750ms first audio**, caption 60ms instant feels <1s streaming, big LLM token from 200ms streaming sentence queue. vs Pipecat ~2200ms cloud, Moshi ~600ms magic but no tools, Qwen-Omni 2.8-4s
+
+---
+
+## 4. Ground-truth lock fix — 3-layer defense (key feature request)
+
+User worry: Pi hears "boy instead of voice" and model keep it as ground truth. FreeFlow fix is not trust acoustic as truth but semantic context.
+
+**Before (current):**
+```python
+user_message = f'[Voice input from ESP32 device "{device_id}"]\n{transcript}'
+```
+=> Hermes thinks typed ground truth = boy mode.
+
+**After — 3 layers:**
+
+**L1 deterministic 0ms:** `deterministic_polish()` from `freeflow_polish.py` — strips fillers/noise, collapses `,,,`, Apple spurious commas — boy still boy here
+
+**L2 semantic ANE 3B with context (fixes boy→voice):**
+- `precedingText = last 2 finals + voice:{id} summary` (FreeFlow `refined_context_history deque 2`)
+- Prompt includes domain `KNOWN_TERMS += voice, gateway, hotkey, dictate, Kokoro, VAD, barge-in, ANE, SpeechAnalyzer, DictationPipeline, FreeFlowKit`
+- FreeFlow `systemPromptEnglish` full (80 lines tested) — self-correction `I mean…` → keep final, etc.
+- ANE 3B `polish_line(substituted, prev1, prev2)` → if context = "voice gateway setup", "boy" acoustically close → semantic mismatch → corrects "voice"
+- Guards: truncate 0.25 / overlap 0.45 / casing / keep tags
+- Alternatives: Apple `SpeechTranscriber` attribute options `[.alternativeTranscriptions]` returns list — already parsed in pipe `--pipe` JSONL `alternatives` — pass to LLM: `Alternatives: boy mode (0.4), voice mode (0.6)` — model picks semantic fit
+
+**L3 Hermes LLM disclaimer (your ask) + multi-partial ask confirm:**
+
+Build `user_message` with explicit uncertainty + polished + raw + alternatives + instruction:
+
+```
+[Voice input — speech-to-text, may contain errors]
+This was spoken, not typed. Homophones like
+boy/voice, four/for, new/knew, dictate/dictation 
+are common STT errors. Don't treat transcription
+as ground truth.
+
+Polished transcript (best guess, 1 edit, overlap 0.83):
+"Set voice mode"
+
+Raw STT alternatives (low confidence word jitter):
+• "set boy mode" (raw final)
+• "set voice mode" (draft volatile alternative, 3x seen)
+Draft volatile instability is normal (speecheech → 
+speech final) — don't treat as rewrite.
+
+Context: user was talking about voice gateway setup
+for MacBook native, previous turn: "add optimization
+doc with FreeFlow ref".
+
+If request unclear or semantically off in voice gateway
+context, paraphrase back what you understood and ask for
+confirmation before acting. Don't hallucinate tool call
+on shaky transcript. Example: "Just to confirm — you mean
+voice mode?"
+
+User originally said (as heard / best):
+"Set voice mode"
+```
+
+Ephemeral system add for whole `voice:{id}` session: `"Voice session: STT imperfect 2.12% WER even polished ANE. If confidence low or term out-of-domain, clarify."` + Hermes domain bias list.
+
+This makes big LLM say "Just to confirm — you mean voice mode?" if still ambiguous instead of silently wrong. Fast ACK uses same polished transcript so <1.7s audio also correct.
+
+---
+
+## 5. Native M2 Air target — single WS endpoint spec (user final ask)
+
+**Endpoint: `WS /api/voice/live`** (keep shape compat with Pi5 `/api/esp32/voice/ws` for future ESP32, but current design MacBook only focus)
+
+**Transport:**
+- Client → Server: binary `PCM s16le 16k mono` chunks 20ms AudioWorklet (replaces `ScriptProcessor`), OR empty binary (<1600B = 50ms guard = trigger), JSON control `{start:{format:"pcm_s16le"|"webm", sample_rate:16000, locale:"en-US"}, stop, ping}`
+- Server → Client: `{ready} → listening{threshold} → draft volatile{draft,isFinal:false,is_volatile, alternatives, level} every ~400ms → interim_transcript → thinking{transcribing} → transcript final{polished,true, overlap, raw, alts, source:"apple" } → instant_response{quick, draft_count} contextual fast ack → instant_audio{b64/wavBase64, voice, quick_ms,speak_ms} → response_text streaming token via sentence buffer → audio_chunk{audio_b64, text, seq} timed to audio tail (start N in last 300ms of N-1 + 150ms crossfade) → done{background, instant_ok, keep_instant} + thinking{stage:waiting}`
+
+**Instant caption feedback:**
+- Apple `SpeechAnalyzer --pipe volatile` keep-alive single process, stdin 4-byte BE len + wav bytes len 0 = EOF, stdout JSONL draft/final word-level per chunk (~60ms granularity, 31 drafts/6.3s validated, EN 196M ES 174M), `bestAvailableAudioFormat` + `AVAudioConverter` + `AudioBufferQueue` actor, report `[.volatileResults, .alternativeTranscriptions]` attribute `[.audioTimeRange, .transcriptionConfidence]`. Volatile jitter normal.
+- In-mem only `ByteArray` accumulator — zero disk, dedup `last_draft_text` same as Pi5 `_do_live_draft` but no file. Composer desktop shows italic .55 live while listening, solid on final `transcript`
+
+**Trigger:**
+- VAD 1250ms dual `effectiveSilenceThreshold` FreeFlow adaptive (far-field built-in 0.001, near-field max(ambient*1.2,0.0005) capped 0.01) — same threshold client AudioWorklet RMS AnalyserNode + server — `heardSpeechRef` pattern from `use-mic-recorder` + Pi5 3-strike `CONTINUOUS_NO_SPEECH_LIMIT=3` halt
+- Empty audio trigger: binary <1600B or `{start}` then immediate `{stop}` → freeze draft → final transcript immediately → if still <2 words → prompt "yes?" clarification via fast ACK, not error — user spec
+
+**Fast local LLM initial voice response custom based on partial captioning:**
+- Foundation 3B ANE (reuse `apple_speech/.build/release/apple-llm-polish` pipe JSONL `{"id","mode":"line"|"paragraph","text","prev1","prev2","context","prevSource","language"}` → `{"id","ok","text","ms"}` KV cache warm, 258-498ms line polish, temp 0.1 line / 0.2 para `permissiveContentTransformations` guardrails, asset `com_apple_MobileAsset_UAF_FM_GenerativeModels` + Overrides 2GB quantized, ANE+GPU) — contextual: draft+final+context → "On it — checking your calendar for tomorrow, 3 meetings…" not generic "one sec" — must be >30 chars else use final[:400] for context else regenerate if generic "Perfect."
+- Spoken instantly: Kokoro ONNX 82M standalone `kokoro==0.9.4` `voice=` param required, model cache `~/.cache/huggingface/hub/hexgrad_Kokoro-82M` 326MB 4-5x RTF 0.36-0.71s inference load once 4-6s daemon resident, MPS 19x SLOWER LSTM must CPU, Edge AriaNeural cached 4.3 MOS fallback, `voicebox_quick_reply` Aiden Qwen 1.7B 4.4 MOS if kokoro down, `say -v Eddy` not Siri 30MB MOS 3.8 don't use. Timed: play fast ACK <500ms Kokoro while big LLM already started parallel — don't wait fast finish to start big
+
+**Big LLM while fast response playing, multiple partials timed to playback:**
+- `voice:{device_id}` persistent conversation, `response_store`, ephemeral system disclaimer
+- Streaming: `POST /api/sessions/{id}/chat/stream` SSE `delta` existing — after final transcript, submit polished transcript + disclaimer + alternatives + preceding context to big LLM (Claude Spark etc) streaming tokens → sentence buffer `/.+?[.!?。！？]/` soft boundary `, ; :` after 180 chars buf>220 (same as Pi5 `takeSpeechChunk`) → `playSpeechText(chunk, {source:'voice-conversation', messageId})` queue `PLAYBACK_STALL_MS=15000` sequence guard `stopVoicePlayback()`
+- Multi-partial behavior: intent ack already spoken (fast), big LLM can send progressive updates while task runs: `"On it — checking your calendar…"` (fast) → `"Found 3 events tomorrow"` (progress while tool runs) → full final. Timed to tail: schedule next audio start in last 300ms of previous Audio element tail — requires audio duration from WAV header or estimate, crossfade 150ms, no overlap
+- Tools: calendar fetch iCloud, memory, kanban etc — Pipecat usually waits all tools then speaks; we stream partials as tool results arrive via `tool_progress_callback("tool.started"/completed)` → inject as contextual voice update "found 5 events" etc
+
+**Barge-in:**
+- Client `AnalyserNode` RMS `level>0.12` while `speaking` → `stopVoicePlayback()` + `startListening()` → server cancel queue via `ping cancel` or `{barge_in}` event — Pi5 solved via `_tts_playing` Event + 300ms gap + cancel recorder before `play_audio_file`, desktop pause analyser during speaking to avoid self-trigger but keep second node for barge
+- Server also `AnalyserNode` echoCancellation true to allow dual-stream half→full duplex for free (steal from Moshi idea)
+
+**Power/perf truth (measured M2 Air Mac14,2):**
+- SpeechAnalyzer: 0mW ANE in 50 powermetrics samples, 33.5% CPU single-core `localspeechrecognition XPC` 442mW idle → 7.3W spike E+P, GPU 67-193mW, 27-38x RTF file bench 18.9s→0.6s RC 31x pipe, RTF 31 drafts +2 finals, asset per-locale 174-196M EN 196M total 1.4G 4 locales preinstalled path `/System/Library/AssetsV2/com_apple_MobileAsset_UAF_...` Whisper Base 140M single multi-lang but venv Torch 800M — system footprint win no venv but heavier per-locale
+- Foundation 3B: ANE+GPU ~258-498ms measured LM, 2GB quant, 4K context offline no key
+- LLM log `logs/llm_requests.jsonl` → `provider:apple model:apple-fm apple_ms 351-498` vs Ollama 4-8s
+- Missing log bug cause Apple LLM not wired fixed `2e6d6d2`
+
+---
+
+## 6. Implementation plan — low-hanging wins ordered
+
+**Phase 0 — this plugin repo already done:**
+- Pi custom gateway persisted as plugin `~/.hermes/plugins/esp32-voice-gateway` auto-restore bundled patch, verifies 400 not 404
+
+**Phase 1 — polish layer (today, 80 lines reuse existing code):**
+
+1. `hermes_plugin/future/freeflow_polish_port.py` — copy `freeflow_polish.py` deterministic + skip + guards + `KNOWN_TERMS += Hermes domain voice/gateway/hotkey/Kokoro/VAD/ANE/SpeechAnalyzer`, unit test boy→voice case with precedingContext `voice gateway setup`
+2. Swift pipe daemons local — already built: `apple_speech/.build/release/apple-speech-transcribe --pipe` + `apple-llm-polish` — M2 Air native, daemon pattern model resident 0.36s inference after load
+3. Alternatives in prompt — enable `.alternativeTranscriptions` report option, pass `alternatives` array in draft event `{"event":"draft","text","alternatives":["boy mode","voice mode"]}`
+4. Guard tuning — overlap 0.45 not 0.7 to allow boy→voice 1-word edit, truncate 0.25 fallback
+5. Composer opacity .55 italic live draft — no changes to Pi5 — future desktop
+6. Skip PIL UDP spam — throttle screen animation to state change only
+7. In-mem pipe — remove `raw_path` s16le temp files, one WAV at STOP `_s16le_to_wav_bytes` same as Pi5 but no file per chunk, bytearray copy every 800ms `snap = buffer.copy()` to avoid race
+
+**Phase 2 — native `/api/voice/live` plugin proper (replace patch):**
+Create `hermes_plugin/plugin_new/` true standalone plugin injecting routes at runtime without touching file — hook `APIServerAdapter.connect` — mount `POST /api/voice + WS /api/voice/live` reusing helpers `_s16le_to_wav_bytes`, `_write_pcm_s16le_wav`, `_resolve_esp32_screen_url` verbatim + audit logger, but using native Apple pipes instead of Mac mini MCP URLs from `MACMINI_MCP_URL/TOKEN` env / `config.yaml mcp_servers.macmini`
+
+**Phase 3 — streaming sentence TTS timed queue:**
+Port `takeSpeechChunk` regex + 15s stall guard from `voice-playback.ts` to server `audio_chunk` events timed 300ms before prev tail end, sentence buffer while LLM token streams, `stopVoicePlayback()` barge-in `level>0.12`
+
+**Phase 4 — Hermes disclaimer + confirmation behavior:**
+Add `build_voice_user_message()` wrapper that injects STT imperfect disclaimer + raw + polished + alts + domain context + instruction paraphrase back & ask if ambiguous — apply to both ESP32 existing `POST /api/esp32/voice` + new `/api/voice/live` — minimal change reduces hallucination risk immediate win
+
+**Phase 5 — desktop `use-voice-conversation-live` hook ChatGPT Live parity:**
+`use-mic-recorder.ts` expose PCM chunks via AudioWorklet 20ms + AnalyserNode RMS, WS client connect `ws://localhost:8642/api/voice/live`, live draft composer, streaming SSE response_text → sentence TTS queue per sentence, barge-in via `stopVoicePlayback() + startListening()`, continuous loop 3-strike silence halt, waveform pill level*100% width, Press Space to send, header dot green listening yellow transcribing blue thinking purple speaking
+
+**Phase 6 — testing harness:**
+Pipecat/LiveKit e2e tester pattern: audio file → WS → asserts caption + audio timeline regression, plus `boy→voice` specific test case with context
+
+---
+
+## 7. Latency target vs OSS
+
+- Hermes M2 Native target (this spec): **caption 60ms volatile ANE** instant confidence, first sound **1750ms** (1250 VAD + 300 ANE ack + 200 Kokoro) but feels <1s because caption at 60ms, streaming token from 200ms sentence N speaks while N+1 gen, barge-in 12ms RMS detect
+- Pipecat Cloud Deepgram+Cartesia: ~800-1200ms e2e streaming but no contextual fast ack, $0.10/min, net jitter, 350ms partials
+- Moshi 7B full-duplex: 160ms codec theoretical + 400ms tokens → 600ms magic UX but needs A100/CUDA 30W GPU, weak tool-use, no caption (needs separate ASR), not offline M2
+- Qwen2.5-Omni: 800ms encoder + 1.2s prefill → 2.8-4s first audio, good quality not streaming sentence TTS, 12GB VRAM
+
+Winner for Hermes tool-use: native pipeline with text middle auditable transcript + freeflow deterministic readable, fast contextual voice unique.
+
+---
+
+## 8. References — in this repo + external
+
+### Inside this repo:
+- `patch/api_server_esp32_live_draft_with_llm.patch` — 1675-line current running code Pi5 verified 2026-07-14, origin `~/.hermes/hermes-agent/gateway/platforms/api_server.py` +1500 custom
+- `hermes_plugin/__init__.py` — auto-restore plugin logic
+- `docs/*.patch` — evolution reference
+
+### External open source:
+- **mrinalwadhwa/freeflow** https://github.com/mrinalwadhwa/freeflow — Apache-2.0 91★ macOS dictation anywhere, 288 commits, last 7f96ccd guard truncation. `FreeFlowKit/Sources/FreeFlowKit/Services/PolishPipeline.swift` + `DictationPipeline.swift` + Prompts `PolishPromptEnglish.swift` local/casual/qwen/hindi/kannada/tamil, Proposals `OpenAIRealtimeProvider.swift` WS WSS streaming backup warm standby 180s `SessionTiming`, adaptive silence `effectiveSilenceThreshold()`. BENCHMARK.md median 0.55s keyRelease→inject 83% skip LLM. mic compatibility issue #2.
+- **Pipecat** https://github.com/pipecat-ai/pipecat — 6.5k★ Python frame processors, frame DAG, context aggregator, TurnTracker, barge-in cancel preserve context, Daily transport — steal TurnTracker enum + aggregator + testing harness
+- **LiveKit Agents + TEN** — WebRTC SFU agents graph editor — steal flow state JSON visualization
+- **Moshi Kyutai** https://github.com/kyutai-labs/moshi — 7B full-duplex Mimi codec 12.5Hz 2.2GB quant 160ms — steal dual-stream while TTS playing keep STT open echoCancellation true half→full duplex free
+- **Qwen2.5-Omni / Mini-Omni2 / Ultavox** — audio LLM raw encoder → LLM → codec, no STT errors but high TTFT weak tools
+
+### Local prior art:
+- `~/Projects/pythonwhisper/` — `freeflow_polish.py` `deterministic_polish()` port, `engine_apple_transcribe.py` `ApplePipeTranscriber` pipe volatile 31 drafts/6.3s 60ms granularity, `engine_apple_llm.py` `AppleLLM` reader thread + pending-id queues ANE 258-498ms, `engine_llm.py` rolling paragraph refine `deque maxlen2` `post_correct_line()` Apple path primary Ollama fallback + `logs/llm_requests.jsonl`, `engine_kokoro_tts.py`, SQLite `transcribe.py` 61K
+- `~/Projects/pythonwhisper/apple_speech/` — Swift CLI `apple-speech-transcribe --pipe`, `apple-llm-polish` 120KB FoundationModels 3B pipe JSONL KV warm, build workaround `swiftc -O -parse-as-library -target arm64-apple-macosx26.0 -sdk .../MacOSX26.5.sdk Sources/... -framework Speech -framework AVFoundation -framework Foundation` (Package.swift v26 needs PD 6.2 but CLTools SPM 6.0), ANE power trace `ane_power 0mW 7.3W CPU 33% XPC`, assets EN 196M, benchmarks Inscribe 2.12% clean vs Whisper Small 3.74% — see `~/.hermes/skills/apple-speech-api/SKILL.md`
+- `~/.hermes/skills/hermes-live-voice/` — full audit refs `pi5-voice-gateway-audit.md` `ws-protocol.md` `desktop-live-voice-plan.md` `native-m2-websocket-only.md` `oss-v2v-comparison-2026.md`
+- HTML diagrams: `~/hermes-live-voice-native.html` (MacBook-only WS-only target) + `~/hermes-voice-oss-comparison.html` + `~/hermes-voice-gateway-design.html` (Pi+Mac mini pain points) + `~/hermes-voice-imperfect-fix.html` (boy→voice 3-layer)
+
+### Protocol docs:
+- Binary `pcm_s16le 16k mono`, WS control `start{format,sample_rate}/stop/ping`, Events order `ready → listening → draft/interim_transcript → thinking{transcribing} → transcript/final → instant_response/quick_reply → instant_audio → response_text → done{background} + thinking{waiting}`, Headers `X-Hermes-Screen-Url`, `X-Device-ID=iphone*` → `local_devices.yaml`, peer_ip `http://{ip}/api/mcp` aliases `little32/esp32_screen`, MCP tools `clear_screen,draw_text,play_audio_base64,play_mp3_base64`, animations `hermes_voice_animations.py ack/captioning/waiting` `ESP32_VOICE_ANIMATION_PATH`
+- Audio tools: `tools/voice_mode.py` recorder RMS 200/3s hallucination filter, `tools/tts_tool.py` 300+ voices Edge/Eleven, `hermes_cli/voice.py` Ctrl+B push-to-talk continuous VAD 3-strike
+
+---
+
+## 9. Quick wins TODO checklist
+
+- [ ] **P0** Add `docs/future-optimizations.md` (this file) — done
+- [ ] **P0** Add `build_voice_user_message()` wrapper with disclaimer + alternatives + confirmation instruction — minimal code, biggest reliability gain
+- [ ] **P0** Add Hermes domain `KNOWN_TERMS` voice/gateway/hotkey/Kokoro/VAD/ANE
+- [ ] **P1** Enable `.alternativeTranscriptions` in Apple pipe + emit in WS draft/final so LLM sees homophone candidates
+- [ ] **P1** Tune guard `overlap 0.45 truncate 0.25` to allow boy→voice single-word fix
+- [ ] **P1** In-mem only ByteArray no file per chunk + snap copy every 800ms race fix
+- [ ] **P1** Adaptive silence `effectiveSilenceThreshold()` far-field 0.001 near-field ambient*1.2 cap 0.01
+- [ ] **P1** Empty audio <1600B trigger as clarification not error
+- [ ] **P2** Sentence TTS queue timed to tail 300ms start before tail + 150ms crossfade stall 15s
+- [ ] **P2** Warm backup Swift pipe daemons model resident 0.36s inference
+- [ ] **P2** Barge-in `level>0.12 && speaking` stop TTS + startListening + server cancel
+- [ ] **P2** Multi-partial while tool runs — instant ack → progress found N → final, stream via tool_progress_callback
+- [ ] **P3** Skip PIL UDP spam throttle to state change
+- [ ] **P3** Desktop hook `use-voice-conversation-live` ChatGPT parity — AudioWorklet 20ms + live draft italic .55 + waveform + header dots
+- [ ] **P3** True standalone plugin without file touch (eliminate patch reapply) + integration test curl 400 not 404 + boy→voice e2e test
+
+---
+
+## 10. Open questions / decisions
+
+1. **TTS choice M2 native:** Kokoro 82M standalone `kokoro==0.9.4` imperative 200ms CPU daemon vs Edge AriaNeural 4.3 MOS cached vs Eleven fallback? Start Kokoro daemon (copy 325MB models from Pi5 `~/kokoro-models/kokoro-v1.0.onnx` + voices.bin) but `mlx-audio` wrapper needs `misaki[en]+spacy+pydantic` conflicts — use standalone `kokoro` pip not `mlx_audio`. MPS is 19x SLOWER LSTM keep CPU.
+2. **Foundation 3B prompt for fast ack:** should we reuse `systemPromptLocal = "Clean up… Return only…"` for polish but new quick reply prompt `draft + final + "quick ack under 20 words contextual on intent"` with low temp 0.1/0.2? Already measured Foundation general + contentTagging UseCases ~2GB quantized ANE+GPU.
+3. **WS compat long-term:** keep `/api/esp32/voice/ws` alias for ESP32 firmware already flashed or only `/api/voice/live` + adapter route? Current plugin runs both by patch; future true plugin could mount both aliases.
+4. **FreeFlow license reuse:** Apache-2.0 clean, prompts can be reused with attribution — ideal for private repo (Reyna family private) not selling.
+5. **Ephemeral vs stored transcript:** Should voice gateway keep both raw + polished in audit for metrics, but only polished + disclaimer in `voice_requests.jsonl`? Recommended: log raw, polished, alts, overlap, guard decisions for future tuning.
+
+---
+
+*Generated 2026-07-15. Sources: Pi5 live gateway audit, Mac mini MCP 37 tools, pythonwhisper ANE traces 0mW ANE 7W CPU, FreeFlow 7f96ccd, Pipecat/Moshi/Qwen OSS landscape.*