 Adolfo Reyna
|
fa4037a2b8
|
docs: future optimizations — FreeFlow polish 3-layer defense, OSS V2V landscape, native M2 WS spec, boy->voice ground-truth lock fix
- Add detailed perf baseline Pi5 (live draft 800ms/8KB, quick ack >30 chars, instant audio Alex<Kokoro, macmini-live-final 753ms vs 11s fallback)
- Rank pain points P0-P3: Mac mini SPOF :7331 down now, patch fragility 1675 lines, ground-truth lock boy->voice, disk IO storm, no sentence TTS queue, single VAD vs adaptive FreeFlow effectiveSilenceThreshold, empty trigger
- Deep dive mrinalwadhwa/freeflow 91★: PolishPipeline 43 rules, <keep> tags, collapseAdjacent, cleanSpuriousCommas, stripNoise/fillers, knownTerms, guard_against_truncation 7f96ccd, skip 83% 0.55s median, warm backup 91% zero handshake, adaptive silence far=0.001 near=ambient*1.2 capped, DictationPipeline actor detached AVAudioEngine + BT SCO 3s timeout, chunk buffer sentence boundary, SessionTiming
- OSS V2V 2025-26: Pipecat 6.5k★ TurnTracker/aggregator/barge-in/e2e tester, LiveKit/TEN, Moshi 7B Mimi codec 12.5Hz 160ms full-duplex dual-stream, Qwen2.5-Omni 2.8-4s — borrow ideas + why stay pipeline text middle = tool-use
- 3-layer ground-truth fix: L1 deterministic 0ms, L2 semantic ANE 3B with precedingText + KNOWN_TERMS + alternatives + overlap 0.45 truncate 0.25, L3 Hermes disclaimer + paraphrase confirmation builder build_voice_user_message()
- Native M2 target WS /api/voice/live single endpoint spec: binary PCM s16le 16k / empty <1600B trigger / JSON start/stop/ping, events ready→listening→draft volatile 60ms 400ms→transcript final polished+raw+alts→instant_response contextual>30→instant_audio Kokoro 200ms→response_text streaming sentence buffer timed 300ms tail + crossfade→done background, VAD dual 1250ms adaptive, barge-in level>0.12, multi-partial while tool runs
- Power truth M2 Air 14,2 macOS 26.5.2: SpeechAnalyzer 0mW ANE 7.3W spike 33% XPC 27-38x RTF 196M EN, Foundation 3B ANE 258-498ms 2GB quant
- Phased plan P0→P3 quick wins: disclaimer wrapper + knownTerms + alt transcriptions + guard tune + in-mem ByteArray + adaptive silence + empty trigger + sentence queue + warm daemons + desktop hook ChatGPT parity
- Refs: patches, hermes_plugin/__init__.py, pythonwhisper ApplePipeTranscriber pipe volatile 31 drafts/6.3s, apple-speech-api skill ANE traces, HTML diagrams hermes-live-voice-native + oss-comparison + imperfect-fix
|
2026-07-15 11:14:45 -04:00 |
|
|
Adolfo Reyna
|
6d593cd9ef
|
feat: initial durable ESP32 voice gateway plugin + patches
- Migrates custom voice gateway from in-tree api_server.py patch to durable plugin
- Auto-restores routes after hermes update (POST /api/esp32/voice + WS /api/esp32/voice/ws)
- Includes live draft ingestion via MacMiniMCP speech_live_transcribe (Apple pipe volatile)
- Includes instant smart reply via apple_llm_quick_reply (3B ANE ~300ms, contextual not generic)
- Guard for missing ipaddress/urlparse imports that caused 500 loop (2026-07-14)
- Tools: esp32_voice_gateway_status, esp32_voice_gateway_restore
- Scripts: restore_after_update.py with 400-not-404 verification
- Installed as ~/.hermes/plugins/esp32-voice-gateway (enabled, survives hermes update)
Verified: routes present, plugin enabled, has_routes=True
|
2026-07-14 23:42:34 -04:00 |
|