adolforeyna/FamReynaBrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

daily auto-sync 2026-07-13

Browse Source
This commit is contained in:
aeroreyna
2026-07-13 23:00:01 -04:00
parent 7ca3659035
commit 5738712a54
33 changed files with 8315 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
projects/voice_agent_platform_research_2026.md
+231
View File
@@ -0,0 +1,231 @@
---
Date: 2026-07-13
Author: Hermes
Tags: [voice-agent, realtime, jarvis, webrtc, platform-research, open-gpt-live, streamcore, project]
---
# Voice Agent Platform Research — Open GPT Live + StreamCore
> Goal: find or compose what we actually want for Reyna home voice agent (Jarvis v2 successor) + potentially family-brain-aware Hermes voice gateway.
Best case: someone else built it already and we just self-host + extend with skills/plugins + brain RAG.
## Use Case — What We Want (Reyna context)
### Current state
- **Jarvis v2** (iMac11,3 Intel i3 550, Ubuntu 26.04): local always-listening wakeword pipeline — PipeWire/ALSA -> Silero VAD -> openWakeWord hey_jarvis TFLite -> faster-whisper base INT8 -> ECAPA speakerID + Haar faceID fusion -> Ollama/Hermes API -> Piper TTS. Works but hardware-limited (internal mic SNR terrible, no PortAudio), needs USB mic, CPU slow.
- **Hermes Agent** (Pi / FamReynaServer): brain at `~/brain` PARA, Telegram/WhatsApp, cron, memory, skills, MCP. No voice realtime path yet.
- **Servers**: Fam .110 Instatic+Sardius, EMI .119 Immich, iMac .124 Chrome CDP 9222
### Ideal target
- **Always-listening + push-to-talk + live barge-in** voice assistant in home
- **Wakeword** "Hey Jarvis" (existing) + followup window (no retrigger)
- **Speaker identification** (Adolfo/Alicia/Grace/Elias) -> adapts persona/context
- **Face corroboration** optional (iMac cam)
- **Brain RAG**: ~/brain PARA searchable (family calendar, projects, habits, EMI)
- **Multi-modal I/O**: Mic/speaker (home device), browser client (any laptop), Telegram voice notes, maybe ESP32/door device later
- **LLM pluggable**: local Ollama qwen2.5:3b + fallback to OpenAI API via Hermes
- **STT/TTS pluggable**: faster-whisper base local + cloud fallback Deepgram/Cartesia
- **Plugin/tools**: dates, weather, family media, Immich photo lookup, Instatic CRUD, calendar create, reminders, kid stories with TTS voices (Grace purple / Aiden etc)
- **Skills/persona**: markdown-defined (British JARVIS wit, adapts per speaker)
- **Low latency**: <500ms STT partial, <1s first token, <800ms first audio barge-in
- **Self-hostable**, no vendor lock, APACHE/MIT preferred
- **WebRTC preferred over raw WS for NAT traversal, Opus, built-in echo cancellation**
---
## Reference 1: Open-GPT-Live — https://github.com/study8677/open-gpt-live
**Snapshot 2026-07-13**
- Stats: 1 ⭐, 1 fork, MIT, TS, Node 22.13+, pnpm 11.7+, v0.2.0 main
- Verified docs: `docs/protocol.md`, `configuration.md`, `local-ai.md`, `deployment.md`, `browser-support.md`, `live-mode-smoke-test.md`
### What it is
Open session layer between browser audio and LLM. Not hosted assistant, but reference impl for realtime voice UX pattern: VAD calibration -> live transcripts -> streamed LLM -> TTS playback -> interrupt, over typed WebSocket you can inspect/replace.
### Architecture
```
apps/web (Next.js 14 + React 18) -> browser capture/VAD/pre-roll(400ms PCM)/playback/reconnect
apps/gateway (Node ws + esbuild CJS) -> session state, validation, STT/LLM/TTS orchestration, :8787/healthz
packages/protocol -> TS shared WS contracts
packages/adapters -> batch + streaming provider interfaces
```
Flow: `Browser (text·PTT·VAD·PCM) --typed WS--> Gateway --RealtimeSTT/BatchSTT/LLM/TTS--> Browser`
### Protocol (parseClientMessage / parseServerMessage validated)
Client->Gw: session.start, user.text {text}, audio.chunk {requestId, chunk base64 webm-opus OR pcm16 24kHz, sequence, isFinal, turnMode ptt|live}, vad.speech_start {live, startedAt, rms}, vad.speech_end {endedAt, durationMs, reason silence|manual|cancelled}, interrupt {requestId, reason}, playback.ack
Gw->Client: session.start ready, transcript.partial {text, sequence, isStable}, transcript.final, llm.delta, llm.done {stop|interrupted|error}, tts.start/chunk/end, error
Latency telemetry built-in (browser performance.now + gateway monotonic, never mixed): first STT partial, final transcript, LLM first token, TTS first audio, speech_end->audio, keyed by requestId/turnMode/sttPath batch|realtime|batch_fallback
### Key strengths to reuse/copy
- **Adaptive RMS VAD + noise-floor calibration + 400ms pre-roll + hangover + playback suppress** — far better than Jarvis v2 energy threshold 0.015. Code in `apps/web/src` VAD tunables (15 NEXT_PUBLIC_VAD_* envs). We should copy/adapt: adaptive multiplier (speech_noise 3x, silence 1.8x), dynamic min/max 0.012-0.12 speech, start debounce 160ms, min speech 200ms, hangover 750ms, playback threshold multiplier 2.5 + 300ms suppress after end.
- **Typed protocol with runtime validation** (not just types) — good pattern for Reyna voice gateway
- **Sentence-segmented TTS streaming** (ttsSegmentMinLength 24, Max 240) + interrupt abort across LLM/TTS/playback
- **Latency logging infrastructure** -- we need this for jarvis v2 tuning
- **Local-AI profile** Docker Compose: Ollama 0.30.8 qwen2.5:1.5b 986MB + Speaches 0.8.3 faster-whisper-small 486MB + Kokoro 82M ONNX 354MB = 1.8GB total, optional deps, named volumes
### Limitations for us
- No wakeword (assumes manual PTT or Live Mode toggle)
- No speakerID, no faceID
- History in-memory per WS, no persistence, no session restore
- No auth/billing, no RAG, no plugin/tool system (LLM only)
- No WebRTC — plain WS, so NAT/echo-cancellation is browser's job
- VAD is browser energy RMS, not Silero ONNX — less robust than jarvis v2 silero-vad
- Chromium baseline only
- Single user, not multi-profile/home device
### Usable pieces for Jarvis v3
- Drop-in replacement for web client if we want browser voice UI: copy VAD calibration logic + adaptive threshold algorithm to Jarvis
- Gateway `createGatewayServer` session state pattern (ActiveRun, ActiveAudioTurn, timing) — cleaner than current pipeline.py state machine
- Protocol validation approach for future Hermes voice gateway
---
## Reference 2: StreamCore Server — https://github.com/streamcoreai/streamcore-server
**Snapshot 2026-07-13**
- Stats: 10 ⭐, 2 forks, Apache-2.0, Go 1.25, size 203KB
- Desc: "Open-source realtime voice agent server in Go with WebRTC (WHIP), barge-in, streaming STT/LLM/TTS pipelines, plugin system, multi-language SDKs, SIP telephony, ESP32 support & fully local mode"
- Org family: streamcoreai — server + examples + js-sdk + go-sdk + python-sdk + rust-sdk + plugin-sdk + esp32 + sip-server
### What it is
Go core realtime voice agent with WebRTC WHIP (RFC 9725) single POST SDP exchange, Opus RTP, Pion (no coturn needed, built-in STUN/TURN 3478 + 50001-60000), barge-in, plus plugin system (Python/TS/JS over JSON-RPC + native Go tools zero-IPC), Markdown skills, built-in RAG (pgvector/Supabase + streamcore-cli ingestion .txt/.md/.csv/.pdf/.docx/.xlsx chunk 512w overlap 64w).
Compared to open-gpt-live: Go vs Node, WebRTC vs WebSocket, full product runtime vs reference UI.
### Architecture
```
Client/SDK (Mic -> WebRTC Opus --RTP--> Go Server (Opion) -> Opus decode -> STT -> RAG -> Skills prompt -> LLM -> tools -> TTS -> Opus encode --RTP--> Client speaker)
Signaling: POST /whip SDP offer/answer + server session ID, no persistent signaling socket
DataChannel <- events: transcripts, LLM deltas, tool calls
internal/: audio, config, llm (openai/ollama), peer, pipeline (agent.go inbound.go outbound.go pipeline.go thinking.go vision.go), plugin (manager LoadAll JSON-RPC), rag (pgvector/supabase), session (manager.go session.go), signaling (WHIPHandler), stt (deepgram/openai/vibevoice), tts (cartesia/deepgram/elevenlabs/vibevoice), turn (Pion), vad, tools (native Go car.* drivetrain example)
plugins/plugins: gmail, math-calculate, time-get, vision-analyze, weather-get
plugins/skills: concise-responder, desktop-car-driver, error-recovery, friendly-conversationalist, gmail-assistant, polite-assistant, tool-savvy, vision-assistant
external/vibeVoice: vibeVoiceAsr/server.py ws://127.0.0.1:8200 (MLX mlx-audio Apple Silicon or torch Linux) + vibeVoiceTTS/server.py http://127.0.0.1:8300 VibeVoice-ASR-4bit + VibeVoice-Realtime-0.5B
examples/ org repo: TS browser client connecting to /whip (similar to open-gpt-live web but WebRTC)
```
### Providers
STT: deepgram nova-3 streaming partials, openai whisper final, vibevoice local WS
LLM: openai gpt-4o-mini, ollama llama3.2/gemma4:e4b/qwen2.5 local
TTS: cartesia, deepgram, elevenlabs eleven_turbo_v2_5, vibevoice local HTTP
RAG: pgvector Postgres vector(1536) + table documents, Supabase RPC match_documents, embeddings OpenAI text-embedding-3-small, top_k 3
STUN/TURN: built-in Pion, enabled when server.public_ip + turn_secret set
### Config (config.toml — excerpt)
```
[server] port 8080 public_ip "" turn_secret "" jwt_secret "" api_key ""
[plugins] directory ./plugins (required to load)
[pipeline] barge_in true greeting "" greeting_outgoing "" debug false (timing events over DataChannel)
[stt] provider deepgram/openai/vibevoice
[llm] provider openai/ollama
[tts] provider cartesia/deepgram/elevenlabs/vibevoice
[deepgram] api_key model nova-3
[openai] api_key model gpt-4o-mini system_prompt "concise conversational"
[ollama] base_url http://localhost:11434 model llama3.2
[cartesia] api_key voice_id
[vibevoice] asr_url ws://127.0.0.1:8200 tts_url http://127.0.0.1:8300 voice en-Emma_woman
[rag] provider supabase|pgvector top_k 3 embedding_model text-embedding-3-small
[supabase] url api_key function match_documents table documents
```
### Key strengths to reuse/copy — directly matches what we want
1. **WebRTC WHIP + Opus + barge-in + DataChannel events + built-in STUN/TURN** — solves open-gpt-live NAT/echo gaps. Opus better than webm-opus over WS, RTP read/decode in Go goroutines + bounded channels predictable latency.
2. **Multi-language SDKs**: @streamcore/js-sdk (browser), github.com/streamcoreai/go-sdk, streamcoreai-sdk (Python), rust-sdk — means Hermes Pi can use Python/Go SDK, browser uses TS, same protocol, no reimplement.
3. **Plugin system over JSON-RPC (Python/TS/JS) + native Go tools**: we could port Hermes skills? plugins example: gmail, weather, math, time, vision-analyze. Perfect for home: Immich search, calendar-create, Instatic CRUD, reminder, kid story pipeline.
4. **Markdown Skills**: behavioral layer separate from capabilities — e.g., concise-responder.md, friendly-conversationalist.md — exactly what we drafted for Jarvis soul.md (British wit per speaker). We can define skills per speaker: Adolfo technical concise, Grace playful purple voice, Elias etc.
5. **Built-in RAG with CLI ingestion**: streamcore-cli ingest docs/*.pdf/*.md/*.xlsx chunk 512/overlap 64 -> embeddings -> pgvector/Supabase. That CLI reads server config.toml for credentials. For us: ingest ~/brain into pgvector locally (FamReynaServer) to make Hermes family-brain aware in voice.
6. **Fully local mode (no API keys)**: Ollama llm + VibeVoice ASR/TTS sidecars — alternative to open-gpt-live local-ai (Ollama+Speaches). VibeVoice uses mlx-audio on Apple Silicon or torch on Linux — could run on iMac? Actually needs mlx for Mac, but our iMac is Intel Linux not Apple Silicon, so torch path needed. Speaches equivalent already works CPU.
7. **JWT auth /token + CORS + /health + debug timing** — production hints.
8. **ESP32 + SIP family**: org has esp32 repo + sip-server — for future doorbell ESP32-S3 voice device or phone calling (outbound greeting_outgoing for SIP).
9. **Thinking sound** audible tone via RTP while slow tool executes — nice UX for kid-facing.
### Limitations / gaps to close
- No wakeword either in server — assumes continuous session (but could add plugin or client-side wake like jarvis v2)
- No speakerID / faceID fusion (we have this in jarvis v2 — ECAPA + face_recognition histogram)
- No VAD details yet verified for local — relies on WebRTC VAD + Deepgram interim partials; custom Silero VAD could be added in internal/vad
- RAG requires OpenAI embeddings API key still (even for pgvector) — not fully local; could swap to local embeddings (e.g., nomic-embed) — TODO.
- Small community (10 ⭐) — less mature than LiveKit? But code is small 203KB + Go idiomatic, easier to fork.
- Brain PARA sync not covered — needs engraft?
- Pi host resource usage not benchmarked — Go core low memory though
### Comparison table — us vs references
| Feature | Jarvis v2 (today) | Open-GPT-Live | StreamCore Server | Ideal v3 |
|---------|-------------------|---------------|-------------------|----------|
| Transport | PipeWire local only | WS typed | WebRTC WHIP Opus RTP + DataChannel | **WebRTC + WHIP** |
| VAD | Silero ONNX + energy fallback | Adaptive RMS energy + calibration | WebRTC + Deepgram interim + internal/vad (unknown) | **Silero ONNX in Go OR keep adaptive RMS but add noise-floor cal from OGL** |
| Wakeword | openWakeWord hey_jarvis TFLite 0.45 + STT keyword fallback | None (PTT/Live toggle) | None (always-on session) | **Keep oWW + followup window 12s max 5 turns** |
| STT | faster-whisper base INT8 CPU | batch + streaming OpenAI Realtime | Deepgram streaming + OpenAI Whisper final + VibeVoice local WS | **faster-whisper local CPU + Deepgram cloud fallback** |
| SpeakerID | ECAPA-TDNN voxceleb 192-d + stats fallback | None | None | **Keep ECAPA, port to Go? or Python plugin** |
| FaceID | Haar + histogram fallback, needs frontal | None | vision.go placeholder | **Keep, as Python plugin via camera** |
| LLM | Ollama qwen2.5:3b / hermes_api | OpenAI compat + Ollama | OpenAI + Ollama | **Ollama local + Hermes API fallback to main server** |
| TTS | Piper ONNX local + spd-say | OpenAI + Speaches Kokoro MP3 | Cartesia/11Labs/Deepgram/VibeVoice | **Piper ONNX or Kokoro + ElevenLabs fallback for Grace/Poppy purple voice** |
| Barge-in | Followup window but no TTS interrupt while playing | Yes abort across LLM/TTS/playback | Yes pipeline.barge_in | **Must have** |
| Skills/Persona | soul.md | systemPrompt env | Markdown skills per dir | **Markdown skills per speaker** |
| Plugins/Tools | None (future MCP) | None (LLM only) | Python/TS/JS JSON-RPC + native Go | **Use plugin system for brain, Immich, calendar, Instatic, reminders** |
| RAG | None (~/brain search via tool needed) | None | Built-in pgvector/Supabase + CLI ingestion | **pgvector local on .110 with brain md + Immich captions** |
| SDKs | CLI only | Web Next.js | TS/Go/Python/Rust + browser example | **Need TS browser + Python Hermes** |
| Multi-device | Single iMac | Single browser | Browser + ESP32 + SIP | **Browser + ESP32 + Telegram voice** |
| Latency telemetry | No (manual) | Yes detailed timing logs | debug true timing via DataChannel | **Both** |
| Self-host | Yes | Yes docker-compose | Yes Docker + Go binary | Yes |
| License | — | MIT | Apache-2.0 | OK |
### Recommendation: What is closest to what we want?
**StreamCore is 80% of what we want — open-gpt-live is 30% UI inspiration.**
- If someone built what we want, **StreamCore is the chassis**: Go realtime core + WebRTC + plugin/tools + markdown skills + RAG + SDKs + ESP32/SIP future — exactly our needs except wakeword + speakerID.
- Open-GPT-Live is excellent **reference for VAD calibration UX + adaptive threshold algo + protocol validation + latency metrics + local-ai Compose pattern**, but lacks extensibility for our use case (no plugins, no RAG, no WebRTC, no ESP32).
- Neither gives us speakerID/fusion — our jarvis v2 uniqueness should be preserved.
### Proposed path (Jarvis v3 / Hermes Voice)
Option A — **Fork + extend StreamCore (recommended)**
- Clone streamcore-server into `~/Projects/streamcore-fork` on .110 or iMac
- Keep Go server as is, add:
1. **Wakeword pre-filter plugin** (Python sidecar runs openWakeWord TFLite, emits event to DataChannel to start session) OR client-side wake in js-sdk before WHIP connect (like jarvis v2 IDLE->WAKE_DETECTED)
2. **SpeakerID plugin**: Python plugin `speaker-identify` loads ECAPA embeddings from Jarvis v2 `data/speakers/` NPY averages, returns speaker name + conf; use as RAG/skill selector (Adolfo vs Grace)
3. **FaceID plugin**: optional, uses iMac cam or ESP32 cam, histogram fallback
4. **Brain RAG**: deploy local Postgres + pgvector on FamReynaServer .110 (or use existing?), run `streamcore-cli ingest ~/brain/projects/*.md ~/brain/areas/*.md journals/*.md`, embeddings via OpenAI or swap to local `nomic-embed-text`
5. **Hermes plugin**: Python plugin that calls Hermes API (main server 192.168.68.110:8000) for tool routing — calendar, reminders, Instatic, Immich via MCP
6. **TTS voices**: keep Piper for local fast, Cartesia/ElevenLabs for Grace/Poppy purple voice
7. Copy OGL adaptive VAD calibration: implement internal/vad upgrade with noise floor smoothing 0.08, speech noise multiplier 3, silence 1.8, dynamic min/max, debounce 160ms, hangover 750ms, playback threshold multiplier 2.5, pre-roll 400ms, calibration 1000ms
- Use existing FAT: AdGuard? Keep .110 low resource, Go core fits (Pi 3.9Gi vs iMac 15Gi)
- Browser client from examples/typescript -> customize with VOUS/Hillsong 120px serif branding from memory (user hates generic boxes)
- Later: ESP32 device using streamcoreai/esp32 repo for kitchen/door voice terminal
Option B — **Combine OGL web + StreamCore Go gateway (hybrid)**
- Use OGL Next.js web for VAD UI polish, but swap transport to StreamCore WHIP / keep WS for text?
- More plumbing, not recommended unless we love OGL UI
Option C — **Keep jarvis v2 and graft pieces**
- Smallest lift: copy OGL VAD calibration logic into jarvis-v2 audio/vad.py adaptive, add StreamCore plugin ideas as LLM tools
- Stays iMac-only, no WebRTC/multi-device, no RAG — acceptable MVP but not what we want long-term
### Concrete next steps (if proceed)
1. Clone both repos locally to .110, run StreamCore Docker quickstart (cp config.toml.example config.toml, deepgram openai keys or ollama local). Verify browser client at :3000->:8080/whip works.
2. Benchmark Go server resource on Pi (top, check ~200MB RAM target)
3. Test local-ai profile: Ollama qwen2.5:3b + VibeVoice sidecars (torch path on Ubuntu i3 = slow, maybe stick to Speaches pipeline initially — compare)
4. Build `streamcore-cli` and ingest 5 sample brain docs into Supabase or local pgvector, verify RAG context injection in LLM call (zero tool overhead)
5. Port wakeword: quick Python plugin openWakeWord score over mic before WHIP connect, or js-sdk audio worklet.
6. Port SpeakerID: re-use jarvis-v2 data/speakers/*.npy, plugin returns speaker label.
7. Define skills dir: `plugins/skills/jarvis-british.md`, `plugins/skills/grace-playful.md` + persona per speaker
8. Integrate Hermes API: plugin `hermes-bridge` JSON-RPC calls main server tools (calendar create, reminders list, etc.) — use existing MCP reyna2026! token?
9. Compare latency telemetry to OGL's telemetry — unify dashboard.
### Links
- https://github.com/study8677/open-gpt-live — MIT, WS, adaptive VAD, Speaches+Ollama local
- https://github.com/streamcoreai/streamcore-server — Apache2, Go, WebRTC WHIP, Pion built-in TURN, barge-in, plugins TS/Python/JS+Go native, markdown skills, pgvector RAG, ESP32+SIP family
- Org: https://github.com/streamcoreai (examples, js-sdk, go-sdk, python-sdk, rust-sdk, plugin-sdk, esp32, sip-server)
- VibeVoice: https://huggingface.co/mlx-community/VibeVoice-ASR-4bit, https://huggingface.co/mlx-community/VibeVoice-Realtime-0.5B-6bit
- Related brain: [[Jarvis v2 - Always Listening Voice Assistant]], [[PARA Brain MCP Server]], [[Reyna CLI Browser Control]], [[Grace Poppy Storybook Image Voice App]]
## TODO
- [ ] Clone streamcore-server + examples locally on .110 and test Docker path with Deepgram key? Do we have keys stored?
- [ ] Check Postgres pgvector availability on FamReynaServer .110 (Instatic uses SQLite/MCP?) Could run docker pgvector.
- [ ] Document USB mic purchase recommendation again (iMac internal too quiet per Jarvis v2 test log peak 0.014 raw)
- [ ] Consider whether Hermes Agent itself should expose /whip WHIP endpoint via skill/plugin vs separate Go service.
- [ ] Evaluate LiveKit as third alternative? (mentioned by many but heavier)
- [ ] Add to brain index Projects list