FamReynaBrain/projects/voice_agent_platform_research_2026.md

Date, Author, Tags

Date	Author	Tags
2026-07-13	Hermes	voice-agent realtime jarvis webrtc platform-research open-gpt-live streamcore project

Voice Agent Platform Research — Open GPT Live + StreamCore

Goal: find or compose what we actually want for Reyna home voice agent (Jarvis v2 successor) + potentially family-brain-aware Hermes voice gateway. Best case: someone else built it already and we just self-host + extend with skills/plugins + brain RAG.

Use Case — What We Want (Reyna context)

Current state

• Jarvis v2 (iMac11,3 Intel i3 550, Ubuntu 26.04): local always-listening wakeword pipeline — PipeWire/ALSA -> Silero VAD -> openWakeWord hey_jarvis TFLite -> faster-whisper base INT8 -> ECAPA speakerID + Haar faceID fusion -> Ollama/Hermes API -> Piper TTS. Works but hardware-limited (internal mic SNR terrible, no PortAudio), needs USB mic, CPU slow.
• Hermes Agent (Pi / FamReynaServer): brain at ~/brain PARA, Telegram/WhatsApp, cron, memory, skills, MCP. No voice realtime path yet.
• Servers: Fam .110 Instatic+Sardius, EMI .119 Immich, iMac .124 Chrome CDP 9222

Ideal target

• Always-listening + push-to-talk + live barge-in voice assistant in home
• Wakeword "Hey Jarvis" (existing) + followup window (no retrigger)
• Speaker identification (Adolfo/Alicia/Grace/Elias) -> adapts persona/context
• Face corroboration optional (iMac cam)
• Brain RAG: ~/brain PARA searchable (family calendar, projects, habits, EMI)
• Multi-modal I/O: Mic/speaker (home device), browser client (any laptop), Telegram voice notes, maybe ESP32/door device later
• LLM pluggable: local Ollama qwen2.5:3b + fallback to OpenAI API via Hermes
• STT/TTS pluggable: faster-whisper base local + cloud fallback Deepgram/Cartesia
• Plugin/tools: dates, weather, family media, Immich photo lookup, Instatic CRUD, calendar create, reminders, kid stories with TTS voices (Grace purple / Aiden etc)
• Skills/persona: markdown-defined (British JARVIS wit, adapts per speaker)
• Low latency: <500ms STT partial, <1s first token, <800ms first audio barge-in
• Self-hostable, no vendor lock, APACHE/MIT preferred
• WebRTC preferred over raw WS for NAT traversal, Opus, built-in echo cancellation

---

Reference 1: Open-GPT-Live — https://github.com/study8677/open-gpt-live

Snapshot 2026-07-13

• Stats: 1 ⭐, 1 fork, MIT, TS, Node 22.13+, pnpm 11.7+, v0.2.0 main
• Verified docs: docs/protocol.md, configuration.md, local-ai.md, deployment.md, browser-support.md, live-mode-smoke-test.md

What it is

Open session layer between browser audio and LLM. Not hosted assistant, but reference impl for realtime voice UX pattern: VAD calibration -> live transcripts -> streamed LLM -> TTS playback -> interrupt, over typed WebSocket you can inspect/replace.

Architecture

apps/web (Next.js 14 + React 18) -> browser capture/VAD/pre-roll(400ms PCM)/playback/reconnect
apps/gateway (Node ws + esbuild CJS) -> session state, validation, STT/LLM/TTS orchestration, :8787/healthz
packages/protocol -> TS shared WS contracts
packages/adapters -> batch + streaming provider interfaces

Flow: Browser (text·PTT·VAD·PCM) --typed WS--> Gateway --RealtimeSTT/BatchSTT/LLM/TTS--> Browser

Protocol (parseClientMessage / parseServerMessage validated)

Client->Gw: session.start, user.text {text}, audio.chunk {requestId, chunk base64 webm-opus OR pcm16 24kHz, sequence, isFinal, turnMode ptt|live}, vad.speech_start {live, startedAt, rms}, vad.speech_end {endedAt, durationMs, reason silence|manual|cancelled}, interrupt {requestId, reason}, playback.ack Gw->Client: session.start ready, transcript.partial {text, sequence, isStable}, transcript.final, llm.delta, llm.done {stop|interrupted|error}, tts.start/chunk/end, error

Latency telemetry built-in (browser performance.now + gateway monotonic, never mixed): first STT partial, final transcript, LLM first token, TTS first audio, speech_end->audio, keyed by requestId/turnMode/sttPath batch|realtime|batch_fallback

Key strengths to reuse/copy

• Adaptive RMS VAD + noise-floor calibration + 400ms pre-roll + hangover + playback suppress — far better than Jarvis v2 energy threshold 0.015. Code in apps/web/src VAD tunables (15 NEXT_PUBLIC_VAD_* envs). We should copy/adapt: adaptive multiplier (speech_noise 3x, silence 1.8x), dynamic min/max 0.012-0.12 speech, start debounce 160ms, min speech 200ms, hangover 750ms, playback threshold multiplier 2.5 + 300ms suppress after end.
• Typed protocol with runtime validation (not just types) — good pattern for Reyna voice gateway
• Sentence-segmented TTS streaming (ttsSegmentMinLength 24, Max 240) + interrupt abort across LLM/TTS/playback
• Latency logging infrastructure -- we need this for jarvis v2 tuning
• Local-AI profile Docker Compose: Ollama 0.30.8 qwen2.5:1.5b 986MB + Speaches 0.8.3 faster-whisper-small 486MB + Kokoro 82M ONNX 354MB = 1.8GB total, optional deps, named volumes

Limitations for us

• No wakeword (assumes manual PTT or Live Mode toggle)
• No speakerID, no faceID
• History in-memory per WS, no persistence, no session restore
• No auth/billing, no RAG, no plugin/tool system (LLM only)
• No WebRTC — plain WS, so NAT/echo-cancellation is browser's job
• VAD is browser energy RMS, not Silero ONNX — less robust than jarvis v2 silero-vad
• Chromium baseline only
• Single user, not multi-profile/home device

Usable pieces for Jarvis v3

• Drop-in replacement for web client if we want browser voice UI: copy VAD calibration logic + adaptive threshold algorithm to Jarvis
• Gateway createGatewayServer session state pattern (ActiveRun, ActiveAudioTurn, timing) — cleaner than current pipeline.py state machine
• Protocol validation approach for future Hermes voice gateway

---

Reference 2: StreamCore Server — https://github.com/streamcoreai/streamcore-server

Snapshot 2026-07-13

• Stats: 10 ⭐, 2 forks, Apache-2.0, Go 1.25, size 203KB
• Desc: "Open-source realtime voice agent server in Go with WebRTC (WHIP), barge-in, streaming STT/LLM/TTS pipelines, plugin system, multi-language SDKs, SIP telephony, ESP32 support & fully local mode"
• Org family: streamcoreai — server + examples + js-sdk + go-sdk + python-sdk + rust-sdk + plugin-sdk + esp32 + sip-server

What it is

Go core realtime voice agent with WebRTC WHIP (RFC 9725) single POST SDP exchange, Opus RTP, Pion (no coturn needed, built-in STUN/TURN 3478 + 50001-60000), barge-in, plus plugin system (Python/TS/JS over JSON-RPC + native Go tools zero-IPC), Markdown skills, built-in RAG (pgvector/Supabase + streamcore-cli ingestion .txt/.md/.csv/.pdf/.docx/.xlsx chunk 512w overlap 64w).

Compared to open-gpt-live: Go vs Node, WebRTC vs WebSocket, full product runtime vs reference UI.

Architecture

Client/SDK (Mic -> WebRTC Opus --RTP--> Go Server (Opion) -> Opus decode -> STT -> RAG -> Skills prompt -> LLM -> tools -> TTS -> Opus encode --RTP--> Client speaker)
Signaling: POST /whip SDP offer/answer + server session ID, no persistent signaling socket
DataChannel <- events: transcripts, LLM deltas, tool calls

internal/: audio, config, llm (openai/ollama), peer, pipeline (agent.go inbound.go outbound.go pipeline.go thinking.go vision.go), plugin (manager LoadAll JSON-RPC), rag (pgvector/supabase), session (manager.go session.go), signaling (WHIPHandler), stt (deepgram/openai/vibevoice), tts (cartesia/deepgram/elevenlabs/vibevoice), turn (Pion), vad, tools (native Go car.* drivetrain example)
plugins/plugins: gmail, math-calculate, time-get, vision-analyze, weather-get
plugins/skills: concise-responder, desktop-car-driver, error-recovery, friendly-conversationalist, gmail-assistant, polite-assistant, tool-savvy, vision-assistant
external/vibeVoice: vibeVoiceAsr/server.py ws://127.0.0.1:8200 (MLX mlx-audio Apple Silicon or torch Linux) + vibeVoiceTTS/server.py http://127.0.0.1:8300 VibeVoice-ASR-4bit + VibeVoice-Realtime-0.5B
examples/ org repo: TS browser client connecting to /whip (similar to open-gpt-live web but WebRTC)

Providers

STT: deepgram nova-3 streaming partials, openai whisper final, vibevoice local WS LLM: openai gpt-4o-mini, ollama llama3.2/gemma4:e4b/qwen2.5 local TTS: cartesia, deepgram, elevenlabs eleven_turbo_v2_5, vibevoice local HTTP RAG: pgvector Postgres vector(1536) + table documents, Supabase RPC match_documents, embeddings OpenAI text-embedding-3-small, top_k 3 STUN/TURN: built-in Pion, enabled when server.public_ip + turn_secret set

Config (config.toml — excerpt)

[server] port 8080 public_ip "" turn_secret "" jwt_secret "" api_key ""
[plugins] directory ./plugins (required to load)
[pipeline] barge_in true greeting "" greeting_outgoing "" debug false (timing events over DataChannel)
[stt] provider deepgram/openai/vibevoice
[llm] provider openai/ollama
[tts] provider cartesia/deepgram/elevenlabs/vibevoice
[deepgram] api_key model nova-3
[openai] api_key model gpt-4o-mini system_prompt "concise conversational"
[ollama] base_url http://localhost:11434 model llama3.2
[cartesia] api_key voice_id
[vibevoice] asr_url ws://127.0.0.1:8200 tts_url http://127.0.0.1:8300 voice en-Emma_woman
[rag] provider supabase|pgvector top_k 3 embedding_model text-embedding-3-small
[supabase] url api_key function match_documents table documents

Key strengths to reuse/copy — directly matches what we want

1. WebRTC WHIP + Opus + barge-in + DataChannel events + built-in STUN/TURN — solves open-gpt-live NAT/echo gaps. Opus better than webm-opus over WS, RTP read/decode in Go goroutines + bounded channels predictable latency.
2. Multi-language SDKs: @streamcore/js-sdk (browser), github.com/streamcoreai/go-sdk, streamcoreai-sdk (Python), rust-sdk — means Hermes Pi can use Python/Go SDK, browser uses TS, same protocol, no reimplement.
3. Plugin system over JSON-RPC (Python/TS/JS) + native Go tools: we could port Hermes skills? plugins example: gmail, weather, math, time, vision-analyze. Perfect for home: Immich search, calendar-create, Instatic CRUD, reminder, kid story pipeline.
4. Markdown Skills: behavioral layer separate from capabilities — e.g., concise-responder.md, friendly-conversationalist.md — exactly what we drafted for Jarvis soul.md (British wit per speaker). We can define skills per speaker: Adolfo technical concise, Grace playful purple voice, Elias etc.
5. Built-in RAG with CLI ingestion: streamcore-cli ingest docs/.pdf/.md/*.xlsx chunk 512/overlap 64 -> embeddings -> pgvector/Supabase. That CLI reads server config.toml for credentials. For us: ingest ~/brain into pgvector locally (FamReynaServer) to make Hermes family-brain aware in voice.
6. Fully local mode (no API keys): Ollama llm + VibeVoice ASR/TTS sidecars — alternative to open-gpt-live local-ai (Ollama+Speaches). VibeVoice uses mlx-audio on Apple Silicon or torch on Linux — could run on iMac? Actually needs mlx for Mac, but our iMac is Intel Linux not Apple Silicon, so torch path needed. Speaches equivalent already works CPU.
7. JWT auth /token + CORS + /health + debug timing — production hints.
8. ESP32 + SIP family: org has esp32 repo + sip-server — for future doorbell ESP32-S3 voice device or phone calling (outbound greeting_outgoing for SIP).
9. Thinking sound audible tone via RTP while slow tool executes — nice UX for kid-facing.

Limitations / gaps to close

• No wakeword either in server — assumes continuous session (but could add plugin or client-side wake like jarvis v2)
• No speakerID / faceID fusion (we have this in jarvis v2 — ECAPA + face_recognition histogram)
• No VAD details yet verified for local — relies on WebRTC VAD + Deepgram interim partials; custom Silero VAD could be added in internal/vad
• RAG requires OpenAI embeddings API key still (even for pgvector) — not fully local; could swap to local embeddings (e.g., nomic-embed) — TODO.
• Small community (10 ⭐) — less mature than LiveKit? But code is small 203KB + Go idiomatic, easier to fork.
• Brain PARA sync not covered — needs engraft?
• Pi host resource usage not benchmarked — Go core low memory though

Comparison table — us vs references

Feature	Jarvis v2 (today)	Open-GPT-Live	StreamCore Server	Ideal v3
Transport	PipeWire local only	WS typed	WebRTC WHIP Opus RTP + DataChannel	WebRTC + WHIP
VAD	Silero ONNX + energy fallback	Adaptive RMS energy + calibration	WebRTC + Deepgram interim + internal/vad (unknown)	Silero ONNX in Go OR keep adaptive RMS but add noise-floor cal from OGL
Wakeword	openWakeWord hey_jarvis TFLite 0.45 + STT keyword fallback	None (PTT/Live toggle)	None (always-on session)	Keep oWW + followup window 12s max 5 turns
STT	faster-whisper base INT8 CPU	batch + streaming OpenAI Realtime	Deepgram streaming + OpenAI Whisper final + VibeVoice local WS	faster-whisper local CPU + Deepgram cloud fallback
SpeakerID	ECAPA-TDNN voxceleb 192-d + stats fallback	None	None	Keep ECAPA, port to Go? or Python plugin
FaceID	Haar + histogram fallback, needs frontal	None	vision.go placeholder	Keep, as Python plugin via camera
LLM	Ollama qwen2.5:3b / hermes_api	OpenAI compat + Ollama	OpenAI + Ollama	Ollama local + Hermes API fallback to main server
TTS	Piper ONNX local + spd-say	OpenAI + Speaches Kokoro MP3	Cartesia/11Labs/Deepgram/VibeVoice	Piper ONNX or Kokoro + ElevenLabs fallback for Grace/Poppy purple voice
Barge-in	Followup window but no TTS interrupt while playing	Yes abort across LLM/TTS/playback	Yes pipeline.barge_in	Must have
Skills/Persona	soul.md	systemPrompt env	Markdown skills per dir	Markdown skills per speaker
Plugins/Tools	None (future MCP)	None (LLM only)	Python/TS/JS JSON-RPC + native Go	Use plugin system for brain, Immich, calendar, Instatic, reminders
RAG	None (~/brain search via tool needed)	None	Built-in pgvector/Supabase + CLI ingestion	pgvector local on .110 with brain md + Immich captions
SDKs	CLI only	Web Next.js	TS/Go/Python/Rust + browser example	Need TS browser + Python Hermes
Multi-device	Single iMac	Single browser	Browser + ESP32 + SIP	Browser + ESP32 + Telegram voice
Latency telemetry	No (manual)	Yes detailed timing logs	debug true timing via DataChannel	Both
Self-host	Yes	Yes docker-compose	Yes Docker + Go binary	Yes
License	—	MIT	Apache-2.0	OK

Recommendation: What is closest to what we want?

StreamCore is 80% of what we want — open-gpt-live is 30% UI inspiration.

• If someone built what we want, StreamCore is the chassis: Go realtime core + WebRTC + plugin/tools + markdown skills + RAG + SDKs + ESP32/SIP future — exactly our needs except wakeword + speakerID.
• Open-GPT-Live is excellent reference for VAD calibration UX + adaptive threshold algo + protocol validation + latency metrics + local-ai Compose pattern, but lacks extensibility for our use case (no plugins, no RAG, no WebRTC, no ESP32).
• Neither gives us speakerID/fusion — our jarvis v2 uniqueness should be preserved.

Proposed path (Jarvis v3 / Hermes Voice)

Option A — Fork + extend StreamCore (recommended)

• Clone streamcore-server into ~/Projects/streamcore-fork on .110 or iMac
• Keep Go server as is, add:
  1. Wakeword pre-filter plugin (Python sidecar runs openWakeWord TFLite, emits event to DataChannel to start session) OR client-side wake in js-sdk before WHIP connect (like jarvis v2 IDLE->WAKE_DETECTED)
  2. SpeakerID plugin: Python plugin speaker-identify loads ECAPA embeddings from Jarvis v2 data/speakers/ NPY averages, returns speaker name + conf; use as RAG/skill selector (Adolfo vs Grace)
  3. FaceID plugin: optional, uses iMac cam or ESP32 cam, histogram fallback
  4. Brain RAG: deploy local Postgres + pgvector on FamReynaServer .110 (or use existing?), run streamcore-cli ingest ~/brain/projects/*.md ~/brain/areas/*.md journals/*.md, embeddings via OpenAI or swap to local nomic-embed-text
  5. Hermes plugin: Python plugin that calls Hermes API (main server 192.168.68.110:8000) for tool routing — calendar, reminders, Instatic, Immich via MCP
  6. TTS voices: keep Piper for local fast, Cartesia/ElevenLabs for Grace/Poppy purple voice
  7. Copy OGL adaptive VAD calibration: implement internal/vad upgrade with noise floor smoothing 0.08, speech noise multiplier 3, silence 1.8, dynamic min/max, debounce 160ms, hangover 750ms, playback threshold multiplier 2.5, pre-roll 400ms, calibration 1000ms
• Use existing FAT: AdGuard? Keep .110 low resource, Go core fits (Pi 3.9Gi vs iMac 15Gi)
• Browser client from examples/typescript -> customize with VOUS/Hillsong 120px serif branding from memory (user hates generic boxes)
• Later: ESP32 device using streamcoreai/esp32 repo for kitchen/door voice terminal

Option B — Combine OGL web + StreamCore Go gateway (hybrid)

• Use OGL Next.js web for VAD UI polish, but swap transport to StreamCore WHIP / keep WS for text?
• More plumbing, not recommended unless we love OGL UI

Option C — Keep jarvis v2 and graft pieces

• Smallest lift: copy OGL VAD calibration logic into jarvis-v2 audio/vad.py adaptive, add StreamCore plugin ideas as LLM tools
• Stays iMac-only, no WebRTC/multi-device, no RAG — acceptable MVP but not what we want long-term

Concrete next steps (if proceed)

1. Clone both repos locally to .110, run StreamCore Docker quickstart (cp config.toml.example config.toml, deepgram openai keys or ollama local). Verify browser client at :3000->:8080/whip works.
2. Benchmark Go server resource on Pi (top, check ~200MB RAM target)
3. Test local-ai profile: Ollama qwen2.5:3b + VibeVoice sidecars (torch path on Ubuntu i3 = slow, maybe stick to Speaches pipeline initially — compare)
4. Build streamcore-cli and ingest 5 sample brain docs into Supabase or local pgvector, verify RAG context injection in LLM call (zero tool overhead)
5. Port wakeword: quick Python plugin openWakeWord score over mic before WHIP connect, or js-sdk audio worklet.
6. Port SpeakerID: re-use jarvis-v2 data/speakers/*.npy, plugin returns speaker label.
7. Define skills dir: plugins/skills/jarvis-british.md, plugins/skills/grace-playful.md + persona per speaker
8. Integrate Hermes API: plugin hermes-bridge JSON-RPC calls main server tools (calendar create, reminders list, etc.) — use existing MCP reyna2026! token?
9. Compare latency telemetry to OGL's telemetry — unify dashboard.

Links

• https://github.com/study8677/open-gpt-live — MIT, WS, adaptive VAD, Speaches+Ollama local
• https://github.com/streamcoreai/streamcore-server — Apache2, Go, WebRTC WHIP, Pion built-in TURN, barge-in, plugins TS/Python/JS+Go native, markdown skills, pgvector RAG, ESP32+SIP family
• Org: https://github.com/streamcoreai (examples, js-sdk, go-sdk, python-sdk, rust-sdk, plugin-sdk, esp32, sip-server)
• VibeVoice: https://huggingface.co/mlx-community/VibeVoice-ASR-4bit, https://huggingface.co/mlx-community/VibeVoice-Realtime-0.5B-6bit
• Related brain: Jarvis v2 - Always Listening Voice Assistant, PARA Brain MCP Server, Reyna CLI Browser Control, Grace Poppy Storybook Image Voice App

TODO

• Clone streamcore-server + examples locally on .110 and test Docker path with Deepgram key? Do we have keys stored?
• Check Postgres pgvector availability on FamReynaServer .110 (Instatic uses SQLite/MCP?) Could run docker pgvector.
• Document USB mic purchase recommendation again (iMac internal too quiet per Jarvis v2 test log peak 0.014 raw)
• Consider whether Hermes Agent itself should expose /whip WHIP endpoint via skill/plugin vs separate Go service.
• Evaluate LiveKit as third alternative? (mentioned by many but heavier)
• Add to brain index Projects list