sync brain updates 2026-08-08
This commit is contained in:
Adolfo Reyna
@@ -0,0 +1,235 @@
|
||||
# Pipecat Lag + STT Accuracy — Tuning Plan (Mac mini M4)
|
||||
|
||||
> **For Hermes:** Use `subagent-driven-development` to implement task-by-task. Verify against real trace.
|
||||
|
||||
**Goal:** Cut end-to-end voice turn from ~7–9s to <2.5s for Grace (child voice) and fix Whisper `base` mis-transcriptions, while keeping the stack local (no new cloud bill) with an optional Apple STT fast-path.
|
||||
|
||||
**Current trace (from `proc_221ead32955d`, session 10:01–10:04, Grace):**
|
||||
- `greeting -> speaking: 0.37s` OK. Kokoro synthesize `0.235s` — not the bottleneck.
|
||||
- **Per turn (e.g. "Teach it to be kind"):**
|
||||
- VAD `stop_secs 0.65s` + Silero confidence `0.62` — OK
|
||||
- `Whisper base` `processing 0.21–0.29s`, **TTFB 0.78–0.94s** (speech-end → text) — high for M4
|
||||
- **SmartTurn V3 hold: +2.24–2.40s** `_idle_watcher: Disabling receiver after 2.3s idle / Discarding old frames` + `stop_ttfb 0.78s` — dominates lag. Second +2–3s `append_audio: End of Turn complete due to stop_secs. Silence in ms: 3000.0` on short turns ("Yes.", "Keep going.") — SmartTurn is waiting 3s extra silence before firing `UserTurnStop`.
|
||||
- `LLM muse-spark-1.2 via 127.0.0.1:8642`: `0.93–6.1s` (median ~3.8s). Grows with context — context had duplicate greeting, 6 turns, 20k tokens prompt.
|
||||
- TTS chunking OK but sequential per sentence, no streaming.
|
||||
- **STT accuracy (Whisper base):**
|
||||
- Good: `and my daughter is here...`, `My name is Grace, nice to meet you too.` ✓
|
||||
- Bad child voice: `I like good stuff` (likely "I like princess stuff"?), `Princess, tough.` (fragmented), `go on and do it, inventors, together.` (should be "go on an adventure together" — the LLM had to guess), `subscribe and watch again Enjoy.` (nonsense split, 2 turns merged as one with 4s gap).
|
||||
- `RNNoise 16000↔48000 resampling`, `TurnAnalyzer VAD stop_secs differs from recommended 0.2s` warning, duplicate greeting in context.
|
||||
|
||||
**Architecture:** Keep Pipecat pipeline. Fix in layers: transport/VAD/SmartTurn → STT model/prompt → LLM context/streaming → TTS streaming → observability. All files under `~/Projects/pipecat-voice-gateway/server/`.
|
||||
|
||||
**Tech Stack:** Pipecat 1.7, faster-whisper 1.2.1, `WhisperSTTService` / `WhisperMLXSTTService`, `SileroVADAnalyzer`, `LocalSmartTurnAnalyzerV3`, `OpenAILLMService` @ `:8642`, Kokoro `:7332` warm + MLX.
|
||||
|
||||
---
|
||||
|
||||
## Task 1 — Baseline observability (metrics you can read)
|
||||
|
||||
**Objective:** Make lag measurable without grepping DEBUG.
|
||||
|
||||
**Files:**
|
||||
- Modify: `server/bot.py:1-40` — logging + PipelineParams
|
||||
- Create: `server/metrics.py` (optional, tiny helper)
|
||||
|
||||
**Step 1: Enable structured per-turn timing**
|
||||
In `run_bot()` after `PipelineParams`, ensure `enable_metrics=True, enable_usage_metrics=True` already set (is). Add a `MetricsLogger` or `on_metrics` handler:
|
||||
```python
|
||||
from pipecat.processors.metrics import MetricsProcessor # if exists, else use logger filter
|
||||
# Add after worker creation:
|
||||
@worker.event_handler("on_metrics")
|
||||
async def on_metrics(metrics): # check Pipecat API: worker.on_metrics or PipelineWorker metrics callback
|
||||
logger.info(f"[metrics] {metrics}")
|
||||
```
|
||||
If API differs, at minimum lower VAD/STT/TTS `logger` to INFO and add a one-line turn summary in `on_client_ready` helper that computes `speech_stopped -> transcript -> llm_first_token -> tts_first_audio`.
|
||||
|
||||
**Step 2: Persist logs sanely**
|
||||
The current bg proc writes to stdout only; `tee /tmp/pipecat-7860.log` mask from earlier run hid logs. Ensure `server/bot.py` launch (both manual and `launchctl`) writes to `StandardOutPath /tmp/pipecat-7860.log` and also `logger.add("/tmp/pipecat-7860.log")` if needed.
|
||||
|
||||
**Step 3: Verify**
|
||||
Run: `tail -f /tmp/pipecat-7860.log | grep -E "TTFB|processing time|EndOfTurn|prompt tokens|kokoro"` shows per-turn numbers. Same via `curl` to public URL.
|
||||
|
||||
**Commit:** `feat: per-turn metrics logging`
|
||||
|
||||
---
|
||||
|
||||
## Task 2 — SmartTurn / VAD tuning (biggest win: −2–4s)
|
||||
|
||||
**Objective:** Remove the `2.3s idle + 3s silence` hold.
|
||||
|
||||
**Files:**
|
||||
- Modify: `server/bot.py:145-180` (`VADProcessor`, `LocalSmartTurnAnalyzerV3`, `LLMUserAggregatorParams`)
|
||||
|
||||
**Step 1: Read current Pipecat docs for SmartTurn config**
|
||||
Search in venv:
|
||||
```bash
|
||||
grep -rn "stop_secs\|idle_watcher\|LLMUserAggregatorParams\|TurnAnalyzer" .venv/lib/python*/site-packages/pipecat --include="*.py" | head -n 80
|
||||
```
|
||||
Note: `SileroVADAnalyzer(params=VADParams(confidence=0.62, start_secs=0.12, stop_secs=0.65))` vs the warning `recommended 0.2s`. And `LLMUserAggregatorParams(vad_analyzer=SileroVADAnalyzer(... stop_secs=0.8))` is duplicated.
|
||||
|
||||
**Step 2: Reduce hold**
|
||||
- Set **VAD `stop_secs 0.35–0.45s`** (child speech has longer pauses but 0.65 is too slow):
|
||||
```python
|
||||
vad = VADProcessor(
|
||||
vad_analyzer=SileroVADAnalyzer(params=VADParams(confidence=0.58, start_secs=0.10, stop_secs=0.40)),
|
||||
audio_idle_timeout=1.0) # was 1.5
|
||||
```
|
||||
- If using `LocalSmartTurnAnalyzerV3`, pass `stop_secs` consistently — do **not** create two different Silero instances with 0.65 and 0.8. Use one set of params shared with the aggregator, or disable SmartTurn for the fast path:
|
||||
```python
|
||||
# Option A: disable SmartTurn, rely on VAD only for snappy turn
|
||||
smart_turn = None
|
||||
# Option B: keep it but lower thresholds
|
||||
smart_turn = LocalSmartTurnAnalyzerV3(stop_secs=0.40) # check constructor — if no arg, wrap via LLMUserAggregatorParams(smart_turn_params=...)
|
||||
```
|
||||
- Set `ttfs_p99_latency` on STT to match measured value so Pipecat doesn't pad:
|
||||
```python
|
||||
stt = WhisperSTTService(settings=WhisperSTTService.Settings(model="base", language="en"), ttfs_p99_latency=0.45)
|
||||
```
|
||||
(measure after tuning; target 0.4–0.6s).
|
||||
|
||||
**Step 3: Fix duplicate context turn**
|
||||
In `bot.py:192-194` both `user_aggregator` and `assistant_aggregator` may double-add greeting. The trace shows assistant message duplicated twice. Deduplicate:
|
||||
```python
|
||||
if llm and context is not None:
|
||||
greeting = "Hey — Hermes on your Mac mini..."
|
||||
context.add_message({"role": "assistant", "content": greeting}) # once
|
||||
await worker.queue_frame(TTSSpeakFrame(text=greeting))
|
||||
```
|
||||
|
||||
**Step 4: Verify**
|
||||
Run: speak "Yes." — should trigger LLM within ~0.6s of VAD stop, not 2.8s. Log should show no `Disabling receiver after 2.3s idle` before turn.
|
||||
|
||||
---
|
||||
|
||||
## Task 3 — STT accuracy: Whisper base → small / MLX + prompting
|
||||
|
||||
**Objective:** Fix child-voice errors (`I like good stuff`, `go on and do it, inventors`).
|
||||
|
||||
**Files:**
|
||||
- Modify: `server/bot.py:147` (STT instantiation)
|
||||
- Modify: `server/kokoro_bridge.py` / `server/apple_stt_bridge.py` (optional Apple path)
|
||||
|
||||
**Why base is suboptimal:** Pipecat `Model.BASE` = 74M params, trained on adult speech; WER higher on high-pitched child voice, especially at `base` + CPU int8. The gateway already installs `pipecat-ai[mlx-whisper]`.
|
||||
|
||||
**Step 1: Try Whisper `small` (244M, ~3× base) — easiest drop-in**
|
||||
```python
|
||||
# Cost: ~0.35s vs 0.25s on M4, but accuracy jump for child voice
|
||||
stt = WhisperSTTService(settings=WhisperSTTService.Settings(model="small", language="en"), ttfs_p99_latency=0.55)
|
||||
```
|
||||
Benchmark: `faster_whisper.WhisperModel("small", device="cpu", compute_type="int8")` or `float16` if available — test offline on a recorded Grace wav before wiring.
|
||||
|
||||
**Step 2 (preferred on M4): Whisper MLX `small` or `large-v3-turbo`**
|
||||
Apple Silicon MLX is faster than faster-whisper CPU:
|
||||
```python
|
||||
from pipecat.services.whisper.stt import WhisperSTTServiceMLX
|
||||
stt = WhisperSTTServiceMLX(settings=WhisperSTTServiceMLX.Settings(model="small", language="en"))
|
||||
# or: model="mlx-community/whisper-large-v3-turbo" # need to check Model enum supports it
|
||||
```
|
||||
Quantized variant `large-v3-turbo-q4` may trade tiny WER for speed — test.
|
||||
|
||||
**Step 3: Prompt/initial_prompt for domain**
|
||||
Whisper respects an initial prompt. Pass kid-domain hints:
|
||||
```python
|
||||
stt = WhisperMLXSTTServiceMLX(settings=..., extra={"prompt": "Grace, princess, inventors, adventure, dragon, castle"})
|
||||
# Check BaseWhisperSTTSettings prompt field vs STTSettings extra
|
||||
```
|
||||
For faster-whisper, verify `prompt` actually forwards to model — read `base_stt.py:148` and `stt.py` `transcribe` call.
|
||||
|
||||
**Step 4: Compare to Apple on-device**
|
||||
`server/apple_stt_bridge.py` already wraps `speech_transcribe_file` (SpeechAnalyzer, 6.3× realtime, 2.12% WER) — test it:
|
||||
```bash
|
||||
# Record 5s Grace utterance to /tmp/grace.wav, then:
|
||||
python -c "from apple_stt_bridge import transcribe_apple_stt; print(transcribe_apple_stt('/tmp/grace.wav', locale='en-US'))"
|
||||
```
|
||||
If WER < faster-whisper small, implement a hybrid: try Apple first (single MCP roundtrip ~120ms + transcribe), fall back to MLX Whisper. Requires a custom `SegmentedSTTService` subclass that buffers VAD segment to temp file then calls `transcribe_apple_stt` — latency ~0.6–1.0s but accuracy wins for kids.
|
||||
|
||||
**Step 5: Filter non-speech**
|
||||
Set `no_speech_prob 0.45` or `push_empty_transcripts=False` already — but check `WhisperSTTSettings(no_speech_prob=...)` to suppress the `subscribe and watch again` hallucination on near-silence. Lower `min_volume 0.55` in `VADParams`.
|
||||
|
||||
**Verify:**
|
||||
Record 5 utterances from Grace, compare transcriptions across `base` / `small` / `MLX small` / `Apple`. Pick lowest WER; log latency per model.
|
||||
|
||||
---
|
||||
|
||||
## Task 4 — LLM latency: context pruning + streaming TTS
|
||||
|
||||
**Objective:** Cut `0.9–6.1s` LLM tail.
|
||||
|
||||
**Files:**
|
||||
- Modify: `server/bot.py:166-180, 198-210` (LLM + context + TTS)
|
||||
|
||||
**Step 1: Prune context**
|
||||
Current context grew to `prompt 20k → 20.7k tokens` over 6 turns (includes full `brain` hint). Symptoms: 6s on "subscribe and watch again". Fix:
|
||||
- Set `LLMContext` max turns (e.g. keep last 8 messages + system, drop older):
|
||||
```python
|
||||
context = LLMContext(messages=[{"role":"system","content": SYSTEM_PROMPT}])
|
||||
# On each turn, after add_message, trim if len(context.messages) > 10
|
||||
```
|
||||
- Or use `LLMContextAggregatorPair` with `context_aggregator` limits — check Pipecat LLMContext API for `max_messages`.
|
||||
|
||||
**Step 2: Stream LLM → TTS (don't wait for full completion)**
|
||||
Current: `OpenAILLMService` generates full completion (`processing 4.7s`) then `_push_tts_frames` synthesizes all sentences sequentially. Switch to incremental:
|
||||
- Ensure `OpenAILLMService` streams deltas (`LLMTextFrame` per token). Then `WarmKokoroTTSService` should use `TTSService` sentence-based chunking (already via `push_frame` in Pipecat 1.7 — verify `WarmKokoroTTSService.run_tts` yields per sentence, not per full response). If not streaming, wrap as:
|
||||
```python
|
||||
# In WarmKokoroTTSService: split on sentence boundaries inside run_tts, yield TTSStartedFrame per sentence
|
||||
```
|
||||
- Enable `PipelineParams(enable_metrics=True)` already does streaming; confirm `on_client_ready` doesn't block.
|
||||
|
||||
**Step 3: Hermes gateway model routing**
|
||||
`muse-spark-1.2` via `:8642` is correct; check if a faster local model (e.g. qwen2.5:3b via Ollama `:11434`) could be used for kid chat with sub-500ms first token — compare `curl http://127.0.0.1:8642/v1/models` vs Ollama latency, but keep Hermes for tool use.
|
||||
|
||||
**Step 4: Deduplicate greeting token cost**
|
||||
Fix Task 2 duplicate greeting — saves ~300 tokens/turn.
|
||||
|
||||
**Verify:**
|
||||
`processing time` drops from 5.3s to <1.5s for short turns; `TTS first audio` appears within 200ms of `LLM TTFB`.
|
||||
|
||||
---
|
||||
|
||||
## Task 5 — Audio input hygiene (RNNoise + transport)
|
||||
|
||||
**Objective:** Stop resampling thrash and timeouts.
|
||||
|
||||
**Files:**
|
||||
- Modify: `server/bot.py:230-245` (`_make_rnnoise`, `TransportParams`)
|
||||
|
||||
**Step 1: Lock sample rate**
|
||||
Log shows `RNNoise enabling resampling: 16000 <-> 48000`. SmallWebRTC negotiates 16k or 48k depending on browser. Set transport explicitly:
|
||||
```python
|
||||
TransportParams(audio_in_enabled=True, audio_out_enabled=True, audio_in_sample_rate=16000, audio_out_sample_rate=24000)
|
||||
```
|
||||
So RNNoise knows input rate up front and avoids SOXR per-frame resample jitter (adds ~20ms).
|
||||
|
||||
**Step 2: Confirm RNNoise not double-instantiated**
|
||||
`_make_rnnoise()` is called per-transport lambda — good. Don't also create in `run_bot`.
|
||||
|
||||
**Step 3: Suppress spurious timeout spam**
|
||||
`Timeout: No audio frame received within the specified time.` every 2–3s is normal idle but clutters logs. Lower log level for `smallwebrtc.transport` to INFO, or increase `audio_idle_timeout` handling in Pipecat — just filter it from metrics.
|
||||
|
||||
---
|
||||
|
||||
## Task 6 — E2E validation script (repeatable)
|
||||
|
||||
**Files:**
|
||||
- Create: `server/scripts/bench_stt.py` — records 5s, runs `base` vs `small` vs `MLX` vs `Apple`, prints WER + latency
|
||||
- Create: `server/scripts/bench_turn.py` — drives a WebRTC client and measures `VAD stop -> LLM first token -> TTS first audio`
|
||||
|
||||
**Run:**
|
||||
```bash
|
||||
cd ~/Projects/pipecat-voice-gateway/server
|
||||
uv run python scripts/bench_stt.py --wav /tmp/grace.wav
|
||||
uv run python scripts/bench_turn.py --host https://voice.reynafamily.com
|
||||
# Targets: VAD->STT <0.6s, STT->LLM <0.4s, LLM->TTS <0.3s, E2E <2.0s p50, WER <5% on kid voice
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Risks
|
||||
|
||||
- `WhisperMLX small` model download ~500MB via HF — cache on first run, may need `HF_TOKEN`.
|
||||
- Apple STT via MCP is **outside** Pipecat's segmented pipeline — adds file I/O + HTTP hop; if used, must be benchmarked against local MLX for p99.
|
||||
- Cutting `VAD stop_secs` too low (0.2s) splits Grace's pauses into multiple turns — test child-specific 0.35–0.45s sweet spot.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Is `muse-spark-1.2` the right LLM for kid chat, or should we route kid sessions to a faster 3B with constrained toolset (`voice` profile already is)?
|
||||
- Should kid STT get a separate model (MLX `small`) vs adult default (`base`)? Could switch via diarizer speaker hint.
|
||||
Reference in New Issue
Block a user