VoiceAgent/ACCURACY.md

Transcription accuracy

What was measured, what shipped, and what turned out to be wrong along the way.

Benchmark throughout: eight sentences of technical speech (85 words) containing the vocabulary this project actually uses — pipecat, Kokoro, Metamate, Phabricator, fbsource, SFSpeechRecognizer. The audio is synthesised, so it is cleaner and more evenly paced than a person at a microphone. Trust the ordering, not the absolute rates, and prefer compare_engines.py on your own voice.

Scoring follows the Inscribe benchmark: lowercased, depunctuated, contractions expanded and digits written out. That last part matters — see the correction below.

Where it ended up

Configuration	WER
old SFSpeechRecognizer, no help	20.0%
SpeechTranscriber, no help	16.5%
old + vocabulary + repair rules	10.6%
DictationTranscriber + vocabulary + repair	9.4%
SpeechTranscriber + repair — the default	9.4%

The top two tie, but not at equal cost. SpeechTranscriber reaches 9.4% with five repair rules and no vocabulary, where the old model needs 39 curated terms and seven rules to match it. It is also better with no configuration at all, which is the state anyone starts in.

The two mistakes I made measuring this

Scoring punished the new model for formatting. My first normaliser lowercased and stripped punctuation but left digits alone, so SpeechTranscriber was charged errors for writing "400 milliseconds" instead of "four hundred milliseconds", and "SF speech recognizer" instead of "SFSpeechRecognizer". Neither is a mishearing. The Inscribe benchmark names this exact trap. Rescored properly, a 12.9%-vs-14.1% loss for the new model became a 9.4% tie that it wins on effort.

"Dilution is cheap" was wrong. Padding the term list with a thousand random dictionary words cost only 1.1 points, so auto-generating 100 terms from project filenames looked safe. It erased the entire benefit — back to no-biasing levels — because filenames like bot, plan and hack are ordinary English, and boosting ordinary words drags correct speech onto them. Obscure words are inert by comparison. Discovered terms are now filtered against the system dictionary, capped at 12, and ranked behind the hand-written list.

What each mechanism is worth

Vocabulary biasing (vocabulary.txt → contextualStrings) removed 30% of errors on the old model and costs nothing at runtime. SpeechTranscriber ignores it — output is byte-identical with and without terms — so on the default engine the file now serves only Claude's system prompt and the fallback engines. Keep it short and specific.

Repair rules (corrections.txt) are deterministic substitutions for mistakes that recur identically. Worth more on the new engine than the old, because its errors land phonetically close to the target — "Kakoro" for Kokoro, where the old model produced "coral". Five rules do what 39 terms plus seven rules do elsewhere.

LLM post-correction took 16.5% to 7.1% on the old engine, and 5-best beat 1-best (7.1% against 8.2%). Not worth a separate pass on the conversation path, since Claude already reads every transcript — the vocabulary goes into his system prompt instead, which is free. Worth a real pass for meeting notes.

Punctuation and task hint on the old API measurably do nothing for accuracy. The new engine punctuates and capitalises automatically.

Noise beats all of it. At 10 dB SNR, WER nearly tripled to 69% and neither vocabulary, high-pass filtering nor gain normalisation recovered any of it. The microphone is the highest-leverage component in the pipeline.

Which model is which

	old SFSpeechRecognizer	DictationTranscriber	SpeechTranscriber
Model	old	byte-identical to old	new
Honours vocabulary	yes	yes	no
Long audio	needs stitching	native	native
Punctuation	no	no	automatic
Best measured	10.6%	9.4%	9.4%, far less tuning

DictationTranscriber is not a different model: it produces character-for- character identical output to SFSpeechRecognizer on the same audio, verified sentence by sentence rather than inferred from the docs. It is the old recogniser reached through the new API, and its only advantage is vocabulary support.

Both new-API modules handle long speech natively — 40 of 40 sentences recovered from 83 seconds — so the transcript stitching in apple_stt.py is only needed on the legacy fallback path.

For scale, the published benchmark measured 5,559 LibriSpeech utterances on an M2 Pro: SpeechAnalyzer 2.12% clean and 4.56% noisy, against SFSpeechRecognizer's 9.02% and 16.25% — the legacy engine scoring worse than Whisper Tiny. It validates its harness by reproducing OpenAI's published Whisper numbers within +0.11 to +0.42. Caveats: English read speech, one machine.

Ruled out

Denoising. When De-noising Hurts (arXiv:2512.17562) applied MetricGAN+ across 500 recordings and 10 conditions: all 40 configurations got worse, Whisper degrading 8.82% → 25.83% at 10 dB SNR, and even clean audio losing 1.3–3.2 points. So Pipecat's RNNoiseFilter is not worth the dependency surgery it needs, and the commercial filters (Koala, Krisp, AIC) are the same bet with a licence. Worth checking macOS Voice Isolation is off, since it is the same class of processing applied for free.

Custom language models. SFSpeechLanguageModel trains on example sentences and supports X-SAMPA pronunciations, so it should beat contextualStrings — but its data builder is Swift-only, and it cannot be combined with SpeechTranscriber anyway.

Still open

• Real speech. Every number here is synthesised audio with config files tuned around it. compare_engines.py records you and runs every engine.
• The external microphone, versus the built-in one. Probably the largest real-world effect available and free to test.
• 5-best into Claude. --alternatives exists in the helper and is unused; measured 8.2% → 7.1% on the old engine.
• Confidence-gated clarification — ask again rather than answer a low-confidence guess.