Files
FamReynaBrain/projects/speech_swift_voice_agent_iteration.md
T
2026-10-01 23:00:12 -04:00

28 lines
2.8 KiB
Markdown

# Speech Swift — voice-agent iteration candidate
Date captured: 2026-10-01
Status: Candidate to evaluate; not integrated or benchmarked.
Tags: #voice-agent #macos #apple-silicon #stt #tts #speaker-id #project-idea
Related: [[voice_agent_platform_research_2026]], [[family_voice_context_and_speaker_id_2026]], [[factory/kids-hermes-voice]]
## Source
- https://github.com/soniqo/speech-swift (user-shared URL, tracking parameter removed)
- Documentation: https://soniqo.audio/
- Repository license: Apache-2.0 (verify individual model/component licenses before deployment; some models listed in the README carry non-commercial/research restrictions).
## Why explore it
A modular Swift Package for local Apple Silicon speech pipelines using MLX Swift and CoreML. The repository lists streaming ASR (Parakeet/Nemotron), Qwen3-ASR, Kokoro and Qwen3 TTS, VAD, wake-word spotting, Smart Turn, diarization, speaker embeddings, denoising, echo cancellation, and full-duplex speech-to-speech models. Its per-model SPM products may allow a *selective* native Mac mini voice-service iteration rather than replacing the existing voice gateway wholesale.
## Fit to Reyna voice architecture
- Evaluate a native Swift STT/turn-detection module against the existing Apple SpeechTranscriber and Pipecat VAD/SmartTurn path on the Mac mini; measure partial latency, end-of-turn accuracy, English/Spanish accuracy, memory, and CPU/GPU/ANE pressure.
- Evaluate speaker embeddings and diarization as candidate complements to the family voice-identification plan; diarization (who spoke when) is not automatically identification of Adolfo/Alicia/Grace/Elías (who exactly), so require enrolled reference voices and threshold tests.
- Consider selective Kokoro/Qwen3-TTS comparison against the existing warm Kokoro/Pocket/Qwen routes, preserving current voices, daemon and fallback behavior.
- Keep Hermes brain/tool routing and device-facing ESP32 protocol boundaries intact. The Swift package is a Mac/iOS model toolkit, not a drop-in ESP32 voice client or complete agent orchestration platform.
## Pilot gate (only if requested)
1. Inspect current package manifest, model downloads/licenses, and API contracts for one narrow feature (e.g. streaming ASR or speaker embeddings).
2. Build a small isolated Swift smoke test on the Mac mini; use a non-sensitive speech fixture and capture true partial/final timings. Compare with the existing Apple SpeechTranscriber baseline under equal audio conditions.
3. If promising, wrap it as a local Mac service/CLI adapter to the existing voice gateway, preserving existing STT/TTS choices and permission boundaries. Verify full audio round-trip on a real device before claiming integration.
Verification scope: live GitHub metadata and README inspected 2026-10-01. No package build, model download, benchmark, or live voice-agent integration performed.