Files
FamReynaBrain/projects/speech_swift_voice_agent_iteration.md
T
2026-10-01 23:00:12 -04:00

2.8 KiB

Speech Swift — voice-agent iteration candidate

Date captured: 2026-10-01 Status: Candidate to evaluate; not integrated or benchmarked. Tags: #voice-agent #macos #apple-silicon #stt #tts #speaker-id #project-idea Related: voice_agent_platform_research_2026, family_voice_context_and_speaker_id_2026, factory/kids-hermes-voice

Source

  • https://github.com/soniqo/speech-swift (user-shared URL, tracking parameter removed)
  • Documentation: https://soniqo.audio/
  • Repository license: Apache-2.0 (verify individual model/component licenses before deployment; some models listed in the README carry non-commercial/research restrictions).

Why explore it

A modular Swift Package for local Apple Silicon speech pipelines using MLX Swift and CoreML. The repository lists streaming ASR (Parakeet/Nemotron), Qwen3-ASR, Kokoro and Qwen3 TTS, VAD, wake-word spotting, Smart Turn, diarization, speaker embeddings, denoising, echo cancellation, and full-duplex speech-to-speech models. Its per-model SPM products may allow a selective native Mac mini voice-service iteration rather than replacing the existing voice gateway wholesale.

Fit to Reyna voice architecture

  • Evaluate a native Swift STT/turn-detection module against the existing Apple SpeechTranscriber and Pipecat VAD/SmartTurn path on the Mac mini; measure partial latency, end-of-turn accuracy, English/Spanish accuracy, memory, and CPU/GPU/ANE pressure.
  • Evaluate speaker embeddings and diarization as candidate complements to the family voice-identification plan; diarization (who spoke when) is not automatically identification of Adolfo/Alicia/Grace/Elías (who exactly), so require enrolled reference voices and threshold tests.
  • Consider selective Kokoro/Qwen3-TTS comparison against the existing warm Kokoro/Pocket/Qwen routes, preserving current voices, daemon and fallback behavior.
  • Keep Hermes brain/tool routing and device-facing ESP32 protocol boundaries intact. The Swift package is a Mac/iOS model toolkit, not a drop-in ESP32 voice client or complete agent orchestration platform.

Pilot gate (only if requested)

  1. Inspect current package manifest, model downloads/licenses, and API contracts for one narrow feature (e.g. streaming ASR or speaker embeddings).
  2. Build a small isolated Swift smoke test on the Mac mini; use a non-sensitive speech fixture and capture true partial/final timings. Compare with the existing Apple SpeechTranscriber baseline under equal audio conditions.
  3. If promising, wrap it as a local Mac service/CLI adapter to the existing voice gateway, preserving existing STT/TTS choices and permission boundaries. Verify full audio round-trip on a real device before claiming integration.

Verification scope: live GitHub metadata and README inspected 2026-10-01. No package build, model download, benchmark, or live voice-agent integration performed.