VoiceAgent/AGENTS.md

VoiceAgent Architecture & Developer Guide (AGENTS.md)

This document contains architectural guidelines, operational patterns, and learnings for AI agents and developers working on the VoiceAgent codebase.

---

1. Project Overview & Architecture

VoiceAgent is a high-performance, real-time voice conversation system built on Pipecat and optimized for macOS.

• Audio Pipeline:
  • Sample Rates: STT / VAD operates natively at 16 kHz; Kokoro TTS synthesizes at 24 kHz.
  • VAD / Push-to-Talk: Silero VAD (SileroVADAnalyzer) and Push-To-Talk (push_to_talk.py, global_hotkey.py).
  • STT: Apple SpeechAnalyzer (macOS 26+) or Whisper MLX / CPU (speech_analyzer_stt.py, apple_stt.py).
  • TTS: Kokoro Neural TTS (KokoroTTSService) or macOS System Speech (apple_tts.py).
• LLM Slot:
  • Primary Harness: Hermes (hermes_llm.py / HermesLLM).
  • Additional Engines: Claude Code (claude_llm.py), local macOS MLX (apple_llm.py).

---

2. Hermes Integration & Session Handling

• CLI Subcommand & Mode:
  • Uses hermes chat -q "<query>" -Q --source voice.
  • The -Q (quiet mode) flag suppresses decorative headers, spinners, and preview output so only cleaned text reaches TTS.
• Stderr Session Tracking:
  • Hermes outputs session headers (e.g. session_id: 20260809_...) on stderr.
  • _remember_session_id() in hermes_llm.py extracts the session ID from stderr and persists it.
• Per-Workspace Session Persistence:
  • Session state is saved per workspace in .hermes-voice-session.json (inside cwd), keeping project histories isolated.
  • On restart, HermesLLM loads the active session ID and resumes with hermes chat ... -r <session_id>.
• Display Renaming:
  • Automatically renames the active session to "Voice Agent" via hermes sessions rename <session_id> "Voice Agent" on the first turn.
• Native Persona & Memory:
  • Hermes manages persona, preferences, and long-term memory natively in ~/.hermes.
  • Avoid injecting redundant system prompt wrappers (Brain memories or AGENTS.md personality prompts) into Hermes turns.
• Model Selection:
  • Default model setting is "default", letting Hermes use its configured model in ~/.hermes/config.yaml. Avoid passing -m unless explicitly overriding the model.

---

3. App Directory Storage (No ~/Workspace Dependency)

The project operates entirely from the app directory without depending on ~/Workspace:

• Vocabulary & Repairs: vocabulary.txt and corrections.txt live directly in the app root folder.
• App Settings: model_settings.json, voice_settings.json, and journal.jsonl are saved in the app folder.
• CLI Helper Scripts:
  • bin/voice_tool.py: List and switch TTS voices (python bin/voice_tool.py list, python bin/voice_tool.py set <voice>).
  • bin/model_tool.py: List and switch LLM models (python bin/model_tool.py list, python bin/model_tool.py set <model>).
  • bin/web_tool.py: Inspect files or URLs in the Companion Web UI drawer (http://localhost:8888).

---

4. Building & Packaging /Applications/VoiceAgent.app

Whenever python or Swift sources are modified, update the standalone macOS application bundle:

bash build_app.sh

Build Workflow:

1. Builds Swift binaries (swift/build.sh -> speech-helper, llm-helper, VoiceAgentLauncher).
2. Bundles Python source files, bin/ tools, and assets into dist/VoiceAgent.app.
3. Signs the app bundle (codesign -s - --deep --force).
4. Replaces /Applications/VoiceAgent.app.

---

5. Verification & Testing

Before committing changes, execute the test suite:

.venv/bin/python3 test_model_manager.py
.venv/bin/python3 test_spoken_text.py
.venv/bin/python3 test_journal.py
.venv/bin/python3 test_working_phrase.py