VoiceAgent/AGENTS.md

VoiceAgent Architecture & Developer Guide (AGENTS.md)

This document contains architectural guidelines, operational patterns, and learnings for AI agents and developers working on the VoiceAgent codebase.

---

1. Project Overview & Architecture

VoiceAgent is a high-performance, real-time voice conversation system built on Pipecat and optimized for macOS.

• Audio Pipeline:
  • Sample Rates: STT / VAD operates natively at 16 kHz; Kokoro TTS synthesizes at 24 kHz.
  • VAD / Push-to-Talk: Silero VAD (SileroVADAnalyzer) and Push-To-Talk (push_to_talk.py, global_hotkey.py).
  • STT: Apple SpeechAnalyzer (macOS 26+) or Whisper MLX / CPU (speech_analyzer_stt.py, apple_stt.py).
  • TTS: Kokoro Neural TTS (KokoroTTSService) or macOS System Speech (apple_tts.py).
• LLM Slot:
  • Primary Harness: Hermes (hermes_llm.py / HermesLLM).
  • Additional Engines: Claude Code (claude_llm.py), local macOS MLX (apple_llm.py).

---

2. Hermes Integration & Session Handling

• CLI Subcommand & Mode:
  • Uses hermes chat -q "<query>" -Q --source voice.
  • The -Q (quiet mode) flag suppresses decorative headers, spinners, and preview output so only cleaned text reaches TTS.
• Stderr Session Tracking:
  • Hermes outputs session headers (e.g. session_id: 20260809_...) on stderr.
  • _remember_session_id() in hermes_llm.py extracts the session ID from stderr and persists it.
• Per-Workspace Session Persistence:
  • Session state is saved per workspace in .hermes-voice-session.json (inside cwd), keeping project histories isolated.
  • On restart, HermesLLM loads the active session ID and resumes with hermes chat ... -r <session_id>.
• Display Renaming:
  • Automatically renames the active session to "Voice Agent" via hermes sessions rename <session_id> "Voice Agent" on the first turn.
• Native Persona & Memory:
  • Hermes manages persona, preferences, and long-term memory natively in ~/.hermes.
  • Avoid injecting redundant system prompt wrappers (Brain memories or AGENTS.md personality prompts) into Hermes turns.
• Model Selection:
  • Default model setting is "default", letting Hermes use its configured model in ~/.hermes/config.yaml. Avoid passing -m unless explicitly overriding the model.

---

3. App Directory Storage (No ~/Workspace Dependency)

The project operates entirely from the app directory without depending on ~/Workspace:

• Vocabulary & Repairs: vocabulary.txt and corrections.txt live directly in the app root folder.
• App Settings: model_settings.json, voice_settings.json, and journal.jsonl are saved in the app folder.
• CLI Helper Scripts:
  • bin/voice_tool.py: List and switch TTS voices (python bin/voice_tool.py list, python bin/voice_tool.py set <voice>).
  • bin/model_tool.py: List and switch LLM models (python bin/model_tool.py list, python bin/model_tool.py set <model>).
  • bin/session_tool.py: Inspect or reset active Hermes session (python bin/session_tool.py get, python bin/session_tool.py reset).
  • bin/profile_tool.py: List and switch Hermes agent profiles (python bin/profile_tool.py list, python bin/profile_tool.py set <profile>).
  • bin/web_tool.py: Inspect files or URLs in the Companion Web UI drawer (http://localhost:8888).

---

4. Building & Packaging /Applications/VoiceAgent.app

VoiceAgentLauncher runs Python scripts dynamically from the project workspace (configured in ~/.voiceagent.env or VOICEAGENT_DIR).

• Python code edits take effect immediately without recompiling or re-signing the macOS app bundle.
• Rebuilding binaries (bash build_app.sh) is only required when modifying Swift sources (swift/*.swift), helper binaries, or Info.plist entitlements.
• build_app.sh automatically updates ~/.voiceagent.env and skips binary re-signing if Swift sources are unchanged, preserving macOS privacy permissions (Microphone, Accessibility, Speech Recognition). Use bash build_app.sh -f to force a full rebuild.

bash build_app.sh

Build Workflow:

1. Configures ~/.voiceagent.env with VOICEAGENT_DIR and VOICEAGENT_PYTHON.
2. Checks if Swift binaries (speech-helper, llm-helper, VoiceAgentLauncher) need recompiling.
3. If up to date, copies Python resources without re-signing the app bundle (preserving macOS TCC permissions).
4. If modified, compiles Swift binaries, creates dist/VoiceAgent.app, signs, and installs to /Applications/VoiceAgent.app.

---

5. Verification & Testing

Before committing changes, execute the test suite:

.venv/bin/python3 test_profile_tool.py
.venv/bin/python3 test_session_tool.py
.venv/bin/python3 test_model_manager.py
.venv/bin/python3 test_spoken_text.py
.venv/bin/python3 test_journal.py
.venv/bin/python3 test_working_phrase.py