Adolfo Reyna
90 lines
4.8 KiB
Markdown
90 lines
4.8 KiB
Markdown
# VoiceAgent Architecture & Developer Guide (AGENTS.md)
|
|
|
|
This document contains architectural guidelines, operational patterns, and learnings for AI agents and developers working on the `VoiceAgent` codebase.
|
|
|
|
---
|
|
|
|
## 1. Project Overview & Architecture
|
|
|
|
`VoiceAgent` is a high-performance, real-time voice conversation system built on Pipecat and optimized for macOS.
|
|
|
|
- **Audio Pipeline**:
|
|
- **Sample Rates**: STT / VAD operates natively at **16 kHz**; Kokoro TTS synthesizes at **24 kHz**.
|
|
- **VAD / Push-to-Talk**: Silero VAD (`SileroVADAnalyzer`) and Push-To-Talk (`push_to_talk.py`, `global_hotkey.py`).
|
|
- **STT**: Apple `SpeechAnalyzer` (macOS 26+) or Whisper MLX / CPU (`speech_analyzer_stt.py`, `apple_stt.py`).
|
|
- **TTS**: Kokoro Neural TTS (`KokoroTTSService`) or macOS System Speech (`apple_tts.py`).
|
|
- **LLM Slot**:
|
|
- Primary Harness: **Hermes** (`hermes_llm.py` / `HermesLLM`).
|
|
- Additional Engines: Claude Code (`claude_llm.py`), local macOS MLX (`apple_llm.py`).
|
|
|
|
---
|
|
|
|
## 2. Hermes Integration & Session Handling
|
|
|
|
- **CLI Subcommand & Mode**:
|
|
- Uses `hermes chat -q "<query>" -Q --source voice`.
|
|
- The `-Q` (quiet mode) flag suppresses decorative headers, spinners, and preview output so only cleaned text reaches TTS.
|
|
- **Stderr Session Tracking**:
|
|
- Hermes outputs session headers (e.g. `session_id: 20260809_...`) on **`stderr`**.
|
|
- `_remember_session_id()` in `hermes_llm.py` extracts the session ID from stderr and persists it.
|
|
- **Per-Workspace Session Persistence**:
|
|
- Session state is saved per workspace in `.hermes-voice-session.json` (inside `cwd`), keeping project histories isolated.
|
|
- On restart, `HermesLLM` loads the active session ID and resumes with `hermes chat ... -r <session_id>`.
|
|
- **Display Renaming**:
|
|
- Automatically renames the active session to `"Voice Agent"` via `hermes sessions rename <session_id> "Voice Agent"` on the first turn.
|
|
- **Native Persona & Memory**:
|
|
- Hermes manages persona, preferences, and long-term memory natively in `~/.hermes`.
|
|
- Avoid injecting redundant system prompt wrappers (`Brain` memories or `AGENTS.md` personality prompts) into Hermes turns.
|
|
- **Model Selection**:
|
|
- Default model setting is `"default"`, letting Hermes use its configured model in `~/.hermes/config.yaml`. Avoid passing `-m` unless explicitly overriding the model.
|
|
|
|
---
|
|
|
|
## 3. App Directory Storage (No `~/Workspace` Dependency)
|
|
|
|
The project operates entirely from the app directory without depending on `~/Workspace`:
|
|
|
|
- **Vocabulary & Repairs**: `vocabulary.txt` and `corrections.txt` live directly in the app root folder.
|
|
- **App Settings**: `model_settings.json`, `voice_settings.json`, and `journal.jsonl` are saved in the app folder.
|
|
- **CLI Helper Scripts**:
|
|
- `bin/voice_tool.py`: List and switch TTS voices (`python bin/voice_tool.py list`, `python bin/voice_tool.py set <voice>`).
|
|
- `bin/model_tool.py`: List and switch LLM models (`python bin/model_tool.py list`, `python bin/model_tool.py set <model>`).
|
|
- `bin/session_tool.py`: Inspect or reset active Hermes session (`python bin/session_tool.py get`, `python bin/session_tool.py reset`).
|
|
- `bin/profile_tool.py`: List and switch Hermes agent profiles (`python bin/profile_tool.py list`, `python bin/profile_tool.py set <profile>`).
|
|
- `bin/web_tool.py`: Inspect files or URLs in the Companion Web UI drawer (`http://localhost:8888`).
|
|
|
|
---
|
|
|
|
## 4. Building & Packaging `/Applications/VoiceAgent.app`
|
|
|
|
`VoiceAgentLauncher` runs Python scripts dynamically from the project workspace (configured in `~/.voiceagent.env` or `VOICEAGENT_DIR`).
|
|
|
|
- **Python code edits take effect immediately** without recompiling or re-signing the macOS app bundle.
|
|
- **Rebuilding binaries** (`bash build_app.sh`) is only required when modifying Swift sources (`swift/*.swift`), helper binaries, or `Info.plist` entitlements.
|
|
- `build_app.sh` automatically updates `~/.voiceagent.env` and skips binary re-signing if Swift sources are unchanged, preserving macOS privacy permissions (Microphone, Accessibility, Speech Recognition). Use `bash build_app.sh -f` to force a full rebuild.
|
|
|
|
```bash
|
|
bash build_app.sh
|
|
```
|
|
|
|
**Build Workflow**:
|
|
1. Configures `~/.voiceagent.env` with `VOICEAGENT_DIR` and `VOICEAGENT_PYTHON`.
|
|
2. Checks if Swift binaries (`speech-helper`, `llm-helper`, `VoiceAgentLauncher`) need recompiling.
|
|
3. If up to date, copies Python resources without re-signing the app bundle (preserving macOS TCC permissions).
|
|
4. If modified, compiles Swift binaries, creates `dist/VoiceAgent.app`, signs, and installs to `/Applications/VoiceAgent.app`.
|
|
|
|
---
|
|
|
|
## 5. Verification & Testing
|
|
|
|
Before committing changes, execute the test suite:
|
|
|
|
```bash
|
|
.venv/bin/python3 test_profile_tool.py
|
|
.venv/bin/python3 test_session_tool.py
|
|
.venv/bin/python3 test_model_manager.py
|
|
.venv/bin/python3 test_spoken_text.py
|
|
.venv/bin/python3 test_journal.py
|
|
.venv/bin/python3 test_working_phrase.py
|
|
```
|