hermes-esp32-voice-gateway/README.md

# Hermes ESP32 Voice Gateway

Durable custom voice gateway for Reyna family ESP32 + iPhone Watch devices. Survives `hermes update` (which wipes in-tree patches to `gateway/platforms/api_server.py`).

Upstream issue: ESP32 push-to-talk endpoints `POST /api/esp32/voice` and `WS /api/esp32/voice/ws` with:

- live draft ingestion via Mac mini `speech_live_transcribe` (ApplePipeTranscriber volatile drafts, ~800ms, 2.12% WER)
- instant smart reply via Mac mini `apple_llm_quick_reply` (Apple 3B ANE ~300ms contextual, not generic template)
- final transcription prefers `macmini-live-final` over local faster-whisper base (753ms vs 11-13s cold)
- parallel instant future started at STOP before final transcribe
- persistence flags `persistent=True`, `is_instant=True`, `keep_instant=True` for iPhone client sticky UI
- screen animation + TTS via `play_audio_base64` / `play_mp3_base64` to `http://<source_ip>/api/mcp` screens
- audit log `~/.hermes/.../audio_cache/esp32/voice_requests.jsonl`

## Why a separate repo + plugin

`hermes update` does `git stash --include-untracked` + pull origin/main, dropping local modifications to `api_server.py`. The plugin `esp32-voice-gateway` lives in `~/.hermes/plugins/` (user plugin dir, never overwritten), and on every Hermes start auto-checks whether the custom routes are present. If missing (after an update), it **auto-restores** them from the bundled patch and verifies.

### What's tracked here

- `patch/api_server_esp32_live_draft_with_llm.patch` — full 1675-line patch that is currently running on Adolfo's Pi (verified 2026-07-14)
- `hermes_plugin/` — Hermes user plugin (`plugin.yaml` + `__init__.py`) installable via `hermes plugins install`
- `scripts/restore_after_update.py` — manual restore + verification (400 not 404 check)
- `docs/` — original excerpts + audit references

## Quick start

### 1. Create repo on Gitea

The token in `~/.git-credentials` only has `write:repository`, not `write:user`, so repo creation via API is blocked. Create an empty repo via Gitea UI:

https://git.reynafamily.com/repo/create

- Name: `hermes-esp32-voice-gateway`
- Visibility: private (contains no secrets, but references family device IPs)
- Uncheck initialize README

Then:

```bash
cd ~/Projects/hermes-esp32-voice-gateway
git remote add origin https://adolforeyna:TOKEN@git.reynafamily.com/adolforeyna/hermes-esp32-voice-gateway.git
git push -u origin main
```

(TOKEN is the existing `~/.git-credentials` token — `git` credential helper already does this)

### 2. Install the durable plugin

```bash
# From this repo
cp -r hermes_plugin ~/.hermes/plugins/esp32-voice-gateway
# or symlink for dev
# ln -s ~/Projects/hermes-esp32-voice-gateway/hermes_plugin ~/.hermes/plugins/esp32-voice-gateway
hermes plugins enable esp32-voice-gateway
hermes gateway restart
```

### 3. Verify

```bash
# Routes should be 400 (not 404) when hit without WS upgrade:
curl -s -o /dev/null -w "%{http_code}" -H "Authorization: Bearer $API_SERVER_KEY" http://127.0.0.1:8642/api/esp32/voice/ws
# -> 400 (means route exists, missing Upgrade header) — NOT 404
# POST empty should be 400 not 404:
curl -s -o /dev/null -w "%{http_code}" -H "Authorization: Bearer $API_SERVER_KEY" http://127.0.0.1:8642/api/esp32/voice -X POST --data-binary @/dev/null
# -> 400
```

### 4. After `hermes update`

The plugin auto-restores on next import (gateway start). Check logs:

```bash
grep -i "esp32.*restore\|esp32.*patch\|ESP32" ~/.hermes/logs/gateway.log | tail -20
# If auto-restore happened, it logs patched file + needs restart:
systemctl --user status hermes-gateway-kids hermes-gateway --no-pager
# The plugin will request one restart cycle; if stuck in deactivating, run:
# pkill -9 -f "hermes_cli.main gateway run"; systemctl --user reset-failed hermes-gateway*; systemctl --user start hermes-gateway
```

Manual restore:

```bash
~/Projects/hermes-esp32-voice-gateway/scripts/restore_after_update.py
```

## Architecture (for reference)

- Incoming: iPhone 192.168.68.150:8080 `iphone6` (WAV POST) + ESP32 tablets 192.168.68.129 `grace-esp32`/`elias-esp32` (WS PCM s16le 16k mono)
- Gateway: `gateway/platforms/api_server.py` `APIServerAdapter`
  - `POST /api/esp32/voice` → saves WAV, transcribes (macmini-live-final preferred), runs Hermes agent turn, TTS → WAV (ffmpeg to 16k mono), `FileResponse`
  - `WS /api/esp32/voice/ws` → ingest s16le binary frames, every 800ms snapshot → `_s16le_to_wav_bytes` → `speech_live_transcribe` (Mac mini pipe volatile) fallback tiny → emit `draft`/`interim_transcript`, on STOP → parallel instant future `apple_llm_quick_reply` (must be >30 chars, else use final transcript for context), final transcribe Mac mini full buffer (11588ms fallback if Mac down), emit `instant_response persistent=True` + `response_text is_instant=True` + `done instant_ok` + `thinking keep_instant`, background ` _run_esp32_voice_background` (screen animation/text/image/audio via `http://<source_ip>/api/mcp`)
- MacMiniMCP: `~/Projects/MacMiniMCP` (canonical on Mac) 37 tools including `speech_live_transcribe`, `apple_llm_quick_reply`, `speech_synthesize_base64`, `voicebox_quick_reply` (Aiden boy)
- Pitfalls documented in `hermes-multi-profile-voice-gateways` skill references: `esp32-voice-500-import-loop.md`, `esp32-voice-gateway-update-survival.md`, etc.

## Contributing

This repo is Reyna family private. Changes to the voice gateway should update both:

1. `patch/api_server_esp32_live_draft_with_llm.patch` (regenerated from `git diff` of `~/.hermes/hermes-agent/gateway/platforms/api_server.py`)
2. The plugin's restore logic (it just reapplies patch, so keeping patch current is enough)

Regenerate patch:

```bash
cd ~/.hermes/hermes-agent
git diff gateway/platforms/api_server.py > ~/Projects/hermes-esp32-voice-gateway/patch/api_server_esp32_live_draft_with_llm.patch
cp ~/.hermes/esp32_live_draft_with_llm_patch.patch ~/Projects/hermes-esp32-voice-gateway/patch/ 2>/dev/null || true
```

## TODO

- [ ] Extract to true standalone plugin that injects routes at runtime without touching file (eliminate patch reapply entirely). Requires hooking `APIServerAdapter.connect` — current plugin does auto-restore via file patch for reliability.
- [ ] Add integration test that curls both endpoints after restore.

## License

Private - Reyna family.