Hermes ESP32 Voice Gateway

Durable custom voice gateway for Reyna family ESP32 + iPhone Watch devices. Survives hermes update (which wipes in-tree patches to gateway/platforms/api_server.py).

Upstream issue: ESP32 push-to-talk endpoints POST /api/esp32/voice and WS /api/esp32/voice/ws with:

• live draft ingestion via Mac mini speech_live_transcribe (ApplePipeTranscriber volatile drafts, ~800ms, 2.12% WER)
• instant smart reply via Mac mini apple_llm_quick_reply (Apple 3B ANE ~300ms contextual, not generic template)
• final transcription prefers macmini-live-final over local faster-whisper base (753ms vs 11-13s cold)
• parallel instant future started at STOP before final transcribe
• persistence flags persistent=True, is_instant=True, keep_instant=True for iPhone client sticky UI
• screen animation + TTS via play_audio_base64 / play_mp3_base64 to http://<source_ip>/api/mcp screens
• audit log ~/.hermes/.../audio_cache/esp32/voice_requests.jsonl

Why a separate repo + plugin

hermes update does git stash --include-untracked + pull origin/main, dropping local modifications to api_server.py. The plugin esp32-voice-gateway lives in ~/.hermes/plugins/ (user plugin dir, never overwritten), and on every Hermes start auto-checks whether the custom routes are present. If missing (after an update), it auto-restores them from the bundled patch and verifies.

What's tracked here

• patch/api_server_esp32_live_draft_with_llm.patch — full 1675-line patch that is currently running on Adolfo's Pi (verified 2026-07-14)
• hermes_plugin/ — Hermes user plugin (plugin.yaml + __init__.py) installable via hermes plugins install
• scripts/restore_after_update.py — manual restore + verification (400 not 404 check)
• docs/ — original excerpts + audit references

Quick start

1. Create repo on Gitea

The token in ~/.git-credentials only has write:repository, not write:user, so repo creation via API is blocked. Create an empty repo via Gitea UI:

https://git.reynafamily.com/repo/create

• Name: hermes-esp32-voice-gateway
• Visibility: private (contains no secrets, but references family device IPs)
• Uncheck initialize README

Then:

cd ~/Projects/hermes-esp32-voice-gateway
git remote add origin https://adolforeyna:TOKEN@git.reynafamily.com/adolforeyna/hermes-esp32-voice-gateway.git
git push -u origin main

(TOKEN is the existing ~/.git-credentials token — git credential helper already does this)

2. Install the durable plugin

# From this repo
cp -r hermes_plugin ~/.hermes/plugins/esp32-voice-gateway
# or symlink for dev
# ln -s ~/Projects/hermes-esp32-voice-gateway/hermes_plugin ~/.hermes/plugins/esp32-voice-gateway
hermes plugins enable esp32-voice-gateway
hermes gateway restart

3. Verify

# Routes should be 400 (not 404) when hit without WS upgrade:
curl -s -o /dev/null -w "%{http_code}" -H "Authorization: Bearer $API_SERVER_KEY" http://127.0.0.1:8642/api/esp32/voice/ws
# -> 400 (means route exists, missing Upgrade header) — NOT 404
# POST empty should be 400 not 404:
curl -s -o /dev/null -w "%{http_code}" -H "Authorization: Bearer $API_SERVER_KEY" http://127.0.0.1:8642/api/esp32/voice -X POST --data-binary @/dev/null
# -> 400

4. After hermes update

The plugin auto-restores on next import (gateway start). Check logs:

grep -i "esp32.*restore\|esp32.*patch\|ESP32" ~/.hermes/logs/gateway.log | tail -20
# If auto-restore happened, it logs patched file + needs restart:
systemctl --user status hermes-gateway-kids hermes-gateway --no-pager
# The plugin will request one restart cycle; if stuck in deactivating, run:
# pkill -9 -f "hermes_cli.main gateway run"; systemctl --user reset-failed hermes-gateway*; systemctl --user start hermes-gateway

Manual restore:

~/Projects/hermes-esp32-voice-gateway/scripts/restore_after_update.py

Architecture (for reference)

• Incoming: iPhone 192.168.68.150:8080 iphone6 (WAV POST) + ESP32 tablets 192.168.68.129 grace-esp32/elias-esp32 (WS PCM s16le 16k mono)
• Gateway: gateway/platforms/api_server.py APIServerAdapter
  • POST /api/esp32/voice → saves WAV, transcribes (macmini-live-final preferred), runs Hermes agent turn, TTS → WAV (ffmpeg to 16k mono), FileResponse
  • WS /api/esp32/voice/ws → ingest s16le binary frames, every 800ms snapshot → _s16le_to_wav_bytes → speech_live_transcribe (Mac mini pipe volatile) fallback tiny → emit draft/interim_transcript, on STOP → parallel instant future apple_llm_quick_reply (must be >30 chars, else use final transcript for context), final transcribe Mac mini full buffer (11588ms fallback if Mac down), emit instant_response persistent=True + response_text is_instant=True + done instant_ok + thinking keep_instant, background _run_esp32_voice_background (screen animation/text/image/audio via http://<source_ip>/api/mcp)
• MacMiniMCP: ~/Projects/MacMiniMCP (canonical on Mac) 37 tools including speech_live_transcribe, apple_llm_quick_reply, speech_synthesize_base64, voicebox_quick_reply (Aiden boy)
• Pitfalls documented in hermes-multi-profile-voice-gateways skill references: esp32-voice-500-import-loop.md, esp32-voice-gateway-update-survival.md, etc.

Contributing

This repo is Reyna family private. Changes to the voice gateway should update both:

1. patch/api_server_esp32_live_draft_with_llm.patch (regenerated from git diff of ~/.hermes/hermes-agent/gateway/platforms/api_server.py)
2. The plugin's restore logic (it just reapplies patch, so keeping patch current is enough)

Regenerate patch:

cd ~/.hermes/hermes-agent
git diff gateway/platforms/api_server.py > ~/Projects/hermes-esp32-voice-gateway/patch/api_server_esp32_live_draft_with_llm.patch
cp ~/.hermes/esp32_live_draft_with_llm_patch.patch ~/Projects/hermes-esp32-voice-gateway/patch/ 2>/dev/null || true

TODO

• Extract to true standalone plugin that injects routes at runtime without touching file (eliminate patch reapply entirely). Requires hooking APIServerAdapter.connect — current plugin does auto-restore via file patch for reliability.
• Add integration test that curls both endpoints after restore.

License

Private - Reyna family.