docs: add websocket audio streaming implementation plan

2026-06-18 23:40:30 -04:00
parent d778391c5c
commit c944d8c48f
1 changed files with 65 additions and 0 deletions
@@ -0,0 +1,65 @@
+# WebSocket Audio Streaming Implementation Plan
+
+Implement a WebSocket-based real-time voice integration between the ESP32 board and the Hermes Voice Gateway (Phase 1: WebSocket streaming with server-side batch fallback) to solve the RAM/Flash storage constraints and lower latency.
+
+## User Review Required
+
+> [!IMPORTANT]
+> - This plan adds a new route `/api/esp32/voice/ws` to the Hermes Gateway. The old POST route `/api/esp32/voice` will remain completely intact to act as a stable fallback.
+> - A lightweight WebSocket client `websocket_client.py` will be added to the MicroPython `lib/` directory since standard MicroPython doesn't bundle a WebSocket client by default.
+
+## Open Questions
+
+> [!NOTE]
+> None at the moment. The proposed wire protocol and incremental phased approach minimize risk.
+
+---
+
+## Proposed Changes
+
+### 1. Hermes Voice Gateway
+
+#### [MODIFY] [api_server_endpoint.py](file:///Users/adolforeyna/Projects/MicroPython/test1/Screen/hermes_voice_gateway/api_server_endpoint.py)
+* Add a `_handle_esp32_voice_websocket` handler method.
+* Listen for a JSON `start` frame to read parameters like `device_id`, `reply_mode`, and `screen_url`.
+* Accept incoming binary frames (raw PCM s16le mono 16 kHz chunks) and write them directly to a temporary WAV file on the host's disk.
+* Upon receiving the JSON `end` frame (or a disconnection/timeout):
+  * Close the WAV file writer.
+  * Invoke `transcribe_audio(wav_path)` using the local `faster-whisper` engine.
+  * Run the agent turn via `_run_esp32_voice_turn`.
+  * Synthesize TTS audio and send the resulting WAV file back as a binary frame to the client.
+* Update `register_esp32_voice_route` to bind the new WebSocket endpoint:
+  ```python
+  app.router.add_get("/api/esp32/voice/ws", api_server._handle_esp32_voice_websocket)
+  ```
+
+#### [MODIFY] [README.md](file:///Users/adolforeyna/Projects/MicroPython/test1/Screen/hermes_voice_gateway/README.md)
+* Update documentation to cover the new `/api/esp32/voice/ws` WebSocket client contract and protocol.
+
+---
+
+### 2. ESP32 MicroPython Firmware
+
+#### [NEW] [websocket_client.py](file:///Users/adolforeyna/Projects/MicroPython/test1/Screen/lib/websocket_client.py)
+* Implement a minimal, robust, memory-efficient WebSocket client helper class (`WebSocketClient`) in MicroPython.
+* Support HTTP handshake, sending/receiving text frames (opcode 1), and binary frames (opcode 2) with proper masking (client-to-server requirement).
+
+#### [NEW] [demo_websocket_voice.py](file:///Users/adolforeyna/Projects/MicroPython/test1/Screen/demo_websocket_voice.py)
+* Create a dedicated firmware script demonstrating how the board communicates with the new gateway:
+  * Connect to `ws://<ip>:8642/api/esp32/voice/ws`.
+  * Send the JSON start frame.
+  * Record continuously from I2S and stream 100ms chunks (3200 bytes mono) over the socket.
+  * Send the JSON end frame on screen release.
+  * Listen for the server's response WAV frame and play it using the ES8311 speaker codec.
+
+---
+
+## Verification Plan
+
+### Automated Tests
+* Run `python3 hermes_voice_gateway/smoke_test.py` to ensure local helper methods (normalization, MIME mapping) remain fully functional.
+
+### Manual Verification
+1. Run the updated Hermes server with the new WebSocket route registered.
+2. Run `demo_websocket_voice.py` on the ESP32 board.
+3. Verify that the ESP32 connects, streams audio chunks on-the-fly without running out of RAM (SRAM memory checks via `gc.mem_free()`), transcribes the speech, and plays back the agent's voice response.