diff --git a/websocket_implementation_plan.md b/websocket_implementation_plan.md new file mode 100644 index 0000000..bbd29bd --- /dev/null +++ b/websocket_implementation_plan.md @@ -0,0 +1,65 @@ +# WebSocket Audio Streaming Implementation Plan + +Implement a WebSocket-based real-time voice integration between the ESP32 board and the Hermes Voice Gateway (Phase 1: WebSocket streaming with server-side batch fallback) to solve the RAM/Flash storage constraints and lower latency. + +## User Review Required + +> [!IMPORTANT] +> - This plan adds a new route `/api/esp32/voice/ws` to the Hermes Gateway. The old POST route `/api/esp32/voice` will remain completely intact to act as a stable fallback. +> - A lightweight WebSocket client `websocket_client.py` will be added to the MicroPython `lib/` directory since standard MicroPython doesn't bundle a WebSocket client by default. + +## Open Questions + +> [!NOTE] +> None at the moment. The proposed wire protocol and incremental phased approach minimize risk. + +--- + +## Proposed Changes + +### 1. Hermes Voice Gateway + +#### [MODIFY] [api_server_endpoint.py](file:///Users/adolforeyna/Projects/MicroPython/test1/Screen/hermes_voice_gateway/api_server_endpoint.py) +* Add a `_handle_esp32_voice_websocket` handler method. +* Listen for a JSON `start` frame to read parameters like `device_id`, `reply_mode`, and `screen_url`. +* Accept incoming binary frames (raw PCM s16le mono 16 kHz chunks) and write them directly to a temporary WAV file on the host's disk. +* Upon receiving the JSON `end` frame (or a disconnection/timeout): + * Close the WAV file writer. + * Invoke `transcribe_audio(wav_path)` using the local `faster-whisper` engine. + * Run the agent turn via `_run_esp32_voice_turn`. + * Synthesize TTS audio and send the resulting WAV file back as a binary frame to the client. +* Update `register_esp32_voice_route` to bind the new WebSocket endpoint: + ```python + app.router.add_get("/api/esp32/voice/ws", api_server._handle_esp32_voice_websocket) + ``` + +#### [MODIFY] [README.md](file:///Users/adolforeyna/Projects/MicroPython/test1/Screen/hermes_voice_gateway/README.md) +* Update documentation to cover the new `/api/esp32/voice/ws` WebSocket client contract and protocol. + +--- + +### 2. ESP32 MicroPython Firmware + +#### [NEW] [websocket_client.py](file:///Users/adolforeyna/Projects/MicroPython/test1/Screen/lib/websocket_client.py) +* Implement a minimal, robust, memory-efficient WebSocket client helper class (`WebSocketClient`) in MicroPython. +* Support HTTP handshake, sending/receiving text frames (opcode 1), and binary frames (opcode 2) with proper masking (client-to-server requirement). + +#### [NEW] [demo_websocket_voice.py](file:///Users/adolforeyna/Projects/MicroPython/test1/Screen/demo_websocket_voice.py) +* Create a dedicated firmware script demonstrating how the board communicates with the new gateway: + * Connect to `ws://:8642/api/esp32/voice/ws`. + * Send the JSON start frame. + * Record continuously from I2S and stream 100ms chunks (3200 bytes mono) over the socket. + * Send the JSON end frame on screen release. + * Listen for the server's response WAV frame and play it using the ES8311 speaker codec. + +--- + +## Verification Plan + +### Automated Tests +* Run `python3 hermes_voice_gateway/smoke_test.py` to ensure local helper methods (normalization, MIME mapping) remain fully functional. + +### Manual Verification +1. Run the updated Hermes server with the new WebSocket route registered. +2. Run `demo_websocket_voice.py` on the ESP32 board. +3. Verify that the ESP32 connects, streams audio chunks on-the-fly without running out of RAM (SRAM memory checks via `gc.mem_free()`), transcribes the speech, and plays back the agent's voice response.