brain backup 2026-09-14

2026-09-14 22:38:47 -04:00
parent ca346cc6e7
commit f0ead4f6fa
147 changed files with 73451 additions and 8 deletions
@@ -0,0 +1,49 @@
+---
+date: 2026-08-23
+type: project
+status: active
+tags: [tactility, esp32, captions, voice, mac-mini, stt]
+related: [[Voice & Dev Infrastructure]]
+---
+
+# Live Captions — ESP32 / Mac mini
+
+## Purpose
+
+A dedicated Tactility ESP32-S3 app streams device microphone PCM to the Mac mini for local Apple Speech captioning. It shows captions on the device and records text locally on its SD card; raw device audio is not retained by the app.
+
+## Active architecture
+
+```text
+Tactility `.113` (kidsos_591c)
+  -> authenticated WebSocket PCM, 16 kHz / mono / s16le
+  -> Mac mini `.102` caption service on :8645
+  -> MacMiniMCP Apple Speech live transcription
+  -> draft + silence-finalized review events
+  -> ESP32 native caption display
+```
+
+The dedicated caption route does not invoke the old generic gateway screen-MCP drawing path, preventing caption UI conflicts.
+
+## Current state
+
+- Live Captions app deployed to `.113`; auto-connect and automatic reconnect are enabled.
+- UI: device toolbar/close control, latest 50 caption words, and a bottom Connecting / Connected / Failed—Reconnecting status line.
+- Mac service accepts multiple sessions and bounds retained audio at 30 seconds / 960 KB per device.
+- The old fixed 64-second stream cap was removed.
+- Silence VAD turns speech pauses into `review` events without closing the microphone stream.
+- Finalized reviews are logged on the Mac mini to `~/.hermes/logs/esp32-captions/YYYY-MM-DD.jsonl` with UTC timestamp and friendly device name.
+- The board is currently connected and the service has confirmed incoming 1,024-byte PCM frames.
+
+## Verified test evidence
+
+- Four caption-service tests are passing, including rolling-buffer and multi-device coverage.
+- Synthesized long audio streams produced live draft/review events and final transcripts.
+- A synthesized speech message plus valid silence produced a `review` before an explicit stop command.
+
+## Next
+
+- Observe a natural on-device speech pause and confirm the review visibly replaces the draft.
+- Tune capture gain/filtering after that UI verification.
+- Install the caption service LaunchAgent from an approved external shell for durability.
+- Commit the app/service once physical acceptance is confirmed.