FamReynaBrain/projects/live_captions_esp32_2026-08-23.md

date, type, status, tags, related

date	type	status	tags	related
2026-08-23	project	active	tactility esp32 captions voice mac-mini stt	Voice & Dev Infrastructure

Live Captions — ESP32 / Mac mini

Purpose

A dedicated Tactility ESP32-S3 app streams device microphone PCM to the Mac mini for local Apple Speech captioning. It shows captions on the device and records text locally on its SD card; raw device audio is not retained by the app.

Active architecture

Tactility `.113` (kidsos_591c)
  -> authenticated WebSocket PCM, 16 kHz / mono / s16le
  -> Mac mini `.102` caption service on :8645
  -> MacMiniMCP Apple Speech live transcription
  -> draft + silence-finalized review events
  -> ESP32 native caption display

The dedicated caption route does not invoke the old generic gateway screen-MCP drawing path, preventing caption UI conflicts.

Current state

• Live Captions app deployed to .113; auto-connect and automatic reconnect are enabled.
• UI: device toolbar/close control, latest 50 caption words, and a bottom Connecting / Connected / Failed—Reconnecting status line.
• Mac service accepts multiple sessions and bounds retained audio at 30 seconds / 960 KB per device.
• The old fixed 64-second stream cap was removed.
• Silence VAD turns speech pauses into review events without closing the microphone stream.
• Finalized reviews are logged on the Mac mini to ~/.hermes/logs/esp32-captions/YYYY-MM-DD.jsonl with UTC timestamp and friendly device name.
• The board is currently connected and the service has confirmed incoming 1,024-byte PCM frames.

Verified test evidence

• Four caption-service tests are passing, including rolling-buffer and multi-device coverage.
• Synthesized long audio streams produced live draft/review events and final transcripts.
• A synthesized speech message plus valid silence produced a review before an explicit stop command.

Next

• Observe a natural on-device speech pause and confirm the review visibly replaces the draft.
• Tune capture gain/filtering after that UI verification.
• Install the caption service LaunchAgent from an approved external shell for durability.
• Commit the app/service once physical acceptance is confirmed.