Adolfo Reyna
50 lines
2.2 KiB
Markdown
50 lines
2.2 KiB
Markdown
---
|
|
date: 2026-08-23
|
|
type: project
|
|
status: active
|
|
tags: [tactility, esp32, captions, voice, mac-mini, stt]
|
|
related: [[Voice & Dev Infrastructure]]
|
|
---
|
|
|
|
# Live Captions — ESP32 / Mac mini
|
|
|
|
## Purpose
|
|
|
|
A dedicated Tactility ESP32-S3 app streams device microphone PCM to the Mac mini for local Apple Speech captioning. It shows captions on the device and records text locally on its SD card; raw device audio is not retained by the app.
|
|
|
|
## Active architecture
|
|
|
|
```text
|
|
Tactility `.113` (kidsos_591c)
|
|
-> authenticated WebSocket PCM, 16 kHz / mono / s16le
|
|
-> Mac mini `.102` caption service on :8645
|
|
-> MacMiniMCP Apple Speech live transcription
|
|
-> draft + silence-finalized review events
|
|
-> ESP32 native caption display
|
|
```
|
|
|
|
The dedicated caption route does not invoke the old generic gateway screen-MCP drawing path, preventing caption UI conflicts.
|
|
|
|
## Current state
|
|
|
|
- Live Captions app deployed to `.113`; auto-connect and automatic reconnect are enabled.
|
|
- UI: device toolbar/close control, latest 50 caption words, and a bottom Connecting / Connected / Failed—Reconnecting status line.
|
|
- Mac service accepts multiple sessions and bounds retained audio at 30 seconds / 960 KB per device.
|
|
- The old fixed 64-second stream cap was removed.
|
|
- Silence VAD turns speech pauses into `review` events without closing the microphone stream.
|
|
- Finalized reviews are logged on the Mac mini to `~/.hermes/logs/esp32-captions/YYYY-MM-DD.jsonl` with UTC timestamp and friendly device name.
|
|
- The board is currently connected and the service has confirmed incoming 1,024-byte PCM frames.
|
|
|
|
## Verified test evidence
|
|
|
|
- Four caption-service tests are passing, including rolling-buffer and multi-device coverage.
|
|
- Synthesized long audio streams produced live draft/review events and final transcripts.
|
|
- A synthesized speech message plus valid silence produced a `review` before an explicit stop command.
|
|
|
|
## Next
|
|
|
|
- Observe a natural on-device speech pause and confirm the review visibly replaces the draft.
|
|
- Tune capture gain/filtering after that UI verification.
|
|
- Install the caption service LaunchAgent from an approved external shell for durability.
|
|
- Commit the app/service once physical acceptance is confirmed.
|