---
date: 2026-08-23
type: project
status: active
tags: [tactility, esp32, captions, voice, mac-mini, stt]
related: [[Voice & Dev Infrastructure]]
---

# Live Captions — ESP32 / Mac mini

## Purpose

A dedicated Tactility ESP32-S3 app streams device microphone PCM to the Mac mini for local Apple Speech captioning. It shows captions on the device and records text locally on its SD card; raw device audio is not retained by the app.

## Active architecture

```text
Tactility `.113` (kidsos_591c)
  -> authenticated WebSocket PCM, 16 kHz / mono / s16le
  -> Mac mini `.102` caption service on :8645
  -> MacMiniMCP Apple Speech live transcription
  -> draft + silence-finalized review events
  -> ESP32 native caption display
```

The dedicated caption route does not invoke the old generic gateway screen-MCP drawing path, preventing caption UI conflicts.

## Current state

- Live Captions app deployed to `.113`; auto-connect and automatic reconnect are enabled.
- UI: device toolbar/close control, latest 50 caption words, and a bottom Connecting / Connected / Failed—Reconnecting status line.
- Mac service accepts multiple sessions and bounds retained audio at 30 seconds / 960 KB per device.
- The old fixed 64-second stream cap was removed.
- Silence VAD turns speech pauses into `review` events without closing the microphone stream.
- Finalized reviews are logged on the Mac mini to `~/.hermes/logs/esp32-captions/YYYY-MM-DD.jsonl` with UTC timestamp and friendly device name.
- The board is currently connected and the service has confirmed incoming 1,024-byte PCM frames.

## Verified test evidence

- Four caption-service tests are passing, including rolling-buffer and multi-device coverage.
- Synthesized long audio streams produced live draft/review events and final transcripts.
- A synthesized speech message plus valid silence produced a `review` before an explicit stop command.

## Next

- Observe a natural on-device speech pause and confirm the review visibly replaces the draft.
- Tune capture gain/filtering after that UI verification.
- Install the caption service LaunchAgent from an approved external shell for durability.
- Commit the app/service once physical acceptance is confirmed.