VoiceAgent/PLAN.md

# Plan: meeting notes, and shipping this as an internal Mac app

Two separate pieces of work. The meeting-notes feature is buildable today. The
packaging work is mostly not a coding problem — it's a signing, permissions and
distribution problem, and one policy question that should be answered before
any of it starts.

Nothing here is implemented yet.

---

## Part 1 — Transcribing the other side of a meeting

### Is it possible?

Yes. macOS 26 has two ways to capture audio that is being *played*, neither of
which needs a virtual audio device like BlackHole or Loopback:

| Approach | API | Notes |
| --- | --- | --- |
| **Core Audio process tap** | `AudioHardwareCreateProcessTap` + `CATapDescription` | Confirmed present in the installed SDK (`AudioHardwareTapping.h`). Can tap one process — e.g. only Zoom — rather than everything. Preferred. |
| **ScreenCaptureKit** | `SCStream` with audio only | `pyobjc-framework-ScreenCaptureKit` 12.2.1 is on PyPI. Simpler bindings, but conceptually "screen recording", and captures system-wide. |

Both need a TCC grant (Screen Recording / Audio Recording), which is currently
not granted. Same one-time prompt as the other permissions.

### Design sketch

Run it as a **second, independent pipeline** in the same process, not as a
branch of the conversation pipeline. It has a different job, a different
lifetime, and must not interfere with turn taking.

```
Zoom process audio ──► tap ──► Apple STT ──► transcript writer ──► notes/2026-08-07-standup.md
your microphone   ──► tap ──► Apple STT ──┘
```

Points that matter:

- **Tap the mic separately from the output.** Two streams gives you speaker
  attribution for free — output is "them", mic is "you" — without any
  diarization model. This is the single biggest quality win available and it's
  nearly free.
- **Reuse `apple_stt.py`.** It already transcribes accurately at 0.05–0.10 s
  per utterance, and needs no model download.
- **Long-form transcription needs chunking.** `SFSpeechRecognizer` is built for
  utterances, not hour-long meetings; recognition tasks end on their own. The
  notes pipeline will need to segment on silence and restart tasks, and that
  behaviour needs to be verified against a real long meeting before trusting
  it. This is the main technical unknown.
- macOS 26 also ships `SpeechAnalyzer`/`SpeechTranscriber`, which is *designed*
  for long-form audio and would be a better fit — but it's a Swift-only API and
  unreachable from pyobjc. See the packaging section; this is one of the
  stronger arguments for a Swift shell later.

### Turning it on and off

Two triggers, and they're not equivalent:

- **Automatic on Zoom.** Detection is trivial — Zoom is running right now
  (`/Applications/zoom.us.app`, pid 89302) and installs its own
  `ZoomAudioDevice` Core Audio driver. But *process running* is a poor signal:
  Zoom idles in the background for hours. Better to trigger on Zoom actually
  holding an active audio stream, or on a calendar event from
  `meta calendar` — which knows when a meeting is genuinely scheduled.
- **By voice.** Don't do this by pattern-matching transcripts. Claude is
  already in the loop, so expose `start_meeting_notes` / `stop_meeting_notes`
  as SDK MCP tools and let him call them. That handles "start taking notes",
  "actually stop that", and "are you recording?" without any string matching,
  and it's a handful of lines.

### Before building this: the consent question

This records other people. Worth settling first, because it shapes the design
and it's much cheaper to answer now:

1. **Meta already has an official AI notetaker** with transcripts and
   summaries, reachable from `meta calendar`. If that covers the need, this
   feature is redundant — and it's already been through review.
2. **Some jurisdictions require all-party consent** to record a conversation.
   A personal tool that silently captures colleagues is a different thing from
   one that announces itself.
3. **Distributing a recorder internally is a compliance matter**, not just an
   engineering one. This is the single biggest risk to the "share it with the
   team" goal, and it applies to the meeting-notes feature specifically — not
   to the voice assistant.

Concrete suggestions if it goes ahead: default it **off**, require an explicit
per-meeting start, make it obvious when it's running, and keep transcripts
local with a retention policy. Get a read from Privacy/Legal before it goes to
anyone else's machine.

**Recommendation:** build it for your own use behind an off-by-default flag,
and treat "ship the notes feature to the team" as a separate decision gated on
that review. The voice assistant itself carries none of this baggage and can
ship first.

---

## Part 2 — Packaging as an internal Mac app

### What's actually hard

Not the code. Three things:

**1. Permissions are the whole reason to do this.** You just granted Input
Monitoring to your terminal and noted it'll take effect on restart — that
fragility *is* the argument for a real app. TCC grants attach to a code-signed
bundle identity. Today the grants belong to your terminal, so they're shared
with everything else you run there and break when the terminal changes. A
signed `.app` with a stable bundle ID asks once, keeps it, and shows up in
System Settings under its own name. This app needs four: Microphone, Speech
Recognition, Input Monitoring, and (for meeting notes) Screen/Audio Recording.

**2. Signing.** `security find-identity -p codesigning` returns **0 valid
identities** on this machine. Unsigned or ad-hoc-signed apps have their TCC
grants invalidated on every rebuild, which makes the app unusable in practice.
This needs a real signing identity from whoever owns Mac app distribution.
It also intersects with the binary-approval policy on managed Macs: a
hand-built `.app` handed to a colleague will be killed on launch unless it
comes through the sanctioned channel.

**3. Distribution.** **Managed Software Center** is installed — that's the
internal channel, and the answer to "how do I share this". It also solves
signing and approval, because packages that ship through it are already
handled. The work is conforming to whatever that team requires, not inventing
a mechanism.

### Size: fixable, and the fix is cheap

The venv is 1.7 GB, which is a non-starter for distribution. Almost all of it
is for engines that are no longer the default:

| Package | Size | Needed by |
| --- | --- | --- |
| torch | 529 MB | mlx-whisper only |
| claude_agent_sdk `_bundled` | 260 MB | its bundled CLI — **we deliberately don't use it** |
| mlx | 183 MB | mlx-whisper |
| llvmlite + numba | 156 MB | mlx-whisper |
| scipy + sympy | 171 MB | mlx-whisper (scipy also used by the resampler) |
| onnxruntime | 69 MB | Kokoro, Silero VAD |
| av | 45 MB | faster-whisper |

Switching the shipped default to **Apple STT + Apple TTS + push-to-talk** drops
Whisper, Kokoro *and* Silero. I verified this: importing only the Apple-path
modules pulls in `scipy` and nothing else heavy, whereas importing `bot.py`
pulls in all nine. A realistic app is **250–300 MB**, most of which is the
Python runtime and pyobjc.

Two concrete prerequisites:

- Make the Whisper/Kokoro imports lazy. `bot.py` imports them at module scope
  today, so they'd be bundled even when unused.
- Exclude `claude_agent_sdk/_bundled` (260 MB). The app must call the installed
  `claude` CLI anyway — it's the only build that can reach the gateway — so
  **Claude Code at Meta becomes a documented prerequisite**, not something the
  app ships.

### Shape of the app

A **menu-bar app** fits the interaction model: no window, hold a key to talk, a
menu for voice/model/notes toggles, and an indicator showing when the mic is
live and when notes are recording.

| Option | Effort | Trade-off |
| --- | --- | --- |
| **A. Python + py2app + `rumps`** | Low | Reuses everything as-is. Bundles a Python runtime; py2app + pyobjc + code signing is fiddly but well-trodden. |
| **B. Swift shell, Python core as a subprocess** | High | Native menu bar and a much cleaner signing/TCC story. Also unlocks `SpeechAnalyzer` for long-form meeting transcription. |
| **C. Stay a CLI, distribute as an internal package** | Lowest | No TCC identity of its own, so the permissions problem stays. Fine for a handful of engineers, not for a team. |

**Recommendation: A for a pilot, with B as the path if it gets real adoption.**
A gets it onto a few machines quickly and proves whether anyone wants it. B is
justified once it needs to survive OS upgrades and support non-engineers, and
its extra value is concentrated exactly where A is weakest — signing and
long-form transcription.

### Suggested sequence

1. **Decide the notes/consent question.** Blocks part 1 only; everything else
   can proceed in parallel.
2. **Slim the dependencies.** Lazy imports, exclude the bundled CLI, verify the
   Apple-only path runs with Whisper/Kokoro uninstalled. Do this first — it's
   pure cleanup with no external dependency, and it makes every later step
   smaller.
3. **Prototype the process tap** against Zoom, behind an off-by-default flag,
   for your own use. Verify long-meeting chunking on a real call.
4. **Talk to whoever owns Managed Software Center** about signing and packaging
   requirements before building the bundle. Their answer may dictate the app's
   shape, and finding that out after building option A would be expensive.
5. **Build the menu-bar bundle** (option A), signed, and install it on your own
   machine. Confirm all four TCC grants survive a rebuild and a reboot — that's
   the acceptance test that matters.
6. **Pilot with two or three teammates** before any wider push.

### Open questions

- Who owns Mac app signing and Managed Software Center packaging?
- Does the official `meta calendar` notetaker already cover the meeting-notes
  need well enough to drop that feature entirely?
- Should the app require Claude Code at Meta as a prerequisite (simple, 260 MB
  smaller, and the only thing that authenticates) or attempt to bundle a CLI?
- Is a menu-bar app the right shape, or would a Raycast/Alfred-style overlay
  suit how people actually work better?