VoiceAgent/PLAN.md

Plan: meeting notes, and shipping this as an internal Mac app

Two separate pieces of work. The meeting-notes feature is buildable today. The packaging work is mostly not a coding problem — it's a signing, permissions and distribution problem, and one policy question that should be answered before any of it starts.

Nothing here is implemented yet.

---

Part 1 — Transcribing the other side of a meeting

Is it possible?

Yes. macOS 26 has two ways to capture audio that is being played, neither of which needs a virtual audio device like BlackHole or Loopback:

Approach	API	Notes
Core Audio process tap	AudioHardwareCreateProcessTap + CATapDescription	Confirmed present in the installed SDK (AudioHardwareTapping.h). Can tap one process — e.g. only Zoom — rather than everything. Preferred.
ScreenCaptureKit	SCStream with audio only	pyobjc-framework-ScreenCaptureKit 12.2.1 is on PyPI. Simpler bindings, but conceptually "screen recording", and captures system-wide.

Both need a TCC grant (Screen Recording / Audio Recording), which is currently not granted. Same one-time prompt as the other permissions.

Design sketch

Run it as a second, independent pipeline in the same process, not as a branch of the conversation pipeline. It has a different job, a different lifetime, and must not interfere with turn taking.

Zoom process audio ──► tap ──► Apple STT ──► transcript writer ──► notes/2026-08-07-standup.md
your microphone   ──► tap ──► Apple STT ──┘

Points that matter:

• Tap the mic separately from the output. Two streams gives you speaker attribution for free — output is "them", mic is "you" — without any diarization model. This is the single biggest quality win available and it's nearly free.
• Reuse apple_stt.py. It already transcribes accurately at 0.05–0.10 s per utterance, and needs no model download.
• Long-form transcription needs chunking. SFSpeechRecognizer is built for utterances, not hour-long meetings; recognition tasks end on their own. The notes pipeline will need to segment on silence and restart tasks, and that behaviour needs to be verified against a real long meeting before trusting it. This is the main technical unknown.
• macOS 26 also ships SpeechAnalyzer/SpeechTranscriber, which is designed for long-form audio and would be a better fit — but it's a Swift-only API and unreachable from pyobjc. See the packaging section; this is one of the stronger arguments for a Swift shell later.

Turning it on and off

Two triggers, and they're not equivalent:

• Automatic on Zoom. Detection is trivial — Zoom is running right now (/Applications/zoom.us.app, pid 89302) and installs its own ZoomAudioDevice Core Audio driver. But process running is a poor signal: Zoom idles in the background for hours. Better to trigger on Zoom actually holding an active audio stream, or on a calendar event from meta calendar — which knows when a meeting is genuinely scheduled.
• By voice. Don't do this by pattern-matching transcripts. Claude is already in the loop, so expose start_meeting_notes / stop_meeting_notes as SDK MCP tools and let him call them. That handles "start taking notes", "actually stop that", and "are you recording?" without any string matching, and it's a handful of lines.

Before building this: the consent question

This records other people. Worth settling first, because it shapes the design and it's much cheaper to answer now:

1. Meta already has an official AI notetaker with transcripts and summaries, reachable from meta calendar. If that covers the need, this feature is redundant — and it's already been through review.
2. Some jurisdictions require all-party consent to record a conversation. A personal tool that silently captures colleagues is a different thing from one that announces itself.
3. Distributing a recorder internally is a compliance matter, not just an engineering one. This is the single biggest risk to the "share it with the team" goal, and it applies to the meeting-notes feature specifically — not to the voice assistant.

Concrete suggestions if it goes ahead: default it off, require an explicit per-meeting start, make it obvious when it's running, and keep transcripts local with a retention policy. Get a read from Privacy/Legal before it goes to anyone else's machine.

Recommendation: build it for your own use behind an off-by-default flag, and treat "ship the notes feature to the team" as a separate decision gated on that review. The voice assistant itself carries none of this baggage and can ship first.

---

Part 2 — Packaging as an internal Mac app

What's actually hard

Not the code. Three things:

1. Permissions are the whole reason to do this. You just granted Input Monitoring to your terminal and noted it'll take effect on restart — that fragility is the argument for a real app. TCC grants attach to a code-signed bundle identity. Today the grants belong to your terminal, so they're shared with everything else you run there and break when the terminal changes. A signed .app with a stable bundle ID asks once, keeps it, and shows up in System Settings under its own name. This app needs four: Microphone, Speech Recognition, Input Monitoring, and (for meeting notes) Screen/Audio Recording.

2. Signing. security find-identity -p codesigning returns 0 valid identities on this machine. Unsigned or ad-hoc-signed apps have their TCC grants invalidated on every rebuild, which makes the app unusable in practice. This needs a real signing identity from whoever owns Mac app distribution. It also intersects with the binary-approval policy on managed Macs: a hand-built .app handed to a colleague will be killed on launch unless it comes through the sanctioned channel.

3. Distribution. Managed Software Center is installed — that's the internal channel, and the answer to "how do I share this". It also solves signing and approval, because packages that ship through it are already handled. The work is conforming to whatever that team requires, not inventing a mechanism.

Size: fixable, and the fix is cheap

The venv is 1.7 GB, which is a non-starter for distribution. Almost all of it is for engines that are no longer the default:

Package	Size	Needed by
torch	529 MB	mlx-whisper only
claude_agent_sdk _bundled	260 MB	its bundled CLI — we deliberately don't use it
mlx	183 MB	mlx-whisper
llvmlite + numba	156 MB	mlx-whisper
scipy + sympy	171 MB	mlx-whisper (scipy also used by the resampler)
onnxruntime	69 MB	Kokoro, Silero VAD
av	45 MB	faster-whisper

Switching the shipped default to Apple STT + Apple TTS + push-to-talk drops Whisper, Kokoro and Silero. I verified this: importing only the Apple-path modules pulls in scipy and nothing else heavy, whereas importing bot.py pulls in all nine. A realistic app is 250–300 MB, most of which is the Python runtime and pyobjc.

Two concrete prerequisites:

• Make the Whisper/Kokoro imports lazy. bot.py imports them at module scope today, so they'd be bundled even when unused.
• Exclude claude_agent_sdk/_bundled (260 MB). The app must call the installed claude CLI anyway — it's the only build that can reach the gateway — so Claude Code at Meta becomes a documented prerequisite, not something the app ships.

Shape of the app

A menu-bar app fits the interaction model: no window, hold a key to talk, a menu for voice/model/notes toggles, and an indicator showing when the mic is live and when notes are recording.

Option	Effort	Trade-off
A. Python + py2app + rumps	Low	Reuses everything as-is. Bundles a Python runtime; py2app + pyobjc + code signing is fiddly but well-trodden.
B. Swift shell, Python core as a subprocess	High	Native menu bar and a much cleaner signing/TCC story. Also unlocks SpeechAnalyzer for long-form meeting transcription.
C. Stay a CLI, distribute as an internal package	Lowest	No TCC identity of its own, so the permissions problem stays. Fine for a handful of engineers, not for a team.

Recommendation: A for a pilot, with B as the path if it gets real adoption. A gets it onto a few machines quickly and proves whether anyone wants it. B is justified once it needs to survive OS upgrades and support non-engineers, and its extra value is concentrated exactly where A is weakest — signing and long-form transcription.

Suggested sequence

1. Decide the notes/consent question. Blocks part 1 only; everything else can proceed in parallel.
2. Slim the dependencies. Lazy imports, exclude the bundled CLI, verify the Apple-only path runs with Whisper/Kokoro uninstalled. Do this first — it's pure cleanup with no external dependency, and it makes every later step smaller.
3. Prototype the process tap against Zoom, behind an off-by-default flag, for your own use. Verify long-meeting chunking on a real call.
4. Talk to whoever owns Managed Software Center about signing and packaging requirements before building the bundle. Their answer may dictate the app's shape, and finding that out after building option A would be expensive.
5. Build the menu-bar bundle (option A), signed, and install it on your own machine. Confirm all four TCC grants survive a rebuild and a reboot — that's the acceptance test that matters.
6. Pilot with two or three teammates before any wider push.

Open questions

• Who owns Mac app signing and Managed Software Center packaging?
• Does the official meta calendar notetaker already cover the meeting-notes need well enough to drop that feature entirely?
• Should the app require Claude Code at Meta as a prerequisite (simple, 260 MB smaller, and the only thing that authenticates) or attempt to bundle a CLI?
• Is a menu-bar app the right shape, or would a Raycast/Alfred-style overlay suit how people actually work better?