Initial commit of current state
This commit is contained in:
Adolfo Reyna
@@ -0,0 +1,203 @@
|
||||
# Plan: meeting notes, and shipping this as an internal Mac app
|
||||
|
||||
Two separate pieces of work. The meeting-notes feature is buildable today. The
|
||||
packaging work is mostly not a coding problem — it's a signing, permissions and
|
||||
distribution problem, and one policy question that should be answered before
|
||||
any of it starts.
|
||||
|
||||
Nothing here is implemented yet.
|
||||
|
||||
---
|
||||
|
||||
## Part 1 — Transcribing the other side of a meeting
|
||||
|
||||
### Is it possible?
|
||||
|
||||
Yes. macOS 26 has two ways to capture audio that is being *played*, neither of
|
||||
which needs a virtual audio device like BlackHole or Loopback:
|
||||
|
||||
| Approach | API | Notes |
|
||||
| --- | --- | --- |
|
||||
| **Core Audio process tap** | `AudioHardwareCreateProcessTap` + `CATapDescription` | Confirmed present in the installed SDK (`AudioHardwareTapping.h`). Can tap one process — e.g. only Zoom — rather than everything. Preferred. |
|
||||
| **ScreenCaptureKit** | `SCStream` with audio only | `pyobjc-framework-ScreenCaptureKit` 12.2.1 is on PyPI. Simpler bindings, but conceptually "screen recording", and captures system-wide. |
|
||||
|
||||
Both need a TCC grant (Screen Recording / Audio Recording), which is currently
|
||||
not granted. Same one-time prompt as the other permissions.
|
||||
|
||||
### Design sketch
|
||||
|
||||
Run it as a **second, independent pipeline** in the same process, not as a
|
||||
branch of the conversation pipeline. It has a different job, a different
|
||||
lifetime, and must not interfere with turn taking.
|
||||
|
||||
```
|
||||
Zoom process audio ──► tap ──► Apple STT ──► transcript writer ──► notes/2026-08-07-standup.md
|
||||
your microphone ──► tap ──► Apple STT ──┘
|
||||
```
|
||||
|
||||
Points that matter:
|
||||
|
||||
- **Tap the mic separately from the output.** Two streams gives you speaker
|
||||
attribution for free — output is "them", mic is "you" — without any
|
||||
diarization model. This is the single biggest quality win available and it's
|
||||
nearly free.
|
||||
- **Reuse `apple_stt.py`.** It already transcribes accurately at 0.05–0.10 s
|
||||
per utterance, and needs no model download.
|
||||
- **Long-form transcription needs chunking.** `SFSpeechRecognizer` is built for
|
||||
utterances, not hour-long meetings; recognition tasks end on their own. The
|
||||
notes pipeline will need to segment on silence and restart tasks, and that
|
||||
behaviour needs to be verified against a real long meeting before trusting
|
||||
it. This is the main technical unknown.
|
||||
- macOS 26 also ships `SpeechAnalyzer`/`SpeechTranscriber`, which is *designed*
|
||||
for long-form audio and would be a better fit — but it's a Swift-only API and
|
||||
unreachable from pyobjc. See the packaging section; this is one of the
|
||||
stronger arguments for a Swift shell later.
|
||||
|
||||
### Turning it on and off
|
||||
|
||||
Two triggers, and they're not equivalent:
|
||||
|
||||
- **Automatic on Zoom.** Detection is trivial — Zoom is running right now
|
||||
(`/Applications/zoom.us.app`, pid 89302) and installs its own
|
||||
`ZoomAudioDevice` Core Audio driver. But *process running* is a poor signal:
|
||||
Zoom idles in the background for hours. Better to trigger on Zoom actually
|
||||
holding an active audio stream, or on a calendar event from
|
||||
`meta calendar` — which knows when a meeting is genuinely scheduled.
|
||||
- **By voice.** Don't do this by pattern-matching transcripts. Claude is
|
||||
already in the loop, so expose `start_meeting_notes` / `stop_meeting_notes`
|
||||
as SDK MCP tools and let him call them. That handles "start taking notes",
|
||||
"actually stop that", and "are you recording?" without any string matching,
|
||||
and it's a handful of lines.
|
||||
|
||||
### Before building this: the consent question
|
||||
|
||||
This records other people. Worth settling first, because it shapes the design
|
||||
and it's much cheaper to answer now:
|
||||
|
||||
1. **Meta already has an official AI notetaker** with transcripts and
|
||||
summaries, reachable from `meta calendar`. If that covers the need, this
|
||||
feature is redundant — and it's already been through review.
|
||||
2. **Some jurisdictions require all-party consent** to record a conversation.
|
||||
A personal tool that silently captures colleagues is a different thing from
|
||||
one that announces itself.
|
||||
3. **Distributing a recorder internally is a compliance matter**, not just an
|
||||
engineering one. This is the single biggest risk to the "share it with the
|
||||
team" goal, and it applies to the meeting-notes feature specifically — not
|
||||
to the voice assistant.
|
||||
|
||||
Concrete suggestions if it goes ahead: default it **off**, require an explicit
|
||||
per-meeting start, make it obvious when it's running, and keep transcripts
|
||||
local with a retention policy. Get a read from Privacy/Legal before it goes to
|
||||
anyone else's machine.
|
||||
|
||||
**Recommendation:** build it for your own use behind an off-by-default flag,
|
||||
and treat "ship the notes feature to the team" as a separate decision gated on
|
||||
that review. The voice assistant itself carries none of this baggage and can
|
||||
ship first.
|
||||
|
||||
---
|
||||
|
||||
## Part 2 — Packaging as an internal Mac app
|
||||
|
||||
### What's actually hard
|
||||
|
||||
Not the code. Three things:
|
||||
|
||||
**1. Permissions are the whole reason to do this.** You just granted Input
|
||||
Monitoring to your terminal and noted it'll take effect on restart — that
|
||||
fragility *is* the argument for a real app. TCC grants attach to a code-signed
|
||||
bundle identity. Today the grants belong to your terminal, so they're shared
|
||||
with everything else you run there and break when the terminal changes. A
|
||||
signed `.app` with a stable bundle ID asks once, keeps it, and shows up in
|
||||
System Settings under its own name. This app needs four: Microphone, Speech
|
||||
Recognition, Input Monitoring, and (for meeting notes) Screen/Audio Recording.
|
||||
|
||||
**2. Signing.** `security find-identity -p codesigning` returns **0 valid
|
||||
identities** on this machine. Unsigned or ad-hoc-signed apps have their TCC
|
||||
grants invalidated on every rebuild, which makes the app unusable in practice.
|
||||
This needs a real signing identity from whoever owns Mac app distribution.
|
||||
It also intersects with the binary-approval policy on managed Macs: a
|
||||
hand-built `.app` handed to a colleague will be killed on launch unless it
|
||||
comes through the sanctioned channel.
|
||||
|
||||
**3. Distribution.** **Managed Software Center** is installed — that's the
|
||||
internal channel, and the answer to "how do I share this". It also solves
|
||||
signing and approval, because packages that ship through it are already
|
||||
handled. The work is conforming to whatever that team requires, not inventing
|
||||
a mechanism.
|
||||
|
||||
### Size: fixable, and the fix is cheap
|
||||
|
||||
The venv is 1.7 GB, which is a non-starter for distribution. Almost all of it
|
||||
is for engines that are no longer the default:
|
||||
|
||||
| Package | Size | Needed by |
|
||||
| --- | --- | --- |
|
||||
| torch | 529 MB | mlx-whisper only |
|
||||
| claude_agent_sdk `_bundled` | 260 MB | its bundled CLI — **we deliberately don't use it** |
|
||||
| mlx | 183 MB | mlx-whisper |
|
||||
| llvmlite + numba | 156 MB | mlx-whisper |
|
||||
| scipy + sympy | 171 MB | mlx-whisper (scipy also used by the resampler) |
|
||||
| onnxruntime | 69 MB | Kokoro, Silero VAD |
|
||||
| av | 45 MB | faster-whisper |
|
||||
|
||||
Switching the shipped default to **Apple STT + Apple TTS + push-to-talk** drops
|
||||
Whisper, Kokoro *and* Silero. I verified this: importing only the Apple-path
|
||||
modules pulls in `scipy` and nothing else heavy, whereas importing `bot.py`
|
||||
pulls in all nine. A realistic app is **250–300 MB**, most of which is the
|
||||
Python runtime and pyobjc.
|
||||
|
||||
Two concrete prerequisites:
|
||||
|
||||
- Make the Whisper/Kokoro imports lazy. `bot.py` imports them at module scope
|
||||
today, so they'd be bundled even when unused.
|
||||
- Exclude `claude_agent_sdk/_bundled` (260 MB). The app must call the installed
|
||||
`claude` CLI anyway — it's the only build that can reach the gateway — so
|
||||
**Claude Code at Meta becomes a documented prerequisite**, not something the
|
||||
app ships.
|
||||
|
||||
### Shape of the app
|
||||
|
||||
A **menu-bar app** fits the interaction model: no window, hold a key to talk, a
|
||||
menu for voice/model/notes toggles, and an indicator showing when the mic is
|
||||
live and when notes are recording.
|
||||
|
||||
| Option | Effort | Trade-off |
|
||||
| --- | --- | --- |
|
||||
| **A. Python + py2app + `rumps`** | Low | Reuses everything as-is. Bundles a Python runtime; py2app + pyobjc + code signing is fiddly but well-trodden. |
|
||||
| **B. Swift shell, Python core as a subprocess** | High | Native menu bar and a much cleaner signing/TCC story. Also unlocks `SpeechAnalyzer` for long-form meeting transcription. |
|
||||
| **C. Stay a CLI, distribute as an internal package** | Lowest | No TCC identity of its own, so the permissions problem stays. Fine for a handful of engineers, not for a team. |
|
||||
|
||||
**Recommendation: A for a pilot, with B as the path if it gets real adoption.**
|
||||
A gets it onto a few machines quickly and proves whether anyone wants it. B is
|
||||
justified once it needs to survive OS upgrades and support non-engineers, and
|
||||
its extra value is concentrated exactly where A is weakest — signing and
|
||||
long-form transcription.
|
||||
|
||||
### Suggested sequence
|
||||
|
||||
1. **Decide the notes/consent question.** Blocks part 1 only; everything else
|
||||
can proceed in parallel.
|
||||
2. **Slim the dependencies.** Lazy imports, exclude the bundled CLI, verify the
|
||||
Apple-only path runs with Whisper/Kokoro uninstalled. Do this first — it's
|
||||
pure cleanup with no external dependency, and it makes every later step
|
||||
smaller.
|
||||
3. **Prototype the process tap** against Zoom, behind an off-by-default flag,
|
||||
for your own use. Verify long-meeting chunking on a real call.
|
||||
4. **Talk to whoever owns Managed Software Center** about signing and packaging
|
||||
requirements before building the bundle. Their answer may dictate the app's
|
||||
shape, and finding that out after building option A would be expensive.
|
||||
5. **Build the menu-bar bundle** (option A), signed, and install it on your own
|
||||
machine. Confirm all four TCC grants survive a rebuild and a reboot — that's
|
||||
the acceptance test that matters.
|
||||
6. **Pilot with two or three teammates** before any wider push.
|
||||
|
||||
### Open questions
|
||||
|
||||
- Who owns Mac app signing and Managed Software Center packaging?
|
||||
- Does the official `meta calendar` notetaker already cover the meeting-notes
|
||||
need well enough to drop that feature entirely?
|
||||
- Should the app require Claude Code at Meta as a prerequisite (simple, 260 MB
|
||||
smaller, and the only thing that authenticates) or attempt to bundle a CLI?
|
||||
- Is a menu-bar app the right shape, or would a Raycast/Alfred-style overlay
|
||||
suit how people actually work better?
|
||||
Reference in New Issue
Block a user