document family voice context project

2026-08-08 23:50:29 -04:00
parent 367bbb1578
commit e559811eb3
2 changed files with 57 additions and 0 deletions
@@ -0,0 +1,56 @@
+---
+date: 2026-08-08
+type: project
+status: active
+tags: [voice, speaker-identification, family, esp32, pipecat, local-ai, privacy]
+---
+
+# Family Voice Context and Speaker Identification 2026
+
+## Idea
+
+Use the family ESP32 voice devices as low-power local audio sensors. While idle, each device measures microphone energy / lightweight VAD, starts a short recording when speech is detected, and sends the clip over the private LAN for local processing.
+
+The Mac mini should perform accurate VAD, transcription, speaker diarization, and speaker verification. Initial target: predict whether Adolfo's voice is present in a short clip, using clear WhatsApp voice recordings as enrollment candidates. Later extend to best-effort profiles for Adolfo, Alicia, Grace, and Elias, always allowing `unknown` and `uncertain` results.
+
+## Privacy and retention policy
+
+- All processing remains local; no cloud transcription, embeddings, or raw audio uploads.
+- ESP32 devices must have a visible recording indicator and physical mute behavior.
+- Raw audio should be temporary and deleted after successful processing by default.
+- Do not automatically write every conversation into the canonical brain.
+- Keep a separate short-retention conversation log; promote only confirmed durable facts, tasks, or decisions into the brain.
+- Treat medical, financial, intimate, and private conversations as excluded or review-only.
+- Never use speaker recognition as sole authorization for high-impact actions.
+
+## Initial prototype
+
+Build a small local speaker-verification service that:
+
+1. Finds and labels candidate Adolfo voice clips from local WhatsApp/ESP32 audio cache.
+2. Creates a speaker embedding profile from approved enrollment clips.
+3. Scores short WAV clips as `adolfo`, `not_adolfo`, or `unknown`.
+4. Reports cosine similarity, duration, source/device, and model version.
+5. Keeps raw recordings out of Git/Gitea and stores only code, metadata, and derived profile artifacts when explicitly approved.
+6. Provides a small adapter boundary suitable for a future Pipecat pipeline after VAD and STT, before LLM/context injection.
+
+## Candidate architecture
+
+```text
+ESP32 mic -> energy/adaptive VAD + pre-roll -> LAN clip upload
+  -> Mac mini voice service -> accurate VAD -> local STT
+  -> speaker embedding / diarization -> local transcript log
+  -> optional reviewed context retrieval -> Hermes / brain suggestion
+```
+
+## Decisions still open
+
+- Exact source labeling for the cached WhatsApp/ESP32 WAV files.
+- ECAPA-TDNN/SpeechBrain versus a lighter embedding model on Mac mini.
+- Whether profiles should live in a private ignored data directory or encrypted local store.
+- Pipecat integration point: audio clip preprocessor or post-STT speaker metadata processor.
+- Family consent, retention period, and device-room recording policy before multi-person rollout.
+
+## Next action
+
+Run the Adolfo-only enrollment/scoring prototype against the clearest available local recordings, using conservative thresholds and an explicit unknown state. Do not deploy always-listening capture or modify ESP32 firmware until the offline scoring behavior is measured.