document family voice context project

index.md

@@ -15,6 +15,7 @@
   - [[Reyna CLI Browser Control]]: Add remote browser-control automations, starting with Gemini Web image generation, into `reyna-cli`.
   - [[Voice Agent Platform Research 2026]]: Compare open-gpt-live (WS+VAD) + StreamCore (Go+WebRTC+plugins+RAG+ESP32) for Jarvis v3 / Hermes voice; fork StreamCore recommended.
   - [[Jarvis v2 - Always Listening Voice Assistant]]: Current iMac local wakeword+speakerID+faceID pipeline; to be extended with OGL VAD cal + StreamCore chassis.
+  - [[Family Voice Context and Speaker Identification 2026]]: Local ESP32-triggered audio context, Adolfo voice verification, and later family speaker identification.
 - **[[Areas]]:** Ongoing responsibilities and household maintenance.
 - **[[Resources]]:** Interests, reference material, and research.
   - [[Family Timeline]]: Chronological milestones.

projects/family_voice_context_and_speaker_id_2026.md

@@ -0,0 +1,56 @@
+---
+date: 2026-08-08
+type: project
+status: active
+tags: [voice, speaker-identification, family, esp32, pipecat, local-ai, privacy]
+---
+
+# Family Voice Context and Speaker Identification 2026
+
+## Idea
+
+Use the family ESP32 voice devices as low-power local audio sensors. While idle, each device measures microphone energy / lightweight VAD, starts a short recording when speech is detected, and sends the clip over the private LAN for local processing.
+
+The Mac mini should perform accurate VAD, transcription, speaker diarization, and speaker verification. Initial target: predict whether Adolfo's voice is present in a short clip, using clear WhatsApp voice recordings as enrollment candidates. Later extend to best-effort profiles for Adolfo, Alicia, Grace, and Elias, always allowing `unknown` and `uncertain` results.
+
+## Privacy and retention policy
+
+- All processing remains local; no cloud transcription, embeddings, or raw audio uploads.
+- ESP32 devices must have a visible recording indicator and physical mute behavior.
+- Raw audio should be temporary and deleted after successful processing by default.
+- Do not automatically write every conversation into the canonical brain.
+- Keep a separate short-retention conversation log; promote only confirmed durable facts, tasks, or decisions into the brain.
+- Treat medical, financial, intimate, and private conversations as excluded or review-only.
+- Never use speaker recognition as sole authorization for high-impact actions.
+
+## Initial prototype
+
+Build a small local speaker-verification service that:
+
+1. Finds and labels candidate Adolfo voice clips from local WhatsApp/ESP32 audio cache.
+2. Creates a speaker embedding profile from approved enrollment clips.
+3. Scores short WAV clips as `adolfo`, `not_adolfo`, or `unknown`.
+4. Reports cosine similarity, duration, source/device, and model version.
+5. Keeps raw recordings out of Git/Gitea and stores only code, metadata, and derived profile artifacts when explicitly approved.
+6. Provides a small adapter boundary suitable for a future Pipecat pipeline after VAD and STT, before LLM/context injection.
+
+## Candidate architecture
+
+```text
+ESP32 mic -> energy/adaptive VAD + pre-roll -> LAN clip upload
+  -> Mac mini voice service -> accurate VAD -> local STT
+  -> speaker embedding / diarization -> local transcript log
+  -> optional reviewed context retrieval -> Hermes / brain suggestion
+```
+
+## Decisions still open
+
+- Exact source labeling for the cached WhatsApp/ESP32 WAV files.
+- ECAPA-TDNN/SpeechBrain versus a lighter embedding model on Mac mini.
+- Whether profiles should live in a private ignored data directory or encrypted local store.
+- Pipecat integration point: audio clip preprocessor or post-STT speaker metadata processor.
+- Family consent, retention period, and device-room recording policy before multi-person rollout.
+
+## Next action
+
+Run the Adolfo-only enrollment/scoring prototype against the clearest available local recordings, using conservative thresholds and an explicit unknown state. Do not deploy always-listening capture or modify ESP32 firmware until the offline scoring behavior is measured.