diff --git a/index.md b/index.md index c006028..66fe6b9 100644 --- a/index.md +++ b/index.md @@ -15,6 +15,7 @@ - [[Reyna CLI Browser Control]]: Add remote browser-control automations, starting with Gemini Web image generation, into `reyna-cli`. - [[Voice Agent Platform Research 2026]]: Compare open-gpt-live (WS+VAD) + StreamCore (Go+WebRTC+plugins+RAG+ESP32) for Jarvis v3 / Hermes voice; fork StreamCore recommended. - [[Jarvis v2 - Always Listening Voice Assistant]]: Current iMac local wakeword+speakerID+faceID pipeline; to be extended with OGL VAD cal + StreamCore chassis. + - [[Family Voice Context and Speaker Identification 2026]]: Local ESP32-triggered audio context, Adolfo voice verification, and later family speaker identification. - **[[Areas]]:** Ongoing responsibilities and household maintenance. - **[[Resources]]:** Interests, reference material, and research. - [[Family Timeline]]: Chronological milestones. diff --git a/projects/family_voice_context_and_speaker_id_2026.md b/projects/family_voice_context_and_speaker_id_2026.md new file mode 100644 index 0000000..dc54922 --- /dev/null +++ b/projects/family_voice_context_and_speaker_id_2026.md @@ -0,0 +1,56 @@ +--- +date: 2026-08-08 +type: project +status: active +tags: [voice, speaker-identification, family, esp32, pipecat, local-ai, privacy] +--- + +# Family Voice Context and Speaker Identification 2026 + +## Idea + +Use the family ESP32 voice devices as low-power local audio sensors. While idle, each device measures microphone energy / lightweight VAD, starts a short recording when speech is detected, and sends the clip over the private LAN for local processing. + +The Mac mini should perform accurate VAD, transcription, speaker diarization, and speaker verification. Initial target: predict whether Adolfo's voice is present in a short clip, using clear WhatsApp voice recordings as enrollment candidates. Later extend to best-effort profiles for Adolfo, Alicia, Grace, and Elias, always allowing `unknown` and `uncertain` results. + +## Privacy and retention policy + +- All processing remains local; no cloud transcription, embeddings, or raw audio uploads. +- ESP32 devices must have a visible recording indicator and physical mute behavior. +- Raw audio should be temporary and deleted after successful processing by default. +- Do not automatically write every conversation into the canonical brain. +- Keep a separate short-retention conversation log; promote only confirmed durable facts, tasks, or decisions into the brain. +- Treat medical, financial, intimate, and private conversations as excluded or review-only. +- Never use speaker recognition as sole authorization for high-impact actions. + +## Initial prototype + +Build a small local speaker-verification service that: + +1. Finds and labels candidate Adolfo voice clips from local WhatsApp/ESP32 audio cache. +2. Creates a speaker embedding profile from approved enrollment clips. +3. Scores short WAV clips as `adolfo`, `not_adolfo`, or `unknown`. +4. Reports cosine similarity, duration, source/device, and model version. +5. Keeps raw recordings out of Git/Gitea and stores only code, metadata, and derived profile artifacts when explicitly approved. +6. Provides a small adapter boundary suitable for a future Pipecat pipeline after VAD and STT, before LLM/context injection. + +## Candidate architecture + +```text +ESP32 mic -> energy/adaptive VAD + pre-roll -> LAN clip upload + -> Mac mini voice service -> accurate VAD -> local STT + -> speaker embedding / diarization -> local transcript log + -> optional reviewed context retrieval -> Hermes / brain suggestion +``` + +## Decisions still open + +- Exact source labeling for the cached WhatsApp/ESP32 WAV files. +- ECAPA-TDNN/SpeechBrain versus a lighter embedding model on Mac mini. +- Whether profiles should live in a private ignored data directory or encrypted local store. +- Pipecat integration point: audio clip preprocessor or post-STT speaker metadata processor. +- Family consent, retention period, and device-room recording policy before multi-person rollout. + +## Next action + +Run the Adolfo-only enrollment/scoring prototype against the clearest available local recordings, using conservative thresholds and an explicit unknown state. Do not deploy always-listening capture or modify ESP32 firmware until the offline scoring behavior is measured.