FamReynaBrain/projects/family_voice_context_and_speaker_id_2026.md

date, type, status, tags

date	type	status	tags
2026-08-08	project	active	voice speaker-identification family esp32 pipecat local-ai privacy

Family Voice Context and Speaker Identification 2026

Idea

Use the family ESP32 voice devices as low-power local audio sensors. While idle, each device measures microphone energy / lightweight VAD, starts a short recording when speech is detected, and sends the clip over the private LAN for local processing.

The Mac mini should perform accurate VAD, transcription, speaker diarization, and speaker verification. Initial target: predict whether Adolfo's voice is present in a short clip, using clear WhatsApp voice recordings as enrollment candidates. Later extend to best-effort profiles for Adolfo, Alicia, Grace, and Elias, always allowing unknown and uncertain results.

Privacy and retention policy

• All processing remains local; no cloud transcription, embeddings, or raw audio uploads.
• ESP32 devices must have a visible recording indicator and physical mute behavior.
• Raw audio should be temporary and deleted after successful processing by default.
• Do not automatically write every conversation into the canonical brain.
• Keep a separate short-retention conversation log; promote only confirmed durable facts, tasks, or decisions into the brain.
• Treat medical, financial, intimate, and private conversations as excluded or review-only.
• Never use speaker recognition as sole authorization for high-impact actions.

Initial prototype

Build a small local speaker-verification service that:

1. Finds and labels candidate Adolfo voice clips from local WhatsApp/ESP32 audio cache.
2. Creates a speaker embedding profile from approved enrollment clips.
3. Scores short WAV clips as adolfo, not_adolfo, or unknown.
4. Reports cosine similarity, duration, source/device, and model version.
5. Keeps raw recordings out of Git/Gitea and stores only code, metadata, and derived profile artifacts when explicitly approved.
6. Provides a small adapter boundary suitable for a future Pipecat pipeline after VAD and STT, before LLM/context injection.

Candidate architecture

ESP32 mic -> energy/adaptive VAD + pre-roll -> LAN clip upload
  -> Mac mini voice service -> accurate VAD -> local STT
  -> speaker embedding / diarization -> local transcript log
  -> optional reviewed context retrieval -> Hermes / brain suggestion

Decisions still open

• Exact source labeling for the cached WhatsApp/ESP32 WAV files.
• ECAPA-TDNN/SpeechBrain versus a lighter embedding model on Mac mini.
• Whether profiles should live in a private ignored data directory or encrypted local store.
• Pipecat integration point: audio clip preprocessor or post-STT speaker metadata processor.
• Family consent, retention period, and device-room recording policy before multi-person rollout.

Next action

Run the Adolfo-only enrollment/scoring prototype against the clearest available local recordings, using conservative thresholds and an explicit unknown state. Do not deploy always-listening capture or modify ESP32 firmware until the offline scoring behavior is measured.

Prototype started 2026-08-08

Local project: ~/Projects/voice-speaker-id. It uses a pretrained SpeechBrain ECAPA speaker encoder, manual clip review, and offline CLI scoring. The safe first test requires explicit approval of clips before enrollment. No HTTP listener, microphone capture, Pipecat integration, ESP32 firmware change, or automatic brain ingestion is enabled.