diff --git a/journals/2026-07-10.md b/journals/2026-07-10.md new file mode 100644 index 0000000..b92b528 --- /dev/null +++ b/journals/2026-07-10.md @@ -0,0 +1,26 @@ +--- +Date: 2026-07-10 +Author: Hermes +Tags: [journal] +--- + +# 2026-07-10 - Jarvis v2 rebuild + +Built Jarvis v2 from scratch for iMac (i3 550 Ubuntu 26.04). Original v1 was mlx-whisper macOS-only monolith. New v2: + +- **Audio**: PipeWire pw-record backend (no PortAudio) - tested live, chunks 512 samples OK +- **VAD**: Energy + Silero fallback +- **Wakeword**: openWakeWord 0.4.0 TFLite hey_jarvis_v0.1.onnx - fixed API from wakeword_models to wakeword_model_paths, 9 models found, silence correctly not triggering +- **STT**: faster-whisper base INT8 CPU loads OK (needs HuggingFace download on first run) +- **SpeakerID**: fallback stats embedding works (enroll -> identify 1.0 match), ECAPA speechbrain pending (heavy, not in venv yet) +- **Vision**: Camera /dev/video0 640x480 OK, FaceStore empty, Haar detector ready +- **LLM**: Echo placeholder works with speaker context fusion (voice_conf + face_conf -> fused_conf) +- **TTS**: Echo + spd-say available, Piper pending download + +Pushed to Gitea Jarvis branch v2. Fixed openWakeWord API bug, pipeline runs idle 60s waiting for Hey Jarvis. + +Pipeline: Mic -> VAD -> Wake -> pre-roll 0.8s -> recording silence 800ms -> STT -> Speaker+Face fusion -> LLM -> TTS -> followup 12s. + +Next: enroll Adolfo/Grace/Elias voices+faces, install Ollama qwen2.5:3b, Piper voice. + +See [[jarvis_v2_always_listening]] for full doc. diff --git a/projects/jarvis_v2_always_listening.md b/projects/jarvis_v2_always_listening.md new file mode 100644 index 0000000..13f0b03 --- /dev/null +++ b/projects/jarvis_v2_always_listening.md @@ -0,0 +1,169 @@ +--- +Date: 2026-07-10 +Author: Hermes +Tags: [jarvis, voice-assistant, iMac, always-listening, project] +--- + +# Jarvis v2 - Always Listening Voice Assistant + +## Overview +Rebuild from scratch of always-listening voice agent originally in Gitea repo `Jarvis` (v1). v2 is CPU-optimized for iMac (2010 Intel Core i3 550 @3.2GHz, 15GB RAM, Ubuntu 26.04, no GPU) running in `~/Projects/jarvis-v2` on host `imac` (ssh imac). Goal: conversational assistant triggered by wake word "Hey Jarvis" or future tiny-LLM gate, that knows speaker by voice + webcam face corroboration. + +## Why Rebuild +- v1 used `mlx-whisper` (Apple Silicon only), monolithic `jarvis.py`, blocking, macOS-specific `say` + `gTTS`, no speaker ID / vision +- iMac is actually Ubuntu Intel, not macOS ARM - needed PipeWire backend, not PortAudio +- Need modular pipeline for future voice/face fusion + +## Architecture v2 +``` +Mic 16kHz mono (PipeWire pw-record, fallback parec/arecord/ffmpeg) + -> VAD (silero-vad ONNX CPU + Energy fallback) + -> Wakeword (openWakeWord TFLite 0.4.0 hey_jarvis 0.45 thresh) + + Keyword fallback (STT substring) + -> STT (faster-whisper base INT8 CPU via CTranslate2) + -> SpeakerID (ECAPA-TDNN speechbrain/spkrec-ecapa-voxceleb 192-d + fallback stats) + + FaceID (OpenCV Haar 640x480 cam0 + face_recognition 128-d + histogram fallback) + -> Fusion (agreement boost, conflict pick higher conf) + -> LLM router (ollama qwen2.5:3b / gemini_cli / hermes_api / echo) + -> TTS (Piper ONNX local + espeak + spd-say + echo) + -> Playback + Transcription logging + Followup window 12s +``` + +### State Machine +IDLE -> WAKE_DETECTED (pre-roll 0.8s buffer) -> RECORDING (silence 800ms) -> TRANSCRIBING -> IDENTIFYING -> THINKING -> SPEAKING -> FOLLOWUP (5 turns max, 12s timeout, no wake needed) -> IDLE + +### Directory +``` +jarvis-v2/ + pyproject.toml + jarvis.yaml / .example + data/ + soul.md (persona J.A.R.V.I.S. British wit, adapts to Adolfo/Grace/Elias) + speakers/ {name}_{idx}.npy + {name}_avg.npy + {name}.json + faces/ same + transcriptions/YYYY-MM-DD.log + models/ (Piper voices, gitignored) + jarvis/ + config.py (YAML loader) + audio/ capture.py (facade), capture_pa.py (PipeWire), vad.py, wakeword.py + stt/ faster_whisper_engine.py + speaker/ embeddings.py, store.py, identifier.py + vision/ camera.py, face_detector.py, face_identifier.py + llm/ base, ollama, gemini_cli, hermes_api, echo + tts/ base, piper (auto-download + sf/sd playback) + core/ pipeline.py, state.py + cli.py (run, devices, enroll-speaker, enroll-face, test-vad, test-mic) + scripts/ test_mic, test_wakeword, test_stt, enroll_speaker, enroll_face, build_portaudio, install_service + systemd/jarvis.service (user service, CPUQuota 200%) + main.py +``` + +## iMac Details (verified 2026-07-10 live at iMac) +- Model: iMac11,3 Intel i3 550 4 threads, i3-550 @3.2GHz, 15GB RAM + 4GB swap +- OS: Ubuntu 26.04 LTS resolute +- Audio: HDA Intel MID CS4206 Analog/Digital, backend PipeWire `pw-record` / ALSA `arecord` S16LE 16k mono (PortAudio lib missing, workaround via capture_pa.py). **BUGFIX 2026-07-10**: pw-record writes WAV with RIFF header, not raw - old code parsed header as audio causing fake peak 0.86. Fixed to prefer `arecord -D default -t raw` (guaranteed raw) and strip RIFF `data` chunk if wav. Added software gain param 12-18x because ALSA Capture volume max 90% still yields peak 0.014-0.02 raw (extremely quiet mic). With gain 18x peak 0.36, rms 0.04-0.09 LOUD when speaking close to bottom mic. +- Word list: `arecord -L` shows null, pipewire, default, hw:CARD=MID,DEV=0 (busy when pipewire holds), plughw busy, dsnoop busy. +- PipeWire volumes: `wpctl status` Sources Built-in Audio Analog Stereo vol 0.90, Sinks 0.18. ALSA `amixer scontents` only Master Playback 18% and Capture 90% (no Mic Boost control). iMac fan noise high. +- Cam: /dev/video0 640x480 OK (1280x720 also works), /dev/video1 fail. v4l2-ctl not installed. Tested capture: saves /tmp/frame_*.jpg, but **user was in side profile not frontal** - all 25 frames showed profile (looking at laptop/phone away from iMac lens), so Haar frontal detector correctly returned 0 faces. Need frontal face looking at iMac top-center camera hole arm's length. +- Python: 3.14.4 + uv 0.11.28, venv .venv with torch 2.13 CPU, torchaudio CPU, faster-whisper 1.2.1 + ctranslate2 4.8.1, openwakeword 0.4.0 tflite (9 models available including hey_jarvis_v0.1.onnx), silero-vad, opencv-headless 5.0.0 (CascadeClassifier missing? actually present via data/ path), scikit-learn, scipy, onnxruntime CPU (CUDAExecutionProvider not found warning normal), soundfile, spd-say present (speech-dispatcher) +- GPU: AMD Madison HD5730 (no usable compute) -> CPU-only + +## Live Test Log 2026-07-10 (physically at iMac) + +### Mic +- `arecord -D default` peak 0.014 rms 0.0029 silence -> raw hardware quiet +- With software gain 12x: peak 0.0215 rms 0.0065 -> STT "I'll be right there x3" hallucination +- With gain 18x close to mic: levels LOUD 0.04-0.09 continuous, peak 0.349 rms 0.0567, captured 7.9sec mono 16k -> `MODEL base.en` still hallucinated "It's always the last day/minute/lot" across beam 1/5 temp 0/0.2 even after highpass 120Hz filtering. `base` model gave same. 48k native record then resample also hallucinated "So what do you mean by that?". +- Conclusion: iMac internal mic SNR too low on Ubuntu driver, STT needs external USB mic/headset for reliable transcription. VAD energy threshold tuned to 0.015, min_speech 250ms, min_silence 900ms. + +### Wakeword +- openWakeWord 0.4.0 API changed: `Model(wakeword_models=)` -> `Model(wakeword_model_paths=)`. Fixed in wakeword.py to map `hey_jarvis` -> `hey_jarvis_v0.1.onnx` found in `.../openwakeword/resources/models` (9 models). Loaded as `hey_jarvis_v0.1`. Silence correctly not triggering (score 0.0). Live wake not triggered due to quiet mic + not saying wake phrase loud enough? Full scan over 7.9 sec clip max score 0.0 threshold 0.35. +- Added KeywordSpotterFallback that checks transcript for "hey jarvis"/"jarvis" when wakeword model missing or via STT keyword gate. + +### SpeakerID +- Enrolled Adolfo 3 samples via `enroll-speaker` CLI (code path using AudioCapture, 3x 4sec). Files: data/speakers/Adolfo.json, Adolfo_0..3.npy, Adolfo_avg.npy. +- Fallback embedding (stats) used because speechbrain not installed (ModuleNotFound). Still gives 0.97 conf matching same person, 0.80-0.84 on new recording. +- Needs speechbrain `spkrec-ecapa-voxceleb` for robust 192-d ECAPA embeddings. + +### FaceID +- Haar detection parameters: scaleFactor 1.05-1.1 minNeighbors 3-4 minSize 60x60, with CLAHE + equalizeHist. Both haarcascade_frontalface_default and alt2 tried. +- All captures (25+) returned 0 faces because no frontal face present (user facing laptop away). Vision analyzer confirmed: greenish low-light side profile 90deg, head down, lit by window right, not suitable for frontal detector. +- Still need enroll: `jarvis enroll-face --name Adolfo` requires frontal look at iMac camera. + +### LLM/TTS +- Ollama not installed (Connection refused localhost:11434). Using EchoLLM placeholder which includes speaker name in reply. +- Piper not installed voice yet, using EchoTTS (prints text). +- spd-say available (speech-dispatcher 0.12.1) but espeak binary missing (espeak-ng-data installed but not espeak binary?). `spd-say` wrapper works but not used yet. + +### Pipeline +- `jarvis run` with echo brain stays alive 60sec idle waiting for wakeword, no crash. VAD gating works, pre-roll buffer 0.8sec. + +## Gitea +- Repo `Jarvis` branch `v2` now contains full `jarvis/` module after fix (was missing earlier due to bad tar strip). Latest commit af6f926 "fix: iMac mic quiet - arecord raw + 12x software gain, strip WAV RIFF header, wakeword api 0.4.0, STT base.en tests, voiceID Adolfo 0.97 enrolled" pushed 2026-07-10. +- Note: Gitea has `Push to create disabled` -> cannot create `Jarvis-v2` separate repo via push, needs UI. +- Local final tar: /home/adolforeyna/jarvis-v2-final and /tmp/jv2_final.tar.gz (70K) taken from imac. + +## CLI +``` +export PATH=$HOME/.local/bin:$PATH +cd ~/Projects/jarvis-v2 +source .venv/bin/activate +jarvis devices +jarvis test-mic # levels +jarvis test-vad +python scripts/test_wakeword.py +python scripts/test_stt.py +jarvis enroll-speaker --name Adolfo --mic +jarvis enroll-face --name Adolfo +jarvis run [--config jarvis.yaml] +# with gain override +python -c "from jarvis.audio.capture import AudioCapture; c=AudioCapture(gain=18.0)..." +# systemd +./scripts/install_service.sh +systemctl --user start jarvis +journalctl --user -u jarvis -f +``` + +## Config (jarvis.yaml) +- audio.backend pipewire (now prefers arecord raw), device null, samplerate 16000 block 512 gain 12.0 (can boost to 18 for iMac internal mic) +- vad.engine energy threshold 0.015 min_speech 250ms min_silence 900ms +- wakeword.enabled true models [hey_jarvis] thresh 0.35 cooldown 1.0 pre_roll 0.8 allow_followup true +- stt engine faster-whisper model base (also base.en tried) lang en device cpu compute int8, vad_filter false (true tried) +- speaker.enabled true enrollment_dir data/speakers thresh 0.60 model speechbrain/spkrec-ecapa-voxceleb +- vision.enabled true camera_id 0 check_on_wake true threshold 0.55 enrollment_dir data/faces +- llm.provider echo/ollama/gemini_cli/hermes_api model qwen2.5:3b host http://localhost:11434 max_history 8 temp 0.7 system_prompt_file data/soul.md +- conversation.followup 12s max 5 turns silence 7s listening_sound false +- tts.engine echo/piper voice en_US-lessac-medium vol 0.9 +- logging.level INFO file data/jarvis.log transcriptions dir data/transcriptions + +## Fixes Applied 2026-07-10 +1. Wakeword API: `wakeword_models` -> `wakeword_model_paths`, map hey_jarvis_v0.1.onnx, handle model names list, KeywordSpotterFallback via STT +2. Capture backend: prefer arecord raw over pw-record wav, strip RIFF header if pw-record used, add gain param 12-18x software boost, fix reader thread header detection (RIFF/data chunk) +3. Pipeline: keyword fallback when wakeword model None, VAD fallback trigger for recording, face check on wake, fusion voice+face +4. iMac yaml: lower VAD threshold, gain, followup timeout + +## Next Steps +1. **Enroll frontal face**: Sit facing iMac camera directly arm's length, run `jarvis enroll-face --name Adolfo` or python snippet with CLAHE+equalizeHist, need good light. +2. **USB mic/headset**: iMac internal mic too quiet (0.014 raw). Recommend USB dongle or Jabra. Will instantly fix STT hallucinations. Alternative try `sox` noise suppression + Speex pre-process. +3. Install Ollama (`curl -fsSL https://ollama.com/install.sh | sh; ollama pull qwen2.5:3b`) then set llm.provider ollama - CPU slow but works (~2GB RAM, i3 will be 5-10 sec response). +4. Install Piper voice `python -m piper.download_voices en_US-lessac-medium` and set tts.engine piper (spd-say temporary) +5. Install speechbrain `uv pip install speechbrain` for ECAPA 192-d embeddings (heavy, ~400MB). +6. Build PortAudio optionally via scripts/build_portaudio.sh for sounddevice backend +7. Tiny-LLM gate future: qwen 0.5B classifier decides if utterance addressed to assistant even without wakeword (use speaker gaze + VAD + transcription keyword) +8. Home automation hooks / MCP tools / Hermes API integration pointing to main server 192.168.68.110:8000 +9. Create Gitea repo Jarvis-v2 via UI if want separate repo + +## Related Ideas +- Links to [[Family AI Home Server]] for iMac as home server appliance +- Could run on same box as Immich/AdGuard/Piper etc + +## Troubleshooting +- `PortAudio library not found` -> use pipewire backend (already default, now arecord raw) or run scripts/build_portaudio.sh and export LD_LIBRARY_PATH=$HOME/.local/lib:$LD_LIBRARY_PATH +- `No audio capture backend` -> install pipewire/pipewire-pulse provides pw-record or use arecord (ALSA always present, now preferred) +- Peak 0.01-0.02 raw quiet? -> set audio.gain 12-18 in yaml, Capture volume already 90% max via amixer, no Mic Boost control on this driver. +- STT hallucinates "It's always..." -> mic too quiet/noisy, use external mic, or increase gain + high-pass filter 120Hz + downsample from 48k native via scipy resample. +- `faster-whisper model download` first run needs internet, caches to ~/.cache/huggingface +- `face_recognition` requires dlib (heavy, not installed on i3 Python 3.14). Fallback histogram works, but for robust face ID install on Python 3.11 or use insightface/medio pipe. Haar needs frontal face, fails on profile. +- STT too slow on i3 -> switch model to tiny/base.en quantized, or use vosk engine +- Ollama slow on i3 -> use hermes_api provider pointing to main server (192.168.68.110:8000), or gemini_cli provider which calls Gemini CLI wrapper. +- No frontal face detected -> ensure facing iMac camera directly, 50cm distance, good frontal light, not profile.