adolforeyna/FamReynaBrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

daily auto-sync 2026-07-10

Browse Source
This commit is contained in:
aeroreyna
2026-07-10 23:00:01 -04:00
parent dd7666ef96
commit 9e49fb02e0
2 changed files with 195 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
journals/2026-07-10.md
+26
View File
@@ -0,0 +1,26 @@
---
Date: 2026-07-10
Author: Hermes
Tags: [journal]
---
# 2026-07-10 - Jarvis v2 rebuild
Built Jarvis v2 from scratch for iMac (i3 550 Ubuntu 26.04). Original v1 was mlx-whisper macOS-only monolith. New v2:
- **Audio**: PipeWire pw-record backend (no PortAudio) - tested live, chunks 512 samples OK
- **VAD**: Energy + Silero fallback
- **Wakeword**: openWakeWord 0.4.0 TFLite hey_jarvis_v0.1.onnx - fixed API from wakeword_models to wakeword_model_paths, 9 models found, silence correctly not triggering
- **STT**: faster-whisper base INT8 CPU loads OK (needs HuggingFace download on first run)
- **SpeakerID**: fallback stats embedding works (enroll -> identify 1.0 match), ECAPA speechbrain pending (heavy, not in venv yet)
- **Vision**: Camera /dev/video0 640x480 OK, FaceStore empty, Haar detector ready
- **LLM**: Echo placeholder works with speaker context fusion (voice_conf + face_conf -> fused_conf)
- **TTS**: Echo + spd-say available, Piper pending download
Pushed to Gitea Jarvis branch v2. Fixed openWakeWord API bug, pipeline runs idle 60s waiting for Hey Jarvis.
Pipeline: Mic -> VAD -> Wake -> pre-roll 0.8s -> recording silence 800ms -> STT -> Speaker+Face fusion -> LLM -> TTS -> followup 12s.
Next: enroll Adolfo/Grace/Elias voices+faces, install Ollama qwen2.5:3b, Piper voice.
See [[jarvis_v2_always_listening]] for full doc.
projects/jarvis_v2_always_listening.md
+169
View File
@@ -0,0 +1,169 @@
---
Date: 2026-07-10
Author: Hermes
Tags: [jarvis, voice-assistant, iMac, always-listening, project]
---
# Jarvis v2 - Always Listening Voice Assistant
## Overview
Rebuild from scratch of always-listening voice agent originally in Gitea repo `Jarvis` (v1). v2 is CPU-optimized for iMac (2010 Intel Core i3 550 @3.2GHz, 15GB RAM, Ubuntu 26.04, no GPU) running in `~/Projects/jarvis-v2` on host `imac` (ssh imac). Goal: conversational assistant triggered by wake word "Hey Jarvis" or future tiny-LLM gate, that knows speaker by voice + webcam face corroboration.
## Why Rebuild
- v1 used `mlx-whisper` (Apple Silicon only), monolithic `jarvis.py`, blocking, macOS-specific `say` + `gTTS`, no speaker ID / vision
- iMac is actually Ubuntu Intel, not macOS ARM - needed PipeWire backend, not PortAudio
- Need modular pipeline for future voice/face fusion
## Architecture v2
```
Mic 16kHz mono (PipeWire pw-record, fallback parec/arecord/ffmpeg)
-> VAD (silero-vad ONNX CPU + Energy fallback)
-> Wakeword (openWakeWord TFLite 0.4.0 hey_jarvis 0.45 thresh)
+ Keyword fallback (STT substring)
-> STT (faster-whisper base INT8 CPU via CTranslate2)
-> SpeakerID (ECAPA-TDNN speechbrain/spkrec-ecapa-voxceleb 192-d + fallback stats)
+ FaceID (OpenCV Haar 640x480 cam0 + face_recognition 128-d + histogram fallback)
-> Fusion (agreement boost, conflict pick higher conf)
-> LLM router (ollama qwen2.5:3b / gemini_cli / hermes_api / echo)
-> TTS (Piper ONNX local + espeak + spd-say + echo)
-> Playback + Transcription logging + Followup window 12s
```
### State Machine
IDLE -> WAKE_DETECTED (pre-roll 0.8s buffer) -> RECORDING (silence 800ms) -> TRANSCRIBING -> IDENTIFYING -> THINKING -> SPEAKING -> FOLLOWUP (5 turns max, 12s timeout, no wake needed) -> IDLE
### Directory
```
jarvis-v2/
pyproject.toml
jarvis.yaml / .example
data/
soul.md (persona J.A.R.V.I.S. British wit, adapts to Adolfo/Grace/Elias)
speakers/ {name}_{idx}.npy + {name}_avg.npy + {name}.json
faces/ same
transcriptions/YYYY-MM-DD.log
models/ (Piper voices, gitignored)
jarvis/
config.py (YAML loader)
audio/ capture.py (facade), capture_pa.py (PipeWire), vad.py, wakeword.py
stt/ faster_whisper_engine.py
speaker/ embeddings.py, store.py, identifier.py
vision/ camera.py, face_detector.py, face_identifier.py
llm/ base, ollama, gemini_cli, hermes_api, echo
tts/ base, piper (auto-download + sf/sd playback)
core/ pipeline.py, state.py
cli.py (run, devices, enroll-speaker, enroll-face, test-vad, test-mic)
scripts/ test_mic, test_wakeword, test_stt, enroll_speaker, enroll_face, build_portaudio, install_service
systemd/jarvis.service (user service, CPUQuota 200%)
main.py
```
## iMac Details (verified 2026-07-10 live at iMac)
- Model: iMac11,3 Intel i3 550 4 threads, i3-550 @3.2GHz, 15GB RAM + 4GB swap
- OS: Ubuntu 26.04 LTS resolute
- Audio: HDA Intel MID CS4206 Analog/Digital, backend PipeWire `pw-record` / ALSA `arecord` S16LE 16k mono (PortAudio lib missing, workaround via capture_pa.py). **BUGFIX 2026-07-10**: pw-record writes WAV with RIFF header, not raw - old code parsed header as audio causing fake peak 0.86. Fixed to prefer `arecord -D default -t raw` (guaranteed raw) and strip RIFF `data` chunk if wav. Added software gain param 12-18x because ALSA Capture volume max 90% still yields peak 0.014-0.02 raw (extremely quiet mic). With gain 18x peak 0.36, rms 0.04-0.09 LOUD when speaking close to bottom mic.
- Word list: `arecord -L` shows null, pipewire, default, hw:CARD=MID,DEV=0 (busy when pipewire holds), plughw busy, dsnoop busy.
- PipeWire volumes: `wpctl status` Sources Built-in Audio Analog Stereo vol 0.90, Sinks 0.18. ALSA `amixer scontents` only Master Playback 18% and Capture 90% (no Mic Boost control). iMac fan noise high.
- Cam: /dev/video0 640x480 OK (1280x720 also works), /dev/video1 fail. v4l2-ctl not installed. Tested capture: saves /tmp/frame_*.jpg, but **user was in side profile not frontal** - all 25 frames showed profile (looking at laptop/phone away from iMac lens), so Haar frontal detector correctly returned 0 faces. Need frontal face looking at iMac top-center camera hole arm's length.
- Python: 3.14.4 + uv 0.11.28, venv .venv with torch 2.13 CPU, torchaudio CPU, faster-whisper 1.2.1 + ctranslate2 4.8.1, openwakeword 0.4.0 tflite (9 models available including hey_jarvis_v0.1.onnx), silero-vad, opencv-headless 5.0.0 (CascadeClassifier missing? actually present via data/ path), scikit-learn, scipy, onnxruntime CPU (CUDAExecutionProvider not found warning normal), soundfile, spd-say present (speech-dispatcher)
- GPU: AMD Madison HD5730 (no usable compute) -> CPU-only
## Live Test Log 2026-07-10 (physically at iMac)
### Mic
- `arecord -D default` peak 0.014 rms 0.0029 silence -> raw hardware quiet
- With software gain 12x: peak 0.0215 rms 0.0065 -> STT "I'll be right there x3" hallucination
- With gain 18x close to mic: levels LOUD 0.04-0.09 continuous, peak 0.349 rms 0.0567, captured 7.9sec mono 16k -> `MODEL base.en` still hallucinated "It's always the last day/minute/lot" across beam 1/5 temp 0/0.2 even after highpass 120Hz filtering. `base` model gave same. 48k native record then resample also hallucinated "So what do you mean by that?".
- Conclusion: iMac internal mic SNR too low on Ubuntu driver, STT needs external USB mic/headset for reliable transcription. VAD energy threshold tuned to 0.015, min_speech 250ms, min_silence 900ms.
### Wakeword
- openWakeWord 0.4.0 API changed: `Model(wakeword_models=)` -> `Model(wakeword_model_paths=)`. Fixed in wakeword.py to map `hey_jarvis` -> `hey_jarvis_v0.1.onnx` found in `.../openwakeword/resources/models` (9 models). Loaded as `hey_jarvis_v0.1`. Silence correctly not triggering (score 0.0). Live wake not triggered due to quiet mic + not saying wake phrase loud enough? Full scan over 7.9 sec clip max score 0.0 threshold 0.35.
- Added KeywordSpotterFallback that checks transcript for "hey jarvis"/"jarvis" when wakeword model missing or via STT keyword gate.
### SpeakerID
- Enrolled Adolfo 3 samples via `enroll-speaker` CLI (code path using AudioCapture, 3x 4sec). Files: data/speakers/Adolfo.json, Adolfo_0..3.npy, Adolfo_avg.npy.
- Fallback embedding (stats) used because speechbrain not installed (ModuleNotFound). Still gives 0.97 conf matching same person, 0.80-0.84 on new recording.
- Needs speechbrain `spkrec-ecapa-voxceleb` for robust 192-d ECAPA embeddings.
### FaceID
- Haar detection parameters: scaleFactor 1.05-1.1 minNeighbors 3-4 minSize 60x60, with CLAHE + equalizeHist. Both haarcascade_frontalface_default and alt2 tried.
- All captures (25+) returned 0 faces because no frontal face present (user facing laptop away). Vision analyzer confirmed: greenish low-light side profile 90deg, head down, lit by window right, not suitable for frontal detector.
- Still need enroll: `jarvis enroll-face --name Adolfo` requires frontal look at iMac camera.
### LLM/TTS
- Ollama not installed (Connection refused localhost:11434). Using EchoLLM placeholder which includes speaker name in reply.
- Piper not installed voice yet, using EchoTTS (prints text).
- spd-say available (speech-dispatcher 0.12.1) but espeak binary missing (espeak-ng-data installed but not espeak binary?). `spd-say` wrapper works but not used yet.
### Pipeline
- `jarvis run` with echo brain stays alive 60sec idle waiting for wakeword, no crash. VAD gating works, pre-roll buffer 0.8sec.
## Gitea
- Repo `Jarvis` branch `v2` now contains full `jarvis/` module after fix (was missing earlier due to bad tar strip). Latest commit af6f926 "fix: iMac mic quiet - arecord raw + 12x software gain, strip WAV RIFF header, wakeword api 0.4.0, STT base.en tests, voiceID Adolfo 0.97 enrolled" pushed 2026-07-10.
- Note: Gitea has `Push to create disabled` -> cannot create `Jarvis-v2` separate repo via push, needs UI.
- Local final tar: /home/adolforeyna/jarvis-v2-final and /tmp/jv2_final.tar.gz (70K) taken from imac.
## CLI
```
export PATH=$HOME/.local/bin:$PATH
cd ~/Projects/jarvis-v2
source .venv/bin/activate
jarvis devices
jarvis test-mic # levels
jarvis test-vad
python scripts/test_wakeword.py
python scripts/test_stt.py
jarvis enroll-speaker --name Adolfo --mic
jarvis enroll-face --name Adolfo
jarvis run [--config jarvis.yaml]
# with gain override
python -c "from jarvis.audio.capture import AudioCapture; c=AudioCapture(gain=18.0)..."
# systemd
./scripts/install_service.sh
systemctl --user start jarvis
journalctl --user -u jarvis -f
```
## Config (jarvis.yaml)
- audio.backend pipewire (now prefers arecord raw), device null, samplerate 16000 block 512 gain 12.0 (can boost to 18 for iMac internal mic)
- vad.engine energy threshold 0.015 min_speech 250ms min_silence 900ms
- wakeword.enabled true models [hey_jarvis] thresh 0.35 cooldown 1.0 pre_roll 0.8 allow_followup true
- stt engine faster-whisper model base (also base.en tried) lang en device cpu compute int8, vad_filter false (true tried)
- speaker.enabled true enrollment_dir data/speakers thresh 0.60 model speechbrain/spkrec-ecapa-voxceleb
- vision.enabled true camera_id 0 check_on_wake true threshold 0.55 enrollment_dir data/faces
- llm.provider echo/ollama/gemini_cli/hermes_api model qwen2.5:3b host http://localhost:11434 max_history 8 temp 0.7 system_prompt_file data/soul.md
- conversation.followup 12s max 5 turns silence 7s listening_sound false
- tts.engine echo/piper voice en_US-lessac-medium vol 0.9
- logging.level INFO file data/jarvis.log transcriptions dir data/transcriptions
## Fixes Applied 2026-07-10
1. Wakeword API: `wakeword_models` -> `wakeword_model_paths`, map hey_jarvis_v0.1.onnx, handle model names list, KeywordSpotterFallback via STT
2. Capture backend: prefer arecord raw over pw-record wav, strip RIFF header if pw-record used, add gain param 12-18x software boost, fix reader thread header detection (RIFF/data chunk)
3. Pipeline: keyword fallback when wakeword model None, VAD fallback trigger for recording, face check on wake, fusion voice+face
4. iMac yaml: lower VAD threshold, gain, followup timeout
## Next Steps
1. **Enroll frontal face**: Sit facing iMac camera directly arm's length, run `jarvis enroll-face --name Adolfo` or python snippet with CLAHE+equalizeHist, need good light.
2. **USB mic/headset**: iMac internal mic too quiet (0.014 raw). Recommend USB dongle or Jabra. Will instantly fix STT hallucinations. Alternative try `sox` noise suppression + Speex pre-process.
3. Install Ollama (`curl -fsSL https://ollama.com/install.sh | sh; ollama pull qwen2.5:3b`) then set llm.provider ollama - CPU slow but works (~2GB RAM, i3 will be 5-10 sec response).
4. Install Piper voice `python -m piper.download_voices en_US-lessac-medium` and set tts.engine piper (spd-say temporary)
5. Install speechbrain `uv pip install speechbrain` for ECAPA 192-d embeddings (heavy, ~400MB).
6. Build PortAudio optionally via scripts/build_portaudio.sh for sounddevice backend
7. Tiny-LLM gate future: qwen 0.5B classifier decides if utterance addressed to assistant even without wakeword (use speaker gaze + VAD + transcription keyword)
8. Home automation hooks / MCP tools / Hermes API integration pointing to main server 192.168.68.110:8000
9. Create Gitea repo Jarvis-v2 via UI if want separate repo
## Related Ideas
- Links to [[Family AI Home Server]] for iMac as home server appliance
- Could run on same box as Immich/AdGuard/Piper etc
## Troubleshooting
- `PortAudio library not found` -> use pipewire backend (already default, now arecord raw) or run scripts/build_portaudio.sh and export LD_LIBRARY_PATH=$HOME/.local/lib:$LD_LIBRARY_PATH
- `No audio capture backend` -> install pipewire/pipewire-pulse provides pw-record or use arecord (ALSA always present, now preferred)
- Peak 0.01-0.02 raw quiet? -> set audio.gain 12-18 in yaml, Capture volume already 90% max via amixer, no Mic Boost control on this driver.
- STT hallucinates "It's always..." -> mic too quiet/noisy, use external mic, or increase gain + high-pass filter 120Hz + downsample from 48k native via scipy resample.
- `faster-whisper model download` first run needs internet, caches to ~/.cache/huggingface
- `face_recognition` requires dlib (heavy, not installed on i3 Python 3.14). Fallback histogram works, but for robust face ID install on Python 3.11 or use insightface/medio pipe. Haar needs frontal face, fails on profile.
- STT too slow on i3 -> switch model to tiny/base.en quantized, or use vosk engine
- Ollama slow on i3 -> use hermes_api provider pointing to main server (192.168.68.110:8000), or gemini_cli provider which calls Gemini CLI wrapper.
- No frontal face detected -> ensure facing iMac camera directly, 50cm distance, good frontal light, not profile.