Adolfo Reyna
88 lines
3.5 KiB
Markdown
88 lines
3.5 KiB
Markdown
# Jarvis v2 - Final Setup (iMac Intel i3 Ubuntu 26.04)
|
|
|
|
## What we built vs v1
|
|
v1 (Jarvis repo main):
|
|
- monolithic jarvis.py, macOS only, mlx-whisper (Apple Silicon), openWakeWord, gTTS, say command, Gemini CLI
|
|
- No speaker ID, no vision, blocking pipeline
|
|
|
|
v2 (Jarvis repo -> branch v2):
|
|
- Modular: audio/capture (PipeWire pw-record backend, no PortAudio needed), VAD (silero + Energy fallback), wakeword (openWakeWord TFLite hey_jarvis 0.4), STT (faster-whisper base INT8 CPU), SpeakerID (ECAPA-TDNN speechbrain + stats fallback), Vision (OpenCV Haar + face_recognition), LLM pluggable (ollama qwen2.5:3b / gemini_cli / hermes_api / echo), TTS (Piper ONNX + espeak + echo)
|
|
- Pipeline state machine: IDLE -> WAKE -> RECORDING (with pre-roll 0.8s) -> TRANSCRIBING -> IDENTIFYING (voice+face fusion) -> THINKING -> SPEAKING -> FOLLOWUP window (12s no wake needed)
|
|
- Tested on iMac: mic via pw-record 16k mono OK, VAD energy OK, faster-whisper base int8 loads, cam0 640x480 OK, SpeakerStore empty (ready to enroll), Echo LLM/TTS OK
|
|
|
|
## iMac paths
|
|
~/Projects/jarvis-v2
|
|
.venv/ (uv, python 3.14, torch CPU, faster-whisper, openwakeword, opencv-headless etc)
|
|
jarvis.yaml (active config) + jarvis.yaml.example
|
|
data/soul.md (persona)
|
|
data/speakers/ (enroll .npy + .json)
|
|
data/faces/ (enroll .npy + .json)
|
|
data/transcriptions/YYYY-MM-DD.log
|
|
models/ (Piper voices, gitignored)
|
|
systemd/jarvis.service (user service)
|
|
|
|
## Commands on iMac
|
|
export PATH=$HOME/.local/bin:$PATH
|
|
cd ~/Projects/jarvis-v2
|
|
source .venv/bin/activate
|
|
|
|
# List audio
|
|
jarvis devices
|
|
|
|
# Mic level test
|
|
jarvis test-mic
|
|
# VAD test
|
|
jarvis test-vad
|
|
|
|
# Low-level wav test
|
|
python scripts/test_mic.py
|
|
python scripts/test_wakeword.py
|
|
python scripts/test_stt.py
|
|
|
|
# Enroll voices (do 3 samples per person for robustness, quiet room)
|
|
jarvis enroll-speaker --name Adolfo --mic
|
|
jarvis enroll-speaker --name Grace --mic
|
|
jarvis enroll-speaker --name Elias --mic
|
|
# or from file: jarvis enroll-speaker --name Adolfo --file /path/to/wav (16k mono)
|
|
|
|
# Enroll faces
|
|
jarvis enroll-face --name Adolfo
|
|
jarvis enroll-face --name Grace
|
|
jarvis enroll-face --name Elias
|
|
|
|
# Run assistant (echo brain - placeholder)
|
|
jarvis run
|
|
# Later with ollama
|
|
# install ollama: curl -fsSL https://ollama.com/install.sh | sh
|
|
# ollama pull qwen2.5:3b
|
|
# edit jarvis.yaml llm.provider=ollama
|
|
# jarvis run
|
|
|
|
# Systemd always-on
|
|
./scripts/install_service.sh
|
|
systemctl --user start jarvis
|
|
journalctl --user -u jarvis -f
|
|
systemctl --user stop jarvis
|
|
systemctl --user disable jarvis
|
|
|
|
## Configurable tiny-LLM gate (future)
|
|
In jarvis.yaml wakeword.allow_followup_without_wake + conversation.followup_timeout
|
|
Future: add local 0.5B Qwen to decide if utterance is addressed to Jarvis even without wakeword (use voice activity + speaker gaze)
|
|
|
|
## Piper TTS voices
|
|
.venv/bin/python -m piper.download_voices en_US-lessac-medium
|
|
# then edit jarvis.yaml tts.engine=piper tts.voice=en_US-lessac-medium
|
|
|
|
## Next steps priority
|
|
1. Enroll speakers (you + kids)
|
|
2. Enroll faces (webcam)
|
|
3. Install Ollama + qwen2.5:3b for local LLM
|
|
4. Test with wakeword: say "Hey Jarvis" + command
|
|
5. Install Piper voice for natural TTS
|
|
6. (Optional) Build PortAudio in ~/.local if you want sounddevice backend: ./scripts/build_portaudio.sh; export LD_LIBRARY_PATH=$HOME/.local/lib:$LD_LIBRARY_PATH
|
|
7. Create repo Jarvis-v2 in Gitea web UI (private) then push main: git push origin main
|
|
|
|
## Gitea
|
|
Current: Jarvis repo branch v2 contains full v2 code (pushed)
|
|
Missing: Jarvis-v2 new repo creation blocked by server setting "Push to create is not enabled for users." Need to create via Gitea UI, then we can push main.
|