fix: iMac mic quiet - arecord raw + 12x software gain, strip WAV RIFF header, wakeword api 0.4.0, STT base.en tests, voiceID Adolfo 0.97 enrolled

.gitignore

@@ -0,0 +1,14 @@
+__pycache__/
+*.pyc
+.venv/
+venv/
+.env
+data/*.log
+data/transcriptions/
+data/speakers/*.npy
+data/faces/*.npy
+models/
+*.wav
+.DS_Store
+/tmp/
+jarvis.yaml

FINAL_NOTES.md

@@ -0,0 +1,87 @@
+# Jarvis v2 - Final Setup (iMac Intel i3 Ubuntu 26.04)
+
+## What we built vs v1
+v1 (Jarvis repo main):
+- monolithic jarvis.py, macOS only, mlx-whisper (Apple Silicon), openWakeWord, gTTS, say command, Gemini CLI
+- No speaker ID, no vision, blocking pipeline
+
+v2 (Jarvis repo -> branch v2):
+- Modular: audio/capture (PipeWire pw-record backend, no PortAudio needed), VAD (silero + Energy fallback), wakeword (openWakeWord TFLite hey_jarvis 0.4), STT (faster-whisper base INT8 CPU), SpeakerID (ECAPA-TDNN speechbrain + stats fallback), Vision (OpenCV Haar + face_recognition), LLM pluggable (ollama qwen2.5:3b / gemini_cli / hermes_api / echo), TTS (Piper ONNX + espeak + echo)
+- Pipeline state machine: IDLE -> WAKE -> RECORDING (with pre-roll 0.8s) -> TRANSCRIBING -> IDENTIFYING (voice+face fusion) -> THINKING -> SPEAKING -> FOLLOWUP window (12s no wake needed)
+- Tested on iMac: mic via pw-record 16k mono OK, VAD energy OK, faster-whisper base int8 loads, cam0 640x480 OK, SpeakerStore empty (ready to enroll), Echo LLM/TTS OK
+
+## iMac paths
+~/Projects/jarvis-v2
+  .venv/ (uv, python 3.14, torch CPU, faster-whisper, openwakeword, opencv-headless etc)
+  jarvis.yaml (active config) + jarvis.yaml.example
+  data/soul.md (persona)
+  data/speakers/ (enroll .npy + .json)
+  data/faces/ (enroll .npy + .json)
+  data/transcriptions/YYYY-MM-DD.log
+  models/ (Piper voices, gitignored)
+  systemd/jarvis.service (user service)
+
+## Commands on iMac
+export PATH=$HOME/.local/bin:$PATH
+cd ~/Projects/jarvis-v2
+source .venv/bin/activate
+
+# List audio
+jarvis devices
+
+# Mic level test
+jarvis test-mic
+# VAD test
+jarvis test-vad
+
+# Low-level wav test
+python scripts/test_mic.py
+python scripts/test_wakeword.py
+python scripts/test_stt.py
+
+# Enroll voices (do 3 samples per person for robustness, quiet room)
+jarvis enroll-speaker --name Adolfo --mic
+jarvis enroll-speaker --name Grace --mic
+jarvis enroll-speaker --name Elias --mic
+# or from file: jarvis enroll-speaker --name Adolfo --file /path/to/wav (16k mono)
+
+# Enroll faces
+jarvis enroll-face --name Adolfo
+jarvis enroll-face --name Grace
+jarvis enroll-face --name Elias
+
+# Run assistant (echo brain - placeholder)
+jarvis run
+# Later with ollama
+# install ollama: curl -fsSL https://ollama.com/install.sh | sh
+# ollama pull qwen2.5:3b
+# edit jarvis.yaml llm.provider=ollama
+# jarvis run
+
+# Systemd always-on
+./scripts/install_service.sh
+systemctl --user start jarvis
+journalctl --user -u jarvis -f
+systemctl --user stop jarvis
+systemctl --user disable jarvis
+
+## Configurable tiny-LLM gate (future)
+In jarvis.yaml wakeword.allow_followup_without_wake + conversation.followup_timeout
+Future: add local 0.5B Qwen to decide if utterance is addressed to Jarvis even without wakeword (use voice activity + speaker gaze)
+
+## Piper TTS voices
+.venv/bin/python -m piper.download_voices en_US-lessac-medium
+# then edit jarvis.yaml tts.engine=piper tts.voice=en_US-lessac-medium
+
+## Next steps priority
+1. Enroll speakers (you + kids)
+2. Enroll faces (webcam)
+3. Install Ollama + qwen2.5:3b for local LLM
+4. Test with wakeword: say "Hey Jarvis" + command
+5. Install Piper voice for natural TTS
+6. (Optional) Build PortAudio in ~/.local if you want sounddevice backend: ./scripts/build_portaudio.sh; export LD_LIBRARY_PATH=$HOME/.local/lib:$LD_LIBRARY_PATH
+7. Create repo Jarvis-v2 in Gitea web UI (private) then push main: git push origin main
+
+## Gitea
+Current: Jarvis repo branch v2 contains full v2 code (pushed)
+Missing: Jarvis-v2 new repo creation blocked by server setting "Push to create is not enabled for users." Need to create via Gitea UI, then we can push main.

README.md

@@ -0,0 +1,63 @@
+# Jarvis v2 - Always-Listening Assistant (CPU)
+
+Designed for Intel iMac Ubuntu 26.04, 15GB RAM, CPU-only.
+
+## Architecture
+```
+Mic 16kHz -> VAD (silero) -> Wakeword (openWakeWord TFLite) -> STT (faster-whisper int8) -> SpeakerID (ECAPA) + FaceID (OpenCV) -> LLM (ollama/gemini_cli/hermes) -> TTS (piper) -> Speaker
+```
+
+## Quick start on iMac
+
+```bash
+cd ~/Projects/jarvis-v2
+# install uv if not present
+curl -LsSf https://astral.sh/uv/install.sh | sh
+export PATH=$HOME/.local/bin:$PATH
+uv venv .venv --python 3.14
+source .venv/bin/activate
+uv pip install -e .
+
+# for audio libs you may need (if sudo works, otherwise pip wheels should work)
+# sudo apt install portaudio19-dev libasound2-dev espeak
+
+# configure
+cp jarvis.yaml.example jarvis.yaml
+# edit jarvis.yaml
+
+# Test components
+python scripts/test_mic.py
+python scripts/test_wakeword.py
+python scripts/test_stt.py
+
+# Enroll family voices (3 samples each)
+python scripts/enroll_speaker.py --name Adolfo --mic
+python scripts/enroll_speaker.py --name Grace --mic
+python scripts/enroll_speaker.py --name Elias --mic
+
+# Enroll faces
+python scripts/enroll_face.py --name Adolfo
+python scripts/enroll_face.py --name Grace
+
+# Run
+jarvis run
+# or
+python -m jarvis.cli run --config jarvis.yaml
+
+# Install as systemd user service (always-on)
+./scripts/install_service.sh
+```
+
+## Config
+See jarvis.yaml.example - covers audio, VAD, wakeword, STT, speaker, vision, LLM, TTS.
+
+## LLM Providers
+- ollama: local qwen2.5:3b (install ollama then `ollama pull qwen2.5:3b`)
+- gemini_cli: uses existing gemini CLI + soul.md
+- hermes_api: points to Hermes API
+- echo: placeholder
+
+## Future
+- voice trigger via small LLM gate (qwen0.5B decides if addressed)
+- Face corroboration on wake
+- Home Assistant / MCP tools

data/soul.md

@@ -0,0 +1,21 @@
+# J.A.R.V.I.S. v2 - The Sophisticated Assistant
+
+## Persona
+You are J.A.R.V.I.S., the sophisticated, witty, highly capable AI assistant living on an iMac in the family home. You are calm, helpful, concise (you are speaking, not writing essays), with dry British wit. You adapt to who is speaking: Adolfo (Sir), Grace (young lady, encouraging), Elias (young sir, fun). If unknown, be politely curious.
+
+## Directives
+1. Be concise - voice output. One to three sentences unless asked for more.
+2. No markdown in speech output - no * or # or backticks.
+3. Personalize based on speaker identification confidence provided.
+4. For tools/actions, state intent clearly and ask confirmation for destructive/sensitive actions.
+5. You have vision when available - you can see via webcam on request.
+6. You were woken by "Hey Jarvis" - you may have conversation context history.
+7. Remember: you are always listening but only active after wake word.
+
+## Knowledge
+- Family in Florida, iMac in home office.
+- Kids: Grace and Elias.
+- User prefers concise, not verbose.
+- Brain vault at ~/brain if referenced.
+
+If speaker confidence is low or unknown, ask who you are speaking with once, remember for conversation.

jarvis.yaml.example

@@ -0,0 +1,69 @@
+audio:
+  device: null
+  samplerate: 16000
+  block_size: 512
+  channels: 1
+
+vad:
+  engine: "silero"
+  threshold: 0.5
+  min_speech_ms: 250
+  min_silence_ms: 800
+
+wakeword:
+  enabled: true
+  models: ["hey_jarvis"]
+  threshold: 0.45
+  cooldown_sec: 1.0
+  pre_roll_sec: 0.8
+  allow_followup_without_wake: true
+
+stt:
+  engine: "faster-whisper"
+  model: "base"
+  language: "en"
+  device: "cpu"
+  compute_type: "int8"
+  vad_filter: false
+
+speaker:
+  enabled: true
+  enrollment_dir: "data/speakers"
+  threshold: 0.60
+  embedding_model: "speechbrain/spkrec-ecapa-voxceleb"
+
+vision:
+  enabled: true
+  camera_id: 0
+  check_on_wake: true
+  continuous_interval_sec: 0
+  threshold: 0.55
+  enrollment_dir: "data/faces"
+
+llm:
+  provider: "ollama"
+  model: "qwen2.5:3b"
+  system_prompt_file: "data/soul.md"
+  ollama_host: "http://localhost:11434"
+  max_history: 8
+  temperature: 0.7
+
+conversation:
+  followup_timeout_sec: 12
+  max_followup_turns: 5
+  silence_timeout_sec: 7
+  listening_sound: true
+  idle_timeout_sec: 0
+
+tts:
+  engine: "piper"
+  voice: "en_US-lessac-medium"
+  volume: 0.9
+
+logging:
+  level: "INFO"
+  file: "data/jarvis.log"
+  transcription_dir: "data/transcriptions"
+
+daemon:
+  user_service: true

jarvis/__init__.py

@@ -0,0 +1 @@
+# jarvis package

jarvis/audio/__init__.py

@@ -0,0 +1,2 @@
+# audio
+from .capture import AudioCapture

jarvis/audio/capture.py

@@ -0,0 +1,76 @@
+
+import logging, queue
+import numpy as np
+from typing import Optional
+from .capture_pa import PipeWireCapture
+log = logging.getLogger(__name__)
+try:
+    import sounddevice as sd
+    HAS_SOUNDDEVICE=True
+except Exception as e:
+    HAS_SOUNDDEVICE=False
+    sd=None
+    log.warning("sounddevice not available: %s, will use PipeWire/ALSA capture" % e)
+
+class SoundDeviceCapture:
+    def __init__(self, samplerate=16000, block_size=512, channels=1, device=None, gain=1.0):
+        self.samplerate=samplerate; self.block_size=block_size; self.channels=channels; self.device=device; self.gain=gain
+        self.q=queue.Queue(); self.stream=None; self._running=False
+    def _callback(self, indata, frames, time_info, status):
+        if status: log.warning("Audio status: %s" % status)
+        data=indata.copy().reshape(-1)
+        if self.gain!=1.0: data=np.clip(data*self.gain, -0.95, 0.95)
+        self.q.put(data)
+    def start(self):
+        if self._running: return
+        import sounddevice as sd
+        self.stream=sd.InputStream(samplerate=self.samplerate, channels=self.channels, device=self.device, blocksize=self.block_size, callback=self._callback, dtype='float32')
+        self.stream.start(); self._running=True
+        log.info("Audio capture (sounddevice) started sr=%s device=%s gain=%s" % (self.samplerate, self.device, self.gain))
+    def stop(self):
+        if self.stream:
+            try: self.stream.stop(); self.stream.close()
+            except Exception as e: log.warning("Error stopping: %s" % e)
+        self._running=False
+    def read(self, timeout=0.1):
+        try: return self.q.get(timeout=timeout)
+        except queue.Empty: return None
+    def flush(self):
+        while not self.q.empty():
+            try: self.q.get_nowait()
+            except queue.Empty: break
+    @staticmethod
+    def list_devices():
+        import sounddevice as sd
+        print(sd.query_devices())
+
+class AudioCapture:
+    def __init__(self, samplerate=16000, block_size=512, channels=1, device=None, backend="auto", gain=12.0):
+        self.samplerate=samplerate; self.block_size=block_size; self.channels=channels; self.device=device; self.backend_name=backend; self.gain=gain
+        chosen=None
+        if backend=="auto":
+            if HAS_SOUNDDEVICE:
+                try:
+                    import sounddevice as sd; sd.query_devices(); chosen="sounddevice"
+                except Exception as e:
+                    log.warning("sounddevice query failed: %s, using pipewire" % e); chosen="pipewire"
+            else: chosen="pipewire"
+        else: chosen=backend
+        if chosen=="sounddevice" and HAS_SOUNDDEVICE:
+            self.impl=SoundDeviceCapture(samplerate, block_size, channels, device, gain=gain)
+            self.backend_name="sounddevice"
+        else:
+            self.impl=PipeWireCapture(samplerate, block_size, channels, device, gain=gain)
+            self.backend_name="pipewire"
+        log.info("AudioCapture using backend: %s gain=%s" % (self.backend_name, self.gain))
+    def start(self): return self.impl.start()
+    def stop(self): return self.impl.stop()
+    def read(self, timeout=0.1): return self.impl.read(timeout=timeout)
+    def flush(self): return self.impl.flush()
+    @staticmethod
+    def list_devices():
+        if HAS_SOUNDDEVICE:
+            try:
+                from .capture import SoundDeviceCapture; SoundDeviceCapture.list_devices(); return
+            except: pass
+        PipeWireCapture.list_devices()

jarvis/audio/capture_pa.py

@@ -0,0 +1,130 @@
+import subprocess, threading, queue, logging, time, shutil
+import numpy as np
+from typing import Optional
+log = logging.getLogger(__name__)
+
+class PipeWireCapture:
+    def __init__(self, samplerate=16000, block_size=512, channels=1, device=None, gain=1.0):
+        self.samplerate=samplerate; self.block_size=block_size; self.channels=channels; self.device=device
+        self.gain = gain
+        self.q: queue.Queue[np.ndarray]=queue.Queue(); self.proc=None; self.reader_thread=None; self._running=False
+
+    def _find_backend(self):
+        if shutil.which("arecord"): return "arecord"
+        if shutil.which("pw-record"): return "pw-record"
+        if shutil.which("parec"): return "parec"
+        return None
+
+    def _build_cmd(self, backend):
+        sr=self.samplerate; ch=self.channels
+        if backend=="arecord":
+            dev=self.device or "default"
+            return ["arecord","-D",dev,"-f","S16_LE","-r",str(sr),"-c",str(ch),"-t","raw","-q"]
+        elif backend=="pw-record":
+            return ["pw-record","--format","s16","--rate",str(sr),"--channels",str(ch),"-"]
+        elif backend=="parec":
+            cmd=["parec","--rate=%d" % sr,"--channels=%d" % ch,"--format=s16le","--latency-msec=50"]
+            if self.device: cmd.append("--device=%s" % self.device)
+            return cmd
+        raise RuntimeError(backend)
+
+    def start(self):
+        if self._running: return
+        backend=self._find_backend()
+        if not backend: raise RuntimeError("No backend")
+        cmd=self._build_cmd(backend)
+        log.info("Starting %s: %s gain=%.1f", backend, " ".join(cmd), self.gain)
+        self.proc=subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, bufsize=1000000)
+        self._running=True
+        gain = self.gain
+        def reader():
+            bpb=self.block_size*2*self.channels
+            leftover=b""; header_buf=b""; header_done=False
+            while self._running:
+                try:
+                    if self.proc is None or self.proc.stdout is None: break
+                    if not header_done:
+                        data=self.proc.stdout.read(4096)
+                        if not data:
+                            time.sleep(0.01)
+                            if self.proc.poll() is not None: break
+                            continue
+                        header_buf+=data
+                        if len(header_buf)<44: continue
+                        if header_buf[:4]==b'RIFF':
+                            idx=header_buf.find(b'data')
+                            if idx!=-1 and len(header_buf)>=idx+8:
+                                hl=idx+8
+                                raw=header_buf[hl:]
+                                header_buf=b""; header_done=True
+                                buf=leftover+raw
+                                while len(buf)>=bpb:
+                                    cb=buf[:bpb]; buf=buf[bpb:]
+                                    arr=np.frombuffer(cb,dtype=np.int16).astype(np.float32)/32768.0
+                                    if gain != 1.0:
+                                        arr = np.clip(arr*gain, -0.95, 0.95)
+                                    self.q.put(arr)
+                                leftover=buf
+                                continue
+                            else:
+                                if len(header_buf)>8192:
+                                    raw=header_buf[44:]; header_buf=b""; header_done=True
+                                    leftover=raw
+                                continue
+                        else:
+                            header_done=True
+                            buf=header_buf; header_buf=b""
+                            while len(buf)>=bpb:
+                                cb=buf[:bpb]; buf=buf[bpb:]
+                                arr=np.frombuffer(cb,dtype=np.int16).astype(np.float32)/32768.0
+                                if gain != 1.0:
+                                    arr = np.clip(arr*gain, -0.95, 0.95)
+                                self.q.put(arr)
+                            leftover=buf
+                            continue
+                    needed=bpb-len(leftover)
+                    chunk=self.proc.stdout.read(needed) if needed>0 else b""
+                    if not chunk:
+                        time.sleep(0.005)
+                        if self.proc.poll() is not None: break
+                        continue
+                    buf=leftover+chunk
+                    if len(buf)<bpb:
+                        leftover=buf; continue
+                    cb=buf[:bpb]; leftover=buf[bpb:]
+                    arr=np.frombuffer(cb,dtype=np.int16).astype(np.float32)/32768.0
+                    if gain != 1.0:
+                        arr = np.clip(arr*gain, -0.95, 0.95)
+                    self.q.put(arr)
+                except Exception as e:
+                    log.error("reader %s", e)
+                    import traceback; traceback.print_exc()
+                    time.sleep(0.05)
+        self.reader_thread=threading.Thread(target=reader,daemon=True)
+        self.reader_thread.start()
+        log.info("Capture started")
+
+    def stop(self):
+        self._running=False
+        if self.proc:
+            try:
+                self.proc.terminate(); self.proc.wait(timeout=2)
+            except:
+                try: self.proc.kill()
+                except: pass
+        self.proc=None
+
+    def read(self, timeout=0.1):
+        try: return self.q.get(timeout=timeout)
+        except queue.Empty: return None
+
+    def flush(self):
+        while not self.q.empty():
+            try: self.q.get_nowait()
+            except: break
+
+    @staticmethod
+    def list_devices():
+        try:
+            subprocess.run(["arecord","-l"])
+        except: pass

jarvis/audio/vad.py

@@ -0,0 +1,52 @@
+import numpy as np
+import logging
+
+log = logging.getLogger(__name__)
+
+class VAD:
+    def is_speech(self, audio: np.ndarray, sr: int = 16000) -> bool:
+        raise NotImplementedError
+
+class SileroVAD(VAD):
+    def __init__(self, threshold=0.5):
+        self.threshold = threshold
+        self.model = None
+        self._load()
+
+    def _load(self):
+        try:
+            from silero_vad import load_silero_vad
+            self.model = load_silero_vad()
+            log.info("Silero VAD loaded")
+        except Exception as e:
+            log.warning(f"Silero VAD load failed: {e}, fallback to energy")
+            self.model = None
+
+    def is_speech(self, audio, sr=16000):
+        if self.model is None:
+            return float(np.abs(audio).mean()) > 0.01
+        try:
+            import torch
+            from silero_vad import get_speech_timestamps
+            if len(audio) < 512:
+                return False
+            tensor = torch.from_numpy(audio.astype(np.float32))
+            stamps = get_speech_timestamps(tensor, self.model, sampling_rate=sr, threshold=self.threshold, min_speech_duration_ms=250, min_silence_duration_ms=100)
+            return len(stamps) > 0
+        except Exception as e:
+            log.debug(f"VAD error: {e}")
+            return float(np.abs(audio).mean()) > 0.01
+
+class EnergyVAD(VAD):
+    def __init__(self, threshold=0.01):
+        self.threshold = threshold
+    def is_speech(self, audio, sr=16000):
+        return float(np.mean(np.abs(audio))) > self.threshold
+
+def create_vad(cfg):
+    engine = cfg.get('vad', {}).get('engine', 'silero')
+    thresh = cfg.get('vad', {}).get('threshold', 0.5)
+    if engine == 'silero':
+        return SileroVAD(threshold=thresh)
+    else:
+        return EnergyVAD(threshold=0.02)

jarvis/audio/wakeword.py

@@ -0,0 +1,101 @@
+import logging, time, os
+import numpy as np
+from typing import List, Optional
+from pathlib import Path
+
+log = logging.getLogger(__name__)
+
+class WakeWordDetector:
+    def __init__(self, models: List[str] = None, threshold: float = 0.45, cooldown: float = 1.0):
+        self.models_requested = models or ["hey_jarvis"]
+        self.threshold = threshold
+        self.cooldown = cooldown
+        self.last_trigger = 0
+        self.oww_model = None
+        self.model_names = []  # actual loaded names
+        self._load()
+
+    def _load(self):
+        try:
+            from openwakeword.model import Model
+            import openwakeword
+            # Get default model dir
+            # openWakeWord stores models in site-packages/openwakeword/resources/models
+            # We need to provide paths
+            import importlib.resources as pkg_resources
+            try:
+                # Try new way
+                base_path = Path(openwakeword.__file__).parent / "resources" / "models"
+            except:
+                base_path = Path.home() / ".cache" / "openwakeword"
+            
+            # List available models
+            if base_path.exists():
+                available = list(base_path.glob("*.onnx"))
+                log.info(f"openWakeWord models dir {base_path} has {len(available)} models: {[p.name for p in available[:10]]}")
+            
+            # Build paths for requested models
+            model_paths = []
+            for req in self.models_requested:
+                # req like "hey_jarvis" -> find file containing that name
+                candidates = []
+                if base_path.exists():
+                    candidates = list(base_path.glob(f"*{req}*.onnx")) + list(base_path.glob(f"*{req.replace('_','')}*.onnx"))
+                    # also lowercase search
+                    candidates += [p for p in base_path.glob("*.onnx") if req.lower() in p.name.lower()]
+                if candidates:
+                    model_paths.append(str(candidates[0]))
+                    log.info(f"Mapped wakeword {req} -> {candidates[0].name}")
+                else:
+                    log.warning(f"Wakeword model not found for {req}, will load all and filter")
+            
+            if model_paths:
+                self.oww_model = Model(wakeword_model_paths=model_paths)
+            else:
+                # Load all
+                self.oww_model = Model()
+            
+            # Get model names
+            try:
+                # Model has attribute models or similar
+                self.model_names = list(self.oww_model.models.keys()) if hasattr(self.oww_model, 'models') else []
+            except:
+                pass
+            
+            log.info(f"openWakeWord loaded, names: {self.model_names}")
+        except Exception as e:
+            log.warning(f"openWakeWord failed: {e}", exc_info=True)
+            self.oww_model = None
+
+    def detect(self, audio_chunk: np.ndarray) -> Optional[dict]:
+        now = time.time()
+        if now - self.last_trigger < self.cooldown:
+            return None
+        if self.oww_model is None:
+            return None
+        try:
+            scores = self.oww_model.predict(audio_chunk)
+            for model_name, score in scores.items():
+                # Filter: only requested models + any if threshold
+                if self.models_requested and not any(r.replace('_','') in model_name.replace('_','').lower() or model_name.lower() in r.lower() for r in self.models_requested):
+                    # If we loaded specific models, this check not needed, but for "load all" case, filter
+                    if not model_name.lower().startswith('hey_jarvis') and 'jarvis' not in model_name.lower():
+                        # allow if requested is subset
+                        pass
+                if score >= self.threshold:
+                    self.last_trigger = now
+                    log.info(f"Wakeword: {model_name} score={score:.2f}")
+                    return {"model": model_name, "score": float(score), "timestamp": now}
+        except Exception as e:
+            log.debug(f"Wakeword error: {e}")
+        return None
+
+    def reset(self):
+        self.last_trigger = 0
+
+class KeywordSpotterFallback:
+    def __init__(self, keywords: List[str] = None):
+        self.keywords = [k.lower() for k in (keywords or ["hey jarvis", "jarvis"])]
+    def check_transcript(self, text: str) -> bool:
+        low = text.lower()
+        return any(k in low for k in self.keywords)

jarvis/audio/wakeword.py.new

@@ -0,0 +1,101 @@
+import logging, time, os
+import numpy as np
+from typing import List, Optional
+from pathlib import Path
+
+log = logging.getLogger(__name__)
+
+class WakeWordDetector:
+    def __init__(self, models: List[str] = None, threshold: float = 0.45, cooldown: float = 1.0):
+        self.models_requested = models or ["hey_jarvis"]
+        self.threshold = threshold
+        self.cooldown = cooldown
+        self.last_trigger = 0
+        self.oww_model = None
+        self.model_names = []  # actual loaded names
+        self._load()
+
+    def _load(self):
+        try:
+            from openwakeword.model import Model
+            import openwakeword
+            # Get default model dir
+            # openWakeWord stores models in site-packages/openwakeword/resources/models
+            # We need to provide paths
+            import importlib.resources as pkg_resources
+            try:
+                # Try new way
+                base_path = Path(openwakeword.__file__).parent / "resources" / "models"
+            except:
+                base_path = Path.home() / ".cache" / "openwakeword"
+            
+            # List available models
+            if base_path.exists():
+                available = list(base_path.glob("*.onnx"))
+                log.info(f"openWakeWord models dir {base_path} has {len(available)} models: {[p.name for p in available[:10]]}")
+            
+            # Build paths for requested models
+            model_paths = []
+            for req in self.models_requested:
+                # req like "hey_jarvis" -> find file containing that name
+                candidates = []
+                if base_path.exists():
+                    candidates = list(base_path.glob(f"*{req}*.onnx")) + list(base_path.glob(f"*{req.replace('_','')}*.onnx"))
+                    # also lowercase search
+                    candidates += [p for p in base_path.glob("*.onnx") if req.lower() in p.name.lower()]
+                if candidates:
+                    model_paths.append(str(candidates[0]))
+                    log.info(f"Mapped wakeword {req} -> {candidates[0].name}")
+                else:
+                    log.warning(f"Wakeword model not found for {req}, will load all and filter")
+            
+            if model_paths:
+                self.oww_model = Model(wakeword_model_paths=model_paths)
+            else:
+                # Load all
+                self.oww_model = Model()
+            
+            # Get model names
+            try:
+                # Model has attribute models or similar
+                self.model_names = list(self.oww_model.models.keys()) if hasattr(self.oww_model, 'models') else []
+            except:
+                pass
+            
+            log.info(f"openWakeWord loaded, names: {self.model_names}")
+        except Exception as e:
+            log.warning(f"openWakeWord failed: {e}", exc_info=True)
+            self.oww_model = None
+
+    def detect(self, audio_chunk: np.ndarray) -> Optional[dict]:
+        now = time.time()
+        if now - self.last_trigger < self.cooldown:
+            return None
+        if self.oww_model is None:
+            return None
+        try:
+            scores = self.oww_model.predict(audio_chunk)
+            for model_name, score in scores.items():
+                # Filter: only requested models + any if threshold
+                if self.models_requested and not any(r.replace('_','') in model_name.replace('_','').lower() or model_name.lower() in r.lower() for r in self.models_requested):
+                    # If we loaded specific models, this check not needed, but for "load all" case, filter
+                    if not model_name.lower().startswith('hey_jarvis') and 'jarvis' not in model_name.lower():
+                        # allow if requested is subset
+                        pass
+                if score >= self.threshold:
+                    self.last_trigger = now
+                    log.info(f"Wakeword: {model_name} score={score:.2f}")
+                    return {"model": model_name, "score": float(score), "timestamp": now}
+        except Exception as e:
+            log.debug(f"Wakeword error: {e}")
+        return None
+
+    def reset(self):
+        self.last_trigger = 0
+
+class KeywordSpotterFallback:
+    def __init__(self, keywords: List[str] = None):
+        self.keywords = [k.lower() for k in (keywords or ["hey jarvis", "jarvis"])]
+    def check_transcript(self, text: str) -> bool:
+        low = text.lower()
+        return any(k in low for k in self.keywords)

jarvis/cli.py

@@ -0,0 +1,154 @@
+import argparse, logging, sys
+from pathlib import Path
+from .config import load_config
+
+def main():
+    parser = argparse.ArgumentParser(description="Jarvis v2 - Always listening assistant")
+    sub = parser.add_subparsers(dest="cmd")
+
+    run_p = sub.add_parser("run", help="Run assistant")
+    run_p.add_argument("--config", type=str, default=None, help="Path to jarvis.yaml")
+
+    devices_p = sub.add_parser("devices", help="List audio devices")
+
+    enroll_sp = sub.add_parser("enroll-speaker", help="Enroll speaker via mic")
+    enroll_sp.add_argument("--name", required=True)
+    enroll_sp.add_argument("--mic", action="store_true", help="Record from mic")
+    enroll_sp.add_argument("--file", type=str, help="Audio file path")
+
+    enroll_face = sub.add_parser("enroll-face", help="Enroll face via webcam")
+    enroll_face.add_argument("--name", required=True)
+    enroll_face.add_argument("--camera", type=int, default=0)
+
+    test_vad = sub.add_parser("test-vad", help="Test VAD")
+    test_mic = sub.add_parser("test-mic", help="Test mic capture")
+
+    parser.add_argument("--verbose", action="store_true")
+
+    args = parser.parse_args()
+
+    if args.verbose:
+        logging.basicConfig(level=logging.DEBUG)
+    else:
+        logging.basicConfig(level=logging.INFO, format='%(asctime)s %(levelname)s %(message)s')
+
+    if args.cmd == "run" or args.cmd is None:
+        cfg_path = args.config if hasattr(args, 'config') else None
+        from .core.pipeline import JarvisPipeline
+        pipeline = JarvisPipeline(config_path=cfg_path)
+        pipeline.run()
+
+    elif args.cmd == "devices":
+        from .audio.capture import AudioCapture
+        AudioCapture.list_devices()
+
+    elif args.cmd == "enroll-speaker":
+        from .audio.capture import AudioCapture
+        from .speaker.identifier import SpeakerIdentifier
+        import numpy as np, time, sys
+
+        identifier = SpeakerIdentifier()
+        audio_data = None
+
+        if args.file:
+            # Load via soundfile
+            import soundfile as sf
+            data, sr = sf.read(args.file)
+            if len(data.shape) > 1:
+                data = data.mean(axis=1)
+            audio_data = (data, sr)
+        elif args.mic:
+            print(f"Recording 3s sample for {args.name}... Speak now!")
+            cap = AudioCapture(samplerate=16000, block_size=512)
+            cap.start()
+            import time as _time
+            bufs = []
+            start = _time.time()
+            while _time.time() - start < 3:
+                chunk = cap.read(timeout=0.2)
+                if chunk is not None:
+                    bufs.append(chunk)
+            cap.stop()
+            if bufs:
+                audio_data = (np.concatenate(bufs), 16000)
+            else:
+                print("No audio captured")
+                sys.exit(1)
+
+        if audio_data:
+            data, sr = audio_data
+            print(f"Enrolling {args.name} with {len(data)/sr:.2f}s audio...")
+            identifier.enroll(args.name, data, sr)
+            print(f"Enrolled {args.name} successfully")
+        else:
+            print("No audio, use --mic or --file")
+            sys.exit(1)
+
+    elif args.cmd == "enroll-face":
+        from .vision.face_identifier import FaceIdentifier
+        fi = FaceIdentifier(camera_id=args.camera)
+        import time
+        print(f"Capturing face for {args.name} in 2 seconds, look at camera...")
+        time.sleep(2)
+        frame = fi.camera.capture_frame()
+        if frame is None:
+            print("Failed to capture camera")
+            sys.exit(1)
+        faces = fi.detector.detect(frame)
+        if not faces:
+            print("No face detected, try again (move closer, better light)")
+            # Save debug image
+            import cv2
+            cv2.imwrite(f"/tmp/{args.name}_no_face.jpg", frame)
+            print(f"Saved debug to /tmp/{args.name}_no_face.jpg")
+            sys.exit(1)
+        # largest face
+        faces = sorted(faces, key=lambda b: b[2]*b[3], reverse=True)
+        bbox = faces[0]
+        print(f"Face detected at {bbox}, enrolling...")
+        ok = fi.store.add_sample(args.name, frame, bbox=bbox)
+        if ok:
+            print(f"Enrolled face {args.name}")
+        else:
+            print(f"Failed to enroll face {args.name}")
+        fi.close()
+
+    elif args.cmd == "test-vad":
+        from .audio.capture import AudioCapture
+        from .audio.vad import create_vad
+        cfg = load_config(None)
+        vad = create_vad(cfg)
+        cap = AudioCapture()
+        cap.start()
+        print("Testing VAD - speak and see detection...")
+        try:
+            while True:
+                chunk = cap.read(timeout=0.5)
+                if chunk is not None:
+                    is_speech = vad.is_speech(chunk)
+                    print("SPEECH" if is_speech else "silence", end="\r")
+        except KeyboardInterrupt:
+            pass
+        finally:
+            cap.stop()
+
+    elif args.cmd == "test-mic":
+        from .audio.capture import AudioCapture
+        import numpy as np
+        cap = AudioCapture()
+        cap.start()
+        print("Mic test - watch levels:")
+        try:
+            while True:
+                chunk = cap.read(timeout=0.5)
+                if chunk is not None:
+                    level = np.abs(chunk).mean()
+                    bars = "#" * int(level*100)
+                    print(f"level {level:.4f} {bars}", end="\r")
+        except KeyboardInterrupt:
+            pass
+        finally:
+            cap.stop()
+
+if __name__ == "__main__":
+    main()

jarvis/config.py

@@ -0,0 +1,28 @@
+import yaml, os
+from pathlib import Path
+from typing import Any, Dict, Optional
+
+def load_config(path: Optional[Path] = None) -> Dict[str, Any]:
+    default = Path(__file__).parent.parent / "jarvis.yaml"
+    example = Path(__file__).parent.parent / "jarvis.yaml.example"
+    if path is None:
+        if default.exists():
+            path = default
+        elif example.exists():
+            path = example
+        else:
+            return {}
+    path = Path(path)
+    if not path.exists():
+        return {}
+    with open(path, 'r') as f:
+        return yaml.safe_load(f) or {}
+
+def get_nested(cfg: dict, dotted: str, default=None):
+    cur = cfg
+    for part in dotted.split('.'):
+        if isinstance(cur, dict) and part in cur:
+            cur = cur[part]
+        else:
+            return default
+    return cur

jarvis/core/__init__.py

@@ -0,0 +1,2 @@
+# core
+from .pipeline import JarvisPipeline

jarvis/core/pipeline.py

@@ -0,0 +1,369 @@
+import logging, time, collections, numpy as np
+from pathlib import Path
+
+from ..config import load_config
+from ..audio.capture import AudioCapture
+from ..audio.vad import create_vad
+from ..audio.wakeword import WakeWordDetector, KeywordSpotterFallback
+from ..stt.faster_whisper_engine import FasterWhisperSTT
+from ..speaker.identifier import SpeakerIdentifier
+from ..vision.face_identifier import FaceIdentifier
+from ..llm.echo import EchoLLM
+from ..tts.piper import PiperTTS, EchoTTS
+
+log = logging.getLogger(__name__)
+
+class JarvisPipeline:
+    def __init__(self, config_path=None):
+        self.cfg = load_config(config_path)
+        self.samplerate = self.cfg.get('audio',{}).get('samplerate',16000)
+        self.block_size = self.cfg.get('audio',{}).get('block_size',512)
+        self.device = self.cfg.get('audio',{}).get('device',None)
+        self.backend = self.cfg.get('audio',{}).get('backend','auto')
+
+        self.capture = AudioCapture(samplerate=self.samplerate, block_size=self.block_size, device=self.device, backend=self.backend)
+        self.vad = create_vad(self.cfg)
+        self.wakeword = WakeWordDetector(
+            models=self.cfg.get('wakeword',{}).get('models',['hey_jarvis']),
+            threshold=self.cfg.get('wakeword',{}).get('threshold',0.45),
+            cooldown=self.cfg.get('wakeword',{}).get('cooldown_sec',1.0)
+        )
+        self.keyword_fallback = KeywordSpotterFallback(["hey jarvis","jarvis"])
+
+        stt_cfg = self.cfg.get('stt',{})
+        self.stt = FasterWhisperSTT(
+            model=stt_cfg.get('model','base'),
+            language=stt_cfg.get('language','en'),
+            device=stt_cfg.get('device','cpu'),
+            compute_type=stt_cfg.get('compute_type','int8'),
+            vad_filter=stt_cfg.get('vad_filter',False)
+        )
+
+        self.speaker_id = None
+        if self.cfg.get('speaker',{}).get('enabled',True):
+            try:
+                self.speaker_id = SpeakerIdentifier(
+                    enrollment_dir=self.cfg.get('speaker',{}).get('enrollment_dir','data/speakers'),
+                    threshold=self.cfg.get('speaker',{}).get('threshold',0.60),
+                    embedding_model=self.cfg.get('speaker',{}).get('embedding_model','speechbrain/spkrec-ecapa-voxceleb')
+                )
+            except Exception as e:
+                log.warning(f"Speaker ID disabled: {e}")
+
+        self.face_id = None
+        if self.cfg.get('vision',{}).get('enabled',True):
+            try:
+                self.face_id = FaceIdentifier(
+                    camera_id=self.cfg.get('vision',{}).get('camera_id',0),
+                    enrollment_dir=self.cfg.get('vision',{}).get('enrollment_dir','data/faces'),
+                    threshold=self.cfg.get('vision',{}).get('threshold',0.55)
+                )
+            except Exception as e:
+                log.warning(f"Face ID disabled: {e}")
+
+        self.llm = self._create_llm()
+        self.tts = self._create_tts()
+
+        self.pre_roll_sec = self.cfg.get('wakeword',{}).get('pre_roll_sec',0.8)
+        self.pre_roll_samples = int(self.pre_roll_sec * self.samplerate)
+        self.pre_roll_buffer = collections.deque(maxlen=self.pre_roll_samples)
+
+        self.silence_ms = self.cfg.get('vad',{}).get('min_silence_ms',800)
+        self.followup_timeout = self.cfg.get('conversation',{}).get('followup_timeout_sec',12)
+        self.max_followup = self.cfg.get('conversation',{}).get('max_followup_turns',5)
+
+        self.history = []
+        self.transcription_dir = Path(self.cfg.get('logging',{}).get('transcription_dir','data/transcriptions'))
+        self.transcription_dir.mkdir(parents=True, exist_ok=True)
+
+        self.running = False
+        self.in_followup = False
+        self.followup_turns = 0
+        self.last_interaction = 0
+        self._last_face_result = {"name":"unknown","face_conf":0.0}
+
+    def _create_llm(self):
+        provider = self.cfg.get('llm',{}).get('provider','echo')
+        log.info(f"LLM provider: {provider}")
+        if provider == 'ollama':
+            try:
+                from ..llm.ollama import OllamaLLM
+                return OllamaLLM(
+                    model=self.cfg.get('llm',{}).get('model','qwen2.5:3b'),
+                    host=self.cfg.get('llm',{}).get('ollama_host','http://localhost:11434'),
+                    temperature=self.cfg.get('llm',{}).get('temperature',0.7)
+                )
+            except Exception as e:
+                log.warning(f"Ollama failed: {e}, echo")
+                return EchoLLM()
+        elif provider == 'gemini_cli':
+            try:
+                from ..llm.gemini_cli import GeminiCLILLM
+                return GeminiCLILLM(workspace_dir="data", system_file=self.cfg.get('llm',{}).get('system_prompt_file','data/soul.md'))
+            except Exception as e:
+                log.warning(f"Gemini CLI failed: {e}")
+                return EchoLLM()
+        elif provider == 'hermes_api':
+            try:
+                from ..llm.hermes_api import HermesAPILLM
+                return HermesAPILLM(base_url=self.cfg.get('llm',{}).get('base_url','http://192.168.68.110:8000'))
+            except Exception as e:
+                log.warning(f"Hermes API failed: {e}")
+                return EchoLLM()
+        else:
+            return EchoLLM()
+
+    def _create_tts(self):
+        engine = self.cfg.get('tts',{}).get('engine','echo')
+        if engine == 'piper':
+            try:
+                return PiperTTS(voice=self.cfg.get('tts',{}).get('voice','en_US-lessac-medium'), models_dir="models", volume=self.cfg.get('tts',{}).get('volume',0.9))
+            except Exception as e:
+                log.warning(f"Piper failed: {e}, echo fallback")
+                return EchoTTS()
+        else:
+            return EchoTTS()
+
+    def log_transcription(self, text, speaker_info=None):
+        if not text:
+            return
+        from datetime import datetime
+        date_str = datetime.now().strftime("%Y-%m-%d")
+        time_str = datetime.now().strftime("%H:%M:%S")
+        speaker = speaker_info.get('name','unknown') if speaker_info else 'unknown'
+        log_file = self.transcription_dir / f"{date_str}.log"
+        try:
+            with open(log_file,'a') as f:
+                f.write(f"[{time_str}] ({speaker}) {text}\n")
+        except Exception as e:
+            log.warning(f"Log failed: {e}")
+
+    def run(self):
+        print("=== Jarvis v2 Starting ===")
+        print(f"Config: wakeword={self.cfg.get('wakeword',{}).get('models')} stt={self.cfg.get('stt',{}).get('model')} llm={self.cfg.get('llm',{}).get('provider')} backend={self.backend}")
+        print(f"Speaker ID: {self.speaker_id is not None} Face ID: {self.face_id is not None}")
+        print(f"Wakeword model loaded: {self.wakeword.oww_model is not None} names={self.wakeword.model_names}")
+
+        self.capture.start()
+        self.running = True
+
+        recording_buffer = []
+        is_recording = False
+        silence_start = None
+
+        system_prompt = ""
+        soul_path = self.cfg.get('llm',{}).get('system_prompt_file','data/soul.md')
+        if soul_path and Path(soul_path).exists():
+            system_prompt = Path(soul_path).read_text()[:4000]
+
+        print("Listening... say 'Hey Jarvis' (or just speak if wakeword model missing, keyword fallback via STT)")
+        # For keyword fallback mode, we need continuous STT? We'll use VAD-gated recording + STT keyword check
+        # If wakeword model missing, we go into VAD recording mode and check transcript for keyword
+
+        try:
+            while self.running:
+                chunk = self.capture.read(timeout=0.2)
+                if chunk is None:
+                    if self.in_followup and (time.time() - self.last_interaction > self.followup_timeout):
+                        print("Followup window expired, idle")
+                        self.in_followup = False
+                        self.followup_turns = 0
+                    continue
+
+                self.pre_roll_buffer.extend(chunk)
+
+                if not is_recording:
+                    if not self.vad.is_speech(chunk, self.samplerate):
+                        if not self.in_followup:
+                            continue
+
+                    detected = None
+                    if not self.in_followup:
+                        detected = self.wakeword.detect(chunk)
+                        # If wakeword model not available, use VAD to start recording and let STT keyword check handle it
+                        if self.wakeword.oww_model is None and self.vad.is_speech(chunk):
+                            # Start recording for keyword fallback
+                            detected = {"model": "vad_fallback", "score": 0.5}
+                    else:
+                        if self.vad.is_speech(chunk, self.samplerate):
+                            detected = {"model": "followup", "score": 1.0}
+
+                    if detected:
+                        print(f"\n>>> Wake detected: {detected['model']} score={detected.get('score',0):.2f}")
+                        try:
+                            import numpy as np
+                            import sounddevice as sd
+                            sr=16000
+                            t=np.linspace(0,0.15,sr//6)
+                            beep = 0.3*np.sin(2*np.pi*880*t)
+                            # try pipewire playback? ignore
+                            try:
+                                sd.play(beep, sr)
+                                sd.wait()
+                            except:
+                                pass
+                        except:
+                            pass
+
+                        is_recording = True
+                        recording_buffer = [np.array(self.pre_roll_buffer, dtype=np.float32)]
+                        silence_start = None
+
+                        face_result = {"name":"unknown","face_conf":0.0}
+                        if self.face_id and self.cfg.get('vision',{}).get('check_on_wake',True):
+                            try:
+                                face_result = self.face_id.check()
+                                print(f"Face: {face_result.get('name')} conf={face_result.get('face_conf',0):.2f} found={face_result.get('faces_found',0)}")
+                            except Exception as e:
+                                log.warning(f"Face check error: {e}")
+                        self._last_face_result = face_result
+                        continue
+                else:
+                    recording_buffer.append(chunk.copy())
+                    if self.vad.is_speech(chunk, self.samplerate):
+                        silence_start = None
+                    else:
+                        if silence_start is None:
+                            silence_start = time.time()
+                        else:
+                            if (time.time() - silence_start)*1000 > self.silence_ms:
+                                total_len = sum(len(c) for c in recording_buffer)
+                                if total_len > self.samplerate * 0.4:
+                                    is_recording = False
+                                    audio = np.concatenate(recording_buffer) if recording_buffer else np.array([])
+                                    recording_buffer = []
+                                    self._process_utterance(audio, system_prompt)
+                                    self.wakeword.reset()
+                                    self.pre_roll_buffer.clear()
+                                else:
+                                    is_recording = False
+                                    recording_buffer = []
+
+                    total_len = sum(len(c) for c in recording_buffer)
+                    if total_len > self.samplerate * 12:
+                        print("Max recording length, processing")
+                        is_recording = False
+                        audio = np.concatenate(recording_buffer) if recording_buffer else np.array([])
+                        recording_buffer = []
+                        self._process_utterance(audio, system_prompt)
+                        self.wakeword.reset()
+                        self.pre_roll_buffer.clear()
+
+        except KeyboardInterrupt:
+            print("\nShutting down...")
+        finally:
+            self.capture.stop()
+            if self.face_id:
+                self.face_id.close()
+
+    def _process_utterance(self, audio, system_prompt):
+        import numpy as np
+        if len(audio) < self.samplerate * 0.2:
+            print("Audio too short")
+            return
+
+        print(f"Transcribing {len(audio)/self.samplerate:.2f}s...")
+        stt_result = self.stt.transcribe(audio, self.samplerate)
+        text = stt_result.get('text','').strip()
+        print(f"STT: '{text}' conf={stt_result.get('confidence',0):.2f}")
+
+        if not text:
+            print("Empty transcription")
+            return
+
+        # Keyword fallback check when wakeword model missing or followup not active
+        is_keyword_trigger = self.keyword_fallback.check_transcript(text)
+        
+        if not self.in_followup:
+            # If we were triggered by VAD fallback, we need keyword in text to confirm it's for Jarvis
+            if self.wakeword.oww_model is None:
+                if not is_keyword_trigger:
+                    print(f"No wake keyword in '{text}', ignoring (keyword fallback)")
+                    return
+                # Extract command after keyword
+                low = text.lower()
+                for kw in ["hey jarvis", "jarvis"]:
+                    if kw in low:
+                        idx = low.find(kw)
+                        text = text[idx+len(kw):].strip(" ,:")
+                        break
+                if not text:
+                    print("Only wakeword, waiting for command in followup")
+                    self.in_followup = True
+                    self.followup_turns = 0
+                    self.last_interaction = time.time()
+                    return
+        else:
+            # in followup, allow empty? no
+            pass
+
+        speaker_info = {"name": "unknown", "voice_conf": 0.0, "face_conf": 0.0, "fused_conf": 0.0}
+        if hasattr(self, '_last_face_result'):
+            speaker_info['face_conf'] = self._last_face_result.get('face_conf',0.0)
+
+        if self.speaker_id:
+            try:
+                voice_result = self.speaker_id.identify(audio, self.samplerate)
+                speaker_info['name'] = voice_result.get('name','unknown')
+                speaker_info['voice_conf'] = voice_result.get('voice_conf',0.0)
+                speaker_info['voice_scores'] = voice_result.get('scores',{})
+            except Exception as e:
+                log.warning(f"Speaker ID error: {e}")
+
+        face_name = getattr(self, '_last_face_result', {}).get('name','unknown')
+        face_conf = getattr(self, '_last_face_result', {}).get('face_conf',0.0) if hasattr(self, '_last_face_result') else 0.0
+        final_name = speaker_info.get('name','unknown')
+        final_conf = speaker_info.get('voice_conf',0.0)
+
+        if face_name != "unknown" and final_name != "unknown":
+            if face_name == final_name:
+                final_conf = min(max(face_conf, final_conf) * 1.1, 0.99)
+                final_name = face_name
+            else:
+                if face_conf > final_conf:
+                    final_name = face_name
+                    final_conf = face_conf
+        elif face_name != "unknown" and final_name == "unknown":
+            final_name = face_name
+            final_conf = face_conf
+
+        speaker_info['name'] = final_name
+        speaker_info['fused_conf'] = final_conf
+        speaker_info['face_conf'] = face_conf
+
+        print(f"Speaker: {final_name} fused={final_conf:.2f} voice={speaker_info.get('voice_conf',0):.2f} face={face_conf:.2f}")
+
+        self.log_transcription(text, speaker_info)
+
+        messages = []
+        if system_prompt:
+            messages.append({"role": "system", "content": system_prompt})
+        for h in self.history[-self.cfg.get('llm',{}).get('max_history',8):]:
+            messages.append(h)
+        messages.append({"role": "user", "content": text})
+
+        print("Thinking...")
+        try:
+            response = self.llm.chat(messages, speaker=speaker_info)
+        except Exception as e:
+            log.error(f"LLM error: {e}")
+            response = f"Error: {e}"
+
+        print(f"Jarvis: {response}")
+        self.history.append({"role": "user", "content": text})
+        self.history.append({"role": "assistant", "content": response})
+
+        try:
+            self.tts.speak(response)
+        except Exception as e:
+            log.error(f"TTS error: {e}")
+            print(f"[TTS error] {response}")
+
+        self.in_followup = True
+        self.followup_turns += 1
+        self.last_interaction = time.time()
+
+        if self.followup_turns >= self.max_followup:
+            print("Max followup reached, idle")
+            self.in_followup = False
+            self.followup_turns = 0

jarvis/core/state.py

@@ -0,0 +1,38 @@
+from enum import Enum, auto
+import time
+
+class State(Enum):
+    IDLE = auto()
+    WAKE_DETECTED = auto()
+    RECORDING = auto()
+    TRANSCRIBING = auto()
+    IDENTIFYING = auto()
+    THINKING = auto()
+    SPEAKING = auto()
+    FOLLOWUP = auto()
+
+class ConversationState:
+    def __init__(self):
+        self.state = State.IDLE
+        self.last_wake_time = 0
+        self.last_speech_time = 0
+        self.history = []  # list of {role, content}
+        self.current_speaker = {"name": "unknown", "voice_conf": 0.0, "face_conf": 0.0, "fused_conf": 0.0}
+        self.turns_in_followup = 0
+
+    def set_state(self, s: State):
+        # print(f"State {self.state} -> {s}")
+        self.state = s
+        if s == State.WAKE_DETECTED:
+            self.last_wake_time = time.time()
+
+    def add_history(self, role: str, content: str, max_len=8):
+        self.history.append({"role": role, "content": content})
+        if len(self.history) > max_len:
+            self.history = self.history[-max_len:]
+
+    def clear_history(self):
+        self.history = []
+
+    def is_in_followup_window(self, timeout_sec):
+        return (time.time() - self.last_wake_time) < timeout_sec

jarvis/llm/__init__.py

@@ -0,0 +1 @@
+# jarvis package

jarvis/llm/base.py

@@ -0,0 +1,10 @@
+from typing import List, Dict, Optional
+class LLMEngine:
+    def chat(self, messages: List[Dict[str,str]], speaker: Optional[dict] = None) -> str:
+        raise NotImplementedError
+    def single(self, prompt: str, system: str = "", speaker: Optional[dict] = None) -> str:
+        msgs = []
+        if system:
+            msgs.append({"role": "system", "content": system})
+        msgs.append({"role": "user", "content": prompt})
+        return self.chat(msgs, speaker=speaker)

jarvis/llm/echo.py

@@ -0,0 +1,6 @@
+from .base import LLMEngine
+class EchoLLM(LLMEngine):
+    def chat(self, messages, speaker=None):
+        last = messages[-1]["content"] if messages else ""
+        who = speaker.get("name","unknown") if speaker else "unknown"
+        return f"Echo from {who}: You said '{last}'. I'm placeholder brain - configure ollama or gemini_cli."

jarvis/llm/gemini_cli.py

@@ -0,0 +1,75 @@
+from .base import LLMEngine
+import subprocess, os, logging, re, tempfile
+
+log = logging.getLogger(__name__)
+
+class GeminiCLILLM(LLMEngine):
+    def __init__(self, yolo=True, workspace_dir="data", system_file="data/soul.md"):
+        self.yolo = yolo
+        self.workspace_dir = workspace_dir
+        self.system_file = system_file
+        self.session_id = None
+
+    def _read_system(self):
+        try:
+            if os.path.exists(self.system_file):
+                with open(self.system_file) as f:
+                    return f.read()
+        except Exception as e:
+            log.warning(f"No system file: {e}")
+        return ""
+
+    def chat(self, messages, speaker=None):
+        system = ""
+        user_prompt = ""
+        for m in messages:
+            if m["role"] == "system":
+                system += m["content"] + "\n"
+            elif m["role"] == "user":
+                user_prompt += f"User: {m['content']}\n"
+            elif m["role"] == "assistant":
+                user_prompt += f"Assistant: {m['content']}\n"
+
+        if speaker and speaker.get("name") != "unknown":
+            user_prompt = f"[Speaker: {speaker.get('name')} conf {speaker.get('fused_conf',0):.2f}]\n" + user_prompt
+
+        soul = self._read_system()
+        full_system = (soul + "\n" + system).strip() if soul else system
+
+        env = os.environ.copy()
+        tmp_sys = None
+        if full_system:
+            fd, tmp_path = tempfile.mkstemp(suffix=".md")
+            os.write(fd, full_system.encode())
+            os.close(fd)
+            tmp_sys = tmp_path
+            env["GEMINI_SYSTEM_MD"] = tmp_sys
+
+        args = ["gemini", "--prompt", user_prompt]
+        if self.yolo:
+            args.append("--yolo")
+        if self.session_id:
+            args.extend(["--resume", self.session_id])
+
+        try:
+            os.makedirs(self.workspace_dir, exist_ok=True)
+            proc = subprocess.run(args, capture_output=True, text=True, cwd=self.workspace_dir, env=env, timeout=120)
+            out = proc.stdout.strip() or proc.stderr.strip()
+            if not self.session_id:
+                try:
+                    res = subprocess.run(["gemini","--list-sessions"], capture_output=True, text=True, cwd=self.workspace_dir, timeout=10)
+                    matches = re.findall(r"\[([a-f0-9\-]+)\]", res.stdout)
+                    if matches:
+                        self.session_id = matches[-1]
+                except:
+                    pass
+            return out or "No response from Gemini"
+        except Exception as e:
+            log.error(f"Gemini CLI error: {e}")
+            return f"Gemini error: {e}"
+        finally:
+            if tmp_sys and os.path.exists(tmp_sys):
+                try:
+                    os.remove(tmp_sys)
+                except:
+                    pass

jarvis/llm/hermes_api.py

@@ -0,0 +1,29 @@
+from .base import LLMEngine
+import requests, logging
+
+log = logging.getLogger(__name__)
+
+class HermesAPILLM(LLMEngine):
+    def __init__(self, base_url="http://192.168.68.110:8000", model="default", api_key=""):
+        self.base_url = base_url.rstrip('/')
+        self.model = model
+        self.api_key = api_key
+
+    def chat(self, messages, speaker=None):
+        if speaker and speaker.get("name") != "unknown":
+            messages = list(messages)
+            hint = f"(Speaker: {speaker['name']})"
+            if messages and messages[-1]["role"]=="user":
+                messages[-1] = {**messages[-1], "content": f"{hint} {messages[-1]['content']}"}
+        try:
+            headers = {}
+            if self.api_key:
+                headers["Authorization"] = f"Bearer {self.api_key}"
+            payload = {"model": self.model, "messages": messages}
+            r = requests.post(f"{self.base_url}/v1/chat/completions", json=payload, headers=headers, timeout=30)
+            r.raise_for_status()
+            data = r.json()
+            return data["choices"][0]["message"]["content"]
+        except Exception as e:
+            log.error(f"Hermes API error: {e}")
+            return f"Hermes API offline: {e}"

jarvis/llm/ollama.py

@@ -0,0 +1,38 @@
+from .base import LLMEngine
+import logging, requests
+
+log = logging.getLogger(__name__)
+
+class OllamaLLM(LLMEngine):
+    def __init__(self, model="qwen2.5:3b", host="http://localhost:11434", temperature=0.7):
+        self.model = model
+        self.host = host.rstrip('/')
+        self.temperature = temperature
+
+    def chat(self, messages, speaker=None):
+        if speaker and speaker.get("name") and speaker["name"] != "unknown":
+            sys_add = f"[Speaker: {speaker['name']} voice={speaker.get('voice_conf',0):.2f} face={speaker.get('face_conf',0):.2f}]"
+            messages = list(messages)
+            if messages and messages[-1]["role"] == "user":
+                messages[-1] = {**messages[-1], "content": f"{sys_add}\n{messages[-1]['content']}"}
+        try:
+            payload = {
+                "model": self.model,
+                "messages": messages,
+                "stream": False,
+                "options": {"temperature": self.temperature}
+            }
+            r = requests.post(f"{self.host}/api/chat", json=payload, timeout=90)
+            r.raise_for_status()
+            data = r.json()
+            return data.get("message", {}).get("content", "") or data.get("response","")
+        except Exception as e:
+            log.error(f"Ollama error: {e}")
+            return f"Sorry, my local brain is offline ({e})."
+
+    def check_available(self):
+        try:
+            r = requests.get(f"{self.host}/api/tags", timeout=3)
+            return r.status_code == 200
+        except:
+            return False

jarvis/speaker/__init__.py

@@ -0,0 +1 @@
+# jarvis package

jarvis/speaker/embeddings.py

@@ -0,0 +1,54 @@
+import numpy as np
+import logging
+
+log = logging.getLogger(__name__)
+
+class SpeakerEmbedding:
+    def __init__(self, model_name="speechbrain/spkrec-ecapa-voxceleb"):
+        self.model_name = model_name
+        self.model = None
+        self._load()
+
+    def _load(self):
+        try:
+            from speechbrain.inference import EncoderClassifier
+            self.model = EncoderClassifier.from_hparams(source=self.model_name, run_opts={"device":"cpu"})
+            log.info(f"Speaker embedding loaded: {self.model_name}")
+        except Exception as e:
+            log.warning(f"Speaker embedding not available: {e}, using fallback")
+            self.model = None
+
+    def embed(self, audio: np.ndarray, sr: int = 16000) -> np.ndarray:
+        if self.model is not None:
+            try:
+                import torch
+                if audio.dtype != np.float32:
+                    audio = audio.astype(np.float32)
+                tensor = torch.from_numpy(audio).unsqueeze(0)
+                with torch.no_grad():
+                    emb, _ = self.model.encode_batch(tensor)
+                emb = emb.squeeze().cpu().numpy()
+                emb = emb / (np.linalg.norm(emb)+1e-9)
+                return emb
+            except Exception as e:
+                log.warning(f"ECAPA embed failed: {e}, fallback")
+        # Fallback: simple stats embedding 192-d
+        try:
+            n = len(audio)
+            if n == 0:
+                return np.zeros(192, dtype=np.float32)
+            chunk_size = max(1, n // 20)
+            feats = []
+            for i in range(0, n, chunk_size):
+                chunk = audio[i:i+chunk_size]
+                if len(chunk)==0:
+                    continue
+                feats.extend([np.mean(chunk), np.std(chunk)])
+            arr = np.array(feats[:192], dtype=np.float32)
+            if len(arr) < 192:
+                arr = np.pad(arr, (0, 192-len(arr)))
+            arr = arr / (np.linalg.norm(arr)+1e-9)
+            return arr
+        except Exception as e:
+            log.error(f"Fallback embed failed: {e}")
+            return np.zeros(192, dtype=np.float32)

jarvis/speaker/identifier.py

@@ -0,0 +1,28 @@
+import numpy as np
+import logging
+from .embeddings import SpeakerEmbedding
+from .store import SpeakerStore
+
+log = logging.getLogger(__name__)
+
+class SpeakerIdentifier:
+    def __init__(self, enrollment_dir="data/speakers", threshold=0.65, embedding_model="speechbrain/spkrec-ecapa-voxceleb"):
+        self.threshold = threshold
+        self.embedder = SpeakerEmbedding(model_name=embedding_model)
+        self.store = SpeakerStore(enrollment_dir=enrollment_dir)
+
+    def identify(self, audio: np.ndarray, sr=16000):
+        emb = self.embedder.embed(audio, sr)
+        result = self.store.identify(emb, threshold=self.threshold)
+        log.info(f"Speaker ID: {result['name']} conf={result['confidence']:.2f}")
+        return {
+            "name": result["name"],
+            "voice_conf": result["confidence"],
+            "scores": result["scores"],
+            "embedding": emb
+        }
+
+    def enroll(self, name: str, audio: np.ndarray, sr=16000, meta=None):
+        emb = self.embedder.embed(audio, sr)
+        self.store.add_sample(name, emb, meta=meta)
+        return emb

jarvis/speaker/store.py

@@ -0,0 +1,78 @@
+import json, numpy as np
+from pathlib import Path
+import logging
+
+log = logging.getLogger(__name__)
+
+class SpeakerStore:
+    def __init__(self, enrollment_dir="data/speakers"):
+        self.dir = Path(enrollment_dir)
+        self.dir.mkdir(parents=True, exist_ok=True)
+        self.speakers = {}
+        self.load()
+
+    def load(self):
+        self.speakers = {}
+        for file in self.dir.glob("*.json"):
+            try:
+                with open(file) as f:
+                    data = json.load(f)
+                name = data.get("name", file.stem)
+                avg_path = self.dir / f"{name}_avg.npy"
+                if avg_path.exists():
+                    avg = np.load(avg_path)
+                else:
+                    continue
+                self.speakers[name] = {"avg": avg, "meta": data.get("meta", {}), "file": file}
+                log.info(f"Loaded speaker {name}")
+            except Exception as e:
+                log.warning(f"Failed loading {file}: {e}")
+
+    def add_sample(self, name: str, embedding: np.ndarray, meta=None):
+        existing = list(self.dir.glob(f"{name}_*.npy"))
+        idx = len([p for p in existing if "_avg" not in p.name])
+        sample_path = self.dir / f"{name}_{idx}.npy"
+        np.save(sample_path, embedding)
+        all_embs = []
+        for p in self.dir.glob(f"{name}_*.npy"):
+            if "_avg" in p.name:
+                continue
+            try:
+                all_embs.append(np.load(p))
+            except:
+                pass
+        if not all_embs:
+            return
+        avg = np.mean(np.stack(all_embs), axis=0)
+        avg = avg / (np.linalg.norm(avg)+1e-9)
+        np.save(self.dir / f"{name}_avg.npy", avg)
+        meta_file = self.dir / f"{name}.json"
+        meta_data = {"name": name, "samples": len(all_embs), "meta": meta or {}}
+        with open(meta_file, 'w') as f:
+            json.dump(meta_data, f, indent=2)
+        self.speakers[name] = {"avg": avg, "meta": meta_data.get("meta", {}), "file": meta_file}
+        log.info(f"Enrolled {name} samples={len(all_embs)}")
+
+    def identify(self, embedding: np.ndarray, threshold: float = 0.65):
+        if not self.speakers:
+            return {"name": "unknown", "confidence": 0.0, "scores": {}}
+        scores = {}
+        best_name = "unknown"
+        best_score = -1
+        for name, data in self.speakers.items():
+            avg = data["avg"]
+            if avg.shape != embedding.shape:
+                min_dim = min(len(avg), len(embedding))
+                sim = float(np.dot(avg[:min_dim], embedding[:min_dim]))
+            else:
+                sim = float(np.dot(avg, embedding))
+            scores[name] = sim
+            if sim > best_score:
+                best_score = sim
+                best_name = name
+        if best_score < threshold:
+            return {"name": "unknown", "confidence": best_score, "scores": scores}
+        return {"name": best_name, "confidence": best_score, "scores": scores}
+
+    def list_speakers(self):
+        return list(self.speakers.keys())

jarvis/stt/__init__.py

@@ -0,0 +1 @@
+# jarvis package

jarvis/stt/base.py

@@ -0,0 +1,4 @@
+import numpy as np
+class STTEngine:
+    def transcribe(self, audio: np.ndarray, sr: int = 16000) -> dict:
+        raise NotImplementedError

jarvis/stt/faster_whisper_engine.py

@@ -0,0 +1,45 @@
+import numpy as np
+import logging
+from .base import STTEngine
+
+log = logging.getLogger(__name__)
+
+class FasterWhisperSTT(STTEngine):
+    def __init__(self, model="base", language="en", device="cpu", compute_type="int8", vad_filter=False):
+        self.model_name = model
+        self.language = language
+        self.device = device
+        self.compute_type = compute_type
+        self.vad_filter = vad_filter
+        self.model = None
+        self._load()
+
+    def _load(self):
+        try:
+            from faster_whisper import WhisperModel
+            log.info(f"Loading faster-whisper {self.model_name} {self.device} {self.compute_type}")
+            self.model = WhisperModel(self.model_name, device=self.device, compute_type=self.compute_type)
+            log.info("faster-whisper loaded")
+        except Exception as e:
+            log.error(f"Failed to load faster-whisper: {e}")
+            self.model = None
+
+    def transcribe(self, audio: np.ndarray, sr=16000) -> dict:
+        if self.model is None:
+            return {"text": "", "language": "", "confidence": 0.0, "error": "model not loaded"}
+        if audio.dtype != np.float32:
+            audio = audio.astype(np.float32)
+        try:
+            segments, info = self.model.transcribe(
+                audio,
+                language=self.language if self.language != "auto" else None,
+                vad_filter=self.vad_filter,
+                beam_size=5,
+                condition_on_previous_text=False,
+            )
+            text = " ".join([s.text for s in segments]).strip()
+            log.debug(f"STT: {text[:200]}")
+            return {"text": text, "language": info.language, "confidence": float(info.language_probability)}
+        except Exception as e:
+            log.error(f"STT error: {e}")
+            return {"text": "", "language": "", "confidence": 0.0, "error": str(e)}

jarvis/tts/__init__.py

@@ -0,0 +1 @@
+# jarvis package

jarvis/tts/base.py

@@ -0,0 +1,3 @@
+class TTSEngine:
+    def speak(self, text: str):
+        raise NotImplementedError

jarvis/tts/piper.py

@@ -0,0 +1,95 @@
+import logging, os, subprocess, tempfile, pathlib
+from .base import TTSEngine
+log = logging.getLogger(__name__)
+
+class PiperTTS(TTSEngine):
+    def __init__(self, voice="en_US-lessac-medium", models_dir="models", volume=0.9):
+        self.voice = voice
+        self.models_dir = pathlib.Path(models_dir)
+        self.models_dir.mkdir(parents=True, exist_ok=True)
+        self.volume = volume
+        self.voice_path = None
+        self.config_path = None
+        self._ensure_model()
+
+    def _ensure_model(self):
+        pattern = self.models_dir / f"{self.voice}.onnx"
+        if pattern.exists():
+            self.voice_path = pattern
+            self.config_path = pathlib.Path(str(pattern) + ".json")
+            log.info(f"Piper voice found: {self.voice_path}")
+            return
+        for p in self.models_dir.rglob("*.onnx"):
+            if self.voice in p.name:
+                self.voice_path = p
+                self.config_path = pathlib.Path(str(p) + ".json")
+                log.info(f"Piper voice found: {p}")
+                return
+        log.warning(f"Piper voice {self.voice} not in {self.models_dir}")
+
+    def _download_voice(self):
+        try:
+            log.info(f"Downloading Piper voice {self.voice}")
+            result = subprocess.run(["python3", "-m", "piper.download_voices", self.voice], cwd=str(self.models_dir), capture_output=True, text=True, timeout=60)
+            log.info(result.stdout[:500])
+            self._ensure_model()
+        except Exception as e:
+            log.warning(f"Piper download failed: {e}")
+
+    def speak(self, text: str):
+        if not text.strip():
+            return
+        clean = text.replace("*","").replace("#","").replace("`","")[:600]
+        if self.voice_path is None:
+            self._download_voice()
+        if self.voice_path and self.voice_path.exists():
+            try:
+                with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tf:
+                    wav_path = tf.name
+                cmd = ["piper", "--model", str(self.voice_path), "--output_file", wav_path]
+                proc = subprocess.run(cmd, input=clean.encode(), capture_output=True, timeout=15)
+                if proc.returncode==0 and os.path.exists(wav_path):
+                    self._play_wav(wav_path)
+                    try:
+                        os.remove(wav_path)
+                    except:
+                        pass
+                    return
+                else:
+                    log.warning(f"Piper failed: {proc.stderr[:300]}")
+            except Exception as e:
+                log.warning(f"Piper speak error: {e}")
+        self._fallback_speak(clean)
+
+    def _play_wav(self, wav_path):
+        try:
+            import soundfile as sf
+            import sounddevice as sd
+            data, sr = sf.read(wav_path)
+            data = data * self.volume
+            sd.play(data, sr)
+            sd.wait()
+        except Exception as e:
+            log.debug(f"sd playback fail {e}, trying aplay")
+            try:
+                subprocess.run(["aplay", wav_path], capture_output=True, timeout=10)
+            except Exception as e2:
+                log.warning(f"aplay fail: {e2}")
+
+    def _fallback_speak(self, text):
+        try:
+            subprocess.run(["espeak", text], capture_output=True, timeout=10)
+            return
+        except:
+            pass
+        try:
+            subprocess.run(["spd-say", text], capture_output=True, timeout=10)
+            return
+        except:
+            pass
+        log.info(f"[TTS fallback] {text}")
+        print(f"Jarvis says: {text}")
+
+class EchoTTS(TTSEngine):
+    def speak(self, text: str):
+        print(f"[Jarvis TTS]: {text}")

jarvis/vision/__init__.py

@@ -0,0 +1 @@
+# jarvis package

jarvis/vision/camera.py

@@ -0,0 +1,41 @@
+import cv2, logging
+import numpy as np
+from typing import Optional
+log = logging.getLogger(__name__)
+class Camera:
+    def __init__(self, camera_id=0, width=640, height=480):
+        self.camera_id = camera_id
+        self.width = width
+        self.height = height
+        self.cap: Optional[cv2.VideoCapture] = None
+    def open(self):
+        if self.cap and self.cap.isOpened():
+            return True
+        try:
+            self.cap = cv2.VideoCapture(self.camera_id)
+            self.cap.set(cv2.CAP_PROP_FRAME_WIDTH, self.width)
+            self.cap.set(cv2.CAP_PROP_FRAME_HEIGHT, self.height)
+            for _ in range(5):
+                self.cap.read()
+            if not self.cap.isOpened():
+                log.warning(f"Camera {self.camera_id} failed to open")
+                return False
+            log.info(f"Camera {self.camera_id} opened")
+            return True
+        except Exception as e:
+            log.error(f"Camera open error: {e}")
+            return False
+    def capture_frame(self) -> Optional[np.ndarray]:
+        if not self.open():
+            return None
+        ret, frame = self.cap.read()
+        if not ret:
+            return None
+        return frame
+    def close(self):
+        if self.cap:
+            try:
+                self.cap.release()
+            except:
+                pass
+            self.cap=None

jarvis/vision/face_detector.py

@@ -0,0 +1,25 @@
+import cv2, numpy as np, logging, os
+from typing import List, Tuple
+log = logging.getLogger(__name__)
+class FaceDetector:
+    def __init__(self):
+        self.cascade=None
+        self._load()
+    def _load(self):
+        try:
+            cascade_path = cv2.data.haarcascades + 'haarcascade_frontalface_default.xml'
+            if os.path.exists(cascade_path):
+                self.cascade = cv2.CascadeClassifier(cascade_path)
+                log.info("Haar face detector loaded")
+        except Exception as e:
+            log.warning(f"Face detector load failed: {e}")
+    def detect(self, frame: np.ndarray) -> List[Tuple[int,int,int,int]]:
+        if self.cascade is None:
+            return []
+        try:
+            gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
+            faces = self.cascade.detectMultiScale(gray, scaleFactor=1.1, minNeighbors=5, minSize=(80,80))
+            return [(int(x),int(y),int(w),int(h)) for (x,y,w,h) in faces]
+        except Exception as e:
+            log.debug(f"Detect error: {e}")
+            return []

jarvis/vision/face_identifier.py

@@ -0,0 +1,128 @@
+import json, numpy as np, cv2, logging
+from pathlib import Path
+from .camera import Camera
+from .face_detector import FaceDetector
+log = logging.getLogger(__name__)
+
+class FaceStore:
+    def __init__(self, enrollment_dir="data/faces"):
+        self.dir = Path(enrollment_dir)
+        self.dir.mkdir(parents=True, exist_ok=True)
+        self.known = {}
+        self.load()
+    def _encoding(self, frame, bbox=None):
+        try:
+            import face_recognition
+            rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
+            if bbox:
+                x,y,w,h = bbox
+                loc = [(y, x+w, y+h, x)]
+                encs = face_recognition.face_encodings(rgb, known_face_locations=loc)
+            else:
+                encs = face_recognition.face_encodings(rgb)
+            if encs:
+                return encs[0]
+        except Exception as e:
+            log.debug(f"face_recog encoding fail: {e}")
+        try:
+            if bbox:
+                x,y,w,h = bbox
+                crop = frame[y:y+h, x:x+w]
+            else:
+                crop = frame
+            if crop.size == 0:
+                return None
+            crop = cv2.resize(crop, (64,64))
+            gray = cv2.cvtColor(crop, cv2.COLOR_BGR2GRAY)
+            flat = gray.flatten().astype(np.float32)
+            flat = flat / (np.linalg.norm(flat)+1e-9)
+            return flat
+        except Exception as e:
+            log.debug(f"Fallback encoding fail: {e}")
+            return None
+    def load(self):
+        self.known={}
+        for jf in self.dir.glob("*.json"):
+            try:
+                with open(jf) as f:
+                    data=json.load(f)
+                name=data["name"]
+                enc_files = list(self.dir.glob(f"{name}_*.npy"))
+                encs=[]
+                for ef in enc_files:
+                    if "_avg" in ef.name:
+                        continue
+                    try:
+                        encs.append(np.load(ef))
+                    except:
+                        pass
+                if encs:
+                    self.known[name]=encs
+                    log.info(f"Loaded face {name}: {len(encs)}")
+            except Exception as e:
+                log.warning(f"Load face {jf}: {e}")
+    def add_sample(self, name, frame, bbox=None):
+        enc=self._encoding(frame,bbox)
+        if enc is None:
+            return False
+        existing=list(self.dir.glob(f"{name}_*.npy"))
+        idx=len([p for p in existing if "_avg" not in p.name])
+        path=self.dir/f"{name}_{idx}.npy"
+        np.save(path,enc)
+        meta_path=self.dir/f"{name}.json"
+        with open(meta_path,'w') as f:
+            json.dump({"name":name,"samples":idx+1},f)
+        self.load()
+        return True
+    def identify(self, frame, bbox=None, threshold=0.55):
+        enc=self._encoding(frame,bbox)
+        if enc is None:
+            return {"name":"unknown","confidence":0.0}
+        best_name="unknown"
+        best_score=-1
+        scores={}
+        for name, enc_list in self.known.items():
+            sims=[]
+            for ke in enc_list:
+                if ke.shape != enc.shape:
+                    min_dim=min(len(ke),len(enc))
+                    sim=float(np.dot(ke[:min_dim], enc[:min_dim]))
+                else:
+                    if len(enc)==128:
+                        dist=np.linalg.norm(ke-enc)
+                        sim=max(0,1-dist)
+                    else:
+                        sim=float(np.dot(ke,enc))
+                sims.append(sim)
+            max_sim=max(sims) if sims else 0
+            scores[name]=max_sim
+            if max_sim>best_score:
+                best_score=max_sim
+                best_name=name
+        if best_score<threshold:
+            return {"name":"unknown","confidence":best_score,"scores":scores}
+        return {"name":best_name,"confidence":best_score,"scores":scores}
+
+class FaceIdentifier:
+    def __init__(self, camera_id=0, enrollment_dir="data/faces", threshold=0.55):
+        self.camera=Camera(camera_id=camera_id)
+        self.detector=FaceDetector()
+        self.store=FaceStore(enrollment_dir=enrollment_dir)
+        self.threshold=threshold
+    def check(self):
+        frame=self.camera.capture_frame()
+        if frame is None:
+            return {"name":"unknown","face_conf":0.0,"error":"no camera"}
+        faces=self.detector.detect(frame)
+        if not faces:
+            return {"name":"unknown","face_conf":0.0,"faces_found":0,"frame":frame}
+        faces_sorted=sorted(faces, key=lambda b: b[2]*b[3], reverse=True)
+        best_bbox=faces_sorted[0]
+        result=self.store.identify(frame,bbox=best_bbox,threshold=self.threshold)
+        result["faces_found"]=len(faces)
+        result["bbox"]=best_bbox
+        result["face_conf"]=result.get("confidence",0.0)
+        result["frame"]=frame
+        return result
+    def close(self):
+        self.camera.close()

main.py

@@ -0,0 +1,7 @@
+#!/usr/bin/env python3
+import sys
+from jarvis.cli import main
+
+if __name__ == '__main__':
+    sys.argv[0] = 'jarvis'
+    main()

pyproject.toml

@@ -0,0 +1,30 @@
+[project]
+name = "jarvis-v2"
+version = "0.2.0"
+description = "Always-listening voice assistant - CPU optimized"
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = [
+    "numpy<2",
+    "sounddevice",
+    "soundfile",
+    "openwakeword",
+    "faster-whisper",
+    "onnxruntime",
+    "torch",
+    "torchaudio",
+    "silero-vad",
+    "webrtcvad-wheels",
+    "scipy",
+    "piper-tts",
+    "scikit-learn",
+    "opencv-python-headless",
+    "pyyaml",
+    "requests",
+    "python-dotenv",
+    "ollama",
+    "tqdm",
+]
+
+[project.scripts]
+jarvis = "jarvis.cli:main"

scripts/build_portaudio.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+# Build PortAudio locally in user space if system lib missing
+set -e
+cd /tmp
+if [ -d portaudio ]; then
+    echo "portaudio src exists, skipping clone"
+else
+    git clone https://github.com/PortAudio/portaudio.git
+fi
+cd portaudio
+./configure --prefix=$HOME/.local --without-jack
+make -j$(nproc)
+make install
+echo "Installed to ~/.local/lib"
+echo "Add to LD_LIBRARY_PATH: export LD_LIBRARY_PATH=\$HOME/.local/lib:\$LD_LIBRARY_PATH"
+ldconfig -n $HOME/.local/lib || true

scripts/enroll_face.py

@@ -0,0 +1,10 @@
+import sys
+from jarvis.cli import main
+if len(sys.argv) == 1:
+    print("Usage: python scripts/enroll_face.py --name NAME")
+    sys.exit(1)
+if sys.argv[1] != "enroll-face" and not sys.argv[1].startswith("--"):
+    sys.argv.insert(1, "enroll-face")
+elif sys.argv[1].startswith("--"):
+    sys.argv.insert(1, "enroll-face")
+main()

scripts/enroll_speaker.py

@@ -0,0 +1,13 @@
+import sys
+from jarvis.cli import main
+# Pass through args
+sys.argv[0] = "jarvis"
+if len(sys.argv) == 1:
+    print("Usage: python scripts/enroll_speaker.py --name NAME --mic")
+    sys.exit(1)
+# ensure first arg is enroll-speaker
+if sys.argv[1] != "enroll-speaker" and not sys.argv[1].startswith("--"):
+    sys.argv.insert(1, "enroll-speaker")
+elif sys.argv[1].startswith("--"):
+    sys.argv.insert(1, "enroll-speaker")
+main()

scripts/install_service.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+set -e
+DIR="$(cd "$(dirname "$0")/.." && pwd)"
+SERVICE_FILE="$DIR/systemd/jarvis.service"
+DEST="$HOME/.config/systemd/user/jarvis.service"
+
+mkdir -p "$HOME/.config/systemd/user"
+cp "$SERVICE_FILE" "$DEST"
+
+echo "Reloading systemd user daemon"
+systemctl --user daemon-reload
+echo "Enabling jarvis service"
+systemctl --user enable jarvis.service
+echo "To start: systemctl --user start jarvis"
+echo "To logs: journalctl --user -u jarvis -f"
+echo "To stop: systemctl --user stop jarvis"

scripts/test_mic.py

@@ -0,0 +1,4 @@
+from jarvis.cli import main
+import sys
+sys.argv = ["jarvis","test-mic"]
+main()

scripts/test_stt.py

@@ -0,0 +1,50 @@
+"""Test STT with mic"""
+from jarvis.audio.capture import AudioCapture
+from jarvis.stt.faster_whisper_engine import FasterWhisperSTT
+from jarvis.audio.vad import create_vad
+from jarvis.config import load_config
+import numpy as np, time, collections
+
+cfg = load_config(None)
+vad = create_vad(cfg)
+stt = FasterWhisperSTT(model=cfg.get('stt',{}).get('model','base'), device='cpu', compute_type='int8')
+cap = AudioCapture()
+cap.start()
+
+print("Speak, with 1.5s silence to trigger transcription. Ctrl+C to exit")
+buffer = []
+silence_start = None
+recording = False
+
+try:
+    while True:
+        chunk = cap.read(timeout=0.1)
+        if chunk is None:
+            continue
+        is_speech = vad.is_speech(chunk)
+        if is_speech:
+            if not recording:
+                print("Speech start")
+                recording = True
+            buffer.append(chunk)
+            silence_start = None
+        else:
+            if recording:
+                if silence_start is None:
+                    silence_start = time.time()
+                buffer.append(chunk)
+                if time.time() - silence_start > 1.2:
+                    # transcribe
+                    audio = np.concatenate(buffer)
+                    print(f"Transcribing {len(audio)/16000:.2f}s...")
+                    res = stt.transcribe(audio)
+                    print(f" => {res.get('text')}")
+                    buffer = []
+                    recording = False
+                    silence_start = None
+                    print("Listening again...")
+            # else silence, continue
+except KeyboardInterrupt:
+    pass
+finally:
+    cap.stop()

scripts/test_wakeword.py

@@ -0,0 +1,29 @@
+"""Test wakeword detection without full pipeline"""
+from jarvis.audio.capture import AudioCapture
+from jarvis.audio.wakeword import WakeWordDetector
+from jarvis.audio.vad import create_vad
+from jarvis.config import load_config
+import time
+
+cfg = load_config(None)
+vad = create_vad(cfg)
+ww = WakeWordDetector(models=cfg.get('wakeword',{}).get('models',['hey_jarvis']), threshold=0.45)
+
+cap = AudioCapture()
+cap.start()
+print("Say 'Hey Jarvis'...")
+try:
+    while True:
+        chunk = cap.read(timeout=0.1)
+        if chunk is None:
+            continue
+        # Optional VAD gate
+        if not vad.is_speech(chunk):
+            continue
+        det = ww.detect(chunk)
+        if det:
+            print(f"Wake detected: {det}")
+except KeyboardInterrupt:
+    pass
+finally:
+    cap.stop()

systemd/jarvis.service

@@ -0,0 +1,24 @@
+[Unit]
+Description=Jarvis v2 Voice Assistant - Always Listening
+After=network.target sound.target
+Wants=network.target
+
+[Service]
+Type=simple
+WorkingDirectory=%h/Projects/jarvis-v2
+ExecStart=%h/Projects/jarvis-v2/.venv/bin/jarvis run --config %h/Projects/jarvis-v2/jarvis.yaml
+Restart=on-failure
+RestartSec=5
+Environment=PATH=%h/.local/bin:/usr/bin:/bin
+Environment=PYTHONUNBUFFERED=1
+
+# Logging
+StandardOutput=journal
+StandardError=journal
+
+# Resource limits - protect old iMac
+Nice= -5
+CPUQuota=200%
+
+[Install]
+WantedBy=default.target