whisper-translation/history.md

# Project History: Python Whisper Live Transcription

This document tracks the evolution, technical decisions, and optimizations made for this live audio transcription and translation tool.

## Phase 1: Basic Live Transcription
- **Goal:** Create a simple script to transcribe live audio.
- **Initial Stack:** `openai-whisper`, `sounddevice`, `numpy`.
- **Approach:** 
    - Captured audio in 2-second chunks using `sounddevice`.
    - Used the `tiny.en` model for initial testing.
- **Outcome:** Successful basic transcription, but limited by continuous processing (even during silence).

## Phase 2: Voice Activity Detection (VAD)
- **Goal:** Improve efficiency by only transcribing when someone is speaking.
- **Stack Addition:** `silero-vad`.
- **Approach:** 
    - Integrated Silero VAD to monitor the audio stream.
    - Transcription is only triggered after a speech segment is followed by a period of silence (500ms).
- **Outcome:** Significantly reduced CPU usage and cleaner output.

## Phase 3: Apple Silicon Optimization (M-Series/M2)
- **Goal:** Leverage the M2's Neural Engine and GPU for better performance.
- **Stack Transition:** `mlx-whisper` (via Apple's MLX framework).
- **Decision:** Switched from `openai-whisper` to `mlx-whisper` and upgraded the model to `small.en` for better accuracy without sacrificing speed.
- **Outcome:** Faster inference and better battery efficiency.

## Phase 4: Local Translation
- **Approach A (LLM):** Tried using `SmolLM2-135M` via `mlx-lm` for translation.
    - **Issue:** The LLM was "too talkative," often adding conversational filler or explaining the translation instead of just providing it.
- **Approach B (Dedicated MT):** Switched to `MarianMT` (`Helsinki-NLP/opus-mt-en-es`).
    - **Decision:** Chose a dedicated Translation Model for cleaner, direct mapping from English to Spanish.
- **Technical Hurdle (LZMA Error):** 
    - The local Python environment lacked `_lzma` support, causing `transformers` and `huggingface_hub` to crash.
    - **Solution:** Implemented a comprehensive `lzma` mock in the script to provide necessary constants (`FORMAT_XZ`, etc.) and bypass the system-level limitation.

## Current Status
The project now features a high-performance, Apple Silicon-optimized pipeline that:
1. Detects speech using **Silero VAD**.
2. Transcribes using **MLX-Whisper (small.en)**.
3. Translates using **MarianMT (EN-ES)**.
4. Operates entirely locally with hardware acceleration.

## Phase 5: Simultaneous Multi-Language Translation
- **Goal:** Provide translations in Spanish, French, and Arabic at the same time.
- **Approach:** 
    - Refactored the script to support a dictionary of multiple `MarianMT` models.
    - Each transcribed English segment is passed through each loaded translation engine sequentially.
- **Performance on M2:** Loading 3-4 specialized models + Whisper is highly efficient, using ~1.5GB of RAM and providing near-instant results.

## Phase 6: Memory & Generation Safety
- **Issue:** Occasionally, long inputs or model glitches caused "runaway" translation generation, which could consume excessive memory.
- **Solution:** 
    - Artificially truncated input transcription to a maximum of 250 characters.
    - Added `max_new_tokens=150` to the translation generation call to ensure the model terminates even if it gets stuck in a loop.

## Phase 7: Multilingual Detection & Bridge Translation
- **Goal:** Support input in any language, detect it, and translate to English + others.
- **Approach:** 
    - Switched to `whisper-small-mlx` (multilingual).
    - **Hub-and-Spoke Model:** If a non-English language is detected, Whisper's `task="translate"` is used to create an English "bridge" text, which is then fed into the specialized MarianMT models.
- **Outcome:** Full support for multilingual input with centralized translation.

## Phase 8: Compilation to Binary
- **Goal:** Distribute the script as a single, standalone executable for macOS terminal.
- **Tool:** `PyInstaller`.
- **Process:** 
    - Used `--onefile` to bundle the entire Python runtime and its heavy dependencies (Torch, MLX, Transformers).
    - Excluded build artifacts (`build/`, `dist/`, `.spec`) from the repository.
- **Build Script:** 
    ```bash
    chmod +x build.sh
    ./build.sh
    ```
- **Troubleshooting:** Fixed a runtime `ModuleNotFoundError: No module named 'mlx._reprlib_fix'` by explicitly adding `--collect-all mlx` and `--hidden-import=mlx._reprlib_fix` to the PyInstaller configuration. Also added `multiprocessing.freeze_support()` to fix infinite loops in the compiled binary.

## Phase 9: Real-Time Server Ingest
- **Goal:** Send live captions and translations to a central ingest server.
- **Backend:** `https://emiapi.reynafamily.com/live-captions/ingest`.
- **Approach:** 
    - Implemented a background `ingest_worker` thread to handle HTTP POST requests without stalling the audio processing.
    - **Flat JSON Schema:** Used a key-value format as requested (e.g., `original`, `es`, `en`, `fr`).
    - **Reliability:** Integrated exponential backoff retries (1s to 15s) to handle network or server failures.

## Phase 10: CLI Configuration & Documentation
- **Goal:** Make the tool fully configurable without editing the code.
- **Approach:** 
    - Integrated `argparse` to allow dynamic selection of translation languages (`-es`, `-en`, `-ar`, `-fr`), data ingestion (`-i`), and audio device (`-d`).
    - Added device listing capability (`-l`).
    - Excluded the `venv` directory from git and generated `requirements.txt`.
    - Created a comprehensive `README.md` containing setup instructions, usage examples, and a technical note on universal models (like NLLB-200).
- **Fix:** Added missing `sentencepiece` dependency required by MarianMT models.

## Phase 11: Speed Optimization (Quantization & Streaming)
- **Goal:** Reduce latency and improve real-time feedback.
- **Approach:**
    - **Streaming Mode (`-s`):** Implemented a 1-second rolling draft transcription that continuously updates the console while the user is still speaking.
    - **Quantization (`-q`):** Added support for dynamically swapping to the 4-bit quantized Whisper model (`mlx-community/whisper-small-mlx-4bit`) for faster inference on Apple Silicon with lower memory bandwidth.

## Phase 12: Accuracy & Context Optimization
- **Goal:** Improve translation quality of short or broken audio chunks.
- **Approach:**
    - **Prompt Caching (`-c`):** Implemented a rolling context buffer that feeds the last 200 characters of previously transcribed text back into Whisper as an `initial_prompt`, maintaining sentence continuity.
    - **Language Bypassing (`--lang`):** Added the ability to hardcode the source language to skip the Whisper language identification phase on every chunk.
    - **Heuristic Punctuation Buffering (Reverted):** Briefly implemented a system to hold English translations until a definitive punctuation mark was reached to prevent grammatical errors. This was reverted because Whisper's punctuation generation is not 100% reliable, leading to translations getting "stuck" in the buffer indefinitely if no period was generated.

## Phase 13: Stability & Stuck Detection (Watchdog)
- **Goal:** Prevent captions from getting "stuck" during long periods of music, singing, or model hallucinations.
- **Approach:**
    - **Transcription Watchdog:** Implemented a logic that monitors the audio buffer duration and the time since the last *unique* draft change. If the buffer exceeds 12s without a change for 7s, or if it hits an absolute limit of 25s, it forces a clean reset.
    - **Hallucination Filtering:** Added regex-based detection for common Whisper hallucinations (e.g., "Thanks for watching", "Please subscribe") and a repetition counter to identify and discard high-frequency loops (e.g., "Hallelujah" repeated 20 times).
    - **Buffer Optimization:** Reduced the default maximum buffer from 30s to 20s (now configurable via `--max-buffer`) to ensure more frequent flushes during continuous sound.
- **Outcome:** Significantly improved reliability during live church services and musical performances where Whisper previously tended to "hang" or hallucinate.

## Phase 14: Multi-Channel & Language Filtering
- **Goal:** Isolate specific speakers or languages when the audio feed contains multiple mixed sources.
- **Approach:**
    - **Stereo Channel Selection:** Added `--channels` and `--pick-channel [0|1]` flags. This allows the tool to pull a specific audio channel (e.g., English on Left, Translation on Right) from a stereo feed, ignoring the other.
    - **Language Filtering (`--filter-lang`):** Integrated a mechanism to discard any segment where the detected language does not match the hardcoded `--lang` parameter. 
- **Outcome:** Enabled the ability to "focus" the transcription engine on a single speaker even when the audio input is a complex mix.

## Phase 15: Potential Future Optimizations (Backlog)
- **Decoding Parameters:**
    - **Temperature Fallback:** Use a tuple `(0.0, 0.2, 0.4, 0.6, 0.8, 1.0)` to allow Whisper to re-try failed transcriptions with higher randomness (crucial for music/singing).
    - **Beam Size Tuning:** Set `beam_size=1` for Draft Mode (3x faster) and `beam_size=5` for final segments (higher accuracy).
    - **Token Suppression:** Native suppression of music symbols or common hallucination tokens at the decoder level.
    - **Heuristic Thresholds:** Utilize `logprob_threshold` (confidence) and `compression_ratio_threshold` (repetition) to programmatically identify and discard bad transcriptions before they reach the user.
- **Structural Features:**
    - **Word-Level Timestamps:** Enable `word_timestamps=True` to provide granular timing data for front-end caption highlighting.

## Phase 16: Decoder Controls + Bridge Context Refinements
- **Goal:** Improve English caption stability and translation quality without sacrificing runtime compatibility.
- **Approach:**
    - **Whisper Decoder Controls (implemented):** Added configurable decoding flags for final segments and drafts, including temperature fallback and quality thresholds (`logprob_threshold`, `compression_ratio_threshold`), with compatibility fallback when unsupported.
    - **Beam Compatibility Hardening:** Defaulted beam sizes to greedy-safe values and auto-retry without `beam_size` when the runtime reports "Beam search decoder is not yet implemented."
    - **Dedicated English Bridge Context:** Split context handling into source-language context and a separate English context used specifically for `task="translate"` bridge generation.
    - **Sentence-Aware MT Chunking:** Replaced hard truncation with sentence-aware splitting for long English bridge text before translation.
    - **Marian Generation Tuning:** Added configurable generation controls (`num_beams`, `no_repeat_ngram_size`, `length_penalty`, `repetition_penalty`, `early_stopping`) to reduce repetitive or unstable outputs.
- **Outcome:** Better continuity for non-English to English bridging, fewer clipped translations on long segments, and cleaner target-language output with safer defaults for current `mlx_whisper` builds.