whisper-translation/history.md

Project History: Python Whisper Live Transcription

This document tracks the evolution, technical decisions, and optimizations made for this live audio transcription and translation tool.

Phase 1: Basic Live Transcription

• Goal: Create a simple script to transcribe live audio.
• Initial Stack: openai-whisper, sounddevice, numpy.
• Approach:
  • Captured audio in 2-second chunks using sounddevice.
  • Used the tiny.en model for initial testing.
• Outcome: Successful basic transcription, but limited by continuous processing (even during silence).

Phase 2: Voice Activity Detection (VAD)

• Goal: Improve efficiency by only transcribing when someone is speaking.
• Stack Addition: silero-vad.
• Approach:
  • Integrated Silero VAD to monitor the audio stream.
  • Transcription is only triggered after a speech segment is followed by a period of silence (500ms).
• Outcome: Significantly reduced CPU usage and cleaner output.

Phase 3: Apple Silicon Optimization (M-Series/M2)

• Goal: Leverage the M2's Neural Engine and GPU for better performance.
• Stack Transition: mlx-whisper (via Apple's MLX framework).
• Decision: Switched from openai-whisper to mlx-whisper and upgraded the model to small.en for better accuracy without sacrificing speed.
• Outcome: Faster inference and better battery efficiency.

Phase 4: Local Translation

• Approach A (LLM): Tried using SmolLM2-135M via mlx-lm for translation.
  • Issue: The LLM was "too talkative," often adding conversational filler or explaining the translation instead of just providing it.
• Approach B (Dedicated MT): Switched to MarianMT (Helsinki-NLP/opus-mt-en-es).
  • Decision: Chose a dedicated Translation Model for cleaner, direct mapping from English to Spanish.
• Technical Hurdle (LZMA Error):
  • The local Python environment lacked _lzma support, causing transformers and huggingface_hub to crash.
  • Solution: Implemented a comprehensive lzma mock in the script to provide necessary constants (FORMAT_XZ, etc.) and bypass the system-level limitation.

Current Status

The project now features a high-performance, Apple Silicon-optimized pipeline that:

1. Detects speech using Silero VAD.
2. Transcribes using MLX-Whisper (small.en).
3. Translates using MarianMT (EN-ES).
4. Operates entirely locally with hardware acceleration.

Phase 5: Simultaneous Multi-Language Translation

• Goal: Provide translations in Spanish, French, and Arabic at the same time.
• Approach:
  • Refactored the script to support a dictionary of multiple MarianMT models.
  • Each transcribed English segment is passed through each loaded translation engine sequentially.
• Performance on M2: Loading 3-4 specialized models + Whisper is highly efficient, using ~1.5GB of RAM and providing near-instant results.

Phase 6: Memory & Generation Safety

• Issue: Occasionally, long inputs or model glitches caused "runaway" translation generation, which could consume excessive memory.
• Solution:
  • Artificially truncated input transcription to a maximum of 250 characters.
  • Added max_new_tokens=150 to the translation generation call to ensure the model terminates even if it gets stuck in a loop.

Phase 7: Multilingual Detection & Bridge Translation

• Goal: Support input in any language, detect it, and translate to English + others.
• Approach:
  • Switched to whisper-small-mlx (multilingual).
  • Hub-and-Spoke Model: If a non-English language is detected, Whisper's task="translate" is used to create an English "bridge" text, which is then fed into the specialized MarianMT models.
• Outcome: Full support for multilingual input with centralized translation.

Phase 8: Compilation to Binary

• Goal: Distribute the script as a single, standalone executable for macOS terminal.
• Tool: PyInstaller.
• Process:
  • Used --onefile to bundle the entire Python runtime and its heavy dependencies (Torch, MLX, Transformers).
  • Excluded build artifacts (build/, dist/, .spec) from the repository.
• Build Script:

  chmod +x build.sh
  ./build.sh
  

• Troubleshooting: Fixed a runtime ModuleNotFoundError: No module named 'mlx._reprlib_fix' by explicitly adding --collect-all mlx and --hidden-import=mlx._reprlib_fix to the PyInstaller configuration. Also added multiprocessing.freeze_support() to fix infinite loops in the compiled binary.

Phase 9: Real-Time Server Ingest

• Goal: Send live captions and translations to a central ingest server.
• Backend: https://emiapi.reynafamily.com/live-captions/ingest.
• Approach:
  • Implemented a background ingest_worker thread to handle HTTP POST requests without stalling the audio processing.
  • Flat JSON Schema: Used a key-value format as requested (e.g., original, es, en, fr).
  • Reliability: Integrated exponential backoff retries (1s to 15s) to handle network or server failures.

Phase 10: CLI Configuration & Documentation

• Goal: Make the tool fully configurable without editing the code.
• Approach:
  • Integrated argparse to allow dynamic selection of translation languages (-es, -en, -ar, -fr), data ingestion (-i), and audio device (-d).
  • Added device listing capability (-l).
  • Excluded the venv directory from git and generated requirements.txt.
  • Created a comprehensive README.md containing setup instructions, usage examples, and a technical note on universal models (like NLLB-200).
• Fix: Added missing sentencepiece dependency required by MarianMT models.

Phase 11: Speed Optimization (Quantization & Streaming)

• Goal: Reduce latency and improve real-time feedback.
• Approach:
  • Streaming Mode (-s): Implemented a 1-second rolling draft transcription that continuously updates the console while the user is still speaking.
  • Quantization (-q): Added support for dynamically swapping to the 4-bit quantized Whisper model (mlx-community/whisper-small-mlx-4bit) for faster inference on Apple Silicon with lower memory bandwidth.

Phase 12: Accuracy & Context Optimization

• Goal: Improve translation quality of short or broken audio chunks.
• Approach:
  • Prompt Caching (-c): Implemented a rolling context buffer that feeds the last 200 characters of previously transcribed text back into Whisper as an initial_prompt, maintaining sentence continuity.
  • Language Bypassing (--lang): Added the ability to hardcode the source language to skip the Whisper language identification phase on every chunk.
  • Heuristic Punctuation Buffering (Reverted): Briefly implemented a system to hold English translations until a definitive punctuation mark was reached to prevent grammatical errors. This was reverted because Whisper's punctuation generation is not 100% reliable, leading to translations getting "stuck" in the buffer indefinitely if no period was generated.

Phase 13: Stability & Stuck Detection (Watchdog)

• Goal: Prevent captions from getting "stuck" during long periods of music, singing, or model hallucinations.
• Approach:
  • Transcription Watchdog: Implemented a logic that monitors the audio buffer duration and the time since the last unique draft change. If the buffer exceeds 12s without a change for 7s, or if it hits an absolute limit of 25s, it forces a clean reset.
  • Hallucination Filtering: Added regex-based detection for common Whisper hallucinations (e.g., "Thanks for watching", "Please subscribe") and a repetition counter to identify and discard high-frequency loops (e.g., "Hallelujah" repeated 20 times).
  • Buffer Optimization: Reduced the default maximum buffer from 30s to 20s (now configurable via --max-buffer) to ensure more frequent flushes during continuous sound.
• Outcome: Significantly improved reliability during live church services and musical performances where Whisper previously tended to "hang" or hallucinate.

Phase 14: Multi-Channel & Language Filtering

• Goal: Isolate specific speakers or languages when the audio feed contains multiple mixed sources.
• Approach:
  • Stereo Channel Selection: Added --channels and --pick-channel [0|1] flags. This allows the tool to pull a specific audio channel (e.g., English on Left, Translation on Right) from a stereo feed, ignoring the other.
  • Language Filtering (--filter-lang): Integrated a mechanism to discard any segment where the detected language does not match the hardcoded --lang parameter.
• Outcome: Enabled the ability to "focus" the transcription engine on a single speaker even when the audio input is a complex mix.

Phase 15: Potential Future Optimizations (Backlog)

• Decoding Parameters:
  • Temperature Fallback: Use a tuple (0.0, 0.2, 0.4, 0.6, 0.8, 1.0) to allow Whisper to re-try failed transcriptions with higher randomness (crucial for music/singing).
  • Beam Size Tuning: Set beam_size=1 for Draft Mode (3x faster) and beam_size=5 for final segments (higher accuracy).
  • Token Suppression: Native suppression of music symbols or common hallucination tokens at the decoder level.
  • Heuristic Thresholds: Utilize logprob_threshold (confidence) and compression_ratio_threshold (repetition) to programmatically identify and discard bad transcriptions before they reach the user.
• Structural Features:
  • Word-Level Timestamps: Enable word_timestamps=True to provide granular timing data for front-end caption highlighting.

Phase 16: Decoder Controls + Bridge Context Refinements

• Goal: Improve English caption stability and translation quality without sacrificing runtime compatibility.
• Approach:
  • Whisper Decoder Controls (implemented): Added configurable decoding flags for final segments and drafts, including temperature fallback and quality thresholds (logprob_threshold, compression_ratio_threshold), with compatibility fallback when unsupported.
  • Beam Compatibility Hardening: Defaulted beam sizes to greedy-safe values and auto-retry without beam_size when the runtime reports "Beam search decoder is not yet implemented."
  • Dedicated English Bridge Context: Split context handling into source-language context and a separate English context used specifically for task="translate" bridge generation.
  • Sentence-Aware MT Chunking: Replaced hard truncation with sentence-aware splitting for long English bridge text before translation.
  • Marian Generation Tuning: Added configurable generation controls (num_beams, no_repeat_ngram_size, length_penalty, repetition_penalty, early_stopping) to reduce repetitive or unstable outputs.
• Outcome: Better continuity for non-English to English bridging, fewer clipped translations on long segments, and cleaner target-language output with safer defaults for current mlx_whisper builds.

Phase 17: Intelligent English Caption Correction (V1)

• Goal: Improve English caption readability and resilience without introducing heavy latency.
• Approach:
  • Confidence-Triggered Re-Decode: Added optional retry logic that re-transcribes low-confidence segments when Whisper quality signals indicate likely errors (avg_logprob / compression_ratio thresholds).
  • Glossary Replacements: Added configurable SOURCE=TARGET term normalization (--glossary-pair) to consistently correct names/terms in English captions.
  • Rolling Caption Revision Buffer: Added a short correction window (--caption-correction-window, default 3s) with a 2-line rolling buffer that can merge incomplete English lines into a cleaner sentence as new context arrives.
• Outcome: Captions now support lightweight post-correction behavior that improves continuity and term consistency while remaining compatible with real-time streaming.