whisper-translation/history.md

Project History: Python Whisper Live Transcription

This document tracks the evolution, technical decisions, and optimizations made for this live audio transcription and translation tool.

Phase 1: Basic Live Transcription

• Goal: Create a simple script to transcribe live audio.
• Initial Stack: openai-whisper, sounddevice, numpy.
• Approach:
  • Captured audio in 2-second chunks using sounddevice.
  • Used the tiny.en model for initial testing.
• Outcome: Successful basic transcription, but limited by continuous processing (even during silence).

Phase 2: Voice Activity Detection (VAD)

• Goal: Improve efficiency by only transcribing when someone is speaking.
• Stack Addition: silero-vad.
• Approach:
  • Integrated Silero VAD to monitor the audio stream.
  • Transcription is only triggered after a speech segment is followed by a period of silence (500ms).
• Outcome: Significantly reduced CPU usage and cleaner output.

Phase 3: Apple Silicon Optimization (M-Series/M2)

• Goal: Leverage the M2's Neural Engine and GPU for better performance.
• Stack Transition: mlx-whisper (via Apple's MLX framework).
• Decision: Switched from openai-whisper to mlx-whisper and upgraded the model to small.en for better accuracy without sacrificing speed.
• Outcome: Faster inference and better battery efficiency.

Phase 4: Local Translation

• Approach A (LLM): Tried using SmolLM2-135M via mlx-lm for translation.
  • Issue: The LLM was "too talkative," often adding conversational filler or explaining the translation instead of just providing it.
• Approach B (Dedicated MT): Switched to MarianMT (Helsinki-NLP/opus-mt-en-es).
  • Decision: Chose a dedicated Translation Model for cleaner, direct mapping from English to Spanish.
• Technical Hurdle (LZMA Error):
  • The local Python environment lacked _lzma support, causing transformers and huggingface_hub to crash.
  • Solution: Implemented a comprehensive lzma mock in the script to provide necessary constants (FORMAT_XZ, etc.) and bypass the system-level limitation.

Current Status

The project now features a high-performance, Apple Silicon-optimized pipeline that:

1. Detects speech using Silero VAD.
2. Transcribes using MLX-Whisper (small.en).
3. Translates using MarianMT (EN-ES).
4. Operates entirely locally with hardware acceleration.

Phase 5: Simultaneous Multi-Language Translation

• Goal: Provide translations in Spanish, French, and Arabic at the same time.
• Approach:
  • Refactored the script to support a dictionary of multiple MarianMT models.
  • Each transcribed English segment is passed through each loaded translation engine sequentially.
• Performance on M2: Loading 3-4 specialized models + Whisper is highly efficient, using ~1.5GB of RAM and providing near-instant results.

Phase 6: Memory & Generation Safety

• Issue: Occasionally, long inputs or model glitches caused "runaway" translation generation, which could consume excessive memory.
• Solution:
  • Artificially truncated input transcription to a maximum of 250 characters.
  • Added max_new_tokens=150 to the translation generation call to ensure the model terminates even if it gets stuck in a loop.

Phase 7: Multilingual Detection & Bridge Translation

• Goal: Support input in any language, detect it, and translate to English + others.
• Approach:
  • Switched to whisper-small-mlx (multilingual).
  • Hub-and-Spoke Model: If a non-English language is detected, Whisper's task="translate" is used to create an English "bridge" text, which is then fed into the specialized MarianMT models.
• Outcome: Full support for multilingual input with centralized translation.

Phase 8: Compilation to Binary

• Goal: Distribute the script as a single, standalone executable for macOS terminal.
• Tool: PyInstaller.
• Process:
  • Used --onefile to bundle the entire Python runtime and its heavy dependencies (Torch, MLX, Transformers).
  • Excluded build artifacts (build/, dist/, .spec) from the repository.
• Build Script:

  chmod +x build.sh
  ./build.sh
  

• Troubleshooting: Fixed a runtime ModuleNotFoundError: No module named 'mlx._reprlib_fix' by explicitly adding --collect-all mlx and --hidden-import=mlx._reprlib_fix to the PyInstaller configuration. Also added multiprocessing.freeze_support() to fix infinite loops in the compiled binary.

Phase 9: Real-Time Server Ingest

• Goal: Send live captions and translations to a central ingest server.
• Backend: https://emiapi.reynafamily.com/live-captions/ingest.
• Approach:
  • Implemented a background ingest_worker thread to handle HTTP POST requests without stalling the audio processing.
  • Flat JSON Schema: Used a key-value format as requested (e.g., original, es, en, fr).
  • Reliability: Integrated exponential backoff retries (1s to 15s) to handle network or server failures.

Phase 10: CLI Configuration & Documentation

• Goal: Make the tool fully configurable without editing the code.
• Approach:
  • Integrated argparse to allow dynamic selection of translation languages (-es, -en, -ar, -fr), data ingestion (-i), and audio device (-d).
  • Added device listing capability (-l).
  • Excluded the venv directory from git and generated requirements.txt.
  • Created a comprehensive README.md containing setup instructions, usage examples, and a technical note on universal models (like NLLB-200).
• Fix: Added missing sentencepiece dependency required by MarianMT models.

Phase 11: Speed Optimization (Quantization & Streaming)

• Goal: Reduce latency and improve real-time feedback.
• Approach:
  • Streaming Mode (-s): Implemented a 1-second rolling draft transcription that continuously updates the console while the user is still speaking.
  • Quantization (-q): Added support for dynamically swapping to the 4-bit quantized Whisper model (mlx-community/whisper-small-mlx-4bit) for faster inference on Apple Silicon with lower memory bandwidth.

Phase 12: Accuracy & Context Optimization

• Goal: Improve translation quality of short or broken audio chunks.
• Approach:
  • Prompt Caching (-c): Implemented a rolling context buffer that feeds the last 200 characters of previously transcribed text back into Whisper as an initial_prompt, maintaining sentence continuity.
  • Language Bypassing (--lang): Added the ability to hardcode the source language to skip the Whisper language identification phase on every chunk.
  • Heuristic Punctuation Buffering (Reverted): Briefly implemented a system to hold English translations until a definitive punctuation mark was reached to prevent grammatical errors. This was reverted because Whisper's punctuation generation is not 100% reliable, leading to translations getting "stuck" in the buffer indefinitely if no period was generated.

Phase 13: Stability & Stuck Detection (Watchdog)

• Goal: Prevent captions from getting "stuck" during long periods of music, singing, or model hallucinations.
• Approach:
  • Transcription Watchdog: Implemented a logic that monitors the audio buffer duration and the time since the last unique draft change. If the buffer exceeds 12s without a change for 7s, or if it hits an absolute limit of 25s, it forces a clean reset.
  • Hallucination Filtering: Added regex-based detection for common Whisper hallucinations (e.g., "Thanks for watching", "Please subscribe") and a repetition counter to identify and discard high-frequency loops (e.g., "Hallelujah" repeated 20 times).
  • Buffer Optimization: Reduced the default maximum buffer from 30s to 20s (now configurable via --max-buffer) to ensure more frequent flushes during continuous sound.
• Outcome: Significantly improved reliability during live church services and musical performances where Whisper previously tended to "hang" or hallucinate.

Phase 14: Multi-Channel & Language Filtering

• Goal: Isolate specific speakers or languages when the audio feed contains multiple mixed sources.
• Approach:
  • Stereo Channel Selection: Added --channels and --pick-channel [0|1] flags. This allows the tool to pull a specific audio channel (e.g., English on Left, Translation on Right) from a stereo feed, ignoring the other.
  • Language Filtering (--filter-lang): Integrated a mechanism to discard any segment where the detected language does not match the hardcoded --lang parameter.
• Outcome: Enabled the ability to "focus" the transcription engine on a single speaker even when the audio input is a complex mix.

Phase 15: Potential Future Optimizations (Backlog)

• Decoding Parameters:
  • Temperature Fallback: Use a tuple (0.0, 0.2, 0.4, 0.6, 0.8, 1.0) to allow Whisper to re-try failed transcriptions with higher randomness (crucial for music/singing).
  • Beam Size Tuning: Set beam_size=1 for Draft Mode (3x faster) and beam_size=5 for final segments (higher accuracy).
  • Token Suppression: Native suppression of music symbols or common hallucination tokens at the decoder level.
  • Heuristic Thresholds: Utilize logprob_threshold (confidence) and compression_ratio_threshold (repetition) to programmatically identify and discard bad transcriptions before they reach the user.
• Structural Features:
  • Word-Level Timestamps: Enable word_timestamps=True to provide granular timing data for front-end caption highlighting.

Phase 16: Decoder Controls + Bridge Context Refinements

• Goal: Improve English caption stability and translation quality without sacrificing runtime compatibility.
• Approach:
  • Whisper Decoder Controls (implemented): Added configurable decoding flags for final segments and drafts, including temperature fallback and quality thresholds (logprob_threshold, compression_ratio_threshold), with compatibility fallback when unsupported.
  • Beam Compatibility Hardening: Defaulted beam sizes to greedy-safe values and auto-retry without beam_size when the runtime reports "Beam search decoder is not yet implemented."
  • Dedicated English Bridge Context: Split context handling into source-language context and a separate English context used specifically for task="translate" bridge generation.
  • Sentence-Aware MT Chunking: Replaced hard truncation with sentence-aware splitting for long English bridge text before translation.
  • Marian Generation Tuning: Added configurable generation controls (num_beams, no_repeat_ngram_size, length_penalty, repetition_penalty, early_stopping) to reduce repetitive or unstable outputs.
• Outcome: Better continuity for non-English to English bridging, fewer clipped translations on long segments, and cleaner target-language output with safer defaults for current mlx_whisper builds.

Phase 17: Intelligent English Caption Correction (V1)

• Goal: Improve English caption readability and resilience without introducing heavy latency.
• Approach:
  • Confidence-Triggered Re-Decode: Added optional retry logic that re-transcribes low-confidence segments when Whisper quality signals indicate likely errors (avg_logprob / compression_ratio thresholds).
  • Glossary Replacements: Added configurable SOURCE=TARGET term normalization (--glossary-pair) to consistently correct names/terms in English captions.
  • Rolling Caption Revision Buffer: Added a short correction window (--caption-correction-window, default 3s) with a 2-line rolling buffer that can merge incomplete English lines into a cleaner sentence as new context arrives.
• Outcome: Captions now support lightweight post-correction behavior that improves continuity and term consistency while remaining compatible with real-time streaming.

Phase 18: File-Based Glossary Loading

• Goal: Make glossary corrections persistent and easier to maintain without long CLI commands.
• Approach:
  • Added --glossary-file (default glossary.txt) to load SOURCE=TARGET term mappings when the file exists.
  • Implemented comment/blank-line support and invalid-line skipping with line-level warnings.
  • Merged file-based glossary entries with repeatable --glossary-pair CLI entries for runtime overrides.
• Outcome: English caption correction can now use a maintained glossary file automatically, while preserving ad-hoc CLI term fixes.

Phase 19: Arabic Terminal Rendering Hardening

• Goal: Improve readability of Arabic captions in terminal environments with mixed LTR/RTL output.
• Approach:
  • Switched Arabic output to a strict two-line format ([AR]: label line + dedicated RTL content line).
  • Wrapped Arabic caption content in explicit RTL embedding marks for better bidirectional layout stability.
  • Added punctuation normalization for Arabic display (،, ؛, ؟) to reduce LTR punctuation artifacts.
• Outcome: Arabic captions render more consistently in terminal output, especially when adjacent to English/French/Spanish caption lines.

Phase 20: Post-Correct Mode (V2 Foundation)

• Goal: Add a practical second-stage correction pass for finalized English captions.
• Approach:
  • Added --post-correct mode to run deterministic rule-based cleanup after ASR (spacing, punctuation cleanup, disfluency reduction, immediate duplicate-word collapse).
  • Added optional local LLM correction via Ollama (--post-correct-llm, model/url/timeout flags) with strict fallback to rules-only when unavailable.
  • Integrated post-correction after bridge-to-English and before downstream translation so improved English text propagates to target-language MT.
• Outcome: The pipeline now supports a two-stage transcription flow (ASR -> correction) with low-latency defaults and optional local generative enhancement.

Phase 21: LLM Merge-Decider for Line Revisions

• Goal: Improve English line-merge quality when smart-correct revision is ambiguous.
• Approach:
  • Added optional --llm-merge-decider to validate/refine heuristic merge candidates using the local post-correct LLM.
  • Enforced strict low-latency behavior with --llm-merge-timeout (default 0.7s) and automatic fallback to heuristic merges on errors/timeouts.
  • Added safety guards to reject overly large LLM rewrites and preserve buffered caption state when merge is rejected.
• Outcome: Merge decisions remain fast and deterministic by default, with optional LLM arbitration for cleaner sentence continuity.

Phase 22: Optional Speaker-Change Line Cutting

• Goal: Split long live captions earlier when a different person starts speaking.
• Approach:
  • Added optional --speaker-change-detect mode with cosine-similarity comparison of lightweight voice signatures from buffered audio.
  • Introduced tuning flags for sensitivity and runtime cost (--speaker-sim-threshold, --speaker-min-buffer, --speaker-check-interval).
  • When a probable speaker switch is detected, the current buffered caption line is force-flushed early instead of waiting only for silence.
  • Added speaker-state resets on watchdog/hallucination/silence reset paths to avoid stale identity drift.
• Outcome: Users can opt into faster caption segmentation at speaker boundaries while keeping default behavior unchanged.

Phase 23: Prompt Hardening + Anti-Drift Guardrails

• Goal: Reduce semantic drift in LLM post-correction and merge decisions.
• Approach:
  • Rewrote post-correction prompt with strict "current line only" and "if uncertain, keep original" constraints.
  • Expanded post-correction context to up to two previous lines but marked as reference-only.
  • Added token-overlap acceptance guard (--post-correct-min-overlap) to reject LLM rewrites that diverge too far from source text.
  • Updated merge-decider prompt to stricter JSON behavior and added overlap-based fallback to heuristic merge.
• Outcome: Improved protection against context bleed (e.g., replacing current line with prior sentence) while keeping optional LLM improvements.

Phase 24: Ollama Warmup + Keep-Alive

• Goal: Reduce intermittent post-correction timeouts caused by cold model loads.
• Approach:
  • Added startup warmup request for the local post-correct model when LLM correction is enabled.
  • Added configurable Ollama keep_alive setting so the model stays resident between caption calls.
  • Routed post-correct and merge-decider requests through a shared Ollama caller with keep-alive support.
  • Added flags for warmup timeout and optional warmup skip for troubleshooting.
• Outcome: Lower first-request latency and fewer fallback-to-rules events due to local model cold starts.

Phase 25: Session Logging + Tuned Live Preset

• Goal: Preserve reliable debugging context while locking in the best-performing live settings discovered through manual testing.
• Approach:
  • Added session file logging hooks so finalized captions, revisions, startup state, and runtime errors can be written to a dedicated log via --session-log-file.
  • Improved interactive terminal behavior by clearing stale draft lines cleanly and right-aligning Arabic output more consistently against terminal width.
  • Recorded the current preferred live preset for English-first filtered transcription with multilingual output and local LLM post-correction:

  python3 transcribe.py --silence 100 -q -s -c -es -en -fr -ar --lang en --filter-lang --mt-max-chars 450 --mt-max-new-tokens 220 --smart-correct --post-correct --post-correct-llm --post-correct-model qwen2.5:3b-instruct --session-log-file debug_session_1.log
  

• Outcome: The project now has a repeatable "known good" runtime profile plus persistent session diagnostics for reviewing caption issues after a run.

Phase 26: Multi-Process Decoupled Architecture

• Goal: Improve system stability, reduce latency, and fully decouple audio capture from heavy LLM/Translation tasks.
• Approach:
  • 4-Process Pipeline: Refactored the monolithic script into four independent services coordinated via multiprocessing.Queue:
    1. engine_transcribe.py (Whisper): Dedicated to high-priority audio capture and ASR.
    2. engine_llm.py (Ollama): Handles asynchronous post-correction and paragraph structuring without blocking the transcription loop.
    3. engine_translate.py (MarianMT): Manages multi-language translation for both raw and refined text.
    4. engine_distribute.py (API/CLI): Handles data delivery and terminal display.
  • Centralized Configuration: Introduced config.json for managing all defaults and parameters in one place, with main_v2.py as the entry point.
  • Dual Payload Strategy: Implemented a robust data flow where the translation and distribution engines receive both raw and LLM-corrected versions of the text, allowing for fallback and comparison.
  • Hardware Isolation: Transcription and Translation processes independently leverage Apple Silicon (MPS), while the LLM process utilizes Ollama's external server, preventing resource contention.
• Outcome: Significantly increased resilience. If the LLM or Translation engine stalls, the Transcription engine continues to capture and buffer audio safely, preventing data loss.

Phase 27: Intelligent LLM Refinement & Pipeline Hardening

• Goal: Transform raw ASR segments into professional paragraphs and stabilize advanced features.
• Approach:
  • Accumulative LLM Engine: Developed a stateful refinement logic in engine_llm.py that maintains a "working paragraph," allowing the LLM to continuously integrate and polish new segments in real-time.
  • OpenAI Integration: Added direct support for OpenAI's GPT API (via requests to avoid subprocess dependency issues), enabling higher-quality refinement than lightweight local models.
  • Rolling Context Window: Implemented a smart context management system that "finalizes" paragraphs once they reach a natural break (detected via \n\n), clearing the prompt history to save tokens and maintain focus.
  • Speaker Diarization Hardening: Refactored engine_transcribe.py to support pyannote.audio 4.0.4, specifically handling the new DiarizeOutput object structure and ensuring speaker labels (SPEAKER_XX) propagate through the entire multi-process pipeline.
  • English Bridge Optimization: Forced a two-pass transcription strategy (ASR first, then optional translation) to ensure word-level timestamps are captured for speaker mapping even when translating to English.
  • Stability Fixes: Added hallucination filtering for Whisper's repetitive loops, enforced line-buffering for terminal logging, and implemented a global lzma mock to ensure compatibility across restricted Python environments.
• Outcome: A robust, production-ready pipeline that produces high-quality, speaker-attributed, and professionally formatted live transcripts.

Phase 28: main_v2 Bridge Contract Fixes + Live Reliability Review

• Goal: Repair stage-contract regressions in the new multi-process pipeline and preserve live throughput under failure.
• Review Findings:
  • The Whisper English bridge in engine_transcribe.py reused source-language prompt context for task="translate" instead of a dedicated English context, which could bias or degrade translated bridge text.
  • --filter-lang was exposed in main_v2.py but not enforced in the new transcription engine, so wrong-language speech could still be bridged and propagated downstream.
  • engine_llm.py told the paragraph refiner to output the source language even though downstream Marian models require English input, breaking llm_paragraph for non-English sources.
  • engine_translate.py could translate one English variant while publishing a different [EN] line, making the visible source text diverge from what target-language MT actually used.
  • engine_distribute.py retried ingest forever in the queue consumer, allowing a network outage to stall the whole live pipeline.
  • main_v2.py exposed post-correction settings that the new pipeline did not fully honor, making CLI behavior drift from the documented workflow.
• Approach:
  • Restored a dedicated rolling English prompt context for the Whisper bridge pass and rebuilt bridge kwargs explicitly instead of copying source-ASR kwargs.
  • Reinstated --filter-lang enforcement before bridge generation and reduced unnecessary ASR cost by only asking Whisper for word timestamps when diarization is enabled.
  • Updated the LLM stage so paragraph refinement always preserves English bridge text, added deterministic post-correction plus optional LLM line cleanup, and provided a safe fallback paragraph when the LLM backend is unavailable.
  • Aligned published English output with the exact text chosen for translation and added the missing Ollama-related config/plumbing to main_v2.
  • Replaced infinite ingest retry loops with bounded retry logic so failed delivery degrades gracefully instead of freezing caption flow.
• Outcome: The main_v2 pipeline now keeps an English-first contract across transcription, refinement, and Marian translation while behaving more predictably under API outages and mixed-language input.

Phase 29: Output Semantics + Model Defaults Cleanup

• Goal: Make the live console reflect the real pipeline stages and choose safer small-model defaults for local LLM use.
• Approach:
  • Split raw ASR, Whisper English bridge, line-level LLM corrections, and paragraph output into distinct console labels instead of grouping multiple stages under [LLM].
  • Updated the default local post-correction model configuration to prefer a smaller Qwen 3.5 variant for experimentation, then kept the code flexible so model choice can be overridden per run.
  • Added metadata flow through the translation/distribution stages so downstream output can distinguish true LLM paragraphs from deterministic paragraph fallback.
• Outcome: Terminal output is easier to interpret during multilingual runs, especially when comparing source ASR against the English bridge and later refinement stages.

Phase 30: Local LLM Warmup + Fallback Transparency

• Goal: Reduce cold-start timeout confusion for local Ollama-backed correction and paragraph modes.
• Approach:
  • Added Ollama startup health checks and model presence checks before live processing begins.
  • Added a warmup request for local models so working Ollama models can pay the cold-load cost before live captions arrive.
  • Split paragraph output labels into true LLM paragraphs versus deterministic fallback paragraphs to avoid overstating when a model actually answered.
• Outcome: Local-model failures are surfaced earlier, cold-start behavior is easier to understand, and fallback output is clearly labeled.

Phase 31: OpenAI/Ollama Prompt Parity + Better API Diagnostics

• Goal: Make remote and local LLM behavior easier to compare and make API failures actionable.
• Approach:
  • Improved OpenAI error reporting to print the full API error body instead of only a generic HTTP 400 message.
  • Removed the explicit Chat Completions temperature override after discovering current GPT-5-family models reject non-default temperature values on this endpoint.
  • Aligned the Ollama path with the OpenAI path by forwarding the same system instructions to local generation requests, including warmup calls.
• Outcome: GPT-backed runs now fail with useful diagnostics, OpenAI requests are compatible with current GPT-5-mini behavior, and local-vs-remote prompting is more consistent.