whisper-translation/README.md

Python Whisper Live Transcription & Translation

A real-time, low-latency audio transcription and translation tool utilizing OpenAI's Whisper (via mlx-whisper for Apple Silicon optimization), Silero VAD for speech detection, and Helsinki-NLP's Opus-MT models for translation.

Features

• Live Transcription: Real-time speech-to-text with automatic language detection.
• On-the-fly Translation: Bridge translations to English and then to target languages (Spanish, Arabic, French, etc.).
• Voice Activity Detection (VAD): Intelligent audio buffering using Silero VAD to process only actual speech.
• Apple Silicon Optimized: Uses MLX for high performance on Mac (MPS).
• Background Ingestion: Optional background thread to send JSON payloads to a remote server.
• Configurable: Command-line parameters to select languages, devices, and ingestion.

Installation

1. Clone the repository:

   git clone <repository-url>
   cd pythonwhisper
   

2. Install dependencies: Ensure you have Python 3.9+ and the required libraries:

   pip install mlx-whisper numpy sounddevice torch requests transformers silero-vad
   

   Note: On Apple Silicon, ensure mlx and torch with MPS support are correctly installed.

Usage

Run the script using python3 transcribe.py with optional flags.

Common Commands

• List available audio devices:

  python3 transcribe.py -l
  

• Caption system audio (speakers) using a loopback device:

  python3 transcribe.py --loopback -es
  

• Transcribe and translate to Spanish (screen only):

  python3 transcribe.py -es
  

• Enable Spanish, Arabic, and French with English bridging, and send to server:

  python3 transcribe.py -es -ar -fr -en -i
  

• Use a specific input device (e.g., index 3) and translate to Spanish:

  python3 transcribe.py -d 3 -es
  

Arguments

• -es: Enable Spanish translation.
• -en: Enable English (shows original if detected, or bridged if not).
• -ar: Enable Arabic translation.
• -fr: Enable French translation.
• -i, --ingest: Enable data transmission to the remote server.
• -l, --list-devices: Show available audio devices and exit.
• -d, --device [ID]: Input device index (bypasses selection prompt).
• --loopback: Automatically select a loopback device (e.g., BlackHole, Stereo Mix) to caption system audio.
• -q, --quantize: Use 4-bit quantized Whisper model for faster transcription (Strategy 3).
• -s, --stream: Enable real-time streaming transcription/draft mode (Strategy 1).
• -c, --context: Enable prompt caching/rolling context to help the model maintain sentence continuity across chunks.
• --lang [CODE]: Hardcode the source language (e.g., en, es) to bypass automatic language detection for faster processing.
• --silence [MS]: Set the minimum silence duration in milliseconds to end a chunk. Defaults to 1000ms. Increase to force longer sentences before translation.
• --max-buffer [SEC]: Maximum buffer duration in seconds before forcing a flush (default: 20).
• --channels [N]: Number of input channels (default: 1).
• --pick-channel [0|1]: If stereo, select channel 0 (Left) or 1 (Right) to focus transcription.
• --filter-lang: If used with --lang, discards segments that do not match the target language.

---

Technical Note: Universal Translation Models

While the current implementation uses specialized, per-language models from the Helsinki-NLP Opus-MT project (e.g., opus-mt-en-es, opus-mt-en-ar), there is an alternative approach: Universal Models.

Universal Model Alternative (e.g., Meta's NLLB-200)

The current per-language model approach is highly accurate and memory-efficient if you only need 1 or 2 target languages. However, if you require support for many languages simultaneously, loading multiple specialized models can consume significant RAM/VRAM.

We have the option to switch the translation engine to a single, universal model such as NLLB-200 (No Language Left Behind):

• Model ID: facebook/nllb-200-distilled-600M
• Benefits:
  • Supports over 200 languages in a single model.
  • Simplified code: no need to load/manage multiple model objects.
  • More efficient for complex multilingual environments.
• Trade-off: Slightly higher memory footprint for the single model compared to a single specialized model, but more efficient than 3+ specialized models.

If you wish to switch to a universal model, the transcribe.py logic can be updated to use a single M2M100 or NLLB pipeline instead of the current MarianMT loop.