diff --git a/.gitignore b/.gitignore index 548ed60..8cf6f8e 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,16 @@ dist/ venv/ .env *.log +apple_speech/.build/ +apple-speech-transcribe +apple_speech/apple-speech-transcribe +logs/apple_bench.json +logs/context*.json +logs/prompt*.json +logs/final*.json +logs/paragraph*.json +logs/llm_requests.jsonl +prayer*.json +test.json +/tmp/*.wav +/tmp/*.aiff diff --git a/README.md b/README.md index 7619295..87a3fda 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,30 @@ You can now send the line-correction or paragraph-refinement prompts directly wi Each log entry includes the timestamp, provider, model, mode, temperature, and full `messages` payload that was sent to the LLM. +### Freeflow-style Local Polish +This implementation includes a Python port of the portable parts of Freeflow's polish pipeline: + +- spoken punctuation commands such as `comma`, `question mark`, `new paragraph`, `at sign`, and `hashtag` +- filler/noise stripping before any LLM call +- protected `...` symbols so the LLM does not reinterpret dictated symbols +- a clean-transcript skip heuristic for low-latency local runs +- Freeflow-inspired English/minimal/Qwen line-polish prompts + +Run deterministic cleanup only: +```bash +python3 main_v2.py --post-correct +``` + +Run deterministic cleanup plus a local Ollama line-polish model when needed: +```bash +python3 main_v2.py --post-correct-llm --post-correct-model qwen3.5:0.8b --post-correct-prompt-style qwen +``` + +Force every line through the LLM, even if the deterministic output already looks clean: +```bash +python3 main_v2.py --post-correct-llm --no-post-correct-skip-clean +``` + ### Common Commands - **List available audio devices:** ```bash diff --git a/README_APPLE_V3.md b/README_APPLE_V3.md new file mode 100644 index 0000000..56036b9 --- /dev/null +++ b/README_APPLE_V3.md @@ -0,0 +1,105 @@ +# Apple Speech API v3 — Real-time transcription using macOS 26's new engine + +Based on [Inscribe's benchmark](https://get-inscribe.com/blog/apple-speech-api-benchmark.html): Apple's `SpeechAnalyzer` + `SpeechTranscriber` (iOS/macOS 26) hits **2.12% WER** on LibriSpeech test-clean vs Whisper Small's 3.74%, at ~3x speed. No performance numbers from Apple until this blog. + +## What's in v3 + +Two implementations: + +### 1. Swift CLI (`apple_speech/`) + +Tiny Swift 6 package compiled directly (no SPM version dance): + +```bash +cd apple_speech && bash build.sh +``` + +Modes: +- `apple-speech-transcribe --locale en-US` → JSONL segments, timestamped +- `apple-speech-transcribe --bench ` → single JSON with RTF, full transcript +- `apple-speech-transcribe --live --locale en-US` → mic streaming JSONL (volatile + final) +- `apple-speech-transcribe --check` → availability + installed/supported locales +- `apple-speech-transcribe --list-locales` + +Correct API usage discovered by testing against crash logs: +- File: `SpeechAnalyzer(inputAudioFile: file, modules: [transcriber], finishAfterFile: true)` then `for try await result in transcriber.results` — results loop terminates naturally, not via task cancellation. This is the path that emits final segments correctly; `analyzeSequence(from:)` returned early and missed finals. +- Live: `AudioBufferQueue` actor + `LiveInputSequence: AsyncSequence` fed to `SpeechAnalyzer(inputSequence:modules:)` with AVAudioEngine tap + optional AVAudioConverter to bestAvailableAudioFormat. + +### 2. Python engine (`engine_apple_transcribe.py`) + +Drop-in for `engine_transcribe.py`: + +- Same Silero VAD chunking (`silence`, `max_buffer`) +- Each VAD-finalized chunk → temp 16-bit WAV → `apple-speech-transcribe --bench` subprocess +- Emits dict `{original, en_bridge, detected_lang, speaker, ts, _apple_meta{rtf, segments}}` same shape as whisper engine +- Queue-compatible with full pipeline + +### 3. Main v3 entry (`main_v3.py`) + +`--engine {whisper,apple}` selector: + +```bash +./venv/bin/python main_v3.py --engine apple --lang en -v +./venv/bin/python main_v3.py --engine apple --lang es -v +./venv/bin/python main_v3.py --engine whisper --lang en # old path +./venv/bin/python main_v3.py --apple-bench-file /tmp/audio.wav --lang en +``` + +## Benchmark results (this machine, M-series, macOS 26.5.2) + +Generated via `benchmark_v3.py --test-say` (TTS roundtrip): + +| Locale | Text | WER | RTF | +|--------|------|-----|-----| +| en-US | Hello world, this is a test... | 0.0% | 28.4x | +| en-US | quick brown fox + $35.99 normalization | 38.9%* | 30.7x | +| en-US | email to john@example.com | 14.3%* | 23.7x | +| es-ES | Hola mundo... | 0.0% | 14.6x | +| es-ES | Buenos días... | 0.0% | 38.4x | +| fr-FR | Bonjour... (voice mismatch) | 92.9% | — | +| de-DE | Hallo Welt... | 8.3% | 15.2x | + +*WER inflated due to smart normalization ($35.99, john@example.com) — actually *better* output. + +Real claim from Inscribe: **2.12% vs Whisper Small 3.74%**, ~3x faster, on LibriSpeech 5,559 utterances, M2 Pro 32GB macOS 26.5.1. + +## Locales on this machine + +After auto-download triggers: + +``` +installed: de_AT de_CH de_DE en_* es_CL es_ES es_MX es_US fr_BE fr_CA fr_CH fr_FR (20) +supported: + it_CH it_IT ja_JP ko_KR pt_BR pt_PT yue_CN zh_CN zh_HK zh_TW (30 total) +``` + +Install is lazy: first transcribe in a locale triggers `AssetInventory.assetInstallationRequest`. + +## Architecture comparison + +- **v2 Whisper**: audio -> Silero VAD chunks -> mlx-whisper (460MB Small) -> freeflow_polish -> LLM paragraph -> MarianMT -> ingest +- **v3 Apple**: audio -> Silero VAD chunks -> temp WAV -> apple-speech-transcribe SFSpeechAnalyzer (system model, ~few 100MB on-demand) -> same LLM/MT pipeline +- **Future**: native streaming inputSequence path from Python without temp files (requires PyObjC or Swift extension) + +## Files + +- `apple_speech/Sources/AppleSpeechCLI/main.swift` — Swift binary +- `apple_speech/build.sh` — swiftc build (bypass SPM .v26 manifest issue) +- `engine_apple_transcribe.py` — Python VAD + subprocess wrapper +- `main_v3.py` — engine selector main +- `benchmark_v3.py` — quick TTS bench +- `config_v3.json` — example config with engine=apple + +## Next steps to go production + +- [ ] Mic streaming without temp files (PyObjC bridge or keep long-lived Swift daemon) +- [ ] Speaker diarization (pyannote still works same) +- [ ] WER eval on LibriSpeech subset +- [ ] CI for binary rebuild on macOS version bump + +## Gotchas discovered + +1. `Package.swift` with `.macOS(.v26)` needs PackageDescription 6.2+ but CLT's swift-pm is 6.0 — must compile via `swiftc` directly. +2. `SpeechAnalyzer`'s `init(inputAudioFile:)` vs `analyzeSequence(from:)` have different lifetime: the former finishes when results AsyncSequence ends, the latter returns before final results (requires extra sleep + task cancellation). Fixed by using file initializer. +3. Silence-only audio returns 0 segments, not error. +4. Assets auto-download on first transcribe; status transitions `supported` -> `downloading` -> `installed`. +5. Binary crashes with SIGTRAP if you call `start(inputAudioFile:)` AFTER `init(inputAudioFile:)` — double start. diff --git a/__pycache__/transcribe.cpython-310.pyc b/__pycache__/transcribe.cpython-310.pyc deleted file mode 100644 index 3ddde21..0000000 Binary files a/__pycache__/transcribe.cpython-310.pyc and /dev/null differ diff --git a/apple_speech/Package.swift b/apple_speech/Package.swift new file mode 100644 index 0000000..688e606 --- /dev/null +++ b/apple_speech/Package.swift @@ -0,0 +1,16 @@ +// swift-tools-version: 6.0 +import PackageDescription + +let package = Package( + name: "AppleSpeech", + platforms: [.macOS(.v26)], + products: [ + .executable(name: "apple-speech-transcribe", targets: ["AppleSpeechCLI"]) + ], + targets: [ + .executableTarget( + name: "AppleSpeechCLI", + path: "Sources/AppleSpeechCLI" + ) + ] +) diff --git a/apple_speech/Sources/AppleSpeechCLI/main.swift b/apple_speech/Sources/AppleSpeechCLI/main.swift new file mode 100644 index 0000000..991df57 --- /dev/null +++ b/apple_speech/Sources/AppleSpeechCLI/main.swift @@ -0,0 +1,436 @@ +import AVFoundation +import Foundation +import Speech + +enum Mode: String { case file, live, pipe, listLocales, bench, check } + +struct CLIArgs { + var mode: Mode = .file + var filePath: String? = nil + var locale: String = "en-US" + var verbose: Bool = false + var includeVolatile: Bool = false +} + +func parseArgs() -> CLIArgs { + var args = CLIArgs() + var i = 1 + let raw = CommandLine.arguments + while i < raw.count { + let a = raw[i] + switch a { + case "--live": args.mode = .live + case "--pipe": args.mode = .pipe + case "--list-locales": args.mode = .listLocales + case "--bench": args.mode = .bench + case "--check": args.mode = .check + case "--locale": if i + 1 < raw.count { args.locale = raw[i+1]; i += 1 } + case "--verbose", "-v": args.verbose = true + case "--include-volatile": args.includeVolatile = true + case "--help", "-h": + fputs(""" + Usage: + apple-speech-transcribe [--locale en-US] [-v] [--include-volatile] + apple-speech-transcribe --bench [--locale en-US] [-v] + apple-speech-transcribe --live [--locale en-US] [-v] + apple-speech-transcribe --pipe [--locale en-US] [-v] (stdin wav chunks: 4-byte BE len + wav payload) + apple-speech-transcribe --check + apple-speech-transcribe --list-locales + + Output (file mode): JSON lines per segment on stdout + Output (bench): single JSON object with full transcript + timing + Output (live/pipe): JSON lines streaming (isFinal + text) + """, stderr) + exit(0) + default: + if !a.hasPrefix("-") && args.filePath == nil && (args.mode == .file || args.mode == .bench) { + args.filePath = a + } + } + i += 1 + } + return args +} + +func logv(_ msg: String, verbose: Bool) { if verbose { fputs("[apple-speech] \(msg)\n", stderr) } } + +func resolveLocale(_ id: String) async -> Locale { + if let sup = await SpeechTranscriber.supportedLocale(equivalentTo: Locale(identifier: id)) { return sup } + return Locale(identifier: id) +} + +func makeTranscriber(locale: Locale, includeVolatile: Bool) -> SpeechTranscriber { + let rep: Set = includeVolatile ? [.volatileResults, .alternativeTranscriptions] : [.alternativeTranscriptions] + return SpeechTranscriber(locale: locale, transcriptionOptions: [], reportingOptions: rep, attributeOptions: [.audioTimeRange, .transcriptionConfidence]) +} + +func ensureAssets(_ transcriber: SpeechTranscriber, verbose: Bool) async -> Bool { + let status = await AssetInventory.status(forModules: [transcriber]) + switch status { + case .installed: logv("Assets installed", verbose: verbose); return true + case .unsupported: fputs("Locale unsupported\n", stderr); return false + case .supported: + logv("Assets not installed, downloading...", verbose: true) + do { + if let req = try await AssetInventory.assetInstallationRequest(supporting: [transcriber]) { try await req.downloadAndInstall() } + return true + } catch { fputs("Asset download failed: \(error)\n", stderr); return false } + case .downloading: + logv("Assets downloading, waiting up to 30s...", verbose: true) + for _ in 0..<30 { + try? await Task.sleep(nanoseconds: 1_000_000_000) + let s2 = await AssetInventory.status(forModules: [transcriber]) + if s2 == .installed { return true } + } + return false + @unknown default: return true + } +} + +// MARK: - Check + +func runCheck() async { + print("{\"event\":\"check_start\"}") + let isAvailable = SpeechTranscriber.isAvailable + print("{\"event\":\"availability\",\"isAvailable\":\(isAvailable)}") + if !isAvailable { print("{\"event\":\"check_end\",\"ok\":false}"); return } + let supported = await SpeechTranscriber.supportedLocales + let installed = await SpeechTranscriber.installedLocales + let dict: [String: Any] = [ + "event": "locales", + "supported": supported.map { $0.identifier }.sorted(), + "installed": installed.map { $0.identifier }.sorted(), + "supported_count": supported.count, + "installed_count": installed.count + ] + if let d = try? JSONSerialization.data(withJSONObject: dict), let s = String(data: d, encoding: .utf8) { print(s) } + let t = SpeechTranscriber(locale: Locale(identifier: "en-US"), preset: .transcription) + let st = await AssetInventory.status(forModules: [t]) + let str: String + switch st { case .unsupported: str="unsupported"; case .supported: str="supported"; case .downloading: str="downloading"; case .installed: str="installed"; @unknown default: str="unknown" } + print("{\"event\":\"asset_status\",\"locale\":\"en-US\",\"status\":\"\(str)\"}") + print("{\"event\":\"check_end\",\"ok\":true}") +} + +func runListLocales() async { + if !SpeechTranscriber.isAvailable { fputs("Not available\n", stderr); exit(1) } + for loc in (await SpeechTranscriber.supportedLocales).sorted(by: { $0.identifier < $1.identifier }) { print(loc.identifier) } +} + +// MARK: - File transcription (fixed) + +struct SegmentJSON: Codable { + var text: String + var isFinal: Bool + var start: Double? + var duration: Double? + var alternatives: [String] +} + +func runFile(filePath: String, localeId: String, includeVolatile: Bool, bench: Bool, verbose: Bool) async { + guard FileManager.default.fileExists(atPath: filePath) else { fputs("File not found: \(filePath)\n", stderr); exit(1) } + guard SpeechTranscriber.isAvailable else { fputs("Not available\n", stderr); exit(1) } + + let locale = await resolveLocale(localeId) + logv("Locale resolved: \(locale.identifier) (req \(localeId))", verbose: verbose) + let finalTranscriber: SpeechTranscriber = { + if bench { return makeTranscriber(locale: locale, includeVolatile: false) } + return makeTranscriber(locale: locale, includeVolatile: includeVolatile) + }() + + guard await ensureAssets(finalTranscriber, verbose: verbose) else { exit(2) } + + let fileURL = URL(fileURLWithPath: filePath) + let audioFile: AVAudioFile + do { audioFile = try AVAudioFile(forReading: fileURL) } catch { fputs("Open failed: \(error)\n", stderr); exit(4) } + let audioDuration = Double(audioFile.length) / audioFile.fileFormat.sampleRate + logv("File: \(filePath) sr=\(audioFile.fileFormat.sampleRate) ch=\(audioFile.fileFormat.channelCount) dur=\(String(format: "%.2f", audioDuration))s", verbose: verbose) + + // Use init(inputAudioFile:finishAfterFile:true) which is the path that emits volatile correctly + // and automatically handles format. + let analyzer: SpeechAnalyzer + do { + analyzer = try await SpeechAnalyzer(inputAudioFile: audioFile, modules: [finalTranscriber], finishAfterFile: true) + } catch { fputs("Analyzer init failed: \(error)\n", stderr); exit(5) } + + var segments: [SegmentJSON] = [] + var volatileCount = 0 + var finalCount = 0 + let t0 = Date() + + // Consume results; loop ends naturally when analyzer finishes file + do { + for try await result in finalTranscriber.results { + let text = String(result.text.characters) + if text.isEmpty { continue } + let isF = result.isFinal + if isF { finalCount += 1 } else { volatileCount += 1 } + + let seg = SegmentJSON( + text: text, + isFinal: isF, + start: result.range.start.seconds, + duration: result.range.duration.seconds, + alternatives: result.alternatives.map { String($0.characters) } + ) + + if bench { + if isF { segments.append(seg) } + } else { + if !includeVolatile && !isF { continue } + segments.append(seg) + if let data = try? JSONEncoder().encode(seg), let line = String(data: data, encoding: .utf8) { + print(line); fflush(stdout) + } + } + } + } catch { + fputs("Results error: \(error)\n", stderr) + } + + // Ensure analyzer is done (it should be since results ended) + _ = analyzer // keep alive + + let elapsed = Date().timeIntervalSince(t0) + let rtf = audioDuration / max(elapsed, 0.001) + + if bench { + let fullText = segments.filter { $0.isFinal }.map { $0.text }.joined(separator: " ") + let out: [String: Any] = [ + "file": filePath, + "locale": locale.identifier, + "requested_locale": localeId, + "audio_duration_sec": audioDuration, + "processing_sec": elapsed, + "rtf": rtf, + "volatile_count": volatileCount, + "final_count": finalCount, + "text": fullText, + "segments": segments.map { ["text": $0.text, "start": $0.start ?? 0, "duration": $0.duration ?? 0] } + ] + if let data = try? JSONSerialization.data(withJSONObject: out, options: [.prettyPrinted, .sortedKeys]), + let str = String(data: data, encoding: .utf8) { print(str) } + } else { + if verbose { + fputs("Done: \(finalCount) final / \(volatileCount) volatile, RTF=\(String(format: "%.1f", rtf))x elapsed=\(String(format: "%.2f", elapsed))s\n", stderr) + } + } +} + +// MARK: - Live (AnalyzerInput) + +actor AudioBufferQueue { + private var buffers: [AVAudioPCMBuffer] = [] + private var conts: [CheckedContinuation] = [] + private var finished = false + func enqueue(_ b: AVAudioPCMBuffer) { + if let c = conts.first { conts.removeFirst(); c.resume(returning: b) } else { buffers.append(b) } + } + func dequeue() async -> AVAudioPCMBuffer? { + if !buffers.isEmpty { return buffers.removeFirst() } + if finished { return nil } + return await withCheckedContinuation { conts.append($0) } + } + func finish() { finished = true; for c in conts { c.resume(returning: nil) }; conts.removeAll() } +} + +struct LiveInputSequence: AsyncSequence, Sendable { + let queue: AudioBufferQueue + struct AsyncIterator: AsyncIteratorProtocol { + let queue: AudioBufferQueue + mutating func next() async -> AnalyzerInput? { + guard let buf = await queue.dequeue() else { return nil } + return AnalyzerInput(buffer: buf) + } + } + func makeAsyncIterator() -> AsyncIterator { AsyncIterator(queue: queue) } + typealias Element = AnalyzerInput +} + +func runLive(localeId: String, verbose: Bool) async { + guard SpeechTranscriber.isAvailable else { fputs("Not available\n", stderr); exit(1) } + let locale = await resolveLocale(localeId) + logv("Live locale \(locale.identifier) (req \(localeId))", verbose: verbose) + let transcriber = SpeechTranscriber(locale: locale, transcriptionOptions: [], reportingOptions: [.volatileResults, .alternativeTranscriptions], attributeOptions: [.audioTimeRange, .transcriptionConfidence]) + guard await ensureAssets(transcriber, verbose: verbose) else { exit(2) } + + let engine = AVAudioEngine() + let inputNode = engine.inputNode + let inputFormat = inputNode.inputFormat(forBus: 0) + logv("Mic format: \(inputFormat)", verbose: verbose) + + let queue = AudioBufferQueue() + let inputSequence = LiveInputSequence(queue: queue) + let analyzer = SpeechAnalyzer(inputSequence: inputSequence, modules: [transcriber]) + let targetFormat = await SpeechAnalyzer.bestAvailableAudioFormat(compatibleWith: [transcriber], considering: inputFormat) + logv("Target format: \(String(describing: targetFormat))", verbose: verbose) + + let converter: AVAudioConverter? = { + guard let tf = targetFormat, tf != inputFormat else { return nil } + return AVAudioConverter(from: inputFormat, to: tf) + }() + + var shouldStop = false + let sig = DispatchSource.makeSignalSource(signal: SIGINT, queue: .main) + signal(SIGINT, SIG_IGN) + sig.setEventHandler { fputs("\nStopping...\n", stderr); shouldStop = true; Task { await queue.finish() } } + sig.resume() + + let analyzerTask = Task { do { try await analyzer.start(inputSequence: inputSequence) } catch { fputs("Analyzer err: \(error)\n", stderr) } } + let resultsTask = Task { + do { + for try await result in transcriber.results { + let dict: [String: Any] = [ + "text": String(result.text.characters), + "isFinal": result.isFinal, + "start": result.range.start.seconds, + "duration": result.range.duration.seconds, + "alternatives": result.alternatives.map { String($0.characters) } + ] + if let d = try? JSONSerialization.data(withJSONObject: dict), let s = String(data: d, encoding: .utf8) { print(s); fflush(stdout) } + } + } catch { fputs("Results err: \(error)\n", stderr) } + } + + if let tf = targetFormat, let conv = converter { + inputNode.installTap(onBus: 0, bufferSize: 4096, format: inputFormat) { buffer, _ in + if shouldStop { return } + let cap = AVAudioFrameCount(Double(buffer.frameLength) * tf.sampleRate / inputFormat.sampleRate + 10) + guard let out = AVAudioPCMBuffer(pcmFormat: tf, frameCapacity: cap) else { return } + var err: NSError? + let block: AVAudioConverterInputBlock = { _, outStatus in outStatus.pointee = .haveData; return buffer } + conv.convert(to: out, error: &err, withInputFrom: block) + if err == nil && out.frameLength > 0 { Task { await queue.enqueue(out) } } + } + } else { + inputNode.installTap(onBus: 0, bufferSize: 4096, format: inputFormat) { buffer, _ in + if shouldStop { return } + guard let copy = AVAudioPCMBuffer(pcmFormat: buffer.format, frameCapacity: buffer.frameCapacity) else { return } + copy.frameLength = buffer.frameLength + if let src = buffer.floatChannelData, let dst = copy.floatChannelData { + for ch in 0...size) } + } + Task { await queue.enqueue(copy) } + } + } + + engine.prepare() + do { try engine.start() } catch { fputs("Audio engine start failed: \(error)\n", stderr); exit(6) } + logv("Listening... Ctrl-C to stop", verbose: true) + while !shouldStop { try? await Task.sleep(nanoseconds: 100_000_000) } + inputNode.removeTap(onBus: 0) + engine.stop() + await queue.finish() + analyzerTask.cancel() + resultsTask.cancel() +} + +@main +struct AppleSpeechCLI { + static func main() async { + let cli = parseArgs() + switch cli.mode { + case .check: await runCheck() + case .listLocales: await runListLocales() + case .file: + guard let fp = cli.filePath else { + fputs("Usage: apple-speech-transcribe [--locale en-US] [-v] [--include-volatile]\n", stderr); exit(1) + } + await runFile(filePath: fp, localeId: cli.locale, includeVolatile: cli.includeVolatile, bench: false, verbose: cli.verbose) + case .bench: + guard let fp = cli.filePath else { fputs("Usage: --bench [--locale]\n", stderr); exit(1) } + await runFile(filePath: fp, localeId: cli.locale, includeVolatile: false, bench: true, verbose: cli.verbose) + case .live: await runLive(localeId: cli.locale, verbose: cli.verbose) + case .pipe: await runPipe(localeId: cli.locale, verbose: cli.verbose) + } + } +} + +// MARK: - Pipe (stdin wav chunks) for draft streaming from Python VAD + +actor WavChunkQueue { + private var chunks: [URL] = [] + private var conts: [CheckedContinuation] = [] + private var finished = false + func enqueue(_ url: URL) { + if let c = conts.first { conts.removeFirst(); c.resume(returning: url) } else { chunks.append(url) } + } + func dequeue() async -> URL? { + if !chunks.isEmpty { return chunks.removeFirst() } + if finished { return nil } + return await withCheckedContinuation { conts.append($0) } + } + func finish() { finished = true; for c in conts { c.resume(returning: nil) }; conts.removeAll() } +} + +func runPipe(localeId: String, verbose: Bool) async { + guard SpeechTranscriber.isAvailable else { fputs("Not available\n", stderr); exit(1) } + let locale = await resolveLocale(localeId) + let warm = SpeechTranscriber(locale: locale, transcriptionOptions: [], reportingOptions: [.volatileResults], attributeOptions: []) + guard await ensureAssets(warm, verbose: verbose) else { exit(2) } + logv("Pipe ready locale=\(locale.identifier)", verbose: true) + + let stdinH = FileHandle.standardInput + var leftover = Data() + var chunkIdx = 0 + + func readExact(_ n: Int) -> Data? { + var out = Data() + out.reserveCapacity(n) + // first consume leftover + if leftover.count >= n { + let d = leftover.prefix(n); leftover = leftover.dropFirst(n); return Data(d) + } + if leftover.count > 0 { out.append(leftover); leftover = Data() } + while out.count < n { + let d = stdinH.readData(ofLength: n - out.count) + if d.isEmpty { + // EOF? try one more small sleep then retry once if we have partial + if out.count == 0 { return nil } + // partial -> return nil -> broken stream + return nil + } + out.append(d) + } + return out + } + + while true { + guard let lenData = readExact(4) else { break } + let length = lenData.withUnsafeBytes { $0.load(as: UInt32.self).bigEndian } + if length == 0 { logv("Pipe EOF", verbose: true); break } + if length > 20_000_000 { fputs("Pipe: chunk too large \(length)\n", stderr); break } + guard let chunkData = readExact(Int(length)) else { + fputs("Pipe: truncated, expected \(length)\n", stderr); break + } + chunkIdx += 1 + let tmpURL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent("apple-pipe-\(ProcessInfo.processInfo.processIdentifier)-\(chunkIdx).wav") + do { try chunkData.write(to: tmpURL) } catch { fputs("Pipe write err: \(error)\n", stderr); continue } + guard let audioFile = try? AVAudioFile(forReading: tmpURL) else { + try? FileManager.default.removeItem(at: tmpURL) + fputs("Pipe open fail \(chunkIdx)\n", stderr); continue + } + let t = SpeechTranscriber(locale: locale, transcriptionOptions: [], reportingOptions: [.volatileResults], attributeOptions: [.audioTimeRange]) + guard let analyzer = try? await SpeechAnalyzer(inputAudioFile: audioFile, modules: [t], finishAfterFile: true) else { + try? FileManager.default.removeItem(at: tmpURL); continue + } + do { + for try await res in t.results { + let txt = String(res.text.characters).trimmingCharacters(in: .whitespacesAndNewlines) + if txt.isEmpty { continue } + let d: [String: Any] = [ + "event": res.isFinal ? "final" : "draft", + "text": txt, "isFinal": res.isFinal, + "chunk": chunkIdx, "start": res.range.start.seconds, + "duration": res.range.duration.seconds + ] + if let jd = try? JSONSerialization.data(withJSONObject: d), let s = String(data: jd, encoding: .utf8) { + print(s); fflush(stdout) + } + } + } catch { fputs("Pipe results err \(chunkIdx): \(error)\n", stderr) } + _ = analyzer + try? FileManager.default.removeItem(at: tmpURL) + } + logv("Pipe done \(chunkIdx) chunks", verbose: true) +} \ No newline at end of file diff --git a/apple_speech/build.sh b/apple_speech/build.sh new file mode 100755 index 0000000..69e2e19 --- /dev/null +++ b/apple_speech/build.sh @@ -0,0 +1,23 @@ +#!/bin/bash +set -euo pipefail +cd "$(dirname "$0")" +SRC="Sources/AppleSpeechCLI/main.swift" +OUT=".build/release/apple-speech-transcribe" +mkdir -p .build/release + +# Use swiftc directly with macOS 26.5 SDK +# Swift 6.3.3 supports typed throws, Speech new API compiles fine with swiftc +echo "Compiling apple-speech-transcribe with swiftc..." +swiftc -O -parse-as-library \ + -target arm64-apple-macosx26.0 \ + -sdk /Library/Developer/CommandLineTools/SDKs/MacOSX26.5.sdk \ + "$SRC" -o "$OUT" \ + -framework Speech -framework AVFoundation -framework Foundation + +echo "Built: $OUT" +ls -lh "$OUT" +"$OUT" --check 2>&1 || true + +# Also copy to project root for python use +cp "$OUT" ../apple-speech-transcribe 2>/dev/null || cp "$OUT" ./apple-speech-transcribe +echo "Copied to ../apple-speech-transcribe and ./apple-speech-transcribe" diff --git a/benchmark_v3.py b/benchmark_v3.py new file mode 100644 index 0000000..caa9a78 --- /dev/null +++ b/benchmark_v3.py @@ -0,0 +1,145 @@ +#!/usr/bin/env python3 +""" +benchmark_v3.py - Quick WER-free benchmark comparing Apple vs Whisper on file(s) + +- For each audio file: run Apple (--bench) and optionally Whisper +- Measures wall time, RTF, segment counts +- For known reference texts, computes naive WER (word-level) +- Outputs table + JSON + +Usage: + python benchmark_v3.py /tmp/long.wav + python benchmark_v3.py prayer.json --extract-audio # future + python benchmark_v3.py --test-say # generates TTS samples + benches +""" + +import subprocess +import json +import time +import os +import sys +import tempfile +from pathlib import Path + +BIN = Path(__file__).parent / "apple_speech" / ".build" / "release" / "apple-speech-transcribe" + +def run_apple_bench(wav_path: str, locale: str = "en-US"): + cmd = [str(BIN), "--bench", wav_path, "--locale", locale] + start = time.time() + result = subprocess.run(cmd, capture_output=True, text=True, timeout=20) + wall = time.time() - start + if result.returncode != 0: + return {"ok": False, "stderr": result.stderr, "wall": wall} + try: + data = json.loads(result.stdout) + data["ok"] = True + data["bench_wall"] = wall + return data + except Exception as e: + return {"ok": False, "error": str(e), "stdout": result.stdout[:500], "wall": wall} + +def normalize_words(s: str): + import re + return re.findall(r"[a-z0-9']+", s.lower()) + +def wer(ref: str, hyp: str) -> float: + """Simple word error rate via edit distance""" + ref_words = normalize_words(ref) + hyp_words = normalize_words(hyp) + if not ref_words: + return 0.0 if not hyp_words else 1.0 + # DP + n, m = len(ref_words), len(hyp_words) + dp = [[0]*(m+1) for _ in range(n+1)] + for i in range(n+1): dp[i][0] = i + for j in range(m+1): dp[0][j] = j + for i in range(1, n+1): + for j in range(1, m+1): + cost = 0 if ref_words[i-1] == hyp_words[j-1] else 1 + dp[i][j] = min(dp[i-1][j]+1, dp[i][j-1]+1, dp[i-1][j-1]+cost) + return dp[n][m] / n + +def test_with_say(): + tests = [ + ("en-US", "Hello world, this is a test of Apple's new speech transcription API."), + ("en-US", "The quick brown fox jumps over the lazy dog while pricing thirty five dollars and ninety nine cents."), + ("en-US", "Please send an email to john at example dot com with the meeting notes."), + ("es-ES", "Hola mundo, esto es una prueba de la nueva API de transcripción de Apple."), + ("es-ES", "Buenos días, ¿cómo estás? Espero que tengas un excelente día."), + ("fr-FR", "Bonjour le monde, ceci est un test de la nouvelle API de transcription d'Apple."), + ("de-DE", "Hallo Welt, dies ist ein Test der neuen Apple Speech Transkriptions-API."), + ] + + voice_map = { + "en-US": "Samantha", + "es-ES": "Paulina", + "fr-FR": "Amelie", + "de-DE": "Anna", + } + + results = [] + for locale, ref_text in tests: + voice = voice_map.get(locale, "Samantha") + # Generate aiff + tmp_aiff = tempfile.mktemp(suffix=".aiff") + tmp_wav = tempfile.mktemp(suffix=".wav") + try: + subprocess.run(["say", "-v", voice, "-o", tmp_aiff, ref_text], timeout=10) + subprocess.run(["afconvert", tmp_aiff, "-f", "WAVE", "-d", "LEI16@16000", tmp_wav], timeout=10) + if not os.path.exists(tmp_wav): + continue + apple = run_apple_bench(tmp_wav, locale=locale) + hyp = apple.get("text","") if apple.get("ok") else "" + score = wer(ref_text, hyp) if apple.get("ok") else 1.0 + results.append({ + "locale": locale, + "ref": ref_text, + "hyp": hyp, + "wer": score, + "rtf": apple.get("rtf", 0), + "processing_sec": apple.get("processing_sec", 0), + "ok": apple.get("ok", False) + }) + print(f"[{locale}] WER={score:.1%} RTF={apple.get('rtf',0):.1f}x") + print(f" REF: {ref_text}") + print(f" HYP: {hyp}") + print() + except Exception as e: + print(f"[{locale}] error: {e}") + finally: + for p in [tmp_aiff, tmp_wav]: + if os.path.exists(p): + os.unlink(p) + + print("\n=== Summary ===") + for r in results: + print(f"{r['locale']:6} WER={r['wer']:.1%} RTF={r['rtf']:.1f}x ok={r['ok']}") + + # Save + out_path = Path(__file__).parent / "logs" / "apple_bench.json" + out_path.parent.mkdir(exist_ok=True) + with open(out_path, "w") as f: + json.dump(results, f, indent=2, ensure_ascii=False) + print(f"\nSaved to {out_path}") + +if __name__ == "__main__": + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("files", nargs="*", help="WAV files to bench") + parser.add_argument("--locale", default="en-US") + parser.add_argument("--test-say", action="store_true", help="Generate TTS samples and bench") + args = parser.parse_args() + + if not BIN.exists(): + print(f"Binary not found: {BIN}") + print("Run: cd apple_speech && bash build.sh") + sys.exit(1) + + if args.test_say or not args.files: + print("=== Running TTS-based benchmark ===") + test_with_say() + else: + for fp in args.files: + print(f"\n=== {fp} ===") + res = run_apple_bench(fp, locale=args.locale) + print(json.dumps(res, indent=2, ensure_ascii=False)) diff --git a/config.json b/config.json index c4cd435..d627fbc 100644 --- a/config.json +++ b/config.json @@ -24,6 +24,9 @@ "post_correct_warmup_timeout": 20.0, "post_correct_min_overlap": 0.45, "post_correct_debug": false, + "freeflow_polish": true, + "post_correct_skip_clean": true, + "post_correct_prompt_style": "freeflow", "llm_paragraph": false, "llm_paragraph_temperature": 0.2, "llm_paragraph_top_p": 0.8, diff --git a/config_v3.json b/config_v3.json new file mode 100644 index 0000000..6757df1 --- /dev/null +++ b/config_v3.json @@ -0,0 +1,45 @@ +{ + "model": "mlx-community/whisper-base-mlx", + "device": null, + "lang": null, + "silence": 1000, + "max_buffer": 20, + "channels": 1, + "es": false, + "fr": false, + "ar": false, + "en": false, + "ingest": false, + "filter_lang": false, + "stream": false, + "context": false, + "quantize": false, + "speaker_diarization": false, + "post_correct": true, + "post_correct_llm": false, + "post_correct_model": "qwen3.5:0.8b", + "post_correct_ollama_url": "http://127.0.0.1:11434/api/generate", + "post_correct_llm_timeout": 8.0, + "post_correct_keep_alive": "30m", + "post_correct_warmup_timeout": 20.0, + "post_correct_min_overlap": 0.45, + "post_correct_debug": false, + "freeflow_polish": true, + "post_correct_skip_clean": true, + "post_correct_prompt_style": "freeflow", + "llm_paragraph": false, + "llm_paragraph_temperature": 0.2, + "llm_paragraph_top_p": 0.8, + "llm_paragraph_repeat_penalty": 1.0, + "llm_request_log_path": "logs/llm_requests.jsonl", + "only_translate_llm": false, + "temperature_fallback": [0.0, 0.2, 0.4, 0.6, 0.8, 1.0], + "logprob_threshold": -0.8, + "compression_threshold": 2.2, + "mt_max_chars": 250, + "mt_max_new_tokens": 150, + "mt_num_beams": 4, + "engine": "apple", + "apple_locale": null, + "verbose": false +} diff --git a/debug_session_1_issues.md b/debug_session_1_issues.md deleted file mode 100644 index a32ffe1..0000000 --- a/debug_session_1_issues.md +++ /dev/null @@ -1,110 +0,0 @@ -# Debug Session 1: Observed Issues and Low-Effort Fixes - -This note summarizes concrete problems observed in `debug_session_1.log`, with examples and pragmatic fixes that do not require a major redesign. - -## 1. Duplicate Final English Captions - -The pipeline frequently emits the same English line twice in quick succession, usually as a raw/final pair or a minimally corrected pair. - -### Examples - -- `It has a lot of sweet things.` appears twice at `debug_session_1.log:449` and `debug_session_1.log:450`. -- `Because the crisis we're going through is the crisis Cuba is going through.` appears twice at `debug_session_1.log:827` and `debug_session_1.log:828`. -- `And one thing I can say about my brother, Daniel.` appears twice at `debug_session_1.log:923` and `debug_session_1.log:924`. - -### Low-Hanging Fix - -- Track the last emitted finalized English caption and suppress exact or near-exact duplicates before printing or sending downstream. -- Prefer a single user-facing final English line instead of exposing both pre-corrected and corrected variants. - -## 2. Over-Eager Merge Logic - -The short-window merge logic sometimes combines adjacent fragments that should remain separate, producing unnatural or noisy caption revisions. - -### Examples - -- `restaurants, bakeries.` is merged into the previous line as `Right now I'm a partner of that business group, which has Restaurants, bakeries.` at `debug_session_1.log:461` and `debug_session_1.log:462`. -- `A-A-M` is preserved and merged into `A-A-M Prepare ourselves for the future in that sense.` at `debug_session_1.log:534` and `debug_session_1.log:541`. -- Short fragments become a clumsy combined sentence around `Right now.` / `What's coming in the future.` at `debug_session_1.log:1019`, `debug_session_1.log:1025`, and `debug_session_1.log:1032`. - -### Low-Hanging Fix - -- Tighten merge conditions so a merge requires stronger evidence that the current line continues the previous one. -- Reject merges when the current line is too short, looks like noise, or resembles a standalone noun phrase. -- Reject merges when either line contains token patterns like `A-A-M` or other obvious non-lexical fragments. -- Require token overlap or a continuation cue before merging. - -## 3. Runaway Repetition and Hallucination - -The current hallucination checks do not catch repeated multi-word loops reliably, so some bad segments still reach output and translation. - -### Examples - -- Repeated sentence loop at `debug_session_1.log:28`. -- `He got a job.` loop at `debug_session_1.log:34`. -- `So what about ...` repetition burst at `debug_session_1.log:83`. -- Extreme `remember` repetition at `debug_session_1.log:152`. - -### Low-Hanging Fix - -- Expand hallucination detection beyond single-word repetition. -- Add repeated n-gram detection for 2- to 5-word phrases. -- Reject long lines with very low lexical diversity. -- Skip downstream translation when the English bridge is flagged as repetitive garbage. - -## 4. Bad ASR Segments Survive Correction - -Some obviously wrong segments remain wrong even after rule-based and LLM post-correction. - -### Examples - -- `red at this moment.` at `debug_session_1.log:516`. -- `Gong!` at `debug_session_1.log:599`. -- `an ungovernmental organization.` at `debug_session_1.log:647`. -- `So I posted what clarified that last part.` at `debug_session_1.log:875`. -- `It's such a fun to help.` at `debug_session_1.log:1109`. - -### Low-Hanging Fix - -- Add deterministic rejection for filler-only, malformed, or obviously low-information English fragments before translation. -- Expand rule-based cleanup to drop pure filler lines such as `Uh.`, `Um.`, or `Hmm`. -- Add heuristics to reject malformed token patterns before they become finalized captions. - -## 5. Wrong English Pollutes All Translations - -Because translation is English-bridge driven, wrong or noisy English becomes confident multilingual output. - -### Examples - -- `Gong!` propagates into ES/FR/AR around `debug_session_1.log:599`, `debug_session_1.log:601`, `debug_session_1.log:602`, and `debug_session_1.log:604`. -- `ungovernmental organization` is normalized into target-language output around `debug_session_1.log:647`, `debug_session_1.log:649`, `debug_session_1.log:650`, and `debug_session_1.log:652`. - -### Low-Hanging Fix - -- Add a quality gate before translation so clearly bad English is not propagated. -- For rejected English segments, prefer skipping translation over producing confident multilingual noise. - -## 6. Language Filtering Was Necessary - -Earlier runs without `--filter-lang` showed stray source-language outputs that do not fit the intended English-first workflow. - -### Examples - -- `[PT]: Que sempre pulamos.` at `debug_session_1.log:26`. -- `[RU]: и просуло им.` at `debug_session_1.log:32`. -- `[IT]: Three, one.` at `debug_session_1.log:133`. - -### Low-Hanging Fix - -- Keep `--filter-lang --lang en` as part of the preferred operating profile for this use case. -- Preserve this as the default recommendation for live runs unless a multilingual-source workflow is explicitly desired. - -## Recommended First Fixes - -If the goal is best return on effort, the first three fixes to implement are: - -1. Suppress duplicate finalized English captions. -2. Tighten merge heuristics so only clear continuations are merged. -3. Add repeated-phrase hallucination detection and skip translation for clearly bad English. - -These changes target the most visible issues in the log without requiring a new model stack or a broader pipeline redesign. diff --git a/engine_apple_transcribe.py b/engine_apple_transcribe.py new file mode 100644 index 0000000..8fe1a4d --- /dev/null +++ b/engine_apple_transcribe.py @@ -0,0 +1,668 @@ +#!/usr/bin/env python3 +""" +engine_apple_transcribe.py - v3 Apple Speech backend + +Drop-in replacement for engine_transcribe.py using Apple's new +SpeechAnalyzer + SpeechTranscriber (macOS 26+). + +Architecture: + - Uses same Silero VAD chunking as whisper engine (audio_buffer + silence threshold) + - Each finalized chunk -> temp WAV 16k mono -> apple-speech-transcribe subprocess + - Parses bench JSON output -> emits same dict shape as whisper engine + +This keeps the rest of the pipeline (LLM polish, translation, distribution) unchanged. + +Binary location: apple_speech/.build/release/apple-speech-transcribe +Fallback: ../apple-speech-transcribe (project root copy) +""" + +from __future__ import annotations +import sys +from unittest.mock import MagicMock +try: + import lzma +except ImportError: + mock_lzma = MagicMock() + mock_lzma.FORMAT_XZ, mock_lzma.FORMAT_ALONE, mock_lzma.FORMAT_RAW = 1, 2, 3 + mock_lzma.CHECK_NONE, mock_lzma.CHECK_CRC32, mock_lzma.CHECK_CRC64, mock_lzma.CHECK_SHA256 = 0, 1, 4, 10 + sys.modules["_lzma"] = MagicMock() + sys.modules["lzma"] = mock_lzma + +import time +import os +import argparse +import json +import subprocess +import tempfile +import shutil +from pathlib import Path +from typing import Optional + +APPLE_BIN_CANDIDATES = [ + Path(__file__).parent / "apple_speech" / ".build" / "release" / "apple-speech-transcribe", + Path(__file__).parent / "apple_speech" / "apple-speech-transcribe", + Path(__file__).parent / "apple-speech-transcribe", + Path("/tmp/apple-speech-transcribe"), +] + +LANG_TO_LOCALE = { + "en": "en-US", + "en-us": "en-US", + "en-gb": "en-GB", + "en-au": "en-AU", + "es": "es-ES", + "es-es": "es-ES", + "es-mx": "es-MX", + "es-us": "es-US", + "es-cl": "es-CL", + "fr": "fr-FR", + "fr-fr": "fr-FR", + "fr-ca": "fr-CA", + "de": "de-DE", + "de-de": "de-DE", + "it": "it-IT", + "ja": "ja-JP", + "ko": "ko-KR", + "pt": "pt-PT", + "pt-br": "pt-BR", + "zh": "zh-CN", + "zh-cn": "zh-CN", + "zh-tw": "zh-TW", + "yue": "yue-CN", +} + +APPLE_SUPPORTED_LANGS = {"en", "es", "fr", "de", "it", "ja", "ko", "pt", "zh", "yue"} + +def resolve_binary() -> Optional[Path]: + for p in APPLE_BIN_CANDIDATES: + if p.exists() and os.access(p, os.X_OK): + return p + return None + +def resolve_locale(lang: Optional[str], apple_locale_arg: Optional[str] = None) -> str: + if apple_locale_arg: + return apple_locale_arg + if not lang: + return "en-US" + lang_lower = lang.lower().strip() + if lang_lower in LANG_TO_LOCALE: + return LANG_TO_LOCALE[lang_lower] + # try prefix + prefix = lang_lower.split("-")[0] + if prefix in LANG_TO_LOCALE: + return LANG_TO_LOCALE[prefix] + # if lang itself looks like a locale (contains -) try as-is + if "-" in lang and len(lang) >= 4: + return lang + return "en-US" + +def list_audio_devices(): + try: + import sounddevice as sd + print("\nAvailable Audio Devices:") + print(sd.query_devices()) + except ImportError: + print("[Error] sounddevice not installed. Cannot list devices.") + +def _ensure_soundfile(): + try: + import soundfile as sf + return sf + except ImportError: + return None + +def transcribe_chunk_apple(audio_np, sample_rate, binary_path: Path, locale: str, verbose: bool = False) -> Optional[dict]: + """ + Transcribe a numpy audio chunk via apple-speech-transcribe --bench + Returns parsed JSON dict or None. + """ + return _transcribe_chunk_wav(audio_np, sample_rate, binary_path, locale, verbose, mode="bench") + +def _wav_bytes_from_np(audio_np, sample_rate: int): + import numpy as np, wave, io + if audio_np.dtype != np.float32: + if audio_np.dtype == np.int16: + audio_float = audio_np.astype(np.float32) / 32768.0 + else: + audio_float = audio_np.astype(np.float32) + else: + audio_float = audio_np + if audio_float.ndim > 1: + audio_float = audio_float.mean(axis=1) + audio_float = np.clip(audio_float, -1.0, 1.0) + if sample_rate != 16000: + duration = len(audio_float) / sample_rate + new_len = int(duration * 16000) + if new_len > 0: + x_old = np.linspace(0, 1, len(audio_float)) + x_new = np.linspace(0, 1, new_len) + audio_float = np.interp(x_new, x_old, audio_float).astype(np.float32) + sample_rate = 16000 + int16_data = (audio_float * 32767).astype(np.int16) + bio = io.BytesIO() + with wave.open(bio, 'w') as wf: + wf.setnchannels(1); wf.setsampwidth(2); wf.setframerate(sample_rate) + wf.writeframes(int16_data.tobytes()) + return bio.getvalue(), sample_rate + +def _transcribe_chunk_wav(audio_np, sample_rate, binary_path: Path, locale: str, verbose: bool, mode: str = "bench") -> Optional[dict]: + import numpy as np + tmp_path = None + try: + import wave, struct, io + wav_bytes, sr = _wav_bytes_from_np(audio_np, sample_rate) + fd, tmp_path = tempfile.mkstemp(suffix=".wav") + os.close(fd) + with open(tmp_path, "wb") as f: + f.write(wav_bytes) + if mode == "bench": + cmd = [str(binary_path), "--bench", tmp_path, "--locale", locale] + else: + # include volatile to get drafts + cmd = [str(binary_path), tmp_path, "--locale", locale, "--include-volatile"] + result = subprocess.run(cmd, capture_output=True, text=True, timeout=8.0) + if result.returncode != 0: + if verbose: + print(f"[Apple] Binary failed {result.returncode}: {result.stderr[:500]}") + return None + if mode == "bench": + try: + payload = json.loads(result.stdout) + except json.JSONDecodeError as e: + if verbose: + print(f"[Apple] JSON parse failed: {e} stdout={result.stdout[:500]}") + return None + return payload + else: + # parse JSONL with volatile+final + finals = [] + volatiles = [] + for line in result.stdout.splitlines(): + line=line.strip() + if not line: continue + try: + seg=json.loads(line) + if seg.get("isFinal"): + finals.append(seg) + else: + volatiles.append(seg) + except: pass + # build combined text from finals + full_text = " ".join(s.get("text","") for s in finals).strip() + full_text = " ".join(full_text.split()) + return {"text": full_text, "segments": finals, "volatiles": volatiles, "audio_duration_sec": len(audio_np)/16000} + except subprocess.TimeoutExpired: + if verbose: print("[Apple] Transcription timed out") + return None + except Exception as e: + if verbose: print(f"[Apple] Error: {e}") + return None + finally: + if tmp_path and os.path.exists(tmp_path): + try: os.unlink(tmp_path) + except: pass + +def transcribe_chunk_apple_with_draft(audio_np, sample_rate, binary_path: Path, locale: str, verbose: bool = False) -> Optional[dict]: + return _transcribe_chunk_wav(audio_np, sample_rate, binary_path, locale, verbose, mode="draft") + + +class ApplePipeTranscriber: + """ + Keeps one Swift binary process alive in --pipe mode. + Write wav chunk -> reads draft/final JSON lines with ultra-low overhead (no subprocess spawn per chunk). + Falls back to bench if pipe not available. + """ + def __init__(self, binary_path: Path, locale: str, verbose=False): + self.binary_path = binary_path + self.locale = locale + self.verbose = verbose + self.proc = None + self.lock = None + self._start() + + def _start(self): + import threading + self.lock = threading.Lock() + cmd = [str(self.binary_path), "--pipe", "--locale", self.locale] + if self.verbose: + cmd.append("-v") + try: + self.proc = subprocess.Popen(cmd, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=False, bufsize=0) + # check alive after 1s + import time + time.sleep(0.2) + if self.proc.poll() is not None: + err = self.proc.stderr.read().decode(errors="ignore") if self.proc.stderr else "" + if self.verbose: + print(f"[ApplePipe] Failed to start, exit={self.proc.returncode} {err[:500]}") + self.proc=None + else: + if self.verbose: + print(f"[ApplePipe] Started pid={self.proc.pid} locale={self.locale}") + except Exception as e: + if self.verbose: + print(f"[ApplePipe] Start error: {e}") + self.proc=None + + def transcribe(self, audio_np, sample_rate) -> Optional[dict]: + if not self.proc or self.proc.poll() is not None: + self._start() + if not self.proc: + return None + try: + wav_bytes, _ = _wav_bytes_from_np(audio_np, sample_rate) + import struct + header = struct.pack(">I", len(wav_bytes)) + with self.lock: + self.proc.stdin.write(header + wav_bytes) + self.proc.stdin.flush() + import time, select + finals=[]; volatiles=[] + start=time.time() + while True: + remaining = 6.0 - (time.time()-start) + if remaining<=0: break + rready,_,_ = select.select([self.proc.stdout], [], [], min(remaining, 0.2)) + if not rready: + if finals: break + continue + line = self.proc.stdout.readline() + if not line: break + try: + txt=line.decode('utf-8', errors='ignore').strip() + if not txt: continue + obj=json.loads(txt) + t=obj.get("text","").strip() + if not t: continue + if obj.get("isFinal") or obj.get("event")=="final": + finals.append(obj) + stall=time.time()+0.35 + while time.time() Optional[dict]: + """ + Streaming version: calls draft_callback(text) for each volatile update (word-by-word), + then returns final dict. Use this for ultra-low latency draft captions. + """ + if not self.proc or self.proc.poll() is not None: + self._start() + if not self.proc: + return None + try: + wav_bytes, _ = _wav_bytes_from_np(audio_np, sample_rate) + import struct, time, select + header = struct.pack(">I", len(wav_bytes)) + with self.lock: + self.proc.stdin.write(header + wav_bytes) + self.proc.stdin.flush() + finals=[]; volatiles=[] + start=time.time() + last_draft="" + while True: + remaining = 6.0 - (time.time()-start) + if remaining<=0: break + rr,_,_ = select.select([self.proc.stdout], [], [], min(remaining, 0.15)) + if not rr: + if finals: break + continue + line=self.proc.stdout.readline() + if not line: break + try: + obj=json.loads(line.decode('utf-8',errors='ignore').strip()) + t=obj.get("text","").strip() + if not t: continue + if obj.get("isFinal") or obj.get("event")=="final": + finals.append(obj) + stall=time.time()+0.35 + while time.time()I", 0)) + self.proc.stdin.flush() + self.proc.stdin.close() + except: pass + self.proc.wait(timeout=3) + except: + try: self.proc.kill() + except: pass + self.proc=None + + + +def run_transcription(out_queue, args): + """ + Main entry compatible with multiprocessing.Process(target=run_transcription) + Same signature as engine_transcribe.py + """ + import numpy as np + import sounddevice as sd + import torch + from silero_vad import load_silero_vad, get_speech_timestamps + import queue + + SAMPLERATE, BLOCK_SIZE, VAD_THRESHOLD = 16000, 512, 0.5 + + binary_path = resolve_binary() + if binary_path is None: + print(f"[AppleTranscribe] ERROR: binary not found. Searched: {APPLE_BIN_CANDIDATES}") + print("[AppleTranscribe] Run: cd apple_speech && bash build.sh") + return + + apple_locale = resolve_locale(getattr(args, "lang", None), getattr(args, "apple_locale", None)) + verbose = getattr(args, "verbose", False) or getattr(args, "post_correct_debug", False) + + print(f"[AppleTranscribe] Using binary: {binary_path}") + print(f"[AppleTranscribe] Locale: {apple_locale} (from lang={getattr(args,'lang',None)})") + print(f"[AppleTranscribe] VAD model loading...") + vad_model = load_silero_vad() + + # Quick health check + try: + chk = subprocess.run([str(binary_path), "--check"], capture_output=True, text=True, timeout=5) + if chk.returncode != 0: + print(f"[AppleTranscribe] --check failed: {chk.stderr}") + else: + # parse last line + print(f"[AppleTranscribe] Binary check OK") + if verbose: + for line in chk.stdout.splitlines(): + print(f" {line}") + except Exception as e: + print(f"[AppleTranscribe] Binary check error: {e}") + + # Optional: preload by transcribing 0.1s silence to warm model + try: + silence = np.zeros(int(SAMPLERATE * 0.1), dtype=np.float32) + _ = transcribe_chunk_apple(silence, SAMPLERATE, binary_path, apple_locale, verbose=False) + print(f"[AppleTranscribe] Warmup done") + except: + pass + + audio_queue = queue.Queue() + + def audio_callback(indata, frames, time_info, status): + if status: + print(status, file=sys.stderr) + audio_queue.put(indata.copy()) + + audio_buffer, speech_started = [], False + last_stream_time = time.time() + last_draft_text = "" + # Feature flags + use_stream = getattr(args, "stream", False) or getattr(args, "apple_stream", True) # default ON for Apple (cheap) + stream_interval = getattr(args, "apple_stream_interval", 1.0) # seconds between draft transcribes + use_pipe = getattr(args, "apple_pipe", True) # keep one Swift process alive (faster than spawn per chunk) + pipe_transcriber = None + + if use_pipe: + try: + pipe_transcriber = ApplePipeTranscriber(binary_path, apple_locale, verbose=verbose) + if pipe_transcriber.proc is None: + pipe_transcriber = None + print("[AppleTranscribe] Pipe mode unavailable, falling back to subprocess per chunk") + except Exception as e: + if verbose: + print(f"[AppleTranscribe] Pipe init failed: {e}") + pipe_transcriber = None + + device = "mps" if torch.backends.mps.is_available() else "cpu" + + def do_transcribe_final(audio_np): + if pipe_transcriber and pipe_transcriber.proc and pipe_transcriber.proc.poll() is None: + return pipe_transcriber.transcribe(audio_np, SAMPLERATE) + return transcribe_chunk_apple(audio_np, SAMPLERATE, binary_path, apple_locale, verbose=verbose) + + def do_transcribe_with_streaming_drafts(audio_np, on_draft): + """Use pipe's volatile streaming for real-time drafts (60ms granularity).""" + if pipe_transcriber and pipe_transcriber.proc and pipe_transcriber.proc.poll() is None: + return pipe_transcriber.transcribe_with_draft_callback(audio_np, SAMPLERATE, draft_callback=on_draft) + # fallback: no streaming, just final + return do_transcribe_final(audio_np) + + try: + with sd.InputStream(samplerate=SAMPLERATE, channels=getattr(args, "channels", 1), + callback=audio_callback, blocksize=BLOCK_SIZE, device=getattr(args, "device", None)): + print(f"[AppleTranscribe] Listening (silence={getattr(args,'silence',1000)}ms max_buffer={getattr(args,'max_buffer',20)}s stream={use_stream} pipe={pipe_transcriber is not None})...") + while True: + while not audio_queue.empty(): + data = audio_queue.get() + if data is not None and data.size > 0: + if getattr(args, "channels", 1) > 1: + data = np.mean(data, axis=1) + audio_buffer.append(data.flatten()) + + if audio_buffer: + current_audio = np.concatenate(audio_buffer) + audio_tensor = torch.from_numpy(current_audio) + + speech_timestamps = get_speech_timestamps( + audio_tensor, vad_model, + sampling_rate=SAMPLERATE, + threshold=VAD_THRESHOLD, + min_silence_duration_ms=getattr(args, "silence", 1000) + ) + + if speech_timestamps: + speech_started = True + # --- draft streaming (non-blocking poll every stream_interval) --- + if use_stream and (time.time() - last_stream_time) > stream_interval: + if len(current_audio) > SAMPLERATE * 0.6: + def _emit_draft(txt): + nonlocal last_draft_text + t = " ".join(txt.split()).strip() + if t and t != last_draft_text and len(t) > 1: + out_queue.put({"draft": t, "ts": time.time(), "isApple": True}) + if verbose: + print(f"\r[DRAFT]: {t} ", end="", flush=True) + last_draft_text = t + + # quick draft via pipe streaming - but we are inside input stream callback thread? + # Use quick final call with draft callback if pipe else interim transcribe + try: + # For low-latency drafts while speaking, transcribe current buffer + # with incremental volatile updates showing word-by-word + snap = current_audio.copy() # avoid racing with audio_buffer appending + # If pipe not available, fall back to fast volatile parse + if pipe_transcriber and pipe_transcriber.proc: + # Use a separate thread to not block VAD loop too long? But okay + # We'll call blocking but it returns within 200ms + _ = pipe_transcriber.transcribe_with_draft_callback( + snap, SAMPLERATE, draft_callback=_emit_draft + ) + else: + dr = transcribe_chunk_apple_with_draft(snap, SAMPLERATE, binary_path, apple_locale, verbose=False) + if dr: + txt = dr.get("text", "").strip() + if txt: + _emit_draft(txt) + except Exception as e: + if verbose: + print(f"[Apple DRAFT err] {e}") + last_stream_time = time.time() + + silence_frames = len(current_audio) - speech_timestamps[-1]['end'] + should_flush = (silence_frames > (SAMPLERATE * getattr(args, "silence", 1000) / 1000)) or \ + len(current_audio) > (SAMPLERATE * getattr(args, "max_buffer", 20)) + + if should_flush: + # Cancel any pending draft display and transcribe final + # Use streaming final for best latency + true partials + t0 = time.time() + + # Use pipe final (fast) without draft callback for this flush + result = do_transcribe_final(current_audio) + dt = time.time() - t0 + + if result is None: + print(f"[AppleTranscribe] Chunk failed (len={len(current_audio)/SAMPLERATE:.2f}s)") + audio_buffer, speech_started = [], False + last_draft_text="" + last_stream_time = time.time() + continue + + text = result.get("text", "").strip() + text = " ".join(text.split()) + + if not text: + audio_buffer, speech_started = [], False + last_draft_text="" + last_stream_time = time.time() + continue + + detected_lang = getattr(args, "lang", None) or apple_locale.split("-")[0] + audio_dur = result.get("audio_duration_sec", len(current_audio)/SAMPLERATE) + rtf = result.get("rtf", result.get("processing_sec", 0) and (audio_dur / max(result.get("processing_sec",0.001),0.001)) or 0) + if use_stream: + print(flush=True) + + out_queue.put({ + "original": text, + "en_bridge": text, + "detected_lang": detected_lang, + "speaker": None, + "ts": time.time(), + "_apple_meta": { + "locale": apple_locale, + "rtf": rtf, + "processing_sec": result.get("processing_sec"), + "segments": result.get("segments", []), + "pipe": result.get("pipe", False), + } + }) + + print(f"[AppleTranscribe] {detected_lang.upper()} [{audio_dur:.1f}s-> {dt*1000:.0f}ms RTF={rtf:.1f}x]: {text}") + audio_buffer, speech_started = [], False + last_draft_text="" + last_stream_time = time.time() + + elif not speech_started and len(current_audio) > SAMPLERATE * 2: + audio_buffer = [] + last_draft_text="" + + except Exception as e: + print(f"[AppleTranscribe] Error: {e}") + import traceback + traceback.print_exc() + finally: + if pipe_transcriber: + pipe_transcriber.close() + + +# For --file mode testing standalone (non-live) +def transcribe_file(file_path: str, locale: str = "en-US", include_volatile: bool = False) -> Optional[dict]: + binary_path = resolve_binary() + if not binary_path: + raise FileNotFoundError(f"Apple speech binary not found. Candidates: {APPLE_BIN_CANDIDATES}") + # For file mode we can call binary directly with JSONL or bench + cmd = [str(binary_path), file_path, "--locale", locale] + if include_volatile: + cmd.append("--include-volatile") + result = subprocess.run(cmd, capture_output=True, text=True, timeout=20) + if result.returncode != 0: + print(f"Failed: {result.stderr}") + return None + # JSONL lines + segments = [] + for line in result.stdout.splitlines(): + line=line.strip() + if not line: + continue + try: + seg = json.loads(line) + segments.append(seg) + except: + pass + full_text = " ".join(s.get("text","") for s in segments if s.get("isFinal")).strip() + full_text = " ".join(full_text.split()) + return {"text": full_text, "segments": segments} + +if __name__ == "__main__": + import argparse + parser = argparse.ArgumentParser() + parser.add_argument("file", nargs="?", help="Audio file to transcribe") + parser.add_argument("--locale", default="en-US") + parser.add_argument("--bench", action="store_true") + parser.add_argument("--include-volatile", action="store_true") + parser.add_argument("--verbose", "-v", action="store_true") + args = parser.parse_args() + + binary_path = resolve_binary() + if not binary_path: + print("Binary not found") + sys.exit(1) + + if args.file: + if args.bench: + cmd = [str(binary_path), "--bench", args.file, "--locale", args.locale] + if args.verbose: + cmd.append("-v") + subprocess.run(cmd) + else: + r = transcribe_file(args.file, args.locale, args.include_volatile) + print(json.dumps(r, indent=2, ensure_ascii=False)) + else: + # live test (just list devices) + list_audio_devices() diff --git a/engine_distribute.py b/engine_distribute.py index 573cd51..a89eb63 100644 --- a/engine_distribute.py +++ b/engine_distribute.py @@ -99,11 +99,16 @@ def run_distribution(in_queue, args): "speaker": payload.get("speaker"), "ts": payload.get("ts") } + english_output = payload.get("english_output") or payload.get("en") + if english_output: + ingest_payload["en"] = english_output has_any_translation = False - for lang in ["es", "fr", "ar", "en"]: + for lang in ["es", "fr", "ar"]: if payload.get(lang): ingest_payload[lang] = payload[lang] has_any_translation = True + if english_output: + has_any_translation = True # If only-translate-llm is on, ONLY send when we have a translation (paragraph-level) should_send = True diff --git a/engine_llm.py b/engine_llm.py index 406be55..8741ef7 100644 --- a/engine_llm.py +++ b/engine_llm.py @@ -18,8 +18,19 @@ import os import re import json import requests +import queue from collections import deque from datetime import datetime, timezone +from freeflow_polish import ( + build_user_prompt, + deterministic_polish, + guard_against_truncation, + is_clean_enough_to_skip_llm, + match_input_casing, + normalize_formatting, + strip_keep_tags, + system_prompt_for_language, +) def build_paragraph_messages(previous_refined_context, previous_source_text, new_source_text): @@ -85,6 +96,15 @@ Hard rules: ], prompt +def build_freeflow_line_messages(text, language=None, model=""): + system_prompt = system_prompt_for_language(language, model=model) + prompt = build_user_prompt(text, language=language) + return [ + {"role": "system", "content": system_prompt}, + {"role": "user", "content": prompt}, + ], prompt + + def append_llm_request_log(log_path, entry): if not log_path: return @@ -229,11 +249,22 @@ def run_llm_prompt_test(args): print(response) if getattr(args, "llm_test_line", None): - messages, prompt = build_line_messages( - getattr(args, "llm_test_prev1", ""), - getattr(args, "llm_test_prev2", ""), - args.llm_test_line, - ) + use_freeflow = getattr(args, "post_correct_prompt_style", "freeflow") != "legacy" + if use_freeflow: + preprocessed, substituted = deterministic_polish(args.llm_test_line) + messages, prompt = build_freeflow_line_messages( + substituted, + language=getattr(args, "lang", None), + model=getattr(args, "post_correct_model", ""), + ) + metadata_input = preprocessed + else: + messages, prompt = build_line_messages( + getattr(args, "llm_test_prev1", ""), + getattr(args, "llm_test_prev2", ""), + args.llm_test_line, + ) + metadata_input = args.llm_test_line response = send_test_request( "line", messages, @@ -242,7 +273,8 @@ def run_llm_prompt_test(args): { "prev1": getattr(args, "llm_test_prev1", ""), "prev2": getattr(args, "llm_test_prev2", ""), - "input": args.llm_test_line, + "input": metadata_input, + "prompt_style": getattr(args, "post_correct_prompt_style", "freeflow"), }, ) print("[LLM TEST] Line response:") @@ -251,6 +283,7 @@ def run_llm_prompt_test(args): if getattr(args, "llm_test_segments", None): messages, prompt = build_paragraph_messages( getattr(args, "llm_test_context", ""), + "", args.llm_test_segments, ) response = send_test_request( @@ -328,6 +361,9 @@ def run_llm_processor(in_queue, out_queue, args): return normalized def apply_rule_based_post_correction(text): + if getattr(args, "freeflow_polish", True): + corrected, _ = deterministic_polish(text) + return corrected corrected = normalize_english_caption(text) corrected = re.sub(r"\s+", " ", corrected).strip() corrected = re.sub(r"\s+([,.;!?])", r"\1", corrected) @@ -417,13 +453,13 @@ def run_llm_processor(in_queue, out_queue, args): entry["duration_ms"] = round((time.time() - started_at) * 1000, 2) append_llm_request_log(log_path, entry) - # State: This is the ONLY text we send to the LLM as context - active_context = "" + # State: Keep a short rolling window of refined paragraphs as context. + refined_context_history = deque(maxlen=2) previous_source_window = "" recent_lines = deque(maxlen=2) - def call_llm_rolling_refine(new_segments_str, context, previous_source_text): - messages, prompt = build_paragraph_messages(context, previous_source_text, new_segments_str) + def call_llm_rolling_refine(new_segments_str, previous_refined_context, previous_source_text): + messages, prompt = build_paragraph_messages(previous_refined_context, previous_source_text, new_segments_str) paragraph_options = { "top_p": getattr(args, "llm_paragraph_top_p", 0.8), "repeat_penalty": getattr(args, "llm_paragraph_repeat_penalty", 1.0), @@ -440,20 +476,92 @@ def run_llm_processor(in_queue, out_queue, args): if not getattr(args, "post_correct", False): return normalize_english_caption(text) - corrected = apply_rule_based_post_correction(text) + corrected, substituted = deterministic_polish(text) if getattr(args, "freeflow_polish", True) else (apply_rule_based_post_correction(text), text) if not getattr(args, "post_correct_llm", False): return corrected + if getattr(args, "post_correct_skip_clean", True) and is_clean_enough_to_skip_llm(corrected): + if getattr(args, "post_correct_debug", False): + print("[LLM] Skipping line LLM polish; deterministic output looks clean.") + return corrected - prev2 = recent_lines[-2] if len(recent_lines) >= 2 else "" - prev1 = recent_lines[-1] if len(recent_lines) >= 1 else "" - messages, prompt = build_line_messages(prev1, prev2, corrected) - candidate = normalize_english_caption(call_llm(messages, prompt, 0.1, "line_warned")) + prompt_style = getattr(args, "post_correct_prompt_style", "freeflow") + if prompt_style == "legacy": + prev2 = recent_lines[-2] if len(recent_lines) >= 2 else "" + prev1 = recent_lines[-1] if len(recent_lines) >= 1 else "" + messages, prompt = build_line_messages(prev1, prev2, corrected) + else: + language = getattr(args, "lang", None) + messages, prompt = build_freeflow_line_messages( + substituted, + language=language, + model=args.post_correct_model if prompt_style == "qwen" else "", + ) + candidate = call_llm(messages, prompt, 0.1, "line_warned").strip() + if prompt_style != "legacy": + fallback = guard_against_truncation(candidate, corrected) + if fallback: + return fallback + candidate = strip_keep_tags(candidate) + candidate = normalize_formatting(candidate) + candidate = match_input_casing(candidate, substituted) + else: + candidate = normalize_english_caption(candidate) if candidate and word_overlap_ratio(corrected, candidate) >= getattr(args, "post_correct_min_overlap", 0.45): return candidate return corrected pending_buffer = [] - last_llm_call_time = time.time() + + def flush_pending_outputs(processed_items): + nonlocal pending_buffer, previous_source_window + if not processed_items: + return + + structured_paragraph = None + paragraph_from_llm = False + if args.llm_paragraph and pending_buffer: + new_batch = " ".join(pending_buffer) + previous_refined_context = "\n\n".join(refined_context_history) + result = call_llm_rolling_refine(new_batch, previous_refined_context, previous_source_window) + + if result: + if "\n\n" in result: + parts = result.split("\n\n") + refined_context_history.clear() + for part in parts: + cleaned_part = part.strip() + if cleaned_part: + refined_context_history.append(cleaned_part) + structured_paragraph = result + else: + cleaned_result = result.strip() + if cleaned_result: + refined_context_history.append(cleaned_result) + structured_paragraph = result + paragraph_from_llm = True + else: + structured_paragraph = new_batch + cleaned_batch = new_batch.strip() + if cleaned_batch: + refined_context_history.append(cleaned_batch) + + previous_source_window = new_batch + + last_index = len(processed_items) - 1 + for index, processed_item in enumerate(processed_items): + out_queue.put({ + "raw": processed_item["raw"], + "corrected": processed_item["corrected"], + "paragraph": structured_paragraph if index == last_index else None, + "en_bridge": processed_item["en_bridge"], + "used_llm_line": bool(getattr(args, "post_correct_llm", False)), + "used_llm_paragraph": paragraph_from_llm if index == last_index else False, + "detected_lang": processed_item.get("detected_lang"), + "speaker": processed_item.get("speaker"), + "ts": processed_item.get("ts", time.time()) + }) + + pending_buffer = [] while True: try: @@ -462,64 +570,41 @@ def run_llm_processor(in_queue, out_queue, args): if "draft" in item: out_queue.put(item) continue + processed_items = [] + pending_item = item - raw_text = item.get("original", "").strip() - bridge_text = item.get("en_bridge", raw_text).strip() - if not raw_text: continue + while pending_item is not None: + raw_text = pending_item.get("original", "").strip() + bridge_text = pending_item.get("en_bridge", raw_text).strip() + if raw_text: + corrected_text = post_correct_line(bridge_text) - corrected_text = post_correct_line(bridge_text) + # Hallucination check + words = corrected_text.lower().split() + if not (len(words) > 10 and len(set(words)) < 3): + pending_buffer.append(corrected_text) + recent_lines.append(corrected_text) + processed_items.append({ + "raw": raw_text, + "corrected": corrected_text, + "en_bridge": bridge_text, + "detected_lang": pending_item.get("detected_lang"), + "speaker": pending_item.get("speaker"), + "ts": pending_item.get("ts", time.time()), + }) - # Hallucination check - words = corrected_text.lower().split() - if len(words) > 10 and len(set(words)) < 3: continue + try: + pending_item = in_queue.get_nowait() + if pending_item is None: + flush_pending_outputs(processed_items) + return + if "draft" in pending_item: + out_queue.put(pending_item) + pending_item = None + except queue.Empty: + pending_item = None - pending_buffer.append(corrected_text) - recent_lines.append(corrected_text) - - should_refine = False - if len(pending_buffer) >= 5: should_refine = True - elif len(pending_buffer) >= 2 and corrected_text.endswith((".", "?", "!")): should_refine = True - elif len(pending_buffer) >= 1 and (time.time() - last_llm_call_time) > 15: should_refine = True - - structured_paragraph = None - paragraph_from_llm = False - if args.llm_paragraph and should_refine: - new_batch = " ".join(pending_buffer) - result = call_llm_rolling_refine(new_batch, active_context, previous_source_window) - - if result: - # Logic to handle paragraph breaks - if "\n\n" in result: - # Split by double newline - parts = result.split("\n\n") - # The last part is the new "Active Context" - active_context = parts[-1].strip() - # The whole result is sent to the user (contains the break) - structured_paragraph = result - else: - # No break, just update the context - active_context = result - structured_paragraph = result - paragraph_from_llm = True - else: - structured_paragraph = new_batch - active_context = new_batch - previous_source_window = new_batch - - pending_buffer = [] - last_llm_call_time = time.time() - - out_queue.put({ - "raw": raw_text, - "corrected": corrected_text, - "paragraph": structured_paragraph, - "en_bridge": bridge_text, - "used_llm_line": bool(getattr(args, "post_correct_llm", False)), - "used_llm_paragraph": paragraph_from_llm, - "detected_lang": item.get("detected_lang"), - "speaker": item.get("speaker"), - "ts": item.get("ts", time.time()) - }) + flush_pending_outputs(processed_items) except Exception as e: print(f"[LLM] Error: {e}") diff --git a/engine_translate.py b/engine_translate.py index 37d015f..76a6323 100644 --- a/engine_translate.py +++ b/engine_translate.py @@ -84,12 +84,14 @@ def run_translation(in_queue, out_queue, args): if args.only_translate_llm: text_to_translate = paragraph_text - if args.en and paragraph_text: - payload["en"] = paragraph_text else: text_to_translate = paragraph_text or corrected_text or raw_text - if args.en: - payload["en"] = text_to_translate + + english_output = text_to_translate or "" + if english_output: + payload["english_output"] = english_output + if args.en and english_output: + payload["en"] = english_output if text_to_translate and translation_engines: for lang_key, (model, tokenizer) in translation_engines.items(): diff --git a/freeflow_polish.py b/freeflow_polish.py new file mode 100644 index 0000000..44e6df1 --- /dev/null +++ b/freeflow_polish.py @@ -0,0 +1,317 @@ +import re + + +# Python port of the Apache-2.0 FreeFlow polish pipeline ideas: +# https://github.com/mrinalwadhwa/freeflow + +SYSTEM_PROMPT_QWEN = """Clean up this dictated text. +Fix punctuation and capitalization. +Return only the cleaned text.""" + +SYSTEM_PROMPT_MINIMAL = """You are a speech-to-text cleanup assistant. The user dictated text in a non-English language and a speech-to-text engine transcribed it. Your job is to clean up the transcription into polished written text. + +Fix filler words, false starts, consecutive repetitions, mid-sentence corrections, punctuation, capitalization, and obvious number formatting. Keep the user's original words. Do not fabricate text. Do not translate. Preserve every ... block exactly as it appears. + +If the transcription is already clean, return it unchanged. Return only the cleaned text.""" + +SYSTEM_PROMPT_ENGLISH = """You are a speech-to-text cleanup assistant. The user dictated text and a speech-to-text engine transcribed it. Your job is to clean up the transcription into polished written text. If the transcription is already clean, return it unchanged. Do not wrap your output in quotes or add any preamble. Return only the cleaned text. If the input starts with a lowercase letter, keep it lowercase. This happens when the user is continuing a sentence. + +Fix punctuation, capitalization, and spelling. Prefer periods, commas, or colons over em-dashes unless an em-dash appears inside a tag. + +When the input already contains explicit punctuation like "!", "?", ":", or ";", preserve it. + +When "period" appears at the end of a clause and does not make sense as a noun, treat it as a punctuation command and replace it with ".". + +Remove filler words and throat-clearing preambles that add no content. Remove unintentional stuttered repetitions. When the speaker corrects themselves with phrases like "no wait", "actually", "sorry", "I mean", "let me rephrase", "never mind", "or rather", or "make that", drop the abandoned version and keep the final version. + +Convert spelled-out numbers to digits where appropriate. Format times, phone numbers, ratios, and currency when the speaker clearly dictated them. + +When the speaker mentions 3 or more items, format them as a vertical list. Use numbered items if the order is explicit; otherwise use bullets. Keep 2-item phrases inline. + +Keep the speaker's word choices. Do not substitute synonyms, rephrase sentences, expand contractions, or add words the speaker did not say. Preserve informal words such as kinda, gonna, wanna, dunno, lemme, sorta, and gotta. + +When "at" appears between a name and a domain, and "dot" appears before the TLD, format it as an email address. + +The input may contain ... tags around symbols inserted by preprocessing. Preserve every ... block in your output exactly as-is, including the tags themselves.""" + + +PUNCTUATION_RULES = [ + (r"\bnew paragraph\b", "[PAR]", True), + (r"\bnew line\b", "[NL]", True), + (r"\bnewline\b", "[NL]", True), + (r"\bquestion mark\b", "?", False), + (r"\bexclamation point\b", "!", False), + (r"\bexclamation mark\b", "!", False), + (r"\bcomma\b", ",", False), + (r"\bcolon\b", ":", False), + (r"\bsemicolon\b", ";", False), + (r"\bem dash\b", "\u2014", True), + (r"\ben dash\b", "\u2013", True), + (r"\bhyphen\b", "-", True), + (r"\bminus\s+(?:sign|symbol)\b", "-", True), + (r"\bopen paren(?:t|thesis)?\b", "(", False), + (r"\bclose paren(?:t|thesis)?\b", ")", False), + (r"\bopen quote\b", "\u201c", False), + (r"\b(?:close|end) quote\b", "\u201d", False), + (r"\bunquote\b", "\u201d", False), + (r"\b(?:apostrophe|single quote)\b", "'", False), + (r"\bopen bracket\b", "[", False), + (r"\bclose bracket\b", "]", False), + (r"\b(?:open )?angle bracket\b", "<", True), + (r"\bless[- ]than sign\b", "<", True), + (r"\bclose angle bracket\b", ">", True), + (r"\bgreater[- ]than sign\b", ">", True), + (r"\bdot dot dot\b", "\u2026", True), + (r"\bellipsis\b", "\u2026", True), + (r"\b(?:ampersand|and sign|and symbol)\b", "&", True), + (r"\b(?:at sign|at symbol)\b", "@", True), + (r"\bhashtag\b", "#", True), + (r"\b(?:back ?slash|slash en)\b", "\\", True), + (r"\bforward slash\b", "/", True), + (r"\b(?:asterisk|asterisk sign)\b", "*", True), + (r"\bunderscore\b", "_", True), + (r"\b(?:percent sign|per cent|percentage symbol)\b", "%", True), + (r"\bdollar sign\b", "$", True), + (r"\b(?:equals sign|equals symbol)\b", "=", True), + (r"\b(?:plus sign|plus symbol)\b", "+", True), + (r"\btrademark sign\b", "\u2122", True), + (r"\btm\b", "\u2122", True), + (r"\bcopyright sign\b", "\u00a9", True), + (r"\bcopyright symbol\b", "\u00a9", True), + (r"\bdegrees?\s+fahrenheit\b", "\u00b0F", True), + (r"\bdegrees?\s+f\b", "\u00b0F", True), + (r"\bdegrees?\s+celsius\b", "\u00b0C", True), + (r"\bdegrees?\s+centigrade\b", "\u00b0C", True), + (r"\b(?:degree sign|degree symbol)\b", "\u00b0", True), +] + +KNOWN_TERMS = { + "redis": "Redis", + "postgresql": "PostgreSQL", + "postgres": "Postgres", + "mongodb": "MongoDB", + "mysql": "MySQL", + "sqlite": "SQLite", + "kubernetes": "Kubernetes", + "docker": "Docker", + "terraform": "Terraform", + "nginx": "Nginx", + "grafana": "Grafana", + "prometheus": "Prometheus", + "cloudflare": "Cloudflare", + "vercel": "Vercel", + "rabbitmq": "RabbitMQ", + "kafka": "Kafka", + "javascript": "JavaScript", + "typescript": "TypeScript", + "python": "Python", + "swift": "Swift", + "rust": "Rust", + "react": "React", + "django": "Django", + "fastapi": "FastAPI", + "nextjs": "Next.js", + "nodejs": "Node.js", + "numpy": "NumPy", + "pandas": "Pandas", + "pytorch": "PyTorch", + "github": "GitHub", + "slack": "Slack", + "figma": "Figma", + "macos": "macOS", + "ios": "iOS", + "iphone": "iPhone", + "xcode": "Xcode", + "aws": "AWS", + "gcp": "GCP", + "api": "API", + "sql": "SQL", + "css": "CSS", + "html": "HTML", + "json": "JSON", + "xml": "XML", + "yaml": "YAML", + "url": "URL", + "cli": "CLI", + "sdk": "SDK", +} + + +def ends_at_sentence_boundary(text): + stripped = (text or "").rstrip() + return bool(stripped) and stripped[-1] in ".?!" + + +def _capitalize_after_pattern(text, pattern): + def repl(match): + return match.group(1) + match.group(2).upper() + + return re.sub(pattern, repl, text) + + +def collapse_adjacent_punctuation(text): + strength = {",": 1, ":": 2, ";": 3, ".": 4, "?": 5, "!": 6} + + def repl(match): + marks = [ch for ch in match.group(0) if ch in strength] + return max(marks, key=lambda ch: strength[ch]) if marks else match.group(0) + + return re.sub(r"([.,;:?!])(?:\s*[.,;:?!])+", repl, text) + + +def strip_noise_phrases(text): + text = re.sub(r"(?i)\b(?:uh huh|uh-huh|mm hmm|mm-hmm)\b[,.]?\s*", "", text) + return re.sub(r" {2,}", " ", text).strip() + + +def strip_filler_sounds(text): + fillers = "um|eh|mmm|uhh|hm|umm|mm|uh|uhhh|uhm|ah|hmm|mh|ehh" + text = re.sub(rf"(?i)\b(?:{fillers})\b[,.]?\s*", "", text) + return re.sub(r" {2,}", " ", text).strip() + + +def clean_spurious_commas(text): + text = re.sub(r",{2,}", ",", text) + text = re.sub(r",\s*(AM|PM|a\.m\.|p\.m\.)", r" \1", text) + text = re.sub(r",(\s*[.!?])", r"\1", text) + text = re.sub(r",\s*$", "", text) + text = re.sub(r",\s*,", ",", text) + return re.sub(r" {2,}", " ", text).strip() + + +def substitute_dictated_punctuation(text, preceding_text=None, casual=False): + result = text or "" + for pattern, replacement, protect in PUNCTUATION_RULES: + value = f"{replacement}" if protect else replacement + result = re.sub(pattern, lambda _m, v=value: v, result, flags=re.IGNORECASE) + + result = result.replace("...", "\u2026") + result = re.sub(r" +([.,;:?!)\]\u201d])", r"\1", result) + result = re.sub(r"([(\[\u201c]) +", r"\1", result) + result = re.sub(r" {2,}", " ", result) + result = collapse_adjacent_punctuation(result) + result = re.sub(r"[,;]\s*(\[(?:PAR|NL)\])", r".\1", result) + result = re.sub(r"([^.!?\s])\s*(\[(?:PAR|NL)\])", r"\1.\2", result) + result = re.sub(r" *\n *", "\n", result) + + if casual: + if result[:1].isupper(): + first_word = re.match(r"[A-Za-z]+", result) + token = first_word.group(0) if first_word else "" + if token != "I" and not (len(token) > 1 and token.isupper()): + result = result[0].lower() + result[1:] + else: + result = _capitalize_after_pattern(result, r"([.!?]\s+)(\w)") + if not (preceding_text and not ends_at_sentence_boundary(preceding_text)): + result = result[:1].upper() + result[1:] if result[:1].isalpha() else result + + result = clean_spurious_commas(result) + result = strip_noise_phrases(result) + result = strip_filler_sounds(result) + return result.strip() + + +def strip_keep_tags(text, casual=False): + result = re.sub(r"(.*?)", r"\1", text or "", flags=re.DOTALL) + result = re.sub(r" *\[PAR\] *", "\n\n", result) + result = re.sub(r" *\[NL\] *", "\n", result) + result = re.sub(r" *\u00b6 *", "\n\n", result) + result = re.sub(r" *\u21b5 *", "\n", result) + result = re.sub(r" +([.,;:?!)\]>\u201d\u2026%\u2122\u00a9\u00b0])", r"\1", result) + result = re.sub(r"([(\[\u201c#$<]) +", r"\1", result) + result = re.sub(r" *([-@/\\_'\u2013\u2014]) +", r"\1", result) + result = re.sub(r"\*[\w* ]*\*", lambda m: m.group(0).replace(" ", ""), result) + result = re.sub(r" {2,}", " ", result) + if not casual: + result = _capitalize_after_pattern(result, r"(\n)(\w)") + return result.strip() + + +def capitalize_known_terms(text): + result = text + for source, replacement in KNOWN_TERMS.items(): + result = re.sub(rf"(? 1 and token.isupper(): + return text + return text[0].lower() + text[1:] + if preprocessed_input[0].isupper() and text[0].islower(): + return text[0].upper() + text[1:] + return text + + +def guard_against_truncation(polished, preprocessed): + if len(preprocessed or "") < 40: + return None + if not polished: + return preprocessed + return preprocessed if len(polished) / len(preprocessed) < 0.25 else None + + +def deterministic_polish(text, preceding_text=None, casual=False): + substituted = substitute_dictated_punctuation(text, preceding_text=preceding_text, casual=casual) + stripped = strip_keep_tags(substituted, casual=casual) + return normalize_formatting(stripped, casual=casual), substituted + + +def build_user_prompt(text, language=None): + parts = [f"Transcription:\n{text}"] + if language: + parts.append(f"Language: {language}") + return "\n\n".join(parts) + + +def system_prompt_for_language(language=None, model=""): + if "qwen" in (model or "").lower(): + return SYSTEM_PROMPT_QWEN + if not language or language == "en": + return SYSTEM_PROMPT_ENGLISH + return SYSTEM_PROMPT_MINIMAL + + +def is_clean_enough_to_skip_llm(text): + normalized = " ".join((text or "").split()) + if not normalized: + return True + if normalized[0].isalpha() and not normalized[0].isupper(): + return False + if not ends_at_sentence_boundary(normalized): + return False + if re.search(r"(?i)\b(um|uh|uhh|umm|ah|hmm|you know|i mean|no wait|actually|sorry|let me rephrase)\b", normalized): + return False + if re.search(r"(?i)\b(\w+)(?:\s+\1\b){1,}", normalized): + return False + return True diff --git a/main_v2.py b/main_v2.py index 0d7b357..1c6d11a 100644 --- a/main_v2.py +++ b/main_v2.py @@ -56,6 +56,11 @@ def main(): parser.add_argument("--post-correct-warmup-timeout", type=float, default=defaults.get("post_correct_warmup_timeout", 20.0)) parser.add_argument("--post-correct-min-overlap", type=float, default=defaults.get("post_correct_min_overlap", 0.45)) parser.add_argument("--post-correct-debug", action="store_true", default=defaults.get("post_correct_debug", False)) + parser.add_argument("--freeflow-polish", action="store_true", default=defaults.get("freeflow_polish", True), help="Use Freeflow-style deterministic punctuation/filler polishing before optional LLM cleanup.") + parser.add_argument("--no-freeflow-polish", action="store_false", dest="freeflow_polish", help="Use the older local regex cleanup instead of Freeflow-style deterministic polishing.") + parser.add_argument("--post-correct-skip-clean", action="store_true", default=defaults.get("post_correct_skip_clean", True), help="Skip the line LLM when deterministic polish already looks clean.") + parser.add_argument("--no-post-correct-skip-clean", action="store_false", dest="post_correct_skip_clean", help="Always call the line LLM when --post-correct-llm is enabled.") + parser.add_argument("--post-correct-prompt-style", choices=["freeflow", "qwen", "legacy"], default=defaults.get("post_correct_prompt_style", "freeflow"), help="Prompt style for line LLM cleanup.") parser.add_argument("--llm-paragraph", action="store_true", default=defaults.get("llm_paragraph", False)) parser.add_argument("--llm-paragraph-temperature", type=float, default=defaults.get("llm_paragraph_temperature", 0.2)) parser.add_argument("--llm-paragraph-top-p", type=float, default=defaults.get("llm_paragraph_top_p", 0.8)) diff --git a/main_v3.py b/main_v3.py new file mode 100644 index 0000000..ce587fd --- /dev/null +++ b/main_v3.py @@ -0,0 +1,242 @@ +import multiprocessing +import argparse +import json as _json +import os +import sys +from dotenv import load_dotenv + +load_dotenv() + +from engine_transcribe import run_transcription as run_whisper_transcription, list_audio_devices + +def _lazy_apple(): + try: + from engine_apple_transcribe import run_transcription as run_apple + return run_apple + except Exception as e: + print(f"[Main] Apple engine import failed: {e}") + return None + +from engine_llm import run_llm_processor, run_llm_prompt_test +# engine_translate is imported lazily inside main() to avoid PIL crash when not needed +from engine_distribute import run_distribution + +def _run_translation_passthrough(in_q, out_q, a): + """No MarianMT import — just map LLM output to distribution shape. Top-level for spawn pickling.""" + import time as _time + while True: + try: + item = in_q.get() + if item is None: + break + if "draft" in item: + out_q.put(item) + continue + raw_text = item.get("raw") + corrected = item.get("corrected") + paragraph = item.get("paragraph") + to_show = paragraph if getattr(a, "only_translate_llm", False) else (paragraph or corrected or raw_text) + payload = { + "original": raw_text, + "corrected": corrected, + "paragraph": paragraph, + "en_bridge": item.get("en_bridge"), + "english_output": to_show or "", + "used_llm_line": item.get("used_llm_line", False), + "used_llm_paragraph": item.get("used_llm_paragraph", False), + "paragraph_fallback": bool(paragraph) and not item.get("used_llm_paragraph", False), + "speaker": item.get("speaker"), + "ts": item.get("ts", _time.time()), + } + if getattr(a, "en", False) and to_show: + payload["en"] = to_show + out_q.put(payload) + except Exception as e: + print(f"[Translate-passthrough] Error: {e}") + +def parse_temperature_fallback(value): + try: + return tuple(float(x.strip()) for x in value.split(",") if x.strip()) + except: + return (0.0, 0.2, 0.4, 0.6, 0.8, 1.0) + +def main(): + config_path = "config.json" + defaults = {} + if os.path.exists(config_path): + with open(config_path, "r") as f: + defaults = _json.load(f) + + parser = argparse.ArgumentParser(description="Multi-process Transcription and Translation (v3: whisper + apple).") + + # Engine selector + parser.add_argument("--engine", choices=["whisper", "apple"], default=defaults.get("engine", "whisper"), + help="Transcription engine: whisper (mlx) or apple (SpeechAnalyzer macOS 26+) (default: whisper)") + parser.add_argument("--apple-locale", type=str, default=defaults.get("apple_locale", None), + help="Apple locale override e.g. en-US, es-ES, fr-FR (default: auto from --lang)") + parser.add_argument("--apple-bench-file", type=str, default=None, + help="Quick bench: transcribe a file with apple engine and exit") + parser.add_argument("--apple-stream", action="store_true", default=defaults.get("apple_stream", True), + help="Apple: enable draft streaming every ~1s while speaking (default: on)") + parser.add_argument("--no-apple-stream", action="store_false", dest="apple_stream") + parser.add_argument("--apple-stream-interval", type=float, default=defaults.get("apple_stream_interval", 1.0), + help="Apple: seconds between draft transcribes (default: 1.0)") + parser.add_argument("--apple-pipe", action="store_true", default=defaults.get("apple_pipe", True), + help="Apple: keep one Swift process alive via --pipe (default: on, faster)") + parser.add_argument("--no-apple-pipe", action="store_false", dest="apple_pipe") + + # Transcribe Args (shared) + parser.add_argument("-l", "--list-devices", action="store_true", help="Show available audio devices and exit") + parser.add_argument("--model", type=str, default=defaults.get("model", "mlx-community/whisper-base-mlx")) + parser.add_argument("--device", type=int, default=defaults.get("device")) + parser.add_argument("--lang", type=str, default=defaults.get("lang")) + parser.add_argument("--silence", type=int, default=defaults.get("silence", 1000)) + parser.add_argument("--max-buffer", type=int, default=defaults.get("max_buffer", 20)) + parser.add_argument("--channels", type=int, default=defaults.get("channels", 1)) + parser.add_argument("--filter-lang", action="store_true", default=defaults.get("filter_lang", False)) + parser.add_argument("-q", "--quantize", action="store_true", default=defaults.get("quantize", False)) + parser.add_argument("-s", "--stream", action="store_true", default=defaults.get("stream", False)) + parser.add_argument("-c", "--context", action="store_true", default=defaults.get("context", False)) + parser.add_argument("--speaker-diarization", action="store_true", default=defaults.get("speaker_diarization", False)) + parser.add_argument("--temperature-fallback", type=parse_temperature_fallback, default=tuple(defaults.get("temperature_fallback", [0.0, 0.2, 0.4, 0.6, 0.8, 1.0]))) + parser.add_argument("--logprob-threshold", type=float, default=defaults.get("logprob_threshold", -0.8)) + parser.add_argument("--compression-threshold", type=float, default=defaults.get("compression_threshold", 2.2)) + parser.add_argument("--verbose", "-v", action="store_true", default=defaults.get("verbose", False)) + + # LLM Args + parser.add_argument("--post-correct", action="store_true", default=defaults.get("post_correct", False)) + parser.add_argument("--post-correct-llm", action="store_true", default=defaults.get("post_correct_llm", False)) + parser.add_argument("--post-correct-model", type=str, default=defaults.get("post_correct_model", "qwen3.5:0.8b")) + parser.add_argument("--post-correct-ollama-url", type=str, default=defaults.get("post_correct_ollama_url", "http://127.0.0.1:11434/api/generate")) + parser.add_argument("--post-correct-llm-timeout", type=float, default=defaults.get("post_correct_llm_timeout", 8.0)) + parser.add_argument("--post-correct-keep-alive", type=str, default=defaults.get("post_correct_keep_alive", "30m")) + parser.add_argument("--post-correct-warmup-timeout", type=float, default=defaults.get("post_correct_warmup_timeout", 20.0)) + parser.add_argument("--post-correct-min-overlap", type=float, default=defaults.get("post_correct_min_overlap", 0.45)) + parser.add_argument("--post-correct-debug", action="store_true", default=defaults.get("post_correct_debug", False)) + parser.add_argument("--freeflow-polish", action="store_true", default=defaults.get("freeflow_polish", True)) + parser.add_argument("--no-freeflow-polish", action="store_false", dest="freeflow_polish") + parser.add_argument("--post-correct-skip-clean", action="store_true", default=defaults.get("post_correct_skip_clean", True)) + parser.add_argument("--no-post-correct-skip-clean", action="store_false", dest="post_correct_skip_clean") + parser.add_argument("--post-correct-prompt-style", choices=["freeflow", "qwen", "legacy"], default=defaults.get("post_correct_prompt_style", "freeflow")) + parser.add_argument("--llm-paragraph", action="store_true", default=defaults.get("llm_paragraph", False)) + parser.add_argument("--llm-paragraph-temperature", type=float, default=defaults.get("llm_paragraph_temperature", 0.2)) + parser.add_argument("--llm-paragraph-top-p", type=float, default=defaults.get("llm_paragraph_top_p", 0.8)) + parser.add_argument("--llm-paragraph-repeat-penalty", type=float, default=defaults.get("llm_paragraph_repeat_penalty", 1.0)) + parser.add_argument("--llm-request-log-path", type=str, default=defaults.get("llm_request_log_path", "logs/llm_requests.jsonl")) + parser.add_argument("--llm-test-line", type=str, default=None) + parser.add_argument("--llm-test-prev1", type=str, default="") + parser.add_argument("--llm-test-prev2", type=str, default="") + parser.add_argument("--llm-test-segments", type=str, default=None) + parser.add_argument("--llm-test-context", type=str, default="") + parser.add_argument("--llm-test-from-log", type=int, default=None) + parser.add_argument("--llm-test-use-logged-model", action="store_true", default=False) + + # Translate Args + parser.add_argument("-es", action="store_true", default=defaults.get("es", False)) + parser.add_argument("-fr", action="store_true", default=defaults.get("fr", False)) + parser.add_argument("-ar", action="store_true", default=defaults.get("ar", False)) + parser.add_argument("-en", action="store_true", default=defaults.get("en", False)) + parser.add_argument("--only-translate-llm", action="store_true", default=defaults.get("only_translate_llm", False)) + parser.add_argument("--mt-max-chars", type=int, default=defaults.get("mt_max_chars", 250)) + parser.add_argument("--mt-max-new-tokens", type=int, default=defaults.get("mt_max_new_tokens", 150)) + parser.add_argument("--mt-num-beams", type=int, default=defaults.get("mt_num_beams", 4)) + + # Distribute + parser.add_argument("-i", "--ingest", action="store_true", default=defaults.get("ingest", False)) + + args = parser.parse_args() + + # Auto-enable + if args.post_correct_llm and not args.post_correct: + args.post_correct = True + if args.only_translate_llm and not args.llm_paragraph: + print("[Main] Enabling --llm-paragraph because --only-translate-llm was requested.") + args.llm_paragraph = True + + # Apple bench file shortcut + if args.apple_bench_file: + from engine_apple_transcribe import resolve_binary, resolve_locale + import subprocess, json + binary = resolve_binary() + if not binary: + print("Apple binary not found. Run cd apple_speech && bash build.sh") + sys.exit(1) + locale = resolve_locale(args.lang, args.apple_locale) + cmd = [str(binary), "--bench", args.apple_bench_file, "--locale", locale, "-v"] + print(f"[Main] Running: {' '.join(cmd)}") + subprocess.run(cmd) + return + + if args.llm_test_line or args.llm_test_segments or args.llm_test_from_log is not None: + run_llm_prompt_test(args) + return + + if args.list_devices: + list_audio_devices() + return + + if args.device is None: + list_audio_devices() + try: + val = input("\nSelect input device index (or press Enter for default): ").strip() + if val: + args.device = int(val) + except EOFError: + pass + except ValueError: + print("[Main] Invalid index, using default.") + + # Choose engine + if args.engine == "apple": + run_transcription_fn = _lazy_apple() + if run_transcription_fn is None: + print("[Main] Falling back to whisper because apple engine unavailable") + run_transcription_fn = run_whisper_transcription + else: + run_transcription_fn = run_whisper_transcription + + print(f"[Main] Engine: {args.engine}") + + needs_translate = bool(args.es or args.fr or args.ar or args.en) + q_trans_to_llm = multiprocessing.Queue() + q_llm_to_tl = multiprocessing.Queue() + q_tl_to_dist = multiprocessing.Queue() + + p_transcribe = multiprocessing.Process(target=run_transcription_fn, args=(q_trans_to_llm, args)) + p_llm = multiprocessing.Process(target=run_llm_processor, args=(q_trans_to_llm, q_llm_to_tl, args)) + + if not needs_translate: + print("[Main] Translate disabled (no -es/-fr/-ar/-en) — using passthrough (no MarianMT/PIL)") + p_translate = multiprocessing.Process(target=_run_translation_passthrough, args=(q_llm_to_tl, q_tl_to_dist, args)) + else: + try: + from engine_translate import run_translation as _real_translate + p_translate = multiprocessing.Process(target=_real_translate, args=(q_llm_to_tl, q_tl_to_dist, args)) + except Exception as e: + print(f"[Main] Translate import failed ({e}) — falling back to passthrough") + p_translate = multiprocessing.Process(target=_run_translation_passthrough, args=(q_llm_to_tl, q_tl_to_dist, args)) + + p_distribute = multiprocessing.Process(target=run_distribution, args=(q_tl_to_dist, args)) + + print("[Main] Starting processes...") + p_transcribe.start() + p_llm.start() + p_translate.start() + p_distribute.start() + + try: + p_transcribe.join() + p_llm.join() + p_translate.join() + p_distribute.join() + except KeyboardInterrupt: + print("\n[Main] Stopping processes...") + p_transcribe.terminate() + p_llm.terminate() + p_translate.terminate() + p_distribute.terminate() + sys.exit(0) + +if __name__ == "__main__": + multiprocessing.freeze_support() + main() diff --git a/prompt_experiments.py b/prompt_experiments.py index b2b5ee1..2da96d1 100644 --- a/prompt_experiments.py +++ b/prompt_experiments.py @@ -401,6 +401,73 @@ Decision rules: """ variants.append((f"grounded_ctx{window_size}_no_prev_source", system_ctx, user_ctx_no_prev_source)) + if window_size == 2: + system_ctx_paragraph = """You are a careful live transcript editor working from noisy translated source text. + +Goal: +Produce faithful, readable English paragraphs. + +Decision rules: +1. NEW SOURCE TEXT is the primary evidence and should dominate the output. +2. Use PREVIOUS SOURCE TEXT and PREVIOUS REFINED CONTEXT only when they clearly help resolve or continue the new material. +3. The final answer must be English only. +4. Never copy non-English words into the output unless they are proper names. +5. If a non-English fragment appears as an isolated clause, trailing fragment, or side comment without a clear English anchor, drop it. +6. Only translate a non-English fragment when it is clearly central to the same idea and its meaning is reasonably obvious from context. +7. If NEW SOURCE TEXT continues the same thought as PREVIOUS REFINED CONTEXT, merge them into one coherent paragraph. +8. If NEW SOURCE TEXT starts a fresh thought, start a new paragraph instead of forcing a merge. +9. Remove exact or near-exact repetition unless the repetition is clearly intentional rhetoric. +10. Make obvious caption-word mistakes make sense only when the intended English wording is reasonably clear from context. +11. Prefer conservative wording over guessed meaning. +12. Use a double newline between distinct paragraphs. +13. Do not add explanations, disclaimers, labels, or meta commentary. +14. Return only the revised paragraph text.""" + user_ctx_paragraph = f"""Synthesize the transcript into coherent paragraphs. + +[PREVIOUS REFINED CONTEXT] +{context_window} + +[PREVIOUS SOURCE TEXT] +"{prev_raw}" + +[NEW SOURCE TEXT] +"{new_raw}" +""" + variants.append(("grounded_ctx2_paragraph_live", system_ctx_paragraph, user_ctx_paragraph)) + + system_ctx_paragraph_soft = """You are a careful live transcript editor working from noisy translated source text. + +Goal: +Produce the most faithful readable English update. + +Decision rules: +1. NEW SOURCE TEXT is the primary evidence and should dominate the output. +2. Use PREVIOUS SOURCE TEXT and PREVIOUS REFINED CONTEXT only when they clearly help resolve or continue the new material. +3. The final answer must be English only. +4. Never copy non-English words into the output unless they are proper names. +5. If a non-English fragment appears as an isolated clause, trailing fragment, or side comment without a clear English anchor, drop it. +6. Only translate a non-English fragment when it is clearly central to the same idea and its meaning is reasonably obvious from context. +7. If NEW SOURCE TEXT clearly continues PREVIOUS REFINED CONTEXT, you may merge them naturally. +8. If NEW SOURCE TEXT starts a fresh thought, output only the new material and do not drag older context forward. +9. If both pieces are relevant but distinct adjacent thoughts, separate them with a double newline. +10. Remove exact or near-exact repetition unless the repetition is clearly intentional rhetoric. +11. Make obvious caption-word mistakes make sense only when the intended English wording is reasonably clear from context. +12. Prefer conservative wording over guessed meaning. +13. Do not add explanations, disclaimers, labels, or meta commentary. +14. Return only the revised transcript text.""" + user_ctx_paragraph_soft = f"""Edit this transcript update as readable paragraph text. + +[PREVIOUS REFINED CONTEXT] +{context_window} + +[PREVIOUS SOURCE TEXT] +"{prev_raw}" + +[NEW SOURCE TEXT] +"{new_raw}" +""" + variants.append(("grounded_ctx2_paragraph_soft", system_ctx_paragraph_soft, user_ctx_paragraph_soft)) + system_ctx_repair = """You are a careful live transcript editor working from noisy translated source text. Goal: