Document debug session issues and quick fixes

debug_session_1_issues.md

@@ -0,0 +1,110 @@
+# Debug Session 1: Observed Issues and Low-Effort Fixes
+
+This note summarizes concrete problems observed in `debug_session_1.log`, with examples and pragmatic fixes that do not require a major redesign.
+
+## 1. Duplicate Final English Captions
+
+The pipeline frequently emits the same English line twice in quick succession, usually as a raw/final pair or a minimally corrected pair.
+
+### Examples
+
+- `It has a lot of sweet things.` appears twice at `debug_session_1.log:449` and `debug_session_1.log:450`.
+- `Because the crisis we're going through is the crisis Cuba is going through.` appears twice at `debug_session_1.log:827` and `debug_session_1.log:828`.
+- `And one thing I can say about my brother, Daniel.` appears twice at `debug_session_1.log:923` and `debug_session_1.log:924`.
+
+### Low-Hanging Fix
+
+- Track the last emitted finalized English caption and suppress exact or near-exact duplicates before printing or sending downstream.
+- Prefer a single user-facing final English line instead of exposing both pre-corrected and corrected variants.
+
+## 2. Over-Eager Merge Logic
+
+The short-window merge logic sometimes combines adjacent fragments that should remain separate, producing unnatural or noisy caption revisions.
+
+### Examples
+
+- `restaurants, bakeries.` is merged into the previous line as `Right now I'm a partner of that business group, which has Restaurants, bakeries.` at `debug_session_1.log:461` and `debug_session_1.log:462`.
+- `A-A-M` is preserved and merged into `A-A-M Prepare ourselves for the future in that sense.` at `debug_session_1.log:534` and `debug_session_1.log:541`.
+- Short fragments become a clumsy combined sentence around `Right now.` / `What's coming in the future.` at `debug_session_1.log:1019`, `debug_session_1.log:1025`, and `debug_session_1.log:1032`.
+
+### Low-Hanging Fix
+
+- Tighten merge conditions so a merge requires stronger evidence that the current line continues the previous one.
+- Reject merges when the current line is too short, looks like noise, or resembles a standalone noun phrase.
+- Reject merges when either line contains token patterns like `A-A-M` or other obvious non-lexical fragments.
+- Require token overlap or a continuation cue before merging.
+
+## 3. Runaway Repetition and Hallucination
+
+The current hallucination checks do not catch repeated multi-word loops reliably, so some bad segments still reach output and translation.
+
+### Examples
+
+- Repeated sentence loop at `debug_session_1.log:28`.
+- `He got a job.` loop at `debug_session_1.log:34`.
+- `So what about ...` repetition burst at `debug_session_1.log:83`.
+- Extreme `remember` repetition at `debug_session_1.log:152`.
+
+### Low-Hanging Fix
+
+- Expand hallucination detection beyond single-word repetition.
+- Add repeated n-gram detection for 2- to 5-word phrases.
+- Reject long lines with very low lexical diversity.
+- Skip downstream translation when the English bridge is flagged as repetitive garbage.
+
+## 4. Bad ASR Segments Survive Correction
+
+Some obviously wrong segments remain wrong even after rule-based and LLM post-correction.
+
+### Examples
+
+- `red at this moment.` at `debug_session_1.log:516`.
+- `Gong!` at `debug_session_1.log:599`.
+- `an ungovernmental organization.` at `debug_session_1.log:647`.
+- `So I posted what clarified that last part.` at `debug_session_1.log:875`.
+- `It's such a fun to help.` at `debug_session_1.log:1109`.
+
+### Low-Hanging Fix
+
+- Add deterministic rejection for filler-only, malformed, or obviously low-information English fragments before translation.
+- Expand rule-based cleanup to drop pure filler lines such as `Uh.`, `Um.`, or `Hmm`.
+- Add heuristics to reject malformed token patterns before they become finalized captions.
+
+## 5. Wrong English Pollutes All Translations
+
+Because translation is English-bridge driven, wrong or noisy English becomes confident multilingual output.
+
+### Examples
+
+- `Gong!` propagates into ES/FR/AR around `debug_session_1.log:599`, `debug_session_1.log:601`, `debug_session_1.log:602`, and `debug_session_1.log:604`.
+- `ungovernmental organization` is normalized into target-language output around `debug_session_1.log:647`, `debug_session_1.log:649`, `debug_session_1.log:650`, and `debug_session_1.log:652`.
+
+### Low-Hanging Fix
+
+- Add a quality gate before translation so clearly bad English is not propagated.
+- For rejected English segments, prefer skipping translation over producing confident multilingual noise.
+
+## 6. Language Filtering Was Necessary
+
+Earlier runs without `--filter-lang` showed stray source-language outputs that do not fit the intended English-first workflow.
+
+### Examples
+
+- `[PT]: Que sempre pulamos.` at `debug_session_1.log:26`.
+- `[RU]: и просуло им.` at `debug_session_1.log:32`.
+- `[IT]: Three, one.` at `debug_session_1.log:133`.
+
+### Low-Hanging Fix
+
+- Keep `--filter-lang --lang en` as part of the preferred operating profile for this use case.
+- Preserve this as the default recommendation for live runs unless a multilingual-source workflow is explicitly desired.
+
+## Recommended First Fixes
+
+If the goal is best return on effort, the first three fixes to implement are:
+
+1. Suppress duplicate finalized English captions.
+2. Tighten merge heuristics so only clear continuations are merged.
+3. Add repeated-phrase hallucination detection and skip translation for clearly bad English.
+
+These changes target the most visible issues in the log without requiring a new model stack or a broader pipeline redesign.