feat: warm up ollama and keep post-correct model loaded

This commit is contained in:
Adolfo Reyna
2026-03-06 20:48:11 -05:00
parent 2b501f9cdd
commit 4115ddfa8b
2 changed files with 52 additions and 21 deletions
+9
View File
@@ -194,3 +194,12 @@ The project now features a high-performance, Apple Silicon-optimized pipeline th
- Added token-overlap acceptance guard (`--post-correct-min-overlap`) to reject LLM rewrites that diverge too far from source text.
- Updated merge-decider prompt to stricter JSON behavior and added overlap-based fallback to heuristic merge.
- **Outcome:** Improved protection against context bleed (e.g., replacing current line with prior sentence) while keeping optional LLM improvements.
## Phase 24: Ollama Warmup + Keep-Alive
- **Goal:** Reduce intermittent post-correction timeouts caused by cold model loads.
- **Approach:**
- Added startup warmup request for the local post-correct model when LLM correction is enabled.
- Added configurable Ollama `keep_alive` setting so the model stays resident between caption calls.
- Routed post-correct and merge-decider requests through a shared Ollama caller with keep-alive support.
- Added flags for warmup timeout and optional warmup skip for troubleshooting.
- **Outcome:** Lower first-request latency and fewer fallback-to-rules events due to local model cold starts.