Add LLM logging and prompt experiment harness

This commit is contained in:
Adolfo Reyna
2026-03-17 11:13:22 -04:00
parent 727b7701c1
commit 7da02a9697
6 changed files with 678 additions and 45 deletions
+26
View File
@@ -29,6 +29,32 @@ A real-time, low-latency audio transcription and translation tool utilizing Open
Run the script using `python3 transcribe.py` with optional flags.
### LLM Prompt Testing
You can now send the line-correction or paragraph-refinement prompts directly without starting live transcription. Every LLM request is appended as JSON Lines to `logs/llm_requests.jsonl` by default.
- **Test the line-correction prompt:**
```bash
python3 main_v2.py --post-correct-model qwen3.5:0.8b --llm-test-line "so um we should probably ship it tomorrow" --llm-test-prev1 "We finished the staging deploy." --llm-test-prev2 "QA signed off this morning."
```
- **Test the paragraph-refinement prompt:**
```bash
python3 main_v2.py --post-correct-model qwen3.5:0.8b --llm-test-context "We reviewed the launch checklist." --llm-test-segments "we confirmed monitoring we confirmed rollback and then talked about the release window"
```
- **Change where requests are logged:**
```bash
python3 main_v2.py --llm-request-log-path tmp/my_llm_requests.jsonl --llm-test-line "example text"
```
- **Replay the most recent logged request:**
```bash
python3 main_v2.py --llm-test-from-log -1
```
- **Replay a logged request using the original model from the log entry:**
```bash
python3 main_v2.py --llm-test-from-log 12 --llm-test-use-logged-model
```
Each log entry includes the timestamp, provider, model, mode, temperature, and full `messages` payload that was sent to the LLM.
### Common Commands
- **List available audio devices:**
```bash