Document Grace Poppy image voice app project

2026-07-07 10:08:11 -04:00
parent 9ac7fb09de
commit 2d2cea9584
2 changed files with 416 additions and 0 deletions
@@ -5,6 +5,7 @@
  - [[Livestream and Video Generation Process]]: Document current livestream/video workflow and prepare for media producer advice.
  - [[EMI Livestream and Media Workflow Report for Ari / Sardius]]: Shareable first-draft brief focused on workflow architecture, translation, archive/search, and AI-assisted media production.
  - [[reMarkable Hermes Communication App]]: Tablet/Hermes communication app over reMarkable Paper Pro.
+  - [[Grace Poppy Storybook Image Voice App]]: Document the Grace/Poppy image-generation and narration playback workflow, with next steps for a triggerable app and correction pipeline.
  - [[PARA Brain MCP Server]]: Server-side MCP/search layer for the Reyna family brain.
 - **[[Areas]]:** Ongoing responsibilities and household maintenance.
 - **[[Resources]]:** Interests, reference material, and research.
@@ -0,0 +1,415 @@
+---
+Date: 2026-07-07
+Author: Hermes
+Tags: [project, family, kids, storybook, esp32, image-generation, voice, codex, tactility]
+---
+
+# Grace/Poppy Storybook Image + Voice App
+
+## Summary
+
+This project documents the workflow used to turn a family storybook package for Princess Grace and Poppy into a synchronized image-and-audio experience on an ESP32/Tactility color screen device. The current working prototype is coordinated by Hermes: Hermes displays each generated image on the device, plays the matching narration audio, waits for playback, then advances to the next page.
+
+The next project phase is to turn this into an app that runs the playback flow itself, so Hermes can trigger it and the device can auto-play the book without Hermes coordinating every image and audio call.
+
+Related idea: this workflow should support future corrections after generation, including language changes, image detail edits, narration pacing, and selecting different voice profiles.
+
+## Source Material
+
+The starting point was an existing Grace/Poppy ESP32 package:
+
+- Local package: `/home/adolforeyna/grace_poppy_esp32_package.zip`
+- Extracted working folder: `/home/adolforeyna/output/grace_poppy_existing/`
+- Original page images: `/home/adolforeyna/output/grace_poppy_existing/pages_300x400/page_XX.jpg`
+- Original narration audio: `/home/adolforeyna/output/grace_poppy_existing/mp3/page_XX.mp3`
+- Original transcript: `/home/adolforeyna/output/grace_poppy_existing/transcript.txt`
+
+The package contained page images, MP3 narration files, and RGB565-ready assets. The new workflow used the original page images and transcript as reference material, then generated new storybook images with improved/consistent styling.
+
+## Important Continuity Rule
+
+Princess Grace is the purple/lavender princess.
+
+This is a key continuity detail. The story text does not explicitly say this, but it is deduced from other pages and should be preserved in all future generation or correction passes.
+
+Generation prompts should include:
+
+- Grace wears purple/lavender.
+- Grace has blonde curly hair and a small tiara.
+- Grace should remain visually primary when she is the focus of the page.
+- Do not make Grace the yellow princess.
+
+## Image Generation Workflow Used
+
+Images were generated on the Mac mini using Codex CLI and the built-in `imagegen` skill. The reliable pattern was to make Codex save or copy the generated image to a deterministic path, write the final response to a known file with `-o`, and then read that file.
+
+Important working details:
+
+- Mac mini host used for generation: Mac mini at `192.168.68.102`
+- Codex CLI path observed: `/Users/adolforeyna/.local/bin/codex`
+- Codex version observed: `codex-cli 0.142.5`
+- Mac mini output folder: `/Users/adolforeyna/output/imagegen/grace-poppy/`
+- Local output folder: `/home/adolforeyna/output/imagegen/grace-poppy/`
+
+The validated Codex pattern:
+
+```bash
+OUT="$HOME/output/imagegen/<project>/<image-name>-$(date +%Y%m%d-%H%M%S).png"
+LAST="/tmp/codex-imagegen-last.txt"
+mkdir -p "$(dirname "$OUT")"
+
+codex exec \
+  --skip-git-repo-check \
+  -C "$HOME" \
+  -s workspace-write \
+  -o "$LAST" \
+  "Use the imagegen skill to generate one PNG image.
+
+Prompt: <image prompt>
+
+Save or copy the final generated image to this exact absolute path:
+$OUT
+
+After the file exists, your final response must be exactly this path and nothing else:
+$OUT" </dev/null
+
+cat "$LAST"
+```
+
+Notes from testing:
+
+- Do not use `-a never`; this Codex version no longer accepts it. Approval already defaulted to `never` in the observed runs.
+- Avoid piping/heredoc content directly in a way that Codex may continue reading from stdin. Use a prompt file or clean argument and redirect stdin from `/dev/null`.
+- Use `--image <reference>` when giving Codex a reference image for scene/style continuity.
+- The generated full-size images were copied back locally and resized to `320x240` for the ESP32/Tactility display.
+
+## Generated Image Set
+
+Corrected/generated ESP32-ready images are in:
+
+- `/home/adolforeyna/output/imagegen/grace-poppy/`
+
+Final ZIP for the generated ESP32 images:
+
+- `/home/adolforeyna/output/imagegen/grace-poppy/grace-poppy-generated-all-esp32-320x240.zip`
+
+Preview contact sheet:
+
+- `/home/adolforeyna/output/imagegen/grace-poppy/grace-poppy-contact-sheet.png`
+
+Generated pages tested:
+
+- Page 01: castle garden, Grace corrected to purple/lavender
+- Page 02: adventurous Grace in castle garden
+- Page 03: hallway/vase/closet
+- Page 04: five kittens
+- Page 05: family finds kittens
+- Page 06: kitten on dining table
+- Page 07: Grace promises to care for Poppy
+- Page 08: princesses play with kittens
+- Page 09: Poppy slides on hallway floor
+- Page 10: Poppy missing
+- Page 11: Grace comforted by the king
+- Page 12: king sends search party
+- Page 13: countryside search
+- Page 14: Black Woods entrance
+- Page 15: Black Woods search
+- Page 16: Grace prays
+- Page 17: Grace finds Poppy
+- Page 18: ride home / feast
+- Page 20: fireplace safe at home
+- Page 22: parable thoughts
+- Page 23: lost sheep parable
+
+## Device Test Target
+
+The synchronized playback was tested on the ESP32/Tactility device:
+
+- IP: `192.168.68.113`
+- Dashboard: `http://192.168.68.113/dashboard.html`
+- MCP endpoint: `POST http://192.168.68.113/api/mcp`
+
+Important discovery:
+
+- `GET /api/mcp` returned `404`, but this did not mean MCP was unavailable.
+- `POST /api/mcp` worked and returned the available tools.
+- The device reported a `320x240` color display and supported `rgb565_base64`, `bmp_base64`, and `pbm_base64`.
+
+Useful device capabilities observed:
+
+```json
+{
+  "color": true,
+  "width": 320,
+  "height": 240,
+  "formats": ["rgb565_base64", "bmp_base64", "pbm_base64"],
+  "audio": {
+    "sampleRate": 16000,
+    "bitsPerSample": 16,
+    "channels": 1
+  },
+  "platform": "tactility",
+  "appVersion": "0.1.0"
+}
+```
+
+## Display Path That Worked
+
+Each `320x240` PNG was converted to big-endian RGB565 and sent through MCP:
+
+- MCP tool: `draw_raw_rgb565`
+- Arguments:
+  - `rgb565_base64`: base64-encoded big-endian RGB565 pixel payload
+  - `x`: `0`
+  - `y`: `0`
+  - `w`: `320`
+  - `h`: `240`
+
+Each displayed image was `153600` bytes of RGB565 payload before base64 encoding:
+
+```text
+320 * 240 * 2 = 153600 bytes
+```
+
+Every page image tested successfully on `.113` with:
+
+```text
+Raw RGB565 data displayed successfully.
+```
+
+## Audio Path That Worked
+
+Original package narration files were MP3s:
+
+- `/home/adolforeyna/output/grace_poppy_existing/mp3/page_XX.mp3`
+
+The speaker path was first verified with the MCP `play_tone` tool.
+
+What did not work well:
+
+- Sending the original full-size MP3 files through `play_mp3_base64` was too large for at least some page files and caused a connection reset.
+- `download_file` successfully downloaded an MP3 to the SD card, but `play_mp3` then reported `MP3 file was not found`, apparently because it expects a simple app-user-data location rather than the same SD-card location used by `download_file`.
+
+What worked:
+
+- Transcode narration MP3s to smaller files:
+  - mono
+  - 16 kHz
+  - 24 kbps MP3
+- Send the smaller file through MCP `play_mp3_base64`.
+
+Example transcode command:
+
+```bash
+ffmpeg -y -hide_banner -loglevel error \
+  -i /home/adolforeyna/output/grace_poppy_existing/mp3/page_01.mp3 \
+  -ac 1 -ar 16000 -b:a 24k \
+  /home/adolforeyna/output/grace_poppy_existing/mp3_small/page_01_24k.mp3
+```
+
+The smaller Page 1 file was about `47,744` bytes and played successfully through:
+
+- MCP tool: `play_mp3_base64`
+- Argument: `mp3_base64`
+- Volume tested: `90`
+
+Successful response:
+
+```text
+Successfully played base64 MP3 audio.
+```
+
+## Synchronized Prototype Behavior
+
+The working Hermes-coordinated prototype performs this loop:
+
+1. Convert page image to RGB565.
+2. Send image to the device with `draw_raw_rgb565`.
+3. Send matching small MP3 narration with `play_mp3_base64`.
+4. Wait for playback to finish.
+5. Continue to the next page.
+
+The full set was tested on `192.168.68.113`.
+
+Tested successfully:
+
+- Pages `01–16`: image displayed and audio played.
+- Then pages `17`, `18`, `20`, `22`, `23`: image displayed and audio played.
+
+Every tested page returned:
+
+```text
+Raw RGB565 data displayed successfully.
+Successfully played base64 MP3 audio.
+```
+
+The prototype ended on Page 23 after playing the lost-sheep narration.
+
+## Current Working Scripts and Artifacts
+
+Local project folder:
+
+- `/home/adolforeyna/output/imagegen/grace-poppy/`
+
+Useful scripts created during testing:
+
+- `generate_remaining_pages.py` — generated remaining pages via Mac mini Codex workflow.
+- `test_device_113.py` — tested Page 1 image display on `.113`.
+- `slideshow_test_device_113.py` — tested sending all images without audio.
+- `test_page1_image_audio_113.py` — tested Page 1 display + original base64 MP3; found payload reset issue.
+- `test_page1_download_play_113.py` — tested URL download + `play_mp3`; found app-data path issue.
+- `run_image_audio_slideshow_113.py` — first synchronized image/audio slideshow; worked but included an extra wait and timed out after Page 16.
+- `continue_image_audio_slideshow_113.py` — continued Pages 17, 18, 20, 22, 23 without extra wait.
+
+Small narration files:
+
+- `/home/adolforeyna/output/grace_poppy_existing/mp3_small/page_XX_24k.mp3`
+
+## Lessons Learned
+
+- The Mac mini Codex CLI + `imagegen` path works well for deterministic image generation when the prompt explicitly tells Codex where to save the final file and `-o` captures the final response path.
+- Always include the Grace-purple continuity rule in generation prompts.
+- Full-size generated images should be resized to `320x240` for this ESP32/Tactility display.
+- For this board, `POST /api/mcp` is the real MCP test; `GET /api/mcp` may return `404`.
+- `draw_raw_rgb565` is reliable for displaying page images.
+- `play_tone` is a good first speaker-path test.
+- Original MP3s may be too large for JSON/base64 playback.
+- Smaller `16kHz`, mono, `24kbps` MP3 files worked with `play_mp3_base64`.
+- `play_mp3_base64` appears to block until playback finishes, so the coordinator should not add a second full-duration sleep after it returns.
+- `download_file` and `play_mp3` may not share the same file lookup location. Do not assume a downloaded SD-card file can be played by `play_mp3` without validating the firmware path behavior.
+
+## Desired Next Phase
+
+Build an app so Hermes is not responsible for coordinating every page.
+
+The app should be able to:
+
+1. Store or access a complete book package containing:
+   - page image assets
+   - narration audio assets
+   - page order
+   - page metadata
+   - duration/pacing settings
+   - language/voice metadata
+2. Play the book locally:
+   - show Page 1 image
+   - play Page 1 audio
+   - advance when audio finishes or after configured pacing
+   - continue until the end
+3. Let Hermes trigger playback:
+   - start book
+   - stop book
+   - pause/resume
+   - jump to page
+   - replay page
+   - maybe select language or voice variant
+4. Allow post-generation corrections:
+   - regenerate or replace an image for a page
+   - edit image details while preserving character continuity
+   - change narration language
+   - change narration pacing
+   - change voice profile
+   - regenerate only affected page audio/images
+   - rebuild a package manifest without regenerating everything
+5. Support future review/approval flow:
+   - generate draft assets
+   - preview contact sheet
+   - mark pages needing correction
+   - apply corrections page-by-page
+   - publish a final playable package
+
+## Suggested Book Package Structure
+
+A future app could use a folder or ZIP format like:
+
+```text
+book.json
+images/page_01.png
+images/page_02.png
+audio/en-US/default/page_01.mp3
+audio/en-US/default/page_02.mp3
+audio/es-US/default/page_01.mp3
+corrections/page_01.md
+```
+
+Possible `book.json` fields:
+
+```json
+{
+  "id": "grace-poppy",
+  "title": "Princess Grace and Poppy",
+  "display": {
+    "width": 320,
+    "height": 240,
+    "pixel_format": "rgb565"
+  },
+  "default_language": "en-US",
+  "default_voice_profile": "default",
+  "pages": [
+    {
+      "page": 1,
+      "image": "images/page_01.png",
+      "audio": "audio/en-US/default/page_01.mp3",
+      "duration_sec": 15.8,
+      "caption": "Once upon a time...",
+      "status": "approved"
+    }
+  ]
+}
+```
+
+## Correction Workflow Ideas
+
+Corrections should be recorded as structured page-level notes rather than ad hoc prompts.
+
+Examples:
+
+```yaml
+page: 1
+correction_type: image
+request: "Make Grace's dress more lavender and make the castle slightly smaller."
+constraints:
+  - Grace must remain the purple/lavender princess.
+  - No visible text, logos, or watermark.
+  - Keep 4:3 storybook composition.
+```
+
+```yaml
+page: 7
+correction_type: voice
+request: "Slow this page down and use a warmer dad-reading voice."
+language: en-US
+voice_profile: adolfo-warm
+pacing: slower
+```
+
+Possible correction types:
+
+- `image_detail`
+- `image_style`
+- `image_continuity`
+- `language_translation`
+- `voice_profile`
+- `voice_pacing`
+- `page_order`
+- `text/narration`
+
+## Open Design Questions
+
+- Should the playable app live as a Tactility app on the ESP32 device, or should it be a Hermes-side app that streams commands to the board?
+- Should full assets be stored on the SD card, or should Hermes serve them on demand?
+- Can the firmware expose a reliable app-local audio file path so `download_file` + `play_mp3` works without base64 payloads?
+- Should generated packages include both source-quality images and device-optimized `320x240` assets?
+- Should narration be pre-generated for each voice/language, or generated on demand when a correction is requested?
+- How should children/family members approve or reject generated page variants?
+
+## Current Status
+
+Working prototype completed.
+
+- Generated images are available.
+- Audio was extracted and transcoded successfully.
+- Full image-only slideshow works on `.113`.
+- Full image + audio synchronized playback works on `.113` when using smaller base64 MP3 files.
+
+No implementation changes should be made from this note alone. This is documentation and planning context for the next build phase.