Document Grace Poppy image voice app project

2026-07-07 10:08:11 -04:00
parent 9ac7fb09de
commit 2d2cea9584
2 changed files with 416 additions and 0 deletions
@@ -5,6 +5,7 @@
  - [[Livestream and Video Generation Process]]: Document current livestream/video workflow and prepare for media producer advice.
  - [[EMI Livestream and Media Workflow Report for Ari / Sardius]]: Shareable first-draft brief focused on workflow architecture, translation, archive/search, and AI-assisted media production.
  - [[reMarkable Hermes Communication App]]: Tablet/Hermes communication app over reMarkable Paper Pro.
  - [[Grace Poppy Storybook Image Voice App]]: Document the Grace/Poppy image-generation and narration playback workflow, with next steps for a triggerable app and correction pipeline.
  - [[PARA Brain MCP Server]]: Server-side MCP/search layer for the Reyna family brain.
 - **[[Areas]]:** Ongoing responsibilities and household maintenance.
 - **[[Resources]]:** Interests, reference material, and research.
@@ -0,0 +1,415 @@
 ---
 Date: 2026-07-07
 Author: Hermes
 Tags: [project, family, kids, storybook, esp32, image-generation, voice, codex, tactility]
 ---
 # Grace/Poppy Storybook Image + Voice App
 ## Summary
 This project documents the workflow used to turn a family storybook package for Princess Grace and Poppy into a synchronized image-and-audio experience on an ESP32/Tactility color screen device. The current working prototype is coordinated by Hermes: Hermes displays each generated image on the device, plays the matching narration audio, waits for playback, then advances to the next page.
 The next project phase is to turn this into an app that runs the playback flow itself, so Hermes can trigger it and the device can auto-play the book without Hermes coordinating every image and audio call.
 Related idea: this workflow should support future corrections after generation, including language changes, image detail edits, narration pacing, and selecting different voice profiles.
 ## Source Material
 The starting point was an existing Grace/Poppy ESP32 package:
 - Local package: `/home/adolforeyna/grace_poppy_esp32_package.zip`
 - Extracted working folder: `/home/adolforeyna/output/grace_poppy_existing/`
 - Original page images: `/home/adolforeyna/output/grace_poppy_existing/pages_300x400/page_XX.jpg`
 - Original narration audio: `/home/adolforeyna/output/grace_poppy_existing/mp3/page_XX.mp3`
 - Original transcript: `/home/adolforeyna/output/grace_poppy_existing/transcript.txt`
 The package contained page images, MP3 narration files, and RGB565-ready assets. The new workflow used the original page images and transcript as reference material, then generated new storybook images with improved/consistent styling.
 ## Important Continuity Rule
 Princess Grace is the purple/lavender princess.
 This is a key continuity detail. The story text does not explicitly say this, but it is deduced from other pages and should be preserved in all future generation or correction passes.
 Generation prompts should include:
 - Grace wears purple/lavender.
 - Grace has blonde curly hair and a small tiara.
 - Grace should remain visually primary when she is the focus of the page.
 - Do not make Grace the yellow princess.
 ## Image Generation Workflow Used
 Images were generated on the Mac mini using Codex CLI and the built-in `imagegen` skill. The reliable pattern was to make Codex save or copy the generated image to a deterministic path, write the final response to a known file with `-o`, and then read that file.
 Important working details:
 - Mac mini host used for generation: Mac mini at `192.168.68.102`
 - Codex CLI path observed: `/Users/adolforeyna/.local/bin/codex`
 - Codex version observed: `codex-cli 0.142.5`
 - Mac mini output folder: `/Users/adolforeyna/output/imagegen/grace-poppy/`
 - Local output folder: `/home/adolforeyna/output/imagegen/grace-poppy/`
 The validated Codex pattern:
 ```bash
 OUT="$HOME/output/imagegen/<project>/<image-name>-$(date +%Y%m%d-%H%M%S).png"
 LAST="/tmp/codex-imagegen-last.txt"
 mkdir -p "$(dirname "$OUT")"
 codex exec \
  --skip-git-repo-check \
  -C "$HOME" \
  -s workspace-write \
  -o "$LAST" \
  "Use the imagegen skill to generate one PNG image.
 Prompt: <image prompt>
 Save or copy the final generated image to this exact absolute path:
 $OUT
 After the file exists, your final response must be exactly this path and nothing else:
 $OUT" </dev/null
 cat "$LAST"
 ```
 Notes from testing:
 - Do not use `-a never`; this Codex version no longer accepts it. Approval already defaulted to `never` in the observed runs.
 - Avoid piping/heredoc content directly in a way that Codex may continue reading from stdin. Use a prompt file or clean argument and redirect stdin from `/dev/null`.
 - Use `--image <reference>` when giving Codex a reference image for scene/style continuity.
 - The generated full-size images were copied back locally and resized to `320x240` for the ESP32/Tactility display.
 ## Generated Image Set
 Corrected/generated ESP32-ready images are in:
 - `/home/adolforeyna/output/imagegen/grace-poppy/`
 Final ZIP for the generated ESP32 images:
 - `/home/adolforeyna/output/imagegen/grace-poppy/grace-poppy-generated-all-esp32-320x240.zip`
 Preview contact sheet:
 - `/home/adolforeyna/output/imagegen/grace-poppy/grace-poppy-contact-sheet.png`
 Generated pages tested:
 - Page 01: castle garden, Grace corrected to purple/lavender
 - Page 02: adventurous Grace in castle garden
 - Page 03: hallway/vase/closet
 - Page 04: five kittens
 - Page 05: family finds kittens
 - Page 06: kitten on dining table
 - Page 07: Grace promises to care for Poppy
 - Page 08: princesses play with kittens
 - Page 09: Poppy slides on hallway floor
 - Page 10: Poppy missing
 - Page 11: Grace comforted by the king
 - Page 12: king sends search party
 - Page 13: countryside search
 - Page 14: Black Woods entrance
 - Page 15: Black Woods search
 - Page 16: Grace prays
 - Page 17: Grace finds Poppy
 - Page 18: ride home / feast
 - Page 20: fireplace safe at home
 - Page 22: parable thoughts
 - Page 23: lost sheep parable
 ## Device Test Target
 The synchronized playback was tested on the ESP32/Tactility device:
 - IP: `192.168.68.113`
 - Dashboard: `http://192.168.68.113/dashboard.html`
 - MCP endpoint: `POST http://192.168.68.113/api/mcp`
 Important discovery:
 - `GET /api/mcp` returned `404`, but this did not mean MCP was unavailable.
 - `POST /api/mcp` worked and returned the available tools.
 - The device reported a `320x240` color display and supported `rgb565_base64`, `bmp_base64`, and `pbm_base64`.
 Useful device capabilities observed:
 ```json
 {
  "color": true,
  "width": 320,
  "height": 240,
  "formats": ["rgb565_base64", "bmp_base64", "pbm_base64"],
  "audio": {
    "sampleRate": 16000,
    "bitsPerSample": 16,
    "channels": 1
  },
  "platform": "tactility",
  "appVersion": "0.1.0"
 }
 ```
 ## Display Path That Worked
 Each `320x240` PNG was converted to big-endian RGB565 and sent through MCP:
 - MCP tool: `draw_raw_rgb565`
 - Arguments:
  - `rgb565_base64`: base64-encoded big-endian RGB565 pixel payload
  - `x`: `0`
  - `y`: `0`
  - `w`: `320`
  - `h`: `240`
 Each displayed image was `153600` bytes of RGB565 payload before base64 encoding:
 ```text
 320 * 240 * 2 = 153600 bytes
 ```
 Every page image tested successfully on `.113` with:
 ```text
 Raw RGB565 data displayed successfully.
 ```
 ## Audio Path That Worked
 Original package narration files were MP3s:
 - `/home/adolforeyna/output/grace_poppy_existing/mp3/page_XX.mp3`
 The speaker path was first verified with the MCP `play_tone` tool.
 What did not work well:
 - Sending the original full-size MP3 files through `play_mp3_base64` was too large for at least some page files and caused a connection reset.
 - `download_file` successfully downloaded an MP3 to the SD card, but `play_mp3` then reported `MP3 file was not found`, apparently because it expects a simple app-user-data location rather than the same SD-card location used by `download_file`.
 What worked:
 - Transcode narration MP3s to smaller files:
  - mono
  - 16 kHz
  - 24 kbps MP3
 - Send the smaller file through MCP `play_mp3_base64`.
 Example transcode command:
 ```bash
 ffmpeg -y -hide_banner -loglevel error \
  -i /home/adolforeyna/output/grace_poppy_existing/mp3/page_01.mp3 \
  -ac 1 -ar 16000 -b:a 24k \
  /home/adolforeyna/output/grace_poppy_existing/mp3_small/page_01_24k.mp3
 ```
 The smaller Page 1 file was about `47,744` bytes and played successfully through:
 - MCP tool: `play_mp3_base64`
 - Argument: `mp3_base64`
 - Volume tested: `90`
 Successful response:
 ```text
 Successfully played base64 MP3 audio.
 ```
 ## Synchronized Prototype Behavior
 The working Hermes-coordinated prototype performs this loop:
 1. Convert page image to RGB565.
 2. Send image to the device with `draw_raw_rgb565`.
 3. Send matching small MP3 narration with `play_mp3_base64`.
 4. Wait for playback to finish.
 5. Continue to the next page.
 The full set was tested on `192.168.68.113`.
 Tested successfully:
 - Pages `01–16`: image displayed and audio played.
 - Then pages `17`, `18`, `20`, `22`, `23`: image displayed and audio played.
 Every tested page returned:
 ```text
 Raw RGB565 data displayed successfully.
 Successfully played base64 MP3 audio.
 ```
 The prototype ended on Page 23 after playing the lost-sheep narration.
 ## Current Working Scripts and Artifacts
 Local project folder:
 - `/home/adolforeyna/output/imagegen/grace-poppy/`
 Useful scripts created during testing:
 - `generate_remaining_pages.py` — generated remaining pages via Mac mini Codex workflow.
 - `test_device_113.py` — tested Page 1 image display on `.113`.
 - `slideshow_test_device_113.py` — tested sending all images without audio.
 - `test_page1_image_audio_113.py` — tested Page 1 display + original base64 MP3; found payload reset issue.
 - `test_page1_download_play_113.py` — tested URL download + `play_mp3`; found app-data path issue.
 - `run_image_audio_slideshow_113.py` — first synchronized image/audio slideshow; worked but included an extra wait and timed out after Page 16.
 - `continue_image_audio_slideshow_113.py` — continued Pages 17, 18, 20, 22, 23 without extra wait.
 Small narration files:
 - `/home/adolforeyna/output/grace_poppy_existing/mp3_small/page_XX_24k.mp3`
 ## Lessons Learned
 - The Mac mini Codex CLI + `imagegen` path works well for deterministic image generation when the prompt explicitly tells Codex where to save the final file and `-o` captures the final response path.
 - Always include the Grace-purple continuity rule in generation prompts.
 - Full-size generated images should be resized to `320x240` for this ESP32/Tactility display.
 - For this board, `POST /api/mcp` is the real MCP test; `GET /api/mcp` may return `404`.
 - `draw_raw_rgb565` is reliable for displaying page images.
 - `play_tone` is a good first speaker-path test.
 - Original MP3s may be too large for JSON/base64 playback.
 - Smaller `16kHz`, mono, `24kbps` MP3 files worked with `play_mp3_base64`.
 - `play_mp3_base64` appears to block until playback finishes, so the coordinator should not add a second full-duration sleep after it returns.
 - `download_file` and `play_mp3` may not share the same file lookup location. Do not assume a downloaded SD-card file can be played by `play_mp3` without validating the firmware path behavior.
 ## Desired Next Phase
 Build an app so Hermes is not responsible for coordinating every page.
 The app should be able to:
 1. Store or access a complete book package containing:
   - page image assets
   - narration audio assets
   - page order
   - page metadata
   - duration/pacing settings
   - language/voice metadata
 2. Play the book locally:
   - show Page 1 image
   - play Page 1 audio
   - advance when audio finishes or after configured pacing
   - continue until the end
 3. Let Hermes trigger playback:
   - start book
   - stop book
   - pause/resume
   - jump to page
   - replay page
   - maybe select language or voice variant
 4. Allow post-generation corrections:
   - regenerate or replace an image for a page
   - edit image details while preserving character continuity
   - change narration language
   - change narration pacing
   - change voice profile
   - regenerate only affected page audio/images
   - rebuild a package manifest without regenerating everything
 5. Support future review/approval flow:
   - generate draft assets
   - preview contact sheet
   - mark pages needing correction
   - apply corrections page-by-page
   - publish a final playable package
 ## Suggested Book Package Structure
 A future app could use a folder or ZIP format like:
 ```text
 book.json
 images/page_01.png
 images/page_02.png
 audio/en-US/default/page_01.mp3
 audio/en-US/default/page_02.mp3
 audio/es-US/default/page_01.mp3
 corrections/page_01.md
 ```
 Possible `book.json` fields:
 ```json
 {
  "id": "grace-poppy",
  "title": "Princess Grace and Poppy",
  "display": {
    "width": 320,
    "height": 240,
    "pixel_format": "rgb565"
  },
  "default_language": "en-US",
  "default_voice_profile": "default",
  "pages": [
    {
      "page": 1,
      "image": "images/page_01.png",
      "audio": "audio/en-US/default/page_01.mp3",
      "duration_sec": 15.8,
      "caption": "Once upon a time...",
      "status": "approved"
    }
  ]
 }
 ```
 ## Correction Workflow Ideas
 Corrections should be recorded as structured page-level notes rather than ad hoc prompts.
 Examples:
 ```yaml
 page: 1
 correction_type: image
 request: "Make Grace's dress more lavender and make the castle slightly smaller."
 constraints:
  - Grace must remain the purple/lavender princess.
  - No visible text, logos, or watermark.
  - Keep 4:3 storybook composition.
 ```
 ```yaml
 page: 7
 correction_type: voice
 request: "Slow this page down and use a warmer dad-reading voice."
 language: en-US
 voice_profile: adolfo-warm
 pacing: slower
 ```
 Possible correction types:
 - `image_detail`
 - `image_style`
 - `image_continuity`
 - `language_translation`
 - `voice_profile`
 - `voice_pacing`
 - `page_order`
 - `text/narration`
 ## Open Design Questions
 - Should the playable app live as a Tactility app on the ESP32 device, or should it be a Hermes-side app that streams commands to the board?
 - Should full assets be stored on the SD card, or should Hermes serve them on demand?
 - Can the firmware expose a reliable app-local audio file path so `download_file` + `play_mp3` works without base64 payloads?
 - Should generated packages include both source-quality images and device-optimized `320x240` assets?
 - Should narration be pre-generated for each voice/language, or generated on demand when a correction is requested?
 - How should children/family members approve or reject generated page variants?
 ## Current Status
 Working prototype completed.
 - Generated images are available.
 - Audio was extracted and transcoded successfully.
 - Full image-only slideshow works on `.113`.
 - Full image + audio synchronized playback works on `.113` when using smaller base64 MP3 files.
 No implementation changes should be made from this note alone. This is documentation and planning context for the next build phase.