From 2d2cea9584d299d4b3edac7508c0985a8f266770 Mon Sep 17 00:00:00 2001 From: aeroreyna Date: Tue, 7 Jul 2026 10:08:11 -0400 Subject: [PATCH] Document Grace Poppy image voice app project --- index.md | 1 + .../grace_poppy_storybook_image_voice_app.md | 415 ++++++++++++++++++ 2 files changed, 416 insertions(+) create mode 100644 projects/grace_poppy_storybook_image_voice_app.md diff --git a/index.md b/index.md index 9c2a7cd..b8131a5 100644 --- a/index.md +++ b/index.md @@ -5,6 +5,7 @@ - [[Livestream and Video Generation Process]]: Document current livestream/video workflow and prepare for media producer advice. - [[EMI Livestream and Media Workflow Report for Ari / Sardius]]: Shareable first-draft brief focused on workflow architecture, translation, archive/search, and AI-assisted media production. - [[reMarkable Hermes Communication App]]: Tablet/Hermes communication app over reMarkable Paper Pro. + - [[Grace Poppy Storybook Image Voice App]]: Document the Grace/Poppy image-generation and narration playback workflow, with next steps for a triggerable app and correction pipeline. - [[PARA Brain MCP Server]]: Server-side MCP/search layer for the Reyna family brain. - **[[Areas]]:** Ongoing responsibilities and household maintenance. - **[[Resources]]:** Interests, reference material, and research. diff --git a/projects/grace_poppy_storybook_image_voice_app.md b/projects/grace_poppy_storybook_image_voice_app.md new file mode 100644 index 0000000..4d04646 --- /dev/null +++ b/projects/grace_poppy_storybook_image_voice_app.md @@ -0,0 +1,415 @@ +--- +Date: 2026-07-07 +Author: Hermes +Tags: [project, family, kids, storybook, esp32, image-generation, voice, codex, tactility] +--- + +# Grace/Poppy Storybook Image + Voice App + +## Summary + +This project documents the workflow used to turn a family storybook package for Princess Grace and Poppy into a synchronized image-and-audio experience on an ESP32/Tactility color screen device. The current working prototype is coordinated by Hermes: Hermes displays each generated image on the device, plays the matching narration audio, waits for playback, then advances to the next page. + +The next project phase is to turn this into an app that runs the playback flow itself, so Hermes can trigger it and the device can auto-play the book without Hermes coordinating every image and audio call. + +Related idea: this workflow should support future corrections after generation, including language changes, image detail edits, narration pacing, and selecting different voice profiles. + +## Source Material + +The starting point was an existing Grace/Poppy ESP32 package: + +- Local package: `/home/adolforeyna/grace_poppy_esp32_package.zip` +- Extracted working folder: `/home/adolforeyna/output/grace_poppy_existing/` +- Original page images: `/home/adolforeyna/output/grace_poppy_existing/pages_300x400/page_XX.jpg` +- Original narration audio: `/home/adolforeyna/output/grace_poppy_existing/mp3/page_XX.mp3` +- Original transcript: `/home/adolforeyna/output/grace_poppy_existing/transcript.txt` + +The package contained page images, MP3 narration files, and RGB565-ready assets. The new workflow used the original page images and transcript as reference material, then generated new storybook images with improved/consistent styling. + +## Important Continuity Rule + +Princess Grace is the purple/lavender princess. + +This is a key continuity detail. The story text does not explicitly say this, but it is deduced from other pages and should be preserved in all future generation or correction passes. + +Generation prompts should include: + +- Grace wears purple/lavender. +- Grace has blonde curly hair and a small tiara. +- Grace should remain visually primary when she is the focus of the page. +- Do not make Grace the yellow princess. + +## Image Generation Workflow Used + +Images were generated on the Mac mini using Codex CLI and the built-in `imagegen` skill. The reliable pattern was to make Codex save or copy the generated image to a deterministic path, write the final response to a known file with `-o`, and then read that file. + +Important working details: + +- Mac mini host used for generation: Mac mini at `192.168.68.102` +- Codex CLI path observed: `/Users/adolforeyna/.local/bin/codex` +- Codex version observed: `codex-cli 0.142.5` +- Mac mini output folder: `/Users/adolforeyna/output/imagegen/grace-poppy/` +- Local output folder: `/home/adolforeyna/output/imagegen/grace-poppy/` + +The validated Codex pattern: + +```bash +OUT="$HOME/output/imagegen//-$(date +%Y%m%d-%H%M%S).png" +LAST="/tmp/codex-imagegen-last.txt" +mkdir -p "$(dirname "$OUT")" + +codex exec \ + --skip-git-repo-check \ + -C "$HOME" \ + -s workspace-write \ + -o "$LAST" \ + "Use the imagegen skill to generate one PNG image. + +Prompt: + +Save or copy the final generated image to this exact absolute path: +$OUT + +After the file exists, your final response must be exactly this path and nothing else: +$OUT" ` when giving Codex a reference image for scene/style continuity. +- The generated full-size images were copied back locally and resized to `320x240` for the ESP32/Tactility display. + +## Generated Image Set + +Corrected/generated ESP32-ready images are in: + +- `/home/adolforeyna/output/imagegen/grace-poppy/` + +Final ZIP for the generated ESP32 images: + +- `/home/adolforeyna/output/imagegen/grace-poppy/grace-poppy-generated-all-esp32-320x240.zip` + +Preview contact sheet: + +- `/home/adolforeyna/output/imagegen/grace-poppy/grace-poppy-contact-sheet.png` + +Generated pages tested: + +- Page 01: castle garden, Grace corrected to purple/lavender +- Page 02: adventurous Grace in castle garden +- Page 03: hallway/vase/closet +- Page 04: five kittens +- Page 05: family finds kittens +- Page 06: kitten on dining table +- Page 07: Grace promises to care for Poppy +- Page 08: princesses play with kittens +- Page 09: Poppy slides on hallway floor +- Page 10: Poppy missing +- Page 11: Grace comforted by the king +- Page 12: king sends search party +- Page 13: countryside search +- Page 14: Black Woods entrance +- Page 15: Black Woods search +- Page 16: Grace prays +- Page 17: Grace finds Poppy +- Page 18: ride home / feast +- Page 20: fireplace safe at home +- Page 22: parable thoughts +- Page 23: lost sheep parable + +## Device Test Target + +The synchronized playback was tested on the ESP32/Tactility device: + +- IP: `192.168.68.113` +- Dashboard: `http://192.168.68.113/dashboard.html` +- MCP endpoint: `POST http://192.168.68.113/api/mcp` + +Important discovery: + +- `GET /api/mcp` returned `404`, but this did not mean MCP was unavailable. +- `POST /api/mcp` worked and returned the available tools. +- The device reported a `320x240` color display and supported `rgb565_base64`, `bmp_base64`, and `pbm_base64`. + +Useful device capabilities observed: + +```json +{ + "color": true, + "width": 320, + "height": 240, + "formats": ["rgb565_base64", "bmp_base64", "pbm_base64"], + "audio": { + "sampleRate": 16000, + "bitsPerSample": 16, + "channels": 1 + }, + "platform": "tactility", + "appVersion": "0.1.0" +} +``` + +## Display Path That Worked + +Each `320x240` PNG was converted to big-endian RGB565 and sent through MCP: + +- MCP tool: `draw_raw_rgb565` +- Arguments: + - `rgb565_base64`: base64-encoded big-endian RGB565 pixel payload + - `x`: `0` + - `y`: `0` + - `w`: `320` + - `h`: `240` + +Each displayed image was `153600` bytes of RGB565 payload before base64 encoding: + +```text +320 * 240 * 2 = 153600 bytes +``` + +Every page image tested successfully on `.113` with: + +```text +Raw RGB565 data displayed successfully. +``` + +## Audio Path That Worked + +Original package narration files were MP3s: + +- `/home/adolforeyna/output/grace_poppy_existing/mp3/page_XX.mp3` + +The speaker path was first verified with the MCP `play_tone` tool. + +What did not work well: + +- Sending the original full-size MP3 files through `play_mp3_base64` was too large for at least some page files and caused a connection reset. +- `download_file` successfully downloaded an MP3 to the SD card, but `play_mp3` then reported `MP3 file was not found`, apparently because it expects a simple app-user-data location rather than the same SD-card location used by `download_file`. + +What worked: + +- Transcode narration MP3s to smaller files: + - mono + - 16 kHz + - 24 kbps MP3 +- Send the smaller file through MCP `play_mp3_base64`. + +Example transcode command: + +```bash +ffmpeg -y -hide_banner -loglevel error \ + -i /home/adolforeyna/output/grace_poppy_existing/mp3/page_01.mp3 \ + -ac 1 -ar 16000 -b:a 24k \ + /home/adolforeyna/output/grace_poppy_existing/mp3_small/page_01_24k.mp3 +``` + +The smaller Page 1 file was about `47,744` bytes and played successfully through: + +- MCP tool: `play_mp3_base64` +- Argument: `mp3_base64` +- Volume tested: `90` + +Successful response: + +```text +Successfully played base64 MP3 audio. +``` + +## Synchronized Prototype Behavior + +The working Hermes-coordinated prototype performs this loop: + +1. Convert page image to RGB565. +2. Send image to the device with `draw_raw_rgb565`. +3. Send matching small MP3 narration with `play_mp3_base64`. +4. Wait for playback to finish. +5. Continue to the next page. + +The full set was tested on `192.168.68.113`. + +Tested successfully: + +- Pages `01–16`: image displayed and audio played. +- Then pages `17`, `18`, `20`, `22`, `23`: image displayed and audio played. + +Every tested page returned: + +```text +Raw RGB565 data displayed successfully. +Successfully played base64 MP3 audio. +``` + +The prototype ended on Page 23 after playing the lost-sheep narration. + +## Current Working Scripts and Artifacts + +Local project folder: + +- `/home/adolforeyna/output/imagegen/grace-poppy/` + +Useful scripts created during testing: + +- `generate_remaining_pages.py` — generated remaining pages via Mac mini Codex workflow. +- `test_device_113.py` — tested Page 1 image display on `.113`. +- `slideshow_test_device_113.py` — tested sending all images without audio. +- `test_page1_image_audio_113.py` — tested Page 1 display + original base64 MP3; found payload reset issue. +- `test_page1_download_play_113.py` — tested URL download + `play_mp3`; found app-data path issue. +- `run_image_audio_slideshow_113.py` — first synchronized image/audio slideshow; worked but included an extra wait and timed out after Page 16. +- `continue_image_audio_slideshow_113.py` — continued Pages 17, 18, 20, 22, 23 without extra wait. + +Small narration files: + +- `/home/adolforeyna/output/grace_poppy_existing/mp3_small/page_XX_24k.mp3` + +## Lessons Learned + +- The Mac mini Codex CLI + `imagegen` path works well for deterministic image generation when the prompt explicitly tells Codex where to save the final file and `-o` captures the final response path. +- Always include the Grace-purple continuity rule in generation prompts. +- Full-size generated images should be resized to `320x240` for this ESP32/Tactility display. +- For this board, `POST /api/mcp` is the real MCP test; `GET /api/mcp` may return `404`. +- `draw_raw_rgb565` is reliable for displaying page images. +- `play_tone` is a good first speaker-path test. +- Original MP3s may be too large for JSON/base64 playback. +- Smaller `16kHz`, mono, `24kbps` MP3 files worked with `play_mp3_base64`. +- `play_mp3_base64` appears to block until playback finishes, so the coordinator should not add a second full-duration sleep after it returns. +- `download_file` and `play_mp3` may not share the same file lookup location. Do not assume a downloaded SD-card file can be played by `play_mp3` without validating the firmware path behavior. + +## Desired Next Phase + +Build an app so Hermes is not responsible for coordinating every page. + +The app should be able to: + +1. Store or access a complete book package containing: + - page image assets + - narration audio assets + - page order + - page metadata + - duration/pacing settings + - language/voice metadata +2. Play the book locally: + - show Page 1 image + - play Page 1 audio + - advance when audio finishes or after configured pacing + - continue until the end +3. Let Hermes trigger playback: + - start book + - stop book + - pause/resume + - jump to page + - replay page + - maybe select language or voice variant +4. Allow post-generation corrections: + - regenerate or replace an image for a page + - edit image details while preserving character continuity + - change narration language + - change narration pacing + - change voice profile + - regenerate only affected page audio/images + - rebuild a package manifest without regenerating everything +5. Support future review/approval flow: + - generate draft assets + - preview contact sheet + - mark pages needing correction + - apply corrections page-by-page + - publish a final playable package + +## Suggested Book Package Structure + +A future app could use a folder or ZIP format like: + +```text +book.json +images/page_01.png +images/page_02.png +audio/en-US/default/page_01.mp3 +audio/en-US/default/page_02.mp3 +audio/es-US/default/page_01.mp3 +corrections/page_01.md +``` + +Possible `book.json` fields: + +```json +{ + "id": "grace-poppy", + "title": "Princess Grace and Poppy", + "display": { + "width": 320, + "height": 240, + "pixel_format": "rgb565" + }, + "default_language": "en-US", + "default_voice_profile": "default", + "pages": [ + { + "page": 1, + "image": "images/page_01.png", + "audio": "audio/en-US/default/page_01.mp3", + "duration_sec": 15.8, + "caption": "Once upon a time...", + "status": "approved" + } + ] +} +``` + +## Correction Workflow Ideas + +Corrections should be recorded as structured page-level notes rather than ad hoc prompts. + +Examples: + +```yaml +page: 1 +correction_type: image +request: "Make Grace's dress more lavender and make the castle slightly smaller." +constraints: + - Grace must remain the purple/lavender princess. + - No visible text, logos, or watermark. + - Keep 4:3 storybook composition. +``` + +```yaml +page: 7 +correction_type: voice +request: "Slow this page down and use a warmer dad-reading voice." +language: en-US +voice_profile: adolfo-warm +pacing: slower +``` + +Possible correction types: + +- `image_detail` +- `image_style` +- `image_continuity` +- `language_translation` +- `voice_profile` +- `voice_pacing` +- `page_order` +- `text/narration` + +## Open Design Questions + +- Should the playable app live as a Tactility app on the ESP32 device, or should it be a Hermes-side app that streams commands to the board? +- Should full assets be stored on the SD card, or should Hermes serve them on demand? +- Can the firmware expose a reliable app-local audio file path so `download_file` + `play_mp3` works without base64 payloads? +- Should generated packages include both source-quality images and device-optimized `320x240` assets? +- Should narration be pre-generated for each voice/language, or generated on demand when a correction is requested? +- How should children/family members approve or reject generated page variants? + +## Current Status + +Working prototype completed. + +- Generated images are available. +- Audio was extracted and transcoded successfully. +- Full image-only slideshow works on `.113`. +- Full image + audio synchronized playback works on `.113` when using smaller base64 MP3 files. + +No implementation changes should be made from this note alone. This is documentation and planning context for the next build phase.