adolforeyna/FamReynaBrain
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9fc352b26354fc76f8e3844935e170b5122f5243
FamReynaBrain/projects/grace_poppy_storybook_image_voice_app.md
T
aeroreyna 9fc352b263 Document Grace Poppy page one audio trial
2026-07-08 22:11:46 -04:00

21 KiB
Raw Blame History

Date, Author, Tags
Date Author Tags
2026-07-07 Hermes
project
family
kids
storybook
esp32
image-generation
voice
codex
tactility

Grace/Poppy Storybook Image + Voice App

Summary

This project documents the workflow used to turn a family storybook package for Princess Grace and Poppy into a synchronized image-and-audio experience on an ESP32/Tactility color screen device. The current working prototype is coordinated by Hermes: Hermes displays each generated image on the device, plays the matching narration audio, waits for playback, then advances to the next page.

The next project phase is to turn this into an app that runs the playback flow itself, so Hermes can trigger it and the device can auto-play the book without Hermes coordinating every image and audio call.

Related idea: this workflow should support future corrections after generation, including language changes, image detail edits, narration pacing, and selecting different voice profiles.

Source Material

The starting point was an existing Grace/Poppy ESP32 package:

  • Local package: /home/adolforeyna/grace_poppy_esp32_package.zip
  • Extracted working folder: /home/adolforeyna/output/grace_poppy_existing/
  • Original page images: /home/adolforeyna/output/grace_poppy_existing/pages_300x400/page_XX.jpg
  • Original narration audio: /home/adolforeyna/output/grace_poppy_existing/mp3/page_XX.mp3
  • Original transcript: /home/adolforeyna/output/grace_poppy_existing/transcript.txt

The package contained page images, MP3 narration files, and RGB565-ready assets. The new workflow used the original page images and transcript as reference material, then generated new storybook images with improved/consistent styling.

Important Continuity Rule

Princess Grace is the purple/lavender princess.

This is a key continuity detail. The story text does not explicitly say this, but it is deduced from other pages and should be preserved in all future generation or correction passes.

Generation prompts should include:

  • Grace wears purple/lavender.
  • Grace has blonde curly hair and a small tiara.
  • Grace should remain visually primary when she is the focus of the page.
  • Do not make Grace the yellow princess.

Image Generation Workflow Used

Images were generated on the Mac mini using Codex CLI and the built-in imagegen skill. The reliable pattern was to make Codex save or copy the generated image to a deterministic path, write the final response to a known file with -o, and then read that file.

Important working details:

  • Mac mini host used for generation: Mac mini at 192.168.68.102
  • Codex CLI path observed: /Users/adolforeyna/.local/bin/codex
  • Codex version observed: codex-cli 0.142.5
  • Mac mini output folder: /Users/adolforeyna/output/imagegen/grace-poppy/
  • Local output folder: /home/adolforeyna/output/imagegen/grace-poppy/

The validated Codex pattern:

OUT="$HOME/output/imagegen/<project>/<image-name>-$(date +%Y%m%d-%H%M%S).png"
LAST="/tmp/codex-imagegen-last.txt"
mkdir -p "$(dirname "$OUT")"

codex exec \
  --skip-git-repo-check \
  -C "$HOME" \
  -s workspace-write \
  -o "$LAST" \
  "Use the imagegen skill to generate one PNG image.

Prompt: <image prompt>

Save or copy the final generated image to this exact absolute path:
$OUT

After the file exists, your final response must be exactly this path and nothing else:
$OUT" </dev/null

cat "$LAST"

Notes from testing:

  • Do not use -a never; this Codex version no longer accepts it. Approval already defaulted to never in the observed runs.
  • Avoid piping/heredoc content directly in a way that Codex may continue reading from stdin. Use a prompt file or clean argument and redirect stdin from /dev/null.
  • Use --image <reference> when giving Codex a reference image for scene/style continuity.
  • The generated full-size images were copied back locally and resized to 320x240 for the ESP32/Tactility display.

Generated Image Set

Corrected/generated ESP32-ready images are in:

  • /home/adolforeyna/output/imagegen/grace-poppy/

Final ZIP for the generated ESP32 images:

  • /home/adolforeyna/output/imagegen/grace-poppy/grace-poppy-generated-all-esp32-320x240.zip

Preview contact sheet:

  • /home/adolforeyna/output/imagegen/grace-poppy/grace-poppy-contact-sheet.png

Generated pages tested:

  • Page 01: castle garden, Grace corrected to purple/lavender
  • Page 02: adventurous Grace in castle garden
  • Page 03: hallway/vase/closet
  • Page 04: five kittens
  • Page 05: family finds kittens
  • Page 06: kitten on dining table
  • Page 07: Grace promises to care for Poppy
  • Page 08: princesses play with kittens
  • Page 09: Poppy slides on hallway floor
  • Page 10: Poppy missing
  • Page 11: Grace comforted by the king
  • Page 12: king sends search party
  • Page 13: countryside search
  • Page 14: Black Woods entrance
  • Page 15: Black Woods search
  • Page 16: Grace prays
  • Page 17: Grace finds Poppy
  • Page 18: ride home / feast
  • Page 20: fireplace safe at home
  • Page 22: parable thoughts
  • Page 23: lost sheep parable

Device Test Target

The synchronized playback was tested on the ESP32/Tactility device:

  • IP: 192.168.68.113
  • Dashboard: http://192.168.68.113/dashboard.html
  • MCP endpoint: POST http://192.168.68.113/api/mcp

Important discovery:

  • GET /api/mcp returned 404, but this did not mean MCP was unavailable.
  • POST /api/mcp worked and returned the available tools.
  • The device reported a 320x240 color display and supported rgb565_base64, bmp_base64, and pbm_base64.

Useful device capabilities observed:

{
  "color": true,
  "width": 320,
  "height": 240,
  "formats": ["rgb565_base64", "bmp_base64", "pbm_base64"],
  "audio": {
    "sampleRate": 16000,
    "bitsPerSample": 16,
    "channels": 1
  },
  "platform": "tactility",
  "appVersion": "0.1.0"
}

Display Path That Worked

Each 320x240 PNG was converted to big-endian RGB565 and sent through MCP:

  • MCP tool: draw_raw_rgb565
  • Arguments:
    • rgb565_base64: base64-encoded big-endian RGB565 pixel payload
    • x: 0
    • y: 0
    • w: 320
    • h: 240

Each displayed image was 153600 bytes of RGB565 payload before base64 encoding:

320 * 240 * 2 = 153600 bytes

Every page image tested successfully on .113 with:

Raw RGB565 data displayed successfully.

Audio Path That Worked

Original package narration files were MP3s:

  • /home/adolforeyna/output/grace_poppy_existing/mp3/page_XX.mp3

The speaker path was first verified with the MCP play_tone tool.

What did not work well:

  • Sending the original full-size MP3 files through play_mp3_base64 was too large for at least some page files and caused a connection reset.
  • download_file successfully downloaded an MP3 to the SD card, but play_mp3 then reported MP3 file was not found, apparently because it expects a simple app-user-data location rather than the same SD-card location used by download_file.

What worked:

  • Transcode narration MP3s to smaller files:
    • mono
    • 16 kHz
    • 24 kbps MP3
  • Send the smaller file through MCP play_mp3_base64.

Example transcode command:

ffmpeg -y -hide_banner -loglevel error \
  -i /home/adolforeyna/output/grace_poppy_existing/mp3/page_01.mp3 \
  -ac 1 -ar 16000 -b:a 24k \
  /home/adolforeyna/output/grace_poppy_existing/mp3_small/page_01_24k.mp3

The smaller Page 1 file was about 47,744 bytes and played successfully through:

  • MCP tool: play_mp3_base64
  • Argument: mp3_base64
  • Volume tested: 90

Successful response:

Successfully played base64 MP3 audio.

Synchronized Prototype Behavior

The working Hermes-coordinated prototype performs this loop:

  1. Convert page image to RGB565.
  2. Send image to the device with draw_raw_rgb565.
  3. Send matching small MP3 narration with play_mp3_base64.
  4. Wait for playback to finish.
  5. Continue to the next page.

The full set was tested on 192.168.68.113.

Tested successfully:

  • Pages 01–16: image displayed and audio played.
  • Then pages 17, 18, 20, 22, 23: image displayed and audio played.

Every tested page returned:

Raw RGB565 data displayed successfully.
Successfully played base64 MP3 audio.

The prototype ended on Page 23 after playing the lost-sheep narration.

Current Working Scripts and Artifacts

Local project folder:

  • /home/adolforeyna/output/imagegen/grace-poppy/

Useful scripts created during testing:

  • generate_remaining_pages.py — generated remaining pages via Mac mini Codex workflow.
  • test_device_113.py — tested Page 1 image display on .113.
  • slideshow_test_device_113.py — tested sending all images without audio.
  • test_page1_image_audio_113.py — tested Page 1 display + original base64 MP3; found payload reset issue.
  • test_page1_download_play_113.py — tested URL download + play_mp3; found app-data path issue.
  • run_image_audio_slideshow_113.py — first synchronized image/audio slideshow; worked but included an extra wait and timed out after Page 16.
  • continue_image_audio_slideshow_113.py — continued Pages 17, 18, 20, 22, 23 without extra wait.

Small narration files:

  • /home/adolforeyna/output/grace_poppy_existing/mp3_small/page_XX_24k.mp3

Lessons Learned

  • The Mac mini Codex CLI + imagegen path works well for deterministic image generation when the prompt explicitly tells Codex where to save the final file and -o captures the final response path.
  • Always include the Grace-purple continuity rule in generation prompts.
  • Full-size generated images should be resized to 320x240 for this ESP32/Tactility display.
  • For this board, POST /api/mcp is the real MCP test; GET /api/mcp may return 404.
  • draw_raw_rgb565 is reliable for displaying page images.
  • play_tone is a good first speaker-path test.
  • Original MP3s may be too large for JSON/base64 playback.
  • Smaller 16kHz, mono, 24kbps MP3 files worked with play_mp3_base64.
  • play_mp3_base64 appears to block until playback finishes, so the coordinator should not add a second full-duration sleep after it returns.
  • download_file and play_mp3 may not share the same file lookup location. Do not assume a downloaded SD-card file can be played by play_mp3 without validating the firmware path behavior.

Desired Next Phase

Build an app so Hermes is not responsible for coordinating every page.

The app should be able to:

  1. Store or access a complete book package containing:
    • page image assets
    • narration audio assets
    • page order
    • page metadata
    • duration/pacing settings
    • language/voice metadata
  2. Play the book locally:
    • show Page 1 image
    • play Page 1 audio
    • advance when audio finishes or after configured pacing
    • continue until the end
  3. Let Hermes trigger playback:
    • start book
    • stop book
    • pause/resume
    • jump to page
    • replay page
    • maybe select language or voice variant
  4. Allow post-generation corrections:
    • regenerate or replace an image for a page
    • edit image details while preserving character continuity
    • change narration language
    • change narration pacing
    • change voice profile
    • regenerate only affected page audio/images
    • rebuild a package manifest without regenerating everything
  5. Support future review/approval flow:
    • generate draft assets
    • preview contact sheet
    • mark pages needing correction
    • apply corrections page-by-page
    • publish a final playable package

Suggested Book Package Structure

A future app could use a folder or ZIP format like:

book.json
images/page_01.png
images/page_02.png
audio/en-US/default/page_01.mp3
audio/en-US/default/page_02.mp3
audio/es-US/default/page_01.mp3
corrections/page_01.md

Possible book.json fields:

{
  "id": "grace-poppy",
  "title": "Princess Grace and Poppy",
  "display": {
    "width": 320,
    "height": 240,
    "pixel_format": "rgb565"
  },
  "default_language": "en-US",
  "default_voice_profile": "default",
  "pages": [
    {
      "page": 1,
      "image": "images/page_01.png",
      "audio": "audio/en-US/default/page_01.mp3",
      "duration_sec": 15.8,
      "caption": "Once upon a time...",
      "status": "approved"
    }
  ]
}

Tactility ESP32 App Plan

Captured on 2026-07-08 from the next app idea: build a dedicated Tactility app that can play local book packages from the SD card, one page at a time, showing the image and playing the matching narration together on the device screen.

App Goal

Create a Tactility app, tentatively called Book Player, that runs locally on the ESP32/Tactility device instead of requiring Hermes to coordinate each page over MCP.

Core behavior:

  1. Scan a books folder on the SD card.
  2. Let the user pick a book.
  3. Display the current page image on the screen.
  4. Play the matching page audio.
  5. Provide on-device controls:
    • play/pause
    • next page
    • previous page
  6. Auto-advance to the next page when audio finishes.
  7. Keep the device safe and responsive by loading only the current page assets.

Tactility App Details for Codex Build

Implementation notes for the future Codex/Tactility build:

  • Follow existing Tactility app structure in the firmware repository rather than inventing a separate framework.
  • Look for existing examples of:
    • app registration
    • screen lifecycle
    • LVGL/UI layout
    • SD-card file access
    • image rendering
    • audio playback
  • Register the app so it appears in the Tactility launcher.
  • Use robust SD-card path handling and clear serial logs.
  • Handle these failures without crashing:
    • no SD card
    • no /sdcard/books/ folder
    • no books found
    • missing or invalid manifest
    • missing image/audio files
    • unsupported image/audio formats
    • audio playback errors
  • If true audio pause/resume is unavailable in the Tactility audio stack, implement clean stop/restart behavior and document the limitation.

Preferred SD Card Book Folder Layout

Use a parent folder containing one subfolder per book:

/sdcard/books/
  grace-poppy/
    manifest.json
    page001.png
    page001.mp3
    page002.png
    page002.mp3

  another-book/
    manifest.json
    page001.jpg
    page001.wav

Each book folder should include a manifest.json like:

{
  "title": "Princess Grace and Poppy",
  "author": "Reyna Family",
  "pages": [
    {
      "image": "page001.png",
      "audio": "page001.mp3",
      "caption": "Optional caption text"
    },
    {
      "image": "page002.png",
      "audio": "page002.mp3",
      "caption": "Optional caption text"
    }
  ]
}

For future multi-language or multi-voice versions, this can later evolve toward the richer book.json package format below, but the first app should stay simple and folder-based.

Desired Device UI

Book picker screen:

  • Title: Books
  • List of detected book titles from manifests
  • Error/empty state if no books are found

Player screen:

  • Top: book title or short title
  • Center: page image scaled to fit the screen while preserving aspect ratio
  • Bottom controls: Prev, Play/Pause, Next
  • Small page indicator such as 1 / 23

Playback Rules

  • Opening a book loads page 1.
  • Pressing Play plays the current page audio while keeping the image visible.
  • When audio finishes, advance to the next page and start playback.
  • At the final page, stop and remain on the final image.
  • Next/Previous should stop the current audio, switch pages, and resume audio if playback was active.
  • Only load the current image/audio metadata; do not load the whole book into memory.

Acceptance Criteria for the Build

The app build is successful when:

  • Firmware builds.
  • The app appears in the Tactility launcher.
  • A test book under /sdcard/books/test-book/ appears in the picker.
  • Selecting the book displays page 1.
  • Play starts page 1 narration.
  • Play/pause, next, and previous work.
  • Audio completion advances pages automatically.
  • Missing/invalid files show or log a useful error instead of crashing.
  • A README documents the SD-card folder format and includes a sample manifest.

Codex Prompt Seed

Use the following as the working direction for Codex when implementing the app:

Create a new Tactility app for ESP32 called “Book Player”. It should scan `/sdcard/books/` for subfolders with `manifest.json`, show a book picker, then play each selected book page by displaying the page image and playing the matching audio. Provide Prev, Play/Pause, and Next controls. Auto-advance when audio finishes. Follow existing Tactility app structure, UI conventions, app registration, SD-card access, image rendering, and audio playback patterns in the repository. Keep memory usage safe by loading only the current page. Handle no SD card, no books, invalid JSON, missing assets, and playback errors gracefully. Add registration/build entries, a README, and a sample manifest. Build and report changed files and limitations.

Initial Example Book Package

A ready-to-copy example package was assembled on 2026-07-08 using the current Grace/Poppy assets:

  • ZIP: /home/adolforeyna/output/tactility-book-packages/grace-poppy-tactility-book-example.zip
  • Folder inside ZIP: books/grace-poppy/
  • Manifest: books/grace-poppy/manifest.json
  • Pages: 21 sequential app pages, with original_page preserving the source page numbers where the original book skipped pages 19 and 21.
  • Images: page001.png through page021.png, 320×240 PNG.
  • Audio: page001.mp3 through page021.mp3, mono 16 kHz 24 kbps MP3.
  • Includes README.md explaining SD-card placement.

To install on the SD card, unzip so the final path is:

/sdcard/books/grace-poppy/manifest.json
/sdcard/books/grace-poppy/page001.png
/sdcard/books/grace-poppy/page001.mp3
...

Package verification at creation:

unzip -t: No errors detected
manifest pages: 21
sha256: e84a5ecdcd44235ab44775aee3867fb71d2f1cd0e1be439002a640d922cd3ed9

Page 1 Audio Improvement Trial

Captured on 2026-07-08: Page 1 narration felt too plain, with phrases gluing together and not enough emotion. A comparison trial was generated using performance text with warmer storytelling direction, clearer pauses, and more deliberate pacing.

Performance direction used:

  • Warm bedtime-story opening.
  • Slightly magical/hushed tone for “Once upon a time.”
  • Wider pause before the castle setting.
  • Gentle affection when introducing the father/king.
  • Clear pauses between the princess names.
  • Soft concluding tone for “They were blessed...”

Artifacts:

  • Folder: /home/adolforeyna/output/tactility-book-packages/page1-audio-comparison/
  • Comparison ZIP: /home/adolforeyna/output/tactility-book-packages/page1-audio-comparison/page001-audio-comparison.zip
  • Adolfo segmented version: page001_adolfo_segmented_24k.mp3, 20.628s, SHA256 42dff81746ea1a708826a9f87c8654ee200e2669faed5956e338a10c6b72a2d1
  • Aiden version: page001_aiden_24k.mp3, 26.100s, SHA256 25b88aaf438ed23abcbff63052a3937a9bd959384ee0fde3d0a9bdf79f54b632
  • Original package Page 1 was 15.840s, so both trials are slower and should reduce phrase-gluing.

If one version is approved, replace only books/grace-poppy/page001.mp3 in the Tactility package first, then regenerate the package ZIP.

Correction Workflow Ideas

Corrections should be recorded as structured page-level notes rather than ad hoc prompts.

Examples:

page: 1
correction_type: image
request: "Make Grace's dress more lavender and make the castle slightly smaller."
constraints:
  - Grace must remain the purple/lavender princess.
  - No visible text, logos, or watermark.
  - Keep 4:3 storybook composition.

page: 7
correction_type: voice
request: "Slow this page down and use a warmer dad-reading voice."
language: en-US
voice_profile: adolfo-warm
pacing: slower

Possible correction types:

  • image_detail
  • image_style
  • image_continuity
  • language_translation
  • voice_profile
  • voice_pacing
  • page_order
  • text/narration

Open Design Questions

  • Should the playable app live as a Tactility app on the ESP32 device, or should it be a Hermes-side app that streams commands to the board?
  • Should full assets be stored on the SD card, or should Hermes serve them on demand?
  • Can the firmware expose a reliable app-local audio file path so download_file + play_mp3 works without base64 payloads?
  • Should generated packages include both source-quality images and device-optimized 320x240 assets?
  • Should narration be pre-generated for each voice/language, or generated on demand when a correction is requested?
  • How should children/family members approve or reject generated page variants?

Current Status

Working prototype completed.

  • Generated images are available.
  • Audio was extracted and transcoded successfully.
  • Full image-only slideshow works on .113.
  • Full image + audio synchronized playback works on .113 when using smaller base64 MP3 files.

No implementation changes should be made from this note alone. This is documentation and planning context for the next build phase.

Reference in New Issue View Git Blame Copy Permalink