---
Date: 2026-07-07
Author: Hermes
Tags: [project, family, kids, storybook, esp32, image-generation, voice, codex, tactility]
---

# Grace/Poppy Storybook Image + Voice App

## Summary

This project documents the workflow used to turn a family storybook package for Princess Grace and Poppy into a synchronized image-and-audio experience on an ESP32/Tactility color screen device. The current working prototype is coordinated by Hermes: Hermes displays each generated image on the device, plays the matching narration audio, waits for playback, then advances to the next page.

The next project phase is to turn this into an app that runs the playback flow itself, so Hermes can trigger it and the device can auto-play the book without Hermes coordinating every image and audio call.

Related idea: this workflow should support future corrections after generation, including language changes, image detail edits, narration pacing, and selecting different voice profiles.

## Source Material

The starting point was an existing Grace/Poppy ESP32 package:

- Local package: `/home/adolforeyna/grace_poppy_esp32_package.zip`
- Extracted working folder: `/home/adolforeyna/output/grace_poppy_existing/`
- Original page images: `/home/adolforeyna/output/grace_poppy_existing/pages_300x400/page_XX.jpg`
- Original narration audio: `/home/adolforeyna/output/grace_poppy_existing/mp3/page_XX.mp3`
- Original transcript: `/home/adolforeyna/output/grace_poppy_existing/transcript.txt`

The package contained page images, MP3 narration files, and RGB565-ready assets. The new workflow used the original page images and transcript as reference material, then generated new storybook images with improved/consistent styling.

## Important Continuity Rule

Princess Grace is the purple/lavender princess.

This is a key continuity detail. The story text does not explicitly say this, but it is deduced from other pages and should be preserved in all future generation or correction passes.

Generation prompts should include:

- Grace wears purple/lavender.
- Grace has blonde curly hair and a small tiara.
- Grace should remain visually primary when she is the focus of the page.
- Do not make Grace the yellow princess.

## Image Generation Workflow Used

Images were generated on the Mac mini using Codex CLI and the built-in `imagegen` skill. The reliable pattern was to make Codex save or copy the generated image to a deterministic path, write the final response to a known file with `-o`, and then read that file.

Important working details:

- Mac mini host used for generation: Mac mini at `192.168.68.102`
- Codex CLI path observed: `/Users/adolforeyna/.local/bin/codex`
- Codex version observed: `codex-cli 0.142.5`
- Mac mini output folder: `/Users/adolforeyna/output/imagegen/grace-poppy/`
- Local output folder: `/home/adolforeyna/output/imagegen/grace-poppy/`

The validated Codex pattern:

```bash
OUT="$HOME/output/imagegen/<project>/<image-name>-$(date +%Y%m%d-%H%M%S).png"
LAST="/tmp/codex-imagegen-last.txt"
mkdir -p "$(dirname "$OUT")"

codex exec \
  --skip-git-repo-check \
  -C "$HOME" \
  -s workspace-write \
  -o "$LAST" \
  "Use the imagegen skill to generate one PNG image.

Prompt: <image prompt>

Save or copy the final generated image to this exact absolute path:
$OUT

After the file exists, your final response must be exactly this path and nothing else:
$OUT" </dev/null

cat "$LAST"
```

Notes from testing:

- Do not use `-a never`; this Codex version no longer accepts it. Approval already defaulted to `never` in the observed runs.
- Avoid piping/heredoc content directly in a way that Codex may continue reading from stdin. Use a prompt file or clean argument and redirect stdin from `/dev/null`.
- Use `--image <reference>` when giving Codex a reference image for scene/style continuity.
- The generated full-size images were copied back locally and resized to `320x240` for the ESP32/Tactility display.

## Generated Image Set

Corrected/generated ESP32-ready images are in:

- `/home/adolforeyna/output/imagegen/grace-poppy/`

Final ZIP for the generated ESP32 images:

- `/home/adolforeyna/output/imagegen/grace-poppy/grace-poppy-generated-all-esp32-320x240.zip`

Preview contact sheet:

- `/home/adolforeyna/output/imagegen/grace-poppy/grace-poppy-contact-sheet.png`

Generated pages tested:

- Page 01: castle garden, Grace corrected to purple/lavender
- Page 02: adventurous Grace in castle garden
- Page 03: hallway/vase/closet
- Page 04: five kittens
- Page 05: family finds kittens
- Page 06: kitten on dining table
- Page 07: Grace promises to care for Poppy
- Page 08: princesses play with kittens
- Page 09: Poppy slides on hallway floor
- Page 10: Poppy missing
- Page 11: Grace comforted by the king
- Page 12: king sends search party
- Page 13: countryside search
- Page 14: Black Woods entrance
- Page 15: Black Woods search
- Page 16: Grace prays
- Page 17: Grace finds Poppy
- Page 18: ride home / feast
- Page 20: fireplace safe at home
- Page 22: parable thoughts
- Page 23: lost sheep parable

## Device Test Target

The synchronized playback was tested on the ESP32/Tactility device:

- IP: `192.168.68.113`
- Dashboard: `http://192.168.68.113/dashboard.html`
- MCP endpoint: `POST http://192.168.68.113/api/mcp`

Important discovery:

- `GET /api/mcp` returned `404`, but this did not mean MCP was unavailable.
- `POST /api/mcp` worked and returned the available tools.
- The device reported a `320x240` color display and supported `rgb565_base64`, `bmp_base64`, and `pbm_base64`.

Useful device capabilities observed:

```json
{
  "color": true,
  "width": 320,
  "height": 240,
  "formats": ["rgb565_base64", "bmp_base64", "pbm_base64"],
  "audio": {
    "sampleRate": 16000,
    "bitsPerSample": 16,
    "channels": 1
  },
  "platform": "tactility",
  "appVersion": "0.1.0"
}
```

## Display Path That Worked

Each `320x240` PNG was converted to big-endian RGB565 and sent through MCP:

- MCP tool: `draw_raw_rgb565`
- Arguments:
  - `rgb565_base64`: base64-encoded big-endian RGB565 pixel payload
  - `x`: `0`
  - `y`: `0`
  - `w`: `320`
  - `h`: `240`

Each displayed image was `153600` bytes of RGB565 payload before base64 encoding:

```text
320 * 240 * 2 = 153600 bytes
```

Every page image tested successfully on `.113` with:

```text
Raw RGB565 data displayed successfully.
```

## Audio Path That Worked

Original package narration files were MP3s:

- `/home/adolforeyna/output/grace_poppy_existing/mp3/page_XX.mp3`

The speaker path was first verified with the MCP `play_tone` tool.

What did not work well:

- Sending the original full-size MP3 files through `play_mp3_base64` was too large for at least some page files and caused a connection reset.
- `download_file` successfully downloaded an MP3 to the SD card, but `play_mp3` then reported `MP3 file was not found`, apparently because it expects a simple app-user-data location rather than the same SD-card location used by `download_file`.

What worked:

- Transcode narration MP3s to smaller files:
  - mono
  - 16 kHz
  - 24 kbps MP3
- Send the smaller file through MCP `play_mp3_base64`.

Example transcode command:

```bash
ffmpeg -y -hide_banner -loglevel error \
  -i /home/adolforeyna/output/grace_poppy_existing/mp3/page_01.mp3 \
  -ac 1 -ar 16000 -b:a 24k \
  /home/adolforeyna/output/grace_poppy_existing/mp3_small/page_01_24k.mp3
```

The smaller Page 1 file was about `47,744` bytes and played successfully through:

- MCP tool: `play_mp3_base64`
- Argument: `mp3_base64`
- Volume tested: `90`

Successful response:

```text
Successfully played base64 MP3 audio.
```

## Synchronized Prototype Behavior

The working Hermes-coordinated prototype performs this loop:

1. Convert page image to RGB565.
2. Send image to the device with `draw_raw_rgb565`.
3. Send matching small MP3 narration with `play_mp3_base64`.
4. Wait for playback to finish.
5. Continue to the next page.

The full set was tested on `192.168.68.113`.

Tested successfully:

- Pages `01–16`: image displayed and audio played.
- Then pages `17`, `18`, `20`, `22`, `23`: image displayed and audio played.

Every tested page returned:

```text
Raw RGB565 data displayed successfully.
Successfully played base64 MP3 audio.
```

The prototype ended on Page 23 after playing the lost-sheep narration.

## Current Working Scripts and Artifacts

Local project folder:

- `/home/adolforeyna/output/imagegen/grace-poppy/`

Useful scripts created during testing:

- `generate_remaining_pages.py` — generated remaining pages via Mac mini Codex workflow.
- `test_device_113.py` — tested Page 1 image display on `.113`.
- `slideshow_test_device_113.py` — tested sending all images without audio.
- `test_page1_image_audio_113.py` — tested Page 1 display + original base64 MP3; found payload reset issue.
- `test_page1_download_play_113.py` — tested URL download + `play_mp3`; found app-data path issue.
- `run_image_audio_slideshow_113.py` — first synchronized image/audio slideshow; worked but included an extra wait and timed out after Page 16.
- `continue_image_audio_slideshow_113.py` — continued Pages 17, 18, 20, 22, 23 without extra wait.

Small narration files:

- `/home/adolforeyna/output/grace_poppy_existing/mp3_small/page_XX_24k.mp3`

## Lessons Learned

- The Mac mini Codex CLI + `imagegen` path works well for deterministic image generation when the prompt explicitly tells Codex where to save the final file and `-o` captures the final response path.
- Always include the Grace-purple continuity rule in generation prompts.
- Full-size generated images should be resized to `320x240` for this ESP32/Tactility display.
- For this board, `POST /api/mcp` is the real MCP test; `GET /api/mcp` may return `404`.
- `draw_raw_rgb565` is reliable for displaying page images.
- `play_tone` is a good first speaker-path test.
- Original MP3s may be too large for JSON/base64 playback.
- Smaller `16kHz`, mono, `24kbps` MP3 files worked with `play_mp3_base64`.
- `play_mp3_base64` appears to block until playback finishes, so the coordinator should not add a second full-duration sleep after it returns.
- `download_file` and `play_mp3` may not share the same file lookup location. Do not assume a downloaded SD-card file can be played by `play_mp3` without validating the firmware path behavior.

## Desired Next Phase

Build an app so Hermes is not responsible for coordinating every page.

The app should be able to:

1. Store or access a complete book package containing:
   - page image assets
   - narration audio assets
   - page order
   - page metadata
   - duration/pacing settings
   - language/voice metadata
2. Play the book locally:
   - show Page 1 image
   - play Page 1 audio
   - advance when audio finishes or after configured pacing
   - continue until the end
3. Let Hermes trigger playback:
   - start book
   - stop book
   - pause/resume
   - jump to page
   - replay page
   - maybe select language or voice variant
4. Allow post-generation corrections:
   - regenerate or replace an image for a page
   - edit image details while preserving character continuity
   - change narration language
   - change narration pacing
   - change voice profile
   - regenerate only affected page audio/images
   - rebuild a package manifest without regenerating everything
5. Support future review/approval flow:
   - generate draft assets
   - preview contact sheet
   - mark pages needing correction
   - apply corrections page-by-page
   - publish a final playable package

## Suggested Book Package Structure

A future app could use a folder or ZIP format like:

```text
book.json
images/page_01.png
images/page_02.png
audio/en-US/default/page_01.mp3
audio/en-US/default/page_02.mp3
audio/es-US/default/page_01.mp3
corrections/page_01.md
```

Possible `book.json` fields:

```json
{
  "id": "grace-poppy",
  "title": "Princess Grace and Poppy",
  "display": {
    "width": 320,
    "height": 240,
    "pixel_format": "rgb565"
  },
  "default_language": "en-US",
  "default_voice_profile": "default",
  "pages": [
    {
      "page": 1,
      "image": "images/page_01.png",
      "audio": "audio/en-US/default/page_01.mp3",
      "duration_sec": 15.8,
      "caption": "Once upon a time...",
      "status": "approved"
    }
  ]
}
```

## Tactility ESP32 App Plan

Captured on 2026-07-08 from the next app idea: build a dedicated Tactility app that can play local book packages from the SD card, one page at a time, showing the image and playing the matching narration together on the device screen.

### App Goal

Create a Tactility app, tentatively called **Book Player**, that runs locally on the ESP32/Tactility device instead of requiring Hermes to coordinate each page over MCP.

Core behavior:

1. Scan a books folder on the SD card.
2. Let the user pick a book.
3. Display the current page image on the screen.
4. Play the matching page audio.
5. Provide on-device controls:
   - play/pause
   - next page
   - previous page
6. Auto-advance to the next page when audio finishes.
7. Keep the device safe and responsive by loading only the current page assets.

### Tactility App Details for Codex Build

Implementation notes for the future Codex/Tactility build:

- Follow existing Tactility app structure in the firmware repository rather than inventing a separate framework.
- Look for existing examples of:
  - app registration
  - screen lifecycle
  - LVGL/UI layout
  - SD-card file access
  - image rendering
  - audio playback
- Register the app so it appears in the Tactility launcher.
- Use robust SD-card path handling and clear serial logs.
- Handle these failures without crashing:
  - no SD card
  - no `/sdcard/books/` folder
  - no books found
  - missing or invalid manifest
  - missing image/audio files
  - unsupported image/audio formats
  - audio playback errors
- If true audio pause/resume is unavailable in the Tactility audio stack, implement clean stop/restart behavior and document the limitation.

### Preferred SD Card Book Folder Layout

Use a parent folder containing one subfolder per book:

```text
/sdcard/books/
  grace-poppy/
    manifest.json
    page001.png
    page001.mp3
    page002.png
    page002.mp3

  another-book/
    manifest.json
    page001.jpg
    page001.wav
```

Each book folder should include a `manifest.json` like:

```json
{
  "title": "Princess Grace and Poppy",
  "author": "Reyna Family",
  "pages": [
    {
      "image": "page001.png",
      "audio": "page001.mp3",
      "caption": "Optional caption text"
    },
    {
      "image": "page002.png",
      "audio": "page002.mp3",
      "caption": "Optional caption text"
    }
  ]
}
```

For future multi-language or multi-voice versions, this can later evolve toward the richer `book.json` package format below, but the first app should stay simple and folder-based.

### Desired Device UI

Book picker screen:

- Title: `Books`
- List of detected book titles from manifests
- Error/empty state if no books are found

Player screen:

- Top: book title or short title
- Center: page image scaled to fit the screen while preserving aspect ratio
- Bottom controls: `Prev`, `Play/Pause`, `Next`
- Small page indicator such as `1 / 23`

### Playback Rules

- Opening a book loads page 1.
- Pressing Play plays the current page audio while keeping the image visible.
- When audio finishes, advance to the next page and start playback.
- At the final page, stop and remain on the final image.
- Next/Previous should stop the current audio, switch pages, and resume audio if playback was active.
- Only load the current image/audio metadata; do not load the whole book into memory.

### Acceptance Criteria for the Build

The app build is successful when:

- Firmware builds.
- The app appears in the Tactility launcher.
- A test book under `/sdcard/books/test-book/` appears in the picker.
- Selecting the book displays page 1.
- Play starts page 1 narration.
- Play/pause, next, and previous work.
- Audio completion advances pages automatically.
- Missing/invalid files show or log a useful error instead of crashing.
- A README documents the SD-card folder format and includes a sample manifest.

### Codex Prompt Seed

Use the following as the working direction for Codex when implementing the app:

```text
Create a new Tactility app for ESP32 called “Book Player”. It should scan `/sdcard/books/` for subfolders with `manifest.json`, show a book picker, then play each selected book page by displaying the page image and playing the matching audio. Provide Prev, Play/Pause, and Next controls. Auto-advance when audio finishes. Follow existing Tactility app structure, UI conventions, app registration, SD-card access, image rendering, and audio playback patterns in the repository. Keep memory usage safe by loading only the current page. Handle no SD card, no books, invalid JSON, missing assets, and playback errors gracefully. Add registration/build entries, a README, and a sample manifest. Build and report changed files and limitations.
```

### Initial Example Book Package

A ready-to-copy example package was assembled on 2026-07-08 using the current Grace/Poppy assets:

- ZIP: `/home/adolforeyna/output/tactility-book-packages/grace-poppy-tactility-book-example.zip`
- Folder inside ZIP: `books/grace-poppy/`
- Manifest: `books/grace-poppy/manifest.json`
- Pages: 21 sequential app pages, with `original_page` preserving the source page numbers where the original book skipped pages 19 and 21.
- Images: `page001.png` through `page021.png`, 320×240 PNG.
- Audio: `page001.mp3` through `page021.mp3`, mono 16 kHz 24 kbps MP3.
- Includes `README.md` explaining SD-card placement.

To install on the SD card, unzip so the final path is:

```text
/sdcard/books/grace-poppy/manifest.json
/sdcard/books/grace-poppy/page001.png
/sdcard/books/grace-poppy/page001.mp3
...
```

Package verification at creation:

```text
unzip -t: No errors detected
manifest pages: 21
sha256: e84a5ecdcd44235ab44775aee3867fb71d2f1cd0e1be439002a640d922cd3ed9
```

### Page 1 Audio Improvement Trial

Captured on 2026-07-08: Page 1 narration felt too plain, with phrases gluing together and not enough emotion. A comparison trial was generated using performance text with warmer storytelling direction, clearer pauses, and more deliberate pacing.

Performance direction used:

- Warm bedtime-story opening.
- Slightly magical/hushed tone for “Once upon a time.”
- Wider pause before the castle setting.
- Gentle affection when introducing the father/king.
- Clear pauses between the princess names.
- Soft concluding tone for “They were blessed...”

Artifacts:

- Folder: `/home/adolforeyna/output/tactility-book-packages/page1-audio-comparison/`
- Comparison ZIP: `/home/adolforeyna/output/tactility-book-packages/page1-audio-comparison/page001-audio-comparison.zip`
- Adolfo segmented version: `page001_adolfo_segmented_24k.mp3`, 20.628s, SHA256 `42dff81746ea1a708826a9f87c8654ee200e2669faed5956e338a10c6b72a2d1`
- Aiden version: `page001_aiden_24k.mp3`, 26.100s, SHA256 `25b88aaf438ed23abcbff63052a3937a9bd959384ee0fde3d0a9bdf79f54b632`
- Original package Page 1 was 15.840s, so both trials are slower and should reduce phrase-gluing.

If one version is approved, replace only `books/grace-poppy/page001.mp3` in the Tactility package first, then regenerate the package ZIP.

### Full Aiden Audio Regeneration

Captured on 2026-07-09 after review: the second male voice, Voicebox profile `Aiden`, sounded better than the Adolfo cloned voice for this storybook narration. All 21 app pages were regenerated with Aiden using paced performance text: line breaks, ellipses, dialogue separation, and slower emotional phrasing to avoid glued-together sentences.

Artifacts:

- Final ZIP: `/home/adolforeyna/output/tactility-book-packages/grace-poppy-tactility-book-aiden-audio.zip`
- Working folder: `/home/adolforeyna/output/tactility-book-packages/grace-poppy-aiden-full-audio/`
- Performance text: `/home/adolforeyna/output/tactility-book-packages/grace-poppy-aiden-full-audio/performance_texts.json`
- Generated MP3 folder: `/home/adolforeyna/output/tactility-book-packages/grace-poppy-aiden-full-audio/mp3_24k/`
- Regeneration script: `/home/adolforeyna/output/tactility-book-packages/regenerate_aiden_book_audio.py`

Package details:

- Pages: 21
- Voice: Aiden
- Audio format: MP3, mono, 16 kHz, 24 kbps
- Total narration duration: 515.268 seconds
- ZIP entries: 45 files, including `AUDIO_REGENERATION.md`
- ZIP SHA256: `18dee4feffa2184fb03c7e459d63c372e81a1a3e0b09c6184c4b269c5f2f3444`
- Verification: `unzip -t` reported no errors.

The package manifest now includes `audio_voice: "Aiden"` and per-page `audio_duration_sec` values. The source package folder under `/home/adolforeyna/output/tactility-book-packages/grace-poppy-tactility-book-example/books/grace-poppy/` was also updated with the Aiden audio files and regenerated manifest.

## Correction Workflow Ideas

Corrections should be recorded as structured page-level notes rather than ad hoc prompts.

Examples:

```yaml
page: 1
correction_type: image
request: "Make Grace's dress more lavender and make the castle slightly smaller."
constraints:
  - Grace must remain the purple/lavender princess.
  - No visible text, logos, or watermark.
  - Keep 4:3 storybook composition.
```

```yaml
page: 7
correction_type: voice
request: "Slow this page down and use a warmer dad-reading voice."
language: en-US
voice_profile: adolfo-warm
pacing: slower
```

Possible correction types:

- `image_detail`
- `image_style`
- `image_continuity`
- `language_translation`
- `voice_profile`
- `voice_pacing`
- `page_order`
- `text/narration`

## Open Design Questions

- Should the playable app live as a Tactility app on the ESP32 device, or should it be a Hermes-side app that streams commands to the board?
- Should full assets be stored on the SD card, or should Hermes serve them on demand?
- Can the firmware expose a reliable app-local audio file path so `download_file` + `play_mp3` works without base64 payloads?
- Should generated packages include both source-quality images and device-optimized `320x240` assets?
- Should narration be pre-generated for each voice/language, or generated on demand when a correction is requested?
- How should children/family members approve or reject generated page variants?

## Current Status

Working prototype completed.

- Generated images are available.
- Audio was extracted and transcoded successfully.
- Full image-only slideshow works on `.113`.
- Full image + audio synchronized playback works on `.113` when using smaller base64 MP3 files.

No implementation changes should be made from this note alone. This is documentation and planning context for the next build phase.
