- [[Livestream and Video Generation Process]]: Document current livestream/video workflow and prepare for media producer advice.
- [[EMI Livestream and Media Workflow Report for Ari / Sardius]]: Shareable first-draft brief focused on workflow architecture, translation, archive/search, and AI-assisted media production.
- [[reMarkable Hermes Communication App]]: Tablet/Hermes communication app over reMarkable Paper Pro.
- [[Grace Poppy Storybook Image Voice App]]: Document the Grace/Poppy image-generation and narration playback workflow, with next steps for a triggerable app and correction pipeline.
- [[PARA Brain MCP Server]]: Server-side MCP/search layer for the Reyna family brain.
- [[Reyna CLI Browser Control]]: Add remote browser-control automations, starting with Gemini Web image generation, into `reyna-cli`.
- **[[Areas]]:** Ongoing responsibilities and household maintenance.
- **[[Resources]]:** Interests, reference material, and research.
This project documents the workflow used to turn a family storybook package for Princess Grace and Poppy into a synchronized image-and-audio experience on an ESP32/Tactility color screen device. The current working prototype is coordinated by Hermes: Hermes displays each generated image on the device, plays the matching narration audio, waits for playback, then advances to the next page.
The next project phase is to turn this into an app that runs the playback flow itself, so Hermes can trigger it and the device can auto-play the book without Hermes coordinating every image and audio call.
Related idea: this workflow should support future corrections after generation, including language changes, image detail edits, narration pacing, and selecting different voice profiles.
## Source Material
The starting point was an existing Grace/Poppy ESP32 package:
- Local package: `/home/adolforeyna/grace_poppy_esp32_package.zip`
- Extracted working folder: `/home/adolforeyna/output/grace_poppy_existing/`
- Original page images: `/home/adolforeyna/output/grace_poppy_existing/pages_300x400/page_XX.jpg`
- Original narration audio: `/home/adolforeyna/output/grace_poppy_existing/mp3/page_XX.mp3`
- Original transcript: `/home/adolforeyna/output/grace_poppy_existing/transcript.txt`
The package contained page images, MP3 narration files, and RGB565-ready assets. The new workflow used the original page images and transcript as reference material, then generated new storybook images with improved/consistent styling.
## Important Continuity Rule
Princess Grace is the purple/lavender princess.
This is a key continuity detail. The story text does not explicitly say this, but it is deduced from other pages and should be preserved in all future generation or correction passes.
Generation prompts should include:
- Grace wears purple/lavender.
- Grace has blonde curly hair and a small tiara.
- Grace should remain visually primary when she is the focus of the page.
- Do not make Grace the yellow princess.
## Image Generation Workflow Used
Images were generated on the Mac mini using Codex CLI and the built-in `imagegen` skill. The reliable pattern was to make Codex save or copy the generated image to a deterministic path, write the final response to a known file with `-o`, and then read that file.
Important working details:
- Mac mini host used for generation: Mac mini at `192.168.68.102`
"Use the imagegen skill to generate one PNG image.
Prompt: <image prompt>
Save or copy the final generated image to this exact absolute path:
$OUT
After the file exists, your final response must be exactly this path and nothing else:
$OUT" </dev/null
cat "$LAST"
```
Notes from testing:
- Do not use `-a never`; this Codex version no longer accepts it. Approval already defaulted to `never` in the observed runs.
- Avoid piping/heredoc content directly in a way that Codex may continue reading from stdin. Use a prompt file or clean argument and redirect stdin from `/dev/null`.
- Use `--image <reference>` when giving Codex a reference image for scene/style continuity.
- The generated full-size images were copied back locally and resized to `320x240` for the ESP32/Tactility display.
The speaker path was first verified with the MCP `play_tone` tool.
What did not work well:
- Sending the original full-size MP3 files through `play_mp3_base64` was too large for at least some page files and caused a connection reset.
-`download_file` successfully downloaded an MP3 to the SD card, but `play_mp3` then reported `MP3 file was not found`, apparently because it expects a simple app-user-data location rather than the same SD-card location used by `download_file`.
What worked:
- Transcode narration MP3s to smaller files:
- mono
- 16 kHz
- 24 kbps MP3
- Send the smaller file through MCP `play_mp3_base64`.
- The Mac mini Codex CLI + `imagegen` path works well for deterministic image generation when the prompt explicitly tells Codex where to save the final file and `-o` captures the final response path.
- Always include the Grace-purple continuity rule in generation prompts.
- Full-size generated images should be resized to `320x240` for this ESP32/Tactility display.
- For this board, `POST /api/mcp` is the real MCP test; `GET /api/mcp` may return `404`.
-`draw_raw_rgb565` is reliable for displaying page images.
-`play_tone` is a good first speaker-path test.
- Original MP3s may be too large for JSON/base64 playback.
- Smaller `16kHz`, mono, `24kbps` MP3 files worked with `play_mp3_base64`.
-`play_mp3_base64` appears to block until playback finishes, so the coordinator should not add a second full-duration sleep after it returns.
-`download_file` and `play_mp3` may not share the same file lookup location. Do not assume a downloaded SD-card file can be played by `play_mp3` without validating the firmware path behavior.
## Desired Next Phase
Build an app so Hermes is not responsible for coordinating every page.
The app should be able to:
1. Store or access a complete book package containing:
- page image assets
- narration audio assets
- page order
- page metadata
- duration/pacing settings
- language/voice metadata
2. Play the book locally:
- show Page 1 image
- play Page 1 audio
- advance when audio finishes or after configured pacing
- continue until the end
3. Let Hermes trigger playback:
- start book
- stop book
- pause/resume
- jump to page
- replay page
- maybe select language or voice variant
4. Allow post-generation corrections:
- regenerate or replace an image for a page
- edit image details while preserving character continuity
- change narration language
- change narration pacing
- change voice profile
- regenerate only affected page audio/images
- rebuild a package manifest without regenerating everything
5. Support future review/approval flow:
- generate draft assets
- preview contact sheet
- mark pages needing correction
- apply corrections page-by-page
- publish a final playable package
## Suggested Book Package Structure
A future app could use a folder or ZIP format like:
```text
book.json
images/page_01.png
images/page_02.png
audio/en-US/default/page_01.mp3
audio/en-US/default/page_02.mp3
audio/es-US/default/page_01.mp3
corrections/page_01.md
```
Possible `book.json` fields:
```json
{
"id":"grace-poppy",
"title":"Princess Grace and Poppy",
"display":{
"width":320,
"height":240,
"pixel_format":"rgb565"
},
"default_language":"en-US",
"default_voice_profile":"default",
"pages":[
{
"page":1,
"image":"images/page_01.png",
"audio":"audio/en-US/default/page_01.mp3",
"duration_sec":15.8,
"caption":"Once upon a time...",
"status":"approved"
}
]
}
```
## Correction Workflow Ideas
Corrections should be recorded as structured page-level notes rather than ad hoc prompts.
Examples:
```yaml
page:1
correction_type:image
request:"Make Grace's dress more lavender and make the castle slightly smaller."
constraints:
- Grace must remain the purple/lavender princess.
- Novisible text, logos, or watermark.
- Keep 4:3 storybook composition.
```
```yaml
page:7
correction_type:voice
request:"Slow this page down and use a warmer dad-reading voice."
language:en-US
voice_profile:adolfo-warm
pacing:slower
```
Possible correction types:
-`image_detail`
-`image_style`
-`image_continuity`
-`language_translation`
-`voice_profile`
-`voice_pacing`
-`page_order`
-`text/narration`
## Open Design Questions
- Should the playable app live as a Tactility app on the ESP32 device, or should it be a Hermes-side app that streams commands to the board?
- Should full assets be stored on the SD card, or should Hermes serve them on demand?
- Can the firmware expose a reliable app-local audio file path so `download_file` + `play_mp3` works without base64 payloads?
- Should generated packages include both source-quality images and device-optimized `320x240` assets?
- Should narration be pre-generated for each voice/language, or generated on demand when a correction is requested?
- How should children/family members approve or reject generated page variants?
## Current Status
Working prototype completed.
- Generated images are available.
- Audio was extracted and transcoded successfully.
- Full image-only slideshow works on `.113`.
- Full image + audio synchronized playback works on `.113` when using smaller base64 MP3 files.
No implementation changes should be made from this note alone. This is documentation and planning context for the next build phase.
Add the remote browser-control workflow into `reyna-cli` so browser automations can become reusable CLI subcommands/scripts instead of one-off Hermes procedures.
## Initial Use Case
Gemini Web image generation through the iMac's headed Chrome browser:
- SSH target: `ssh imac`
- Remote Chrome CDP: `localhost:9222` on the iMac
- Local tunnel: `localhost:9222 -> imac:localhost:9222`
- Gemini URL: `https://gemini.google.com/app`
- Current prototype script lives in Hermes skill support files:
Create a `reyna-cli` browser-control namespace that can host reusable scripts, starting with Gemini image generation.
Potential command shape:
```bash
reyna-cli browser gemini-image \
--prompt "Create a square image of ..."\
--out ~/generated-images/gemini-web/output.png
```
or:
```bash
reyna-cli image gemini-web \
--prompt "Create a square image of ..."\
--out ~/generated-images/gemini-web/output.png
```
## Automation Requirements
The command should automate:
1. Ensure SSH host key works for `imac`.
2. Ensure headed Chrome with `--remote-debugging-port=9222` is running on the iMac.
3. Open/reuse an SSH tunnel to Chrome CDP.
4. Connect via Playwright CDP.
5. Open Gemini Web.
6. Start a new chat unless `--reuse-chat` is passed.
7. Submit an image prompt.
8. Wait for a real generated image, filtering out avatars/Gemini sparkle images.
9. Extract blob-backed images by drawing to canvas and exporting PNG.
10. Save the image to a deterministic local path.
11. Print machine-readable output with path, bytes, and dimensions.
## Speed Improvements to Add Later
- Keep Chrome CDP running on the iMac at boot/login.
- Keep a persistent SSH tunnel/service instead of opening a new tunnel per image.
- Default output folder: `~/generated-images/gemini-web/` with timestamped filenames.
- Add `--json` output for agent/tool consumption.
- Add batch mode: prompt file or multiple prompts.
- Add optional screenshot/debug artifacts on failure.
- Reuse the existing `remote-browser-control` Hermes skill as implementation notes.
## Related Notes
- [[Projects]]
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
aeroreyna