From b84341063cee954fc2216c6936ee5c14a8b9a1c0 Mon Sep 17 00:00:00 2001 From: aeroreyna Date: Wed, 8 Jul 2026 21:56:32 -0400 Subject: [PATCH] Update Grace Poppy book player app plan --- .../grace_poppy_storybook_image_voice_app.md | 133 ++++++++++++++++++ 1 file changed, 133 insertions(+) diff --git a/projects/grace_poppy_storybook_image_voice_app.md b/projects/grace_poppy_storybook_image_voice_app.md index 4d04646..5500dcd 100644 --- a/projects/grace_poppy_storybook_image_voice_app.md +++ b/projects/grace_poppy_storybook_image_voice_app.md @@ -358,6 +358,139 @@ Possible `book.json` fields: } ``` +## Tactility ESP32 App Plan + +Captured on 2026-07-08 from the next app idea: build a dedicated Tactility app that can play local book packages from the SD card, one page at a time, showing the image and playing the matching narration together on the device screen. + +### App Goal + +Create a Tactility app, tentatively called **Book Player**, that runs locally on the ESP32/Tactility device instead of requiring Hermes to coordinate each page over MCP. + +Core behavior: + +1. Scan a books folder on the SD card. +2. Let the user pick a book. +3. Display the current page image on the screen. +4. Play the matching page audio. +5. Provide on-device controls: + - play/pause + - next page + - previous page +6. Auto-advance to the next page when audio finishes. +7. Keep the device safe and responsive by loading only the current page assets. + +### Tactility App Details for Codex Build + +Implementation notes for the future Codex/Tactility build: + +- Follow existing Tactility app structure in the firmware repository rather than inventing a separate framework. +- Look for existing examples of: + - app registration + - screen lifecycle + - LVGL/UI layout + - SD-card file access + - image rendering + - audio playback +- Register the app so it appears in the Tactility launcher. +- Use robust SD-card path handling and clear serial logs. +- Handle these failures without crashing: + - no SD card + - no `/sdcard/books/` folder + - no books found + - missing or invalid manifest + - missing image/audio files + - unsupported image/audio formats + - audio playback errors +- If true audio pause/resume is unavailable in the Tactility audio stack, implement clean stop/restart behavior and document the limitation. + +### Preferred SD Card Book Folder Layout + +Use a parent folder containing one subfolder per book: + +```text +/sdcard/books/ + grace-poppy/ + manifest.json + page001.png + page001.mp3 + page002.png + page002.mp3 + + another-book/ + manifest.json + page001.jpg + page001.wav +``` + +Each book folder should include a `manifest.json` like: + +```json +{ + "title": "Princess Grace and Poppy", + "author": "Reyna Family", + "pages": [ + { + "image": "page001.png", + "audio": "page001.mp3", + "caption": "Optional caption text" + }, + { + "image": "page002.png", + "audio": "page002.mp3", + "caption": "Optional caption text" + } + ] +} +``` + +For future multi-language or multi-voice versions, this can later evolve toward the richer `book.json` package format below, but the first app should stay simple and folder-based. + +### Desired Device UI + +Book picker screen: + +- Title: `Books` +- List of detected book titles from manifests +- Error/empty state if no books are found + +Player screen: + +- Top: book title or short title +- Center: page image scaled to fit the screen while preserving aspect ratio +- Bottom controls: `Prev`, `Play/Pause`, `Next` +- Small page indicator such as `1 / 23` + +### Playback Rules + +- Opening a book loads page 1. +- Pressing Play plays the current page audio while keeping the image visible. +- When audio finishes, advance to the next page and start playback. +- At the final page, stop and remain on the final image. +- Next/Previous should stop the current audio, switch pages, and resume audio if playback was active. +- Only load the current image/audio metadata; do not load the whole book into memory. + +### Acceptance Criteria for the Build + +The app build is successful when: + +- Firmware builds. +- The app appears in the Tactility launcher. +- A test book under `/sdcard/books/test-book/` appears in the picker. +- Selecting the book displays page 1. +- Play starts page 1 narration. +- Play/pause, next, and previous work. +- Audio completion advances pages automatically. +- Missing/invalid files show or log a useful error instead of crashing. +- A README documents the SD-card folder format and includes a sample manifest. + +### Codex Prompt Seed + +Use the following as the working direction for Codex when implementing the app: + +```text +Create a new Tactility app for ESP32 called “Book Player”. It should scan `/sdcard/books/` for subfolders with `manifest.json`, show a book picker, then play each selected book page by displaying the page image and playing the matching audio. Provide Prev, Play/Pause, and Next controls. Auto-advance when audio finishes. Follow existing Tactility app structure, UI conventions, app registration, SD-card access, image rendering, and audio playback patterns in the repository. Keep memory usage safe by loading only the current page. Handle no SD card, no books, invalid JSON, missing assets, and playback errors gracefully. Add registration/build entries, a README, and a sample manifest. Build and report changed files and limitations. +``` + ## Correction Workflow Ideas Corrections should be recorded as structured page-level notes rather than ad hoc prompts.