tactility_apps/Apps/DiscoveryMountain/CRASH_ANALYSIS.md

DiscoveryMountain – Crash After Install – Root Cause & Fix

Board: CYD ES3C28P (es3c28p) 192.168.68.132
Firmware: 0.8.0-dev / IDF 5.5.2
App ID: one.tactility.discoverymountain
Branch: personal/discovery-mountain-player → merged into main
Date: 2026-07-21

---

Symptoms

• After tactility.py install 192.168.68.132, device reboots or shows blue dialog Application failed to start: missing symbol
• Serial logs: E ELF: Can't find symbol ..., then Guru Meditation (LoadProhibited, StoreProhibited, IllegalInstruction, Cache error)
• Even when it did start, fetch fell back to 6 dummy episodes (S01E01…) with empty audio_url, so download always failed
• When forced to download 4.3 MB MP3 from http://192.168.68.110:8098/.../esp32_24k.mp3, device hard-faulted during play_task

---

1. Missing Symbol Loader Failures

Detected via xtensa-esp32s3-elf-nm -D …elf | grep " U "

U lv_font_montserrat_14
U lv_roller_create
U lv_roller_set_options
U lv_roller_get_selected
U lv_roller_set_selected
U qsort
U cJSON_* (via json component)

Why:

• Firmware exports defined in firmware/Modules/lvgl-module/source/symbols.c – roller not exported
• lv_font_montserrat_14 is a data symbol, not exported (AGENTS.md known missing)
• qsort not in g_esp_libc_elfsyms nor main_symbols
• cJSON not exported – apps must vendor cJSON.c (see BookPlayer, McpScreen)

Fix:

• manifest.properties V1 INI → V2 flat:

  manifest.version=0.2
  target.sdk=0.8.0-dev
  target.platforms=esp32s3
  app.id=one.tactility.discoverymountain
  app.version.name=0.1.0
  app.version.code=1
  

• main/CMakeLists.txt:

  set(CJSON_SOURCE "$ENV{IDF_PATH}/components/json/cJSON/cJSON.c")
  idf_component_register(SRCS ${SOURCE_FILES} ${CJSON_SOURCE}
                         INCLUDE_DIRS Source "$ENV{IDF_PATH}/components/json/cJSON"
                         REQUIRES TactilitySDK esp_http_client lwip)
  

• Replace roller with lv_dropdown (exported, see symbols.c:272-291)
• Replace qsort with bubble sort
• Replace lv_font_montserrat_14 with lvgl_get_text_font(FONT_SIZE_DEFAULT) from tactility/lvgl_fonts.h

After fix: nm -D shows no missing roller/qsort/font, verify script passes.

---

2. Fatal memset Bug

memset(&G,0,0); // size 0!

Static G is zero-initialized first boot, but second onShow leaves dangling lv_obj_t*, play_handle, etc.

Fix: memset(&G,0,sizeof(G))

---

3. Blocking LVGL Thread → Watchdog Reboot

Original onShow:

load_state(); fetch_data(); build_ui();

fetch_data() did 2× esp_http_client_perform with 8s timeout + cJSON_Parse of 91KB JSON on LVGL thread → 16s block → TWDT.

Fix:

• Build UI immediately with "Fetching..."
• xTaskCreate(fetch_task_fn,"dm_fetch",16384, ...)
• fetch_task_fn does fetch_data() in background, then tt_lvgl_lock to update labels
• Same for download: dl_task 12288 stack

---

4. esp_http_client vs Raw Sockets

• esp_http_client failed on device for PB http://192.168.68.110:8095 (returned chunked 1634\r\n{...}), while curl --http1.0 returned plain JSON.

Raw socket version via lwip_socket, ipaddr_addr, my_htons (like RobotArm) is more reliable and allows PSRAM buffers.

PocketBase:

• HTTP/1.1 → Transfer-Encoding: chunked → 1634\r\n{...}\r\n0\r\n\r\n
• HTTP/1.0 → Content-Length absent, body until close → 5684 bytes seasons, 91293 bytes episodes

Fix: Implement http_get_raw with:

• PSRAM heap_caps_malloc(65536, MALLOC_CAP_SPIRAM|8BIT) growing to 600KB
• Manual realloc via malloc+memcpy+free (avoid heap_caps_realloc not exported)
• strstr(buf,"\r\n\r\n") to find header end
• Detect chunked, decode hex chunk sizes via strtol(hex, NULL, 16)
• Return de-chunked body

Result: DM: seasons items count 37, episodes items 232 (was fallback 37/6)

---

5. Download Buffer & PSRAM

• dl_ep used esp_http_client + FILE* with 2KB stack buf, but also rename with static sd_path() race
• 4.3MB MP3 download needs streaming, not full RAM
• malloc(600000) in internal heap (120KB free) → OOM → cache error

Fix:

• make_sd_path(out,len,slug) caller-provided buffer
• ensure_dir() does NOT mkdir("/sdcard")
• dl_ep_raw uses PSRAM hdr 8192 + recv_buf 4096, streams directly to FILE*
• Progress via lv_label_set_text under tt_lvgl_lock
• LRU keeps 8 files, skips current playing

Result: DL done total 4329701 expected 4329701 – file correctly saved to /sdcard/dm/

---

6. Audio: I2S Direct vs audio-stream + PSRAM

Original used i2s_controller_* directly:

device_lock(i2s_dev); i2s_controller_set_config(...); i2s_controller_write(...);

Conflicts with Audio service, no resampling, vTaskDelete while holding lock → deadlock, samples*2 overflow for mono.

Fix (from BookPlayer):

• find_audio_device() via device_find_by_name("audio-stream")
• audio_stream_open_output(dev, cfg, &handle)
• audio_stream_write(handle, pcm+off, ...)
• close_stream() via audio_stream_close
• wait_play_exit() uses tt_lvgl_unlock(); vTaskDelay(10); tt_lvgl_lock() pattern
• Buffers: inbuf 16384 + pcm 1152*2*2 from PSRAM
• Volume scaling with proper clipping
• Stack 12288 for play/dl/fetch (was 6144/8192 → overflow on cJSON 91KB)
• Rate-limit ui_timer 500ms, only update bar if pct != last_pct

Playback crash – stack overflow:

***ERROR*** A stack overflow in task dm_play has been detected.
...
Guru Meditation: BREAK instr at vTaskGenericNotifyGiveFromISR
...
PC : 0x40385c6e : vPortYieldFromInt

Root: mp3dec_t (~2KB) + large locals on 8192 stack + malloc for buffers + fread + memmove caused overflow.

Fix:

• Increase dm_play stack 8192 → 16384
• Allocate mp3dec_t* dec from PSRAM via heap_caps_malloc(..., SPIRAM|8BIT) instead of stack
• Allocate inbuf / pcm via malloc (internal) first, fallback PSRAM
• Add ID3v2 skip:

  uint8_t id3hdr[10];
  if(fread(id3hdr,1,10,file)==10 && memcmp(id3hdr,"ID3",3)==0){
    int sz = (id3hdr[6]&0x7F)<<21 | ...;
    fseek(file, sz+10, SEEK_SET);
  }
  

• Add logs open_stream sr=%d ch=%d, open_stream ok, decoding frame buffered=%d

Result after fix:

DM: play task start /sdcard/dm/463_...mp3 size 4329701
DM: open_stream sr=16000 ch=1
esp32_i2s: Configuring I2S pins...
Adev_Codec: Open codec device OK
DM: open_stream ok

Stays alive 50s+ (previously crashed <1s).

7. UI Progress Not Showing (user report: "It is playing now, the UI does not show the progress, maybe due to the autoplay.")

Root:

G.pos_sec += samples / info.hz; // integer division 1152/16000 = 0!

pos_sec never increments, so pct = pos_sec*100/total_sec stays 0, bar never moves. Also last_pct not reset on new episode.

Fix:

• Track total_decoded_samples += samples; G.pos_sec = total_decoded_samples / hz;
• Reset last_pct=-1 in select_ep and start_idx_internal
• Keep ui_timer 500ms but always update time label, only bar when pct changes
• Remove auto-play for final stable (user presses Play manually) – auto-play from fetch_task caused race with LVGL lock and hid progress issue

After fix, progress bar moves and time label updates 0:01 / 4:30 etc. No immediate crash, stays alive 50s+ (previously crashed <1s).

Remaining TODO: test actual audio output via speaker, try 44.1k file if 16k resampling still issues, add taskYIELD() and volume ramp to avoid pop.

---

Build & Deploy (correct env)

Per AGENTS.md:

cd /Users/adolforeyna/Projects/Tactility/apps
unset PYTHONPATH; unset PYTHONHOME
export IDF_PYTHON_ENV_PATH=/Users/adolforeyna/.espressif/python_env/idf5.5_py3.10_env
source /Users/adolforeyna/esp/esp-idf-v5.5.2/export.sh
export TACTILITY_SDK_PATH=/Users/adolforeyna/Projects/Tactility/firmware/release/TactilitySDK

$IDF_PYTHON_ENV_PATH/bin/python tactility.py Apps/DiscoveryMountain build esp32s3 --local-sdk
curl -X PUT http://192.168.68.132/api/apps/install -F "file=@Apps/DiscoveryMountain/build/DiscoveryMountain.app"
curl -X POST "http://192.168.68.132/api/apps/run?id=one.tactility.discoverymountain"
# serial
python3 -c "import serial; s=serial.Serial('/dev/cu.usbmodem101',115200); ..."

Verify symbols:

xtensa-esp32s3-elf-nm -D Apps/DiscoveryMountain/build/cmake-build-esp32s3/DiscoveryMountain.app.elf | grep " U " | grep -E "roller|qsort|montserrat"
# should be empty

---

Final State (2026-07-21 post-fix)

• Stable install: No missing symbol, no memset garbage, no watchdog. Boots to launcher, shows UI, stays alive.
• Fetch: 37 seasons / 232 episodes via raw LWIP sockets + PSRAM 200KB buffer, HTTP/1.0 to avoid chunked. Previously fallback 6 dummy.
• Download: DL done total 4329701 expected 4329701 via simple streaming lwip_recv → FILE* with PSRAM hdr/recv buffers, LRU keeps 8 files. Previously OOM and crash.
• Playback: Fixed stack overflow in dm_play (8192 → 16384) + mp3dec_t heap allocated + ID3 skip. Now:

  DM: play task start ... size 4329701
  DM: open_stream sr=16000 ch=1
  DM: open_stream ok
  

  No immediate crash, stays alive 50s+. Audio output via audio_stream (ES8311) should work, but needs manual play test (auto-play removed for stability).
• UI: Roller → dropdown, font → lvgl_get_text_font, no HideStatusBar per user request.
• Build: Verified nm -D clean, tactility.py build esp32s3 --local-sdk ok.

Verified Logs (final)

DM: fetch task start
DM: GET .../dm_seasons... → GET ok 5684
DM: seasons items count 37
DM: GET .../dm_episodes... → GET ok 91293
DM: episodes items 232
DM: fetch done seasons=37 eps=232

---

TODO

• Fix playback: skip ID3v2 before decode, try internal RAM for mp3dec_t, test with 44.1k mp3 from other episode (not esp32_24k)
• Use device_get_by_name / device_put instead of deprecated find
• Use tt_app_get_user_data_path for state file, not hardcoded /sdcard/apps/...
• Rate-limit screenshot API (2s gap) to avoid heap spike