chore: checkpoint app work before firmware sync
This commit is contained in:
Adolfo Reyna
@@ -0,0 +1,89 @@
|
||||
# Pipecat SmallWebRTC ESP32 feasibility spike
|
||||
|
||||
Date: 2026-08-10
|
||||
|
||||
## Decision
|
||||
|
||||
**NO-GO for a Tactility external ELF on the current SDK; GO only for a separate, full ESP-IDF firmware application.**
|
||||
|
||||
Pipecat has an official native ESP32 client, so the required transport is real and source-proven. It cannot presently be used as a Tactility runtime ELF without a firmware/SDK integration project: the client is a full ESP-IDF firmware with private, static component dependencies and system configuration that the ELF loader does not provide or export. Do not replace the rejected raw-WebSocket implementation with another transport until that integration is designed and proven.
|
||||
|
||||
This is not a proposal to flash anything. No device was deployed or flashed during this spike.
|
||||
|
||||
## Source-pinned native client
|
||||
|
||||
| Item | Evidence |
|
||||
| --- | --- |
|
||||
| Client | `https://github.com/pipecat-ai/pipecat-esp32`, commit `e70e3b1f0576e502af9e390434e1a6e8a5cd0d2e`, cloned with all recursive submodules |
|
||||
| Top-level licence | MIT (`LICENSE`, copyright Daily/OpenAI) |
|
||||
| Pipecat server | local installed `pipecat-ai 1.7.0`, Python 3.11 virtual environment |
|
||||
| Server ESP32 support | `SmallWebRTCRequestHandler(..., esp32_mode=True, host=...)` munges the SDP; `smallwebrtc_sdp_munging()` removes SHA-384/SHA-512 fingerprints and retains only the chosen host's ICE candidates |
|
||||
| ESP-IDF build used | local ESP-IDF `v5.5.2`, environment `idf5.5_py3.9_env` |
|
||||
| Native build result | upstream `esp32-s3-box-3` built successfully, including `peer`, `srtp`, `esp-libopus`, Wi-Fi, HTTP client, DTLS/SRTP, Opus, and the ESP-BOX-3 BSP |
|
||||
|
||||
The official client is designed for ESP32-S3 and uses the `libpeer` API. Its `PeerConfiguration` sets `CODEC_OPUS`, creates a peer connection, installs ICE/data/audio callbacks, and invokes `peer_connection_create_offer()`. This is the source-proven native WebRTC implementation; it owns ICE, DTLS-SRTP, RTP, and Opus rather than hand-implementing any of them.
|
||||
|
||||
## Required SmallWebRTC contract
|
||||
|
||||
The product transport is SmallWebRTC HTTP signaling plus WebRTC media, never the old raw WebSocket PCM/JSON protocol:
|
||||
|
||||
1. `POST /start` with `transport: "webrtc"`, `enableDefaultIceServers: false`, and optional `body`; retain the returned `sessionId`.
|
||||
2. `POST /sessions/{sessionId}/api/offer` with `{ "sdp": ..., "type": "offer", "pc_id": optional, "restart_pc": optional, "requestData": optional }`; Pipecat returns SDP answer, type, and `pc_id`.
|
||||
3. `PATCH /sessions/{sessionId}/api/offer` with `{ "pc_id": ..., "candidates": [{ "candidate": ..., "sdp_mid": ..., "sdp_mline_index": ... }] }` for trickle ICE. An empty candidate is the end-of-candidates marker.
|
||||
4. Use the negotiated WebRTC audio track continuously. The runner starts the bot after the offer is processed.
|
||||
|
||||
The local Pipecat source also supports the direct `/api/offer` route used by the current official ESP32 example. The session form above is the approved application contract because it supports Pipecat runner session lifecycle. The live endpoint returned HTTP 200 to `/status`, but it was not restarted with `--esp32`; therefore no live offer/candidate exchange is represented as ESP32 validation.
|
||||
|
||||
## Audio adapter boundary
|
||||
|
||||
The official client source (`media.cpp`) uses 16 kHz, mono, signed 16-bit PCM (`640` bytes = 320 samples = 20 ms) and encodes it as Opus for `peer_connection_send_audio()`. Inbound WebRTC audio reaches the `onaudiotrack` callback as Opus, is decoded to the same PCM shape, and is written to the speaker codec.
|
||||
|
||||
For a future firmware-level integration, Tactility must keep ownership at the following boundary (no hard-coded board pins):
|
||||
|
||||
- acquire the existing Tactility `audio_stream` / `i2s_controller` service;
|
||||
- pull fixed 20 ms frames, 16 kHz mono S16LE, into the native client encoder;
|
||||
- feed decoded remote S16LE frames to the existing output service;
|
||||
- serialize I/O ownership, keep bounded queues, and drop stale audio rather than accumulating latency;
|
||||
- close peer/media callbacks before releasing the audio device.
|
||||
|
||||
The current kernel exports `audio_stream_open_input`, `audio_stream_open_output`, `audio_stream_read`, `audio_stream_write`, `audio_stream_close`, and `i2s_controller_read`/`i2s_controller_write`. Those APIs are the usable boundary, not a reason to configure physical pins in the app.
|
||||
|
||||
## Full-firmware build evidence
|
||||
|
||||
The following was run in a temporary checkout; non-secret placeholder Wi-Fi values were used and no flash command was run:
|
||||
|
||||
```text
|
||||
cd /tmp/pipecat-esp32-spike
|
||||
# cloned pipecat-esp32 at e70e3b1... and initialized all recursive submodules
|
||||
cd esp32-s3-box-3
|
||||
unset PYTHONPATH PYTHONHOME
|
||||
export IDF_PYTHON_ENV_PATH=/Users/adolforeyna/.espressif/python_env/idf5.5_py3.9_env
|
||||
export WIFI_SSID=spike
|
||||
export WIFI_PASSWORD=spike
|
||||
export PIPECAT_SMALLWEBRTC_URL=http://192.168.68.112:7860/api/offer
|
||||
source /Users/adolforeyna/esp/esp-idf/export.sh
|
||||
idf.py build
|
||||
```
|
||||
|
||||
Actual result: `src.elf` and `src.bin` were produced; the IDF build ended with `Project build complete`. `src.bin` is **1,493,408 bytes** and the upstream 1.5 MiB app partition reported **79,712 bytes (5%) free**. `xtensa-esp32s3-elf-size src.elf` reported text `1,304,360`, data `201,012`, bss `2,863,205` (total `4,368,577`). The linked firmware has no undefined dynamic symbols.
|
||||
|
||||
This is important capacity evidence: even before adapting it to the target board and Tactility services, the supported client nearly fills its own dedicated application partition and has a 2.86 MiB BSS footprint.
|
||||
|
||||
## Why this does not link as a Tactility ELF
|
||||
|
||||
Tactility's `TactilitySDK.cmake` calls `project_elf()`. Its loader CMake builds a PIC shared ELF with `-nostartfiles -nostdlib -shared -e app_main`, and links only `main` plus explicitly listed `ELF_COMPONENTS` / `ELF_LIBS`. The current PipecatVoice component declares only `REQUIRES TactilitySDK lwip`.
|
||||
|
||||
The upstream client instead requires full firmware components including `peer`, `srtp`, `esp-libopus`, `esp_http_client`, `esp_wifi`, `nvs_flash`, `esp_psram`, `esp_netif`, mbedTLS, and ESP-BOX-3 BSP. The official `peer` static archive has unresolved references to the linked firmware environment such as `mbedtls_ssl_conf_dtls_srtp_protection_profiles`, `mbedtls_ssl_config_defaults`, `lwip_inet_ntop`, and socket/ICE helpers. The Tactility kernel export table contains the audio service APIs listed above but no `peer_connection`, `opus_*`, `srtp_*`, `mbedtls_*`, `esp_http_client*`, `esp_wifi*`, or `esp_netif*` exports.
|
||||
|
||||
Attempting the ordinary app build also hit a concrete local SDK packaging blocker before linking: `tactility.py Apps/PipecatVoice build esp32s3 --local-sdk` reported that `Buildscripts/TactilitySDK/0.8.0-dev-esp32s3/TactilitySDK` is missing. This must be corrected for later normal ELF builds, but it is distinct from the component/loader incompatibility.
|
||||
|
||||
Therefore copying the client sources or merely adding `REQUIRES peer` would not make a runnable ELF: it would either fail to find the private IDF component libraries during the ELF link or produce imports that the firmware loader cannot resolve. Statically embedding all dependencies is unproven and high-risk because of ELF size, duplicate runtime/library state, SDK configuration, and Wi-Fi/codec ownership conflicts.
|
||||
|
||||
## Security, licensing, and follow-up gate
|
||||
|
||||
- Do not log SDP, ICE details, credentials, or raw audio. The source-level HTTP helper currently logs offer/answer in debug mode; any reused code must remove that logging.
|
||||
- The upstream defaults deliberately disable TLS certificate verification for its demo. Production must use HTTPS with a pinned/validated trust chain; do not inherit that setting.
|
||||
- Preserve MIT notices for Pipecat ESP32 and audit each pinned submodule separately (`libpeer`, SRTP/libSRTP, Opus, and Espressif managed components have their own licenses).
|
||||
- Do not use the existing PipecatVoice raw WebSocket code or its historical configuration as a fallback; it is protocol-incompatible with SmallWebRTC.
|
||||
|
||||
A firmware-level project must first export/package the required WebRTC dependency set, prove an external ELF link with zero unresolved loader symbols (or move the client into firmware), set deterministic memory budgets, and then perform an `--esp32` SmallWebRTC live offer/ICE/media test. Only after that gate may the approved minimal auto-start UI be implemented.
|
||||
Reference in New Issue
Block a user