tactility/Documentation/simulator-audio.md

Simulator audio

The desktop simulator exposes an SDL speaker and microphone through Tactility's standard audio_stream_* API. On macOS, SDL uses CoreAudio. Audio Settings controls the simulator's input/output volume, mute, and enabled state; these controls do not change macOS's system volume.

Running on macOS

Build the simulator in the usual host build environment (with python, lark, and pyyaml available, and without ESP_IDF_VERSION):

cmake -S . -B buildsim
cmake --build buildsim --target Tactility -j 8

Create a fresh application bundle (the script deliberately refuses to overwrite an existing bundle):

sh Buildscripts/release-simulator-macos-app.sh buildsim release/Tactility-audio.app
open release/Tactility-audio.app

Alternatively, run ../buildsim/Tactility/Tactility with Data/ as the working directory. Both the executable and the application bundle include a microphone usage description. macOS asks for microphone access on the first actual recording request, not at simulator startup. If denied, enable access in System Settings → Privacy & Security → Microphone and relaunch. For command-line launches, macOS may attribute the permission to the terminal or launching application.

A Mac mini needs an external input device, such as a USB mic or headset. Without an input device, speaker output still works and input is reported unavailable. Connect the input before launch for predictable discovery/UI behavior.

Selecting devices

By default, each stream opens the system's default device. Startup logs list SDL's device names. Optional environment variables select an exact name:

SIM_AUDIO_OUTPUT="Mac mini Speakers" SIM_AUDIO_INPUT="USB Microphone" \
  release/Tactility-audio.app/Contents/MacOS/Tactility

• SIM_AUDIO_OUTPUT: exact output name, or none to disable output.
• SIM_AUDIO_INPUT: exact input name, or none to disable input.
• Unset or empty values use the system default.

Selection is applied when opening a stream. An already-open stream does not automatically switch when the system default changes; close/reopen it or relaunch. A missing selected device causes an open failure rather than silently selecting a different device. Permission/device errors are logged under SdlAudio.

Formats and behavior

• Signed 16-bit PCM; the shared stream module converts app sample rates and channel counts to 48 kHz mono capture / stereo playback. SDL handles host format conversion.
• One input and one output stream can be open simultaneously.
• Read/write from a worker task, never the LVGL thread.
• Audio callbacks use bounded lock-free buffers, with silence on playback underrun and dropped incoming frames on capture overflow.
• Reads/writes report partial byte counts on timeout, including converted streams.
• Closing output drains its bounded buffer for up to 250 ms. Closing input does not affect output, and vice versa.
• No audio mixing or acoustic echo cancellation is added by this backend. Use headphones when testing simultaneous microphone capture and playback.

Verification

cmake --build buildsim --target SimulatorAudioTests -j 8
ctest --test-dir buildsim/Tests -R SimulatorAudio --output-on-failure

These tests use SDL's dummy and disk backends without requiring microphone access. They cover full duplex, 16/44.1/48 kHz app formats, bounded buffering, partial timeouts, enable/disable, unavailable inputs, and sample-level input/output gain and mute. They also cover disabling audio while a slow codec open is pending, as can happen during a microphone permission prompt. Disk fixtures and output files are temporary and removed after the run.

To explicitly play a quiet half-second 440 Hz tone through real Mac audio:

SDL_AUDIODRIVER=coreaudio SIM_AUDIO_OUTPUT= \
  buildsim/Tests/simulator/SimulatorAudioTests \
  --no-skip --test-case="hardware output smoke test"

Physical microphone capture, the first-use permission prompt, and live device unplugging require a separate manual check with an input device attached.