mcp_screen/iphone_app/README.md

# iPhone MCP

Native UIKit MCP screen server for a jailbroken arm64 iPhone running iOS 12. It shares the ESP32 screen's MCP tool shape while using the iPhone display, speaker, microphone, battery, and front camera.

## Features

- HTTP JSON-RPC MCP endpoint: `POST http://<iphone-ip>:8080/api/mcp`
- Full-screen landscape canvas with idle sleep disabled while the app is active
- ESP32-compatible 400x300 monochrome streaming: TCP `8081`, UDP `8082`
- Framed RGB565 color streaming: TCP `8083`
- UDP discovery on `5000`: `DISCOVER_SCREEN` -> `SCREEN_IP_8080`
- True pitched tones, WAV playback, microphone recording, and selfie capture
- Hermes voice gateway turn: record -> transcribe/agent/TTS -> play response
- Stream statistics and MCP screenshots

## Requirements

- Jailbroken arm64 iPhone (tested on iPhone 6 / iOS 12.5.8)
- OpenSSH and `ldid` installed on the phone
- Theos at `/var/mobile/theos` with `iPhoneOS12.4.sdk`
- The phone and build computer on the same LAN

## Copy To The Phone

From the repository root:

```sh
rsync -av iphone_app/ root@<iphone-ip>:/var/mobile/IPhoneMCP/
```

Keep the phone unlocked while launching the app. On iOS 12, `uiopen` may refuse to launch a locked device.

## Build

The included direct build script is the most reliable path on the iPhone 6:

```sh
ssh root@<iphone-ip>
su - mobile -c 'chmod +x /var/mobile/IPhoneMCP/manual_build_iphone.sh && /var/mobile/IPhoneMCP/manual_build_iphone.sh /var/mobile/IPhoneMCP'
```

This creates and signs:

```text
/var/mobile/IPhoneMCP/.manual_build/IPhoneMCP.app
```

The regular Theos build is also supported:

```sh
su - mobile -c 'env THEOS=/var/mobile/theos PATH=/usr/bin:/bin:/usr/sbin:/sbin make -C /var/mobile/IPhoneMCP clean package SDKVERSION=12.4 INCLUDE_SDKVERSION=12.4 FAKEROOT="bash /var/mobile/theos/bin/fakeroot.sh -p /var/mobile/IPhoneMCP/.theos/fakeroot" _THEOS_PLATFORM_DPKG_DEB=dpkg-deb THEOS_PLATFORM_DEB_COMPRESSION_TYPE=gzip'
```

## Package And Install

For a direct-build bundle:

```sh
PKGROOT=/var/mobile/IPhoneMCP/.manual_pkg
DEB=/var/mobile/IPhoneMCP/packages/com.reynafamily.iphonemcp_manual_iphoneos-arm.deb
rm -rf "$PKGROOT"
mkdir -p "$PKGROOT/DEBIAN" "$PKGROOT/Applications/IPhoneMCP.app" /var/mobile/IPhoneMCP/packages
cp /var/mobile/IPhoneMCP/control "$PKGROOT/DEBIAN/control"
cp /var/mobile/IPhoneMCP/.manual_build/IPhoneMCP.app/IPhoneMCP "$PKGROOT/Applications/IPhoneMCP.app/IPhoneMCP"
cp /var/mobile/IPhoneMCP/.manual_build/IPhoneMCP.app/Info.plist "$PKGROOT/Applications/IPhoneMCP.app/Info.plist"
chmod 755 "$PKGROOT/Applications/IPhoneMCP.app/IPhoneMCP"
chown -R root:wheel "$PKGROOT"
dpkg-deb -b "$PKGROOT" "$DEB"
dpkg -i "$DEB"
uicache -p /Applications/IPhoneMCP.app
uiopen com.reynafamily.iphonemcp
```

Verify listeners:

```sh
netstat -an | grep -E '\.8080|\.8081|\.8082|\.8083|\.5000'
```

## MCP Bridge

```sh
python3 iphone_app/iphone_mcp_bridge.py --ip <iphone-ip>
```

Example client configuration:

```json
{
  "mcpServers": {
    "iphone6-mcp": {
      "command": "python3",
      "args": ["/absolute/path/to/mcp_screen/iphone_app/iphone_mcp_bridge.py", "--ip", "192.168.68.150"]
    }
  }
}
```

## Monochrome Streaming

- TCP `8081`: consecutive 15,000-byte ESP32 RLCD frame buffers
- UDP `8082`: fifteen 1002-byte packets per frame
  - byte 0: frame id
  - byte 1: chunk index 0-14
  - bytes 2-1001: 1000-byte payload

The mapping is identical to `test_stream.py` in the repository.

## RGB565 Color Streaming

Color frames use TCP port `8083`. Every frame has a 16-byte big-endian header followed by RGB565 data:

| Offset | Size | Meaning |
|---|---:|---|
| 0 | 4 | ASCII `IMCR` |
| 4 | 1 | Version `1` |
| 5 | 1 | Format `1` = RGB565 big-endian |
| 6 | 2 | Width |
| 8 | 2 | Height |
| 10 | 4 | Payload length (`width * height * 2`) |
| 14 | 2 | Flags, currently `0` |

Run the included example:

```sh
python3 -m pip install Pillow
python3 iphone_app/tools/stream_color.py --ip <iphone-ip>
```

## Hermes Voice

The `hermes_voice_turn` MCP tool records a mono 16 kHz, 16-bit WAV, posts it to Hermes, plays the returned WAV, and returns the transcript and answer headers.

Default gateway:

```text
http://192.168.68.126:8642/api/esp32/voice
```

Example JSON-RPC call:

```json
{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "hermes_voice_turn",
    "arguments": {
      "url": "http://192.168.68.126:8642/api/esp32/voice",
      "api_token": "YOUR_HERMES_API_TOKEN",
      "device_id": "iphone6",
      "duration_sec": 4,
      "volume": 80,
      "reply_mode": "sync",
      "screen_url": "http://192.168.68.150:8080/api/mcp"
    }
  }
}
```

Do not commit the Hermes API token. It is accepted per call and is not persisted by the app.

## Implemented Tools

`clear_screen`, `draw_text`, `draw_image`, `get_screenshot`, `get_battery`, `play_tone`, `play_audio`, `play_audio_base64`, `record_voice`, `hermes_voice_turn`, `capture_selfie`, `write_file`, `read_file`, `download_file`, `get_video_streaming_instructions`, `get_stream_stats`, `reset_stream_stats`, and `set_stream_debug`.

Compatibility stubs remain for `set_led`, `get_sensors`, `scan_ble`, and `execute_python`.

## Operational Notes

- Keep the app foregrounded for reliable network service on iOS 12.
- The idle timer is disabled only while the app is active.
- Give each device a unique static DHCP reservation; duplicate IPs can make traffic reach the wrong host.