Add jailbroken iPhone MCP screen app

2026-06-21 21:25:55 -04:00
parent 8dbc5f4c7b
commit ad30662a48
21 changed files with 2206 additions and 2 deletions
@@ -0,0 +1,174 @@
+# iPhone MCP
+
+Native UIKit MCP screen server for a jailbroken arm64 iPhone running iOS 12. It shares the ESP32 screen's MCP tool shape while using the iPhone display, speaker, microphone, battery, and front camera.
+
+## Features
+
+- HTTP JSON-RPC MCP endpoint: `POST http://<iphone-ip>:8080/api/mcp`
+- Full-screen landscape canvas with idle sleep disabled while the app is active
+- ESP32-compatible 400x300 monochrome streaming: TCP `8081`, UDP `8082`
+- Framed RGB565 color streaming: TCP `8083`
+- UDP discovery on `5000`: `DISCOVER_SCREEN` -> `SCREEN_IP_8080`
+- True pitched tones, WAV playback, microphone recording, and selfie capture
+- Hermes voice gateway turn: record -> transcribe/agent/TTS -> play response
+- Stream statistics and MCP screenshots
+
+## Requirements
+
+- Jailbroken arm64 iPhone (tested on iPhone 6 / iOS 12.5.8)
+- OpenSSH and `ldid` installed on the phone
+- Theos at `/var/mobile/theos` with `iPhoneOS12.4.sdk`
+- The phone and build computer on the same LAN
+
+## Copy To The Phone
+
+From the repository root:
+
+```sh
+rsync -av iphone_app/ root@<iphone-ip>:/var/mobile/IPhoneMCP/
+```
+
+Keep the phone unlocked while launching the app. On iOS 12, `uiopen` may refuse to launch a locked device.
+
+## Build
+
+The included direct build script is the most reliable path on the iPhone 6:
+
+```sh
+ssh root@<iphone-ip>
+su - mobile -c 'chmod +x /var/mobile/IPhoneMCP/manual_build_iphone.sh && /var/mobile/IPhoneMCP/manual_build_iphone.sh /var/mobile/IPhoneMCP'
+```
+
+This creates and signs:
+
+```text
+/var/mobile/IPhoneMCP/.manual_build/IPhoneMCP.app
+```
+
+The regular Theos build is also supported:
+
+```sh
+su - mobile -c 'env THEOS=/var/mobile/theos PATH=/usr/bin:/bin:/usr/sbin:/sbin make -C /var/mobile/IPhoneMCP clean package SDKVERSION=12.4 INCLUDE_SDKVERSION=12.4 FAKEROOT="bash /var/mobile/theos/bin/fakeroot.sh -p /var/mobile/IPhoneMCP/.theos/fakeroot" _THEOS_PLATFORM_DPKG_DEB=dpkg-deb THEOS_PLATFORM_DEB_COMPRESSION_TYPE=gzip'
+```
+
+## Package And Install
+
+For a direct-build bundle:
+
+```sh
+PKGROOT=/var/mobile/IPhoneMCP/.manual_pkg
+DEB=/var/mobile/IPhoneMCP/packages/com.reynafamily.iphonemcp_manual_iphoneos-arm.deb
+rm -rf "$PKGROOT"
+mkdir -p "$PKGROOT/DEBIAN" "$PKGROOT/Applications/IPhoneMCP.app" /var/mobile/IPhoneMCP/packages
+cp /var/mobile/IPhoneMCP/control "$PKGROOT/DEBIAN/control"
+cp /var/mobile/IPhoneMCP/.manual_build/IPhoneMCP.app/IPhoneMCP "$PKGROOT/Applications/IPhoneMCP.app/IPhoneMCP"
+cp /var/mobile/IPhoneMCP/.manual_build/IPhoneMCP.app/Info.plist "$PKGROOT/Applications/IPhoneMCP.app/Info.plist"
+chmod 755 "$PKGROOT/Applications/IPhoneMCP.app/IPhoneMCP"
+chown -R root:wheel "$PKGROOT"
+dpkg-deb -b "$PKGROOT" "$DEB"
+dpkg -i "$DEB"
+uicache -p /Applications/IPhoneMCP.app
+uiopen com.reynafamily.iphonemcp
+```
+
+Verify listeners:
+
+```sh
+netstat -an | grep -E '\.8080|\.8081|\.8082|\.8083|\.5000'
+```
+
+## MCP Bridge
+
+```sh
+python3 iphone_app/iphone_mcp_bridge.py --ip <iphone-ip>
+```
+
+Example client configuration:
+
+```json
+{
+  "mcpServers": {
+    "iphone6-mcp": {
+      "command": "python3",
+      "args": ["/absolute/path/to/mcp_screen/iphone_app/iphone_mcp_bridge.py", "--ip", "192.168.68.150"]
+    }
+  }
+}
+```
+
+## Monochrome Streaming
+
+- TCP `8081`: consecutive 15,000-byte ESP32 RLCD frame buffers
+- UDP `8082`: fifteen 1002-byte packets per frame
+  - byte 0: frame id
+  - byte 1: chunk index 0-14
+  - bytes 2-1001: 1000-byte payload
+
+The mapping is identical to `test_stream.py` in the repository.
+
+## RGB565 Color Streaming
+
+Color frames use TCP port `8083`. Every frame has a 16-byte big-endian header followed by RGB565 data:
+
+| Offset | Size | Meaning |
+|---|---:|---|
+| 0 | 4 | ASCII `IMCR` |
+| 4 | 1 | Version `1` |
+| 5 | 1 | Format `1` = RGB565 big-endian |
+| 6 | 2 | Width |
+| 8 | 2 | Height |
+| 10 | 4 | Payload length (`width * height * 2`) |
+| 14 | 2 | Flags, currently `0` |
+
+Run the included example:
+
+```sh
+python3 -m pip install Pillow
+python3 iphone_app/tools/stream_color.py --ip <iphone-ip>
+```
+
+## Hermes Voice
+
+The `hermes_voice_turn` MCP tool records a mono 16 kHz, 16-bit WAV, posts it to Hermes, plays the returned WAV, and returns the transcript and answer headers.
+
+Default gateway:
+
+```text
+http://192.168.68.126:8642/api/esp32/voice
+```
+
+Example JSON-RPC call:
+
+```json
+{
+  "jsonrpc": "2.0",
+  "id": 1,
+  "method": "tools/call",
+  "params": {
+    "name": "hermes_voice_turn",
+    "arguments": {
+      "url": "http://192.168.68.126:8642/api/esp32/voice",
+      "api_token": "YOUR_HERMES_API_TOKEN",
+      "device_id": "iphone6",
+      "duration_sec": 4,
+      "volume": 80,
+      "reply_mode": "sync",
+      "screen_url": "http://192.168.68.150:8080/api/mcp"
+    }
+  }
+}
+```
+
+Do not commit the Hermes API token. It is accepted per call and is not persisted by the app.
+
+## Implemented Tools
+
+`clear_screen`, `draw_text`, `draw_image`, `get_screenshot`, `get_battery`, `play_tone`, `play_audio`, `play_audio_base64`, `record_voice`, `hermes_voice_turn`, `capture_selfie`, `write_file`, `read_file`, `download_file`, `get_video_streaming_instructions`, `get_stream_stats`, `reset_stream_stats`, and `set_stream_debug`.
+
+Compatibility stubs remain for `set_led`, `get_sensors`, `scan_ble`, and `execute_python`.
+
+## Operational Notes
+
+- Keep the app foregrounded for reliable network service on iOS 12.
+- The idle timer is disabled only while the app is active.
+- Give each device a unique static DHCP reservation; duplicate IPs can make traffic reach the wrong host.