mcp_screen/iphone_app/README.md

iPhone MCP

Native UIKit MCP screen server for a jailbroken arm64 iPhone running iOS 12. It shares the ESP32 screen's MCP tool shape while using the iPhone display, speaker, microphone, battery, and front camera.

Features

• HTTP JSON-RPC MCP endpoint: POST http://<iphone-ip>:8080/api/mcp
• Full-screen landscape canvas with idle sleep disabled while the app is active
• ESP32-compatible 400x300 monochrome streaming: TCP 8081, UDP 8082
• Framed RGB565 color streaming: TCP 8083
• UDP discovery on 5000: DISCOVER_SCREEN -> SCREEN_IP_8080
• True pitched tones, WAV playback, microphone recording, and selfie capture
• Hermes voice gateway turn: record -> transcribe/agent/TTS -> play response
• Stream statistics and MCP screenshots

Requirements

• Jailbroken arm64 iPhone (tested on iPhone 6 / iOS 12.5.8)
• OpenSSH and ldid installed on the phone
• Theos at /var/mobile/theos with iPhoneOS12.4.sdk
• The phone and build computer on the same LAN

Copy To The Phone

From the repository root:

rsync -av iphone_app/ root@<iphone-ip>:/var/mobile/IPhoneMCP/

Keep the phone unlocked while launching the app. On iOS 12, uiopen may refuse to launch a locked device.

Build

The included direct build script is the most reliable path on the iPhone 6:

ssh root@<iphone-ip>
su - mobile -c 'chmod +x /var/mobile/IPhoneMCP/manual_build_iphone.sh && /var/mobile/IPhoneMCP/manual_build_iphone.sh /var/mobile/IPhoneMCP'

This creates and signs:

/var/mobile/IPhoneMCP/.manual_build/IPhoneMCP.app

The regular Theos build is also supported:

su - mobile -c 'env THEOS=/var/mobile/theos PATH=/usr/bin:/bin:/usr/sbin:/sbin make -C /var/mobile/IPhoneMCP clean package SDKVERSION=12.4 INCLUDE_SDKVERSION=12.4 FAKEROOT="bash /var/mobile/theos/bin/fakeroot.sh -p /var/mobile/IPhoneMCP/.theos/fakeroot" _THEOS_PLATFORM_DPKG_DEB=dpkg-deb THEOS_PLATFORM_DEB_COMPRESSION_TYPE=gzip'

Package And Install

For a direct-build bundle:

PKGROOT=/var/mobile/IPhoneMCP/.manual_pkg
DEB=/var/mobile/IPhoneMCP/packages/com.reynafamily.iphonemcp_manual_iphoneos-arm.deb
rm -rf "$PKGROOT"
mkdir -p "$PKGROOT/DEBIAN" "$PKGROOT/Applications/IPhoneMCP.app" /var/mobile/IPhoneMCP/packages
cp /var/mobile/IPhoneMCP/control "$PKGROOT/DEBIAN/control"
cp /var/mobile/IPhoneMCP/.manual_build/IPhoneMCP.app/IPhoneMCP "$PKGROOT/Applications/IPhoneMCP.app/IPhoneMCP"
cp /var/mobile/IPhoneMCP/.manual_build/IPhoneMCP.app/Info.plist "$PKGROOT/Applications/IPhoneMCP.app/Info.plist"
chmod 755 "$PKGROOT/Applications/IPhoneMCP.app/IPhoneMCP"
chown -R root:wheel "$PKGROOT"
dpkg-deb -b "$PKGROOT" "$DEB"
killall IPhoneMCP 2>/dev/null || true
dpkg -i "$DEB"

For an update, launch iPhone MCP again from its Home Screen icon. Do not run uiopen com.reynafamily.iphonemcp: this jailbreak's uiopen command expects a URL, not an application bundle identifier.

Only run the following after the first installation if the icon does not appear:

uicache -p /Applications/IPhoneMCP.app

Stop the app before running it and let uicache exit normally. Interrupting a refresh can leave the application database busy and make app launches appear to crash. If that happens, recover it over SSH with killall -9 uicache, then tap the app icon again.

Verify listeners:

netstat -an | grep -E '\.8080|\.8081|\.8082|\.8083|\.5000'

MCP Bridge

python3 iphone_app/iphone_mcp_bridge.py --ip <iphone-ip>

Example client configuration:

{
  "mcpServers": {
    "iphone6-mcp": {
      "command": "python3",
      "args": ["/absolute/path/to/mcp_screen/iphone_app/iphone_mcp_bridge.py", "--ip", "192.168.68.150"]
    }
  }
}

Monochrome Streaming

• TCP 8081: consecutive 15,000-byte ESP32 RLCD frame buffers
• UDP 8082: fifteen 1002-byte packets per frame
  • byte 0: frame id
  • byte 1: chunk index 0-14
  • bytes 2-1001: 1000-byte payload

The mapping is identical to test_stream.py in the repository.

RGB565 Color Streaming

Color frames use TCP port 8083. Every frame has a 16-byte big-endian header followed by RGB565 data:

Offset	Size	Meaning
0	4	ASCII IMCR
4	1	Version 1
5	1	Format 1 = RGB565 big-endian
6	2	Width
8	2	Height
10	4	Payload length (width * height * 2)
14	2	Flags, currently 0

Run the included example:

python3 -m pip install Pillow
python3 iphone_app/tools/stream_color.py --ip <iphone-ip>

Hermes Voice

The hermes_voice_turn MCP tool records a mono 16 kHz, 16-bit WAV, posts it to Hermes, plays the returned WAV, and returns the transcript and answer headers.

Default gateway:

http://192.168.68.126:8642/api/esp32/voice

Example JSON-RPC call:

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "hermes_voice_turn",
    "arguments": {
      "url": "http://192.168.68.126:8642/api/esp32/voice",
      "api_token": "YOUR_HERMES_API_TOKEN",
      "device_id": "iphone6",
      "duration_sec": 4,
      "volume": 80,
      "reply_mode": "sync",
      "screen_url": "http://192.168.68.150:8080/api/mcp"
    }
  }
}

Do not commit the Hermes API token. It is accepted per call and is not persisted by the app.

Implemented Tools

clear_screen, draw_text, draw_image, get_screenshot, get_battery, play_tone, play_audio, play_audio_base64, record_voice, hermes_voice_turn, capture_selfie, write_file, read_file, download_file, get_video_streaming_instructions, get_stream_stats, reset_stream_stats, and set_stream_debug.

Compatibility stubs remain for set_led, get_sensors, scan_ble, and execute_python.

Operational Notes

• Keep the app foregrounded for reliable network service on iOS 12.
• The idle timer is disabled only while the app is active.
• Give each device a unique static DHCP reservation; duplicate IPs can make traffic reach the wrong host.