Driving it from an agent
There is exactly one channel between an agent and the XPhoneUse desktop app: http://127.0.0.1:7070/mcp, plus a skill that teaches it how to use it. This page is the verb table and the handful of rules that matter.
Coordinates are normalized
Every coordinate is a 0–1 ratio: (0,0) is the top-left corner, (1,1) the bottom-right. It does not matter whether the device is 1080p or 2K, or how far the mirror is scaled down — the same command means the same thing on Android and iPhone.
python scripts/xphoneuse-cli.py tap 0.5 0.92 # bottom centre
python scripts/xphoneuse-cli.py longtap 0.5 0.3 1200 # press and hold for 1.2s
Verbs
| Command | Notes |
|---|---|
tap <x> <y> |
The default way to touch anything |
longtap <x> <y> [ms] |
Long press, 800 ms by default, 100–10000 |
swipe <fx> <fy> <tx> <ty> |
Drag from one point to another |
scroll <x> <y> <dx> <dy> |
Negative dy scrolls down |
text <TEXT> [--enter] |
Full Unicode; --enter presses Enter (which sends, in chat apps) |
unicode-text <TEXT> [--method clipboard|agentinput] |
Choose how iOS delivers non-ASCII: clipboard paste (default; nothing to install, but iOS shows the Allow-Paste alert) or the AgentInput keyboard (no alerts, clipboard untouched; it must be installed and active) |
press home|back|recents|power|volume_up|volume_down |
iOS has no back |
screenshot [--compare ref.png] |
With --compare: a pixel-diff verdict and the region that changed |
ui-tree / find --text … |
Android: read the control tree, locate an element by text |
launch <PACKAGE> / open-url <URL> |
Android |
macro record|play |
Record a flow once, replay it forever - Android and iOS |
wifi enable|connect|ip |
Android: Wi-Fi debugging |
screenshot, list, shell and wifi * work without a session; the input verbs need connect first.
Three rules of thumb
Look before you act, and verify after. Unverified actions on a real device are a coin flip.
Read text instead of pixels. On Android ui-tree and find return the whole screen as text with coordinates, at a fraction of the cost of an image. Spend screenshots on canvases — games, maps, photo grids.
One action, one verification. Tap, look, tap. Five taps in a row before looking is how you end up typing into the wrong conversation.
Screenshot diffing
The agent's most expensive sense is sight. --compare diffs locally so it only needs to look when something actually changed:
python scripts/xphoneuse-cli.py screenshot --compare xphoneuse-screenshot.png
# VERDICT: CHANGED (86.95% of pixels differ, threshold 0.50%)
# changed region: x 0-1178, y 0-2555 (normalized 0.000-1.000, 0.000-1.001)
UNCHANGED means skip the image; CHANGED means look. The changed-region line often answers the question by itself: "the bottom third" is a keyboard, "a small box in the middle" is a dialog.
MCP clients
{
"mcpServers": {
"xphoneuse": { "url": "http://127.0.0.1:7070/mcp" }
}
}
MCP and the CLI share one session: a device connected from the command line is immediately usable from MCP.
Last updated 2026-09-17