Use operating system accessibility tree APIs (AX APIs) to build resilient desktop UI automation that survives visual changes
domain: developer.apple.com · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗
Steps
On macOS, query the accessibility tree via the `Accessibility` framework using `AXUIElementCreateApplication(pid)` to obtain the root element for a process, then traverse with `AXUIElementCopyAttributeValue` to read properties like `AXRole`, `AXTitle`, and `AXChildren`.
On Windows, use the UI Automation API (see the UIA route) or the lower-level IAccessible/MSAA interface for legacy apps — both expose a hierarchical tree of elements with roles, names, and states independent of pixel positions.
Identify target elements by semantic attributes (role + name or identifier) rather than by screen coordinates — coordinate-based clicking breaks when windows are resized or moved, while AX tree paths are stable across those changes.
Trigger actions through AX-exposed action methods rather than simulated mouse clicks where possible: on macOS, list available actions with `AXUIElementCopyActionNames` and call `AXUIElementPerformAction(element, kAXPressAction)` to press a button.
For cross-platform scripts, use a library that abstracts the platform AX API (e.g., `pywinauto` on Windows, `atomacos` or `AppiumForMac` on macOS) to write unified automation code without calling the raw C APIs directly.
Known gotchas
AX API access requires explicit permission grants on both macOS (Accessibility in Privacy & Security settings) and Windows (process integrity level matching). An automation script that worked as an admin may fail when run as a standard user because it cannot attach to elevated processes.
Applications built with non-native UI frameworks (some Electron apps, Java Swing with default settings, some Qt builds) may expose a flat or incomplete accessibility tree where interactive elements appear as a single generic container. In those cases, AX-based automation cannot reach individual controls — use browser automation for Electron or check whether the framework has an accessibility plugin.
The AX tree is a live view of the application state and can change during traversal — elements can appear, disappear, or be recycled by the app while you are iterating. Always check for null or invalid element handles after each step and handle errors from AX API calls defensively.
Give your agent this knowledge — and 15,500+ more routes
One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp
Need this verified for your stack — or a route we don't have yet?