Locate on-screen text in a screenshot for automation using Tesseract OCR when no accessibility tree is available

domain: desktop-automation · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. Capture a screenshot of the target region, then run `tesseract <screenshot> <outputbase>` for basic recognition (English + page segmentation mode 3 by default)
  2. Select the right page segmentation mode with `--psm` when the default general-layout assumption doesn't match your UI (e.g. a single line of text vs. a full-screen layout)
  3. Set the recognition language(s) with `-l eng` (or `-l eng+deu` etc.) to match the UI's actual language rather than relying on the English default
  4. Choose the OCR engine mode with `--oem 1` (LSTM/neural, generally more accurate) vs `--oem 0` (legacy engine) depending on your Tesseract build's available models
  5. Request `hocr` or `tsv` output instead of plain text to get per-word bounding boxes and confidence scores (e.g. `x_wconf`), which is what lets automation code click at a specific word's coordinates rather than just reading the text
  6. Treat this as a fallback strategy behind accessibility-API or DOM-based locators — reserve it for legacy/non-instrumented apps where no accessible tree or automation ID exists

Known gotchas

Related routes

Use operating system accessibility tree APIs (AX APIs) to build resilient desktop UI automation that survives visual changes
developer.apple.com · 5 steps · unrated
run automated accessibility scans in playwright using @axe-core/playwright's axebuilder
github.com/dequelabs/axe-core-npm · 5 steps · unrated
Use Playwright ARIA snapshot testing to detect accessibility-tree regressions
playwright.dev · 5 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans