Before automating any merchant interaction, fetch and parse the merchant's robots.txt; respect Disallow rules for paths your agent intends to access and honor any agent-specific Crawl-delay directives.
Review the merchant's Terms of Service for clauses that restrict automated access, scraping, or bot usage; if the ToS prohibits automation, use only the merchant's official API or MCP server rather than web automation.
Identify yourself honestly in the User-Agent header: include your agent's name, version, and a contact URL so the merchant can reach you if your agent causes unintended load or policy issues.
Implement rate limiting that errs well below what the merchant's infrastructure can handle; treat absence of a rate limit header as a signal to be conservative, not permissive.
If you receive a 429 Too Many Requests or a CAPTCHA challenge, do not attempt to bypass it; back off, reduce request rate, and consider whether the task requires switching to an official API.
Periodically re-read the merchant's ToS (e.g., quarterly) and alert if terms change in ways that affect your automation; maintain a ToS-change audit log.
Known gotchas
ToS for large merchants are frequently updated and may not send active notifications; subscribing to their developer newsletter or API changelog is more reliable than periodic page diffs.
Automated checkout that mimics human behavior to evade bot detection violates most merchants' ToS even if no explicit scraping clause exists; always prefer the merchant's official API or agent-facing interface.
Using another user's credentials or sharing session tokens across accounts is a ToS violation on virtually every platform; each agent session must use its own authorized credentials.
Give your agent this knowledge — and 200+ more routes
One MCP install gives any agent live access to the full route map, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp