Ensure agent ToS compliance when automating interactions with merchant websites

Steps

1. Before automating any merchant interaction, fetch and parse the merchant's robots.txt; respect Disallow rules for paths your agent intends to access and honor any agent-specific Crawl-delay directives.
2. Review the merchant's Terms of Service for clauses that restrict automated access, scraping, or bot usage; if the ToS prohibits automation, use only the merchant's official API or MCP server rather than web automation.
3. Identify yourself honestly in the User-Agent header: include your agent's name, version, and a contact URL so the merchant can reach you if your agent causes unintended load or policy issues.
4. Implement rate limiting that errs well below what the merchant's infrastructure can handle; treat absence of a rate limit header as a signal to be conservative, not permissive.
5. If you receive a 429 Too Many Requests or a CAPTCHA challenge, do not attempt to bypass it; back off, reduce request rate, and consider whether the task requires switching to an official API.
6. Periodically re-read the merchant's ToS (e.g., quarterly) and alert if terms change in ways that affect your automation; maintain a ToS-change audit log.