SOURCE-LINKED INTELLIGENCE

terms.txt: A Consent and Compensation Protocol for Agentic Web Access

arXiv · AI, language, vision and robotics · article · Sep 10, 2026 · UTC

The open web ran on an unwritten bargain: sites admitted crawlers, and search engines sent visitors back. Public measurements show that bargain breaking under AI crawlers and agents. Automated clients now make up most requests, training dominates Cloudflare-classified crawling, and the largest AI platforms fetch thousands of pages for each visitor they return. The web's common control, robots.txt, cannot express identity, purpose, terms, or price, can be circumvented, and newer alternatives are largely proprietary CDN features. We specify terms.txt, a robots.txt-style file for per-path, per-pu

Read original source ↗ Open in workspace

recordType

paper

region

Global