There is no single settled answer. Today, machine readers often access public web content without paying, some publishers negotiate individual licenses, and new technical mechanisms for automatic payment are being tried. Who should pay, for what, and how much remains an open question.
Why the question exists
The web's economics grew around human visitors: a person arrives, sees an ad or hits a paywall, and the site earns something. When a machine reads the page instead, the usual triggers may never fire. The content is used, but nothing in the system registers that the use had a value.
What exists today
- Open access by default. Public pages can generally be fetched by anyone or anything. A site can ask machines not to (robots.txt) but can't force them to comply.
- Blocking and limiting. Many sites restrict declared automated readers at the network edge.
- Individual licensing. Some publishers have negotiated direct agreements. It's a negotiation-by-negotiation model, out of reach for most sites.
- Experiments in automatic payment. HTTP has long reserved a "402 Payment Required" status. Open protocols such as x402 and CDN-level pay-per-crawl tools explore using it so machines can pay per request.
The hard questions
- Who owes what? Reading a page to answer a question, quoting it, and using it for training may deserve different treatment.
- What's a fair price? A flat per-page fee treats a throwaway page and a landmark investigation the same.
- How is the price set? Someone has to decide what a page is worth, and by what evidence.
- Who is a legitimate payer? Agents can impersonate browsers or one another.
- What's lost if everything is paywalled? Search, research, accessibility and archives depend on openness.
Where Widgetail's thinking comes in
One of those questions, what a page is worth, is one we think can be answered with evidence. Real human engagement with a page (how deeply people read it) is a signal of its value that a single flat fee can't capture. Widgetail measures engagement today with pageviews, unique sessions and scroll depth, and is building a content value score that machine-access pricing could be based on.
What to do in the meantime
- Decide what you'd actually want: excluded, credited, or paid, and for which uses.
- Check your server logs for machine traffic your analytics can't see.
- Keep an eye on emerging standards rather than betting on one.
- Start measuring which of your pages are truly valuable, because whatever pricing emerges will need that evidence.
For a fuller tour of the terrain, see AI agents and websites, or the difference between AI crawlers and AI agents.