AI agents and websites: the web has new readers.

What AI agents are

An AI agent is software that uses an AI model to pursue a goal on someone's behalf: researching a question, comparing products, summarizing a page, filling in a form. To do that, many agents read websites, sometimes hundreds of pages in a few seconds.

They sit alongside older kinds of automated readers. Search crawlers index pages so people can find them. Scrapers copy content in bulk, for many different purposes. AI crawlers collect content for AI systems. The lines between these blur, and we untangle them in AI crawlers vs AI agents.

How agents differ from ordinary visitors

A human visitor loads a page in a browser, sees the design and the ads, scrolls, reads, and leaves. A machine consumer behaves differently in ways that matter for anyone who publishes:

• No ads seen. Many machine readers fetch the text and never render an advertisement, so no impression is produced.
• Often no JavaScript. Many fetch raw HTML and never run a page's scripts. That includes most analytics scripts, so these visits are usually invisible to browser-based analytics.
• Volume. A single agent can read far more pages than a person would in the same time.
• Different purpose. The content may be summarized, quoted, used to answer a question, or used to train or ground a model. The reader may never visit the source at all.
• Self-declared identity. Well-behaved automated readers announce themselves in the request's user-agent string. Nothing forces them to.

How machine consumption changes the web's economics

For most of the web's history, the deal was simple: people visit a page, and the visit is worth something, usually through advertising or a subscription. That deal assumes a human on the other end.

When a machine reads the same page, the assumptions break down. There may be no ad impression, no subscription, no visit to count and no relationship with the reader. The content still carries value, because someone or something is using it, but the usual mechanisms for capturing that value don't fire.

What's being tried today

Site owners currently have a limited set of tools for managing machine access. None of them is a complete answer:

• robots.txt. A long-standing convention for telling crawlers what they may fetch. It's a request, not enforcement: it works only when the crawler chooses to respect it.
• Blocking at the network edge. Rules that identify and block or limit automated traffic. Effective against declared bots, harder against ones that don't announce themselves.
• Licensing. Some publishers negotiate direct agreements. That works one deal at a time and is out of reach for most sites.
• Payment protocols. HTTP has had a "402 Payment Required" status code reserved since its early days. Open protocols such as x402 and CDN-level "pay per crawl" tools are experiments in using it, so that a machine can be charged for access automatically.

What Widgetail is exploring

Widgetail's thesis is that access should be priced by value, not a flat fee, and that the value of a page can be measured from how real visitors engage with it. It's a different angle on the same problem: rather than only deciding who may read, price what reading is worth.

Here is what it looks like when a declared AI crawler is sent a payment challenge, using illustrative activity:

Measuring real engagement, the evidence that value can be priced from, is covered in what Widgetail measures. The full journey is on the roadmap.

Open questions

• Who should pay, and for what? Access, use in an answer, use in training: these may deserve different treatment.
• How is value priced fairly? A flat fee ignores that some pages are worth far more than others.
• How do you tell agents apart? Legitimate agents versus impersonators, and helpful automation versus abuse.
• How is privacy protected? Any measurement of human engagement has to stay anonymous and aggregate.
• What about the open web? Charging for access has costs. Search, accessibility and archiving all rely on open reading.

We don't claim to have answers to all of these. We think measuring real value is a necessary starting point for any of them, and we're building toward it in the open. For the wider economic question, read who pays when AI uses web content.

If you build agents or AI products

Developers and AI builders will be on the other side of whatever mechanisms emerge. A few practical habits will age well: identify your agent honestly in the user-agent string, respect robots.txt, don't rely on scraping as a permanent free supply, and expect that some sites will respond to machine access with a payment request rather than content. If you'd like to shape how that works from the builder's side, join early access.