Cloudflare Pay Per Use: Getting Paid When AI Answers Use Your Content

Every GPTBot fetch used to cost you nothing and pay you nothing. Cloudflare just built a toll booth into that exchange, and the toll booth has a price field you control. The feature is called pay per crawl, though most people searching for it type “pay per use,” because that is a plainer description of what it actually does: a site charges per fetch instead of blocking the fetch outright. If you run a site behind Cloudflare, you can now set a dollar amount on every page an AI crawler touches, and the crawler either pays it or walks away empty handed.

I get asked about this on nearly every client call since Cloudflare flipped AI crawlers to blocked by default earlier this year. That earlier shift was about the gate: allow, block, or charge. This piece goes deeper into the third option specifically, because charging is the one most business owners do not understand and most need to think through carefully before they touch it. Blocking is obvious. Allowing is obvious. Pricing a crawler request is a new decision nobody has made before, and the wrong call either leaves money on the table or quietly removes your brand from every AI answer it could have earned.

The question behind the question is usually some version of fairness. A site spends years building an archive, a knowledge base, or a product catalog. An AI crawler reads the whole thing in seconds, folds it into a model, and a user somewhere else gets an answer built partly from your work without ever visiting your domain. No click, no ad impression, no lead form. For publishers whose entire business model depended on that click, the math stopped working. Pay per use is Cloudflare's answer to that complaint, built as an actual payment mechanism rather than a legal threat or a blanket block.

That context matters because it explains who this feature was built for first. It was not built with a local HVAC company's service pages in mind. It was built with newsrooms, research databases, and subscription publishers in mind, the businesses whose entire revenue model is reading access itself. Everyone else inherited the same dashboard toggle, which is exactly why so many business owners are now staring at a Charge setting and wondering whether they are missing free money.

Pay Per Use vs Pay Per Crawl: Same Feature, Different Name

Cloudflare's product name is pay per crawl. “Pay per use” is not a separate feature. It is the phrase search engines and AI tools surface when someone types a plain-language question about charging crawlers for access, because “use” describes the transaction better than Cloudflare's own brand name does. When you see “pay per use” and “pay per crawl” in the same sentence, they are pointing at the identical mechanism, a priced gate for AI crawler requests, enforced at Cloudflare's network edge.

The naming confusion matters because people searching for “pay per use” are often trying to answer a business question, not a Cloudflare dashboard question. They want to know whether they can get paid when an AI model reads their page. The answer is yes, conditionally, and the rest of this article covers exactly what that condition is.

Part of the confusion also comes from how many different phrases describe adjacent but distinct ideas. Pay per crawl is Cloudflare's specific product. Content licensing refers to a direct deal negotiated between a publisher and an AI company, outside any marketplace. Pay per use sometimes gets used loosely for both, and occasionally for unrelated ideas like usage-based API pricing that has nothing to do with crawlers at all. When you are researching this topic, confirm which of these three you are actually reading about before you act on any advice, including this article's.

How the Charge Setting Actually Works

Inside Cloudflare's AI Crawl Control dashboard, every AI crawler category gets one of three settings: allow, block, or charge. Charge is the one that turns your content into a metered resource instead of a free one. Set a price for a crawler category, and any request from that category gets a different response than it would have gotten a year ago.

Here is the sequence. A crawler requests a page on your domain. Cloudflare's edge checks the category policy for that crawler before the request ever touches your origin server. If the policy is Charge, Cloudflare returns an HTTP 402 Payment Required response along with the price you set. A crawler enrolled in Cloudflare's pay per crawl marketplace reads that response, pays the stated price automatically, and the fetch completes. A crawler that is not enrolled, or that declines the price, gets nothing. No partial content, no cached summary, no fallback page. The request simply fails and the crawler moves on.

HTTP 402 is not a new status code Cloudflare invented for this. It has existed in the HTTP specification for decades as a reserved, rarely used response meaning payment is required to proceed. Cloudflare is one of the first infrastructure providers to actually wire that status code into a working marketplace at scale, which is part of why this feature gets attention. A piece of the web's oldest, least used protocol just became load bearing.

The transaction happens between Cloudflare and the crawler, not between you and the crawler directly. You set a price inside the dashboard. Cloudflare handles billing, collects payment from enrolled crawlers, and credits your account according to its own payout terms. You never negotiate with OpenAI or Anthropic directly through this flow, and you never see a per-request invoice arrive from a specific AI company. From your side, it looks like a single number in a Cloudflare dashboard, a price you set once and a balance that moves when enrolled crawlers pay it.

That also means you are trusting Cloudflare as the counterparty for every transaction. If a dispute arises over whether a fetch happened or whether the right price applied, you are resolving it through Cloudflare's support and billing systems, not through a direct relationship with the AI company reading your content. For a publisher used to negotiating licensing terms directly with a counterpart, this intermediary structure is a meaningful change in how the relationship works, even though it removes the friction of negotiating dozens of separate deals one at a time.

Verification Comes Before Price

None of this works unless Cloudflare can tell a real crawler from one pretending to be real. Any bot can set its user agent string to claim it is GPTBot. Cloudflare does not take that string at face value. It verifies crawler identity cryptographically before applying whichever policy you chose, allow, block, or charge, which means the price you set only ever reaches the crawler it was actually built for.

Our field guide to GPTBot, ClaudeBot, and PerplexityBot covers why this verification step matters on its own. User agent strings are case sensitive and trivially spoofed, and a site relying purely on robots.txt string matching can be fooled by a bot that is not what it claims to be. Cloudflare's cryptographic check closes that specific gap at the edge, for the crawlers it has verification agreements with. A spoofed request claiming to be GPTBot gets routed to your fallback rule for unverified traffic, not to the price you set for the real thing.

Why Most Sites Should Not Flip This On Yet

Here is the part that surprises people who come to this feature expecting a new revenue line. Pay per crawl only produces revenue when an AI company has actually enrolled its crawlers in Cloudflare's marketplace and agreed to pay the prices site owners set. Most AI companies have not done that broadly, for most of their crawler traffic, as of this writing. That means a Charge setting configured today frequently behaves closer to a soft block for whichever crawlers have not signed up for the marketplace yet. The crawler hits the 402 response, has no mechanism to pay, and simply does not fetch the page.

That is a meaningfully different outcome than the one implied by getting paid when AI answers use your content. Right now, for most sites, setting Charge on a crawler category mostly removes that crawler, the same functional result as Block, minus the small subset of enrolled crawlers that do pay. The marketplace is real and it is live, but it is young, and the gap between theoretically monetizable and actually generating payments is wide for a typical business site today.

This is the detail that should shape your decision more than anything else in this article. If your goal is citation, being named in a ChatGPT or Perplexity answer that sends you a qualified visitor, Charge is the wrong lever to pull for most crawler categories. If your goal is protecting content that has standalone commercial value independent of any inquiry it generates, Charge is worth a serious look, understanding that today it functions more as a selective gate than a reliable income stream.

There is a second-order effect worth naming directly. If your competitor leaves their content open under Allow while you set a price under Charge, and most crawlers in that category have not enrolled to pay, the practical outcome is that your competitor gets cited and you do not, for the exact same query, in the exact same AI answer. The crawler did not choose your competitor over you. It simply could not read your page because you priced it out of a marketplace most of its traffic has not joined yet. That is a real cost, and it is easy to miss if you evaluate the Charge setting purely as a potential revenue line without weighing the visibility you are trading away to get it.

Content as Product vs Content as Vehicle

The decision comes down to one question, the same one we raise in our broader piece on Cloudflare's crawler default change: is your content the product you sell, or is it the vehicle that earns attention and inquiries for something else you sell?

A services business, a local provider, or a SaaS company with a content-led blog sits in the second category. The blog post exists so that a prospective buyer finds it, trusts it, and books a call. Citation in an AI answer is the entire point of having written the page. Charging a crawler a few cents per fetch, in a marketplace most crawlers have not joined, trades away free distribution for a payment that will not materialize for the vast majority of requests. That math does not favor Charge.

A paid research firm, a subscription news publisher, or any business where the content itself is the thing a customer pays for sits in the first category. For that business, every free crawl is a free copy of the product handed to a company that may resell the derived value through a chatbot subscription. Charge is a defensible, even obvious, position for that content, with the same caveat that today's payout will be partial until more AI companies enroll.

Setting a Price: What the Dashboard Asks You to Decide

If you decide Charge fits your content, AI Crawl Control lets you set a price per crawler category rather than one blanket number for the whole domain. This matters because training crawlers and retrieval crawlers behave differently. A training crawler like GPTBot gathers content that shapes a future model update, an effect that shows up on a delay of months. A retrieval crawler like PerplexityBot fetches a page to answer a specific question happening right now, an effect that is immediate.

Some publishers price these two categories differently on purpose. A low or zero price for retrieval crawlers preserves the chance of same-week citation, while a meaningful price for training crawlers reflects the ongoing commercial value of letting a model train on the content indefinitely. This is a defensible split, but it only works if you set it deliberately. Leaving a category on Cloudflare's fallback rule, because you never opened the dashboard, means a decision got made for you.

Pricing itself is still an emerging practice. There is no established market rate yet the way there is for digital ad impressions, so most site owners are setting prices based on their own judgment about what a single page fetch is worth to their business, not on an industry benchmark. Expect that to firm up as more AI companies enroll and more publishers compare notes.

What Happens to Pages You Have Already Priced

A priced page does not disappear from the open web. A human visitor using a normal browser never sees an HTTP 402 response and never pays anything. The Charge setting only applies to the specific crawler categories you assigned it to inside AI Crawl Control. Search engine crawlers, for instance, are typically governed by a separate policy than AI training and retrieval crawlers, so a Charge setting aimed at GPTBot does not touch how Googlebot indexes your page for traditional search results.

This separation is worth confirming directly in your dashboard rather than assuming it. AI Crawl Control groups crawlers into categories, and it is possible to misconfigure a category boundary in a way that catches more traffic than intended. Before setting a price on any category, open the list of crawlers assigned to it and read through the names. A category labeled broadly enough to include a crawler you did not mean to price is a common, avoidable mistake.

It is also worth checking what a priced page looks like to a crawler that declines to pay. Some site owners expect a partial preview, a truncated summary, or a paywall-style teaser, the kind of experience a news site might show a human reader who has hit a subscription limit. That is not what happens here. The crawler that will not pay gets the 402 response and nothing else, no partial content negotiation. If you want crawlers to see a public summary regardless of payment status, that has to be built separately, often through a distinct public-facing page or through the kind of structured summary an llms.txt file provides.

Where Robots.txt and llms.txt Fit Into This

A Charge setting in AI Crawl Control overrides your robots.txt whenever the two disagree. Robots.txt is a request a well-behaved crawler reads and honors. Cloudflare's edge decision is enforcement, applied before the request reaches your server or your robots.txt file at all. If your robots.txt says GPTBot is welcome but AI Crawl Control has that category set to Charge with a price no crawler is paying, the edge setting is what actually governs the outcome, not the file you wrote.

This is why pairing a pricing decision with a clear llms.txt file matters even when you charge for access. An llms.txt file gives AI models a structured, plain-language summary of your site, your organization, and your key pages, and it is a reasonable place to state your access terms in human-readable form alongside the machine-enforced Cloudflare policy. It will not override a 402 response, but it removes any ambiguity about why a crawler got turned away and what it would take to be let back in, which matters if you are negotiating a direct licensing relationship outside the marketplace entirely.

A Decision Checklist Before You Touch the Charge Setting

Work through these in order before setting a price on any crawler category.

1. Confirm your site sits behind Cloudflare. Check your nameservers or look for a cf-ray header in your site's response. If you are not on Cloudflare, this specific feature does not apply to you yet.
2. Answer the product-versus-vehicle question honestly. Does a prospective customer pay you for reading this exact content, or does this content exist to generate an inquiry for something else you sell?
3. Separate training crawlers from retrieval crawlers. Decide whether the immediate citation value of a retrieval crawler outweighs the delayed training value you would be pricing away.
4. Check your robots.txt for contradictions. Since the Cloudflare edge setting wins any disagreement, a contradictory robots.txt rule is not protecting anything. Fix the mismatch either way.
5. Set explicit prices rather than leaving categories on default. A category left untouched inherits Cloudflare's fallback rule, which is not a choice you made on purpose.
6. Revisit the setting quarterly. As more AI companies enroll in the marketplace, the payoff from Charge will change. A setting that behaves like a soft block today may start generating real payments later.

Most AEO Hunt clients land on step two and stop there, because the honest answer for a services business or a SaaS company with a content-led blog is that the content is a vehicle. For those businesses, the right move is Allow for every crawler category that matters to their niche, not Charge.

Common Mistakes We See With This Setting

The most frequent mistake is not a wrong price. It is no decision at all. A site owner opens AI Crawl Control once, reads the three options, feels unsure which one applies, and closes the tab without picking anything. Every category left untouched inherits Cloudflare's fallback rule, which for most new domains is Block. That means the most common outcome of indecision is an accidental block, not an accidental charge, and it is the opposite of what most business owners actually want.

A second mistake is treating Charge as a one-time setup rather than a setting that needs revisiting. The pay per crawl marketplace is still adding participating AI companies. A price that produced zero payments six months ago might start producing real revenue as more crawlers enroll, and a business that set Charge once and never checked back has no way of knowing that changed.

A third mistake is pricing an entire domain uniformly when the content on it is not uniform. A site with both a public marketing blog and a gated research archive should not apply the same policy to both. The blog wants Allow, because its job is to generate inquiries. The archive might warrant Charge or Block, because its job is to be the product. Applying one blanket setting across a mixed-content domain sacrifices the upside of whichever half you got wrong.

What This Means for the Broader Web

Step back from any single site's decision and the pay per crawl marketplace represents something larger: an attempt to build a real payment rail for content access that previously had none. Search engines paid publishers in traffic. AI answer engines, by design, often answer the question directly and never send the visitor anywhere. That broke the implicit trade that kept the open web funded for two decades, content for traffic, and nobody had built a working substitute until Cloudflare wired HTTP 402 into a verified, enforced marketplace at the infrastructure layer.

Whether this becomes a meaningful revenue stream for most publishers, or stays a niche tool for a subset of premium content, depends entirely on how many AI companies choose to enroll their crawlers and pay the prices site owners set. That is a business decision happening at companies like OpenAI, Anthropic, and Perplexity, not something Cloudflare alone controls. What Cloudflare has built is the plumbing. Whether the money flows through it at scale is still an open question, and one worth revisiting every few months rather than assuming today's answer is permanent.

The Practical Takeaway

If someone on your team asks whether you can get paid when ChatGPT reads your blog, the honest answer is: technically, yes, through Cloudflare's pay per crawl marketplace, but only from the subset of crawlers that have actually enrolled to pay, and only if you are willing to trade away the free citation that a competitor leaving their content open will keep earning in the meantime. For the overwhelming majority of businesses working with an AEO agency, the content exists to be found, not to be sold by the fetch. Allow remains the right default. Charge is a tool for a specific kind of publisher, used deliberately, not a free money lever every site should reach for the moment it appears in a dashboard.

At AEO Hunt, we review this setting as part of every technical AEO audit we run, alongside robots.txt, schema validation, and crawler access more broadly. Getting the Charge decision wrong in either direction, pricing away citations you needed or leaving a paywall-worthy archive sitting open and unpriced, is an easy mistake to make from inside a dashboard you only opened once.