Our AI training reservation was addressed to nobody. The line Content-Signal: search=yes, ai-input=yes, ai-train=no sat in the robots.txt file of wppoland.com exactly once, inside the User-agent: * group, with twenty named crawler groups underneath it. Under RFC 9309 none of those twenty groups inherits anything from the star group.
GPTBot, ClaudeBot, PerplexityBot, Google-Extended and the rest of the list read only their own sections, and the line was not in any of them. Every crawler we had bothered to name by hand was the one crawler that could not see what we had written for it.
We are writing this up because it is our file and our mistake. We had reviewed robots.txt for whether it accidentally blocked something, and never for whether a declaration in it reaches its addressee at all. Those are two different questions, and only the first one has a tool that answers it.
What was actually in the file
The structure looked the way most files we have seen look: one general group at the top, then a list of named bots with extended Allow rules for machine endpoints.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Disallow: /cdn-cgi/
Sitemap: https://wppoland.com/sitemap-index.xml
Agentmap: https://wppoland.com/.well-known/ai-catalog.json
User-agent: GPTBot
Allow: /
Allow: /llms.txt
Allow: /facts.json
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /It looks tidy. The signal is there, the named groups are there, the agent endpoints are listed out separately.
The problem is that those two halves never touch each other. Reading the file as a document, you see a policy at the top and a set of permissions below it, and the eye supplies a relationship between them that the parser does not. A parser does not read documents. It matches one token against a list of names, keeps the block that wins, and throws the remainder of the file away before it has read a single directive.
Why one line does not cover the whole file
The client compares its product token against the names in the User-agent lines, case-insensitively, and applies one group: the one that names it. The asterisk is not the least specific pattern, it is the fallback, and RFC 9309 reaches for it only when no group names the client at all. Once a named group exists, the client applies only that group’s rules and the star group stops applying to it entirely.
That last sentence is the whole bug. It reads as obvious in the specification and stops being obvious the moment you hold it next to a file you wrote yourself, because a file you wrote yourself carries your intent, and the intent is invisible to the reader that matters.
This is not a special case for Content Signals. The same rule has always governed Allow and Disallow, which is exactly why a file where somebody added Disallow: /wp-admin/ only to the star group protects nothing against a bot with its own section further down.
The difference is what happens next. With Disallow the consequence is visible: the page shows up in the index or it does not, and a week of indexing data tells you which. With Content-Signal nothing is visible, in either direction. A rights declaration has no feedback loop, so a wrong one and a right one produce the same observable history for as long as you care to watch.
Content Signals is a Cloudflare proposal, not part of RFC 9309, and Cloudflare turns the policy on by default in its managed robots.txt file. The group mechanism the line lives inside, though, is entirely an RFC 9309 mechanism, so the one-group rule binds it just like everything else in that file. A new directive in an old container inherits the container’s rules, including the ones the person adding the directive never thought to look up.
Three signals, and what they do not promise
The specification defines three signals, each taking the value yes or no:
| Signal | What it covers |
|---|---|
search | building a search index and returning results, meaning links and short snippets |
ai-input | feeding content into an AI model at answer time, meaning RAG and answer grounding |
ai-train | training and fine-tuning models |
Our set, search=yes, ai-input=yes, ai-train=no, is a deliberate choice, and the trade-off inside it is worth stating plainly. We want to be cited in generative answers, because that is now one of the channels through which somebody finds us. We do not want to hand over the corpus for training, because a trained model carries the text forward with no path back to the site it came from. Saying yes to the first and no to the second is us accepting that the same crawler may be doing both jobs on different days, and that we are relying on it to tell the days apart.
No signal at all means a third thing, different from no: the site operator neither grants nor refuses permission by this route. Silence is not a refusal here, which is the whole reason the line exists.
None of these signals blocks anything. It is a line of text in a file the client may fetch or not, read or not, and respect or not. The server answers the request regardless of what that file says.
Blocking is a separate layer, evaluated ahead of the application, and you build it with WAF rules, rate limits or bot identity verification. Those cost something: they run on every request, they can be wrong about a human, and they need maintenance as the identity signals change. A declaration costs nothing and enforces nothing, which is a reasonable trade as long as you know which of the two you just shipped. The signal is worth exactly as much as an unambiguous message sent to the right address. Ours was unambiguous and sent to the wrong one.
Check your own file with one command
It takes a few seconds and works on any domain:
curl -s https://yourdomain.com/robots.txt \
| awk 'tolower($0) ~ /^user-agent:/ {ua++} tolower($0) ~ /^content-signal:/ {cs++} END {print "User-agent groups: " ua "\nContent-Signal lines: " cs}'One caveat before you act on the output: this counts User-agent lines, not groups, and the ABNF in RFC 9309 allows several User-agent lines to open a single group, so a file that stacks names that way will report a shortfall it does not have. If your file does that, read the second command’s list rather than the first command’s arithmetic.
If the group count is higher than the signal count, the difference tells you how many groups cannot see your reservation. Ours read twenty one to one.
The more precise variant prints the names of the groups where the signal is missing:
curl -s https://yourdomain.com/robots.txt \
| awk '/^[Uu]ser-agent:/ {if (name != "" && !sig) print "no signal: " name; name=$2; sig=0} /^[Cc]ontent-[Ss]ignal:/ {sig=1} END {if (name != "" && !sig) print "no signal: " name}'The second command is the one worth wiring into a pipeline, because its output is a list to fix rather than a number to interpret.
How to fix it
The fix is boring: repeat the Content-Signal line in every group it should cover. There is no shorthand, no compact form and no inheritance you can switch on.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Disallow: /cdn-cgi/
User-agent: GPTBot
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
Allow: /llms.txt
Allow: /facts.json
User-agent: ClaudeBot
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
User-agent: PerplexityBot
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /We shipped that change on 18 September 2026, the day this post went out. The signal now stands in 18 groups out of 23, and the five we skipped are scrapers that already carry a Disallow from us, where a permission statement would only muddy what we are saying to them.
While making the fix you have to settle one thing the repeated line does not settle by itself: whether the same set of signals fits every group. For an indexing crawler and for a crawler collecting a training corpus the answer differs, and a file with twenty identical lines suggests nobody thought about it. A group where search=yes makes no sense, because the client builds no index, deserves its own line, and writing that line is the point at which the file stops being a copy-paste and starts being a policy.
The second thing we were missing: groups per role
The same reading of the file turned up that we had a ClaudeBot group and neither of the other two: Claude-User, which handles user-initiated fetches, and Claude-SearchBot, which handles search grounding. Those are three different roles and three different intents on the client side. A bot fetching a page because a human just pasted its address into a conversation is doing something other than a bot building a corpus, and it makes sense to answer it differently.
The effect of the missing groups is exactly the one above, seen from the other end: a client with no section of its own falls back to the star group. In our case that meant Claude-User saw the correct signal, because the star group had it.
So in a file where the signal failed to reach twenty named bots, it did reach the unnamed ones. Precisely the opposite of what we intended, and produced by the same rule working exactly as written.
Why this bug survives review so easily
Three reasons, all structural, none of them carelessness.
A robots.txt file reads top to bottom and the general section looks like the parent. Putting anything into it triggers the intuition you carry from CSS or from server configuration, where a general setting is the default and a specific one overrides it. RFC 9309 has no overriding. It has the selection of one group and the discarding of the rest of the file.
Second reason: no tool reports it. The robots.txt testers check whether a given URL is allowed for a given bot, and they know nothing about Content-Signal, because it is not an RFC directive. The line passes as a comment in the validator’s eyes and as policy in a human’s, and both readings are internally consistent.
Third, and the important one: a rights declaration has no return signal. Place a Disallow wrong and a week later you see it in the indexing report. Place a Content-Signal wrong and nothing happens that you could observe, and the absence of an event looks identical in both cases. This class of defect has to be found by reading the file, not by watching outcomes, which is an uncomfortable thing to schedule because reading has no trigger.
Lines outside groups mislead, because those really are global
There is one thing in the file that actively suggests the wrong intuition. Alongside User-agent groups, robots.txt has records that are independent of groups, and Sitemap is one of them. In our file it sits right after the star group:
Sitemap: https://wppoland.com/sitemap-index.xml
Agentmap: https://wppoland.com/.well-known/ai-catalog.jsonSitemap applies to the whole file no matter where you put it or how many groups sit below it. Nobody repeats it twenty times and nobody has to.
A person who knows how Sitemap behaves, and who sees Content-Signal placed in the same region of the file two lines above it, has every right to assume the line behaves the same way. It does not. One is a global record, the other a group-scoped directive, and nothing in the syntax signals the difference. Both are a two-part line with a colon, sitting next to each other.
The practical takeaway when reading somebody else’s file: indentation and ordering mean nothing, the only thing that means anything is whether a given directive is defined as group-scoped. Allow, Disallow and Content-Signal are. Sitemap is not.
What this post does not settle
Two things stay out of scope on purpose, because they are not ours to settle.
The first is the legal effect of the reservation. Content Signals refers to a rights reservation expressed in copyright law, and text to that effect does sit in the comment at the top of our file. Whether and when such a reservation holds is for a lawyer, not for the person editing robots.txt. Our contribution is technical only: if the declaration is to carry any weight, it has to at least reach the client it addresses, and ours did not.
The second is the behaviour of individual operators. We did not measure which crawler reads Content-Signal, which one respects it, and what any of them do with inconsistent signals across groups. Without our own measurement we have nothing to say here, and we are not going to repeat someone else’s numbers.
The fix we describe is worth making regardless, because it repairs a file that says something other than its author intended, and that is a defect in itself.
What to do about it on the process side
Checking the signal’s reach is a one-off job of a few minutes, but keeping it right is not a one-off. Every new group added to the file, and they get added regularly, because the list of AI crawlers grows month over month, is a new group without a signal until somebody remembers it. That is why the second command above belongs in your tests, not in a note. A rule that checks itself survives; a rule written in a document survives until the next person edits the file without having read it.
It also helps to remember what share of traffic we are talking about. In our August 2026 measurement, written up in the post on bot traffic to a small site, 72% of requests did not come from a browser. The robots.txt file is the only place where we talk to that majority at all, and the only thing that majority knows about our terms. If a sentence in it is addressed to the wrong group, that is not a soft error, it is silence.
Which leaves a question worth asking about your own file before you close this tab: when did anyone last read it as a machine would, one group at a time, rather than as a page?
Short checklist
- Count the
User-agentgroups and theContent-Signallines in your file. If the numbers differ, the difference is the number of groups with no reservation. - Repeat the line in every group it should cover. Repetition is the correct form here, not a duplicate.
- Decide deliberately whether every group gets the same set of signals. Twenty identical lines usually means nobody decided.
- Check whether you have separate groups for the roles of one provider, for example user-initiated fetches and corpus collection.
- Wire the check into a pipeline. The crawler list grows and every group added starts out without a signal.
- Do not tell yourself the signal blocks anything. Blocking is a layer in front of the application, this is a declaration.
