This is our scanner.
Here is what it does.
You are most likely here because AUTHORITY44-Scanner appeared in your logs and you followed the link in its user-agent string. This page tells you who runs it, what it asked your server for, how it behaves when you push back, and the two lines of robots.txt that will stop it for good.
A registered Australian company, with a name on it.
AUTHORITY44 is a digital authority measurement service. The scanner is the part of it that reads your public pages. There is no third-party crawl vendor in between — the company below runs it directly and answers for it.
Public pages, read the way a search engine reads them.
A scan is triggered when someone requests an assessment of a domain — usually the site owner, sometimes an agency or consultant evaluating it. It is not a standing index and it is not a content archive.
robots.txt, before anything else.
Every assessment begins with a request for /robots.txt. Nothing else is requested until that response has been read and applied. If your file tells the scanner to go away, the first request is also the last one.
The homepage, and a small number of pages linked from it.
Ordinary GET requests for HTML that any visitor could load without signing in. The scanner follows public links; it does not guess at URLs, enumerate directories, or probe for files that were never linked.
What it reads out of those responses.
Response status and headers; the title, meta description, and canonical tag; structured data; heading structure; and the internal linking between the pages it fetched. These are the same signals a search crawler or an AI assistant's retriever would use to decide what your site is and whether it can be relied on.
The result is a score and a written report for the person who requested the assessment. Your pages are read to produce that reading — they are not republished, resold, or served to anyone as content.
Honoured properly — including the merged groups.
Plenty of crawlers claim to respect robots.txt and then only match a single exact user-agent line. The scanner implements RFC 9309, which is stricter than that, and the difference matters most in the case people expect us to get wrong.
Group selection is case-insensitive on the product token.
The scanner looks for a group whose User-agent line names AUTHORITY44-Scanner, matched without regard to case. If no group names it specifically, it obeys the User-agent: * group instead. A specific group always wins over the wildcard.
Matching groups are merged, not picked between.
RFC 9309 §2.2.1 requires that where a product token matches more than one group, the rules of all matching groups are combined into one group before they are evaluated. The scanner does this.
In practice that covers the two layouts that trip up naive parsers: several User-agent lines stacked above a single set of rules, and the same token appearing again in a second group further down the file. Both bind. If our token appears anywhere in a stack whose rules say Disallow: /, we are disallowed from your whole site — there is no reading of the file where we escape on a technicality.
Longest match wins; Allow breaks the tie.
Within the merged group, rules are applied per RFC 9309 §2.2.2: the most specific — longest — matching path decides, and where an Allow and a Disallow rule are equally specific, the Allow wins. Path rules are honoured, so you can fence off part of a site without blocking all of it.
A disallowed path is simply never requested.
Disallowed URLs are not fetched, and they are not counted against you for being missing. The scanner does not treat a path it was told to leave alone as a gap in your site.
Where your robots.txt disallows us outright, the scan is refused rather than performed, and the report says the site instructed us not to read it. That wording is deliberately distinct from the one used when a site blocks us at the edge — an instruction we chose to obey is not the same event as a door held shut, and the report does not blur them.
If robots.txt cannot be read, we assume the answer is no.
A 5xx, a timeout, or a network failure is treated as a complete disallow — RFC 9309 §2.3.1.4. If we cannot read your instructions, we do not help ourselves to the benefit of the doubt.
We retry once before concluding that, on the chance the first failure was transient. So a genuinely unreachable robots.txt shows up in your logs as two requests and nothing further — not one, and not a series. After the second failure the assessment ends.
A 4xx is treated as no robots.txt existing at all, which per §2.3.1.3 means no restrictions. A missing file is a site that never expressed a preference; an unreachable file is a site whose preference we failed to obtain. They are not the same and we do not read them the same way.
Crawl-delay is not honoured today.
Stating it plainly rather than leaving it out: the scanner does not currently read or apply Crawl-delay. It is not part of RFC 9309 and we have not implemented it. If you are relying on it to pace us, it is not doing anything — the request ceiling in Section IV is what actually bounds our load, and a 429 stops us outright.
Fewer than twenty-five pages, then it is gone.
This is the part most worth checking against your logs. A scan is a short, bounded job against one domain — not a crawl that settles in.
A typical assessment fetches fewer than 25 pages.
Counted as requests rather than pages — which is what your logs will show — a complete assessment is roughly 25 to 35 requests. The gap between the two figures is robots.txt and the other non-page reads that go with a scan.
Every scan is triggered by a customer asking for one. Nothing runs on our own schedule, and there is no standing crawl. If a domain is read again later because a rescore was requested, that is another discrete job of the same size, not a crawler that took up residence.
Peak concurrency is five.
Requests go out in fans of at most five parallel fetches, with 500ms between fans. That is the ceiling, not an average, and it is why a full assessment takes tens of seconds rather than arriving as a burst.
If what you are seeing in your logs is heavier than that, faster than that, or still going after a minute, it is not us — or it is a bug worth reporting. Either way the address is at the bottom of this page.
A 429 is a stop signal, not a retry cue.
If your server returns 429 Too Many Requests, the scanner stops fetching from that host. It does not retry into the limit, it does not sit in a back-off-and-try-again loop, and it does not go looking for another route in. The assessment ends with whatever was already read.
You should never see us push against a rate limit. If you do, that is a bug on our side and we would like to hear about it — the address is at the bottom of this page.
One user-agent string. Always the same one.
Everything the scanner does is meant to be visible in your logs. If it were trying to hide, publishing this page would be pointless.
What this costs us, and what it means for your report.
Refusing to impersonate other crawlers puts one question permanently out of our reach. Your robots.txt tells us what your site declares — which crawlers it invites and which it turns away. It cannot tell us what your edge actually enforces. Whether your WAF quietly refuses GPTBot is not observable from outside without arriving dressed as GPTBot, and we do not do that.
So we report what you declare, and we do not guess at the rest. A site whose robots.txt welcomes every AI crawler while its firewall turns them away will read, in our report, as a site that welcomes them — because that is the only half we can honestly see.
If you need the other half, it is already in your possession: the answer is in your own logs. Search them for the crawler you care about and you will know in a minute what no external scanner can tell you without lying about who it is.
You cannot verify us by IP address.
The scanner runs on Cloudflare Workers and egresses from shared infrastructure. We hold no dedicated addresses, so there is no egress range for us to publish — and any range you inferred from your logs would carry unrelated traffic that is not ours. We would rather say so here than hand you an allowlist that does not mean what it appears to mean.
The user-agent string at the top of this page is the identifier we can stand behind today. If you need a harder guarantee than that before allowing the scanner, block it — Section VI, and no score is issued.
Two lines, and we are gone.
Add this to the robots.txt at the root of your site. It is picked up on the next assessment, because the file is read at the start of every one.
That is the whole thing. No form to fill in, no account, no waiting period.
If you would rather keep us out of part of the site only, path rules work as you would expect:
Blocking at the edge works just as well. A firewall rule on the user-agent string, a WAF rule, or a plain 403 will all stop the scanner, and it will not look for a way around any of them.
The report states that the site could not be read, and no score is issued. There is no penalty band, no inferred grade, and no version of our output in which blocking us is made to look like a failing result. Declining the crawl costs you the assessment and nothing else.
A person reads this address, and answers.
Questions about a specific request in your logs, a complaint about our behaviour, or a request to be left alone — all to the same place.
About a request you have seen
Send the hostname, the timestamp with its timezone, and the requesting IP if you have it. That is enough to identify the assessment on our side and tell you who asked for it and what was fetched. We answer within one business day.
doug@digitaldominator.com.auIf you would rather not edit robots.txt
Email the domain and ask us to stop, and we will stop fetching it — no reason required and no argument in return. Editing robots.txt is faster because it takes effect on the next assessment without waiting on us, but it is not the only way.