How an AI agent actually reads a website
Not a browser, not a model's guess — a fixed set of server-side requests and deterministic checks, exactly as the published rubric defines them.
Published · updated · by Julian Joseph, founder
Get an Agent Trust Gap BriefWhat does an agent actually fetch when it reads a page?
Per the published scope, one thing: "one URL plus the site-wide files at its root: robots.txt, llms.txt, sitemap.xml, /.well-known/security.txt, /.well-known/ai.txt." Six requests total, all server-side, none requiring a browser or JavaScript runtime.
Does the reading agent execute JavaScript?
No signal in the rubric assumes it does. The ma.no_js_content check exists specifically because, per its own fix text, "agents rarely execute your JS bundle" — content only earns credit if it is present in server-rendered HTML.
What counts as "structured" to the engine, versus just readable prose?
Six of the 36 checks are JSON-LD specific: whether any block exists, whether it declares an Organization or Product, an FAQPage, a BreadcrumbList, an Article-family type, and whether a sameAs identity link is present.
| Pillar | Weight (of 100) |
|---|---|
| Machine Access | 18 |
| Structured Data | 18 |
| Answer Extractability | 16 |
| Evidence & Citability | 14 |
| Entity Clarity | 12 |
| Agent Interaction Surface | 12 |
| Governance & Trust | 10 |
Source: methodology.json, spec date 2026-08-24. Weights sum to 100.
Which signals are visible for free, and which sit behind the full report?
9 of the rubric's 36 checks are marked in_free_tier: true — reachability, AI-crawler blocking, no-JS content, any JSON-LD, question-form headings, a machine-readable date, About/Contact reachability, a booking link, and a stated AI-access policy. The other 27 require the full report.
| Check | Title | Pillar |
|---|---|---|
ma.reachable | Page returns 200 to an agent user-agent | Machine Access |
ma.ai_crawlers | AI crawlers not blocked in robots.txt | Machine Access |
ma.no_js_content | Primary content present without JavaScript | Machine Access |
sd.any | JSON-LD structured data present | Structured Data |
ex.question_heads | Question-form headings | Answer Extractability |
ev.dated | Content carries a machine-readable date | Evidence & Citability |
en.about_contact | About and Contact reachable from this page | Entity Clarity |
ag.booking | Booking endpoint linked | Agent Interaction Surface |
gv.ai_policy | Stated policy for AI/agent access | Governance & Trust |
What is the difference between a PASS, a PARTIAL, and an UNKNOWN verdict?
A PASS earns the full weight, a PARTIAL earns half. An UNKNOWN — a check that genuinely could not be evaluated — is removed from the denominator entirely, per the published scoring rules, rather than scored as a fail.
Does this mean a page's score can differ from one crawl to the next?
Only if the page itself changes, or the rubric version does. The spec is dated (2026-08-24) and versioned (engine ARB/1.0); the changelog records exactly one entry so far — the rubric's first publication.
Why weight the pillars unevenly instead of scoring every check the same?
Because a missing sitemap and a missing signed receipt are not the same size of gap. Machine Access and Structured Data each carry 18% of the total; Agent Interaction Surface carries only 12%, reflecting how early that surface still is.
- Reachability and structure come first because an agent that cannot fetch or parse a page cannot act on anything else on it — hence 18% and 18% respectively, the two largest pillars.
- Evidence & Citability and Entity Clarity sit in the middle: useful for trust and attribution, but meaningless if the page was never reachable in the first place.
- Agent Interaction Surface and Governance & Trust carry the smallest weights today, 12% and 10% — not because they matter less, but because the rubric dates from 2026-08-24 and these are the newest categories of signal.
Does the rubric read the same six site-wide files for every page on a domain?
Yes. robots.txt, llms.txt, sitemap.xml, /.well-known/security.txt and /.well-known/ai.txt are fetched once from the domain root regardless of which specific URL is being scored, alongside the one page itself. A site publishing those five files correctly benefits on every page it has, not only the one that was audited.
That also means a domain-level gap — a missing llms.txt, say — shows up identically on every page scored on that domain, while the page-specific checks (headings, dates, JSON-LD, a booking link) vary page to page. Two pages on the same site can therefore share half their findings and differ completely on the other half.
Method note
Everything on this page is read directly from the published rubric at methodology.json (engine ARB/1.0, spec date 2026-08-24) — no behavior is inferred beyond what that document states about scope, checks, and scoring.
The rubric's Governance & Trust pillar cross-references OWASP Agentic Top 10, NIST AI RMF, ISO/IEC 42001 and the EU AI Act.
By Julian Joseph, Founder, ApexClaw. Every figure on this page is recomputed at build time from a named source file and date — see the method notes above. Reviewed against the claims policy: sourced, first-party, or labelled.