Conformance vectors for ARB/1.1 and AGA/1.1
Runnable adversarial test vectors for both engines: a clean fixture that must not trip each check, a dirty fixture that must, and a generic runner that compares actual verdicts against a declared expectation. This is the artifact behind the methodology, not a description of it — the thing referenced when either engine is cited to a standards body.
Origin. Feedback on an earlier proposal, via the W3C's AI-agent verifiable-statements community discussion, raised two real gaps: no distinction between "we checked and it failed" and "we could not check," and no recomputable, independent vector set to verify a claimed check against. ARB/1.1's three-state verdicts (detail here) answer the first. This directory existing at all answers the second.
Layout
| Path | Contents |
|---|---|
arb_v11/ | Engine source (bundled, standalone), vectors.json, run.py, and the clean/dirty HTML fixtures + tiny local servers that serve them. |
aga_v11/ | Engine source, vectors.json, run.py, and the clean/dirty source-tree fixtures. No servers needed — static file trees. |
README.md | Full instructions, reproduced below. |
Run the ARB/1.1 vectors
ALLOW_PRIVATE=1 is required only because the fixtures are loopback addresses. A real audit target is never loopback and the flag has no effect on auditing one. fixtures/tls_dirty_cert/ generates a fresh, throwaway self-signed certificate on first run — no private key ships in this package.
Run the AGA/1.1 vectors
No setup required.
Two declared network exceptions
Every fixture is loopback-only except two, both inherent to what the check measures and disclosed in arb_v11/vectors.json's network_exceptions field:
- offsite.wikidata calls the real, public, unauthenticated Wikidata API. Faking this locally would mean testing a fake API, not the check.
- sec.cert_valid's VERIFIED vector uses the real, live
apexclawai.com, because chain-of-trust validity is inherently rooted in real, publicly-trusted certificate authorities. The UNVERIFIED path is tested fully locally, with a self-signed certificate.
What the vectors actually assert
Each row in vectors.json is a triple: a fixture URL or file tree, a check id, and
the single verdict the runner requires — VERIFIED (the evidence was observed and it satisfies
the check), UNVERIFIED (the evidence was observed and it does not satisfy the check), or
UNOBSERVABLE (the evidence needed to decide is absent, e.g. a network failure, or a runtime control
with no evidence supplied). A run that produces any verdict other than the one declared for that row is a
failure of the runner or the engine, not a judgment call to be argued about — there is no partial
credit in either engine's scoring, 0% for a state that misses the bar, full weight for one that
clears it. See
the published rubric for how each verdict feeds into
a score.
By the numbers: ARB/1.1 holds every pillar's weights to a total of 100%, caps its heaviest single pillar at 14%, and a typical vector run against the full ARB/1.1 fixture set completes in under 2 seconds. AGA/1.1's 27 controls map to four published standards, and OWASP's Agentic Security Initiative alone accounts for 44.4% of them (12 of 27) — the largest single reference set. Full citation counts for NIST, ISO/IEC 42001 and the EU AI Act are drawn from the OWASP ASI Top 10, the NIST AI Risk Management Framework, and the EU AI Act itself — the same four references cited on every AGA control, never invented per-page.
Why a runnable vector set matters
A benchmark that only describes its checks in prose is asking to be trusted rather than verified. Publishing the fixtures, the expected verdict for each one, and a runner that exits nonzero on any mismatch moves the claim from "we say it works this way" to "run it yourself." Anyone with a Python interpreter can reproduce every result referenced on this page without contacting ApexClaw, without an API key, and without uploading anything to a third party — the same posture the governance audit's local scanner takes with a customer's own source tree: it runs on their machine and never leaves it.
The adversarial pairing matters as much as the fixtures themselves. A clean fixture that never trips a check proves the engine does not cry wolf; a dirty fixture that always trips it proves the engine actually looks. Testing only one half of that pair is how a benchmark quietly drifts into a rubber stamp — a check that always passes is indistinguishable, from the outside, whether it is faithfully observing the right evidence.
Common questions
What are these vectors for?
Proving that ARB/1.1 and AGA/1.1's checks behave exactly as documented — a clean fixture that must not trip a check, a dirty fixture that must, and a runner that fails loudly on any mismatch.
Do I need network access to run them?
Almost none. Every fixture is loopback-only except the two exceptions above.
Does passing these vectors certify my site?
No. It proves the engine's own logic is internally consistent with what it documents. It says nothing about any site's actual security posture, product quality, or reputation.
Related
The three-state verdict model, the new pillars, and the re-scored 30-site cohort: ARB/1.1 scoring update. Machine methodology: methodology.json.
Licensed CC-BY 4.0. Raw files: README.md, arb_v11/vectors.json, aga_v11/vectors.json.