Conformance
An all-green result means nothing unless the checker could have failed
A conformance result is only meaningful if the instrument that produced it can be shown to bite. The method below records a verdict per check, a controlled cause for every non-verdict, and the exact versions and digests needed to reproduce the run.
Per-check verdicts
Five outcomes, and only one of them is a pass. The distinction between the last three is where most conformance reporting quietly fails.
| Verdict | Meaning |
|---|---|
| PASS | The check ran and the property held. |
| FAIL | The check ran and the property did not hold. |
| NOT-EXERCISED | No vector reached this check. It is not a pass, and it may not be reported as one. |
| INCONCLUSIVE | The check ran and could not decide. A stated cause from a controlled list is required. |
| VOID | The instrument itself failed. Never reported as a result about the system under test. |
What a reproducible run records
Exact versions
The checker version and the corpus version, so a later run can be compared against this one rather than against a moving target.
Artifact digests
A digest of every artifact under test, so the thing that was checked is identifiable afterwards.
Positive controls
Cases that must pass. If a positive control fails, the run is void — the instrument is broken, and its other results say nothing.
Must-fail vectors
Deliberately defective inputs that the checker is required to reject. If a must-fail vector passes, the checker is not biting and the run is void.
Verdict
Reproducible conformance method
CandidatePer-check verdicts of pass, fail, not-exercised, inconclusive or void, with a controlled cause for every non-verdict, positive controls, must-fail vectors, the exact checker and corpus versions, and artifact digests — so that an all-green result means something only when the checker was demonstrably capable of failing.
Limitation · Reproducible by design. Not yet reproduced by an independent party, which is the only thing that would move this to proven.
On the W3C affiliation
ApexClaw's founder, Julian Joseph, chairs the W3C Agent Conformance and Benchmarking Community Group. Participation in a community group is not endorsement: the W3C does not certify ApexClaw and does not endorse it. Community Group work is not a W3C Standard and does not carry W3C consensus.