What a deterministic audit catches that a language model can't

What makes an audit control "deterministic" rather than a model's opinion?

Per aga.py's own description, every control reads a specific file and line and returns that exact path as evidence — nothing is inferred or guessed, and the same input always produces the same verdict.

Can you show a case where pattern-matching alone would flag something wrongly?

The tricky_clean fixture contains an AWS-style key, the string DROP TABLE, and rm -rf /var/lib/apt/lists/* — three strings a naive scan would flag. Re-run today (2026-08-24), both relevant controls PASS.

Source: /root/apexclaw-audit/aga_fixtures/tricky_clean/ and tricky_dirty/, scored by re-running aga.run_controls() at build time, 2026-08-24.

Why does tricky_clean pass when it contains the same strings that fail tricky_dirty?

Context the control checks: the AWS key is a published SigV4 example inside a test file, DROP TABLE sits inside a quoted blocklist a guard script uses to refuse it, and the rm -rf targets an apt cache path.

And tricky_dirty — what does it actually get wrong?

A real-shaped AWS key sits in config.py, not an environment file; a JWT-shaped secret sits in an .env with no .gitignore protecting it; and release.sh runs rm -rf /, an unconditioned rsync --delete, and a force-push, unquoted.

Why can't a single regex, or a single model prompt, reliably tell these apart?

Telling them apart needs several facts at once: file path, whether the match sits inside quotes or a comment, and whether the containing .env is git-ignored. A deterministic control checks all of them explicitly, every run.

What is the practical takeaway for someone building an agent system?

Two identical-looking strings can be a false alarm or a real leak, depending on file, quoting and ignore-state a keyword scan discards. A control naming the exact file, line and reason is checkable by a human in seconds either way.

How did the two controls score across both fixtures, in full?

4 of 4 verdicts matched the fixtures' intended design (100%): both controls PASS on tricky_clean and both FAIL on tricky_dirty. Combined, id.no_secrets (weight 6, 5.6% of the raw total) and sc.safe_deploy (weight 3, 2.8%) account for 9 of the 108 raw control-weight points across all 27 aga.py controls — 8.3% together.

What would a human reviewer have to check to confirm the same verdict?

Exactly what the evidence string already names: open the cited file, go to the cited line, and read the ten words around it. That review takes seconds because the control already did the search — a human is confirming one named line, not re-scanning an entire unfamiliar tree from scratch with no starting point.

This is the trade a deterministic control makes deliberately: it does less reasoning than a model could in principle apply, in exchange for evidence a second person can check without re-deriving the judgment call from nothing.

Method note

Both fixture directories are deliberately adversarial reference material shipped alongside aga.py at /root/apexclaw-audit/aga_fixtures/, built to test exactly this distinction — the same file-and-line evidence style as the separate website-facing rubric published at methodology.json. The verdicts and evidence strings above are not historical or cached — they were produced by importing aga.py and calling run_controls() against each fixture directory at build time, 2026-08-24.

Both controls compared here map to OWASP ASI05 and ISO/IEC 42001 (Annex A.8/A.10); see the full standards crosswalk. Reference frameworks: NIST AI RMF, EU AI Act.