Zero LLM Judge, Programmatic Verification
Every score is a deterministic program run — results are reproducible
Legacy approach: LLM-as-judge
deprecated
✕
Model bias · randomness
✕
Prompt engineering skews scores
✕
Different sampling temperature, different score
replaced by
Agent output
exploit script · findings report · detection rule
Docker isolated execution
The deterministic scorer runs inside the verifier container
PoC execution verification
Run the script, check the flag
YARA / Ground Truth
Match rate · F1 · Kendall-tau
Structured report diffing
Field matching · precision floor
reward = w₁×artifact + w₂×correctness + w₃×robustness
Numeric 0-1 final score
Core guarantees
Deterministic
Same task, same output — identical score on repeat runs
Reproducible
No model bias, unaffected by prompt engineering
Partial credit
macro-F1 + smooth ramp + precision floor
Avoids binary pass/fail