ClaimPromoteBench: Paired Regression Tests for Versioned Agent Policies
Abstract
Repository policies are versioned inputs to coding agents, yet evaluations usually score versions independently. We present ClaimPromoteBench, a paired regression protocol that binds two policy versions, an unchanged case, and a target action relation. For finite executable policies, it enumerates action-changing contexts and derives cardinality-minimum suites with relative rule and transition coverage. For prose policies, targets are declared author mappings, not policy-owner ground truth. Route records retain transport, syntax, schema, and semantic failures in end-to-end denominators. Eight public histories illustrate the supported fault classes. On ten author-mapped pairs from two AGENTS.md revisions, eight hosted routes yield 128/160 item agreements and 52/80 pair agreements. In a prespecified non-ALLOW extension, every valid item agrees with its author target; incomplete outputs leave 50/54 stable pairs but only 6/18 changed pairs complete. On selected routes, a same-item request-cap comparison changes complete target-agreeing pairs from 0/15 to 7/15 and 0/15 to 9/15, with one call per cell. The protocol supports auditable frozen-case regression tests. The executable studies also retain exact action confusion matrices and source-bound verification receipts. Deployment validity and model-family effects remain unmeasured.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.