ClaimPromoteBench: Paired Regression Tests for Versioned Agent Policies
Desk rejected at ICLR 2027, so left out of the paper lists and the map.
Abstract
Repository policies are versioned inputs to coding agents, yet evaluations usually score versions independently. We present ClaimPromoteBench, a paired regression protocol that binds two policy versions, an unchanged case, and a target action relation. For finite executable policies, it enumerates action-changing contexts and derives cardinality-minimum suites with relative rule and transition coverage. For prose policies, targets are declared author mappings, not policy-owner ground truth. Route records retain transport, syntax, schema, and semantic failures in end-to-end denominators. Eight public histories illustrate the supported fault classes. On ten author-mapped pairs from two AGENTS.md revisions, eight hosted routes yield 128/160 item agreements and 52/80 pair agreements. In a prespecified non-ALLOW extension, every valid item agrees with its author target; incomplete outputs leave 50/54 stable pairs but only 6/18 changed pairs complete. On selected routes, a same-item request-cap comparison changes complete target-agreeing pairs from 0/15 to 7/15 and 0/15 to 9/15, with one call per cell. The protocol supports auditable frozen-case regression tests. The executable studies also retain exact action confusion matrices and source-bound verification receipts. Deployment validity and model-family effects remain unmeasured.
- Primary area:
- datasets and benchmarks