CEDAR: An Identifiability Audit for Verifier-Defined Credit in Plan Revision
Abstract
Multi-step agent benchmarks often score only the final plan, even when a late instruction changes dependencies among earlier actions. We ask whether verifier-defined credit supplies information beyond the edits between an initial and gold final plan. In 12,288 cases from CEDAR's original four-action schema, credit equals the changed-position set in every case, so a direct plan-difference baseline fully recovers the target. We then test a 4,096-case variable-length schema with source-only credit, dependent edits, and no-change controls. Its identity fraction is 0.253662: all 1,039 controls satisfy the identity by construction, while all 3,057 revisions do not; an oracle reaches 1.000 on both final-plan and credit metrics. A bounded check with Qwen2.5-7B-Instruct covers two case seeds and 256 cases per condition. We compare its emitted credit sets with edit sets implied by its predicted plans. The audit separates final-plan scoring from source-credit localization and gives a four-step procedure for testing whether intermediate benchmark labels add information beyond serialized outputs.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.