acceptodds
Under review as a conference paper at ICLR 2027

CEDAR: An Identifiability Audit for Verifier-Defined Credit in Plan Revision

Abstract

Multi-step agent benchmarks often score only the final plan, even when a late instruction changes dependencies among earlier actions. We ask whether verifier-defined credit supplies information beyond the edits between an initial and gold final plan. In 12,288 cases from CEDAR's original four-action schema, credit equals the changed-position set in every case, so a direct plan-difference baseline fully recovers the target. We then test a 4,096-case variable-length schema with source-only credit, dependent edits, and no-change controls. Its identity fraction is 0.253662: all 1,039 controls satisfy the identity by construction, while all 3,057 revisions do not; an oracle reaches 1.000 on both final-plan and credit metrics. A bounded check with Qwen2.5-7B-Instruct covers two case seeds and 256 cases per condition. We compare its emitted credit sets with edit sets implied by its predicted plans. The audit separates final-plan scoring from source-credit localization and gives a four-step procedure for testing whether intermediate benchmark labels add information beyond serialized outputs.

Primary area:
datasets and benchmarks
Keywords:
agent evaluation, plan revision, credit assignment, benchmark identifiability, verifier-defined labels

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MuWNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CEDAR, an identifiability audit for verifier-defined credit in multi-step plan revision. It questions whether intermediate credit labels provide information beyond simply comparing the initial and final plans. In the original four-action schema, the authors find that credit exactly equals changed plan positions across all 12,288 cases, revealing a structural shortcut. They therefore construct a variable-length stress suite where source actions receive credit while dependent actions may also change. This breaks the shortcut: only 25.37% of cases satisfy the identity, entirely due to no-change controls. Experiments with Qwen2.5-7B further illustrate the distinction between final-plan correctness and source-credit localization.

Critical strengths
  1. The paper makes a compelling distinction between predicting an intermediate credit label and merely reconstructing that label from observable differences between initial and final plans. The finding that the canonical CEDAR credit target equals the changed-position set in all 12,288 cases is striking and demonstrates that apparently meaningful intermediate supervision can be structurally redundant.
  2. The paper does more than expose the shortcut: it constructs a variable-length stress suite with source-only credit, dependent edits, single/multiple source revisions, and explicit no-change controls. The resulting decomposition is particularly informative: all 3,057 revision cases violate the shortcut, whereas all 1,039 controls satisfy it by construction. The deterministic generation procedure, frozen manifests, unit tests, and released audit scripts further strengthen reproducibility.
  3. A notable strength is that the authors repeatedly distinguish structural benchmark findings from claims about model capability or causal attribution. For example, they explicitly state that the Qwen experiments do not establish successful source-responsibility recovery and that CEDAR's “credit” is verifier-defined rather than causal responsibility. This restraint makes the central measurement argument more credible.
Critical weaknesses
  1. The main result is established on controlled serialized plans containing only 3–7 actions. There is no live tool execution, environment interaction, partial observability, execution failure, or realistic long-horizon agent trajectory. The authors themselves acknowledge that the study does not establish transfer to open-world agent tasks. Thus, while the identifiability issue is convincing within CEDAR, its prevalence and practical importance in existing agent benchmarks remain unclear.
  2. The revised stress suite is evaluated using only Qwen2.5-7B-Instruct, two case-generation seeds, three prompting modes, and 256 examples per condition. More importantly, final exactness on every revision subset is zero. Consequently, the model experiments provide relatively little evidence about how credit identifiability affects capable agents, learning, reranking, or actual repair performance. Experiments with stronger models and training on the stress schema would make the practical implications substantially more convincing. 3.The proposed four-step audit protocol is potentially useful, but the paper applies it only to CEDAR. The authors explicitly state that they have not audited external benchmarks and that testing the protocol elsewhere remains open. Applying the audit to at least one established benchmark, or demonstrating that removing an identifiability shortcut changes conclusions about model ranking/training/repair, would considerably strengthen the claim that this is a broadly relevant evaluation methodology rather than primarily a correction to CEDAR's original design.
Questions for authors

NA

2 comments

The deterministic audit establishes a label-construction issue within the serialized CEDAR cases. The stress suite contains plans with three to seven actions, and the experiments omit tool execution, partial observability, and environment feedback. The model check uses one 7B instruction-tuned model, two generation seeds, three prompting interfaces, and 256 cases per condition. Exact final-plan accuracy is zero on revision cases. The analysis separates terminal-plan accuracy from source-credit localization; it does not measure reliable repair or downstream utility.

This audit is applied only to CEDAR in this submission. No external benchmark audit, model-ranking comparison, or training result is reported. The evidence supports the label-structure diagnosis and the measurements on the CEDAR stress suite. It does not establish how often the same issue occurs in other benchmarks or whether source-credit scores improve a repair policy.

The revised paper now adds the component-level checks requested in discussion and narrows the scope to the evidence they support. A parser that reads only the visible revision instruction and matches the explicit update stage N template recovers the named source set in 368/368 revisions (179/179 and 189/189 by seed). In the saved Qwen outputs, normalized plan syntax parses in 97.8–100.0% of revision cases, while exact source-state, dependent-state, dependent-action source-field, and full-plan matches are 60.9–67.6%, 8.4–11.6%, 0%, and 0%. These results separate source extraction from dependent-state propagation; they do not measure cross-model discrimination or downstream repair benefit.

The tau-bench Airline slice contains no explicit source-credit labels among 3,618 action steps, so it cannot provide an external identity rate. The 512 generated initial/gold plan pairs are all unique, leaving different credit sets for an identical pair untested. The manuscript now states both limits directly and presents the four-step procedure as demonstrated on CEDAR. The revised PDF and supplementary archive include the table and rerunnable semantic audit.

Reviewer_e7mMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the identifiability of verifier-defined credit labels in plan-revision benchmarks. The authors show that, in an earlier four-action CEDAR schema, the gold credit set is exactly recoverable from the positions changed between the initial and gold final plans across all 12,288 audited cases, indicating that the credit target does not provide information independent of the final plan. They then introduce a 4,096-case variable-length stress suite that separates source actions from dependent edits, so that source-credit labels are no longer equivalent to plan differences on revision cases. Structural baselines and a bounded Qwen2.5-7B-Instruct evaluation are used to demonstrate that final-plan correctness and source-credit localization can be measured separately under the revised schema. The paper’s broader contribution is a four-step audit protocol for checking whether intermediate benchmark labels are identifiable beyond simpler post-hoc functions of serialized outputs.

Critical strengths

The paper identifies a concrete and easy-to-overlook evaluation shortcut. The central audit is simple but important: in the original four-action schema, the verifier-defined credit label is exactly recoverable from the set of positions changed between the initial and gold plans across all 12,288 audited cases. This exposes a genuine identifiability issue in the original benchmark design rather than merely showing another model failure mode.

Critical weaknesses

The paper’s core contribution largely consists of identifying and repairing a design flaw in the authors’ own benchmark, while its broader external relevance remains unvalidated. The central finding is that, in the original four-action CEDAR schema, the credit label can be perfectly recovered from the initial-to-final plan difference; the authors then introduce a variable-length stress schema specifically designed to remove this shortcut. The main concern is that this failure mode arises in a benchmark constructed by the same authors, making the work look more like a retrospective audit and repair of CEDAR than the discovery of a problem that has been shown to affect existing benchmarks broadly. Although the paper presents the result as a general identifiability-audit methodology, it explicitly acknowledges that external benchmarks such as PlanBench were not audited. As a result, the broader applicability and significance of the proposed methodology are not yet well established.

The model evaluation is limited, and the current results do not yet establish the practical value of the revised benchmark. The experiments use only Qwen2.5-7B-Instruct, two case-generation seeds, and three prompting conditions, with 256 cases per condition. More importantly, final exactness is zero on all actual revision cases, with the nonzero pooled final accuracy coming entirely from no-change controls. Thus, the experiments mainly demonstrate that the revised interface can numerically separate final-plan correctness from source-credit localization. They do not yet show that the benchmark meaningfully discriminates among models of different capability levels, nor that source-credit prediction is related to successful plan repair in practice.

Questions for authors

The paper repeatedly refers to an “original” or “earlier” four-action CEDAR schema and discusses substantial prior experiments conducted on it. Could the authors clarify the status of this earlier version? Specifically, was the four-action CEDAR benchmark previously published or publicly released, or was it an internal/preliminary version developed during the preparation of this submission? This distinction would affect my assessment of whether the paper is primarily auditing an existing benchmark or retrospectively correcting a design issue discovered during the development of the benchmark itself.

7 comments

The submitted manuscript identifies the earlier four-action schema as canonical and original, but does not state whether it was published, publicly released, or internal. Those labels describe its role in the study and do not establish its release history; the paper provides no verified release record.

The evidence supports a focused CEDAR case study. The canonical label is exactly recoverable from the gold-plan difference in 12,288 cases; the stress suite separates source-credit labels from plan edits in its 3,057 revision cases and 1,039 controls. The model check uses one Qwen2.5-7B-Instruct model, two case seeds, three output interfaces, and 256 cases per condition. Exact final-plan accuracy is zero on revision cases. These results test label construction and separate terminal-plan scoring from credit scoring.

The submission does not audit an external benchmark or measure effects on model ranking, training, or downstream repair. The four-step procedure is demonstrated on CEDAR only, so broader applicability remains untested.

Thank you for the response. However, this does not read like a conventional author rebuttal. It reads more like a third-person summary of the manuscript's limitations. Rather than directly addressing my question, the response merely reiterates what the paper does and does not establish.

Was the original four-action CEDAR benchmark publicly released before this submission, or was it an internal preliminary version? Please provide a direct answer and, if applicable, a release link or date.

If the latter, the paper essentially identifies and corrects a design flaw in the authors' own preliminary benchmark. I question whether this constitutes a sufficiently novel, standalone research contribution, rather than a routine part of benchmark development. This concern is further reinforced by the lack of external validation and demonstrated downstream benefits.

My concerns remain unresolved, and I therefore maintain my rating of Strong Reject.

Direct answer: the original four-action CEDAR schema was an internal preliminary version and had not been publicly released before the 14 September 2026 submission. We plan to release it after the paper is accepted. Accordingly, the 12,288-case result is an audit of a design flaw in our own preliminary benchmark. The revised stress suite and four-step procedure are also demonstrated only on CEDAR. We agree that the submission does not establish the flaw’s prevalence in external benchmarks or show downstream repair benefits, so broader validity and practical value remain unverified.

Reviewer_V3aXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In the context of the plan-auditing task, is the “verifier-defined credit” label determined solely by the locations of changes between the initial plan and the gold-standard final plan? In the original four-action model – comprising 12,288 cases – the set of “credit” elements consistently aligns with the set of change locations; thus, once the correct final plan is obtained, this label can be directly reconstructed.

The authors further constructed a dataset of 4,096 variable-length cases, which include source actions, dependent actions, and unchanged control elements, with only the source actions counted toward the “credit.” In this new setup, the set of “credit” elements matches the set of change locations for the 1,039 unchanged control cases, but fails to do so for all 3,057 revision cases – a ratio of 0.253662. A limited evaluation using Qwen2.5-7B across six conditions revealed a zero exact-match rate for the final plans in all revision cases. The paper proposes a four-step auditing framework – consisting of information-boundary checks, simple structural baselines, stress-test cases, and paired output transformations – and emphasizes that plan correctness, “credit” attribution, and downstream utility should be interpreted as distinct aspects.

Critical strengths
  1. The audit questions are well-defined and falsifiable. The paper examines whether intermediate labels are implicitly determined by the final output, employing explicit differential rules rather than relying solely on a comparison of model scores. This approach directly reveals label redundancy within the four-action interface, cautioning researchers against misinterpreting redundantly encoded labels as evidence of independent diagnostic capabilities.
  2. The study demonstrates a rigorous approach to defining evidentiary boundaries and handling denominators. The authors distinguish between “gold-standard” structural probes and baselines accessible to the model, reporting results for both control and revised cases – thereby clarifying that the model is not yet capable of performing revisions – while also correcting scoring issues related to unsolicited or invalid “credit” fields.
Critical weaknesses
  1. Innovation and information gain remain limited. The original framework defined all modified locations as “credit,” whereas the new framework distinguishes between “source-only credit” and “dependency-based credit” based on structural configuration; the 1.000 score (and the 0.000 score for the subset of revisions) primarily serves to validate the labeling construction. The condition C≠DC \neq D merely excludes the rule of “directly copying the entire diff” – it does not rule out other simple functions that could reconstruct the labels. As the paper has yet to demonstrate any new insights gained from this auditing process through external interfaces, its current contribution is best characterized as a narrow case study in construct validity. Given that the authors have explicitly defined the scope of their work, it would be unfair to criticize it for failing to achieve causal attribution in an open-ended environment.
  2. The experimental setup has yet to establish the diagnostic discriminative power of the stress-test dataset. The six experimental conditions are derived from a single 7B model, three types of prompting interfaces, and two batches of generated examples – they do not represent six distinct models or independent training runs. For all revision cases, the exact match rate for the final plan is zero; any non-zero “pooled” score stems solely from the unmodified control cases. However, the action F1 score ranges from 0.594 to 0.626, suggesting that the model’s outputs are at least partially correct. To determine exactly what these interfaces are measuring, it is necessary to disaggregate the results – separating parsing errors, exact match failures, errors in source-action updates, and errors in dependency propagation.
Questions for authors
  1. Please clarify which category of recovery rules is excluded by the concept of “identifiability.” Aside from the case where C=DC = D, is it possible for the current generator to produce pairs of examples that share the same initial and final plans but differ in their “source credit”? If not, please qualify your conclusions by limiting them to the specific “shortcuts” that have been empirically tested.
  2. What level of performance can be achieved by a simple extraction rule that relies solely on the initial plan and the revision instructions – information already visible to the model? Please distinguish this from probing methods based on fixed positions or the earliest/latest modifications, and ensure that “gold standard” information – accessible only to an evaluator – is not misconstrued as a deployable baseline.
4 comments

The paper uses identifiability for a specific structural test: whether verifier-defined credit C* equals the changed-position set D(P0,P*) on the generated cases. The equality holds in all 12,288 canonical cases. In the stress suite, it holds for all 1,039 no-change controls and none of the 3,057 revision cases. These results test direct plan differences and the listed structural baselines; they do not enumerate all functions that might recover credit. C* ≠ D* rules out direct copying of the full difference set, but the paper reports no search for pairs with identical initial and final plans but different credit. The submitted results do not establish whether the generator permits such pairs, so the conclusion remains limited to the tested shortcuts.

Table 3's gold-final-plan-plus-differences strategy uses evaluator-only information. The control-aware first-position strategy also uses the evaluator-only no-change flag. The paper reports no baseline that extracts credit from only the initial plan and revision instructions, so it provides no performance value for the deployable rule in the reviewer's question.

The Qwen stress check reports zero exact final plans on every revision subset. It does not disaggregate parsing, source-action updates, or dependency propagation, so those failure modes remain unresolved.

Thank you for the specific requests. We added two checks to the revised paper and supplement.

First, the instruction-only regex baseline extracts the explicitly named source stages in 368/368 revision cases (179/179 for seed 7; 189/189 for seed 19). It uses the current update stage N template, so this is a prompt-visible extraction baseline, not general language understanding.

Second, we replayed the saved Qwen outputs by failure component. Across six conditions, normalized plan syntax parses in 97.8–100.0% of revisions; source-state exactness is 60.9–67.6%, dependent-state exactness 8.4–11.6%, dependent-action source-field exactness 0%, and full-plan exactness 0%. This separates source extraction from propagation: recognizing the named stage does not yield a correct dependent repair. The table and a rerunnable audit are in the revised PDF and supplement.

Finally, all 512 initial/gold plan pairs are unique. Thus the current generated cases contain no repeated plan pair with which to test different credit labels. We now limit the conclusion to the directly tested plan-difference rule and this instruction parser; we do not claim that all possible recovery rules have been excluded. The revised source archive includes the audit.

Thank you for adding the instruction-only baseline and the error breakdown, which address my diagnostic questions. However, the limited contribution beyond the constructed schema remains my main concern, so I maintain my rating of 2 (Weak rejection).

AuthorsOpenReview ↗

Scope and contribution

Thank you for recognizing the added baseline and error breakdown. The remaining question is the scope of the contribution. The empirical study is confined to one controlled benchmark; it does not estimate prevalence in other benchmarks or show downstream repair benefit. The contribution is a falsifiable measurement result about what verifier-defined credit scores encode. In 12,288 canonical cases, the evaluator's initial-to-gold-plan difference recovers every credit label. In the stress suite, it recovers 0/3,057 revision labels, while all 1,039 no-change controls match by construction. A parser using only the visible instruction recovers the named source stages in 368/368 revisions, yet the saved Qwen outputs reach only 8.4–11.6% exactness on dependent states and 0% on dependent-action source fields and complete plans. These checks separate label redundancy, source extraction, and dependency propagation; a single terminal or credit score would conflate them. The reusable contribution is an audit sequence that tests candidate recovery rules against their information access and then separates label recovery from task execution. Our 200-record tau-bench Airline slice has no explicit source-credit labels across 3,618 action steps. Assigning labels from edits would assume the relation being tested, so an external identity rate is undefined. We present this as a bounded CEDAR methods case study and maintain that the measurement distinction and diagnostic protocol are substantive, while external prevalence and downstream utility remain open.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.