- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CEDAR, an identifiability audit for verifier-defined credit in multi-step plan revision. It questions whether intermediate credit labels provide information beyond simply comparing the initial and final plans. In the original four-action schema, the authors find that credit exactly equals changed plan positions across all 12,288 cases, revealing a structural shortcut. They therefore construct a variable-length stress suite where source actions receive credit while dependent actions may also change. This breaks the shortcut: only 25.37% of cases satisfy the identity, entirely due to no-change controls. Experiments with Qwen2.5-7B further illustrate the distinction between final-plan correctness and source-credit localization.
Critical strengths
- The paper makes a compelling distinction between predicting an intermediate credit label and merely reconstructing that label from observable differences between initial and final plans. The finding that the canonical CEDAR credit target equals the changed-position set in all 12,288 cases is striking and demonstrates that apparently meaningful intermediate supervision can be structurally redundant.
- The paper does more than expose the shortcut: it constructs a variable-length stress suite with source-only credit, dependent edits, single/multiple source revisions, and explicit no-change controls. The resulting decomposition is particularly informative: all 3,057 revision cases violate the shortcut, whereas all 1,039 controls satisfy it by construction. The deterministic generation procedure, frozen manifests, unit tests, and released audit scripts further strengthen reproducibility.
- A notable strength is that the authors repeatedly distinguish structural benchmark findings from claims about model capability or causal attribution. For example, they explicitly state that the Qwen experiments do not establish successful source-responsibility recovery and that CEDAR's “credit” is verifier-defined rather than causal responsibility. This restraint makes the central measurement argument more credible.
Critical weaknesses
- The main result is established on controlled serialized plans containing only 3–7 actions. There is no live tool execution, environment interaction, partial observability, execution failure, or realistic long-horizon agent trajectory. The authors themselves acknowledge that the study does not establish transfer to open-world agent tasks. Thus, while the identifiability issue is convincing within CEDAR, its prevalence and practical importance in existing agent benchmarks remain unclear.
- The revised stress suite is evaluated using only Qwen2.5-7B-Instruct, two case-generation seeds, three prompting modes, and 256 examples per condition. More importantly, final exactness on every revision subset is zero. Consequently, the model experiments provide relatively little evidence about how credit identifiability affects capable agents, learning, reranking, or actual repair performance. Experiments with stronger models and training on the stress schema would make the practical implications substantially more convincing. 3.The proposed four-step audit protocol is potentially useful, but the paper applies it only to CEDAR. The authors explicitly state that they have not audited external benchmarks and that testing the protocol elsewhere remains open. Applying the audit to at least one established benchmark, or demonstrating that removing an identifiability shortcut changes conclusions about model ranking/training/repair, would considerably strengthen the claim that this is a broadly relevant evaluation methodology rather than primarily a correction to CEDAR's original design.
Questions for authors
NA
2 comments
The revised paper now adds the component-level checks requested in discussion and narrows the scope to the evidence they support. A parser that reads only the visible revision instruction and matches the explicit update stage N template recovers the named source set in 368/368 revisions (179/179 and 189/189 by seed). In the saved Qwen outputs, normalized plan syntax parses in 97.8–100.0% of revision cases, while exact source-state, dependent-state, dependent-action source-field, and full-plan matches are 60.9–67.6%, 8.4–11.6%, 0%, and 0%. These results separate source extraction from dependent-state propagation; they do not measure cross-model discrimination or downstream repair benefit.
The tau-bench Airline slice contains no explicit source-credit labels among 3,618 action steps, so it cannot provide an external identity rate. The 512 generated initial/gold plan pairs are all unique, leaving different credit sets for an identical pair untested. The manuscript now states both limits directly and presents the four-step procedure as demonstrated on CEDAR. The revised PDF and supplementary archive include the table and rerunnable semantic audit.
The deterministic audit establishes a label-construction issue within the serialized CEDAR cases. The stress suite contains plans with three to seven actions, and the experiments omit tool execution, partial observability, and environment feedback. The model check uses one 7B instruction-tuned model, two generation seeds, three prompting interfaces, and 256 cases per condition. Exact final-plan accuracy is zero on revision cases. The analysis separates terminal-plan accuracy from source-credit localization; it does not measure reliable repair or downstream utility.
This audit is applied only to CEDAR in this submission. No external benchmark audit, model-ranking comparison, or training result is reported. The evidence supports the label-structure diagnosis and the measurements on the CEDAR stress suite. It does not establish how often the same issue occurs in other benchmarks or whether source-credit scores improve a repair policy.