- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether invocation-level evaluation can distinguish deviation from a reference trajectory from a loss of local competence after an early tool-use error. It compares baseline performance under correct histories with free-running performance under the model's own histories on matched tasks, defining relative propagation loss as . Its central result is a conditional construct-validity statement: when the canonical post-divergence target is difficult to predict from the observable history, reference agreement is bounded by the target's guessing probability and provides limited information about local competence. The authors apply conditional-on-state replay to cached trajectories and examine five open-weight models on synthetic routing tasks, supplemented by targeted posterior predictive checks, error injection, and assigned state repair. The paper distinguishes correct local continuation, return to the canonical state, and completion of the original task.
Critical strengths
1. The paper identifies an important issue in interpreting agent evaluations. An upstream error can cause subsequent locally correct actions to remain inconsistent with the reference trajectory. Later reference mismatches therefore do not directly establish new local decision errors. Section 5.3 states the relevant condition through conditional guessing probability and discusses exceptions involving reconstructible targets, resets, and alternative valid paths. This gives a concrete basis for interpreting trajectory scores without dismissing all fixed-reference evaluations.
2. The paired protocol and cached replay make the different factors easier to examine separately. The two arms use matched tasks and the same retry budget. The primary depth-level estimates use paired task-level bootstrap intervals, and the paper reports absolute gaps and error amplification alongside relative loss. Conditional-on-state replay changes the argument-scoring target on the same cached outputs without new model queries, directly showing how the scoring regime changes the interpretation of identical behavior. The paper also makes clear that tool selection was already scored against the actual state in the original protocol.
3. The diagnostics and negative findings offer useful methodological lessons. Appendix C shows a fit passing an aggregate posterior predictive check while failing a check targeted at corrupted contexts, illustrating what aggregate fit can miss. Appendix F separately reports post-repair reference agreement and original-task completion, without treating improved reference agreement as clear evidence of task recovery. The discussion of ceiling effects, parser issues, and unexecuted experimental conditions also helps establish the limits of the findings.
Critical weaknesses
1. The claimed roughly 2.9-fold observational overestimate appears to mix different estimands. Figure 4 defines severity as the relative loss , with an observational estimate of 0.316 for allam. However, Table 7 reports corrected/corrupted accuracies of 0.411/0.304 and an effect of 0.108, which appears to be an absolute accuracy difference. Using the displayed rounded rates, relative severity would be approximately . Thus, dividing 0.316 by 0.108 appears to compare a relative loss with an absolute difference. The resulting claim occurs in the contributions, Section 5.4, and Appendix E, and should be recomputed under a common definition or removed. Even after aligning normalization, the state and position distributions of the observational and intervention samples need to be comparable before their difference can quantify bias.
2. The generator-level justification of the theoretical condition and its relationship to prior work need further detail. The proposition holds under its stated condition, but Section 5.3 sets under a conditional-uniformity assumption without deriving it from the complete generation process. For , hidden constants alone do not exclude information revealed by previous input-output observations. The constant distribution, tool reuse, and observable history need to be specified. Related Work should also discuss Exposure Bias versus Self-Recovery (EMNLP 2021), which already compares data prefixes with model-generated prefixes and warns that retaining a fixed reference after changing the prefix can mislead evaluation. The contribution is better positioned around the formal tool-use condition, executable scoring, and diagnostic evidence, with a clearer account of what these add to the earlier analysis.
3. The empirical evidence remains concentrated in narrow task and model conditions. The primary tasks involve parity routing and number copying, stronger models are mostly at ceiling, and the injection and recovery experiments use only allam-2-7b. This model is itself anti-correlated with the routing rule, so the selection-channel effect may depend on parity distributions and tool presentation. Section 7 acknowledges that conditional-on-state scoring does not directly transfer to BFCL multi-turn tasks whose correct actions depend on user intent. The evidence therefore supports a controlled evaluation diagnosis but does not establish its prevalence in realistic agents. In addition, Appendix E tests multiple injection positions on the same task collection but resamples pairs; if a source task contributes multiple pairs, dependence between those pairs needs to be accounted for.
Questions for authors
1. Does Table 7 report an absolute effect or relative severity? Please recompute the observational and intervention estimates and intervals under a common definition. After aligning the estimand and accounting for state and position distributions, does the roughly 2.9-fold overestimate remain supported? Otherwise, which conclusions about causal effects still hold?
2. Could the authors specify constant sampling, tool reuse, and history visibility, and use these rules to derive or bound ? Please also explain what new conclusions or capabilities the tool-use setting adds beyond existing exposure-bias prefix comparisons and conditional-reference evaluation.
3. Could the authors add a check with non-copy arguments or randomized tool presentation on an additional reproducible model that is not at ceiling? For the existing injection data, please specify the number of pairs contributed by each source task and provide task-cluster bootstrap intervals where tasks are shared, to assess whether the intervals and effect directions remain robust.