- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies step-level failure attribution in multi-agent traces using the continuous uncertainty of a fixed observer instead of its binary judgment. For each trajectory prefix, the observer provides a probability of correctness; the method converts it to uncertainty, applies Otsu thresholding, and selects the first high-uncertainty step. It requires no attribution-specific training or labeled failure steps at inference. Experiments use both Who&When subsets, reporting agent- and step-level exact-match accuracy with and without ground truth. Seven open observers and seven closed judges are compared, with same-observer binary readouts, alternative uncertainty measures, and localization rules. The main result is 38.1% step accuracy on Algorithm-Generated traces without ground truth, versus 27.8% for the strongest closed judge. The paper also reports positive continuous-over-binary gains and higher mean failure-step AUROC.
Critical strengths
- The central comparison is well controlled: Section 4.3 uses the same observer, prompt, and inference call for continuous and binary readouts, isolating whether retaining confidence values helps rather than conflating the effect with a different model or extra sampling.
- Tables 2 through 4 and Figure 6 vary observer, uncertainty estimator, and localization rule; continuous scores improve 51 of 56 observer-setting pairs.
- Section 5 explicitly limits evaluation to Who&When and notes prefix-wise inference costs on long traces.
- The method avoids labeled attribution data, which is useful for an inexpensive first-pass diagnostic.
Critical weaknesses
- Performance is benchmark dependent: on Algorithm-Generated data, step accuracy is 38.1%, but on Hand-Crafted trajectories it is 23.6%, below the strongest closed judge at 27.0% (Table 1). How well does uncertainty localize failures beyond Who&When?
- Absolute step localization remains weak. In Table 3, mean continuous-score AUROC is only 57.3 on Algorithm-Generated and 58.7 on Hand-Crafted data, and several individual observers are near chance or below it. Otsu converts these modest ranking gains into an exact step, but there is no analysis of how often errors are adjacent-step confusions versus distant misattributions.
- Baseline currency is a concern for an ICLR 2027 submission. The related-work section discusses Who&When, ECHO, and CHIEF, but Table 1 does not compare against recent structured or causal attribution systems. These are not exact uncertainty baselines, but they are pertinent system-level comparisons for the headline claim of improved attribution.
- The uncertainty is a proxy for observer confidence, not a calibrated probability that a step caused failure. There is no calibration or breakdown by trace length and step type. Otsu also assumes a useful two-regime distribution, but the robustness study does not show when this assumption holds.
- Scores are exact-match point estimates without confidence intervals or repeated-run variation, leaving gains of only a few points (Table 1) difficult to interpret.
Questions for authors
- Can you report results on an additional attribution benchmark or system family, including a comparison with structured methods such as CHIEF and EMFA?
- Can you break down step errors by distance from the annotated failure step, trace length, and step type, and report confidence intervals over traces?
- What evidence supports Otsu's two-regime assumption, and how often does the selected high-uncertainty regime begin before, at, or after the annotated decisive step?
- Can you report calibration of the observer uncertainty against step correctness, or explain why calibration is unnecessary for the thresholding rule?
3 comments
Official Comment by Authors (2/2)
Q2. Analysis by Trace Length and Step Type
Step-level accuracy declines with trajectory length across all methods, including closed-source judges. Our method achieves the highest exact-match and ±2-step accuracy on 6–10-step trajectories, which account for the majority of the benchmark.
| Length (steps) | Ours | Binary readout | Scope Delin. | Gemini 3.5 Flash Lite | GPT-4.1 |
|---|---|---|---|---|---|
| 1–5 | 71.4 / 100 | 57.1 / 100 | 42.9 / 71.4 | 42.9 / 85.7 | 71.4 / 100 |
| 6–10 | 35.0 / 68.3 | 26.8 / 60.2 | 24.4 / 56.9 | 26.8 / 61.8 | 20.3 / 57.7 |
| 11+ | 2.0 / 17.6 | 7.8 / 17.6 | 9.8 / 27.5 | 7.8 / 21.6 | 13.7 / 19.6 |
(Values indicate exact-match / ±2-step accuracy (%).)
Since Who&When lacks step-type annotations, we analyze failures by the responsible agent as a proxy. WebSurfer-related failures are the most challenging, with all methods achieving at most 13% accuracy. Trace-level 95% confidence intervals are provided in Appendix B.6.
Q4. Calibration of the Observer Score
Our method uses observer scores as relative failure evidence rather than calibrated probabilities of failure. Otsu determines a threshold from the score distribution within each trajectory, without requiring probability calibration. Although raw scores are poorly calibrated (ECE: 0.543 on HC and 0.449 on AG), Platt scaling substantially reduces ECE to 0.001 and 0.017, respectively. Thus, probability calibration is not a prerequisite for our training-free localization approach (Appendix B.8).
W5. Confidence Intervals and Run-to-Run Variance
Repeated experiments confirm consistent improvements on Algorithm-Generated logs. Across four runs, our method outperforms the binary readout in step-level accuracy in every run, achieving 35.9% vs. 31.7% (+4.2pp) with less than half the standard deviation (1.6 vs. 3.6).
| Readout | AG step | AG agent | HC step | HC agent |
|---|---|---|---|---|
| Ours (Otsu) | 35.9 ± 1.6 | 46.0 ± 1.3 | 11.8 ± 3.1 | 40.0 ± 2.6 |
| Ours (map) | 32.1 ± 0.5 | 44.4 ± 0.6 | 11.8 ± 2.3 | 42.7 ± 1.0 |
| Binary readout | 31.7 ± 3.6 | 43.3 ± 2.6 | 13.2 ± 4.0 | 45.5 ± 3.0 |
On Hand-Crafted logs, performance varies more across runs, with no consistent advantage over binary readout. On Algorithm-Generated logs, paired bootstrap tests further confirm significant improvements over binary readout (+10.3pp, p=0.004), all same-observer baselines, and GPT-4.1 (p≤0.022) in the reported single-run comparison. Paired bootstrap results are reported in Appendix B.1, and run-to-run variance in Appendix B.2
Brief Summary of Our Response and Gentle Reminder
Thank you again for your thoughtful and constructive feedback, and for recognizing the value of our controlled comparison between continuous and binary readouts and the practical benefits of our training-free approach.
Regarding your concerns:
Benchmark dependence and generalization (W1, Q1): We extended our evaluation to Who&When Pro and TraceElephant, where our method achieves the highest step-level accuracy on both benchmarks, outperforming the strongest baseline by 3.9pp and binary readout by 12.9pp on the pooled evaluation.
Error distance analysis (W2): We analyzed localization errors and found that 53% of incorrect predictions on Algorithm-Generated trajectories fall within ±2 steps of the gold step, compared with 16% on longer Hand-Crafted trajectories, clarifying the relationship between localization accuracy and trajectory length.
Comparison with recent attribution systems (W3): We demonstrated that our method achieves the highest overall step-level accuracy (30.4%) under the same observer, outperforming Scope Delineation (+9.4pp), CHIEF (+10.5pp), and OAT (+13.6pp), without additional training.
Validity of the two-regime assumption (W4, Q3): We provided empirical evidence of increased failure-evidence scores after the gold step in 84–93% of trajectories and showed that change-point detection mitigates early-onset errors, improving Hand-Crafted accuracy from 12.7% to 16.4%.
Analysis by trace length and step type (Q2): We demonstrated that localization accuracy declines with trajectory length across all methods, while our approach achieves the highest exact-match and ±2-step accuracy on 6–10-step trajectories. We also provided agent-level failure analysis and confidence intervals.
Calibration of the observer score (Q4): We clarified that our method relies on relative failure evidence rather than calibrated probabilities, and showed that probability calibration is not required for our trajectory-adaptive thresholding approach.
Confidence intervals and run-to-run variance (W5): We conducted four repeated runs and confirmed consistent improvements over binary readout on Algorithm-Generated trajectories (35.9% vs. 31.7%), while acknowledging the higher variance and lack of consistent improvement on Hand-Crafted trajectories.
As today is the final day of the rebuttal period, we hope that these additional experiments and clarifications address your concerns, and would sincerely appreciate it if you could reconsider your assessment of our manuscript in light of these updates.
Official Comment by Authors (1/2)
Thank you for highlighting our
controlled comparisonof continuous and binary readouts and thepractical valueof our training-free approach. Below, we address your concerns one by one.W1, Q1. Benchmark Dependence and Generalization
The lower accuracy on Hand-Crafted trajectories is partly associated with their greater length. Most Hand-Crafted logs exceed 10 steps, where all methods, including closed-source judges, achieve below 14% step-level accuracy. Nevertheless, our best observer outperforms six of seven closed-source judges on Hand-Crafted logs. Beyond Who&When, our method achieves the highest step-level accuracy on two additional benchmarks, Who&When Pro and TraceElephant, outperforming the strongest baseline by 3.9pp and the binary readout by 12.9pp on the pooled evaluation.
Under the same observer, our method also outperforms CHIEF and OAT overall (see W3). We could not evaluate EMFA because its implementation is not publicly available.
W2. Error Distance Analysis
Localization errors are predominantly near misses on Algorithm-Generated logs but more distant on Hand-Crafted logs. Among incorrect predictions, 53% fall within ±2 steps of the gold step on Algorithm-Generated logs, compared with only 16% on Hand-Crafted logs.
This difference is associated with trajectory length: all Algorithm-Generated logs contain at most 10 steps, whereas most Hand-Crafted logs exceed 10 steps. Unlike AUROC, which evaluates ranking across all steps, our method localizes the onset of elevated failure-evidence scores. Thus, moderate AUROC can coexist with near-miss localization errors (Appendix B.6).
W3. Comparison with Recent Attribution Systems
Our method achieves the highest overall step-level accuracy (30.4%) under the same observer (Qwen3.6-35B-A3B), outperforming Scope Delineation (+9.4pp), CHIEF (+10.5pp), and OAT (+13.6pp). On Algorithm-Generated logs, it also surpasses CHIEF by +12.7pp and OAT by +17.8pp.
Unlike CHIEF's multi-stage generation and retrieval or OAT's training on successful trajectories, our method requires no training and only a single-token probability readout per step. We have added these comparisons to Table 2. We also tried to compare with EMFA, but its implementation is not publicly available.
W4, Q3. Validity of the Two-Regime Assumption
The two-regime assumption is empirically supported by a clear increase in failure-evidence scores after the gold step. Mean scores rise from 0.41 to 0.62 on Algorithm-Generated logs and from 0.38 to 0.61 on Hand-Crafted logs, with higher post-failure averages in 84% and 93% of logs, respectively. Regarding Q3, predicted onsets tend to precede the gold step, particularly on longer Hand-Crafted trajectories, as shown below.
Change-point detection mitigates early-onset errors by identifying shifts in mean scores rather than isolated threshold crossings, improving Hand-Crafted accuracy from 12.7% to 16.4% (Appendix B.9).