- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors propose a framework for learning step representations geared towards failure attribution in LLM-based agent systems. The method combines layer selection, dual-branch feature construction, and contrastive learning to identify root-cause steps. Experiments on four benchmarks show improvements over existing baselines.
Critical strengths
-
The paper provides a clear motivation for learning attribution-oriented step representations.
-
The experiments cover multiple benchmarks, with consistent improvements over existing methods. And the ablation studies support the effectiveness of the proposed components.
Critical weaknesses
-
Section 3.1 defines the root cause through counterfactual intervention, but the labels in ReCast-2K are obtained through LLM voting and manual review. It is unclear whether these annotations actually satisfy the counterfactual definition. This raises some concerns about the reliability of the training labels.
-
A key motivation of this paper is that existing step representations are not well suited for failure attribution. However, ReCast uses a different learning framework from the embedding-based baselines, making it difficult to tell whether the improvements come from the hidden-state features or the proposed learning framework. A comparison using text embeddings within the same framework would help clarify this.
Questions for authors
-
How do the authors ensure that the root-cause labels in ReCast-2K are consistent with the counterfactual definition in Section 3.1?
-
How sensitive is ReCast to the successful trajectories used to construct the reference mean, especially when the reference comes from a different dataset, as in Who&When Pro?
4 comments
Response to Reviewer 96fd (2/3)
W2: Text embeddings within ReCast
A2: We thank the reviewer for this constructive suggestion. We have added a prefix-encoded Qwen3-Embedding-8B control within the same ReCast learning framework and training setup. The table reports Hit@1 (%) as mean ± standard deviation over three seeds; Overall pools trajectories within each benchmark.
Table 2. Input-representation comparison within ReCast: Hit@1 (%).
| ReCast input | Algorithm | Handcrafted | Captain | Magentic | Who&When Overall | TraceElephant Overall |
|---|---|---|---|---|---|---|
| Text embeddings | 31.99 ± 2.31 | 21.26 ± 2.15 | 25.10 ± 2.93 | 19.63 ± 2.77 | 28.57 ± 1.55 | 22.29 ± 0.00 |
| Hidden states | 45.43 ± 2.01 | 33.33 ± 1.63 | 35.69 ± 2.93 | 27.04 ± 1.89 | 41.58 ± 1.04 | 31.24 ± 1.17 |
Within the same ReCast learning framework, the original hidden-state version achieves 13.01 and 8.95 percentage points higher Overall Hit@1 on Who&When and TraceElephant, respectively. The tested text embeddings still show a substantial gap even with prefix encoding and the ReCast framework. This supports our central motivation that existing step representations are not well suited for failure attribution and provides complementary evidence for the effectiveness of ReCast's attribution-oriented representation design.
Response to Reviewer 96fd (3/3)
Q2: Successful-reference sensitivity
A3: We thank the reviewer for asking about sensitivity to successful references. To examine reference-size sensitivity, we use 25%, 50%, or 100% of the successful reference trajectories to estimate the reference mean, keeping all other training and evaluation settings fixed. The table reports Hit@1 (%) as mean ± standard deviation over three seeds; Overall pools trajectories within each benchmark.
Table 3. Successful-reference size sensitivity on the main benchmarks: Hit@1 (%).
| Reference | Algorithm | Handcrafted | Captain | Magentic | Who&When Overall | TraceElephant Overall |
|---|---|---|---|---|---|---|
| 25% | 44.09 ± 2.31 | 32.76 ± 1.41 | 34.90 ± 1.11 | 27.78 ± 2.40 | 40.48 ± 1.13 | 31.24 ± 1.50 |
| 50% | 43.55 ± 2.37 | 32.76 ± 1.41 | 34.12 ± 1.92 | 26.30 ± 2.92 | 40.11 ± 1.35 | 30.10 ± 1.89 |
| 100% | 45.43 ± 2.01 | 33.33 ± 1.63 | 35.69 ± 2.93 | 27.04 ± 1.89 | 41.58 ± 1.04 | 31.24 ± 1.17 |
Increasing the reference fraction from 50% to 100% improves Overall Hit@1 by 1.47 and 1.14 percentage points on Who&When and TraceElephant, respectively, indicating that additional successful reference trajectories can further improve attribution, although the gains are modest. Smaller reference subsets already achieve performance close to that of the full reference.
To address your concern about cross-dataset references, we also conducted the same reference-size experiment on Who&When Pro, with results shown below.
Table 4. Successful-reference size sensitivity on Who&When Pro.
| Reference | Hit@1 (%) | AUROC | AUPRC |
|---|---|---|---|
| 25% | 91.63 ± 0.31 | 0.9852 ± 0.0027 | 0.9639 ± 0.0039 |
| 50% | 91.58 ± 0.25 | 0.9874 ± 0.0014 | 0.9657 ± 0.0011 |
| 100% | 91.42 ± 0.21 | 0.9887 ± 0.0012 | 0.9667 ± 0.0017 |
On Who&When Pro, Hit@1 varies by only about 0.21 percentage points across the three reference sizes, while AUROC/AUPRC improve modestly with larger references. These results indicate that ReCast is robust to the tested changes in successful-reference size, maintaining high attribution accuracy even with only 25% of the external successful reference pool.
Overall, both the main experiments and Who&When Pro show limited variation in attribution performance as the number of successful reference trajectories changes, supporting ReCast's robustness over the tested reference-size range.
Thanks for your time and effort in reviewing our paper. Your constructive comments have helped us strengthen the experimental evidence. As the author–reviewer discussion period is approaching its end, we would kindly ask you to review our rebuttal, including the clarification of our annotation procedure and the additional experiments on text embeddings and successful-reference sensitivity, and let us know whether it addresses your concerns. We would be happy to answer any remaining questions. Thanks again for your time and valuable feedback.
Response to Reviewer 96fd (1/3)
A1: Thank you for highlighting the need to connect the counterfactual definition with the annotation procedure. The definition specifies the root-cause target. Verifying naturally occurring failures requires restoring the state before a candidate step, constructing a feasible correction, and rerunning the continuation; establishing earliestness also requires checking earlier candidates. Execution stochasticity and difficulties reproducing tool/environment states further increase dataset-scale verification costs. We therefore estimate this target through retrospective annotation of complete trajectories.
A relevant precedent is Who&When [1], which defines decisive errors through corrections that turn failure into success and selects the earliest decisive error. Its labels are obtained through three experts' inspection of failure logs, discussion of uncertain cases, and cross-review. This provides a methodological precedent for retrospective annotation, without implying that every label is intervention-tested.
Our scalable procedure uses step-level voting by three independent LLM judges to obtain 1,112 consensus labels, followed by manual review of the remaining 384 failed trajectories. It reduces reliance on a single judge and provides review for unresolved cases. Given the high cost of per-trajectory counterfactual verification, combining multi-judge voting with manual review offers a practical way to estimate the earliest decisive error from execution evidence and provide supervision for large-scale failure-attribution training.
[1] Zhang et al. (2025). Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. ICML 2025. Paper.