acceptodds
Under review as a conference paper at ICLR 2027

ReCast: Attribution-Oriented Step Representation Learning for LLM-Based Agent Systems

Abstract

In LLM-based agent systems, failures can originate from early steps whose effects propagate through subsequent interactions, making their origins difficult to identify. To trace such failures back to their origin, failure attribution has been formulated as the task of identifying the earliest step responsible for the failure. Recent methods leverage LLM internal signals for failure attribution, typically using hidden states as step representations. We therefore conduct an empirical study to evaluate how effectively these representations distinguish root-cause steps from other steps and find limited separation. Motivated by this observation, we propose ReCast, a step representation learning method that transforms hidden states from a frozen LLM into attribution-oriented step representations. ReCast first selects attribution-relevant layers, then constructs complementary pattern and deviation features, and finally learns contextualized step representations through an encoder trained with contrastive and ranking objectives. We also introduce ReCast-2K, a training dataset for failure attribution. ReCast achieves the best Hit@1 across four benchmarks, surpassing the strongest baseline by 5.65 and 9.19 pp on Who&When Algorithm and Handcrafted, respectively. Code is available at https://anonymous.4open.science/r/ReCast-5FB6.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
LLM-Based Multi-Agent Systems, Failure Attribution, Step Representation Learning, Model Internal Signals

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_96fdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose a framework for learning step representations geared towards failure attribution in LLM-based agent systems. The method combines layer selection, dual-branch feature construction, and contrastive learning to identify root-cause steps. Experiments on four benchmarks show improvements over existing baselines.

Critical strengths
  1. The paper provides a clear motivation for learning attribution-oriented step representations.

  2. The experiments cover multiple benchmarks, with consistent improvements over existing methods. And the ablation studies support the effectiveness of the proposed components.

Critical weaknesses
  1. Section 3.1 defines the root cause through counterfactual intervention, but the labels in ReCast-2K are obtained through LLM voting and manual review. It is unclear whether these annotations actually satisfy the counterfactual definition. This raises some concerns about the reliability of the training labels.

  2. A key motivation of this paper is that existing step representations are not well suited for failure attribution. However, ReCast uses a different learning framework from the embedding-based baselines, making it difficult to tell whether the improvements come from the hidden-state features or the proposed learning framework. A comparison using text embeddings within the same framework would help clarify this.

Questions for authors
  1. How do the authors ensure that the root-cause labels in ReCast-2K are consistent with the counterfactual definition in Section 3.1?

  2. How sensitive is ReCast to the successful trajectories used to construct the reference mean, especially when the reference comes from a different dataset, as in Who&When Pro?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 96fd (1/3)

Q1 & W1: Annotation and the counterfactual definition

A1: Thank you for highlighting the need to connect the counterfactual definition with the annotation procedure. The definition specifies the root-cause target. Verifying naturally occurring failures requires restoring the state before a candidate step, constructing a feasible correction, and rerunning the continuation; establishing earliestness also requires checking earlier candidates. Execution stochasticity and difficulties reproducing tool/environment states further increase dataset-scale verification costs. We therefore estimate this target through retrospective annotation of complete trajectories.

A relevant precedent is Who&When [1], which defines decisive errors through corrections that turn failure into success and selects the earliest decisive error. Its labels are obtained through three experts' inspection of failure logs, discussion of uncertain cases, and cross-review. This provides a methodological precedent for retrospective annotation, without implying that every label is intervention-tested.

Our scalable procedure uses step-level voting by three independent LLM judges to obtain 1,112 consensus labels, followed by manual review of the remaining 384 failed trajectories. It reduces reliance on a single judge and provides review for unresolved cases. Given the high cost of per-trajectory counterfactual verification, combining multi-judge voting with manual review offers a practical way to estimate the earliest decisive error from execution evidence and provide supervision for large-scale failure-attribution training.

[1] Zhang et al. (2025). Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems. ICML 2025. Paper.

AuthorsOpenReview ↗

Response to Reviewer 96fd (2/3)

W2: Text embeddings within ReCast

A2: We thank the reviewer for this constructive suggestion. We have added a prefix-encoded Qwen3-Embedding-8B control within the same ReCast learning framework and training setup. The table reports Hit@1 (%) as mean ± standard deviation over three seeds; Overall pools trajectories within each benchmark.

Table 2. Input-representation comparison within ReCast: Hit@1 (%).

ReCast inputAlgorithmHandcraftedCaptainMagenticWho&When OverallTraceElephant Overall
Text embeddings31.99 ± 2.3121.26 ± 2.1525.10 ± 2.9319.63 ± 2.7728.57 ± 1.5522.29 ± 0.00
Hidden states45.43 ± 2.0133.33 ± 1.6335.69 ± 2.9327.04 ± 1.8941.58 ± 1.0431.24 ± 1.17

Within the same ReCast learning framework, the original hidden-state version achieves 13.01 and 8.95 percentage points higher Overall Hit@1 on Who&When and TraceElephant, respectively. The tested text embeddings still show a substantial gap even with prefix encoding and the ReCast framework. This supports our central motivation that existing step representations are not well suited for failure attribution and provides complementary evidence for the effectiveness of ReCast's attribution-oriented representation design.

AuthorsOpenReview ↗

Response to Reviewer 96fd (3/3)

Q2: Successful-reference sensitivity

A3: We thank the reviewer for asking about sensitivity to successful references. To examine reference-size sensitivity, we use 25%, 50%, or 100% of the successful reference trajectories to estimate the reference mean, keeping all other training and evaluation settings fixed. The table reports Hit@1 (%) as mean ± standard deviation over three seeds; Overall pools trajectories within each benchmark.

Table 3. Successful-reference size sensitivity on the main benchmarks: Hit@1 (%).

ReferenceAlgorithmHandcraftedCaptainMagenticWho&When OverallTraceElephant Overall
25%44.09 ± 2.3132.76 ± 1.4134.90 ± 1.1127.78 ± 2.4040.48 ± 1.1331.24 ± 1.50
50%43.55 ± 2.3732.76 ± 1.4134.12 ± 1.9226.30 ± 2.9240.11 ± 1.3530.10 ± 1.89
100%45.43 ± 2.0133.33 ± 1.6335.69 ± 2.9327.04 ± 1.8941.58 ± 1.0431.24 ± 1.17

Increasing the reference fraction from 50% to 100% improves Overall Hit@1 by 1.47 and 1.14 percentage points on Who&When and TraceElephant, respectively, indicating that additional successful reference trajectories can further improve attribution, although the gains are modest. Smaller reference subsets already achieve performance close to that of the full reference.

To address your concern about cross-dataset references, we also conducted the same reference-size experiment on Who&When Pro, with results shown below.

Table 4. Successful-reference size sensitivity on Who&When Pro.

ReferenceHit@1 (%)AUROCAUPRC
25%91.63 ± 0.310.9852 ± 0.00270.9639 ± 0.0039
50%91.58 ± 0.250.9874 ± 0.00140.9657 ± 0.0011
100%91.42 ± 0.210.9887 ± 0.00120.9667 ± 0.0017

On Who&When Pro, Hit@1 varies by only about 0.21 percentage points across the three reference sizes, while AUROC/AUPRC improve modestly with larger references. These results indicate that ReCast is robust to the tested changes in successful-reference size, maintaining high attribution accuracy even with only 25% of the external successful reference pool.

Overall, both the main experiments and Who&When Pro show limited variation in attribution performance as the number of successful reference trajectories changes, supporting ReCast's robustness over the tested reference-size range.

Thanks for your time and effort in reviewing our paper. Your constructive comments have helped us strengthen the experimental evidence. As the author–reviewer discussion period is approaching its end, we would kindly ask you to review our rebuttal, including the clarification of our annotation procedure and the additional experiments on text embeddings and successful-reference sensitivity, and let us know whether it addresses your concerns. We would be happy to answer any remaining questions. Thanks again for your time and valuable feedback.

Reviewer_46WHOpenReview ↗
Rating
3
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies step-level failure attribution in LLM-based agent systems to identify the earliest error in a failed execution trajectory. The authors first examine the separation between root-cause and non-root steps in text embeddings and frozen LLM hidden states. Then they propose ReCast based on their insights, which transforms hidden states into attribution-oriented step representations. The paper also introduces ReCast-2K. Extensive experiments demonstrate the effectiveness of ReCast.

Critical strengths
  1. ReCast explicitly distinguishes the attribution target from the step with the root cause instead of other steps that may also contain errors.

  2. Extensive evaluation includes four benchmarks and different baselines. Repeated experiments further demonstrate the effectiveness of ReCast.

Critical weaknesses
  1. Lack of experiments to improve the reasoning at inference time, which undermines the practical value of ReCast.
Questions for authors

Adding more experiments regarding repairing the failure reasoning at inference-time evaluation will increase the importance of this paper.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 46WH

Q1 & W1: Inference-time repair utility

A1: We thank the reviewer for suggesting an evaluation of inference-time repair utility. We have added a repair evaluation on all 165 GAIA validation tasks using Captain-Agent with Qwen3.5-9B. Both methods share the same initial executions (seed 42): 46 successes and 119 failures. Only failed tasks receive up to four repair rounds under matched model and budget settings, stopping upon success.

Self-Refine uses Qwen3.5-9B to generate corrective advice from the latest failed trajectory and restarts execution. ReCast instead re-localizes the root cause at each round, provides fixed-template feedback, restores the state before the predicted step, and regenerates the suffix, without an additional LLM call for feedback generation.

Table 2. Cumulative task success after repair on GAIA validation.

StageSelf-RefineReCast
Initial execution46/165 (27.88%)46/165 (27.88%)
Repair Round 155/165 (33.33%)58/165 (35.15%)
Repair Round 260/165 (36.36%)66/165 (40.00%)
Repair Round 362/165 (37.58%)70/165 (42.42%)
Repair Round 463/165 (38.18%)74/165 (44.85%)

After four rounds, ReCast repairs 28 of the 119 initial failures versus 17 for Self-Refine, recovering 11 additional tasks and achieving a 6.67-percentage-point higher final success rate. Its success rate increases from 27.88% to 44.85%, providing downstream evidence for the practical value of ReCast-guided repair.

Reviewer_46WHOpenReview ↗

Thanks for your rebuttal

Thanks for your rebuttal! I maintain my positive assessment, I think this is a good work and your additional experiments show that ReCast can be used to further localize the error and help the model to fix them. I will increase my confidence to 4.

AuthorsOpenReview ↗

Thanks for Your Review

Thanks for your thoughtful review and positive feedback! We appreciate your increased confidence in the work, and your constructive suggestion helped us strengthen the experimental evidence for the practical value of ReCast in failure attribution and repair. We sincerely appreciate your time and thoughtful feedback, and we will incorporate these additional results into the revised manuscript.

Reviewer_hzVoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a method for failure attribution in LLM agent systems. It extracts step features from a frozen LLM and constructs features that capture activation patterns and deviations from successful executions. It then learns step representations through a ranking objective and two contrastive objectives.

The authors also proppose a new dataset called ReCast-2K, which contains success and failure agent trajectories. The authors evaluate their method on 4 benchmarks against multiple attribution baselines and include ablations on components of the method. ReCast achieves the highest reported Hit@1 in every evaluated setting.

Overall I find the method novel and have enought empirical support, but I have a few concerns regarding the experimental settings that I hope the authors can help clarify.

Critical strengths

Learning step representations specifically for attribution is an interesting approach, and the ablations justify the usage of each component in the method. Results provide concrete evidence that selecting and adapting internal features can benefit attribution.

ReCast outperforms baselines consistently across four benchmarks and several agent configurations, although the gains are smaller on AFTraj-2K and TraceElephant.

Critical weaknesses

ReCast receives prefix-conditioned hidden states from Qwen3.5-27B, but StepFinder uses independently encoded text from a 0.6B embedding model. This could confound the learning method with the information available in its input features. Showing improvement given the same prefix-conditioned features would make the claim stronger.

ReCast-2K is generated from GAIA and AssistantBench validation tasks. The authors claim that TraceElephant trajectories do not exactly overlap with training trajectories but does not clarify that their underlying tasks are disjoint.

Questions for authors

Are ReCast-2K’s training trajectories disjoint from Who&When and TraceElephant at the underlying task/query level?

Can the authors evaluate StepFinder or ASCon using the same prefix-conditioned Qwen3.5-27B features available to ReCast?

5 comments
AuthorsOpenReview ↗

Response to Reviewer hzVo (1/2)

Q2 & W1: Baselines with prefix-conditioned inputs

A1: We thank the reviewer for this constructive suggestion to better examine the role of input representations in the comparison. In the added comparison, StepFinder and ASCon retain their basic configurations and use the same prefix-conditioned Qwen3.5-27B features as ReCast. The table reports Hit@1 (%) as mean ± standard deviation over three seeds; Overall pools evaluation trajectories within each benchmark separately.

Table 1. Baseline comparison with prefix-conditioned inputs: Hit@1 (%).

Method / settingAlgorithmHandcraftedCaptainMagenticWho&When OverallTraceElephant Overall
StepFinder (original)22.85 ± 2.1218.97 ± 2.4427.84 ± 2.0019.26 ± 0.5221.61 ± 2.2123.43 ± 0.81
StepFinder (+ prefix)31.45 ± 2.8722.41 ± 0.0032.94 ± 0.9622.96 ± 2.7728.57 ± 1.9627.81 ± 1.89
ASCon (original)26.34 ± 1.0121.84 ± 0.8130.98 ± 1.1125.93 ± 0.5224.91 ± 0.6928.38 ± 0.71
ASCon (+ prefix)36.02 ± 4.4824.71 ± 0.8134.51 ± 0.5527.41 ± 0.5232.42 ± 3.2430.86 ± 0.47
ReCast (original)45.43 ± 2.0133.33 ± 1.6335.69 ± 2.9327.04 ± 1.8941.58 ± 1.0431.24 ± 1.17

Prefix-conditioned inputs improve both StepFinder and ASCon. This is consistent with our central motivation: existing step representations based on independent step encoding, whether embeddings or hidden states, do not fully exploit the execution context needed for failure attribution. The results support prefix-conditioned hidden-state features as more suitable attribution inputs.

Although prefix inputs improve both baselines, ReCast retains higher Overall Hit@1 on both benchmarks. These results support our view that improving input representations should be complemented by attribution-oriented representation learning, transforming contextual features into representations that distinguish root-cause steps from other steps and further improve attribution. ReCast follows this approach to better exploit prefix-conditioned features for root-cause localization.

AuthorsOpenReview ↗

Response to Reviewer hzVo (2/2)

Q1 & W2: Task/query overlap

A2: Thank you for asking about task overlap between training and evaluation. ReCast-2K, Who&When, and TraceElephant all draw tasks from GAIA and AssistantBench. Since AssistantBench's development split contains only 33 tasks, we retained all of them during data collection, resulting in some task reuse across datasets.

Despite this partial task overlap, ReCast-2K trajectories were collected through our own independent executions rather than reused from the evaluation sets. Existing checks found no identical execution sequences shared between training and evaluation. A subset was generated using temperature = 0.1 and top-p = 0.95, retaining sampling stochasticity that allows repeated executions of the same task to produce different action sequences.

Attribution labels identify root-cause steps in specific executions rather than task answers. Different executions can contain different errors and therefore have different root-cause labels. Thus, seeing the same task during training is not equivalent to obtaining the attribution label of its evaluation trajectory.

Dear Reviewer hzVo,

Thanks for your time and effort in reviewing our paper. Your constructive comments have helped us strengthen the experimental evidence and clarify the evaluation setup. As the author–reviewer discussion period is approaching its end, we would kindly ask you to review our rebuttal, including the additional prefix-conditioned baseline comparisons and clarification of task overlap, and let us know whether it addresses your concerns. We would be happy to answer any remaining questions. Thanks again for your time and valuable feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.