- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the LongRCA benchmark and the RCTA method for root-cause attribution in long agent executions. LongRCA releases 1,140 human-annotated failed trajectories from five task domains and workflows (mean 156.3 steps; natural failures, no error injection), plus a 200-trajectory Mini subset. The authors propose RCTA, a training-free two-stage method that recalls failure-relevant records and compares them with upstream handoff instructions to distinguish inherited errors from newly introduced ones. RCTA achieves the highest observed point estimates on both the full and Mini benchmarks.
Critical strengths
-
The benchmark is meaningful. Existing failure-attribution benchmarks are short. LongRCA Bench targets trajectories longer
-
The 1,140 trajectories come from ordinary runs across five heterogeneous workflows, so the dataset reflects real agent failure modes
-
The method's design is tightly matched to a well-diagnosed failure mode, with empirical evidence to support it
Critical weaknesses
-
Long-horizon tasks are typically decomposable into subtasks, and sub-task-level error analysis may already be sufficient; moreover, many early errors are automatically repaired during execution, and the paper's annotation rules explicitly exclude repaired errors. The incremental value of "locating the earliest root cause" for practical debuggers therefore needs to be further argued.
-
AgenTracer should be discussed and compared, which uses counterfactual replay plus programmatic fault injection to automatically annotate 2,000+ trajectories, trains an 8B dedicated tracer.
-
The paper itself reports that among 77 cases where RCTA cited the correct root step, 21 still selected a different step as the root cause; among 126 trajectories whose input excerpts contained the root record, only 52 were correctly localized. This shows that A3's causal judgment is unreliable.
-
The benchmark's supervision signal is insufficiently precise. Inter-annotator exact-step agreement is only 40.3%, ±5-step agreement 50.3%, with a median step gap of 5 and an interquartile range of 0–46. As a benchmark, the ground truth itself exhibits substantial disagreement.
-
Table 2 reports 24.1% versus 13.2% for the strongest baseline, but the paper reports no token usage, call counts, or latency, nor does it control for cost across methods. Since RCTA is a multi-stage pipeline with repeated model calls, the comparison may confound attribution quality with additional compute. An equal-budget comparison is needed to establish that the gain is not merely from more inference budget..
-
TravelPlanner accounts for 60.1% (685/1,140), while the other four sources together contribute less than 40%. Mini has only 40 per source, and the paper explicitly states this does not support definitive source-level comparisons.
-
On the generator side, MiniMax-M2.5 produced 681 trajectories (59.7%), Kimi-K2.5 276, and Qwen3.5-Plus 183; SWE and Terminal are entirely generated by M2.5. So "cross-model diversity" exists in name only—most trajectories come from a single generator, and root-cause patterns may be tied to M2.5's characteristics.
Questions for authors
Please check weak points above.
5 comments
Response (2/2): Label Precision, Compute Cost, Source Balance, and Generator Diversity
Continued from Part 1. We address the remaining points below.
W4. Label precision and the 40.3% agreement
Thank you for this concern. The 40.3% and 50.3% figures are measured between Stage-1 candidate annotations; the released labels come from Stage-2 expert adjudication. We provide four analyses.
1. Independent reproduction. Two experts new to the annotation re-examined all 178 multiply annotated trajectories with Stage-1 disagreement, blind to the final labels, and agree with them on 81.5% and 82.6% of cases.
2. Human bounds. Individual annotators reach 47.6%–85.7% exact-step accuracy against the final labels (median 67.2%) and 61.6%–90.9% role accuracy (median 74.5%). On the same trajectories, RCTA reaches 22.9% Step Acc and 51.1% Agent Acc, far below every annotator.
3. Large gaps reflect individual errors. With pairwise agreement p² ≈ 0.403, each annotator is correct with p ≈ 0.635, and an erroneous pick on a median 120-step trajectory usually lands far from the true root, which explains the wide interquartile range. The Stage-2 candidate set contains the true root with probability 0.87–0.95.
4. Rankings are robust to label ambiguity. On the trajectories where all annotators agree, the ranking is identical to the full benchmark (RCTA 39.7%, ECHO 28.7%); of RCTA's 82 misses there, 45 are more than 20 steps from the label. If any annotator-proposed step counts as correct, RCTA rises from 24.1% to 28.4% and ECHO from 13.2% to 15.8%; the ranking is unchanged and RCTA's lead grows from 10.9 to 12.6 pp.
W5. Compute cost and comparable budgets
Thank you for this suggestion. On LongRCA-Mini, with tokens measured on DeepSeek-V4.1-Flash:
| Method | Agent Acc | Step Acc | Calls/traj. | Tokens/traj. |
|---|---|---|---|---|
| ECHO (V4-Flash) | 46.5 | 13.5 | 3.0 | 52.5K |
| FALAT (V4-Flash) | 34.5 | 3.0 | 9.6 | 167.6K |
| RCTA (V4-Flash) | 61.0 | 27.0 | 13 | 268.6K |
| TrajDebug (V4.1-Flash) | 33.0 | 17.0 | 232 | ≈5.25M |
| OpenCode agent (V4.1-Flash) | 58.5 | 29.0 | 12.1 | 361.4K |
The gain is not explained by compute. FALAT uses about 3× ECHO's tokens yet scores 10.5 pp lower; TrajDebug uses about 19.5× RCTA's tokens yet scores 10.0 pp lower. No method with a comparable or larger budget exceeds RCTA: a general log-reading agent at 1.35× RCTA's tokens performs on par (p = 0.68), while RCTA's calls are fixed in advance by trajectory length. We report calls and tokens rather than wall-clock latency, which is dominated by API load; at similar per-call cost, latency scales with the number of calls.
W6. Source imbalance
Thank you for this observation. Source sizes reflect how many qualifying failures each source yields; we release every trajectory passing quality control. Mini is designed for balanced comparison, with 40 per source and the equal-source Macro-5. RCTA has the highest Step Acc in every source (Table 2: SWE 38.3, TB2 26.2, TP 18.8, Vita 25.0, WA 33.3), so it ranks first under any source weighting. These per-source results come from Full; we draw no source-level conclusions from Mini. Full and Mini under the same weighting, Step Acc %:
| Method | Full, traj.-weighted | Full, Macro-5 | Mini, Macro-5 |
|---|---|---|---|
| All-at-once | 7.6 | 9.9 | 8.5 |
| Step-by-step | 5.3 | 6.5 | 6.5 |
| Binary search | 3.4 | 4.9 | 7.5 |
| ECHO | 13.2 | 17.3 | 13.5 |
| FALAT | 2.8 | 2.7 | 3.0 |
| RCTA | 24.1 | 28.3 | 27.0 |
We will add a Macro-5 column to the main Full table.
W7. Generator diversity
Thank you for raising this point. We agree that most trajectories come from MiniMax-M2.5 and will state this clearly. Two analyses show that conclusions are not tied to it.
1. Within-source generator split. In TravelPlanner and VitaBench, tasks are assigned to generators by a fixed-seed shuffle, so only the generator differs. RCTA leads under each generator, with Step Acc of 18.9% for MiniMax-M2.5, 18.1% for Kimi-K2.5, and 23.5% for Qwen3.5-Plus, and a significant margin over ECHO in each (McNemar p = .027, < .001, < .001).
2. New generators in single-generator sources. Given the limited rebuttal period, we have so far collected, annotated, and verified 12 new failed trajectories from other generators, 6 from Kimi-K3 and 6 from DeepSeek-V4.1-Flash, covering WebArena Verified (9) and Terminal-Bench 2 (3), and ran all six methods on DeepSeek-V4.1-Flash with unchanged prompts. Step and Agent Acc out of 12: All-at-once 0 and 7, Step-by-step 1 and 6, Binary search 0 and 5, ECHO 0 and 7, FALAT 3 and 6, RCTA 5 and 8. On trajectories from generators not previously represented in these sources, RCTA again ranks first. In the camera-ready version, we will provide at least 20–40 annotated trajectories from other generators for each single-generator source, including SWE-bench Pro.
Thank you again, especially for the comments on the debugging value of root-cause localization and on compute cost. These analyses will be included in the revision.
Dear Reviewer Z5NW,
Thank you again for your thoughtful feedback on our submission. As the discussion period draws to a close, we wanted to follow up on our rebuttal. We hope our responses addressed your concerns, and we would be happy to clarify any points that remain open.
If our responses have changed your assessment of the paper, we would appreciate your considering an update to your evaluation. Thank you for your time and consideration.
Best regards, The authors of Submission 36831
Response (1/2): Value of Root-Cause Localization, AgenTracer, and Attribution Diagnostics
We sincerely thank you for the thorough review. Your comments helped us sharpen the motivation for root-cause localization, the positioning against AgenTracer, and the interpretation of our diagnostics. We respond point by point.
W1. Value of locating the earliest unrepaired root cause
Thank you for raising this question. We clarify three points.
1. Excluding repaired errors is the intended criterion. An error that is later repaired did not cause the final failure, so fixing it would not change the outcome. A practical debugger therefore needs the earliest error that remains unrepaired, which is exactly what LongRCA labels. Targeting the earliest error overall could instead point developers to issues the agent had already fixed.
2. Subtask-level analysis is not sufficient. The subtask where the failure surfaces is often not the one that introduced it:
3. Practical value. Localizing where an error was introduced is the prerequisite for targeted repair, as in Who&When and related benchmarks. RCTA returns the responsible role, the root step, and the retrieved evidence, so a developer can check a diagnosis instead of re-reading a trajectory of median 145 steps. We will strengthen this motivation in the introduction.
W2. AgenTracer
Thank you for pointing us to AgenTracer; we will add it to the related work. A direct evaluation is not possible: the official repository states that the AgenTracer-8B weights will not be released, and no training scripts are public. We therefore provide three forms of comparison.
1. Reference comparison on Who&When. Under the same setting without ground-truth access, Step / Agent Acc (%):
The training-free RCTA is comparable to or higher than AgenTracer's reported Step Acc. Backbones and role-scoring conventions differ, so this is a reference point rather than a controlled comparison.
2. Context length. AgenTracer-8B is built on Qwen3-8B with a native 32K context, and 72% of LongRCA trajectories exceed 32K tokens. Applying it would require truncation or chunking, which is precisely the long-history setting LongRCA is designed to test.
3. Complementary designs. AgenTracer derives labels from counterfactual replay and fault injection, which requires re-executable environments. LongRCA instead consists of natural failures in real environments such as SWE-bench Pro and WebArena, where deterministic replay is generally unavailable, and its labels are human-annotated. A trained tracer could also serve as the attribution stage inside RCTA.
W3. Citation without correct localization
We appreciate the careful reading of these diagnostics. We report the 77/21 and 126/52 counts precisely to separate three stages: whether the root record reaches the input, whether it is cited, and whether it is finally chosen as the root. We read them as follows.
1. This gap is the core difficulty LongRCA measures. Seeing the root record is not the same as recognizing it as the cause rather than a consequence. That more than half of the covered roots are still missed shows that cause-versus-consequence judgment, not retrieval alone, limits long-horizon attribution.
2. The attribution stage is effective, though not solved. When RCTA cites the root, it selects it as the root in 56 of 77 cases (73%). On handoff-root trajectories (n = 41), removing the backward-attribution package A3 lowers Step Acc from 51.2% to 31.7%, and removing handoff evidence A2 lowers it to 24.4%, while non-handoff cases are unchanged. RCTA's overall Step Acc is also 10.9 pp above the strongest baseline.
3. Room for improvement. The 21 cited-but-missed cases are concrete targets for future attribution methods; we will analyze them in the appendix.
Thank you again for these comments; we will incorporate this discussion into the revision.