acceptodds
Under review as a conference paper at ICLR 2027

LongRCA Bench: Root-Cause Localization in Long-Horizon Failed Agent Trajectories

Abstract

Failure attribution is a critical component of the failure-driven agent improve- ment loop: identifying which workflow role introduced a decisive error and when can turn failed executions into actionable feedback for repair, retry, and workflow refinement. However, existing benchmarks are dominated by short execution his- tories. In Who&When, 71% of trajectories contain at most 10 recorded steps, a regime in which the visible erroneous step is often close to the final failure and failure attribution can collapse toward local failure detection. Long-horizon failures create a different diagnostic problem: an early error may be propagated, reused, validated, or left uncorrected across dozens or hundreds of subsequent steps before the execution terminates. We introduce LONGRCA BENCH, com- prising 1,140 complete naturally failed trajectories from five sources, all human- annotated with responsible roles and earliest decisive root-cause steps. Under the same DeepSeek-V4-Flash backbone and scoring protocol, five existing meth- ods achieve 30.4–36.4% exact root-step accuracy on Who&When but only 2.8– 13.2% on LONGRCA BENCH, exposing a substantial gap between detecting nearby errors and tracing distant error origins. Motivated by the benchmark’s challenges, we propose Root-Cause Trajectory Attribution (RCTA), a training- free method that combines hierarchical trajectory summarization, original-record retrieval, and upstream instruction–execution comparison to distinguish inherited errors from newly introduced ones. RCTA achieves 41.8% exact root-step accu- racy on Who&When and 24.1% on LONGRCA BENCH, compared with 36.4% and 13.2% for the strongest evaluated baselines, respectively.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, root-cause localization, failure attribution, long-horizon trajectories, datasets and benchmarks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Z5NWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the LongRCA benchmark and the RCTA method for root-cause attribution in long agent executions. LongRCA releases 1,140 human-annotated failed trajectories from five task domains and workflows (mean 156.3 steps; natural failures, no error injection), plus a 200-trajectory Mini subset. The authors propose RCTA, a training-free two-stage method that recalls failure-relevant records and compares them with upstream handoff instructions to distinguish inherited errors from newly introduced ones. RCTA achieves the highest observed point estimates on both the full and Mini benchmarks.

Critical strengths
  1. The benchmark is meaningful. Existing failure-attribution benchmarks are short. LongRCA Bench targets trajectories longer

  2. The 1,140 trajectories come from ordinary runs across five heterogeneous workflows, so the dataset reflects real agent failure modes

  3. The method's design is tightly matched to a well-diagnosed failure mode, with empirical evidence to support it

Critical weaknesses
  1. Long-horizon tasks are typically decomposable into subtasks, and sub-task-level error analysis may already be sufficient; moreover, many early errors are automatically repaired during execution, and the paper's annotation rules explicitly exclude repaired errors. The incremental value of "locating the earliest root cause" for practical debuggers therefore needs to be further argued.

  2. AgenTracer should be discussed and compared, which uses counterfactual replay plus programmatic fault injection to automatically annotate 2,000+ trajectories, trains an 8B dedicated tracer.

  3. The paper itself reports that among 77 cases where RCTA cited the correct root step, 21 still selected a different step as the root cause; among 126 trajectories whose input excerpts contained the root record, only 52 were correctly localized. This shows that A3's causal judgment is unreliable.

  4. The benchmark's supervision signal is insufficiently precise. Inter-annotator exact-step agreement is only 40.3%, ±5-step agreement 50.3%, with a median step gap of 5 and an interquartile range of 0–46. As a benchmark, the ground truth itself exhibits substantial disagreement.

  5. Table 2 reports 24.1% versus 13.2% for the strongest baseline, but the paper reports no token usage, call counts, or latency, nor does it control for cost across methods. Since RCTA is a multi-stage pipeline with repeated model calls, the comparison may confound attribution quality with additional compute. An equal-budget comparison is needed to establish that the gain is not merely from more inference budget..

  6. TravelPlanner accounts for 60.1% (685/1,140), while the other four sources together contribute less than 40%. Mini has only 40 per source, and the paper explicitly states this does not support definitive source-level comparisons.

  7. On the generator side, MiniMax-M2.5 produced 681 trajectories (59.7%), Kimi-K2.5 276, and Qwen3.5-Plus 183; SWE and Terminal are entirely generated by M2.5. So "cross-model diversity" exists in name only—most trajectories come from a single generator, and root-cause patterns may be tied to M2.5's characteristics.

Questions for authors

Please check weak points above.

5 comments
AuthorsOpenReview ↗

Response (1/2): Value of Root-Cause Localization, AgenTracer, and Attribution Diagnostics

We sincerely thank you for the thorough review. Your comments helped us sharpen the motivation for root-cause localization, the positioning against AgenTracer, and the interpretation of our diagnostics. We respond point by point.

W1. Value of locating the earliest unrepaired root cause

Thank you for raising this question. We clarify three points.

1. Excluding repaired errors is the intended criterion. An error that is later repaired did not cause the final failure, so fixing it would not change the outcome. A practical debugger therefore needs the earliest error that remains unrepaired, which is exactly what LongRCA labels. Targeting the earliest error overall could instead point developers to issues the agent had already fixed.

2. Subtask-level analysis is not sufficient. The subtask where the failure surfaces is often not the one that introduced it:

  • In the 80 trajectories with logged handoffs, 41 have their root in a handoff, i.e., the error enters through an upstream delegation and surfaces in the downstream subtask.
  • Errors of omission, such as premature completion or missing verification, leave no local error signal and surface many steps later; RCTA's Step Acc on them is only 10.3% and 11.4%.
  • Role-level attribution, the closest analogue to subtask-level analysis, is far easier than step-level localization: RCTA reaches 51.1% Agent Acc but 24.1% Step Acc on the full benchmark. Knowing the responsible role leaves most of the search unresolved.

3. Practical value. Localizing where an error was introduced is the prerequisite for targeted repair, as in Who&When and related benchmarks. RCTA returns the responsible role, the root step, and the retrieved evidence, so a developer can check a diagnosis instead of re-reading a trajectory of median 145 steps. We will strengthen this motivation in the introduction.

W2. AgenTracer

Thank you for pointing us to AgenTracer; we will add it to the related work. A direct evaluation is not possible: the official repository states that the AgenTracer-8B weights will not be released, and no training scripts are public. We therefore provide three forms of comparison.

1. Reference comparison on Who&When. Under the same setting without ground-truth access, Step / Agent Acc (%):

MethodHand-craftedAlgorithm-generated
AgenTracer-8B, reported20.7 / 63.837.3 / 63.7
RCTA, DeepSeek-V4-Flash22.4 / 60.350.8 / 68.3

The training-free RCTA is comparable to or higher than AgenTracer's reported Step Acc. Backbones and role-scoring conventions differ, so this is a reference point rather than a controlled comparison.

2. Context length. AgenTracer-8B is built on Qwen3-8B with a native 32K context, and 72% of LongRCA trajectories exceed 32K tokens. Applying it would require truncation or chunking, which is precisely the long-history setting LongRCA is designed to test.

3. Complementary designs. AgenTracer derives labels from counterfactual replay and fault injection, which requires re-executable environments. LongRCA instead consists of natural failures in real environments such as SWE-bench Pro and WebArena, where deterministic replay is generally unavailable, and its labels are human-annotated. A trained tracer could also serve as the attribution stage inside RCTA.

W3. Citation without correct localization

We appreciate the careful reading of these diagnostics. We report the 77/21 and 126/52 counts precisely to separate three stages: whether the root record reaches the input, whether it is cited, and whether it is finally chosen as the root. We read them as follows.

1. This gap is the core difficulty LongRCA measures. Seeing the root record is not the same as recognizing it as the cause rather than a consequence. That more than half of the covered roots are still missed shows that cause-versus-consequence judgment, not retrieval alone, limits long-horizon attribution.

2. The attribution stage is effective, though not solved. When RCTA cites the root, it selects it as the root in 56 of 77 cases (73%). On handoff-root trajectories (n = 41), removing the backward-attribution package A3 lowers Step Acc from 51.2% to 31.7%, and removing handoff evidence A2 lowers it to 24.4%, while non-handoff cases are unchanged. RCTA's overall Step Acc is also 10.9 pp above the strongest baseline.

3. Room for improvement. The 21 cited-but-missed cases are concrete targets for future attribution methods; we will analyze them in the appendix.

Thank you again for these comments; we will incorporate this discussion into the revision.

AuthorsOpenReview ↗

Response (2/2): Label Precision, Compute Cost, Source Balance, and Generator Diversity

Continued from Part 1. We address the remaining points below.

W4. Label precision and the 40.3% agreement

Thank you for this concern. The 40.3% and 50.3% figures are measured between Stage-1 candidate annotations; the released labels come from Stage-2 expert adjudication. We provide four analyses.

1. Independent reproduction. Two experts new to the annotation re-examined all 178 multiply annotated trajectories with Stage-1 disagreement, blind to the final labels, and agree with them on 81.5% and 82.6% of cases.

2. Human bounds. Individual annotators reach 47.6%–85.7% exact-step accuracy against the final labels (median 67.2%) and 61.6%–90.9% role accuracy (median 74.5%). On the same trajectories, RCTA reaches 22.9% Step Acc and 51.1% Agent Acc, far below every annotator.

3. Large gaps reflect individual errors. With pairwise agreement p² ≈ 0.403, each annotator is correct with p ≈ 0.635, and an erroneous pick on a median 120-step trajectory usually lands far from the true root, which explains the wide interquartile range. The Stage-2 candidate set contains the true root with probability 0.87–0.95.

4. Rankings are robust to label ambiguity. On the trajectories where all annotators agree, the ranking is identical to the full benchmark (RCTA 39.7%, ECHO 28.7%); of RCTA's 82 misses there, 45 are more than 20 steps from the label. If any annotator-proposed step counts as correct, RCTA rises from 24.1% to 28.4% and ECHO from 13.2% to 15.8%; the ranking is unchanged and RCTA's lead grows from 10.9 to 12.6 pp.

W5. Compute cost and comparable budgets

Thank you for this suggestion. On LongRCA-Mini, with tokens measured on DeepSeek-V4.1-Flash:

MethodAgent AccStep AccCalls/traj.Tokens/traj.
ECHO (V4-Flash)46.513.53.052.5K
FALAT (V4-Flash)34.53.09.6167.6K
RCTA (V4-Flash)61.027.013268.6K
TrajDebug (V4.1-Flash)33.017.0232≈5.25M
OpenCode agent (V4.1-Flash)58.529.012.1361.4K

The gain is not explained by compute. FALAT uses about 3× ECHO's tokens yet scores 10.5 pp lower; TrajDebug uses about 19.5× RCTA's tokens yet scores 10.0 pp lower. No method with a comparable or larger budget exceeds RCTA: a general log-reading agent at 1.35× RCTA's tokens performs on par (p = 0.68), while RCTA's calls are fixed in advance by trajectory length. We report calls and tokens rather than wall-clock latency, which is dominated by API load; at similar per-call cost, latency scales with the number of calls.

W6. Source imbalance

Thank you for this observation. Source sizes reflect how many qualifying failures each source yields; we release every trajectory passing quality control. Mini is designed for balanced comparison, with 40 per source and the equal-source Macro-5. RCTA has the highest Step Acc in every source (Table 2: SWE 38.3, TB2 26.2, TP 18.8, Vita 25.0, WA 33.3), so it ranks first under any source weighting. These per-source results come from Full; we draw no source-level conclusions from Mini. Full and Mini under the same weighting, Step Acc %:

MethodFull, traj.-weightedFull, Macro-5Mini, Macro-5
All-at-once7.69.98.5
Step-by-step5.36.56.5
Binary search3.44.97.5
ECHO13.217.313.5
FALAT2.82.73.0
RCTA24.128.327.0

We will add a Macro-5 column to the main Full table.

W7. Generator diversity

Thank you for raising this point. We agree that most trajectories come from MiniMax-M2.5 and will state this clearly. Two analyses show that conclusions are not tied to it.

1. Within-source generator split. In TravelPlanner and VitaBench, tasks are assigned to generators by a fixed-seed shuffle, so only the generator differs. RCTA leads under each generator, with Step Acc of 18.9% for MiniMax-M2.5, 18.1% for Kimi-K2.5, and 23.5% for Qwen3.5-Plus, and a significant margin over ECHO in each (McNemar p = .027, < .001, < .001).

2. New generators in single-generator sources. Given the limited rebuttal period, we have so far collected, annotated, and verified 12 new failed trajectories from other generators, 6 from Kimi-K3 and 6 from DeepSeek-V4.1-Flash, covering WebArena Verified (9) and Terminal-Bench 2 (3), and ran all six methods on DeepSeek-V4.1-Flash with unchanged prompts. Step and Agent Acc out of 12: All-at-once 0 and 7, Step-by-step 1 and 6, Binary search 0 and 5, ECHO 0 and 7, FALAT 3 and 6, RCTA 5 and 8. On trajectories from generators not previously represented in these sources, RCTA again ranks first. In the camera-ready version, we will provide at least 20–40 annotated trajectories from other generators for each single-generator source, including SWE-bench Pro.

Thank you again, especially for the comments on the debugging value of root-cause localization and on compute cost. These analyses will be included in the revision.

Dear Reviewer Z5NW,

Thank you again for your thoughtful feedback on our submission. As the discussion period draws to a close, we wanted to follow up on our rebuttal. We hope our responses addressed your concerns, and we would be happy to clarify any points that remain open.

If our responses have changed your assessment of the paper, we would appreciate your considering an update to your evaluation. Thank you for your time and consideration.

Best regards, The authors of Submission 36831

Reviewer_aQZpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LongRCA Bench, a dataset of 1,140 failed agent executions annotated with the responsible role and earliest decisive error step. It also proposes RCTA, a training-free pipeline that summarizes trajectories, retrieves original records, and compares upstream instructions with downstream execution. RCTA improves exact-step accuracy from 13.2% to 24.1% over the strongest evaluated baseline on the benchmark.

Critical strengths

C1. The benchmark addresses a useful problem. Complete, human-annotated histories of naturally occurring failures could support research on debugging long agent executions. The dataset’s scale and trajectory lengths make it a potentially valuable resource.

C2. RCTA shows meaningful improvements over the evaluated baselines, including the highest exact-step accuracy across five model backbones on LongRCA-Mini. The paired statistical analyses and sensitivity analysis using alternative human annotations strengthen the evaluation.

Critical weaknesses

W1. The reliability of a single exact root-step label remains my main concern. Before adjudication, annotators agreed on the exact step in only 40.3% of annotation pairs, and 38.1% differed by more than 20 steps. This suggests substantive disagreement about causation! Expert adjudication and label-sensitivity analyses help, but stable method rankings do not establish that the final references are independently reproducible. Since exact-step matching is the primary metric, this uncertainty directly affects the benchmark’s interpretation.

W2. The distinctive value of RCTA is not sufficiently established. The pipeline combines familiar techniques, and closely related approaches such as SAFARI and TrajDebug are discussed without experimental comparisons. Matched-budget comparisons would help establish what the proposed procedure adds. Moreover, none of the four overall component ablations is significant after correction, leaving limited evidence for the individual design choices. The paper also does not test whether its diagnoses improve recovery or reduce debugging effort, so its practical value remains uncertain.

Questions for authors

Q1. How do annotators distinguish an error that makes execution “irrecoverable” from an error that simply remains uncorrected in the observed run? Was an independently annotated subset used to assess the reproducibility of the final adjudicated labels?

Q2. Are comparisons with SAFARI or TrajDebug available under comparable inference budgets? A general-purpose agent allowed to read and search the logs would also be an informative baseline.

Q3. Could the authors clarify when multiple plausible root steps remain acceptable, even after excluding trajectories with independent causes? How would allowing alternative valid roots affect the eval?

As a suggestion for future work, a small diagnosis-guided repair or human-debugging study would help establish whether improvements in the benchmark metric translate into practical benefits

5 comments
AuthorsOpenReview ↗

Response (1/2): Label Reliability, Attribution Criterion, and Alternative Roots

We sincerely thank you for the thoughtful and detailed review. Label reliability was also our foremost concern in building LongRCA, and your questions helped us clarify both the annotation criterion and how the benchmark should be read. We respond point by point.

W1 and Q1. Reliability and reproducibility of the final labels

Annotation had two stages. In Stage 1, annotators independently proposed a root step and role; in Stage 2, experts examined each trajectory with all candidate labels to set the ground truth. The 40.3% exact agreement and the 38.1% of pairs more than 20 steps apart are measured between Stage-1 candidates, whereas the released labels come from Stage-2 adjudication. We add four analyses.

1. Independent reproduction of the final labels. To answer Q1 directly, two experts who had not taken part in any prior annotation or adjudication re-examined all 178 multiply annotated trajectories with Stage-1 disagreement, blind to the adjudicated labels. Their judgments agree with the final labels on 81.5% and 82.6% of these trajectories, the hardest cases in the benchmark. The remaining 33 and 31 cases will be re-reviewed for future releases.

2. Human accuracy bounds. On all multiply annotated trajectories, individual annotators reach 47.6%–85.7% exact-step accuracy against the final labels (median 67.2%). RCTA reaches 22.9% on the same trajectories, far below every annotator.

3. Large disagreements mostly reflect individual errors, not competing causes. If each annotator is correct with probability p, pairwise agreement gives p² ≈ 0.403, so p ≈ 0.635, matching pooled and per-annotator mean accuracy (61.6–65.6%). Under this model, a pair disagrees whenever either annotator errs, and on a median 120-step trajectory an erroneous pick usually lands far from the true root. The candidate set seen in Stage 2 thus contains the true root with probability 0.87–0.95, and adjudication resolves the choice from the submitted rationales.

4. Rankings do not depend on ambiguous labels. On the 136 trajectories where all annotators chose the same step and role, the method ranking is identical to the full benchmark (RCTA 39.7%, ECHO 28.7%, All-at-once 15.4%, Step-by-step 10.3%, Binary search 7.4%, FALAT 2.9%). Of RCTA's 82 misses there, 45 are more than 20 steps from the label.

Q1. "Irrecoverable" versus "uncorrected in the observed run"

Thank you for raising this distinction. The criterion is defined on the observed run, not counterfactually: the root is the earliest step that introduces a decisive error relevant to the failed requirement and that remains unrepaired at completion. Annotators trace the failed requirement backward and apply three rules:

  • A later step that propagates the error or fails to catch it is not the root; the step that introduced it is.
  • An error that is successfully repaired later is disqualified, even if it occurred earlier.
  • Each label carries a rationale citing the original records, which Stage-2 adjudication checks.

We do not claim that the run could not have been rescued after the root step. In the revision, we will replace "irrecoverable" with "decisive and unrepaired in the observed run" to match the protocol.

Q3. Multiple plausible roots and their effect on evaluation

Trajectories with independent causes or with ambiguity that adjudication could not resolve are excluded during quality control, and Step ±5 already gives tolerance for neighboring steps. To quantify the effect of alternative roots, we re-scored all methods so that any step proposed by an annotator counts as correct. This applies to the 283 multiply annotated trajectories with disagreement, which have 2.2 candidate steps on average (Step Acc, %):

MethodDisagree, final labelDisagree, any candidateFull, final labelFull, any candidate
RCTA14.832.224.128.4
ECHO8.519.113.215.8
All-at-once4.28.17.68.6
Step-by-step3.57.45.36.2
Binary search1.84.93.44.2
FALAT2.17.12.84.0

Allowing alternative roots raises absolute scores but leaves the ranking unchanged, and RCTA's lead over ECHO on the full benchmark grows from 10.9 to 12.6 pp. Even under this lenient scoring, the best method localizes fewer than one in three roots. We will report this multi-reference score in the appendix as a complement to the exact-step metric.

Thank you again for these questions, which led us to state the attribution criterion more precisely.

AuthorsOpenReview ↗

Response to Reviewer (2/2): SAFARI, TrajDebug, and General-Agent Baselines; Component Evidence

Continued from Part 1. We address the remaining points below.

W2 and Q2. Comparisons with SAFARI, TrajDebug, and a general-purpose agent

Thank you for this suggestion. SAFARI (June 2026) and TrajDebug (August 2026) are concurrent works that appeared within three months of the submission deadline. Nevertheless, for completeness and following your suggestion, we evaluated TrajDebug and a general-purpose agent on LongRCA-Mini (200 trajectories, 40 per source), scored with the released scorer. DeepSeek-V4-Flash has been retired, so the new baselines run on its successor, DeepSeek-V4.1-Flash.

MethodAgent AccStep AccStep ±5Calls/traj.Tokens/traj.
ECHO (V4-Flash)46.513.529.03.052.5K
FALAT (V4-Flash)34.53.018.09.6167.6K
RCTA (V4-Flash)61.027.041.013268.6K
TrajDebug (V4.1-Flash)33.017.038.0232≈5.25M
OpenCode agent (V4.1-Flash)58.529.041.012.1361.4K

1. General-purpose log-reading agent. We used OpenCode (v1.19.5), an open-source coding agent that can freely read, grep, and revisit the step files of each trajectory, with reference labels removed from its workspace. It is a strong baseline: its Step Acc is slightly higher than RCTA's, Step ±5 is tied, and Agent Acc is slightly lower; the Step Acc difference is not significant (exact McNemar, p = 0.68). The two methods are also complementary, as their correct exact steps cover 82 of 200 trajectories together. RCTA thus matches a strong general agent at a comparable budget, and RCTA has the highest Agent Acc; the two best methods still localize fewer than one third of roots, so the benchmark remains far from solved.

What RCTA adds is cost and control. The agent uses 1.35× RCTA's tokens because each round re-sends the dialogue and every log excerpt read so far; its cumulative prompt is a median 6.3× the trajectory length, and its cost is hard to predict (90th percentile 758K tokens, maximum 1.2M). RCTA reads each step roughly once, its number of calls is fixed in advance by trajectory length, it needs no tool runtime, and its intermediate summaries and retrieved evidence can be inspected directly.

2. TrajDebug. We ran TrajDebug with its publicly released code. It calls the model at least once per step, about 232 calls and 5.25M tokens per trajectory, roughly 19.5× RCTA's tokens, yet reaches 17.0% Step Acc, 10.0 pp below RCTA. Because its cost grows with trajectory length by design, it cannot be reduced to RCTA's budget without changing the method, so we report it at its native budget.

3. SAFARI. SAFARI's code has not been released, and reimplementing its investigation agent from the paper alone would not give a faithful comparison. The OpenCode agent above is an investigation-style baseline of the same kind. We will add SAFARI once its code is available and expand the related-work discussion of how RCTA differs from both methods.

W2. Component evidence

All four removals lower Step Acc, but at n = 200 with four Holm-corrected tests the minimum detectable effect is about 9 pp, so the end-to-end test is underpowered for gains of this size. Each component is therefore tested on the quantity it is designed to change:

  • A1, candidate retrieval: +31.5 pp reference-root coverage over an equal-size uniform control (95% CI [24, 39]).
  • A2, handoff evidence, and A3, backward attribution: on handoff-root trajectories (n = 41), removing A2 or A3 lowers Step Acc from 51.2% to 24.4% (p = .0034) or 31.7% (p = .077), while non-handoff cases are unchanged (28.2%, n = 39).
  • A4, output validation: valid-output coverage rises from 89.0% to 97.0%; 16 trajectories gain a usable output and none lose one (p = 3.1e-5).

We will clarify in Sec. 4.3 that these targeted diagnostics, not the underpowered end-to-end test, are the primary component evidence.

W2. Practical value for recovery and debugging

LongRCA targets attribution itself, as do Who&When and related benchmarks: localizing where an error was introduced is the prerequisite for any targeted repair, and a wrong root sends the developer to the wrong part of the log. RCTA also returns the responsible role, the root step, and the retrieved evidence behind its decision, so a developer can check a diagnosis instead of re-reading a trajectory of median 145 steps. We agree that a diagnosis-guided repair or human-debugging study would directly measure this downstream benefit. We will discuss it in the revision and pursue it as future work, following your suggestion.

Thank you again, especially for suggesting the general-purpose agent baseline and the multi-reference view of the labels. Both gave us a clearer picture of what LongRCA measures, and these analyses will be included in the revision.

Thank you for the additional experiments. Clarifying the root-cause criterion addresses my concerns about label reliability. Please clarify why the audit refers to 178 disputed trajectories while the alternative-root analysis refers to 283.

I also appreciate the OpenCode comparison. Its 29% exact-step accuracy versus RCTA’s 27% supports positioning RCTA as a competitive baseline. However, the different backbones, unmatched budgets, and inconclusive overall ablations leave uncertainty about the contribution of its specific design.

The rebuttal strengthens the benchmark contribution, which I consider the paper’s main value. Given my limited direct research experience in this area, I would welcome the other reviewers’ perspectives on its usefulness and remain open to raising my rating.

Reviewer_82X3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces longrca bench which includes 1140 naturally failed mutiagent trajectories with human labels.

It also proposes RCTA which is a trainign free method that retriees original records and compares with upstream handoff instructions. The author reports 24.1 exact step accuracy against 12.2 for the best baseline.

Critical strengths
  1. The benchmark fils a real gap. Existing attribution datasets are mostly short, and this one has a much longer trajectory.
  2. The evaluation seems to be very careful. It covers five backbones, paired tests with Holm correction, label- resampling sensitivity, etc. The author are also upfront about what their results do not show.
  3. The handoff analysis is also pretty informative
Critical weaknesses
  1. I guess my main concern is that the label reliability. Before adjudication, annotators agreed on the exact step only 40.3% of the time, and pairs that disagreed on role were a medain 42 steps apart. With only 24% model accuracy, it is unclear how much of the remaining gap is model failure and how much is label ambiguity. I think at least a human vs final label upper bound would help a lot :)
  2. I also think the long horizon framing is not well supported. The cross dataset gap does not isolate length and within sources, accuracy shows no consistent decline with root-to-end distance. The paper would be much stronger if it showed direclty what makes these cases hard.
  3. The dataset itself is also skewed. It looks like travelplanermakes up 60% of the data, and three sources were genreated by a single model. The full results use trajectory weighting while Mini uses equal-source weighting which makes the things hard to compare
  4. lastly I think the evidence for rcta's component is weak right now. No ablation seems to survive Holm correction, and on the 30 trajectory repatability panel RCTA ranges from 26 to 43 withi only 50-57 agreement between runs. Run to run variance looks comparable to some of the reported gains, and the compute cost relative to the baselines is not reported
Questions for authors

See weakness. Happy to discuss during rebuttal :)

4 comments
AuthorsOpenReview ↗

Response: label reliability, long-horizon framing

We sincerely thank you for the careful and constructive review. Label reliability was also our foremost concern in building LongRCA, and your comments helped us sharpen several claims. We respond point by point.

W1. Label reliability and the 40.3% agreement

In Stage 1, annotators independently proposed a root step and role; in Stage 2, experts examined each trajectory with all candidate labels to set the ground truth. The 40.3% agreement is between individual Stage-1 candidates, whereas the released labels come from Stage-2 adjudication. We add four analyses.

1. Human accuracy bounds. On all multiply annotated trajectories, individual annotators reach 47.6%–85.7% exact-step accuracy (median 67.2%) and 61.6%–90.9% role accuracy (median 74.5%). On the same trajectories, RCTA reaches 22.9% Step Acc and 51.1% Agent Acc.

2. The gap reflects method capability. On trajectories where all annotators independently chose the same root step and role, labels are essentially unambiguous, and the ranking matches the full benchmark: RCTA 39.7%, ECHO 28.7%, All-at-once 15.4%, Step-by-step 10.3%, Binary search 7.4%, FALAT 2.9% Step Acc. RCTA still falls below the human lower bound of 47.6% and 27.5 pp below the human median, although the bounds use the harder full set. Of RCTA's 82 misses here, only 13 are within 5 steps of the label, 45 are >20 steps away, and 46 also miss the role. Ambiguity-driven errors would cluster near the label.

3. Candidate-set coverage. If each annotator is correct with probability p, pairwise agreement gives p² ≈ 0.403, since coincident errors on a median 120-step trajectory are negligible. Hence p ≈ 0.635, matching observed per-annotator accuracy (61.6–65.6%). The chance that no annotator selects the true root is then about 0.13 with two annotators and 0.05 with three, so the Stage-2 candidate set contains the true answer with probability 0.87–0.95.

4. Independent verification of Stage 2. Two experts new to the annotation re-examined all 178 multiply annotated trajectories with Stage-1 disagreement, blind to the adjudicated labels. They agree with the ground truth on 81.5% and 82.6% of cases. The other 33 and 31 cases will be re-reviewed for future releases.

No human-annotated benchmark is error-free, but these results indicate the labels are reliable for benchmarking. We will add these analyses to the revision and document label changes in versioned releases. Thank you again for this valuable suggestion.

W2. Long-horizon framing

We appreciate this comment, which lets us clarify the framing and show directly what makes these cases hard. Our framing concerns the total history an attributor must search to localize the root cause (median 145 steps vs. 10 for Who&When; Sec. 1, Fig. 4). Root-to-end distance (Sec. 2.4) describes how much activity is recorded after the root; as noted there, it is not the difficulty measure the framing relies on, so the lack of a consistent within-source trend agrees with Sec. 4.2 and App. F.1. We agree the cross-source comparison alone does not isolate length, and provide direct evidence below.

1. New within-trajectory intervention. We fix each trajectory and label, varying only the visible history. The cohort is all 95 Mini trajectories over 128 steps (median 216). Nested windows of 32, 64, and 128 steps contain the root at a random offset vs. the full trajectory, always keeping the task instruction and final record. With All-at-once on DeepSeek-V4.1-Flash, from W = 32 to 64, 128, and full:

  • Step Acc falls 6.3 → 4.2 → 2.1 → 0.0%, and Agent Acc 54.7 → 48.4 → 45.3 → 37.9%.
  • Median error distance grows 7 → 14 → 25 → 40 steps.
  • Controlling for the root's relative position, each doubling of history gives OR = 0.40 [0.23, 0.70] (p = .001). W = 32 vs. full yields 19 vs. 3 discordant trajectories on Agent Acc (p < .001).

The same failure becomes harder to attribute as more history surrounds it.

2. Total length tracks difficulty within source. Within each source, total length predicts lower Step Acc for all six methods (OR 0.10–0.63 per doubling, all p < .01). This holds after controlling for root-to-end distance (pooled OR = 0.46 [0.38, 0.55]).

3. What makes long cases hard.

  • Early anchoring: across within-source length tertiles, reference roots in the first 5% of the trajectory fall from 11.2% to 3.7%, while baseline predictions there rise, FALAT 39.1% → 58.9% and All-at-once 21.1% → 29.0%. Baseline errors are mostly too early, with median signed error −12 to −39 steps.
  • Errors of omission: roots with no local error signal are hardest. RCTA Step Acc is 10.3% for premature completion and 11.4% for missing verification vs. 52.7% for wrong tool/arguments. Verifier, terminal, and planner roots (13.9 / 13.5 / 17.3%) are also harder than executor roots (27.8%).

The revision will state that the framing refers to total history length and add these analyses, with details in the appendix.

AuthorsOpenReview ↗

Response: source balance, and RCTA components/cost

Continued from Part 1. We address the remaining two points.

W3. Source imbalance, single-generator sources, and weighting

1. Full and Mini serve different purposes. Source sizes reflect how many qualifying failures each source yields; we release all that pass quality control. Mini targets low-cost comparison: proportional sampling at n = 200 would give TravelPlanner about 120 and TB2 only about 7, so Mini samples 40 per source with a fixed seed and reports the equal-source Macro-5. n = 200 is the smallest tested size meeting our predefined score-error criteria (Sec. 3.3, App. C, Table 9).

2. The source mix does not determine the ranking. RCTA leads Step Acc in every source (Table 2: SWE 38.3, TB2 26.2, TP 18.8, Vita 25.0, WA 33.3). Any source weighting averages these, so it changes absolute values, not RCTA's rank.

3. Full and Mini under the same weighting. Step Acc, %:

MethodFull, traj.-weightedFull, Macro-5Mini, Macro-5Mini, post-strat.
All-at-once7.69.98.56.9
Step-by-step5.36.56.53.6
Binary search3.44.97.57.7
ECHO13.217.313.59.6
FALAT2.82.73.02.8
RCTA24.128.327.024.7

Table 10 compares Macro-5 with Macro-5 (mean absolute deviation 1.665 pp over 18 method–metric pairs); Table 12 uses the original mixture. We will add a Macro-5 column to the Full table.

4. Conclusions do not depend on the generator. In TravelPlanner and VitaBench, tasks are assigned to three generators by a fixed-seed shuffle (App. M), so only the generator differs. Re-scoring existing predictions, Step Acc %:

MethodM2.5 (n=334)K2.5 (n=276)Q3.5+ (n=183)
All-at-once / Step-by-step8.1 / 7.22.2 / 1.810.4 / 4.4
Binary search / FALAT5.7 / 4.52.2 / 1.81.1 / 2.2
ECHO13.57.210.9
RCTA18.918.123.5
RCTA − ECHO, McNemar+5.4, p=.027+10.9, p<.001+12.6, p<.001

The other three sources were generated by MiniMax-M2.5. We collected and annotated 12 new failed trajectories, 6 from Kimi-K3 and 6 from DeepSeek-V4.1-Flash, and ran all six methods on DeepSeek-V4.1-Flash, as V4-Flash is retired. Prompts and scoring are unchanged. Step and Agent Acc, out of 12: All-at-once 0 and 7, Step-by-step 1 and 6, Binary search 0 and 5, ECHO 0 and 7, FALAT 3 and 6, RCTA 5 and 8. RCTA again ranks first. We will expand this to 20–40 per source for the camera-ready.

W4. Component evidence, variance, and compute cost

1. The method-level gain exceeds run-to-run noise. On the 30-trajectory panel, one trajectory moves accuracy by 3.3 pp. Across three runs, 6 trajectories are solved in all, 15 in none, and 9 in some, giving a run-to-run Step Acc SD of 5.8 pp at n = 30, 2.2 pp at n = 200, and 0.9 pp at n = 1,140. RCTA's gain over the strongest baseline is 10.9 pp on Full (24.1% vs. 13.2%; McNemar p ≈ 1e-14) and 10.0–14.0 pp on Mini per paired backbone (Holm-adjusted p ≤ .008, Table 13). The 50–57% exact-step agreement mostly reflects trajectories RCTA fails in every run: of 90 run pairs, only 18 differ in correctness, while 23 pick two different wrong steps.

2. Component evidence. We agree the end-to-end ablations are directional: at n = 200 with four Holm-corrected tests, the minimum detectable effect is about 9 pp. We will revise the contributions and Sec. 4.3 to present A1–A4 as design choices with directional evidence. Yet A1, A2, and A4 each move their target quantity:

  • A1, candidate retrieval: +31.5 pp reference-root coverage over an equal-size uniform control (95% CI [24, 39]).
  • A2, handoff evidence: removing it lowers Step Acc from 51.2% to 24.4% on handoff-root trajectories (n = 41, p = .0034), with no change elsewhere (28.2% → 28.2%, n = 39).
  • A4, output validation: valid-output coverage rises from 89.0% to 97.0%; 16 trajectories gain a usable output and none lose one (p = 3.1e-5).

A3 has no isolated target metric and is supported only by the end-to-end ablation.

3. Compute cost. RCTA vs. the two strongest multi-call baselines, on Mini with DeepSeek-V4-Flash:

MethodAgent AccStep AccStep ±5Calls/traj.Tokens/traj.Total
ECHO46.513.529.03.052.5K10.49M
FALAT34.53.018.09.6167.6K33.51M
RCTA61.027.041.013268.6K53.72M

More compute alone does not yield accuracy: FALAT uses about 3× ECHO's tokens yet scores 10.5 pp lower in Step Acc. At about 270K tokens per trajectory, RCTA is far cheaper than human step-level annotation.

Thank you again, especially for suggesting the human upper bound, which separates model failure from label ambiguity, and for pushing us to show directly what makes long trajectories hard and to report compute cost. These analyses substantially strengthen the paper and will all be included in the revision.

Thank you for the careful response! My concerns regarding W1 and W2 have been largely addressed by the additional analyses.

I still have some reservations regarding W4, particularly the lack of compute-matched comparisons and the limited evidence for individual component contributions. Nevertheless, I appreciate the authors' efforts to address these concerns and have raised my score accordingly. :)

Thank you for the follow-up and for raising your score. We appreciate your constructive feedback throughout the review.

In the revision, we will clarify the scope of the component-level findings, highlighting the diagnostic evidence alongside the end-to-end ablation results, and add the Mini compute-cost table for transparency. We will also discuss compute-matched comparisons and larger-scale ablations as directions for further evaluation.

Thank you again for your time and helpful suggestions.:)

Comments on the paper

1 comment
Area_Chair_ZcwpOpenReview ↗

Please engage in the discussion

Reviewers: if you have not responded to the authors' reply, please read it and engage in the discussion. Thanks!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.