- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper reframes root-cause attribution for long-horizon AI agent failures as a search problem rather than a one-shot judgment: since key evidence is sparse and scattered across huge execution logs (up to hundreds of thousands of tokens), an LLM judge tends to settle on a plausible diagnosis early and miss critical evidence. The authors propose Continual Search, an iterative framework that prompts the judge over successive turns to challenge its standing answer and actively perform operations, using a generic decision-tree-assembled prompt that generalizes across benchmarks. They also introduce MegaRCA-Mix, a 50-trial benchmark with a median log size of 286K tokens. Across long-log benchmarks, Continual Search consistently improves attribution compared with other methods.
Critical strengths
-
This paper draws an interesting finding: long-horizon RCA is not a judgment problem but a search problem.
-
The proposed method improves the performance some benchmark significantly. For example, On MegaRCA-Mix, Opus-4.8's F1 rises from 0.471 to 0.608 (+29%).
-
This paper honestly reports that the proposed method does not to help on short traces.
Critical weaknesses
-
The finding that better retrieval beats stronger reasoning may hold for this setting, but plausibly in the future. More fundamentally, attention over context is a form of retrieval, so the paper's contrast between explicit search and internal reasoning is an engineering distinction, not a mechanistic one.
-
What the judge retrieves is raw log content, whereas the target is a structured root cause (edge, fault side, failure mode). The paper should detail the mapping between these two sides
-
There is no control removing a turn—most importantly, no condition skipping turn 2—so its de-anchoring role is only argued qualitatively. In addition, costs are reported in USD, which is confounded by differing API rates; token consumption would be the fairer basis for comparing methods, and it is not clear that the baselines consume comparable amounts.
-
The paper should compare the previous work like agenTracer. A clear analysis of conceptual and empirical differences is needed to establish what is genuinely new.
-
The short-vs-long trace discrepancy raises two unanswered questions. (a) Why do the degradation factors observed on short traces (AgentRx, Who&When) not operate on long traces? (b) How to perform adaptively for a given problem? The paper fixes four turns without providing a selection rule or evidence that four is appropriate across settings.
-
MegaRCA-Mix is drawn solely from Harbor Index; adding data from other domains would strengthen claims of adaptability.
-
Figure quality could also be improved, including font legibility and the amount of information conveyed per panel.
Questions for authors
Please check weak points above.
2 comments
Dear Reviewer QRJY,
Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.
Best regards,
The Authors
Ablation isolates the search directive; token-based costs; AgenTracer comparison
Thank you for the detailed comments. Tables G1–G5 are in the general response.
W3 (controls). No ablation isolating the re-derivation step; cost in USD only.
Both are now done. For the ablation we used TRAIL, where all arms can be forked from one turn-1 session (Opus-4.8, n = 147; table G3). The turn-2 prompt has two parts: a stance that the first answer is untested, and a directive to re-read the least-covered parts of the trace. With the directive alone the judge reaches 0.529 Weighted F1 (+0.040 over Passive, p = 0.0002). With the stance alone it reaches 0.491, the same as Passive (0.489). So the effect does not come from de-anchoring or from pressure to doubt the answer; it comes from sending the judge back to evidence it under-read. We will rewrite the mechanism section accordingly.
In tokens (G4), Opus-4.8 on TRAIL: single-turn 989K per trajectory, Passive 1,663K, Continual Search 2,172K, self-consistency 4,197K, judge panel 2,251K. Continual Search costs 1.3× Passive and less than either ensemble; cut at turn 2, inside Passive's budget, it scores 0.521 against 0.488.
W1. "Retrieval vs. reasoning" is a loose dichotomy.
Fair. We will define it operationally. Our judges read the record through tools, so text they never opened is not in context at all, and no amount of reasoning in that turn can use it. A first pass opens 27% (Opus-4.8) to 42% (GPT-5.5) of trace tokens on TRAIL. The claim we can support is narrower than the slogan: effort directed back at under-read parts of the record helps, while the same or more effort spent as higher reasoning effort, ensembling, or self-doubt does not (G3, G4).
W2. How a raw log becomes a structured label.
The judge (i) finds the first step where the run departs from correct behaviour and quotes it, (ii) picks the interaction edge at that step (for example agent → tool), and (iii) picks the failure mode on that edge. The fault side follows from the mode by lookup in the taxonomy. Only edge and failure mode are parsed and scored. An example from the benchmark (
sql-14, Opus-4.8): after turns 1–2 the label is Tool Feedback Neglect at step 27. At turn 3 the judge opens the trial log, the config and the raw session, quotes the task instruction and the step-20 call that violated it, and moves the label to Suboptimal Arguments at the earlier step, which matches the benchmark label. We will add this walk-through and the full mapping to the appendix.W4. No comparison to AgenTracer.
We will cite and discuss it. AgenTracer trains a dedicated attributor (Qwen3-8B, reinforcement learning on counterfactual-replay data) that reads the whole trajectory in one pass. Continual Search trains nothing and wraps any tool-using judge, for records too large for one pass, so the two are complementary. We could not run it: the weights are not released, and its base context (32K) is below our record sizes. The only shared ground is the Who&When automated subset, where AgenTracer reports 69.6 / 42.9 agent / step accuracy and our single-turn GPT-5.5 judge gets 66.7 / 51.6. That is not a controlled comparison, and Who&When is a short-record benchmark where we do not recommend Continual Search.
W5. Degradation on short traces; why it should not appear on long ones; adaptive use.
It does appear on long records, just rarely: a correct first answer is overturned in 1 of 183 cases on long records and in 24 of 133 on short ones. On a short record the first pass has already read everything, so a further "search" turn has nothing new to find and acts only as pressure to change the answer. On a long record there is unread material to check the answer against.
For deployment we propose a size gate that needs no labels: use Continual Search only when the record exceeds 20K tokens (TELBench's own definition of a long trajectory). This scores 0.354 macro against 0.299 for always single-turn and 0.334 for always Continual Search, and is at least as good as both in all 10 benchmark × judge cells. The threshold coincides with the benchmark boundary in our data and has not been validated on held-out benchmarks, so we will present it as guidance. On turns: gains per turn are +0.039 / +0.019 / +0.003, so four is where an extra turn stops paying.
W6. Single source for MegaRCA-Mix.
Correct: it spans 25 task families but one ecosystem (Harbor Index), and we will say so. Evidence outside that ecosystem comes from TRAIL (GAIA and SWE-bench agent traces), which is also where we now show the effect across 8 judge families (G2).
W7. Figures.
We will enlarge the fonts and split the dense panels.