acceptodds
Under review as a conference paper at ICLR 2027

Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures

Abstract

The increasing deployment of AI agents in long-horizon tasks yields massive execution logs. Diagnosing failures within these records is crucial for reliability, as it transforms outcome-level signals into actionable interventions. The sheer scale of the data renders human review impractical, driving the need for automated root-cause attribution (RCA). However, automated RCA methods using LLMs suffer from low diagnostic accuracy, especially as execution traces grow larger. They struggle because relevant information is often sparse, distributed across distant actions, and disconnected from the visible failure, reducing root-cause attribution to a massive search problem. Existing RCA methods typically rely on one-shot LLM judgments to diagnose failures from execution traces. While effective for shorter trajectories, these judges tend to settle on a plausible diagnosis early, leaving critical evidence in longer traces unexamined. We introduce Continual Search, an iterative framework that nudges the judge, over successive turns, to keep searching for unresolved diagnostic evidence. We evaluate Continual Search across four existing RCA benchmarks. Recognizing the lack of massive execution traces in current benchmarks, we introduce MegaRCA-Mix to evaluate RCA at scale. MegaRCA-Mix provides a challenging testbed of 50 human-annotated failure trials spanning long-horizon, execution-heavy tasks. Across multiple benchmark suites and model families, Continual Search consistently improves attribution performance. On MegaRCA-Mix, for example, it improves Opus-4.8's F1 score by 29%, from to . More interestingly, within the same model family, lower-tier models can even surpass their higher-tier counterparts, demonstrating that effective search supersedes raw model scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
root-cause attribution, agent failure diagnosis, LLM-as-a-judge, agent-as-a-judge, long-horizon agents, test-time search, agent evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QRJYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper reframes root-cause attribution for long-horizon AI agent failures as a search problem rather than a one-shot judgment: since key evidence is sparse and scattered across huge execution logs (up to hundreds of thousands of tokens), an LLM judge tends to settle on a plausible diagnosis early and miss critical evidence. The authors propose Continual Search, an iterative framework that prompts the judge over successive turns to challenge its standing answer and actively perform operations, using a generic decision-tree-assembled prompt that generalizes across benchmarks. They also introduce MegaRCA-Mix, a 50-trial benchmark with a median log size of 286K tokens. Across long-log benchmarks, Continual Search consistently improves attribution compared with other methods.

Critical strengths
  1. This paper draws an interesting finding: long-horizon RCA is not a judgment problem but a search problem.

  2. The proposed method improves the performance some benchmark significantly. For example, On MegaRCA-Mix, Opus-4.8's F1 rises from 0.471 to 0.608 (+29%).

  3. This paper honestly reports that the proposed method does not to help on short traces.

Critical weaknesses
  1. The finding that better retrieval beats stronger reasoning may hold for this setting, but plausibly in the future. More fundamentally, attention over context is a form of retrieval, so the paper's contrast between explicit search and internal reasoning is an engineering distinction, not a mechanistic one.

  2. What the judge retrieves is raw log content, whereas the target is a structured root cause (edge, fault side, failure mode). The paper should detail the mapping between these two sides

  3. There is no control removing a turn—most importantly, no condition skipping turn 2—so its de-anchoring role is only argued qualitatively. In addition, costs are reported in USD, which is confounded by differing API rates; token consumption would be the fairer basis for comparing methods, and it is not clear that the baselines consume comparable amounts.

  4. The paper should compare the previous work like agenTracer. A clear analysis of conceptual and empirical differences is needed to establish what is genuinely new.

  5. The short-vs-long trace discrepancy raises two unanswered questions. (a) Why do the degradation factors observed on short traces (AgentRx, Who&When) not operate on long traces? (b) How to perform adaptively for a given problem? The paper fixes four turns without providing a selection rule or evidence that four is appropriate across settings.

  6. MegaRCA-Mix is drawn solely from Harbor Index; adding data from other domains would strengthen claims of adaptability.

  7. Figure quality could also be improved, including font legibility and the amount of information conveyed per panel.

Questions for authors

Please check weak points above.

2 comments
AuthorsOpenReview ↗

Ablation isolates the search directive; token-based costs; AgenTracer comparison

Thank you for the detailed comments. Tables G1–G5 are in the general response.

W3 (controls). No ablation isolating the re-derivation step; cost in USD only.

Both are now done. For the ablation we used TRAIL, where all arms can be forked from one turn-1 session (Opus-4.8, n = 147; table G3). The turn-2 prompt has two parts: a stance that the first answer is untested, and a directive to re-read the least-covered parts of the trace. With the directive alone the judge reaches 0.529 Weighted F1 (+0.040 over Passive, p = 0.0002). With the stance alone it reaches 0.491, the same as Passive (0.489). So the effect does not come from de-anchoring or from pressure to doubt the answer; it comes from sending the judge back to evidence it under-read. We will rewrite the mechanism section accordingly.

In tokens (G4), Opus-4.8 on TRAIL: single-turn 989K per trajectory, Passive 1,663K, Continual Search 2,172K, self-consistency 4,197K, judge panel 2,251K. Continual Search costs 1.3× Passive and less than either ensemble; cut at turn 2, inside Passive's budget, it scores 0.521 against 0.488.

W1. "Retrieval vs. reasoning" is a loose dichotomy.

Fair. We will define it operationally. Our judges read the record through tools, so text they never opened is not in context at all, and no amount of reasoning in that turn can use it. A first pass opens 27% (Opus-4.8) to 42% (GPT-5.5) of trace tokens on TRAIL. The claim we can support is narrower than the slogan: effort directed back at under-read parts of the record helps, while the same or more effort spent as higher reasoning effort, ensembling, or self-doubt does not (G3, G4).

W2. How a raw log becomes a structured label.

The judge (i) finds the first step where the run departs from correct behaviour and quotes it, (ii) picks the interaction edge at that step (for example agent → tool), and (iii) picks the failure mode on that edge. The fault side follows from the mode by lookup in the taxonomy. Only edge and failure mode are parsed and scored. An example from the benchmark (sql-14, Opus-4.8): after turns 1–2 the label is Tool Feedback Neglect at step 27. At turn 3 the judge opens the trial log, the config and the raw session, quotes the task instruction and the step-20 call that violated it, and moves the label to Suboptimal Arguments at the earlier step, which matches the benchmark label. We will add this walk-through and the full mapping to the appendix.

W4. No comparison to AgenTracer.

We will cite and discuss it. AgenTracer trains a dedicated attributor (Qwen3-8B, reinforcement learning on counterfactual-replay data) that reads the whole trajectory in one pass. Continual Search trains nothing and wraps any tool-using judge, for records too large for one pass, so the two are complementary. We could not run it: the weights are not released, and its base context (32K) is below our record sizes. The only shared ground is the Who&When automated subset, where AgenTracer reports 69.6 / 42.9 agent / step accuracy and our single-turn GPT-5.5 judge gets 66.7 / 51.6. That is not a controlled comparison, and Who&When is a short-record benchmark where we do not recommend Continual Search.

W5. Degradation on short traces; why it should not appear on long ones; adaptive use.

It does appear on long records, just rarely: a correct first answer is overturned in 1 of 183 cases on long records and in 24 of 133 on short ones. On a short record the first pass has already read everything, so a further "search" turn has nothing new to find and acts only as pressure to change the answer. On a long record there is unread material to check the answer against.

For deployment we propose a size gate that needs no labels: use Continual Search only when the record exceeds 20K tokens (TELBench's own definition of a long trajectory). This scores 0.354 macro against 0.299 for always single-turn and 0.334 for always Continual Search, and is at least as good as both in all 10 benchmark × judge cells. The threshold coincides with the benchmark boundary in our data and has not been validated on held-out benchmarks, so we will present it as guidance. On turns: gains per turn are +0.039 / +0.019 / +0.003, so four is where an extra turn stops paying.

W6. Single source for MegaRCA-Mix.

Correct: it spans 25 task families but one ecosystem (Harbor Index), and we will say so. Evidence outside that ecosystem comes from TRAIL (GAIA and SWE-bench agent traces), which is also where we now show the effect across 8 judge families (G2).

W7. Figures.

We will enlarge the fonts and split the dense panels.

Dear Reviewer QRJY,

Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.

Best regards,

The Authors

Reviewer_BP4zOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes to perform iterative root cause analysis by a method called as "continual search". In this process, an LLM judge first examines the failed agent's trajectory and proposes a cause. It then performs three additional analysis rounds on top of the first analysis. In each round a predefined follow-up prompt is used. That is, if its found after first analysis that (i) evidence is spread across files, the prompt guides the LLM to inspect unread files. (ii) if a list of errors is found then the prompt guides the LLM to find missed errors. (iii) And, if the analysis gives a single causal step, the prompt guides the LLM to check for earlier mistakes w.r.t. that step.

The paper also proposes a new human labelled dataset "MegaRCA-Mix" to evaluate RCA at scale.

W.r.t. the impact on using the proposed method for RCA - it improves the RCA capability of Opus 4.8 by 29% for long horizon tasks. On shorter horizon tasks, however, there is no consistent gains observed over single-shot RCA.

Critical strengths
  • The dataset proposed by the paper contains agent trajectories that are 3x longer (w.r.t. median) than any previous datasets - making the contribution quite important for the community.
  • Continual search helps improve over single-shot RCA on long-horizon tasks using a simple approach.
Critical weaknesses
  • The proposed method lacks novelty as it only combines pre-defined follow-up prompts in a chain.
  • The experimental results have only been shared with 2 LLMs on MegaRCA-Mix dataset (only TRAIL dataset has results from 4 different LLMs). Thus, the generality of "continual search" is hard to infer - across model family and model sizes.
  • The proposed dataset, MegaRCA-Mix, though contains long traces, it only has 50 failed runs. This limits the statistical significance of the conclusions one can draw.
Questions for authors
  • How does Continual Search compare with single-shot RCA that whose prompt explicitly mentions the need for comprehensive evidence inspection, under a matched token or cost budget?
  • How were MegaRCA-Mix annotations generated and validated? How many annotators assessed each run, what was their agreement, and how were disagreements resolved?
  • Why four rounds were used in the proposed continual search method? Does performance saturate or deteriorate with more rounds?
  • Which prompt components are responsible for the gains over single-shot RCA? Can the authors separately ablate unread-evidence instructions, reconsideration instructions, and the earlier-cause revision constraint?
3 comments
AuthorsOpenReview ↗

Exhaustive single-pass baseline, component ablation, and 8 additional model families

Thank you for the review. We ran new experiments for Q1 and Q4; tables G1–G5 are in the general response.

Q1. A single-shot judge told to inspect the evidence comprehensively, at matched budget.

We tested the strongest version of this we could build: a step-by-step judge that is shown every LLM and tool span of the trace in order and must rule on each (GPT-5.5, TRAIL, n = 147; table G5).

MethodTokens / traceWeighted F1Joint
Single-turn360K0.4290.143
Step-by-step, every span1,375K0.4450.285
Continual Search952K0.4940.207

Exhaustive inspection spends 1.4× the tokens of Continual Search and gets lower Weighted F1 (−0.049 [−0.087, −0.013], p = 0.012). It does get higher Joint accuracy. Joint is a recall measure and the step-by-step judge lists 14.5 errors per trace against 8.3, so part of that is list length; we report both numbers rather than claim a clean win. Note also that the single-turn prompt already instructs the judge to examine every span before answering, so the single-turn row is itself an "inspect comprehensively" baseline. The other matched-budget controls (G4) point the same way: self-consistency and a judge panel use more tokens than Continual Search and score lower.

Q4. Which part of the prompt is responsible.

We ran this ablation (Opus-4.8, TRAIL, n = 147, all arms forked from the same turn-1 session; table G3). Keeping only the search directive ("re-read the parts you covered least and add what you find") gives 0.529 Weighted F1, +0.040 over Passive (p = 0.0002), which is the whole effect. Keeping only the stance ("your answer has not been tested") gives 0.491, the same as Passive (0.489). The active ingredient is the instruction to return to under-read evidence, not pressure on the model to doubt itself.

Q3. Why four rounds, and does it saturate.

The gain per extra turn on TRAIL is +0.040 / +0.019 / +0.006 Weighted F1 for GPT-5.5 and +0.039 / +0.019 / +0.003 for Opus-4.8, so it is flattening by turn 4 and has not turned negative. Four turns is where another turn stops paying for its tokens. We have no runs beyond four turns on long records and will say that. On benchmarks where whole files can remain unopened, the procedure ends early by itself once nothing is left unread (GPT-5.5 stops after turn 2 on 22 of 47 MegaRCA-Mix trajectories). On short records quality drops within four turns, which is why we recommend the method only above a record-size threshold (see below).

W2. Only two LLMs.

On TRAIL we now report Continual Search against Passive for 12 further judge/effort settings from 8 model families (Haiku-4.5, Sonnet-5, Opus-4.8 low effort, Fable-5, GLM-4.7, GLM-5.3, GPT-5.6-Sol, GPT-5.6-Terra; table G2). Continual Search is at or above Passive in all 12 on both metrics, pooled +0.065 Weighted F1 [+0.053, +0.081]. The gain is largest for weaker judges (Haiku-4.5: +0.11 to +0.16) and small for Sonnet-5 at high effort.

W3. 50 runs limit statistical significance.

Agreed, and we now report confidence intervals everywhere (G1). MegaRCA-Mix supports one claim at its size: multi-turn adjudication beats the single-turn judge (Opus-4.8 +0.136, 6 trajectories up and 0 down, p = 0.031). It cannot resolve Continual Search against Passive, and we will not claim that from it. That claim rests on TRAIL (n = 147), where it is significant for both judges, replicates in 4 of 4 independent reruns, and extends to the other model families above.

Q2. Annotation of MegaRCA-Mix.

Each run was labelled by one annotator, an author of this paper, working from the run records without seeing any model output. The 50 runs were sampled at random. With one annotator per run there is no agreement statistic or disagreement-resolution step to report; we will state this as a limitation and add an independent second annotation of a subset, with agreement, in the revision. From the released labels: 49 scored trajectories from 25 task families; the fault lies with the model in 25, the task owner in 16, the environment in 8 and the orchestrator in 1.

W1. Novelty relative to chains of follow-up prompts.

A chain of predefined follow-ups is exactly our Passive control, and it does not work: with Opus-4.8 on TRAIL it adds +0.006 Weighted F1 over the single-turn judge, against +0.061 for Continual Search. The ablation shows a "doubt yourself" follow-up does not work either. The contribution is identifying which follow-up helps on long records, showing it holds across model families and beats ensembling and exhaustive inspection per token, and showing where it hurts.

Dear Reviewer BP4z,

Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.

Best regards,

The Authors

Dear authors, please find my comments below.

Addressed:

  • The authors added 12 settings from 8 model families on TRAIL.

  • The authors added paired bootstrap CIs and tests. On TRAIL the gains are visible, however, as the authors themselves concede, MegaRCA-Mix supports only multi-turn > single turn claim.

  • The authors also added token based costs, and prompt clause ablation.

Not addressed (critical)

  • Annotation quality of MegaRCA-Mix is not up to the mark as each trial was labelled by a single annotator who is an author, so there is no agreement statistic and adjudication. A second annotation of a subset is promised for the revision but no evidence is provided.

  • Scope of the "search supersedes scale" claim is a bit weak - as gains are not significant for 5.6 sol or for sonnet 5 at default/max effort.

Owing to the above reasons, I have increased my score, but I lean towards rejection.

Reviewer_XGD2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper models root-cause attribution in long-horizon agent failures as an evidence-search problem, observing that one-shot LLM judges often suffer from premature commitment by settling on plausible symptoms while leaving critical evidence unread. The authors propose Continual Search, an iterative prompting framework where a generic decision tree assembles prompts across turns 2–4 to guide an agentic judge to independently re-derive its diagnosis, open unread files/spans, and restrict updates to earlier divergence points supported by quotes. The authors also introduce MegaRCA-Mix, a suite of 50 human-annotated failure trials from Harbor Index containing complete execution artifacts. Across five RCA benchmarks, Continual Search improves attribution over single-turn baselines, passive reconsideration, self-consistency, and judge panels on long-horizon logs. Conversely, on short trajectories like AgentRx and Who&When, additional turns degrade accuracy due to conversational pressure acting on an already fully ingested context.

Critical strengths
  1. The evaluation spans five diverse benchmarks and tests key alternatives (self-consistency, judge panels, reasoning effort scaling). The authors transparently report that Continual Search degrades performance on short traces (AgentRx, Who&When) due to conversational drift, establishing clear operational boundaries.

  2. Correct attribution determines whether a fix targets the model, harness, environment, or grader. The paper raised a real problem that existing RCA benchmarks are short on this front. It introduced MegaRCA-Mix, a benchmark that adds records an order of magnitude larger, includes non-trajectory artifacts, and labels fault side.

Critical weaknesses
  1. Limited methodological novelty: Continual Search is primarily an iterative prompting strategy, and its contribution over existing prompting approaches is not fully isolated. The evaluation does not clearly disentangle the benefits of targeted evidence acquisition from those of additional computation or stronger prompting.

  2. Small dataset size and limited statistical power: Key evaluation suites are very small—MegaRCA-Mix contains only 50 trials and TELBench has 33. Headline improvements on these benchmarks often represent shifts in just 1 or 2 trajectories. Also, the paper lacks confidence intervals or statistical significance tests to establish that these gains are robust against sample variance.

  3. Inconsistent performance and degradation on short trajectories: The method does not generalize across execution scales; on shorter trajectories (AgentRx and Who&When), Continual Search fails to help and frequently degrades accuracy. Because the initial pass already processes all available context, subsequent prompting acts as mere conversational pressure, causing judges to flip away from correct diagnoses.

Questions for authors
  1. Mechanism ablation: How much performance is retained if you ablate (a) the {unread_block} injection, (b) the Turn 2 re-derivation prompt, and (c) the "earlier cause only" constraint?

  2. Coverage discrepancy: Given that ~85% of MegaRCA-Mix F1 gains occur at Turn 2 under virtually identical coverage to Passive Continuation, how do you substantiate the claim that new evidence search is the primary driver of performance?

  3. Statistical uncertainty: Can you provide paired bootstrap confidence intervals, McNemar tests, and multi-run standard deviations for MegaRCA-Mix and TELBench?

  4. Benchmark curation: Can you document the MegaRCA-Mix annotation process including number of annotators, expertise, inter-annotator agreement, and the proportion of trials with multiple acceptable gold sets?

  5. Prompt details: What are the "phase-1 votes" in the MegaRCA-Mix Turn 1 prompt, which models generated them, and do they risk label leakage or anchoring?

3 comments
AuthorsOpenReview ↗

Component ablation run, confidence intervals added, coverage claim clarified

Thank you for the precise review. Your questions led us to run a new ablation. Tables G1–G5 are in the general response.

Q1. Which component matters.

We ran it on TRAIL, where every arm is forked from the same turn-1 session and differs only in the turn-2 prompt (Opus-4.8, n = 147, table G3). The prompt has two substantive clauses, a stance ("your answer is untested") and a search directive ("re-read the parts you covered least and add what you find").

  • Search directive only: 0.529 Weighted F1, +0.040 over Passive [+0.019, +0.062], p = 0.0002.
  • Stance only: 0.491, +0.002 over Passive [−0.010, +0.014].
  • Full prompt: 0.521.

The directive to return to under-read evidence carries the entire gain; the "doubt your answer" framing carries none, although that judge does more work than Passive (2.7 vs 1.7 tool calls). For the earlier-cause-only rule (c) our evidence is stability rather than an ablation: on long records Continual Search overturned 1 of 183 correct turn-1 answers. MegaRCA-Mix (n = 49) is too small to separate components of this size.

Q2. Coverage is equal at turn 2, yet most of the gain arrives there.

This is a fair reading of the MegaRCA-Mix figure, and we will not claim that coverage explains the turn-2 gain there. The cleaner test is TRAIL, where the arms share turn 1: Continual Search reads more of the trace than Passive (Opus-4.8 30.8% vs 28.1% of trace tokens; GPT-5.5 51.0% vs 41.9%), 51 of 147 trajectories gain a correct category at turn 2 against 10 of 147 under Passive, and the ablation shows the gain disappears when the search directive is removed and survives when everything else is removed. We will state the mechanism as "the continuation must direct the judge back to the record" and drop wording that equates the gain with a coverage percentage.

Q3 / W2. Confidence intervals and seeds.

Added throughout (G1). On TRAIL the Continual Search − Passive difference is significant for both judges on both metrics, replicates in 4 of 4 independent Opus-4.8 reruns, and holds in direction for all 12 additional judge/effort settings (G2; pooled +0.065 Weighted F1 [+0.053, +0.081]). You are right about the small benchmarks. On TELBench (n = 33) each cell differs by one or two trajectories. On MegaRCA-Mix (n = 49) the gain over the single-turn judge is significant (Opus-4.8 +0.136, 6 up / 0 down, p = 0.031), but the difference to Passive depends on the Passive seed and is not resolved. We will present both as descriptive and rest the Continual-vs-Passive claim on TRAIL.

W1. Evidence acquisition vs. simply more computation.

All in tokens (G4, G5). Passive is more computation in the same session: +674K tokens over single-turn buys +0.006 Weighted F1 (Opus-4.8). Self-consistency and the judge panel use more tokens than Continual Search and score lower. Exhaustive step-by-step inspection of every span uses 1.4× the tokens for lower Weighted F1. The stance-only arm spends more effort than Passive for no gain. What helps is where the extra computation is pointed.

W3. Degradation on short traces.

We agree, and it is statistically real: pooled over AgentRx and Who&When, −0.052 [−0.101, −0.005]; a correct first answer is overturned in 24 of 133 cases on short records against 1 of 183 on long ones. The practical rule is a record-size gate. Applied without labels at 20K tokens (TELBench's own long-trajectory threshold), it scores 0.354 macro against 0.299 for always single-turn and 0.334 for always Continual Search, and is at least as good as both in all 10 benchmark × judge cells. It is not validated on held-out benchmarks, so we present it as guidance.

Q5. Phase-1 votes.

One vote per trajectory, produced by Opus-4.8 with the single-turn protocol. The identical vote is shown to both arms and both judges (checked in all 294 sessions), so it cannot produce a between-arm difference. It does mean the MegaRCA-Mix adjudicator starts from an Opus-4.8 answer even when the judge is GPT-5.5; we will state this in the setup.

Q4. Curation of MegaRCA-Mix.

Each trajectory was labelled by one annotator, an author of this paper, from the run records and without seeing any model output. The 50 trajectories were drawn at random from the Harbor Index runs. With a single annotator there is no agreement statistic; we will state this as a limitation and add an independent second annotation of a subset, with agreement, in the revision. One trajectory has alternative acceptable label sets; ten require 2–4 root causes, with partial credit. Scoring uses 49 trajectories because one labelled task has no run.

Novelty.

The method is simple; the finding is not obvious. On long records a neutral "reconsider" follow-up, a "doubt yourself" follow-up, ensembling, and exhaustive span-by-span inspection all fail to deliver what a short directive back to under-read evidence delivers, across 8 model families, and we show where it backfires.

Dear Reviewer XGD2,

Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.

Best regards,

The Authors

Thank you for the detailed rebuttal, the new experiments and your recalibration of the claims. The component ablation on TRAIL (G3) separates the search directive (+0.040+0.040 W-F1) from generic self-doubt (+0.002+0.002 W-F1). The comparison against exhaustive step-by-step inspection (G5) provides a useful empirical takeaway.

However, looking at the revised picture as a whole, I am keeping my score at 2 for four main reasons:

  1. The core statistical advantage over passive reconsideration rests only on one existing benchmark TRAIL. The paper evaluates five datasets. Two of them (AgentRx and Who&When) show statistically significant degradation. The other two long horizon datasets, MegaRCA-Mix (n=49) and TELBench (n=33), are treated as unresolved against Passive Continuation in the rebuttal. This leaves TRAIL (n=147) as the only benchmark where Continual Search statistically outperforms Passive Continuation.

  2. There are limited gains on strong frontier models. As shown in G2, the positive pooled effect is driven largely by weaker models. On stronger judges, the gains are negligible or statistically indistinguishable from zero.

  3. MegaRCA-Mix is too preliminary to be seen as a standalone benchmark contribution. We now know MegaRCA-Mix has n=49 usable trials, annotated by a single author with no inter-annotator agreement, and injects an Opus-4.8 "phase-1 vote" into the Turn-1 prompt for both judge models (meaning GPT-5.5 is not diagnosing from scratch). Combined with the fact that it fails to separate Continual Search from Passive Continuation, the benchmark needs another round of curation before publication.

  4. Modest methodological novelty. Now we know that TRAIL is the only benchmark where Continual Search statistically outperforms Passive Continuation. It is a single JSON trace rather than a multi-file environment, so it doesn't use the {unread_block} mechanism. On TRAIL, the surviving method is simply a generic prompt sentence instructing the judge to re-read the parts of the trace it covered least on the first pass.

I appreciate the rigor of your rebuttal analyses but the overall contribution remains just below my acceptance threshold.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2): New ablation shows the search directive alone carries the gain and the effect holds across 8 model families

We thank all three reviewers for the careful reading. The reviews converge on four requests: (i) statistical uncertainty, (ii) evidence for which part of the prompt is responsible, (iii) a budget comparison in tokens rather than dollars, and (iv) a stronger "inspect everything" single-pass baseline. We ran all four during the rebuttal period. The tables below are referenced from the individual responses as G1–G5.

G1. Uncertainty on the main comparison (TRAIL, n = 147)

On TRAIL the Passive arm is forked from the Continual Search turn-1 session, so the two arms share an identical first answer and differ only in the continuation prompt. Differences are paired over trajectories (10,000 bootstrap resamples; paired permutation test).

JudgeMetricPassiveContinual SearchDifference [95% CI]
GPT-5.5Weighted F10.4540.494+0.040, p = 0.001
GPT-5.5Joint Acc.0.1800.207+0.027 [+0.001, +0.051], p = 0.031
Opus-4.8Weighted F10.4880.543+0.056 [+0.032, +0.084], p < 0.0001
Opus-4.8Joint Acc.0.1350.152+0.017 [+0.008, +0.028], p = 0.0005 (12 up, 0 down)

Four fully independent Opus-4.8 reruns of both arms give +0.042 ± 0.018 Weighted F1 and +0.046 ± 0.008 Joint; Continual Search is higher in 4 of 4. (For GPT-5.5 we now report the Passive run in which all tool calls succeeded; Table 2 will change from 0.459 / 0.170 to the values above.)

Against the single-turn judge, pooled over the three long-record benchmarks and both judges: +0.092 [+0.058, +0.131]; 71 trajectories improve and 1 gets worse.

G2. The effect is not specific to two judges (TRAIL, same protocol)

We scored the Passive and Continual Search arms for every additional judge and effort setting we have run on TRAIL, without selection: 12 settings from 8 model families.

Judge (effort)Passive W-F1Continual Search W-F1Difference [95% CI]
Haiku-4.5 (low / high / max)0.270 / 0.300 / 0.2950.407 / 0.410 / 0.454+0.137 / +0.110 / +0.160, all p < 0.0001
GLM-4.70.3760.473+0.096 [+0.057, +0.143]
GLM-5.30.4800.539+0.059 [+0.026, +0.094]
GPT-5.6-Terra0.4000.447+0.047 [+0.011, +0.084]
GPT-5.6-Sol0.5040.517+0.013 [−0.013, +0.041]
Sonnet-5 (low / default / max)0.460 / 0.510 / 0.5080.521 / 0.518 / 0.511+0.061 / +0.009 / +0.002
Opus-4.8 (low)0.4760.530+0.055 [+0.028, +0.085]
Fable-50.4920.528+0.036 [+0.012, +0.065]

Continual Search is at or above Passive in 12 of 12 settings (8 of 8 families) on both metrics; individually significant in 9 of 12 on Weighted F1 and 10 of 12 on Joint. Pooled: +0.065 Weighted F1 [+0.053, +0.081], +0.040 Joint [+0.031, +0.049]. The gain is largest for weak judges and small for Sonnet-5 at high effort. These runs use the hand-written version of the prompts (the appendix variant), not the template version behind Table 2.

G3. Component ablation (new runs; Opus-4.8, TRAIL, n = 147, turn 2)

The Continual Search continuation prompt has two substantive clauses: a stance ("your list has not been tested; do not treat it as established") and a search directive ("re-read the trace, focusing on the parts you covered least, and add newly found errors"). We removed each in turn, forking the same turn-1 sessions used for the other arms.

Turn-2 promptWeighted F1Jointvs Passive (W-F1) [95% CI]
(turn 1, shared)0.4820.133
Passive ("reconsider and reverify")0.4890.136
Stance only0.4910.135+0.002 [−0.010, +0.014]
Search directive only0.5290.158+0.040 [+0.019, +0.062], p = 0.0002
Full Continual Search0.5210.144+0.032 [+0.014, +0.053], p = 0.001

The search directive alone reproduces the full gain; the stance alone reproduces none of it (search-only − stance-only = +0.038 [+0.017, +0.060], p = 0.0005). Telling the judge to doubt its answer makes it work more (2.7 tool calls vs 1.7 for Passive) without changing the outcome; telling it to go back to under-read parts of the record is what helps. This is one judge, one run per arm, at turn 2; the full-vs-search-only difference (−0.007 [−0.026, +0.011]) is within run noise.

(Continued in part 2: G4 token budget, G5 exhaustive baseline, changes to the paper.)

AuthorsOpenReview ↗

General Response (2/2): Continual Search beats ensembles and exhaustive inspection at lower token cost

G4. Budget in tokens (TRAIL, mean total tokens per trajectory)

MethodGPT-5.5 tokensW-F1Opus-4.8 tokensW-F1
Single-turn360K0.429989K0.482
Passive Continuation, 4 turns771K0.4591,663K0.488
Continual Search, 4 turns952K0.4942,172K0.543
Self-consistency, 4 samples1,419K0.3834,197K0.430
Judge panel2,251K0.4312,251K0.431

Continual Search uses fewer tokens than either ensemble and scores higher. It uses 1.2–1.3× the tokens of Passive; stopped after turn 2, where it is inside Passive's four-turn budget, it is still ahead (0.469 vs 0.459 for GPT-5.5; 0.521 vs 0.488 for Opus-4.8).

G5. Exhaustive single-pass baseline (GPT-5.5, TRAIL, n = 147)

As the strongest form of "inspect everything", we scored a step-by-step judge that is shown every LLM/tool span of the trace in order and must rule on each one (18 model calls per trace on average).

MethodTokensWeighted F1JointErrors listed per trace
Single-turn360K0.4290.1435.1
Step-by-step (every span)1,375K0.4450.28514.5
Continual Search952K0.4940.2078.3

Exhaustive inspection costs 1.4× the tokens of Continual Search and has lower Weighted F1 (−0.049 [−0.087, −0.013], p = 0.012). It has higher Joint accuracy; Joint is a recall measure and step-by-step lists 75% more errors per trace, so we report both and do not claim Continual Search dominates it on every metric.

Changes to the paper

  1. Confidence intervals, tests and up/down counts in Table 2; TRAIL presented as the primary evidence for Continual Search vs Passive. On MegaRCA-Mix (n = 49) and TELBench (n = 33) the gain over the single-turn judge stands, but the difference to Passive is not resolved at these sample sizes and we will say so.
  2. G2 and G3 added as new results; the mechanism section rewritten around the search directive.
  3. Token-based cost next to USD everywhere.
  4. The short-record degradation presented as a boundary condition, with a record-size rule for when to use the method.
  5. Annotation protocol for MegaRCA-Mix, AgenTracer and related work, and redrawn figures.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.