acceptodds
Under review as a conference paper at ICLR 2027

Locating Hidden Failures Makes Long-Horizon Agents More Reliable

Abstract

AI agents now carry out long tasks that take tens to hundreds of steps, yet they are judged almost entirely by whether their final answer passes. An outcome-based pass or fail signal cannot show where a trajectory went wrong or what harm the agent caused along the way. Identifying errors in an agent's thoughts and actions, step by step, would support robust monitoring, debugging, and dense, step-level rewards for training agents. This is challenging because humans cannot check long trajectories at scale, and the most visible errors are usually consequences of an earlier, hidden mistake. Existing process verification benchmarks cover only single-turn reasoning or short agentic tasks, often with artificially injected errors, and do not reflect how long-horizon agents fail in real settings, where they go wrong early, fake success, or cause real harm. We introduce Traverse, a benchmark for long-horizon agentic process verification consisting of real agent trajectories from software engineering, computer use, and bioinformatics data analysis, with human labels marking where the agent goes wrong. Current frontier models struggle to identify these process errors. Alongside the benchmark, we build a taxonomy of failure types, grouped into families, for long-horizon agents. We find that the mistakes that start a failure are most often reasoning and planning errors, and that agents take harmful, irreversible actions along the way, even in trajectories that pass the outcome check. We then train Scout, a B verifier, with supervised fine-tuning and reinforcement learning. It localizes mistakes more accurately than frontier models and transfers to a domain it never sees in training. At test time, it improves task success by percentage points on Terminal-Bench 2.0 and points on SWE-bench Verified. Our benchmark, failure taxonomy, and verifier training recipe offer a practical way to detect and localize failures in long-horizon agents and oversee them at scale.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-horizon agents, Failure localization, Scalable oversight

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YtTROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark of 1423 agent trajectories from SWE, computer use and bioinformatics domain with human annotations on each agent step. It also proposes a taxonomy of 88 failure types to further analyze the data collected. Finally, the paper trains a 4B verifier model via SFT and RL to predict the step-level scores - trained on the human curated data. On the first mistake detected by the verifier, a Best-of-N trajectory selection mechanism is put into place to improve agent performance.

Critical strengths
  • The paper tries to address an important limitation of outcome-based evaluation. That is, a final result can conceal intermeditate mistakes, making step-level analysis valuable for understanding agent reliability.
  • The proposed benchmark covers diverse agent settings across SWE, computer use and bio-informatics tasks. Along with the human annotation, this makes the benchmark quite useful.
Critical weaknesses
  • W.r.t. the definition of step label in section 2.1 - it requires all the four conditions to hold. I have a few questions on the validity of these conditions:

    • (Reasoning for condition 2) Why exclude a decision that is reasonable given evidence till that step, but nevertheless directs the task incorrectly or contributes to failure?
    • (difference between condition 3 and 4) Its unclear to me what distiguishes "leads the task in an incorrect direction" from "directly contributes to a wrong outcome". Please give concrete examples where one condition holds and the other does not.
  • The paper misses a lot of relevant works like AgentTracer [1], Aegis[2], "Efficient Forecasting of Task Failures in LLM Agents" [3] & [4]. Please discuss these works and clarify how the proposed method differs from them. This is a major drawback of this work - novelty of a work's method and findings can only be established in context to existing literature.

[1] AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?

[2] Aegis: Automated Error Generation and Attribution for Multi-Agent Systems

[3] Efficient Forecasting of Task Failures in LLM Agents through Adaptive Fault Injection

[4] Which Agent Causes Task Failures & When? On Automated Failure Attribution of LLM Multi-Agent Systems

  • No report on F1 score on correct & incorrect steps. That is, the reported trajectory-level detection and first-mistake localization metrics do not fully characterize the model's per-step judgments. Please report class-specific step-level F1 scores to show how well SCOUT recognizes both correct and incorrect steps.

  • No report on additional cost incurred. That is, the best-of-N results improve task success but require generating and judging multiple candidate trajectories. Please report the total cost.

Questions for authors
  • Please compare the error definition in Section 2.1 with the definition in [4]. They seem to be very close to me.
  • W.r.t. the new taxonomy proposed, how is it better than or different from "Where LLM Agents Fail and How They can Learn From Failures"
  • In the proposed dataset, whats the ratio of correct vs incorrect steps? If its skewed then how is that taken into consideration during training?
  • W.r.t. context limit of 128k of the model, please clarify whether every example fits within that limit. If it does not then how are longer contexts handled?

[4] Which Agent Causes Task Failures & When? On Automated Failure Attribution of LLM Multi-Agent Systems

2 comments

We thank the reviewer for the careful review and for recognizing that step-level analysis addresses an important limitation of outcome-based evaluation, and that the benchmark's coverage of software engineering, computer use, and bioinformatics tasks with human annotation makes it useful. We address each concern below and will include all changes in the revised paper.

W1 & Q1: Error definition in Section 2.1 and its relation to [4]

(1) Why condition 2. We label mistakes that the agent itself makes. If the information available to the agent points to one file and the agent edits a different file, that is the agent's mistake, and we label it as a mistake. If the information points to a file, the agent edits that file, and the real cause later turns out to be elsewhere, that is not the agent's mistake, because it followed the information it had.

(2) Conditions 3 and 4. Condition 3 is about direction; condition 4 is about consequence. Condition 3 covers a step where the agent goes the wrong way, even when the final outcome check does not catch it. For example, the agent makes a clearly wrong assumption or takes a harmful action, such as deleting an important file, and the task still passes the final test. Condition 4 covers a step that changes the result. For example, the agent takes an action that breaks the solution, the unit tests fail, and the task ends unsolved.

(3) Relation to [4]. The two are close: both look for the earliest error responsible for a failure. The main difference is that [4] defines an error by the outcome (correcting the step would make a failed run succeed), so it applies only to failed runs. We also consider tasks that pass the outcome-based test but still contain a mistake, as described under condition 3 above: 343 trajectories (14%) are solved yet contain a mistake. In addition, [4] annotates one step per failed multi-agent run, while we label every step of a single-agent trajectory. We will cite [4] and add this comparison in Section 2.1.

[4] Zhang et al., "Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems."

W2: Relation to prior work

We thank the reviewer for these references. Our paper focuses on process-level verification of long-horizon tasks by a single agent and on training a verifier, so we did not cover the multi-agent work the reviewer mentions. Our work differs from these papers in three ways. First, [1], [2], and [4] study multi-agent systems and ask which agent caused a failure, while we study a single agent over a long trajectory: what kinds of mistakes it makes and how to detect them reliably. Second, [1], [2], and [3] build their data mostly by injecting faults or by automatic replay, while we use naturally occurring trajectories with every label verified by expert human annotators. Third, [3] predicts whether a run will fail in the future and all four focus on failed runs, while we locate the mistake that has already happened, including in runs that pass the outcome check, and analyze the failure modes behind it. We will cite all four in Section 5 and add them to Table 1.

On the taxonomy of Zhu et al. [5], which we cite in Section 5: both works find that an early root-cause error cascades into later ones. The main difference is horizon and scale. Our taxonomy focuses on long-horizon agentic tasks, with trajectories of up to 200 steps, and has 88 types in 10 families built from 6,967 mistakes. Their taxonomy covers relatively shorter-horizon tasks (ALFWorld, WebShop, and GAIA) and has 17 error types built from 200 failed trajectories.

[1] Zhang et al., "AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?" [2] Kong et al., "Aegis: Automated Error Generation and Attribution for Multi-Agent Systems." [3] Sengar et al., "Efficient Forecasting of Task Failures in LLM Agents through Adaptive Fault Injection." [4] Zhang et al., "Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems." [5] Zhu et al., "Where LLM Agents Fail and How They Can Learn From Failures."

W3: Step-level F1 on correct and incorrect steps

We agree that the trajectory-level and first-mistake metrics do not fully describe per-step judgments. SCOUT gives a verdict for every step, so we report the F1 score for each class on the held-out test set:

ModelF1 on correct stepsF1 on incorrect steps
SCOUT (SFT)87.3%33.5%
SCOUT (SFT+RL)95.8%49.6%

Reinforcement learning improves both classes: F1 rises by 8.5 points on correct steps and by 16.1 points on incorrect steps.

The F1 on incorrect steps is lower for two reasons. Incorrect steps are rare, only 9.4% of all steps, and this score requires locating every incorrect step in a trajectory.

We will include the detailed step-level scores (precision, recall, and F1 for both classes) in the revised paper.

W4: Cost of best-of-N selection

Best-of-N has two costs: generating the candidate trajectories and judging them.

(1) Generation. We use 4 candidate trajectories per task on average (3 on SWE-bench Verified and 5 on Terminal-Bench 2.0), so generation costs about 4 times a single agent run. One trajectory costs about 1.10 USD, so the candidates cost about 4.40 USD per task. This cost is the same for every verifier, because all verifiers select from the same candidates.

(2) Judging. Each verifier reads every candidate once. With Gemini 3.1 Pro as the verifier, judging costs about 0.20 USD per trajectory, or 0.80 USD per task. SCOUT is a 4B open model that runs on a single GPU, so it has no API fee. It writes up to 8K tokens per trajectory, or up to 32K tokens per task. We will add a full cost breakdown to Section 4.3.

Q3: Ratio of correct to incorrect steps and how training handles it

Yes, the data is skewed. In long-horizon trajectories, correct steps far outnumber incorrect ones, and in our data there are 9.7 correct steps for every incorrect step (Table 8).

We handle this in the reinforcement learning reward (Section 4.1). The reward multiplies the recall on correct steps by the recall on incorrect steps, so a verifier that labels every step correct gets a reward of zero. To earn any reward, the verifier must find the real mistakes while leaving correct steps alone. With this reward, reinforcement learning raises the F1 on incorrect steps from 33.5% to 49.6% (see W3).

Q4: Whether every trajectory fits in the 128K context

Yes. During training data filtering, we remove trajectories that exceed 128K tokens, so every trajectory used for training fits within the context.

Reviewer_Tp1cOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

They introduce TRAVERSE, a benchmark for testing how well models can localize mistakes in agentic trajectories. To my understanding, most of their data comes from CodeTracer and TRAIL. They introduce a small set of additional human annotations, in particular on BixBench. Where the human data comes from overall is rather unclear. Additionally, they train SCOUT, a 4B parameter fine-tune from Qwen3. They train it to perform well localizing mistakes in agentic trajectories. They validate SCOUT by showing that using it as a verifier on Terminal-Bench increases the Terminal-Bench score by 8.2%. Although Terminal-Bench traces are also used for training SCOUT, meaning data-contamination is a serious concern.

Critical strengths
  1. overall good writing, with good details on metrics used.
Critical weaknesses
  1. missing attribution / limited contribution. It seems that criteria 1-3 for TRAVERSE are taken almost word for word from CodeTracer (Li et al., 2026) without attribution. Similarly, the trajectories mainly come from there, which the authors only attribute quite late in the paper. The abstact gives the impression that one of their main contribution is human annotation, which it is not.
  2. unclear how the dataset is being constructed from CodeTracer and TRAIL (Deshpande et al., 2025).
Questions for authors
  1. why did you choose to only report a single step where the trajectory went wrong in BixBench? Could it not also be that there are multiple, independent misguided steps?
  2. "The judges agree well on these labels, with Cohen’s κ of 0.66 for failure family, 0.64 for root versus cascade, and 0.81 for phase." This could be because you only use Gemini judges. Does this still hold when using judges e.g. from OpenAI or Anthropic models. In my experience, OpenAI models are the best monitors in such cases.
  3. Does SCOUT generalize to longer trajectories than are present in it's training data?
4 comments

We thank the reviewer for the careful review and for recognizing the clear writing and the detailed metrics. We address each concern below and will include all changes in the revised paper.

W1 & W2: Attribution to CodeTracer and TRAIL, and how TRAVERSE is constructed

We agree that the attribution should be earlier and more explicit. Below we clarify what comes from prior work, how TRAVERSE is built, and what we contribute.

(1) What comes from CodeTracer and TRAIL. Agentic coding trajectories (SWE-bench and TerminalBench) and their initial step labels come from CodeTracer (Li et al., 2026b) and TRAIL (Deshpande et al., 2025). We collect about 8,000 trajectories from these two sources and, after extensive filtering and label verification, keep 1,321 of them in the final benchmark. The agentic science trajectories (BixBench) do not come from either work; we generate and annotate all ourselves. We will make this credit clear in the abstract, the introduction, and Section 2.1.

(2) How TRAVERSE is constructed.

  1. Collect and filter. We collect the coding and terminal trajectories available in CodeTracer and TRAIL (five backbones, four harnesses), about 8,000 in total, and keep only the high-quality ones after filtering.

  2. Standardize and verify the labels. We parse each remaining trajectory into thought, action, and observation steps. Each trajectory is checked by at least two domain-expert annotators, another round of annotation resolves any disagreement, and trajectories that stay unresolved are removed. Steps 1 and 2 reduce the 8,000 trajectories to 2,416. After the final quality filter described in Section 2, these 2,416 trajectories are reduced to 1,321.

  3. Add agentic science. We generate the agentic science trajectories ourselves by running eight models under two harnesses on BixBench, and domain-expert annotators label the first mistake in each one from scratch. We apply the same verification and filtering as in step 2.

(3) Our contributions.

  • Benchmark. Although the agentic coding trajectories come from existing datasets, we filter them extensively, verify their labels with domain-expert annotators, and add a new domain, agentic science. The result is TRAVERSE, a benchmark for long-horizon agentic process verification with 1,423 real agent trajectories from three domains.
  • Failure analysis. A comprehensive analysis of when and why long-horizon agents fail, with 88 failure types in 10 families.
  • SCOUT. A process verifier trained with supervised fine-tuning and reinforcement learning, which localizes mistakes more accurately than frontier models and improves agents' task success at test time.

Q1: Why BixBench is annotated with only the first mistake

We follow prior work on process verification [1], which defines the task as identifying the first mistake. Our analysis also shows that the first mistake is where a failure starts and that later errors build on it (Figure 8d). TRAVERSE has the same goal, locating the first error, and BixBench is part of TRAVERSE, so we annotate the first mistake there. We appreciate the suggestion: a trajectory can contain several independent mistakes, and we will extend BixBench with labels beyond the first mistake and add the failure patterns of agentic science to our analysis.

[1] Zheng et al., "ProcessBench: Identifying Process Errors in Mathematical Reasoning."

Q2: Judge agreement beyond Gemini judges

We agree that two judges from the same model family could agree more with each other. Following the reviewer's suggestion, we add GPT-5.5, an OpenAI model, as a judge and measure its agreement with Gemini 3.1 Pro. The agreement is only slightly lower than what we report in the paper: Cohen's κ is 0.62 for failure family (0.66 in the paper), 0.61 for root versus cascade (0.64), and 0.78 for phase (0.81). The agreement therefore holds across model families.

We will add these results to Section 3.1.

Q3: Generalization to longer trajectories

We appreciate the reviewer's question. SCOUT's training trajectories average 31 steps. We agree that SCOUT is most accurate when the horizon is short: on our held-out test set, it finds the exact first-mistake step in 44% of trajectories with up to 50 steps. It still localizes errors in very long trajectories: on trajectories with more than 150 steps, about five times the training average, it finds the exact step in 33% of cases.

SCOUT also generalizes to a different domain and length distribution: on BixBench, an agentic science benchmark that it never sees in training, it reaches 68.4% exact match.

We will provide a breakdown of SCOUT's performance by trajectory length in the revised paper.

Clarification on data contamination issue

We understand the concern about overlap between training and evaluation. SCOUT never sees a best-of-N task or trajectory during training:

(1) No task overlap. 190 of the 500 SWE-bench Verified tasks and 38 of the 89 Terminal-Bench 2.0 tasks also appear in SCOUT's training data. We remove them, and Table 4 reports results only on the remaining 310 and 51 tasks, which SCOUT never saw. All results, including the agent-alone baselines, are on these tasks and averaged over 5 runs. We will clarify this in Section 4.2.

(2) No model overlap. Best-of-N trajectories are generated by Claude Opus 4.6, which is not among the models in SCOUT's training data (Table 5). New: we also run best-of-N with a second agent, Claude Haiku 5.5, on Terminal-Bench-Science: task success is 20.0% for the agent alone, 20.9% with the Gemini 3.1 Pro verifier, and 24.9% with SCOUT.

(3) Out-of-distribution domain. New: we also run best-of-N on BixBench-Verified-50, an agentic science benchmark that is fully out of distribution for SCOUT:

BenchmarkAgent aloneGemini 3.1 Pro verifierSCOUT
BixBench-Verified-5065.8%67.3%73.2%

Despite being a 4B model, SCOUT improves task success by 7.4 points over the agent alone and by 5.9 points over the Gemini 3.1 Pro verifier, on a domain it never saw in training.

We will add this table to the revised paper.

Thank you for your response.

I still have serious concerns about the training / test overlap. Not in terms of individual datapoints which overlap, but the distributions overlap, being both trained on SWE-bench + Terminal-Bench and tested on them (even while using different train / test splits). So out of distribution generalization becomes a serious concern.

What are the 150 step trajectories which you test for? How is the performance of frontier models on the 150 step trajectories?

It is still not clear to me how exactly you filter the CodeTracer and TRAIL datasets. More details would be appreciated.

We thank the reviewer for the follow-up. We answer the three points below.

1. Out-of-distribution generalization

We understand the reviewer's concern: even with disjoint tasks, SWE-bench Verified and Terminal-Bench 2.0 are in-distribution for SCOUT, because it is trained on agentic coding trajectories from the same two benchmark families. For out-of-distribution evidence, we therefore evaluate SCOUT on agentic science tasks that it never sees in training: BixBench (bioinformatics) and Terminal-Bench-Science.

Benchmark (not in training)MetricAgent aloneGPT-5.5SCOUT
BixBenchFirst-mistake exact match–63.2%68.4%
BixBenchDetection F1–80.0%84.5%
BixBench-Verified-50Best-of-N task success65.8%69.3%73.2%
Terminal-Bench-ScienceBest-of-N task success20.0%21.7%24.9%

On all of these, SCOUT, a specialized 4B verifier, outperforms the much larger general-purpose model.

2. The trajectories with more than 150 steps

The reviewer asked whether SCOUT generalizes to very long-horizon trajectories. To answer this, we evaluate on the trajectories in our held-out test set that have more than 150 steps and contain a mistake. On these same trajectories, SCOUT finds the exact first-mistake step in 32.8% of cases, compared with 29.5% for GPT-5.5 and 27.1% for Gemini 3.1 Pro.

3. How we filter CodeTracer and TRAIL

We start from about 8,000 SWE-bench and TerminalBench trajectories in CodeTracer and TRAIL, filter them down to 2,416 in four steps, and then select 1,321 of these for the benchmark. The exact steps are:

  1. Step labels. We keep only trajectories that come with step-level labels in the source data.
  2. Parsing. We keep only trajectories that parse cleanly into thought, action, and observation steps.
  3. Expert verification. At least two domain-expert annotators check the labels of each trajectory, another round of annotation resolves disagreements, and trajectories that stay unresolved are removed.
  4. Matching explanation. For each trajectory, we generate a step-by-step explanation of why each step is correct or incorrect (Section 4.1). We keep only trajectories whose explanation agrees exactly with the verified labels. 2,416 trajectories remain.

For the benchmark, we then discard trajectories with malformed or missing steps and keep both correct and incorrect trajectories in every domain (Section 2), which gives 1,321 trajectories.

So although the coding trajectories originate from CodeTracer and TRAIL, only those that pass every step above enter TRAVERSE. The agentic science trajectories (BixBench) are entirely new: we generate and annotate them ourselves, using the same verification process.

We will add these steps to Section 2 and release the filtering script with the data. We hope this clarifies the reviewer's remaining concern.

Reviewer_9K8fOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to study about finding the first error appeared in an agent task. In a long-horizon agent task, the authors find the agent may make mistakes even the outcome is correct. For example, the agent deletes files that should be kept to solve the no space left task. To study this important problem, the authors propose a benchmark, Traverse, where humans label the first error in each agent trajectory. The proposed Traverse benchmark consists of 1,423 real agent trajectories ranging from different scenarios including software engineering, computer use, agentic science, etc, produced by six model families and agent harnesses. On top of the proposed benchmark, the authors first trained a 4B expert verifier, SCOUT, using SFT and RL. SCOUT clearly outperform the best frontier model in detecting the first error in an agent trajectory.

Critical strengths
  1. The topic that this paper proposes to study is an interesting and important research problem. As frontier labs develop more advanced models and agents, it is more and more important to monitor agent's behviaor and avoid catastrophic errors that the agent may cause. In real agent execution or even in agent outcome-based RL training, it is difficult to find and diagnose how bad an intermediate agent error may cause to the system. To better help this, the authors proposed Traverse benchmark to evaluate the agent's intermediate error, which is important to model developers to assess and improve agent's reliabilities.
  2. The authors trained a small verifier SCOUT outperforming frontier models in identifying the first error in an agent task. It is promising to see this result as the small verifier can be potentially used to improve the larger frontier agent model training with fewer costs.
Critical weaknesses

My most concerns are about the statistics of the collected benchmark:

  • The benchmarks only has ~2k trajectories which is not large enough compared to exisiting agent model training set, especially considering the 2k examples are across different domains, which makes each domain evern less sufficient.
  • Table 7 caption says "SCOUT training data: all 2,416 label-faithful trajectories by source". However, adding up all the numbers together, it is only 1,423 trajectories.
  • In the test-time performance experiments, the authors applied best-of-N selection using the trained verifier and tested the SWE-bench Verified and Terminal-Bench 2.0 benchmarks. Does the training data for the verifier contain these benchmarks eval samples already? If so, the result comparison would be questionable.
  • The paper says "This yields 2,416 labelfaithful trajectories (Table 8), which we will release; we train SCOUT on 1,904 of them and hold out 258 for testing (Section 4.2)." What are the rest 2416 - 1904 - 258 = 254 trajectories?
Questions for authors

N/A

1 comment

We thank the reviewer for the careful review and for recognizing that locating intermediate agent errors is an important problem in long-horizon tasks, and that SCOUT, a small verifier, can outperform frontier models at identifying them. We address each concern below and will include all changes in the revised paper.

W1: Size of the benchmark

We would like to clarify that TRAVERSE is an evaluation benchmark, not a training set. It contains 1,423 carefully curated trajectories (44,341 steps) across three domains: 600 SWE-bench, 721 TerminalBench, and 102 BixBench, each labeled by expert human annotators. For reference, widely used agent benchmarks are of similar or smaller size (e.g., Terminal-Bench 2.0 has only 89 tasks). We believe this is enough to cover diverse long-horizon failures: our analysis in Section 3 finds 78 distinct failure types across 10 families.

Table 1 compares TRAVERSE with existing process-level benchmarks. It is the only one that combines long trajectories (20–200 steps), naturally occurring failures, and human-annotated first-mistake localization across multiple domains.

W2: Table 7 caption

Thank you for catching this. The caption is wrong; the numbers are correct. Table 7 reports the TRAVERSE benchmark statistics: 600 SWE-bench, 721 TerminalBench, and 102 BixBench trajectories, 1,423 in total. SCOUT's training data (2,416 trajectories) is reported in Table 8 and does not include BixBench. We will correct the caption to: "TRAVERSE benchmark statistics by domain (1,423 trajectories)."

W3: Overlap between SCOUT's training data and the best-of-N evaluation

We understand the concern about overlap between training and evaluation. SCOUT never sees a best-of-N task or trajectory during training:

(1) No task overlap. 190 of the 500 SWE-bench Verified tasks and 38 of the 89 Terminal-Bench 2.0 tasks also appear in SCOUT's training data. We remove them, and Table 4 reports results only on the remaining 310 and 51 tasks, which SCOUT never saw. All results, including the agent-alone baselines, are on these tasks and averaged over 5 runs. We will clarify this in Section 4.2.

(2) No model overlap. Best-of-N trajectories are generated by Claude Opus 4.6, which is not among the models in SCOUT's training data (Table 5). New: we also run best-of-N with a second agent, Claude Haiku 5.5, on Terminal-Bench-Science: task success is 20.0% for the agent alone, 20.9% with the Gemini 3.1 Pro verifier, and 24.9% with SCOUT.

(3) Out-of-distribution domain. New: we also run best-of-N on BixBench-Verified-50, an agentic science benchmark that is fully out of distribution for SCOUT:

BenchmarkAgent aloneGemini 3.1 Pro verifierSCOUT
BixBench-Verified-5065.8%67.3%73.2%

Despite being a 4B model, SCOUT improves task success by 7.4 points over the agent alone and by 5.9 points over the Gemini 3.1 Pro verifier, on a domain it never saw in training.

We will add this table to the revised paper.

W4: Remaining 254 trajectories in SCOUT's training data

Remaining 254 trajectories are the validation set. We use them only to select checkpoints while training SCOUT. The three splits are fully separate: 1,904 trajectories for training, 254 for validation, and 258 for held-out testing (2,416 in total), and no trajectory appears in more than one. We will add this split to Section 4.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.