- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a benchmark of 1423 agent trajectories from SWE, computer use and bioinformatics domain with human annotations on each agent step. It also proposes a taxonomy of 88 failure types to further analyze the data collected. Finally, the paper trains a 4B verifier model via SFT and RL to predict the step-level scores - trained on the human curated data. On the first mistake detected by the verifier, a Best-of-N trajectory selection mechanism is put into place to improve agent performance.
Critical strengths
- The paper tries to address an important limitation of outcome-based evaluation. That is, a final result can conceal intermeditate mistakes, making step-level analysis valuable for understanding agent reliability.
- The proposed benchmark covers diverse agent settings across SWE, computer use and bio-informatics tasks. Along with the human annotation, this makes the benchmark quite useful.
Critical weaknesses
-
W.r.t. the definition of step label in section 2.1 - it requires all the four conditions to hold. I have a few questions on the validity of these conditions:
- (Reasoning for condition 2) Why exclude a decision that is reasonable given evidence till that step, but nevertheless directs the task incorrectly or contributes to failure?
- (difference between condition 3 and 4) Its unclear to me what distiguishes "leads the task in an incorrect direction" from "directly contributes to a wrong outcome". Please give concrete examples where one condition holds and the other does not.
-
The paper misses a lot of relevant works like AgentTracer [1], Aegis[2], "Efficient Forecasting of Task Failures in LLM Agents" [3] & [4]. Please discuss these works and clarify how the proposed method differs from them. This is a major drawback of this work - novelty of a work's method and findings can only be established in context to existing literature.
[1] AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?
[2] Aegis: Automated Error Generation and Attribution for Multi-Agent Systems
[3] Efficient Forecasting of Task Failures in LLM Agents through Adaptive Fault Injection
[4] Which Agent Causes Task Failures & When? On Automated Failure Attribution of LLM Multi-Agent Systems
-
No report on F1 score on correct & incorrect steps. That is, the reported trajectory-level detection and first-mistake localization metrics do not fully characterize the model's per-step judgments. Please report class-specific step-level F1 scores to show how well SCOUT recognizes both correct and incorrect steps.
-
No report on additional cost incurred. That is, the best-of-N results improve task success but require generating and judging multiple candidate trajectories. Please report the total cost.
Questions for authors
- Please compare the error definition in Section 2.1 with the definition in [4]. They seem to be very close to me.
- W.r.t. the new taxonomy proposed, how is it better than or different from "Where LLM Agents Fail and How They can Learn From Failures"
- In the proposed dataset, whats the ratio of correct vs incorrect steps? If its skewed then how is that taken into consideration during training?
- W.r.t. context limit of 128k of the model, please clarify whether every example fits within that limit. If it does not then how are longer contexts handled?
[4] Which Agent Causes Task Failures & When? On Automated Failure Attribution of LLM Multi-Agent Systems
2 comments
W4: Cost of best-of-N selection
Best-of-N has two costs: generating the candidate trajectories and judging them.
(1) Generation. We use 4 candidate trajectories per task on average (3 on SWE-bench Verified and 5 on Terminal-Bench 2.0), so generation costs about 4 times a single agent run. One trajectory costs about 1.10 USD, so the candidates cost about 4.40 USD per task. This cost is the same for every verifier, because all verifiers select from the same candidates.
(2) Judging. Each verifier reads every candidate once. With Gemini 3.1 Pro as the verifier, judging costs about 0.20 USD per trajectory, or 0.80 USD per task. SCOUT is a 4B open model that runs on a single GPU, so it has no API fee. It writes up to 8K tokens per trajectory, or up to 32K tokens per task. We will add a full cost breakdown to Section 4.3.
Q3: Ratio of correct to incorrect steps and how training handles it
Yes, the data is skewed. In long-horizon trajectories, correct steps far outnumber incorrect ones, and in our data there are 9.7 correct steps for every incorrect step (Table 8).
We handle this in the reinforcement learning reward (Section 4.1). The reward multiplies the recall on correct steps by the recall on incorrect steps, so a verifier that labels every step correct gets a reward of zero. To earn any reward, the verifier must find the real mistakes while leaving correct steps alone. With this reward, reinforcement learning raises the F1 on incorrect steps from 33.5% to 49.6% (see W3).
Q4: Whether every trajectory fits in the 128K context
Yes. During training data filtering, we remove trajectories that exceed 128K tokens, so every trajectory used for training fits within the context.
We thank the reviewer for the careful review and for recognizing that step-level analysis addresses an important limitation of outcome-based evaluation, and that the benchmark's coverage of software engineering, computer use, and bioinformatics tasks with human annotation makes it useful. We address each concern below and will include all changes in the revised paper.
W1 & Q1: Error definition in Section 2.1 and its relation to [4]
(1) Why condition 2. We label mistakes that the agent itself makes. If the information available to the agent points to one file and the agent edits a different file, that is the agent's mistake, and we label it as a mistake. If the information points to a file, the agent edits that file, and the real cause later turns out to be elsewhere, that is not the agent's mistake, because it followed the information it had.
(2) Conditions 3 and 4. Condition 3 is about direction; condition 4 is about consequence. Condition 3 covers a step where the agent goes the wrong way, even when the final outcome check does not catch it. For example, the agent makes a clearly wrong assumption or takes a harmful action, such as deleting an important file, and the task still passes the final test. Condition 4 covers a step that changes the result. For example, the agent takes an action that breaks the solution, the unit tests fail, and the task ends unsolved.
(3) Relation to [4]. The two are close: both look for the earliest error responsible for a failure. The main difference is that [4] defines an error by the outcome (correcting the step would make a failed run succeed), so it applies only to failed runs. We also consider tasks that pass the outcome-based test but still contain a mistake, as described under condition 3 above: 343 trajectories (14%) are solved yet contain a mistake. In addition, [4] annotates one step per failed multi-agent run, while we label every step of a single-agent trajectory. We will cite [4] and add this comparison in Section 2.1.
[4] Zhang et al., "Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems."
W2: Relation to prior work
We thank the reviewer for these references. Our paper focuses on process-level verification of long-horizon tasks by a single agent and on training a verifier, so we did not cover the multi-agent work the reviewer mentions. Our work differs from these papers in three ways. First, [1], [2], and [4] study multi-agent systems and ask which agent caused a failure, while we study a single agent over a long trajectory: what kinds of mistakes it makes and how to detect them reliably. Second, [1], [2], and [3] build their data mostly by injecting faults or by automatic replay, while we use naturally occurring trajectories with every label verified by expert human annotators. Third, [3] predicts whether a run will fail in the future and all four focus on failed runs, while we locate the mistake that has already happened, including in runs that pass the outcome check, and analyze the failure modes behind it. We will cite all four in Section 5 and add them to Table 1.
On the taxonomy of Zhu et al. [5], which we cite in Section 5: both works find that an early root-cause error cascades into later ones. The main difference is horizon and scale. Our taxonomy focuses on long-horizon agentic tasks, with trajectories of up to 200 steps, and has 88 types in 10 families built from 6,967 mistakes. Their taxonomy covers relatively shorter-horizon tasks (ALFWorld, WebShop, and GAIA) and has 17 error types built from 200 failed trajectories.
[1] Zhang et al., "AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?" [2] Kong et al., "Aegis: Automated Error Generation and Attribution for Multi-Agent Systems." [3] Sengar et al., "Efficient Forecasting of Task Failures in LLM Agents through Adaptive Fault Injection." [4] Zhang et al., "Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems." [5] Zhu et al., "Where LLM Agents Fail and How They Can Learn From Failures."
W3: Step-level F1 on correct and incorrect steps
We agree that the trajectory-level and first-mistake metrics do not fully describe per-step judgments. SCOUT gives a verdict for every step, so we report the F1 score for each class on the held-out test set:
Reinforcement learning improves both classes: F1 rises by 8.5 points on correct steps and by 16.1 points on incorrect steps.
The F1 on incorrect steps is lower for two reasons. Incorrect steps are rare, only 9.4% of all steps, and this score requires locating every incorrect step in a trajectory.
We will include the detailed step-level scores (precision, recall, and F1 for both classes) in the revised paper.