- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies action-level test-time scaling for terminal agents through Mid-Harness, a wrapper that samples multiple candidate actions from the same interaction history, verifies them, and forwards one candidate to the execution harness. The generator and harness control loop remain fixed. The study compares listwise, pointwise, and pairwise verification, investigates verifier distillation, and evaluates composition with trajectory selection and sequential refinement. On TerminalBench-Lite, the paper reports that a GPT-5.6 Sol verifier improves a fixed TMAX-9B generator from 50.00% to 68.03% Pass@1. Zero-shot pairwise self-verification achieves 54.76%, while verifier distillation raises this to 57.14%. Additional experiments examine decision-only verification, other models and harnesses, and several task benchmarks. The central finding is that the benefit of sampling more actions depends strongly on the verifier’s ability to select useful candidates.
Critical strengths
-
The paper addresses a practically important decision point in agent execution. Selecting an action before it changes the environment is a useful intervention for terminal agents. The implementation requires only one live environment per trajectory and avoids executing or restoring the environments of discarded candidates.
-
The fixed-generator design provides an informative separation between generation and selection. The frontier-verifier result shows that the same generator can support substantially more successful trajectories under improved action selection. Activating the distilled adapter only for verification also helps isolate the contribution of verifier training.
-
The evaluation examines several complementary aspects of the approach. Beyond the main performance comparison, the paper studies model scale, transfer across harnesses and benchmarks, composition with parallel and sequential trajectory scaling, and the trade-off between reasoning-based and decision-only verification. The decision-only results are particularly relevant to practical deployment.
Critical weaknesses
-
Pairwise verification receives substantially more compute than competing mechanisms. It uses 125.7k verifier output tokens per run, versus 7.0k for pointwise and 0.6k for listwise, alongside different reasoning prompts. The comparison therefore does not establish whether the gain comes from pairwise comparison itself. A matched-budget comparison is needed.
-
The candidate-width ablation is not fully controlled. Appendix B.2 specifies an 8,000-character history limit for zero-shot , without stating an equivalent limit for ; the tournament structure also changes. These differences should be controlled before attributing performance changes to candidate width.
-
The incremental gains are not supported by direct uncertainty estimates. The reported confidence intervals compare variants against the base agent, but do not establish the benefit of distillation—54.76% to 57.14%—or pairwise verification over other mechanisms. Paired confidence intervals are needed for these central comparisons.
-
The contribution beyond existing action-selection methods remains unclear. Prior work already studies per-step action verification and its combination with trajectory selection, while the pivot tournament is adopted from Kwok et al. A comparison with the closest existing method under the same generator and budget would clarify the empirical advance.
-
The efficiency comparison mixes compute allocation with verifier capability. The strongest Mid-Harness configuration uses a distilled verifier, whereas Best-of-T uses a zero-shot trajectory verifier. The advantage may therefore partly reflect better verification rather than where compute is allocated. A stronger trajectory selector and measured end-to-end runtimes would strengthen the claim.
Questions for authors
- Does pairwise verification retain its advantage when the mechanisms use a common evaluation rubric and comparable total verifier budgets?
- Which configurations use the 8,000-character history limit, and do the candidate-width findings persist under identical history preprocessing? 3.What are the paired task-level confidence intervals for distilled versus zero-shot verification and pairwise versus the alternative mechanisms?
- Does the cost–success advantage over trajectory scaling persist with a stronger trajectory selector and under measured, fixed-resource execution?
6 comments
Response to Reviewer LG7N (2/3)
W2. The candidate-width ablation is not fully controlled. Appendix B.2 specifies an 8,000-character history limit for zero-shot (N=4), without stating an equivalent limit for (N=8); the tournament structure also changes. These differences should be controlled before attributing performance changes to candidate width.
Q2. Which configurations use the 8,000-character history limit, and do the candidate-width findings persist under identical history preprocessing?
All N=4 and N=8 configurations use the same history preprocessing, which keeps the most recent 8,000 characters, and with the tournament also matched to N=4, distilled pairwise verification still improves from N=4 to N=8.
- The history limit is shared. Every configuration in the candidate-width comparison, including the frontier references, uses the same history preprocessing, so the candidate-width findings already hold under identical preprocessing. We have corrected Appendix B.2, whose wording suggested a narrower scope.
- With the tournament matched to N=4, distilled pairwise verification still improves with more candidates. We have run distilled pairwise verification at N=8 with the full tournament used at N=4, which compares every unordered pair. It reaches 56.12% Pass@1 and 73.47% Pass@3, against 55.44% and 70.41% at N=4, so its candidate-width trend holds under the same comparison structure. We have added this configuration to Table 6 and Appendix B.2.
W3. The incremental gains are not supported by direct uncertainty estimates. The reported confidence intervals compare variants against the base agent, but do not establish the benefit of distillation—54.76% to 57.14%—or pairwise verification over other mechanisms. Paired confidence intervals are needed for these central comparisons.
Q3. What are the paired task-level confidence intervals for distilled versus zero-shot verification and pairwise versus the alternative mechanisms?
Distillation and pairwise verification improve Pass@1 consistently across settings. Distilled verification exceeds zero-shot verification in 13 of 14 comparisons, and pairwise verification stays highest under a matched verifier budget, although the paired intervals of the three N=8 differences (+2.38 to +3.74 pp) include zero on the 98 tasks. We resample the 98 common TerminalBench-Lite tasks and keep each task's three runs together.
| Comparison (N=8) | Pass@1 difference (pp) | 95% CI (pp) |
|---|---|---|
| Distilled - Zero-shot pairwise | +2.38 | [-3.40, +8.16] |
| Zero-shot pairwise - Zero-shot listwise | +3.74 | [-3.06, +10.88] |
| Zero-shot pairwise - Zero-shot pointwise | +2.38 | [-3.40, +8.16] |
- Distillation helps consistently. Distilled verification exceeds zero-shot verification in Pass@1 in 13 of 14 comparisons across model sizes (4B, 9B, and 27B), candidate widths (N=4 and 8), compositions (alone, with Best-of-3, and with SR), and benchmarks (Tables 2, 3, and 6).
- Pairwise verification stays highest. With a common rationale instruction and comparable verifier output tokens, pairwise verification keeps the highest observed Pass@1 and Pass@3 (see our response to W1).
We have added these intervals to Appendix C.4 (Table 13), next to the existing comparisons with the base agent, and described these differences as consistent observed trends.
Response to Reviewer LG7N (3/3)
W4. The contribution beyond existing action-selection methods remains unclear. Prior work already studies per-step action verification and its combination with trajectory selection, while the pivot tournament is adopted from Kwok et al. A comparison with the closest existing method under the same generator and budget would clarify the empirical advance.
Our advance over prior work is not a new selection method but a systematic study of when action-level verification pays off and by how much (see our response to Reviewer MpQg W1). To compare with the closest existing method, we have evaluated the critic of Zainullina et al. [2] in our setting, with the same generator and training data. It falls below the base agent, while our distilled decision-only verification, which like the critic decodes no rationale, reaches 59.18% Pass@1.
- Our study shows when action-level verification improves performance and by how much. Prior work, including Zainullina et al. [2] and Kwok et al. [3], shows that per-step verification can improve agents, and we adopt the pivot tournament of Kwok et al. as one component. Our study instead shows when this verification pays off before each action changes the environment. Verification, not sampling, governs the gain (Section 4.2), the generator's own backbone can serve as the verifier at 4B, 9B, and 27B (Table 2), and action selection raises Pass@1 over both parallel and sequential trajectory scaling in all 12 comparisons (Table 2).
- In our setting, the closest existing method falls below the base agent. We evaluate the action-value critic of Zainullina et al. with the same TMAX-9B generator, N=8 candidates, and the data used for our distillation, training it to predict the terminal reward of each candidate. Unlike their separate 70B critic, it shares the generator's backbone, so the comparison holds the model and training data fixed. It reaches 48.64% Pass@1 and 66.33% Pass@3, against 50.00% and 69.39% for the base agent. Our distilled decision-only verification, which like the critic decodes no rationale but reads candidates in pairs, reaches 59.18% and 73.47% (Table 6).
W5. The efficiency comparison mixes compute allocation with verifier capability. The strongest Mid-Harness configuration uses a distilled verifier, whereas Best-of-T uses a zero-shot trajectory verifier. The advantage may therefore partly reflect better verification rather than where compute is allocated. A stronger trajectory selector and measured end-to-end runtimes would strengthen the claim.
Q4. Does the cost–success advantage over trajectory scaling persist with a stronger trajectory selector and under measured, fixed-resource execution?
In two separate checks, the composition still exceeds Best-of-7 in success: without action-verifier training, at lower token cost, and with a stronger trajectory verifier. Its advantage therefore does not come from the verifier difference alone. On the same infrastructure, its runs also take less measured time in total.
- Without action-verifier training. With the same trajectory verifier and three environment executions, zero-shot Mid-Harness raises Best-of-3 Pass@1 from 55.10% to 61.22%, above Best-of-7 (59.18%, Table 8), at 0.710 dollar per output against 0.907 dollar (our response to Reviewer x4k7 W1-2 gives the full cost table).
- With a stronger trajectory verifier. With GPT-5.6 Sol as the trajectory verifier, which sees the task instruction and agent-visible traces but no rewards, distilled Mid-Harness + Best-of-3 reaches 72.45% at 9B, against 67.35% for Best-of-7, with three rather than seven environment executions. This reverses the verifier gap, because Sol selects trajectories directly while our action verifier is TMAX-9B distilled from Sol. We compare success only in this setting, because the cost of Sol is not on the reference price scale. We have added this result to Section 6.1 and Table 8.
- Measured runtime. On the same infrastructure with TMAX-9B, one run of distilled Mid-Harness at N=8 takes 6.4 minutes on average against 3.8 minutes for the base agent, with verification taking 3.8 seconds per step. Summing per-run times as if runs execute one at a time in a single environment, followed by trajectory selection, the composition takes 20.1 minutes per task on average (0.9 minutes for selection), against 29.6 minutes for Best-of-7 (3.0 minutes for selection). We have added these measurements to Appendix B.4.
References
[2] Zainullina et al., Guided Search Strategies in Non-Serializable Environments with Applications to Software Engineering Agents
[3] Kwok et al., LLM-as-a-Verifier: A General-Purpose Verification Framework
Response to Reviewer LG7N (1/3)
We thank the reviewer for the detailed feedback. We are glad the reviewer found that (1) the paper addresses a practically important decision point, selecting an action before it changes the environment, (2) the fixed-generator design informatively separates generation from selection, and (3) the evaluation examines complementary aspects, with the decision-only results particularly relevant to practical deployment. We address each concern and question below.
With a common evaluation rubric, a common rationale instruction, and comparable verifier output tokens, pairwise verification has the highest observed Pass@1 and Pass@3, ahead of listwise by 1.36 pp and of pointwise by 3.74 pp.
We have added this matched-budget comparison to Appendix C.3 (Table 11) and cited it where Section 4.2 reports pairwise as the best evaluated mechanism.
References
[1] Zhang et al., Generative Verifiers: Reward Modeling as Next-Token Prediction