acceptodds
Under review as a conference paper at ICLR 2027

Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents

Abstract

Terminal agents act through stochastic model generations, yet the ability to generate a useful action does not ensure its reliable execution. A poor command (e.g., wrong package install) can change the environment in ways that hinder subsequent progress, even when the model could generate a better alternative. We investigate whether allocating test-time compute at the model-harness boundary can improve action reliability and trajectory success, and what makes this allocation effective. To study these questions, we introduce Mid-Harness, which samples and verifies candidate actions before forwarding one for execution, while keeping the generator and harness unchanged. With a TMAX-9B generator, more action sampling yields little benefit under weak verification, whereas a capable verifier can exploit useful alternatives from the same generator. On TerminalBench-Lite, a GPT-5.6 Sol verifier raises Pass@1 from 50.00% for the base agent to 68.03% with 8 sampled actions. When the same TMAX-9B model serves as the verifier, pairwise verification performs best among the evaluated verification mechanisms. Distilling responses from the stronger verifier into TMAX-9B further improves Pass@1, while leaving the action generator unchanged. With TMAX-9B on TerminalBench-Lite, combining action and trajectory scaling reaches higher success at lower estimated token cost than generating more trajectories alone. Mid-Harness also improves performance across additional models, benchmarks, and harnesses. These findings identify action scaling as a promising target for test-time compute scaling in terminal agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, agent, test-time scaling, harness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_LG7NOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies action-level test-time scaling for terminal agents through Mid-Harness, a wrapper that samples multiple candidate actions from the same interaction history, verifies them, and forwards one candidate to the execution harness. The generator and harness control loop remain fixed. The study compares listwise, pointwise, and pairwise verification, investigates verifier distillation, and evaluates composition with trajectory selection and sequential refinement. On TerminalBench-Lite, the paper reports that a GPT-5.6 Sol verifier improves a fixed TMAX-9B generator from 50.00% to 68.03% Pass@1. Zero-shot pairwise self-verification achieves 54.76%, while verifier distillation raises this to 57.14%. Additional experiments examine decision-only verification, other models and harnesses, and several task benchmarks. The central finding is that the benefit of sampling more actions depends strongly on the verifier’s ability to select useful candidates.

Critical strengths
  • The paper addresses a practically important decision point in agent execution. Selecting an action before it changes the environment is a useful intervention for terminal agents. The implementation requires only one live environment per trajectory and avoids executing or restoring the environments of discarded candidates.

  • The fixed-generator design provides an informative separation between generation and selection. The frontier-verifier result shows that the same generator can support substantially more successful trajectories under improved action selection. Activating the distilled adapter only for verification also helps isolate the contribution of verifier training.

  • The evaluation examines several complementary aspects of the approach. Beyond the main performance comparison, the paper studies model scale, transfer across harnesses and benchmarks, composition with parallel and sequential trajectory scaling, and the trade-off between reasoning-based and decision-only verification. The decision-only results are particularly relevant to practical deployment.

Critical weaknesses
  • Pairwise verification receives substantially more compute than competing mechanisms. It uses 125.7k verifier output tokens per run, versus 7.0k for pointwise and 0.6k for listwise, alongside different reasoning prompts. The comparison therefore does not establish whether the gain comes from pairwise comparison itself. A matched-budget comparison is needed.

  • The candidate-width ablation is not fully controlled. Appendix B.2 specifies an 8,000-character history limit for zero-shot N=4N=4, without stating an equivalent limit for N=8N=8; the tournament structure also changes. These differences should be controlled before attributing performance changes to candidate width.

  • The incremental gains are not supported by direct uncertainty estimates. The reported confidence intervals compare variants against the base agent, but do not establish the benefit of distillation—54.76% to 57.14%—or pairwise verification over other mechanisms. Paired confidence intervals are needed for these central comparisons.

  • The contribution beyond existing action-selection methods remains unclear. Prior work already studies per-step action verification and its combination with trajectory selection, while the pivot tournament is adopted from Kwok et al. A comparison with the closest existing method under the same generator and budget would clarify the empirical advance.

  • The efficiency comparison mixes compute allocation with verifier capability. The strongest Mid-Harness configuration uses a distilled verifier, whereas Best-of-T uses a zero-shot trajectory verifier. The advantage may therefore partly reflect better verification rather than where compute is allocated. A stronger trajectory selector and measured end-to-end runtimes would strengthen the claim.

Questions for authors
  1. Does pairwise verification retain its advantage when the mechanisms use a common evaluation rubric and comparable total verifier budgets?
  2. Which configurations use the 8,000-character history limit, and do the candidate-width findings persist under identical history preprocessing? 3.What are the paired task-level confidence intervals for distilled versus zero-shot verification and pairwise versus the alternative mechanisms?
  3. Does the cost–success advantage over trajectory scaling persist with a stronger trajectory selector and under measured, fixed-resource execution?
6 comments
AuthorsOpenReview ↗

Response to Reviewer LG7N (1/3)

We thank the reviewer for the detailed feedback. We are glad the reviewer found that (1) the paper addresses a practically important decision point, selecting an action before it changes the environment, (2) the fixed-generator design informatively separates generation from selection, and (3) the evaluation examines complementary aspects, with the decision-only results particularly relevant to practical deployment. We address each concern and question below.


W1. Pairwise verification receives substantially more compute than competing mechanisms. It uses 125.7k verifier output tokens per run, versus 7.0k for pointwise and 0.6k for listwise, alongside different reasoning prompts. The comparison therefore does not establish whether the gain comes from pairwise comparison itself. A matched-budget comparison is needed.

Q1. Does pairwise verification retain its advantage when the mechanisms use a common evaluation rubric and comparable total verifier budgets?

With a common evaluation rubric, a common rationale instruction, and comparable verifier output tokens, pairwise verification has the highest observed Pass@1 and Pass@3, ahead of listwise by 1.36 pp and of pointwise by 3.74 pp.

  • The rubric was already common, and the prompts differed in the rationale they require. The three prompts in the paper share one evaluation rubric, apart from mechanism-specific instructions such as score ranges (Appendix B.6). Pairwise verification asks for a structured rationale covering the unresolved requirement, the supporting evidence, each candidate's next-state effect and failure risk, and their contrast, while listwise and pointwise verification ask for a short free-form rationale before the decision.
  • We give listwise and pointwise the same structured-rationale instruction at a comparable budget. We sample multiple responses per step to bring total verifier output tokens close to those of pairwise verification, following Generative Verifiers [1]. Listwise aggregates samples by majority vote and pointwise by mean score. All rows use TMAX-9B at N=8 with the same history preprocessing, and token counts are per run as in Figure 3.
MechanismRationaleResponsesVerifier output tokensPass@1 (%)Pass@3 (%)
PairwiseStructured1125.7k54.7671.43
ListwiseShort10.6k51.0267.35
ListwiseShort168.4k52.7267.35
ListwiseStructured17.0k51.0269.39
ListwiseStructured16128.4k53.4068.37
PointwiseShort17.0k52.3867.35
PointwiseStructured4145.6k51.0265.31
  • Matched compute narrows the gap to listwise but does not close it. Sixteen samples with the structured rationale raise listwise from 51.02% to 53.40% at 128.4k tokens, still below pairwise verification at 54.76% with 125.7k tokens. With one call, the structured rationale alone leaves listwise at 51.02%.
  • Pointwise does not benefit from the structured rationale with more samples. It reaches 51.02% with the structured rationale at 145.6k tokens, against 52.38% with the short rationale. This suggests that a larger budget helps verification that compares candidates (listwise) but not verification that scores each candidate alone (pointwise).

We have added this matched-budget comparison to Appendix C.3 (Table 11) and cited it where Section 4.2 reports pairwise as the best evaluated mechanism.

References

[1] Zhang et al., Generative Verifiers: Reward Modeling as Next-Token Prediction

AuthorsOpenReview ↗

Response to Reviewer LG7N (2/3)

W2. The candidate-width ablation is not fully controlled. Appendix B.2 specifies an 8,000-character history limit for zero-shot (N=4), without stating an equivalent limit for (N=8); the tournament structure also changes. These differences should be controlled before attributing performance changes to candidate width.

Q2. Which configurations use the 8,000-character history limit, and do the candidate-width findings persist under identical history preprocessing?

All N=4 and N=8 configurations use the same history preprocessing, which keeps the most recent 8,000 characters, and with the tournament also matched to N=4, distilled pairwise verification still improves from N=4 to N=8.

  • The history limit is shared. Every configuration in the candidate-width comparison, including the frontier references, uses the same history preprocessing, so the candidate-width findings already hold under identical preprocessing. We have corrected Appendix B.2, whose wording suggested a narrower scope.
  • With the tournament matched to N=4, distilled pairwise verification still improves with more candidates. We have run distilled pairwise verification at N=8 with the full tournament used at N=4, which compares every unordered pair. It reaches 56.12% Pass@1 and 73.47% Pass@3, against 55.44% and 70.41% at N=4, so its candidate-width trend holds under the same comparison structure. We have added this configuration to Table 6 and Appendix B.2.

W3. The incremental gains are not supported by direct uncertainty estimates. The reported confidence intervals compare variants against the base agent, but do not establish the benefit of distillation—54.76% to 57.14%—or pairwise verification over other mechanisms. Paired confidence intervals are needed for these central comparisons.

Q3. What are the paired task-level confidence intervals for distilled versus zero-shot verification and pairwise versus the alternative mechanisms?

Distillation and pairwise verification improve Pass@1 consistently across settings. Distilled verification exceeds zero-shot verification in 13 of 14 comparisons, and pairwise verification stays highest under a matched verifier budget, although the paired intervals of the three N=8 differences (+2.38 to +3.74 pp) include zero on the 98 tasks. We resample the 98 common TerminalBench-Lite tasks and keep each task's three runs together.

Comparison (N=8)Pass@1 difference (pp)95% CI (pp)
Distilled - Zero-shot pairwise+2.38[-3.40, +8.16]
Zero-shot pairwise - Zero-shot listwise+3.74[-3.06, +10.88]
Zero-shot pairwise - Zero-shot pointwise+2.38[-3.40, +8.16]
  • Distillation helps consistently. Distilled verification exceeds zero-shot verification in Pass@1 in 13 of 14 comparisons across model sizes (4B, 9B, and 27B), candidate widths (N=4 and 8), compositions (alone, with Best-of-3, and with SR), and benchmarks (Tables 2, 3, and 6).
  • Pairwise verification stays highest. With a common rationale instruction and comparable verifier output tokens, pairwise verification keeps the highest observed Pass@1 and Pass@3 (see our response to W1).

We have added these intervals to Appendix C.4 (Table 13), next to the existing comparisons with the base agent, and described these differences as consistent observed trends.

AuthorsOpenReview ↗

Response to Reviewer LG7N (3/3)

W4. The contribution beyond existing action-selection methods remains unclear. Prior work already studies per-step action verification and its combination with trajectory selection, while the pivot tournament is adopted from Kwok et al. A comparison with the closest existing method under the same generator and budget would clarify the empirical advance.

Our advance over prior work is not a new selection method but a systematic study of when action-level verification pays off and by how much (see our response to Reviewer MpQg W1). To compare with the closest existing method, we have evaluated the critic of Zainullina et al. [2] in our setting, with the same generator and training data. It falls below the base agent, while our distilled decision-only verification, which like the critic decodes no rationale, reaches 59.18% Pass@1.

  • Our study shows when action-level verification improves performance and by how much. Prior work, including Zainullina et al. [2] and Kwok et al. [3], shows that per-step verification can improve agents, and we adopt the pivot tournament of Kwok et al. as one component. Our study instead shows when this verification pays off before each action changes the environment. Verification, not sampling, governs the gain (Section 4.2), the generator's own backbone can serve as the verifier at 4B, 9B, and 27B (Table 2), and action selection raises Pass@1 over both parallel and sequential trajectory scaling in all 12 comparisons (Table 2).
  • In our setting, the closest existing method falls below the base agent. We evaluate the action-value critic of Zainullina et al. with the same TMAX-9B generator, N=8 candidates, and the data used for our distillation, training it to predict the terminal reward of each candidate. Unlike their separate 70B critic, it shares the generator's backbone, so the comparison holds the model and training data fixed. It reaches 48.64% Pass@1 and 66.33% Pass@3, against 50.00% and 69.39% for the base agent. Our distilled decision-only verification, which like the critic decodes no rationale but reads candidates in pairs, reaches 59.18% and 73.47% (Table 6).

W5. The efficiency comparison mixes compute allocation with verifier capability. The strongest Mid-Harness configuration uses a distilled verifier, whereas Best-of-T uses a zero-shot trajectory verifier. The advantage may therefore partly reflect better verification rather than where compute is allocated. A stronger trajectory selector and measured end-to-end runtimes would strengthen the claim.

Q4. Does the cost–success advantage over trajectory scaling persist with a stronger trajectory selector and under measured, fixed-resource execution?

In two separate checks, the composition still exceeds Best-of-7 in success: without action-verifier training, at lower token cost, and with a stronger trajectory verifier. Its advantage therefore does not come from the verifier difference alone. On the same infrastructure, its runs also take less measured time in total.

  • Without action-verifier training. With the same trajectory verifier and three environment executions, zero-shot Mid-Harness raises Best-of-3 Pass@1 from 55.10% to 61.22%, above Best-of-7 (59.18%, Table 8), at 0.710 dollar per output against 0.907 dollar (our response to Reviewer x4k7 W1-2 gives the full cost table).
  • With a stronger trajectory verifier. With GPT-5.6 Sol as the trajectory verifier, which sees the task instruction and agent-visible traces but no rewards, distilled Mid-Harness + Best-of-3 reaches 72.45% at 9B, against 67.35% for Best-of-7, with three rather than seven environment executions. This reverses the verifier gap, because Sol selects trajectories directly while our action verifier is TMAX-9B distilled from Sol. We compare success only in this setting, because the cost of Sol is not on the reference price scale. We have added this result to Section 6.1 and Table 8.
  • Measured runtime. On the same infrastructure with TMAX-9B, one run of distilled Mid-Harness at N=8 takes 6.4 minutes on average against 3.8 minutes for the base agent, with verification taking 3.8 seconds per step. Summing per-run times as if runs execute one at a time in a single environment, followed by trajectory selection, the composition takes 20.1 minutes per task on average (0.9 minutes for selection), against 29.6 minutes for Best-of-7 (3.0 minutes for selection). We have added these measurements to Appendix B.4.

References

[2] Zainullina et al., Guided Search Strategies in Non-Serializable Environments with Applications to Software Engineering Agents

[3] Kwok et al., LLM-as-a-Verifier: A General-Purpose Verification Framework

Reviewer_kdFMOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

As agent action modify the environment (for ex. coding agents), an incorrect action can introduce changes that affect subsequent decisions. The paper investigates whether test-time compute is better spent generating and verifying multiple candidate actions before execution, rather than relying on a single generated action or scaling entire agent trajectories.

The central findings are:

  • Strong verification increases TMAX-9B's Pass@1 from 50.00% to 68.03% on TerminalBench-Lite.
  • Pairwise verification outperforms the evaluated listwise and pointwise mechanisms in the main self-verification experiments.
  • Distilling a stronger verifier improves performance without modifying the generator.
  • Action-level verification can complement trajectory-level scaling.
Critical strengths

The paper asks an interesting question: Is agent performance limited by an inability to generate good actions, or by an inability to select good actions among alternatives? Verifier experiments also provide some useful evidence that action selection is a bottleneck.

Critical weaknesses
  • The claim about candidate coverage is supported indirectly. The verifier's improvement demonstrates that useful alternatives can be selected from the candidate distribution. However, the paper does not measure how often a suitable candidate exists or how frequently the verifier selects it.

  • The offline verifier analysis measures agreement with a stronger teacher rather than independent action correctness. There is also a methodological difference: Appendix D.1.1 computes offline verification agreement using simple win counts, whereas online action selection uses margin-weighted ranking. This makes the connection between offline agreement and the deployed selection mechanism less direct.

Questions for authors
  • The paper would benefit from a clearer distinction from existing test-time scaling approaches, particularly Kim et al. (2026) [The authors have already cited this work]. Rather than primarily positioning action scaling as another compute-scaling axis, the authors could emphasize the challenges of selecting actions before they modify the environment, and how verification quality affects long-horizon agent reliability.

  • Better disentangle candidate quality from verification quality. The improvement with a good verifier suggests that the generator produces useful alternative actions. However, the current evidence does not directly distinguish whether failures arise because good candidates are unavailable or because the verifier cannot identify them. A controlled analysis on a subset of agent states could help quantify these two factors.

  • Compare verification mechanisms under matched compute budgets. Pairwise verification performs better than listwise and pointwise verification, but also requires substantially more verifier calls. A compute-matched comparison would help establish whether these improvements arise from the verification mechanism itself or from additional inference computation.

  • Strengthen the comparison with trajectory-level scaling. Given the Kim et al. (2026), it would be useful to discuss whether Mid-Harness remains complementary to stronger trajectory selection and refinement strategies beyond the evaluated Best-of-T and Sequential Refine baselines. Even a limited comparison would help clarify the practical benefits of allocating compute at different stages of an agent's execution.

6 comments
AuthorsOpenReview ↗

Response to Reviewer kdFM (1/2)

We thank the reviewer for the detailed feedback. We are glad the reviewer found that (1) the paper asks an important question of generating versus selecting good actions, (2) the verifier experiments provide useful evidence that action selection is a bottleneck, (3) a unified framework covers sampling, verification, verifier training, and composition with trajectory scaling, and (4) the evaluation across model sizes, harnesses, and benchmarks strengthens the paper. We address each concern and question below.


W1. The claim about candidate coverage is supported indirectly. The verifier's improvement demonstrates that useful alternatives can be selected from the candidate distribution. However, the paper does not measure how often a suitable candidate exists or how frequently the verifier selects it.

Q2. Better disentangle candidate quality from verification quality. The improvement with a good verifier suggests that the generator produces useful alternative actions. However, the current evidence does not directly distinguish whether failures arise because good candidates are unavailable or because the verifier cannot identify them. A controlled analysis on a subset of agent states could help quantify these two factors.

To quantify whether failures come from missing candidates or from verification, we ran an LLM-based audit of the failed runs of distilled Mid-Harness with GPT-5.6 Sol. A good candidate is available in 79.44% of steps, and the verifier selects it in 54.30% of steps (68.36% of those with a good candidate). Failures therefore involve both factors: the verifier misses an available good candidate in 25.14% of steps, and no candidate is good in 17.45%.

  • Audit setup. We use the 126 failed runs among the 294 runs of distilled Mid-Harness with TMAX-9B at N=8 on TerminalBench-Lite. In 2,184 steps from 120 failed runs with at least two distinct valid candidates, GPT-5.6 Sol labels each candidate as a good, bad, or uncertain next action given the state. It sees the task, history, and candidates as shown to the verifier, but not the selected action, verifier scores, later observations, or the outcome.
Step category on failed runsStepsShare (%)
Selected candidate good1,18654.30
Selected candidate bad, another good (missed selection)54925.14
All candidates bad (candidate gap)38117.45
Unresolved due to uncertain labels683.11
  • Both factors appear in most failed runs. Of the 120 failed runs, 116 (96.67%) contain at least one missed selection and 96 (80.00%) at least one candidate gap. Later states depend on earlier selections, so these counts do not assign each failure to a single cause.
  • The labels are judgments, not outcomes. A candidate action cannot be labeled by its outcome, because its effect on success depends on the actions that follow, and runs with different verifiers reach different states (Section 4.1, Appendix A). Unselected candidates are also never executed, so a good label does not show that selecting it would have rescued the run.

We have added this audit to Appendix D.4 (Table 15).


W2. The offline verifier analysis measures agreement with a stronger teacher rather than independent action correctness. There is also a methodological difference: Appendix D.1.1 computes offline verification agreement using simple win counts, whereas online action selection uses margin-weighted ranking. This makes the connection between offline agreement and the deployed selection mechanism less direct.

Recomputed with the margin-weighted ranking used online, offline agreement with the teacher rises from 37.49% to 57.49% with distillation, nearly the same as with win counts (38.52% to 57.79%). This agreement is a diagnostic of what distillation transfers, while our performance claims rest on online Pass@1 and Pass@3.

  • The result does not depend on the aggregation rule. We re-aggregate the same 1,355 states and 8,702 saved ring comparisons with the margin-weighted ranking used in online verification, instead of the win counts in Appendix D.1.1. Agreement is 37.49% for zero-shot and 57.49% for distilled verification, against 38.52% and 57.79% with win counts, so distillation raises it by 20.00 pp rather than 19.26 pp.
  • Teacher agreement is a diagnostic, and performance claims rest on online results. Candidate actions have no outcome labels (see our response to W1), so we compare verifiers with the GPT-5.6 Sol teacher on the same states, which reaches 68.03% Pass@1 as an online verifier (Section 4.1). Section 5 states that this measures teacher agreement, not action correctness or trajectory success. Every comparison of verifiers instead uses online Pass@1 and Pass@3 with the deployed mechanism (Table 6), where distillation raises Pass@1 from 54.76% to 57.14%.

We have added this recomputation to Appendix D.1.1.

AuthorsOpenReview ↗

Response to Reviewer kdFM (2/2)

Q1. Positioning relative to prior work.

We thank the reviewer for this suggestion. As the reviewer notes, unlike trajectory-level methods such as Kim et al. [1], Mid-Harness must choose each action before it modifies the environment and without an outcome label. Verification quality therefore governs long-horizon reliability, and in return Mid-Harness raises the reliability of each trajectory within one environment.

  • The challenge: an action is chosen before its effect is known, and it has no outcome label. Kim et al. select and refine completed trajectories, each with a task outcome against which the choice can be judged. A candidate action has no such label. Only the chosen action is executed, it changes the environment and every later candidate set, and its value depends on the actions that follow. We therefore analyze verifier judgments through agreement with a frontier teacher (Section 5, Appendix A).
  • The choice decides success, and verification quality governs it over long horizons. With the generator fixed, Pass@1 ranges from 49.66% when the first runnable candidate is executed to 68.03% under frontier verification (Section 4). Distilled verification improves Pass@1 by 11.1, 5.1, and 14.1 points on the tasks with the longest base-agent executions at 4B, 9B, and 27B (Appendix C.5), while agreement with the teacher falls from 68.13% at turns 1-4 to 54.07% at turns 17-32 (Section 5).
  • The benefit: action-level scaling works within one environment and raises the reliability of each trajectory. Only the chosen action runs, so no additional environment executions are needed (Section 2). Each trajectory becomes more likely to succeed, so trajectory methods, including those of Kim et al., can run on top of it (see our response to Q4).

Following the reviewer's suggestion, we have revised the Introduction to state this challenge before our research question, and the Related Work to contrast action selection with the trajectory-level selection and refinement of Kim et al. (Sections 1 and 7).


Q3. Compare verification mechanisms under matched compute budgets. Pairwise verification performs better than listwise and pointwise verification, but also requires substantially more verifier calls. A compute-matched comparison would help establish whether these improvements arise from the verification mechanism itself or from additional inference computation.

Pairwise verification keeps the highest observed Pass@1 when the number of verifier output tokens is matched, so its lead is not explained by additional decoding alone. With a common structured-rationale instruction and total verifier output tokens matched to pairwise verification (128.4k and 145.6k against 125.7k tokens per run), listwise reaches 53.40% Pass@1 and pointwise 51.02%, against 54.76% for pairwise. Our response to Reviewer x4k7 W1-1 gives the full table and detailed explanation.


Q4. Strengthen the comparison with trajectory-level scaling.

Mid-Harness is complementary to trajectory selection and refinement by construction, because it changes how each rollout is produced, and with a stronger trajectory verifier it still adds 7.14 pp to Best-of-3 at 9B.

  • Any trajectory method runs on rollouts from the same harness loop, so Mid-Harness plugs in unchanged. Mid-Harness replaces the model call inside this loop (Section 2). Kim et al.'s Recursive Tournament Voting selects among rollout summaries and Parallel-Distill-Refine starts fresh rollouts from the selected summaries, and both still run the same loop. Our SR already uses their summary format, and Mid-Harness raises it from 55.10% to 60.20% at 9B (Section 6.1).
  • The gain comes from a higher Pass@3. With GPT-5.6 Sol as the trajectory verifier, Mid-Harness raises Best-of-3 from 65.31% to 72.45% at 9B. Best-of-3 cannot exceed Pass@3, and with GPT-5.6 Sol it already comes close, so the gain comes from Mid-Harness raising Pass@3 from 69.39% to 75.51% (Table 2).
  • Three executions with Mid-Harness exceed seven without it. With the same trajectory verifier, Mid-Harness + Best-of-3 exceeds Best-of-7 with both GPT-5.6 Sol (72.45% versus 67.35%) and the base TMAX trajectory verifier (66.33% versus 59.18%, Table 8).

We have added the GPT-5.6 Sol trajectory-verifier results to Section 6.1 and Table 8.

References

[1] Kim et al., Scaling Test-Time Compute for Agentic Coding

AuthorsOpenReview ↗

Gentle Follow-Up on Our Responses

Dear Reviewer kdFM,

Thank you again for your thoughtful review and for your positive assessment of our work. As the discussion period ends on November 18 (AoE), we would like to follow up on our response. If you have any additional question or concern, please let us know. We are more than happy to address them. Thank you.

Reviewer_x4k7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a systematic empirical study of action-level test-time scaling for terminal agents. Mid-Harness samples multiple candidate actions from the same interaction history, selects one through verification, and executes only that action, keeping the generator and execution harness fixed. The study compares listwise, pointwise, and pairwise verification, examines frontier-verifier distillation and decision-only outputs, and combines action selection with trajectory selection and sequential refinement. Evaluation covers TerminalBench-Lite and additional models, harnesses, and software-engineering benchmarks. Its principal contribution is empirical understanding of when verification makes candidate sampling useful, rather than the introduction of pre-execution action selection itself.

Critical strengths
  1. The fixed-generator experiments provide persuasive evidence that verifier capability determines how effectively sampled alternatives can be exploited.** With eight candidates, first-runnable selection achieves 49.66% Pass@1, zero-shot listwise verification achieves 51.02%, and frontier listwise verification reaches 68.03%, compared with the 50.00% base agent (Figure 2; Table 6). The same-format listwise comparison is particularly informative about the importance of verifier capability. Moreover, distilled pairwise verification reaches 57.14%, with a task-paired 95% bootstrap interval of [1.36, 12.93] percentage points for its improvement over the base agent. These results establish a meaningful practical benefit without requiring environment branching or modifying the generator.

  2. The study provides useful breadth and reports limitations that make its practical conclusions more credible. Experiments cover multiple model sizes, alternative harnesses, additional benchmarks, and composition with trajectory-level scaling. Adding distilled Mid-Harness raises Best-of-3 performance from 55.10% to 66.33% while retaining three environment executions per selected output, although using additional inference compute. The appendices include complete-pipeline token accounting under explicit assumptions, decision-only variants, additional-compute experiments, and domain-specific regressions. The explicit distinction between teacher agreement and action correctness is also important. Together, these experiments offer useful guidance about both the opportunities and limitations of action verification.

Critical weaknesses
  1. The mechanism and compute-allocation comparisons do not fully isolate the sources of the gains. Given the precedents for action ranking and composition in Zainullina et al. (2025), and per-step Terminal-Bench verification in Kwok et al. (2026), the incremental contribution depends substantially on the empirical design guidance. With eight distinct valid candidates, reasoning-producing pairwise verification uses 22–25 comparisons, versus one listwise call, with different prompting and reasoning requirements. Appendix C.3 provides useful within-pairwise compute controls but does not isolate these cross-mechanism differences. A related attribution limitation concerns the prominent cost comparisons: distilled action scaling uses a specially trained verifier, whereas trajectory selection uses the base TMAX model. Importantly, zero-shot composition already improves Best-of-3 from 55.10% to 61.22%, so composition gains cannot be attributed entirely to distillation. Nevertheless, the highlighted distilled cost comparisons do not isolate the allocation strategy from differences in verifier training. These limitations do not invalidate the reported operating points, but they constrain conclusions about verification format or compute allocation independently of the other design choices.

  2. Uncertainty is incompletely characterized for several central incremental gains. Appendix C.4 appropriately uses task-paired bootstrap intervals, but only for comparisons against the base agent. These intervals do not establish the additional benefit of distillation over zero-shot verification or the ordering between verification mechanisms. At N=8N=8, the distilled–zero-shot pairwise difference is 2.38 percentage points, and the zero-shot pairwise–listwise difference is 3.74 points. Direct uncertainty estimates are also absent for the prominent composition improvement. This does not undermine the supported distilled-versus-base result, but it leaves uncertainty about which added design choices reliably improve performance and how strongly those findings should guide practitioners.

Questions for authors
  1. Analysis of existing results — uncertainty: Could you report task-paired confidence intervals for the direct N=8N=8 comparisons of distilled versus zero-shot pairwise verification, pairwise versus listwise verification, and Mid-Harness + Best-of-3 versus Best-of-3? These would help distinguish the established benefit over the base agent from the incremental benefits of particular design choices.

  2. Targeted additional validation — verification format: With the generator, candidate count, and available history fixed, does the pairwise advantage persist against a competitive listwise or pointwise configuration using a common evaluation rubric and a comparable verifier-token budget? A focused comparison would clarify whether the practical recommendation concerns the comparison structure or the particular prompting and compute allocation evaluated here.

  3. Analysis of existing results — allocation without distillation: Could you report the complete-pipeline token cost of the zero-shot Mid-Harness + Best-of-3 configuration that achieves 61.22%, alongside the Best-of-T cost curve and using the same accounting as Figure 5? This would clarify how much of the composition's cost-success advantage persists without dedicated action-verifier training, without requiring a new verifier-training experiment.

3 comments
AuthorsOpenReview ↗

Response to Reviewer x4k7 (1/2)

We thank the reviewer for the constructive feedback. We are glad the reviewer found that (1) the fixed-generator experiments provide persuasive evidence that verifier capability determines how well sampled alternatives are exploited, and (2) the breadth of the study and its reported limitations offer useful guidance on action verification. We address each concern and question below.


W1-1. The mechanism and compute-allocation comparisons do not fully isolate the sources of the gains. … With eight distinct valid candidates, reasoning-producing pairwise verification uses 22–25 comparisons, versus one listwise call, with different prompting and reasoning requirements.

Q2. Targeted additional validation — verification format: With the generator, candidate count, and available history fixed, does the pairwise advantage persist against a competitive listwise or pointwise configuration using a common evaluation rubric and a comparable verifier-token budget?

With a common evaluation rubric, a common rationale instruction, and comparable verifier output tokens, pairwise verification has the highest observed Pass@1 and Pass@3, ahead of listwise by 1.36 pp and of pointwise by 3.74 pp.

  • The rubric was already common, and the prompts differed in the rationale they require. The three prompts in the paper share one evaluation rubric, apart from mechanism-specific instructions such as score ranges (Appendix B.6). Pairwise verification asks for a structured rationale covering the unresolved requirement, the supporting evidence, each candidate's next-state effect and failure risk, and their contrast, while listwise and pointwise verification ask for a short free-form rationale before the decision.
  • We give listwise and pointwise the same structured-rationale instruction at a comparable budget. We sample multiple responses per step to bring verifier total output tokens close to those of pairwise verification, following Generative Verifiers [1]. Listwise aggregates samples by majority vote and pointwise by mean score. All rows use TMAX-9B at N=8, and token counts are per run as in Figure 3.
MechanismRationaleResponsesVerifier output tokensPass@1 (%)Pass@3 (%)
PairwiseStructured1125.7k54.7671.43
ListwiseShort10.6k51.0267.35
ListwiseShort168.4k52.7267.35
ListwiseStructured17.0k51.0269.39
ListwiseStructured16128.4k53.4068.37
PointwiseShort17.0k52.3867.35
PointwiseStructured4145.6k51.0265.31
  • Matched compute narrows the gap to listwise but does not close it. 16 samples with the structured rationale raise listwise from 51.02% to 53.40% at 128.4k tokens, still below pairwise verification at 54.76% with 125.7k tokens. With one call, the structured rationale alone leaves listwise at 51.02%.
  • Pointwise does not benefit from the structured rationale with more samples. It reaches 51.02% with the structured rationale at 145.6k tokens, against 52.38% with the short rationale. This suggests that a larger budget helps verification that compares candidates (listwise) but not verification that scores each candidate alone (pointwise).

We have added this comparison to Appendix C.3 (Table 11) and cited it in Section 4.2.


W1-2. ... Nevertheless, the highlighted distilled cost comparisons do not isolate the allocation strategy from differences in verifier training.

Q3. Analysis of existing results — allocation without distillation: Could you report the complete-pipeline token cost of the zero-shot Mid-Harness + Best-of-3 configuration that achieves 61.22%, alongside the Best-of-T cost curve and using the same accounting as Figure 5?

Without action-verifier training, zero-shot Mid-Harness + Best-of-3 reaches 61.22% at 0.710 dollar, exceeding Best-of-7 (59.18%, 0.907 dollar) at 22% lower reference-priced cost and with three rather than seven environment executions. We agree that the Figure 5 comparison with distilled action verification combines two factors, where verification occurs and whether the action verifier was distilled. Zero-shot Mid-Harness + Best-of-3 uses the generator itself as the action verifier, which removes the second factor. Using the Figure 5 accounting, the point lies above the Best-of-T curve.

ConfigurationPass@1 (%)Input tokensOutput tokensCost (USD)# Env.
Best-of-355.104.107M80.6K0.3393
Best-of-557.147.518M139.0K0.6195
Best-of-759.1811.010M198.5K0.9077
Zero-shot Mid-Harness + Best-of-361.227.523M833.9K0.7103

We have added this point to Figure 5 and Section 6.1.

References

[1] Zhang et al., Generative Verifiers: Reward Modeling as Next-Token Prediction

AuthorsOpenReview ↗

Response to Reviewer x4k7 (2/2)

W2. Uncertainty is incompletely characterized for several central incremental gains.

Q1. Analysis of existing results — uncertainty: Could you report task-paired confidence intervals for the direct (N=8) comparisons of distilled versus zero-shot pairwise verification, pairwise versus listwise verification, and Mid-Harness + Best-of-3 versus Best-of-3?

Adding Mid-Harness to Best-of-3 improves Pass@1 by 11.22 pp, with a 95% interval above zero. The distillation and mechanism differences are also positive and consistent in direction across further comparisons, although their intervals include zero. We report 95% task-paired bootstrap intervals on the same 98 TerminalBench-Lite tasks, following Appendix C.4. For the distillation and mechanism comparisons, each task contributes its mean success over three runs. For the Best-of-3 comparison, Best-of-3 uses its three runs to return one trajectory, so each task contributes a single success or failure, which widens the confidence interval.

ComparisonPass@1 difference (pp)95% CI (pp)
Distilled - Zero-shot pairwise, N=8+2.38[-3.40, +8.16]
Zero-shot pairwise - Zero-shot listwise, N=8+3.74[-3.06, +10.88]
Distilled Mid-Harness + Best-of-3 - Best-of-3+11.22[+1.02, +21.43]
  • Mid-Harness adds to trajectory scaling. With the same three environment executions, the composition gain over Best-of-3 is supported by the interval, as are the gains of distilled Mid-Harness over the base agent at 9B and 27B (Table 12).
  • Distillation helps consistently. Distilled verification exceeds zero-shot verification in Pass@1 in 13 of 14 comparisons across model sizes (4B, 9B, and 27B), candidate widths (N=4 and 8), compositions (alone, with Best-of-3, and with SR), and benchmarks (Tables 2, 3, and 6).
  • Pairwise verification stays highest. With a common rationale instruction and comparable verifier output tokens, pairwise verification keeps the highest observed Pass@1 and Pass@3 (see our response to W1-1).

We have added these intervals to Appendix C.4 (Table 13), next to the comparisons with the base agent, and described the distillation and mechanism differences as consistent observed trends.

AuthorsOpenReview ↗

Gentle Follow-Up on Our Responses

Dear Reviewer x4k7,

Thank you again for your constructive review and for your positive assessment of our work. As the discussion period ends on November 18 (AoE), we would like to follow up on our response. If you have any additional question or concern, please let us know. We are more than happy to address them. Thank you.

Reviewer_MpQgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Mid-Harness, which samples multiple candidate actions at each step and selects one of them through verification before execution. It keeps the generator model unchanged. It first estimates the improvement room by pairing a generator with GPT-5.6 Sol verifier. It then compares three verification methods (listwise, pointwise, and pairwise) with the generator itself as the verifier, and finds pairwise works best. It distills pairwise judgments from GPT-5.6 Sol into a 9B Qwen model. Experiments show that Mid-Harness also improves under different test-time scaling settings, and transfers to other model sizes, benchmarks, and harnesses.

Critical strengths
  1. The paper is logically organized: it first estimates the room for improvement, then analyzes how different verification methods recover it.

  2. It provides comprehensive empirical studies. Beyond performance comparison, the paper also provides a cost analysis across verification methods, including parallelized output tokens, total verifier output tokens, and dollar cost.

Critical weaknesses
  1. The idea is not entirely new. The contribution is mainly a systematic study rather than a new method.

  2. Several simple baselines are missing. Majority voting or the generator’s own action confidence could select candidates without any verifier calls, and would show whether the gains really come from verification.

Questions for authors
  1. Verifying at every step is costly. To make Mid-Harness more practical, could the authors analyze when verification actually matters, for example, how often the selected action differs from the first sampled candidate?

  2. The decision-only distilled verifier outperforms the reasoning-based one at N=8 but underperforms it at N=4. What could be the possible explanations here? Since GPT only provides a short rationale rather than its full reasoning, could this limit what the reasoning-based verifier learns from distillation?

7 comments
AuthorsOpenReview ↗

Response to Reviewer MpQg (1/3)

We thank the reviewer for the thoughtful feedback. We are glad the reviewer found that (1) the paper is logically organized, estimating the room for improvement before analyzing how verification recovers it, and (2) the empirical study is comprehensive, including a cost analysis across verification methods. We address each concern and question below.


W1. The idea is not entirely new. The contribution is mainly a systematic study rather than a new method.

Our contribution is a rigorous systematic study of action-level compute scaling in terminal agents. It shows when this scaling improves performance and by how much, and beyond the study, it provides a lens for interpreting this scaling (Section 1). Prior work, including action selection in [1, 2], shows that sampling and verification can improve agents, but not when they pay off before each action changes the environment. Our study answers this by examining sampling and verification jointly.

  • When it works: verification, not sampling, governs the gain. Without a verifier, sampling eight candidates leaves Pass@1 at the base level (first-runnable 49.66% and majority voting 50.34%, against 50.00% for the base agent, in our response to W2), and wider sampling adds little under weak verification (Section 4.2). With the same generator, frontier verification reaches 68.03%, so the generator already proposes useful alternatives (Figure 2, Table 6). The generator's own backbone can serve as the verifier at 4B, 9B, and 27B (Table 2), and the remaining bottleneck is judging command semantics and execution feasibility (Section 5).
  • How much it helps. Mid-Harness with distilled pairwise verifier improves Pass@1 over the base agent by 5.10, 7.14, and 5.10 pp at 4B, 9B, and 27B (Table 2). Across three harnesses, four benchmarks, and six models from 9B to 550B, including one model and one harness added during the rebuttal, zero-shot Mid-Harness improves Pass@3 and matches or improves Pass@1 (Table 3).
    • In additional experiments during the rebuttal with the recent Nemotron 3.5 Super VL and zero-shot self-verification at N=8, Mid-Harness also raises Terminal-Bench 2.1 Pass@1 from 40.82% to 51.69% with the Terminus-2 harness and from 49.81% to 58.43% with the Opencode harness, and Pass@3 by 6.74 and 11.24 pp, confirming its effectiveness on a recent model. We have added these results to Table 3 and Section 6.2.
  • Beyond the study, it provides a lens for action-level scaling. Mid-Harness defines a layer between the model and the harness, where compute can be added without changing either (Section 2). Through this layer, action scaling becomes an axis of test-time compute complementary to trajectory scaling (Section 8), and trajectory methods run on top of it unchanged (see our response to Reviewer kdFM Q4 for an evaluation with a stronger trajectory verifier). It raises Pass@1 over both parallel (Best-of-3) and sequential (SR) trajectory scaling in all 12 comparisons in Table 2, including 11.22 pp over Best-of-3 with the same three environment executions.

We believe this systematic study offers useful takeaways, and the other reviewers noted related strengths. It addresses the important question of whether agents are limited by generating or by selecting good actions (kdFM), provides persuasive evidence that verifier capability determines how effectively sampled alternatives are exploited (x4k7), and offers useful guidance about the opportunities and limitations of action verification (x4k7), with decision-only results particularly relevant to practical deployment (LG7N).

References

[1] Zainullina et al., Guided Search Strategies in Non-Serializable Environments with Applications to Software Engineering Agents

[2] Kwok et al., LLM-as-a-Verifier: A General-Purpose Verification Framework

AuthorsOpenReview ↗

Response to Reviewer MpQg (2/3)

W2. Several simple baselines are missing. Majority voting or the generator’s own action confidence could select candidates without any verifier calls, and would show whether the gains really come from verification.

We thank the reviewer for suggesting these baselines. Following the reviewer's suggestion, we evaluate majority voting and action confidence with TMAX-9B at N=8 following Figure 2, and both remain near the base-agent level. The gains come from the generator verifying its own candidates by comparing them. All rows use TMAX-9B with N=8, except the base agent.

SelectionSignalPass@1 (%)Pass@3 (%)
Base agent (N=1)n/a50.0069.39
First-runnableFirst candidate that runs49.6666.33
Majority votingMost frequent exact action50.3466.33
Action confidenceMean action-token probability50.6868.37
Value critic [1]Predicted terminal reward48.6466.33
Zero-shot pairwiseGenerator compares candidates54.7671.43
Distilled pairwiseSame, distilled from GPT-5.6 Sol57.1475.51
  • Without verification, Pass@1 stays near the base level. No selection rule without a verifier moves Pass@1 by more than 0.7 pp from the base agent, and all lower Pass@3, whereas pairwise verification raises both. A verifier is required because choosing well requires judging what each candidate will do in the current environment (Section 1), which none of these rules measures.
  • Majority voting needs exactly repeated answers. Unlike multiple-choice or math answers, bash commands with the same purpose often differ in text, as when two candidates rewrite jest.config.js with different options (Appendix D.3). Only 39.5% of steps contain an exact duplicate, and the rest fall back to first-runnable in majority voting.
  • Action confidence does not measure correctness. It averages the probabilities of the action tokens, which shift with the preceding reasoning, so a confident action can still be wrong.
  • A trained value critic does not help either. A critic that predicts terminal reward, following Zainullina et al. [1] and trained on the same data as our distilled verifier, also falls below the base agent (our response to Reviewer LG7N W4 gives the setup and results).

We have added majority voting, action confidence, and the value critic to Figure 2 and Table 6, and discussed the reasons for their limited improvement in Section 4.2, with their procedures in Appendix B.2.


Q1. Verifying at every step is costly. To make Mid-Harness more practical, could the authors analyze when verification actually matters, for example, how often the selected action differs from the first sampled candidate?

Verification changes the executed action in 87.94% of steps, but this rate alone does not show when verification matters. Mid-Harness can allocate sampling and verification per step, so compute could go only to the steps that matter. Identifying those steps, however, requires knowing how an action affects later states and how difficult the current state is, which needs further research (Appendix A).

  • Verification executes an action other than the first sample in most steps. Candidates with identical tool names and arguments are removed first, and the verifier is not called when only one distinct valid candidate remains (Appendix B.2). With eight distinct valid candidates, distilled pairwise verification selects an action other than the first sample in 87.94% of steps.
  • Whether a changed action matters is not observable when it is chosen. A different action matters only if it changes the states that follow and, through them, the task outcome. Whether a step matters therefore depends on the action's effect on later states and on the difficulty of the current state. Neither is observed when the action is chosen, and no per-step label indicates them (our response to Reviewer kdFM W1 explains this in detail).
  • Step-level allocation is a direction Mid-Harness enables. Because Mid-Harness wraps each model call, it can set candidate width and verification effort per step. Estimating action impact and state difficulty before execution requires additional methods, which we discuss as future work in Appendix A.

References

[1] Zainullina et al., Guided Search Strategies in Non-Serializable Environments with Applications to Software Engineering Agents

AuthorsOpenReview ↗

Response to Reviewer MpQg (3/3)

Q2-1. The decision-only distilled verifier outperforms the reasoning-based one at N=8 but underperforms it at N=4. What could be the possible explanations here?

To check the direction, we additionally evaluate decision-only verification at 4B and 27B with N=8, where it underperforms reasoning-based verification. Decision-only verification therefore has no consistent advantage, and as with generative reward models, generating reasoning generally helps. Its value is the trade-off: it cuts verification cost substantially with only a small loss in success.

  • Reasoning-based verification is higher in three of four settings. Switching distilled verification to decision-only changes Pass@1 from 43.88% to 41.50% at 4B and from 76.19% to 74.15% at 27B with N=8, and from 55.44% to 54.42% at 9B with N=4. Only at 9B with N=8 is it higher, from 57.14% to 59.18%. Zero-shot decision-only verification is also lower than zero-shot reasoning-based verification at 4B (38.10% versus 41.50%) and 27B (71.77% versus 73.13%).
  • Decision-only verification is a cost-success trade-off. At 9B, it lowers the reference-priced cost of distilled Mid-Harness by 20.8% at N=4 and 24.1% at N=8 (Appendix C.2), while Pass@1 stays within about two points of reasoning-based verification in all four settings.

We have revised Section 4.4 to present decision-only verification as this trade-off, and added the 4B and 27B results to Appendix C.1 (Table 7).


Q2-2. Since GPT only provides a short rationale rather than its full reasoning, could this limit what the reasoning-based verifier learns from distillation?

Yes, it may, and we have not compared against distilling the teacher's full reasoning. Our reasoning-based verifier is trained on the short rationale that our output format requires before each decision, 45 to 160 tokens (Appendix B.6). It does not learn the teacher's thinking (the full reasoning the reviewer refers to), which is not available from GPT-5.6 Sol.

  • Our verifiers run with thinking disabled, and enabling it adds little for its cost. Verifier thinking is off in all main configurations (Appendix B.2). Enabling it raises zero-shot pairwise Pass@1 from 54.76% to 56.12%, at 4.86 times the verifier output tokens (Table 10). This tests the student's own thinking rather than distilled teacher reasoning.
  • Distilling the teacher's thinking remains an open question. Open-weight large models such as Kimi-K3 and GLM expose their thinking traces, and distilling them is a direction we leave to future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank the reviewers for their time and constructive feedback. We are glad that the reviewers found:

  1. An important and practical question. The paper asks whether agents are limited by generating or by selecting good actions (kdFM), at the practically important decision point before an action changes the environment (LG7N).
  2. Informative evidence from a fixed generator. Holding the generator fixed separates generation from selection (LG7N) and provides persuasive evidence that verifier capability determines how well sampled alternatives are exploited (x4k7), showing that action selection is a bottleneck (kdFM).
  3. A comprehensive and well-organized study. The paper estimates the room for improvement before analyzing how verification recovers it (MpQg), and covers sampling, verification, verifier training, composition, and cost across model sizes, harnesses, and benchmarks (MpQg, kdFM, LG7N).
  4. Useful guidance for practice. The breadth of the study and its reported limitations offer guidance on action verification (x4k7), and the decision-only results are particularly relevant to practical deployment (LG7N).

We summarize the main points below:

  1. Contribution and positioning. We do not claim per-step verification as entirely new. Our contribution is a systematic study of when and how action-level compute scaling improves terminal agents, including its composition with trajectory scaling. Evaluated in our setting, the closest existing method, the critic of Zainullina et al., falls below the base agent (48.64% Pass@1), while our distilled decision-only verification reaches 59.18%. (MpQg W1, kdFM Q1, LG7N W4)
  2. Mechanism comparison under matched budgets. With a common structured-rationale instruction and total verifier output tokens matched to pairwise verification (128.4k and 145.6k against 125.7k tokens per run), listwise reaches 53.40% Pass@1 and pointwise 51.02%, against 54.76% for pairwise. Pairwise verification keeps the highest observed Pass@1. (x4k7 W1-1, kdFM Q3, LG7N W1)
  3. Direct uncertainty estimates. Adding distilled Mid-Harness to Best-of-3 improves Pass@1 by 11.22 pp (95% task-paired CI [1.02, 21.43]) with the same three environment executions. The intervals for distilled versus zero-shot verification and for pairwise versus listwise and pointwise include zero, so we describe those differences as consistent observed trends. (x4k7 W2, LG7N W3)
  4. Without a verifier, Pass@1 stays near the base level. At N=8, majority voting (50.34%), action confidence (50.68%), and first-runnable (49.66%) stay within 0.7 pp of the base agent (50.00%) and all lower Pass@3, while zero-shot pairwise verification reaches 54.76%. (MpQg W2)
  5. Composition beyond the verifier difference. Without action-verifier training, zero-shot Mid-Harness + Best-of-3 exceeds Best-of-7 (61.22% versus 59.18%) at 22% lower reference-priced cost. With GPT-5.6 Sol as a stronger trajectory verifier, distilled Mid-Harness + Best-of-3 reaches 72.45% against 67.35% for Best-of-7. Summing measured per-run times, its three runs and trajectory selection take 20.1 minutes per task on average, against 29.6 for Best-of-7. (x4k7 W1-2, kdFM Q4, LG7N W5)

We have uploaded a revised paper, with all changes shown in blue:

  • Sections 1 and 7: stated the challenge of choosing an action before its effect is known, and contrasted it with trajectory-level methods. (kdFM Q1)
  • Section 4.2, Figure 2, Table 6, and Appendix B.2: added majority voting, action confidence, and the value critic, with their procedures. (MpQg W2, LG7N W4)
  • Section 4.2 and Appendix C.3 (Table 11): added the mechanism comparison under a common rationale instruction and matched verifier output tokens. (x4k7 W1-1, kdFM Q3, LG7N W1)
  • Appendix C.4 (Table 13): added task-paired intervals for the design choices and described the distillation and mechanism differences as consistent observed trends. (x4k7 W2, LG7N W3)
  • Section 4.4 and Appendix C.1 (Table 7): presented decision-only verification as a cost-success trade-off and added the 4B and 27B results. (MpQg Q2-1)
  • Section 6.1 and Figure 5: added zero-shot Mid-Harness + Best-of-3. (x4k7 W1-2)
  • Section 6.1, Table 8, and Appendices B.1 and B.4: added Best-of-T with GPT-5.6 Sol as the trajectory verifier and measured runtime. (kdFM Q4, LG7N W5)
  • Section 6.2, Table 3, and Appendix B.1: added Nemotron 3.5 Super VL with Terminus-2 and Opencode. (MpQg W1)
  • Appendix B.2 and Table 6: corrected the scope of the history cap and added distilled pairwise verification at N=8 with the N=4 tournament. (LG7N W2)
  • Appendix D.1.1: added offline verification agreement recomputed with the margin-weighted ranking used online. (kdFM W2)
  • Appendix D.4 (Table 15): added an audit of candidate quality on failed runs. (kdFM W1, Q2)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.