- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies when a critic for terminal agents should be consulted and what environment access it needs. On all 89 Terminal-Bench 2.1 tasks (five trials each, GPT-5.6 Luna with Terminus-2), neither snapshot/restore tools nor MCTS with a trajectory-only critic improves pass@1. Measured directly, critics rank intermediate states near chance early in a trajectory, with little help from environment access. At claimed completions, however, a critic that executes commands in a fork cuts misclassification from 36–39% to 22%, and a fork of the initial state helps further. The resulting Probe-and-Resume calls the critic only when the actor claims completion, lets it probe copies of the candidate and initial states, and returns unmet requirements to an actor that resumes from the preserved state. Pass@1 rises from 63.4% to 72.4% at 2.2× model cost, with a similar gain on a SWE-bench Pro subset.
Critical strengths
- The experiments are reliable, the arguments are clear, and the content is substantial.
- Non-significant results are reported for the strong model and the difficult subset of SWE-bench Pro.
Critical weaknesses
- The algorithm budget is completely fixed, and more parameter adjustments and experiments should be conducted to demonstrate the effectiveness of the parameters.
- How much benefit do you get from a cheaper second pass when the cost is 2.2x?
Questions for authors
- Public completion evaluation resumes after checking on the current state (action_critic.py lines 343–358, 410–474), while Figures 1 and 5 depict discardable forks. Which one was used in the experiment?
5 comments
Response to Reviewer JeL5 (1/3)
We thank the reviewer for the positive assessment and for reading our code closely.
[W1 Fixed budget and parameter choices]
Probe-and-Resume has three parameter limits. The first is the time limit T, which is each task’s official timeout in the corresponding benchmark and is shared by the actor, the critic, and every snapshot and restore operation. We never change T, so every task keeps its own official limit and all methods, including the Codex CLI reference, follow the benchmark’s rules. The other two limits are our choices. The loop stops after six critic calls (the critic is called each time the actor claims it has finished), and after the first rejection, the actor has at most 50 more turns. To test how much these two choices matter, we vary each of them on the same 445 Terminal-Bench 2.1 trials as in Table 3, and every task keeps its own T.
| Claims | Turns after first rejection | Pass rate | Δ vs. baseline [95% CI] | Total model cost |
|---|---|---|---|---|
| 6 | 50 (paper) | 72.4% | +9.0 [+5.6, +12.6] | $23.14 |
| 1 | – | 63.4% | 0.0 | $10.70 |
| 2 | 50 | 70.3% | +7.0 [+3.8, +10.1] | $20.45 |
| 3 | 50 | 72.6% | +9.2 [+5.8, +12.8] | $21.80 |
| 6 | 10 | 71.9% | +8.5 [+5.2, +12.1] | $20.05 |
| 6 | 25 | 72.4% | +9.0 [+5.6, +12.6] | $21.60 |
| 6 | 100 | 72.4% | +9.0 [+5.6, +12.6] | $24.95 |
| 6 | no limit | 72.4% | +9.0 [+5.6, +12.6] | $30.45 |
- Critic calls beyond the third add nothing. A second call gives +7.0 points, and a third gives the full gain. This matches Figure 6b. Only 11 of the 445 trials (2.5%) were stopped by the limit of six critic calls.
- The turn limit mainly controls cost. From 25 turns to no limit, the pass rate stays at 72.4% while the cost rises from \30.45. Even with 10 turns, the gain is +8.5 points. Only 14 trials (3.1%) were stopped by the 50-turn limit, all on hard tasks, and removing the limit does not change the pass rate.
Across these settings, the gain stays between +8.5 and +9.2 points, and it drops only when the loop allows two critic calls or fewer. Three critic calls keep the full gain at 2.0x the baseline cost, which also bears on the cost question in W2. We will add this table to a new appendix.
Response to Reviewer JeL5 (2/3)
[W2 Benefit of a cheaper second pass at 2.2x cost]
Probe-and-Resume raises the model cost from \23.14 (Table 3). Of the extra \7.48 and the actor’s revisions after a rejection account for \$4.96. We read “a cheaper second pass” in the three ways listed in our earlier question and answer each below.
(a) A cheaper evaluation step inside the loop
We tested two alternatives to the acting critic C_act in the loop. Both use the same time limit T, limit on critic calls, and turn limit as Probe-and-Resume.
- A read-only critic. We use C_read (§3.2) in place of C_act in Probe-and-Resume. A single evaluation costs less than with C_act (App. G), but in the loop it rejects 64% of the actor’s first answers that already pass, against 21% for C_act (§4.2), because many requirements cannot be checked by reading files alone. The actor therefore has to revise correct work more often. 30% of trials use all six critic calls, against 2.5% with C_act, and the critic and the actor’s revisions together cost 1.5x as much as with C_act.
- The actor as its own critic. No separate critic is used here. The actor’s own conversation runs the same probe on a fork of the candidate state, and the probe turns are removed from its context afterwards. This variant reaches 65.2%, 7.2 points below Probe-and-Resume, at a similar total cost (\23.14). Its verdicts are also less reliable. Over all rounds, it accepts 28% of the failing answers and rejects 36% of the passing ones, against 21% and 21% for a separate critic.
(b) A different use of the same budget
Best-of-2 runs the actor twice on each task, each run within the task’s own time limit T, and C_act judges the final state of both runs. It submits the run the critic accepts, and picks one at random when the critic accepts both or neither. Best-of-2 reaches 69.0% (+5.7 [+3.6, +7.8]) at \23.14, 2.2x). Even with an oracle selector that always picks a passing run whenever one exists, at no cost, Best-of-2 would reach 74.2% at \38.12 (Table 3, App. K).
(c) Stopping after the second claim
Most of the gain comes from the second claim. With at most two claims, Probe-and-Resume reaches 70.3% (+7.0 [+3.8, +10.1]) at \21.80 (2.0x). Further claims add cost but no gain (Figure 6b).
Thank you for the review. Before we finalize our response, we would like to better understand the second weakness you mention: “How much benefit do you get from a cheaper second pass when the cost is 2.2x?”
To make sure we answer it directly, could you clarify which comparison you have in mind as “a cheaper second pass”:
(a) a cheaper evaluation step inside the loop, such as a trajectory-only critic or a self-check prompt in place of the execution-based critic
(b) a different use of the same 2.2x budget, such as best-of-2 sampling with critic selection
(c) the cost of stopping after the second round, which already recovers 78% of the gain (Fig. 6b)
Please let us know if you meant something different.
Also, we want to ask why you flagged an ethics violation so that we can promptly address it.