- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the question of when a runtime controller should spend a limited budget to challenge an agent’s proposed action. This is called “forking” in which a controller can either trust the agent or restore the current state, and try the proposed action as well as an alternative, choosing the better option overall. The authors use learned estimates to allocate challenges across a trajectory and evaluate on a controlled fault sandbox, Taxi-v4, and ALFWorld. They find a learned allocation improves success over direct execution in the first two settings, but does not beat a hand-designed policy in Taxi success and does not improve task success over direct execution in the ALFWorld.
Critical strengths
The intervention that this paper presents is straightforward and simple to understand. Before taking an action, a separate controller can either trust the agent or restore the current state, try the proposed action and an alternative, compare what happens, and choose one. These “forks” are limited in number so the controller must learn where to spend them.
Critical weaknesses
This paper is difficult to read and the general writing style seems like LLMs may have been used too much in the writing process. For instance, I have found that one action that LLMs tend to take in writing up results is simply listing a series of numbers that the readers probably would not care about (e.g. many of the numbers in line 22-33 are not important for a reader to immediately know). The authors are encouraged to clean up the writing and make the most important points more evident.
Additionally, the empirical contribution is difficult to assess against existing methods. Although the paper discusses selective reflection, verifier-guided action selection, and learned allocation of agent compute, the experiments do not compare against representative techniques from those lines of work or explain clearly why such comparisons are infeasible. The gains are confined largely to controlled settings and Taxi. In the better-controlled ALFWorld evaluation, the learned controller does not improve task success or total runtime cost over direct execution. As a result, the paper does not establish that this method offers any advantage.
Questions for authors
N/A
Thank you for the close read. You are right about the writing, and the abstract is the worst of it. It lists success rates for the sandbox, for Taxi at three error rates, and for two ALFWorld sets, one after another, before the reader knows what the paper is for. We have rewritten it for the revision. It now carries two numbers: the Taxi gain over direct execution at error rate 0.10, and the ALFWorld result that all three methods finish 23/24. Everything else moves to the results section. The introduction has been rewritten the same way: it opens with a concrete case — a tool agent about to run a destructive command in a sandbox — states the scope limit about irreversible actions in the second paragraph, and states the narrowed contribution in one paragraph. We have also removed the orphan bibliography entry for Kim et al. on agentic test-time compute, which was listed but never cited.
You are right that the paper discusses selective reflection, verifier-guided selection, and learned allocation without comparing against any of them. VeGAS is the closest: it samples 16 candidate actions and picks the one a generative verifier scores highest, without running any of them. We did not run it head-to-head. It does not run on our locked 24-task ALFWorld set or on Taxi at a matched fork budget without a rebuild, and a rebuild inside a rebuttal would be a new method, not a baseline. We ran that ablation. Score-only also finishes 23/24, with the same sole failure, 0 forks and 0 replay, but 5,711.8 tokens per task against 5,347.1 for executed branches, 24 challenges against 18, and 2 overrides against 5. Executing the branches adds no success on this set; with 24 tasks and one direct failure, that is an exploratory null. The revision will also add a Related Work paragraph for each of CATTS, VeGAS, CHILL-Harness, AgentCheck, and EvoHarness-RL, saying what each does and where this paper differs. That does not replace a head-to-head. It stops the paper reading as if those methods did not exist.
Your summary of where the gains sit is accurate. They sit in the controlled sandbox and in Taxi against direct execution and equal allocation. The sandbox gives 17.0% success against 14.1% for equal allocation and 9.3% for direct execution at a fork budget of 12, but that is a controlled setting with noisy diagnostics, and it does not carry the paper on its own. On Taxi, the learned controller beats direct execution by 13.9, 10.1, and 6.5 points at error rates 0.10, 0.20, and 0.30, and loses to reserve-aware by 3.4, 4.1, 5.1, and 5.4 points. At matched fork counts it never beats reserve-aware; the full table is in our reply to Reviewer pWNY. The better-controlled ALFWorld test shows no success gain and no total-cost gain: 23/24 for all three methods, with replay rising from 14.92 to 16.75 steps per task even as forks fall from 2.50 to 1.50.
We will not dress that up. The revision will frame the ALFWorld result as a null result with a cost breakdown, and the Taxi result as a win over direct execution and over a harmful baseline, not over the best hand rule. Four changes carry that framing: the rewritten abstract and introduction, the matched-fork table added to the Taxi section with reserve-aware named as the best rule for success, the ALFWorld cost table in the main text (forks, replay, critic calls, tokens), and the Related Work paragraphs above. If that leaves the paper thinner, it also leaves it accurate, and we would rather submit the thinner accurate version.