- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies "model-free actions": tool calls an agent executes without querying the LLM, built by reusing argument values from earlier calls (as in AutoTool). It proposes certify–repair–ask (CRA), a training-free gate. CRA executes the proposed tuple if its preconditions are certified in strong Kleene logic over observation-grounded predicates. Otherwise it executes the highest-ranked certified cached tuple, and otherwise it asks the LLM.
The evaluation has three protocols on ScienceWorld and ALFWorld: counterfactual execution at 2,339 restored decision points, same-state branches, and closed-loop runs with Gemini 3.8 Flash and Claude Sonnet 4.6. With an expert catalog, CRA lowers the refusal rate of model-free actions from 80.2%/93.7% to 28.5%/26.6%. However, in closed loop, no gate detectably improves success or model calls, and in ALFWorld CRA lowers success relative to simple rejection.
Critical strengths
- The abstract and conclusion state that no gate improves closed-loop success or calls, which is uncommon and valuable.
- The finding that precondition checks cannot distinguish progress from repetition (“executable is not the same as right”) is relevant to anyone building caching or speculative execution for agents.
Critical weaknesses
W1. The proposed method shows no end-to-end benefit.
The title, abstract, and contributions present CRA as the solution for safely skipping the model to save inference cost. However, the paper's own closed-loop results, its only reported test of that goal, do not support this.
In ScienceWorld, CRA does not detectably improve success or model calls over unguarded reuse (Table 9). In ALFWorld, it performs worse than the simpler reject gate, clearly so with Claude, while taking more steps and more model calls (Table 10; Section 5.4). In no reported setting is CRA either the most successful or the cheapest arm (Table 2).
CRA does reduce refused actions, but this does not reduce model calls (Section 5.4). Its repairs often send the agent back to earlier states, which costs extra steps and calls later (Section 5.3; Table 6). The only variant that removes the ALFWorld loss, effect-aware CRA, does so by asking the model instead of repeating actions, and it merely matches rejection. It was also run separately, a day after the other arms (Appendix D).
The authors acknowledge these results (Section 5.4, "Answer to RQ3"; Section 7), but they leave the paper's main methodological claim unsupported.
Questions for authors
- Is there any setting where CRA reduces model calls or cost relative to the reject gate at matched success? If not, why is CRA presented as the paper’s method rather than as a studied baseline?
- Can you report a cost–success frontier (model calls or tokens vs. success) across all closed-loop arms in Table 2, so readers can see where, if anywhere, CRA is the preferred operating point?
5 comments
Response to Reviewer TeFe (1/2)
Thanks for your precise and constructive comments. We agree with W1 and have changed the paper's claims accordingly (see the general response). Both questions are answered with new runs that were registered before they ran, with every arm run three times together.
Q1: Is there any setting where CRA reduces model calls or cost relative to the reject gate at matched success? If not, why is CRA presented as the paper's method rather than as a studied baseline?
A1: With Claude yes, with GPT-5.6 and Grok 4.7 no. During the discussion we built and registered a progress-aware CRA (general response, part 3). On 140 ALFWorld games that no earlier run used, with Claude it changes calls by −0.7 per episode relative to Reject (CI [−1.1, −0.3]) and success by 3.1 points (CI [0.5, 5.7]). In registered replications on the same games, calls change by −0.1 with GPT-5.6 (CI [−0.5, 0.4]) and −0.4 with Grok 4.7 (CI [−0.9, 0.1]), and success by −1.0 and 3.6 points (CIs [−4.3, 2.1] and [−1.0, 8.1]). So we can claim the saving for one of the three models, not in general. For CRA as submitted, the answer is no.
The closest for CRA as submitted is ScienceWorld with AutoTool's own filler. There CRA asks the model less than Reject (−1.5 calls per episode, CI [−2.9, −0.1]), but success changes by −5.7 points (CI [−14.3, 1.0]), an interval too wide to call the two matched. Native AutoTool without any gate reaches 20.0% at 19.7 calls, against 20.0% at 22.2 for Reject and 14.3% at 20.7 for CRA. The earlier single run in this setting had CRA ahead of Reject by 17.1 points; the repeated runs do not confirm it, and the revision reports both. In ALFWorld, CRA uses more calls than Reject with Gemini and about as many with Claude (0.6 and −0.1 per episode) and lowers success by 4.0 and 6.5 points.
In tokens the answer is the same. CRA minus Reject is −1.7k tokens per episode on AutoTool's filler (CI [−21.4, 18.9]), and 1.5k and 0.3k in ALFWorld (CIs [0.5, 2.5] and [−3.0, 3.5]; Claude's counts include a fixed prefix that the endpoint adds to every request). If each refused action cost λ model calls (Appendix D, not registered), CRA would be undominated only with Claude, for 0.2 < λ < 1.4 (its success detectably below that of native AutoTool, Reject and effect-aware CRA and not detectably different from that of the unguarded filler), and on AutoTool's filler, for λ > 2.5 (its success not detectably different from that of native AutoTool and Reject); with Gemini, at no price. Effect-aware CRA, undominated only with Claude, does not detectably reduce cost relative to Reject either: it changes calls by −0.1 and −0.2 per episode (CIs [−0.3, 0.1] and [−0.7, 0.3]) and tokens by −0.3k and −1.3k, at 0.2 and 0.5 points of success.
Why CRA is in the paper: it is the strongest test of the hypothesis that checking executability makes model-free actions worthwhile. It certifies in three-valued logic, repairs from the cache, and acts model-free wherever rejection does, so its failure cannot be blamed on a weak check. The revision presents CRA as the gate under study, and the contribution as the framework and the diagnosis it gives: certification fixes executability but not progress, because preconditions still hold after an action has taken effect. One point you found valuable changes with the repeated runs: the submission's statement that no gate detectably improves success or calls over unguarded reuse or native AutoTool no longer holds with Claude, and the revision drops it, while repair itself still improves on neither baseline.
Thank you for the extensive additional experiments and the more transparent presentation of the results. I appreciate the authors' willingness to revise their claims.
However, my main concern remains unresolved. CRA itself is dominated across the reported cost–success frontiers. The progress-aware variant shows promising results only with Claude on ALFWorld, but fails to replicate with GPT-5.6 and relies on a manually designed, environment-specific heuristic. While reframing the paper as a diagnostic study is more appropriate, I remain unconvinced that its insights provide sufficient novelty and methodological contribution. I maintain my original rating.
Response to Reviewer TeFe (2/2)
Q2: Can you report a cost–success frontier across all closed-loop arms in Table 2?
A2: Yes. New Figure 4 plots model calls per episode against success for every closed-loop arm, with 95% intervals on both axes. An arm inside the shaded region is dominated, because another arm of the panel is no worse in success and calls and better in one. Panels (a) and (b) are the earlier ScienceWorld runs (all arms of Table 2); (c) to (e) are the new repeated runs. The undominated arms, panel by panel: (a) native AutoTool; (b) native AutoTool and the unguarded filler; (c) native AutoTool; (d) native AutoTool; (e) native AutoTool, the unguarded filler and effect-aware CRA. Figure 5 repeats the figure with tokens per episode: (a) native AutoTool; (b) the unguarded filler; (c) native AutoTool; (d) native AutoTool; (e) native AutoTool, the unguarded filler and effect-aware CRA. For the single ALFWorld runs that the submission reported in Table 2 (now Table 10), the frontier is Reject with Gemini and native AutoTool, the unguarded filler and Reject with Claude. On every one of these frontiers, on calls and on tokens, CRA itself is dominated; effect-aware CRA is undominated only with Claude, where it and Reject do not differ detectably in success or calls.
Panel (f) is E4, the progress-aware run on 140 new games (general response, part 3); its frontier holds progress-aware CRA on calls and on tokens (Figure 5f for tokens).
Q3 (from W1): Effect-aware CRA was run a day after the other arms.
A3: It is now run together with all other arms, three times. It raises success over CRA by 4.2 and 7.0 points (CIs [1.5, 7.5] and [2.5, 11.9]) and changes success relative to Reject by 0.2 and 0.5 points, so the earlier reading holds: it recovers what repair loses and stops at the level of rejection.