acceptodds
Under review as a conference paper at ICLR 2027

Skip the Model, Not the Checks: Certifying Model-Free Actions of LLM Agents

Abstract

To save inference cost, LLM agents increasingly act without the model: a lightweight proposer predicts the next tool and fills its arguments with values reused from earlier calls. Such actions are cheap but barely checked. We propose certify–repair–ask (CRA), a training-free gate that executes a proposed action only if its preconditions are certified in three-valued logic over observation-grounded predicates, otherwise executes the best certified alternative among the cached values, and asks the model only when none exists. We restore ScienceWorld and ALFWorld to 2339 decision points of an AutoTool agent, execute each gate's action there, and branch episodes from the same states. First, certification cuts refusals: with an expert catalog, CRA lowers the refusal rate of model-free actions from 80.2% and 93.7% to 28.5% and 26.6%, where a reject gate almost never acts. Second, executable is not the same as right: catalogued preconditions still hold after an action has taken effect, so repair replays the agent's recent past (where the model acted, 35.4% of CRA's accepted ScienceWorld actions re-execute a state change already made, against 3.9% of the model's, and every accepted ALFWorld navigation repair returns to the previous location), and branched from the same states they cost ALFWorld agents extra steps and calls without detectable success gains. Third, in closed loop no gate detectably improves success or model calls over unguarded reuse or native AutoTool, and in ALFWorld repair lowers success relative to rejection, clearly with Claude, less so with Gemini; asking instead of repeating an accepted action restores it. Our results suggest that gates on model-free actions need a progress-sensitive signal beyond executability: a refused action costs a step, an executable wrong one can cost the task.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Tool Use, Argument Binding, Constraint Satisfaction, Training-Free Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TeFeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies "model-free actions": tool calls an agent executes without querying the LLM, built by reusing argument values from earlier calls (as in AutoTool). It proposes certify–repair–ask (CRA), a training-free gate. CRA executes the proposed tuple if its preconditions are certified in strong Kleene logic over observation-grounded predicates. Otherwise it executes the highest-ranked certified cached tuple, and otherwise it asks the LLM.

The evaluation has three protocols on ScienceWorld and ALFWorld: counterfactual execution at 2,339 restored decision points, same-state branches, and closed-loop runs with Gemini 3.8 Flash and Claude Sonnet 4.6. With an expert catalog, CRA lowers the refusal rate of model-free actions from 80.2%/93.7% to 28.5%/26.6%. However, in closed loop, no gate detectably improves success or model calls, and in ALFWorld CRA lowers success relative to simple rejection.

Critical strengths
  • The abstract and conclusion state that no gate improves closed-loop success or calls, which is uncommon and valuable.
  • The finding that precondition checks cannot distinguish progress from repetition (“executable is not the same as right”) is relevant to anyone building caching or speculative execution for agents.
Critical weaknesses

W1. The proposed method shows no end-to-end benefit.

The title, abstract, and contributions present CRA as the solution for safely skipping the model to save inference cost. However, the paper's own closed-loop results, its only reported test of that goal, do not support this.

In ScienceWorld, CRA does not detectably improve success or model calls over unguarded reuse (Table 9). In ALFWorld, it performs worse than the simpler reject gate, clearly so with Claude, while taking more steps and more model calls (Table 10; Section 5.4). In no reported setting is CRA either the most successful or the cheapest arm (Table 2).

CRA does reduce refused actions, but this does not reduce model calls (Section 5.4). Its repairs often send the agent back to earlier states, which costs extra steps and calls later (Section 5.3; Table 6). The only variant that removes the ALFWorld loss, effect-aware CRA, does so by asking the model instead of repeating actions, and it merely matches rejection. It was also run separately, a day after the other arms (Appendix D).

The authors acknowledge these results (Section 5.4, "Answer to RQ3"; Section 7), but they leave the paper's main methodological claim unsupported.

Questions for authors
  1. Is there any setting where CRA reduces model calls or cost relative to the reject gate at matched success? If not, why is CRA presented as the paper’s method rather than as a studied baseline?
  2. Can you report a cost–success frontier (model calls or tokens vs. success) across all closed-loop arms in Table 2, so readers can see where, if anywhere, CRA is the preferred operating point?
5 comments
AuthorsOpenReview ↗

Response to Reviewer TeFe (2/2)

Q2: Can you report a cost–success frontier across all closed-loop arms in Table 2?

A2: Yes. New Figure 4 plots model calls per episode against success for every closed-loop arm, with 95% intervals on both axes. An arm inside the shaded region is dominated, because another arm of the panel is no worse in success and calls and better in one. Panels (a) and (b) are the earlier ScienceWorld runs (all arms of Table 2); (c) to (e) are the new repeated runs. The undominated arms, panel by panel: (a) native AutoTool; (b) native AutoTool and the unguarded filler; (c) native AutoTool; (d) native AutoTool; (e) native AutoTool, the unguarded filler and effect-aware CRA. Figure 5 repeats the figure with tokens per episode: (a) native AutoTool; (b) the unguarded filler; (c) native AutoTool; (d) native AutoTool; (e) native AutoTool, the unguarded filler and effect-aware CRA. For the single ALFWorld runs that the submission reported in Table 2 (now Table 10), the frontier is Reject with Gemini and native AutoTool, the unguarded filler and Reject with Claude. On every one of these frontiers, on calls and on tokens, CRA itself is dominated; effect-aware CRA is undominated only with Claude, where it and Reject do not differ detectably in success or calls.

Panel (f) is E4, the progress-aware run on 140 new games (general response, part 3); its frontier holds progress-aware CRA on calls and on tokens (Figure 5f for tokens).

Q3 (from W1): Effect-aware CRA was run a day after the other arms.

A3: It is now run together with all other arms, three times. It raises success over CRA by 4.2 and 7.0 points (CIs [1.5, 7.5] and [2.5, 11.9]) and changes success relative to Reject by 0.2 and 0.5 points, so the earlier reading holds: it recovers what repair loses and stops at the level of rejection.

AuthorsOpenReview ↗

Response to Reviewer TeFe (1/2)

Thanks for your precise and constructive comments. We agree with W1 and have changed the paper's claims accordingly (see the general response). Both questions are answered with new runs that were registered before they ran, with every arm run three times together.

Q1: Is there any setting where CRA reduces model calls or cost relative to the reject gate at matched success? If not, why is CRA presented as the paper's method rather than as a studied baseline?

A1: With Claude yes, with GPT-5.6 and Grok 4.7 no. During the discussion we built and registered a progress-aware CRA (general response, part 3). On 140 ALFWorld games that no earlier run used, with Claude it changes calls by −0.7 per episode relative to Reject (CI [−1.1, −0.3]) and success by 3.1 points (CI [0.5, 5.7]). In registered replications on the same games, calls change by −0.1 with GPT-5.6 (CI [−0.5, 0.4]) and −0.4 with Grok 4.7 (CI [−0.9, 0.1]), and success by −1.0 and 3.6 points (CIs [−4.3, 2.1] and [−1.0, 8.1]). So we can claim the saving for one of the three models, not in general. For CRA as submitted, the answer is no.

The closest for CRA as submitted is ScienceWorld with AutoTool's own filler. There CRA asks the model less than Reject (−1.5 calls per episode, CI [−2.9, −0.1]), but success changes by −5.7 points (CI [−14.3, 1.0]), an interval too wide to call the two matched. Native AutoTool without any gate reaches 20.0% at 19.7 calls, against 20.0% at 22.2 for Reject and 14.3% at 20.7 for CRA. The earlier single run in this setting had CRA ahead of Reject by 17.1 points; the repeated runs do not confirm it, and the revision reports both. In ALFWorld, CRA uses more calls than Reject with Gemini and about as many with Claude (0.6 and −0.1 per episode) and lowers success by 4.0 and 6.5 points.

In tokens the answer is the same. CRA minus Reject is −1.7k tokens per episode on AutoTool's filler (CI [−21.4, 18.9]), and 1.5k and 0.3k in ALFWorld (CIs [0.5, 2.5] and [−3.0, 3.5]; Claude's counts include a fixed prefix that the endpoint adds to every request). If each refused action cost λ model calls (Appendix D, not registered), CRA would be undominated only with Claude, for 0.2 < λ < 1.4 (its success detectably below that of native AutoTool, Reject and effect-aware CRA and not detectably different from that of the unguarded filler), and on AutoTool's filler, for λ > 2.5 (its success not detectably different from that of native AutoTool and Reject); with Gemini, at no price. Effect-aware CRA, undominated only with Claude, does not detectably reduce cost relative to Reject either: it changes calls by −0.1 and −0.2 per episode (CIs [−0.3, 0.1] and [−0.7, 0.3]) and tokens by −0.3k and −1.3k, at 0.2 and 0.5 points of success.

Why CRA is in the paper: it is the strongest test of the hypothesis that checking executability makes model-free actions worthwhile. It certifies in three-valued logic, repairs from the cache, and acts model-free wherever rejection does, so its failure cannot be blamed on a weak check. The revision presents CRA as the gate under study, and the contribution as the framework and the diagnosis it gives: certification fixes executability but not progress, because preconditions still hold after an action has taken effect. One point you found valuable changes with the repeated runs: the submission's statement that no gate detectably improves success or calls over unguarded reuse or native AutoTool no longer holds with Claude, and the revision drops it, while repair itself still improves on neither baseline.

Thank you for the extensive additional experiments and the more transparent presentation of the results. I appreciate the authors' willingness to revise their claims.

However, my main concern remains unresolved. CRA itself is dominated across the reported cost–success frontiers. The progress-aware variant shows promising results only with Claude on ALFWorld, but fails to replicate with GPT-5.6 and relies on a manually designed, environment-specific heuristic. While reframing the paper as a diagnostic study is more appropriate, I remain unconvinced that its insights provide sufficient novelty and methodological contribution. I maintain my original rating.

Reviewer_rxTAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper considers LLM agents that solve multi-step tasks by emitting calls to different tools at each step. However, using an LLM agent to choose a tool at each step may be computationally expensive. To address this cost inefficiency, recent work has explored the possibility of executing model-free actions, i.e. picking the next tool without running an LLM model, but instead inferring the next tool by caching previous responses and statistically predicting the next action. The present work builds upon the prior cache-and-predict method AutoTool of (Jia & Li, 2026), and addresses its limitation that the predicted action is not checked to be executable. To address this limitation, the present work proposes using a guard (or a precondition), which is a conjunction of predicates in Kleene three-valued logic, that must be satisfied for the action to be executable. Then, a predicted action is allowed to be executed only if it satisfied the guard. Otherwise, the action is repaired by considering other cached actions and selecting a highest ranked action that satisfies the guard, based on internally inferred action ranks. Finally, if it is not possible to repair the action, the method calls the LLM model. This leads to the certify-repair-ask (CRA) pipeline. Experiments show that (1) CRA leads to a reduced refusal rate of non-executable predicted actions, while (2) executable actions do not necessarily lead to good actions, and (3) reduced refusal rate does not necessarily lead to better episodes.

Critical strengths
  1. Relevance. The paper considers an important problem of improved cost-efficiency of LLM models for multi-step tasks by reducing the number of calls to a model, and instead caching and reusing previous actions to predict new actions in a model-free manner.

  2. Novelty. The proposed certify-repair-ask pipeline is a very neat idea that is to the best of my knowledge novel. The paper provides a detailed empirical analysis of this pipeline.

Critical weaknesses
  1. Executable action is not necessarily a good action. While the proposed certify-repair-ask idea is neat, most conclusions of the empirical analysis unfortunately seem to be negative, so it is unclear how useful CRA is in practice. The fact that the refusal rate is decreased is obvious and expected, since CRA allows repairing some of the refused actions. On the other hand, as authors also point out, the experiments show that an action being executable is not sufficient for the action to be good. Hence, CRA alone is not a good criterion for deciding when to use or not to use an LLM model to choose the next action.
    Perhaps one way to address this issue would be to design guards that not only consist of predicates that specify what actions are executable, but also what actions yield good performance. But this leads to the question of how do design good guards, which is my second Weakness point.

  2. Design of guards. The CRA method assumes that the guards are provided a priori. However, it is unclear how to design these guards in practice. Moreover, I imagine the guards would need to be very dependent on the nature of tasks that an LLM needs to solve, hence it would be hard to come up with a principled framework for guard design. The paper briefly mentions that the guards are hand-built, but it does not provide much insight on how to design good guards.

Questions for authors
  1. Please comment on the Weakness point 1 above.

  2. What were the guards that you used in the experiments? And, in general, how can one design good guards?

5 comments
AuthorsOpenReview ↗

Response to Reviewer rxTA (2/2)

Q3 (Weakness 2): Which guards were used, and how can one design good guards? Do they depend on the task?

A3: We used three catalogs, listed in full in Tables 4 and 5: an expert catalog, a catalog mined from accepted steps of execution logs, and a catalog written once, offline, by the LLM from the tool documentation. The new Section 3.3 describes how we build and refine them in four steps, each checked by counterfactual execution on development states:

  1. A small predicate library (visible, in inventory, reachable, container, known location, not the current location, distinct) expresses preconditions of multi-argument tools in both environments.
  2. Each tool gets the predicates of its arguments, by hand, by mining, or in one LLM call. The LLM-written catalog needs no per-task work and is refused 8.2 points more often than the expert catalog in ScienceWorld (36.7% vs. 28.5%).
  3. Refusals show where a predicate is missing. We execute certified actions at restored states and sort each refusal into "precondition the catalog lacks" or "grounding error", which points to the fix. For example, "not the current location" removed stale navigations, and "the agent is at the receptacle" turns all 55 refused ALFWorld take actions of CRA into asks without changing an accepted one.
  4. Tools without predicates pass on record checks alone; a fail-closed guard sends them to the model instead, at a measured cost in calls.

On coverage, a new and registered analysis (Appendix D): of the 425 refused ScienceWorld tuples whose simulator message names a precondition, at least one library check is Violated for 421 (94 of 95 at held-out states); the other 390 refusals fail on the surface form of the action. Choosing the checks is the hard part: all 39 checks together also flag 47.5% of accepted tuples, while the 12 checks that flag no accepted development tuple catch 45.0% of refused ones and, frozen, 39.0% at held-out states without flagging any of the 160 accepted ones.

On task dependence: predicates are written per tool and environment, not per task, so one catalog serves all tasks of an environment, including held-out ones. Frozen, the expert catalog is refused 21.1% of the time at 86 restored states from 20 held-out tasks that did not shape it, against 82.6% for the unguarded filler (Appendix D). The guards encode applicability only, and this is exactly why they cannot tell progress from repetition.

AuthorsOpenReview ↗

Response to Reviewer rxTA (1/2)

Thanks for your constructive comments, and for finding the idea neat. We provide our feedback and revise our paper as follows.

Q1 (Weakness 1): An executable action is not necessarily a good one, most conclusions are negative, and the lower refusal rate is obvious.

A1: We agree that executability is the wrong criterion for skipping the model, and the revision makes this the main finding. We would qualify one point: the lower refusal rate does not follow from repair as such. Repair without certificates accepts about as many actions, but it is refused 36.8 points (distinct repair, CI [29.7, 44.9]) and 25.7 points (Unknown read as a pass, CI [19.3, 32.5]) more often than CRA in ScienceWorld. Checking without repair, the reject gate, almost never acts. So the reduction comes from certifying the repaired tuple.

Beyond that, the paper measures what else changes when the gate acts, and the answer is mostly negative. The revision presents it this way: CRA is the strongest executability gate we could build and the object of a diagnostic study, not the recommended method. The protocols show why executability is the wrong criterion. Catalogued preconditions still hold after an action has taken effect, so recency-ranked repair replays the recent past (35.4% of CRA's accepted ScienceWorld actions re-execute a state change already made, against 3.9% of the model's). Also, the fatal ScienceWorld actions come from tools the catalog has no predicates for.

Q2 (your suggestion in Weakness 1): guards that encode not only what is executable, but what yields good performance.

A2: This is the direction our results point to. We tested its simplest form, effect-aware abstention: ask the model instead of executing a model-free action that repeats an accepted action of the episode. In the new registered runs, with all arms run together three times, it raises success over CRA by 4.2 points with Gemini (CI [1.5, 7.5]; 9 vs. 1 games in the first repetition, p = 0.021) and 7.0 with Claude (CI [2.5, 11.9]), with fewer calls (Gemini) and at about the same number of calls (Claude): it recovers the loss and ends level with Reject (0.2 and 0.5 points relative to Reject). So a useful progress signal has to be more than "not a repeat", and the paper says so.

We then built a stronger signal for navigation: progress-aware CRA leaves a receptacle only once it shows nothing closed and nothing of a type named in the goal, and then goes to the next unvisited receptacle of its type. Registered before it ran on 140 ALFWorld games that no earlier run used, with Claude it uses fewer model calls than both effect-aware CRA and Reject, with higher success than Reject and no detectable difference from effect-aware CRA; registered replications with GPT-5.6 and Grok 4.7 confirm neither call reduction (general response, part 3).

Reviewer_rxTAOpenReview ↗

Official Comment by Reviewer rxTA

Thank you for the detailed response and for clarifying which guards were used in the experiments.

However, I still find the paper's contributions insufficient to justify acceptance, and will keep my score. The main conclusion is largely negative, i.e. that an action being executable is not sufficient for it to provide good performance. I don't think that this is enough to justify acceptance at ICLR.

The additional experiment on designing guards that lead to good performance is a promising step, but it goes beyond the scope of the original submission. Including such experiments would require repositioning the whole paper. That is a major revision, which would require the paper to go through a full new round of reviews.

Reviewer_xUmHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers the paradigm of model-free actions where actions are taken by agents without consulting/asking an LLM. For that purpose, they propose the certify-repair-ask gate which checks if the proposer's action satisfies the required predicates, and if not, checks for the closest alternative via repair, and if that doesn't exist, only then asks a model for the next action. They evaluate the impact of this proposed gate on the decision points of ScienceWorld and ALFWorld upon using the AutoTool agent and show the positive impact of the refusal rate.

Critical strengths
  • Presents the importance of certifying actions, and the need for gates, by quantifying the lowered refusal rates.
  • Presented with evidence the argument that executable is not the same as right. And one has to look for signal beyond executability, since refused action costs steps but wrong executable costs the entire failed task.
Critical weaknesses
  • The second strength is not fully validated: One has to look for signal beyond executability, since refused action costs steps but wrong executable costs the entire failed task. But in their results, the success rate is best for unguarded.
Questions for authors
  • Fix Fig. 1a the top arm (yes)
  • Table 2 success % goes down with CRAs, especially for ScienceWorld.. but isn't that a negative result? No detailing and discussion was presented about what we gain while we forego the success rate.
2 comments
AuthorsOpenReview ↗

Response to Reviewer xUmH

Thanks for your constructive comments. We provide our feedback and revise our paper as follows.

Q1: Fix Fig. 1a, the top arm (yes).

A1: Fixed. The "yes: execute g" arm now leaves the certificate box from the top and enters "Execute" from above, with a visible arrowhead.

Q2: Table 2 success goes down with CRA, especially for ScienceWorld. Isn't that a negative result, and what do we gain?

A2: Yes, it is a negative result for CRA as a way to save cost, and the revision says so plainly. Two points.

(1) In ScienceWorld the differences are within noise. On the development tasks, CRA changes success by −1.7 points relative to the unguarded filler (CI [−15.5, 13.8]); in the new repeated runs on AutoTool's own filler, by −5.7 points relative to native AutoTool (CI [−15.2, 3.8]). In ALFWorld the loss is detectable: in the repeated runs, CRA lowers success relative to Reject by 4.0 points with Gemini (CI [−7.5, −1.0]) and 6.5 with Claude (CI [−11.2, −2.2]).

(2) What CRA gains is executability. It removes most refused model-free actions: 12 of 171 are refused for CRA, against 121 of 240 for the unguarded filler on the same tasks. This pays only where a failed call costs more than a step, for example a billed or side-effecting API. We now quantify it (Appendix D, not registered): if each refused action cost λ model calls, CRA would be undominated only with Claude, for 0.2 < λ < 1.4 (its success detectably below that of native AutoTool, Reject and effect-aware CRA and not detectably different from that of the unguarded filler), and on AutoTool's filler, for λ > 2.5 (its success not detectably different from that of native AutoTool and Reject); with Gemini, at no price. In these benchmarks a refused action costs one step, so CRA's fewer refusals turn into neither success nor fewer calls, and Section 7 says this directly.

The paper now explains why. Catalogued preconditions still hold after an action has taken effect, so certified repairs replay the agent's recent past (35.4% of CRA's accepted ScienceWorld actions re-execute a state change already made, against 3.9% of the model's). From the same states, such repairs cost ALFWorld agents steps and calls.

What the diagnosis gains beyond CRA is a target: a gate needs a progress signal. We built one during the discussion and registered its test on 140 ALFWorld games that no earlier run used: with Claude, progress-aware CRA uses fewer model calls than both effect-aware CRA and Reject, with higher success than Reject and no detectable difference from effect-aware CRA; registered replications with GPT-5.6 and Grok 4.7 confirm neither call reduction (general response, part 3).

Q3: The second strength is not fully validated, since unguarded has the best success rate.

A3: We agree the wording was too strong, and we removed the sentence from the abstract, so it no longer reads as a claim that CRA beats unguarded reuse. The sentence "a refused action costs a step, an executable wrong one can cost the task" describes a mechanism. The unguarded arm doing as well as CRA, or better, is consistent with it.

On which arm is best: in Table 2 of the submission, unguarded reuse was the most successful arm only on the held-out tasks: the most successful arm was native AutoTool (46.6%) on the development tasks, the unguarded filler (43.9%) on the held-out tasks, Reject (100.0%) in ALFWorld with Gemini and native AutoTool (89.6%) with Claude. In the repeated runs it is native AutoTool in ALFWorld with both models (98.8% and 85.8%), and native AutoTool and Reject (20.0%) on AutoTool's filler (Table 12).

On the mechanism: on the ScienceWorld development tasks, 5 of the 7 episodes that CRA ends with a negative score follow a model-free focus on that passed on record checks (the catalog has no predicate for it), against 1 of 3 for the unguarded filler. Such an action ends the task, while a refused one costs a step. The held-out runs show the limit: sending focus on to the model removes these endings but not the failures, because the model then focuses on the wrong object itself (Section 5.4). The counts are small, so we present this as a mechanism, not a significant difference.

We also registered a branch test of this mechanism; it stopped incomplete when the endpoint stopped serving Gemini, and as registered we analyze the states complete at the stop. At 5 states where the unguarded filler's action was refused, executing that refused action instead of CRA's accepted one changes success by 0.0 points (CI [−8.3, 33.3]), so the registered hypothesis that executing the refused action leaves more success than CRA's is not confirmed. At 10 states before a fatal focus on, re-executing it ended the episode again in all three runs at 10 of them, and the model, acting from the same state, succeeded in 16.7% of its runs (Appendix D).

AuthorsOpenReview ↗

Follow-up for Reviewer xUmH

Thank you again for your review. As the discussion closes on November 18, here is a short guide to where the revised PDF answers your three points, in case it saves you time.

Q1 (Fig. 1a). Fixed: the "yes: execute g" arm now enters "Execute" from above, with a visible arrowhead (Figure 1a).

Q2 (lower success with CRA). We agree it is a negative result for CRA as a way to save cost, and the abstract, the answer to RQ3 (Section 5.4) and Section 7 now say so: CRA is dominated on every cost–success frontier. What it gains is executability, which pays only where a failed call costs more than a step. The revision also measures why success does not follow: catalogued preconditions still hold after an action has taken effect, so 35.4% of CRA's accepted ScienceWorld actions redo a state change already made, against 3.9% of the model's (Section 5.3).

Q3 (the second strength). We agree it was too strong and removed the sentence from the abstract. The mechanism it described is now reported with its counts, which are small, as a mechanism and not as a significant difference (Section 5.4 and Table 2).

The experiments we added during the discussion, all registered before their first episode and reported whatever their outcome, are listed in the general response (part 2).

We hope this resolves your concerns; if so, we would be grateful if you would consider updating your rating. If anything is still open, we are glad to answer before the deadline.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Summary of changes in the revised PDF

The main text uses the additional page allowed for the discussion. New and revised text is in blue; cuts that only shorten the wording are not marked. Changes, in order of appearance:

  • Abstract and Section 1 (framework paragraph, findings, contributions): diagnostic framing.
  • Section 2 and Appendix A: Proposition 3, a formal statement of the paper's mechanism (an executed action stays certified while its checks hold, so a gate that decides from preconditions alone treats its repetition as a first execution), which is the replay that Section 5.3 measures; Sections 3.3 and 7 refer to it.
  • Section 3.3 (new): building and refining guards.
  • Figure 1a: the "yes" arm.
  • Section 4: one sentence on the repeated runs and one on how calls, tokens and steps are counted; how heat and cool are rendered moved to Appendix C.
  • Section 5.2: same results; the ablation counts moved to Appendix D.
  • Section 5.3: same results, fewer numbers in the running text.
  • Section 5.4 with the new Figure 4: frontier reading, repeated runs, the corrected result on AutoTool's own filler, the effect-aware rule from the repeated runs, and what the registered rerun shows. To make room, the ALFWorld single runs moved from Table 2 to Table 10 in the appendix.
  • Abstract, Section 1, Sections 3.3 and 5.4, Figure 4f, Section 7, Appendix D (new Tables 19, 20 and 21) and Table 8: the progress-aware rule, its registered run on new games, and its registered replications with GPT-5.6 and Grok 4.7.
  • Table 8 and Appendix D: a second registered Grok 4.7 test, on 600 new games, stopped incomplete when the endpoint's usage balance ran out; as registered, it is not analyzed.
  • Section 6: shorter wording, same content.
  • Section 7: what certification buys and where it does not pay; limitations.
  • Figures 2, 4, 5 and 6: encodings that survive grayscale printing and stay the same across figures (gate colors and markers), with outlined markers and darker intervals in the frontier panels.
  • Appendix C: branch sign test and first-run analysis, moved from the main text.
  • Appendix D: protocol of the repeated runs, including a change in the endpoint's token accounting and an interrupted first Claude pass that was set aside and rerun in full; new Table 12 and Figure 5; charging refused actions; the registered confirmation run (new Table 18); updated native-filler and effect-aware paragraphs (with why the gates help only with Claude) and Table 17; the secondary statistics moved from Sections 5.2 to 5.4.
  • Appendix D and Table 8: the registered branch experiments at refused and fatal ScienceWorld states, which stopped incomplete when the endpoint stopped serving Gemini, analyzed on the states complete at the stop, as registered.

All numbers in the paper and in these responses are generated by scripts from the raw run logs.

AuthorsOpenReview ↗

General response to all reviewers (3/3)

8. A gate built from the diagnosis. rxTA (Q2) asked for guards that encode what yields good performance, TeFe (Q1) for a setting where a gate saves calls relative to Reject at matched success, and xUmH (Q2) what we gain. Our diagnosis names what is missing, a progress signal. So during the discussion we built one and tested it like everything else, registered before it ran.

  • The rule. Progress-aware CRA adds one rule to effect-aware CRA, for navigation only. A model-free go may leave a receptacle only once it is exhausted: its arrival or opening observation shows nothing closed and no object of a type named in the goal, and the agent holds nothing. It then goes to the next receptacle of the same type (shelf 3 after shelf 2), if the first observation lists it and it is unvisited. Any other model-free destination must be a listed, unvisited receptacle, or the model is asked. We derived the rule from the logs of the earlier ALFWorld runs without new model calls; where it would fire, Claude itself took the same step 97% of the time.
  • The test. ALFWorld's 140 in-distribution evaluation games (valid_seen), which none of our runs had used. The five arms (native AutoTool, unguarded filler, Reject, effect-aware CRA, progress-aware CRA) ran together three times with Claude (24,436 requests). Registered before the first episode: H1, fewer calls than effect-aware CRA, and H2, fewer calls than Reject, each confirmed only if the 95% CI lies below zero; success differences reported two-sided; a stopping rule fixed in advance.
  • The result. Progress-aware CRA uses fewer model calls than both effect-aware CRA and Reject, with higher success than Reject and no detectable difference from effect-aware CRA. Relative to effect-aware CRA, calls change by −0.5 per episode (CI [−0.8, −0.1]) and success by 0.5 points (CI [−1.4, 2.6]); relative to Reject, by −0.7 (CI [−1.1, −0.3]) and 3.1 points (CI [0.5, 5.7]). As registered, both H1 and H2 are confirmed. Relative to the baselines (secondary), it uses fewer model calls than both the unguarded filler and native AutoTool, with higher success than the unguarded filler and no detectable difference from native AutoTool. On these games the frontier holds progress-aware CRA on calls and on tokens.
  • Replication with a second model. The Gemini model of the paper was not available during the discussion, so we repeated the registered test with GPT-5.6 (OpenAI), registered before its first episode with the same games, arms, code, hypotheses and stopping rule (26,310 requests). Progress-aware CRA uses about as many model calls as either effect-aware CRA or Reject, with no detectable difference in success from either; relative to effect-aware CRA and Reject, calls change by −0.1 and −0.1 per episode (CIs [−0.5, 0.3] and [−0.5, 0.4]) and success by 0.0 and −1.0 points. As registered, neither H1 nor H2 is confirmed. On these games the frontier holds native AutoTool, Reject and progress-aware CRA on calls and on tokens. Per episode the rule moved model-free 1.10 times, and the model was asked instead of repeating an action 1.37 times against 1.31 in the effect-aware arm (with Claude: 0.74 moves, 0.98 against 1.34 asks).
  • Replication with a third model. We repeated the registered test once more with Grok 4.7 (xAI), registered before its first episode with the same games, arms, code, hypotheses and stopping rule (28,048 requests). Progress-aware CRA uses about as many model calls as either effect-aware CRA or Reject, with no detectable difference in success from either; relative to effect-aware CRA and Reject, calls change by −0.4 and −0.4 per episode (CIs [−0.9, 0.1] and [−0.9, 0.1]) and success by −0.2 and 3.6 points. As registered, neither H1 nor H2 is confirmed. On these games the frontier holds native AutoTool, effect-aware CRA and progress-aware CRA on calls and on tokens.
  • A larger Grok 4.7 test, stopped. Since both Grok intervals reached zero, we registered, before it ran, a second Grok 4.7 test on 600 new games, sized to decide. The endpoint's usage balance for the model ran out after 1,288 of 5,400 episodes, before any repetition was complete; as registered, it is not analyzed (Table 8).

So the rule saves calls relative to effect-aware CRA and Reject without a detectable loss of success, on games that played no part in building it, with Claude; registered replications with GPT-5.6 and Grok 4.7 confirm neither call reduction. Its limits: the rule is specific to ALFWorld navigation, written by hand like the catalogs, and tested with Claude, GPT-5.6 and Grok 4.7 only; the paper states this among its limitations. The paper adds the rule to Section 3.3, the result to Section 5.4 and Figure 4f, and the details to Appendix D (Tables 19, 20 and 21).

AuthorsOpenReview ↗

General response to all reviewers (2/3)

4. New experiments. To answer TeFe's questions with more than single runs, we registered a batch before running it (arms, games and tasks, estimators, primary comparisons). We ran every arm three times, all arms together against the same endpoint.

  • E1: ALFWorld, 134 games with each of Gemini 3.8 Flash and Claude Sonnet 4.6, five arms: native AutoTool, the unguarded filler, Reject, CRA, and effect-aware CRA (it asks the model instead of repeating an accepted action). This also answers the concern that effect-aware CRA ran a day after the other arms.
  • E2: ScienceWorld, the 35 tasks on which Reject and CRA wrap AutoTool's own filler.
  • E3: registered ScienceWorld development reruns and branch tests with Gemini; they stopped incomplete when the endpoint stopped serving Gemini (Appendix D; answer to xUmH, Q3).
  • Confirmation rerun: after we saw the Claude comparisons with the two baselines, which we had not registered, we registered a rerun before running it: Claude, the four arms other than CRA, three new repetitions, 21,214 requests (Appendix D, Table 18).
  • E4: a progress-aware CRA, built from the diagnosis during the discussion and registered before it ran on 140 ALFWorld games that no earlier run used (part 3).
  • E5: a registered replication of E4 with GPT-5.6 (OpenAI) on the same games, because Gemini was no longer served (part 3).
  • E6: a second registered replication of E4, with Grok 4.7 (xAI), on the same games (part 3).
ALFWorld (success %, calls per episode)native AutoToolunguardedRejectCRAeffect-aware CRA
Gemini98.8, 8.296.8, 8.698.0, 8.694.0, 9.298.3, 8.5
Claude85.8, 14.179.9, 12.484.1, 13.177.6, 13.084.6, 12.9

We report the results whatever their direction:

  • CRA lowers success relative to Reject by 4.0 points with Gemini (CI [−7.5, −1.0]) and 6.5 with Claude (CI [−11.2, −2.2]).
  • The registered effect-aware rule raises success over CRA by 4.2 and 7.0 points: it recovers the loss and ends level with Reject, and effect-aware CRA is undominated only with Claude, where it and Reject do not differ detectably in success or calls.
  • On AutoTool's own filler, the earlier single run had CRA ahead of Reject by 17.1 points; the repeated runs do not confirm this (−5.7 points, CI [−14.3, 1.0]).
  • Not registered in the batch: the comparisons with the two baselines. With Claude, Reject and effect-aware CRA improve on native AutoTool in calls at no detectable loss of success, again in a registered rerun, and on unguarded reuse in success at the price of more calls, which the rerun does not confirm; with Gemini, they improve on neither. In the batch, with Claude, their success differences from the unguarded filler are 4.2 and 4.7 points (CIs [0.5, 8.2] and [0.2, 9.2]) and their call differences from native AutoTool −1.0 and −1.2 per episode (CIs [−1.5, −0.5] and [−1.8, −0.6]). The registered rerun confirms neither success gain: on its new repetitions the success differences are 3.5 and 3.7 points (CIs [−0.5, 7.7] and [−0.2, 8.0]). In its secondary comparisons with native AutoTool, calls change by −1.1 and −1.1 per episode (CIs [−1.6, −0.7] and [−1.6, −0.6]) and success by −1.0 and −0.7 points (CIs [−3.7, 2.0] and [−3.7, 2.0]). Pooled over all six Claude repetitions, including the three that suggested the hypotheses, the success differences from the unguarded filler are 3.9 and 4.2 points (CIs [1.0, 6.8] and [1.0, 7.6]). The submission's statement that no gate detectably improves success or calls over unguarded reuse or native AutoTool therefore no longer holds with Claude, and the revision drops it.

5. Cost–success frontier. New Figure 4 plots model calls per episode against success for every closed-loop arm, with intervals on both axes, in six panels (the two earlier ScienceWorld runs, the three new ones, and E4). Figure 5 does the same with tokens. On calls, the undominated arms are: ScienceWorld development, native AutoTool; held-out, native AutoTool and the unguarded filler; AutoTool's filler, native AutoTool; ALFWorld with Gemini, native AutoTool; with Claude, native AutoTool, the unguarded filler and effect-aware CRA; E4, progress-aware CRA. On calls and on tokens, CRA is dominated on every cost–success frontier. Appendix D also charges each refused action a price in model calls (see our answers to xUmH and TeFe). Table 12 gives means, paired differences, tokens and McNemar tests.

6. How guards are built. New Section 3.3 describes how we build and refine guards (predicate library; instantiation by an expert, by mining, or by one LLM call; refusal-driven refinement; coverage), with the evidence for each step.

7. Figure 1a and readability. Figure 1a is fixed. Sections 5.3 and 5.4 carry fewer numbers in the running text; secondary statistics moved to Appendices C and D, and no conclusion changed. Section 7 now says plainly what certification buys and where it does not pay.

AuthorsOpenReview ↗

General response to all reviewers (1/3)

We thank all reviewers for their constructive comments. All three reviewers raise the same main concern: the paper presents certify–repair–ask (CRA) as its method, but in the submitted closed-loop results CRA brings no detectable gain over unguarded reuse, native AutoTool, or a simple reject gate. We agree, and the new runs confirm it: CRA is dominated on every cost–success frontier. So we changed what the paper claims, not how the results look, and we added the experiments the reviewers asked for. The revised PDF is uploaded, and our comment "Summary of changes in the revised PDF" lists every change with its location.

The diagnosis also points to a fix. During the discussion we built a gate with a progress signal from it, registered its test before running it, and ran it on 140 ALFWorld games that no earlier run used. With Claude, it uses fewer model calls than every other arm at no detectable loss of success; not with GPT-5.6 or Grok 4.7 (part 3).

1. What the paper claims now. The paper is now a diagnostic study of gates on model-free actions. It asks three questions of any gate and answers each with its own protocol:

  • Is the gated action executable? Counterfactual execution from restored simulator states, with no new model call.
  • Is it the right action? Episodes branched from the same state, which separates the gate from the policy it changes.
  • What does the gate do to whole episodes? Repeated closed-loop runs, read on a cost–success frontier.

CRA is the gate under study. It is the strongest executability check we could build, and it provably acts model-free wherever rejection does (Proposition 1). We rewrote the abstract, introduction, contributions and conclusion accordingly. ICLR does not allow title changes during the discussion; if the paper is accepted, we plan to change the title to "Skip the Model, Not the Checks? Diagnosing Gates on Model-Free Actions of LLM Agents".

2. What the study shows. Three findings:

  1. Certification, not repair, removes refused actions. With an expert catalog, CRA lowers the refusal rate of model-free actions from 80.2% and 93.7% to 28.5% and 26.6% (ScienceWorld, ALFWorld). Repair without certificates is refused 36.8 points more often than CRA, and a gate that only checks almost never acts.
  2. An executable action is often the wrong one, for a measurable reason. Catalogued preconditions still hold after an action has taken effect, so repair replays the agent's recent past: 35.4% of CRA's accepted ScienceWorld actions redo a state change already made, against 3.9% of the model's. Branched from the same states, these repairs cost ALFWorld agents steps and calls. The first two protocols show this before any closed-loop run.
  3. In closed loop, repair costs success; refusing and asking do not. In ALFWorld, repair lowers success relative to Reject with both models. With Claude, Reject and effect-aware CRA improve on native AutoTool in calls at no detectable loss of success, again in a registered rerun, and on unguarded reuse in success at the price of more calls, which the rerun does not confirm; with Gemini, they improve on neither. A likely reason for the difference between the two models: with Gemini, native AutoTool already succeeds in 98.8% of games and Reject acts model-free only 0.68 times per episode (2.65 with Claude), so there is little success to gain and few calls to save.

3. Practical reading. Refuse rather than repair. Ask the model rather than repeat an action that has already taken effect. Certify where a failed call costs more than a step. A gate needs a progress signal; the one we built during the discussion saved calls on new games with Claude; registered replications with GPT-5.6 and Grok 4.7 confirm neither call reduction (part 3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.