acceptodds
Under review as a conference paper at ICLR 2027

Propose Before You Plan: Inferring Changed Legality Rules from Refusals

Abstract

When a deployed policy must learn which actions are legal from refusals, the usual lever is better probing: choosing informative actions and planning them ahead. On legality rules from third-party codebases, a policy frozen across a rule change trails a specialist retrained on the new rule. Our deployable policy closes 0% of that gap on average. We split the policy into proposing rule hypotheses, updating a belief from refusals and choosing actions, and vary one stage at a time. Planning probes ahead adds nothing measurable over pricing them one at a time, even with exact belief updates. An explicit information term adds nothing significant over exploiting the belief. We test proposal by planting a chosen rule among the hypotheses. Pooled over rule shapes, a near-copy of the deployed rule, wrong in one machine or command class, retains 60% of what the deployed rule itself gains over a wrong rule. The policy's own hypotheses stay further from the deployed rule than that near-copy, which places the bottleneck at proposal. This pooled result holds under a grammar that can express the deployed rules. For such policies, better proposal should come before better planning, and planting gives proposers a measurable target.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, action legality, feasible action sets, distribution shift, active learning, bayesian inference, belief-state planning, hypothesis proposal, in-context reinforcement learning, tool-use agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_17oCOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies a policy that must infer a changed legality rule from accept/refuse feedback alone. It splits a deployable policy into propose, infer, and decide stages, varies one stage at a time, and introduces "planting" a chosen rule among the hypotheses as a diagnostic. It concludes that proposal, not probe planning, is the bottleneck. A benchmark of 38 rule shifts derived from three codebases is released.

Critical strengths
  • The question is focused, and the stage-wise intervention design and planting diagnostic are clean ideas.
  • Success criteria are fixed in advance, and negative results are reported honestly.
Critical weaknesses
  1. The main conclusion follows from the design. The policy draws 24 random hypotheses, and in five of six shapes none of them can express the deployed rule. If the correct rule is not in the hypothesis set, no inference or planning stage can recover it, and planting it must help. Locating the bottleneck at proposal is therefore a consequence of the setup, not an empirical finding.
  2. The planning result is confounded. Planning is compared on a hypothesis set that does not contain the deployed rule, so little gain should be expected. In addition, with H=40 attempts over 36 actions, most actions are tried within an episode anyway, which leaves little room for query selection to matter. The claim that planning adds nothing is not tested under conditions where it could.
  3. The proposer appears under-provisioned. The only change to proposal that helps is drawing more hypotheses, and the gain shows no sign of saturating. This points to too few hypotheses in this particular policy rather than a general finding that proposal is the bottleneck.
Questions for authors

See Weaksnss.

Reviewer_HqANOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The author studies how an agent adapts to changes in hidden legality rules under the setting of having binary feedback on whether the actions are either accepted or refused. In particular, they introduce ACTIONSET-SHIFT which is a benchmark that has 38 rule changes from 3 different codebases and they study why the policies are hard to adapt without training from scratch again. There are a couple of main contributions, which includes the formulation and benchmark and how they propose separating adaptation into proposing rules, updating beliefs from refusals and selecting actions based on the beliefs. They also did some ablation and found the bottleneck of the proposal, where having a good hypothesis significantly improves performance.

Critical strengths

I found the paper well written because the authors didn’t just propose another algorithm, and instead studied why adaptations fail by looking at inference, generation of hypothesis and decision making. The authors are also honest in reporting the negative results and had a very detailed empirical evaluation section studying different decision strategies and rule representations. Empirically, the paper encourages reproducibility and also gives uncertainty intervals and costs, which allowed me to better understand the system’s behavior.

Critical weaknesses

One main weakness I found from the paper is that we cannot use the hypothesis grammar to write the deployed rule in five out of the six evaluated rule shapes. Naturally this means we would expect bad hypothesis generation as a limitation of the system. Another weakness is that the planning comparisons do not consider more general belief space typed of planning or objectives about information gain. This means that the negative results derived from using a specific type of uncertainty based objective do not show that the more general/complex planning method is not very beneficial. It also seems that the main comparisons against the retrained specialists also use a different action set size and runs. I put some of the weaknesses in the questions to be addressed as well.

Questions for authors
  • It’s unclear how sensitive the limited value of planning to the uncertainty-based objective and search procedure is. I would appreciate it if the authors could clarify whether the ranking holds when we evaluate task reward or failed actions instead of just using accuracy?
  • How much does the conclusion depend on the mismatch in representation given that the default grammar cannot represent more of the deployed rules (as aforementioned)? This would be useful in trying to understand whether the proposal is a bottleneck under a more effective proposal mechanism.
Reviewer_K5bPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines how a deployed policy can infer changes in action-legality rules from binary acceptance/refusal feedback. It introduces ACTIONSET-SHIFT, a benchmark of 38 rule shifts drawn from three codebases, and evaluates a Bayesian policy with three stages: proposal, inference, and decision. Policy experiments use two of the rule sources and primarily assess balanced accuracy in predicting action legality on a common scoring trajectory. Shallow lookahead adds little accuracy over one-step selection, while inserting an exact or approximate version of the deployed rule into the hypothesis pool produces substantial gains. The authors take these findings to suggest that improving hypothesis proposal and retention should take priority over more elaborate planning.

Critical strengths
  1. The evaluation design addresses a meaningful problem in adaptive-agent assessment. Policies that choose different actions can also change the states on which they are evaluated. The common scoring trajectory separates evidence collection from evaluation, allowing policies to act independently while being scored on the same states (Section 2.4). This makes the comparison easier to interpret and could be useful beyond this particular task.

  2. The study offers a useful way to investigate where adaptation fails. The three-stage framework allows the authors to vary action selection and candidate hypotheses within a common inference framework. Planting exact, approximate, and incorrect rules measures the value of having useful hypotheses available, while the expressive-grammar control tests whether the findings depend on representational limitations (Section 4.4). These diagnostics give the paper value even though its proposed planner does not meet the stated success criteria.

Critical weaknesses
  1. The headline adaptation result is not based on a matched evaluation. The trained generalist and specialist are evaluated with 120 available actions per episode, whereas the inference policies have 36 (Section 4.1; Appendix B). The reported gap-closure ratio therefore combines scores from different interfaces. Its uncertainty is also substantial, with a mean of 0% and a Student interval of [−74%, +74%] (Appendix F). This comparison does not isolate adaptation performance under a common task setting. It weakens the headline claim, while leaving the paired planning and planting comparisons intact.

  2. The explanation of the proposal bottleneck remains incomplete. Planting establishes that access to a useful, protected hypothesis improves performance, but does not fully explain why the unmodified policy fails to obtain the same benefit. Under the richer grammar, the closest generated hypothesis and the one-swap planted rule each have a reported disagreement rate of 8% on the rail rules; the planted rule remains closer on the permission rules (Appendix I). Protecting an exact hypothesis from replacement also adds 8.1 pp under the richer grammar, pooled over the six shapes (Table 10). These findings leave the roles of generation quality, retention, and posterior weighting unresolved. The authors acknowledge retention failures, so this does not contradict their broader proposal-and-maintenance argument. It does, however, call for a clearer justification of the abstract’s distance-based explanation and of what the planting diagnostic identifies.

Questions for authors
  1. Are matched-interface results available for the trained references and inference policies? If so, how do the gap-closure estimate and its uncertainty change? If not, please clarify what conclusion can be drawn from the current cross-interface comparison.

  2. Under the richer grammar, are the disagreement rates for generated and planted hypotheses computed over the same steps and scoring actions? When do the closest generated hypotheses appear, and do they retain appreciable posterior weight? Within the proposal-and-resampling process, what evidence distinguishes limitations in generating useful hypotheses from limitations in retaining and weighting them?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.