acceptodds
Under review as a conference paper at ICLR 2027

ProPS: Posterior Sampling for Programmatic World Models

Abstract

Programmatic world models represent an environment’s dynamics as code that a language model synthesizes from observed transitions, and recent methods decompose this code into many small “laws”. We study how an agent can plan with such laws. Because each law states only part of what an action does, we compile laws into planning rules that each state an action’s whole effect. However, since a rule's context can stay partial, two uncertainties remain: 1) whether an action can run at all, and 2) which of the rival rules compiled for an action is right. We introduce Program Posterior Sampling (ProPS), a controller that keeps an availability posterior and a rule posterior over these two unknowns, plans under a sample from both, and updates both from each outcome. For a small library of complete programs, we prove that an exact version of ProPS is optimal under Bayes-adaptive dynamic programming, while a general version trades this guarantee for scale. On Crafter-OO, ProPS outperforms all baselines given the same laws, including a product of experts and an LLM agent that reads the laws as text.

Primary area:
reinforcement learning
Keywords:
programmatic world models, world modeling, bayesian rl, planning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Vk9oOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies planning with learned programmatic world models composed of partial laws. ProPS compiles these laws into planning rules, maintains beliefs over competing rules and action availability, and repeatedly samples a model, plans, executes an action, and updates its beliefs. The paper also presents an exact controller for a restricted setting with a small library of complete programs, with a Bayes-optimality result for that decision problem. Experiments include controlled tests of program ambiguity and open-ended control in Crafter-OO. Under a shared rule library and action interface, ProPS achieves more achievements than the evaluated planning-based and LLM-agent baselines.

Critical strengths
  • The paper addresses an important problem: how to plan effectively with learned world models that are partial, uncertain, and potentially stochastic. Predictive accuracy alone does not establish that a learned model is useful for control. The proposed compilation and control pipeline makes concrete progress toward using such models for sequential decision-making.

  • The empirical analysis provides useful evidence about which components help control. The shared planner and action adapter make the comparisons informative about alternative ways of using the same learned knowledge. The ablations and failure analyses are also valuable: the paper examines availability estimation, retention of competing rules, replanning, and joint guard induction, and explicitly discusses limitations of its approximate belief updates.

Critical weaknesses
  • Evaluation is limited to Crafter-OO with one learned source portfolio. The system also relies on hand-designed compilation heuristics, planning restrictions, and a supplied action adapter, leaving unclear how much engineering would be needed to transfer it to other environments. The LLM comparisons use only Gemini 2.5 Flash with low reasoning effort, further limiting the scope of the empirical conclusions.
  • The baselines are adapted versions of prior methods rather than their full implementations. For example, WorldCoder omits its optimism constraint, exploration, and planner, while the open-ended PoE baseline freezes its fitted weights. These comparisons help isolate components but provide limited evidence of improvement over the original methods.
Questions for authors
  • Are there results on more benchmarks with newer LLMs?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Vk9o (Part 1/2)

Dear Reviewer Vk9o,

Thank you for your thoughtful and constructive review of our paper. We are very pleased that you found our compilation and control pipeline to make concrete progress on planning with partial, uncertain world models, and that you found our shared-interface comparisons and empirical analyses informative. We greatly appreciate your positive assessment of our work, and your suggestions on broadening the evaluation and on the baselines. In response to your feedback, we have conducted new experiments, adding (1) a second law portfolio synthesized by a newer LLM (Gemini 3.8 Flash), (2) a preliminary study in a second environment (ALFWorld), and (3) two of the baseline components you highlighted (an optimism rule following WorldCoder's optimism constraint, and product-of-experts weights refit while acting), and uploaded a revised manuscript. We provide detailed answers to each question below.


W1.Q1 & Q1: Evaluation on more benchmarks and with newer LLMs

A: Thank you for this great suggestion. Following your suggestion, we ran two additional sets of experiments.

A second law portfolio from a newer LLM We conducted an additional evaluation of our open-ended Crafter-OO setting using a second portfolio of laws, obtained by running OneLife's exploration and synthesis pipeline with Gemini 3.8 Flash on 88 new source episodes. We found that ProPS achieves the highest score at 8.408.40 achievements (up from 6.906.90 with the original Gemini 2.5 Flash laws), outperforming Best-supported rule + availability (8.068.06, vs. 3.293.29 originally) and Product of experts + availability (7.687.68 over the 41/4841/48 worlds completed within the search limit, vs. 5.885.88 originally). Please find the full comparison in Common response 2.

A second environment (ALFWorld). As a preliminary step beyond Crafter-OO, we evaluated ProPS on one task type, pick-and-place, in ALFWorld (Shridhar et al., 2021), using laws synthesized by Gemini 3.8 Flash. We synthesized laws from 88 source games using OneLife's prompt (with Crafter-OO's state schema replaced by ALFWorld's), compiling them into rules as in Sec. 2.3, and evaluated on 2424 held-out games (6464 decisions per game):

ControllerGames solved (of 2424)
Best-supported rule + availability2424
Product of experts + availability1313 (no plan in 1010)
ProPS (ours)2424

ProPS and Best-supported rule + availability, which both plan with one compiled rule per action, solve all 2424 held-out games with 00 LLM calls during control. Product of experts + availability, whose search scores plans over all competing rules at once, returned no plan within the 600600-second search limit in 1010 games and solves 1313. Overall, this preliminary study shows that the compiled rules support planning in a second environment, and that ProPS runs end to end with laws from a newer LLM.


W1.Q2: Engineering effort needed to transfer to other environments

A: Our preliminary study on ALFWorld allowed us to test this directly. We are able to adapt our method to ALFWorld with only modest engineering: The compile procedure, the rule posterior, the availability posterior, their updates, and the planning limits were the same as in Crafter-OO, and no action adapter was needed. The environment-specific work was in implementing the compile step, and the synthesis prompt adapted for ALFWorld's state schema. More specifically:

  • Action adapter and planning restrictions: ALFWorld's native commands are already high-level, with no grid pathfinding, so each decision issues one native ALFWorld command instantiated with objects from the environment state. We added no planning restriction for ALFWorld: plans have at most twelve steps and use one rule per action, as in Crafter-OO.
  • Compilation: For the compile step, we perform some engineering to adapt to ALFWorld's state schema and extend the guard language to support comparisons between two state fields, which the ALFWorld laws use. Otherwise, laws are merged into whole-effect rules the same as in Sec. 2.3.

For law synthesis, we use OneLife's synthesis prompt with ALFWorld's state schema in place of Crafter-OO's, and an exploration instruction that asks the explorer to complete the task and to open and close a receptacle. Every controller observes the simulator's full symbolic state, represented as a set of fluents as in WorldCoder (Tang et al., 2024). Full details and prompts are provided in Appendix G.7.

Reference: Shridhar et al., ALFWorld: Aligning Text and Embodied Environments for Interactive Learning, ICLR 2021.

AuthorsOpenReview ↗

Response to Reviewer Vk9o (Part 2/2)

W1.Q3: LLM comparisons with Gemini 2.5 Flash at low reasoning effort

A: The ReAct agents use Gemini 2.5 Flash at temperature 00 with low reasoning effort (App. D.4). We chose the same LLM that synthesized the original laws so that the comparison holds the underlying model fixed and varies how the laws are used. Given the 161161 law records as text, ReAct + law memory makes 3,0643{,}064 LLM calls across the 4848 worlds and reaches 3.983.98 achievements (App. F), whereas ProPS compiles the same laws into rules and reaches 6.906.90 with 00 LLM calls during control. While we scoped the ReAct baselines to this setting, we have now also evaluated the planning controllers with laws synthesized by a newer model, Gemini 3.8 Flash (W1.Q1 above), where ProPS consistently achieves the best score.


W2: Fidelity of the baselines to prior methods

A: Thank you for raising this important point. Based on your feedback, we implemented and evaluated two further baseline variants, and added them to the revised Tables 1 and 2:

  • Optimism constraint (Table 1): We added Optimistic commitment, which applies WorldCoder's optimism principle to the three candidate programs, committing to the first program under which a plan to defeat the creature fits within the remaining budget, and switching whenever an observed outcome contradicts that program. It succeeds on 48.6% of instances, the same instances as Retain all, never test, against 83.3% for ProPS.
  • Refit weights (Table 2): We added Product of experts, weights refit, which refits the rule weights after every fired action on the source transitions and its own transitions, using the fitting settings of PoE-World (its Appendix A.1). It obtains 6.06±0.286.06 \pm 0.28 achievements with our availability posterior (5.88±0.255.88 \pm 0.25 with fixed weights) and 1.06±0.041.06 \pm 0.04 without it, against 6.90±0.276.90 \pm 0.27 for ProPS.

As you noted, sharing the planner and the action adapter makes the comparisons informative about alternative ways of using the same learned knowledge. Within this shared setting, both additions test each method's update mechanism without calling an LLM while acting. The optimism rule selects among our three supplied programs, and the refit keeps our rule set fixed. While the full implementations of WorldCoder and PoE-World involve additional LLM calls during interaction to rewrite the program (WorldCoder) or synthesize new experts (PoE-World), neither was developed for Crafter-OO: WorldCoder's monolithic synthesis loop, which refines a single program for the entire transition function, is difficult to scale to complex environments (Piriyakulkij et al., 2025), and PoE-World's Atari-specific law synthesis is incompatible with Crafter's state (Khan et al., 2026, Appendix I). Our implementation protocol therefore follows OneLife in adapting both methods to Crafter-OO, to present a fair comparison. We provide the full details of each baseline in Appendix D. Overall, when compared with optimistic commitment and weight refitting, ProPS achieves the best score in both settings.

Thank you again for your detailed suggestions on the evaluation and the baselines, which made the empirical side of our paper stronger.

AuthorsOpenReview ↗

Response to Reviewer Vk9o

Dear Reviewer Vk9o,

We sincerely appreciate your detailed suggestions regarding our evaluation and baselines. Through our additional experiments with Gemini 3.8 Flash, additional comparisons with WorldCoder and PoE-World, and preliminary study on ALFWorld, we hope that we have addressed your concerns.

As the discussion period is nearing its end, we would greatly appreciate it if you could let us know whether any concerns remain. If you have any further comments or questions, please do not hesitate to let us know.

Thank you again for your positive assessment of our method and empirical results, and your valuable feedback.

Reviewer_y4GmOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper addresses a problem in learning symbolic (i.e. programmatic) world models, which is, once you have such a world model, how do you plan to achieve goals with that world model? They investigate a representation used by recent papers on symbolic world modeling, which is a mixture of programmatic experts / laws. They point out that these laws cannot actually be used directly for planning to achieve goals (i.e. as in STRIPS-based planning). They propose a pipeline which compiles these laws / experts into rules which can be fed to a planner and then proposes a posterior sampling procedure that can be used to learn which of these rules are actually correct. The method is validated on the Crafter-OO benchmark, and seems to produce good results.

Critical strengths

Symbolic world modeling is one of my favorite areas, and I think this paper addresses a real scientific gap in symbolic world modeling right now: how do we get representations that are useful for planning, and then how do we learn which of our symbolic rules that can be used for planning are right or wrong? I think this is consequential because without this, it's hard to use symbolic world models for goal-based planning except in places where the environment dynamics are simple enough that forward planning works. I also think the technique used by this paper is pretty clever (work backwards from the desired change to figure out whether we can compile more compositional rules out of the atomic laws).

I also like that the method is conceptually simple. This is important to me because I feel that usually methods that have too much complexity are hiding things, but I think that the method used by this paper, which is effectively Thompson sampling, is quite intuitive and makes sense. There's also quite a bit of math to back up the paper. However, I cannot say I've checked all the math, although a skim over the math does make sense to me.

Critical weaknesses

Despite having worked in this area, I found the paper very difficult to understand because it feels entirely LLM-written with little work done to make it easier for readers to understand. I do appreciate that the figures seem human-designed. However, it was very difficult for me to figure out what the details of the method are and what problem each of the components is solving. Another big issue is that the experiment section seems to be full of Claude-like statements that don't really explain themselves. For example, consider the phrase on line 353: "A posteriori over programs resolves what one fitted program cannot." This phrase seems like it came out of a movie script, and does not belong in a scientific paper. This is just one example. There are many throughout. Another issue is that, at no point (at least, in my understanding), does the paper clearly state the research question it is solving, why that research question matters, etc. I'm only able to make these arguments for myself because I sort of understand what the problems are in this area.

Questions for authors

I think that this paper has a great technical contribution and should be accepted, provided that the authors significantly rewrite it for clarity and intuition. It is fine if the authors use an LLM to assist with the rewrite (I am not opposed to this), but the rewrite should be guided by the authors with a focus on clarity and human understanding. I strongly urge the authors to follow standard good practices, such as making sure that:

  • the research question is clearly stated up front
  • the importance of solving the research statement is explained
  • in the experiment section, each takeaway is written in clear and simple language

I wish I could give more detailed feedback here, but as it is, I barely understood the experiment section. I was only able to understand what is happening because of a familiarity with Crafter as well as some of the baselines used in the paper. Essentially, the experiment section seems to be devoid of meaningful semantic content and almost looks like a commentary by Claude on internal experiments that are not designed to be read by outside reviewers.

In short, I really like this paper, but the writing is absolutely unacceptable and is holding back the technical content. It should not be difficult to at least write the main takeaways, the research question, and the table/figure captions with a human touch to ensure that the skeleton of the paper is understandable to humans. If the authors do this rewrite during the ICLR rebuttal period, I will raise my score to a clear accept.

4 comments
AuthorsOpenReview ↗

Response to Reviewer y4Gm (Part 1/2)

Dear Reviewer y4Gm,

Thank you for your insightful and detailed review of our paper. We are especially glad to hear that you found our paper's focus on symbolic representation for planning to address a current gap in symbolic world modeling, and that you found our method conceptually intuitive. We greatly appreciate your positive assessment of our paper's technical contributions, and your constructive comments on how to improve the presentation of the paper. During the rebuttal period, we have made significant efforts to ensure that we fully reflect your feedback through a revised manuscript, and we hope that the revision provides a presentation of ProPS with enhanced clarity. Below, we summarize the revisions, and provide detailed answers to your questions.


W1.Q1: Clear statement of the research question and its importance

A: Thank you for your valuable feedback. Following your suggestion, we have revised the Introduction (L46–L82) and the opening of the Method section (L133–L138) to improve the presentation of our research question—given a portfolio of synthesized partial laws, how can an agent construct plans and act to achieve goals?—and why we believe it is important. In the revised Introduction, we highlight the gap in the existing symbolic world modeling literature in connecting learned models to goal-directed planning, and discuss the classical STRIPS planning representation as a concrete framework to show why partial laws (which leave effects fragmented and preconditions open) cannot be chained directly for multi-step planning. We believe the revised sections significantly strengthen the motivation and clarity of the paper, and we thank you again for your constructive feedback.


W1.Q2: Details of the method and the problem each component solves

A: In the revision, we have clarified the alignment of each component of ProPS with the challenge it resolves through the following additions. Section 2.1 first summarizes the three challenges of planning with synthesized laws (L202–L211): incomplete effects, context unavailability, and context ambiguity. Then, we provide a framework overview (L212–L230) that links each component to its challenge: the offline compiler to incomplete effects, the availability posterior to context unavailability, and the rule posterior to context ambiguity. Finally, we updated Section 2.3 to detail Steps (1)–(4) under the same numbering.


W1.Q3: Clear and simple takeaways in the experiment section

A: Following your suggestion, we have reorganized and rewritten Section 3 around five explicit research questions in Crafter-OO (RQ1 in Sec. 3.1, and RQ2–RQ5 in Sec. 3.2), each answered with a direct takeaway:

  • RQ1 (Sec. 3.1, controlled setting): When several candidate programs fit the offline data equally, keeping all of them and testing them before committing is essential. With three programs that disagree on which sword defeats the target creature, committing to one fitted model (WorldCoder fit, PoE-World / OneLife fit) succeeds in at most 29.9% of the 144144 instances, retaining all programs without testing in 48.6%, and the exact version of ProPS, which tests before committing, in 83.3%, close to the 85.4% of an oracle given the true program.
  • RQ2 (Sec. 3.2, open-ended setting): Compiling partial laws into plannable rules and updating the rule and availability posteriors online enables multi-step planning toward goals. Across 4848 held-out worlds, ProPS unlocks 6.906.90 distinct achievements on average, the highest score among all planning-based and LLM-agent baselines, with 00 LLM calls during control.
  • RQ3 (Sec. 3.2, rule posterior): Maintaining a posterior over competing rules outperforms both committing to a single rule and combining all rules with fixed weights. Replacing the rule posterior with the best-supported rule lowers ProPS from 6.906.90 to 3.293.29 achievements, and replacing it with a product of experts, which keeps its weights fixed, lowers it to 5.885.88.
  • RQ4 (Sec. 3.2, availability posterior): Tracking which actions fail to run keeps the planner from repeating unreachable actions. Removing the availability posterior lowers ProPS from 6.906.90 to 5.965.96 achievements, and adding it raises the product of experts from 1.061.06 to 5.885.88.
  • RQ5 (Sec. 3.2, a second set of laws): ProPS remains the highest-scoring controller when a newer LLM writes the laws. With laws synthesized by Gemini 3.8 Flash from eight new source episodes and evaluated on the same 4848 worlds, it reaches 8.408.40 achievements (Please refer to Common response 2 for the full comparison).

In addition, we rewrote the body paragraphs of Section 3 to remove opaque phrasing (including the L353 heading you noted). We believe these changes made Section 3 easier to follow, and we thank you again for your close reading.

AuthorsOpenReview ↗

Response to Reviewer y4Gm (Part 2/2)

Q1: Scope of the rewrite, including table and figure captions

A: We recap the points addressed through the revised manuscript: The research question and its importance (W1.Q1), the method and the problem each component solves (W1.Q2), and the takeaways of the experiments (W1.Q3). We also revised the captions of Figures 1 and 2 and clarified the caption of Table 2.

We hope that the revised manuscript addresses your concerns, and we thank you again for your encouraging and detailed feedback.

Thanks, authors. I have raised my score to a 3. The writing of the paper has improved substantially. However, it is still very difficult to understand, and I find myself having to read the takeaways multiple times to understand what they are even talking about. I'm only able to understand because of a deep familiarity with the environments / game being discussed. This is genuinely an excellent technical contribution, and I highly encourage the authors to please rewrite the paper in a more reasonable form and resubmit if rejected. I really like this work. I'm just not sure why it was written in such a manner.

AuthorsOpenReview ↗

Response to Reviewer y4Gm

Dear Reviewer y4Gm,

We sincerely appreciate your acknowledgement of the improved draft with a raised score. Thank you for your valuable suggestions, which helped us sharpen the framing of the research question and its importance, and strengthen the presentation of the experimental results and takeaways.

We are once again grateful for your positive assessment of the technical contribution of our work, and we are fully committed to reflecting further clarity refinements in the next version of the manuscript.

Reviewer_4arpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies programmatic world models, which represent the world's dynamics as sets of rules. The rules, basing on OneLife (Khan et al., 2026), are pairs of preconditions and effects, and are proposed by an LLM from a few episodes of interaction with the world. The paper proposes maintaining two probability distributions over these rules -- whether the rule is available in the current context of the world, and whether the rule is the correct one describing the action. Experimental results on the Crafter-OO domain demonstrate this method has evidence of performing better than prior work.

Critical strengths

S1. Experimental results seem promising -- if the method is robust enough, it shows evidence to perform much better than prior work. Compared to LLM agents, it seems to be using significantly fewer LLM calls (based on the method). I encourage the authors to specify the comparison of LLM calls in the paper.

Critical weaknesses

W1. Clarity of setup & formal definition. There are many details missing. E.g., my attempt at inferring the setup from figure 1 and multiple careful reads of the setup: "We query the LLM with a prompt (see appendix for the prompt) and demonstrations (see appendix for the format of the demonstrations) to get output LL -- a set of laws li∈Ll_i \in L (or hh?) in the form of li=(...)l_i=(...). The compiler makes all possible law combinations up a maxumum length of 7(?) into a set of rules RR (or Hf\mathcal{H}_f?), where each rule ri∈Rr_i \in R (or κ\kappa?) is of the form ri=(...)r_i = (...). Our method defines two posteriors -- availability posterior and rule posterior -- over every element rir_i -- piap^\text{a}_i and pirp^\text{r}_i respectively. The goal is to ... select actions that maximize achievements / success which is defined as ..." Etc. The above is likely not a true setup of your method, but my best guess at what it is. In addition, the caption for figure 1 is confusing -- it talks about a baseline, but up until this point no baselines have been defined. In the current writeup there are many things missing: How is an adapter defined? What is the prompt to the LLM to get the laws? What does the compiler do? Are the preconditions chc_h simply a predicate? What is the write scope ShS_h? Is it a subset of the variables in the state? In eq. 1, what is the domain of zz, β\beta, mm, Q\mathcal{Q}? I suppose the domain of V\mathcal{V} is R\mathbb{R}? What is the domain of gg, O\mathcal{O} What are ff, Hf\mathcal{H}_f, F\mathcal{F}? What are their domains? In eq. 2 is the outcome class cc the same as the precondition cc? I could not appropriately follow section 2 due to these issues.

W2. Unclear experimental details. The experiments cover a sufficient variety of baselines, although the current writeup is missing some explanations and details. In section 3.1 what is random tier? Why does the oracle only solve 85.4% of the worlds? Are some of the worlds unsolvable? Perhaps then it makes more sense to discard those worlds, or normalize the score w.r.t. the oracle? It is unclear whether this method is robust. Does it depend on the randomness of the LLM writing the laws? In section 3.2, what is the standard deviation over? In general, is it possible to have multiple replications / random seeds to test the robustness of this method? Or perhaps test on different environments to see if the success translates?

Questions for authors
  1. Could you present a concise framework and setup for the method (see W1 above)? Personally my capability to understand the paper and contributions relies on a clear setup and problem statement.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 4arp (Part 1/4)

Dear Reviewer 4arp,

Thank you very much for your careful review of our paper. We greatly appreciate your time, effort and constructive feedback, and we are pleased to hear that you found our method's results promising, while leveraging significantly fewer LLM calls than LLM agents. We are especially grateful for your precise questions on the problem setup and formal definitions. To address your comments, we have uploaded a revised manuscript with a rewritten Introduction, Section 2 and Section 3, and conducted new experiments, adding a second set of laws synthesized by a newer LLM (Gemini 3.8 Flash) and a preliminary study in a second environment (ALFWorld). We hope that the revision provides a clear and precise presentation of our method, and we answer each question in more detail below.


W1 & Q1: Formal setup and a concise framework for the method

A: Thank you for your close reading of our paper, and your constructive suggestions. Based on your suggestions, we have revised Section 2 to open with a clear problem setup (L140–L196), and present a concise framework in Section 2.1 (L212–L230). We describe brief version of the framework overview below:

ProPS consists of an offline compiler that turns partial laws into whole-effect planning rules, followed by an online control loop that resolves context ambiguity and unavailability via posterior sampling:

  • (1) Compile: In the first offline step, the compiler merges the laws that jointly reproduce an observed state change into one whole-effect rule. Because laws can omit preconditions, it induces them from each source episode's successes and failures, which yields a set Hf\mathcal H_f of competing rules for the same action (a factor ff).
  • (2) Maintain two posteriors: Next, online during agent control, ProPS maintains a rule posterior bf∈Δ(Hf)b_f \in \Delta(\mathcal H_f) over the competing compiled rules of each factor f∈Ff \in \mathcal F to address context ambiguity, and an availability posterior over the firing probability ρo∈[0,1]\rho_o \in [0,1] of each action o∈Oo \in \mathcal O to address context unavailability.
  • (3) Sample, plan, and act: At each control decision step, ProPS samples one rule per factor and one firing probability per action, chains the sampled rules backward from each unearned achievement, and executes the first action of the highest-scoring plan.
  • (4) Update: From the result of executing the action, ProPS updates both posteriors, adjusting the action's availability based on whether it fired and shifting the rule posterior away from contradicted rules toward consistent rivals.

W1.C1: Caption of Figure 1

A: We appreciate your bringing this to our attention. The caption of Figure 1 now names the two controllers shown in the figure (a product-of-experts controller and a best-supported-rule controller), and no longer refers to a baseline before one is defined.


W1.Q1: How is an adapter defined?

A: The adapter is a fixed, hand-written routine shared by every controller. Because the laws we compile model interactions, crafting mechanics, and placements once the agent already faces a target, we use the adapter to factor out local pathfinding, so that every controller plans at the task level. An action o∈Oo \in \mathcal O is one of six kinds: explore, interact with a visible material or entity, craft a named item, place a named item, sleep, or wait. For an interaction, the adapter computes a shortest path over the visible walkable tiles in the 9×99 \times 9 view to a tile next to the target, faces it, and issues the interaction, calling Crafter's step function once per primitive action, one to nine per interaction (explore, craft, and place each take one primitive action). It returns a receipt yt=(xt,xt+1,rt)y_t = (x_t, x_{t+1}, r_t), where rt∈{0,1}r_t \in \lbrace 0,1 \rbrace records whether the adapter issued the intended interaction, craft, or placement, whether or not the state then changed. Failed navigation or targeting reports rt=0r_t = 0: the intended action is never issued, so no rule's weight changes (Sec. 2, Problem setup; Appendix B.4).


W1.Q2: What is the prompt to the LLM to get the laws?

A: We use OneLife's released synthesis prompt without modification. Each prompt shows a language model one recorded transition (x,a,x′)(x, a, x') from an exploration episode (the state xx as JSON, a text map of the surroundings, the primitive action aa, a diff from xx to x′x', and one aspect of the state to model, such as the player's inventory, with its changes listed) and asks for Python law classes (small programs with a precondition method and an effect method), one per distinct change. We reproduce the prompt in Appendix E.

AuthorsOpenReview ↗

Response to Reviewer 4arp (Part 2/4)

W1.Q3: Are the preconditions simply a predicate?

A: Yes. In a law ℓ=(φℓ,eℓ,Sℓ)\ell = (\varphi_\ell, e_\ell, S_\ell) (previously written h=(ch,fh,Sh)h = (c_h, f_h, S_h), renamed so that hh no longer denotes a law and cc denotes only an outcome), the precondition φℓ:X×Aprim→{0,1}\varphi_\ell : \mathcal X \times \mathcal A_{\text{prim}} \to \lbrace 0, 1 \rbrace is a boolean predicate on the observation and the primitive action. In Python, it is a law's precondition(current_state, action) -> bool method (e.g., return action == "make_wood_sword" and current_state.player.inventory.wood >= 1).


W1.Q4: What is the write scope ShS_h? Is it a subset of the variables in the state?

A: Yes. The agent observes xt∈Xx_t \in \mathcal X, an assignment of values to a fixed set VV of named state variables, such as inventory counts, the faced tile, the objects in view, vitals, and achievements. A law's write scope Sℓ⊆VS_\ell \subseteq V is the subset of variables its effect method writes to. In Fig. 2a, CraftWoodSwordConsumeWood writes inventory.wood, so Sℓ={inventory.wood}S_\ell = \lbrace \texttt{inventory.wood} \rbrace. This is why we call a law partial, as it predicts nothing outside SℓS_\ell, and nothing where φℓ\varphi_\ell is false.


W1.Q5: What does the compiler do?

A: The compiler merges partial laws, each of which states only one piece of what an action does, into whole-effect rules that a planner can chain, without enumerating law combinations (Sec. 2.3, step (1); Fig. 2a; Algorithm 2). In each source episode, when an action oo produces a state change Δ\Delta (in Fig. 2a, \Delta = \lbrace \texttt{wood} - 1, \texttt{wood_sword} + 1 \rbrace, where one law predicts only wood−1\texttt{wood} - 1 and another only \texttt{wood_sword} + 1), the compiler replays that episode's laws for oo and greedily adds the law that predicts the most still-unexplained variables in Δ\Delta, until all of Δ\Delta is accounted for. If any inventory or target variable in Δ\Delta goes unpredicted, no rule is emitted.

The selected laws then become one rule, whose outcome is the full change Δ\Delta. Its guard gg (its precondition) is the conjunction of the selected laws' preconditions, enough inventory for what Δ\Delta uses up (at least one wood if Δ\Delta removes one), and conditions induced by comparing that episode's successes against its failures. These are additional conditions, each a material next to the agent or a tool it holds, that held at every success and whose absence explains the most failures. An episode with no failed crafts therefore adds nothing (rule A in Fig. 2b), while one whose crafts failed away from a table and a tree adds both (rule C). Since one episode holds only a few attempts of an action, an induced guard can be wrong, so each one is kept as a rival rule for the rule posterior to weigh during control.


W1.Q6: In eq. 1, what is the domain of zz, β\beta, mm, Q\mathcal Q? I suppose the domain of V\mathcal V is R\mathbb R?

A: The domain of each symbol in Eq. 1 is as follows:

  • z∈Zz \in \mathcal Z is the physical task state: the world state plus the highest sword tier crafted so far.
  • β∈Δ(H)\beta \in \Delta(\mathcal H) is the posterior, a probability distribution over the finite set H\mathcal H of complete candidate programs.
  • m∈{0,…,B}m \in \lbrace 0, \dots, B \rbrace is the remaining budget in primitive actions.
  • Qcommit\mathcal Q_{\text{commit}} and Qa\mathcal Q_a each map (z,β,m)(z, \beta, m) to a pair (p,−e)∈[0,1]×R≤0(p,-e) \in [0,1]\times\mathbb R_{\le 0} of completion probability and negated expected number of actions. Qcommit\mathcal Q_{\text{commit}} is the value of committing to the best plan now, and Qa\mathcal Q_a is the value of first running diagnostic a∈Adiag(z,m)a \in \mathcal A_{\text{diag}}(z, m).
  • V\mathcal V maps (z,β,m)(z, \beta, m) to the same kind of pair, so it is not real-valued. It is the lexmax⁡\operatorname{lexmax} of the Q\mathcal Q pairs, which prefers the higher probability, then the fewer actions.

W1.Q7: What is the domain of gg, O\mathcal O?

A: In the revised draft, we write a rule as κ=(o,g,Cκ∪{∅},Sκ)\kappa = (o, g, \mathcal C_\kappa \cup \lbrace \emptyset \rbrace, S_\kappa):

  • g:X→{0,1}g : \mathcal X \to \lbrace 0,1 \rbrace is a boolean predicate on the observation. It is the rule's guard and indicates when the rule applies.
  • Cκ\mathcal C_\kappa (previously written O\mathcal O) is a finite set of outcome clauses. Each clause is a specific change to the variables in the write scope Sκ⊆VS_\kappa \subseteq V, and ∅\emptyset stands for every outcome that matches no clause, including the case where nothing changed.
  • O\mathcal O is now only the finite set of high-level actions, and o∈Oo \in \mathcal O is the rule's action.
AuthorsOpenReview ↗

Response to Reviewer 4arp (Part 3/4)

W1.Q8: What are ff, Hf\mathcal H_f, and F\mathcal F? What are their domains?

A:

  • F\mathcal F is the finite set of factors, ∣F∣=16\lvert \mathcal F \rvert = 16 in our experiments. A factor f∈Ff \in \mathcal F is one high-level action together with its rival rules, such as crafting a wood sword with rules A, B, and C in Fig. 2b.
  • Hf\mathcal H_f is the finite set of those rival rules. The 1616 factors hold 4747 rules in total. We write h∈Hfh \in \mathcal H_f for a rule and κh\kappa_h for its tuple.
  • The rule posterior keeps one independent distribution bf∈Δ(Hf)b_f \in \Delta(\mathcal H_f) per factor, so the belief over all rules is a product over factors.

W1.Q9: In eq. 2 is the outcome class cc the same as the precondition chc_h?

A: Thank you for bringing this to our attention. No, they are different objects. In Eq. 2, cc is an outcome of a rule. It ranges over the rule's outcome clauses and the residual, c∈Cκ∪{∅}c \in \mathcal C_\kappa \cup \lbrace \emptyset \rbrace. nκ(c)n_\kappa(c) counts how often outcome cc was observed when the rule's action fired and its guard held, and q(κ,c)q(\kappa, c) is the probability the rule then assigns to cc. The precondition belongs to a law. The revised manuscript writes the precondition as φℓ\varphi_\ell, so cc denotes only an outcome.


W2.Q1: In section 3.1 what is random tier?

A: Random tier is a baseline that picks one of the three sword tiers (wood, stone, or iron) uniformly at random, crafts that sword, and attacks the designated creature, with no test beforehand. Only the true tier defeats the creature, so it is right one time in three and then succeeds as often as the Oracle: 85.4% / 3 = 28.5% of instances (Sec. 3.1).


W2.Q2: Why does the oracle only solve 85.4% of the worlds? Are some of the worlds unsolvable?

A: We give the Oracle the correct sword tier, so it needs no test. But it still needs to craft the sword, reach the creature, and strike it multiple times while the creature fights back. It fails to complete this on 2121 of the 144144 instances (14.6%), due to dying, running out of actions, or finding no path. Of these 2121, ProPS solves 1414, while the other 77 (4.9% of all instances) are solved by no controller and may be unsolvable.

Discarding these 7 would scale every controller’s success rate by the same factor, so we report success over all 144144 instances. For reference, relative to the Oracle’s success rate of 85.4%, ProPS reaches 97.6% of Oracle, whereas “Retain all, never test” and “PoE-World / OneLife fit” reach 56.9% and 35.0%, respectively.


W2.Q3: Is the method robust? Does it depend on the randomness of the LLM writing the laws?

A: Thank you for this important question. Based on your suggestion, we ran the open-ended experiment with a second set of laws written by a different LLM (Gemini 3.8 Flash) from 8 new source episodes to verify the robustness of our method, evaluating on the same 48 worlds (Sec. 3.2, RQ5; App. G.6). ProPS achieves the best score under both sets, at 6.906.90 achievements with the original laws and 8.408.40 with the second set. Please refer to Common response 2 for the full comparison.

The findings support the robustness of our approach. The best-supported rule varies from 3.293.29 to 8.068.06 between the two sets, whereas ProPS, which retains rival rules, achieves the best score under both. Overall, across the two sets ProPS obtains the highest mean in each and on average, consistent with our original conclusion.

To answer the second part of your question, our method does not rely on the randomness of the LLM in writing laws. Rival rules arise because laws are partial: a law can omit a precondition (e.g., the crafting table), and the missing part must be induced from few source transitions. When the rules agree, the rule posterior concentrates on one rule and nothing is lost. With laws from a stronger LLM, we expect the rule posterior to matter less, but still help where the laws leave a precondition open, and the second set is consistent with this (the best-supported rule reaches 8.068.06 against 8.408.40). This is consistent with our original observation in Sec. 3.2, that the rule posterior helps most when the rules are ambiguous.

Finally, the availability posterior is independent of the LLM, since no law states whether a target can be reached in a new world. With the second set it still raises the product of experts from 6.266.26 to 7.687.68.

Thank you again for this suggestion, which further strengthened our evaluation.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Dear Reviewers and AC,

We greatly appreciate your insightful and constructive reviews, and are glad to hear your acknowledgement of the positive aspects of our paper, including:

  • Addressing an important gap in symbolic world modeling: Making concrete progress on how to plan with learned world models that are partial, uncertain, and potentially stochastic (y4Gm, Vk9o).
  • Technical contribution with a conceptually simple and intuitive method: Combining a clever technique for compiling atomic laws into planning rules with a posterior-sampling control loop inspired by Thompson sampling, supported by mathematical analysis (y4Gm).
  • Promising results and informative empirical analysis: Showing promising results (4arp, y4Gm) with significantly fewer LLM calls than LLM agents (4arp), along with informative comparisons, ablations, and failure analyses (Vk9o).

In response to your feedback, we have uploaded a revised manuscript and conducted additional experiments, which we further detail in individual responses. Below, we address common questions regarding the clarity of the paper, the robustness of our method, and its application to other environments.


[Common response 1: Clarity and presentation (4arp, y4Gm)]

We appreciate the reviewers’ constructive suggestions on the exposition. In the revised manuscript, we made the following revisions:

  • Research question and motivation (Sec. 1). We clarify our focus on an important open problem of how an agent can use synthesized partial laws for goal-directed planning, and how our solution ProPS, which consists of an offline compilation step followed by an online posterior-sampling loop, addresses specific challenges in planning from a portfolio of synthesized partial laws.
  • Formal setup and framework overview (Sec. 2 and Sec. 2.1). We enhance the organization around the formal setup, the action adapter, and law synthesis. We present a concise overview of the ProPS framework in the context of the challenges introduced in Sec. 1 (incomplete effects, context unavailability, and context ambiguity).
  • Experiments organized by research question (Sec. 3). We organize around five research questions (RQ1 to RQ5), and state each takeaway in clear and simple language.

We provide further details in the individual responses to Reviewers 4arp and y4Gm. We believe these revisions make our method and its contributions much easier to follow, and we thank the reviewers for their constructive suggestions.


[Common response 2: Robustness (4arp, Vk9o)]

We appreciate the reviewers for raising this important point. To verify the robustness of our approach, we ran the open-ended experiment with a second set of laws written by a newer LLM (Gemini 3.8 Flash), evaluating on the same 48 worlds. We report the new results below:

ControllerOriginal laws (Gemini 2.5 Flash)Second set (Gemini 3.8 Flash)Average
Product of experts1.06±0.041.06 \pm 0.046.26±0.256.26 \pm 0.25 (38/48)3.663.66
Best-supported rule + availability3.29±0.283.29 \pm 0.288.06±0.318.06 \pm 0.315.685.68
Product of experts + availability5.88±0.255.88 \pm 0.257.68±0.317.68 \pm 0.31 (41/48)6.786.78
ProPS (ours)6.90±0.276.90 \pm 0.278.40±0.258.40 \pm 0.257.657.65

Entries are mean distinct achievements ±\pm standard error over the 48 evaluation worlds. (n/48) marks a mean over the n worlds in which the planner finished within the search limit.

The findings support the robustness of our approach, with ProPS achieving the best score under both sets, consistent with our original conclusion, and we thank the reviewers for this valuable suggestion.


[Common response 3: Application to other environments (4arp, Vk9o)]

As a preliminary step beyond Crafter-OO, in a second environment, ALFWorld (Shridhar et al., 2021), we ran a small study on one task type, pick-and-place. We synthesize laws from 88 source games with OneLife's prompt adapted to ALFWorld's state schema, compile them into whole-effect rules as in Sec. 2.3, and evaluate on 2424 held-out games. We report the number of held-out games solved below:

ControllerGames solved (of 2424)
Best-supported rule + availability2424
Product of experts + availability1313 (no plan in 1010)
ProPS (ours)2424

ProPS and the best-supported rule both solve all 24 held-out games with no LLM call during control, while the product of experts returned no plan within the 600-second search limit in 10 games. We describe the setup and the adaptations in our response to Reviewer Vk9o. Overall, this preliminary study confirms that ProPS is applicable end-to-end in a second environment, and we thank the reviewers for encouraging this study.

References:

Shridhar et al., ALFWorld: Aligning Text and Embodied Environments for Interactive Learning, ICLR 2021.

Dear reviewers,

Thank you for your work reviewing this paper!

The authors have submitted their responses. If you haven't already, please acknowledge them in a comment and adjust your reviews as needed. The discussion period finishes on November 18.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.