- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies planning with learned programmatic world models composed of partial laws. ProPS compiles these laws into planning rules, maintains beliefs over competing rules and action availability, and repeatedly samples a model, plans, executes an action, and updates its beliefs. The paper also presents an exact controller for a restricted setting with a small library of complete programs, with a Bayes-optimality result for that decision problem. Experiments include controlled tests of program ambiguity and open-ended control in Crafter-OO. Under a shared rule library and action interface, ProPS achieves more achievements than the evaluated planning-based and LLM-agent baselines.
Critical strengths
-
The paper addresses an important problem: how to plan effectively with learned world models that are partial, uncertain, and potentially stochastic. Predictive accuracy alone does not establish that a learned model is useful for control. The proposed compilation and control pipeline makes concrete progress toward using such models for sequential decision-making.
-
The empirical analysis provides useful evidence about which components help control. The shared planner and action adapter make the comparisons informative about alternative ways of using the same learned knowledge. The ablations and failure analyses are also valuable: the paper examines availability estimation, retention of competing rules, replanning, and joint guard induction, and explicitly discusses limitations of its approximate belief updates.
Critical weaknesses
- Evaluation is limited to Crafter-OO with one learned source portfolio. The system also relies on hand-designed compilation heuristics, planning restrictions, and a supplied action adapter, leaving unclear how much engineering would be needed to transfer it to other environments. The LLM comparisons use only Gemini 2.5 Flash with low reasoning effort, further limiting the scope of the empirical conclusions.
- The baselines are adapted versions of prior methods rather than their full implementations. For example, WorldCoder omits its optimism constraint, exploration, and planner, while the open-ended PoE baseline freezes its fitted weights. These comparisons help isolate components but provide limited evidence of improvement over the original methods.
Questions for authors
- Are there results on more benchmarks with newer LLMs?
3 comments
Response to Reviewer Vk9o (Part 2/2)
W1.Q3: LLM comparisons with Gemini 2.5 Flash at low reasoning effort
A: The ReAct agents use Gemini 2.5 Flash at temperature with low reasoning effort (App. D.4). We chose the same LLM that synthesized the original laws so that the comparison holds the underlying model fixed and varies how the laws are used. Given the law records as text, ReAct + law memory makes LLM calls across the worlds and reaches achievements (App. F), whereas ProPS compiles the same laws into rules and reaches with LLM calls during control. While we scoped the ReAct baselines to this setting, we have now also evaluated the planning controllers with laws synthesized by a newer model, Gemini 3.8 Flash (W1.Q1 above), where ProPS consistently achieves the best score.
W2: Fidelity of the baselines to prior methods
A: Thank you for raising this important point. Based on your feedback, we implemented and evaluated two further baseline variants, and added them to the revised Tables 1 and 2:
- Optimism constraint (Table 1): We added Optimistic commitment, which applies WorldCoder's optimism principle to the three candidate programs, committing to the first program under which a plan to defeat the creature fits within the remaining budget, and switching whenever an observed outcome contradicts that program. It succeeds on 48.6% of instances, the same instances as Retain all, never test, against 83.3% for ProPS.
- Refit weights (Table 2): We added Product of experts, weights refit, which refits the rule weights after every fired action on the source transitions and its own transitions, using the fitting settings of PoE-World (its Appendix A.1). It obtains achievements with our availability posterior ( with fixed weights) and without it, against for ProPS.
As you noted, sharing the planner and the action adapter makes the comparisons informative about alternative ways of using the same learned knowledge. Within this shared setting, both additions test each method's update mechanism without calling an LLM while acting. The optimism rule selects among our three supplied programs, and the refit keeps our rule set fixed. While the full implementations of WorldCoder and PoE-World involve additional LLM calls during interaction to rewrite the program (WorldCoder) or synthesize new experts (PoE-World), neither was developed for Crafter-OO: WorldCoder's monolithic synthesis loop, which refines a single program for the entire transition function, is difficult to scale to complex environments (Piriyakulkij et al., 2025), and PoE-World's Atari-specific law synthesis is incompatible with Crafter's state (Khan et al., 2026, Appendix I). Our implementation protocol therefore follows OneLife in adapting both methods to Crafter-OO, to present a fair comparison. We provide the full details of each baseline in Appendix D. Overall, when compared with optimistic commitment and weight refitting, ProPS achieves the best score in both settings.
Thank you again for your detailed suggestions on the evaluation and the baselines, which made the empirical side of our paper stronger.
Response to Reviewer Vk9o
Dear Reviewer Vk9o,
We sincerely appreciate your detailed suggestions regarding our evaluation and baselines. Through our additional experiments with Gemini 3.8 Flash, additional comparisons with WorldCoder and PoE-World, and preliminary study on ALFWorld, we hope that we have addressed your concerns.
As the discussion period is nearing its end, we would greatly appreciate it if you could let us know whether any concerns remain. If you have any further comments or questions, please do not hesitate to let us know.
Thank you again for your positive assessment of our method and empirical results, and your valuable feedback.
Response to Reviewer Vk9o (Part 1/2)
Dear Reviewer Vk9o,
Thank you for your thoughtful and constructive review of our paper. We are very pleased that you found our compilation and control pipeline to make concrete progress on planning with partial, uncertain world models, and that you found our shared-interface comparisons and empirical analyses informative. We greatly appreciate your positive assessment of our work, and your suggestions on broadening the evaluation and on the baselines. In response to your feedback, we have conducted new experiments, adding (1) a second law portfolio synthesized by a newer LLM (Gemini 3.8 Flash), (2) a preliminary study in a second environment (ALFWorld), and (3) two of the baseline components you highlighted (an optimism rule following WorldCoder's optimism constraint, and product-of-experts weights refit while acting), and uploaded a revised manuscript. We provide detailed answers to each question below.
W1.Q1 & Q1: Evaluation on more benchmarks and with newer LLMs
A: Thank you for this great suggestion. Following your suggestion, we ran two additional sets of experiments.
A second law portfolio from a newer LLM We conducted an additional evaluation of our open-ended Crafter-OO setting using a second portfolio of laws, obtained by running OneLife's exploration and synthesis pipeline with Gemini 3.8 Flash on 8 new source episodes. We found that ProPS achieves the highest score at 8.40 achievements (up from 6.90 with the original Gemini 2.5 Flash laws), outperforming Best-supported rule + availability (8.06, vs. 3.29 originally) and Product of experts + availability (7.68 over the 41/48 worlds completed within the search limit, vs. 5.88 originally). Please find the full comparison in Common response 2.
A second environment (ALFWorld). As a preliminary step beyond Crafter-OO, we evaluated ProPS on one task type, pick-and-place, in ALFWorld (Shridhar et al., 2021), using laws synthesized by Gemini 3.8 Flash. We synthesized laws from 8 source games using OneLife's prompt (with Crafter-OO's state schema replaced by ALFWorld's), compiling them into rules as in Sec. 2.3, and evaluated on 24 held-out games (64 decisions per game):
ProPS and Best-supported rule + availability, which both plan with one compiled rule per action, solve all 24 held-out games with 0 LLM calls during control. Product of experts + availability, whose search scores plans over all competing rules at once, returned no plan within the 600-second search limit in 10 games and solves 13. Overall, this preliminary study shows that the compiled rules support planning in a second environment, and that ProPS runs end to end with laws from a newer LLM.
W1.Q2: Engineering effort needed to transfer to other environments
A: Our preliminary study on ALFWorld allowed us to test this directly. We are able to adapt our method to ALFWorld with only modest engineering: The compile procedure, the rule posterior, the availability posterior, their updates, and the planning limits were the same as in Crafter-OO, and no action adapter was needed. The environment-specific work was in implementing the compile step, and the synthesis prompt adapted for ALFWorld's state schema. More specifically:
For law synthesis, we use OneLife's synthesis prompt with ALFWorld's state schema in place of Crafter-OO's, and an exploration instruction that asks the explorer to complete the task and to open and close a receptacle. Every controller observes the simulator's full symbolic state, represented as a set of fluents as in WorldCoder (Tang et al., 2024). Full details and prompts are provided in Appendix G.7.
Reference: Shridhar et al., ALFWorld: Aligning Text and Embodied Environments for Interactive Learning, ICLR 2021.