acceptodds
Under review as a conference paper at ICLR 2027

OPINE-World: Programmatic World Modeling with Ontology-error-Prioritized Interactive Exploration for ARC-AGI-3

Abstract

Learning a useful world model from minimal interaction is central to building agents that adapt to unfamiliar tasks. Programmatic world modeling approaches such as WorldCoder quickly learn transition models when given pre-supplied symbolic representations; however, such approaches are insufficient when the symbolic representation and goal are unknown and continually changing, and are intractable under large action spaces due to rigid planning heuristics. Extending the programmatic world modeling approach, OPINE-World additionally maintains an abstracted representation in which provisional objects, incomplete causal explanations, and unresolved interactions can exist in a Bayesian, exploration-centric hypothesis space. This enables learning an unknown, dynamically evolving state, transition, and goal. Exploration and program revision share this evolving abstraction, allowing new evidence to revise both the hypothesized mechanics and their implementation in tandem at test time. ARC-AGI-3 presents this open-world problem as a general-intelligence learning benchmark. Through our OPINE-World model learning paradigm, we improve the Relative Human Action Efficiency score of Claude Opus 4.8 high from 1.5% to 78.40% on the ARC-AGI-3 benchmark.

Primary area:
reinforcement learning
Keywords:
World model, ARC-AGI-3, program synthesis, object-centric, Bayesian, epistemic uncertainty, planning, agents, agentic, LLM, programmatic

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LTRbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

In my understanding, this paper introduces a test-time world-model learning and planning pipeline that resembles model-based RL for ARC-AGI-3. During interaction, the agent learns an executable transition model for next-state prediction, a reward model for goal evaluation, and an abstraction module that maps pixel observations into an object-oriented structured state. In parallel, it maintains a natural-language proto-world-model for unresolved hypotheses. Based on the learned world model, the agent either performs reward-directed planning or conducts epistemic exploration. Compared with WorldCoder, OPINE-World additionally introduces a Bayesian belief state to quantify uncertainty over object-action effects. High uncertainty is interpreted as evidence that the current ontology may be missing relevant state distinctions, which can trigger targeted exploration and state refinement, and may subsequently lead to revisions of the transition and reward models.

Critical strengths

I think the paper's major advantage is its framing in test-time ``RL'' for composite reasoning: The introduction of the Episodic Replay Buffer (memory) and Bayesian Belief State to quantify uncertainty seems to be a reasonable novelty to me, as I regard them as answering two key questions: what to learn from, and where to explore to collect data for continual world-model learning.

Audience in agentic reasoning/planning, harness design, and recursive improvement may find this work interesting. That's why I think this paper is worth a score of at least 2 for ICLR2027.

Critical weaknesses
  1. The major weakness of this paper is its writing. Section 3 is particularly hard to follow due to the large number of notations and interacting representations introduced within a short space. For example, the method simultaneously maintains (SNL,TNL,RNL)(S^{NL}, T^{NL}, R^{NL}), SBS_B, and (TP,RP,P,f−1)(T^P, R^P, P, f^{-1}), and later introduces additional quantities such as Φ1\Phi_1, Φ2\Phi_2, and j=(τ,a,c)j=(\tau,a,c). The presentation frequently switches between the natural-language model, Bayesian belief state, executable model, and planning loop, making it difficult to understand the overall workflow and the role of each component on a first read. I suggest that the authors reframe Sec. 3 with less but necessary notation, add a necessary illustration of how the pipeline works, with relatively complete examples in the appendix.

  2. The proposed framework seems highly complex and computationally expensive (2B tokens for individual environments), and its advantages or weaknesses against baselines under compute normalization have not been discussed (e.g. matched token usage, wall-clock time, or inference cost).

Questions for authors

To the authors: I am confused about several details in the paper, although some of them may be due to my own misunderstanding. I would appreciate it if the authors could answer the following questions transparently. It is completely acceptable if some design choices not fully optimal, as long as their motivations and limitations are clearly explained.

To the AC and the authors: Since I cannot fully understand the paper before the discussion period, I may ask further questions during the discussion.


  1. The authors demonstrate the effectiveness of their framework using Claude, while I am not sure whether the findings generalize to a broader range of models. Could the authors try models from different model families/providers and add some small-scale controlled experiments?

  2. Do the authors claim that their method is optimal against other baselines (e.g., WorldCoder) for all problems due to its planning mechanism? If so, I would like to better understand why. Could the authors provide a brief explanation of where the advantage comes from? If not, I would still appreciate a discussion or qualitative comparison with related methods to clarify the inductive biases introduced by the proposed framework.

  3. Continue from Q2, ARC-AGI-V3 is an interesting benchmark, but the current experiments are limited to it. I think the paper would be stronger if the authors could also evaluate on tasks used by WorldCoder, ideally with a comparably weak LLMs, to better illustrate the advantages and limitations of the proposed method relative to other baselines.


I think addressing these questions would improve the paper from an Occam's Razor perspective, by helping identify the minimal, controllable advantages and limitations introduced by the method's inductive biases.

5 comments
AuthorsOpenReview ↗

Response to Reviewer LTRb (1/3)

Thank you for your review and suggestions. We address each point below.

Critical Weakness 1

The major weakness of this paper is its writing. Section 3 is particularly hard to follow due to the large number of notations and interacting representations introduced within a short space.

We interpret this point as: Section 3 introduces too much notation at once for the overall workflow, and the role of each component is not introduced in a sensible manner, to be clear on a first read.

I suggest that the authors reframe Sec. 3 with less but necessary notation, add a necessary illustration of how the pipeline works, with relatively complete examples in the appendix.

We agree that section 3 of our paper is notationally dense and hard to follow on a first read, so we have restructured Section 3 to reduce the necessary notation to the components that minimally formalize the system shown in Figure 1, we have added an illustration of the system on an example domain in the new Figure 2, which Section 3 now references off during the introduction of components, and we have added concrete examples on each representation in Appendix A.

Critical Weakness 2

The proposed framework seems highly complex and computationally expensive (2B tokens for individual environments), and its advantages or weaknesses against baselines under compute normalization have not been discussed (e.g. matched token usage, wall-clock time, or inference cost).

We interpret this point as: the cost of the method has not been compared against the baselines under matched tokens, wall-clock time, or inference costs. Specifically: In the case of a large cost disparity, the value and usability of our system is degraded, of which has not been discussed.

We agree that a compute-normalized comparison is needed to assess the method's advantages and weaknesses, therefore, we compare score under wall-clock budgets and inference cost in Table 18 (Appendix F.9). All systems use GPT-5.5 high.

RHAE within wall-clock budget

Method2 h3 h4 h8 hFinal
OPINE-World38.950.058.782.984.3
baseline128.236.847.254.463.7
NOOA50.2–––50.2
DreamTeam––––38.1
Continual14.515.515.515.522.2

Tokens and cost

MethodTokens in / outCost per game (USD)
OPINE-World17.6B / 19.9M693.36
baseline128.6B / 49.3M744.00
NOOA–17.85
DreamTeam–720.00
Continual0.46B / 4.3M39.06

We were able to accrue enough compute to run our system across ARC-3 on GPT-5.5 high, and we report results that are accessible to us via the respective author’s publicly released results if they have a valid GPT 5.5-high run. For methods that have a blank entry, this result was not reported in their original paper or run artifacts.

On GPT-5.5 high, OPINE-World costs less than baseline1 and DreamTeam on the same model and scores higher at all wall-clock budgets. NOOA is reported by the NOOA team to score higher at that budget and at a lower cost, however, their final score is significantly lower. Specifically: The compute disparity comes from the single-agent nature of NOOA, combined with the lack of repeated tool calls and low total runtime. We do not claim that OPINE-World is superior under highly limited compute and time constraints as this is not the setting we aim to optimize, but it is a good direction for future work. Alongside baseline1 and DreamTeam approaches, we aim to demonstrate the capability of a system with tools and structures for pushing performance as high as is reasonable within open world exploration tasks.

AuthorsOpenReview ↗

Response to Reviewer LTRb (2/3)

Question #1: Do the findings generalize beyond Claude?

The authors demonstrate the effectiveness of their framework using Claude, while I am not sure whether the findings generalize to a broader range of models. Could the authors try models from different model families/providers and add some small-scale controlled experiments?

We interpret this point as: the harness may be turned to work on Claude, and the full system should be tested on a different model family.

We agree that results on a second model family are needed to show the findings are not specific to Claude, so we ran the same system across GPT-5.5 high. The RHAE for this model was improved from 0.5% to 84.3%, demonstrating similar performance improvements across model families that OPINE-World was not built on. We have updated Table 1, which now additionally compares the system on GPT-5.5 high:

SystemMean RHAE ± SELevels completedGames completed
OPINE-World (GPT-5.5 high)84.3 ± 3.5176/18321/25
OPINE-World (Opus 4.8 high)78.4 ± 6.3160/18320/25
No S^B\widehat{\mathcal S}_{\mathrm B} (Opus 4.8 high)58.2 ± 8.3126/18312/25
baseline163.7 ± 7.4146/18314/25
DreamTeam38.1 ± 7.698/1836/25
NOOA50.2118/183–
Continual22.2 ± 5.986/1835/25

Per-game results are in Appendix F.1.

AuthorsOpenReview ↗

Response to Reviewer LTRb (3/3)

Question #2: Is the method claimed to be optimal over WorldCoder, and where does the advantage come from?

Do the authors claim that their method is optimal against other baselines (e.g., WorldCoder) for all problems due to its planning mechanism? If so, I would like to better understand why. Could the authors provide a brief explanation of where the advantage comes from?

We interpret this point as: whether OPINE-World is claimed to outperform WorldCoder on every problem, and if not, which of its design choices produce the gains or losses.

We agree that the paper should more specifically break down the benefits over prior work such as WorldCoder, therefore, we have run a WorldCoder comparison and added results to Section 5.1 (Table 2), and a discussion of where WorldCoder outperforms OPINE-World to Appendix E.

Solved episodes (x optimal)

EnvironmentWorldCoderOPINE-World
Empty-5x51.45 ± 0.021.52 ± 0.35
DoorKey-5x51.27 ± 0.121.08 ± 0.04
Unlock1.32 ± 0.051.10 ± 0.04
Fetch-6x6-N21.59 ± 0.041.21 ± 0.14
UnlockPickup–1.31 ± 0.10
BlockedUnlockPickup–1.60 ± 0.19
Sokoban, 2 boxes1.41 ± 0.201.23 ± 0.01
Sokoban, 3 boxes1.04 ± 0.041.04 ± 0.02

Episodes to mastery

EnvironmentWorldCoderOPINE-World
Empty-5x511.011.0
DoorKey-5x511.013.3 ± 2.3
Unlock11.011.0
Fetch-6x6-N211.011.0
UnlockPickup–11.0
BlockedUnlockPickup–15.0 ± 4.0
Sokoban, 2 boxes12.0 ± 1.016.7 ± 3.0
Sokoban, 3 boxes–15.7 ± 4.2

In a direct comparison on WorldCoder-compatible domains, OPINE-World produces action sequences closer to the optimal solution despite both systems having access to a planner. WorldCoder's planner fails to find a reward-reaching plan in UnlockPickup due to the absence of synthesized heuristics and cannot progress to BlockedUnlockPickup, while OPINE-World eventually masters all six environments. On Sokoban, WorldCoder masters the 2-box environment in fewer episodes; however, even with a grid search across a 1000-12000 node depth limit for its MCTS planner, its most successful configuration at 6000 nodes exhausts its generous 24h search budget on the 3-box layouts and is unable to progress. This suggests that OPINE-World provides more efficient and computable solutions to simpler domains.

OPINE-World does not improve on domain specific approaches, such as WorldCoder, in every case. In these domains, the symbolic state, action names, and goal may be supplied, which removes the uncertainty that OPINE-World's object abstraction, goal inference, and exploration are designed for. Outside of these domains, WorldCoder is unable to progress as it lacks the systems to acquire this information. Within these domains, due to the rigid heuristic and sub-optimality of its MCTS planner, WorldCoder is unable to progress in certain levels due to planner intractability. These are the cases where OPINE-World demonstrates a higher capability.

Question #3: Can you evaluate on WorldCoder's tasks with a comparably weak LLM?

I think the paper would be stronger if the authors could also evaluate on tasks used by WorldCoder, ideally with a comparably weak LLMs, to better illustrate the advantages and limitations of the proposed method relative to other baselines.

We interpret this point as: an evaluation on WorldCoder's own tasks, with a weak model, would show the advantages and limitations of the method outside ARC-AGI-3.

We agree that a comparison on WorldCoder's tasks shows both the advantages and the limitations of the method, so we have added the WorldCoder comparison to Section 5.1 (Table 2) and Appendix E (refer to above table in Question #2). We run OPINE-World and WorldCoder on WorldCoder's MiniGrid and Sokoban domains, and give each agent information exposed in the original WorldCoder benchmark, with each run capped at 24 hours of planner CPU compute time on a Ryzen 7600X. We measure actions relative to the optimal plan and episodes to mastery. Both systems in the WorldCoder comparison use a weaker GPT-6 Luna so the domains don't saturate.

See our response to Question #2 for findings.

Reviewer_YcVDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents OPINE-World, an inference-time structured world modeling harness for ARC-AGI-3 that combines natural language (NL), Bayesian, and programmatic representations. The paper shows that its bayesian belief state representation is able to propose targeted exploration goals and more efficient program repairs for its world model by tracking under-sampled or conflicting object-action effects. Furthermore, the NL representation aids in initial online learning compared to purely using a programmatic model, suggesting it helps the LLM interpret the observations. These components are combined in a learning loop where the agent can plan using its world model, using it to pursue goals or explore, and update its model based on prediction mismatches. Overall, the contribution of OPINE-World is showing that the NL, Bayesian, and programmatic structure for world modeling has potential for efficient repair and improved exploration. They validate their approach on ARC-AGI-3 games and show that their harness is able to improve the Relative Human Action Efficiency score of Claude Opus 4.8 from 1.5% to 78.40%.

Critical strengths

S1. The paper's main claim for using the NL and Bayesian representations are supported by ablations that show its usefulness for more efficient learning in terms of fewer actions, synthesis rounds and tokens. This is the main factor for my soundness assessment of the paper.

Critical weaknesses

W1. It is unclear what the base model for each of the baselines are. The paper mentions NOOA having a different base model. However, why do the authors not use the same base model for the methods and compare (i.e. Claude Opus 4.8 high with NOOA or OPINE-World with GPT-5.6-sol xhigh)? This is a potential concern because these differences may make it difficult to assess OPINE-World's improvements over the baselines, and thus its significance.

Questions for authors
  1. Regarding W1, could the authors clarify which base LLM was used for each baseline and whether model-matched comparisons to NOOA are possible?
3 comments
AuthorsOpenReview ↗

Response to Reviewer YcVD

Thank you for the review, we address each point below.

Critical Weakness 1

It is unclear what the base model for each of the baselines are. The paper mentions NOOA having a different base model. However, why do the authors not use the same base model for the methods and compare (i.e. Claude Opus 4.8 high with NOOA or OPINE-World with GPT-5.6-sol xhigh)? This is a potential concern because these differences may make it difficult to assess OPINE-World's improvements over the baselines, and thus its significance.

We interpret this point as: comparing systems that use different base models makes it difficult to attribute the improvement to OPINE-World rather than to the base model.

We agree that the comparison should be model-matched, so we have run the full system on GPT-5.5 high, the same base model as other methods in Table 1, and were able to observe similar increases in performance compared to base model capability. Using GPT-5.5 high, it improves RHAE from 0.5% to 84.3%. We have updated Table 1:

SystemMean RHAE ± SELevels completedGames completed
OPINE-World (GPT-5.5 high)84.3 ± 3.5176/18321/25
OPINE-World (Opus 4.8 high)78.4 ± 6.3160/18320/25
No S^B\widehat{\mathcal S}_{\mathrm B} (Opus 4.8 high)58.2 ± 8.3126/18312/25
baseline1 (GPT-5.5 high)63.7 ± 7.4146/18314/25
DreamTeam (GPT-5.5 high)38.1 ± 7.698/1836/25
NOOA (GPT-5.5 high)50.2118/183–
Continual (GPT-5.5 high)22.2 ± 5.986/1835/25

OPINE-World achieves the highest RHAE among comparators and completes the most games (Table 1, per-game results in Appendix F.1). On GPT-5.5 high, OPINE-World also costs less than baseline1 and scores higher at all budgets (Appendix F.9).

Question #1: Which base LLM was used for each baseline, and are model-matched comparisons to NOOA possible?

Regarding W1, could the authors clarify which base LLM was used for each baseline and whether model-matched comparisons to NOOA are possible?

We interpret this point as: the base model of each baseline should be stated, and NOOA should be compared on the same base model as OPINE-World.

We agree that the base model of every baseline should be stated. To address this, all systems in Table 1 and Table 18 have been updated to report results on GPT-5.5 high: baseline1, DreamTeam, NOOA and Continual, and OPINE-World in its GPT-5.5 column (See our response to Critical Weakness 1). NOOA's reported final GPT-5.5 high result is 50.2 RHAE (118/183 levels), against 84.3 (176/183 levels) for OPINE-World on the same model.

MethodRHAE at 2 hFinal RHAELevels completed
OPINE-World38.984.3176/183
NOOA50.250.2118/183

We have additionally added Table 18 (Appendix F.9), which compares score under wall-clock budgets and inference cost. NOOA reports 85.1 on GPT-5.6-sol xhigh. We did not run OPINE-World on GPT-5.6-sol, so the model-matched comparison to NOOA is the GPT-5.5 high result above.

Dear Reviewer YcVD, thank you again for your review. As the discussion period closes today, we would appreciate it if you could let us know whether our response addresses your concerns. In summary, we have run OPINE-World on GPT-5.5 high, the same base model as the other methods, and updated Table 1, and added Table 18 (Appendix F.9). We are happy to answer any further questions.

Since the discussion is ending and Reviewer YcVD has not yet participated in the discussion, as a fellow reviewer, I think the authors’ responses appear to address Reviewer YcVD’s concerns.

Reviewer_sJqPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

World modeling is an essential and highly active research area. One approach for world modeling is programmatic models (WorldCoder). Such models require a predetermined symbolic representation of the world, and fail when the symbolic state and goal are unknown or changing. To handle this, the paper introduces OPINE-World, an expansion of WorldCoder. The main idea is maintaining 2 levels of representation: a. A quantitative, executable world model (python program, saved in T_P, R_P) b. A qualitative representation in natural language (saved in S_NL, T_NL, R_NL). In addition, a Bayesian belief state B_S is maintained to track empirical transition uncertainties. Section 3 of the paper formally defines the different components of the OPINE-World model and the learning loop for it.
The following research questions are asked:

  1. Does Bayesian epistemic guidance improve exploration? To evaluate this question, the full OPINE-World model is compared with a no bayesian state ablation over 25 ARC-AGI-3 games (budget of 2,000 steps). It is also compared with an epsilon-greedy exploration. Results indicate that the Bayesian state variant achieves higher RHAE, and solves more levels and games than all other baselines.
  2. How does the program structure affect generalizability, adaptivity and maintainability? The full OPINE-World (hybrid) is tested against monolithic, strict OOP and unrestricted baselines on a fixed action trajectory of a single game, under 3 rule perturbations M1, M2, M3. The hybrid structure shows high test accuracy on unseen transitions and requires significantly less code repair and output tokens under local rule changes.
  3. Does the hybrid representation (NL+code) help world modeling? Evaluated by comparing the full OPINE-World against an ablation that drops the NL files and code comments on a single game. The OPINE-World model clears the game faster with fewer synthesis rounds.
Critical strengths
  1. Combining natural language, qualitative description of the world model is a good idea aimed at solving a real problem with programmatic world modeling. Adding the Bayesian belief state also helps in encoding uncertainty. Overall these are good ideas worth exploring.
  2. Results indicate the suggested hybrid framework outperforms all baselines on ARC-AGI-3, demonstrating the potential of the hybrid system.
  3. Experimental design is mostly done well, indicating the advantages in adding the NL description of the world, and mostly adding the belief state.
Critical weaknesses
  1. RQ1 asks whether the Bayesian belief state helps exploration. The results indicate that adding it increases RHAE and number of games/levels solved over baselines/no-bayesian-state/epsilon-greedy exploration. This, however, does not necessarily indicate the Bayesian state results in better exploration - the gap might also come from better program synthesis. It would be interesting to see a direct metric for exploration.
  2. While the S_B component is evaluated over all 25 games, the NL representation, which was introduced as a main contribution, is only evaluated on the first 3 levels of a single game. Moreover, the selection of this game (g50t) might be a bit biased, as it is a game on which the no-S_B ablation from RQ1 did not succeed at all, while the full OPINE_World got all 7 levels correct. It would be useful to see a larger testbed for this component, as it is an important component of the suggested solution.
  3. (Minor) The ablation in RQ3 completely removes code comments as well as the NL representation. Since code is usually generated with comments, it would be interesting to add this to the ablation - ablating over code comments as a separate component to really see the utility of the NL representations.
Questions for authors
  1. Can you decouple exploration from program synthesis for RQ1? I.e. is there a direct metric/qualitative way of evaluating the benefit to exploration rather than RHAE?
  2. How does RQ3 ablation look like when code comments are kept (but NL representation is removed)?
  3. Can you provide the RQ3 ablation results on other games? Preferably those in which the no-S_B baseline did not perform as well.
3 comments
AuthorsOpenReview ↗

Response to Reviewer sJqP (1/2)

Thank you for your review and suggestions. We address each point below.

Critical Weakness 1

RQ1 asks whether the Bayesian belief state helps exploration. The results indicate that adding it increases RHAE and number of games/levels solved over baselines/no-bayesian-state/epsilon-greedy exploration. This, however, does not necessarily indicate the Bayesian state results in better exploration - the gap might also come from better program synthesis.

We interpret this point as: assessing the RHAE by itself is not sufficient to conclude that the Bayesian belief state helps exploration.

It would be interesting to see a direct metric for exploration.

We agree that RHAE measures action efficiency, a correlative measure of exploration efficiency and doesn’t measure exploration by itself. Specifically, in the case where both systems have been exposed to the same information, the system with the most competent planner and world model may perform better, therefore, we have added Table 3 to Section 5.2 further analyzing the games where the two systems differ in performance. To look at exploration more directly, we report the share of repeated action sequences resulting in no knowledge gain. For the purpose of easy access, we provide the tables in a readable form below:

Levels completed

GameFullS^B\widehat{\mathcal S}_{\mathrm B} Abl.
cn0464
dc2262
g50t70
s5i542
sc2563
sp8065
su1597
tn3671
wa3093
Total6027

Unique effects discovered

GameFullS^B\widehat{\mathcal S}_{\mathrm B} Abl.
cn0410160
dc2220350
g50t4428
s5i58047
sc257022
sp805849
su159362
tn363611
wa30235120
Total920449

Discovered effects modeled (%)

GameFullS^B\widehat{\mathcal S}_{\mathrm B} Abl.
cn04100.0100.0
dc22100.0100.0
g50t100.075.0
s5i5100.0100.0
sc25100.0100.0
sp80100.0100.0
su15100.0100.0
tn36100.0100.0
wa3098.792.4
Total99.996.4

Repeated action sequences (%)

GameFullS^B\widehat{\mathcal S}_{\mathrm B} Abl.
cn041.16.1
dc229.410.3
g50t15.232.8
s5i56.021.5
sc255.97.9
sp802.24.6
su153.317.9
tn367.717.0
wa303.311.5
Total6.014.4

A high share of repeated actions indicates a lack of progress or a narrow trajectory. Additionally, while both systems model nearly all unique effects they observe (99.9% and 96.4%), the ablation repeats more action sequences (14.4% vs. 6.0%). While the number of discovered effects are directly related to the levels seen, the count shown proves the percentage of effects modeled has a significant enough sample size to be a meaningful number.

Therefore, the ablation's losses arise predominantly from weaker acquisition of evidence rather than modeling failure. We are able to conclude that exploration capability is negatively affected by the belief state ablation, in either the planner heuristic or the Bayesian belief state, both a fundamental component to exploration, rather than purely a difference in modeling capability.

AuthorsOpenReview ↗

Response to Reviewer sJqP (2/2)

Critical Weakness 2

While the S_B component is evaluated over all 25 games, the NL representation, which was introduced as a main contribution, is only evaluated on the first 3 levels of a single game. Moreover, the selection of this game (g50t) might be a bit biased, as it is a game on which the no-S_B ablation from RQ1 did not succeed at all, while the full OPINE_World got all 7 levels correct.

We interpret this point as: assessing the g50t domain by itself does not test a large enough sample of the representative domains to draw a reliable conclusion for RQ3, especially with the Belief state ablation for the chosen domain scoring significantly lower.

It would be useful to see a larger testbed for this component, as it is an important component of the suggested solution.

We agree that a larger representation of the domain space with various RQ1 relative ablation scores will better support or reject the RQ3 claim. To address this, we ran the NL ablation on four random additional games: cd82, cn04, ft09, and tn36, of which contain games where the S^B\widehat{\mathcal S}_{\mathrm B} ablations perform comparably (ft09, cd82), and reran g50t under these new testing conditions. We believe our results under these 5 games are sufficient to demonstrate the usefulness of natural language in world modeling. We have updated Section 5.4 and Figure 6, and added Table 17 (Appendix F.8).

RHAE

GameFullNLCNL
cd8291.17.47.9
cn0454.848.03.1
ft0990.1100.045.8
g50t24.620.824.6
tn3618.212.34.7
Mean ± SE55.8 ± 15.537.7 ± 17.117.2 ± 8.1

Actions per level

GameFullNLCNL
cd82283427
cn04143836
ft09211616
g50t88145117
tn36783682
Mean ± SE46.0 ± 15.453.7 ± 23.155.4 ± 19.0

Synthesis rounds per level

GameFullNLCNL
cd825.06.04.0
cn041.04.06.0
ft092.21.81.2
g50t5.38.07.7
tn364.02.03.5
Mean ± SE3.5 ± 0.84.3 ± 1.24.5 ± 1.1

Synthesis tokens per level (M)

GameFullNLCNL
cd827.19.811.1
cn043.66.114.3
ft092.83.62.7
g50t11.922.128.3
tn363.34.15.0
Mean ± SE5.7 ± 1.79.2 ± 3.412.3 ± 4.5

On g50t, the game originally used for RQ3, RHAE is tied under this new run, likely due to inter-run variance, though the full system still uses fewer actions (88 vs. 117), synthesis rounds (5.3 vs. 7.7) and tokens (11.9M vs. 28.3M) per level. The cross-game gain is therefore not driven by the original game choice.

On average across games, the full system outperformed the NL ablation in RHAE, actions per level, synthesis rounds, and synthesis cost.

Critical Weakness 3

The ablation in RQ3 completely removes code comments as well as the NL representation. Since code is usually generated with comments, it would be interesting to add this to the ablation - ablating over code comments as a separate component to really see the utility of the NL representations.

We interpret this point as: Outside of the complete NL ablation, it would be interesting to see how ablating only the natural language representation and leaving in code comments would affect the system’s ability to progress.

We agree that this would be a good experiment to see the utility of the NL representations in any form, so we have added this condition (NLC) to Section 5.4, Figure 6 and Table 17. NLC retains code comments and docstrings.

See Tables under Weakness 2.

The NLC run is able to store the world model within the code, but an explicit NL state is not instructed to be provided. The NLC run was able to score much closer to our full system (37.7 vs. 55.8 RHAE, against 17.2 without NL). However, it remained much more expensive (9.2M vs. 5.7M synthesis tokens per level) and needed more synthesis rounds (4.3 vs. 3.5 per level).

We observed that most of these comments explained why the code was created and what world component it was created for. As a result, we found that these code comments often contained natural language information about the world, not merely being isolated to the code itself. We have added these comments for each game to Appendix F.8 (Examples 1–5). This suggests that any form of natural language representation has high utility in representing information necessary for exploration and problem solving.

Question #1: Can you decouple exploration from program synthesis for RQ1?

See Weakness #1.

Question #2: How does the RQ3 ablation look when code comments are kept?

See Weakness #3.

Question #3: Can you provide RQ3 results on other games, preferably those where no-S^B\widehat{\mathcal S}_{\mathrm B} did not perform well?

See Weakness #2.

Dear Reviewer sJqP, thank you again for your review. As the discussion period closes today, we would appreciate it if you could let us know whether our response addresses your concerns. In summary, we have added Table 3 (Section 5.2) on exploration, extended the NL ablation to five games and added the NLC condition (Section 5.4, Figure 6, Table 17), and added the NLC comment examples to Appendix F.8. We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.