acceptodds
Under review as a conference paper at ICLR 2027

WorldEngine: Reverse Engineering Visual Worlds with Executable Coding Agents

Abstract

Adapting AI models to novel visual domains typically requires humans to hand-engineer simulators, training data, and evaluators, whereas learned neural world models remain difficult to inspect, verify, or control. We introduce WorldEngine, a framework in which a coding agent turns a minimal task description and sparse visual examples into an executable world hypothesis: an engine that generates task instances, renders solution trajectories, and validates candidate solutions, with configurable difficulty and appearance. In Part I, evaluations across four visual reasoning and planning domains expose an executability-fidelity gap: engines that run without error still encode wrong rules, leak answers, or collapse controls into fixed templates. Our structured construction workflow improves rule fidelity and validity, and explicit diversity objectives activate controllable variation. In Part II, we use a synthesized engine to adapt a video-based visual planner in two regimes. Used open-loop, the engine supplies verified trajectories for fine-tuning, raising the solving rate on an independent simulator's puzzles from 2% to 24%. Used closed-loop, its validator feeds per-difficulty success back into sampling from a fixed pool, improving mean hard-level validation solving rate by up to 12.0 points over a static mix. Together, these results establish executable world reverse engineering as an inspectable, controllable engine for adapting visual planners.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agents, Recursive Improvement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_K6HGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

WorldEngine asks whether an AI coding agent can turn a task description and a few visual examples into a simulation, ensure that simulation is correct, and be used to train video-generation model to solve visual puzzles. It's worth noting that the agent/WorldEngine's rules, rendering, and validation logic can therefore be inspected and modified directly. The paper’s motivation is that adapting a visual model to a new task usually requires someone to manually build a simulator, generate correct demonstrations, and write an evaluator. WorldEngine tries to automate that infrastructure from sparse task evidence.

Critical strengths

The authors proposes an interesting idea of whether automatically built environments can be useful teachers. The authors combine environment reconstruction through agentic frameworks to training a separate pixel-space video planner.

The improvement from 2% to 24% on independently generated puzzles shows that the synthesized engine produces useful data. However, I would still say that, it is unclear if the WorldEngine produces worse data, does that actually affect the improvement?

Critical weaknesses
  1. I am of the opinion that this is primarily a system paper and lacks methodological insights. Coding agents for inverse graphics or inverse physics are popular. The video generated models are not the novelty here.
  2. I have some serious concerns about overclaiming: the authors demonstrated the construction -> generation -> solving puzzles pipeline in very limited settings, yet calling this a WorldEngine. There is substantial overlap between the video generator's fine-tuning distribution and its test distribution, which makes this much more limited, i.e., implementing mechanics already supplied in text and using the generated images mainly to reproduce the mechanics.
  3. The paper largely demonstrates that a structured agent workflow catches some defects, but far from a systematic proof that the agent workflow is reliable.
Questions for authors

Please answer the weakness section, particularly about train/test distribution and why this would be generalizable to unseen dynamics.

Reviewer_LeK2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

WorldEngine uses a coding agent to construct an executable environment from a task description and a few visual examples. The agent writes an explicit rule hypothesis, implements generation, rendering, and validation interfaces, and revises either the rules or their implementation based on feedback from an independent critic. The resulting engine is then used to produce training trajectories and evaluate a downstream visual planner.

The paper evaluates engine construction on four visual puzzle domains, with construction ablations on Her-Trees and downstream experiments on RushHour. Fine-tuning Wan2.2 on engine-generated trajectories raises the reported solving rate from 2% to 24% on puzzles from an independent simulator. A separate curriculum experiment adjusts difficulty proportions within a fixed training pool, improving mean hard-level controller-validation performance by up to 12 percentage points.

Critical strengths
  1. The distinction between a runnable engine and a faithful training environment is important. The audits document cases where generated software executes successfully but implements incorrect constraints or exposes answer-bearing information. These examples make a useful point: passing build and unit-execution checks is not enough when the generated program will supply supervision for another

  2. The construction workflow supports inspection and targeted repair. Separating the rule specification from its implementation gives the critic a concrete basis for diagnosing failures.

  3. The downstream evaluation is not confined to the engine that supplied the training trajectories. The improvement on unseen puzzles from a separately implemented simulator is therefore more informative than an in-engine result alone.

Critical weaknesses
  1. The empirical scope is narrow relative to the broader framing. The four domains are stylized 2D puzzles centered on geometric arrangement or constrained movement. Construction ablations are limited to Her-Trees, and downstream adaptation is demonstrated only on RushHour at two difficulty levels. Therefore, they leave substantial uncertainty about how the method handles less/more constrained environments or more varied interaction rules. I thinking it establishes transfer between implementations and distributions of the same task, rather than transfer to a different set of mechanics. The demonstrated range of applicability remains limited.

  2. The ablations do not isolate the contributions of the proposed construction mechanisms. The no-skill ablation removes the entire workflow, leaving the individual effects of hypothesis articulation, critic feedback, and failure routing unclear. A compute-matched implementation–review–repair baseline would be more informative. The diversity intervention also bundles multiple instructions, so its gains cannot be attributed to diversity alone. Finally, comparison with an untuned model demonstrates the value of task-specific supervision, but not whether this workflow produces better training data than a simpler coding-agent baseline.

  3. The curriculum experiment does not isolate the benefit of feedback. The largest gains occur on the validation set repeatedly queried by the controllers. External gains are only 2.2 and 2.9 percentage points, with both confidence intervals including zero. With one training run per policy, claims of improved generalization or reduced overfitting remain unsupported. Moreover, comparison only with a uniform 1:1 baseline cannot distinguish feedback-driven adaptation from simply emphasizing harder cases. A fixed hard-heavy baseline (e.g., L2:L3 = 1:3) is needed to separate these effects.

Questions for authors
  1. The improvement from 2% to 24% shows that the generated data is useful, but does not isolate the value of the proposed workflow. How does it compare with a standard coding agent using a generic review-and-repair loop, given the same inputs and construction budget? The resulting datasets should be compared under matched data sizes and fine-tuning budgets.

  2. The evaluator parses the initial state from the generated first frame. Is this checked against the original input state? Could the authors repeat the evaluation using the original state for replay, reporting both plan success and joint parse validity/plan completion?

  3. How do EMA and Kalman compare with a fixed L2:L3 = 1:3 mixture or a preset schedule matched for cumulative per-level exposure? Do the gains persist across training seeds and on a separate in-distribution test set that the controller never queries?

Reviewer_Ba1XOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed WorldEngine, using a coding agent to construct an executable task engine from a task description and a small set of visual demonstrations. The engine implements generation, rendering, and validation with a structured workflow. They also use a separate critic to route failures to rule revision or code repair. In RushHour, engine-generated videos improve Wan2.2's reported cross-engine action-plan solving rate from 2% to 24%. The results suggest that these environments can provide useful training data, although the evaluation of the resulting video plans needs further clarification.

Critical strengths
  1. The paper addresses a useful problem and provides concrete examples of why checking whether generated code runs is not enough. The Tangram example is particularly helpful: the engine's own solution fails because of an incorrectly handled polygon hole, which the critic identifies after ordinary tests pass. These examples make the motivation convincing.

  2. The authors go beyond evaluating the generated environments themselves. In Table 3, training on the generated data improves action-plan success on an independent RushHour engine from 2% to 24%. This is encouraging evidence that the data are useful for training.

Critical weaknesses
  1. My main concern is whether the planning metric measures solving the original puzzle. Appendices E.2 and F.3 infer the initial state from the generated first frame and then replay the extracted actions. If the model changes the puzzle in that first frame, a successful replay could correspond to a different problem. I could not find a check that this state matches the input puzzle. Also, replaying a valid action sequence does not establish that the motions shown in the video are legal. This distinction matters for the paper's claims about visual planning.

  2. Some of the proposed components need stronger controls. The task descriptions already provide important rules, so it is unclear how much the visual examples contribute to rule recovery without a description-only baseline. For the curriculum experiment, the large gains are on the validation set used by the controller, while both confidence intervals for the independent-engine gains include zero. A fixed curriculum that simply samples more hard cases would help determine whether the feedback mechanism itself is useful.

Questions for authors

See as Weaknesses. & Is the state parsed from the first generated frame checked against the input puzzle? How do the results change when this agreement and legal transitions in the generated video are required?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.