acceptodds
Under review as a conference paper at ICLR 2027

WorldForge: Benchmarking Persistent Embodied Agents in Living Worlds

Abstract

Embodied agents must often act with incomplete world state, yet missing information can be handled in fundamentally different ways: an agent can spend more computation reasoning over what it already knows, or act to acquire new evidence. We introduce WorldForge, a simulated household platform for studying such decisions under controlled, replayable world conditions. WorldForge provides generated homes, certified physical interactions, matched task instances, and a shared interface for embodied agents, while supporting changes in information, task structure, and world dynamics. As one controlled study within this platform, we examine everyday object relocation and vary only whether target locations are given or must be recovered through interaction. Across two model–interface systems and multiple reasoning effort settings, withholding locations reduces task completion and induces substantially more search. Increasing fixed reasoning does not reliably close this gap, whereas self-selected reasoning matches the best fixed-effort GPT performance with substantially less computation. Trajectory analysis reveals that most remaining failures occur before the target ever becomes observable, particularly for objects hidden inside closed containers. These results expose a gap between allocating internal computation and allocating information-gathering actions, and illustrate how WorldForge can be used to isolate and study such failures in embodied agents.

Primary area:
applications to robotics, autonomy, planning
Keywords:
WorldForge: How Reasoning Models Choose Between Remembering, Thinking, and Looking

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_PmNyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper introduces WorldForge, a household simulation platform with generated environments, simulation-verified interactions, and replayable world states. It studies whether increasing reasoning effort helps embodied agents recover missing information by comparing otherwise identical tasks with and without target-object locations. Experiments evaluate GPT-6 Astra through Codex CLI and Claude Opus 5.5 through Claude Code under three fixed reasoning settings and an adaptive setting. Although the study includes 12 tasks across five homes, its central analysis uses seven selected tasks across three homes. The authors report that additional reasoning does not consistently improve completion, failures frequently occur before targets become visible, and adaptive reasoning reduces GPT’s reasoning-token usage by 78% while matching its best observed completion.

Critical strengths
  • Controlled information comparison. The Known/Unknown pairing preserves the initial world state and task requirements while changing the supplied location information. This provides a useful experimental design for examining the consequences of missing information.

  • Informative trajectory analysis. Separating failures into acquisition, use, and execution stages provides more insight than aggregate completion rates alone. The distinction between reaching the relevant furniture and opening the correct compartment is particularly useful.

Critical weaknesses
  • Limited significance of the research motivation. The paper does not convincingly establish why varying native reasoning effort on household search tasks addresses a sufficiently significant research gap. Its main observation—that agents must perform appropriate interactions to reveal hidden objects—provides limited actionable insight into how better agents should be designed. The broader motivation of choosing among remembering, thinking, and looking is also not fully investigated: the experiments keep the environment static and do not manipulate memory freshness or the environmental cost of deliberation.

  • Insufficient methodological novelty. The technical contribution primarily consists of integrating scene generation, interaction verification, and checkpoint replay into a simulation platform, followed by comparisons across information conditions and reasoning settings. The manuscript does not clearly establish what substantive methodological advance this combination provides. For a platform contribution, a convincing comparison with existing environments is particularly important to demonstrate which new experimental capabilities WorldForge enables. The current evaluation does not sufficiently establish this distinction.

  • Insufficient comparison with relevant baseline methods. The evaluation compares only two model–interface pairs under different reasoning settings, without including alternative strategies for embodied information acquisition. Relevant baselines would include active-exploration policies, agents with explicit uncertainty tracking, and simple heuristics that systematically inspect plausible closed containers.

  • Broken cross-references to the supplementary material. Although the appendix is provided in a separate file, the main manuscript contains unresolved references such as “Appendix ??” and “Table ??”. These broken references make it difficult to locate the corresponding benchmark results, prompts, action specifications, execution budgets, and supporting analyses. The authors should repair the links and explicitly identify the relevant appendix sections and tables so that readers can readily trace the evidence supporting the paper’s claims.

Questions for authors
  • How does adaptive reasoning compare with fixed-low reasoning under the same cost-aware prompt? Does it retain an advantage when average reasoning-token usage is matched?

  • Would a simple policy that systematically inspects plausible closed containers substantially improve completion? Can interventions supplying the correct furniture or compartment distinguish spatial uncertainty from failures to perform information-gathering actions?

  • Which conclusions are intended to generalize to memory use and dynamic environments? Can you either provide experiments manipulating memory freshness and deliberation time or narrow the framing to the static setting actually evaluated?

Reviewer_iri2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces WorldForge, a MuJoCo-based household platform with persona-generated homes, simulation-certified interaction affordances, checkpointed exact replay, and a shared skill interface for embodied agents. In a controlled study, 12 object-relocation tasks (evaluated on a 7-task, 14-target subset) are run under matched Known/Unknown conditions that differ only in whether target initial locations are stated, across fixed reasoning efforts (medium/high/extra-high) and an adaptive self-selected-effort condition, for two model–interface systems (GPT-6 Astra via Codex CLI; Claude Opus 5.5 via Claude Code). Findings: withholding locations lowers completion (GPT 4–5/7, Claude 3–4/7); raising effort multiplies reasoning tokens (22× for GPT) without reliably closing the gap; per-decision reasoning does not rise; adaptive effort matches best fixed completion at lower cost; most failures are acquisition failures on closed-container targets (12/14); an offline probe shows both models rank the true source furniture highly (median rank 2), yet GPT fails to reach sources while Claude reaches them but fails to open the correct compartment — concluding that knowing where to look is separable from taking the action that exposes evidence.

Critical strengths

S1. The matched-condition design is sound and the right one. Paired Known/Unknown episodes on identical world checkpoints, with only two sentences removed, cleanly attribute the gap to information rather than task difficulty; certified interactions and replay make this reproducible.

S2. Stage-resolved failure attribution is the paper’s best idea. The acquisition/use/execution funnel and the reached/opened/visible/completed breakdown (Tables 2–3) yield the most interesting observations — the GPT-reaches-but vs. Claude-reaches-but-never-opens dissociation at equal completion, and the probe-vs-action gap in §4.4.

S3. Unusually careful scoping. The authors repeatedly refuse overclaiming: no cross-system token comparisons, no claims on process tasks, explicit acknowledgment that adjacent-effort differences are within run-to-run variation, and a candid Limitations section.

Critical weaknesses

W1. The study is underpowered for its central negative claims. Seven tasks, three homes, 14 targets, one run per configuration. Footnote 2 shows a single repeat of Claude medium moves completion 3/7 → 4/7 — the same magnitude as the effort effects interpreted in §4.2. The headline result that “raising effort does not close the gap” rests on 4→5→4 and 3→3→4 with no confidence intervals; the authors themselves concede adjacent-setting differences are within run-to-run variation (L315). The 22× token multiplication and the 78% adaptive-cost reduction are likewise single-run quantities.

W2. The central mechanism claim is confounded by the harness. The paper’s most striking finding — GPT fails to reach, Claude reaches but fails to open (Tables 3; §§4.4, 5) — is precisely the cross-system contrast the authors disclaim elsewhere (L244–247: interfaces serialize context differently; treat model–interface pairs as separate systems). “Current models lack the judgment to expose evidence” is a model-level conclusion drawn from system-level evidence; Claude’s “passes a reached source without opening” could be a context-serialization or affordance-salience artifact of Claude Code rather than a model trait.

W3. The probe–action gap admits a perceptual confound, and no baseline calibrates difficulty. The location probe queries furniture-level rankings from a static map; acting requires identifying the correct compartment from egocentric RGB. Claude reaching the sideboard but not opening it may reflect failure to visually ground which drawer/joint exposes the target — a perception failure, not a judgment failure — and the paper does not analyze what the model perceived at the source. Moreover, there is no reference point for 4–5/7 Unknown completion: no scripted sweep-and-open policy, no semantic-prior navigation baseline (e.g., Dorbala et al., 2024, cited), and no smaller or non-reasoning model. Without these, neither “models are bad at this” nor “reasoning is not the bottleneck” has a yardstick.

W4. Framing outruns the design. (i) That computation cannot reveal the contents of an unopened cabinet is true by construction; the non-trivial question — whether reasoning improves search planning — is answered only indirectly via the probe. (ii) The subset is admitted precisely because Known succeeds in every configuration, so the gap is measured only on tasks that are trivial with information supplied. (iii) The title and framing promise a three-way choice among remembering, thinking, and looking, but with residents fixed and the clock paused, memory is never operative; the study tests two of the three advertised regimes.

W5. Related-work completeness. The discussion of embodied architectures omits recent model-side work that hierarchically decomposes perception–reasoning–action or adaptively truncates reasoning between stages, which are the natural counterparts of this platform’s findings — e.g., A0: An Affordance-Aware Hierarchical Model for General Robotic Manipulation, ICCV 2025; and A1: Adaptive Truncated Vision-Language-Action Model from Affordance to Action, CVPR 2026. A brief positioning would improve completeness.

W6. Presentation. Numerous unresolved cross-references (“Appendix ??”) throughout §§3–4 — the task list, execution budgets, probe prompts, and generation statistics cited as evidentiary support are not resolvable in this version of the PDF. Reasoning effort is a provider-soft knob with interface-reported tokens, further qualifying the token-based claims.

Questions for authors

Can you repeat each configuration ≥3 times on the subset and report per-configuration intervals? Does the Known–Unknown gap and the effort non-monotonicity survive? Can the reach/open dissociation be unbundled from the harness — e.g., running one model under both interfaces, or ablating context serialization? For “reached, not opened” episodes: is the correct compartment visually identifiable in the delivered image (affordance-visibility analysis)? This would separate perception failure from decision failure (W3). How does a scripted exhaustive policy (visit and open all certified containers) score on the seven Unknown tasks? Where do the models sit relative to it? Why does Claude adaptive take more decisions (46.1) than fixed high (39.3) yet complete fewer targets — is there loop/dithering behavior worth reporting?

Reviewer_FooUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WorldForge is a household simulation platform for studying how embodied agents allocate reasoning and information-gathering actions. The authors compare tasks with known and unknown target locations in identical scenes across different reasoning-effort settings. Increasing reasoning effort does not consistently improve task completion, and many failures occur because agents never reach or open the containers holding the targets.

Critical strengths
  1. The paper explores an interesting and novel question by connecting reasoning effort with active information acquisition in embodied agents: when information is missing, should an agent think more or act to obtain new evidence?
  2. The experimental analysis is relatively comprehensive, combining Known/Unknown comparisons, fixed and adaptive reasoning effort, offline location probes, and stage-level failure analysis to examine the reasons behind the observed performance.
Critical weaknesses
  1. The main analysis relies on only seven tasks across three homes, with one run per configuration. The authors report that repeating one configuration changes completion from 3/7 to 4/7, making it difficult to distinguish stable effects of reasoning effort from run-to-run variability.
  2. The finding that additional reasoning cannot reliably compensate for missing observations is largely intuitive, and the analysis offers limited new insight into how to address the failures. A simple baseline that systematically inspects likely containers would help establish whether these failures require better reasoning or can be mitigated by a basic search strategy.
Questions for authors

How does a simple search baseline that systematically inspects likely containers perform under the Unknown condition, and can it resolve the reported information-acquisition failures without increasing reasoning effort?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.