- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper introduces WorldForge, a household simulation platform with generated environments, simulation-verified interactions, and replayable world states. It studies whether increasing reasoning effort helps embodied agents recover missing information by comparing otherwise identical tasks with and without target-object locations. Experiments evaluate GPT-6 Astra through Codex CLI and Claude Opus 5.5 through Claude Code under three fixed reasoning settings and an adaptive setting. Although the study includes 12 tasks across five homes, its central analysis uses seven selected tasks across three homes. The authors report that additional reasoning does not consistently improve completion, failures frequently occur before targets become visible, and adaptive reasoning reduces GPT’s reasoning-token usage by 78% while matching its best observed completion.
Critical strengths
-
Controlled information comparison. The Known/Unknown pairing preserves the initial world state and task requirements while changing the supplied location information. This provides a useful experimental design for examining the consequences of missing information.
-
Informative trajectory analysis. Separating failures into acquisition, use, and execution stages provides more insight than aggregate completion rates alone. The distinction between reaching the relevant furniture and opening the correct compartment is particularly useful.
Critical weaknesses
-
Limited significance of the research motivation. The paper does not convincingly establish why varying native reasoning effort on household search tasks addresses a sufficiently significant research gap. Its main observation—that agents must perform appropriate interactions to reveal hidden objects—provides limited actionable insight into how better agents should be designed. The broader motivation of choosing among remembering, thinking, and looking is also not fully investigated: the experiments keep the environment static and do not manipulate memory freshness or the environmental cost of deliberation.
-
Insufficient methodological novelty. The technical contribution primarily consists of integrating scene generation, interaction verification, and checkpoint replay into a simulation platform, followed by comparisons across information conditions and reasoning settings. The manuscript does not clearly establish what substantive methodological advance this combination provides. For a platform contribution, a convincing comparison with existing environments is particularly important to demonstrate which new experimental capabilities WorldForge enables. The current evaluation does not sufficiently establish this distinction.
-
Insufficient comparison with relevant baseline methods. The evaluation compares only two model–interface pairs under different reasoning settings, without including alternative strategies for embodied information acquisition. Relevant baselines would include active-exploration policies, agents with explicit uncertainty tracking, and simple heuristics that systematically inspect plausible closed containers.
-
Broken cross-references to the supplementary material. Although the appendix is provided in a separate file, the main manuscript contains unresolved references such as “Appendix ??” and “Table ??”. These broken references make it difficult to locate the corresponding benchmark results, prompts, action specifications, execution budgets, and supporting analyses. The authors should repair the links and explicitly identify the relevant appendix sections and tables so that readers can readily trace the evidence supporting the paper’s claims.
Questions for authors
-
How does adaptive reasoning compare with fixed-low reasoning under the same cost-aware prompt? Does it retain an advantage when average reasoning-token usage is matched?
-
Would a simple policy that systematically inspects plausible closed containers substantially improve completion? Can interventions supplying the correct furniture or compartment distinguish spatial uncertainty from failures to perform information-gathering actions?
-
Which conclusions are intended to generalize to memory use and dynamic environments? Can you either provide experiments manipulating memory freshness and deliberation time or narrow the framing to the static setting actually evaluated?