acceptodds
Under review as a conference paper at ICLR 2027

Words to World: Benchmarking LLMs in Interactive Physical Environments

Abstract

Words are increasingly becoming an interface for action: large language models (LLMs) can now invoke tools, operate devices, and make decisions that affect the physical world. However, current benchmarks largely evaluate LLMs through static text responses, offline tool calls, or simulated tasks, leaving open the question of how well they can act in real physical environments. We present a benchmark for measuring LLMs in closed-loop physical interaction, where models control a harness system connected to real-world components including robotic arms, cameras, and sensors. In each task, the model must issue instructions, receive physical feedback from the environment, interpret the resulting state, and adapt its next action accordingly. This setting evaluates capabilities beyond language understanding, including grounded perception, action planning, tool selection, spatial reasoning, feedback-driven correction, and robustness to execution uncertainty. Through experiments with representative LLMs, we characterize their ability to translate linguistic reasoning into physical actions and identify common failure modes such as hallucinated state changes, poor recovery from unexpected outcomes, and unstable long-horizon plans. Our benchmark aims to provide a systematic foundation for evaluating LLMs as interactive agents that connect words, tools, and the physical world.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Physical Grounding Simulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8oUUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Words to World introduces W2W-Bench, evaluating complete model–harness systems across eleven task families. Tasks cover active observation, time-series analysis, device control, construction, experiments, and planning, with independent outcome checks and process records used to assess performance. The paper further interprets observed failures in terms of spatial, temporal, coordination, and recovery limitations.

Critical strengths

Eleven task families examine observation, action, and feedback within a common evaluation framework, exposing system problems that single-turn tests can miss. Outcome evaluators kept outside the agent loop and independent functional LEGO tests ground success judgments in task requirements. Together with process records, this resource can support comparisons of complete model–harness configurations and inspection of failures.

Critical weaknesses
  1. The claimed real-world contribution does not match the reported evaluation scope. The abstract distinguishes the benchmark from simulated tasks and explicitly describes closed-loop interaction with real robotic arms, cameras, and sensors. Appendix C instead places several central tasks, including scanning, LEGO construction, laboratory operation, and tabletop games, in simulation, while the health task uses recorded data. The paper does not map the reported scores to hardware, simulation, or replay conditions.
  2. The cross-task diagnosis lacks explicit and auditable failure attribution. Reconstruction and assembly errors in Figures 4–5 do not distinguish a missed geometric deviation from an incorrect correction parameter or unsuccessful execution. Likewise, the lack of a relationship between occlusion and exploration steps in Figure 6 does not establish insufficient evidence gathering. The paper uses these observations to identify millimeter-scale perception and evidence-acquisition bottlenecks without corresponding trace analysis or controlled diagnosis. Recovery accounting is also unresolved: the three categories in Table 5 sum to 6 and 11 cases despite a denominator of 10 for each configuration, with no explanation of missing or overlapping cases. The analysis does not yet reliably map the observed failures to the claimed capability limitations.
Questions for authors
  1. Please clarify which tasks were run on real devices, which were run in simulation, and which used previously collected data. Which scores in Table 1 come from actual hardware runs and how many trials support them? Based on this mapping, which tasks provide the paper's additional evidence about real physical interaction?
  2. For the assembly deviations in Figure 5, could you show an existing failure trace with the observations supplied to the model, the generated correction parameters, and the executed outcome, explaining how perceptual, parameter-planning, and execution errors are distinguished? Please also clarify whether the three categories in Table 5 are mutually exclusive and provide complete counts accounting for all ten cases in each configuration.
3 comments

Thank you for your careful review! We appreciate your questions and address each of them in detail below.

Q1. Which tasks used physical hardware, simulation, or previously collected data? What evidence does Table 1 provide about real-world physical interaction?

We did all the experiments in the simulation. The evaluation settings were as follows.

Evaluation settingTask families
Physics simulation or renderingDocument Image Capture, 3D Scanning, Object Recognition, 3D Video Generation, LEGO Building, Functional LEGO Design, and Tabletop Games
Task-specific simulationSimuHome and Automated Biology Laboratory (AutoBio, using Fast Twin)
Previously collected dataMultipage QA (MP-DocVQA documents) and Health Time-Series (MIMIC-III Demo physiological recordings)

We believe our simulation results help us design targeted tests on physical hardware. For example, the LEGO traces suggest testing upright and inverted bricks under matched camera conditions, recording the agent’s orientation estimate before it moves, and checking the brick’s pose after the grasp. This would let us distinguish a visual misinterpretation from a correctly planned action that fails during execution. Similarly, the SimuHome results suggest checking both whether a real device accepts a scheduled command and whether it reaches the requested state by the deadline. These findings identify specific failure modes to test and measurements to collect. Their frequency and impact on real hardware remain to be established.

We will add this explanation and revise the abstract and introduction to describe the evaluation settings accurately.

Q2. What do Figures 4–5 establish, and how can an existing trace distinguish perceptual, parameter-selection, and execution errors?

Thank you for asking us to clarify how we attribute failures. Figure 4 reports reconstruction F-scores at specified geometric tolerances. Figure 5 reports lateral and insertion-depth errors in the final assembly. To investigate the cause of a deviation, we compare what the agent observed, the command it issued, and the resulting simulator state. Three LEGO episodes illustrate the distinctions we can make.

  • Perception error (basic_10, Claude Code + Opus 5). At step 36, the agent inspected a fallen brick through its wrist camera. It estimated the brick’s center to within 0.2 mm of its recorded position and reported seeing its 2×4 studs. At that same instant, the simulator recorded a 180° tilt, with the studs facing down. The discrepancy was present before the agent issued its recovery command. It selected a grasp with a yaw of 0.42 radians and returned the brick to the platform without turning it over. The base remained inverted as the agent built the next layer. This localizes the earliest identifiable recovery error to the agent’s interpretation of which face was up.

  • Parameter-selection error(basic_03, Codex CLI + GPT-6 Astra). At step 3, the agent requested a grasp height of 6 mm. The workspace validator rejected the command before any motion occurred. On retry, the agent changed only the height to 12 mm, and the gripper reached 12.2 mm. This single-parameter change identifies the requested height as the reason for the rejection.

  • Execution error (basic_01, Codex CLI + GPT-6 Astra). At step 27, the agent commanded the gripper to open and withdraw to 65 mm. The command executed, but the upper brick rose 18.95 mm with the gripper while the base stayed in place. The agent detected the lift, lowered the brick, closed the jaws, and withdrew successfully. The trace establishes that the first withdrawal failed to release the brick as intended. The precise contact mechanism remains uncertain.

We will revise Section 3.2 to distinguish these episode-specific findings from the aggregate measurements in Figures 4–5.

Q3. Does Figure 6 show that agents gathered insufficient visual evidence?

We agree that Figure 6 alone cannot establish this. We found no significant association between occlusion and the number of exploration steps. One well-chosen view may reveal the relevant feature, while several similar views may leave it occluded. We will remove the inference that agents systematically under-explored difficult scenes or over-inspected easy ones.

Q4. Are the Table 5 categories mutually exclusive, and how do they account for all ten episodes per configuration?

Thank you for flagging the ambiguity. Each anomaly event receives one outcome, but an episode can contain multiple events with different outcomes. For example, in Opus 5 episode 08, the agent successfully corrected one anomaly and later began preparing a repair for a second anomaly before the run ended. That episode therefore counts once under successful local recovery and once under unsuccessful or incomplete recovery.

We retained the episode as the unit of analysis. Each cell gives the number of distinct episodes out of ten, followed by their IDs.

ConfigurationSuccessful local recoveryUnsuccessful or incomplete recoveryUndetected or misinterpreted anomalyNo annotated anomaly
Codex CLI + GPT-6 Astra4/10 (01, 08, 09, 10)1/10 (07)1/10 (03)4/10 (02, 04, 05, 06)
Claude Code + Opus 55/10 (02, 04, 07, 08, 10)4/10 (06, 07, 08, 10)2/10 (02, 07)4/10 (01, 03, 05, 09)

Thank you for the detailed response. The task breakdown clarifies the evaluation settings, and the episode IDs resolve my concern about the Table 5 accounting. I also appreciate the explicit withdrawal of the under-/over-exploration inference from Figure 6. The LEGO examples provide useful episode-specific evidence. My remaining concern is how these findings relate to the broader capability claims. In particular, an orientation misinterpretation does not directly establish a millimeter-scale perception limitation, and the unsuccessful release establishes an unintended outcome without identifying its underlying cause. Could you confirm whether those broader claims will also be narrowed consistently throughout the paper? Brief supporting excerpts of the observations, commands, and resulting states would help make the case analyses auditable. These clarifications improve my assessment. I will reassess the recommendation based on the benchmark’s contribution under the clarified evaluation scope and the conclusions supported by the evidence.

Reviewer_qnUWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The motivation of this paper is to build a "true physical-grounded" benchmark where the LLM can be evaluated in closed-loop physical interactions; The delivery is a suite of 11 tasks covering spatial understanding, temporal understanding, low-level execution etc. The paper evaluates mostly latest gpt and claude models, and find that harness can significantly change model performance; successful tool-using does not guarantee precise execution.

Critical strengths
  • the benchmark covers a board collection of interactive tasks across different domains
Critical weaknesses
  • I find it hard to convince myself that this is a useful benchmark according to what is being motivated in the abstract and intro:

    • the statement that "current benchmarks largely evaluate LLMs through static text responses, offline tool calls, or simulated tasks, leaving open the question of how well they can act in real physical environments." is not valid; back in 2021 Alfworld-like benchmark already provides real interaction capabilities connecting LLM to interacting in a physical environment; very early papers such as Saycan also evaluates LLM-backed robots directly in real world environments
    • a lot of the tasks included in W2W benchmark shows very little "physical" property, e.g., health time-series; multipage QA, etc; so it makes readers very confused on the motivation of putting up this suite
    • the main contribution of this paper seems to be aggregating diverse tasks from existing environments; datasets
  • Although there are quite a few observations from the evaluation results; I found very few actionable insights, e.g., we pinpoint the problem to an aspect that prior benchmarks does not show, and what might be a good way to address. Most observations are intuitive, e.g., harness matters;

Questions for authors

N/A

2 comments

Thank you for recognizing the breadth of the interactive tasks. We address each of your concerns below.

Q1. How does W2W-Bench relate to prior interactive benchmarks and real-robot evaluations?

We agree that our opening statement was too broad. ALFWorld already evaluates agents through interaction in simulated households, and SayCan demonstrated language-guided execution on real robots. The question we set out to study is how a complete model–harness system chooses observations, acts on feedback, and verifies outcomes across different phyiscal tasks. For example, with GPT-6 Astra held fixed, Pi scored higher than Codex CLI on 3D Scanning (77.91 vs. 62.68), while Codex CLI scored higher on AutoBio (73.80 vs. 58.97). This reversal illustrates why we evaluate a configuration across several settings. The trajectories then let us examine what happened in each setting, including cases where a tool accepted a command but the intended outcome never occurred.

Q2. What did you build on top of existing resources, and why include Multipage QA and Health Time-Series?

We use existing resources for assets, records, and simulation environments. Our task designs specify what decisions the agent must make, what feedback it can use, and how the evaluator checks the result. For example, ABO supplies the product models for our five-shot capture task. We define the photography assignment, let the agent revise its shots after inspection, and evaluate both image quality and the feasibility of moving between camera poses. In Functional LEGO, public parts provide the inventory. The agent chooses a design and assembly sequence, and an independent simulation tests whether the released structure performs its intended function under a load or perturbation. We will make these contributions explicit alongside adaptations of established environments such as AutoBio and SimuHome.

Multipage QA and Health Time-Series examine observation choices that a physical agent also faces. Multipage QA asks where to look: the agent selects pages, can magnify a region, and earns 25% of its score by visiting and correctly citing the supporting page. Health Time-Series asks when to measure: the agent selects a signal and time window and decides when it has enough evidence to stop. These tasks support conclusions about visual evidence acquisition and temporal observation policies. Scanning, assembly, and timed device control provide the more direct tests of geometry, manipulation, and intervention.

Q3. What actionable insights emerge beyond the observation that harness choice matters?

One finding concerns what a harness should verify after scheduling an action. In SimuHome, some agents failed to register the required workflow. In other episodes, registration succeeded, yet the device still failed to reach its target state by the checkpoint. The two failures call for checks at different times. Reading back the schedule immediately can detect a missing workflow. Detecting the second failure requires observing the device state later, while there is still time to intervene. This distinction comes directly from comparing the tool response with the subsequent environment state. It identifies a specific harness change whose benefit can be tested.

A second finding concerns the cost of acquiring evidence. On the same 30 recorded health cases, Codex CLI × GPT-5.6 Sol selected 6.4 hours of observations on average and answered 17 cases correctly. A fixed nine-hour window yielded 16 correct answers, and the two policies agreed on correctness in 29 cases. The one discordant case provides little evidence of an accuracy advantage. The clearer finding is that reporting accuracy alone would conceal a 2.6-hour difference in observation use. We will foreground these results in Section 3.2 and distinguish the behaviors established by the traces from proposed improvements that still require testing.

I thank the reviewers for the response and additional information; after examine the response, I still feel that applying different harnesses to existing benchmarks does not have sufficient contributoin. also in response to Q3 I was actually expecting to see some explanation or insights relates the Pi vs Codex observation as mentioned in response to Q1 (as in given a task, how one may choose the best harness? what metric should I consider etc) but it does not seem to be covered; Overall I am keeping my score based considering the original manuscript content and additional rebuttal clarifications.

Reviewer_TehAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces W2W-Bench, a benchmark for evaluating complete model–harness systems on language-specified tasks in physical or simulated environments, motivated by the observation that tool access alone does not ensure reliable task completion—the harness must close the loop between issuing device commands, observing heterogeneous feedback, and verifying intended effects; its contributions are (1) a capability-oriented framework organizing tasks around four interacting dimensions (3D spatial understanding, temporal understanding, low-level execution, and high-level planning), (2) a suite of 11 device-grounded task families spanning cameras, robot arms, lab instruments, health monitors, and smart-home devices, evaluated with task-specific outcome metrics plus process-level diagnostics under an independent evaluator with hidden ground truth, and (3) an empirical study of multiple model–harness configurations that identifies complementary strengths, recurring failure patterns, and task-dependent harness differences. Validation pairs harnesses (Codex CLI, Claude Code, Pi, Hermes, Factory Droid) with LLMs (GPT-5.6 variants, GPT-6 Astra, Claude Opus 5, Qwen 3.8 Max) and finds that no configuration dominates: GPT-6 Astra excels at planning and execution but struggles with basic spatial/temporal understanding, whereas Claude Opus 5 with Claude Code performs best overall without winning any single task; harnesses show complementary strengths, and documented failures include undetected millimeter-scale spatial deviations, exploration effort not scaling with occlusion severity, model- and harness-dependent temporal observation trade-offs, differing dual-arm coordination failures, and missed physical anomalies.

Critical strengths

The strongest and most consequential contribution is the Section 3.2 diagnostic analysis of why LLM–harness systems fail in closed-loop physical interaction, rather than merely reporting aggregate scores. By dissecting the full loop, perception, i.e., evidence acquisition, timing, execution, and recovery, the authors surface concrete, mechanism-level failure patterns: millimeter-scale spatial deviations go undetected even with multi-view feedback (low 3D-scanning F-scores at 10 mm tolerance; LEGO pieces overlapping rather than interlocking); exploration effort does not scale with occlusion severity (no significant correlation between occlusion ratio and exploration steps); temporal observation windows trade off against diagnostic accuracy in a model- and harness-dependent way (Pi × Opus 5 uses 48 h for only 55.6% Exact P@1, versus Codex × Sol's 56.7% with 6.4 h), showing more observation is not uniformly better; dual-arm failures differ qualitatively by model (GPT-6 dominated by collision avoidance, Opus 5 by insufficient coordination and sequence misalignment); and physical anomalies such as a 180°-flipped brick are sometimes undetected or unrecovered. This is consequential because it turns the benchmark from a ranking table into a diagnostic instrument, yielding actionable guidance for harness design (better observation, verification, and feedback-driven correction) and substantiating the paper's central thesis that the harness, not just the model, determines whether language plans become reliable physical actions.

Critical weaknesses

The main concern is that W2W-Bench appears to be a stitched-together assemblage of existing datasets and environments rather than a coherently designed benchmark, despite its claimed four-dimension capability framework. The 11 task families largely reuse prior resources (ModelNet10/ABO20, MP-DocVQA, MIMIC-III, AutoBio, SimuHome, public LEGO assets) with only 10–30 generated cases per system, and the four dimensions look like post-hoc labels for a task's "primary demand" rather than design principles that generated the tasks.

I would prefer to see a coherent interactive environment evaluation from which these capability dimensions can be decomposed internally, rather than a collection of different benchmarks placed together. This is because the issues exposed on tasks such as multipage question answering and health time-series analysis do not necessarily accurately reveal the 3D spatial understanding and temporal understanding problems in real physical interaction.

Questions for authors

None.

1 comment

We appreciate your constructive review and address each of your points in detail below.

Q1. Task originality, design principles, and sample size

We chose the four dimensions before constructing the tasks. Each informed a decision we wanted the agent to make: where to look for visual evidence, when and for how long to observe an evolving process, how to translate a goal into an executable command and verify its effect, and how to plan or revise a sequence of actions. Figure 1 outlines this framework. We will make the connection between each design choice and the resulting task more explicit in Section 2.

W2W-Bench uses public assets, datasets, and simulators as starting points. Our contribution includes the interactive tasks, the tools through which agents observe and act, and the independent checks of their outcomes. For example, ABO supplies the product meshes for our five-shot capture task. We define the photography assignment, let the agent inspect and revise its shots, and evaluate both image quality and the feasibility of moving between camera poses. In Document Image Capture, we generate scenes with glare, shadows, and occlusions so the agent must decide how to adjust the camera before trying again. In our AutoBio adaptation, we use Fast Twin to generate episodes from ten drug-screening templates. Missing consumables, delays, and partial action effects require the agent to inspect what happened before choosing a recovery action. In Tabletop Games, a legal move advances the game only after the corresponding simulated robotic transfer succeeds. We will identify what each source provides and what we built or adapted for each family.

The 10–30 cases per setting allow us to trace specific failures from an observation through an action to its outcome. We agree that estimates of failure frequency and differences between closely ranked configurations are less precise at this scale. We will reflect that limitation in our interpretation of the results.

Q2: Examining capabilities within a coherent interactive environment

Your suggestion is especially relevant to our two LEGO tasks, which share a simulated dual-arm workspace with wrist-camera feedback. Basic LEGO provides a target structure and construction steps, allowing us to trace how the agent executes and verifies a given plan. Functional LEGO provides a functional objective and a parts inventory. The agent must choose both the design and the assembly sequence, after which an independent simulation applies task-specific loads or perturbations to the released structure. This pairing makes the additional design decisions visible while retaining common perception and manipulation requirements.

We can also distinguish outcomes within a single build. In one Basic LEGO episode, the agent noticed that withdrawing its gripper had lifted the upper brick by about 19 mm. It lowered the brick and withdrew again without lifting it. The released stack was stable, yet none of the eight required studs met the geometric insertion criterion. The trace therefore shows that the agent detected and corrected the displacement, while the evaluator identifies the connection as the requirement that remained unmet. We will give this within-environment analysis greater prominence.

Q3. What Multipage QA and Health Time-Series measure

We agree that the original category headings invite a broader interpretation than these two tasks support. Multipage QA examines where to look for visual evidence. The agent can select pages and magnify regions, and 25% of its score depends on visiting and correctly citing the supporting page. This measures active inspection of document images. Scanning and assembly provide the direct tests of 3D geometry. We will broaden the heading to Active Visual Perception and 3D Spatial Understanding to make that scope clear.

Health Time-Series examines when another observation is worth acquiring. The agent chooses a signal and an observation window and decides when it has sufficient evidence to stop. Replay allows different observation policies to be compared on the same underlying physiological trajectory. SimuHome extends this question to intervention by checking whether a device reaches the required state at the specified time. We will clarify that the four dimensions describe decisions that can overlap within an episode, and that each task score supports conclusions only about the decisions its protocol actually tests.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.