- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Words to World introduces W2W-Bench, evaluating complete model–harness systems across eleven task families. Tasks cover active observation, time-series analysis, device control, construction, experiments, and planning, with independent outcome checks and process records used to assess performance. The paper further interprets observed failures in terms of spatial, temporal, coordination, and recovery limitations.
Critical strengths
Eleven task families examine observation, action, and feedback within a common evaluation framework, exposing system problems that single-turn tests can miss. Outcome evaluators kept outside the agent loop and independent functional LEGO tests ground success judgments in task requirements. Together with process records, this resource can support comparisons of complete model–harness configurations and inspection of failures.
Critical weaknesses
- The claimed real-world contribution does not match the reported evaluation scope. The abstract distinguishes the benchmark from simulated tasks and explicitly describes closed-loop interaction with real robotic arms, cameras, and sensors. Appendix C instead places several central tasks, including scanning, LEGO construction, laboratory operation, and tabletop games, in simulation, while the health task uses recorded data. The paper does not map the reported scores to hardware, simulation, or replay conditions.
- The cross-task diagnosis lacks explicit and auditable failure attribution. Reconstruction and assembly errors in Figures 4–5 do not distinguish a missed geometric deviation from an incorrect correction parameter or unsuccessful execution. Likewise, the lack of a relationship between occlusion and exploration steps in Figure 6 does not establish insufficient evidence gathering. The paper uses these observations to identify millimeter-scale perception and evidence-acquisition bottlenecks without corresponding trace analysis or controlled diagnosis. Recovery accounting is also unresolved: the three categories in Table 5 sum to 6 and 11 cases despite a denominator of 10 for each configuration, with no explanation of missing or overlapping cases. The analysis does not yet reliably map the observed failures to the claimed capability limitations.
Questions for authors
- Please clarify which tasks were run on real devices, which were run in simulation, and which used previously collected data. Which scores in Table 1 come from actual hardware runs and how many trials support them? Based on this mapping, which tasks provide the paper's additional evidence about real physical interaction?
- For the assembly deviations in Figure 5, could you show an existing failure trace with the observations supplied to the model, the generated correction parameters, and the executed outcome, explaining how perceptual, parameter-planning, and execution errors are distinguished? Please also clarify whether the three categories in Table 5 are mutually exclusive and provide complete counts accounting for all ten cases in each configuration.
3 comments
Q4. Are the Table 5 categories mutually exclusive, and how do they account for all ten episodes per configuration?
Thank you for flagging the ambiguity. Each anomaly event receives one outcome, but an episode can contain multiple events with different outcomes. For example, in Opus 5 episode 08, the agent successfully corrected one anomaly and later began preparing a repair for a second anomaly before the run ended. That episode therefore counts once under successful local recovery and once under unsuccessful or incomplete recovery.
We retained the episode as the unit of analysis. Each cell gives the number of distinct episodes out of ten, followed by their IDs.
| Configuration | Successful local recovery | Unsuccessful or incomplete recovery | Undetected or misinterpreted anomaly | No annotated anomaly |
|---|---|---|---|---|
| Codex CLI + GPT-6 Astra | 4/10 (01, 08, 09, 10) | 1/10 (07) | 1/10 (03) | 4/10 (02, 04, 05, 06) |
| Claude Code + Opus 5 | 5/10 (02, 04, 07, 08, 10) | 4/10 (06, 07, 08, 10) | 2/10 (02, 07) | 4/10 (01, 03, 05, 09) |
Thank you for the detailed response. The task breakdown clarifies the evaluation settings, and the episode IDs resolve my concern about the Table 5 accounting. I also appreciate the explicit withdrawal of the under-/over-exploration inference from Figure 6. The LEGO examples provide useful episode-specific evidence. My remaining concern is how these findings relate to the broader capability claims. In particular, an orientation misinterpretation does not directly establish a millimeter-scale perception limitation, and the unsuccessful release establishes an unintended outcome without identifying its underlying cause. Could you confirm whether those broader claims will also be narrowed consistently throughout the paper? Brief supporting excerpts of the observations, commands, and resulting states would help make the case analyses auditable. These clarifications improve my assessment. I will reassess the recommendation based on the benchmark’s contribution under the clarified evaluation scope and the conclusions supported by the evidence.
Thank you for your careful review! We appreciate your questions and address each of them in detail below.
Q1. Which tasks used physical hardware, simulation, or previously collected data? What evidence does Table 1 provide about real-world physical interaction?
We did all the experiments in the simulation. The evaluation settings were as follows.
We believe our simulation results help us design targeted tests on physical hardware. For example, the LEGO traces suggest testing upright and inverted bricks under matched camera conditions, recording the agent’s orientation estimate before it moves, and checking the brick’s pose after the grasp. This would let us distinguish a visual misinterpretation from a correctly planned action that fails during execution. Similarly, the SimuHome results suggest checking both whether a real device accepts a scheduled command and whether it reaches the requested state by the deadline. These findings identify specific failure modes to test and measurements to collect. Their frequency and impact on real hardware remain to be established.
We will add this explanation and revise the abstract and introduction to describe the evaluation settings accurately.
Q2. What do Figures 4–5 establish, and how can an existing trace distinguish perceptual, parameter-selection, and execution errors?
Thank you for asking us to clarify how we attribute failures. Figure 4 reports reconstruction F-scores at specified geometric tolerances. Figure 5 reports lateral and insertion-depth errors in the final assembly. To investigate the cause of a deviation, we compare what the agent observed, the command it issued, and the resulting simulator state. Three LEGO episodes illustrate the distinctions we can make.
Perception error (
basic_10, Claude Code + Opus 5). At step 36, the agent inspected a fallen brick through its wrist camera. It estimated the brick’s center to within 0.2 mm of its recorded position and reported seeing its 2×4 studs. At that same instant, the simulator recorded a 180° tilt, with the studs facing down. The discrepancy was present before the agent issued its recovery command. It selected a grasp with a yaw of 0.42 radians and returned the brick to the platform without turning it over. The base remained inverted as the agent built the next layer. This localizes the earliest identifiable recovery error to the agent’s interpretation of which face was up.Parameter-selection error(
basic_03, Codex CLI + GPT-6 Astra). At step 3, the agent requested a grasp height of 6 mm. The workspace validator rejected the command before any motion occurred. On retry, the agent changed only the height to 12 mm, and the gripper reached 12.2 mm. This single-parameter change identifies the requested height as the reason for the rejection.Execution error (
basic_01, Codex CLI + GPT-6 Astra). At step 27, the agent commanded the gripper to open and withdraw to 65 mm. The command executed, but the upper brick rose 18.95 mm with the gripper while the base stayed in place. The agent detected the lift, lowered the brick, closed the jaws, and withdrew successfully. The trace establishes that the first withdrawal failed to release the brick as intended. The precise contact mechanism remains uncertain.We will revise Section 3.2 to distinguish these episode-specific findings from the aggregate measurements in Figures 4–5.
Q3. Does Figure 6 show that agents gathered insufficient visual evidence?
We agree that Figure 6 alone cannot establish this. We found no significant association between occlusion and the number of exploration steps. One well-chosen view may reveal the relevant feature, while several similar views may leave it occluded. We will remove the inference that agents systematically under-explored difficult scenes or over-inspected easy ones.