- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks what a multi-view world model has to predict so that alternative actions can be compared from a shared current state, and argues that agreement between synchronized views on the current scene does not imply agreement on the finite-horizon change those views attribute to the same physical transition. Stage A learns a predictive coordinate over frozen visual features and aligns, across the agent and wrist views, both the synchronized states and their endpoint displacements, adding an auxiliary context branch and a decorrelation term against collapse. Stage B freezes that coordinate and trains a transformer, conditioned on a candidate action chunk, to predict the resulting displacement, with an executed-versus-mismatched ranking loss over in-batch action chunks. A downstream controller scores local candidates by the distance between the predicted future coordinate and a goal prototype built from successful terminal demonstrations, and replaces the nominal action only when ensemble uncertainty, a damage estimate, a trust region, a nominal-failure classifier and an intervention budget all pass. A policy-only student is distilled from the accepted corrections. Evidence is eight simulated manipulation tasks under paired resets, fixed-state candidate ranking on two tasks, and a protocol that executes all candidates in the simulator and reports rank correlation with the executed outcomes.
Critical strengths
- The transition-level framing earns the architecture. The decomposition shows why approximate state agreement leaves the displacement free, and the matched fixed-capacity control turns that into a measurement, holding State Match at 0.999 while Delta Match moves from 0.786 to 0.988 and demonstrated-action MRR from 0.2748 to 0.4522. Freezing the coordinate before Stage B keeps the two questions separable.
- The physical evaluation is stronger than the offline ranking this area usually reports. Restoring sixty-four simulator states and executing all fifty candidates, then reporting Spearman , Top-1 and regret with state-level bootstrap intervals, validates the scorer against outcomes rather than against another representation. The paper also reports where it loses, including weaker future-state retrieval than DINO-WM.
Critical weaknesses
- The diagnostic that motivates the paper is the quantity Stage A is trained to maximise, so the cross-method comparison does not establish the motivating claim. Delta Match is the mean cosine similarity between the two views' finite-horizon displacements, and the transition loss is with as its second term, so a high Delta Match for the proposed method records a training outcome rather than an independent property. The baselines were never trained on that objective, and the table records the asymmetry, scoring DINO-WM on mean-pooled raw features. Within the same table the statistic does not track the behaviour it is meant to explain. Across the seven baselines the rank correlation between Delta Match and demonstrated-action MRR is , the method with the lowest Delta Match has the highest baseline MRR, and the method with Delta Match 0.4530 has MRR 0.3104. The figure caption concedes that the relation is not monotonic. What survives is the matched control in the appendix, where transition alignment is added inside one fixed-capacity architecture and State Match barely moves while MRR rises by 0.18. That is the evidence supporting the first contribution, and it is not in the main text. Leading with it, and reporting Delta Match only within an architecture, would make the claim as strong as the data.
- The headline comparison measures the world model and the search machinery together, and no baseline row reports the machinery. The complete system scores a hundred candidates per decision, filters them with two learned classifiers trained on additional simulator steps, and applies an intervention budget, while the baselines in the same table are each method's own system, which the caption leaves at one sentence. The authors' own ablation shows how much this matters, since the row labelled without online correction reaches 71.33 against VILA's 70.57. Nothing here says the world model is not better. It says the 5.28-point margin reported for the complete system cannot be attributed to the model, and the paper has what is needed to settle the question, because the candidate bank and the gates are described as identical for every learned scorer and a DINO-WM-scored correction rollout already appears as one qualitative panel. One closed-loop row in which a baseline scorer drives the same search and gating would convert the headline from a system demonstration into evidence about the model.
Questions for authors
- Appendix Table 9 gives the online teacher 82.3 and the student 84.1, while Tables 1 and 5 give the complete online system 84.00 (378/450). Neither is attainable over 450 episodes, and Table 9 shares the Table 5 split, both reporting 77.8 for always intervening. Which denominators, and which number carries the 1.8-point claim?
- Does any closed-loop row place a baseline's own scorer inside the shared bank and gates? The text says every learned scorer receives the same bank and gates.
- The artifact README says source code only, while the text claims configurations, thresholds, optimizer settings, seeds and episode outcomes. Where are those, and the sampling and decoder details the text places in the appendix?
- The Door scorer uses sixteen prediction steps while the reported Stage B horizon is eight. How is that produced, and which horizon does the goal distance use?
- Are the regret columns of Tables 13 and 15 the same quantity? Table 15 puts the model at 0.3293, below nominal BC at 0.3391, while Table 13 reports 0.3801.
4 comments
Part(2/3)
Thank you for your suggestions. We should have explained each experiment's role more clearly. We have revised the manuscript, marking changes in blue. Below we address W1, W2, Q1 and Q2.
W1. Transition-alignment evidence
We agree that Delta Match is linked to the training objective. Our motivating observation is that high cross-view state agreement can coexist with inconsistent transition displacements. Delta Match captures this gap and motivated Stage A; its post-training increase records alignment. Success and Demo MRR remain the primary performance metrics. We have retained cross-method Delta Match in Table 2 to describe the observation; the matched intervention tests the downstream effect of alignment.
We have foregrounded the control (Fig. 1, p. 2; Table 17, p. 24), as suggested. At fixed architecture/capacity, each frozen variant trains its own Stage B without OPD. Results:
| Metric | State-only | + Transition |
|---|---|---|
| State Match ↑ | 0.9994 | 0.9998 |
| Delta Match ↑ | 0.7862 | 0.9884 |
| Demo MRR ↑ | 0.2748 | 0.4522 |
| Physical Spearman ↑ | 0.1518 | 0.3017 |
| Regret ↓ | 0.1442 | 0.0890 |
State Match barely changes, while MRR, physical correlation and Regret improve.
Q2. Baseline scorers in shared OPD
Thank you for proposing this control. We have expanded the G1 Door comparison to seven baseline scorers and TAM-WM. All share nominal BC, risk/safety gates, rescue proposer, 64-local + 36-rescue candidate rule, intervention budget and 450 resets. Run 1–3 vary model training; controller components remain fixed. Entries are success percentages over 450 episodes per run; SD is the sample SD across runs. BC is held fixed across runs.
| Method | Run 1 | Run 2 | Run 3 | Component | Full OPD | Mean ± SD (%) |
|---|---|---|---|---|---|---|
| BC | 78.22% | 78.22% | 78.22% | -- | -- | 78.22 |
| DINO-WM | 75.33% | 76.00% | 77.33% | 8.84/11.84 | 123.04/132.19 | 76.22 ± 1.02 |
| ReViWo | 75.56% | 75.11% | 76.00% | 9.05/10.05 | 133.42/139.44 | 75.56 ± 0.44 |
| MV-MWM | 78.89% | 79.78% | 77.78% | 11.32/12.03 | 126.47/134.05 | 78.81 ± 1.00 |
| MVP-LAM | 78.22% | 79.56% | 80.67% | 57.94/72.49 | 221.17/277.56 | 79.48 ± 1.22 |
| MVISTA-4D | 77.56% | 77.33% | 74.89% | 68.91/69.43 s | 300.36/306.31 s | 76.59 ± 1.48 |
| Ctrl-World | 75.78% | 77.56% | 76.89% | 7.30/7.37 s | 1461.44/1476.59 s | 76.74 ± 0.90 |
| VILA | 76.22% | 77.78% | 76.44% | 49.38/54.62 | 212.48/248.16 | 76.81 ± 0.84 |
| TAM-WM | 84.00% | 83.78% | 83.11% | 10.01/13.98 | 126.51/165.62 | 83.63 ± 0.46 |
TAM-WM reaches 83.63% ± 0.46%, exceeding the strongest baseline, MVP-LAM (79.48% ± 1.22%), by 4.15 pp, DINO-WM by 7.41 pp and BC by 5.41 pp, from unrounded counts. It leads all seven baselines in each run.
The gates use shared simulator labels; rescue uses training demonstrations. Candidate tensors follow each rollout state. Uncertainty thresholds follow the same validation-quantile rule on each scorer's scale. Matching covers additional supervision, candidate count and intervention budget; total model-training compute varies. Table 1's RL policy-only row is a separate deployment experiment.
We have added two p50/p95 columns to Table 19 (ms unless marked s). BC uses no world-model prediction or OPD (--). Full OPD includes observation processing, nominal BC, K=100 candidate evaluation, gates, rescue and final selection. Component timing covers K=100 scoring for DINO/ReViWo/MV-MWM/MVP-LAM/VILA/TAM, one video candidate for Ctrl-World, and one native action chunk for MVISTA-4D.
Q1. Door teacher/student results
Thank you for checking Table 9. We apologize for the inaccurate percentages and incomplete reporting; we have corrected them from episode records. Tables 5/9 use fixed models on the same 450 original Door episodes (0–449), separate from G1 above; we have reported three-run means in Table 1. “Current” is the 378/450 Full OPD teacher; “Earlier” is the earlier conservative-carrier teacher. “Conservative student” abbreviates Conservative-Carrier Student.
| Method | Teacher | Deployment | Search | Success k/450 |
|---|---|---|---|---|
| Nominal BC | — | Policy-only | No | 352/450 (78.22%) |
| Always intervene | Current | Ungated | Yes | 350/450 (77.78%) |
| Online teacher | — | Online OPD | Yes | 378/450 (84.00%) |
| Matched student | Current | Policy-only | No | 355/450 (78.89%) |
| Matched student | Current | Online OPD | Yes | 377/450 (83.78%) |
| Conservative student | Earlier | Policy-only | No | 367/450 (81.56%) |
| Conservative student | Earlier | Online OPD | Yes | 379/450 (84.22%) |
Policy-only students omit online world-model search; +OPD rows include it. The counts do not support the earlier 1.8-point search-free gain claim; we have withdrawn it and report the deployment trade-off. These students are distilled from corrections; Table 1's w/o OPD policy is trained by RL. We have added the matched student's image-to-action timing to Table 9: 0.72/0.90 ms (p50/p95; A100, 100 calls).
We have revised Fig. 1 (p. 2) and the discussion (p. 8) to foreground Table 17 (p. 24), expanded Table 19 (p. 27) with shared-controller results and separate timing scopes, corrected Table 9 (p. 20), and updated the teacher/student discussion (p. 8).
Part(3/3)
Thank you for checking the artifact and evaluation protocols. The original text did not bring the settings together or fully explain the task-specific horizons and Regret definitions. We have revised the manuscript and highlighted the changes in blue. Below we address Q3–Q5 with the settings and corresponding revisions.
Q3. Artifact and implementation settings
For Q3, the original text incorrectly listed per-episode outputs among the supplied materials; the source/configuration archive does not contain them. We have corrected that statement. The sampling and architecture descriptions also needed detail; we list the settings below, separating Can PH defaults from the selected Door configuration.
We have listed the Can PH defaults in Table 20: both stages use , seed 261, batch 256 and 20,000 updates.
| Setting | Stage A | Stage B |
|---|---|---|
| Optimizer / LR / WD | AdamW / 1e-4 / 1e-4 | AdamW / 2e-5 / 1e-4 |
| LR schedule | Constant | Cosine; minimum 10% of initial LR |
| Dimensions | Content 256; context 64; hidden 512 | Residual in frozen content coordinate |
| Dropout | Network 0.1; context 0.2 | 0.1 |
| Loss weights | Align 0.1; transition 0.2; cross/self reconstruction 1/1; orthogonal 0.01; bottleneck 0 | Action contrastive 0.01; variance 0; temperature 0.1 |
We have reported the online gate settings in Table 21. Below are the selected Door settings:
| Gate/control | Setting |
|---|---|
| Trigger rescue (both) | ; |
| Accept rescue (both) | ; |
| Acceptance / distance | Score > 0.400144589683965; Euclidean distance from nominal BC ≤ 0.2 (otherwise score -1e9) |
| Budget / burst | At most steps/episode; at most 3 consecutive steps |
| Continue burst | ; ; budget remains |
| Candidates | 64 base + 36 rescue = 100 |
For the sampling details you requested, we distinguish context pairing, training negatives and evaluation candidates:
| Stage | Rule |
|---|---|
| Stage-A context | Same demo, , deterministic from seed/sample index; cross-combination context pairing |
| Stage-B negatives | Seven in-batch action chunks/positive (eight total, temperature 0.1); circularly shift actions by 1–7, keeping current/target states fixed |
| Can offline bank | Four families: magnitude-matched, action-nearest, state-matched, hybrid; 50 candidates/query, demo index 0, non-demo indices 1–49; shared indices, seed 83083 |
| Hybrid negatives | Round-robin over temporal-adjacent, magnitude-matched, action-nearest, state-matched, other-demo and random pools; deduplicate indices, random fallback |
Regarding network structure, the Stage-A encoder applies LayerNorm(384) → Linear(384,512) → GELU → Dropout(0.1) → Linear(512,512) → GELU per patch, mean-pools, then uses residual-MLP heads for 256-D content and 64-D context, with separate weights. The feature readout reconstructs frozen DINO features:
| Component | Structure |
|---|---|
| Input | Concatenate content/context (320-D); LayerNorm(320), Linear(320,512), GELU, Dropout(0.1) |
| ResidualMLP | LayerNorm(512), Linear(512,1024), GELU, Dropout(0.1), Linear(1024,512); residual addition |
| Patch/output | Broadcast 512-D + learned positions [256,512]; LayerNorm(512), Linear(512,512), GELU, Linear(512,384); output [B,256,384] |
| Views | Separate agent/wrist readouts; identical structures, unshared weights |
Q4. Door prediction horizon
For Q4, we should have stated the horizons explicitly. Door uses to cover progress after handle contact and sustained pushing; the other seven tasks use . It predicts directly. Online, each 4-D candidate is repeated for 16 steps; endpoint MSE to is averaged across three members. Control re-observes every step. Door shares the recovery workflow with Can/Square (p. 18).
Q5. Regret protocols
For Q5, we apologize for the unclear Regret labels. Table 13 uses an expert-centered bank and expert-terminal discrepancy; Table 15 uses a BC-centered bank and task-progress cost . Let . The original Table 13 reports mean raw gap (0.3801); Table 15 averages (TAM-WM 0.3293; BC 0.3391). Constant costs give zero. We have added normalized Regret to Table 13 for all eight methods, using Can's floor: TAM-WM 0.0980, DINO-WM 0.1353. The original raw column remains. Table 15's Selection Efficiency averages the selected/oracle gain ratio, clipped to [0,1], over states with oracle gain above 1e-6.
We have corrected the artifact statement on p. 22 and added settings in Tables 20–23 (pp. 28–29). We have clarified the Door horizon in Table 6 (p. 19), its implementation on p. 21, and the Regret protocols in Tables 13/15 (pp. 23–24) and definitions on pp. 25–26. Thank you for helping us clarify these details.
Thank you again for your time and thoughtful review. As the discussion period is approaching its end, we would like to follow up on our responses and the revised manuscript. We would appreciate hearing whether the clarifications and additional results address your concerns. If any questions remain, we would be happy to discuss them further.
Part(1/3)
Thank you for your careful review and for recognizing the transition-level formulation and simulator-executed evaluation. Your comments helped us see that the relationship between the motivating observation, controlled model evidence and complete-system results needed clearer explanation. We have revised these explanations and added the missing protocol details. These manuscript revisions are highlighted in blue. This overview introduces our response; Parts 2 and 3 give the evidence and settings.
W1. Motivation and controlled evidence. Delta Match captures the motivating observation: strong state agreement can coexist with inconsistent transitions. We agree that its post-training increase also reflects optimization; success/MRR remain primary. Following your suggestion, we have foregrounded the matched control (Fig. 1, p. 2; Table 17, p. 24). At fixed architecture/capacity, each frozen variant trains its own Stage B, with OPD disabled:
State Match barely changes, while action ranking and physical selection improve. Part 2 explains the control.
Q2. Shared-controller comparison. We acknowledge that the original shared-bank/gate scope was unclear. We have added the G1 Door comparison in Table 19 (p. 27): seven baseline scorers and TAM-WM share BC, risk/safety gates, rescue, candidate rules and intervention budgets. Below are results across three training runs (450 episodes/run; BC fixed):
TAM-WM exceeds MVP-LAM by 4.15 pp and DINO-WM by 7.41 pp. Part 2 reports each run, supervision, calibration and timing scopes.
Q1. Teacher/student results. We have corrected Table 9 (p. 20) from the same 450 original Door episodes as Table 5, and updated the discussion (p. 8). “Current” denotes the 84.00% Full OPD teacher; “Earlier” denotes the conservative-carrier teacher. Results are:
These counts distinguish policy-only and online-search deployment. We have withdrawn the earlier 1.8-point search-free gain claim. Part 2 gives the teacher/student details.
Q3. Artifact and implementation settings. For Q3, we corrected the artifact-content statement and added the training, gate, sampling and encoder/readout settings.
Q4–Q5. Horizons and Regret protocols. For Q4–Q5, we explain direct 16-step Door prediction and the common recovery and metric definitions across tasks. We have added Can's normalized-Regret definition to Table 13, retaining its original raw gaps. Tables 13/15 use distinct candidate banks and physical references, which we now explain explicitly.
Manuscript revisions. We have revised Figure 1 (p. 2) and the matched-intervention discussion (p. 8) to foreground the control in Table 17 (p. 24), and added the shared-controller comparison in Table 19 (p. 27). We have corrected Table 9 (p. 20) and updated the teacher/student discussion (p. 8). We have also corrected the artifact statement (p. 22), added settings in Tables 20–23 (pp. 28–29), and clarified the Door horizon in Table 6 (p. 19) and its implementation (p. 21). The Regret protocols are now explained in Tables 13/15 (pp. 23–24) and the definitions on pp. 25–26. Thank you for helping us make the connection between evidence and claims clearer.