acceptodds
Under review as a conference paper at ICLR 2027

TAM-WM: Transition-Aligned Multi-View World Model for Visual Robot Manipulation

Abstract

Multi-view observations provide complementary information for visual robot manipulation, but they also require a world model to represent the consequences of an action consistently across viewpoints. Existing approaches often encourage synchronized observations to have similar representations, which primarily enforces agreement about the current state. For action selection, however, the model must also assign consistent finite-horizon changes to the same physical transition; strong state agreement alone does not guarantee such transition agreement. To address the issue of inconsistent transition representations across viewpoints, we propose TAM-WM, which learns a shared visual predictive coordinate in two stages. Stage A aligns the displacements induced by the same observed physical transition across synchronized views. Stage B freezes this coordinate and learns state- and action-conditioned residual dynamics, enabling candidate actions from a shared current state to be compared through their predicted changes. A downstream online correction teacher uses these predictions to score local proposals and selectively correct a nominal policy action. On Can PH, a fixed-scorer ablation shows that removing transition alignment reduces demonstrated-action ranking, while the learned scores correlate with simulator-executed candidate outcomes. Across eight simulated manipulation tasks, the complete system achieves the highest equal-weight average success-rate point estimate among the evaluated methods: 75.85%, 5.28 percentage points above VILA and 8.99 points above the behavior-cloning anchor.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models, Robot Manipulation, Multi-View Learning, Transition-Aligned Representation Learning, Action-Conditioned Dynamics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZSuJOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks what a multi-view world model has to predict so that alternative actions can be compared from a shared current state, and argues that agreement between synchronized views on the current scene does not imply agreement on the finite-horizon change those views attribute to the same physical transition. Stage A learns a predictive coordinate over frozen visual features and aligns, across the agent and wrist views, both the synchronized states and their endpoint displacements, adding an auxiliary context branch and a decorrelation term against collapse. Stage B freezes that coordinate and trains a transformer, conditioned on a candidate action chunk, to predict the resulting displacement, with an executed-versus-mismatched ranking loss over in-batch action chunks. A downstream controller scores local candidates by the distance between the predicted future coordinate and a goal prototype built from successful terminal demonstrations, and replaces the nominal action only when ensemble uncertainty, a damage estimate, a trust region, a nominal-failure classifier and an intervention budget all pass. A policy-only student is distilled from the accepted corrections. Evidence is eight simulated manipulation tasks under paired resets, fixed-state candidate ranking on two tasks, and a protocol that executes all candidates in the simulator and reports rank correlation with the executed outcomes.

Critical strengths
  1. The transition-level framing earns the architecture. The decomposition Δcta−Δctw=et+H−et\Delta c^a_t - \Delta c^w_t = e_{t+H} - e_t shows why approximate state agreement leaves the displacement free, and the matched fixed-capacity control turns that into a measurement, holding State Match at 0.999 while Delta Match moves from 0.786 to 0.988 and demonstrated-action MRR from 0.2748 to 0.4522. Freezing the coordinate before Stage B keeps the two questions separable.
  2. The physical evaluation is stronger than the offline ranking this area usually reports. Restoring sixty-four simulator states and executing all fifty candidates, then reporting Spearman ρ\rho, Top-1 and regret with state-level bootstrap intervals, validates the scorer against outcomes rather than against another representation. The paper also reports where it loses, including weaker future-state retrieval than DINO-WM.
Critical weaknesses
  1. The diagnostic that motivates the paper is the quantity Stage A is trained to maximise, so the cross-method comparison does not establish the motivating claim. Delta Match is the mean cosine similarity between the two views' finite-horizon displacements, and the transition loss is dϵ(Δca,Δcw)d_\epsilon(\Delta c^a, \Delta c^w) with 1−cos⁡1-\cos as its second term, so a high Delta Match for the proposed method records a training outcome rather than an independent property. The baselines were never trained on that objective, and the table records the asymmetry, scoring DINO-WM on mean-pooled raw features. Within the same table the statistic does not track the behaviour it is meant to explain. Across the seven baselines the rank correlation between Delta Match and demonstrated-action MRR is −0.46-0.46, the method with the lowest Delta Match has the highest baseline MRR, and the method with Delta Match 0.4530 has MRR 0.3104. The figure caption concedes that the relation is not monotonic. What survives is the matched control in the appendix, where transition alignment is added inside one fixed-capacity architecture and State Match barely moves while MRR rises by 0.18. That is the evidence supporting the first contribution, and it is not in the main text. Leading with it, and reporting Delta Match only within an architecture, would make the claim as strong as the data.
  2. The headline comparison measures the world model and the search machinery together, and no baseline row reports the machinery. The complete system scores a hundred candidates per decision, filters them with two learned classifiers trained on additional simulator steps, and applies an intervention budget, while the baselines in the same table are each method's own system, which the caption leaves at one sentence. The authors' own ablation shows how much this matters, since the row labelled without online correction reaches 71.33 against VILA's 70.57. Nothing here says the world model is not better. It says the 5.28-point margin reported for the complete system cannot be attributed to the model, and the paper has what is needed to settle the question, because the candidate bank and the gates are described as identical for every learned scorer and a DINO-WM-scored correction rollout already appears as one qualitative panel. One closed-loop row in which a baseline scorer drives the same search and gating would convert the headline from a system demonstration into evidence about the model.
Questions for authors
  1. Appendix Table 9 gives the online teacher 82.3 and the student 84.1, while Tables 1 and 5 give the complete online system 84.00 (378/450). Neither is attainable over 450 episodes, and Table 9 shares the Table 5 split, both reporting 77.8 for always intervening. Which denominators, and which number carries the 1.8-point claim?
  2. Does any closed-loop row place a baseline's own scorer inside the shared bank and gates? The text says every learned scorer receives the same bank and gates.
  3. The artifact README says source code only, while the text claims configurations, thresholds, optimizer settings, seeds and episode outcomes. Where are those, and the sampling and decoder details the text places in the appendix?
  4. The Door scorer uses sixteen prediction steps while the reported Stage B horizon is eight. How is that produced, and which horizon does the goal distance use?
  5. Are the regret columns of Tables 13 and 15 the same quantity? Table 15 puts the model at 0.3293, below nominal BC at 0.3391, while Table 13 reports 0.3801.
4 comments

Thank you for your careful review and for recognizing the transition-level formulation and simulator-executed evaluation. Your comments helped us see that the relationship between the motivating observation, controlled model evidence and complete-system results needed clearer explanation. We have revised these explanations and added the missing protocol details. These manuscript revisions are highlighted in blue. This overview introduces our response; Parts 2 and 3 give the evidence and settings.

W1. Motivation and controlled evidence. Delta Match captures the motivating observation: strong state agreement can coexist with inconsistent transitions. We agree that its post-training increase also reflects optimization; success/MRR remain primary. Following your suggestion, we have foregrounded the matched control (Fig. 1, p. 2; Table 17, p. 24). At fixed architecture/capacity, each frozen variant trains its own Stage B, with OPD disabled:

MetricState-only+ Transition
State Match ↑0.99940.9998
Delta Match ↑0.78620.9884
Demo MRR ↑0.27480.4522
Physical Spearman ↑0.15180.3017
Regret ↓0.14420.0890

State Match barely changes, while action ranking and physical selection improve. Part 2 explains the control.

Q2. Shared-controller comparison. We acknowledge that the original shared-bank/gate scope was unclear. We have added the G1 Door comparison in Table 19 (p. 27): seven baseline scorers and TAM-WM share BC, risk/safety gates, rescue, candidate rules and intervention budgets. Below are results across three training runs (450 episodes/run; BC fixed):

MethodMean SR ± sample SD (%)
BC78.22
DINO-WM76.22 ± 1.02
ReViWo75.56 ± 0.44
MV-MWM78.81 ± 1.00
MVP-LAM79.48 ± 1.22
MVISTA-4D76.59 ± 1.48
Ctrl-World76.74 ± 0.90
VILA76.81 ± 0.84
TAM-WM83.63 ± 0.46

TAM-WM exceeds MVP-LAM by 4.15 pp and DINO-WM by 7.41 pp. Part 2 reports each run, supervision, calibration and timing scopes.

Q1. Teacher/student results. We have corrected Table 9 (p. 20) from the same 450 original Door episodes as Table 5, and updated the discussion (p. 8). “Current” denotes the 84.00% Full OPD teacher; “Earlier” denotes the conservative-carrier teacher. Results are:

MethodTeacherDeploymentSearchSuccess k/450
Nominal BC—Policy-onlyNo352/450 (78.22%)
Always interveneCurrentUngatedYes350/450 (77.78%)
Online teacher—Online OPDYes378/450 (84.00%)
Matched studentCurrentPolicy-onlyNo355/450 (78.89%)
Matched studentCurrentOnline OPDYes377/450 (83.78%)
Conservative studentEarlierPolicy-onlyNo367/450 (81.56%)
Conservative studentEarlierOnline OPDYes379/450 (84.22%)

These counts distinguish policy-only and online-search deployment. We have withdrawn the earlier 1.8-point search-free gain claim. Part 2 gives the teacher/student details.

Q3. Artifact and implementation settings. For Q3, we corrected the artifact-content statement and added the training, gate, sampling and encoder/readout settings.

Q4–Q5. Horizons and Regret protocols. For Q4–Q5, we explain direct 16-step Door prediction and the common recovery and metric definitions across tasks. We have added Can's normalized-Regret definition to Table 13, retaining its original raw gaps. Tables 13/15 use distinct candidate banks and physical references, which we now explain explicitly.

Manuscript revisions. We have revised Figure 1 (p. 2) and the matched-intervention discussion (p. 8) to foreground the control in Table 17 (p. 24), and added the shared-controller comparison in Table 19 (p. 27). We have corrected Table 9 (p. 20) and updated the teacher/student discussion (p. 8). We have also corrected the artifact statement (p. 22), added settings in Tables 20–23 (pp. 28–29), and clarified the Door horizon in Table 6 (p. 19) and its implementation (p. 21). The Regret protocols are now explained in Tables 13/15 (pp. 23–24) and the definitions on pp. 25–26. Thank you for helping us make the connection between evidence and claims clearer.

Thank you for your suggestions. We should have explained each experiment's role more clearly. We have revised the manuscript, marking changes in blue. Below we address W1, W2, Q1 and Q2.

W1. Transition-alignment evidence

We agree that Delta Match is linked to the training objective. Our motivating observation is that high cross-view state agreement can coexist with inconsistent transition displacements. Delta Match captures this gap and motivated Stage A; its post-training increase records alignment. Success and Demo MRR remain the primary performance metrics. We have retained cross-method Delta Match in Table 2 to describe the observation; the matched intervention tests the downstream effect of alignment.

We have foregrounded the control (Fig. 1, p. 2; Table 17, p. 24), as suggested. At fixed architecture/capacity, each frozen variant trains its own Stage B without OPD. Results:

MetricState-only+ Transition
State Match ↑0.99940.9998
Delta Match ↑0.78620.9884
Demo MRR ↑0.27480.4522
Physical Spearman ↑0.15180.3017
Regret ↓0.14420.0890

State Match barely changes, while MRR, physical correlation and Regret improve.

Q2. Baseline scorers in shared OPD

Thank you for proposing this control. We have expanded the G1 Door comparison to seven baseline scorers and TAM-WM. All share nominal BC, risk/safety gates, rescue proposer, 64-local + 36-rescue candidate rule, intervention budget and 450 resets. Run 1–3 vary model training; controller components remain fixed. Entries are success percentages over 450 episodes per run; SD is the sample SD across runs. BC is held fixed across runs.

MethodRun 1Run 2Run 3ComponentFull OPDMean ± SD (%)
BC78.22%78.22%78.22%----78.22
DINO-WM75.33%76.00%77.33%8.84/11.84123.04/132.1976.22 ± 1.02
ReViWo75.56%75.11%76.00%9.05/10.05133.42/139.4475.56 ± 0.44
MV-MWM78.89%79.78%77.78%11.32/12.03126.47/134.0578.81 ± 1.00
MVP-LAM78.22%79.56%80.67%57.94/72.49221.17/277.5679.48 ± 1.22
MVISTA-4D77.56%77.33%74.89%68.91/69.43 s300.36/306.31 s76.59 ± 1.48
Ctrl-World75.78%77.56%76.89%7.30/7.37 s1461.44/1476.59 s76.74 ± 0.90
VILA76.22%77.78%76.44%49.38/54.62212.48/248.1676.81 ± 0.84
TAM-WM84.00%83.78%83.11%10.01/13.98126.51/165.6283.63 ± 0.46

TAM-WM reaches 83.63% ± 0.46%, exceeding the strongest baseline, MVP-LAM (79.48% ± 1.22%), by 4.15 pp, DINO-WM by 7.41 pp and BC by 5.41 pp, from unrounded counts. It leads all seven baselines in each run.

The gates use shared simulator labels; rescue uses training demonstrations. Candidate tensors follow each rollout state. Uncertainty thresholds follow the same validation-quantile rule on each scorer's scale. Matching covers additional supervision, candidate count and intervention budget; total model-training compute varies. Table 1's RL policy-only row is a separate deployment experiment.

We have added two p50/p95 columns to Table 19 (ms unless marked s). BC uses no world-model prediction or OPD (--). Full OPD includes observation processing, nominal BC, K=100 candidate evaluation, gates, rescue and final selection. Component timing covers K=100 scoring for DINO/ReViWo/MV-MWM/MVP-LAM/VILA/TAM, one video candidate for Ctrl-World, and one native action chunk for MVISTA-4D.

Q1. Door teacher/student results

Thank you for checking Table 9. We apologize for the inaccurate percentages and incomplete reporting; we have corrected them from episode records. Tables 5/9 use fixed models on the same 450 original Door episodes (0–449), separate from G1 above; we have reported three-run means in Table 1. “Current” is the 378/450 Full OPD teacher; “Earlier” is the earlier conservative-carrier teacher. “Conservative student” abbreviates Conservative-Carrier Student.

MethodTeacherDeploymentSearchSuccess k/450
Nominal BC—Policy-onlyNo352/450 (78.22%)
Always interveneCurrentUngatedYes350/450 (77.78%)
Online teacher—Online OPDYes378/450 (84.00%)
Matched studentCurrentPolicy-onlyNo355/450 (78.89%)
Matched studentCurrentOnline OPDYes377/450 (83.78%)
Conservative studentEarlierPolicy-onlyNo367/450 (81.56%)
Conservative studentEarlierOnline OPDYes379/450 (84.22%)

Policy-only students omit online world-model search; +OPD rows include it. The counts do not support the earlier 1.8-point search-free gain claim; we have withdrawn it and report the deployment trade-off. These students are distilled from corrections; Table 1's w/o OPD policy is trained by RL. We have added the matched student's image-to-action timing to Table 9: 0.72/0.90 ms (p50/p95; A100, 100 calls).

We have revised Fig. 1 (p. 2) and the discussion (p. 8) to foreground Table 17 (p. 24), expanded Table 19 (p. 27) with shared-controller results and separate timing scopes, corrected Table 9 (p. 20), and updated the teacher/student discussion (p. 8).

Thank you for checking the artifact and evaluation protocols. The original text did not bring the settings together or fully explain the task-specific horizons and Regret definitions. We have revised the manuscript and highlighted the changes in blue. Below we address Q3–Q5 with the settings and corresponding revisions.

Q3. Artifact and implementation settings

For Q3, the original text incorrectly listed per-episode outputs among the supplied materials; the source/configuration archive does not contain them. We have corrected that statement. The sampling and architecture descriptions also needed detail; we list the settings below, separating Can PH defaults from the selected Door configuration.

We have listed the Can PH defaults in Table 20: both stages use H=8H=8, seed 261, batch 256 and 20,000 updates.

SettingStage AStage B
Optimizer / LR / WDAdamW / 1e-4 / 1e-4AdamW / 2e-5 / 1e-4
LR scheduleConstantCosine; minimum 10% of initial LR
DimensionsContent 256; context 64; hidden 512Residual in frozen content coordinate
DropoutNetwork 0.1; context 0.20.1
Loss weightsAlign 0.1; transition 0.2; cross/self reconstruction 1/1; orthogonal 0.01; bottleneck 0Action contrastive 0.01; variance 0; temperature 0.1

We have reported the online gate settings in Table 21. Below are the selected Door settings:

Gate/controlSetting
Trigger rescue (both)pfailure>0.5p_{\mathrm{failure}}>0.5; precoverability>0.4p_{\mathrm{recoverability}}>0.4
Accept rescue (both)prescue>0.5p_{\mathrm{rescue}}>0.5; pdamage<0.4p_{\mathrm{damage}}<0.4
Acceptance / distanceScore > 0.400144589683965; Euclidean distance from nominal BC ≤ 0.2 (otherwise score -1e9)
Budget / burstAt most round⁡(0.08×150)=12\operatorname{round}(0.08\times150)=12 steps/episode; at most 3 consecutive steps
Continue burstpfailure>0.4p_{\mathrm{failure}}>0.4; pdamage<0.5p_{\mathrm{damage}}<0.5; budget remains
Candidates64 base + 36 rescue = 100

For the sampling details you requested, we distinguish context pairing, training negatives and evaluation candidates:

StageRule
Stage-A contextSame demo, m≠tm\ne t, deterministic from seed/sample index; cross-combination context pairing
Stage-B negativesSeven in-batch action chunks/positive (eight total, temperature 0.1); circularly shift actions by 1–7, keeping current/target states fixed
Can offline bankFour families: magnitude-matched, action-nearest, state-matched, hybrid; 50 candidates/query, demo index 0, non-demo indices 1–49; shared indices, seed 83083
Hybrid negativesRound-robin over temporal-adjacent, magnitude-matched, action-nearest, state-matched, other-demo and random pools; deduplicate indices, random fallback

Regarding network structure, the Stage-A encoder applies LayerNorm(384) → Linear(384,512) → GELU → Dropout(0.1) → Linear(512,512) → GELU per patch, mean-pools, then uses residual-MLP heads for 256-D content and 64-D context, with separate weights. The feature readout reconstructs frozen DINO features:

ComponentStructure
InputConcatenate content/context (320-D); LayerNorm(320), Linear(320,512), GELU, Dropout(0.1)
ResidualMLPLayerNorm(512), Linear(512,1024), GELU, Dropout(0.1), Linear(1024,512); residual addition
Patch/outputBroadcast 512-D + learned positions [256,512]; LayerNorm(512), Linear(512,512), GELU, Linear(512,384); output [B,256,384]
ViewsSeparate agent/wrist readouts; identical structures, unshared weights

Q4. Door prediction horizon

For Q4, we should have stated the horizons explicitly. Door uses H=16H=16 to cover progress after handle contact and sustained pushing; the other seven tasks use H=8H=8. It predicts t+16t+16 directly. Online, each 4-D candidate is repeated for 16 steps; endpoint MSE to cgc_g is averaged across three members. Control re-observes every step. Door shares the recovery workflow with Can/Square (p. 18).

Q5. Regret protocols

For Q5, we apologize for the unclear Regret labels. Table 13 uses an expert-centered bank and expert-terminal discrepancy; Table 15 uses a BC-centered bank and task-progress cost c=−uc=-u. Let gi=ci,ji∗−min⁡jcijg_i=c_{i,j_i^*}-\min_j c_{ij}. The original Table 13 reports mean raw gap (0.3801); Table 15 averages gi/max⁡(max⁡jcij−min⁡jcij,10−8)g_i/\max(\max_j c_{ij}-\min_j c_{ij},10^{-8}) (TAM-WM 0.3293; BC 0.3391). Constant costs give zero. We have added normalized Regret to Table 13 for all eight methods, using Can's 10−1210^{-12} floor: TAM-WM 0.0980, DINO-WM 0.1353. The original raw column remains. Table 15's Selection Efficiency averages the selected/oracle gain ratio, clipped to [0,1], over states with oracle gain above 1e-6.

We have corrected the artifact statement on p. 22 and added settings in Tables 20–23 (pp. 28–29). We have clarified the Door horizon in Table 6 (p. 19), its implementation on p. 21, and the Regret protocols in Tables 13/15 (pp. 23–24) and definitions on pp. 25–26. Thank you for helping us clarify these details.

Thank you again for your time and thoughtful review. As the discussion period is approaching its end, we would like to follow up on our responses and the revised manuscript. We would appreciate hearing whether the clarifications and additional results address your concerns. If any questions remain, we would be happy to discuss them further.

Reviewer_rLrGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TAM-WM aims to make predictions of robot action consequences consistent across camera views. Stage A learns a shared visual predictive coordinate by aligning latent changes across synchronized views of the same transition. Stage B freezes this representation and learns state- and action-conditioned residual dynamics. A controller uses the predicted outcomes to selectively correct a nominal policy. Validation combines representation diagnostics, candidate-action ranking, simulator-executed outcomes, and closed-loop evaluation across eight simulated tasks. The reported results support improved action comparison and higher average task success under the evaluated configurations.

Critical strengths

The ablations help clarify what the main components contribute. Transition alignment improves demonstrated-action ranking with little change in State Match (Table 17). Removing transition alignment or online correction also lowers average success across eight simulated tasks (Table 1), supporting benefits beyond feature similarity.

Critical weaknesses
  1. The physical-ranking metrics and tie handling need clarification. The PDF does not clearly define the score assigned to simulator-executed outcomes, the Top-1 metric in Table 4, regret normalization, or how correlations are aggregated. This leaves unclear how executed outcomes are evaluated and whether the physical-ranking test supports the candidate-scoring rule used at deployment. Separately, the MRR formula on page 6 ranks the demonstrated action first when all candidates tie. This can inflate Demo MRR without better action discrimination. Could you please clarify the implemented tie-handling rule and its effect on the reported results?
  2. Key implementation details of the main comparison and ablation remain unclear. The PDF does not fully explain how VILA and MVP-LAM score candidate actions. The "fixed scorer" in Table 3(a) also leaves unclear whether Stage B is trained separately for each variant's representation. For "Ours w/o OPD" in Table 1, the authors should explain how actions are generated, whether and how the retained world model affects decisions, and whether this variant uses the phase-conditioned hybrid controller in Figure 4. These missing details limit interpretation of the comparisons and component gains.
Questions for authors
  1. How are the model score and the executed physical score calculated in Tables 4 and 13? Please define Top-1 physical rank (Table 4) and normalized regret. Is Physical Spearman ρ\rho computed for each restored state and then averaged? Does the model score use the distance between predicted future coordinates and the task-specific goal prototype, as at deployment?
  2. For the "fixed scorer" in Table 3(a), was Stage B trained separately for each variant, or were the same trained weights reused across variants?
  3. How do VILA and MVP-LAM score candidate actions and generate actions in closed-loop evaluation? Please identify the modules used.
  4. How does “Ours w/o OPD” in Table 1 generate actions? Does the world model still affect these actions? Does this variant use the phase-conditioned hybrid controller shown in Figure 4?
  5. For Demo MRR, how often does the demonstrated action receive the same score as another candidate, and how are these ties handled? If ties occur, please check whether random tie-breaking changes the main MRR comparisons.
8 comments

We sincerely thank you for the time and care invested in reviewing our work. We acknowledge that several metric definitions and implementation details were not clearly explained in the original manuscript. We also appreciate your observation that the original ranking rule places the demonstrated action first among candidates with equal scores. We have expanded the explanations and recomputed MRR using expected reciprocal rank under random tie-breaking. We have revised the manuscript and highlighted the changes in blue. This overview introduces the responses; Parts 2 and 3 provide the definitions and implementation details.

Q1. Physical metrics and targets. We have clarified the metric definitions (pp. 25–26) and added normalized Regret for all eight methods to Table 13 (p. 23), retaining raw gaps. Success/MRR are primary; physical metrics provide auxiliary checks against executed outcomes. Tables 4/13 give:

TaskTAM MRRDINO MRRTAM Norm. RegretDINO Norm. Regret
Can0.70570.58630.06060.0944
Square0.43050.33070.09800.1353

TAM-WM leads these MRR/Regret comparisons; DINO-WM leads Can Top-1 (0.3862 vs. 0.3807) and Square Spearman (0.2638 vs. 0.2101). Door/Can/Square share the recovery workflow and metrics, with task-specific costs. TAM-WM's offline scores target expert transitions; online scores target the goal prototype. Part 2 explains aggregation, Bootstrap and constant-vector rules.

Q5. Demo MRR ties. We have added the tie-aware formula and eight-method audit in Table 18 (p. 26). Each method has 8,656 Can PH queries; both rules use the same scores within each method. Demo Ties and Pair Ties count queries with an exact demo/non-demo tie and any exact candidate-pair tie, respectively:

MethodStrictTie-awareDemo TiesPair Ties
TAM-WM0.70570.705700
DINO-WM0.58630.586303
MV-MWM0.43800.438001
ReViWo0.18470.184701
MVP-LAM0.24280.242805
VILA0.36270.362700
Ctrl-World0.32930.329305
MVISTA-4D0.31040.310401

All ties involve non-demo actions, so expected MRR under random tie-breaking is unchanged. As a positive control, Rank-average has 2,355/8,656 demo-tied queries and its MRR changes from 0.685547 to 0.646210. Part 2 gives the formula and protocol.

Q1. Deployment scoring. We have added the shared-OPD G1 Door comparison in Table 19 (p. 27): three training runs, 450 resets/run, shared BC, gates, rescue, K=100 candidate rule and intervention budget. BC is fixed; SD is across runs. Results are:

MethodMean SR ± sample SD (%)
BC78.22
DINO-WM76.22 ± 1.02
ReViWo75.56 ± 0.44
MV-MWM78.81 ± 1.00
MVP-LAM79.48 ± 1.22
MVISTA-4D76.59 ± 1.48
Ctrl-World76.74 ± 0.90
VILA76.81 ± 0.84
TAM-WM83.63 ± 0.46

TAM-WM exceeds the strongest baseline, MVP-LAM, by 4.15 pp and DINO-WM by 7.41 pp. This evaluates deployment scoring in closed loop, complementing the offline expert-target diagnostic.

Q2–Q4. Training and action generation. For W2 and Q2–Q4, we recognize that labels such as “fixed scorer” and “w/o OPD” did not adequately explain what was trained or used at deployment. Each frozen Stage-A variant has its own trained Stage-B scorer, with architecture and scoring form fixed. VILA/MVP-LAM use separate offline ranking adapters and closed-loop policy modules. “Ours w/o OPD” directly executes a COMBO-trained RL policy; Figure 4's hybrid controller serves the G1 qualitative visualization. Part 3 connects these modules to their respective comparisons.

Manuscript revisions. We have clarified the metric definitions and tie handling on pp. 25–26 (Table 18), and added the deployment comparison in Table 19 (p. 27). We have also clarified separate Stage-B training in Table 3(a) (p. 9), baseline adapters and policies on pp. 21–22, and the RL/Figure 4 distinction on p. 7, with COMBO details on p. 22. Parts 2–3 explain these revisions. Thank you for helping us make the evaluation easier to assess.

Thank you for checking the metrics and tie handling. We have revised the manuscript and highlighted the changes in blue. Below we address W1, Q1 and Q5 with the physical-metric definitions and a tie-aware recomputation of MRR, starting with the auxiliary metrics.

Q1. Physical metrics and targets

Success rate and Demo MRR are our primary metrics; physical metrics additionally compare model rankings with executed outcomes. From each of N=64N=64 restored states, we execute K=50K=50 chunks of H=8H=8 actions. Cost measures terminal discrepancy from the executed demo (j=0j=0); lower is closer. For Can:

cijCan=λp∣Δpo∣2+λθΔθo+λe∣Δ(pe−po)∣2+λq∣Δqg∣2+λG∣ΔG∣+λC∣ΔC∣+λS∣ΔS∣.c_{ij}^{\mathrm{Can}}=\lambda_p\\|\Delta p_o\\|_2+\lambda_\theta\Delta\theta_o+\lambda_e\\|\Delta(p_e-p_o)\\|_2+\lambda_q\\|\Delta q_g\\|_2+\lambda_G|\Delta G|+\lambda_C|\Delta C|+\lambda_S|\Delta S|.

Here Δ\Delta is the candidate/demo terminal difference; po,pe,qgp_o,p_e,q_g are object, end-effector and gripper-joint positions; Δθo\Delta\theta_o is relative rotation angle; G,C,SG,C,S indicate grasp, contact and success. Effective weights are λp=λe=20\lambda_p=\lambda_e=20, λθ=1/(2π)\lambda_\theta=1/(2\pi), λq=6.25\lambda_q=6.25, λG=1\lambda_G=1, λC=0.5\lambda_C=0.5, λS=4\lambda_S=4. Square sets λp=λe=100/3\lambda_p=\lambda_e=100/3 and adds μpeg∣Δ(po−ppeg)∣2\mu_{\mathrm{peg}}\\|\Delta(p_o-p_{\mathrm{peg}})\\|_2 with μpeg=100/3\mu_{\mathrm{peg}}=100/3; other weights are unchanged. Within each task, methods share candidates, executed outcomes and weights. This cost is used only after execution for evaluation.

Regarding the target, offline TAM-WM uses sij=d−1∣c^i,t+H(A(j))−ci,t+H⋆∣22s_{ij}=d^{-1}\\|\hat c_{i,t+H}(A^{(j)})-c_{i,t+H}^{\star}\\|_2^2, MSE to the expert future coordinate (dd: coordinate dimension), and selects ji∗=arg⁡min⁡jsijj_i^*=\arg\min_j s_{ij}. Online OPD uses the task goal prototype cgc_g within the shared recovery workflow (p. 18).

We then assess ranking and selection in three ways, computing each measure per state and averaging over the 64 states:

Spearman measures list-wise agreement: ρi=Spearman(−si,−ci)\rho_i=\mathrm{Spearman}(-s_i,-c_i), using average ranks for ties. Nonfinite correlations, including constant vectors, are set to zero and retained. 95% percentile CIs bootstrap the 64 states with replacement.

Top-1 focuses on the chosen candidate: 1−(ri,ji∗−1)/(K−1)1-(r_{i,j_i^*}-1)/(K-1), where rr is ascending average physical-cost rank (1 is best). This measures the selected candidate's rank percentile.

Regret measures excess physical cost. Let gi=ci,ji∗−min⁡jcijg_i=c_{i,j_i^*}-\min_j c_{ij}. Tables 4/13 use the mean of gi/max⁡(max⁡jcij−min⁡jcij,10−12)g_i/\max(\max_j c_{ij}-\min_j c_{ij},10^{-12}); constant costs give zero. Table 13 retains raw gaps and adds normalized values for all eight methods.

TaskTAM MRRDINO MRRTAM Norm. RegretDINO Norm. Regret
Can0.70570.58630.06060.0944
Square0.43050.33070.09800.1353

TAM-WM leads the MRR and normalized-Regret comparisons above; DINO-WM has higher Can Top-1 (0.3862 vs. 0.3807) and Square Spearman (0.2638 vs. 0.2101). These assess different aspects of ranking. Regret comparisons are within each protocol. Part 1's shared-OPD test evaluates goal-prototype scoring in closed loop, complementing this expert-target diagnostic.

Q5. Demo MRR tie handling

Thank you for raising the tie issue. We added a tie-aware definition and recomputed MRR. The original rank ristrict=1+bir_i^{\mathrm{strict}}=1+b_i assigns the best position in a tied group. Here bib_i counts strictly better candidates and mim_i counts exact ties with the demo, including itself. Uniform random tie-breaking gives expected reciprocal rank

RRitie-aware=1mi∑r=bi+1bi+mi1r.\mathrm{RR}_i^{\mathrm{tie\text{-}aware}}=\frac{1}{m_i}\sum_{r=b_i+1}^{b_i+m_i}\frac{1}{r}.

To check its effect on the results, we recomputed both rules. Each method has 2,164 states × 4 candidate families = 8,656 queries (K=50K=50); both rules use the same score records. Both tie columns count queries: Demo Ties requires an exact demo/non-demo tie; Pair Ties any exact tie between distinct candidates.

MethodPaperStrictTie-awareDemo TiesPair Ties
TAM-WM0.70570.70570.705700
DINO-WM0.58630.58630.586303
MV-MWM0.43800.43800.438001
ReViWo0.18470.18470.184701
MVP-LAM0.24280.24280.242805
VILA0.36270.36270.362700
Ctrl-World0.32930.32930.329305
MVISTA-4D0.31040.31040.310401

All observed ties involve non-demo actions, so demo MRR is unchanged. To check that the audit responds to ties, the independent Rank-average control has 2,355/8,656 demo-tied queries; MRR changes from 0.685547 to 0.646210.

We have revised the metric and ranking explanations on p. 6 and Tables 4/13 (pp. 9/23), and added full definitions and statistics on pp. 25–26, the tie-aware formula on p. 26 and Table 18 on p. 26. We also specify shared BC/gates/rescue, 100-candidate search, intervention budget, 450 resets and per-scorer calibration in Table 19 (p. 27).

Thank you for identifying these gaps in the implementation description. We did not clearly connect the training, offline scoring and deployed policy modules. We have revised the manuscript and highlighted the changes in blue. Below we explain these processes in response to W2 and Q2–Q4.

Q2. Stage-B training

Thank you for asking us to clarify “fixed scorer”. Each representation has its own trained Stage B.

In Table 3(a), "fixed scorer" means we keep the Stage-B dynamics architecture and scoring formula fixed, while independently training a separate Stage-B model for each frozen Stage-A representation variant so each dynamics model matches its own latent coordinate.

Q3. Baseline scoring and action generation

For VILA and MVP-LAM, we should have distinguished scoring from action generation. The offline adapters share candidate indices but use different conditioning: VILA uses current observations, while MVP-LAM uses an expert-transition target. We have separately reported closed-loop action generation in Table 1.

For offline scoring on Can PH (K=50K=50 chunks/state), VILA conditions on current two-view features, adds noise to each normalized candidate chunk at a fixed diffusion timestep, and runs noise_pred_net. Cost is the MSE between predicted and injected noise (all 50 candidates/state share the same noise draw). This VILA adapted denoising score is a conditional denoising-consistency proxy using only current images and candidate actions, without using executed outcomes.

MVP-LAM instead uses frozen vq_encode to encode each view's current-to-expert-future image pair and concatenate the quantized embeddings into z_q. We then fit a training-split ridge-regression head from "current proprioception + candidate chunk" to z_q. Cost is the MSE between predicted and expert transition codes. We label this MVP-LAM (adapted): frozen latent code plus a ridge action-to-latent mapping head, distinct from the full 7B VLA policy.

For the main closed-loop comparison in Table 1, these rows use policy modules to generate actions; the above adapters provide offline ranking scores.

VILA uses a trained conditional diffusion policy that generates action sequences for execution. The G1 Stage-2 implementation uses a single-view policy trained from the multi-view setup.

MVP-LAM uses an adapted latent-action policy that predicts action codes, queries the frozen codebook, and uses a trained action head to output actions or chunks. In the G1 implementation, code prediction uses visual features, proprioception, and episode progress; the chunked decoder also receives visual features.

Q4. RL actions and Figure 4

For Q4, the label “w/o OPD” did not explain how the retained model affects actions. The variant directly executes a COMBO-trained RL policy.

This row evaluates policy-only use of the learned representation, with a different action-generation path from the full BC-plus-OPD system. In the shared-controller study, BC versus BC + OPD evaluates correction at a fixed nominal policy, while TAM-WM versus DINO-WM under shared OPD evaluates the scorer. We have separately reported the gate evaluation in Table 5.

Training uses COMBO offline model-based RL. The policy encodes the two views via the frozen multi-view encoder, zt=fθ(otagent,otwrist)z_t = f_\theta(o_t^{\mathrm{agent}}, o_t^{\mathrm{wrist}}), and concatenates normalized task/proprioceptive state s~t\tilde{s}_t into xt=[zt,s~t]x_t = [z_t, \tilde{s}_t] (pre-cached, fixed during RL).

Using cached real transitions (xt,at,rt,xt+1,dt)(x_t, a_t, r_t, x_{t+1}, d_t), COMBO trains a separate probabilistic dynamics ensemble in xtx_t space to predict next-step deltas and rewards; short synthetic rollouts under the current policy are mixed with real transitions. TAM-WM provides ztz_t; the ensemble is trained separately from Stage B.

The policy is then learned with two critics using Bellman targets on real and synthetic transitions with conservative value regularization; a stochastic actor with auto-tuned entropy temperature α\alpha is updated via:

Lactor=Ex,,a∼πϕ(⋅∣x)[αlog⁡πϕ(a∣x)−min⁡i=1,2Qi(x,a)].\mathcal{L}_{\mathrm{actor}}=\mathbb{E}_{x,\\,a\sim\pi_\phi(\cdot\mid x)}\left[\alpha\log\pi_\phi(a\mid x)-\min_{i=1,2}Q_i(x,a)\right].

At deployment, at=tanh⁡(μϕ(xt))a_t = \tanh(\mu_\phi(x_t)) is output directly—no BC proposer, candidate bank, online rollout, risk gate, or rescue. The representation shapes every step via xtx_t; COMBO's dynamics shape learning offline. Figure 4's phase-conditioned hybrid controller is used only for G1 qualitative visualizations.

We have clarified separate Stage-B training in Table 3(a) and its discussion (p. 9), and baseline scoring and action generation on pp. 21–22. We have also clarified the distinction between the RL policy and the qualitative controller in Table 1/Figure 4 (p. 7), and expanded the COMBO training and deployment description on p. 22.

Reviewer_obUZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TAM-WM, a multi-view world model for robotic manipulation that explicitly aligns action-induced state transitions across camera viewpoints. The key motivation is that agreement between representations of synchronized observations does not necessarily imply agreement between their predicted temporal changes.

The proposed method uses two training stages. Stage A learns a shared latent coordinate by aligning observed transition displacements across views. Stage B learns action-conditioned residual dynamics in this fixed coordinate, enabling the model to compare predicted outcomes of candidate actions. A downstream selective correction mechanism uses these predictions to improve a nominal manipulation policy.

Experiments on eight simulated manipulation tasks report an average success rate of 75.85%, compared with 70.57% for VILA and 66.86% for behavior cloning. Additional experiments examine transition alignment, candidate-action ranking, and correlation with simulator-executed outcomes.

Critical strengths
  1. A meaningful distinction between state alignment and transition alignment. The observation that cross-view consistency of state representations does not necessarily ensure consistency of action-induced changes is interesting and relevant to multi-view world modeling. Explicitly aligning transition displacements is a reasonable approach. The controlled experiments provide evidence supporting this design: the matched transition-alignment intervention improves demonstrated-action ranking and the correlation between model scores and physically executed outcomes.

  2. Evaluation connects learned representations to action selection. Rather than relying exclusively on representation similarity or future prediction metrics, the paper evaluates candidate-action ranking and actual outcomes in simulation, and further connects the model to closed-loop policy correction. This is an important aspect of evaluating world models for robot manipulation. The eight-task results and within-system ablations provide encouraging evidence that the learned representations can be useful for control.

Critical weaknesses
  1. The main closed-loop improvements are not sufficiently isolated from the additional control system. The complete approach includes not only transition-aligned dynamics, but also task-specific goal prototypes, candidate search, failure-risk estimation, safety filtering, and selective intervention. Some of these components require additional simulator interaction data and considerable inference-time computation. Although the paper provides component ablations and controlled scorer comparisons, it remains difficult to assess the general utility of transition alignment independently of this fairly elaborate downstream correction pipeline. This limits the strength of the broader world-model claims.

  2. The evaluation is not yet sufficiently convincing in terms of robustness and real-world generalization. All quantitative success rates are obtained in simulation, while the real-robot results are qualitative demonstrations. Moreover, the main eight-task comparison reports one finalized training run per method and task, without independent training-seed variation. Some predictive metrics are also less consistently favorable across tasks: for example, DINO-WM achieves stronger physical-ranking correlation on Square, and its Top-1 physical ranking on Can is slightly higher. The reported average success improvement is encouraging, but stronger evidence is needed to establish its reliability and generality.

Questions for authors
  1. Can the authors more clearly quantify the additional data and computational costs associated with the online correction system, including simulator-generated risk labels and candidate evaluation? How much of the overall gain remains when comparing methods under a strictly matched additional supervision and inference budget?

  2. Can the authors provide training-seed variation for the main closed-loop results, at least on representative tasks? This would help determine whether the reported average improvements are robust.

  3. Can the authors provide quantitative real-robot evaluations or additional evidence that transition alignment improves performance under realistic viewpoint changes, contact uncertainty, and execution failures? This is particularly important for establishing the practical significance of the proposed world model.

2 comments

Thank you for your careful review. We have revised the manuscript and highlighted the changes in blue. Below we address your questions with experimental results.

Q1. Model contribution and OPD costs

We have expanded the fixed-capacity control discussion (Fig. 1, p. 2; Table 17, p. 24). With OPD disabled and Stage B separately trained for each frozen representation:

MetricState-only+ Transition
State Match ↑0.99940.9998
Delta Match ↑0.78620.9884
Demo MRR ↑0.27480.4522
Physical Spearman ↑0.15180.3017
Regret ↓0.14420.0890

Alignment improves action ranking and physical selection while State Match changes little. Table 1's w/o OPD separately tests an RL actor using TAM-WM representations, without online search/gates/rescue.

We have added the G1 Door shared-OPD comparison in Table 19 (p. 27). Three model-training runs share fixed BC, risk/safety gates, rescue, 64-local + 36-rescue candidates (K=100), intervention budgets and 450 resets/run. Results and full-OPD p50/p95 follow (ms unless marked s):

MethodMean SR ± sample SD (%)Full OPD p50/p95
BC78.22--
DINO-WM76.22 ± 1.02123.04/132.19
ReViWo75.56 ± 0.44133.42/139.44
MV-MWM78.81 ± 1.00126.47/134.05
MVP-LAM79.48 ± 1.22221.17/277.56
MVISTA-4D76.59 ± 1.48300.36/306.31 s
Ctrl-World76.74 ± 0.901461.44/1476.59 s
VILA76.81 ± 0.84212.48/248.16
TAM-WM83.63 ± 0.46126.51/165.62

TAM-WM leads every baseline in each run: +4.15 pp over MVP-LAM, +7.41 over DINO-WM and +5.41 over BC, from unrounded counts. Matching covers additional supervision, candidate count and intervention budgets. Candidates follow rollout states; thresholds use the same validation rule on each scorer's scale. Total training compute varies.

We have added cost accounting on pp. 28–29; Table 19 reports component/full-OPD timings. G1 Door uses 32 training/8 validation resets:

ItemRecorded cost
Risk / candidate labels3,840 / 1,280
Corresponding physics substeps100,160 / 491,520 (591,680 total)
Stage A / B training~2.5 / ~3.3 GPU-h
Risk / safety / rescue training2.45 / 1.14 / 33.42 s
TAM K=100 scoring p50/p9510.01 / 13.98 ms
Matched student p50/p950.72 / 0.90 ms
Square BC / OPD median9.6 / 27.5 ms (+17.9 ms)

Candidate labels span 20,480 control steps (24 physics substeps/step); replay/calibration/evaluation are separate. Stage B uses Q=2 and 20k updates. Gates/rescue train once per task on cached inputs. Full OPD includes observation processing, BC, candidate generation/evaluation, gates, rescue and selection. The student is search-free (Table 9, p. 20; A100, 100 image-to-action calls).

Q2. Training-seed variation

We changed seeds and independently retrained all non-BC methods three times across eight tasks. We have updated Table 1 (p. 7) and added full runs in Tables 25–27 (p. 30). Eight-task mean SR (%) and sample SD across runs follow; BC is fixed:

MethodRun 1Run 2Run 3Mean ± SD
BC66.8666.8666.8666.86
DINO-WM64.8364.6165.0364.82 ± 0.21
ReViWo65.8365.3566.1065.76 ± 0.38
MV-MWM66.9466.0766.8266.61 ± 0.47
MVP-LAM67.5667.2568.1867.66 ± 0.47
MVISTA-4D68.4467.6968.0868.07 ± 0.38
Ctrl-World70.1070.3270.1370.18 ± 0.12
VILA70.5770.0670.9070.51 ± 0.43
Ours w/o transition72.3571.9071.7872.01 ± 0.30
Ours w/o OPD71.3373.3873.0472.58 ± 1.10
Ours + OPD75.8575.8975.9475.89 ± 0.05

The mean gain over VILA is 5.38 pp; TAM-WM + OPD leads in each run.

W2. Physical-ranking results. We have clarified the auxiliary metrics (p. 25) and added normalized Regret for all eight methods to Table 13 (p. 23). Tables 4/13 give:

TaskTAM MRRDINO MRRTAM Norm. RegretDINO Norm. Regret
Can0.70570.58630.06060.0944
Square0.43050.33070.09800.1353

DINO-WM leads Can Top-1 (0.3862 vs. 0.3807) and Square Spearman (0.2638 vs. 0.2101); TAM-WM leads MRR/Regret above. Success/MRR are primary; physical metrics assess ordering and selection. Regret is compared within each protocol; raw gaps remain in Table 13.

Q3. Real-robot and view robustness

We have added real-robot block-to-bowl results in Table 28 (p. 31): 15 trials/method (105 total), varied object positions, fixed cameras; success requires placing the block inside the bowl. The w/o OPD variant uses RL.

MethodSuccess rate
BC46.67%
DINO-WM53.33%
ReViWo46.67%
MV-MWM60.00%
Ours w/o transition66.67%
Ours w/o OPD73.33%
Ours + OPD86.67%

Full OPD has the highest observed success rate.

We have clarified Table 16 (p. 24): Square/Tool Hang, 180 episodes/task/condition. Mean simulated SR is:

ViewsFullAgent droppedWrist droppedMaskedHeld-last
SR (%)73.0667.5070.5666.6767.50

Missing or outdated views reduce SR by 2.50–6.39 pp, showing the contribution of synchronized views.

Thank you again for your time and thoughtful review. As the discussion period is approaching its end, we would like to follow up on our responses and the revised manuscript. We would appreciate hearing whether the clarifications and additional results address your concerns. If any questions remain, we would be happy to discuss them further.

Comments on the paper

2 comments

We sincerely thank obUZ, rLrG, and ZSuJ for their careful reviews. Your questions helped us distinguish what transition alignment contributes to action ranking from what the full controller achieves in closed loop. We have revised the manuscript to make that distinction explicit, added results across training seeds and on a real robot, and clarified the evaluation protocols. Blue text marks the substantive revisions in the PDF.

Revision map. Page numbers refer to the revised manuscript.

Reviewer questionsWhat we revisedLocation
obUZ W1, ZSuJ W1We brought the fixed-capacity transition-alignment comparison into the main discussion and clarified the role of Delta Match as a diagnostic.Fig. 1 (p. 2), discussion (p. 8), Table 17 (p. 25)
obUZ W1/Q1, ZSuJ W2/Q2We compared seven baseline scorers and TAM-WM inside the same Door controller and reported simulator supervision, training costs, and decision latency.Table 19 (p. 27), cost breakdown (pp. 28–29)
obUZ W2/Q2/Q3We added three-run simulation results and a quantitative real-robot evaluation. We also clarified the simulated camera-perturbation results.Table 1 (p. 7), Table 16 (p. 24), Tables 25–27 (p. 30), Table 28 (p. 31)
obUZ W2, rLrG W1/Q1/Q5, ZSuJ Q5We defined the physical metrics and their aggregation, distinguished the offline expert target from the deployment goal, clarified the two Regret protocols, and audited exact ties in Demo MRR.Table 4 (p. 9), Tables 13 and 15 (pp. 23–24), definitions and Table 18 (pp. 25–26)
rLrG W2/Q2–Q4We clarified how Stage B is trained for each representation, how VILA and MVP-LAM score and generate actions, and how the RL policy in “w/o OPD” selects actions.Table 1 and Fig. 4 (p. 7), Table 3(a) (p. 9), baseline and RL details (pp. 21–22)
ZSuJ Q1We corrected the Door teacher/student success rates on the same 450 episodes, identified each student's teacher and deployment mode, and withdrew the earlier claim of a 1.8-point gain without search.Discussion (p. 8), Table 9 (p. 20)
ZSuJ Q3/Q4We corrected the description of the submitted archive, listed training, sampling, decoder, and gate settings, and explained how the Door scorer directly predicts the 16-step endpoint.Table 6 (p. 19), implementation (pp. 21–22), Tables 20–23 (pp. 28–29)

Main findings.

  • Contribution of transition alignment. With model capacity fixed and online correction disabled, adding transition alignment raises Demo MRR from 0.2748 to 0.4522 while State Match changes from 0.9994 to 0.9998. In the separate G1 Door experiment, TAM-WM achieves 83.63% success versus 79.48% for the strongest baseline scorer, MVP-LAM, under the same supervision, candidate count, and intervention budget.
  • Reliability across training runs. Across three independent runs, TAM-WM with OPD leads on eight-task average success in every run. Its mean is 75.89% versus 70.51% for VILA. The sample standard deviation of the eight-task average is 0.05 percentage points for TAM-WM and 0.43 for VILA.
  • Real-robot manipulation. With varied block positions and fixed cameras, TAM-WM with OPD placed the block in the bowl in 86.67% of trials, compared with 73.33% for the RL policy without OPD and 66.67% without transition alignment (15 trials per method).
  • Effect of ties on Demo MRR. Across 8,656 Can PH queries per method, none of the eight methods assigned the demonstrated action exactly the same score as another candidate. Random tie-breaking therefore gives the same Demo MRR as the original rule. In a separate rank-average control, 2,355 queries had a tie involving the demonstration, and the correction reduced MRR from 0.685547 to 0.646210.

We appreciate the opportunity to improve the paper. Our separate replies to each reviewer give the experimental details and address the individual questions.

AuthorsOpenReview ↗

Note to Area Chair: Summary of Key Rebuttal Evidence

Dear Area Chair,

Thank you for your time and effort in overseeing the review process. As discussions conclude, we would like to briefly summarize the core scientific contribution of TAM-WM and how our rebuttal evidence directly addresses the reviewers' primary concerns.

Core Scientific Contribution. Conventional multi-view world models rely on state-level representation alignment, yet cross-view agreement in state space does not guarantee agreement on the underlying physical transitions. TAM-WM addresses this by explicitly aligning cross-view transition displacements and learning action-conditioned dynamics within the resulting shared predictive coordinate space.

Key Rebuttal Evidence. During the rebuttal, we focused on isolating the world model's independent contribution from the downstream control stack and strengthening empirical rigor:

Isolating transition alignment: In a fixed-capacity intervention with online correction disabled, transition alignment improves Demo MRR from 0.2748 to 0.4522 and physical Spearman correlation from 0.1518 to 0.3017, while reducing regret from 0.1442 to 0.0890.

Controlled downstream comparison: Under a shared controller with identical control components and search budgets, TAM-WM achieves 83.63% ± 0.46% success across three runs, outperforming all seven baseline scorers and surpassing the strongest baseline by 4.15 percentage points.

Reviewer Consensus & Feedback. We are encouraged that Reviewer rLrG raised their rating from 2 to 3 after reviewing the shared-controller results, and that Reviewer ZSuJ recognized the value of both our transition-level formulation and controlled intervention study.

We hope this concise summary is helpful as you weigh the methodological insight alongside the new empirical evidence in your final assessment. Thank you again for your careful consideration.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.