- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies visual nuisance information in reconstruction-trained latent action models and its effect on action-conditioned world models. CD-LAM combines foreground-weighted reconstruction, contrastive learning with coarse manipulation-primitive labels, and latent calibration using free-bit KL and a duplicated-frame zero-transition objective. The pipeline fine-tunes the LAM, continues world-model training with its latent actions, and grounds robot controls through a separately fitted, frozen MLP bridge. Experiments evaluate latent- and robot-action-conditioned DreamDojo rollouts at 2B/14B, action-replacement probes, adaptation efficiency, LTX-2.3 adaptation, and matched policy pretraining. The policy study includes three paired post-training seeds on LIBERO/LIBERO-Plus and a predefined 12-task RoboTwin clean-to-random subset.
Critical strengths
-
Evidence connects latent supervision to downstream behavior. The paper evaluates encoder diagnostics, latent-conditioned rollouts, and robot-conditioned rollouts at two model scales. Stage 2 uses the same video subset and update counts for the continued-training baseline. The substantial latent-conditioned FDCE improvements, alongside visual-quality measurements and action-replacement probes, provide useful evidence for the practical value of the corrected supervision beyond reconstruction alone. This staged evaluation helps establish that modifying the upstream representation can affect downstream world-model behavior.
-
Matched policy experiments provide complementary closed-loop validation. The two LAM-based LIBERO policies use matched training recipes and three paired post-training seeds. LIBERO-Plus improves from 75.69±0.33% to 78.25±0.37% over all 10,030 variants (Appendix G). The RoboTwin subset is defined before model results, and the reported paired interval is appropriately conditional on its fixed policies and tasks. These results support the usefulness of the learned representation and reduce dependence on video-generation proxies alone.
Critical weaknesses
-
The robot-adaptation and efficiency gains are not isolated from the changed grounding procedure. Appendix B.4 fits a bridge on 10,560 training pairs for 300 epochs and freezes it during robot adaptation. The original DreamDojo procedure, Section 3.3.3, instead resets the action-conditioning input layer and jointly fine-tunes it with the world model. The Stage-3 comparison against released 50k-update checkpoints therefore changes both the latent representation and the adaptation procedure. Matching data, batch size, and optimizer does not resolve this difference. Figure 5 also lacks an otherwise matched baseline learning curve, so the released 50k endpoint does not establish when that baseline would first reach the reference performance. Without an original-LAM baseline using the same bridge fitting and staged adaptation, the robot-action and update-efficiency gains cannot be attributed specifically to latent debiasing. The Stage-2 and matched policy evidence remain useful, but do not establish this central attribution.
-
The causal and controllability interpretation needs stronger validation. Appendix C’s FDCE compares unordered foreground displacement sets and can miss incorrect assignment of motion to entities. Under action transfer, identical commands in different source states or cameras need not imply identical image-space displacement, while the construction of a physically compatible reference is not sufficiently specified. A smaller duplicated-frame latent norm also demonstrates calibration to a chosen coordinate reference rather than uniquely establishing removal of confounding. Finally, removing the combined calibration term improves FDCE from 17.23 to 16.10 in Table 5, so the necessity of the full objective for control is unresolved. These limitations affect the paper’s central explanation of improved action following.
Questions for authors
-
Matched adaptation control: What results are obtained with the original DreamDojo LAM using the same bridge architecture, fitting data/epochs, freezing, Stage-2 continuation, and Stage-3 schedule, including its adaptation learning curve? Does the robot-conditioned FDCE and adaptation-update advantage persist when all grounding steps are matched?
-
Intervention and metric validity: Is the zero-action replacement applied before or after standardization, and why is it a physical no-op for every control dimension? How are transferred actions and reference videos matched for source state and camera geometry? Can an entity-aware end-effector/object trajectory metric corroborate the FDCE ordering?
-
Objective interpretation: What backbone, evaluation split, and training budgets are used in Table 5? Can KL and zero-transition calibration be ablated separately, including their effects on ordinary and zero-action rollouts and policy success? How should the full objective’s worse FDCE than the no-calibration variant be interpreted, and what evidence supports nuisance removal beyond a change of latent coordinates?
7 comments
Part II: Metric Validation, Calibration, and Nuisance Removal
4. How are transferred actions matched to the source state and camera? We acknowledge that the original transfer pairs did not match initial states, an oversight in our experiment design. Donor-video displacement is not necessarily a valid physical counterfactual under different source states. We therefore constructed episode-start set, matched by initial arm pose (worst-joint gap about 0.10 rad, 3.1% of active range), with head-camera difference within 0.02 rad. CD-LAM reduces transfer FDCE by 30%/45% for the left/right arms, or 41% overall, relative to released DreamDojo (Table R4). Our evaluation focuses on action-induced arm motion rather than object dynamics. With closely matched initial joint poses and camera geometry, donor-arm displacement provides a meaningful reference for assessing whether the transferred robot commands are followed. For a qualitative example, see Figure 2(b) and Figure E.2 (top): the target action moves only the right arm, the released DreamDojo rollout moves the left arm instead, and CD-LAM follows the target motion at both scales.
Table R4. Episode-start pairs with matched initial pose: transfer FDCE to the target video (px, ↓).
| Model | Transfer, left arm | Transfer, right arm | Mean of the two arms |
|---|---|---|---|
| DreamDojo (released) | 9.1 | 23.9 | 16.5 |
| CD-LAM (ours) | 6.4 | 13.2 | 9.8 |
| Difference | -30% | -45% | -41% |
Reading the table. The two recordings of a pair start from nearly the same arm pose and camera view. The rollout starts from one, follows the actions of the other, and is scored against it.
5. Can an entity-aware trajectory metric corroborate the FDCE ordering? We apologize for the unclear FDCE description. We segment and track each arm separately, comparing its displacement only against the corresponding reference arm, so left/right motion swaps are penalized (Table R3, Part I). While within-arm Chamfer matching does not enforce pointwise correspondence, an independent gripper-tip trajectory error preserves the same model ordering. These results corroborate arm-specific motion following, rather than merely matching unordered foreground displacement sets. We will clarify Appendix C and release the evaluation script.
6. What setting does Table 5 use, and why is the full objective's FDCE worse? We apologize for two oversights in Table 5: the w/o L_cal row removes only the zero-transition term (KL is kept in every row because the original DreamDojo LAM objective already includes it), and we omitted the zero-motion latent residual, which is essential for evaluating calibration. Table 5 uses a 2B, 4-H100 reduced-budget ablation on an evaluation subset rather than the full main protocol. Adding the zero-motion latent column (Table R5) reveals that removing zero-motion calibration improves ordinary FDCE 17.23→16.10 and transfer FDCE (by approximately 6 px), but substantially worsens zero-motion latent residual motion from 4.88 to 15.27 px when static. This reveals an important trade-off: calibration establishes a stable null-latent reference rather than uniformly improving all motion metrics. We will correct the label.
Table R5. Table 5 with a zero-motion latent column (px, ↓).
| Variant | FDCE (Table 5) | Zero-motion latent residual |
|---|---|---|
| Full | 17.23 | 4.88 |
| w/o L_emb | 18.40 | 4.88 |
| w/o L_ctr | 19.07 | 4.93 |
| w/o L_zero ("w/o L_cal" in Table 5) | 16.10 | 15.27 |
Reading the table. Each row removes one loss term. Zero-motion latent residual: arm motion left in a rollout under a zero-motion latent (ideal: 0).
7. What supports nuisance removal beyond a change of latent coordinates? Our evidence extends beyond simply anchoring latent coordinates to zero. The null-action reference is defined by actual duplicated-frame transitions (o_t, o_t), rather than an arbitrary latent vector. Importantly, the null behavior does not rely on manually setting z=0: at the start of Stage 3, under real robot actions passed through the same bridge, an arm that the commanded action holds still moves 3.6 px with CD-LAM versus 10.0 px with the original LAM. Before any robot-action adaptation, the world model driven by real latents from its own LAM has 61%/31% lower reconstruction/transfer FDCE with CD-LAM (6.2 vs 15.9 px and 21.1 vs 30.8 px). We also observe reduced camera-shift sensitivity (0.555→0.156) and shortcut leakage (0.151→0.014). Under matched Stage-2 videos, backbones, and updates, reconstruction/transfer FDCE improve 34.00→19.63 / 42.74→33.81, before any robot-action bridge. Finally, matched policy evaluations improve LIBERO-Plus success 75.69%→78.25% and RoboTwin randomized success by 5 points. Together, these results support reduced nuisance dependence across latent representations, world-model behavior, and downstream control, rather than merely a change of latent coordinates.
Rebuttal Response and Further Clarifications
Dear Reviewer egCH,
Thank you for your valuable feedback. We have carefully addressed your concerns and provided additional experiments in our rebuttal.
As the discussion period is approaching its end, we would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns. We would be happy to clarify any remaining questions.
Thank you for your time and consideration.
Part I: Matched Adaptation and Action-Intervention Validity
Thank you for the detailed review.
1. Do the gains persist when the original LAM uses the same bridge and schedule? We re-ran two baselines: DreamDojo (re-run) with the official adaptation procedure, and DreamDojo + bridge, the original LAM with the same bridge architecture, fitting procedure, auxiliary objectives, and Stage-3 schedule as CD-LAM. The official post-training (50k updates, on data that are not specified) could not be reproduced, so all three runs are adapted on the same subset of AgiBot World for 3k updates; absolute errors are not comparable with Table 3. The answer differs by behaviour. In reconstruction and transfer the advantage does not persist: the gaps of 55%/46% narrow to 7%/-1% once the bridge is matched (Table R1). Under a zero action it does persist: DreamDojo + bridge still has 105% higher residual (17.4 vs 8.5 px), and CD-LAM adapts more stably (SD of reconstruction FDCE: 1.1 vs 3.9 px). The bridge is our own design: instead of resetting the action layer and retraining it, we map robot actions into the latent space the world model already understands, with a purpose-built objective (Appendix B.4), fitted first and then frozen. It is the main source of the faster adaptation, while debiasing accounts for the zero-action stability. We will attribute the Stage-3 gains to each component separately, present the bridge as a contribution in its own right, and replace the 12× claim with the measured learning curves (Table R2). Stage-2 debiasing is evaluated independently in Table 2, before any bridge.
Table R1. Stage 3: FDCE (px, ↓) averaged over all 20 saved checkpoints. In parentheses, increase relative to CD-LAM.
Reading the table. Recon.: rollout under the clip's own actions. Transfer: under another clip's actions, scored against that clip. Zero: under a zero action, where any motion is residual (ideal: 0).
Table R2. Stage 3: FDCE (px, ↓) by number of updates. In parentheses, increase relative to CD-LAM.
Reading the table. Zero input here is a zero robot action for both models. The re-run reaches CD-LAM's 0.6k level at about 1.4k updates in reconstruction and 2.4k in zero input, and does not reach it in transfer within 3k. The released checkpoint scores 12.7 / 31.9 / 6.9 px on the same clips.
2. Where is the zero action applied, and why is it a physical no-op? We apologize for the unclear definitions. We distinguish the physical zero action
u=0from the zero latentz=0and zero-motion latentz=LAM(O_t, O_t). For Stage 3,u=0is applied before bridge standardization. Arm joints, grippers, head, and waist use relative commands, so zero holds their current targets. The two base-velocity commands are also zero. This is distinct from setting the normalized action or bridge output to zero. Tables R1 and R2 and R3 are under the physical zero action for both models. For CD-LAM the two give similar residuals (3.0 vs 3.7 px at the 3k checkpoint).3. Can FDCE miss motion assigned to the wrong arm? FDCE tracks 16 points per arm, scoring each arm against its corresponding reference rather than pooling foreground motion. Thus, left/right motion swaps are penalized, and we will clarify Appendix C. While within-arm Chamfer matching does not establish point wise correspondence or contact correctness, gripper-tip trajectory error preserves the same model ordering, and an arm commanded to remain still moves only 1.5 px (median). Table R3 reports the per-arm results, supporting arm-motion fidelity rather than object/contact correctness.
Table R3. Per-arm FDCE (px, ↓) of CD-LAM and of the released DreamDojo checkpoint.
Reading the table. Each arm is scored separately. CD-LAM is lower on both arms in reconstruction and under a zero input, and on the left arm in transfer.