acceptodds
Under review as a conference paper at ICLR 2027

Causally Debiased Latent Action Model for Embodied Action-Conditioned World Models

Abstract

Action-conditioned world models (ACWMs) aim to simulate future observations conditioned on embodied actions. However, learning controllable ACWMs requires large-scale action-labeled data, which remains costly to collect. Latent action models (LAMs) mitigate this bottleneck by inferring latent actions from videos without executable action labels, but existing LAMs are typically trained with reconstruction-only objectives and therefore entangle action-relevant dynamics with action-irrelevant visual confounders. With such objectives, LAM favors visual context over action dynamics, therefore downstream ACWMs exhibit residual motion under zero actions and fail to reproduce supplied dynamics. We propose CD-LAM, a causally debiased framework for LAM-based ACWMs. CD-LAM introduces three debiasing objectives: embodiment-centric reconstruction, action-centric contrastive learning, and latent space calibration, which together encourage embodiment-focused, action-aware, and well-calibrated representations. CD-LAM reduces action-following error by up to 42% and 35% when conditioned on latent and robot actions, while improving fidelity, and matching DreamDojo reference with over 12 fewer adaptation updates. CD-LAM also adapts LTX-2.3-22B into an ACWM with competitive quality using only 3.2% of DreamDojo-14B's sample exposures. On X-VLA, pretraining with CD-LAM instead consistently improves success rates by up to 5% across LIBERO, LIBERO-Plus, and RoboTwin C2R.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
latent action models, representation learning, action-conditioned world models, causal debiasing, embodied AI, VLA

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_egCHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies visual nuisance information in reconstruction-trained latent action models and its effect on action-conditioned world models. CD-LAM combines foreground-weighted reconstruction, contrastive learning with coarse manipulation-primitive labels, and latent calibration using free-bit KL and a duplicated-frame zero-transition objective. The pipeline fine-tunes the LAM, continues world-model training with its latent actions, and grounds robot controls through a separately fitted, frozen MLP bridge. Experiments evaluate latent- and robot-action-conditioned DreamDojo rollouts at 2B/14B, action-replacement probes, adaptation efficiency, LTX-2.3 adaptation, and matched policy pretraining. The policy study includes three paired post-training seeds on LIBERO/LIBERO-Plus and a predefined 12-task RoboTwin clean-to-random subset.

Critical strengths
  1. Evidence connects latent supervision to downstream behavior. The paper evaluates encoder diagnostics, latent-conditioned rollouts, and robot-conditioned rollouts at two model scales. Stage 2 uses the same video subset and update counts for the continued-training baseline. The substantial latent-conditioned FDCE improvements, alongside visual-quality measurements and action-replacement probes, provide useful evidence for the practical value of the corrected supervision beyond reconstruction alone. This staged evaluation helps establish that modifying the upstream representation can affect downstream world-model behavior.

  2. Matched policy experiments provide complementary closed-loop validation. The two LAM-based LIBERO policies use matched training recipes and three paired post-training seeds. LIBERO-Plus improves from 75.69±0.33% to 78.25±0.37% over all 10,030 variants (Appendix G). The RoboTwin subset is defined before model results, and the reported paired interval is appropriately conditional on its fixed policies and tasks. These results support the usefulness of the learned representation and reduce dependence on video-generation proxies alone.

Critical weaknesses
  1. The robot-adaptation and efficiency gains are not isolated from the changed grounding procedure. Appendix B.4 fits a bridge on 10,560 training pairs for 300 epochs and freezes it during robot adaptation. The original DreamDojo procedure, Section 3.3.3, instead resets the action-conditioning input layer and jointly fine-tunes it with the world model. The Stage-3 comparison against released 50k-update checkpoints therefore changes both the latent representation and the adaptation procedure. Matching data, batch size, and optimizer does not resolve this difference. Figure 5 also lacks an otherwise matched baseline learning curve, so the released 50k endpoint does not establish when that baseline would first reach the reference performance. Without an original-LAM baseline using the same bridge fitting and staged adaptation, the robot-action and update-efficiency gains cannot be attributed specifically to latent debiasing. The Stage-2 and matched policy evidence remain useful, but do not establish this central attribution.

  2. The causal and controllability interpretation needs stronger validation. Appendix C’s FDCE compares unordered foreground displacement sets and can miss incorrect assignment of motion to entities. Under action transfer, identical commands in different source states or cameras need not imply identical image-space displacement, while the construction of a physically compatible reference is not sufficiently specified. A smaller duplicated-frame latent norm also demonstrates calibration to a chosen coordinate reference rather than uniquely establishing removal of confounding. Finally, removing the combined calibration term improves FDCE from 17.23 to 16.10 in Table 5, so the necessity of the full objective for control is unresolved. These limitations affect the paper’s central explanation of improved action following.

Questions for authors
  1. Matched adaptation control: What results are obtained with the original DreamDojo LAM using the same bridge architecture, fitting data/epochs, freezing, Stage-2 continuation, and Stage-3 schedule, including its adaptation learning curve? Does the robot-conditioned FDCE and adaptation-update advantage persist when all grounding steps are matched?

  2. Intervention and metric validity: Is the zero-action replacement applied before or after standardization, and why is it a physical no-op for every control dimension? How are transferred actions and reference videos matched for source state and camera geometry? Can an entity-aware end-effector/object trajectory metric corroborate the FDCE ordering?

  3. Objective interpretation: What backbone, evaluation split, and training budgets are used in Table 5? Can KL and zero-transition calibration be ablated separately, including their effects on ordinary and zero-action rollouts and policy success? How should the full objective’s worse FDCE than the no-calibration variant be interpreted, and what evidence supports nuisance removal beyond a change of latent coordinates?

7 comments
AuthorsOpenReview ↗

Part I: Matched Adaptation and Action-Intervention Validity

Thank you for the detailed review.

1. Do the gains persist when the original LAM uses the same bridge and schedule? We re-ran two baselines: DreamDojo (re-run) with the official adaptation procedure, and DreamDojo + bridge, the original LAM with the same bridge architecture, fitting procedure, auxiliary objectives, and Stage-3 schedule as CD-LAM. The official post-training (50k updates, on data that are not specified) could not be reproduced, so all three runs are adapted on the same subset of AgiBot World for 3k updates; absolute errors are not comparable with Table 3. The answer differs by behaviour. In reconstruction and transfer the advantage does not persist: the gaps of 55%/46% narrow to 7%/-1% once the bridge is matched (Table R1). Under a zero action it does persist: DreamDojo + bridge still has 105% higher residual (17.4 vs 8.5 px), and CD-LAM adapts more stably (SD of reconstruction FDCE: 1.1 vs 3.9 px). The bridge is our own design: instead of resetting the action layer and retraining it, we map robot actions into the latent space the world model already understands, with a purpose-built objective (Appendix B.4), fitted first and then frozen. It is the main source of the faster adaptation, while debiasing accounts for the zero-action stability. We will attribute the Stage-3 gains to each component separately, present the bridge as a contribution in its own right, and replace the 12× claim with the measured learning curves (Table R2). Stage-2 debiasing is evaluated independently in Table 2, before any bridge.

Table R1. Stage 3: FDCE (px, ↓) averaged over all 20 saved checkpoints. In parentheses, increase relative to CD-LAM.

ModelRecon.TransferZero
CD-LAM (ours)12.030.98.5
DreamDojo (re-run)18.6 (+55%)45.3 (+46%)18.4 (+116%)
DreamDojo + bridge12.8 (+7%)30.6 (-1%)17.4 (+105%)

Reading the table. Recon.: rollout under the clip's own actions. Transfer: under another clip's actions, scored against that clip. Zero: under a zero action, where any motion is residual (ideal: 0).

Table R2. Stage 3: FDCE (px, ↓) by number of updates. In parentheses, increase relative to CD-LAM.

MetricModel0.6k1.2k1.8k2.4k3k
ReconstructionCD-LAM (ours)12.110.39.47.98.2
ReconstructionDreamDojo (re-run)26.5 (+120%)13.6 (+31%)9.5 (+1%)8.2 (+3%)7.9 (-4%)
TransferCD-LAM (ours)30.031.130.730.629.9
TransferDreamDojo (re-run)57.9 (+93%)45.3 (+45%)39.0 (+27%)32.5 (+7%)31.1 (+5%)
Zero inputCD-LAM (ours)4.63.32.52.13.0
Zero inputDreamDojo (re-run)35.8 (+681%)15.3 (+363%)8.6 (+240%)4.3 (+99%)3.8 (+24%)

Reading the table. Zero input here is a zero robot action for both models. The re-run reaches CD-LAM's 0.6k level at about 1.4k updates in reconstruction and 2.4k in zero input, and does not reach it in transfer within 3k. The released checkpoint scores 12.7 / 31.9 / 6.9 px on the same clips.

2. Where is the zero action applied, and why is it a physical no-op? We apologize for the unclear definitions. We distinguish the physical zero action u=0 from the zero latent z=0and zero-motion latent z=LAM(O_t, O_t). For Stage 3, u=0 is applied before bridge standardization. Arm joints, grippers, head, and waist use relative commands, so zero holds their current targets. The two base-velocity commands are also zero. This is distinct from setting the normalized action or bridge output to zero. Tables R1 and R2 and R3 are under the physical zero action for both models. For CD-LAM the two give similar residuals (3.0 vs 3.7 px at the 3k checkpoint).

3. Can FDCE miss motion assigned to the wrong arm? FDCE tracks 16 points per arm, scoring each arm against its corresponding reference rather than pooling foreground motion. Thus, left/right motion swaps are penalized, and we will clarify Appendix C. While within-arm Chamfer matching does not establish point wise correspondence or contact correctness, gripper-tip trajectory error preserves the same model ordering, and an arm commanded to remain still moves only 1.5 px (median). Table R3 reports the per-arm results, supporting arm-motion fidelity rather than object/contact correctness.

Table R3. Per-arm FDCE (px, ↓) of CD-LAM and of the released DreamDojo checkpoint.

ModelRecon., leftRecon., rightTransfer, leftTransfer, rightZero input, leftZero input, right
DreamDojo (released)9.615.925.637.75.08.7
CD-LAM (ours)6.99.622.837.12.44.9
Difference-28%-40%-11%-2%-52%-43%

Reading the table. Each arm is scored separately. CD-LAM is lower on both arms in reconstruction and under a zero input, and on the left arm in transfer.

AuthorsOpenReview ↗

Part II: Metric Validation, Calibration, and Nuisance Removal

4. How are transferred actions matched to the source state and camera? We acknowledge that the original transfer pairs did not match initial states, an oversight in our experiment design. Donor-video displacement is not necessarily a valid physical counterfactual under different source states. We therefore constructed episode-start set, matched by initial arm pose (worst-joint gap about 0.10 rad, 3.1% of active range), with head-camera difference within 0.02 rad. CD-LAM reduces transfer FDCE by 30%/45% for the left/right arms, or 41% overall, relative to released DreamDojo (Table R4). Our evaluation focuses on action-induced arm motion rather than object dynamics. With closely matched initial joint poses and camera geometry, donor-arm displacement provides a meaningful reference for assessing whether the transferred robot commands are followed. For a qualitative example, see Figure 2(b) and Figure E.2 (top): the target action moves only the right arm, the released DreamDojo rollout moves the left arm instead, and CD-LAM follows the target motion at both scales.

Table R4. Episode-start pairs with matched initial pose: transfer FDCE to the target video (px, ↓).

ModelTransfer, left armTransfer, right armMean of the two arms
DreamDojo (released)9.123.916.5
CD-LAM (ours)6.413.29.8
Difference-30%-45%-41%

Reading the table. The two recordings of a pair start from nearly the same arm pose and camera view. The rollout starts from one, follows the actions of the other, and is scored against it.

5. Can an entity-aware trajectory metric corroborate the FDCE ordering? We apologize for the unclear FDCE description. We segment and track each arm separately, comparing its displacement only against the corresponding reference arm, so left/right motion swaps are penalized (Table R3, Part I). While within-arm Chamfer matching does not enforce pointwise correspondence, an independent gripper-tip trajectory error preserves the same model ordering. These results corroborate arm-specific motion following, rather than merely matching unordered foreground displacement sets. We will clarify Appendix C and release the evaluation script.

6. What setting does Table 5 use, and why is the full objective's FDCE worse? We apologize for two oversights in Table 5: the w/o L_cal row removes only the zero-transition term (KL is kept in every row because the original DreamDojo LAM objective already includes it), and we omitted the zero-motion latent residual, which is essential for evaluating calibration. Table 5 uses a 2B, 4-H100 reduced-budget ablation on an evaluation subset rather than the full main protocol. Adding the zero-motion latent column (Table R5) reveals that removing zero-motion calibration improves ordinary FDCE 17.23→16.10 and transfer FDCE (by approximately 6 px), but substantially worsens zero-motion latent residual motion from 4.88 to 15.27 px when static. This reveals an important trade-off: calibration establishes a stable null-latent reference rather than uniformly improving all motion metrics. We will correct the label.

Table R5. Table 5 with a zero-motion latent column (px, ↓).

VariantFDCE (Table 5)Zero-motion latent residual
Full17.234.88
w/o L_emb18.404.88
w/o L_ctr19.074.93
w/o L_zero ("w/o L_cal" in Table 5)16.1015.27

Reading the table. Each row removes one loss term. Zero-motion latent residual: arm motion left in a rollout under a zero-motion latent (ideal: 0).

7. What supports nuisance removal beyond a change of latent coordinates? Our evidence extends beyond simply anchoring latent coordinates to zero. The null-action reference is defined by actual duplicated-frame transitions (o_t, o_t), rather than an arbitrary latent vector. Importantly, the null behavior does not rely on manually setting z=0: at the start of Stage 3, under real robot actions passed through the same bridge, an arm that the commanded action holds still moves 3.6 px with CD-LAM versus 10.0 px with the original LAM. Before any robot-action adaptation, the world model driven by real latents from its own LAM has 61%/31% lower reconstruction/transfer FDCE with CD-LAM (6.2 vs 15.9 px and 21.1 vs 30.8 px). We also observe reduced camera-shift sensitivity (0.555→0.156) and shortcut leakage (0.151→0.014). Under matched Stage-2 videos, backbones, and updates, reconstruction/transfer FDCE improve 34.00→19.63 / 42.74→33.81, before any robot-action bridge. Finally, matched policy evaluations improve LIBERO-Plus success 75.69%→78.25% and RoboTwin randomized success by 5 points. Together, these results support reduced nuisance dependence across latent representations, world-model behavior, and downstream control, rather than merely a change of latent coordinates.

AuthorsOpenReview ↗

Rebuttal Response and Further Clarifications

Dear Reviewer egCH,

Thank you for your valuable feedback. We have carefully addressed your concerns and provided additional experiments in our rebuttal.

As the discussion period is approaching its end, we would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns. We would be happy to clarify any remaining questions.

Thank you for your time and consideration.

Reviewer_uEteOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CD-LAM, a framework for mitigating action-irrelevant visual confounders in latent action models (LAMs). It introduces three objectives: embodiment-centric reconstruction, action-centric contrastive learning, and latent space calibration. A three-stage training pipeline integrates the debiased latent actions into action-conditioned world models and aligns them with executable robot actions. Experiments on DreamDojo, LTX-2.3, and X-VLA demonstrate improvements in action following, visual fidelity, and downstream policy performance.

Critical strengths

1.The paper addresses visual confounding in LAMs, which is a meaningful challenge for learning controllable world models. The proposed diagnostic evaluations help illustrate this issue. 2.Experiments across multiple world-model backbones and VLA benchmarks demonstrate consistent improvements over the selected baselines, suggesting practical value beyond a single architecture.

Critical weaknesses

1.Limited novelty. The three proposed objectives largely build on existing techniques. In particular, foreground-weighted reconstruction and action-centric contrastive learning have already been explored in related works such as MaskLAM, ConLA, and CLAP. 2.The experiments primarily compare against DreamDojo without directly evaluating existing LAM debiasing methods under matched settings. It remains unclear whether the improvements arise from genuinely new contributions or simply combining established techniques. 3.Although the paper motivates its approach through causal confounding, the proposed losses are primarily representation regularization techniques. The experiments demonstrate reduced sensitivity to visual perturbations but do not sufficiently establish causal disentanglement. 4.In Table 5, removing the latent space calibration loss improves FDCE from 17.23 to 16.10, despite this loss being presented as a key component. This raises concerns about its necessity and the effectiveness of the complete objective.

Questions for authors

Clarify the methodological differences from MaskLAM, ConLA, and CLAP, and provide controlled comparisons or a straightforward combination baseline under the same DreamDojo training setup.

5 comments
AuthorsOpenReview ↗

Clarify the methodological distinctions from prior work, provide controlled objective-level comparisons, and strengthen the evidence for bias propagation from latent actions to downstream control.

1. How does CD-LAM differ from MaskLAM, ConLA and CLAP? (W1, Q) CD-LAM is not a combination of MaskLAM, ConLA, and CLAP. We identify a specific failure mechanism: action-irrelevant visual factors leak into LAM representations and propagate through ACWM conditioning to impair downstream control. We respectively design (i) embodiment-centric reconstruction that emphasizes embodiment and interacted objects while retaining background supervision, unlike MaskLAM's agent-only masking. We use (ii) action-centric contrastive learning with weighted pairwise sigmoid supervision on continuous latents, distinct from ConLA's separate action/visual contrastive objectives and quantization, and CLAP's visual–proprioceptive alignment. We add (iii) latent calibration to establish a stable zero-transition reference. These differences concern formulation, representation, and supervision, not just applications (Table R1). Crucially, our objectives are derived from failure modes across the LAM → WM → robot-control pipeline. We systematically validate their effects through encoder diagnostics, matched Stage-2 rollouts, action interventions, and closed-loop policy evaluations.

Table R1. Representation, objectives, and supervision.

MethodRepresentation / reconstructionContrastive objectiveSupervision & downstream
MaskLAMBinary agent-only reconstruction (zero background loss)NoneUnlabeled video → BC policy
ConLAExplicit action/visual latent split; VQ-discretized action tokensAction-class SupCon and temporal-reversal visual InfoNCEVideo + action-class labels → VLA
CLAPRobot-action VQ codebook; separate visual action/nuisance latentsSigLIP visual–proprio alignment (robot); EMA-teacher positives (human)Action-labeled robot + human video → VLA
CD-LAMContinuous latents; embodiment/object-weighted reconstruction with nonzero background lossWeighted pairwise sigmoid across action primitives; no latent splitting or VQAction-free video + coarse primitive labels → WM → robot control → policy

2. Are the gains new or a combination of existing techniques, and is calibration necessary? (W2, W4, Q) Directly adapting MaskLAM, ConLA, or CLAP as published would drop background supervision needed for full-frame generation or change DreamDojo's continuous action interface or available supervision. We instead provide objective-level controls in the same DreamDojo pipeline, including a combination baseline. Removing foreground weighting/contrast raises FDCE from 17.23 to 18.40/19.07. The straightforward foreground + contrast + KL combination, without zero-transition calibration (Table R2), improves FDCE to 16.10 but worsens zero-motion latent residual from 4.88 to 15.27 px. This isolates what the additional calibration supplies: a stable null-latent reference, not a guarantee of lower FDCE on every rollout.

We apologize for a Table 5 labeling error: w/o L_cal removes only the zero-transition term. KL is kept in every row because the original DreamDojo LAM objective already includes it. We have corrected the label in the revised manuscript.

Table R2. Table 5 with a zero-motion latent column (px, ↓).

VariantFDCE (Table 5)Zero-motion latent residual
Full17.234.88
w/o L_emb18.404.88
w/o L_ctr19.074.93
w/o L_zero ("w/o L_cal" in Table 5)16.1015.27

Reading the table. Each row removes one loss term. Zero-motion latent residual: arm motion left in a rollout under a zero-motion latent (ideal: 0). Removing the zero-transition term lowers FDCE by 1.1 px but triples the motion left under a zero-motion latent (4.88 to 15.27 px). Removing either other term leaves it unchanged.

AuthorsOpenReview ↗

Causal Claim

3. Do regularization losses establish causal disentanglement? We agree that these regularizers do not prove formal causal disentanglement, and we will make that scope explicit. Table R3 puts the evidence side by side.

(1) In this paper we focus on debiasing the ACWM with causality-driven regularizers, rather than on perfectly identifying the causal structure.

(2) Our motivation starts from the poor action following of the original ACWM (Tables 2-3): under a zero action the arms still move by 10.71 px, a transferred action is missed (42.74 px against the target), and even the model's own latent actions are followed poorly (34.00 px). A 7× larger backbone does not help (9.36, 50.27 and 40.29 px at 14B).

(3) We find that the confounded, collapsed latent-action representation is the fundamental problem (Table 1): the original LAM encodes a 3-px camera shift as 0.555 of a typical action, gives two identical frames a latent of relative norm 0.527, and groups latents by scene more than by action (shortcut leakage 0.151).

(4) The problem is therefore to reduce the effect of these confounders in the ACWM, i.e. to debias the model efficiently, instead of learning a new causal structure from scratch, which requires large-scale data and GPU resources (DreamDojo's latent-action pretraining alone uses 140k updates on 256 H100 GPUs).

(5) Thus we debias directly, and the results show that this is enough to greatly improve performance at low cost (1k LAM updates and 2k ACWM updates): the camera-shift response drops 0.555→0.156, Stage-2 reconstruction FDCE 34.00→19.63 and transfer FDCE 42.74→33.81 before any robot-action bridge, and LIBERO-Plus success rises 75.69%→78.25% (Table R3). Direct debiasing is also more efficient for large-scale ACWMs.

(6) Our next step is to test identifiability directly with interventional data, for example simulator-rendered pairs in which only the camera, the background or the action is changed.

Table R3. Original WM (DreamDojo) vs. our causally-debiased WM (CD-LAM). Numbers are from the paper unless marked "new".

LevelMeasurementIdealOriginal WM (DreamDojo)Causally-debiased WM (CD-LAM)
Latent action (Table 1)Response to a 3-px camera shift, action fixed ↓00.5550.156
Latent action (Table 1)Static frame pair, relative latent norm ↓00.5270.043
Latent action (Table 1)Shortcut leakage ↓≤ 00.1510.014
World model, Stage 2 (Table 2)FDCE under its own latent actions (px) ↓034.0019.63
World model, Stage 2 (Table 2)FDCE under transferred latent actions (px) ↓042.7433.81
World model before robot adaptation (new)FDCE under its own latent actions, robot videos (px) ↓015.96.2
World model before robot adaptation (new)FDCE under transferred latent actions, robot videos (px) ↓030.821.1
World model, Stage 3 (Table 3)Residual motion under a zero action (px) ↓010.715.03
Policy, X-VLA (Table 6)LIBERO-Plus success ↑100%75.69%78.25%
Policy, X-VLA (Table 6)RoboTwin randomized-scene success ↑100%28.83%33.83%

Reading the table. Rows go from the latent action to the world model (2B) and the policy. Camera shift / static pair: latent norm for a shifted or duplicated frame, relative to a real transition. No action happens, so the ideal is 0. Shortcut leakage: similarity of latents from the same episode with different actions, minus that of the same action across episodes. new: measured for this rebuttal, before any robot-action adaptation. Every level improves: the latent responds less to a camera shift (0.555 to 0.156), the world model follows its actions more closely (34.00 to 19.63 px), and policy success rises (75.69% to 78.25%).

AuthorsOpenReview ↗

Rebuttal Response and Further Clarifications

Dear Reviewer uEte,

Thank you for your valuable feedback. We have carefully addressed your concerns and provided additional experiments in our rebuttal.

As the discussion period is approaching its end, we would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns. We would be happy to clarify any remaining questions.

Thank you for your time and consideration.

Reviewer_S6T1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how visual distractions in latent action models affect action-conditioned world models. CD-LAM combines foreground-weighted reconstruction, action-category contrastive learning, and latent-space calibration. The resulting latents are used to continue world-model training, followed by robot-action adaptation through a learned action bridge. Experiments on DreamDojo, LTX, and X-VLA evaluate video prediction, action following, and downstream policy performance.

Critical strengths
  1. The Stage 2 comparison is well controlled. The authors continue training DreamDojo on the same videos with matched update counts. At 2B, FDCE decreases from 34.00 to 19.63, while visual quality also improves. This supports the value of changing the latent action representation; additional world-model training alone does not explain the gain.

  2. The evaluation covers latent diagnostics, world-model predictions, and downstream policies. The matched X-VLA comparison is particularly useful. CD-LAM improves LIBERO-Plus success from 75.69% to 78.25% and RoboTwin randomized-scene success from 28.83% to 33.83%. These results show that the learned representations have practical value beyond video prediction.

Critical weaknesses
  1. The methodological novelty is limited. Foreground-weighted reconstruction closely follows the idea behind MaskLAM, while action-category contrastive learning has been explored in ConLA. The authors acknowledge these connections, but mainly compare the full method against the original DreamDojo. There is no direct comparison with existing debiasing approaches under the same pipeline. The clearest contribution is applying these ideas to large world models and evaluating their effects. The paper has not yet established a substantially different debiasing method.

  2. The Stage 3 comparison does not isolate the contribution of latent-action debiasing. This stage also introduces an action bridge trained with action reconstruction and cycle consistency, whereas the baseline is DreamDojo’s released 50k-update checkpoint. The paper does not report the original LAM with the same bridge and adaptation procedure. The gains may therefore come from both the representation and the adaptation method. The efficiency claim has a related limitation: Figure 5 shows only the baseline’s final score, not its learning curve. Reaching that score in fewer updates does not establish that the baseline needed 50k updates or that convergence is over 12 times faster.

  3. The target-action transfer experiment does not establish a valid counterfactual reference. Appendix B.5 transfers actions from another video and expects the generated motion to match that video. However, the same action can produce different motion under different initial poses, object locations, and contact states. The paper does not explain how these differences are controlled. FDCE compares sets of foreground displacements in image space, which cannot resolve whether the reference motion is appropriate for the source scene. These experiments support improved motion transfer, but provide limited evidence of physically correct action responses.

  4. The role of calibration in improving control remains unclear. In Table 5, removing LcalL_{\mathrm{cal}} substantially worsens camera-shift responses but improves Stage 3 FDCE from 17.23 to 16.10. The authors acknowledge this trade-off. Still, it shows that better latent diagnostics do not necessarily lead to better action following. The paper neither separates the KL and zero-transition terms nor reports zero-action and target-action transfer results for this ablation. It is therefore unclear which control behaviors benefit from calibration.

Questions for authors
  1. How much of the Stage 3 gap remains when the original DreamDojo LAM uses the same action bridge, losses, and adaptation procedure? Can you provide learning curves for both methods?
  2. Under matched backbones and training budgets, what does CD-LAM add beyond foreground reconstruction alone, category-based contrastive learning alone, and their combination?
  3. How do you ensure that the target video provides an appropriate motion reference for the source scene? Can you evaluate different actions from the same initial state using actual resulting trajectories?
  4. What happens to standard prediction, zero-action behavior, and action transfer when the KL and zero-transition terms are removed separately? Can these results explain the trade-off in Table 5?
7 comments
AuthorsOpenReview ↗

Part I: Novelty and What Each Objective Adds

Thank you for the thoughtful review.

  1. Is the method substantially different from MaskLAM and ConLA? We study a mechanism, not only a loss. CD-LAM is not a combination of MaskLAM, ConLA or other loss. We trace how action-irrelevant visual factors leak into latent actions and impair control through WM conditioning, and we design each stage of the pipeline around this. On the LAM side, we design (i) embodiment-centric reconstruction that preserves scene consistency while emphasizing embodiment and interacted objects, unlike MaskLAM's agent-only masking; (ii) action-centric contrastive learning with a weighted pairwise SigLIP objective on continuous latents, distinct from ConLA's SupCon/InfoNCE with feature splitting and quantization; and (iii) latent calibration to establish a stable zero-transition reference. On the WM training and Policy Model training side, we design (iv) a latent-space bridge: a pre-fitted, frozen bidirectional action-to-latent mapping trained with a cycle-consistency objective (Sec. 4.3, Appendix B.4). MaskLAM and ConLA connect robot actions on the policy side, through an action decoder or a replaced action head, while we instead map robot actions into the latent space the world model already uses. These designs target failure modes across the LAM→WM→robot-policy control pipeline, rather than latent representations in isolation. We validate them through encoder diagnostics, matched world-model experiments, action interventions, a matched-bridge control (Part II), and closed-loop policy evaluations.

Table R1. What each work studies.

MaskLAMConLACD-LAM (ours)
Latent action drivesa BC policya VLA (discrete tokens)a video world model, and a policy
World-model control studiednonoyes (2B, 14B)
WM interventionsnoone qualitative figureyes: zero action, transferred action
Robot actions enter viaan action decoder after the latent policya new action head replacing the latent headour pre-fitted, frozen bridge into the latent space

2. What does CD-LAM add beyond foreground weighting, contrast and their combination? We apologize for an error in the paper's Table 5 description: the w/o L_cal row removes only the zero-transition term. KL is kept in every row because the original DreamDojo LAM objective already includes it. We added zero-motion latent residual scores to clarify its contribution (Table R2). Removing foreground weighting or contrast raises ordinary FDCE from 17.23 to 18.40 or 19.07. Their straightforward combination without zero-transition improves ordinary FDCE to 16.10, but residual motion rises from 4.88 to 15.27 px. Thus, calibration establishes a usable null-action reference rather than uniformly optimizing reconstruction. Regarding Q4: with KL kept, only the zero-transition term is ablated. Removing it lowers standard-prediction FDCE (17.23→16.10) and transfer FDCE (by about 6 px in this setting) but raises the zero-motion latent residual motion (4.88→15.27 px), which explains the trade-off in Table 5.

Table R2. Table 5 with a zero-motion latent column (px, ↓).

VariantFDCE (Table 5)Zero-motion latent residual
Full17.234.88
w/o L_emb18.404.88
w/o L_ctr19.074.93
w/o L_zero ("w/o L_cal" in Table 5)16.1015.27

Reading the table. Each row removes one loss term. Zero-motion latent residual: arm motion left in a rollout under a zero motion latent (ideal: 0). Removing the zero-transition term lowers FDCE by 1.1 px but triples the motion left under a zero-motion latent (4.88 to 15.27 px). Removing either other term leaves it unchanged.

AuthorsOpenReview ↗

Part II: Stage-3 Attribution with a Matched Bridge

3. How much of the Stage-3 gap remains with the same bridge, and what do the learning curves show? Thank you for highlighting this missing control. As the review notes, Stage 3 changes two things relative to DreamDojo, and both are our designs: the debiased latent and the bridge. DreamDojo resets the action layer and retrains it, while our bridge gg maps robot actions uu into the latent space the world model already understands. It is trained with a purpose-built objective (Appendix B.4), which combines latent regression, cycle consistency through an action readout hh, and action decoding from the latent. It is fitted first and then frozen, and serves both our world-model and policy pipelines. To separate the two, we re-ran DreamDojo (re-run) with its official procedure and DreamDojo + bridge, the original LAM with our bridge. On average, the official re-run has 55%/46% higher reconstruction/transfer FDCE than CD-LAM (Table R4); with our bridge the gap narrows to 7% in reconstruction and vanishes in transfer. The bridge is therefore the main source of the faster Stage-3 adaptation, and we will present it as a contribution in its own right. What remains with the same bridge is due to debiasing: under a zero action, DreamDojo + bridge has 105% more residual motion (17.4 vs 8.5 px), and CD-LAM adapts more stably (SD of reconstruction FDCE: 1.1 vs 3.9 px). Matched Stage-2 and policy results support debiasing independently of any bridge. Table R3 gives the learning curves, which we will add to the paper. Table R3. Stage 3: FDCE (px, ↓) by number of updates. In parentheses, increase relative to CD-LAM.

MetricModel0.6k1.2k1.8k2.4k3k
ReconstructionCD-LAM (ours)12.110.39.47.98.2
ReconstructionDreamDojo (re-run)26.5 (+120%)13.6 (+31%)9.5 (+1%)8.2 (+3%)7.9 (-4%)
TransferCD-LAM (ours)30.031.130.730.629.9
TransferDreamDojo (re-run)57.9 (+93%)45.3 (+45%)39.0 (+27%)32.5 (+7%)31.1 (+5%)
Zero actionCD-LAM (ours)4.63.32.52.13.0
Zero actionDreamDojo (re-run)35.8 (+681%)15.3 (+363%)8.6 (+240%)4.3 (+99%)3.8 (+24%)

Reading the table. Reconstruction: rollout under the clip's own actions. Transfer: under another clip's actions, scored against that clip. Zero action: under a zero robot action, where any motion is residual (ideal: 0). DreamDojo (re-run): the official procedure, no bridge. The re-run reaches CD-LAM's 0.6k level at about 1.4k updates in reconstruction and 2.4k under a zero action, and does not reach it in transfer within 3k. The released checkpoint scores 12.7 / 31.9 / 6.9 px on the same clips.

Table R4. Stage 3: FDCE (px, ↓) averaged over all 20 saved checkpoints. In parentheses, increase relative to CD-LAM.

ModelRecon.TransferZero action
CD-LAM (ours)12.030.98.5
DreamDojo (re-run)18.6 (+55%)45.3 (+46%)18.4 (+116%)
DreamDojo + bridge12.8 (+7%)30.6 (-1%)17.4 (+105%)

Reading the table. Columns as in Table R3. DreamDojo + bridge: the original LAM with our bridge. The bridge closes most of the reconstruction/transfer gap (55%/46% to 7%/-1%), while the zero-action gap remains (17.4 vs 8.5 px).

4. Does Figure 5 establish 12× faster convergence? We agree that the original "12×" does not establish convergence speed. We could not reproduce the official post-training budget (50k updates, on data that are not specified), so we ran an independent comparison: the official DreamDojo procedure and CD-LAM are adapted on the same subset of AgiBot World with the same optimizer settings and schedule for 3k updates. In this setting adaptation converges quickly. Early in adaptation, DreamDojo requires 2.4–4× more updates to reach the same reconstruction FDCE, and it does not reach CD-LAM's transfer FDCE within 3k updates (Table R5). The gap narrows as training proceeds. We will revise this 12× claim.

Table R5. Stage-3 updates needed to reach the same FDCE.

MetricCD-LAM reaches the level afterDreamDojo (re-run) reaches the same level afterRe-run / ours
Reconstruction300 updates1,200 updates4.0×
Reconstruction500 updates1,200 updates2.4×
Reconstruction2,000 updates2,400 updates1.2×
Transfer300 updatesnot reached by 3k–
Transfer500 updatesnot reached by 3k–
Transfer2,000 updatesnot reached by 3k–

Reading the table. Each row takes the FDCE CD-LAM has after the given number of updates and gives the first saved checkpoint at which the re-run reaches it. In reconstruction the re-run needs 2.4–4× more updates early and 1.2× by 2k. In transfer it does not reach CD-LAM's level within 3k.

AuthorsOpenReview ↗

Part III: Transfer Reference

5. Is the target video an appropriate motion reference for the source scene? We acknowledge that donor-video motion is not a strict physical counterfactual under different initial states, and will clarify this limitation. To provide a more controlled evaluation, we constructed an episode-start set with closely matched initial arm poses (worst-joint gap about 0.10 rad, 3.1% of range, with the camera gap within 0.02 rad). As shown in Table R6, CD-LAM reduces transfer FDCE by 41% on average. For unmatched pairs, transfer FDCE remains a quantitative reference rather than a measure of physical correctness. For reference Video, please check appendix Figure E.2 and Figure E.3.

Table R6. Episode-start pairs with matched initial pose: transfer FDCE to the target video (px, ↓).

ModelTransfer, left armTransfer, right armMean of the two arms
DreamDojo (released)9.123.916.5
CD-LAM (ours)6.413.29.8
Difference-30%-45%-41%

Reading the table. The two recordings of a pair start from nearly the same arm pose and camera view. The rollout starts from one, follows the actions of the other, and is scored against it. With the initial pose matched, CD-LAM follows the transferred action more closely on both arms (-30% left, -45% right).

Reviewer_cvtCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies visual-context leakage in latent action models (LAMs), arguing that reconstruction-based training entangles embodied motion with background and appearance information, weakening downstream world-model controllability. It proposes CD-LAM, which combines foreground-weighted reconstruction, contrastive learning using coarse action-primitive annotations, and latent-space calibration. A three-stage pipeline debiases the LAM, fine-tunes the world model with its latents, and aligns executable robot actions through an MLP bridge.

Validation includes latent-space diagnostics, action-conditioned rollouts, zero-action and action-transfer probes, objective ablations, and downstream policy evaluation. On DreamDojo-2B/14B, CD-LAM reduces latent-conditioned FDCE by approximately 42%/26% and robot-action-conditioned FDCE by 35%/30%, while improving visual fidelity. Additional experiments demonstrate LTX adaptation and policy gains on LIBERO, LIBERO-Plus, and a selected RoboTwin subset.

Critical strengths

The paper examines an important failure mode at both representation and behavioral levels. Encoder diagnostics are complemented by downstream rollouts and action interventions, providing substantially stronger evidence than visual-fidelity improvements alone. The gains persist after executable-action adaptation at both model scales.

The representation has demonstrated downstream utility. The matched CD-LAM versus original-LAM policy experiments show benefits beyond video reconstruction. In particular, LIBERO-Plus improves by 2.56 percentage points across three paired post-training seeds, and RoboTwin randomized-scene success improves by 5 points on the evaluated subset.

Critical weaknesses

FDCE provides incomplete evidence of correct action following. It compares unordered sets of foreground displacement vectors, so it can overlook which arm or object executes a motion. Furthermore, transferring an action sequence to a different initial state does not generally imply that the resulting image-space motion should match the donor video, because motion depends on state, viewpoint, and contact. These limitations weaken the interpretation of the central controllability results, especially the transfer probes.

The evidence only partially isolates the proposed mechanism. In Table 5, removing calibration substantially worsens camera-shift diagnostics but improves downstream FDCE from 17.23 to 16.10. This shows that improved diagnostic scores do not consistently translate into better control. The separate ablation setting is also insufficiently connected to the main results. Additionally, LTX lacks a matched original-LAM control, so that experiment establishes feasibility more clearly than a CD-LAM-specific efficiency advantage.

Questions for authors

Metric clarification: How are source and target clips selected for action transfer, and what makes donor-video displacement a valid reference under the source initial state? Can the main gains be corroborated with arm/object-specific correspondence metrics or simulator-grounded action outcomes?

Ablation clarification: What differs between Table 5’s setting and the main experiments? Does removing calibration also improve FDCE under the main protocol, and how does it affect zero-action and action-transfer performance?

Attribution clarification: Does the Stage-3 baseline use the same bridge-fitting procedure and auxiliary objectives? For LTX, is there a matched comparison using the original LAM under identical training budgets?

5 comments
AuthorsOpenReview ↗

Part I: Metric Validity: FDCE and Transfer Reference

Thank you for the careful review. We clarify FDCE, add state-matched transfer and matched-bridge controls, and revise efficiency attribution.

1. Does FDCE capture which arm moves, and do other metrics agree? We apologize for the unclear metric description. FDCE is computed separately for left and right arms, not by pooling all foreground tracks: 16 points per arm are compared only with that arm's reference, so left/right swaps are penalized. Within-arm Chamfer matching does not establish point or object identity; FDCE is a proxy for arm displacement, not grasp/contact success. Per-arm results and gripper-tip tracking preserve the model ordering (Table R1): an independent gripper-tip trajectory error ranks the models in the same order as FDCE (Kendall tau = 1.0), and an arm that the transferred action holds still moves only 1.5 px (median). As independent simulator-grounded evidence, matched X-VLA training with CD-LAM versus original-LAM targets improves LIBERO-Plus success 75.69→78.25% (3 paired seeds) and RoboTwin-C2R randomized success 28.8→33.8%. These are policy outcomes, not direct validation of every world-model rollout.

Table R1. Per-arm FDCE (px, ↓) of CD-LAM and of the released DreamDojo checkpoint.

ModelRecon., leftRecon., rightTransfer, leftTransfer, rightZero-motion, leftZero-motion, right
DreamDojo (released)9.615.925.637.75.08.7
CD-LAM (ours)6.99.622.837.12.44.9
Difference-28%*-40%*-11%*-2%-52%*-43%*

Reading the table. Each arm is scored separately. Recon.: rollout under the clip's own actions; Transfer: under another clip's actions, scored against that clip; Zero-motion: under a zero-motion latent encoded by LAM, where any motion is residual (ideal: 0). Difference: CD-LAM relative to the released checkpoint. CD-LAM is lower on both arms in reconstruction (-28%, -40%) and under a zero input (-52%, -43%), and on the left arm in transfer (-11%); right-arm transfer is on par.

2. Is the donor video a valid reference under a different initial state? We agree that the original transfer pairs did not match initial states, and donor-video displacement is not generally the correct outcome under a different initial state. We will correct the original “reproduce target dynamics” claim. We therefore construct an episode-start set with matched initial arm poses (worst-joint gap about 0.10 rad, 3.1% of range, and head-camera orientation within 0.02 rad). With target actions under the matched source frame, CD-LAM has lower per-arm error (R2), 40% lower than the baseline. This supports arm-motion alignment under restricted initial-state variation, not contact-accurate counterfactual prediction: object/contact states are not matched. We will report unmatched transfer as a descriptive test, and ordinary reconstruction and policy outcomes remain independent evidence. For a qualitative example, see Figure 2(b) and Figure E.2 (top): the target action moves only the right arm, the released DreamDojo rollout moves the left arm instead, and CD-LAM follows the target motion at both scales.

Table R2. Episode-start pairs with matched initial pose: transfer FDCE to the target video (px, ↓).

ModelTransfer, left armTransfer, right armMean of the two arms
DreamDojo (released)9.123.916.5
CD-LAM (ours)6.413.29.8
Difference-30%*-45%*-41%*

Reading the table. The two recordings of a pair start from nearly the same arm pose and camera view; the rollout starts from one, follows the actions of the other, and is scored against it. With the initial pose matched, CD-LAM follows the transferred action more closely on both arms (-30% left, -45% right).

AuthorsOpenReview ↗

Part II: Ablation Setting and Stage-3 Attribution

3. How does Table 5 differ from the main setting, and why does removing calibration improve FDCE? Table 5 uses a separate, smaller 2B/4-GPU, shorter-schedule setting, not the main 128-GPU protocol. We also identified a labeling error: "w/o L_cal" removes only the zero-transition term. The KL term is kept in every variant because the original DreamDojo LAM objective already includes it. We will relabel it “w/o L_zero”. Removing this term improves ordinary FDCE 17.23→16.10 (and transfer in this small setting), but worsens the zero-motion-latent rollout residual 4.88→15.27 px (R3). Thus calibration improves the designated null-latent behavior, not every downstream motion metric. We will explicitly present this trade-off: whether it holds in the full main protocol is not established by this ablation.

Table R3. Table 5 with a zero-motion-latent column (px, ↓).

VariantFDCE (Table 5)Zero-motion latent residual
Full17.234.88
w/o L_emb18.404.88
w/o L_ctr19.074.93
w/o L_zero ("w/o L_cal" in Table 5)16.1015.27

Reading the table. Each row removes one loss term. Zero-motion latent residual: arm motion left in a rollout under a zero-motion latent (ideal: 0). Removing the zero-transition term lowers FDCE by 1.1 px but triples the motion left under a zero-motion latent (4.88 to 15.27 px) while removing either other term leaves it unchanged.

4. Does the Stage-3 baseline use the same bridge fitting and auxiliary objectives? No. The bridge is our own design: DreamDojo's official procedure resets the action layer and retrains it with the world model, whereas we map robot actions into the latent action space the world model already understands, using a three-layer MLP trained with a purpose-built objective (latent regression, cycle consistency through an action readout, and action decoding; Appendix B.4), fitted before adaptation and then frozen. To separate this design from debiasing, we re-ran DreamDojo for 3k updates both with its official procedure and with our bridge and the same auxiliary objectives. With our bridge, the original LAM closes most of the gap: the reconstruction/transfer FDCE gaps to CD-LAM narrow from 55%/46% to 7%/-1% (Table R4). Latent-space grounding through our bridge is therefore the main source of the faster Stage-3 adaptation. The paper credited this gain to debiasing alone; we revise the attribution and will present the bridge as a contribution in its own right. Debiasing contributes what the bridge does not: with the same bridge, DreamDojo + bridge still has 105% higher zero-action residual than CD-LAM (17.4 vs 8.5 px), where the zero input is a zero robot action u=0u=0 passed through the bridge. CD-LAM also adapts more stably (SD of reconstruction FDCE: 1.1 vs 3.9 px). Independently of any bridge, the matched Stage-2 comparison and the policy results support debiasing, and before any robot-action adaptation the CD-LAM world model driven by its own LAM's latents already has much lower reconstruction/transfer FDCE than the original DreamDojo model (6.2 vs 15.9 px and 21.1 vs 30.8 px).

Table R4. Stage 3: FDCE (px, ↓) averaged over all 20 saved checkpoints. In parentheses, increase relative to CD-LAM.

ModelRecon.TransferZero action
CD-LAM (ours)12.030.98.5
DreamDojo (re-run)18.6 (+55%)45.3 (+46%)18.4 (+116%)
DreamDojo + bridge12.8 (+7%)30.6 (-1%)17.4 (+105%)

Reading the table. Columns as in Table R1. DreamDojo (re-run): the official procedure, no bridge. DreamDojo + bridge: the original LAM with our bridge. The bridge closes most of the reconstruction/transfer gap (55%/46% to 7%/-1%), while the zero-action gap remains (17.4 vs 8.5 px).

5. Is there a matched original-LAM control for LTX? We agree that there is no original-LAM LTX control at identical budget. This comparison is compute-heavy and is still running. We will add it once it finishes. Until then we present LTX as cross-backbone feasibility, not as evidence that CD-LAM uniquely improves LTX adaptation efficiency.

AuthorsOpenReview ↗

Rebuttal Response and Further Clarifications

Dear Reviewer cvtC,

Thank you for your valuable feedback. We have carefully addressed your concerns and provided additional experiments in our rebuttal.

As the discussion period is approaching its end, we would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns. We would be happy to clarify any remaining questions.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of New Experiments in Our Rebuttal

We thank all reviewers for the careful and constructive reviews. To make the discussion easier to follow, we summarize the experiments we added during the rebuttal.

  1. Stage-3 decomposition with a matched bridge. We separated the two designs in our Stage 3 by training the original LAM with our bridge. We find that our specific bridge design contributes the fast adaptation to robot actions, while latent debiasing contributes zero-action stability: with the same bridge, the original LAM still shows about twice the residual motion under a zero action.
  2. Learning curves. We re-ran the official DreamDojo procedure under the same data and schedule and compared the Stage-3 learning curves. We observe that CD-LAM adapts markedly faster early in training.
  3. Per-arm FDCE and an end-effector metric. We report FDCE separately for each arm and add a gripper-tip trajectory error. We find that CD-LAM is lower on both arms in reconstruction and under a zero input, and that the gripper-tip metric gives the same model ordering.
  4. State-matched transfer. We built a transfer set in which source and target start from matched arm poses and camera views. On this set, we show that CD-LAM follows the transferred action more closely on both arms (41% lower transfer FDCE).
  5. Ablation with a zero-motion-latent column. We added the missing zero-motion-latent evaluation to the ablation. We show that the zero-transition term is what provides a stable null reference: removing it triples the residual motion under a zero-motion latent.
  6. Latent controllability before robot adaptation. We evaluated the world models on robot videos before any robot-action adaptation, driven by the latents of their own LAM. We find that the CD-LAM world model already follows actions far more closely than the original model.

Details and tables are in our responses to each reviewer, and the paper is being revised accordingly. We would be grateful if the reviewers could let us know whether these additions address their concerns, and we are happy to provide any further clarification before the discussion period ends.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.