acceptodds
Under review as a conference paper at ICLR 2027

DOLA: Latent Action Pretraining from Delta Observations

14 views

Abstract

Latent action models learn action representations from action-free video, scaling visuomotor learning beyond the limited labeled demonstrations available in robotics. These models typically train a latent inverse dynamics model jointly with a forward dynamics model on paired observations, hoping the latent captures the underlying action through bottlenecked reconstruction. However, since actions cause state changes, an action representation should reflect the change in state it produces. Building on this inductive prior, we present DOLA (Delta-Obs Latent Actions), a new latent action model that explicitly conditions the inverse dynamics on the difference between consecutive observation embeddings, jointly trained with the encoder and an autoregressive forward dynamics model that reconstructs multiple future embeddings. On a suite of four simulated environments, DOLA learns latent actions that linearly recover ground-truth actions across both manipulation and navigation tasks. A single DOLA model trained on unpaired data from two robots, grounded by heterogenous joint velocity actions, learns a shared latent space that linearly recovers end-effector Cartesian deltas and supports open-loop trajectory transfer across embodiments. On a real Franka arm, DOLA executes zero-shot video plans from an off-the-shelf video generator with 75 minutes of task-agnostic random calibration data. Finally, we ablate key components of DOLA and measure its impact on representation quality and downstream task performance. Videos are best viewed at https://dola-anon.github.io

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Latent action, Robotics, Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_39yXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DOLA, which learns latent action representations from differences between consecutive visual embeddings through joint training with forward dynamics prediction and action supervision. The authors evaluate action representations and downstream policies across four simulated environments, and demonstrate cross-embodiment transfer and real-robot execution of generated video plans.

Critical strengths

The method is simple, and the ablation results support the benefit of delta observations across all four environments. With relatively few parameters, DOLA learns competitive action representations and achieves clear downstream policy improvements on LIBERO and MimicGen.

Critical weaknesses

The methodological contribution is incremental. Difference-based representations and auxiliary action supervision have already been discussed in prior work on Linear LAM. DOLA incorporates these ideas into a jointly learned nonlinear temporal model, providing some empirical value, but its conceptual advance over prior work remains limited.

The application experiments lack aggregate statistics on task execution. Although the cross-embodiment experiments provide quantitative analyses of the learned representations, task execution is primarily illustrated through individual examples. The real-robot video plans are manually screened, without reporting the proportion retained or the overall execution success rate. These results support feasibility but do not provide sufficient evidence to assess execution reliability, limiting the support for the method’s practical utility.

Questions for authors

With the visual encoder, delta-conditioned inverse dynamics model, action-label budget, and inference procedure held fixed, how do linear probing and downstream policy performance change when the forward dynamics prediction loss is removed? This comparison would help establish the additional benefit of forward prediction training beyond action supervision.

Please report the total number of attempts and successes in the existing cross-embodiment and real-robot experiments, along with the manual screening criteria and the proportion of generated video plans retained. These statistics would affect my assessment of the application contributions.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

(Edit: we’ve copied some of the Global Response tables and details down here so that it’s easier to read, as well as added more details in our response to your contribution/novelty question. Let us know if you have more questions!)

As a summary, we clarify our contribution, add the requested action-supervision-only ablation, and quantitative cross-embodiment and real-robot execution evaluations below. Most of the details are in the Global Responses at the very top of the page, so we’ve reproduced the relevant tables below in the comments for your convenience.

W1 - Methodological contribution

Thanks for your comment on Linear LAM! First, we would like to note that Linear LAM defines its IDM as z = Co + Do′, with independently learned matrices C and D, and does not explicitly compute an observation difference. We implement it as formulated as a baseline in the original manuscript.

To clarify our contribution, DOLA learns action representations from explicit differences in a jointly learned feature space. We jointly learn the visual encoder, IDM, FDM, and action head with JEPA-style feature prediction and limited action supervision. This formulation achieves the lowest action-probe MSE on the three manipulation benchmarks and strong downstream policy performance with a compact model.

In Global Responses #1 and #2, our controlled ablations show why both explicit differencing and learning the feature space matter. At a fixed two-frame context, delta inputs raise MimicGen Stack policy success from 20% to 85%. With the same ResNet-34 encoder architecture and delta inputs, jointly learning the encoder raises success from 26% with frozen ImageNet features to 90%. Global Response #1 reports the full encoder comparison, and #2 separates differencing from temporal context.

Regarding action supervision, prior works such as Linear LAM and LAOM have shown that action supervision is necessary to align the latent action space, and we do not claim this as a contribution. We want to show empirically that this is the case, and indeed, our Table 1 corroborates this observation. For more details on action supervision, please also see our response to gusE W2/Q2.

W2, Q2 - Task execution statistics, and video-plan screening

Cross-embodiment transfer. DOLA achieves lower per-step cross-embodiment decoding error than DreamDojo in both directions, though accumulated drift limits open-loop success for both methods. In closed-loop transfer to a new task, Panda co-training raises UR5e Stack success from 50% to 72%, compared with 64% for co-trained DreamDojo, without UR5e Stack action labels. Please see Global Response #3 for the full discussion. For your convenience, we put GR#3 as a comment below.

Real-robot video plans. We added a quantitative pushing evaluation across five objects, with automated plan generation and screening. Claude Opus 5.5 produces 25 final video plans through 327 calls to the Seedance 2.5 video generation API. DOLA executes all 25 accepted plans, with 21 successful pushes (84%). Please see Global Response #4 for the planning procedure, per-object counts, and failure modes. For your convenience, we put GR#4 as a comment below.

Q1 - Ablation: action supervision only

We find that video supervision lowers action-probe MSE and improves downstream policy performance on Push-T, MimicGen Stack, and VizDoom compared with action supervision alone. We see no difference on LIBERO 90, where all training trajectories already have action labels. We added the action-probe ablation to Table 4; Global Response #2 reports both probe and policy results.

Each cell reports linear action-probe MSE (↓) / downstream latent policy performance (↑). Policy performance is target coverage on Push-T and success rate on the other environments.

Push-TMimicGen StackLIBERO 90/10VizDoom
DOLA0.01 / 0.560.17 / 0.900.10 / 0.840.39 / 0.98
Action supervision only0.03 / 0.510.31 / 0.640.10 / 0.840.56 / 0.75
AuthorsOpenReview ↗

(Global Response #3) Cross-embodiment quantitative evals, reproduced here for your reference

We quantify cross-embodiment transfer on MimicGen with a 7-DoF Franka Panda and a 6-DoF UR5e, and compare DOLA with DreamDojo. In summary, DOLA achieves lower per-step cross-embodiment decoding error than DreamDojo in both directions, though accumulated drift limits open-loop success for both methods. In closed-loop transfer to a new task, Panda co-training raises UR5e Stack success from 50% to 72%, compared with 64% for co-trained DreamDojo, without UR5e Stack action labels.

Per-step decoding. We train DOLA and DreamDojo on the same unpaired demonstrations of both robots, each with one linear delta-joint head per robot, and evaluate on 50 held-out demonstrations per direction. At each step, we encode the source robot video and use the target robot action head to decode target joint actions. We measure end-effector position error against the source motion retargeted onto the target robot (mm per step, averaged over all steps).

Panda (7-DoF) → UR5e (6-DoF)UR5e (6-DoF) → Panda (7-DoF)
Mean EE Motion (mm)6.567.19
DOLA (mm)1.291.56
DreamDojo (mm)1.351.77

DOLA has lower per-step error than DreamDojo in both directions, including from the 6-DoF UR5e to the 7-DoF Panda.

Open-loop execution. The UR5e’s delta-joint head executes the Panda’s held-out trajectories without feedback control. As the failure cases in Appendix G and the website show, while each step is decoded well, open-loop rollout gradually accumulates tracking error over time. Fully open-loop transfer succeeds in 4/50 trajectories for both DOLA and DreamDojo, with a mean end-effector tracking error over all timesteps and trajectories of 4.4 cm for DOLA and 4.7 cm for DreamDojo. We next evaluate how this shared action space supports closed-loop policy learning.

Panda → UR5e open-loop replayStack successesMean tracking error over all timesteps and trajectories
DOLA4/504.4 cm
DreamDojo4/504.7 cm

Closed-loop transfer across tasks and embodiments. We test whether demonstrations from another robot help the target robot learn a closed-loop policy on a new task from videos. We use two MimicGen tasks, Square and Stack. The target robot, the UR5e, has action-labeled Square demonstrations and unlabeled Stack videos. The source robot, the Panda, has action labels for both tasks.

We compare a LAM trained on UR5e data alone (UR5e only) with one co-trained on both robots (UR5e + Panda). For each LAM, we train a latent policy on the UR5e Stack videos and decode its predictions using a linear head fit on UR5e Square demonstrations. We evaluate each policy on Stack over 100 closed-loop rollouts.

Method and training dataSuccess rate
DOLA, UR5e only0.50
DreamDojo, UR5e only0.31
DOLA, UR5e + Panda0.72
DreamDojo, UR5e + Panda0.64

Panda demonstrations improve transfer from Square to Stack on the UR5e, raising DOLA’s success from 50% to 72% without UR5e Stack action labels. DreamDojo achieves 64% under the same co-training setup. We find that cotraining helps both DOLA and DreamDojo, and DOLA outperforms DreamDojo under matched evaluation settings.

AuthorsOpenReview ↗

(Global Response #4) Real-robot video planning evaluation results, reproduced here for your reference

Task suite. We evaluate our method on a real-world push task, which requires the robot to reach and push an object toward the side-view camera. We test five objects (bowl, box, teddy bear, glass-jar candle, and cap) at five random initial positions each, for 25 episodes in total. Two fixed cameras provide side and top-down views.

Training. We collect 699 random-motion calibration episodes on a Franka arm using two cameras and train DOLA with the same background-compositing augmentation as in the paper.

Automated video planning. Claude Opus 5.5 drives the planning loop through tool calls to the Seedance 2.5 API. For each episode, we concatenate the two initial camera images side by side into a single frame. Claude calls the video generation tool with this frame and a prompt describing the scene and the desired push, revising the prompt and retrying until it judges the generated plan visually acceptable. Claude checks the video for plausible motion, consistency across views, and correct fingertip contact with the object. In total, Claude makes 327 video generation calls and accepts 25 final plans (7.6%).

Inference and results. DOLA decodes each accepted video plan into a sequence of end-effector actions, which we execute open-loop on the robot arm. A success requires making contact with the object and pushing it toward the side camera. The robot succeeds on 21 of 25 accepted plans (84%).

ObjectVideos generatedSuccessesSuccess rate
Green bowl394/580%
White box675/5100%
Teddy bear565/5100%
Glass-jar candle1324/580%
Black cap333/560%
Total32721/2584%

Failure modes and limitations. Per-step IDM errors can accumulate during open-loop execution, especially when the camera views provide ambiguous motion cues. Errors along the push direction or in height cause the fingers to stop short, overshoot, or slide over the object. Obtaining physically accurate video plans also remains an open problem. For example, Seedance 2.5 struggles to depict a clean sideways push of the glass-jar candle, requiring 132 generation calls to obtain five accepted plans.

AuthorsOpenReview ↗

Let us know if you have any additional questions!

Let us know if you have any additional questions! As the author-reviewer discussion period draws to an end, we’d like to thank you again for your helpful and detailed feedback. We will incorporate these changes in our final revision.

Please let us know if you have any remaining concerns, and assuming we have addressed them, we’d like to politely ask if you’d be willing to increase your score. Thanks!

Reviewer_gusEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a LAM with the learning from delta observations between consecutive frames. It also uses autoregressive forward prediction and action grounding to further align the continuous latent action with the actions. The downstream linear probing evaluation and the policy trained in latent space shows that the method achieves the lowest linear probe error and strongest policy performance compared with some other LAM models.

Critical strengths
  1. The paper learns the latent action from delta observations instead of the normal observations, the ablation study shows the improvement over this choice.
  2. The method can achieve comparable performance with models like DreamDojo but with much fewer model parameters.
Critical weaknesses
  1. The paper didn't do thorough research on more LAM works like Mogo-GPT [1], Villa-X[2], DiAL[3], CoMo[4].... As a result, the comparison of LAMs in Table 1 is not sufficient. [1]Chen, Yi, et al. "Moto: Latent motion token as the bridging language for learning robot manipulation from videos." 2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025. [2]Chen, Xiaoyu, et al. "Villa-x: enhancing latent action modeling in vision-language-action models." International Conference on Learning Representations. Vol. 2026. 2026. [3]Zhang, Tianqiu, et al. "DiLA: Disentangled Latent Action World Models." arXiv preprint arXiv:2605.15725 (2026). [4]Yang, Jiange, et al. "Como: Learning continuous latent motion from internet videos for scalable robot learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.
  2. Table 1 shows that without gt action supervision, DOLA shows performance similar to Linear LAM and DreamDojo, while AdaWorld has the best performance. Does that mean delta observation is not the key point for LAM learning? But adding only 5% GT actions, DOLA achieves the best performance. Why GT labels make such a big difference is not deeply analyzed in the DOLA architecture design.
  3. The biggest issue is that delta observation was not first proposed in DOLA for LAM learning, as Como already did.
  4. The parameter comparison (Contribution 2) may not be an advantage. Currently, all methods are trained on each test set, which is actually a small dataset; a 47M model size may be enough for learning LAM at this model size. However, LAPA, Dreamdojo are all designed to train across a large scale of human videos, the LAM model needs to be larger for model capacity. Maybe a scaling experiment can be considered to prove the model size advantage.
Questions for authors
  1. Can you provide the direct comparison with CoMo, in the fair settings, like both only use vision-input or adding action head, given CoMo also uses the delta observations, in Table 1-3.
  2. Can you further analyse why adding only 5% gt labels make such a big difference to DOLA compared with other methods. I think Table 1 needs further clarification over the improvements, when without the gt labels, delta observation learned LAM does not show its priority. The correlation bettwen them needs to be well conveyed.
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

(Edited: formatting)

Thank you for the thoughtful review! We are glad that you acknowledge our ablation “shows the improvement over this choice” of delta observations, and that our model “can achieve comparable performance with models like DreamDojo but with much fewer model parameters.” We address each of your concerns below, as well as in the Global Rebuttal Response.

W1, Q1 - Additional baseline comparisons

Thank you for suggesting these comparisons! We added CoMo, DiLA, Moto, and villa-X to Tables 1-3 using the same action-probing and downstream policy protocols as the original baselines. Overall, we find that DOLA achieves the lowest action-probe MSE on the three manipulation benchmarks and the highest policy success on LIBERO and MimicGen, with competitive policy performance on Push-T and VizDoom. We find DiLA to be a strong baseline. Please see Global Response #5 for the full comparisons.

Additionally, we compare frozen vs. learned observation embeddings in Global Response #1, contrasting the setting of DOLA with CoMo. In general, we observe that learned embedding deltas perform significantly better than frozen embeddings. We report one seed per new baseline here due to compute and time constraints, and will report three per baseline in the final manuscript.

W2, Q2 - Effect of grounding with GT labels

Without action grounding, all methods in Table 1 have high action-probe error. On Push-T, the models trained without action labels score between 0.73 and 1.11, compared with 1.10 for the mean-action baseline and 0.01–0.05 for the best grounded models. The 0%-label results show that delta inputs alone do not yield accurately decodable physical actions.

Observation-only prediction does not uniquely determine the latent representation. Composing an IDM f with an invertible map φ and compensating in the FDM g with φ⁻¹ leaves the prediction unchanged, but can change how easily physical actions can be decoded from the latent. This reflects the identifiability problem [3]. The latent can also encode changes that the agent did not cause [1, 2]. A small amount of action supervision helps ground this otherwise ambiguous representation [1, 2].

The benefit of 5% action labels is shared across methods: DreamDojo improves from 0.86 to 0.05, DiLA from 0.80 to 0.04, and DOLA from 0.84 to 0.01. We therefore compare methods under the same label budget, following prior work [1, 2]. Within this grounded setting, replacing DOLA’s delta inputs with raw inputs raises Push-T MSE from 0.01 to 0.62 (Table 4), showing that delta inputs contribute beyond the action labels.

W3 - Novelty of delta observation

DOLA learns action representations from explicit differences in a jointly learned feature space. Prior work such as CoMo computes differences of frozen pretrained features. Holding the rest of DOLA’s setup fixed, learning the encoder improves action-probe MSE and downstream policy performance over all three frozen alternatives on all four benchmarks. On MimicGen Stack, learning the same ResNet-34 architecture raises policy success from 26% to 90%. Please see Global Response #1 for the controlled comparison and #5 for direct comparisons with CoMo.

W4 - Model parameter size and scaling experiment

Thank you for raising this point on scaling! We agree that the datasets used in this work are small compared with the large corpora that LAPA and DreamDojo were designed for. This work is framed and conducted in an academic setting, without access to large-scale compute and data, and all methods are compared in a controlled setting with the same data and label budget. Our point is to show that the formulation achieves strong performance, not to argue against scale, and scaling DOLA up would be an exciting next step for future work.

Data. As a step towards more diverse data, we have added multi-task experiments (see our response to Reviewer hLRG and Global Response #5), where one model is trained on four tasks per environment: DOLA outperforms DreamDojo on both MimicGen multi-task and VizDoom multitask.

Model. As a proof of concept that DOLA is compatible with much bigger parameter sizes, we additionally train a 736M-parameter version of DOLA with scaled up encoder, IDM, and FDM on PushT. The final linear probe MSE is 0.01, comparable with the smaller DOLA model in the manuscript.

We will clarify this claim in our final manuscript.

In light of the above clarifications, assuming we have addressed your concerns, we would like to politely ask if you are willing to increase your score. Otherwise, please let us know if you have additional questions and we would be more than happy to discuss further.

References

[1] Zhang et al. 2025, What Do Latent Action Models Actually Learn?

[2] Nikulin et al. 2025, Latent Action Learning Requires Supervision in the Presence of Distractors

[3] Hyvärinen & Pajunen 1999, Nonlinear Independent Component Analysis: Existence and Uniqueness Results

AuthorsOpenReview ↗

Tables 1/2/3, reproduced here for your reading convenience

We added CoMo, DiLA, Moto, and villa-X to Tables 1-3 using the same action-probing and downstream policy protocols as the original baselines. We find DiLA and villa-X to be strong baselines. DOLA remains the best overall in both latent action representation and downstream policy performance. We report one seed per new baseline here due to compute and time constraints, and will report three per baseline in the final manuscript.

Table 1. Action grounding on Push-T. Without action labels, probe MSE ranges from 0.73 to 1.11 across methods (mean-action baseline, 1.10). Labeling 5% of demonstrations reduces DOLA’s MSE from 0.84 to 0.01, the lowest in this comparison. Video prediction alone admits multiple latent representations with the same prediction loss, leaving their relationship to physical actions ambiguous. As prior work on Linear LAM and LAOM shows, a small amount of action supervision helps ground the learned latent space. We discuss this in more detail in our response to gusE W2/Q2.

Method0% labels5% labels
Mean baseline1.101.10
Linear LAM0.86 ± 0.010.70 ± 0.10
LAPA1.03 ± 0.011.11 ± 0.01
AdaWorld0.74 ± 0.020.35 ± 0.17
DreamDojo0.86 ± 0.010.05 ± 0.01
CoMo0.730.50
DiLA0.800.04
Moto1.110.65
villa-X0.930.09
DOLA (Ours)0.84 ± 0.050.01 ± 0.01

Table 2. Linear action-probe MSE (↓) on held-out trajectories. DOLA has the lowest error on the three manipulation benchmarks, and DiLA has the lowest error on VizDoom.

MethodPush-TLIBERO 90/10MimicGen StackVizDoom
Mean baseline1.100.911.210.99
Linear LAM0.70 ± 0.100.56 ± 0.040.53 ± 0.010.57 ± 0.01
LAPA1.11 ± 0.010.74 ± 0.120.74 ± 0.050.63 ± 0.03
AdaWorld0.35 ± 0.170.14 ± 0.010.67 ± 0.040.62 ± 0.03
DreamDojo0.05 ± 0.010.13 ± 0.010.24 ± 0.010.42 ± 0.01
CoMo0.500.490.280.38
DiLA0.040.190.220.31
Moto0.650.590.600.47
villa-X0.090.180.350.44
DOLA (Ours)0.01 ± 0.010.10 ± 0.010.17 ± 0.010.39 ± 0.01

Table 3. Downstream latent policy performance (↑) with DynaMo features. DOLA leads on LIBERO and MimicGen. On Push-T, DOLA scores 0.56 and DreamDojo scores 0.57. Several methods reach 0.98–1.00 on VizDoom.

MethodPush-TLIBERO 90/10MimicGen StackVizDoom
Oracle0.64 ± 0.010.90 ± 0.010.92 ± 0.021.00 ± 0.01
Linear LAM0.09 ± 0.010.26 ± 0.060.22 ± 0.020.99 ± 0.01
LAPA0.07 ± 0.010.04 ± 0.040.09 ± 0.010.88 ± 0.09
AdaWorld0.17 ± 0.060.62 ± 0.050.02 ± 0.010.41 ± 0.10
DreamDojo0.57 ± 0.020.71 ± 0.010.82 ± 0.040.96 ± 0.01
CoMo0.170.230.741.00
DiLA0.450.610.750.96
Moto0.110.220.070.99
villa-X0.510.700.550.98
DOLA (Ours)0.56 ± 0.020.84 ± 0.010.90 ± 0.010.98 ± 0.02
AuthorsOpenReview ↗

Novelty/contribution comparison with CoMo (Global Response #1), reproduced here for your reading convenience

DOLA learns action representations from video using explicit differences in a jointly learned feature space. With limited action supervision, it achieves state-of-the-art performance in action decoding and downstream control. Our controlled ablations show that explicit feature differences, temporal context, and learning the feature space contribute to these gains. We further clarify our contributions below and compare with the prior work highlighted in the reviews.

(CoMo, gusE W3) Learned vs. frozen feature deltas. DOLA takes differences of learned features. Prior work such as CoMo takes differences of frozen pretrained features. Holding the rest of DOLA’s training setup fixed, the learned encoder yields lower action-probe error and better downstream policies than all three frozen encoders on all four benchmarks. On MimicGen Stack, learning the same ResNet-34 architecture raises policy success from 26% to 90%.

Here we compare frozen vs learned encoder performance.

Encoder comparison. Each cell is linear action-probe MSE (↓) / downstream policy performance (↑).

EncoderPushTMimicGenLIBERO-90→10VizDoom
Learned ResNet-34 (DOLA)0.01 / 0.560.17 / 0.900.10 / 0.840.39 / 0.98
Frozen ImageNet ResNet-340.42 / 0.110.48 / 0.260.42 / 0.540.58 / 0.57
Frozen DINOv2 ViT-S/14, patch tokens0.18 / 0.170.23 / 0.340.21 / 0.770.43 / 0.91
Frozen DINOv2 ViT-S/14, pooled0.33 / 0.120.38 / 0.250.36 / 0.540.46 / 0.74
AuthorsOpenReview ↗

Let us know if you have any additional questions!

Let us know if you have any additional questions! As the author-reviewer discussion period draws to an end, we’d like to thank you again for your helpful and detailed feedback. We will incorporate these changes in our final revision.

Please let us know if you have any remaining concerns, and assuming we have addressed them, we’d like to politely ask if you’d be willing to increase your score. Thanks!

Reviewer_hLRGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DOLA, a latent action model that builds a delta-observation prior into jointly learned visual and action representations. By constructing latent actions from observation embedding differences and training with autoregressive forward image and action predictions, DOLA achieves strong performance with linear probe error and downstream policy across four simulated benchmarks. DOLA also supports zero-shot execution of generated video plans on a real Franka arm.

Critical strengths
  • Simple, clear idea, and intuitive inductive bias of using delta observation embeddings to learn latent actions which are mostly the transitions between frames caused by the agent.
  • The authors evaluate DOLA with different evaluation metrics like MSE with linear probing and also with success rate for downstream policy learning in four different benchmarks.
  • The authors show that DOLA could learn a shared action representations across different robot embodiments of different action space. and that it can use an off-the-shelf video generation model that can generate a video from an initial observation and text, and then DOLA generate executable actions from this video.
Critical weaknesses
  • The authors claim that the main contribution in DOLA it coming from providing delta features as input to the IDM. But the gains seem to be coming from many different design choices, not only the delta features as shown in the ablations (Table 4). The other baselines like LAPA, AdaWorld, DreamDojo are lacking these design choices. Design choices like: observations context window, forward autoregressive rollout, and temporal aggregations. These baselines only take frame pairs (h=2), while their FDM predict one frame. while DOLA uses h=8 and it's shown that it's worse when it uses h=2. So it's hard to only attribute the gains to the delta features, but it's more of combination of temporal understanding and delta features.

  • The cross-embodiment transfer results shown in Figure 4 are not actual cross-embodiment transfer because the model is trained on both embodiments. So, it only shows that a shared action space is learned. But no new unseen embodiment is used. Also, only qualitative results are shown, no quantitative results on that.

  • The authors use four different benchmarks, which is good. but they use only one task per benchmark, except LIBERO. which is very limited. Evaluating the latent action representations with a shared action head trained across multi-tasks within the same environment would provide stronger evidence of their generality compared to other baselines.

Questions for authors
  • I suggest providing a more controllable comparison with a context window of 2 or 3. no temporal aggregation. no forward autoregressive prediction, no temporal aggregation. and provide two variants with raw features and delta features. to better represent other baselines in your setup. and report the policy performance (success rate) on downstream tasks on LIBERO, MimicGen, or PushT
  • For cross-embodiment transfer, do you think that DOLA can work with unseen embodiment? with only an embodiment-specific action decoder. You also showed the transfer from a 7DoF robot arm to a 6DoF robot arm. do you think will it work if you did the transfer in the opposite way?
  • Could you provide multi-task latent action and policy training on VizDoom or MimicGen. and compare with a multi-task baseline as well like DreamDojo?
  • For zero-shot video plan execution. Could you provide a baseline with a FDM and a CEM planning method similar to JEPA-style and DINO-WM methods? and report how this compares to using DOLA's IDM.
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for your insightful and thorough feedback! We are glad that you find our method a “simple, clear idea” with “intuitive inductive bias,” and that it can be used for multiple downstream applications. We address each of your concerns below as well as in the Global Rebuttal Response.

W1, Q1 - Sources of gains for DOLA

We thank the reviewer for this point, and we agree that temporal context contributes to performance. As described in Section 2.1, the two play different roles: delta inputs provide the IDM with differences between consecutive visual embeddings, while a longer window allows bidirectional attention across neighboring transitions. The ablations in Table 4 already remove each of these choices in turn (delta inputs, context window, temporal aggregation, autoregressive rollout), and removing delta inputs causes the largest degradation. To separate the two effects further, we ran the controlled comparison the reviewer suggested: we reduce the window to a single frame pair (h=2), which leaves no temporal aggregation and no autoregressive rollout, and vary only the IDM input (raw features vs. delta of raw features). This matches the frame-pair setting of the baselines. The table reports downstream latent policy performance (↑).

MethodPush-TLIBERO 90/10MimicGen StackVizDoom
DOLA, delta, h=80.560.840.900.98
DOLA, delta, h=20.590.820.850.99
DOLA, raw, h=20.090.380.201.00
DreamDojo0.570.710.820.96
villa-X0.510.700.550.98
DiLA0.450.610.750.96
CoMo0.170.230.741.00
Moto0.110.220.070.99

Our results show that delta observations provide the largest gains. With only a single frame pair, switching from raw to delta inputs raises policy performance from 0.09 to 0.59 on Push-T, from 0.38 to 0.82 on LIBERO, and from 0.20 to 0.85 on MimicGen. In this setting DOLA already outperforms the strongest baseline, DreamDojo, on all three manipulation benchmarks with the same temporal context. Going from h=2 to h=8 improves the latent representation (see the linear probe ablation table in Global Response #2 and in the original manuscript), and slightly improves policy performance. On VizDoom, the benchmark saturates and all DOLA variants perform comparably.

W2, Q2 - Cross embodiment results and quantitative evaluations

DOLA achieves lower per-step cross-embodiment decoding error than DreamDojo in both directions, though accumulated drift limits open-loop success for both methods. In closed-loop transfer to a new task, Panda co-training raises UR5e Stack success from 50% to 72%, compared with 64% for co-trained DreamDojo, without UR5e Stack action labels. Please see Global Response #3 for the full discussion.

AuthorsOpenReview ↗

Rebuttal by Authors (continued)

W3, Q3 - Additional multi-task results

We extend our evaluation to two multi-task benchmarks, each comprising four tasks with 1,000 demonstrations per task: MimicGen: Stack, Square, Stack Three, and Threading. VizDoom: My Way Home, Health Gathering, Deadly Corridor, and Arena, covering survival and goal-reaching objectives across different maps. For each task, 5% of demonstrations have ground-truth action labels. Following the main paper, we report linear probe MSE and downstream policy success rate, comparing DOLA with DreamDojo.

MimicGen, 4 tasks: policy ↑MimicGen probe ↓VizDoom, 4 tasks: policy ↑VizDoom probe ↓
GT action0.57–1.00–
DOLA0.500.080.990.17
DreamDojo0.350.140.950.23
Table: Multi-task evaluation on MimicGen and VizDoom, reporting downstream policy success rate (↑) and linear probe MSE (↓). Each benchmark comprises four tasks with 1,000 demonstrations per task and 5% ground-truth action supervision.

Results. As shown in the table above, DOLA consistently outperforms DreamDojo across both benchmarks. On MimicGen, DOLA achieves a lower probe MSE (0.078 vs. 0.138) and a higher policy success rate (0.50 vs. 0.35). Similarly, on VizDoom, DOLA reduces probe MSE (0.169 vs. 0.232) and improves policy success (0.988 vs. 0.953). These results demonstrate that DOLA's advantages extend beyond single-task settings to multi-task learning. We will include these results in the revised appendix. Please also see Global Response #5 for a summary.

Q4 - Comparison with JEPA-style and DINO-WM

We thank the reviewer for raising this point. Following the reviewer's suggestion, we trained DINO-WM on the same real-robot calibration dataset used to train DOLA, and used it as an inverse dynamics model: for each pair of adjacent frames in the generated video plan, we run CEM planning with the trained world model to find the action that explains the transition. We train DINO-WM with both frameskip 1, which matches DOLA's pipeline of operating on strictly neighboring frames, and frameskip 5, the DINO-WM default. Since the generated video plans have no ground-truth actions, we use as the reference the action trajectories produced by DOLA, which were executed on the real robot and successfully completed the tasks. For each method we integrate the predicted actions from the same initial pose, and report the end-effector pose discrepancy from the successful DOLA trajectory at the end of the video together with the inference time.

MethodPosition error (cm)Orientation error (°)Time per frame pairTime per 8 s video
DOLA (reference)001.6 ms8.1 ms
DINO-WM, frameskip 530.6 ± 7.172 ± 131.73 s81 s
DINO-WM, frameskip 139.5 ± 4.4107 ± 19~1.7 s~325 s

Mean ± std over four generated video plans. Time is model compute on a single NVIDIA H200 GPU.

The end-effector pose of the trajectory planned with DINO-WM deviates from the successful DOLA trajectory by more than 30 cm on average across four videos, with frameskip 1 worse than frameskip 5 as expected (errors accumulate over more steps), which would fail and not be able to reach the object when executed on the real robot. Repurposing world-model planning as an IDM also introduces significant latency, taking seconds per frame pair where DOLA takes milliseconds, since each action requires a search for the action that explains the transition, whereas DOLA needs a single forward pass. We would like to point out that DOLA operates under a different paradigm from JEPA-style world models: it learns an IDM directly, which yields more accurate action decoding and fast inference. We hypothesize that DINO-WM planning performs worse when acting as an IDM because it needs much broader transition coverage. For DOLA, observing a single transition (s_t, a_t → s_{t+1}) is already enough supervision to output a_t given s_t and s_{t+1}. For DINO-WM, the model must have seen many different actions at s_t before planning can tell which particular a_t explains a given transition; with limited coverage, the search cannot single out the right action.

In light of the above clarifications, assuming we have addressed your concerns, we would like to politely ask if you are willing to increase your score. Otherwise, please let us know if you have additional questions and we would be more than happy to discuss further.

I would like to thank the authors for the detailed response.

Q1. It's still not convincing for me that delta features provide the largest gains? and why DOLA raw fails substantially? Methods like DreamDojo and villa-X are using raw features and h=2, and they provide comparable performance to DOLA with delta and h=2. why is it the case for DOLA to degrade that much?

Q2. Cross-embodiment experiment was really helpful to show the zero-shot transfer and the co-training performance of DOLA. Of course the co-training is expected to work well for both DOLA and DreamDojo. but could you also provide a variant of DreamDojo similar to DOLA with zero-shot transfer? because this is more important here for zero-shot transfer, not co-training.

Q4. why DOLA is the reference and you compute error based on the DOLA predicted trajectory? This shouldn't be the ground truth. The true metric is to provide the success rate for each predicted trajectory using different methods.

Reviewer_oA9zOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DOLA, a latent action model that constructs action representations from differences between learned observation embeddings. The visual encoder and inverse dynamics model are trained jointly with autoregressive forward prediction and action grounding. Evaluation combines linear probing of frozen representations with downstream policies trained in latent action space. DOLA achieves strong results across four simulated environments with a relatively small model. Additional experiments explore cross-embodiment trajectory transfer and execution of generated video plans using task-agnostic calibration data.

Critical strengths

1.The central design is simple and supported by direct ablations. Removing delta-observation inputs increases linear-probe error in all four environments (Table 4), including 0.01→0.62 on Push-T and 0.17→0.34 on MimicGen. This provides concrete evidence that the proposed input representation is useful.

2.The representation evaluation is reasonably controlled. Methods share the action-label budget within each environment, latent dimensionality, and downstream policy setup. Reporting three seeds and repeating policy evaluation with WebSSL features also strengthens confidence that the results are not specific to one downstream configuration.

3.The benefits extend to downstream control. DOLA achieves 0.84 success on LIBERO and 0.90 on MimicGen, compared with DreamDojo’s 0.71 and 0.82 (Table 3). The LIBERO result is particularly interesting because the action representation transfers from source tasks to new tasks with video demonstrations.

4.The model is substantially smaller than the strongest overall baseline. DOLA uses approximately 47M parameters versus DreamDojo’s 710M while maintaining competitive policy performance. Achieving these results with a compact architecture makes the approach practically attractive and suggests that model scale alone does not explain the gains.

Critical weaknesses

1.The contribution of unlabeled video is not isolated. The paper lacks a matched delta-IDM trained only with action supervision on the same labeled subset. Consequently, it remains unclear how much benefit comes from the forward-dynamics objective and additional unlabeled videos beyond supervised inverse dynamics learning.

2.Some architectural claims are stronger than the evidence supports. Differences between jointly learned nonlinear features can still contain absolute scene information. Likewise, a causal FDM does not itself rule out information already encoded by the bidirectional IDM. The exclusion and shortcut-prevention claims in Section 2.1 need stronger evidence or narrower wording.

3.Cross-embodiment transfer is demonstrated mainly through selected examples. Section 3.4 and Appendix G provide encouraging Panda→UR5e executions and failure cases, but no aggregate transfer success rate or tracking error. The CCA analysis reveals shared structure, yet does not establish how reliably that structure supports execution.

4.The real-robot evaluation does not establish execution reliability. Generated plans are manually screened, and results are qualitative (Section 3.5). Without candidate counts, screening rates, execution success rates, or a comparison method, it is difficult to assess the robustness of the proposed video-to-action pipeline.

Questions for authors

1.Value of video learning: How does DOLA compare with the same delta-IDM trained using only L_act and the same action-label budget? What additional gains come from unlabeled videos and L_dyn?

2.Representation mechanism: What evidence shows that delta inputs reduce scene-dependent information or reconstruction shortcuts relative to the no-delta variant under matched temporal context? Can the structural claims in Section 2.1 be qualified accordingly?

3.Cross-embodiment evaluation: On a predefined set of held-out source trajectories, what are the Panda→UR5e transfer success rate and tracking error, and how do they compare with another latent-action method?

4.Real-robot protocol: How many plans were generated, retained, and executed per task? Please report execution success conditional on screening, and specify the video generator, generation settings, and screening criteria.

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors

Thank you for the constructive review! We are glad you found our design “simple and supported by direct ablations” and that its “benefits extend to downstream control.” We address your concerns below, as well as in the Global Rebuttal Response.

W1, Q1 - Ablation: action supervision only

We find that video supervision lowers action-probe MSE and improves downstream policy performance on Push-T, MimicGen Stack, and VizDoom compared with action supervision alone. We see no difference on LIBERO 90, where all training trajectories already have action labels. We added the action-probe ablation to Table 4; Global Response #2 reports both probe and policy results.

W2, Q2 - State information and reconstruction shortcuts

Thank you for raising this important point! We agree that feature differences can still contain absolute state information. To test their effect, we fit linear probes from latent actions to ground-truth states, comparing raw and delta inputs under the same temporal context and training recipe. We report held-out R² (mean ± SD over three seeds); lower values indicate less linearly decodable state information.

Raw inputsDelta inputs (DOLA)
PushT0.59 ± 0.010.22 ± 0.01
MimicGen0.40 ± 0.000.20 ± 0.00
LIBERO-900.39 ± 0.030.21 ± 0.01
VizDoom0.55 ± 0.020.49 ± 0.05

Delta inputs reduce linearly decodable state information in the action latent, but do not make it uncorrelated with state. We also agree that a causal FDM cannot rule out future information encoded by the bidirectional IDM. We will revise Sec. 2.1 to qualify the claims about excluding absolute state and preventing reconstruction shortcuts.

W3, Q3 - Quantitative results for cross-embodiment transfer

DOLA achieves lower per-step cross-embodiment decoding error than DreamDojo in both directions, though accumulated drift limits open-loop success for both methods. In closed-loop transfer to a new task, Panda co-training raises UR5e Stack success from 50% to 72%, compared with 64% for co-trained DreamDojo, without UR5e Stack action labels. Please see Global Response #3 for the full discussion.

W4, Q4 - Quantitative results for real-robot evaluation

We added a quantitative pushing evaluation across five objects, with automated plan generation and screening. Claude Opus 5.5 produces 25 final video plans through 327 calls to the Seedance 2.5 video generation API. DOLA executes all 25 accepted plans, with 21 successful pushes (84%). Please see Global Response #4 for the planning procedure, per-object counts, and failure modes.

In light of the above clarifications, assuming we have addressed your concerns, we would like to politely ask if you are willing to increase your score. Otherwise, please let us know if you have additional questions and we would be more than happy to discuss further.

AuthorsOpenReview ↗

Let us know if you have any additional questions!

Let us know if you have any additional questions! As the author-reviewer discussion period draws to an end, we’d like to thank you again for your helpful and detailed feedback. We will incorporate these changes in our final revision.

Please let us know if you have any remaining concerns, and assuming we have addressed them, we’d like to politely ask if you’d be willing to increase your score. Thanks!

Comments on the paper

5 comments
AuthorsOpenReview ↗

Global Rebuttal Response

We thank the reviewers for their constructive feedback! We are glad you found the design simple (oA9z, hLRG, 39yX), the benefits of delta observations supported by ablations (oA9z, gusE, 39yX), and the downstream policy improvements clear (oA9z, 39yX). We have run all the requested experiments within our compute budget and time constraints. We address your shared concerns in the global response below and give individual responses under each review. If you have any additional questions, we are happy to answer them and discuss further.

1. Novelty and contribution (gusE, 39yX)

DOLA learns action representations from video using explicit differences in a jointly learned feature space. With limited action supervision, it achieves state-of-the-art performance in action decoding and downstream control. Our controlled ablations show that explicit feature differences, temporal context, and learning the feature space contribute to these gains. We further clarify our contributions below and compare with the prior work highlighted in the reviews.

(CoMo, gusE W3) Learned vs. frozen feature deltas. DOLA takes differences of learned features. Prior work such as CoMo takes differences of frozen pretrained features. Holding the rest of DOLA’s training setup fixed, the learned encoder yields lower action-probe error and better downstream policies than all three frozen encoders on all four benchmarks. On MimicGen Stack, learning the same ResNet-34 architecture raises policy success from 26% to 90%.

Encoder comparison. Each cell is linear action-probe MSE (↓) / downstream policy performance (↑).

EncoderPushTMimicGenLIBERO-90→10VizDoom
Learned ResNet-34 (DOLA)0.01 / 0.560.17 / 0.900.10 / 0.840.39 / 0.98
Frozen ImageNet ResNet-340.42 / 0.110.48 / 0.260.42 / 0.540.58 / 0.57
Frozen DINOv2 ViT-S/14, patch tokens0.18 / 0.170.23 / 0.340.21 / 0.770.43 / 0.91
Frozen DINOv2 ViT-S/14, pooled0.33 / 0.120.38 / 0.250.36 / 0.540.46 / 0.74

(Linear LAM, 39yX W1) Taking the difference explicitly. We would like to note that Linear LAM defines its IDM as z = Co + Do′, with independently learned matrices C and D, which we implement as formulated. DOLA explicitly takes the difference between observation embeddings and learns these feature differences jointly with forward dynamics and action supervision. The controlled two-frame comparison below in Global Response #2 isolates the benefit of explicit differencing.

Comparisons with Linear LAM, CoMo, and additional baselines appear in the updated manuscript and Global Response #5 below.

2. Additional ablations

Observation context (hLRG W1/Q1). Our original ablations show that h=8 yields lower action-probe MSE than h=2 across all four environments. In the requested downstream policy comparison at h=2, DOLA still outperforms or matches baselines using a single frame pair, without temporal aggregation or multi-step autoregressive rollout.

Feature deltas at fixed context. Notably, further removing the delta input at h=2 substantially reduces policy success on all three manipulation tasks, including from 85% to 20% on MimicGen Stack. On VizDoom, success is near saturation for both variants.

Downstream latent policy performance (↑).

MethodPush-TLIBERO 90/10MimicGen StackVizDoom
DOLA, delta, h=80.560.840.900.98
DOLA, delta, h=20.590.820.850.99
DOLA, raw, h=20.090.380.201.00
DreamDojo0.570.710.820.96
villa-X0.510.700.550.98
DiLA0.450.610.750.96
CoMo0.170.230.741.00
Moto0.110.220.070.99

Both temporal context and feature deltas improve representation learning. We will clarify their contributions in the manuscript.

Action supervision only (oA9z W1/Q1, 39yX Q1). We added an action-supervision-only baseline to the manuscript’s ablation table. Training on the labeled trajectories alone increases action-probe MSE and lowers downstream policy performance on Push-T, MimicGen Stack, and VizDoom. We see no difference on LIBERO 90, where all training trajectories already have action labels.

Linear action-probe MSE (↓). Values with error bars give mean ± SD over three seeds.

MethodPush-TLIBERO 90/10MimicGen StackVizDoom
DOLA full model0.01 ± 0.010.10 ± 0.010.17 ± 0.010.39 ± 0.01
No delta-obs0.62 ± 0.020.15 ± 0.010.34 ± 0.010.57 ± 0.05
No covariance regularization0.02 ± 0.010.10 ± 0.010.14 ± 0.010.40 ± 0.01
Causal IDM0.01 ± 0.010.20 ± 0.010.16 ± 0.010.41 ± 0.01
No temporal aggregation0.02 ± 0.010.27 ± 0.010.20 ± 0.010.46 ± 0.01
Short context (h=2)0.02 ± 0.010.24 ± 0.010.23 ± 0.010.50 ± 0.03
No autoregressive rollout0.01 ± 0.010.10 ± 0.010.16 ± 0.010.56 ± 0.04
Action supervision only0.030.100.310.56

Downstream latent policy performance (↑) for the action-supervision-only ablation (39yX Q1).

MethodPush-TLIBERO 90/10MimicGen StackVizDoom
DOLA full model0.560.840.900.98
Action supervision only0.510.840.640.75

3. Quantitative results for cross-embodiment transfer (oA9z, hLRG, 39yX)

We quantify cross-embodiment transfer on MimicGen with a 7-DoF Franka Panda and a 6-DoF UR5e, and compare DOLA with DreamDojo. In summary, DOLA achieves lower per-step cross-embodiment decoding error than DreamDojo in both directions, though accumulated drift limits open-loop success for both methods. In closed-loop transfer to a new task, Panda co-training raises UR5e Stack success from 50% to 72%, compared with 64% for co-trained DreamDojo, without UR5e Stack action labels.

Per-step decoding. We train DOLA and DreamDojo on the same unpaired demonstrations of both robots, each with one linear delta-joint head per robot, and evaluate on 50 held-out demonstrations per direction. At each step, we encode the source robot video and use the target robot action head to decode target joint actions. We measure end-effector position error against the source motion retargeted onto the target robot (mm per step, averaged over all steps).

Panda (7-DoF) → UR5e (6-DoF)UR5e (6-DoF) → Panda (7-DoF)
Mean EE Motion (mm)6.567.19
DOLA (mm)1.291.56
DreamDojo (mm)1.351.77

DOLA has lower per-step error than DreamDojo in both directions, including from the 6-DoF UR5e to the 7-DoF Panda.

Open-loop execution. The UR5e’s delta-joint head executes the Panda’s held-out trajectories without feedback control. As the failure cases in Appendix G and the website show, while each step is decoded well, open-loop rollout gradually accumulates tracking error over time. Fully open-loop transfer succeeds in 4/50 trajectories for both DOLA and DreamDojo, with a mean end-effector tracking error over all timesteps and trajectories of 4.4 cm for DOLA and 4.7 cm for DreamDojo. We next evaluate how this shared action space supports closed-loop policy learning.

Panda → UR5e open-loop replayStack successesMean tracking error over all timesteps and trajectories
DOLA4/504.4 cm
DreamDojo4/504.7 cm

Closed-loop transfer across tasks and embodiments. We test whether demonstrations from another robot help the target robot learn a closed-loop policy on a new task from videos. We use two MimicGen tasks, Square and Stack. The target robot, the UR5e, has action-labeled Square demonstrations and unlabeled Stack videos. The source robot, the Panda, has action labels for both tasks.

We compare a LAM trained on UR5e data alone (UR5e only) with one co-trained on both robots (UR5e + Panda). For each LAM, we train a latent policy on the UR5e Stack videos and decode its predictions using a linear head fit on UR5e Square demonstrations. We evaluate each policy on Stack over 100 closed-loop rollouts.

Method and training dataSuccess rate
DOLA, UR5e only0.50
DreamDojo, UR5e only0.31
DOLA, UR5e + Panda0.72
DreamDojo, UR5e + Panda0.64

Panda demonstrations improve transfer from Square to Stack on the UR5e, raising DOLA’s success from 50% to 72% without UR5e Stack action labels. DreamDojo achieves 64% under the same co-training setup. We find that cotraining helps both DOLA and DreamDojo, and DOLA outperforms DreamDojo under matched evaluation settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.