- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes DOLA, which learns latent action representations from differences between consecutive visual embeddings through joint training with forward dynamics prediction and action supervision. The authors evaluate action representations and downstream policies across four simulated environments, and demonstrate cross-embodiment transfer and real-robot execution of generated video plans.
Critical strengths
The method is simple, and the ablation results support the benefit of delta observations across all four environments. With relatively few parameters, DOLA learns competitive action representations and achieves clear downstream policy improvements on LIBERO and MimicGen.
Critical weaknesses
The methodological contribution is incremental. Difference-based representations and auxiliary action supervision have already been discussed in prior work on Linear LAM. DOLA incorporates these ideas into a jointly learned nonlinear temporal model, providing some empirical value, but its conceptual advance over prior work remains limited.
The application experiments lack aggregate statistics on task execution. Although the cross-embodiment experiments provide quantitative analyses of the learned representations, task execution is primarily illustrated through individual examples. The real-robot video plans are manually screened, without reporting the proportion retained or the overall execution success rate. These results support feasibility but do not provide sufficient evidence to assess execution reliability, limiting the support for the method’s practical utility.
Questions for authors
With the visual encoder, delta-conditioned inverse dynamics model, action-label budget, and inference procedure held fixed, how do linear probing and downstream policy performance change when the forward dynamics prediction loss is removed? This comparison would help establish the additional benefit of forward prediction training beyond action supervision.
Please report the total number of attempts and successes in the existing cross-embodiment and real-robot experiments, along with the manual screening criteria and the proportion of generated video plans retained. These statistics would affect my assessment of the application contributions.
4 comments
(Global Response #3) Cross-embodiment quantitative evals, reproduced here for your reference
We quantify cross-embodiment transfer on MimicGen with a 7-DoF Franka Panda and a 6-DoF UR5e, and compare DOLA with DreamDojo. In summary, DOLA achieves lower per-step cross-embodiment decoding error than DreamDojo in both directions, though accumulated drift limits open-loop success for both methods. In closed-loop transfer to a new task, Panda co-training raises UR5e Stack success from 50% to 72%, compared with 64% for co-trained DreamDojo, without UR5e Stack action labels.
Per-step decoding. We train DOLA and DreamDojo on the same unpaired demonstrations of both robots, each with one linear delta-joint head per robot, and evaluate on 50 held-out demonstrations per direction. At each step, we encode the source robot video and use the target robot action head to decode target joint actions. We measure end-effector position error against the source motion retargeted onto the target robot (mm per step, averaged over all steps).
| Panda (7-DoF) → UR5e (6-DoF) | UR5e (6-DoF) → Panda (7-DoF) | |
|---|---|---|
| Mean EE Motion (mm) | 6.56 | 7.19 |
| DOLA (mm) | 1.29 | 1.56 |
| DreamDojo (mm) | 1.35 | 1.77 |
DOLA has lower per-step error than DreamDojo in both directions, including from the 6-DoF UR5e to the 7-DoF Panda.
Open-loop execution. The UR5e’s delta-joint head executes the Panda’s held-out trajectories without feedback control. As the failure cases in Appendix G and the website show, while each step is decoded well, open-loop rollout gradually accumulates tracking error over time. Fully open-loop transfer succeeds in 4/50 trajectories for both DOLA and DreamDojo, with a mean end-effector tracking error over all timesteps and trajectories of 4.4 cm for DOLA and 4.7 cm for DreamDojo. We next evaluate how this shared action space supports closed-loop policy learning.
| Panda → UR5e open-loop replay | Stack successes | Mean tracking error over all timesteps and trajectories |
|---|---|---|
| DOLA | 4/50 | 4.4 cm |
| DreamDojo | 4/50 | 4.7 cm |
Closed-loop transfer across tasks and embodiments. We test whether demonstrations from another robot help the target robot learn a closed-loop policy on a new task from videos. We use two MimicGen tasks, Square and Stack. The target robot, the UR5e, has action-labeled Square demonstrations and unlabeled Stack videos. The source robot, the Panda, has action labels for both tasks.
We compare a LAM trained on UR5e data alone (UR5e only) with one co-trained on both robots (UR5e + Panda). For each LAM, we train a latent policy on the UR5e Stack videos and decode its predictions using a linear head fit on UR5e Square demonstrations. We evaluate each policy on Stack over 100 closed-loop rollouts.
| Method and training data | Success rate |
|---|---|
| DOLA, UR5e only | 0.50 |
| DreamDojo, UR5e only | 0.31 |
| DOLA, UR5e + Panda | 0.72 |
| DreamDojo, UR5e + Panda | 0.64 |
Panda demonstrations improve transfer from Square to Stack on the UR5e, raising DOLA’s success from 50% to 72% without UR5e Stack action labels. DreamDojo achieves 64% under the same co-training setup. We find that cotraining helps both DOLA and DreamDojo, and DOLA outperforms DreamDojo under matched evaluation settings.
(Global Response #4) Real-robot video planning evaluation results, reproduced here for your reference
Task suite. We evaluate our method on a real-world push task, which requires the robot to reach and push an object toward the side-view camera. We test five objects (bowl, box, teddy bear, glass-jar candle, and cap) at five random initial positions each, for 25 episodes in total. Two fixed cameras provide side and top-down views.
Training. We collect 699 random-motion calibration episodes on a Franka arm using two cameras and train DOLA with the same background-compositing augmentation as in the paper.
Automated video planning. Claude Opus 5.5 drives the planning loop through tool calls to the Seedance 2.5 API. For each episode, we concatenate the two initial camera images side by side into a single frame. Claude calls the video generation tool with this frame and a prompt describing the scene and the desired push, revising the prompt and retrying until it judges the generated plan visually acceptable. Claude checks the video for plausible motion, consistency across views, and correct fingertip contact with the object. In total, Claude makes 327 video generation calls and accepts 25 final plans (7.6%).
Inference and results. DOLA decodes each accepted video plan into a sequence of end-effector actions, which we execute open-loop on the robot arm. A success requires making contact with the object and pushing it toward the side camera. The robot succeeds on 21 of 25 accepted plans (84%).
| Object | Videos generated | Successes | Success rate |
|---|---|---|---|
| Green bowl | 39 | 4/5 | 80% |
| White box | 67 | 5/5 | 100% |
| Teddy bear | 56 | 5/5 | 100% |
| Glass-jar candle | 132 | 4/5 | 80% |
| Black cap | 33 | 3/5 | 60% |
| Total | 327 | 21/25 | 84% |
Failure modes and limitations. Per-step IDM errors can accumulate during open-loop execution, especially when the camera views provide ambiguous motion cues. Errors along the push direction or in height cause the fingers to stop short, overshoot, or slide over the object. Obtaining physically accurate video plans also remains an open problem. For example, Seedance 2.5 struggles to depict a clean sideways push of the glass-jar candle, requiring 132 generation calls to obtain five accepted plans.
Let us know if you have any additional questions!
Let us know if you have any additional questions! As the author-reviewer discussion period draws to an end, we’d like to thank you again for your helpful and detailed feedback. We will incorporate these changes in our final revision.
Please let us know if you have any remaining concerns, and assuming we have addressed them, we’d like to politely ask if you’d be willing to increase your score. Thanks!
Rebuttal by Authors
(Edit: we’ve copied some of the Global Response tables and details down here so that it’s easier to read, as well as added more details in our response to your contribution/novelty question. Let us know if you have more questions!)
As a summary, we clarify our contribution, add the requested action-supervision-only ablation, and quantitative cross-embodiment and real-robot execution evaluations below. Most of the details are in the Global Responses at the very top of the page, so we’ve reproduced the relevant tables below in the comments for your convenience.
W1 - Methodological contribution
Thanks for your comment on Linear LAM! First, we would like to note that Linear LAM defines its IDM as z = Co + Do′, with independently learned matrices C and D, and does not explicitly compute an observation difference. We implement it as formulated as a baseline in the original manuscript.
To clarify our contribution, DOLA learns action representations from explicit differences in a jointly learned feature space. We jointly learn the visual encoder, IDM, FDM, and action head with JEPA-style feature prediction and limited action supervision. This formulation achieves the lowest action-probe MSE on the three manipulation benchmarks and strong downstream policy performance with a compact model.
In Global Responses #1 and #2, our controlled ablations show why both explicit differencing and learning the feature space matter. At a fixed two-frame context, delta inputs raise MimicGen Stack policy success from 20% to 85%. With the same ResNet-34 encoder architecture and delta inputs, jointly learning the encoder raises success from 26% with frozen ImageNet features to 90%. Global Response #1 reports the full encoder comparison, and #2 separates differencing from temporal context.
Regarding action supervision, prior works such as Linear LAM and LAOM have shown that action supervision is necessary to align the latent action space, and we do not claim this as a contribution. We want to show empirically that this is the case, and indeed, our Table 1 corroborates this observation. For more details on action supervision, please also see our response to gusE W2/Q2.
W2, Q2 - Task execution statistics, and video-plan screening
Cross-embodiment transfer. DOLA achieves lower per-step cross-embodiment decoding error than DreamDojo in both directions, though accumulated drift limits open-loop success for both methods. In closed-loop transfer to a new task, Panda co-training raises UR5e Stack success from 50% to 72%, compared with 64% for co-trained DreamDojo, without UR5e Stack action labels. Please see Global Response #3 for the full discussion. For your convenience, we put GR#3 as a comment below.
Real-robot video plans. We added a quantitative pushing evaluation across five objects, with automated plan generation and screening. Claude Opus 5.5 produces 25 final video plans through 327 calls to the Seedance 2.5 video generation API. DOLA executes all 25 accepted plans, with 21 successful pushes (84%). Please see Global Response #4 for the planning procedure, per-object counts, and failure modes. For your convenience, we put GR#4 as a comment below.
Q1 - Ablation: action supervision only
We find that video supervision lowers action-probe MSE and improves downstream policy performance on Push-T, MimicGen Stack, and VizDoom compared with action supervision alone. We see no difference on LIBERO 90, where all training trajectories already have action labels. We added the action-probe ablation to Table 4; Global Response #2 reports both probe and policy results.
Each cell reports linear action-probe MSE (↓) / downstream latent policy performance (↑). Policy performance is target coverage on Push-T and success rate on the other environments.