acceptodds
Under review as a conference paper at ICLR 2027

HORAE-WM: Action Conditioned RGBD Keyframe World Modeling for Long Horizon Robot Manipulation

Abstract

Action conditioned video world models have made progress in robot manipulation, but with a limited number of output frames, uniform sampling often covers only part of a long-horizon manipulation trajectory, while larger sampling intervals can extend coverage at the cost of missing brief, critical interaction events. We introduce HORAE-WM, an RGBD keyframe world model designed to balance coverage of long manipulation trajectories with the capture of brief critical interactions under a limited output budget. An action-aware sampling selects key frames to be predicted based on arm motion and gripper state changes. Our curriculum progressively introduces prediction of condition videos representing robot actions, metric depth videos capturing scene geometry, and RGB videos depicting robot manipulation under frame-wise time and action condition, using sequences uniformly sampled at different frame rates, followed by training on keyframes. Keyframe prediction evaluations on simulated and real world manipulation show that HORAE-WM outperforms the compared baselines in visual fidelity, arm region consistency, and metric depth accuracy. Sampling analysis shows that our sampler achieves higher event coverage than uniform sampling while allocating more frame budget to events that are both task critical and challenging to capture.

Primary area:
generative models
Keywords:
Embodied World Model, Action-Conditioned Video Prediction, Keyframe Prediction, RGBD Generation, Long-Horizon Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NkDTOpenReview ↗
Rating
2
Confidence
4
Informativeness
3(+1)
Presentation
3(+2)
Soundness
2
Read review
Paper summary

This paper studies a practical limitation of action-conditioned video world models for long-horizon robot manipulation: under a fixed number of generated frames, high-rate uniform sampling preserves local motion but covers only a short time horizon, while low-rate sampling extends the horizon but can skip short yet important interaction events such as grasping or releasing. The paper proposes HORAE-WM, which instead predicts non-uniform RGBD keyframes selected according to robot action dynamics

The method has four main ingredients:

  1. Action-aware keyframe selection. Frames around gripper events and arm-motion boundaries receive higher priority, while the remaining budget is allocated across relatively static or slowly changing intervals.
  2. Frame-wise physical-time conditioning. Each predicted frame is conditioned on its actual elapsed physical time and aligned robot action, rather than merely its ordinal frame index.
  3. Tri-modal prediction of condition video, metric depth, and RGB, implemented using a Mixture-of-Transformers-style DiT with progressive cross-modality attention.
  4. Progressive training, first using multiple fixed temporal intervals and then non-uniform keyframe sequences, while progressively introducing condition video, depth, and RGB supervision.
    Experiments use 16 simulated and real-world dual-arm manipulation tasks from AgiBot World, with 7,669 training and 285 test episodes. The main evaluation uses 1,242 held-out sequences. HORAE-WM reports substantial improvements over GE-Sim 2.0, EVAC, Cosmos3-Nano, OSCAR, and DreamDojo-14B in RGB quality, arm consistency, and depth-related metrics.
Critical strengths
  1. The paper identifies a real and underexplored problem in long-horizon world modeling. The fixed-output-budget argument is compelling. Robot trajectories indeed contain long periods of relatively uninformative motion interspersed with short interaction-critical transitions, so allocating frames uniformly is not necessarily optimal. Framing temporal abstraction as an action-conditioned sampling problem is sensible and particularly relevant for manipulation.
  2. The proposed formulation is simple and well motivated. Using actual physical timestamps is an appropriate solution once non-uniform sampling is introduced. Without explicit timing information, the model cannot distinguish, for example, two adjacent output slots separated by 0.1 s from those separated by 2 s. This design follows naturally from the problem formulation rather than appearing as an arbitrary architectural modification.
Critical weaknesses
  1. The primary comparison does not appear sufficiently controlled for training data and adaptation This is my largest concern. HORAE-WM is initialized from DreamDojo-2B and extensively trained on the target manipulation dataset, including approximately 146k fixed-interval sequences plus 24.8k keyframe sequences. In contrast, the main baselines are evaluated using their publicly released checkpoints, with model-specific input adapters.
    It is therefore unclear how much of Table 1 reflects:
  • the proposed temporal representation,
  • the tri-modal architecture,
  • in-domain AgiBot training,
  • substantially different training budgets,
  • or differences in the underlying pretrained models. A particularly important comparison would be DreamDojo-2B fine-tuned on exactly the same training data and for approximately the same optimization budget, differing only in the HORAE-specific components. Without such a comparison, I do not think the paper convincingly establishes that HORAE-WM's large advantage over external models results primarily from its proposed methodological innovations.
  1. The non-uniform keyframe comparison intrinsically advantages HORAE-WM HORAE-WM receives an explicit physical timestamp for every target keyframe. The competing models do not support this interface, so the paper converts the full non-uniform trajectory into a single average frame rate, losing the actual per-frame temporal intervals.
    This means that the main experiment effectively asks: a model explicitly designed to know the exact non-uniform target times versus models that are not told those target times.

This is useful as a systems comparison, but it is not a clean test of predictive modeling quality. The fixed-frame-rate evaluation in Table 10 substantially mitigates this concern and is a good addition. However, I would still like to see a stronger controlled baseline, for example:

  • timestamp conditioning added to the same DreamDojo backbone;
  • piecewise generation of baseline predictions using the appropriate local interval;
  • or a common backbone trained under identical non-uniform schedules.
  1. Some important ablations are not compute matched The modality-progression ablation is potentially confounded by total optimization budget. According to Appendix A.4, the curriculum variants use different total numbers of epochs: the full C → C+D → C+D+R curriculum receives substantially more total training than the direct tri-modal control.
    Yet the paper concludes that progressive learning itself explains the improvement: C+D+R direct: PSNR 16.958 progressive C→C+D→C+D+R: PSNR 17.388.

A compute-matched direct tri-modal baseline is needed to separate the effect of curriculum ordering from simply receiving more gradient updates. A similar issue exists for the mixed-interval pretraining ablation: one variant receives additional fixed-interval pretraining before the same keyframe training. Thus the experiment establishes that additional mixed-interval training helps, but does not fully demonstrate that the benefit comes specifically from mixed temporal intervals rather than extra optimization/data exposure. I view this as an important experimental weakness because progressive training is presented as one of the main methodological contributions. 4. The strongest depth number is not directly comparable to baseline depth numbers HORAE-WM predicts depth natively and is explicitly trained with metric-depth targets, while most competing models generate RGB and then obtain depth through DA3. The headline result of 83.4 mm versus ≈300–400 mm therefore mixes two rather different settings. The paper appropriately includes a more comparable experiment where DA3 is applied to HORAE-WM's RGB, yielding 263.8 mm, still better than the strongest baseline.
That source-matched comparison is convincing, but I would recommend emphasizing it as the primary cross-model depth result rather than juxtaposing native depth against RGB-derived depth in the main table. 5. No downstream robotic utility is demonstrated The paper motivates HORAE-WM as a long-horizon robot world model, but all principal evaluations measure:

  • RGB reconstruction quality,
  • perceptual similarity,
  • arm segmentation overlap,
  • depth reconstruction,
  • human visual preference,
  • and keyframe information retention. There is no evaluation showing that the better world model improves:
  • action selection,
  • planning,
  • policy ranking,
  • model-predictive control,
  • RL,
  • or manipulation success. The authors acknowledge this limitation explicitly.
    This matters because a visually good future predictor is not necessarily a useful dynamics model. For an ICLR paper on robotic world modeling, even a small experiment such as ranking candidate action trajectories using HORAE-WM would significantly strengthen the contribution.
  1. Generalization is not extensively evaluated The training/test split is episode-disjoint but remains within the same 16 tasks. There is no clear evaluation on:
  • unseen tasks,
  • unseen object categories,
  • substantially different environments,
  • unseen robot embodiments,
  • or new camera configurations. Moreover, the data preprocessing filters for body motion and camera stability. This makes the setup substantially easier than mobile manipulation settings where camera motion and base motion interact with object dynamics. The paper should therefore be careful about broad claims regarding long-horizon robot world modeling; the evidence is strongest for episode-level generalization within a specific dual-arm manipulation distribution.
  1. Improvements from the action-aware sampler are somewhat modest on the most intuitive metrics On 100 real-world segments, coverage increases from 96.86% to 98.26%, while caption completeness increases from 4.45 to 4.56 / 5. The largest improvement is enrichment, from 0.997 to 1.223.
    This does indicate more concentrated sampling of information-dense events, but uniform sampling is already extremely strong in terms of event coverage. I would therefore like to see:
  • performance as a function of increasingly severe frame budgets, e.g. 81 / 40 / 20 / 10 frames;
  • bootstrap confidence intervals;
  • per-task breakdowns;
  • and event coverage specifically for short/hard events. I suspect the advantage of action-aware sampling would become considerably clearer under tighter budgets, which would strengthen the central argument.
  1. The model-assisted caption metric has reproducibility concerns Caption completeness uses independent GPT-6 Astra calls both to produce descriptions and to grade them. The prompt is documented carefully, which is good, but the metric remains dependent on a proprietary, potentially changing model. At minimum I would prefer:
  • repeated runs / variance estimates;
  • agreement with human evaluation on a subset;
  • or an additional deterministic event-recall metric. This is not fatal because caption completeness is supplementary rather than the primary evidence.
Questions for authors

Questions for the authors

  1. Were GE-Sim 2.0, EVAC, Cosmos3-Nano, OSCAR, and DreamDojo-14B fine-tuned on the same AgiBot training split, or are the reported numbers from their original checkpoints? If not fine-tuned, can you provide an in-domain fine-tuning comparison?
  2. Can you provide a DreamDojo-2B + AgiBot fine-tuning baseline with identical data, updates, resolution, and action inputs, but without keyframe-specific HORAE modifications?
  3. For Table 4, can you provide a total-update-matched direct C+D+R baseline? This would help establish that the improvement comes from the modality curriculum rather than extra optimization.
  4. Similarly, can mixed-interval pretraining be compared against an equal-compute baseline that receives the same number of extra updates using only keyframe data?
  5. How does action-aware sampling compare with uniform sampling when the frame budget is much smaller, e.g. 10, 20, and 40 frames? I expect this would provide a more discriminative evaluation of the proposed sampler.
  6. Can HORAE-WM improve an actual downstream decision, e.g. candidate-action ranking or planning success, compared with the baseline world models?
  7. How sensitive is performance to errors in the future action trajectory? In deployment, commanded actions and realized robot states may differ substantially during contact.
4 comments
AuthorsOpenReview ↗

NkDT-1 · Same-backbone control and the role of the training scheme

We provide the same-backbone control requested (GR-1). The pretrained DreamDojo-2B reaches 13.848 dB PSNR without training. Standard fine-tuning on our keyframe training set, with the same initialization, resolution (400×640), 16-D action inputs, optimizer, batch size and number of epochs as the keyframe stage of HORAE-WM, reaches 18.087 dB PSNR, 0.632 Arm IoU and 288.6 mm DA3 depth error. This fine-tuning is RGB-only and uses the backbone's native temporal interface and loss. HORAE-WM reaches 19.000 dB, 0.678 Arm IoU and 263.8 mm, and leads on all six metrics. Most of the gain over the pretrained backbone thus comes from in-domain adaptation, and HORAE-WM adds a further +0.91 dB PSNR, +0.046 Arm IoU and −24.8 mm DA3 depth error.

HORAE-WM is built on top of the standard RGB training of DreamDojo. Its contributions are the formulation (Section 1), frame-wise physical-time conditioning, and the dual-axis progressive learning, in which the condition video and metric depth are added to the RGB objective as additional representations of the scene. The schedule extends the RGB training by adding these streams in stages. To check the schedule against the amount of training, we continue the direct tri-modal baseline to 4 and 6 epochs (GR-3), matching the curriculum in compute (12 stream-epochs) and in total updates (2,328). At equal updates the curriculum is better on all six metrics (17.388 vs. 17.238 dB PSNR; 0.2383 vs. 0.2396 LPIPS), using two thirds of the compute. At the same 1,552 updates, direct tri-modal training outperforms C→C+R, which has no depth stage, on PSNR, SSIM, LPIPS, Arm IoU and depth L1, which shows the value of the depth modality itself.

Mixed fixed-interval pretraining changes the data sampling, not the learning algorithm. The gain it brings is the gain from the additional sampled sequences under the same algorithm, rather than an algorithmic improvement. It re-samples the same 7,669 training episodes at four frame rates, adds no episodes, and provides the uniform-rate interface evaluated in Table 10. Table 3(b) reports its effect with the keyframe stage held at the same schedule: Arm IoU rises from 0.589 to 0.637 and PSNR from 17.30 to 18.12 dB.

AuthorsOpenReview ↗

NkDT-2 · Non-uniform protocol, depth and tighter budgets

We appreciate the reviewer's point that the keyframe comparison also reflects a systems interface, and we thank the reviewer for noting that Table 10 "substantially mitigates this concern". Explicit timestamps are the natural interface for non-uniform schedules, and the public action-conditioned models expose only a chunk-level frame rate. For the controls the reviewer suggested: a common backbone under identical non-uniform schedules is the standard fine-tuned DreamDojo-2B in GR-1, trained on exactly our keyframe schedules with the backbone's native temporal input; timestamp conditioning within the same model is covered by Table 3(a), where removing frame-wise time changes PSNR by 0.28 dB. Each baseline receives the keyframe-aligned action of every output and the most accurate temporal signal its interface admits, the average frame rate; in Table 10, where this signal is exact, the margins persist.

For depth, we restructure Table 1 into two blocks. The first is a common protocol for all RGB-based models, in which DA3 is applied to the generated frames. This is our primary cross-model depth comparison: HORAE-WM reaches 263.8 mm and 0.2242 AbsRel, against 300.3 mm and 0.2748 for the strongest baselines. The second block reports native metric depth for HORAE-WM only (83.4 mm, 0.0864), a capability that RGB-only models do not provide.

Budget K includes the initial frame, so K = 81, 41, 21 and 11 correspond to 80, 40, 20 and 10 predicted frames. We report event and hard-event coverage, enrichment, CH and the reconstruction error of selected keyframes at each budget (GR-2a), with per-task results in the appendix. Hard-event coverage shows the clearest tight-budget advantage: at K = 11 our schedule covers 66.07% of hard events, against 61.55% for uniform sampling. The margin in hard-event coverage is larger at each tighter budget than at K = 81 (+1.0, +1.5, +1.9 and +4.5 points at K = 81, 41, 21 and 11). At K = 81 the budget is generous for these segments, which average about 730 frames (about 24 s), so uniform sampling already falls within nearly every event. Hard events, as defined in Appendix B, include the brief but consequential subactions and reversals the reviewer refers to. Predicted video is evaluated at all four budgets (GR-2b), where ours is best on PSNR, SSIM and LPIPS over all frames and over hard-event frames. Any one of the three controls suggested by the reviewer addresses this point; we provide the third (GR-1) and the first (Table 3a). A piecewise variant with local intervals would change only the frame-rate signal for the baselines, whose effect is bounded by Table 3(a) and Table 10.

AuthorsOpenReview ↗

NkDT-3 · Scope, downstream utility, caption metric, action errors and presentation

Scope (Weakness 6). Our evidence concerns episode-level generalization within 16 dual-arm tasks, in simulation and in the real world, with a fixed base and a stable head camera. Segments with body motion or camera instability are filtered (Appendix A.1). We will qualify the claims in the abstract and in Sections 1 and 6, and list unseen tasks, object categories, embodiments, camera configurations and mobile manipulation as future work.

Downstream utility (Weakness 5, Question 6). Our study concerns predictive fidelity under a fixed output budget. Candidate-action ranking, planning and closed-loop evaluation are the strongest tests of downstream value, as the reviewer notes, and we list them as future work for this dual-arm platform.

Caption metric (Weakness 8). Caption completeness is a supplementary metric, and its generation and scoring prompts are documented in Appendix B. The primary sampling evidence is deterministic: event coverage and enrichment are computed directly from human annotations of the full RGB videos, with no model in the loop, and the GT-level reconstruction metrics in GR-2a and GR-2b are likewise deterministic.

Sensitivity to action errors (Question 7). The 16-D model input consists of commanded joint and gripper actions, and the model is conditioned on the commanded actions of the selected keyframes. All methods, including the baselines, are evaluated open-loop against the executed video. Robustness to the discrepancy between commanded and realized states during contact is important for deployment, and we list it as future work.

Presentation. We have revised the manuscript as follows: (1) an inference overview at the start of Section 3 that states what the model receives (initial RGBD observation, keyframe-aligned actions, physical times) and what it outputs; (2) Algorithm 1 for the sampler, with all hyperparameters; (3) the training schedule and the baseline-adaptation protocol moved from the appendix into the main text; (4) a table listing every ablation configuration with its total number of updates; and (5) unified terminology (keyframe, target time, condition video), with the condition video defined at first use.

Reviewer_NkDTOpenReview ↗

good response

sorry to see that most of your review, include mine, are ai generated.

And your feedback is good enough. But again, when we talk about the paper quality for acceptance, attitude is important but not the most important part.

I think a weak reject is the proper score for this paper for two reason, first , i do not like this contribution: "HORAE-WM is built on top of the standard RGB training of DreamDojo. Its contributions are the formulation (Section 1), frame-wise physical-time conditioning, and the dual-axis progressive learning, in which the condition video and metric depth are added to the RGB objective as additional representations of the scene. " since this is really boring contribution for pure robotic community, and i can just find this incremental result by using agentic recursive learning.

Second, this work's contribution is very incremental and i really do not like cosmos or vision based world model setting. Since this is not the right way for doing robotic.

Depth is good and important, but this contribution is limited, since this is not the first paper to purpose depth.

Thus, i remain my score. But your feed back is comprihensive enough. Thus, if other people raise their score, I will also raise my score. Because i think this paper is really bordline and either accept or reject should be fine.

Reviewer_pHMAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates long-horizon robot world modeling as action-conditioned keyframe prediction under a fixed output frame budget. Given an initial RGBD observation, the complete future action trajectory, and non-uniform target times chosen by an action-aware sampler, the model predicts the corresponding condition-video, metric-depth, and RGB keyframes. The method has three components: a data-side sampler that allocates the frame budget according to arm-motion boundaries and gripper events; a frame-wise conditioning mechanism that injects each target frame's physical time relative to the initial observation, together with its aligned action, through AdaLN modulation; and a curriculum that progresses along both a modality axis and a temporal-sampling axis, built on the DreamDojo backbone with an MoT structure that adds three modality streams. On simulated and real-world dual-arm manipulation data from AgiBot World, the paper reports consistent improvements over the compared world models in visual fidelity, arm-region consistency, and metric-depth accuracy.

Critical strengths
  1. The motivation is clear and addresses a practical problem. Action-conditioned world models can output only a limited number of frames, so uniform sampling must trade off trajectory coverage against brief interaction details, and the paper illustrates this tension with an intuitive example. More interestingly, the authors exploit a property of the action-conditioned setting: the complete future action trajectory is known before generation, so the timing of critical interactions can be read directly from the actions rather than estimated after the fact, as in keyframe methods that first generate frames and then infer the intervals between them. As a result, the sampler requires no learning, and the model receives the exact physical time of every target frame.

  2. Frame-wise physical-time conditioning provides a unified temporal interface. The same model can predict non-uniform keyframes and can also predict directly at different uniform frame rates without retraining, and its predicted keyframes can be interpolated into dense video. Pretraining on mixed fixed-interval sequences exposes the model to both short and long gaps, and the ablation shows that this step clearly benefits subsequent keyframe prediction.

  3. The tri-modal joint prediction is well designed. Predicting metric depth alongside RGB provides geometric information for downstream use. Treating the rendered action video as a prediction target rather than only as an input, and pairing it with a progressive attention mask so that depth and RGB successively attend to motion and geometry, is an interesting architectural choice. The ablation results are consistent with the curriculum direction of learning motion first, then geometry, and finally appearance, and the experiments cover both simulated and real-world data.

Critical weaknesses
  1. The evaluation targets in the main comparison are defined by the proposed method itself, while none of the baselines has been fine-tuned for this target format, so the comparison is not on equal footing. The set of evaluation targets is exactly the output of the proposed sampler, whereas the baselines, which do not support frame-wise time conditioning, are only given the average frame rate of the whole trajectory; the appendix itself notes that this conveys the average temporal spacing rather than the individual non-uniform intervals. The baselines therefore generate at uniform spacing, yet their i-th frame is compared against the i-th non-uniform target, so wherever the sampler departs from uniform spacing they depict the scene at a different moment, producing a systematic temporal misalignment by construction. The arm-region metric amplifies this misalignment: both the predicted and reference masks are obtained by prompting the same segmentation model with bounding boxes projected from ground-truth kinematics, so the box points to where the arm should be while the predicted arm is elsewhere, which turns timing errors into near-zero overlap and gives a well-formed but temporally shifted prediction a score similar to that of a deformed arm; the arm-pose mismatches repeatedly highlighted in the qualitative figures are consistent with this effect. On top of this is a training imbalance: all baselines use public checkpoints, whereas the proposed model is trained specifically for this format for about twelve days with a dedicated temporal curriculum, and the worst-performing baseline is a larger model from the same family as the proposed backbone whose pretraining data consist solely of human videos, exclude the robot domain used here, and lack a depth head, so its ranking reflects domain transfer rather than capacity, yet this is not discussed.

  2. All baselines are uniform or fixed-frame-rate world models; prior work on keyframe world models is not discussed, and the proposed keyframe scheme itself lacks an effective comparison. SKIP (He et al., arXiv:2606.00664, 2026) and KeyWorld (Li et al., arXiv:2509.21027, 2025) proposed keyframe prediction under a fixed budget before this submission: the former likewise treats gripper events as anchors that must be preserved while keeping the budget fixed, and the latter extracts keyframes by simplifying robot motion trajectories. Neither generator accepts action conditioning, both output only RGB and target single-arm settings, and neither has released runnable code or weights, so a direct comparison is not realistic; nevertheless, this prior formulation of the problem should be acknowledged in the related work, together with a clear statement of how this paper differs in action conditioning, frame-wise physical time, and RGBD prediction. Even without these methods, the evaluation of the proposed keyframe scheme remains insufficient: the sampling analysis only compares how much manipulation information is retained by frames selected from ground-truth videos, does not involve world-model predictions, and compares against uniform sampling only. Consequently, the central claim that action-aware keyframes lead to a better world model currently lacks direct evidence, whereas SKIP has already validated the downstream value of the keyframe abstraction with closed-loop policies, which this paper leaves to future work.

Questions for authors
  1. Has any baseline been fine-tuned for the target format used in this paper? If not, could the authors report the same evaluation for the DreamDojo backbone before their training, to quantify the net gain contributed by the proposed training? For baselines without a frame-rate conditioning interface, where exactly is the "single effective frame rate" described in the appendix injected? If the actual procedure is to construct inputs with a fixed stride and run a forward pass, please describe it as such. In addition, what is the distribution of physical durations of the non-uniform keyframe sequences in the main experiment, and how does it compare with the prediction horizons of the four fixed frame rates in the appendix? This quantity determines how far the single effective frame rate deviates from the actual schedule, and hence how severe the concern in W1 is, but it is not reported.

  2. Could the authors discuss SKIP and KeyWorld in the related work and clarify how this paper differs from them? In addition, could the sampling analysis include a trajectory-simplification-based selector as a comparison, such as AWE, which the paper already cites, or the classic RDP algorithm? Such methods rely only on proprioception, are inexpensive to implement, and solve the same class of optimization problem as the proposed sampler, so this comparison would show whether the anchor design provides additional benefit over purely error-driven selection.

5 comments
AuthorsOpenReview ↗

pHMA-1 · Temporal alignment, the temporal interface and Arm IoU

For the i-th output, each baseline receives the action of the i-th selected keyframe k_i (Appendix A.3, A.5), and the i-th output is compared with the GT frame at the same k_i. The arm configuration of every output is therefore specified by the keyframe-aligned action. The remaining difference concerns the temporal signal. Where a public baseline accepts a temporal input, it is a chunk-level frame rate injected through a dedicated positional encoding, and we provide the average frame rate of the sequence, the most accurate signal that interface admits. For models without a frame-rate input, the keyframe-aligned actions are provided in a single forward pass. We will revise Appendix A.3 to describe this per-model protocol precisely. Two results in the paper bound the effect of the interface. Removing frame-wise time from HORAE-WM changes PSNR by 0.28 dB (Table 3a). In Table 10, where all models receive exact uniform timing, HORAE-WM's margin over the strongest baseline (+1.84 to +3.09 dB) lies in the same range as in the keyframe setting (+1.99 dB).

Arm Mask-IoU measures action following: whether the arm lies where the conditioning action places it at the target time. Predicted and reference masks use the same segmentation procedure and the same GT-kinematics boxes, applied identically to all models. For an action-conditioned model, an arm at the wrong pose is an action-following error, which is what this metric captures. Appearance is measured separately by SSIM and LPIPS, where HORAE-WM also leads (LPIPS 0.193 versus 0.283 for GE-Sim 2.0). In Table 10, where no timing mismatch exists, the Arm IoU advantage persists (HORAE-WM 0.62–0.76, GE-Sim 2.0 0.45–0.58, EVAC 0.49–0.58).

AuthorsOpenReview ↗

pHMA-2 · Same-backbone control, training and prediction durations

None of the five public baselines was fine-tuned on our split; each is evaluated from its released checkpoint. To quantify the effect of training on our data, we add the same-backbone rows (GR-1). The pretrained DreamDojo-2B reaches 13.848 dB PSNR without training. Standard fine-tuning on our keyframe data, with the same initialization, resolution, action inputs, optimizer, batch size and number of epochs as the keyframe stage of HORAE-WM, reaches 18.087 dB, and HORAE-WM reaches 19.000 dB and leads on all six metrics. Relative to standard fine-tuning, HORAE-WM gains +0.91 dB PSNR, +0.046 Arm IoU and reduces DA3 depth error by 24.8 mm.

DreamDojo-14B is the AgiBot post-trained checkpoint, so its data domain matches ours. It has no depth head, and its depth is estimated by DA3, as for all RGB baselines. Table 1 reports DA3 depth for every RGB model under the same protocol (HORAE-WM 263.8 mm; strongest baseline 300.3 mm).

Durations (Question 1). The keyframe test sequences have a median physical duration of 10.3 s (p10–p90: 5.9–33.2 s; range 4.5–129.7 s). 51.7% fall in [5.33, 10.67) s, 31.2% in [10.67, 21.33) s, 7.9% in [21.33, 42.67] s, and 7.6% exceed 42.67 s; 1.7% are shorter than 5.33 s. Table 10 evaluates the four horizons of 5.33, 10.67, 21.33 and 42.67 s, which cover the bulk of this distribution.

AuthorsOpenReview ↗

pHMA-3 · Predictions against GT keyframes, the tri-modal design, and related work

We add evidence on generated predictions (GR-2b). With the same trained model and the same segments, we compare predictions made from uniform, AWE and our schedules at each budget (K = 81, 41, 21 and 11). Predicted keyframes are interpolated to the original frame rate with the interpolator of Appendix A.5 and compared with the full ground-truth video. Ours is best on PSNR, SSIM and LPIPS, over all frames and over frames inside annotated hard events, at every budget.

We thank the reviewer for highlighting the tri-modal design. The condition video that renders the robot's actions and the metric depth are additional, complementary representations of the same scene, learned jointly with RGB as part of the dual-axis progressive learning (Section 1, contribution 3). Rendering the action video as a prediction target, not only as an input, lets the model learn motion before geometry and appearance, consistent with the progression in Table 4.

We thank the reviewer for suggesting AWE and RDP. We compare both with our sampler at K ∈ {81, 41, 21, 11} on GT keyframes (GR-2a) and on predicted video at all four budgets (GR-2b). RDP and AWE select frames by reconstruction error alone, while our sampler also keeps segment endpoints, gripper events and arm-motion boundary anchors. Ours achieves the highest event and hard-event coverage at every budget, and the highest reconstruction quality on hard-event frames. On whole-video reconstruction, RDP and AWE are below uniform sampling at every budget, indicating that error-driven selection alone does not preserve the manipulation process as well as the anchor-based design.

We have added SKIP (He et al., 2026) and KeyWorld (Li et al., 2025) to the related work as prior formulations of keyframe prediction under a fixed budget. The main differences are summarized below.

KeyWorldSKIPHORAE-WM
Keyframe generator conditioned ontext task descriptioninitial observation and language instructioninitial RGBD observation, keyframe-aligned actions, per-frame physical time
Keyframe selectioniterative simplification of robot motion trajectoriesrobot-aware multimodal features, with gripper-event protectionsegment-endpoint and gripper-event anchors, arm-motion boundaries, reconstruction-error allocation under a fixed budget
OutputsRGBRGBrendered action video, metric depth, RGB
DataLIBEROLIBERO and real robotAgiBot dual-arm, simulation and real

SKIP validates the downstream value of keyframes with closed-loop policies. Our study concerns predictive fidelity under a fixed output budget and makes no policy-level claim; closed-loop evaluation on this dual-arm platform is future work, which we state in Sections 1 and 6.

Reviewer_ayL1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HORAE-WM, an action-conditioned RGBD world model designed to balance long-horizon prediction and the preservation of short but important manipulation events under a limited output-frame budget. The main idea is to select keyframes based on arm and gripper dynamics, rather than sampling uniformly in time. The model further incorporates physical-time conditioning, jointly predicts rendered condition videos, metric depth, and RGB frames, and uses a progressive training curriculum across modalities and temporal sampling settings. Experiments on 16 simulated and real-world AgiBot World manipulation tasks show improvements in visual quality, robot-arm consistency, and depth estimation over existing world models. The proposed sampling strategy also retains more manipulation events than uniform sampling.

Critical strengths
  1. The paper addresses a relevant problem in long-horizon manipulation prediction. With a fixed number of output frames, uniform sampling can easily miss short interaction events. The proposed action-aware selection strategy instead places more frames around arm-motion changes and gripper events. The results in Table 2 are encouraging: event coverage increases from 96.86% to 98.26%, while hard-event enrichment improves from 0.997 to 1.223. These results suggest that using robot-action information to allocate frames is a reasonable way to preserve manipulation-relevant events.

  2. The improvements in RGB prediction quality and robot-arm consistency are substantial. Compared with GE-Sim 2.0, the proposed method improves PSNR from 17.008 to 19.000 dB, reduces LPIPS from 0.2832 to 0.1934, and increases Arm Mask-IoU from 0.5348 to 0.6780 (Table 1). The depth results are also promising. While the native depth predictions are not directly comparable to the DA3-derived depth used for the baselines, the common-DA3 evaluation still shows an improvement in depth L1 error from 300.3 to 263.8 mm. Overall, the gains across multiple metrics make the experimental results fairly convincing.

  3. The paper includes useful ablations and additional evaluations under different temporal sampling settings. Table 3 reports improvements from physical-time conditioning (+0.277 dB PSNR), mixed-interval pretraining (+0.816 dB), and structure-aware loss weighting. I also appreciate the evaluation in Table 10, where the model consistently outperforms GE-Sim 2.0 and EVAC at four uniform sampling rates, ranging from 15 to 1.875 fps. These experiments help support the proposed design beyond the main non-uniform sampling setting, although there are still some concerns about differences in training budgets.

Critical weaknesses
  1. The comparison with existing world models is not fully controlled. One concern is that HORAE-WM is fine-tuned on the target AgiBot World training split, while the baselines in Table 1 appear to use publicly released checkpoints without comparable in-domain fine-tuning. There is also a difference in how temporal information is provided. According to Appendix A.3, the baselines receive a single effective FPS based on the average spacing of the selected keyframes, whereas HORAE-WM receives the actual physical timestamps. This may put models designed for uniform temporal intervals at a disadvantage. The uniform-rate evaluation in Table 10 is helpful, but it does not completely resolve this issue. A comparison using the same backbone and training data would make it easier to understand how much of the improvement comes from the proposed method itself, rather than differences in adaptation or temporal conditioning.

  2. The benefit of action-aware sampling for the actual generated predictions is not sufficiently demonstrated. Table 2 mainly measures how much information is preserved when selecting frames from ground-truth trajectories. This is useful, but it does not necessarily mean the model generates more accurate predictions at those important moments. In fact, uniform sampling already achieves 96.86% event coverage, and the gain in caption completeness is relatively modest (4.45 to 4.56). I would like to see a more direct comparison between action-aware and uniform sampling using the same trained backbone, prediction horizon, and frame budget. Evaluating the quality of generated frames around critical interaction events, especially under tighter frame budgets, would provide stronger evidence for the paper's main claim.

  3. The methodological novelty seems to lie mainly in combining several existing ideas. For example, the action-reconstruction-based keyframe selection is related to automatic waypoint extraction in Shi et al. (2023). DLFR-Gen (Yuan et al., 2025) also considers non-uniform temporal information density, although it relies on visual information rather than robot actions. In addition, the model builds heavily on DreamDojo, while rendered action conditioning has connections to EVAC, and physical-time embeddings are incorporated through a fairly standard AdaLN mechanism. The combination of these components for action-conditioned RGBD prediction is useful, but I am less convinced about the novelty of the individual designs. Comparisons with simpler action-waypoint selection methods or alternative temporal-conditioning schemes would help clarify what is specifically gained from the proposed formulation.

  4. The curriculum ablations are confounded by differences in training duration. In Table 4 and Appendix A.4, the progressive C → C+D → C+D+R training uses six epochs in total, whereas the direct C+D+R baseline is trained for only two epochs. A similar issue appears in Table 3(b), where mixed-interval pretraining introduces two additional epochs, with only the subsequent keyframe-training stage matched. As a result, it is difficult to tell whether the gains come from the curriculum design or simply from additional optimization. Since progressive training is presented as an important part of the method, I think a compute-matched comparison with direct training would be necessary to better support this claim.

  5. Some aspects of the evaluation and implementation need further clarification. The test set contains 1,242 sequences from held-out episodes of the same 16 tasks, with considerably more real-world sequences than simulated ones (1,162 vs. 80). It is unclear how well the method generalizes beyond these tasks. More detailed results across prediction horizons, temporal gaps, and different levels of interaction difficulty would also be valuable, particularly given the emphasis on long-horizon prediction. The paper does not report variability across independent training runs. On the implementation side, the action-reconstruction-based keyframe selection in Sec. 3.1 and Appendix B is not described in enough detail to reproduce confidently, especially regarding its optimization procedure and what happens when the number of anchors exceeds the frame budget.

Questions for authors
  1. Could the authors provide a comparison with DreamDojo-2B using the same backbone and training data, ideally controlling for target-task fine-tuning, action representation, and physical-time conditioning? This would help clarify the source of the improvements reported in Table 1.

  2. How does action-aware sampling compare with uniform sampling and simpler action-waypoint selection methods when the prediction horizon, frame budget, and trained backbone are kept the same? I am particularly interested in smaller frame budgets and the quality of generated frames around critical manipulation events.

  3. Could the authors include compute-matched ablations for mixed-interval pretraining and progressive modality training? In particular, direct and progressive training should ideally have comparable numbers of optimizer updates and total data exposure.

  4. Could the authors explain the keyframe-selection procedure in more detail, including the action-reconstruction error, temporal-gap constraints, and handling of anchors that exceed the frame budget? Also, Eq. (1) uses the sampled actions AKA_K rather than the full action trajectory. Does the predictor receive any information about the intermediate actions that were not selected?

4 comments
AuthorsOpenReview ↗

ayL1-1 · Same-backbone control and the temporal interface

We thank the reviewer for the careful reading. We fine-tune DreamDojo-2B on our keyframe training data (GR-1), using the same initialization, resolution, 16-D action input, optimizer, batch size and number of epochs as our keyframe stage, with RGB only and none of our components. This model reaches 18.087 dB PSNR and 0.632 Arm IoU, and HORAE-WM outperforms it on all six metrics (19.000 dB, 0.678). The remaining gain, +0.91 dB PSNR, +0.046 Arm IoU and −24.8 mm DA3 depth error, is attributable to our method.

On the temporal interface, none of the public baselines accepts frame-wise timestamps. Where a baseline accepts a temporal input, it is a chunk-wise frame rate injected through a dedicated positional encoding; for these models we provide the average frame rate of the selected keyframe sequence, the most accurate temporal signal their interface admits. For the remaining models, the keyframe-aligned actions are provided in a single forward pass. For every output we provide the action of its own keyframe, and the i-th output is compared with the GT frame at the same keyframe k_i. Two results in the paper bound the effect of this interface. Removing frame-wise time from HORAE-WM changes PSNR by 0.28 dB (Table 3a). In Table 10, where every model receives exact uniform timing, HORAE-WM's margin over the strongest baseline (+1.84 to +3.09 dB) lies in the same range as in the keyframe setting (+1.99 dB).

AuthorsOpenReview ↗

ayL1-2 · Evidence on predictions and the position of the novelty

We add evidence on the generated predictions, not only on the selected ground-truth frames. With the same trained model and the same segments, we run inference with uniform, AWE and our schedules at each budget (K = 81, 41, 21 and 11), interpolate the predicted keyframes to the original frame rate with the interpolator of Appendix A.5, and compare with the full ground-truth video (GR-2b). Ours is best on PSNR, SSIM and LPIPS, both over all frames and over frames inside annotated hard events. The GT-level analysis (GR-2a) covers the same four budgets and includes RDP and AWE, and the ordering holds at every budget. The advantage is clearest under tight budgets: at K = 11 our schedule covers 66.07% of hard events, against 61.55% for uniform sampling, and it gives the highest reconstruction quality on hard-event frames.

The individual building blocks, such as AdaLN conditioning and the DreamDojo backbone, follow established designs. The contributions of HORAE-WM are its formulation and the auxiliary modalities it adds. In the formulation, future actions are known before generation, so the target schedule can be fixed from them and the exact physical time of each target can be given to the generator. Keyframe-first methods infer intervals only after generating frames, and DLFR-Gen adapts the latent frame rate to visual motion. Reviewer pHMA made the same observation, noting that critical timings "can be read directly from the actions rather than estimated after the fact". The auxiliary modalities, the condition video that renders the robot's actions and the metric depth, are additional, complementary representations of the same scene. They form a central part of the dual-axis progressive learning (Section 1, contribution 3) and extend what an RGB-only backbone learns.

Compared with Shi et al. (2023), our selection also uses reconstruction error, but it is designed for a fixed output budget. It keeps segment endpoints, gripper events and arm-motion boundaries, bounds the temporal gap, and allocates the remaining frames by reconstruction error. Under identical budgets we compare directly with the waypoint-style selectors RDP and AWE (GR-2a). Both fall below uniform sampling on whole-video reconstruction at every budget, and ours is best on every coverage, reconstruction and prediction metric. The anchors and the budget-aware allocation therefore add value beyond error-driven waypoint selection. We will also describe the connection to EVAC in the related work.

AuthorsOpenReview ↗

ayL1-3 · Training budgets, mixed pretraining, scope and the sampler procedure

Training budgets. The modality curriculum extends the standard RGB training of DreamDojo. The base objective is kept, and the auxiliary streams are added progressively, so the gains come from learning these additional representations on top of the base training. The 4- and 6-epoch direct-training results in GR-3 show that the curriculum is the more effective way to learn them, using about two thirds of the compute at the same number of updates.

Mixed fixed-interval pretraining. This stage changes the data sampling, not the learning algorithm. The gain it brings is therefore a gain from the additional sampled sequences, rather than from an algorithmic improvement, and the appropriate reading is the effect of these data under the same algorithm. Table 3(b) reports this effect with the keyframe stage held at the same schedule: Arm IoU rises from 0.589 to 0.637 and PSNR from 17.30 to 18.12 dB. The stage re-samples the same 7,669 training episodes at four frame rates, so it adds no episodes, and it provides the uniform-rate interface evaluated in Table 10.

Prediction horizons, interaction difficulty and domains. The keyframe test sequences have a median physical duration of 10.3 s (p10–p90: 5.9–33.2 s; range 4.5–129.7 s). 51.7% fall in [5.33, 10.67) s, 31.2% in [10.67, 21.33) s, 7.9% in [21.33, 42.67] s, and 7.6% exceed 42.67 s; 1.7% are shorter than 5.33 s. Table 10 reports the four horizons of 5.33, 10.67, 21.33 and 42.67 s, and HORAE-WM leads at each of them. Hard events, which are annotated brief and consequential subactions, reversals and salient state changes, are part of the test segments; results inside them are in GR-2a and GR-2b, and HORAE-WM is best on all three metrics at every budget. The test set follows the observed composition (1,162 real and 80 simulated sequences). Table 8 reports each domain separately (PSNR 19.02 real, 18.78 simulated), and both domains support the same conclusions.

Scope. The split is episode-disjoint within 16 dual-arm tasks. Our claims concern in-distribution tasks with a fixed head camera. Cross-task and cross-embodiment generalization are important directions for future work. The conclusions rest on consistent results across all budgets, subsets and metrics (GR-2a, GR-2b), rather than on any single configuration.

Keyframe-selection procedure and intermediate actions. The sampler (Section 3.1, Appendix B) first fixes the anchors: segment endpoints, the start and end of gripper opening and closing, and representative frames of arm-motion boundaries. When the anchors fit within the budget, all of them are retained. When they exceed it, surplus anchors are removed by selecting, among the ways of removing them that satisfy the budget, the one with the lowest action reconstruction error over the complete trajectory. The remaining frames then cover the large temporal gaps, and among the feasible schedules that satisfy the gap constraint, the one with the lowest reconstruction error is selected. We will add Algorithm 1 to Section 3.1, listing the reconstruction-error definition, the gap constraint, the rule for surplus anchors and all hyperparameters, and we will release the implementation. The predictor does not receive the actions of unselected frames: Equation (1) conditions on the sampled actions A_K and the selected times τ_K, and the complete trajectory is used only by the sampler to choose keyframes. We will revise the phrase "complete future action trajectory" in Sections 1–2 so that the predictor's input is stated unambiguously.

Reviewer_ayL1OpenReview ↗

Official Comment to Authors

Thanks for the responses, which address some of my concerns. However, the minor distinctions between the proposed method and existing priors limit the contributions and can not support acceptance for me. Therefore, I retain my rating.

Comments on the paper

4 comments
AuthorsOpenReview ↗

GR-1 · Same-backbone control

We thank all reviewers for their careful reading and constructive comments. All three reviewers find the problem practical and the formulation well motivated, in particular the use of known future actions to place keyframes and the exact physical time given to each target. Three shared concerns were raised: (i) the comparison with external baselines needs a same-backbone control; (ii) the sampler should be evaluated on predictions and under tighter budgets; and (iii) the training ablations should be compared at matched compute. We address them in GR-1 to GR-3.

To give a same-backbone control, we add two rows initialized from DreamDojo-2B, the backbone used by HORAE-WM. The zero-shot row is the pretrained DreamDojo-2B with our 16-D action input and no further training. The standard fine-tuning row trains the same model on our 24,843 keyframe training sequences for 10 epochs (970 updates, batch size 256, 400×640). It is RGB-only and uses the backbone's native temporal interface and loss. All rows are evaluated on the same 1,242 keyframe test sequences.

ModelPSNR ↑SSIM ↑LPIPS ↓Arm IoU ↑DA3 L1 (mm) ↓DA3 AbsRel ↓
DreamDojo-2B, zero-shot13.8480.36790.41730.3165452.30.3619
DreamDojo-2B, standard fine-tuning18.0870.57900.22920.6324288.60.2440
GE-Sim 2.017.0080.58730.28320.5348300.70.2748
HORAE-WM19.0000.60710.19340.6780263.80.2242

Standard fine-tuning on in-domain data is the strongest baseline in our study: it exceeds GE-Sim 2.0 on five of the six metrics. Its gain over the pretrained backbone is +4.24 dB PSNR. HORAE-WM leads standard fine-tuning on all six metrics, with a further +0.91 dB PSNR, +0.046 Arm IoU and a 24.8 mm reduction in DA3 depth error. HORAE-WM also predicts metric depth natively (83.4 mm L1), which RGB-only models do not provide. For RGB-only models, depth is estimated by DA3 from the generated frames, so the DA3-based value (263.8 mm) is the cross-model comparison.

Three points follow from GR-1. First, most of the gain over the pretrained backbone (+4.24 dB PSNR) comes from adapting the model to the target data, which confirms the value of in-domain training for this task. Second, once the backbone is adapted to the same data, HORAE-WM still leads on all six metrics. The remaining margin (+0.91 dB PSNR, +0.046 Arm IoU, −24.8 mm depth error) is therefore attributable to the auxiliary streams and the staged schedule, rather than to in-domain training. Third, the depth result is the most direct test of the auxiliary depth stream: HORAE-WM predicts depth natively, while the RGB-only models must rely on DA3, and the 263.8 mm value is obtained under the same DA3 protocol as the standard fine-tune (288.6 mm), a relative reduction of 8.6%.

AuthorsOpenReview ↗

GR-2a · Keyframe sampler: GT-level evaluation

We compare our sampler with uniform sampling and with two trajectory-simplification methods, RDP and AWE (Shi et al., 2023). RDP and AWE also select keyframes from the action trajectory, but they use reconstruction error alone, without gripper events or arm-motion boundaries. All methods select the same number of frames K, which includes the initial frame. Our sampler keeps all segment endpoints, gripper events and arm-motion boundary anchors whenever they fit within the budget; otherwise it removes surplus anchors in the way that yields the lowest reconstruction error. Among feasible candidate schedules it selects the one with the lowest reconstruction error, as described in Section 3.1 and Appendix B.

(a) Coverage on the annotated events of the 100 real-world segments (Appendix B). The event coverage, enrichment and CH values of Ours and Uniform at K = 81 match Table 2. Hard-event coverage uses the same annotations.

KMethodEvent cov. (%) ↑Hard-event cov. (%) ↑Enrichment H ↑CH ↑
81Ours98.2697.211.2231.205
81Uniform96.8696.170.9970.963
81RDP96.1495.631.2191.202
81AWE96.1794.241.2031.171
41Ours94.4293.571.2911.191
41Uniform93.9092.061.0520.953
41RDP91.9689.821.2851.182
41AWE92.3091.101.2811.184
21Ours87.9485.171.3901.161
21Uniform86.5783.241.0770.914
21RDP84.5979.631.3621.154
21AWE84.3979.541.2891.100
11Ours76.0766.071.3791.059
11Uniform73.7061.551.1770.844
11RDP72.1458.221.3120.939
11AWE73.2264.411.3130.985

Ours has the highest event coverage, hard-event coverage, enrichment and CH at every budget. At K = 11 its hard-event coverage is 4.5 points above uniform sampling.

(b) Reconstruction of the full video from GT keyframes (Practical-RIFE interpolation to 30 Hz, all frames).

KMethodPSNR ↑SSIM ↑LPIPS ↓
81Ours33.9370.97160.0161
81Uniform32.2230.96380.0177
81RDP30.5540.95020.0262
81AWE30.6110.95030.0264
41Ours30.5640.95930.0273
41Uniform29.1520.94430.0294
41RDP27.7710.92980.0389
41AWE27.8680.92980.0391
21Ours26.8370.92570.0435
21Uniform26.0480.91720.0469
21RDP25.1020.90280.0576
21AWE25.1250.90320.0578
11Ours23.6230.88740.0695
11Uniform23.4440.88340.0712
11RDP22.8840.87190.0808
11AWE22.8640.87410.0804

Ours is best on all three metrics at every budget. On whole-video reconstruction, RDP and AWE are below uniform sampling at every budget.

(c) The same reconstruction, restricted to frames inside annotated hard events.

KMethodPSNR ↑SSIM ↑LPIPS ↓
81Ours32.9750.96240.0171
81Uniform32.4850.96170.0184
81RDP32.3250.96070.0195
81AWE32.2860.96060.0201
41Ours30.2270.94460.0289
41Uniform29.3730.94140.0305
41RDP28.9840.93920.0329
41AWE29.2250.94050.0323
21Ours26.9590.91750.0458
21Uniform26.1670.91410.0480
21RDP26.0620.91210.0506
21AWE25.9570.91240.0501
11Ours24.1310.89090.0702
11Uniform23.4740.88290.0711
11RDP23.4620.87970.0749
11AWE23.4790.88200.0732

Ours is best on all three metrics at every budget within hard events.

The GT-level results show that the anchors and the budget-aware allocation retain information that error-driven selection does not. At K = 81, uniform sampling already covers 96.86% of events, and the difference is concentrated in hard events: our schedule is 1.0 point above uniform at K = 81 and 4.5 points above at K = 11. RDP and AWE also select frames by action reconstruction error, and both fall below uniform on whole-video reconstruction at every budget. Their hard-event coverage is also below uniform at K = 81, 41 and 21. Selecting by reconstruction error alone therefore does not reproduce our gains in hard-event coverage. Enrichment and CH, which measure how concentrated the selected frames are on hard events, follow the same ordering at every budget.

AuthorsOpenReview ↗

GR-2b · Keyframe sampler: predicted video

Building on the GT-level analysis in GR-2a, we evaluate the predictions of the trained model.

(d) Predicted video from the trained HORAE-WM model. With the same trained model, we run inference with the uniform, AWE and ours schedules at each budget. For K < 81 the output sequence is shortened, which the model supports directly. Predictions are interpolated to the original video length and compared with the full video.

KMethodPSNR ↑ (all)SSIM ↑ (all)LPIPS ↓ (all)PSNR ↑ (hard)SSIM ↑ (hard)LPIPS ↓ (hard)
81Ours18.7710.65880.181218.7230.63730.1893
81Uniform18.7360.62960.190018.6370.62430.1924
81AWE18.7200.63240.191218.6850.62700.1928
81RDP18.6680.63150.192118.6120.62610.1934
41Ours18.1890.61770.211318.1520.61370.2203
41Uniform18.0690.60280.223818.0310.59900.2239
41AWE18.0950.60500.223118.1190.60070.2214
41RDP18.0190.60360.224918.0470.59970.2235
21Ours16.1970.53790.321716.0490.49860.3651
21Uniform16.0820.50260.358815.9750.48760.3743
21RDP16.0490.50090.363615.9620.48570.3763
21AWE16.0140.49770.366015.9080.48220.3775
11Ours12.9160.38910.512912.7760.35380.5713
11Uniform12.8450.36350.548412.6500.33720.5805
11RDP12.7850.35780.553412.4770.32490.5903
11AWE12.8630.36180.552712.5340.33180.5863

Ours is best on every metric at every budget, for both the whole video and the hard-event frames.

The predicted-video results test whether the schedule advantage survives the generation step. Predictions include model error that is common to all schedules, so the relevant evidence is the consistency of the ordering: ours is best on PSNR, SSIM and LPIPS, over all frames and over hard-event frames, at every budget. Together with the GT-level results in GR-2a, this shows that a schedule preserving more manipulation-relevant frames also yields better generated video. The same ordering holds at K < 81, where the output sequence is shortened without retraining.

AuthorsOpenReview ↗

GR-3 · Training scheme under matched compute

The modality curriculum of Table 4 extends the standard RGB training of DreamDojo: the base objective is kept, and the condition-video and depth streams are added in stages. To check the effect of the schedule against the amount of training, we continue the direct C+D+R baseline to 4 and 6 epochs. Relative compute counts the video streams predicted in each stage (C = 1, C+D = 2, C+D+R = 3) multiplied by the number of epochs. All streams share the same resolution and length, so this count is proportional to token computation. Each curriculum stage runs for two epochs, giving 2,328 updates in total. All rows use keyframe data only.

TrainingEpochsUpdatesRel. computePSNR ↑SSIM ↑LPIPS ↓Arm IoU ↑DA3 L1 ↓AbsRel ↓
direct C+D+R2776616.9580.55250.25200.5364289.20.2608
direct C+D+R41,5521217.1550.56110.24390.5567285.10.2576
direct C+D+R62,3281817.2380.56190.23960.5633282.70.2528
C → C+D → C+D+R (curriculum)62,3281217.3880.56250.23830.5742282.20.2499

The results support three conclusions. First, the auxiliary streams add information beyond the RGB objective: at the same 1,552 updates, direct tri-modal training outperforms C→C+R, which has no depth stage, on PSNR, SSIM, LPIPS, Arm IoU and depth L1. Second, the staged schedule is an effective way to learn these streams: at equal updates the curriculum outperforms direct training on all six metrics while using two thirds of the compute, and at equal compute (4 direct epochs) its lead is +0.23 dB PSNR and +0.018 Arm IoU. Third, longer direct training gains less with each additional epoch (+0.20 dB from 2 to 4 epochs, +0.08 dB from 4 to 6). We will replace Table 4 with these matched comparisons and revise Section 4.4 accordingly.

The three conclusions share one reading. The first shows that the auxiliary streams carry information that the RGB objective does not capture, since at the same number of updates the tri-modal model outperforms the RGB-plus-condition-video model. The second shows that the staged schedule is an effective way to learn these streams, since at the same number of updates it is better than direct training on all six metrics. The third shows that longer direct training improves the model only at a decreasing rate. Together, these results indicate that the benefit of the schedule does not come from additional optimization steps alone.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.