- Rating
- 2
- Confidence
- 4
- Informativeness
- 3(+1)
- Presentation
- 3(+2)
- Soundness
- 2
Read review
Paper summary
This paper studies a practical limitation of action-conditioned video world models for long-horizon robot manipulation: under a fixed number of generated frames, high-rate uniform sampling preserves local motion but covers only a short time horizon, while low-rate sampling extends the horizon but can skip short yet important interaction events such as grasping or releasing. The paper proposes HORAE-WM, which instead predicts non-uniform RGBD keyframes selected according to robot action dynamics
The method has four main ingredients:
- Action-aware keyframe selection. Frames around gripper events and arm-motion boundaries receive higher priority, while the remaining budget is allocated across relatively static or slowly changing intervals.
- Frame-wise physical-time conditioning. Each predicted frame is conditioned on its actual elapsed physical time and aligned robot action, rather than merely its ordinal frame index.
- Tri-modal prediction of condition video, metric depth, and RGB, implemented using a Mixture-of-Transformers-style DiT with progressive cross-modality attention.
- Progressive training, first using multiple fixed temporal intervals and then non-uniform keyframe sequences, while progressively introducing condition video, depth, and RGB supervision.
Experiments use 16 simulated and real-world dual-arm manipulation tasks from AgiBot World, with 7,669 training and 285 test episodes. The main evaluation uses 1,242 held-out sequences. HORAE-WM reports substantial improvements over GE-Sim 2.0, EVAC, Cosmos3-Nano, OSCAR, and DreamDojo-14B in RGB quality, arm consistency, and depth-related metrics.
Critical strengths
- The paper identifies a real and underexplored problem in long-horizon world modeling. The fixed-output-budget argument is compelling. Robot trajectories indeed contain long periods of relatively uninformative motion interspersed with short interaction-critical transitions, so allocating frames uniformly is not necessarily optimal. Framing temporal abstraction as an action-conditioned sampling problem is sensible and particularly relevant for manipulation.
- The proposed formulation is simple and well motivated. Using actual physical timestamps is an appropriate solution once non-uniform sampling is introduced. Without explicit timing information, the model cannot distinguish, for example, two adjacent output slots separated by 0.1 s from those separated by 2 s. This design follows naturally from the problem formulation rather than appearing as an arbitrary architectural modification.
Critical weaknesses
- The primary comparison does not appear sufficiently controlled for training data and adaptation
This is my largest concern.
HORAE-WM is initialized from DreamDojo-2B and extensively trained on the target manipulation dataset, including approximately 146k fixed-interval sequences plus 24.8k keyframe sequences. In contrast, the main baselines are evaluated using their publicly released checkpoints, with model-specific input adapters.
It is therefore unclear how much of Table 1 reflects:
- the proposed temporal representation,
- the tri-modal architecture,
- in-domain AgiBot training,
- substantially different training budgets,
- or differences in the underlying pretrained models. A particularly important comparison would be DreamDojo-2B fine-tuned on exactly the same training data and for approximately the same optimization budget, differing only in the HORAE-specific components. Without such a comparison, I do not think the paper convincingly establishes that HORAE-WM's large advantage over external models results primarily from its proposed methodological innovations.
- The non-uniform keyframe comparison intrinsically advantages HORAE-WM
HORAE-WM receives an explicit physical timestamp for every target keyframe. The competing models do not support this interface, so the paper converts the full non-uniform trajectory into a single average frame rate, losing the actual per-frame temporal intervals.
This means that the main experiment effectively asks: a model explicitly designed to know the exact non-uniform target times versus models that are not told those target times.
This is useful as a systems comparison, but it is not a clean test of predictive modeling quality. The fixed-frame-rate evaluation in Table 10 substantially mitigates this concern and is a good addition. However, I would still like to see a stronger controlled baseline, for example:
- timestamp conditioning added to the same DreamDojo backbone;
- piecewise generation of baseline predictions using the appropriate local interval;
- or a common backbone trained under identical non-uniform schedules.
- Some important ablations are not compute matched
The modality-progression ablation is potentially confounded by total optimization budget.
According to Appendix A.4, the curriculum variants use different total numbers of epochs: the full C → C+D → C+D+R curriculum receives substantially more total training than the direct tri-modal control.
Yet the paper concludes that progressive learning itself explains the improvement: C+D+R direct: PSNR 16.958 progressive C→C+D→C+D+R: PSNR 17.388.
A compute-matched direct tri-modal baseline is needed to separate the effect of curriculum ordering from simply receiving more gradient updates.
A similar issue exists for the mixed-interval pretraining ablation: one variant receives additional fixed-interval pretraining before the same keyframe training. Thus the experiment establishes that additional mixed-interval training helps, but does not fully demonstrate that the benefit comes specifically from mixed temporal intervals rather than extra optimization/data exposure.
I view this as an important experimental weakness because progressive training is presented as one of the main methodological contributions.
4. The strongest depth number is not directly comparable to baseline depth numbers
HORAE-WM predicts depth natively and is explicitly trained with metric-depth targets, while most competing models generate RGB and then obtain depth through DA3.
The headline result of 83.4 mm versus ≈300–400 mm therefore mixes two rather different settings. The paper appropriately includes a more comparable experiment where DA3 is applied to HORAE-WM's RGB, yielding 263.8 mm, still better than the strongest baseline.
That source-matched comparison is convincing, but I would recommend emphasizing it as the primary cross-model depth result rather than juxtaposing native depth against RGB-derived depth in the main table.
5. No downstream robotic utility is demonstrated
The paper motivates HORAE-WM as a long-horizon robot world model, but all principal evaluations measure:
- RGB reconstruction quality,
- perceptual similarity,
- arm segmentation overlap,
- depth reconstruction,
- human visual preference,
- and keyframe information retention. There is no evaluation showing that the better world model improves:
- action selection,
- planning,
- policy ranking,
- model-predictive control,
- RL,
- or manipulation success.
The authors acknowledge this limitation explicitly.
This matters because a visually good future predictor is not necessarily a useful dynamics model. For an ICLR paper on robotic world modeling, even a small experiment such as ranking candidate action trajectories using HORAE-WM would significantly strengthen the contribution.
- Generalization is not extensively evaluated The training/test split is episode-disjoint but remains within the same 16 tasks. There is no clear evaluation on:
- unseen tasks,
- unseen object categories,
- substantially different environments,
- unseen robot embodiments,
- or new camera configurations. Moreover, the data preprocessing filters for body motion and camera stability. This makes the setup substantially easier than mobile manipulation settings where camera motion and base motion interact with object dynamics. The paper should therefore be careful about broad claims regarding long-horizon robot world modeling; the evidence is strongest for episode-level generalization within a specific dual-arm manipulation distribution.
- Improvements from the action-aware sampler are somewhat modest on the most intuitive metrics
On 100 real-world segments, coverage increases from 96.86% to 98.26%, while caption completeness increases from 4.45 to 4.56 / 5. The largest improvement is enrichment, from 0.997 to 1.223.
This does indicate more concentrated sampling of information-dense events, but uniform sampling is already extremely strong in terms of event coverage. I would therefore like to see:
- performance as a function of increasingly severe frame budgets, e.g. 81 / 40 / 20 / 10 frames;
- bootstrap confidence intervals;
- per-task breakdowns;
- and event coverage specifically for short/hard events. I suspect the advantage of action-aware sampling would become considerably clearer under tighter budgets, which would strengthen the central argument.
- The model-assisted caption metric has reproducibility concerns Caption completeness uses independent GPT-6 Astra calls both to produce descriptions and to grade them. The prompt is documented carefully, which is good, but the metric remains dependent on a proprietary, potentially changing model. At minimum I would prefer:
- repeated runs / variance estimates;
- agreement with human evaluation on a subset;
- or an additional deterministic event-recall metric. This is not fatal because caption completeness is supplementary rather than the primary evidence.
Questions for authors
Questions for the authors
- Were GE-Sim 2.0, EVAC, Cosmos3-Nano, OSCAR, and DreamDojo-14B fine-tuned on the same AgiBot training split, or are the reported numbers from their original checkpoints? If not fine-tuned, can you provide an in-domain fine-tuning comparison?
- Can you provide a DreamDojo-2B + AgiBot fine-tuning baseline with identical data, updates, resolution, and action inputs, but without keyframe-specific HORAE modifications?
- For Table 4, can you provide a total-update-matched direct C+D+R baseline? This would help establish that the improvement comes from the modality curriculum rather than extra optimization.
- Similarly, can mixed-interval pretraining be compared against an equal-compute baseline that receives the same number of extra updates using only keyframe data?
- How does action-aware sampling compare with uniform sampling when the frame budget is much smaller, e.g. 10, 20, and 40 frames? I expect this would provide a more discriminative evaluation of the proposed sampler.
- Can HORAE-WM improve an actual downstream decision, e.g. candidate-action ranking or planning success, compared with the baseline world models?
- How sensitive is performance to errors in the future action trajectory? In deployment, commanded actions and realized robot states may differ substantially during contact.
4 comments
NkDT-2 · Non-uniform protocol, depth and tighter budgets
We appreciate the reviewer's point that the keyframe comparison also reflects a systems interface, and we thank the reviewer for noting that Table 10 "substantially mitigates this concern". Explicit timestamps are the natural interface for non-uniform schedules, and the public action-conditioned models expose only a chunk-level frame rate. For the controls the reviewer suggested: a common backbone under identical non-uniform schedules is the standard fine-tuned DreamDojo-2B in GR-1, trained on exactly our keyframe schedules with the backbone's native temporal input; timestamp conditioning within the same model is covered by Table 3(a), where removing frame-wise time changes PSNR by 0.28 dB. Each baseline receives the keyframe-aligned action of every output and the most accurate temporal signal its interface admits, the average frame rate; in Table 10, where this signal is exact, the margins persist.
For depth, we restructure Table 1 into two blocks. The first is a common protocol for all RGB-based models, in which DA3 is applied to the generated frames. This is our primary cross-model depth comparison: HORAE-WM reaches 263.8 mm and 0.2242 AbsRel, against 300.3 mm and 0.2748 for the strongest baselines. The second block reports native metric depth for HORAE-WM only (83.4 mm, 0.0864), a capability that RGB-only models do not provide.
Budget K includes the initial frame, so K = 81, 41, 21 and 11 correspond to 80, 40, 20 and 10 predicted frames. We report event and hard-event coverage, enrichment, CH and the reconstruction error of selected keyframes at each budget (GR-2a), with per-task results in the appendix. Hard-event coverage shows the clearest tight-budget advantage: at K = 11 our schedule covers 66.07% of hard events, against 61.55% for uniform sampling. The margin in hard-event coverage is larger at each tighter budget than at K = 81 (+1.0, +1.5, +1.9 and +4.5 points at K = 81, 41, 21 and 11). At K = 81 the budget is generous for these segments, which average about 730 frames (about 24 s), so uniform sampling already falls within nearly every event. Hard events, as defined in Appendix B, include the brief but consequential subactions and reversals the reviewer refers to. Predicted video is evaluated at all four budgets (GR-2b), where ours is best on PSNR, SSIM and LPIPS over all frames and over hard-event frames. Any one of the three controls suggested by the reviewer addresses this point; we provide the third (GR-1) and the first (Table 3a). A piecewise variant with local intervals would change only the frame-rate signal for the baselines, whose effect is bounded by Table 3(a) and Table 10.
NkDT-3 · Scope, downstream utility, caption metric, action errors and presentation
Scope (Weakness 6). Our evidence concerns episode-level generalization within 16 dual-arm tasks, in simulation and in the real world, with a fixed base and a stable head camera. Segments with body motion or camera instability are filtered (Appendix A.1). We will qualify the claims in the abstract and in Sections 1 and 6, and list unseen tasks, object categories, embodiments, camera configurations and mobile manipulation as future work.
Downstream utility (Weakness 5, Question 6). Our study concerns predictive fidelity under a fixed output budget. Candidate-action ranking, planning and closed-loop evaluation are the strongest tests of downstream value, as the reviewer notes, and we list them as future work for this dual-arm platform.
Caption metric (Weakness 8). Caption completeness is a supplementary metric, and its generation and scoring prompts are documented in Appendix B. The primary sampling evidence is deterministic: event coverage and enrichment are computed directly from human annotations of the full RGB videos, with no model in the loop, and the GT-level reconstruction metrics in GR-2a and GR-2b are likewise deterministic.
Sensitivity to action errors (Question 7). The 16-D model input consists of commanded joint and gripper actions, and the model is conditioned on the commanded actions of the selected keyframes. All methods, including the baselines, are evaluated open-loop against the executed video. Robustness to the discrepancy between commanded and realized states during contact is important for deployment, and we list it as future work.
Presentation. We have revised the manuscript as follows: (1) an inference overview at the start of Section 3 that states what the model receives (initial RGBD observation, keyframe-aligned actions, physical times) and what it outputs; (2) Algorithm 1 for the sampler, with all hyperparameters; (3) the training schedule and the baseline-adaptation protocol moved from the appendix into the main text; (4) a table listing every ablation configuration with its total number of updates; and (5) unified terminology (keyframe, target time, condition video), with the condition video defined at first use.
good response
sorry to see that most of your review, include mine, are ai generated.
And your feedback is good enough. But again, when we talk about the paper quality for acceptance, attitude is important but not the most important part.
I think a weak reject is the proper score for this paper for two reason, first , i do not like this contribution: "HORAE-WM is built on top of the standard RGB training of DreamDojo. Its contributions are the formulation (Section 1), frame-wise physical-time conditioning, and the dual-axis progressive learning, in which the condition video and metric depth are added to the RGB objective as additional representations of the scene. " since this is really boring contribution for pure robotic community, and i can just find this incremental result by using agentic recursive learning.
Second, this work's contribution is very incremental and i really do not like cosmos or vision based world model setting. Since this is not the right way for doing robotic.
Depth is good and important, but this contribution is limited, since this is not the first paper to purpose depth.
Thus, i remain my score. But your feed back is comprihensive enough. Thus, if other people raise their score, I will also raise my score. Because i think this paper is really bordline and either accept or reject should be fine.
NkDT-1 · Same-backbone control and the role of the training scheme
We provide the same-backbone control requested (GR-1). The pretrained DreamDojo-2B reaches 13.848 dB PSNR without training. Standard fine-tuning on our keyframe training set, with the same initialization, resolution (400×640), 16-D action inputs, optimizer, batch size and number of epochs as the keyframe stage of HORAE-WM, reaches 18.087 dB PSNR, 0.632 Arm IoU and 288.6 mm DA3 depth error. This fine-tuning is RGB-only and uses the backbone's native temporal interface and loss. HORAE-WM reaches 19.000 dB, 0.678 Arm IoU and 263.8 mm, and leads on all six metrics. Most of the gain over the pretrained backbone thus comes from in-domain adaptation, and HORAE-WM adds a further +0.91 dB PSNR, +0.046 Arm IoU and −24.8 mm DA3 depth error.
HORAE-WM is built on top of the standard RGB training of DreamDojo. Its contributions are the formulation (Section 1), frame-wise physical-time conditioning, and the dual-axis progressive learning, in which the condition video and metric depth are added to the RGB objective as additional representations of the scene. The schedule extends the RGB training by adding these streams in stages. To check the schedule against the amount of training, we continue the direct tri-modal baseline to 4 and 6 epochs (GR-3), matching the curriculum in compute (12 stream-epochs) and in total updates (2,328). At equal updates the curriculum is better on all six metrics (17.388 vs. 17.238 dB PSNR; 0.2383 vs. 0.2396 LPIPS), using two thirds of the compute. At the same 1,552 updates, direct tri-modal training outperforms C→C+R, which has no depth stage, on PSNR, SSIM, LPIPS, Arm IoU and depth L1, which shows the value of the depth modality itself.
Mixed fixed-interval pretraining changes the data sampling, not the learning algorithm. The gain it brings is the gain from the additional sampled sequences under the same algorithm, rather than an algorithmic improvement. It re-samples the same 7,669 training episodes at four frame rates, adds no episodes, and provides the uniform-rate interface evaluated in Table 10. Table 3(b) reports its effect with the keyframe stage held at the same schedule: Arm IoU rises from 0.589 to 0.637 and PSNR from 17.30 to 18.12 dB.