- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates the limitations of scaling egocentric human video for action-conditioned world models, training Cosmos 3 variants on up to 30,000 hours of data. The authors separately evaluate agent and object-interaction fidelity, finding that agent fidelity saturates earlier while object fidelity remains substantially lower. They introduce skeleton-based conditioning and object-centric adaptive noise scheduling to improve prediction accuracy, and validate their approach through large-scale experiments and downstream humanoid video prediction.
Critical strengths
1.The paper provides a systematic study of large-scale egocentric world models, revealing an important gap between agent fidelity and object-interaction fidelity. 2. The proposed object-centric adaptive noise scheduling improves object prediction fidelity, supported by controlled ablations and downstream humanoid evaluations.
Critical weaknesses
-
The estimated fidelity ceilings rely on extrapolation from a limited scaling range, while data volume and training compute increase together, making the conclusions about fundamental limits less convincing.
-
Humanoid evaluations focus on short open-loop predictions, with success agreement measured on a small set of selected interaction windows rather than long-horizon closed-loop execution.
Questions for authors
-
Can the authors provide compute-matched scaling experiments to validate the robustness of the estimated fidelity ceilings?
-
Can the authors evaluate longer-horizon predictions or closed-loop robot interactions to demonstrate the practical implications of improved object fidelity?
-
Can the authors separately ablate skeleton conditioning and object-centric noise scheduling in the downstream humanoid experiments?
Response to Reviewer 7MJe
Thank you for the thoughtful and constructive feedback. We appreciate your recognition of the importance of our finding that agent and object-interaction fidelity diverge, and of the effectiveness of our adaptive noise scheduling approach. We respond to your comments below.
Compute-matched scaling experiments
Thank you for pointing this out. Note, however, that, because the smaller budgets receive proportionally fewer optimization steps, they are further from convergence than the larger ones, and training each to convergence would raise the left of the curve more than the right and therefore lower the fitted asymptote. The ceilings we report are in this sense optimistic. To validate this, we have trained the 30k and 10k hours data budgets at matched iteration count using the Nano + Skeleton variant of the model and report the results below.
At matched compute, the 10k-hour budget reaches 0.522 object SCS against 0.513 under a single pass, while the 30k-hour run reaches 0.527; agent SCS is essentially unchanged, so at equal compute the additional data brings nothing on the agent axis, as expected, with the agent already close to its limit at 10k hours. Read against the single-pass curve, this means that roughly two thirds of the apparent 10k→30k object gain (+0.014) is recovered by optimizing longer on the same 10k hours (+0.009), leaving about +0.005 attributable to the additional data. Because the smaller budgets are the ones furthest from convergence, the correction is largest on the left of the curve, which steepens it and lowers the fitted asymptote.The qualitative conclusion is therefore unaffected, and the gap to the agent asymptote is if anything wider than reported. We will finish the entire compute-matched curve and include it in the camera-ready version of the paper.
Effect of adaptive noise schedule in humanoid experiments
Following your suggestion, we have trained a variant with the same human pre-training and humanoid fine-tuning, without the object-centric objective (denoted as SKEL) and report the results on the 11 tasks of Table 7 below.
The objective’s contribution on humanoids is comparable in absolute terms to what it contributes on ego-centric human video at 30k hours. In relative terms the gain is smaller, reflecting the higher baseline: the EgoVLA objects are large, rigid and rarely occluded, and object SCS is already comparable to agent SCS on the human benchmark, so there is less for the objective to recover. Nonetheless, it is effective in both scenarios.
Practical implications of improved object fidelity
Following your suggestion we have evaluated closed-loop execution: we roll the model out autoregressively across all 66 test episodes, up to 28.8 s, with each 1.6 s chunk conditioned on the model’s own last predicted frame. The evaluation therefore spans complete episodes rather than the curated interaction windows used for success agreement.
First, we observe that absolute fidelity degrades substantially in closed loop for both variants: object SCS falls from 0.790 under short-horizon open-loop prediction to 0.611 over the full episode and 0.494 beyond 11.2 s. Second, the comparison is about the rate of that degradation, and there the advantage from improved object fidelity grows with rollout length from +0.014 over the whole episode to +0.033 beyond 11.2 s. Under closed-loop execution each prediction becomes the next step’s input, so object errors compound, and the component that reduces them matters more the longer the rollout runs. This is the setting in which improved object fidelity has its clearest practical consequence.