acceptodds
Under review as a conference paper at ICLR 2027

What 30,000 Hours of Ego-centric Video Does Not Teach

10 views

Abstract

World models offer a promising alternative to physics-based simulators, yet remain far from practical deployment. We ask how far scaling ego-centric human video takes them, using a dataset of 30,000 hours spanning over 1,000 scene types and 14,000 contributors. Rather than relying on opaque downstream metrics, we directly evaluate agent and object-interaction fidelity on a challenging out-of-distribution benchmark. Increasing training data by 100× improves both, but unevenly: the agent is modeled well, while object fidelity remains far lower and improves slowly. We show that the agent gains need not come from data, and a careful visual conditioning design saturates fidelity with a fraction of it, which lets us measure object fidelity on its own and discover its saturation point. We then introduce a supervision scheme that shifts capacity from scene appearance toward object dynamics, improving object fidelity though a substantial gap remains. Finally, our conclusions transfer to downstream humanoid modeling. Overall, our results suggest that scaling ego-centric data brings agent modeling close to its limit while leaving its effects on the world far behind, and that closing this gap will depend on how models are trained, not only on how much data they see.

Primary area:
generative models
Keywords:
world models, video generation, egocentric video

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7MJeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the limitations of scaling egocentric human video for action-conditioned world models, training Cosmos 3 variants on up to 30,000 hours of data. The authors separately evaluate agent and object-interaction fidelity, finding that agent fidelity saturates earlier while object fidelity remains substantially lower. They introduce skeleton-based conditioning and object-centric adaptive noise scheduling to improve prediction accuracy, and validate their approach through large-scale experiments and downstream humanoid video prediction.

Critical strengths

1.The paper provides a systematic study of large-scale egocentric world models, revealing an important gap between agent fidelity and object-interaction fidelity. 2. The proposed object-centric adaptive noise scheduling improves object prediction fidelity, supported by controlled ablations and downstream humanoid evaluations.

Critical weaknesses
  1. The estimated fidelity ceilings rely on extrapolation from a limited scaling range, while data volume and training compute increase together, making the conclusions about fundamental limits less convincing.

  2. Humanoid evaluations focus on short open-loop predictions, with success agreement measured on a small set of selected interaction windows rather than long-horizon closed-loop execution.

Questions for authors
  1. Can the authors provide compute-matched scaling experiments to validate the robustness of the estimated fidelity ceilings?

  2. Can the authors evaluate longer-horizon predictions or closed-loop robot interactions to demonstrate the practical implications of improved object fidelity?

  3. Can the authors separately ablate skeleton conditioning and object-centric noise scheduling in the downstream humanoid experiments?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 7MJe

Thank you for the thoughtful and constructive feedback. We appreciate your recognition of the importance of our finding that agent and object-interaction fidelity diverge, and of the effectiveness of our adaptive noise scheduling approach. We respond to your comments below.

Compute-matched scaling experiments

Thank you for pointing this out. Note, however, that, because the smaller budgets receive proportionally fewer optimization steps, they are further from convergence than the larger ones, and training each to convergence would raise the left of the curve more than the right and therefore lower the fitted asymptote. The ceilings we report are in this sense optimistic. To validate this, we have trained the 30k and 10k hours data budgets at matched iteration count using the Nano + Skeleton variant of the model and report the results below.

Training scheduleUpdatesAgent SCSObject SCS
10k hours, three epoch65,4540.7860.522
30k hours, one epoch65,4540.7850.527

At matched compute, the 10k-hour budget reaches 0.522 object SCS against 0.513 under a single pass, while the 30k-hour run reaches 0.527; agent SCS is essentially unchanged, so at equal compute the additional data brings nothing on the agent axis, as expected, with the agent already close to its limit at 10k hours. Read against the single-pass curve, this means that roughly two thirds of the apparent 10k→30k object gain (+0.014) is recovered by optimizing longer on the same 10k hours (+0.009), leaving about +0.005 attributable to the additional data. Because the smaller budgets are the ones furthest from convergence, the correction is largest on the left of the curve, which steepens it and lowers the fitted asymptote.The qualitative conclusion is therefore unaffected, and the gap to the agent asymptote is if anything wider than reported. We will finish the entire compute-matched curve and include it in the camera-ready version of the paper.

Effect of adaptive noise schedule in humanoid experiments

Following your suggestion, we have trained a variant with the same human pre-training and humanoid fine-tuning, without the object-centric objective (denoted as SKEL) and report the results on the 11 tasks of Table 7 below.

MetricSKELOURSDifference
Object SCS0.7720.79+0.018

The objective’s contribution on humanoids is comparable in absolute terms to what it contributes on ego-centric human video at 30k hours. In relative terms the gain is smaller, reflecting the higher baseline: the EgoVLA objects are large, rigid and rarely occluded, and object SCS is already comparable to agent SCS on the human benchmark, so there is less for the objective to recover. Nonetheless, it is effective in both scenarios.

Practical implications of improved object fidelity

Following your suggestion we have evaluated closed-loop execution: we roll the model out autoregressively across all 66 test episodes, up to 28.8 s, with each 1.6 s chunk conditioned on the model’s own last predicted frame. The evaluation therefore spans complete episodes rather than the curated interaction windows used for success agreement.

Object SCSSKELOURSDifference
Whole episode (28.8 s)0.5970.611+0.014
Beyond 11.2 s0.4610.494+0.033

First, we observe that absolute fidelity degrades substantially in closed loop for both variants: object SCS falls from 0.790 under short-horizon open-loop prediction to 0.611 over the full episode and 0.494 beyond 11.2 s. Second, the comparison is about the rate of that degradation, and there the advantage from improved object fidelity grows with rollout length from +0.014 over the whole episode to +0.033 beyond 11.2 s. Under closed-loop execution each prediction becomes the next step’s input, so object errors compound, and the component that reduces them matters more the longer the rollout runs. This is the setting in which improved object fidelity has its clearest practical consequence.

Reviewer_KWZDOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the impact of scaling dataset size on agent and object-interaction fidelity in ego-centric human video generation. The paper finds that while increasing the data size improves both, agent fidelity converges well reaching 86% its measurable range, while object fidelity remains only at 63% and improves slower. By introducing additional conditioning on agent's arms skeletons the authors improve convergence speed with the same ceiling. To raise the ceiling of object-interaction fidelity the authors suggest shifting the supervision signal by moving the noise schedule in automatically detected high dynamic regions. Additionally they demonstrate that their method transfers well to humanoid manipulation.

Critical strengths
  1. The paper takes a systematic approach to studying the scaling behavior of models trained on ego-centric human data. While the findings are intuitive and the behavior is known in the video generation domain, this is to my knowledge the first documented evidence of this type.
  2. The paper is well-written and easy to follow. The paper is equipped with a detailed appendix.
Critical weaknesses
  1. The paper's claim is quite broad, while only one model family is studied. Moreover, the outcome is not decoupled from the initialization with Cosmos 3's weights. Multiple model families and directly comparable model sizes would strengthen the paper.
  2. The paper focuses on studying the agent and object fidelities as functions of data size while keeping the dataset as diverse as possible. This leaves the data type out of focus. E.g. how does object fidelity change when the model is exposed to clearer object manipulation data?
Questions for authors
  • Have the authors tried looking into the data type? E.g. what portion of the time does the actual object manipulation span in the data? Can the object fidelity ceiling be moved by shifting the supervision on the data level instead of the loss level? E.g. by exposing the model to more data with clearer manipulation.
  • Do the findings persist across model families and capacities?
4 comments
AuthorsOpenReview ↗

Response to Reviewer KWZD

Thank you for the thoughtful and constructive feedback. We appreciate your recognition of our systematic approach and the novelty of documenting scaling behavior in models trained on egocentric human data. We respond to your comments below.

Generalization across model families and capacities.

As noted in Section 3.1, the two variants in the submission differ in design and pre-training as well as in size, so the difference between them cannot be read as a pure capacity effect. To separate capacity cleanly, and to address your point directly, we have added two models. Cosmos 3 Super (32B trainable against Nano’s 8B) isolates capacity, holding the architecture and pre-training recipe fixed. Wan2.2 (5B trainable) comes from a different family and is initialized from its own pre-trained weights, learned on a different corpus with a different recipe, which also addresses the concern that our findings could be inherited from Cosmos 3’s initialization. Both are trained with skeleton conditioning across the full data ladder, and their SCS curves are added to Figure 4 of the revised manuscript.

Quadrupling trainable parameters leaves agent fidelity essentially unchanged and raises the fitted object asymptote by 0.034. Read one way this supports the scaling-centric view: capacity does move the object ceiling where additional data largely doesn’t. Read quantitatively it does not. At 0.034 per quadrupling, closing the remaining gap to the agent ceiling of 0.805 would take roughly six further quadruplings: more than three orders of magnitude in parameters, with training compute rising in proportion. Since the per-quadrupling gain should be expected to diminish rather than hold, this is a lower bound on the scale required.

Wan2.2’s absolute scores are closer to Cosmos 3 Edge than to Nano, so the level of interaction fidelity is family-dependent. We do not attribute this to a single factor, since the two models differ in architecture, pre-training and size together. The agent–object ceiling gap, however, barely changes with model family: 0.269 for Wan2.2, against 0.239 for Cosmos 3 Edge and 0.235 for Cosmos 3 Nano, with agent fidelity saturating first in both. Our findings are therefore not specific to the Cosmos 3 family or to its initialization.

Effect of the data quality.

Thank you for this insightful suggestion. Our ego-centric dataset is already curated to emphasize human-object interactions. To quantify this, we randomly sampled 200 of the 17-frame training windows the model actually sees and annotated each manually, identifying the manipulated object from the activity description and recording whether the agent interacts with it and whether it is visible. 91% of sampled windows depict active object manipulation, the remaining 9% being navigation, repositioning and similar activity, and the object is visible in 90%. Hence, further shifting the supervision towards clear, manipulation-centric clips is unlikely to change the results.

We have also experimented with decreasing diversity of the training set. To this end, we have trained the Nano-Skeleton variant on a 10k hour budget while keeping only the most common environments and scenes (i.e. dropping the long tail) and report the results below. Decreasing data diversity doesn’t meaningfully improve object performance and does not reduce the agent-object fidelity gap.

Training budgetDiversityEnvironmentsScenesObject SCS ↑
10,000 hDefault1021,0810.513
10,000 hReduced262610.515

We agree that other properties of the training data, including object size, and object category distribution, could affect the observed object fidelity ceiling. Systematically studying these factors would be a valuable extension of our analysis, which focuses on scaling with data size, and we leave this investigation to future work.

Reviewer_KWZDOpenReview ↗

Response to the Rebuttal

Thank you for the detailed response. Observing the same gap in a different model family is valuable, and the level of fidelity looks consistent with the model's size. Regarding the data, I would suggest to be explicit and include these details in the paper. This would allow readers to better understand the limitations of the study.

Additional question: Both plots in Figure 4 start at 300 hours of training exposure. Could you provide more points closer to the beginning of the fine-tuning stage? This would allow to better understand what part of the gap is already present in the pretrained models.

Thank you for the follow-up and for the constructive suggestions.

Data details in the paper. We agree, and have added these details to the revised manuscript. The dataset description (Section 3.2, with details in the new Appendix A.5) now reports that, of 200 randomly sampled training windows, 91% depict active object manipulation and the manipulated object is visible in 90%. The reduced-diversity experiment is reported in the new Appendix C.3. The limitations (Section 7) now state that we do not vary other properties of the training data, such as object size and the distribution of object categories, which may affect the object fidelity ceiling we observe.

Earlier points in Figure 4. Thank you for this suggestion. We evaluated two earlier checkpoints of the training runs in Figure 4, at 30 and 100 hours of training exposure, on the same evaluation clips and with the same protocol. Results for the skeleton-conditioned Cosmos 3 Edge, Nano and Super, and Wan2.2 are below, as agent SCS / object SCS; the gap is agent SCS minus object SCS.

Training exposureEdge agent / objectEdge gapNano agent / objectNano gapSuper agent / objectSuper gapWan2.2 agent / objectWan2.2 gap
30 h0.633 / 0.3350.2990.598 / 0.3270.2710.505 / 0.2900.2150.505 / 0.2850.220
100 h0.684 / 0.3680.3160.688 / 0.3840.3040.643 / 0.3710.2720.657 / 0.3470.310
300 h0.717 / 0.4090.3080.723 / 0.4370.2870.712 / 0.4370.2750.700 / 0.3740.326

The gap is present essentially from the start of fine-tuning. At 30 hours it is already 0.21–0.30, and by 100 hours 0.27–0.32, within 0.05 of where it remains after two and a half further orders of magnitude of data. This sharpens our main finding: the asymmetry is not objects lagging and catching up later, but a difference that is in place almost immediately and that additional ego-centric video does not close.

An important caveat is that the action projection is re-initialized from scratch and the skeleton projector is zero-initialized, so at 30 hours the conditioning interface is itself still being learned. Part of the agent score at that point reflects how quickly the model learns to read the action input rather than what it has learned about dynamics. The compute-matched runs we are completing for the camera-ready version address this directly, since every budget is optimized till convergence. We will report these early budgets in that setting.

Reviewer_KWZDOpenReview ↗

Response to the Comment by the Authors

Thank you for the additional clarifications. The fact that the gap is present from the very beginning slightly undermines the findings, in my view, as it hints that the problem might be in the pretraining. Nevertheless, I remain positive about the paper and therefore keep my original rating.

Reviewer_3HznOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether scaling egocentric human video leads to accurate modeling of agent motion and its effects on manipulated objects. The authors train Cosmos 3 variants on 300 to 30,000 nominal hours and evaluate agent and object fidelity separately. Agent fidelity improves quickly and saturates early, while object fidelity remains substantially lower. The paper further introduces object-centric adaptive noise scheduling to improve interaction fidelity and evaluates transfer to humanoid manipulation.

Critical strengths
  1. An important and timely question, studied at scale. Separating agent and object fidelity, rather than relying only on perceptual or downstream metrics, reveals a failure that aggregate metrics hide: plausible hand motion with wrong object outcomes.
  2. Useful empirical findings. Skeleton conditioning substantially reduces the data needed for high agent fidelity, and the dynamic-region supervision improves object SCS (0.527 -> 0.546 at 30K) with unchanged LPIPS.
Critical weaknesses
  1. The object-fidelity ceiling is not sufficiently established. The asymptote is estimated from five checkpoints using one functional form, without uncertainty estimates or sensitivity analysis, and object fidelity continues to improve at the largest observed budget. Although Section 7 acknowledges that data exposure and optimization steps increase together, the stronger ceiling claims elsewhere remain insufficiently supported. Moreover, the unconditioned baseline's object asymptote remains undetermined, and skeleton conditioning changes object prediction as well as agent rendering. The results establish diminishing returns under the tested training recipe more convincingly than a general limit of egocentric data scaling.

  2. The evaluation leaves important ambiguities in the interpretation of the interaction gap and transfer gains. Future camera and hand trajectories specify agent motion more directly than object responses, while mask IoU is sensitive to object size, visibility, and occlusion, so the raw hand-object SCS gap does not by itself isolate a difference in learned dynamics. In the humanoid experiment, skeleton conditioning and dynamic-region supervision are introduced together, leaving their individual contributions unclear. Success agreement is measured on only 42 windows with one evaluator, and the reference success rate of 0.86 exceeds the model's 0.81 agreement; the reported recalls imply approximately 0.75 balanced accuracy, indicating predictive value that should be assessed with paired uncertainty estimates. Finally, checkpoint selection uses the reported evaluation clips, and the stated maximum 0.005 difference from fixed-iteration results appears inconsistent with Tables 6-7.

Questions for authors
  1. How robust are the fitted asymptotes to alternative functional forms and leaving out individual budget points? Please provide uncertainty intervals and clarify whether the five seeds in Table 1 are independent training runs. Can the ceiling claim be supported beyond the observed joint data/compute regime?
  2. Does the hand-object gap persist after controlling for mask area, visibility, and occlusion, or under an independent measure of object motion or interaction outcomes? These checks would help distinguish dynamics errors from differences in metric sensitivity.
  3. Can the humanoid gains be separated using a +Human + Skeleton baseline? Please provide paired uncertainty estimates for class-balanced success prediction, clarify checkpoint selection, and reconcile the object SCS values in Tables 2/7 with Table 6 and the stated 0.005 bound.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 3Hzn (1/4)

Thank you for the thoughtful and detailed feedback, which helped improve the paper. We are encouraged by your positive assessment of the importance and timeliness of our study and the value of our empirical findings. We respond to your comments below.

Asymptote robustness

Thank you for suggesting these additional controls. Firstly, we would like to clarify that, although we only plot the 5 data budgets in Figure 4, the curves are fit to 18 checkpoints (the five budgets and 13 intermediate) for improved robustness. We have additionally fit two alternative functional forms (a stretched exponential and a logistic in log training hours), and run a leave-one-out analysis and a paired bootstrap for the Nano + Skeleton variant, the configuration the ceiling claim rests on. As noted in Section 4 we do not fit an asymptote to the unconditioned object curve, which is not yet in its saturating regime.

The robust quantity is the gap rather than the level: across all three functional forms, every leave-one-out refit, and the full span of the bootstrap intervals, the object ceiling lies at least 0.21 below the agent’s. The object asymptote itself falls between 0.539 and 0.565 across specifications, a spread of 0.026, which is an order of magnitude smaller than the gap it would need to explain away, and both alternative forms place it below the 0.565 we report, so the choice of functional form moves the estimate in the conservative direction. Leaving out any single checkpoint moves the power-law estimate by at most 0.007.

Fitting functionObject SCS ceilingObject−hand ceiling gap
Power law0.5650 [0.5484, 0.5869]−0.2348 [−0.2512, −0.2131]
Stretched exponential0.5390 [0.5273, 0.5518]−0.2523 [−0.2637, −0.2398]
Logistic in log hours0.5561 [0.5418, 0.5732]−0.2424 [−0.2564, −0.2257]

Leave-one-out results are below. Ranges are the minimum and maximum across 18 refits, each omitting one checkpoint and retaining the other 17.

Fitting functionObject ceiling rangeObject−hand gap range
Power law[0.5618, 0.5720][−0.2376, −0.2257]
Stretched exponential[0.5378, 0.5466][−0.2537, −0.2453]
Logistic in log hours[0.5537, 0.5639][−0.2444, −0.2330]

On object fidelity still improving at the largest budget: it does, and that is what a saturating curve predicts. What identifies the ceiling is the deceleration, which Section 4 reports: gains of 0.056 between 300 and 3k hours against 0.035 between 3k and 30k, with 93% of the fitted asymptote already realized at 30k. We address the related point that data and optimization compute increase together under Compute-matched scaling below.

Clarification for Table 1

Thank you for pointing out this missing detail. The seeds in Table 1 are different input noise samples, not independent training runs. To verify that this does not understate variability, we have additionally trained the Nano + Skeleton configuration at 10k hours three times with different random seeds for parameter initialization and data ordering, evaluating each run identically.

Training budgetAgent SCSObject SCS
10k hours0.779 ± 0.0010.512 ± 0.001

Run-to-run variation is smaller than the inference-seed variation already reported in Table 1 (±0.002 agent, ±0.004 object), and the means agree with it to within 0.001. The intervals in Table 1 are therefore not optimistic. For context, this spread is nearly twenty times smaller than the smallest effect we claim (+0.019 object SCS for our objective over the skeleton baseline).

AuthorsOpenReview ↗

Response to Reviewer 3Hzn (2/4)

Compute-matched scaling

Note that, because the smaller budgets receive proportionally fewer optimization steps, they are further from convergence than the larger ones, and training each to convergence would raise the left of the curve more than the right and therefore lower the fitted asymptote. The ceilings we report are in this sense optimistic. To validate this, we have trained the 30k and 10k hours data budgets at matched iteration count using the Nano + Skeleton variant of the model and report the results below.

Training scheduleUpdatesAgent SCSObject SCS
10k hours, three epoch65,4540.7860.522
30k hours, one epoch65,4540.7850.527

At matched compute, the 10k-hour budget reaches 0.522 object SCS against 0.513 under a single pass, while the 30k-hour run reaches 0.527; agent SCS is essentially unchanged, so at equal compute the additional data brings nothing on the agent axis, as expected, with the agent already close to its limit at 10k hours. Read against the single-pass curve, this means that roughly two thirds of the apparent 10k→30k object gain (+0.014) is recovered by optimizing longer on the same 10k hours (+0.009), leaving about +0.005 attributable to the additional data. Because the smaller budgets are the ones furthest from convergence, the correction is largest on the left of the curve, which steepens it and lowers the fitted asymptote. The qualitative conclusion is therefore unaffected, and the gap to the agent asymptote is if anything wider than reported. We will finish the entire compute-matched curve and include it in the camera-ready version of the paper.

Scope of the ceiling claim

Section 7 scopes the claim on the data, compute and collection axes. We note that our key observation of the fidelity gap between the agent and the object is not specific to a single configuration: it holds across two conditioning designs, across the objective (Section 5 raises the object ceiling without closing the gap), across model scale, and across backbones differing in pretraining and architecture, including, in this rebuttal, Wan2.2 (see updated Figure 4 in the manuscript and our response to Reviewer KWZD). The object ceiling itself is established under skeleton conditioning, where the curve is in its saturating regime; under standard conditioning it remains unidentifiable within our range, which is why we use conditioning as an instrument rather than reporting an asymptote there. The compute confound is also partially addressed above, and correcting it lowers rather than raises the fitted ceiling. What remains fixed is the prediction horizon, which we have added to the limitations.

AuthorsOpenReview ↗

Response to Reviewer 3Hzn (3/4)

Controlled hand-object gap evaluation

Thank you for these suggestions. We agree the two axes are not symmetric in what the conditioning specifies, and we do not read the gap as an information-matched comparison. The hand curve serves as a reference for what this model class attains, under identical training data, schedule and metric, on a quantity the conditioning does specify directly; the finding is that object fidelity converges far below it. The paper already reports the comparison in a form that removes the per-category component of metric sensitivity: normalized by the maximum the metric can register for each category, the agent reaches 86% of its measurable range and objects 63%. To quantify how much of the remaining difference is attributable to mask geometry rather than to prediction error, we have run the controls you ask for, on the 30k-hour Nano + Skeleton checkpoint.

We split clips by ground-truth object mask area (threshold 9,216 px, i.e. 96×96) and by occlusion, measured as the fraction of a target’s visible-mask convex hull covered by other annotated masks outside the target itself; a clip counts as highly occluded when this exceeds 25% for the same target in at least 4 of 16 future frames. The results are shown below.

SubsetObject SCS @30kObject ceilingObject, % of achievableHand ceilingGap
All0.5270.56563%0.8−0.235
Larger objects0.6960.71180%0.789−0.078
Smaller objects0.4690.52558%0.799−0.274
Low occlusion0.5780.61768%0.8−0.183
High occlusion0.4160.45250%0.799−0.347

The gap is present in every stratum, and its size tracks the difficulty of the interaction. It is largest exactly where an object’s response is least inferable (−0.274 on smaller objects and −0.347 under substantial occlusion, against −0.235 overall), and narrows where the interaction is easiest to observe, to −0.183 on low-occlusion clips and −0.078 on larger objects. These controls therefore localize the gap rather than remove it: mask geometry accounts for part of its magnitude, but the difference concentrates in precisely the conditions (small, occluded objects at the moment of contact) that our account of the failure predicts. And the centroid measure below shows the ordering also holds under a metric that does not depend on mask overlap at all.

To measure the difference under a metric that involves neither segmentation overlap nor mask area, we are taking the centroid of each mask as the mean 2D position of its pixels, and compare predicted and ground-truth centroid displacement from the conditioning frame over the 16 future frames, reporting mean Euclidean error in pixels below.

MeasureHandObjectGap
Centroid displacement error (px)8.621.713.1
Fitted floor (px)8.521.212.7

Object centroid error is roughly 2.5× the hand’s and converges to a floor with a similar gap, reproducing the ordering without any dependence on mask overlap.

Effect of adaptive noise schedule in humanoid experiments

Following your suggestion, we have trained a variant with the same human pre-training and humanoid fine-tuning, without the object-centric objective (denoted as SKEL) and report the results on the 11 tasks of Table 7 below.

MetricOURSSKELDifference [95% CI]
Object SCS0.790.772+0.018 [+0.004, +0.034]

The objective’s contribution on humanoids is comparable in absolute terms to what it contributes on ego-centric human video at 30k hours. In relative terms the gain is smaller, reflecting the higher baseline: the EgoVLA objects are large, rigid and rarely occluded, and object SCS is already comparable to agent SCS on the human benchmark, so there is less for the objective to recover. Nonetheless, it is effective in both scenarios.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.