acceptodds
Under review as a conference paper at ICLR 2027

EgoForesight: Effectively Scaling Policy Learning with Noisy Egocentric Actions

Abstract

Scaling policy learning requires broad experience and reliable supervision. Robot demonstrations provide high-quality trajectory supervision but are costly to collect. Egocentric videos broaden task and environment coverage, yet motion reconstruction and retargeting introduce noisy action labels. To address this challenge, we introduce EgoForesight, a vision-language-action framework that learns jointly from both sources by reducing the influence of unreliable action labels and exploiting demonstrated outcomes as complementary supervision. A joint vision-action flow couples future visual representation prediction with action generation, unifying reliability-aware action learning with outcome-informed representation transfer. Specifically, reliability-conditioned corruption allocates action supervision across noise levels according to source reliability, while reliability-weighted regression regulates each source's contribution. Complementing this treatment of noisy labels, privileged-future representation distillation draws supervision from demonstrated outcomes. An exponential-moving-average teacher observes the current scene and clean future to encode action-relevant structure from the resulting scene change. Its deeper representations supervise an earlier student layer receiving a corrupted future, providing outcome-informed structure beyond noisy action annotations. Together, these mechanisms support effective scaling with noisy egocentric actions. Over the tested range, EgoForesight exhibits steeper scaling slopes than the baseline, reducing real-world open-loop action MSE by 20.5% and improving RoboCasa few-shot success by 8.1 percentage points at 30,000 hours of pretraining data. After post-training, it also achieves strong absolute performance in both simulated and real-world dexterous manipulation.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Model, Ego-centric Data, Dexterous Manipulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_FUMSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EgoForesight, a framework for scaling robot policy learning with noisy action annotations from egocentric human videos. It combines reliability-conditioned action corruption with privileged-future representation distillation in a joint visual-feature and action flow model. Experiments on RoboCasa and real-world dexterous manipulation and ablation settings support the practical benefits of the combined approach.

Critical strengths
  1. The paper is well-motivated. It addresses the important problem of learning from noisy egocentric data. Reliability-conditioned corruption and privileged-future distillation provide complementary ways to exploit broader human experience despite imperfect action labels.
  2. Component ablations show gains from both designs. Matched scaling comparisons and evaluations in simulation and real-world dexterous manipulation further support the method’s practical value.
Critical weaknesses

Camera-related uncertainty in teacher supervision is not fully explained. Appendix C.4 compensates action labels using estimated camera poses, while future visual features still contain viewpoint changes. It remains unclear how the teacher distinguishes camera motion from manipulation outcomes across data sources, and how residual pose-estimation errors affect its action representations.

Questions for authors
  1. Could you clarify whether camera motion is handled explicitly in the privileged-future teacher or learned implicitly? In particular, how are viewpoint changes in future observations and residual pose-estimation errors in action inputs accounted for when constructing distillation targets across datasets?
  2. The method name closely resembles “Ego-Foresight: Self-supervised Learning of Agent-Aware Representations for Improved RL (https://arxiv.org/abs/2407.01570)” (Nunes et al., ICLR 2026). Would you consider adopting a distinct name to avoid confusion with this existing work?
1 comment

We sincerely thank Reviewer FUMS for your thoughtful review and insightful questions regarding camera motion handling and method naming. We have added action-to-image cross-attention visualizations in Appendix D.3 of the revised manuscript. Below, we address each of your comments and questions.

W1 & Q1: Handling Camera Motion, Viewpoint Changes, and Residual Pose-Estimation Errors in the Teacher

In large-scale, in-the-wild egocentric videos, geometric preprocessing (Appendix C.4) can only remove the dominant first-order camera-frame offset—residual camera-pose errors, hand-tracking drift, and retargeting mismatch are fundamentally unavoidable in the reconstructed labels a~\widetilde{a} (Appendix C.5). Rather than assuming preprocessing yields clean labels, EgoForesight is specifically designed to tolerate and overcome these unavoidable residual errors during training:

Signal / PathwayHandling ModeHow It Accounts for Camera Motion & Residual Errors
1. Action Label Construction (a~\widetilde{a})Explicit (First-order)Future end-effector poses are transformed into the current camera frame CtC_t via estimated camera poses (Eq. 27–28, Appendix C.4) to strip dominant ego-motion.
2. Residual Pose Errors in Teacher Action Input (xτaax_{\tau_a}^a)Explicit Noise SuppressionThe teacher does not take raw a~\widetilde{a} as input; it shares the student's corrupted input xτaa=(1−τa)a~+τaϵax_{\tau_a}^a = (1-\tau_a)\widetilde{a} + \tau_a\epsilon_a (Eq. 9). Under qd=0.2q_d = 0.2 (αd=2.5\alpha_d = 2.5), τa∼Beta⁡(αd+1,1)\tau_a \sim \operatorname{Beta}(\alpha_d+1, 1) is skewed toward τa→1\tau_a \to 1, reducing the expected weight on a~\widetilde{a} to E[1−τa]≈0.22\mathbb{E}[1-\tau_a] \approx 0.22 and contracting flow-field errors at rate O((1−τa)/τa3)\mathcal{O}((1-\tau_a)/\tau_a^3) (Appendix A.1, Eq. 21). Fine-grained residual pose errors are drowned out by τaϵa\tau_a\epsilon_a, forcing the teacher to infer hTℓth_T^{\ell_t} primarily from the visual transition (ot,ot+K)(o_t, o_{t+K}).
3. Viewpoint Changes in Future Vision (D(ot+K)D(o_{t+K}))Implicit (Data-driven)The teacher receives unwarped D(ot+K)D(o_{t+K}) (τv′=0\tau_v' = 0) alongside Cψ(ct)C_\psi(c_t) (oto_t), without an explicit visual ego-motion separation module. Because first-order camera compensation and S1 initialization ground the action tokens to encode hand–object state changes rather than camera shift, joint attention over oto_t and ot+Ko_{t+K} in DINOv3 feature space implicitly tracks the active hands and manipulated objects while ignoring background viewpoint shifts. To verify what the student learns from the teacher, Figure 11 (Appendix D.3) visualizes action-to-image cross-attention at layer 5 (the exact layer where the student aligns its action tokens to the teacher via LPFD\mathcal{L}_{\text{PFD}} in Eq. 10), confirming that the distilled layer attends sharply to the active hands and manipulated objects rather than background ego-motion.

Q2: Method Name Similarity to "Ego-Foresight"

Thank you for pointing this out. We named our method EgoForesight to directly reflect Egocentric human data and Foresight from future-aware supervision. Ego-Foresight (Nunes et al., ICLR 2026) investigates a different research domain (self-supervised agent-aware visual representation learning for RL), whereas our work focuses on scaling VLA policy pretraining with noisy egocentric human action annotations. Given the clear separation between the two domains and problem settings, we believe the likelihood of confusion in practice is small; moreover, renaming the method at this stage would require extensive changes across the manuscript, codebase, and pretrained checkpoints, so we prefer to keep EgoForesight.

Reviewer_Bb1hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes EgoForesight, a vision-language-action framework that learns from both robot data and egocentric videos. While egocentric videos provide the data coverage of broader tasks and diverse behaviors, the action annotations are usually noisy due to the motion construction and retargeting process. To reduce the influence of noisy action labels, this paper introduces reliability-conditioned correction. Then the joint vision-action flow is applied to predict future visual features and action chunks. On the other hand, to supply effective supervision during training, this approach performs privileged-future distillation in a teacher-student manner. Experiments are conducted on both the simulation benchmark and real-robot manipulation.

Critical strengths
  1. Leveraging egocentric videos to VLA training has potential for scaling up policy learning. While robot demonstrations contain direct and precise control signals, egocentric videos cover more diverse task and environment coverage, allowing robust policy models for real-world deployment.
  2. The experiments contain both the simulation benchmark (i.e., RoboCasa) and real-world robots (i.e, R1 Pro with Sharpa Wave hands), which is generally comprehensive.
  3. The overall paper is well-organized and easy to follow.
Critical weaknesses
  1. While the proposed method is reasonable, the novelty of this method is relatively insufficient. For example, the joint vision-action prediction, DINO feature as prediction targets, and MoT architecture that already exist in previous video-action models or world-action models (e.g., LDA-1B, InternVLA-A1). The noisy corruption and teacher-student distillation modules are similar to existing semi-supervised or self-supervised learning works (e.g., Unbiased Teacher [A]). It would be beneficial to highlight the method novelty to clarify the contributions of the proposed method, not just a combination of existing techniques.
  2. For the vision prediction, this paper applies the DINOv3 feature as the prediction target. The motivation behind the use of feature-space future prediction instead of in the pixel space are not clearly illustrated. Also, for feature-level prediction, other latent spaces like V-JEPA-2 features, etc are also feasible choices. What’s the advantage of using DINO features in the tasks this paper chooses?

[A] Unbiased Teacher for Semi-Supervised Object Detection. ICLR 2021

Questions for authors
  1. Is the proposed method model-agnostic? The current method is based on the Qwen3.5 4B backbone. Whether the proposed approach can be applied to different VLMs or different model sizes?
5 comments

We sincerely thank Reviewer Bb1h for your time and helpful suggestions on clarifying our technical contributions, visual target choice, and model generality. We have updated the revised manuscript with the suggested reference and additional experimental analyses in Appendix D.3 and Appendix G.2. Below, we address each of your comments and questions.

W1: Highlighting Method Novelty Beyond Existing Ingredients

While our base architecture adopts established building blocks—MoT, joint vision–action prediction, DINO targets, and EMA teacher–student learning (we have added Unbiased Teacher, Liu et al., ICLR 2021 in Appendix G.2)—a naive combination of these ingredients fails to scale on noisy egocentric human data:

  1. Why standard EMA teacher–student distillation fails here (Table 11b): In our new 100-episode ablation, a standard EMA teacher that receives the same corrupted future (τv′=τv\tau_v' = \tau_v) as the student yields almost no gain (54.8% vs. 54.6% without a teacher), because the projection head rapidly shortcuts the alignment loss (cosine similarity >0.999> 0.999). Our Privileged-Future Representation Distillation (PFD) places the teacher at the clean observed future (τv′=0\tau_v' = 0) so it acts as an outcome-informed inverse dynamics teacher, preventing collapse (similarity ≈0.90\approx 0.90) and boosting success to 58.6% (+4.0 pp).
  2. Coupled Vision–Action Reliability Corruption: Unlike action-only noise truncation via an auxiliary classifier (X-Diffusion), we continuously reshape the Beta action corruption density via source reliability qdq_d while enforcing τv<τa\tau_v < \tau_a over (0,1)(0,1), grounding noisy actions in cleaner predicted visual futures (+9.8 pp on precision-critical Articulated tasks).
  3. Demonstrated Scaling on Noisy Egocentric Data: Together with our S1 →\to S2 curriculum, these mechanisms unlock a ~25% steeper scaling slope up to 30,000 hours of mixed robot and noisy human videos (Figure 5), and yield +2.7 pp from adding noisy human data when S1-only continuation has already saturated (Table 11a).

W2: Why Feature-Space Prediction and DINOv3 vs. V-JEPA-2

  • Feature-space vs. pixel-space prediction: A rich body of work in world modeling and control—including DreamerV3 (Hafner et al., 2023), TD-MPC2 (Hansen et al., ICLR 2024), I-JEPA / V-JEPA (Assran et al., CVPR 2023; Bardes et al., 2024), and recent embodied world–action policies such as FLARE (Zheng et al., CoRL 2025), UniVLA (Bu et al., 2025), and LDA-1B (Lyu et al., 2026)—has established that predicting future states in a compact latent representation space is substantially more effective for control than reconstructing raw pixels. Pixel-level generation forces the model to allocate significant capacity to high-frequency background textures, lighting variations, and appearance details that are irrelevant to action decisions, whereas feature-space prediction focuses learning on task-relevant object and hand state transitions.
  • Advantages of DINO features for dexterous manipulation: For fine-grained tabletop and articulated manipulation, DINOv2 / DINOv3 (Oquab et al., TMLR 2024; Siméoni et al., 2025) are widely recognized for preserving dense, patch-level geometric and part-level spatial correspondences. Consequently, DINO representations have become a standard choice across robotics—used for spatial grounding in OpenVLA (Kim et al., CoRL 2024) and DINOBot (Di Palo & Johns, ICRA 2024), and directly validated as future-prediction targets in DINO-WM (Zhou et al., 2025), UniVLA, and LDA-1B. In addition, DINOv3 features can be directly decoded into RGB via an off-the-shelf RAEv2 decoder (Singh et al., 2026; Figure 1c), providing a convenient diagnostic tool for inspecting predicted futures during development.
  • Orthogonality to our core contribution: At the same time, the choice of visual target encoder is orthogonal to our proposed framework. Both reliability-conditioned corruption and privileged-future representation distillation operate on the joint flow-matching schedule and action-token representations without relying on any DINO-specific operation. Alternative self-supervised latent spaces such as V-JEPA-2 are therefore naturally compatible with our method, while adopting the well-validated DINOv3 target space allows us to isolate and study the core problem of scaling policy learning under noisy egocentric action supervision.

Q1: Is the Proposed Approach Model-Agnostic Across VLM Backbones and Sizes?

Yes, the method is completely model-agnostic. Because the VLM only provides contextual conditioning features Cψ(ct)C_\psi(c_t) and is excluded from the EMA teacher, our framework does not depend on any Qwen3.5-specific structure and can be paired with other VLM families or model sizes.

Thanks to the authors for providing the response. Regarding the model-agnostic claim, the authors state that the method is “completely model-agnostic,” but do not explain how the current architecture would accommodate VLMs with different structures, such as different numbers of layers, and provide experiments with other VLM backbones or sizes. Therefore, this strong claim remains insufficiently supported. The rebuttal partially addresses my original concerns, and I maintain my original score.

Thank you for your follow-up. Since our responses and new ablations for W1 and W2 appear to have addressed those concerns, we concisely clarify how our architecture accommodates different VLMs, share an early cross-backbone experiment, and place VLM selection in the context of VLA norms.

1. How Our Architecture Accommodates Arbitrary VLM Depths and Dimensions

Even in our default setup (Appendix C.2), Qwen3.5-4B (LVLM=32L_{\text{VLM}}=32 layers, width dVLM=2560d_{\text{VLM}}=2560) and our coupled vision–action MoT module (LMoT=16L_{\text{MoT}}=16 blocks, width dMoT=1536d_{\text{MoT}}=1536) already have different layer counts and feature dimensions. Unlike 1-to-1 layer-locked designs, our coupled module interfaces with the VLM strictly via stage-wise cross-attention:

  • Different Layer Counts (Feature Selection & Reuse): We extract MM intermediate VLM layers and reuse each feature across r=LMoT/Mr = L_{\text{MoT}}/M consecutive MoT blocks. With 32-layer Qwen3.5-4B, we stride-sample M=8M=8 layers ({4,8,…,32}\{4,8,\dots,32\}) and reuse each across r=2r=2 blocks (8×2=168\times 2=16). Shallower or deeper VLMs (e.g., 18, 26, or 36+ layers) simply adjust the selected layer subset or reuse factor rr (including reusing selected/final layers when LVLM<LMoTL_{\text{VLM}} < L_{\text{MoT}}), leaving the 16-block coupled module unchanged.
  • Different Hidden Dimensions (Latent Projection): A linear projection (Wctx∈RdVLM×dMoTW_{\text{ctx}} \in \mathbb{R}^{d_{\text{VLM}} \times d_{\text{MoT}}}) maps any VLM width into the MoT latent space (15361536), while cross-attention naturally absorbs varying token counts, visual patch layouts, and head structures.
  • Algorithmic Independence: Both Reliability-Conditioned Corruption (Sreg(qd)\mathcal{S}_{\text{reg}}(q_d), qdq_d) and Privileged-Future Distillation (ℓs=5→ℓt=11\ell_s=5 \to \ell_t=11 inside the coupled module; VLM excluded from the EMA teacher) have zero dependency on VLM internals.
  • To avoid any ambiguity, we will refine the phrasing from "completely model-agnostic" to "VLM-backbone-agnostic" and add the explicit layer-mapping/projection details above to Appendix C.2.

2. Early Empirical Test Across VLM Families & Sizes (Gemma-2 2B vs. Qwen3.5-4B)

During early development, we directly tested Gemma-2 2B—the 2B Gemma-family VLM architecture used in π0.5\pi_{0.5}, differing from Qwen3.5-4B in model family, scale (2B vs. 4B), layer count, and hidden width—under matched settings:

VLM BackboneScaleSuccess Rate (%)
Gemma-2 2B (π0.5\pi_{0.5}-style VLM family)2B50.7
Qwen3.5-4B (Adopted in paper)4B57.4

This confirms both seamless compatibility with a structurally distinct VLM and why we selected Qwen3.5-4B (+6.7 pp) while omitting this preliminary check as peripheral to our main thesis (we will include it in Appendix D.3).

3. Orthogonality and Standard Practice in MoT / VLAs

Fixing a strong VLM backbone to isolate core policy-learning contributions is standard across the MoT and VLA/WAM literature: canonical MoT VLAs (π0\pi_0, Black et al., 2025b; π0.5\pi_{0.5}, Black et al., 2025a; Knowledge Insulating VLAs, Driess et al., 2025) and structurally similar coupled foresight/world–action VLAs (InternVLA-A1.5, Ma et al., 2026, which also pairs Qwen3.5 with MoT latent foresight and flow matching; InternVLA-M1, Chen et al., 2025; Motus, Bi et al., 2026; DIAL, Chen et al., 2026) all adopt a single fixed VLM and do not ablate across VLM families, especially given pretraining costs.

Qwen3.5 is an excellent open-source VLM choice orthogonal to our method. Having resolved W1, W2, and now Q1 both architecturally and empirically, we respectfully ask that our work be evaluated on its central focus—effectively scaling policy learning with noisy egocentric human actions—rather than penalized on an orthogonal axis.

Reviewer_fMYyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EgoForesight studies how to improve robot policy learning using large amounts of egocentric human video with imperfect reconstructed action labels. It focuses on a joint future-visual-feature/action flow model with two mechanisms: reliability-conditioned corruption and weighting of action supervision, and representation distillation from an EMA teacher conditioned on the observed future. During training, it first establishes visual-action correspondences on robot and curated human data, then expands to a larger human-video mixture. The experiments show reduction in error of real-world action prediction by 20% and an improvement in a simulated robotic benchmark.

Critical strengths
  1. Controlled evidence that the two training mechanisms provide complementary benefits. At reduced pretraining scale, reliability treatment and future-conditioned distillation improve RoboCasa success from 52.0% to 54.3% and 56.3%, respectively; combining them reaches 58.6% (Table 2). This is the strongest evidence that the proposed combination contributes beyond the shared architecture and training data. However, missing run-to-run uncertainty limits confidence in the improvement’s reproducibility.

  2. Evidence that the complete recipe benefits downstream robot control, beyond action prediction. The method improves few-shot RoboCasa success by 8.1 percentage points over the controlled baseline at full pretraining scale (Fig. 5). It also achieves 63.8% average success on four real-world dexterous tasks versus 51.3% for GR00T N1.7 (Fig. 4). These results support practical usefulness, although the real-world comparison does not isolate the proposed mechanisms because the systems have different pretraining histories.

Critical weaknesses

The experiments do not clearly show how much the added noisy human data helps. In Figure 5, all data sources grow together, including robot data. Table 2 shows that the proposed methods improve performance on the mixed dataset. But these methods might also help without the additional noisy data.

An important comparison is missing: continue training on S1 data alone, or add noisy human data, using the same training budget. Another useful test would increase only the noisy human data while keeping S1 fixed. These comparisons would clarify whether the method delivers its main promise that I understand is "making more noisy human data useful for robot learning".

Questions for authors

With the same S1 initialization and S2 training budget, how does EgoForesight perform when S2 uses only S1 replay versus S1 replay plus the internal human dataset? Are results available that scale only the additional human data while holding S1 fixed?

Do you have quantitative evidence that the future-conditioned teacher produces more accurate action estimates than reconstructed labels, or controls distinguishing future information from ordinary EMA feature regularization? Please identify any existing measurements or ablations. Such evidence would substantially strengthen the interpretation that the method mitigates unreliable action supervision. Otherwise, narrowing the mechanism claims would improve alignment with the demonstrated results.

For Tables 2 and 3 and Figure 5, please specify each variant’s S2 time sampler, clean-future boundary sampling, downstream distillation settings, and evaluated checkpoint weights. In particular, does the baseline retain the same downstream distillation recipe, and how are EMA weights handled for teacher-off variants? Appendix D.3 states that adaptation recipes are shared, but the exact settings remain important for attributing the improvements.

Please provide the raw values at each scale, update/batch-size budgets across scales, available seed variability, and checkpoint-selection rules for both methods. Was a separate validation set used, or was the baseline checkpoint selected directly using the reported test MSE? Existing-run uncertainty and consistent validation-based selection could materially strengthen the scaling conclusion.

What additional training compute does the teacher require, and are any comparisons matched by FLOPs or wall-clock budget? If comparisons are matched only by data and updates, please state this explicitly. This determines whether the demonstrated benefit should be interpreted as data efficiency, compute efficiency, or both.

2 comments

We sincerely thank you for your rigorous and insightful review. Your suggestions on isolating the contribution of noisy human data, testing ordinary EMA regularization, and documenting per-variant configurations and compute overhead have significantly strengthened our paper. Below, we address each of your comments and questions.

W1 & Q1: Isolating the Contribution of Additional Noisy Human Data (Fixed-S1 Controls)

We agree that holding S1 fixed is essential to isolate the value of adding noisy egocentric human data from extra optimization steps or robot-data growth. Using the same full-scale S1 checkpoint (which combines robot data with curated human trajectories from EgoDex/T-Rex) and identical downstream adaptation recipes, we compared four settings on both RoboCasa few-shot closed-loop evaluation and real-world open-loop evaluation (added as Table 11a in Appendix D.3):

Pretraining Setting (Fixed S1 Initialization)Added Noisy Human DataPnP (%)Art. (%)Avg. (%)Gain vs. S1 ReplayOpen-Loop MSEMSE Change vs. S1 Replay
1. Original S1 Checkpoint0 h (No S2)62.065.562.9—0.051—
2. S2 Continuation: S1 Replay Only (saturates at ~20k steps)+0 h62.366.263.3Reference0.050Reference
3. S2 Continuation: S1 Replay + 5k h Internal Human+5,000 h65.766.065.8+2.5 pp0.042−16.0%
4. S2 Continuation: S1 Replay + 20k h Internal Human (Ours)+20,000 h65.168.766.0+2.7 pp0.031−38.0%
  1. S1 replay only vs. adding internal human data (Q1): Continuing S2 with S1 replay alone plateaus by ~20k updates (62.9% -> 63.3% in simulation and 0.051 -> 0.050 in open-loop MSE), showing that extra updates on S1 yield negligible gain (+0.4 pp / −2.0% MSE). Adding noisy internal human data raises RoboCasa average success to 65.8% (+5k h) and 66.0% (+20k h)—a +2.5 to +2.7 pp gain—while reducing real-world open-loop MSE to 0.042 (−16.0%) and 0.031 (−38.0%) relative to S1 replay alone.
  2. Scaling only the noisy human data (Q1): Holding S1 fixed and increasing noisy human data from 0 h →\to 5k h →\to 20k h monotonically improves closed-loop RoboCasa success (63.3% →\to 65.8% →\to 66.0%, with Articulated manipulation reaching 68.7% at 20k h) and substantially lowers real-world open-loop MSE (0.050 →\to 0.042 →\to 0.031), directly confirming that EgoForesight turns additional noisy human videos into stronger robot policies.

Q2: Quantitative Control for the Future-Conditioned Teacher and Narrowing Mechanism Claims

  1. Quantitative ablation distinguishing future information from ordinary EMA regularization: In the Table 2 setting (1/10 scale, 100 ep/task; Table 11b in Appendix D.3), we trained an identical EMA teacher that receives the student's corrupted future (τv′=τv\tau_v' = \tau_v) instead of the clean observed future (τv′=0\tau_v' = 0):
Pretraining Variant (1/10 Scale, Few-Shot)Teacher Visual InputFeature Cosine Sim.PnP (%)Art. (%)Avg. (%)
No Teacher (Reliability-only)——53.458.054.6
Same-Corrupted-Future EMA TeacherCorrupted (τv′=τv\tau_v' = \tau_v)>0.999> 0.999 (collapsed)53.658.554.8 (+0.2)
Clean-Future EMA Teacher (Ours)Observed (τv′=0\tau_v' = 0)≈0.90\approx 0.9057.462.258.6 (+4.0)

When τv′=τv\tau_v' = \tau_v, the student projection head PϕP_\phi rapidly shortcuts the alignment loss (cosine similarity >0.999> 0.999), yielding only +0.2 pp. Conditioning the teacher on τv′=0\tau_v' = 0 prevents collapse (cosine similarity ≈0.90\approx 0.90) and accounts for 95% of the teacher's gain (+3.8 out of +4.0 pp). 2. Tightening mechanism claims: Because our teacher distills latent action-token representations (LPFD\mathcal{L}_{\text{PFD}}) rather than explicit action labels—and because ground-truth 3D hand trajectories cannot be measured on uninstrumented in-the-wild videos without adding mocap/wearables that introduce visual OOD artifacts—we have narrowed our mechanism claim in Section 3.3 to providing outcome-informed, complementary action-representation supervision (where conditioning on τv′=0\tau_v' = 0 lets the flow-matching expert learn a lower-variance conditional representation anchored by the visual outcome).

Q3: Per-Variant Configurations for Tables 2, 3, and Figure 5

All nuisance factors are unified across variants to ensure clean attribution (documented in Appendix D.3):

Table / VariantS2 Sampler (Eq. 13) & Human (qd,wd)(q_d, w_d)Pretrain Teacher & Boundary (pedge=0.02p_{\text{edge}}=0.02)Post-Train Sampler & (qd,wd)(q_d, w_d)Post-Train Teacher & BoundaryEvaluated Weights & Path
Table 2 / Fig. 5: BaselineDecoupled, (1.0,1.0)(1.0, 1.0)OffEq. 13, (1.0,1.0)(1.0, 1.0)OnEMA, τv=τa1.5\tau_v = \tau_a^{1.5} (20 steps)
Table 2 / Table 10: w/ reliabilityDecoupled, (0.2,0.2)(0.2, 0.2)OffEq. 13, (1.0,1.0)(1.0, 1.0)OnEMA, τv=τa1.5\tau_v = \tau_a^{1.5} (20 steps)
Table 2: w/ teacherDecoupled, (1.0,1.0)(1.0, 1.0)OnEq. 13, (1.0,1.0)(1.0, 1.0)OnEMA, τv=τa1.5\tau_v = \tau_a^{1.5} (20 steps)
Table 1 / 2 / 3a / Fig. 5: OursDecoupled, (0.2,0.2)(0.2, 0.2)OnEq. 13, (1.0,1.0)(1.0, 1.0)OnEMA, τv=τa1.5\tau_v = \tau_a^{1.5} (20 steps)
Table 3(b): Post-train Teacher offDecoupled, (0.2,0.2)(0.2, 0.2)OnEq. 13, (1.0,1.0)(1.0, 1.0)OffEMA, τv=τa\tau_v = \tau_a (20 steps)
Table 3(b): Post-train Teacher onDecoupled, (0.2,0.2)(0.2, 0.2)OnEq. 13, (1.0,1.0)(1.0, 1.0)OnEMA, τv=τa\tau_v = \tau_a or τa1.5\tau_a^{1.5}

For teacher-off pretraining variants, EMA weights (μ=0.999\mu = 0.999) are still maintained solely for evaluation so parameter averaging is never a confounder. Sharing the same downstream recipe across all pretraining variants attributes performance differences strictly to pretraining.


Q4: Raw Scaling Values, Budgets, Seed Variability, and Checkpoint Selection

  1. Budgets and raw values across scales (added as Table 9 in Appendix D.3):
Data ScaleBatch SizeS1 + S2 UpdatesSamplesOpen-Loop MSE: BaselineOpen-Loop MSE: OursRoboCasa (%) [95% CI]: BaselineRoboCasa (%) [95% CI]: Ours
0 h———0.1240.12448.4 [46.4, 50.4]48.4 [46.4, 50.4]
3k h (1/10)25650k + 100k38.4M0.0900.08852.1 [50.1, 54.1]58.6 [56.6, 60.6]
10k h (1/3)1,02450k + 100k153.6M0.0590.05454.1 [52.1, 56.1]63.0 [61.0, 64.9]
30k h (Full)2,048100k + 200k614.4M0.039*0.03158.1 [56.1, 60.1]66.0 [64.1, 67.9]
  1. Checkpoint selection: Because the Baseline overfits in late S2 training on noisy human data (its 30k h open-loop MSE worsens from 0.039 to 0.042 at the endpoint, Fig. 1b), the held-out open-loop trajectories serve as a validation signal to select the lowest-MSE checkpoint (0.039) and its corresponding closed-loop evaluation (explicitly clarified in Appendix D.3).
  2. Seed variability: Across 3 independent training seeds (42, 0, 100) at the 3k h scale, Ours achieves 58.7% ± 0.1% (58.6%–58.8%), matching 3-seed evaluation variance (58.7% ± 0.1%) and confirming that seed noise is negligible relative to the scaling gains (+6.5 to +8.9 pp).

Q5: Training Compute Overhead of the EMA Teacher and Budget Matching

All comparisons are matched by training data and optimizer-update budgets, not by FLOPs or wall-clock time. Because the EMA teacher excludes the 4B VLM backbone (reusing detached student VLM features) and runs without backpropagation, its single-GPU overhead is only +6.5% (Table 12 in Appendix D.3, measured on an NVIDIA H200 with BS = 16, grad accum = 1, 16 warmup + 40 timed steps on identical real batches):

Measured Metric (Single NVIDIA H200, 140 GiB)EMA Teacher OffEMA Teacher OnChange
Model forward + backward time (s)1.0351.100+6.2%
Optimizer step time, excl. data loading (s)1.3201.405+6.4% (incl. ~19.2 ms EMA update)
Full step time, incl. data loading (s)1.3441.431+6.5%
Training throughput (samples/s)11.9111.18−6.1%
NVML process memory (GiB)96.37102.37+6.00 GiB

In multi-GPU distributed training, the no_grad teacher requires zero additional gradient all-reduce communication, so shared communication overhead further amortizes this 6.5% single-GPU difference into a negligible wall-clock gap.

Reviewer_dthxOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper trains a VLA, which is a Qwen3.5-4B backbone with coupled vision and action experts that jointly predicts future DINOv3 features and 30-step action chunks with flow matching, using about 30k hours of data, of which 20k hours are internal egocentric human video. Two components target the noise in action labels extracted from human video. The first gives each data source a reliability value, 1.0 for robot data and 0.2 for human data, that keeps the action labels of less reliable sources noisier during training and down-weights their action loss. The second is an EMA teacher that sees the clean future and the same noisy actions, and supervises intermediate action features of the student. On the 24 tasks of RoboCasa GR1, the model reaches 66.0% few-shot and 73.9% full-data success, against 58.3% and 70.2% for DIAL and 61.1% few-shot for DIAL + EgoDex. On four real-robot tasks it reaches 63.8%, against 51.3% for GR00T N1.7.

Critical strengths
  1. The paper gives a working recipe for pretraining on a relatively large amount of egocentric human data with noisy action labels, and the gains show up both in simulation and on a real dexterous robot: few-shot RoboCasa success is 66.0% against 61.1% for DIAL + EgoDex, and real-robot success is 63.8% against 51.3% for GR00T N1.7. Making noisy human-video actions usable is one of the main open problems in scaling VLAs, and this is the main reason I lean toward acceptance.

  2. With the data held fixed, turning on both components moves RoboCasa success from 52.0% to 58.6% in Table 2. With 1,200 evaluation episodes per configuration, a 6.6-point gap is about three standard errors of evaluation noise, so the method seems to help beyond the extra data. No training seeds are reported, though.

Critical weaknesses

The paper is built around two components, and the components themselves sound reasonable, but the evidence for each one on its own is weak. No seeds or error bars are reported, and most individual gains are about one standard error of a difference or less: +2.3 for reliability conditioning in Tables 2 and 9, and +2.1 for the post-training teacher and +1.1 for the inference path in Table 3(b). The teacher alone during pretraining gives +4.3 in Table 2, about two standard errors, again with no seeds reported. Table 9 is the hardest to read: qd∈{0.8,0.5,0.2}q_d \in \{0.8, 0.5, 0.2\} all give 54.0–54.3, so I cannot tell whether the reliability value matters, or whether the benefit comes from something else, such as switching from synchronized to decoupled vision/action flow-time sampling. On the real robot, with 20 trials per task, the method beats GR00T N1.7 on two of the four tasks, 17/20 vs 11/20 and 6/20 vs 1/20, ties on handover and is slightly behind on pick-and-place in Fig. 4.

I believe the full system works with mild improvement, but I'm more concerned about the correctness of attribution, i.e., what may have contributed to the improvement.

Questions for authors
  1. The "S1+S2 (joint)" row of Table 3(a) has exactly the numbers of the "w/ reliability" row of Table 2, 53.1 / 58.0 / 54.3. I thought S1 -> S2 is the default choice reported in all ablations except the training-stage one. Why does the joint recipe appear in the reliability ablation?

  2. Does the qd=1q_d = 1 baseline use the synchronized sampler, τv=τa\tau_v = \tau_a, or the decoupled sampler of Eq. 13? If it is synchronized, the gain attributed to reliability conditioning may actually come from decoupling.

  3. Can you report several training seeds for Tables 2, 3 and 9? Do the three data scales in Fig. 5 use the same training budget? The 1/10-scale runs seem to use batch 256 and 150k steps, against 2,048 and 300k at full scale.

  4. How does reliability-conditioned corruption differ in practice from X-Diffusion, which also treats human actions as noisy robot actions and, building on Ambient Diffusion, trains on them selectively at high noise levels? The paper mentions it in one clause in App. G.1.

  5. Which denoising path is used for the few-shot results in Table 1, and which path, number of integration steps and weights are used on the real robot?

I will consider raising my score if the questions above are properly resolved.

Minor suggestion that did not affect my score: some previous techniques show that EMA self-distillation might improve model performance. If possible, the authors could add an ablation showing whether an EMA teacher without access to the clean future brings a similar improvement.

2 comments

We thank Reviewer dthx for the insightful comments on attribution and statistical reliability. In the revised manuscript, all RoboCasa evaluations are expanded from 50 to 100 episodes per task, with 95% Wilson confidence intervals added to Figure 5(b).

W1: Evidence for Individual Components, Seed Stability, Table 9 Attribution, and Real-Robot Results

1. Complementary component gains under 100 episodes/task: Doubling the evaluation budget reduces sampling noise and confirms consistent, task-specific gains. In particular, while Pick-and-Place (PnP) mainly probes object generalization, Articulated (Art.) tasks require precise multi-step trajectories to manipulate and close fixtures—where noisy human action labels hurt most. Reliability conditioning specifically mitigates this label noise (+9.8 pp on Art.), while combining it with the privileged-future teacher lifts both categories:

Stage & SettingVariantPnP (%)Art. (%)Avg. (%)Avg. Gain
Pretraining (Table 2, 1/10 scale, few-shot)Baseline53.448.252.1—
w/ reliability (qd=0.2q_d = 0.2)53.458.054.6+2.5 pp
w/ teacher (τv′=0\tau_v' = 0)55.956.856.1+4.0 pp
Ours (reliability + teacher)57.462.258.6+6.5 pp
Post-training (Table 3b, full scale, full data)Teacher off, τv=τa\tau_v = \tau_a66.482.570.4—
Teacher on, τv=τa\tau_v = \tau_a71.575.772.5+2.1 pp
Teacher on, τv=τa1.5\tau_v = \tau_a^{1.5} (Ours)72.279.774.0+3.6 pp

2. Training-seed vs. evaluation-seed stability: We repeated Ours in Table 2 across 3 independent training seeds (42, 0, 100) and evaluated the Seed-42 checkpoint across 3 evaluation seeds (42, 0, 100):

Repetition Type (1/10 Scale, 100 ep/task)SeedsPnP Range (Mean ± SD)Art. Range (Mean ± SD)Avg. Range (Mean ± SD)
3 Independent Training Runs42, 0, 10057.2–57.5 (57.4 ± 0.2)62.2–63.0 (62.7 ± 0.4)58.6–58.8 (58.7 ± 0.1)
3 Independent Evaluation Runs42, 0, 10057.4–57.7 (57.6 ± 0.2)61.7–62.2 (62.0 ± 0.3)58.6–58.8 (58.7 ± 0.1)

Training-seed variance (±0.1%\pm 0.1\%) is on the same small scale as 100-episode evaluation variance and an order of magnitude smaller than the component gains (+2.5 to +6.5 pp).

3. Why qdq_d matters in Table 9 (and optimal range qd∈q_d \in {0.5,0.20.5, 0.2}): The qd=1.0q_d = 1.0 baseline already uses the decoupled sampler of Eq. 13, not synchronized sampling. Under 100 episodes/task (Table 10 in Appendix D.3), lowering human qdq_d reveals a monotonic increase in both Art. and Avg. success:

Human qdq_dLoss Weight wdw_dPnP (%)Art. (%)Avg. (%)
1.0 (Baseline, Eq. 13)1.053.448.252.1
0.80.853.455.053.8
0.50.553.257.254.2
0.20.253.458.054.6
0.2 (Unit loss weight)1.052.858.054.1

A mild reduction at qd=0.8q_d = 0.8 brings only a partial gain (53.8%), whereas stronger corruption priors qd∈q_d \in {0.5,0.20.5, 0.2} consistently achieve the strongest performance (54.2%–54.6% Avg., 57.2%–58.0% Art.). This shows both that the reliability level matters and that performance is stable within the low-reliability regime qd∈q_d \in {0.5,0.20.5, 0.2}.

4. Real-robot evaluation: Across 4 tasks (20 trials/task, 80 trials/method with matched initial configurations per trial), note that GR00T N1.7 was pretrained directly on the target R1 Pro + Sharpa embodiment, whereas EgoForesight was not. On simpler tasks (Pick-and-Place: 17/20 vs. 18/20; Handover: 11/20 tie, with ours leading in progress score 79 vs. 76), EgoForesight matches the in-domain baseline; on precision-critical, long-horizon dexterous tasks (Bottle Opening: 17/20 vs. 11/20; 7-stage Syringe Transfer: 6/20 vs. 1/20, progress 84 vs. 50), EgoForesight wins decisively, achieving +15.0 pp higher mean success (63.8% vs. 48.8%) and +14.2 higher progress score (84.8 vs. 70.6).


Q1: Results for "S1+S2 (joint)" in Table 3(a) and "w/ reliability" in Table 2

These rows come from two independent training experiments: Table 2 ("w/ reliability") uses the two-stage S1 →\to S2 schedule without the teacher, whereas Table 3(a) ("S1+S2 joint") trains a single stage on combined S1+S2 data with both components. Their 50-episode averages coincided at one decimal place; at 100 episodes/task, they separate to 53.4 / 58.0 / 54.6 (Table 2) and 52.8 / 58.3 / 54.2 (Table 3a).

Q2: Does the qd=1q_d = 1 Baseline Use Synchronized or Decoupled Sampling?

The qd=1q_d = 1 baseline in S2 uses the decoupled sampler of Eq. 13, not synchronized sampling (τv=τa\tau_v = \tau_a). Setting qd=1q_d = 1 in Eq. 4 gives αd=αmin⁡=0.5\alpha_d = \alpha_{\min} = 0.5, which still samples τv∼U(0,1)\tau_v \sim \mathcal{U}(0,1) and τa=τv+(1−τv)g\tau_a = \tau_v + (1-\tau_v)g with g∼Beta⁡(0.5,2)g \sim \operatorname{Beta}(0.5, 2), ensuring 0<τv<τa<10 < \tau_v < \tau_a < 1. As shown in the Table 9 breakdown under W1, changing human qdq_d from 1.0→0.21.0 \to 0.2 while keeping unit loss weight (wd=1.0w_d = 1.0) already raises average success from 52.1% to 54.1% (Art.: 48.2% →\to 58.0%), and adding loss downweighting (wd=0.2w_d = 0.2) reaches 54.6%.


Q3: Multi-Seed Runs and Training Budgets Across Scales in Figure 5

  1. Multi-seed stability: As shown in the table under W1, repeating 1/10-scale training across 3 seeds yields 58.7% ± 0.1%, matching 3-seed evaluation variance (58.7% ± 0.1%). Given the compute cost of pretraining (22k H200 GPU-hours at full scale), running multiple pretraining seeds for every ablation row is infeasible.
  2. Training budgets in Figure 5 (Table 9 in Appendix D.3): Within each scale, Baseline and Ours share identical data subsets, batch sizes, and update budgets, while budgets scale with dataset size across scales:
Pretraining ScaleUnique Video HoursGlobal Batch SizeS1 + S2 UpdatesManipulation Sample Presentations
1/10 Scale3k h25650k + 100k (150k)38.4M
1/3 Scale10k h1,02450k + 100k (150k)153.6M
Full Scale30k h2,048100k + 200k (300k)614.4M

Because the Baseline overfits in late training on noisy data (its full-scale open-loop MSE worsens from 0.039 to 0.042 at the end of training, Fig. 1b), we early-stop the Baseline at its best open-loop MSE checkpoint (0.039) and report the corresponding closed-loop result.


Q4: Practical Differences from X-Diffusion

While we share the high-noise insight of X-Diffusion and Ambient Diffusion, our method differs in three key ways:

  1. Coupled Vision–Action Density Reshaping vs. Discriminator Truncation: X-Diffusion trains an auxiliary cross-embodiment classifier to set hard per-trajectory noise truncation thresholds on actions alone. In contrast, we operate in a coupled vision–action flow, using a coarse source prior qdq_d to smoothly reshape the Beta action-time density over (0,1)(0,1) while enforcing τv<τa\tau_v < \tau_a. This avoids leaving low-noise intervals unsupervised across domains and encourages the action expert to rely on cleaner predicted visual futures.
  2. Active Outcome Supervision via Privileged-Future Distillation (PFD): Noise scheduling only downweights noisy action labels; our observed-future EMA teacher (τv′=0\tau_v' = 0) actively extracts outcome-informed action representations from visual transitions (ot,ot+K)(o_t, o_{t+K}).
  3. Two-Stage Scaling to 30k Hours: Combining S1 →\to S2 curriculum pretraining with both mechanisms enables ~25% steeper scaling slopes up to 30k hours on 58-DoF dexterous manipulation.

Q5: Denoising Path, Integration Steps, and Weights for Table 1 and Real Robot

Both Table 1 (RoboCasa few-shot and full-data) and the real-robot evaluations in Figure 4 use the vision-leading path τv=τa1.5\tau_v = \tau_a^{1.5} (γ=1.5\gamma = 1.5), 20 flow integration steps, and EMA checkpoint weights (conditioning only on current context ctc_t at deployment).


M1: Ablation with a Standard EMA Teacher Without Clean-Future Access

We trained a Standard EMA Teacher control in the Table 2 setting (1/10 scale, 100 ep/task; added as Table 11b in Appendix D.3) where the teacher receives the student's corrupted future (τv′=τv\tau_v' = \tau_v) instead of the clean future (τv′=0\tau_v' = 0), keeping all other settings identical:

Pretraining Variant (1/10 Scale, Few-Shot)Teacher Visual InputFeature Cosine Sim.PnP (%)Art. (%)Avg. (%)
No Teacher (Reliability-only)——53.458.054.6
Standard EMA TeacherCorrupted (τv′=τv\tau_v' = \tau_v)>0.999> 0.999 (collapsed)53.658.554.8 (+0.2)
Privileged-Future EMA Teacher (Ours)Observed (τv′=0\tau_v' = 0)≈0.90\approx 0.9057.462.258.6 (+4.0)

When τv′=τv\tau_v' = \tau_v, the student's projection head PϕP_\phi quickly shortcuts the cosine loss (feature similarity rapidly exceeds 0.999). Conditioning the teacher on the clean future (τv′=0\tau_v' = 0) injects privileged outcome information that prevents collapse (similarity stabilizes at ~0.90), driving the +3.8 pp gain over standard EMA.


Comments on the paper

2 comments
Area_Chair_ipQDOpenReview ↗

Please review the authors’ rebuttal and update your assessment

Dear reviewers,

Thank you for your reviews, and thank you to the authors for their responses. Please read the authors’ rebuttal and discuss whether it addresses your main concerns. Please update your review and rating if warranted, and briefly explain your current assessment, including any concerns that remain unresolved. If your assessment is unchanged, please explain why.

Thank you for your continued engagement.

We sincerely thank the Area Chair and all four reviewers for their time, thoughtful critiques, and engagement throughout the review and discussion period. Below, we summarize the key experiments, analyses, and manuscript revisions added during the rebuttal:

  1. Statistical Reliability & Seed Stability (Reviewers dthx):

    • Expanded all RoboCasa evaluations from 50 to 100 episodes per task (2,400 episodes per configuration across Tables 1–3 and 8–11) and added 95% Wilson confidence intervals to Figure 5(b).
    • Added 3 independent training-seed and 3 evaluation-seed runs at 1/10 scale (58.7% ± 0.1% for both), confirming that run-to-run variance is an order of magnitude smaller than our component gains (+2.5 to +6.5 pp).
    • Confirmed monotonic reliability gains under 100 episodes/task (Table 10: 52.1% → 53.8% → 54.2% → 54.6% as human qdq_d decreases from 1.0 to 0.2, with stable performance across qd∈q_d \in {0.5, 0.2}).
  2. Isolating the Contribution of Noisy Human Data (Reviewers fMYy):

    • Added fixed-S1 expansion controls (Table 11a): continuing S2 with S1 replay alone plateaus by ~20k steps (62.9% → 63.3%; MSE 0.051 → 0.050), whereas adding +5k h and +20k h of noisy egocentric human data raises RoboCasa few-shot success to 65.8% and 66.0% (+2.5 / +2.7 pp; Art. 68.7%) and reduces real-world open-loop MSE by 16.0% (0.042) and 38.0% (0.031).
  3. Privileged-Future vs. Standard EMA Teacher & Compute Cost (Reviewers dthx, fMYy):

    • Added a Standard EMA Teacher control (Table 11b): receiving the student's corrupted future (τv′=τv\tau_v' = \tau_v) collapses representation alignment (cosine similarity >0.999> 0.999) and yields only +0.2 pp (54.8% vs. 54.6%), whereas our clean-future teacher (τv′=0\tau_v' = 0) maintains informative alignment (~0.90) and reaches 58.6% (+4.0 pp)—driving 95% of the teacher gain.
    • Profiled single-GPU H200 training overhead (Table 12): because the teacher excludes the 4B VLM, it adds only +6.5% step time.
  4. Visual Grounding, Protocol Clarity & Architectural Details (Reviewers FUMS, fMYy, dthx, Bb1h):

    • Added Layer-5 (ℓs=5\ell_s = 5) action-to-image cross-attention visualizations (Figure 11), showing that distilled action tokens attend sharply to active hands and manipulated objects rather than background ego-motion.
    • Documented per-variant training/evaluation configurations, optimization budgets, raw scaling values (Table 9), and checkpoint-selection rules in Appendix D.3; tightened mechanism claims in Section 3.3 and expanded related work in Appendix G.2.
    • Clarified cross-VLM compatibility via stage-wise cross-attention layer selection/reuse and latent projection, and shared our controlled 1/10-scale Gemma-2 2B vs. Qwen3.5-4B comparison (50.7% vs. 57.4% PnP).

We thank the reviewers and Area Chair again for helping us substantially strengthen the empirical rigor and clarity of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.