acceptodds
Under review as a conference paper at ICLR 2027

World Action Learning via Interaction-Centric Spectral Latent Guidance

20 views

Abstract

Learning general-purpose robot policies requires large-scale real-world interaction data, yet collecting robot demonstrations through teleoperation remains expensive and difficult to scale. Egocentric videos provide a rich source of human interaction experience that shares task-relevant semantics with robotic manipulation, creating an opportunity to align human and robot actions in a shared latent action space for cross-embodiment knowledge transfer. However, existing latent action approaches typically infer actions through reconstruction between consecutive frames, which are not inherently interaction-centric and can be dominated by nuisance variations, such as ego-camera motion. Moreover, although human interactions and robot actions share interaction semantics, they often exhibit substantially different temporal dynamics, making direct transfer to robot policies challenging. To address these challenges, we propose **WING** (**W**orld Action Learning via **IN**teraction-Centric Spectral Latent **G**uidance), a framework for transferring interaction knowledge from egocentric videos to robot policies. First, we introduce an interaction-motion decoupling mechanism that separates interaction-relevant signals from task-irrelevant motion and selectively distills the interaction-centric components into latent actions. Second, motivated by the observation that cross-embodiment task semantics are primarily encoded in slowly varying temporal structures, we identify shared components between egocentric latent actions and robot behaviors in the spectral domain and use them as guidance for action generation at inference time. With these designs, WING achieves average success rates of 99.20% on LIBERO, 93.80% on RoboTwin 2.0, and 57.7% on RoboCasa-GR1, while also demonstrating strong performance across four real-world tasks under diverse generalization settings. These results demonstrate that WING can effectively distill embodied interaction knowledge from large-scale egocentric videos and transfer it to robot control, providing a scalable pathway for acquiring physical interaction knowledge from human experience.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Models, Egocentric Data, Robotics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_B5jXOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes WING for transferring interaction knowledge from egocentric human videos to robot policies. The method introduces an interaction-centric latent action model that explicitly separates camera motion from interaction-related motion, and further uses low-frequency DCT components of latent-action trajectories as cross-embodiment guidance. Extensive representation analyses, ablations, simulation benchmarks, and real-world experiments consistently support the proposed design.

Critical strengths

Clear and well-motivated decomposition of cross-embodiment transfer. The paper identifies two concrete sources of mismatch in transferring egocentric human videos to robot policies: contamination of latent actions by observer-induced camera motion, and differences in fine-grained temporal dynamics across embodiments. The proposed interaction-centric latent action decomposition and low-frequency spectral guidance directly target these two issues, resulting in a method that is conceptually simple, intuitive, and well aligned with the underlying problem.

Exceptionally thorough validation of the proposed mechanisms. The experimental analysis goes substantially beyond downstream performance. The paper directly evaluates whether the interaction latent becomes less sensitive to camera motion while preserving interaction semantics, and whether low-frequency latent components indeed exhibit stronger correspondence across human and robot executions. Together with careful component ablations and consistently strong downstream results, these analyses provide a particularly convincing connection between the proposed mechanisms and the final performance gains.

Critical weaknesses

None

Questions for authors

Scaling to larger and less curated egocentric datasets. The latent-action model is pretrained on approximately 200 hours selected from Ego4D, EgoDex, and EgoVerse, although the full datasets are substantially larger. Could the authors clarify how these subsets were selected? In particular, were videos with strong camera motion, tracking failures, or other difficult conditions filtered out? I would also be interested in whether the authors have explored scaling to substantially larger and less curated subsets.

3 comments
AuthorsOpenReview ↗

Response to B5jX

We are truly grateful to the reviewer for the time and effort devoted to reviewing our paper and for the thoughtful and insightful comments!

Q1: Scaling to Larger and Less Curated Egocentric Datasets

We follow commonly adopted data selection practices to construct our 200-hour egocentric pretraining subset, consisting of approximately 95 hours from EgoVerse, 85 hours from EgoDex, and 20 hours from Ego4D. Specifically, we prioritize videos with meaningful hand–object interactions while maintaining diversity in tasks and scenes. For Ego4D, we additionally leverage the provided action annotations to identify relevant interaction segments. Our selection is driven by general interaction relevance and data quality. In particular, we do not explicitly filter videos based on camera motion magnitude.

Regarding scalability, our current study focuses on validating the proposed representation-learning mechanism under a controlled 200-hour pretraining budget, rather than conducting a systematic data-scaling study. Nevertheless, this relatively modest pretraining corpus already yields meaningful downstream benefits: with only 25% of robot demonstrations, WING achieves 96.2% success on LIBERO, compared to 92.6% for the baseline. While this result demonstrates downstream data efficiency, it highlights the effectiveness of the learned representations under a limited pretraining budget. We agree that extending WING to larger and less curated egocentric datasets would provide a valuable assessment of its scalability and robustness, which we leave for future work.

Thank you for your clarification regarding the data selection process and the scalability of WING. They fully address my minor concern. I will maintain my positive assessment.

Thanks for your inspiring feedback! We are pleased to know that you agree with our response. We sincerely thank you for your dedication and effort on our work, and your expertise will definitely motivate our future research!

Reviewer_zFPLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces WING, a framework for transferring interaction knowledge from egocentric human videos to robotic manipulation. It addresses two challenges: latent action representations can entangle task-relevant interactions with observer-induced camera motion, and human and robot executions can differ in their temporal dynamics.

WING first uses point tracking and global background motion estimation to separate camera motion from local interaction residuals. A motion-supervised teacher learns interaction-centric representations, which are subsequently distilled into an RGB-only latent action encoder. The framework then applies a temporal discrete cosine transform (DCT) to latent action sequences and retains low-frequency components as supervision targets, motivated by their stronger correspondence across human and robot interactions. A predictor estimates this guidance from the current observation and task context, and the predicted guidance conditions robot action generation through gated cross-attention.

Training data consists of egocentric videos and robot data. The paper reports average success rates of 99.20% on LIBERO, 93.80% on RoboTwin 2.0, and 57.7% on RoboCasa–GR1, alongside 75.0% across four real-world manipulation tasks under the standard setting. These evaluations are complemented by representation analyses, real-world generalization tests, component ablations, comparisons of pretraining strategies, and robot-data efficiency experiments.

Critical strengths
  1. Well-motivated interaction-centric latent action learning. The paper directly addresses the entanglement of observer-induced camera motion and task-relevant interaction dynamics. Geometric motion decomposition, camera-consistency training, and RGB-only distillation provide a concrete approach to reducing this interference, making the learned representations more useful for robot learning.

  2. Broad experimental support. Representation analyses, three simulation benchmarks, and real-world manipulation experiments provide complementary evidence for the method’s effectiveness. Component ablations, pretraining comparisons, and reduced-data experiments further support the contributions of the proposed designs and their relevance to improving robot-data efficiency.

  3. Coherent integration with world-action modeling. WING incorporates interaction-centric latent actions into a world-action model through predicted low-frequency guidance. This provides a practical interface for transferring human interaction knowledge while allowing robot demonstrations to determine embodiment-specific controls.

Critical weaknesses
  1. The multi-stage training pipeline adds practical complexity, while its incremental computational cost remains unclear. WING involves motion-cue preprocessing, teacher training, RGB-student distillation, offline latent-target construction, egocentric-video WAM pretraining, and robot-policy post-training. Appendix C reports 60,000 optimization steps each for teacher training and distillation, followed by egocentric pretraining for 40 epochs on 32 H100 GPUs. These stages introduce additional dependencies and implementation effort. However, the reported WAM pretraining configuration does not isolate the overhead attributable to WING’s additional components, and the paper does not report stage-wise training times or GPU-hours. A breakdown of preprocessing and training costs, together with a comparison against a corresponding WAM baseline, would help assess whether the downstream gains justify the additional complexity.

  2. The downstream benefit of low-frequency guidance is not sufficiently disentangled from compression and prediction difficulty. Figure 3 and Appendix E.2 support stronger cross-embodiment correspondence in low-frequency components. However, Table 3 and the bandwidth ablation in Figure 5(a) also change the target and guidance dimensionality, leaving open whether better control primarily reflects this correspondence, easier prediction, or a stronger information bottleneck. The dimensionality control in Appendix E.2.3 concerns clustering rather than policy performance. A downstream comparison at matched guidance dimensionality would strengthen the proposed explanation.

  3. The encoder replacement ablation does not isolate the contribution of explicit camera–interaction motion separation. Replacing WING-LAM with DreamDojo in Table 3 and the “w/o OD.” variants changes multiple aspects of representation learning at once, rather than selectively removing geometric motion separation while keeping camera-consistency training, regularization, and distillation fixed. These results demonstrate the utility of WING-LAM as a whole, but do not establish the specific benefit of its geometric motion routing beyond the other training components. The head-pose readout and camera-perturbation results provide supporting evidence, but leave this attribution unresolved.

Questions for authors
  1. Appendix C reports 60,000 steps each for teacher training and distillation, followed by WAM pretraining for 40 epochs on 32 H100 GPUs. Could you provide the wall-clock time or GPU-hours for motion preprocessing, teacher training, distillation, and latent-target construction, and clarify the incremental cost relative to the video-only pretraining baseline in Figure 5(b)? Which costs are incurred once and which recur for each downstream dataset?

  2. Figure 3 and Appendix E.2 support stronger cross-embodiment correspondence in low-frequency components. However, Table 3 and Figure 5(a) also vary the prediction-target dimensionality and the number of guidance tokens. Do you have downstream policy results comparing low-frequency guidance with an equally sized alternative, such as a projection of the full trajectory, while matching the encoder, predictor capacity, and training budget? The PCA control in E.2.3 addresses clustering rather than policy performance.

  3. Appendix C.2.2 describes normalized guidance targets. How is this normalization performed across latent dimensions and frequency components, and how does held-out guidance prediction error vary between the full time-domain target and the K = 2, 4, and 8 spectral targets? Can these results help determine whether the downstream advantage of K = 4 reflects stronger cross-embodiment correspondence, easier prediction, or a combination of both?

  4. The head-pose readout and camera-perturbation results support the selectivity of WING-LAM as a whole. However, Table 3 and the “w/o OD.” variants replace the complete encoder with DreamDojo. Is there a controlled ablation that isolates geometric camera–interaction motion separation from camera-consistency training, representation regularization, and distillation, with the remaining components matched as closely as possible? Such evidence would clarify which part of WING-LAM drives its representation and policy gains.

  5. How many independent training seeds underlie the policy results in Table 3 and Figure 5, and what uncertainty accompanies the reported success rates? Given that some differences are small, could you clarify which improvements remain consistent across training runs and evaluation episodes? This would affect how strongly the component ablations support the proposed contributions.

5 comments
AuthorsOpenReview ↗

Response to Reviewer zFPL (1/3)

We are truly grateful to the reviewer for the time and effort devoted to reviewing our paper and for the thoughtful and constructive comments!

Q1 & W1: Computational Cost and Reusability

We thank the reviewer for raising this important concern. We will provide a detailed breakdown of the computational costs and clarify how these costs are amortized across downstream tasks.

Regarding computational costs, motion preprocessing requires approximately 150 GPU processing hours across the three datasets. Teacher training, latent-target extraction, and RGB-student distillation take roughly 4.6, 2.5, and 18.8 H100 GPU-hours, respectively. Ego WAM pretraining takes 42 hours on 32 H100 GPUs, incurring only a 10% increase in training time compared with video-only pretraining under the same hardware configuration.

Importantly, motion preprocessing and teacher–student training are one-time costs that can be amortized across downstream tasks. For new robot datasets, the pretrained RGB student can be directly reused, requiring only latent-target extraction and policy post-training. Therefore, although WING introduces additional offline representation-learning costs, these stages do not need to be repeated for each downstream task, while WAM pretraining remains computationally comparable to the video-only baseline.

Q2&W2: Matched-Dimensional Guidance Comparison

To further clarify the benefit of low-frequency guidance beyond dimensionality reduction, we conducted additional downstream policy experiments with matched guidance dimensionality. We compare DCT with three alternatives: temporal averaging (Avg), Gaussian smoothing with downsampling (Gaussian), and PCA projection of the full latent trajectory. All methods produce four 32-dimensional guidance tokens and share the same frozen latent encoder, predictor capacity, policy initialization, training data, and training budget. Each policy is evaluated on 40 LIBERO tasks with 50 rollouts per task.

Table. Matched-dimensional guidance comparison on LIBERO (success rate %, ↑).

GuidanceSpatialObjectGoalLong
Ours (DCT)99.6100.099.296.2
Avg97.098.497.296.8
Gaussian94.699.096.892.2
PCA97.096.696.895.2

As shown in Table above, DCT achieves the highest average success rate, outperforming Avg, Gaussian, and PCA. Notably, PCA projects the full trajectory into the same 128-dimensional space, yet still underperforms DCT. These results suggest that improvement cannot be attributed to dimensionality reduction alone. Together with the stronger cross-embodiment correspondence of low-frequency components (Fig. 3 and Appendix E.2), they further support the benefit of explicitly selecting low-frequency interaction structure for robot policy learning.

AuthorsOpenReview ↗

Response to Reviewer zFPL (2/3)

Q3:Guidance Normalization and Prediction Difficulty

We clarify the guidance normalization procedure and provide additional analyses to examine prediction difficulty and semantic preservation across different target representations.

  • Guidance Normalization. Given an interaction latent trajectory Z∈R8×32Z\in\mathbb{R}^{8\times32}, we apply an orthonormal temporal DCT and retain the first KK frequency components. Each frequency–latent-dimension coordinate is independently standardized using training-set statistics: qk,d~=qk,d−μk,dσk,d+10−6.\tilde{q_{{k,d}}}=\frac{q_{k,d}-\mu_{k,d}}{\sigma_{k,d}+10^{-6}}. The statistics remain fixed during training, without target clipping. Separate statistics are estimated for egocentric pretraining and robot post-training.
  • Controlled Prediction Analysis. We train separate capacity-matched auxiliary predictors for 10k steps using identical current-frame visual and language inputs. We report normalized MSE (NMSE) relative to a training-mean predictor, truncation MSE measuring omitted spectral energy, and clustering NMI measuring action-semantic preservation.

Table. Prediction difficulty and semantic preservation across target representations.

TargetNMSE ↓Truncation MSE ↓Clustering NMI ↑
Time-domain (H=8)0.799800.25 ± 0.01
DCT K=20.8020.02970.40 ± 0.01
DCT K=40.8530.01420.55 ± 0.02
DCT K=80.95100.31 ± 0.01

Prediction errors are measured using auxiliary predictors. Clustering NMI is evaluated after projecting all representations to the same 32-dimensional space.

  • As shown in the table above, K=4 provides a favorable balance between predictability and preservation of task-relevant interaction structure, rather than simply yielding an easier prediction objective. Although the time-domain target achieves the lowest NMSE, it exhibits the weakest semantic clustering. Similarly, K=2 is easier to predict but incurs greater truncation error and lower NMI (0.40 vs. 0.55), while K=8 preserves the full trajectory at the cost of higher prediction error and weaker semantic clustering. These findings suggest that the benefit of K=4 cannot be explained by easier prediction alone and are consistent with the stronger cross-embodiment correspondence of low-frequency components observed in Appendix E.2. Moreover, as shown in the downstream bandwidth ablation in Fig. 5(a), other choices of K also achieve competitive policy performance, indicating that WING is relatively robust to the frequency cutoff, with K=4 providing the best overall trade-off.

Q4 & W3: Controlled Ablation of WING-LAM Components

While our original experiments have demonstrated the effectiveness of WING-LAM, we conducted additional controlled ablations to further understand the contributions of its individual components and address the reviewer's concern.

Specifically, we compare Full WING-LAM with three variants: w/o Geometric Compensation, which replaces camera-compensated interaction residuals with raw image-space displacements and adjusts the reconstruction objective accordingly; w/o Camera Consistency, which removes the camera-consistency loss while retaining geometric compensation; and w/o Variance/Covariance Regularization, which removes the variance and covariance regularization terms from the training objective. All variants use the same egocentric dataset, architecture, initialization, and training budget, with the remaining objectives unchanged. We evaluate frozen representations using held-out MLP interaction-motion readout probes on EgoDex, EgoVerse, and Ego4D.

Table. Interaction-motion readout (held-out MLP R² ↑).

VariantEgoDexEgoVerseEgo4D
Full WING-LAM0.480.760.38
w/o Geometric Compensation0.270.610.30
w/o Camera Consistency0.350.660.32
w/o Variance/Covariance Reg.0.420.640.35

As shown in Table above, removing geometric compensation leads to the largest R2R^2 reductions across all three datasets, highlighting its importance in learning interaction-centric representations. This is consistent with our geometric design: mapping tracked points back to the reference frame using the estimated inverse camera warp allows interaction residuals to be computed with reduced influence from global camera motion, providing more interaction-focused supervision. Meanwhile, removing camera consistency or variance/covariance regularization also degrades performance, suggesting that these objectives provide complementary benefits in improving representation quality.

AuthorsOpenReview ↗

Response to Reviewer zFPL (3/3)

Q5: Training Seeds and Evaluation Uncertainty

We thank the reviewer for raising this concern. We acknowledge that the original results in Table 3 and Figure 5 were obtained from one training seed per configuration. Given the substantial computational cost of multiple independent training runs, we followed the commonly adopted practice of extensive rollout-based evaluation in robotic manipulation benchmarks such as LIBERO and RoboTwin. Specifically, on LIBERO, each policy is evaluated over 50 rollouts per task across 40 tasks (2,000 episodes in total). On RoboTwin 2.0, each task is evaluated over 100 rollouts per condition, while our real-world experiments use 20 trials per task. These repeated evaluations help reduce evaluation-level randomness and provide more reliable estimates of policy performance.

To further assess training stability and address the reviewer's concern, we conducted two additional independent training runs on LIBERO under the controlled setting without egocentric WAM pretraining. The three seeds achieve average success rates of 98.70%, 98.75%, and 98.75%, respectively, yielding 98.73% ± 0.03% (mean ± SD). These results indicate consistent overall performance across training seeds, complementing our extensive rollout-based evaluations. We acknowledge that small differences in the original single-seed ablations should still be interpreted with appropriate caution.

Reviewer_H214OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes WING, a framework for transferring interaction knowledge from egocentric human videos to robot manipulation policies. The method introduces two components: (1) an interaction-centric latent action model that disentangles camera motion from hand-object interaction dynamics, and (2) spectral latent guidance that extracts low-frequency components of latent action trajectories to guide robot action generation.

The authors evaluate the approach on three simulation benchmarks and four real-world manipulation tasks, reporting improvements over existing VLA and world-action models.

Critical strengths
  1. Interesting motivation and empirical observations. The paper identifies two relevant challenges in transferring human video knowledge to robot control: observer-motion interference and temporal discrepancies across embodiments. The empirical analysis showing stronger human–robot correspondence in low-frequency latent components is interesting and provides motivation for the proposed design.

  2. Extensive experimental evaluation. The paper provides comprehensive evaluations across simulated and real-world robotic manipulation tasks. The experiments include representation analysis, ablation studies, and comparisons with strong baselines. The overall experimental results are competitive and demonstrate the practical effectiveness of the proposed framework.

Critical weaknesses
  1. Limited methodological novelty. The proposed framework mainly combines existing techniques, including motion disentanglement, teacher–student distillation, and frequency-domain processing. Previous works have already investigated action-irrelevant information in latent action learning and disentangled motion representations for cross-embodiment transfer.

While the integration is reasonable, I find the fundamental methodological advancement somewhat limited. In particular, applying DCT to extract low-frequency latent dynamics is relatively straightforward. The paper does not sufficiently demonstrate why this specific spectral formulation is necessary compared with simpler temporal smoothing or filtering techniques.

  1. Insufficient justification for the additional complexity. The proposed framework introduces a relatively complex multi-stage training pipeline and additional computational overhead. However, the improvements over strong baselines on some benchmarks are relatively modest.

More importantly, it remains unclear whether the observed improvements specifically require a world-action-model architecture or whether similar benefits could be obtained by incorporating interaction-centric guidance into a simpler VLA policy. This limits the strength of the paper's broader methodological claims.

Questions for authors
  1. Can the authors compare the proposed DCT-based spectral guidance against simpler approaches such as temporal averaging, Gaussian smoothing, or other low-pass filters? This would help clarify whether the improvement comes from the proposed spectral representation or general temporal smoothing.

  2. Can the authors further clarify the key methodological differences from previous latent action disentanglement approaches, especially CD-LAM and motion-focused latent action learning? Which specific component contributes the most beyond existing techniques?

  3. Is the world-action-model architecture necessary for obtaining these improvements? Could the proposed interaction-centric spectral guidance be incorporated into a conventional VLA model without the additional video-generation component?

3 comments
AuthorsOpenReview ↗

Response to Reviewer H214 (1/3)

We are truly grateful to the reviewer for the time and effort devoted to reviewing our paper and for the valuable and insightful suggestions!

Q1&W1: Comparison with Temporal Smoothing Alternatives

We adopt DCT not merely for temporal smoothing, but for its fixed, orthogonal frequency basis, which provides a structured way to preserve transferable temporal dynamics. Specifically, retaining the first four DCT coefficients captures distinct low-frequency modes of interaction trajectories while suppressing high-frequency variations that are less consistent across embodiments. Unlike temporal averaging or Gaussian smoothing, which attenuate fluctuations without explicitly separating temporal frequency components, DCT preserves coarse temporal evolution through a compact set of non-redundant coefficients. Moreover, unlike PCA, which maximizes explained variance and may capture embodiment-specific variations, DCT explicitly selects temporal scales using a shared, data-independent basis for both human and robot trajectories. To further validate this motivation, we conducted controlled comparisons with temporal averaging (Avg), Gaussian smoothing (Gaussian), and full-trajectory PCA projection (PCA). All methods produce the same four 32-dimensional guidance target, with matched latent encoder, predictor capacity, policy initialization, training data, and optimization budget.

Table: Comparison of guidance representations on LIBERO (success rate %, ↑).

MethodSpatialObjectGoalLong
Ours (DCT)99.610099.296.2
Avg97.098.497.296.8
Gaussian94.699.096.892.2
PCA97.096.696.895.2

As shown above, DCT achieves the highest average success rate, outperforming Avg, Gaussian, and PCA by 1.40, 3.10, and 2.35 percentage points, respectively. Although Avg performs slightly better on LIBERO-Long, DCT consistently achieves stronger overall performance. Together with the stronger low-frequency cross-embodiment correspondence observed in Figure 3 and Appendix E.2, these results support our hypothesis that explicitly selecting shared temporal frequency components provides benefits beyond generic smoothing or dimensionality reduction.

AuthorsOpenReview ↗

Response to Reviewer H214 (2/3)

Q2&W1: Methodological Differences and Component Contributions

While prior work has investigated action-irrelevant information in latent action learning, WING differs in both its interaction-motion supervision and the temporal structures transferred across embodiments.

(1) Methodological Differences. CD-LAM reduces action-irrelevant biases through embodiment-centric reconstruction, action-centric contrastive learning, and latent calibration, while Motion-Focused separates action and background representations using mask-guided reconstruction and disentangled codebooks. In contrast, WING explicitly compensates for observer motion when constructing interaction-motion supervision. This distinction is important because foreground motion can still contain substantial camera-induced displacement, even after foreground–background separation. Specifically, WING estimates global camera motion from background trajectories and learns canonical interaction residuals after geometric compensation, together with a corresponding geometric reconstruction objective. The resulting interaction representations are further distilled into an RGB-only student. Beyond spatial disentanglement, WING additionally addresses human–robot temporal differences by selecting shared low-frequency interaction structures as guidance, rather than directly transferring full latent action trajectories.

(2) Component Contributions. To further identify the benefits of our geometric design beyond foreground-based motion disentanglement, we conducted additional controlled ablations with matched backbone, training data, latent dimensions, and optimization budget. We compare Mask-only, which separates foreground and background motion; Global + Raw, which additionally models global camera motion but retains raw foreground displacement; and WING Full, which introduces geometry-compensated interaction residuals with matched geometric reconstruction.

Table: Controlled comparison of motion modeling strategies across three egocentric datasets.

MetricDatasetMask-onlyGlobal + RawWING Full
Interaction R² ↑EgoDex0.270.270.48
EgoVerse0.700.710.76
Ego4D0.350.340.38
Camera sensitivity ↓EgoDex0.0350.0360.010
EgoVerse0.0550.0570.020
Ego4D0.0290.030.013

Camera sensitivity measures the mean dynamic-minus-static response across five perturbation levels; lower is better.

As shown above, Mask-only and Global + Raw achieve comparable results, indicating that introducing a separate camera branch alone provides limited additional benefit. In contrast, WING's geometry-compensated interaction modeling consistently improves interaction-motion readout across all three datasets, while reducing camera perturbation sensitivity. Importantly, the improved robustness is accompanied by stronger interaction-motion readout, rather than simply suppressing motion information. These results identify geometry-compensated interaction modeling as the most consequential addition among the evaluated motion-disentanglement strategies, providing benefits beyond foreground–background separation alone.

W2: Additional Complexity and Performance Gains

Although WING involves multiple training stages, most of the additional offline costs are incurred only once and can be amortized across downstream tasks. Moreover, WING leverages only 200 hours of egocentric video pretraining, whereas several recent foundation models rely on substantially larger embodied-data pretraining corpora, including π₀.₅ [1] (10k+ hours of robot data), LingBot-VA [2] (16k hours of robot manipulation data), LaWAM [3] (3k hours of robot videos and 1.5k hours of egocentric human videos), and GR00T-N1.7 [4] (20k hours of human videos alongside robot demonstrations).

Despite this relatively limited egocentric pretraining corpus, WING achieves 99.2% on LIBERO, 93.8% on RoboTwin 2.0, and 57.7% on RoboCasa–GR1, along with 75.0% success on real-world manipulation tasks. More directly, under the same 200-hour pretraining budget, WING improves real-world success from 60.0% with video-only pretraining to 75.0%, an absolute gain of 15 percentage points. These results demonstrate that the additional representation-learning stages yield meaningful downstream benefits, particularly in real-world manipulation, while their offline costs can be amortized across downstream tasks.

References

[1] Black K, Brown N, Darpinian J, et al. π₀.₅: a Vision-Language-Action Model with Open-World Generalization

[2] Li L, Zhang Q, Luo Y, et al. Causal World Modeling for Robot Control

[3] Chen J, Wang K, Chen K, et al. LaWAM: Latent World Action Models for Efficient Dynamics-Aware Robot Policies

[4] NVIDIA. NVIDIA Isaac GR00T N1.7: A Foundation Model for Generalist Robots

AuthorsOpenReview ↗

Response to Reviewer H214 (3/3)

Q3&W2: Is WAM Architecture Necessary?

We adopt WAM because our primary goal is to leverage action-free egocentric videos to learn transferable interaction dynamics for robot control. WAM provides a suitable framework for this purpose, as its video-pretraining pathway naturally supports learning dynamics from temporal visual observations. Our interaction-centric spectral guidance is specifically designed to complement this framework by extracting transferable interaction dynamics and incorporating them into action prediction. Under the same 200-hour egocentric pretraining budget, WING improves real-world success to 75.0%, demonstrating the effectiveness of our design within WAM.

Importantly, our spectral guidance is not inherently restricted to WAM or dependent on video generation. The guidance targets are constructed independently from latent trajectories using the pretrained WING-LAM encoder. Although our current implementation predicts these targets using WAM's visual features, the same guidance could potentially be incorporated into conventional VLAs through an auxiliary prediction objective and spectral-token conditioning, without requiring explicit video generation. We focus on WAM to validate the proposed approach in the setting of action-free video pretraining, while evaluating its generality across other policy architectures remains future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal Period

Dear Area Chair and Reviewers,

First, we would like to express our sincere gratitude for your time, careful reviews, insightful comments, and constructive suggestions!

Before Rebuttal

In summary, our paper has received positive initial assessments from all three reviewers, each of whom recommended acceptance (Ratings 3/3/3; confidence 3/3/3). Reviewer H214 highlighted the finding that low-frequency interaction dynamics exhibit stronger human–robot correspondence and concluded that the results “demonstrate the practical effectiveness of the proposed framework.” Reviewer zFPL praised the “Well-motivated interaction-centric latent action learning” and “Coherent integration with world-action modeling,” highlighting the complementary evidence from representation analyses, simulation benchmarks, and real-world manipulation. Reviewer B5jX described our method as “conceptually simple, intuitive, and well aligned with the underlying problem” and found that the experiments established “a particularly convincing connection between the proposed mechanisms and the final performance gains.”

Our Efforts during Rebuttal

During the rebuttal period, we carefully addressed the reviewers' concerns through additional experiments and clarifications:

  • We compared low-frequency spectral guidance with temporal averaging, Gaussian smoothing, and PCA at matched dimensionality. We also clarified target normalization and analyzed prediction difficulty and semantic preservation across frequency cutoffs. (H214@Q1, zFPL@Q2–Q3)
  • We clarified how WING-LAM differs from prior latent-action methods and added controlled ablations to isolate the contributions of geometric motion compensation, camera consistency, and representation regularization. (H214@Q2, zFPL@Q4)
  • We provided a stage-wise breakdown of computational costs, identified which components can be reused across downstream tasks, and compared WING with video-only pretraining under the same data budget. (H214@W2, zFPL@Q1)
  • We discussed why we use a world-action model for action-free video pretraining and how the same spectral guidance could be incorporated into a conventional VLA policy. (H214@Q3)
  • We clarified how the egocentric pretraining data were selected, including how camera motion was treated during selection, and discussed the scope for scaling to larger, less curated datasets. (B5jX@Q1)
  • We added multi-seed statistics for a controlled LIBERO setting and clarified the training-seed and rollout counts underlying the original policy evaluations. (zFPL@Q5)

Discussion and Feedback

We are grateful that these responses were positively received. All three reviewers explicitly maintained their positive assessments after rebuttal.

We will incorporate the additional experiments and clarifications into the final manuscript, including the matched-dimensional guidance comparisons, analyses of prediction difficulty and semantic preservation, controlled ablations of WING-LAM's motion decomposition, multi-seed evaluation results, and clearer discussions of data selection, computational costs, and the scope of the WAM implementation. We believe the reviews and additional evidence reinforce our central contribution.

Once again, thank you for your time and effort. We greatly appreciate your assistance in improving our manuscript!

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.