acceptodds
Withdrawn from ICLR 2027

InterACT: Grounding Latent Actions through Camera–Physics Interventions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Latent action models learn transitions from video, but visual changes reflect both physical interaction and camera motion. Temporal consistency does not guarantee camera invariance. We introduce , which adapts a pretrained latent action model through crossed camera and physical interventions. Each quartet pairs two interactions with two camera trajectories. A bounded residual aligns views, a frozen reference anchors physical distance scale, and separate gradient paths train the correction and encoder. achieves lower camera ratios and action errors and higher outcome gains than video continuation and an existing ego-supervised adaptation across three controlled domains. On unseen camera trajectories, it lowers the camera-to-physical distance ratio from to while retaining near-reference physical scale, and reduces outcome prediction error by relative to a state-only probe. Controlled cross-view Recall@5 increases from to . A separate comparison with a supervised baseline reveals a trade-off: the full method better preserves reference scale, while supervised adaptation yields higher outcome gains. For downstream policy learning, the frozen tokenizer yields success with on MetaWorld MT50 and with ABot-M0.5 on RoboCasa365 atomic-seen tasks, exceeding the reported references of and , respectively.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Representation Learning, Latent Action

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cXEPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the confounding between physical interactions and camera motion in latent action models. The authors propose InterACT, which constructs crossed camera–physics intervention quartets in simulation by varying camera trajectories while holding the physical interaction fixed, and varying physical trajectories while holding the camera fixed, thereby explicitly separating camera-induced variation from genuine physical changes. Methodologically, InterACT builds on a pretrained latent-action encoder and introduces a constrained residual correction module, while using a frozen copy of the pretrained encoder as a reference for preserving physical scale. The model is adapted through camera alignment, physical-scale preservation, action and outcome supervision, and separated gradient paths. Experiments show that InterACT substantially reduces camera-related representation variation while preserving physical interaction information under controlled ID, Camera-OOD, and Physics-OOD settings, and also improves downstream policy learning performance.

Critical strengths

The problem formulation is valuable, and the crossed camera–physics intervention design is clear and effective. The paper correctly identifies the confounding between camera motion and physical interaction in latent action representations. By constructing paired interventions with the same physical interaction under different cameras and different physical interactions under the same camera, the method provides direct supervision for learning representations that are invariant to camera motion while remaining sensitive to physical changes. This design is intuitive, interpretable, and constitutes the main novelty of the paper.

The representation-learning objective and evaluation protocol are carefully designed to balance nuisance suppression and preservation of task-relevant information. The method not only minimizes camera-induced variation, but also uses a frozen reference and a physical-scale constraint to prevent artificially improving alignment by expanding or shrinking physical distances. In addition, action error and outcome gain are used to verify that interaction-relevant information is still retained in the learned representation.

Critical weaknesses

The reliance on synthetic intervention data limits the generality of the approach, and the evidence for transfer to real-world settings is insufficient. InterACT relies on controllable camera–physics interventions generated in simulation, whereas camera motion in real videos is substantially more complex. In addition, existing approaches can suppress camera-related nuisance factors implicitly through large-scale real-video training and view-robust representation learning. The current experiments are mainly built on ALAM, whose representation is relatively camera-sensitive, and the paper does not demonstrate whether InterACT still provides substantial gains when applied to stronger camera-robust latent-action models. More importantly, the improvement in controlled retrieval does not transfer to CALVIN: R@5 remains at 2.73%, while real-video reconstruction performance slightly degrades. This leaves the effectiveness and necessity of the proposed intervention-based approach in real-world settings insufficiently established.

The experiments do not sufficiently disentangle the contribution of the intervention data itself from that of the more complex training framework. The full InterACT model jointly uses crossed quartets, a frozen reference, scale preservation, gradient routing, action/outcome supervision, and several auxiliary losses, but it lacks a direct comparison against a simple contrastive or alignment baseline trained on exactly the same intervention data. Moreover, the key pairing and routing ablations are conducted mainly on the action-bottleneck variant rather than on the full model. As a result, it remains unclear whether the observed gains primarily come from the proposed methodological design or simply from access to additional high-quality controlled supervision, which weakens the evidence for the algorithmic contribution and novelty of the paper.

Questions for authors

Does InterACT remain effective when applied to a stronger camera-robust baseline? The current experiments mainly build on ALAM, whose Camera-OOD performance shows substantial camera sensitivity, with Rcp=3.08R_{cp}=3.08. Could the authors clarify or provide additional evidence on whether InterACT still brings significant and complementary improvements when applied to latent-action models that have already acquired strong camera robustness through large-scale real-video training or explicit view-/camera-invariant objectives?

Does the invariance learned from synthetic interventions genuinely transfer to real videos? Controlled retrieval improves substantially, with R@5 increasing from 3.39% to 29.43%, whereas R@5 on CALVIN remains at 2.73%, and real-video reconstruction performance slightly degrades. How do the authors explain this clear discrepancy between the synthetic controlled setting and the real-video setting?

Do the gains primarily come from the intervention data itself or from the specific InterACT training design? The full InterACT model combines crossed pairing, a frozen reference, scale preservation, gradient routing, action/outcome supervision, and several auxiliary losses. Could the authors compare the full method against a simple contrastive or same-interaction alignment baseline trained on exactly the same intervention quartets?

Reviewer_xeU1OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces InterACT, a framework for adapting pretrained latent action models to reduce camera dependence while retaining information about physical interactions. The method constructs simulator-generated quartets by crossing two physical trajectories with two camera trajectories. It combines a bounded residual correction, frozen-reference constraints on physical representation geometry, and separate gradient paths for camera alignment and physical supervision. The adapted tokenizer supplies continuous latent targets for downstream policy learning. The experiments demonstrate substantial improvements in camera alignment, action prediction, and cross-view retrieval on controlled transitions relative to pretrained ALAM, video continuation, and ego-supervised adaptation. The paper also reports improved policy success on MetaWorld MT50 and RoboCasa365. However, the practical benefit of preserving reference scale, the contribution of the individual components in the main model, and the generalization of the controlled improvements remain insufficiently established.

Critical strengths
  1. The paper addresses camera–interaction entanglement in latent action learning through a well-motivated crossed-intervention design.
  2. InterACT reduces Camera-OOD action error from 1.44 to 0.34 and improves controlled cross-view Recall@5 from 3.39% to 29.43%.
  3. he paper jointly assesses alignment, scale retention, and predictive information, while explicitly reporting negative results and limitations.
Critical weaknesses
  1. All downstream control experiments are conducted in simulation, and the controlled camera shifts include translation but no rotation. Moreover, CALVIN cross-view retrieval shows no improvement in Recall@5. These results leave unclear whether the learned invariance transfers to realistic viewpoint changes and real-robot control.
  2. The practical benefit of reference-scale preservation is unclear. Preserving distances from a camera-sensitive pretrained encoder does not necessarily preserve physically meaningful information. At the matched 975-update budget, the supervised baseline achieves higher outcome gains in all three domains despite substantial scale expansion. The paper therefore demonstrates scale retention more clearly than its benefit for prediction or control.
  3. Routing and pairing ablations use an action-bottleneck variant trained for 975 updates, whereas the main results use a different outcome predictor and 3.9k updates. The pairing evidence is also mixed: correct pairing slightly improves alignment and action error but yields substantially worse outcome gain than shuffled pairing. The contributions of these components to the final model remain insufficiently isolated.
Questions for authors
  1. Can the authors evaluate real-robot control under unseen camera viewpoints, or at least provide closed-loop simulation results under camera rotations and broader viewpoint changes? What explains the absence of improvement in CALVIN cross-view retrieval?
  2. Can the authors compare InterACT with supervised adaptation and a simple scale-regularized baseline at the full training budget, including downstream control, to establish the practical value of preserving reference scale?
  3. Can the routing and pairing ablations be repeated using the final architecture and training budget? Does the better outcome gain under shuffled pairing persist, and how should this result be interpreted?
Reviewer_6DqYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

InterACT addresses camera-motion contamination in latent action representations. Starting from pretrained ALAM, it constructs crossed camera–physics interventions and adapts the encoder using a bounded residual, frozen-reference geometry constraints, gradient routing, and action supervision. Experiments show substantial improvements in controlled camera-invariance metrics and report higher downstream policy success on MetaWorld and RoboCasa.

Critical strengths
  • Well-motivated problem and intervention design. Camera motion is an important confounder in latent action learning. The crossed camera–physics interventions provide a clear way to distinguish viewpoint changes from physical interactions, while adapting pretrained representations is a sensible direction.

  • Careful representation-level evaluation. The paper jointly examines camera invariance, physical-scale preservation, and action prediction. The controlled improvements are substantial, and the supervised comparison reveals important trade-offs between representation geometry and predictive performance.

Critical weaknesses
  • The method's core design choices lack convincing justification.

    • Physical reference. InterACT uses frozen ALAM latent distances to preserve physical action structure, despite ALAM's strong camera sensitivity and poor action prediction (Table 1). It is unclear whether these distances meaningfully reflect physical differences. Moreover, the paper provides no evidence that preserving this pretrained geometry improves downstream control.
    • Intervention pairing. Shuffling camera–interaction correspondences barely affects camera ratio or action error, while yielding a less negative outcome gain (Table 4). This raises questions about the contribution of correct pairing.
    • Complexity and ablations. The method combines numerous constraints and auxiliary objectives without systematic full-model ablations. Routing and pairing are tested only in a separate action-bottleneck variant. Comparisons against simpler supervised cross-view alignment under matched supervision are missing.
  • Limited generalization and insufficient downstream validation.

    • Generalization. Improvements are largely confined to synthetic interactions with translation-only cameras. CALVIN retrieval does not improve, and real-video reconstruction degrades (Table 2).
    • Downstream control. MetaWorld and RoboCasa gains are compared against previously reported results rather than matched tokenizer-only reruns. Neither experiment directly evaluates policy robustness under camera shifts, despite this being the central motivation.
    • Scope of invariance. Camera motion may carry action-relevant information in egocentric and mobile settings. The method does not distinguish nuisance camera motion from agent-induced viewpoint changes, leaving its broader applicability unclear.
Questions for authors
  1. Physical reference. Why should frozen ALAM geometry be trusted as a physical reference, given its camera sensitivity and poor action prediction? What evidence shows that preserving this geometry is beneficial, especially when supervised adaptation achieves stronger outcome prediction?
  2. Method necessity. Given access to simulator action and outcome labels, does InterACT outperform simpler supervised cross-view alignment under matched supervision? also, ablation studies of the reference constraints, residual correction, and gradient routing in the full model would help clarify the contribution of each component.
  3. Generalization. Why do the gains on controlled synthetic data not transfer to CALVIN retrieval, and why does broader intervention coverage worsen performance? Can the method be evaluated on realistic camera variation, including rotation, and on human egocentric video, where camera motion is coupled with physical action and invariance may remove action-relevant information?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.