InterACT: Grounding Latent Actions through Camera–Physics Interventions

Latent action models learn transitions from video, but visual changes reflect both physical interaction and camera motion. Temporal consistency does not guarantee camera invariance. We introduce , which adapts a pretrained latent action model through crossed camera and physical interventions. Each quartet pairs two interactions with two camera trajectories. A bounded residual aligns views, a frozen reference anchors physical distance scale, and separate gradient paths train the correction and encoder. achieves lower camera ratios and action errors and higher outcome gains than video continuation and an existing ego-supervised adaptation across three controlled domains. On unseen camera trajectories, it lowers the camera-to-physical distance ratio from $3.08$ to $0.71$ while retaining near-reference physical scale, and reduces outcome prediction error by $32.15\%$ relative to a state-only probe. Controlled cross-view Recall@5 increases from $3.39\%$ to $29.43\%$. A separate comparison with a supervised baseline reveals a trade-off: the full method better preserves reference scale, while supervised adaptation yields higher outcome gains. For downstream policy learning, the frozen tokenizer yields $88.6\%$ success with ${\pi }_0$ on MetaWorld MT50 and $76.1\%$ with ABot-M0.5 on RoboCasa365 atomic-seen tasks, exceeding the reported references of $85.0\%$ and $70.6\%$, respectively.