Equivariant Counterfactual Supervision for Hidden-Dynamics Representation Learning

We introduce equivariant counterfactual supervision for learning physical context from a robot's interaction history. The method compares simulated responses under controlled changes to an environmental property, using comparison rules appropriate to material and geometric changes. Subtracting the mean response across the compared conditions produces residual targets. A reservoir feature bank summarizes the interaction history, and a learned code predicts these targets at a query from another rollout in the same environment. We evaluate what physical information the code makes accessible and how useful it is to new prediction tasks. Relative to Full-transition supervision, Residual raises steepness linear-probe $R^2$ from 0.555 to 0.907 and improves mean exact-cell retrieval in all ten factor-level holdouts. A paired target-centering ablation corroborates the accessibility difference under matched heads and scales. Fresh predictors show a modest in-distribution full-transition benefit, whereas Full has lower mean error on all four held-out factor-response tasks. The results establish a supervision-dependent accessibility effect whose predictive value depends on the receiving task.