LA-Control: A Control-Oriented Latent Action Model for Autonomous Driving

Latent Action Models (LAMs) learn representations of visual transitions from video, enabling imitation learning from videos without explicit action labels. For autonomous driving, however, representations learned through visual prediction may not retain the underlying control information. Visually salient states and scene changes can dominate the learned representation, while subtle control differences leave weak visual signatures. We propose LA-Control a latent action model tailored to driving control that grounds video-learned representations in vehicle state and state changes. Through physical supervision during causal future prediction, LA-Control allocates latent capacity to these complementary quantities while retaining room for other visual dynamics. This organization encourages control-relevant distinctions to remain accessible in the learned latent actions. Once trained, the model produces control-proxy targets for driving videos without action labels. We evaluate how readily small probes recover physical signals from these representations and how effectively the representations support downstream planner pretraining. Across two driving datasets and three planner architectures, LA-Control outperforms alternative latent-action models in physical readout and downstream open-loop trajectory prediction.