acceptodds
Withdrawn from ICLR 2027

LA-Control: A Control-Oriented Latent Action Model for Autonomous Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Latent Action Models (LAMs) learn representations of visual transitions from video, enabling imitation learning from videos without explicit action labels. For autonomous driving, however, representations learned through visual prediction may not retain the underlying control information. Visually salient states and scene changes can dominate the learned representation, while subtle control differences leave weak visual signatures. We propose LA-Control a latent action model tailored to driving control that grounds video-learned representations in vehicle state and state changes. Through physical supervision during causal future prediction, LA-Control allocates latent capacity to these complementary quantities while retaining room for other visual dynamics. This organization encourages control-relevant distinctions to remain accessible in the learned latent actions. Once trained, the model produces control-proxy targets for driving videos without action labels. We evaluate how readily small probes recover physical signals from these representations and how effectively the representations support downstream planner pretraining. Across two driving datasets and three planner architectures, LA-Control outperforms alternative latent-action models in physical readout and downstream open-loop trajectory prediction.

Primary area:
applications to robotics, autonomy, planning
Keywords:
latent action model, autonomous driving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9UiWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LA‑CONTROL is a structured latent‑action model for autonomous driving, which partitions latent space into dedicated subspaces for vehicle physical states, state‑change control signals and visual residuals, and applies physical supervision to preserve subtle control information with weak visual signatures. Evaluations on two driving datasets show improved physical signal readout and better open‑loop trajectory prediction for downstream planners.

Critical strengths

This Work identifies a core limitation of existing video‑driven latent‑action models: visually salient signals overwhelm subtle control signals. It further proposes the idea of structured latent‑space partitioning, assigning separate subspaces for physical states and state changes, offering a new direction for control‑representation learning from unlabeled videos.

Critical weaknesses

The experimental datasets rely heavily on the private Internal dataset; most key results are produced on this non‑public dataset. Considerable performance degradation is observed on nuScenes, with a substantial drop in R2R^2 for first‑order control quantities such as acceleration and curvature rate. The model yields limited performance on public datasets, and the paper’s conclusions are largely built upon private data, leading to poor reproducibility.

Only open‑loop simulation evaluation is conducted, with no closed‑loop validation whatsoever. Solely performing open‑loop trajectory prediction cannot verify the effectiveness of the learned latent actions in real interactive closed‑loop driving scenarios. The paper acknowledges the absence of closed‑loop testing, which constitutes a critical shortcoming for autonomous‑driving tasks.

The Subtle subset for minor‑correction cases is a post‑hoc filtered test subset. This corresponds to post‑hoc hypothesis testing and carries risks of p‑hacking. Conclusions drawn from this subset lack sufficient credibility and cannot serve as strong evidence to support the model’s advantages.

Questions for authors

See Weaknesses.

Reviewer_47U7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose LA-Control, a latent action model tailored to learning driving control through physical supervision during causal future prediction. As the results, LA-Control demonstrates SOTA performance across existing LAM variants under both nuscenes and an internal dataset in ADE/FDE.

Critical strengths
  • The authors propose LA-Control to resolve issues of LAM not picking up high order control signals from visual representations alone
  • The authors evaluate LA-Control on two driving dataset and achieve SOTA results across LAM variants
  • The authors demondtrates LA-Control can be applied to exisitng planners and performance gain transfers well
Critical weaknesses
  • The improvement margin of LA-Control compared to the naive auxillary methods seems to be small.
  • The evaluation only focuses on openloop setting, the authors should also include closed-loop evaluation to unveil the performance in a more realistic setting.
Questions for authors
  • What's the key perfomance gain between LA-Control vs control signal auxilary head? Since the subtle set is the place where key performance gain were observerd, but the authors also mentioned the set was curated later which raises concerns on whether it reflects true performance gap in drivng capability.
  • Did the authors conducted any evaluation in closed loop settings and whether the perfomance gain persist if so?
Reviewer_mW76OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes LA-CONTROL, which utilizes labeled states and videos to pretrain a latent action model. This model is used to supervise the pretraining of a downstream planner, which is subsequently fine-tuned with actual expert demonstrations. Overall, the direction is very interesting, as it explores effective ways to pretrain a planner without action labels. The main weakness of this paper is: the evaluation focuses on open-loop metrics (ADE/FDE) rather than closed-loop planning performance.

Critical strengths
  • The idea of utilizing physical quantity in latent embedding design is an interesting direction
  • The paper is well-written and the direction is important and have good potential of impact
  • How to extract knowledge utilizing videos with state information, the paper showed that 5 percent of action labels from (800k videos) can achieve similar quality as 100 percent.
Critical weaknesses
  • Evaluating on open-loop nuscenes, the open-loop eval is old, where open-loop prediction error is not informative for actual planning perfomance and I strongly suggest this work anchored on planning benchmarks such as Waymo E2E challenge or Nuplan.
  • Modest gain from strucutred latent to Naive aux latent. In addition, is 1st-order derivatives really necessary?
  • What does counterfactual mean in the first part of the supp video?
Questions for authors
  • Is the latent action aim to understand the dynamics of the real world(what actions lead to what states) or to learn driving behavior prior(how to plan in the realworld)
  • Follow up Q1, the setup in this work is more on the latter side, but the first direction is also promising direction. For example, how to utilize the Forward dynamics model to evaluate different planners given different actions. It would be informative to understand how different Latent Action Model perform in this direction.
  • I would not recommend the term "residual action", it usually implies correction on top of a prior, target = prior + res, can consider "aux latent" instead

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.