acceptodds
Withdrawn from ICLR 2027

What Should Robot Policies Learn to Predict? Rethinking Supervision Targets in Vision-Language-Action Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) policies are typically trained with action prediction as the primary supervision signal, since robots ultimately execute actions. Auxiliary objectives such as latent-action (LA) and future-vision (FV) prediction are often added to improve policy learning, yet they compete with action prediction for the same training budget. We ask whether, under a fixed budget, allocating a substantial share of training samples to such objectives, rather than to further action prediction, can improve policy learning. We build a unified VLA framework that couples a two-stream multimodal diffusion transformer (MMDiT) with a sample-level router, which assigns each training sample to either action prediction or an auxiliary objective and thus makes supervision allocation explicit. Within this framework, we instantiate three LA targets based on LAPA and its DINO-feature variants, and three FV targets based on DINOv3, V-JEPA 2, and Wan2.2 representations. Across LIBERO, CALVIN, and four real-robot tasks, suitable LA and FV supervision outperforms action-only training under the same budget, even when action prediction receives no more than half of the training samples. FV generally outperforms LA, although the strongest LA target remains competitive on CALVIN. Diagnostic analyses suggest that DINO-based LA targets emphasize relative motion, whereas FV targets emphasize absolute position and task identity. We therefore argue that what robot policies learn to predict, and how training is allocated among these targets, should be treated as explicit design choices rather than defaulting to action-dominant training.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, robot learning, auxiliary supervision, latent actions, future prediction, supervision allocation, imitation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xZcnOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how training samples should be allocated between direct action prediction and auxiliary objectives in VLA learning. A two-stream MMDiT with sample-level routing enables comparisons of three latent-action (LA) targets and three future-vision (FV) targets under fixed policy-update and sample-exposure budgets. Experiments on LIBERO, CALVIN, and four real-robot tasks show that suitable auxiliary supervision can improve policy performance even when direct action prediction receives half or fewer training samples. FV generally performs better among the tested configurations, while the strongest LA target remains competitive on CALVIN.

Critical strengths
  1. The paper addresses a useful training-design question with substantial controlled evidence. The source-matched FV controls and allocation–budget sweeps strengthen the finding that direct action prediction need not dominate training. For example, V-JEPA 2 at P25 on CALVIN exceeds both its matched P100 control and the shared action-only baseline; Table 6 reports an SR-5 of 80.5% versus 71.6% for the latter. This provides a concrete and practically relevant finding about how predictive supervision can benefit action execution.
  2. The experimental coverage is extensive and the conclusions are appropriately qualified. The study compares six target configurations across two simulation benchmarks and four real-robot tasks, with additional loss-weight ablations and representation analyses. It also reports settings where auxiliary supervision hurts and where LA remains competitive with FV. This breadth makes the comparison informative beyond a single favorable configuration.
Critical weaknesses
  1. Training-level variability remains unclear. Table 6 reports multiple evaluation seeds, but independent training-seed results are not reported. Some source-matched LIBERO improvements are relatively small, and fine-grained target rankings vary across checkpoints or evaluation seeds. Repeating a small subset of the key comparisons would strengthen confidence in the reported rankings and preferred allocations.
  2. The real-robot results do not fully isolate the contribution of FV supervision. Unlike the simulation experiments, Table 15 does not include source-matched FV P100 controls. Since FV configurations also receive additional current-frame features, the real-robot gains over the shared baseline may reflect both improved visual inputs and auxiliary supervision. This limits the attribution of those gains, although the matched simulation comparisons provide direct evidence for the supervision effect.
Questions for authors
  1. Were the key comparisons repeated with independent training seeds? If available, please report training-level variability for V-JEPA 2 P25 on CALVIN and the strongest P50 FV configurations on LIBERO.
  2. Are source-matched FV P100 results available for the real-robot tasks? These would help determine how much of the improvement comes from auxiliary supervision beyond the additional current-frame features.
Reviewer_CV8iOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper questions whether the action prediction should take the majority of the training process. To analyze this, it trains a MMDiT on top of a VLM where it works with an action expert and an auxiliary expert per samples. Within a fixed training samples, it uses a sample-level router to use only one of the two objectives, either an action prediction or an auxiliary target. The paper compares each three latent action (LA) and future vision (FV) on LIBERO, CALVIN, and 4 real robot tasks.

Critical strengths
  • The paper throws an interesting question to whether predicting how much portion to include auxiliary predictions in the latent state would have a better effect of just predicting actions.
  • The router turns the proportion of the action supervision within the fixed training budget into a clean independent variable, which makes easier comparison for different levels of action supervision.
Critical weaknesses
  • FV configurations receive frozen-encoder features (f_t) as extra input, so gains must be separated from this input effect. In LIBERO, the input-only control (97.55) already beats action-only (96.50). Also, there’s also no source-matched controls reported in real robot experiments.
  • The main claims rely on post-hoc selection from a large configuration grid evaluated directly on test benchmarks, with single-seed LIBERO results and ~1 pp margins. Neither LA nor FV consistently beats action-only (e.g., no LA target on LIBERO, Wan2.2 on CALVIN), and the paper offers no a prior criterion for choosing the target or allocation. The finding is therefore closer to an existence result than a reliable recipe.
  • Some settings collapse, and every LIBERO P25 configuration underperforms its control. DINOv3 FV is below its source-matched control at every LIBERO checkpoint in Table 8. Several claims, such as the LIBERO ranking, “FV > LA” may therefore not be robust.
  • The paper’s contribution is sample-level routing, yet it never compares against the standard approach in Fig. 1, which is using action + λ·auxiliary loss on every sample at the same number of updates. Without this, you cannot tell whether routing matters or whether any gain simply comes from the presence of the auxiliary signal, which prior work (Lin et al. 2026; FLARE - Zheng et al., 2025) already reports.
  • The novelty is limited. The architecture follows existing VLA designs of VLM + flow-matching action-expert, and all six targets are off-the-shelf encoders or quantizers. The FV route is similar to action-conditioned future prediction in UWM (Zhu et al., 2025)/FLARE (Zheng et al., 2025), and sample-level routing is essentially objective sampling.
Questions for authors
  • Can the authors report Wan2.2/V-JEPA 2 P100 controls on the real robot to separate the effect of the FV loss from that of the extra input?
  • Why do the source-matched FV P100 controls underperform the shared action-only baseline on CALVIN? How large are the gains if each FV run is compared only against the shared baseline across all allocations?
  • How much of the FV gain comes from conditioning on clean demonstrated actions, for instance, world-model-style learning, versus predicting the future itself?
Reviewer_ANkiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how training samples should be allocated between direct action prediction and auxiliary objectives in vision-language-action learning. The authors introduce a two-stream MMDiT framework with sample-level routing and compare three latent-action (LA) targets and three future-vision (FV) targets. Experiments on LIBERO, CALVIN, and four real-robot tasks show that suitable auxiliary objectives can improve performance even when direct action prediction receives half or fewer of the training samples. Frozen-representation probes and attention visualizations provide additional descriptive analyses.

Critical strengths
  1. Relevant research question. Explicitly studying supervision allocation under a fixed sample-exposure budget is useful for VLA training.
  2. Broad evaluation. The paper covers multiple targets, routing ratios, training checkpoints, simulation benchmarks, and real-robot tasks, including negative results.
  3. Useful matched controls. Source-matched FV controls strengthen the within-configuration comparisons, particularly on CALVIN.
Critical weaknesses
  1. LA–FV comparisons remain confounded. The configurations differ in current visual inputs, clean-action conditioning, encoder checkpoints, objective types, and temporal coverage. The results establish differences between complete configurations rather than isolate the effect of supervision target.
  2. Routing is not sufficiently separated from loss weighting. Changing the routing probability also changes relative gradient contributions. A comparison against appropriately weighted joint training would clarify what routing contributes beyond conventional auxiliary-loss tuning.
  3. Real-robot attribution is incomplete. The real-robot experiments lack source-matched FV P100 controls and trial-level uncertainty estimates, preventing clear attribution of improvements to auxiliary supervision.
  4. Mechanistic evidence is limited. Frozen-source probes and a small number of attention examples are descriptive and do not establish why auxiliary prediction improves the learned policy.
Questions for authors
  1. Does FV retain its advantage when LA and FV use the same visual backbone, current-feature inputs, action conditioning, and temporal coverage?

  2. How does routed training compare with joint action-and-auxiliary training using matched expected loss weights and the corresponding route-specific input constructions?

  3. What is the performance of the strongest FV configuration’s source-matched P100 control on the real robot?

  4. The shared LA P100 baseline uses 32 queries, whereas the LAPA-DINO configurations use 128. Since these queries participate in joint attention, would query-count-matched P100 controls change the findings?

  5. Do target-specific validity masks produce different eligible training anchors across LA and FV? If so, how is this handled in the controlled comparisons?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.