- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes IG-VLA, built on ACoT-VLA. During training, a latent reasoning module predicts future SigLIP feature residuals to guide action generation. This pathway is then distilled into a compact Scene Gist vector predicted from the current context alone, so inference skips the future rollout. The paper evaluates on LIBERO, LIBERO-Plus, VLABench and one real UR3 task, and reports results on par with or slightly above ACoT-VLA.
Critical strengths
-
The idea of training with future imagination and deploying without it is practical and well motivated by deployment cost. The latency breakdown in Figure 4 makes clear where the time goes.
-
Predicting latent residuals instead of full future features is a sensible design. It focuses supervision on scene change and avoids pixel generation.
-
The evaluation covers three benchmarks plus a real robot test. The VLABench comparison with ACoT-VLA uses identical training data and schedule. The gist policy matches the reasoning policy almost exactly on all benchmarks, so the distillation step itself works.
Critical weaknesses
-
The paper lacks two baselines: the same backbone trained with identical data and steps but without the reasoning module, and a gist policy trained without the reasoning teacher. Without them, the gains over ACoT-VLA cannot be attributed to future reasoning rather than extra training, extra parameters, or the auxiliary loss acting as a regularizer.
-
The improvements are small. The margin over ACoT-VLA is 0.4 points on LIBERO and 0.5 points on the LIBERO-Plus average, and the VLABench progress score is lower than ACoT-VLA (46.11 vs 47.40). The one large gain, on the LIBERO-Plus Language suite (79.7 to 85.6), is left unexplained, and it is unclear why visual future prediction would help specifically under language perturbation.
-
The efficiency claim uses the wrong reference. Speedups are measured against the paper’s own reasoning policy, and against world action models whose latencies are copied from the ImageWAM paper rather than measured on the same stack. According to Figure 4, the gist policy is essentially the ACoT-VLA pipeline plus a 0.13 ms predictor.
-
The method does not match the motivation. The introduction describes imagining futures under candidate actions and choosing among their consequences. The module actually regresses a single deterministic future with no action conditioning and no selection. “Scene Gist Memory” also stores nothing across time or episodes.
-
Method details are missing or inconsistent. Several components are never described: the implicit action reasoner, the coarse and fine action modules, and the auxiliary pixel reconstruction head. Equation 2 and Algorithm 1 disagree on whether the policy receives the residual or the absolute latent.
-
In Figure 5, raising the cosine loss weight from 0.3 to 0.5 drops LIBERO-Spatial from 98.6 to 37.0, an implausibly sharp collapse that points to training instability and needs multiple seeds. Residual versus absolute latent prediction is not ablated.
-
The real-world evidence is weak. With one single-arm task and 16 trials per method, the reported 62.5, 56.2 and 50.0 differ by only one or two successes. The paper also does not say how the reasoning module was trained on the 77 demonstrations.
Questions for authors
-
What is the performance of the identical backbone without the reasoning module, and of a gist policy trained without the reasoning teacher, under the same training budget?
-
What are the latencies of ACoT-VLA and pi0.5 measured on your A6000 setup?
-
Is there direct evidence that the policy uses the predicted futures? For example, how does performance change when they are zeroed or replaced with futures from another episode?
-
Why does the gain concentrate on the LIBERO-Plus Language suite?
9 comments
[Additional WAM Latency Measurements]
We agree with the reviewer that the cross-paper speedup ratios in the submission used latencies quoted from the ImageWAM paper rather than measured on our stack, and that the 1,081 ms figure belonged to Fast-WAM-IDM rather than Fast-WAM itself. Below, we compare our paper's latency numbers (169.5 ms for Scene Gist Policy and 420.7 ms for Reasoning Policy, measured on a single NVIDIA RTX A6000 GPU with batch size 1 and BF16 precision) against same-GPU measurements of Fast-WAM and ImageWAM using the variant names from ImageWAM Table 11:
| Method | Implementation | Mean (ms) |
|---|---|---|
| π0.5 | Uncompiled (BF16) | 138.85 |
| ACoT-VLA | Uncompiled (BF16) | 171.58 |
| IG-VLA Scene Gist Policy (Paper) | Uncompiled (BF16) | 169.50 |
| IG-VLA Reasoning Policy (Paper) | Uncompiled (BF16) | 420.70 |
| Fast-WAM | Prefix Only | 180.60 |
| Fast-WAM | + Compiled | 89.30 |
| ImageWAM-FLUX.2 4B | Prefix Only | 225.60 |
| ImageWAM-FLUX.2 4B | + Action Loop Compile | 206.70 |
| ImageWAM-FLUX.2 4B | + Image Prefill Compile | 184.10 |
| ImageWAM-FLUX.2 4B | + Action Static Graph | 183.60 |
Measured on the same GPU, our uncompiled Scene Gist Policy (169.5 ms) is on par with ACoT-VLA (171.6 ms) and prefix-only Fast-WAM (180.6 ms), and faster than the tested ImageWAM configurations (183.6–225.6 ms), while being slower than compiled Fast-WAM (89.3 ms). We will therefore remove the 6.38×, 1.78×, and 1.55× cross-paper speedup claims from the abstract, introduction, and Table 4, and retain the internal speedup relative to our explicit Reasoning Policy (420.7 ms to 169.5 ms, 2.48× speedup).
[Clarification of Future Imagination and Scene Gist Memory]
We thank the reviewer for highlighting the need to clarify the connection between our motivation and implementation.
Future Imagination vs. Candidate-Action Evaluation. Our central motivation is to enable VLA policies to anticipate future environmental changes before generating actions, rather than relying solely on current observations and language instructions. IG-VLA realizes this through a latent spatiotemporal reasoning module that predicts a sequence of future visual representations conditioned on the current observation and task instruction. These predicted representations subsequently guide action generation.
Importantly, our method does not explicitly enumerate candidate actions or select among simulated outcomes. Instead, it learns a task-conditioned future trajectory that provides intermediate guidance for action prediction. This design avoids the additional computational overhead of multi-candidate future simulation while retaining the benefits of explicit future reasoning.
Scene Gist Memory vs. Episodic Memory. Scene Gist Memory is designed to internalize future reasoning knowledge into a compact representation, rather than maintain an external episodic memory bank. During training, the Scene Gist Compressor distills reasoning-enhanced representations into a compact Gist Token, and the Scene Gist Encoder learns to recover this representation directly from current observations and instructions.
Consequently, the learned Gist can provide future-informed guidance at inference without explicitly generating the full future trajectory. The term "memory" refers to this learned, parameterized internalization of future reasoning knowledge, not the storage or retrieval of historical episodes. We will clarify this distinction in the revised manuscript.
[Clarification of Method Components and Future Latent Representations]
We thank the reviewer for identifying these missing details and the inconsistency between Eq. (2) and Algorithm 1. We will revise the method and implementation sections to describe all components explicitly.
Our architecture is built directly on the ACoT-VLA action backbone. The coarse action module, also referred to as the Explicit Action Reasoner (EAR), uses a flow-matching head to generate a coarse action trajectory that serves as an explicit action-level plan. The Implicit Action Reasoner (IAR) extracts a compact latent action prior from the intermediate VLM representations. The fine action module is the final flow-matching action expert, which predicts the executable action chunk conditioned on the visual-language context and the coarse and implicit action-reasoning signals. These components are inherited from ACoT-VLA. Our contribution is the future-reasoning module and its distillation into the Gist policy. We will make this architectural dependency and the roles of EAR and IAR explicit.
The auxiliary pixel reconstruction head is used only during Stage-1 future-feature training. It encourages the predicted future representation to retain spatially meaningful visual information by reconstructing the corresponding future RGB observation. This auxiliary head is not used during action fine-tuning or inference and therefore introduces no inference-time overhead.
Clarification of the future latent representation. The action policy directly consumes the predicted future latent residuals ), rather than reconstructed absolute future latents. These residuals represent the anticipated changes in the visual scene and provide future-aware guidance for action generation. We apologize for the inconsistent notation between Eq. (2) and Algorithm 1. We will carefully check the entire manuscript and revise the equations, algorithm, figures, and accompanying text so that they consistently reflect the actual implementation.
[Training Stability and Residual vs. Absolute Future Prediction Ablation]
We thank the reviewer for raising this concern. We agree that the sharp decrease in LIBERO-Spatial success at $\lambda_{\mathrm{dir}}=0.5$ warrants an examination of training robustness.
We repeated the Stage-1 training with two additional random seeds, keeping the initialization, data, batch size, optimizer, and training schedule unchanged. All three runs completed 20k training steps, and their loss curves closely overlap throughout training. The following table reports the loss metrics from the completed 20k-step runs:
| Seed | Total Loss | Latent MSE | Cosine Loss | Cosine Similarity (k=10) | Reconstruction Loss | Predicted Transition Norm |
|---|---|---|---|---|---|---|
| 42 | 0.1605 | 0.0274 | 0.2459 | 0.8084 | 0.0526 | 8.4618 |
| 43 | 0.1559 | 0.0268 | 0.2377 | 0.8185 | 0.0522 | 8.6714 |
| 44 | 0.1578 | 0.0265 | 0.2449 | 0.8101 | 0.0518 | 8.5736 |
The three runs converge to nearly identical total losses, latent-regression losses, cosine alignment, reconstruction losses, and predicted transition norms. We observe no loss divergence, exploding gradients, or latent-scale drift across the three seeds. These results indicate that the setting does not cause anomalous or seed-specific instability during Stage-1 future-representation training.
Due to the computational cost of completing the subsequent Stage-2 action training and full LIBERO rollout evaluation, we were unable to finish end-to-end policy evaluations for all three seeds within the rebuttal period. We therefore report the completed multi-seed Stage-1 analysis as an initial robustness check. We will complete the matched Stage-2 training and downstream evaluation for the additional seeds and report the resulting mean and variance in the revised paper.
Regarding residual versus absolute latent prediction, our method predicts the latent transition ), rather than directly predicting the absolute future latent ). This residual parameterization removes much of the static observation content and focuses the prediction objective on task-relevant temporal changes. This is the design motivation, rather than evidence of empirical superiority over absolute latent prediction. We agree that the original submission did not include a complete matched residual-versus-absolute downstream ablation. We will add this full ablation in the revised version.
[Controlled Ablations to Isolate the Contribution of Future Reasoning]
We thank the reviewer for requesting these controls. We compare the following variants under the same π0.5/ACoT-VLA backbone and initialization. The matched ACoT-VLA control omits future-feature prediction, latent future reasoning, and Gist distillation. The auxiliary-future-prediction-only control retains future-feature prediction training but removes future/Gist inputs from the action path. The Gist Policy without the reasoning teacher uses dataset ground-truth actions for supervision.
Against the 97.00% matched ACoT-VLA control, the full Scene Gist Policy reaches 98.90% average success (+1.90 points) and improves LIBERO-Long from 93.00% to 97.80% (+4.80 points). Compared with auxiliary future prediction only, the gains are +1.50 points on average and +3.60 points on LIBERO-Long. Reasoning-teacher distillation further improves over direct ground-truth action supervision by +0.90 points on average and +2.00 points on LIBERO-Long.
[About Performance Gains]
While the improvements on the near-saturated LIBERO benchmark are modest, IG-VLA achieves substantial gains on challenging instruction-related evaluations. Specifically, it improves success rates on LIBERO-Plus Language from 79.7% to 85.6% (+5.9 points). On VLABench Semantic-Instruction, IG-VLA improves both Intention Score (56.8 to 65.2, +8.4 points) and Progress Score (39.6 to 48.49, +8.89 points). Moreover, IG-VLA achieves a higher overall VLABench Intention Score (66.68 vs. 63.50), demonstrating stronger performance in instruction-sensitive manipulation scenarios.
While the overall VLABench Progress Score is slightly lower (46.11 vs. 47.40) due to minor trade-offs on standard non-paraphrased subtasks, IG-VLA achieves higher overall Intention Score (66.68 vs. 63.50) and large gains on Semantic-Instruction (+8.40 Intention, +8.89 Progress), showing clear advantages when accurate instruction grounding is required.
[Language Perturbation Robustness]
The improvement on LIBERO-Plus Language (+5.9 points) can be explained by the task-grounded guidance provided by future visual reasoning. Prior work [1] demonstrates that aligning language with desired visual changes supports generalization to unseen instructions.
Moreover, [2] finds that 80–96% of failures under language paraphrasing arise from planning-level trajectory divergence rather than low-level execution errors. This finding highlights the importance of accurately understanding task goals and grounding them in the physical environment.
Motivated by these findings, IG-VLA predicts fine-grained future visual representations to guide action generation, potentially reducing sensitivity to surface-level linguistic variations while preserving task-relevant scene dynamics. This provides an explanation for the larger improvement on the Language suite.
[1] Myers V, He A W, Fang K, et al. Goal representations for instruction following: A semi-supervised language interface to control. Conference on Robot Learning, PMLR, 2023: 3894–3908.
[2] Kim et al. LIBERO-Para: A Diagnostic Benchmark and Metrics for Paraphrase Robustness in VLA Models. EMNLP 2026.