acceptodds
Under review as a conference paper at ICLR 2027

Latent Lookahead for Action Conditioned Predictive Structure in Frozen Vision Language Models

Abstract

Pretrained vision–language models (VLMs) are increasingly reused as internal representations for multimodal agents, yet it remains unclear whether states optimized for understanding preserve information about how visual evidence changes after an action. We ask whether question- and history-conditioned representations from a frozen VLM contain recoverable, local action-conditioned predictive structure. Latent Lookahead uses a shallow readout and a low-capacity transition model to predict future latent states, then ranks candidate actions through short latent rollouts while leaving the backbone fixed. Across three VLMs and image, document, and video decision environments, learned transitions outperform persistence and action-independent predictors. The advantage persists over short open-loop rollouts, but weakens with horizon, policy-induced distribution shift, and poorer training coverage. From identical decision states, predicted futures increase best-realized-action match from 55.9% to 64.3% and reduce action regret from 0.0283 to 0.0146 relative to direct scoring. Latent Lookahead also improves task performance by about two points on average with roughly 30% additional compute and no extra backbone calls per real interaction step. These results show that local predictive information can be recovered from frozen VLM representations and used for short-horizon visual evidence selection.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Predictive Representations, Vision-Language Models, Latent World Models, Test-Time Control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BRhDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Latent Lookahead, a lightweight framework for recovering action-conditioned predictive information from frozen vision-language models (VLMs). The method extracts compact latent states through mean pooling and linear projection, trains an MLP-based transition model to predict future latent states, and uses short-horizon rollouts with value estimation and local prediction-error penalties for action selection. Experiments are conducted on three visual decision environments (GQA-PatchEnv, SlideVQA-Env, and NExT-QA-Env) using three frozen VLM backbones. The results show improved latent prediction accuracy over simple baselines and modest downstream performance gains over reactive policies and direct action scoring, without additional VLM backbone calls for simulated transitions.

Critical strengths
  1. Practical and computationally efficient framework. The method keeps the VLM backbone frozen and performs future-state prediction in a compact latent space, avoiding additional backbone calls during lookahead. This is computationally appealing for visual decision-making tasks.

  2. Evaluation across multiple VLMs and environments. The paper evaluates three VLM backbones across image, document, and video decision environments. The results consistently show improvements over reactive policies, suggesting that frozen VLM representations retain some useful action-conditioned predictive information.

Critical weaknesses
  1. Limited methodological novelty and overly simplified planning.

The proposed method mainly combines mean pooling, a linear projection, a small MLP transition model, and value-based action selection. The technical contribution appears incremental. More importantly, the latent rollouts repeatedly apply the same action without considering mixed-action sequences or branching. This substantially restricts the planning space and makes it unclear whether the method provides meaningful planning capabilities beyond simple action scoring.

  1. Insufficient experimental validation and weak comparisons.

The experimental comparisons are not strong enough to establish the effectiveness of the proposed approach. In particular, the paper lacks comparisons with stronger lightweight model-based baselines, such as ensemble dynamics, mixed-action latent beam search, or simple Q-learning using the same representations. The improvements over direct action scoring are also relatively small (0.45–0.80 points). Furthermore, important representation choices, including pooling strategies, token selection, and latent dimensionality, are not sufficiently investigated. These omissions make it difficult to determine whether the additional complexity of latent lookahead is justified.

Questions for authors

Q1. Representation design.

Why is mean pooling over top-layer tokens chosen as the latent readout? Have the authors compared it with attention pooling, selected visual tokens, or multi-token representations? How sensitive are the results to these choices?

Q2. Planning limitations.

Why does the proposed lookahead repeatedly apply the same action rather than considering mixed-action sequences? Can the authors compare the current approach with a simple branching or beam-search planner in the same latent space under a matched computational budget?

Q3. Stronger baselines.

How does Latent Lookahead compare with simple Q-learning or an ensemble-based dynamics model using the same frozen VLM representations and training data? Without these comparisons, how can the improvements be attributed specifically to the proposed planning mechanism?

Q4. Practical significance.

The improvements over direct action scoring are relatively small (0.45–0.80 points). Can the authors provide stronger evidence that these gains justify the additional transition model, value head, and uncertainty-related computation, particularly under matched computational budgets?

Reviewer_4jCoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Latent Lookahead for visual evidence selection using frozen VLMs. A trainable readout, action-conditioned dynamics model, and value head support short latent rollouts to select observation actions. Experiments across image, document, and video environments show consistent improvements over reactive policies and smaller gains over a direct action scorer.

Critical strengths
  1. The approach is lightweight and keeps the VLM frozen, without additional backbone evaluations for imagined transitions.
  2. The evaluation includes useful controls, particularly the direct scorer and same-origin action comparisons with a shared continuation policy, which help isolate the contribution of predictive action selection.
Critical weaknesses
  1. The gradient treatment of the target latent is unclear. Since the shared readout is jointly trained with the dynamics model, reduced prediction error could partly reflect representational degeneration rather than improved prediction. The paper does not sufficiently explain or verify how this is avoided.
  2. The accuracy–latency trade-off lacks sufficient context. Roughly 20% additional latency yields around 1–2 points of improvement over the reactive baseline, with smaller gains over direct scoring. Comparisons with relevant methods at similar computational budgets are needed to establish whether this trade-off is competitive.
Questions for authors
  1. Is stop-gradient applied to the target latent branch? How do you prevent representational degeneration and verify that lower prediction error reflects meaningful predictive ability?
  2. Is the reported accuracy–latency trade-off competitive? Could related work or budget-matched baselines clarify how difficult these benchmarks are to improve and whether the additional computation is justified?
Reviewer_MCEeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether frozen VLM hidden states, conditioned on the query and interaction history, contain recoverable action-conditioned predictive structure. The method trains four lightweight components on offline trajectories: a linear readout, a two-layer MLP transition model, a value head, and a kNN residual-based error penalty. At inference, each candidate visual-access action is rolled out for H=2 latent steps and scored. Evaluation covers three backbones and three wrapped environments (GQA, SlideVQA, NExT-QA). The method improves over a reactive policy by ~1.9 points on average, and over a direct action scorer by 0.45–0.80 points.

Critical strengths
  • The problem is clearly formulated, and the empirical decomposition is careful: one-step fidelity, open-loop composition, policy shift, and coverage.
  • The same-origin diagnostic cleanly isolates next-action quality from downstream policy differences.
  • Gains are consistent across three backbones and three access modalities, and the limitations are stated candidly.
Critical weaknesses
  1. The cost–benefit case is incomplete. The proper reference for the value of prediction is the direct scorer. Gains over it are only 0.45–0.80 points, and its cost is not reported. The reported 1.15–1.30× FLOPs are hard to reconcile with latent modules of roughly a million parameters running alongside an 8B backbone. If the overhead comes from longer episodes, “no extra backbone calls per step” understates the true cost, which matters especially in navigation, where interaction steps are expensive. Please report: a FLOPs breakdown, average interaction steps per episode, per-step latency, and the offline collection cost (~60k VLM trajectories per backbone). Multi-sample reactive inference is a weak reference point.
  2. Generality is not demonstrated. Training and evaluation use the same three environments and action vocabularies; only instances are held out. Joint training is not transfer, and there is no leave-one-environment-out or unseen-task evaluation. The module must be retrained per backbone, the learned action embeddings fix a small discrete action set, and performance degrades under policy shift and low coverage. All environments have cheap, reversible actions. As presented, the method is an in-distribution add-on rather than a general mechanism.
Questions for authors
  1. Where does the 30% FLOPs overhead come from, and does Latent Lookahead change episode length?
  2. How does the same pipeline perform on non-VLM and random representations?
  3. Can the authors provide leave-one-environment-out results, and at least one environment where interaction is costly?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.