acceptodds
Withdrawn from ICLR 2027

How Long Does a Model Need to See? When Vision Becomes State

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion vision-language models (DVLMs) keep the image in the computation throughout denoising, tying seeing to answer formation by design. We ask a simple question: how long does a model actually need to see? By terminating visual computation at different stages, we find that seeing and thinking have different lifetimes: visually grounded answers can continue to form after the image has left the computation. Visual influence changes substantially over denoising and can re-emerge after becoming weak, while visual attention largely misses this transition. Meanwhile, visual information becomes increasingly reflected in the evolving generation state, whose structure reveals whether vision will matter again later. This signal is surprisingly simple to read out: a 305-parameter predictor can forecast future visual influence, determine when visual computation can end, and transfer across DVLMs. It preserves approximately full task performance while substantially reducing decoding attention and visual KV usage. Together, these results reveal a simple visual lifecycle in diffusion multimodal generation: seeing can end once vision becomes state.

Primary area:
interpretability and explainable AI
Keywords:
Diffusion vision-language models, Multimodal Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jvpeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether discrete diffusion vision-language models need continuous access to visual information throughout the denoising process. The authors analyze how visual influence changes over time through visual KV removal experiments, further investigate the relationship between the generation state and future visual requirements, and propose a lightweight predictor to determine when visual computation can end. Experiments show that the method can substantially reduce vision-related computation while largely maintaining task performance, with some cross-model transferability.

Critical strengths
  1. The research question is clear, and the approach is coherent. The paper centers on the question of how long a model needs to see, progressing from intervention experiments that directly remove visual KV to representation analysis, and then applying the findings to visual exit decisions. This establishes a relatively complete connection between the analysis of the observed phenomena and the practical method.
  2. The experiment demonstrating the predictive ability of hidden states is interesting. Through experiments on representation changes, the authors show that visual information enters the hidden states involved in generation. Meanwhile, after controlling for the denoising step and current visual influence, the authors find that states with similar hidden representations have more similar future visual requirements. This indicates that the generation state provides predictive signals.
Critical weaknesses
  1. The choice of predictor input features lacks direct validation. The right panel of Figure 7 shows that using only the mean hidden state yields smaller differences in future visual requirements among neighbors than using the combination of the mean hidden state and the state difference. However, the final predictor still uses the mean hidden state, the state difference, and generation progress. Is this contradictory? In addition, the paper lacks corresponding ablations, making it difficult to determine how much improvement the state difference and generation progress each contribute.
  2. Issues with the central argument and figures. The authors interpret generation representations moving progressively closer to visual representations as the internalization of visual information, but the connection between this interpretation and changes in visual requirements over time is not fully developed. In particular, visual influence may first decrease and then increase again, and the authors need to explain how this phenomenon relates to the narrative of gradual internalization. The left and middle panels of Figure 1 also lack clear definitions of the axes and graphical elements, making it difficult to understand their specific meanings and the corresponding experimental evidence.
Questions for authors
  1. Could the authors provide ablations of the predictor input features, comparing generation progress alone, the mean hidden state alone, the mean hidden state combined with generation progress, and the full input? How do these choices affect prediction error, task performance, and the effectiveness of visual exit?
  2. Could the authors clearly explain the axes, curves, and magnified inset in the left and middle panels of Figure 1, along with the experimental evidence underlying them? Please also explain the relationship between generation representations moving closer to the visual representational region and the non-monotonic changes in visual influence.
Reviewer_mkisOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how long diffusion vision-language models (DVLMs) need direct access to visual information during iterative denoising. While conventional DVLMs retain visual representations throughout generation, the authors hypothesize that visual computation, defined as direct computation over visual representations, and visual thinking, defined as the continued formation of visually grounded answers, do not necessarily need to persist for the same duration.

To investigate this hypothesis, the authors conduct intervention experiments in which visual key-value (KV) representations are permanently removed at different denoising steps. They further define visual influence using the Jensen–Shannon divergence between the prediction distributions of unresolved tokens with and without visual input, evaluated from the same generation state. Their analysis suggests that visual influence evolves non-monotonically during denoising and that visual attention alone does not adequately capture these changes.

The authors then analyze hidden-state geometry and observe that generation representations progressively approach visual representations as denoising proceeds. They further demonstrate that the geometry of the current generation state is associated with future visual influence. Based on these findings, they propose a state-guided visual exit mechanism using a lightweight 305-parameter MLP to predict future visual influence and determine when visual computation can be terminated.

Experiments on Dimple and DreamVL across several multimodal understanding benchmarks show that the proposed method largely preserves the original task performance while substantially reducing decoding attention computation and visual KV usage. The authors also evaluate cross-model transfer by applying a predictor trained on Dimple to DreamVL, LaViDa, and MMaDA without additional training.

Overall, the paper provides an interesting investigation of the temporal role of visual information in DVLMs and attempts to connect these observations to practical inference optimization.

Critical strengths
  1. A well-motivated research question supported by direct interventions that distinguish visual computation from visual thinking.

The main strength of this paper is that it poses a clear and interesting research question: Does a model need continuous access to visual information in order to keep generating visually grounded answers?

The authors investigate this question through direct interventions, permanently removing visual KV representations at different denoising stages and examining the resulting generation behavior. This provides more direct functional evidence than analyses based solely on attention patterns.

In particular, the authors measure visual influence using the Jensen–Shannon divergence between prediction distributions with and without visual representations, evaluated from the same generation state. Their analysis reveals that visual influence does not necessarily decrease monotonically during denoising and may re-emerge after becoming relatively weak (Figures 2–4).

These observations suggest that visual information is not utilized uniformly throughout the generation process and provide useful insights into the internal dynamics of diffusion-based multimodal generation.

  1. Connecting representation-level observations to practical inference optimization, with validation across multiple DVLMs.

Another important strength is that the authors translate their observations about generation states into an adaptive visual exit mechanism.

The proposed predictor contains only 305 trainable parameters and estimates future visual influence to determine when visual computation can be terminated. In Table 2, under the 16-step/32-token setting, the method achieves 99.57% average performance retention with 41.04% decoding attention saving on Dimple, and 101.31% average performance retention with 60.70% decoding attention saving on DreamVL.

The authors also demonstrate cross-model transfer by applying a predictor trained on Dimple to DreamVL, LaViDa, and MMaDA without retraining. Although the achievable computational savings vary considerably across models, these results suggest that the proposed state signal may not be entirely model-specific.

The connection between representation analysis and inference optimization, together with the relatively broad experimental validation across DVLMs, constitutes a meaningful strength of the work.

Critical weaknesses
  1. Insufficient evidence that state-guided visual exit provides meaningful advantages over simpler early-exit strategies.

My primary concern is that the paper does not sufficiently establish whether the proposed predictor genuinely leverages informative generation-state features to select appropriate visual exit points, or whether much of its effectiveness can be explained by the progression of denoising itself.

Figure 2 shows that removing visual KV representations after the initial denoising stages often leads to relatively small performance degradation. While this supports the paper's motivating observation, it also raises the possibility that a simple fixed-step visual exit strategy could achieve comparable accuracy and efficiency.

Moreover, the predictor explicitly includes normalized denoising progress, t/T, as part of its input. Importantly, the target future visual influence, y_t, is defined as the maximum visual influence over all subsequent steps and is therefore monotonically non-increasing with respect to t by construction, as also noted in Appendix E.2. Consequently, a predictor using only denoising progress might already achieve reasonable predictive performance without relying on generation-state representations.

Although the nearest-neighbor analysis in Section 3.2 controls for denoising step and current visual influence, the actual visual-exit evaluation does not provide sufficient comparisons against fixed-step exit, time-only prediction, or feature-ablated predictors.

As a result, it is difficult to determine whether the reported computational savings are attributable to meaningful state-dependent adaptation or simply to the fact that visual computation becomes largely unnecessary after the early denoising stages.

Additionally, the paper reports reductions in attention computation and visual KV participation, but does not provide end-to-end inference latency, peak GPU memory, or throughput measurements. The practical speedup after accounting for PCA projection and feature preprocessing overhead therefore remains unclear.

This is a consequential limitation because it directly affects the practical significance and methodological contribution of the proposed state-guided visual exit mechanism.

  1. Insufficient evidence supporting the central mechanistic claim that visual information becomes internalized in the generation state.

The authors observe that generation representations progressively approach visual representations during denoising and interpret this as evidence that visual information becomes internalized in the evolving generation state.

However, the current analysis primarily establishes geometric changes and correlations with future visual influence, rather than directly demonstrating the proposed information-transfer mechanism.

In particular, it is unclear whether the 2D PCA containment ratio and high-dimensional nearest-neighbor cosine distance used in Section 3.1 are sufficient to establish that image-specific semantic information is being transferred into the generation state. A reduction in representational distance could also arise from changes in token distributions, representation variance, or other properties of denoising dynamics.

Furthermore, Table 1 shows substantial differences across models: the final 2D containment ratio reaches 61.47% for Dimple but only 11.65% for DreamVL. This raises questions about how consistently the proposed geometric interpretation applies across models.

Similarly, the analysis in Section 3.2 shows that generation states with similar representations tend to exhibit similar future visual influence. However, it does not directly establish what image-specific information these states preserve or whether such information is causally responsible for the observed behavior.

Additional experiments involving image perturbation or replacement, or probing whether image-specific semantic attributes can be recovered from generation states, would help distinguish actual visual information internalization from more general changes in representation geometry.

There is also a related issue concerning the definition of future visual influence. The supervision target is computed along the original unpruned trajectory, whereas removing visual KV changes the subsequent generation trajectory. Therefore, low predicted future influence along the unpruned trajectory does not necessarily guarantee that the final answer will remain unchanged after visual exit. The theoretical analysis in Appendix E primarily characterizes local differences in prediction distributions and does not directly establish a guarantee on final task accuracy.

Overall, the representation analyses provide interesting circumstantial evidence for the proposed mechanism, but additional causal or functional validation is needed to substantiate the stronger claim that "vision becomes state."

Questions for authors
  1. How much does state-guided visual exit outperform fixed-step or time-only baselines?

Figure 2 suggests that removing visual KV after the first few denoising steps often preserves most of the original performance. Moreover, the proposed predictor includes t/T as an input, while its prediction target y_t is monotonically non-increasing by definition.

Could the authors compare the following approaches under matched task performance or attention-computation budgets?

  • Fixed-step visual exit
  • A time-only predictor using t/T
  • A predictor without generation-state features
  • The full state-guided predictor

An explicit accuracy–computation trade-off comparison would be particularly helpful in establishing the actual benefit of state-dependent adaptation.

  1. Is there additional evidence that the generation state actually preserves image-specific semantic information?

The PCA containment and cosine-distance analyses in Section 3.1 show that generation representations become geometrically closer to visual representations. However, it is unclear whether this necessarily indicates the internalization of visual semantic information.

For example, how would the representation geometry and future visual influence predictions change if the input image were replaced or its visual content were perturbed while keeping the question fixed? Can image-specific attributes be identified or recovered from the generation state alone?

Such experiments would help establish whether the proposed representation-level handoff can be distinguished from general denoising dynamics.

  1. How sensitive is the proposed method to prediction errors and the choice of visual-exit threshold?

The experiments use a fixed threshold of δ = 0.05. However, underestimating future visual influence could cause the predictor to terminate visual computation prematurely.

How does the accuracy–efficiency trade-off change under different thresholds? In particular, could the authors report the frequency of premature visual exits and the proportion of cases in which such decisions change the final answer or turn a previously correct answer into an incorrect one?

Additionally, how do the reported attention savings and visual KV reductions translate into actual wall-clock inference latency and peak GPU memory improvements?

  1. How can the cross-model transfer results be distinguished from the effects of shared denoising progress?

According to Appendix A, the predictor is trained using 7,168-dimensional features extracted from Dimple, standardized and projected to 16 principal components. The same preprocessing and predictor are then applied to other DVLMs.

Since hidden-state coordinates across independently trained models are not necessarily semantically aligned, could the authors elaborate on why the fixed PCA transformation and predictor transfer successfully across models?

In particular, for MMaDA, the authors randomly select 3,584 channels to match the feature dimensionality. How sensitive are the results to the channel-selection seed?

Comparing cross-model transfer performance against a time-only predictor would also help clarify whether the observed transferability reflects a shared visual state signal rather than general similarities in denoising progress.

Reviewer_4nBaOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary
  • This paper asks, “How long does a model actually need to see?” in DVLMs. The authors distinguish visual computation, the direct use of visual representations during denoising, from visual thinking, the continued formation of an image-grounded answer. They define visual influence to quantify the effect of visual KV on predictions. Through visual-KV removal experiments and temporal analysis, they show that visual computation and visual thinking can have different lifetimes, and that visual influence can weaken and re-emerge during answer formation.
  • The authors explain this temporal decoupling as visual information becoming internalized in generation tokens, reducing the need for continued image access. Although this transition is not apparent in attention analysis, they use PCA visualizations, containment ratios, and high-dimensional cosine distances to support this interpretation.
  • They then train a small predictor to estimate maximum future visual influence and determine whether to retain or remove visual KV. The resulting adaptive state-guided visual exit reportedly reduces computation while maintaining task performance. The predictor also transfers across different DVLMs.
Critical strengths
  • S1. Interesting research question. Understanding how long DVLMs need direct image access provides an interesting perspective on multimodal generation.
  • S2. Simple and transferable method. The small predictor transfers across different DVLMs and reduces computation while largely preserving task performance.
Critical weaknesses
  • W1. The necessity of the predictor remains unclear. A direct comparison with fixed-step exit is needed to establish the added value of the adaptive predictor.
  • W2. The practical efficiency gains need further validation. Actual inference time and memory measurements are needed to substantiate the practical benefits of the reported attention savings and visual KV reductions.
Questions for authors
  • Q1-1. Section 2.1 states that removing vision at very early steps causes substantial degradation, but this is not visually apparent in Figure 2. Could you identify the specific results supporting this statement and quantify the performance drop at the earliest exit steps relative to full visual computation?

  • Q1-2. Does predictor-guided exit perform better than simply exiting at a fixed step? This question is closely related to Q1-1: Figure 2 appears to show little degradation even with early fixed-step exits, and the plotted performance seems broadly comparable to the state-guided results in Table 2. However, without the exact numerical results for Figure 2, this comparison cannot be confirmed. Please provide a direct ablation. If the predictor does not offer a meaningful advantage over fixed exit, its added value as the paper’s third contribution would remain unclear. This comparison is therefore central to my assessment.

  • Q2. How much do the reported attention savings and visual KV reductions improve actual inference time and memory usage? These measurements would provide stronger evidence of practical utility than computational reduction percentages alone. I consider this a necessary revision to substantiate the method’s efficiency claims.

  • Q3. Are the left and middle panels of Figure 1 conceptual illustrations or data-derived visualizations? Insufficient explanations of the axes in both panels and the connecting lines in the middle panel, together with the unclear correspondence between the zoomed-in and zoomed-out views in the middle panel, make these panels difficult to interpret. Please reorganize and clarify these panels so that readers can more easily follow the central idea.

  • Q4. Could the supplementary material include sample-wise, case-by-case qualitative analyses of visual influence? Please show and discuss individual trajectories, including cases with low or high late-stage influence and fluctuating patterns, to explain the behaviors summarized in Figure 3.

  • Q5. How accurately does the predictor estimate maximum future visual influence on the test set and after transfer to other DVLMs? Reporting its predictive performance in the appendix, beyond the loss curves, would be helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.