acceptodds
Withdrawn from ICLR 2027

Which Computations Drive Visual-to-Text Transfer? A Causal Ledger for Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models can incorporate visual evidence into text states, but how much each decoder block contributes to the answer-relevant visual effect remains difficult to quantify. We introduce reciprocal nested interventions that decompose a common image-conditioned answer effect into ordered block contributions. By replacing visual rows at successive decoder boundaries while retaining text states, we measure how much of the visual effect has entered text states by each depth. These contributions sum exactly to the total effect, yielding a layerwise ledger of cumulative transfer and residual visual dependence. On Qwen3.5-0.8B and Qwen3.5-4B, just two dominant writer blocks carry and of the total visual effect, respectively. This rapid accumulation establishes early causal closures at (0.8B) and (4B). The ledger directly guides compression: closure determines when visual tokens can be evicted, while writer share prioritizes which blocks' inputs to protect in BF16 under quantization. These predictions depend on task and readout: transfer schedules shift across tasks, and generation can still require late visual access beyond an immediate-answer closure.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-language models, Mechanistic interpretability, Causal interventions, Visual-to-text transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_fzKTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how vision-language models (VLMs) transfer answer-relevant visual information into text representations across decoder depth and how this schedule governs model compression. To overcome the non-additivity of standard activation patching and the non-causal nature of attention mass, the authors propose a reciprocal nested intervention protocol on matched image pairs that swaps clean donor and recipient visual hidden states at layer kk while preserving text states, yielding an exact telescoping decomposition of the total visual effect into block-level writer shares (rkr_k) and a causal closure boundary. Validating this framework primarily on Qwen3.5 and Qwen3-VL models, the authors demonstrate that visual-to-text transfer is heavily concentrated in just two dominant full-attention blocks (carrying 71%–83% of the total visual effect), which subsequently guides a "Phase Rule" for compression: preserving BF16 precision solely at the K/V projection inputs of top writer blocks under low-bit activation quantization. Finally, the paper reveals that transfer schedules are task-dependent and that causal closure calibrated on single-token likelihood margins fails under multi-step generative reasoning due to ongoing visual re-access during autoregressive decoding.

Critical strengths
  1. Mathematically rigorous causal accounting: The reciprocal nested intervention cleanly resolves the non-additivity flaw of conventional activation patching via a telescoping sum (∑krk=1\sum_k r_k = 1). Using matched clean visual states instead of zero ablation convincingly avoids out-of-distribution intervention artifacts.
  2. Sharp isolation of the K/V reading port: By controlling for layer depth, module type, and quantization-error energy on A-OKVQA, the authors convincingly prove that protecting the K/V input of dominant writer L7 (only 6.8% of window error energy) recovers 94% of full-layer protection gains, ruling out trivial error-magnitude confounders.
  3. Candid diagnosis of the prefill-decode readout mismatch: The decode-only visual blocking experiment cleanly isolates why static-depth token eviction collapses on multi-step math and logical reasoning, offering a valuable cautionary insight for token pruning methods.
Critical weaknesses
  1. Reliance on aligned counterfactual image pairs: Computing the ledger requires paired images (A,B)(A, B) with opposite semantics and identical token grids (requiring synthetic pairs in Appendix F or forced padding in Appendix K). The authors should test whether unpaired random donor images can approximate rkr_k under native dynamic resolution. 
  2. Missing related work on layer-wise visual truncation and quantization: The finding that single-token VQA tolerates early truncation while multi-step reasoning requires deeper visual access closely mirrors prior work that must be cited and discussed: (1) Ghosh et al., "Do Vision Language Models Need to Process Image Tokens?", CVPR 2026 Workshop Oral (arXiv:2604.09425); (2) Zhu & Feng, "From Attenuation to Attention: Variational Information Flow Manipulation for Fine-Grained Visual Perception", arXiv:2604.12508; and (3) Ye et al., "ImpQuant: Fine-Grained Importance-Aware Quantization for Large Vision-Language Models", ICML 2026. 
  3. Idealized compression setup without system-level metrics: Main experiments only simulate quantization on pre-closure visual-row activations while leaving text states and weights untouched. Reporting wall-clock latency, KV-cache memory savings, and end-to-end PTQ comparisons is needed.
Questions for authors

See the weaknesses. I will improve my score after the author addresses my concerns. I hope the author can supplement the corresponding experimental results before the end of the rebuttal and mark them in different colors in the updated article for my review and score improvement.

Reviewer_c5wAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests whether a layer-wise account of visual transfer predicts which computations need protection under compression. Reciprocal state replacements define cumulative transfer, and differences between adjacent layers give ordered writer shares. The ledger then determines an eviction boundary and ranks blocks for precision protection. Experiments test these predictions with visual-row activation quantization and examine exceptions across tasks and readouts.

Critical strengths

Section 4.3 makes a useful distinction between a component's importance, sensitivity to precision, and dispensability. Protecting the dominant writer's K/V inputs captures most of the benefit of protecting the whole block. MLP precision protection offers little benefit, but disabling the MLP costs much more than reducing its precision. These results show why a small protection benefit alone is insufficient grounds for removing a component.

The ledger also predicts behavior beyond the cohort used to construct it. Table 1 compares protection targets at a fixed budget; Table 2 combines writer protection with eviction on additional benchmarks. Edge-pulse and physical-drop experiments provide empirical support beyond the exact accounting identity.

Critical weaknesses

The proportional prediction in P2 has less support than the ranking used for protection. Section 4.3 shows that protecting a dominant writer is useful, but that alone does not establish how well writer share predicts the size of the benefit across blocks. The numerical check relies mainly on two K/V targets, L7 and L11, in Qwen3.5-0.8B. Their recovery ratio is about 1.97, while their writer-share ratio is 2.35. The matched controls support specificity, but two targets provide limited evidence for a general quantitative relationship.

The compression claim also needs to retain the scope of the experiments: targeted perturbations of visual-row activations. The weight-only pilot in Appendix I finds no reliable writer-specific protection benefit, so it provides little basis for extending the rule to weights.

Questions for authors

Do the existing block-protection measurements show a relationship between writer share and recovery beyond L7 and L11? Plotting those measurements would help distinguish a useful ranking from the proportional prediction in P2. Please also clarify whether the proposed compression rule applies specifically to visual-row activations.

Reviewer_23kSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In this work, the authors propose a method for calculating how much each VLM decoder block contributes to conditioning the text with the input image. They show that visual information stops being relevant at relatively early layers, and demonstrate that identifying and retaining salient blocks' K/V inputs and retaining them can result in efficient activation quantization.

Critical strengths
  • The finding that mechanistic understanding of visual information flow into text tokens enables inference time quantization is novel and timely. If model compression results generalize to larger models, the findings could be very useful.

  • The authors explore their conclusions beyond a fixed-answer generation framing to see if the ability for input modality eviction during inference generalizes to sequential output generation, which makes the paper more scientifically rigorous.

Critical weaknesses
  • The conclusions are highly specific to 3 small (0.8B, 4B, 4B) Qwen3-3.5 checkpoints. This limits generality of findings on the model family front, as well as on the capability and size front. Previous works (such as [1]) have shown that mechanistic principles in one model family will not naively transfer to others. If compute is an issue, there are plenty of other small VLM checkpoints without a Qwen LLM backbone.

  • S4.4, while a good scientific addition to the work, seems to diminish the significance of the model compression finding. The paper does not propose any means by which to incorporate their mechanistic findings into more efficient VLM inference in the sequential generation case. This, in addition to the fact that conclusions hinge on 3 small same-family models, greatly restricts the generality of the findings.

  • The finding that visual encodings cease to enrich text tokens past a certain layer, due to the text tokens having sufficiently absorbed relevant visual context, is not novel - this was shown at scale for VLMs in [2], and for video-LLMs in [3], under different terminology.

Minor Weaknesses:

  • The writing and formatting of this manuscript could use a lot of improvement. Most paragraphs past the introduction start with a bold heading, even when they are only a few sentences long. Table 3 description is just a concatenation of sentence segments. Sentence structures are very repetitive, and a great abundance of unnecessary ":"s appear. Finally, figures would be better if they used a unified font. My rating for the paper does not hinge on these factors, but I believe addressing them would improve the presentation of this work.

[1] Mechanistic Analysis Of Universality: Numerical Comparison CircuitsAcross Transformer Architectures, ACL 2026, https://aclanthology.org/2026.acl-srw.84.pdf [2] Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models, ICLR 2026, https://arxiv.org/pdf/2601.12626 [3] Map the Flow: Revealing Hidden Pathways of Information in VideoLLMs, ICLR 2026, https://arxiv.org/abs/2510.13251

Questions for authors
  • S4.4 seems to suggest that in some cases, VLMs can exhibit a renewed need for visual access. But does this actually mean new information is needed, or was the relevant information extracted at some point but not retained in the text? It seems like the intervention required for successful quantization would be different in either case.

  • Does this ability to evict an input modality once it sufficiently enriches the text also exist for audio-LLMs and video-LLMs? If it is hard for the authors to scale to bigger models, this may be another way to extend the breadth of this work for the future.

Reviewer_c2qEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a causal ledger framework based on reciprocal nested intervention to quantify the visual-to-text transfer in VLMs. The authors measure the writer share and causal closure per layer to identify key Writer Blocks that transfer visual information, and discover that few writer blocks take up the most visual role. Based on this, they guide token eviction and quantization.

The experiments find that only two layers in Qwen3.5-0.8B and Qwen3.5-4B take up around 83% and 71% visual effect. They also show that visual information transfer is task-dependent, with complex reasoning tasks requiring continued visual access during generation.

Critical strengths

Propose an innovative causal ledger framework: related works have insufficient discussions on the contribution of each layer, this work decomposes the whole visual effect into computable layer-wise contribution, providing a new way to analyse cross-modal information flow.

Critical weaknesses
  1. Lack discussion on scaling to more VLMs and methods: main evaluations are conducted on 4B models and Qwen-series. The experiments should include discussions on larger models, MoE models, or different model architectures. Besides, the compared baselines are relatively limited. FastV and VTW are used only as attention-based layer selection heuristics rather than complete implementations of the original methods. Additional comparisons with stronger sensitivity-based precision allcoation methods are anticipated.

  2. Insufficient generalization ability of reasoning-heavy tasks: causal closure is mainly determined by an answer-level readout. However, in reasoning-heavy tasks, when a visual dependence disappears in one step, the latter generation steps may still require it. On MMStar, visual token eviction at K16 causes an overall accuracy drop of 5.53 percentage points, with particularly severe degradation in logical reasoning and mathematics, indicating that the static causal closure may not effectively guide multi-step generation.

Questions for authors
  1. Can the method be generalized to larger VLMs, MoE models and OCR or reasoning-heavy tasks?
  2. How much latency reduction, memory reduction, and throughput acceleration can this method bring in practical deployment?
  3. If taken with counterfactual images, can writer share and writer blocks stay stable?
  4. Would recomputing the Causal Ledger on a quantized model lead to better Writer Block selection and precision allocation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.