- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether visual information in hybrid vision-language models is stored in the recurrent state or primarily in the attention KV cache. Through linear probing, Mamba2 state-decay analysis, KV-cache eviction, cross-sample state swapping, and cache-compression experiments, the authors argue that visual information decodable from the recurrent state is largely re-injected through attention and that long-range responses depend primarily on the KV cache. The experiments provide useful evidence for visual interventions. However, the current results more directly support the narrower conclusion that long-range use of visual information relies primarily on the attention KV cache under the tested conditions. They do not yet establish the stronger claim that the recurrent state itself lacks useful visual memory.
Critical strengths
-
The state and KV-swapping experiments provide strong causal evidence. In a set of 100 filtered image pairs for which the model correctly answered based on each input image, swapping the entire recurrent state caused the model to follow the host's original image in all 100 cases. Swapping KV caches across seven attention layers caused the model to follow the donor image in 94 of 100 cases (Table 3). These interventions demonstrate the direct role of KV pairs more clearly than probe accuracy alone. Control experiments, including probes retrained under specific conditions and eviction of an equivalent volume of text-based KV pairs, further strengthen the evidence.
-
The compression experiments reveal a notable distance effect. When 5% of image-related KV pairs are retained, the accuracy drop relative to the full cache increases from 10.4 percentage points at d=0 to 33.3 percentage points at d=512 (Table 5). This suggests that visual-cache compression should not be evaluated only at short temporal intervals and merits further study.
Critical weaknesses
-
The claim that recurrent states contain no useful memory is not sufficiently supported, and the temporal boundaries of the interventions are unclear. Replacing the recurrent state while retaining the original KV pairs only shows that the state is redundant for the current response under that condition. It does not rule out the possibility that the KV pairs later reconstruct or overwrite the replaced state. Likewise, exponential decay in the contribution of an old state within one head does not imply that visual information has disappeared from the broader network, and the median half-life across heads is not an upper bound on information retention. The theoretical and empirical half-lives diverge in opposite directions with model scale in Table 4, so the statement that functional memory lifespan can be pre-calculated should be substantially qualified. Most importantly, the text states that recurrent states remain unchanged at the moment of image-KV eviction, whereas Table 2 shows that fast-head probe accuracy at d=0 drops from 79.7% to 42.2%. If these observations refer to the same point in the sequence, they appear inconsistent. If special tokens or other update mechanisms intervene, the paper should state this explicitly because it changes the mechanistic interpretation at short time scales.
-
The novelty relative to closely related work is not clearly established, and the compression recommendations extend beyond the experiments. Rethinking the Role of Efficient Attention in Hybrid Architectures by Qiao et al. (June 2026; https://arxiv.org/html/2606.15378v1) reports that long-term retrieval in hybrid architectures relies primarily on full attention. Stateful Token Reduction for Long-Video Hybrid VLMs by Jiang et al. (revised July 2026; https://arxiv.org/html/2603.00198v2) directly studies visual-token compression in hybrid VLMs. These works do not fully cover the state interventions in this paper, but they narrow the scope for claiming novelty about the underlying mechanism and its compression motivation. The authors should clarify whether the contribution is a new mechanistic finding or a more granular empirical analysis of the division of labor in these architectures. In addition, the compression experiments use only random deletion of image KV pairs, with 48 samples per condition, and do not evaluate selection or merging methods such as FastV or VisionZip. The results support the limited conclusion that random compression can incur a performance cost, but they do not justify broad claims about compression strategies or establish 25% retention as a generally safe threshold.
Questions for authors
-
Please provide a precise timeline of the token sequence. When do the image sequence, KV eviction, special-token insertion, filler text, question input, and state readout occur? How do you reconcile the statement that the state is identical at the moment of eviction with the fast-head probe accuracy drop from 79.7% to 42.2% at d=0 in Table 2?
-
How do you distinguish between the recurrent state failing to preserve useful information and information in the recurrent state being superseded or overwritten when the full KV cache is available? If subsequent visual-information reinjection is blocked, does recurrent-state replacement remain ineffective? Could later text-based KV pairs capture information originating from the recurrent state?
-
What is the target of the half-life prediction: the direct contribution of a single-head legacy state, the decodable information in that state, or final task performance? How do you explain the divergent trends across model scales in Table 4, and why is the population median the appropriate measure of information decay?
-
Compared with the recent works cited above, what mechanisms are uniquely identified by this paper? Can the compression conclusions be tested with at least one practical selection or merging method, together with uncertainty estimates from repeated sampling or pairing?