acceptodds
Under review as a conference paper at ICLR 2027

A Memory That Only Looks Like One: A Hybrid Vision-Language Model Relays the Picture Rather Than Storing It

Abstract

A hybrid vision-language model carries two kinds of memory. Its attention layers keep a growing cache: every token the model reads, including every image patch, is stored as a key-value entry that later tokens can look up. Its recurrent layers keep a single state of fixed size, and every new token overwrites part of that state. Prior work proved that a fixed-size state must eventually lose information, but not when a given model forgets, what it keeps while forgetting, or what breaks downstream. We answer these three questions for Zamba2-VL-7B, whose 81 decoder layers all carry a Mamba2 mixer while 13 also carry attention. First, just after reading an image the state holds the gist, not the details: a linear probe reads which object was present at on the best layer against chance, but can barely read how many objects there were or where they sat. Second, how fast the state forgets can be computed from the weights alone: each head has a half-life of tokens, with medians of , and at three scales, nothing fitted. Third, and centrally, the memory a probe appears to find at distance is mostly not memory. With write amplitude held fixed, heads with a -token half-life still decode identity at sixty-four tokens after the image, where their own parameters retain of what was written. Evicting the image's key-value entries, without touching the recurrent state, drops the fast tercile from to at thirty-two tokens, below chance, while evicting an equal amount of pre-image text moves the readout the other way; at matched amplitude, eviction costs fast heads times what it costs slow ones. Causally, exchanging all 81 recurrent states between runs that saw different images leaves of answers following the host's own image; exchanging seven of thirteen attention caches moves to the donor's. The recurrent channel relays what attention holds; it does not store the picture. The redundancy premise behind visual token pruning survives down to a quarter of the entries, but the cost of pruning harder roughly doubles with distance from the image.

Primary area:
interpretability and explainable AI
Keywords:
vision-language models, visual memory, state-space models, hybrid architectures, mechanistic interpretability, KV cache, visual token pruning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GmAXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether visual information in hybrid vision-language models is stored in the recurrent state or primarily in the attention KV cache. Through linear probing, Mamba2 state-decay analysis, KV-cache eviction, cross-sample state swapping, and cache-compression experiments, the authors argue that visual information decodable from the recurrent state is largely re-injected through attention and that long-range responses depend primarily on the KV cache. The experiments provide useful evidence for visual interventions. However, the current results more directly support the narrower conclusion that long-range use of visual information relies primarily on the attention KV cache under the tested conditions. They do not yet establish the stronger claim that the recurrent state itself lacks useful visual memory.

Critical strengths
  1. The state and KV-swapping experiments provide strong causal evidence. In a set of 100 filtered image pairs for which the model correctly answered based on each input image, swapping the entire recurrent state caused the model to follow the host's original image in all 100 cases. Swapping KV caches across seven attention layers caused the model to follow the donor image in 94 of 100 cases (Table 3). These interventions demonstrate the direct role of KV pairs more clearly than probe accuracy alone. Control experiments, including probes retrained under specific conditions and eviction of an equivalent volume of text-based KV pairs, further strengthen the evidence.

  2. The compression experiments reveal a notable distance effect. When 5% of image-related KV pairs are retained, the accuracy drop relative to the full cache increases from 10.4 percentage points at d=0 to 33.3 percentage points at d=512 (Table 5). This suggests that visual-cache compression should not be evaluated only at short temporal intervals and merits further study.

Critical weaknesses
  1. The claim that recurrent states contain no useful memory is not sufficiently supported, and the temporal boundaries of the interventions are unclear. Replacing the recurrent state while retaining the original KV pairs only shows that the state is redundant for the current response under that condition. It does not rule out the possibility that the KV pairs later reconstruct or overwrite the replaced state. Likewise, exponential decay in the contribution of an old state within one head does not imply that visual information has disappeared from the broader network, and the median half-life across heads is not an upper bound on information retention. The theoretical and empirical half-lives diverge in opposite directions with model scale in Table 4, so the statement that functional memory lifespan can be pre-calculated should be substantially qualified. Most importantly, the text states that recurrent states remain unchanged at the moment of image-KV eviction, whereas Table 2 shows that fast-head probe accuracy at d=0 drops from 79.7% to 42.2%. If these observations refer to the same point in the sequence, they appear inconsistent. If special tokens or other update mechanisms intervene, the paper should state this explicitly because it changes the mechanistic interpretation at short time scales.

  2. The novelty relative to closely related work is not clearly established, and the compression recommendations extend beyond the experiments. Rethinking the Role of Efficient Attention in Hybrid Architectures by Qiao et al. (June 2026; https://arxiv.org/html/2606.15378v1) reports that long-term retrieval in hybrid architectures relies primarily on full attention. Stateful Token Reduction for Long-Video Hybrid VLMs by Jiang et al. (revised July 2026; https://arxiv.org/html/2603.00198v2) directly studies visual-token compression in hybrid VLMs. These works do not fully cover the state interventions in this paper, but they narrow the scope for claiming novelty about the underlying mechanism and its compression motivation. The authors should clarify whether the contribution is a new mechanistic finding or a more granular empirical analysis of the division of labor in these architectures. In addition, the compression experiments use only random deletion of image KV pairs, with 48 samples per condition, and do not evaluate selection or merging methods such as FastV or VisionZip. The results support the limited conclusion that random compression can incur a performance cost, but they do not justify broad claims about compression strategies or establish 25% retention as a generally safe threshold.

Questions for authors
  1. Please provide a precise timeline of the token sequence. When do the image sequence, KV eviction, special-token insertion, filler text, question input, and state readout occur? How do you reconcile the statement that the state is identical at the moment of eviction with the fast-head probe accuracy drop from 79.7% to 42.2% at d=0 in Table 2?

  2. How do you distinguish between the recurrent state failing to preserve useful information and information in the recurrent state being superseded or overwritten when the full KV cache is available? If subsequent visual-information reinjection is blocked, does recurrent-state replacement remain ineffective? Could later text-based KV pairs capture information originating from the recurrent state?

  3. What is the target of the half-life prediction: the direct contribution of a single-head legacy state, the decodable information in that state, or final task performance? How do you explain the divergent trends across model scales in Table 4, and why is the population median the appropriate measure of information decay?

  4. Compared with the recent works cited above, what mechanisms are uniquely identified by this paper? Can the compression conclusions be tested with at least one practical selection or merging method, together with uncertainty estimates from repeated sampling or pairing?

Reviewer_sZqyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks how a hybrid vision-language model uses its recurrent states and attention caches to remember an image. The experiments focus on Zamba2-VL-7B, with some results also reported for two smaller models in the same family. Using eight-way tasks based on COCO, the authors combine linear probes, estimates of state decay, removal of visual KV entries, and swaps of states or caches between runs. Identity is easier to decode than count or position. Removing visual KV reduces decoding accuracy in some recurrent head groups, and swapping attention caches often makes the model answer about the other image. The authors argue that recurrent states mainly relay information stored by attention. They also examine how the cost of removing visual KV entries changes as the question moves farther from the image.

Critical strengths
  1. The swap experiment is the strongest part of the paper. Table 3 uses 100 image pairs for which both original answers were correct. Swapping all 81 recurrent states and their convolution buffers leaves all 100 answers following the original image, while swapping seven attention caches makes 94 answers follow the other image. Within these selected pairs, this is direct evidence that attention caches strongly influence which image the model answers about.

  2. The authors also include useful controls: they train probes separately for each condition, compare visual-KV removal with removing the same number of text entries, test early and late attention layers, and repeat part of the analysis with different projection matrices. I appreciate the discussion of the failed prediction across model sizes and the unsuccessful Granite probe. These results make the limits of the findings clearer.

Critical weaknesses
  1. My main concern is that relying on attention does not by itself show that the recurrent state stores no useful image information. In Table 3, the original image remains in the attention cache after the recurrent states are swapped. The model could therefore recover that image's information before answering, even if the replaced states had been useful. Table 5 has a related limitation: removing visual KV leaves text KV available, and later text entries could receive image information from earlier recurrent states. The remaining accuracies of 37.5% at d=32 and 22.9% at d=512 therefore do not measure the recurrent channel working alone. The authors acknowledge that probes can miss information, but Section 11 still makes a stronger claim about the absence of useful image memory. The experiments need to distinguish these possibilities before supporting that conclusion.

  2. The decay analysis also needs a clearer connection to what the model remembers. Equation (2) correctly tracks how an earlier state component shrinks, but later writes could still carry image information passed through other recurrent layers. Section 5.1 and Figure 4 use the fast group's 2.3-token median half-life to construct a retention-only prediction. The comparison in Figure 4 is suggestive, but it is unclear how well a prediction based on the fast group's median half-life represents the combined contribution of heads with different decay rates. This matters when interpreting the gap between the predicted and observed probe accuracies. Section 6.2 tests the probe by mixing features from different trials, but it is not clear whether this reflects how real states decay and receive new writes. Matching mean Δ also does not ensure equal write magnitudes, since the full update is ΔxBᵀ. Finally, Δ is measured on the experimental inputs, and Section 8 reports that the predicted and measured half-lives do not agree across scales. These points need to be reconciled with Section 11's claim that a four-token memory lifetime can be calculated before measurement.

Questions for authors
  1. When exactly are the d=0 states in Table 2 read? Section 4 describes d=0 as immediately after the image, while Section 6 says the recurrent states are identical when visual KV is removed. Are question, option, or end-marker tokens processed before the probe reads the state? That could explain the 79.7% versus 42.2% result. Please show the order of these steps and where the Table 3 swap occurs.
  2. If visual KV is removed first, does swapping the recurrent states and convolution buffers between two images change the model's answer when the question and options are held fixed? This would help test whether those states matter when the visual cache cannot restore the original image. The comparison should also account for image information that could enter later text KV. Please clarify whether Table 3 swaps those text entries as well; changes in answer logits would be useful even when the final choice stays the same.
  3. How is Figure 4's predicted decoding curve calculated from the fast group's median half-life of 2.3 tokens? Since individual heads decay at different rates, please explain how their contributions are combined and why the detection threshold from Section 6.2 applies after further writes. This would help assess whether the calculation can rule out retained information as an explanation for the observed decoding accuracy.
  4. What are the model's own count and position accuracies on the same examples used for the probes in Section 4? If the model itself struggles with these questions, the lower probe accuracies may reflect task difficulty or limitations of the readout, rather than a selective difference in what the recurrent states encode. Reporting these behavioral results alongside the probe scores would make the comparison with object identity easier to interpret.
Reviewer_rtUYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors explore the memory in a hybrid linear-attention vision language model. Most of its layers keep a recurrent state of fixed size, whereas a small number of layers keep ordinary attention caches, which grow with the context. The authors aim to find when such a model reads an image and answers a question, which of the two memories the answer comes from. The authors primarilty aim at investigating how much information the fixed state carries. They present results on Zamba2-VL-7B. They further examine visual KV-cache pruning and report that substantial pruning is tolerated, while more aggressive pruning becomes increasingly harmful as the distance from the image grows.

Critical strengths
  1. The paper investigates the roles of recurrent states and attention KV caches in maintaining visual information in hybrid vision-language models, and how much information the fixed recurrent state provides for image-based question-answering.

  2. They investigate this with practical implications for visual token pruning, and report that substantial pruning is tolerated though more aggressive pruning become increasingly harmful.

Critical weaknesses
  1. The paper is very poorly written and difficult to read. The Abstract introduces several terms that have not been introduced before including half-life, fast and slow heads, write amplitude etc. In fact notations are used in the abstract and contributions that are never introduced before. The authors switch between figures and words in the same phrase "26.6% sixty-four tokens".

  2. Johnson-Lindenstrauss projection introduced in Section 3 (Models and Instruments) is not described nor cited. Zamba-2 contains 458,752 numbers per layer has been mentioned several times. The details related to the shape of a layer, layer indices etc are unnecessary.

  3. Results are presented only for a single model and a single dataset, therefore it is not evident if the results generailze.

Questions for authors
  1. What does "identity decoded" in Figure 1 mean?
  2. What is write amplitude mentioned in the Abstract?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.