acceptodds
Withdrawn from ICLR 2027

Good Memory Has ECC: Evaluating the Memory of Vision-Language Models Beyond Accuracy

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Memory is widely viewed as an important unsolved problem for LLMs and VLMs, and current benchmarks typically evaluate it by testing accuracy over long text or video. However, accuracy alone misses properties that matter for real long-horizon tasks. We introduce ECCBench, a benchmark and evaluation protocol that measures memory beyond a system's capacity–its raw accuracy at a specific budget–via three axes we call ECC: efficiency–the computation, in FLOPs, needed to answer from memory; compression–whether compressible inputs are remembered more accurately or efficiently; and calibration–whether the system abstains in response to its own uncertainty and the cost of an error. We find that pretrained VLMs compress their memory over text but not video and are poorly calibrated on both. Among a broader set of memory backbones, several non-Transformer architectures achieve better compression-calibration tradeoffs than RoPE Transformers, suggesting they may be useful components for agents operating over long horizons.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
memory, vlms, llms, video understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WvZsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a benchmark called ECCBench, which aims to measure memory in VLMs along the axes of efficiency, compression, and calibration. Efficiency is motivated by the observation that measuring accuracy alone is insufficient for evaluating memory, since even a system with poor memory could artificially achieve high accuracy through brute-force rechecking of inputs. The compression axis aims to address the tendency of models to exploit biases in training datasets, which can inflate measures of accuracy. Finally, calibration aims to measure abstention to ensure that a model's response adequately reflects its uncertainty and the cost of being wrong. ECCBench evaluates VLMs along these three axes by measuring answer generation cost (in FLOPs), comparing accuracy for varying levels of information content in the input, and by giving models an explicit extension option with an associated cost in the prompt.

The benchmark itself consists of two evaluation sets (one synthetically generated and one taken from existing natural videos) and experiments are conducted using 9 VLMs. With regard to compression, the results show that VLM performance drops as entropy increases in the text setting and that models perform only at chance-level under both entropy settings for synthetic videos. In terms of calibration, the VLMs perform poorly across both modalities and rarely respond to error costs. Additional analyses suggest that non-transformer backbones compress better and are better calibrated than RoPE transformers.

Critical strengths
  1. The three axes of efficiency, compression, and calibration for evaluating memory are intuitive and well-motivated. While prior work has addressed individual components, this work is the first (to the best of my knowledge) to combine them under a single framework for evaluating memory.
  2. The experiments cover a broad range of VLMs spanning different model sizes and provide interesting insights into how compression & calibration vary across the text and image modalities.
Critical weaknesses
  1. There seems to be a mismatch between how this work is motivated & described in the introduction, and the actual experimental results that are presented in Section 4. The "efficiency" axis is motivated on lines 38-40 by the following thought experiment: "If given unlimited time, a capable but memoryless model can achieve high accuracy by simply brute-force checking every chunk of the original input, like how a person can re-read every page of a book to find the answer." Based on this, I was expecting this work to evaluate VLMs under a scenario where the input exceeds the context window length and therefore requires an external memory system in order to avoid a "brute-force" re-reading of the input to answer. Instead, it appears that all the experiments used inputs which fit within a single prompt, so the model already sees the entire input at every step of decode. This seems like a significant conceptual mismatch between how efficiency is framed in the intro and what is actually being evaluated. Furthermore, the efficiency axis is barely measured or described in Section 4, with most of the results discussion devoted to calibration and compression. Overall I feel that there is a disconnect between the narrative framing of the intro and the experimental results.

  2. The compression axis only seems relevant for synthetically generated data since the input data and query distributions are unknown for natural videos (as noted in lines 271-273). Since the synthetic data itself does not seem representative of natural data that would be encountered in real-world usage, I question how well the findings actually generalize beyond synthetically generated streams of comma-separated integers and one-letter-per-frame images. I recognize that synthetic data helps disentangle saliency from compression but this nevertheless seems to limit what we can actually conclude about a model's compression capabilities for real data.

Questions for authors
  1. How do you interpret the "negative compression" in Figure 3?

  2. Section 4.2.2 suggests that VLMs do not compress their memory on synthetic videos and that there is little evidence of compression on natural videos. Do you think all tested models are simply deficient when it comes to video compression? Given that all models perform similarly in this regard, I am left wondering whether it is a model issue or a problem with how compression is being measured.

Reviewer_7mEYOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Existing memory benchmarks typically evaluate memory through recall accuracy over long text or video, which overlooks important properties required for real long-horizon tasks. The authors argue that accuracy alone is insufficient to evaluate good memory, as it does not capture efficiency, compression, and calibration. To address this limitation, they introduce ECCBench, a benchmark and evaluation protocol that measures these three dimensions in addition to capacity, using synthetic text and video streams with controlled entropy and natural videos. Experiments on pretrained VLMs show that they can compress their memory over text but generally not video, and are poorly calibrated on both modalities. Furthermore, experiments across different memory backbones show that several non-Transformer architectures achieve better compression-calibration tradeoffs than RoPE Transformers.

Critical strengths
  • Memory Evaluation Beyond Accuracy: The paper introduces ECCBench, which evaluates memory through efficiency, compression, and calibration in addition to capacity. This provides a broader evaluation of memory systems beyond conventional recall accuracy.
  • Controlled Evaluation of Compression: The paper introduces synthetic text and video streams with controlled entropy levels to evaluate compression while reducing the influence of saliency. This design allows for a systematic comparison of recall accuracy between low- and high-entropy inputs.
  • Broad Evaluation Across Models and Modalities: The paper evaluates various pretrained VLMs across text and video, as well as different memory backbones, including Transformers, recurrent models, and state-space models. These experiments provide useful insights into how memory properties differ across modalities and architectures.
Critical weaknesses
  • Unclear Validity of Behavioral Compression: The authors explicitly state that "Our notion of compression is behavioral and can be determined empirically" (Line 137). Specifically, compression is measured by the recall accuracy difference between low- and high-entropy inputs (Equation 2, Lines 188–195). However, higher recall accuracy on predictable inputs does not necessarily indicate that the model exploits stream regularity to retain information more efficiently. Such improvements could also arise because predictable inputs are simply easier to recognize or recall. While a behavioral measure is reasonable for black-box evaluation, the paper does not provide sufficient evidence to distinguish these explanations. Since compression is one of the main evaluation dimensions of ECCBench, this raises concerns about whether the proposed metric measures the intended memory property.

  • Insufficient Separation of Compression and Saliency: The authors claim that their synthetic task isolates compression from saliency by controlling the input and question distributions (Lines 200–215). However, changing input entropy may also change how easily questions can be answered without recalling the input. For example, in low-entropy streams with repeated patterns, a model may correctly determine that a sequence never appeared simply because it does not follow those patterns. The authors acknowledge this possibility in Appendix A.1 (Lines 851–854). Thus, higher recall accuracy may partly result from exploiting patterns in the input rather than retaining information more efficiently. This raises doubts about whether the benchmark successfully separates compression from other factors that improve recall accuracy.

  • Insufficient Separation of Perception and Memory in Video Evaluation: The paper concludes that VLMs struggle to compress their memory on video (Section 4.2.2). The authors conduct a short-sequence perception check (L=8) and exclude models with poor performance on short videos (Figure 4, Appendix B.5). However, this check does not rule out the possibility that visual processing becomes less reliable as the number of frames increases. Therefore, the observed performance degradation on longer videos may reflect difficulties in processing visual inputs rather than memory alone. Without further controls, it is difficult to determine whether the poor video results are primarily caused by limitations in memory compression.

  • Limited Validation on Real-World Memory Tasks: The benchmark focuses on compression while controlling for saliency (Lines 043–051, 200–209). However, exploiting priors about which information is important can also be useful in real-world memory systems. More importantly, the paper does not examine whether models that perform well on ECCBench also perform well on existing long-memory benchmarks. Without such analysis, it is unclear how well the proposed evaluation dimensions reflect memory performance in real-world tasks.

  • Limited Efficiency Evaluation: The paper measures efficiency using only total FLOPs and assumes one query per stream (Lines 181–186). However, FLOPs alone do not capture practical costs such as memory access, memory bandwidth, and latency, which can vary significantly across memory architectures. Moreover, real-world memory systems often answer multiple queries using the same stored information, allowing the write cost to be shared across queries. As the authors acknowledge in Appendix E (Lines 2136–2139), this can change the relative rankings of memory systems. These limitations raise concerns about whether the proposed efficiency evaluation provides a reliable comparison of memory architectures in practical settings.

  • Limited Evidence for Generalizing Beyond Repetition: The paper distinguishes its work from Arora et al. (2024) by claiming to extend beyond repetition to more general predictable structures (Lines 135–137). However, the low-entropy streams are generated using variable-order n-gram processes, which can also produce strong repetition patterns (Appendix A.1). It is therefore unclear whether the observed compression reflects an ability to exploit more general predictable structures or mainly benefits from repetition. This distinction matters because evaluating a broader range of predictable structures is part of the paper's claimed contribution beyond prior work.

  • Insufficient Discussion of Prior Work on Calibration: The paper claims that existing memory benchmarks have not been designed to evaluate calibration (Line 086). However, LongMemEval [1] already considers calibration by evaluating whether models abstain when they cannot reliably answer questions. The paper does not clearly explain how its calibration evaluation differs from this prior work.

    Reference

    [1] Wu, Di, et al. "Longmemeval: Benchmarking chat assistants on long-term interactive memory." ICLR (2025).

  • Insufficient Justification for the Selection of ECC Dimensions: While the paper provides reasonable motivations for evaluating efficiency, compression, and calibration individually, it does not sufficiently justify why these three dimensions were specifically selected. Other potentially important aspects of memory, such as robustness to interference or memory updating, are not considered. Consequently, it remains unclear how comprehensively the proposed ECC framework captures the essential properties of a good memory system.

Questions for authors
  • Could the authors analyze the correlation between ECCBench results and performance on existing long-memory benchmarks to better demonstrate the practical relevance of the proposed evaluation framework?

  • Could the authors clarify how the proposed calibration evaluation differs from that in LongMemEval?

  • Could the authors clarify the rationale for selecting these three dimensions and discuss whether other important aspects of memory should also be considered?

Reviewer_ykkAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a novel benchmark that evaluates VLMs across different metrics than the conventional accuracy. It proposes three new metrics: compression, efficiency, and calibration. While the study is much needed and has a solid amount of work, it suffers from an incomplete and unsound solution.

Critical strengths

S1. A great problem.

S2. Several good ideas.

Critical weaknesses

Thank you for submitting to ICLR. I loved your work, but it suffers from an incomplete and unsound solution.

W1. The solution needs substantial clarification.

a. What is S_t^{+} in Eq. 1?

b. What is the depth of computation?

Section 3; calibration:

c. First, what is “a selective sense”?

d. What is coverage? It is undefined. R is the probability of responding. Are these the same? If so, please define.

e. Accuracy-on-answered is defined, but it is not clear if it is used later in the paper…

f. Do you take accuracy-on-answered from El-Yaniv & Wiener 2010?

g. The section refers to “calibration is measured” twice within the two sentences using different methods:

“We measure calibration through coverage, R , and accuracy-on-answered, C/R , at different reward and cost profiles”

and

“Calibration is measured through responsiveness and selectivity.”

Which one is true?

h. The section refers to “ do not attempt to elicit a system’s stated confidence”, yet it defines a confidence in the following paragraph using lambda…

i. How do you measure confidence? Using the penultimate layer’s logits?

j. What is selectivity? It is mathematically undefined.

k. Section 4.1; synthetic subset: how do you define D_low? D_high is defined, but not D_low. This is a major problem. It is also not clear how you use the metric you described before, the Lempel-Ziv metric.

l. Details of the natural dataset are not clear in the Appendix. It is important, as this is supposed to be a benchmark and everyone will use it.

What is the coding rate exactly in B.6.2? It implies lossless compression, but I think it refers to lossy compression of frames? Does it mean you decode images at different quality levels, as JPEG can do with different quality levels? Coding usually uses Huffman coding, etc., which is lossless encoding and does not really change the information content of the dataset…

m. What do you mean by high/low “band”? Are you referring to frequency bands?

n. Why do you choose the cutoff 0.0401? There needs to be some justification, and also some variation of that in the benchmark…

W2. The solution is not sound.

a. One of the major problems with the paper is Eq. 2. I am not convinced that it is different than saliency… Eq. 2 does not change the queries. It only changes the input data (the stream). The query might still be asking for the same repetitive patterns (e.g., all the faces as the intro exemplifies), and the model might be capturing just the task-specific patterns rather than actually compressing the data to the level where any question can be answered. For us to understand whether the model can answer any question, there has to be different sets of questions, which ask for saliency and for more diverse set of questions than the saliency. The Q must be varied, not the D.

Interestingly, the authors recognize this issue, but do not provide a satisfactory explanation (the paragraph starting with line 200). These lines corroborates with my comment:

“If Q is narrow, many differences between streams never matter, and a system can discard them using a saliency prior. On the other hand, if some questions are more likely it can further favor the information those questions require.”

Yes, Q should not be narrow, and it should not have any questions that can favor one type of information over the other. The proposed synthetic dataset does have this property, but a metric should capture all kinds of datasets, not only one synthetic dataset… There could be another dataset where the entropy is increased in the dataset such that the saliency patterns are still there and models simply rely on the saliency. The used natural video dataset is one such dataset. The low and high entropy datasets do not really shuffle the patterns. So, the questions might already be favoring a specific pattern that we don’t know. I will have more comments for the natural dataset below.

Also, the paragraphs from 200 to 215 do not really add anything to the paper… Maybe I am missing something. What is K_DQ? It doesn’t have any definition… And, the arguments do not really provide any useful information on saliency vs. compression… The proposed method does not separate saliency from compression. There is no such mechanism whatsoever.

The only way to remove saliency is to actually stress test the model with various types of patterns. The synthetic dataset could be advanced for that. Or the natural dataset could somehow be processed to gain this.

Or, if a model can capture a large number of saliencies, such as being able to answer queries with so many diverse patterns, that could also be considered as “compression”. Because then you adversarially stress the model to recognize different patterns, which, if it does, means it can actually compress the information well.

b. Regarding “For instance, duplicating frames in a video should score as more compressible on average (relative to a video of the same length without duplicated frames). The comparison in an embedding space fits these desiderata.”

This is another major problem. I am not sure about the rationality of the compression for natural images. You just reduce the quality of images... There is no duplication or so. Low-quality images do not mean the model captures "redundancies" across frames. It only does so within each frame. Low-quality images indeed get rid of intra-frame details, and hence if a model can still answer questions, it can learn the patterns from it. But not across frames, which contradicts the synthetic dataset. Synthetic dataset has inter-frame redundancies.

This still does not address the saliency comment. Low-resolution images can also have saliency.

This suggests there could be a better synthetic dataset that can reflect natural videos better -- having some intra-frame and inter-frame redundancy together… now, synthetic is mainly about inter-frame.

W3. Experimental results need substantial clarification.

a. Negative compression needs some discussion.

b. The compression results are all about the accuracy difference between low- and high-entropy datasets… Section 3 promises more…

c. I am not sure if the calibration experiments are sufficient… The increased cost is only a little… It goes from +3/-1 to +1/-4… Maybe, if you were to go to +1/-200, the results would be different… What is the rationale for choosing these values? Surely, the results are valuable. But it is not clear that it actually delivers what it promises.

d. The right figure in Figure 7 is not clear… There are two regimes, but only one is chosen (arm B). Then, what are the columns?

e. How do you measure coverage? By doing a pass over the dataset and counting how many of them had responses and how many did not?

f. “and many models abstain more under the more favorable payoff scheme.” —> What is a “favorable payoff scheme” here? Is it +3/-1? If so, isn’t it counterintuitive that models abstain more when there is less cost? This needs a thorough explanation… Maybe the measurement methodology is wrong, then…

g. “The questions they drop are generally ones they answered incorrectly under arm A (middle)” —> What happens to accuracy then? Does it increase? Or remains the same? If it remains the same, why? Is it that models respond wrongly to those they start to respond to after cost reduction? I feel there is an intertwined interplay here.

h. Figure 8 isn’t really clear. What is each line? The trained and untrained VLMs? Which ones? The dots follow a color scheme, which should then also form a line? What is each point in the graph? A different version of the same model? Are different architectures all trained from scratch? This last detail is not specified. There is only training of small models that are then tested with large Ls. How is Figure 8 obtained eventually?

W4. Some discussion and limitations sections should be added.

a. Why is it only VLMs that are interesting to test compression? Why not LLMs? VLMs are made for multimodal queries, rather than text-only queries. While it is quite valuable to understand the text-processing capacity of VLMs, as one would like to perhaps use only VLMs for all types of inputs, it raises the question of “LLMs” for text-only processing, especially for a benchmark. A “discussion and limitations” section would be valuable to have in the paper, where you can have a paragraph on this as your reflections on LLMs. I see a limitations section in the appendix, but the main body would be a better place for it.

b. Section 3; efficiency: “In other settings, it may be more useful to use other metrics such as latency or the depth of computation” —> another nice item for the discussion and limitations section. What other settings could there be? A concurrent set of users, perhaps, all accessing the model at the same time? With or without any sharing of internal structures in the model (e.g., KV-cache).

W5. Writing should be improved.

a. This sentence is grammatically complete: “If given unlimited time, a capable but memoryless model can achieve high accuracy by simply brute-force checking every chunk of the original input, like how a person can re-read every page of a book to find the answer”

b. I am not sure the terminology “memory backbone” is sound (line 99). I think it is the backbone architecture of the VLM, which is not really a “memory” backbone. It is an architecture for autoregressive prediction. There actually is no memory…

c. I am not sure “Background” is the right title for Section 3. It really defines your metrics, which is not necessarily all background. Some of them are new, at least newly used in this context. If none of them are new, that’s a problem. There is still novelty, as the benchmark itself is new, but it still has to be explicitly articulated if all metrics are taken from an existing work.

Questions for authors

Q1. How do you differentiate saliency from compression without varying the question's domain, Q?

Q2. How do you measure coverage? Where and how is confidence used?

Q3. What is selectivity, and how is it used for measuring calibration?

Q4. What is negative compression, and what causes it?

Q5. How would the results change if we had increased the cost of error more?

Q6. What is each of the points in Figure 8?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.