Thank you for submitting to ICLR. I loved your work, but it suffers from an incomplete and unsound solution.
W1. The solution needs substantial clarification.
a. What is S_t^{+} in Eq. 1?
b. What is the depth of computation?
Section 3; calibration:
c. First, what is “a selective sense”?
d. What is coverage? It is undefined. R is the probability of responding. Are these the same? If so, please define.
e. Accuracy-on-answered is defined, but it is not clear if it is used later in the paper…
f. Do you take accuracy-on-answered from El-Yaniv & Wiener 2010?
g. The section refers to “calibration is measured” twice within the two sentences using different methods:
“We measure calibration through coverage, R , and accuracy-on-answered, C/R , at different reward and cost profiles”
and
“Calibration is measured through responsiveness and selectivity.”
Which one is true?
h. The section refers to “ do not attempt to elicit a system’s stated confidence”, yet it defines a confidence in the following paragraph using lambda…
i. How do you measure confidence? Using the penultimate layer’s logits?
j. What is selectivity? It is mathematically undefined.
k. Section 4.1; synthetic subset: how do you define D_low? D_high is defined, but not D_low. This is a major problem. It is also not clear how you use the metric you described before, the Lempel-Ziv metric.
l. Details of the natural dataset are not clear in the Appendix. It is important, as this is supposed to be a benchmark and everyone will use it.
What is the coding rate exactly in B.6.2? It implies lossless compression, but I think it refers to lossy compression of frames? Does it mean you decode images at different quality levels, as JPEG can do with different quality levels? Coding usually uses Huffman coding, etc., which is lossless encoding and does not really change the information content of the dataset…
m. What do you mean by high/low “band”? Are you referring to frequency bands?
n. Why do you choose the cutoff 0.0401? There needs to be some justification, and also some variation of that in the benchmark…
W2. The solution is not sound.
a. One of the major problems with the paper is Eq. 2. I am not convinced that it is different than saliency… Eq. 2 does not change the queries. It only changes the input data (the stream). The query might still be asking for the same repetitive patterns (e.g., all the faces as the intro exemplifies), and the model might be capturing just the task-specific patterns rather than actually compressing the data to the level where any question can be answered. For us to understand whether the model can answer any question, there has to be different sets of questions, which ask for saliency and for more diverse set of questions than the saliency. The Q must be varied, not the D.
Interestingly, the authors recognize this issue, but do not provide a satisfactory explanation (the paragraph starting with line 200). These lines corroborates with my comment:
“If Q is narrow, many differences between streams never matter, and a system can discard them using a saliency prior. On the other hand, if some questions are more likely it can further favor the information those questions require.”
Yes, Q should not be narrow, and it should not have any questions that can favor one type of information over the other. The proposed synthetic dataset does have this property, but a metric should capture all kinds of datasets, not only one synthetic dataset… There could be another dataset where the entropy is increased in the dataset such that the saliency patterns are still there and models simply rely on the saliency. The used natural video dataset is one such dataset. The low and high entropy datasets do not really shuffle the patterns. So, the questions might already be favoring a specific pattern that we don’t know. I will have more comments for the natural dataset below.
Also, the paragraphs from 200 to 215 do not really add anything to the paper… Maybe I am missing something. What is K_DQ? It doesn’t have any definition… And, the arguments do not really provide any useful information on saliency vs. compression… The proposed method does not separate saliency from compression. There is no such mechanism whatsoever.
The only way to remove saliency is to actually stress test the model with various types of patterns. The synthetic dataset could be advanced for that. Or the natural dataset could somehow be processed to gain this.
Or, if a model can capture a large number of saliencies, such as being able to answer queries with so many diverse patterns, that could also be considered as “compression”. Because then you adversarially stress the model to recognize different patterns, which, if it does, means it can actually compress the information well.
b. Regarding “For instance, duplicating frames in a video should score as more compressible on average (relative to a video of the same length without duplicated frames). The comparison in an embedding space fits these desiderata.”
This is another major problem. I am not sure about the rationality of the compression for natural images. You just reduce the quality of images... There is no duplication or so. Low-quality images do not mean the model captures "redundancies" across frames. It only does so within each frame. Low-quality images indeed get rid of intra-frame details, and hence if a model can still answer questions, it can learn the patterns from it. But not across frames, which contradicts the synthetic dataset. Synthetic dataset has inter-frame redundancies.
This still does not address the saliency comment. Low-resolution images can also have saliency.
This suggests there could be a better synthetic dataset that can reflect natural videos better -- having some intra-frame and inter-frame redundancy together… now, synthetic is mainly about inter-frame.
W3. Experimental results need substantial clarification.
a. Negative compression needs some discussion.
b. The compression results are all about the accuracy difference between low- and high-entropy datasets… Section 3 promises more…
c. I am not sure if the calibration experiments are sufficient… The increased cost is only a little… It goes from +3/-1 to +1/-4… Maybe, if you were to go to +1/-200, the results would be different… What is the rationale for choosing these values? Surely, the results are valuable. But it is not clear that it actually delivers what it promises.
d. The right figure in Figure 7 is not clear… There are two regimes, but only one is chosen (arm B). Then, what are the columns?
e. How do you measure coverage? By doing a pass over the dataset and counting how many of them had responses and how many did not?
f. “and many models abstain more under the more favorable payoff scheme.” —> What is a “favorable payoff scheme” here? Is it +3/-1? If so, isn’t it counterintuitive that models abstain more when there is less cost? This needs a thorough explanation… Maybe the measurement methodology is wrong, then…
g. “The questions they drop are generally ones they answered incorrectly under arm A (middle)” —> What happens to accuracy then? Does it increase? Or remains the same? If it remains the same, why? Is it that models respond wrongly to those they start to respond to after cost reduction? I feel there is an intertwined interplay here.
h. Figure 8 isn’t really clear. What is each line? The trained and untrained VLMs? Which ones? The dots follow a color scheme, which should then also form a line? What is each point in the graph? A different version of the same model? Are different architectures all trained from scratch? This last detail is not specified. There is only training of small models that are then tested with large Ls. How is Figure 8 obtained eventually?
W4. Some discussion and limitations sections should be added.
a. Why is it only VLMs that are interesting to test compression? Why not LLMs? VLMs are made for multimodal queries, rather than text-only queries. While it is quite valuable to understand the text-processing capacity of VLMs, as one would like to perhaps use only VLMs for all types of inputs, it raises the question of “LLMs” for text-only processing, especially for a benchmark. A “discussion and limitations” section would be valuable to have in the paper, where you can have a paragraph on this as your reflections on LLMs. I see a limitations section in the appendix, but the main body would be a better place for it.
b. Section 3; efficiency: “In other settings, it may be more useful to use other metrics such as latency or the depth of computation” —> another nice item for the discussion and limitations section. What other settings could there be? A concurrent set of users, perhaps, all accessing the model at the same time? With or without any sharing of internal structures in the model (e.g., KV-cache).
W5. Writing should be improved.
a. This sentence is grammatically complete: “If given unlimited time, a capable but memoryless model can achieve high accuracy by simply brute-force checking every chunk of the original input, like how a person can re-read every page of a book to find the answer”
b. I am not sure the terminology “memory backbone” is sound (line 99). I think it is the backbone architecture of the VLM, which is not really a “memory” backbone. It is an architecture for autoregressive prediction. There actually is no memory…
c. I am not sure “Background” is the right title for Section 3. It really defines your metrics, which is not necessarily all background. Some of them are new, at least newly used in this context. If none of them are new, that’s a problem. There is still novelty, as the benchmark itself is new, but it still has to be explicitly articulated if all metrics are taken from an existing work.