- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a unified framework for improving the memory and computational efficiency of Vision-Language Models (VLMs) by prioritizing informative visual representations. The key idea is to estimate the information content of visual embeddings using their geometric relationship with the background mean in the latent space. By modeling the embedding distribution with a von Mises-Fisher distribution, the authors derive a relationship between Shannon surprisal and cosine similarity to the inverse background mean. Based on this formulation, the paper introduces three components: (1) Surprisal-Weighted Mixed-Precision (SWMP), which allocates different numerical precisions to visual embeddings based on their estimated information content; (2) Prior-Biased Attention, which incorporates surprisal-based weights into attention logits to prioritize informative representations; and (3) Temporal SWMP, which uses an exponential moving average (EMA) and statistical thresholding to select informative frames from continuous video streams. Experiments are conducted using MS-COCO and nuScenes. The proposed SWMP strategy achieves a 57.50% reduction in embedding storage memory compared to uniform FP16 storage. For temporal frame selection, the authors evaluate event recall under constrained frame budgets and provide a qualitative Video Question Answering example using Qwen2-VL-2B, where the proposed method produces the correct response while retaining only 2.92% of the original frames.
Critical strengths
S1. Information-theoretic formulation for visual representation prioritization. The paper provides an interesting perspective on visual representation prioritization by connecting Shannon surprisal with the geometric properties of the VLM latent space. The proposed formulation offers a unified criterion for identifying informative representations and allocating computational resources accordingly.
S2. Practical potential for memory-efficient video processing. The proposed framework addresses an important problem in resource-constrained VLM inference. The reported 57.50% reduction in embedding storage and the qualitative video understanding example suggest potential benefits for efficient processing of continuous video streams.
Critical weaknesses
W1. Lack of quantitative downstream performance evaluation. The most significant concern is the lack of quantitative evaluation of downstream task performance before and after compression. Although the paper reports a 57.50% reduction in embedding storage, it does not demonstrate whether the compressed representations preserve the original model's performance. Similarly, the temporal routing method is evaluated mainly through event recall and a limited qualitative VQA example, without quantitative comparisons on established video understanding benchmarks such as VideoMME or MVBench. Without these evaluations, it is difficult to determine whether the proposed framework provides meaningful efficiency improvements while maintaining downstream performance.
W2. Limited validation of the proposed surprisal-based importance criterion. The proposed method assumes that representations deviating from the background mean are more informative. However, a high surprisal score does not necessarily indicate that a frame is important for a downstream task. The paper provides limited empirical evidence that this criterion consistently identifies task-relevant information across diverse scenarios. In addition, the theoretical connection between Shannon surprisal and cosine similarity relies on the assumed von Mises-Fisher distribution, but the suitability of this assumption for real VLM embeddings is not sufficiently validated. These limitations raise concerns about the generalizability of the proposed importance estimation method.
Questions for authors
Q1. Downstream performance after compression In Section 5.2, the proposed SWMP achieves a 57.50% reduction in embedding storage. However, no quantitative downstream performance comparison between the original FP16 representations and the compressed representations is reported. Could the authors provide evaluations demonstrating how the proposed mixed-precision compression affects downstream task accuracy?
Q2. Quantitative evaluation on video understanding benchmarks In Section 5.4, the effectiveness of Temporal SWMP is demonstrated through a qualitative Video QA example. Could the authors provide quantitative comparisons on established video understanding benchmarks, such as VideoMME or MVBench, under the same frame or token budget? Such evaluations would help establish whether the proposed method consistently outperforms existing frame selection strategies.
Q3. Validity of the surprisal-based importance criterion The proposed framework assumes that representations with higher surprisal are more informative. However, a larger deviation from the background mean does not necessarily imply greater relevance to a downstream task. Could the authors provide additional evidence supporting the correlation between the proposed surprisal scores and task-relevant information?
3 comments
Thank you for your detailed rebuttal and for addressing my concerns. However, I remain unconvinced that the proposed method offers a meaningful advantage over simple cosine similarity. Moreover, the downstream utility of the resulting surprisal scores remains unclear. Even in the newly added Section 5.5, the difference between Static SWMP and MassMatched is not statistically significant. Therefore, it is still unclear what additional benefit the proposed scoring method provides. For these reasons, I maintain my original score of 1.
Thank you for your response. We acknowledge that your observation is valid. Based on the experimental results, it would be difficult to characterize the proposed method as offering a significant state-of-the-art advantage. As you pointed out, when considering the statistical significance of its performance relative to one of the alternative methods evaluated in a particular experimental setting, such superiority is not established. This is also explicitly acknowledged in the revised Section 5.5.
However, the main contribution of our research is not to demonstrate that SWMP achieves exceptional performance over other methods on a particular task. Rather, a central aspect of our contribution is that our approach differs from existing evaluation and measurement methods precisely because they do not measure or assign weights according to the proposed information-theoretic criterion.
In this context, we hope that the newly added Section 5.5 can be understood as demonstrating the potential applicability of this criterion, and its variants, to tasks where such prioritization can be effective.
This also means that our research takes a different approach from studies conducted by large research groups that establish their contributions through extensive experiments across numerous benchmarks. We do not have the capacity to pursue that scale of experimentation, nor is it the primary direction our research aims to take.
Instead, what we regard as the most significant potential contribution of this work is the mathematical and empirical connection between several well-established and actively researched areas, including information theory, the structural properties of VLM latent spaces, and attention-bias injection.
This is precisely why our paper is titled "Bridging Shannon Surprisal and Latent Geometry."
The primary purpose of our downstream experiments is to examine and demonstrate the conditions and possibilities under which this connection can be practically applied. Rather than claiming overwhelming superiority over other methods in a particular domain, the scope of our work is to demonstrate that a single underlying principle can be applied across multiple computational settings.
Empirical experimentation accounts for a substantial proportion of contemporary artificial intelligence research. Viewed from another perspective, however, this also makes research that primarily investigates theoretically grounded possibilities for practical application particularly valuable in the present research landscape.
We understand why our approach may appear to offer limited advantages in downstream task performance. We also acknowledge that one possible reason for the relatively low Soundness or Confidence assessment may be that, when viewed primarily through the lens of downstream performance, our research can appear overly broad, making it difficult to identify exactly what it seeks to establish. We recognize this possibility and respect your assessment.
Nevertheless, our research is fundamentally about demonstrating that a single principle can be applied across different settings, and that such applications are practically realizable.
We believe that demonstrating how one relatively simple principle can extend across multiple domains is an important foundation upon which the more specialized and sophisticated follow-up studies you have suggested can subsequently develop.
With this perspective in mind, we hope that this conference can also serve as a forum for discussing research questions worthy of academic consideration, exchanging ideas necessary for further progress, and exploring shared understandings of the underlying principles behind empirical results, together with their potential applications.
From the perspective of our research objectives, we recognize that your criticisms are valid in certain respects. Nevertheless, we respectfully ask you to reconsider the relationship between the specific concerns you have raised, your understanding of the paper's central research motivation, and what our study ultimately seeks to demonstrate.
We believe that a mismatch between these perspectives could contribute to a lower Soundness assessment and, consequently, influence the overall Rating.
We therefore respectfully request that you reconsider your Soundness assessment and overall Rating in light of the intended contribution and scope of our research.
Thank you again for your careful consideration and for engaging with our work.
From Geometric Surprisal to Downstream Utility
Thank you for your thoughtful review and for recognizing the value of connecting information-theoretic surprisal with latent geometry. Your concerns helped us clarify our central objective: to establish a query-independent, background-relative information-prioritization criterion and demonstrate its practical utility in resource-constrained VLM processing.
To strengthen the empirical evidence, we added Section 5.5, "Downstream Validation of Prior-Biased Attention."
Q1. Downstream performance after compression.
The reported 57.50% reduction represents nominal embedding-storage savings from surprisal-guided precision allocation, not a guarantee of semantic preservation after quantization. Appendix C.2 explicitly discusses the degradation caused by aggressive low-bit precision.
An important distinction in our framework is that the information criterion and its application mechanisms are separate. Geometric surprisal can guide precision allocation, hard frame selection, or soft attention redistribution.
Section 5.5 provides new downstream evidence for the latter, independently of quantization. Thus, it strengthens the empirical utility of the shared criterion, while FP16-versus-quantized accuracy remains a distinct compression-specific question.
Q2. Broader video-understanding benchmarks.
We agree that VideoMME and MVBench would be valuable for evaluating broader generalization.
However, our criterion is deliberately query-agnostic: it estimates which representations deviate from the background distribution without requiring a downstream question. This is particularly relevant to continuous streams, where information prioritization may precede any linguistic query.
Section 5.4 therefore examined whether a salient semantic transition could be preserved under extreme frame constraints, rather than claiming general query-conditioned reasoning performance.
To move beyond that single qualitative demonstration, Section 5.5 evaluates 224 new CLEVRER scenes using simulator-defined answers and identical high-redundancy inputs containing 4,608 visual tokens.
Importantly, although the prior is computed independently of the question, its effectiveness is evaluated through objective downstream recognition. This directly tests whether query-independent information prioritization can benefit query-specific predictions.
Broader video understanding remains an important extension. PMI-based contextual surprisal, discussed in our future directions, provides a natural path toward explicitly query-conditioned importance estimation.
Q3. Does geometric surprisal indicate task-relevant information?
We agree that geometric rarity does not universally imply task relevance. Our vMF derivation establishes an information ordering relative to the modeled background distribution, not guaranteed relevance to arbitrary questions.
Our hypothesis is more specific: under substantial visual redundancy, background-relative surprisal can provide a useful computational prior even without knowing the downstream query.
Section 5.5 directly tests this hypothesis. Each input contains one evidence pair among 127 repeated background pairs. We inject the surprisal-derived prior into the pre-softmax attention logits of frozen Qwen2-VL-2B-Instruct, without changing input frames, visual-token budgets, or model parameters.
Static SWMP improves recognition accuracy from 61.61% to 71.88% (+10.27 percentage points, paired exact p = 0.00219).
This demonstrates that a query-independent geometric prior can improve the recovery of task-relevant evidence through actual attention redistribution.
The controls further clarify this finding: Shuffled weighting achieves 43.30%, indicating the importance of prior placement, while FrequencyMatched achieves 72.77%, showing that simpler redundancy-aware weighting can also be effective.
Therefore, we do not claim that surprisal universally identifies the most task-relevant information. We demonstrate that background-relative geometric surprisal provides a mathematically interpretable and operationally useful prioritization criterion under the evaluated redundancy conditions.
We believe Section 5.5 substantially strengthens the empirical connection between our theoretical formulation and downstream VLM performance, while clarifying the distinction between query-independent information prioritization and query-specific semantic relevance.
Thank you again for your constructive feedback.