acceptodds
Under review as a conference paper at ICLR 2027

Bridging Shannon Surprisal and Latent Geometry: A Unified Framework for VLM Storage, Attention Routing, and Data Pruning

Abstract

As Vision-Language Models (VLMs) expand into unconstrained, in-the-wild continuous spatiotemporal domains, uniform vector processing creates severe KV-Cache and computational attention bottlenecks. We propose a sparsity-driven methodology that bridges these geometric latent constraints with Shannon's Information Theory. By modeling dense background representations as a von Mises-Fisher (vMF) distribution, we derive that local surprisal exhibits an affine relationship with a vector's cosine similarity with the inverse global mean (). To overcome representation degeneration—where vectors are constrained within narrow anisotropic boundaries—we formulate an exponential weighting function equipped with a Dimensionality Compensator () and a Structural Domain Shifter (). This formulation drives a dual-phase optimization framework. First, in static environments, our Surprisal-Weighted Mixed-Precision (SWMP) strategy achieves a 57.50% memory reduction by quantizing low-surprisal representations. Second, for continuous video streams, we introduce Temporal SWMP, leveraging an Exponential Moving Average (EMA) to transition the framework into an Information-Theoretic Attention Router. We further observe strong metric coupling between adjacent-difference routing and a second-difference kinematic evaluation proxy. Under extreme Iso-KV Cache constraints (3% of frames), Temporal SWMP exceeds uniform sampling in recall while exhibiting selection behavior distinct from adjacent-difference routing. Together, these results support an information-theoretic criterion for attention routing in resource-constrained continuous VLM deployments across unpredictable real-world environments.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Vision-Language Models (VLMs), Representation Degeneration, Information Theory, Surprisal, Attention Monopoly, Surprisal-Weighted Mixed-Precision (SWMP)

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JjhUOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a unified framework for improving the memory and computational efficiency of Vision-Language Models (VLMs) by prioritizing informative visual representations. The key idea is to estimate the information content of visual embeddings using their geometric relationship with the background mean in the latent space. By modeling the embedding distribution with a von Mises-Fisher distribution, the authors derive a relationship between Shannon surprisal and cosine similarity to the inverse background mean. Based on this formulation, the paper introduces three components: (1) Surprisal-Weighted Mixed-Precision (SWMP), which allocates different numerical precisions to visual embeddings based on their estimated information content; (2) Prior-Biased Attention, which incorporates surprisal-based weights into attention logits to prioritize informative representations; and (3) Temporal SWMP, which uses an exponential moving average (EMA) and statistical thresholding to select informative frames from continuous video streams. Experiments are conducted using MS-COCO and nuScenes. The proposed SWMP strategy achieves a 57.50% reduction in embedding storage memory compared to uniform FP16 storage. For temporal frame selection, the authors evaluate event recall under constrained frame budgets and provide a qualitative Video Question Answering example using Qwen2-VL-2B, where the proposed method produces the correct response while retaining only 2.92% of the original frames.

Critical strengths

S1. Information-theoretic formulation for visual representation prioritization. The paper provides an interesting perspective on visual representation prioritization by connecting Shannon surprisal with the geometric properties of the VLM latent space. The proposed formulation offers a unified criterion for identifying informative representations and allocating computational resources accordingly.

S2. Practical potential for memory-efficient video processing. The proposed framework addresses an important problem in resource-constrained VLM inference. The reported 57.50% reduction in embedding storage and the qualitative video understanding example suggest potential benefits for efficient processing of continuous video streams.

Critical weaknesses

W1. Lack of quantitative downstream performance evaluation. The most significant concern is the lack of quantitative evaluation of downstream task performance before and after compression. Although the paper reports a 57.50% reduction in embedding storage, it does not demonstrate whether the compressed representations preserve the original model's performance. Similarly, the temporal routing method is evaluated mainly through event recall and a limited qualitative VQA example, without quantitative comparisons on established video understanding benchmarks such as VideoMME or MVBench. Without these evaluations, it is difficult to determine whether the proposed framework provides meaningful efficiency improvements while maintaining downstream performance.

W2. Limited validation of the proposed surprisal-based importance criterion. The proposed method assumes that representations deviating from the background mean are more informative. However, a high surprisal score does not necessarily indicate that a frame is important for a downstream task. The paper provides limited empirical evidence that this criterion consistently identifies task-relevant information across diverse scenarios. In addition, the theoretical connection between Shannon surprisal and cosine similarity relies on the assumed von Mises-Fisher distribution, but the suitability of this assumption for real VLM embeddings is not sufficiently validated. These limitations raise concerns about the generalizability of the proposed importance estimation method.

Questions for authors

Q1. Downstream performance after compression In Section 5.2, the proposed SWMP achieves a 57.50% reduction in embedding storage. However, no quantitative downstream performance comparison between the original FP16 representations and the compressed representations is reported. Could the authors provide evaluations demonstrating how the proposed mixed-precision compression affects downstream task accuracy?

Q2. Quantitative evaluation on video understanding benchmarks In Section 5.4, the effectiveness of Temporal SWMP is demonstrated through a qualitative Video QA example. Could the authors provide quantitative comparisons on established video understanding benchmarks, such as VideoMME or MVBench, under the same frame or token budget? Such evaluations would help establish whether the proposed method consistently outperforms existing frame selection strategies.

Q3. Validity of the surprisal-based importance criterion The proposed framework assumes that representations with higher surprisal are more informative. However, a larger deviation from the background mean does not necessarily imply greater relevance to a downstream task. Could the authors provide additional evidence supporting the correlation between the proposed surprisal scores and task-relevant information?

3 comments
AuthorsOpenReview ↗

From Geometric Surprisal to Downstream Utility

Thank you for your thoughtful review and for recognizing the value of connecting information-theoretic surprisal with latent geometry. Your concerns helped us clarify our central objective: to establish a query-independent, background-relative information-prioritization criterion and demonstrate its practical utility in resource-constrained VLM processing.

To strengthen the empirical evidence, we added Section 5.5, "Downstream Validation of Prior-Biased Attention."

Q1. Downstream performance after compression.

The reported 57.50% reduction represents nominal embedding-storage savings from surprisal-guided precision allocation, not a guarantee of semantic preservation after quantization. Appendix C.2 explicitly discusses the degradation caused by aggressive low-bit precision.

An important distinction in our framework is that the information criterion and its application mechanisms are separate. Geometric surprisal can guide precision allocation, hard frame selection, or soft attention redistribution.

Section 5.5 provides new downstream evidence for the latter, independently of quantization. Thus, it strengthens the empirical utility of the shared criterion, while FP16-versus-quantized accuracy remains a distinct compression-specific question.

Q2. Broader video-understanding benchmarks.

We agree that VideoMME and MVBench would be valuable for evaluating broader generalization.

However, our criterion is deliberately query-agnostic: it estimates which representations deviate from the background distribution without requiring a downstream question. This is particularly relevant to continuous streams, where information prioritization may precede any linguistic query.

Section 5.4 therefore examined whether a salient semantic transition could be preserved under extreme frame constraints, rather than claiming general query-conditioned reasoning performance.

To move beyond that single qualitative demonstration, Section 5.5 evaluates 224 new CLEVRER scenes using simulator-defined answers and identical high-redundancy inputs containing 4,608 visual tokens.

Importantly, although the prior is computed independently of the question, its effectiveness is evaluated through objective downstream recognition. This directly tests whether query-independent information prioritization can benefit query-specific predictions.

Broader video understanding remains an important extension. PMI-based contextual surprisal, discussed in our future directions, provides a natural path toward explicitly query-conditioned importance estimation.

Q3. Does geometric surprisal indicate task-relevant information?

We agree that geometric rarity does not universally imply task relevance. Our vMF derivation establishes an information ordering relative to the modeled background distribution, not guaranteed relevance to arbitrary questions.

Our hypothesis is more specific: under substantial visual redundancy, background-relative surprisal can provide a useful computational prior even without knowing the downstream query.

Section 5.5 directly tests this hypothesis. Each input contains one evidence pair among 127 repeated background pairs. We inject the surprisal-derived prior into the pre-softmax attention logits of frozen Qwen2-VL-2B-Instruct, without changing input frames, visual-token budgets, or model parameters.

Static SWMP improves recognition accuracy from 61.61% to 71.88% (+10.27 percentage points, paired exact p = 0.00219).

This demonstrates that a query-independent geometric prior can improve the recovery of task-relevant evidence through actual attention redistribution.

The controls further clarify this finding: Shuffled weighting achieves 43.30%, indicating the importance of prior placement, while FrequencyMatched achieves 72.77%, showing that simpler redundancy-aware weighting can also be effective.

Therefore, we do not claim that surprisal universally identifies the most task-relevant information. We demonstrate that background-relative geometric surprisal provides a mathematically interpretable and operationally useful prioritization criterion under the evaluated redundancy conditions.

We believe Section 5.5 substantially strengthens the empirical connection between our theoretical formulation and downstream VLM performance, while clarifying the distinction between query-independent information prioritization and query-specific semantic relevance.

Thank you again for your constructive feedback.

Thank you for your detailed rebuttal and for addressing my concerns. However, I remain unconvinced that the proposed method offers a meaningful advantage over simple cosine similarity. Moreover, the downstream utility of the resulting surprisal scores remains unclear. Even in the newly added Section 5.5, the difference between Static SWMP and MassMatched is not statistically significant. Therefore, it is still unclear what additional benefit the proposed scoring method provides. For these reasons, I maintain my original score of 1.

Thank you for your response. We acknowledge that your observation is valid. Based on the experimental results, it would be difficult to characterize the proposed method as offering a significant state-of-the-art advantage. As you pointed out, when considering the statistical significance of its performance relative to one of the alternative methods evaluated in a particular experimental setting, such superiority is not established. This is also explicitly acknowledged in the revised Section 5.5.

However, the main contribution of our research is not to demonstrate that SWMP achieves exceptional performance over other methods on a particular task. Rather, a central aspect of our contribution is that our approach differs from existing evaluation and measurement methods precisely because they do not measure or assign weights according to the proposed information-theoretic criterion.

In this context, we hope that the newly added Section 5.5 can be understood as demonstrating the potential applicability of this criterion, and its variants, to tasks where such prioritization can be effective.

This also means that our research takes a different approach from studies conducted by large research groups that establish their contributions through extensive experiments across numerous benchmarks. We do not have the capacity to pursue that scale of experimentation, nor is it the primary direction our research aims to take.

Instead, what we regard as the most significant potential contribution of this work is the mathematical and empirical connection between several well-established and actively researched areas, including information theory, the structural properties of VLM latent spaces, and attention-bias injection.

This is precisely why our paper is titled "Bridging Shannon Surprisal and Latent Geometry."

The primary purpose of our downstream experiments is to examine and demonstrate the conditions and possibilities under which this connection can be practically applied. Rather than claiming overwhelming superiority over other methods in a particular domain, the scope of our work is to demonstrate that a single underlying principle can be applied across multiple computational settings.

Empirical experimentation accounts for a substantial proportion of contemporary artificial intelligence research. Viewed from another perspective, however, this also makes research that primarily investigates theoretically grounded possibilities for practical application particularly valuable in the present research landscape.

We understand why our approach may appear to offer limited advantages in downstream task performance. We also acknowledge that one possible reason for the relatively low Soundness or Confidence assessment may be that, when viewed primarily through the lens of downstream performance, our research can appear overly broad, making it difficult to identify exactly what it seeks to establish. We recognize this possibility and respect your assessment.

Nevertheless, our research is fundamentally about demonstrating that a single principle can be applied across different settings, and that such applications are practically realizable.

We believe that demonstrating how one relatively simple principle can extend across multiple domains is an important foundation upon which the more specialized and sophisticated follow-up studies you have suggested can subsequently develop.

With this perspective in mind, we hope that this conference can also serve as a forum for discussing research questions worthy of academic consideration, exchanging ideas necessary for further progress, and exploring shared understandings of the underlying principles behind empirical results, together with their potential applications.

From the perspective of our research objectives, we recognize that your criticisms are valid in certain respects. Nevertheless, we respectfully ask you to reconsider the relationship between the specific concerns you have raised, your understanding of the paper's central research motivation, and what our study ultimately seeks to demonstrate.

We believe that a mismatch between these perspectives could contribute to a lower Soundness assessment and, consequently, influence the overall Rating.

We therefore respectfully request that you reconsider your Soundness assessment and overall Rating in light of the intended contribution and scope of our research.

Thank you again for your careful consideration and for engaging with our work.

Reviewer_RDSTOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper connects negative log density under a single von Mises–Fisher background model to cosine similarity with the inverse background direction. It uses exponentially transformed scores for mixed-precision storage, an attention-logit bias, and online frame selection using a normalized EMA. Validation comprises storage statistics for global MS-COCO CLIP embeddings, kinematic-proxy event analysis on nuScenes, and one Qwen2-VL-2B video-QA example. The mixture-vMF, patch-level dynamic pooling, and halo-retention proposals in the appendix are future architectures rather than demonstrated systems.

Critical strengths
  1. An inexpensive and interpretable historical-background deviation signal. The affine surprisal ordering is directly checkable under the specified vMF model, and N-EMA requires O(d) computation and state per frame. The authors explicitly distinguish recursive N-EMA from an exact Fréchet mean. This makes the method a meaningful, readily implementable candidate for online routing and is a substantive reason to value the work.

  2. Concrete examples demonstrate selection behavior distinct from instantaneous differences. Figures 4–5 show examples with high kinematic-proxy scores but low SWMP scores, and low adjacent differences but high SWMP scores. These observations suggest that historical context can supply a complementary signal. The appendix also identifies missed small-object/vehicle-intersection events and drift-related false positives, helping delimit the current method. These observations are informative, although they do not estimate overall performance.

Critical weaknesses
  1. The central evidence chain for semantic routing and attention improvement is incomplete. Section 5.3 evaluates events defined by latent second differences. The authors reasonably diagnose its motion bias, but proxy disagreement and selected visual examples do not establish aggregate semantic superiority. The downstream evidence in Section 5.4, Table 1, and Figure 2 is concentrated on one 20-second clip. I account for the footnote reporting the same SWMP answer at a 5% budget; the concern remains coverage and reliability. The vMF relationship and large-lambda softmax concentration are conditional results and do not establish that geometric rarity implies task relevance. Moreover, the QA experiment does not sufficiently specify whether the Section 4 logit intervention was actually executed. The evidence therefore does not isolate benefits from simple background-based preselection, the exponential transformation, and actual attention intervention. This affects the central semantic-preservation and unified-mechanism claims, rather than optional extensions.

  2. A useful quality–memory/compute tradeoff has not been established. Section 5.2 allocates 5% FP16, 55% INT8, and 40% INT4, yielding an average of 6.8 bits and hence exactly 57.5% ideal raw-payload savings relative to FP16. The arithmetic is correct, but it does not establish that surprisal-based precision allocation better preserves semantics than random or simple allocation at the same byte budget. Full-model KV bytes, metadata, and end-to-end overhead are not measured. Appendix C.2 acknowledges low-bit quality loss; the dynamic-pooling blueprint in C.3 is future work and cannot replace measurements for the current method. Consequently, the claimed storage and deployment value remains difficult to assess.

Questions for authors
  1. For the existing data, are there results on multiple preselected clips using fixed questions/reference answers or independent semantic labels, with matched actual visual-token or KV-byte budgets? Please specify prompts, scoring, threshold selection, decoding, quality, misses, and false positives. The uncertainty is whether the displayed example represents reliable task gains, which directly affects CW1. A new large-scale dataset is not required to answer this question.

2.Was the Section 4 lambda-W bias actually integrated into Qwen2-VL? If so, which layers/heads, frame-to-token mapping, and lambda were used, and what is the effect of enabling it while holding the input frames fixed? If only preselection was implemented, please clarify and compare simple distance scores with exponential weights using the same N-EMA background. The uncertainty is which algorithmic component has demonstrated benefit; this directly affects CW1.

  1. On the reported data and at the same actual storage budget, how does surprisal-based precision allocation compare with random/simple allocation in semantic task quality? Do the memory figures include quantization parameters, packing, indices, and actual KV state? The uncertainty is whether the nominal 57.5% saving produces a useful quality–resource tradeoff; this directly affects CW2.

  2. Is the baseline associated with ToMe adjacent-CLIP frame-difference selection or full in-network token matching/merging? Please define the operation and budget. If it is only a proxy, the extrapolations in B.3 and Figure 5 should be correspondingly limited. The uncertainty is the scope of superiority that the reported comparison supports, which affects the comparability assessment in CW1.

3 comments
AuthorsOpenReview ↗

Clarifying the Mechanism: From Hard Selection to Validated Attention Intervention

Thank you for your careful review. We particularly appreciate your recognition that historical-background deviation provides an interpretable signal distinct from instantaneous frame differences. Your comments helped us clarify our central contribution: a shared geometric-surprisal criterion implemented through hard frame selection and soft attention redistribution. We added Section 5.5 to strengthen the downstream evidence.

Q1. Quantitative downstream reliability.

Section 5.5 evaluates 224 new CLEVRER scenes with simulator-defined ground truth. The fixed question template is "What color is the {material} {shape} in the video?", followed by eight shuffled color options. Predictions are scored using candidate logits without subjective evaluation.

Each high-redundancy input contains one evidence pair and 127 repeated background pairs. All methods receive identical inputs with 4,608 visual tokens.

Static SWMP improves accuracy from 61.61% (138/224) to 71.88% (161/224), with 38 corrected and 15 regressed predictions (paired exact p = 0.00219). Temporal SWMP achieves 72.32%.

This provides quantitative evidence across multiple scenes, rather than relying on the original single-clip demonstration. The experiment specifically tests controlled redundancy; natural-video event-selection reliability remains a distinct evaluation question.

Q2. Actual attention implementation and component contribution.

Section 5.4 evaluates hard selection: Temporal SWMP selects informative frames before VLM inference, without modifying internal attention.

Section 5.5 separately implements soft attention intervention in frozen Qwen2-VL-2B-Instruct. At decoder layer 14 (zero-indexed), the surprisal-derived bias is added to pre-softmax logits across all attention heads and post-video suffix queries, targeting visual keys. Each frame-pair weight is mapped to its 36 visual tokens, using the development-set-normalized strength η=3\eta=3.

We cache causally independent prefix K/V states and replay the entire question suffix through subsequent decoder layers. Zero-bias replay matches Native logits within 2∗10−42 *10^{-4}.

Thus, the +10.27 percentage-point improvement with identical input frames directly demonstrates the downstream effect of actual attention intervention, independently of frame preselection.

Controls further delimit the mechanism: Shuffled achieves 43.30%, MassMatched 68.30% (difference from Static not significant, p = 0.22948), and FrequencyMatched 72.77%. These results support redundancy-aware attention redistribution, without establishing the exclusive superiority of geometric surprisal or its exponential transformation.

Q3. Storage and quality trade-off.

The reported 57.50% reduction represents nominal embedding storage, derived from the 5% FP16 / 55% INT8 / 40% INT4 allocation (6.8 versus 16 bits per element), not measured full-model KV-cache savings.

Our purpose is to investigate a common information criterion for precision allocation and attention prioritization. Section 5.5 establishes downstream utility for the latter, not semantic preservation after quantization. A matched-byte compression-quality comparison remains distinct from this result, consistent with the low-bit distortion discussed in Appendix C.2.

Q4. ToMe baseline clarification.

Our baseline uses first-order adjacent-CLIP differences as a ToMe-like input-frame selection proxy, not ToMe's in-network token matching/merging.

The comparison concerns local-difference versus historical-background-based selection under constrained frame budgets. It does not establish superiority over full ToMe. Model-internal activation-based routing and token merging are separate architectural extensions.

Overall, Section 5.5 closes the previously missing experimental connection between our theoretical attention prior and measurable downstream improvement, complementing the original hard-selection results while keeping compression-specific claims separate.

If any specific concerns remain, we would welcome your feedback and the opportunity to clarify them further.

Thank you for the rebuttal. The experiment shows that attention reweighting improves accuracy under controlled redundancy. However, simple inverse-frequency weighting performs comparably to both SWMP variants, so the added value of geometric surprisal remains unclear. The evaluation also does not substantiate the broader claims about natural-video routing or KV-cache efficiency. I therefore maintain my rating of 1

Thank you for your follow-up and for acknowledging that the new experiment demonstrates downstream improvement through actual attention reweighting.

Given more time, we would certainly welcome the opportunity to investigate these distinctions further and present additional results. We also share some disappointment that neither SWMP variant demonstrated statistically significant superiority over the alternative methods in our rebuttal experiment. However, setting aside factors beyond our immediate control and considering this result in relation to the academic contribution claimed by our paper, we do not believe that this particular limitation, although regrettable, is sufficiently fundamental to warrant extensive follow-up research and experiments solely to address it more convincingly.

This is because our research does not primarily aim to demonstrate superiority within each individual application. Rather, we believe that proposing a claim and establishing its potential validity can itself make a substantial academic contribution.

We suspect that this aspect of our contribution may not have been sufficiently apparent, leading to a lower assessment than we intended our work to warrant. This does not mean that your perspective or concerns are incorrect; on the contrary, they are reasonable from the standpoint of comparative downstream performance. We believe, however, that the difference lies in the relative importance assigned to different aspects of the research. In other words, we do not find it particularly surprising that this paper has received a mixture of positive and negative reviews. In any case, we believe that properly aligning the discussion with our central claim and research objectives is essential to explaining our work.

Therefore, when evaluating the contribution and soundness of our research, and considering what you already recognized in your review alongside the new evidence in Section 5.5, we would like to emphasize the following: The comparable performance of FrequencyMatched and SWMP does not imply that the geometric-surprisal criterion lacks an independent contribution.

Although we have introduced several experiments and application settings, the ultimate contribution of this research is to connect a background-relative geometric criterion to Shannon surprisal under a model in which inputs are mapped into a latent space, and to demonstrate that the resulting principle can be applied across different computational mechanisms.

Looking back at your original review, you identified the inexpensive and interpretable nature of historical-background deviation, the directly verifiable affine surprisal relationship under the specified vMF model, and its potential as a practical online-routing signal as positive aspects of the work. At the same time, you initially questioned whether the proposed attention-logit intervention had actually been implemented in the VLM and whether its downstream effect could be distinguished from simple frame preselection.

In light of our research objectives and intended utility, Section 5.5 has now conclusively addressed the question of actual attention intervention and its separation from frame preselection, by directly implementing the proposed mechanism in a frozen VLM and demonstrating downstream improvement while keeping the visual inputs identical. Plus, More specifically of 'the added value of geometric surprisal', the derivations and numerical-stability analyses in the appendix already address the prevention of numerical underflow and the mitigation of associated risks, and these considerations are also applied in the experiments presented in the appendix.

We therefore believe that what you describe as "the added value of geometric surprisal" has, in fact, already been substantiated through our theoretical and experimental arguments, making it difficult for us to regard assigning and maintaining a score of 1 as a natural conclusion from our perspective.

Considering these points—the successful demonstration of our paper's central objective and the resolution of an important concern raised in your original review—we respectfully believe that this research is worthy of discussion at this conference. For the reasons, we respectfully ask if you could reconsider the grounds for accepting this work, taking into account the conference's role in fostering academic exchange and diversity, as well as the potential for future research arising from a study whose contribution differs from that of large-scale empirical investigations.

We sincerely thank you for following up on our work and for sharing your thoughtful assessment.

Reviewer_JbfUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses KV Cache and computational bottlenecks in Vision-Language Models (VLMs) processing continuous video streams, proposing a unified framework that bridges Shannon's Information Theory with high-dimensional latent geometry.

1.Mathematical discovery: Proves that local surprisal exhibits an affine relationship with the cosine similarity between vectors and the inverse global mean direction

2.SWMP strategy: Quantizes low-surprisal background data, achieving 57.50% memory reduction

3.Prior-Biased Attention Mechanism: Injects surprisal weights into Transformer layers to overcome representation degeneration

4.Temporal SWMP: Functions as an information-theoretic attention router under extreme ISO-KV Cache constraints (~3% of frames), outperforming uniform sampling and adjacent-difference routing

This method achieves information-theoretic attention routing for resource-constrained continuous VLM deployments, solving the attention dilution problem caused by redundant background frames.

Critical strengths
  1. Mathematical Unification of Information Theory and High-Dimensional Geometry: The paper establishes a rigorous mathematical mapping between Shannon's Information Theory and latent space geometry (Lemma 1), proving affine relationship between surprisal and cosine similarity with inverse global mean. This transforms heuristic attention mechanisms into theoretically-driven routing based on von Mises-Fisher distribution and Maximum Entropy Principle. Unlike existing KV-Cache compression methods (ToMe, H2O, StreamingLLM) that rely on heuristic attention scores or activation magnitudes, this framework provides a provable mathematical foundation for understanding representation degeneration and designing principled sparsity strategies.

  2. Practical Value Under Extreme Constraints: In resource-constrained deployment scenarios (autonomous driving, edge devices), the paper achieves 57.50% memory reduction while maintaining performance under extreme 3% frame constraints. The Temporal SWMP routing outperforms uniform sampling and adjacent-difference routing in recall, addressing real-world bottlenecks in continuous video streams.

Critical weaknesses
  1. Insufficient Experimental Validation The paper claims 57.50% memory reduction and superior performance but lacks critical evidence. There are no quantitative metrics on downstream task quality (question answering accuracy, image retrieval mAP), only recall is mentioned. The comparison with existing methods (ToMe, H2O, StreamingLLM) is vague, making it impossible to quantify marginal contribution. Additionally, the abstract itself admits "severe downstream semantic degradation" from extreme quantization (INT4), yet no systematic evaluation of semantic quality trade-offs is provided. Without rigorous ablation studies and parameter sensitivity analysis, the core contribution's validity remains unproven. In 2026's research standards, theoretical elegance alone is insufficient—solid empirical evidence is essential.

  2. Overly Idealized Theoretical Assumptions The method relies on von Mises-Fisher distribution assumptions that may not hold for real-world data. The paper does not explain how to practically determine the anisotropic cone boundary (δmax) or whether N-EMA can accurately track global mean in online scenarios. The three parameters (α, β, γb) lack systematic guidance for selection—whether task-specific re-tuning is required remains unclear. Most critically, the computational overhead per frame for cosine similarity, EMA updates, and surprisal weight calculation is never quantified. In resource-constrained deployment, extra computation could offset memory savings. This gap between theory and practice significantly undermines the paper's practical contribution.

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Strengthening the Empirical Foundation of Geometric Surprisal

Thank you for your encouraging and constructive review. We particularly appreciate your recognition of the mathematical connection between Shannon surprisal and latent geometry, and its potential for resource-constrained VLM inference.

To address W1 (downstream validation), we added Section 5.5, evaluating actual Prior-Biased Attention in frozen Qwen2-VL-2B-Instruct across 224 CLEVRER scenes.

With identical high-redundancy inputs containing 4,608 visual tokens, Static SWMP improves recognition accuracy from 61.61% to 71.88% (+10.27 percentage points, paired exact p = 0.00219). Unlike our earlier qualitative frame-selection example, this experiment directly implements the attention-logit intervention and includes controls examining the improvement.

Regarding W2 (theoretical assumptions and practical considerations), the vMF relationship is derived under an explicit background-model assumption. N-EMA requires O(d) computation and state per frame, while α controls geometric sensitivity and β adjusts the numerical operating range. We distinguish these analytical properties from unmeasured end-to-end runtime and KV-cache overhead.

Our central contribution is a shared geometric-surprisal criterion applicable to storage allocation, hard frame selection, and soft attention redistribution. The new experiment strengthens the empirical connection between this criterion and downstream performance.

Thank you again for your valuable assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.