- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This study investigates how visual token compression affects frozen video-language models (VLMs). The authors introduce "matched-packet audits" to isolate feature-value changes by keeping token structures, positions, and prompts fixed. Using a "reference-path identity," the framework decouples packet-structure contrasts from value contrasts, applying an orthogonal map to test behavioral responses to feature corrections.
Experiments test four visual interfaces—including PruneVid with PLLaVA-7B across three intensities—using 150 queries from Charades videos. Metrics evaluate lexical recall, teacher-forced token NLL, and MVBench accuracy.
Results show that orthogonal interventions preserve geometric summaries but degrade downstream performance. Spatial and newline perturbation rankings reverse between absolute and high-relative doses, though low-relative doses yield no recall separation. Native aggregation increases directional concentration without ordering recall gaps. Lastly, a Medium-setting held-out map improves video-reference recall and diversity.
Critical strengths
-
Matched-packet and scale controls reveal that structural-token vulnerability rankings depend heavily on intervention definitions. By fixing consumer-side metadata and varying specific values, a spatial-newline comparison progressively matches traffic features before altering the dose definition. This shifts the recall contrast from +0.0946 under absolute matching to -0.1050 under high local-relative matching, while low-dose separation remains undetected. This provides a practical design principle for interpreting compression ablations.
-
A native audit linking mathematical invariance to mechanistic exclusion disproves a common explanation for model gains. Aggregation reduces directional rank across all strengths after multiplicity adjustment. Because the Medium correction improves the video-reference recall estimate while preserving the delivered normalized-row Gram matrix, restoring this specific diversity statistic cannot explain the performance gain. Detailed baseline, sensitivity, and alternative source data provide a comprehensive evaluation framework over isolated metrics.
Critical weaknesses
Limited empirical generality weakens the application to realistic compression. The core evaluation relies on a small cohort of 150 Charades queries from 138 videos. Its positive Medium correction holds only under video-reference lexical recall, becoming inconclusive under query-reference recall, answer-token NLL, and the adjusted follow-up
Questions for authors
Does the Medium correction benefit remain under reasonable changes to the held-out calibration-video subset, and do gain-only versus orthogonal-only maps produce the same behavior?
2 comments
Thank you again for your review. In response to your question, we refitted the alignment on five calibration subsets and tested the orthogonal map and uniform scaling separately. All five refits had positive point estimates, although the gains varied, and the orthogonal map alone reproduced nearly all of the observed full-map gain (Section 4.3 and Appendix C.8). If these results address your concern, would you consider revisiting your overall rating?
Thank you for asking how sensitive the Medium-setting gain is to calibration-video selection and whether it comes from the orthogonal map or the uniform gain. We tested both on the same 150 evaluation queries from 138 videos.
Sensitivity to the calibration subset. We refitted the alignment on five randomly sampled subsets of 48 videos from the original 96-video calibration pool. All calibration videos remained disjoint from evaluation. The paired gains over native compression in video-reference recall were +0.0218, +0.0016, +0.0190, +0.0077, and +0.0226, averaging +0.0145, compared with +0.0237 for the original 96-video fit. Three of the five individual 95% paired video-cluster bootstrap intervals excluded zero before multiplicity adjustment. The estimated gain was positive across all five refits, but its magnitude varied substantially, and two subsets yielded little improvement. Since the refits share a calibration pool and evaluation cohort, they assess sensitivity to calibration choice rather than independent replication.
Uniform gain versus orthogonal map. We used the original fitted Q and s to compare the native packet C, gain-only sC, orthogonal-only CQ, and full alignment sCQ. Relative to C, orthogonal-only improved video-reference recall by +0.0228 (adjusted 99.17% CI [+0.0012, +0.0518]), close to the full alignment gain of +0.0237 [+0.0021, +0.0523]. Gain-only yielded +0.0073 [−0.0136, +0.0342]. Adding s to CQ yielded +0.0008 [−0.0158, +0.0172]. These intervals account for the six-contrast family, which comprises five pairwise contrasts and one interaction contrast. Thus Q alone reproduced nearly all of the observed gain, with no detected additional benefit from applying s on top of Q.
These follow-ups clarify the Medium result on the existing cohort. The full-map query-reference recall and answer-token NLL results remain inconclusive, so the evidence for improvement is specific to video-reference lexical recall. Finally, to echo the point noted in your second strength: alignment leaves the normalized-row Gram matrix unchanged, so the observed recall gain cannot be attributed to restoring the delivery-stage directional diversity measured in the paper.