acceptodds
Under review as a conference paper at ICLR 2027

When Preservation Misleads: Scale-Dependent Audits of Frozen Video-Language Interfaces

Abstract

Visual-token compression changes both the selected evidence and the values delivered to a frozen language model. Explaining a performance change requires distinguishing these effects. We introduce matched-packet audits: paired value interventions that hold source identities, token count, order, positions, and layout fixed. In native PruneVid, aggregation increases directional concentration at all three tested strengths, but the largest concentration change does not coincide with the largest reference-relative recall gap. At Medium, held-out alignment increases recall by 0.0237 under video-level scoring references while preserving delivery-stage directional diversity. The improvement therefore cannot be explained by recovery of that diversity; the query-level scoring estimate is inconclusive. Controlled experiments expose two further interpretation problems. Orthogonal interventions preserve norms, pairwise inner products, and linear CKA while reducing recall by 80.0–87.9% across four frozen consumers. Spatial–newline comparisons change sign between absolute and high relative perturbation amplitudes, while a common low-relative-dose window shows no detected recall separation. These audits distinguish representation change, behavioral loss, and response to a correction. They show how explicit source references and controlled interventions can rule out misleading explanations of compression outcomes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Video-language models, visual token compression, representation similarity, multimodal learning, model evaluation, feature alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zo3DOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This study investigates how visual token compression affects frozen video-language models (VLMs). The authors introduce "matched-packet audits" to isolate feature-value changes by keeping token structures, positions, and prompts fixed. Using a "reference-path identity," the framework decouples packet-structure contrasts from value contrasts, applying an orthogonal map to test behavioral responses to feature corrections.

Experiments test four visual interfaces—including PruneVid with PLLaVA-7B across three intensities—using 150 queries from Charades videos. Metrics evaluate lexical recall, teacher-forced token NLL, and MVBench accuracy.

Results show that orthogonal interventions preserve geometric summaries but degrade downstream performance. Spatial and newline perturbation rankings reverse between absolute and high-relative doses, though low-relative doses yield no recall separation. Native aggregation increases directional concentration without ordering recall gaps. Lastly, a Medium-setting held-out map improves video-reference recall and diversity.

Critical strengths
  1. Matched-packet and scale controls reveal that structural-token vulnerability rankings depend heavily on intervention definitions. By fixing consumer-side metadata and varying specific values, a spatial-newline comparison progressively matches traffic features before altering the dose definition. This shifts the recall contrast from +0.0946 under absolute matching to -0.1050 under high local-relative matching, while low-dose separation remains undetected. This provides a practical design principle for interpreting compression ablations.

  2. A native audit linking mathematical invariance to mechanistic exclusion disproves a common explanation for model gains. Aggregation reduces directional rank across all strengths after multiplicity adjustment. Because the Medium correction improves the video-reference recall estimate while preserving the delivered normalized-row Gram matrix, restoring this specific diversity statistic cannot explain the performance gain. Detailed baseline, sensitivity, and alternative source data provide a comprehensive evaluation framework over isolated metrics.

Critical weaknesses

Limited empirical generality weakens the application to realistic compression. The core evaluation relies on a small cohort of 150 Charades queries from 138 videos. Its positive Medium correction holds only under video-reference lexical recall, becoming inconclusive under query-reference recall, answer-token NLL, and the adjusted follow-up

Questions for authors

Does the Medium correction benefit remain under reasonable changes to the held-out calibration-video subset, and do gain-only versus orthogonal-only maps produce the same behavior?

2 comments

Thank you for asking how sensitive the Medium-setting gain is to calibration-video selection and whether it comes from the orthogonal map or the uniform gain. We tested both on the same 150 evaluation queries from 138 videos.

Sensitivity to the calibration subset. We refitted the alignment on five randomly sampled subsets of 48 videos from the original 96-video calibration pool. All calibration videos remained disjoint from evaluation. The paired gains over native compression in video-reference recall were +0.0218, +0.0016, +0.0190, +0.0077, and +0.0226, averaging +0.0145, compared with +0.0237 for the original 96-video fit. Three of the five individual 95% paired video-cluster bootstrap intervals excluded zero before multiplicity adjustment. The estimated gain was positive across all five refits, but its magnitude varied substantially, and two subsets yielded little improvement. Since the refits share a calibration pool and evaluation cohort, they assess sensitivity to calibration choice rather than independent replication.

Uniform gain versus orthogonal map. We used the original fitted QQ and ss to compare the native packet CC, gain-only sCsC, orthogonal-only CQCQ, and full alignment sCQsCQ. Relative to CC, orthogonal-only improved video-reference recall by +0.0228 (adjusted 99.17% CI [+0.0012, +0.0518]), close to the full alignment gain of +0.0237 [+0.0021, +0.0523]. Gain-only yielded +0.0073 [−0.0136, +0.0342]. Adding ss to CQCQ yielded +0.0008 [−0.0158, +0.0172]. These intervals account for the six-contrast family, which comprises five pairwise contrasts and one interaction contrast. Thus QQ alone reproduced nearly all of the observed gain, with no detected additional benefit from applying ss on top of QQ.

These follow-ups clarify the Medium result on the existing cohort. The full-map query-reference recall and answer-token NLL results remain inconclusive, so the evidence for improvement is specific to video-reference lexical recall. Finally, to echo the point noted in your second strength: alignment leaves the normalized-row Gram matrix unchanged, so the observed recall gain cannot be attributed to restoring the delivery-stage directional diversity measured in the paper.

Thank you again for your review. In response to your question, we refitted the alignment on five calibration subsets and tested the orthogonal map and uniform scaling separately. All five refits had positive point estimates, although the gains varied, and the orthogonal map alone reproduced nearly all of the observed full-map gain (Section 4.3 and Appendix C.8). If these results address your concern, would you consider revisiting your overall rating?

Reviewer_cuBsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to interpret performance changes caused by visual-token compression in frozen video-language models. Its main idea is a matched-packet audit: compare original source vectors, compressed vectors, and aligned compressed vectors while keeping token count, ordering, positions, and the downstream model fixed. The experiments show that orthogonal transformations can preserve several geometric measures while substantially degrading model behavior, and that spatial-versus-newline sensitivity depends on how perturbation magnitude is defined. In native PruneVid, aggregation increases directional concentration at all three compression strengths, while a held-out alignment improves Medium-setting recall without restoring delivery-stage directional diversity. The paper also reports sensitivity to scoring references and largely null results on additional tasks and compression methods.

Critical strengths
  • The matched-packet comparisons are carefully constructed. Keeping the realized groups and consumer-side metadata fixed makes the value-replacement contrast much easier to interpret than a conventional full-versus-compressed comparison. Reporting the absolute condition means is also useful: the Strong-setting results show clearly why a smaller reference-relative gap need not mean better preservation.

  • The native alignment experiment provides a concrete test of one proposed explanation. Because the orthogonal map and uniform gain preserve the normalized Gram matrix, the Medium-setting recall improvement cannot be attributed to restoring delivery-stage directional diversity. I also appreciate that the paper reports the inconclusive query-reference results and unsuccessful extensions rather than presenting alignment as a generally effective repair.

Critical weaknesses
  • The evidence for the main native-compression finding is still narrow. At Medium, alignment improves video-reference word recall by 0.0237, but the query-reference estimate is 0.0124 with a confidence interval spanning zero. The corresponding answer-token NLL improvement is also inconclusive (Appendix C.1). Neither transferred nor newly fitted alignment improves MVBench accuracy, and the ToMe/STTM extensions provide no clear positive repair result. These outcomes do not invalidate the video-reference finding, and the paper correctly avoids claiming that the scoring references produce statistically different effects. Still, I am left uncertain whether the central improvement reflects better video understanding or a change in lexical coverage. This matters because that improvement carries much of the paper’s practical argument.

  • The controlled experiments establish useful cautions, but their implications for realistic compression are less convincing. A large rotation at a frozen interface directly changes the coordinates read by the downstream network; the resulting failure is therefore unsurprising even when pairwise geometry is preserved. Likewise, the high-relative-dose spatial intervention uses a displacement around five times the source norm and increases absolute energy by thousands of times relative to N208 (Appendix B.5). It demonstrates a scale-dependent reversal, but its relevance to ordinary compression distortions remains unclear. The low-dose experiment is helpful, although it finds no detected recall separation. Taken together, these results motivate careful auditing more strongly than they establish a broadly useful diagnosis of compression failures.

Questions for authors
  • For the native Medium-setting alignment, can you provide a paired assessment of whether the changed generations describe the video more accurately, beyond matching more reference words? A blinded assessment of changed outputs would help resolve my main concern about the behavioral endpoint.

  • How much of the Medium-setting gain comes from the orthogonal map versus the uniform gain? A scale-only comparison would clarify whether the observed response requires a coordinate correction.

  • Where do the perturbation magnitudes produced by native compression fall relative to the controlled dose experiments? In particular, does the spatial–newline reversal occur at amplitudes comparable to those encountered in practical compression?

3 comments

Thank you for asking whether the Medium-setting gain reflects more accurate action descriptions, which component of alignment contributes to the gain, and how the controlled doses compare with native compression. We ran three follow-up analyses to address these questions.

1. Do the aligned generations describe the video more accurately?

We conducted a blinded, paired assessment on the same 150 queries from 138 videos. Two vision-language evaluators independently saw the 16 sampled frames, the question, and the two complete answers in randomized A/B order. They scored action correctness on a 0–2 scale without seeing condition labels or reference captions.

The mean score was 1.5800 before alignment and 1.6033 after alignment. The mean paired difference across all 150 queries was +0.0233; among the 67 queries whose generated outputs changed, the mean paired difference was +0.0597. Thus, the blinded action-accuracy ratings move in the same positive direction as the video-reference recall gain, although the estimated improvement is small.

2. Does the gain come from the orthogonal map QQ or the uniform scale ss?

We evaluated CC, sCsC, CQCQ, and sCQsCQ on the same realized packets using the original fitted QQ and ss. We report paired differences in video-reference recall. The 99.17% marginal intervals provide nominal 95% familywise coverage for five pairwise contrasts and one interaction contrast.

ContrastVideo-reference recall difference99.17% CI
Scale only − compressed+0.0073[−0.0136, +0.0342]
Orthogonal only − compressed+0.0228[+0.0012, +0.0518]
Full alignment − compressed+0.0237[+0.0021, +0.0523]
Full alignment − scale only+0.0164[−0.0036, +0.0390]
Full alignment − orthogonal only+0.0008[−0.0158, +0.0172]
Interaction: (full − orthogonal) − (scale − compressed)−0.0065[−0.0376, +0.0200]

The orthogonal map alone reproduced nearly all of the observed video-reference recall gain. We did not detect an additional benefit from adding the fitted scale to QQ, and the interaction estimate was inconclusive. The corresponding query-reference contrasts across the same six-comparison family were inconclusive.

3. Are the controlled doses comparable to native compression?

For merged PruneVid groups, the median relative displacement from the source representative was 0.193, 0.232, and 0.280 at Light, Medium, and Strong, respectively. With equal video weighting, 0.76%, 0.96%, and 6.19% of merged tokens exceeded 0.4. The high-relative-dose spatial intervention (r≈4.97r\approx4.97, the synthetic relative displacement amplitude in Section 4.2) was therefore far larger than typical native displacements. These magnitudes provide context only; native and synthetic perturbations differ in direction and in the tokens they affect.

To test magnitude directly along a native compression direction, we evaluated six values of α\alpha in R+α(C−R)R+\alpha(C-R) at Medium, keeping the realized groups, token count, and consumer settings fixed. Here RR is the source representative and α=1\alpha=1 delivers the native aggregate CC. Reducing α\alpha to 0.75—moving partway from the native aggregate toward the source representative—increased video-reference recall by +0.0203 (Bonferroni-adjusted 97.5% CI [+0.0028, +0.0427]). The other prespecified contrast, α=1.25\alpha=1.25 versus 1, was inconclusive, as were both query-reference contrasts. This shows that changing the magnitude of a native aggregation displacement can affect recall at a fixed token budget. Whether native compression produces the spatial–newline reversal remains untested.

Thank you again for your insightful review and questions. As the Thank you again for your insightful review and questions. As the discussion period is drawing to a close, we wanted to kindly draw your attention to our recent response above, which directly addresses your core questions regarding the Medium-setting gain:

  1. Calibration Subset Stability & Blinded Evaluation: We refitted the alignment on five randomly sampled calibration subsets (all yielding positive point estimates) and conducted a blinded paired assessment, showing that the aligned generations move in the same positive direction as the video-reference recall gain (Section 4.3 and Appendix C.8).
  2. Orthogonal Map vs. Uniform Scale: We isolated the two components and demonstrated that the orthogonal map (QQ) alone reproduces nearly all of the observed full-map gain, with no additional benefit from the uniform scale (ss).

If these additional analyses and results help address your concerns regarding our behavioral endpoints and alignment mechanics, would you be willing to kindly reconsider your rating and evaluation? Thank you once again for your valuable time and guidance.

Reviewer_cuBsOpenReview ↗

When Preservation Misleads: Scale-Dependent Audits of Frozen Video-Language Interfaces Download PDF

Thanks for running these follow-up analyses. The ablation makes it clearer that most of the recall gain comes from the orthogonal map. The blinded ratings are helpful, but the improvement is small, and the query-reference results are still inconclusive. It also remains unclear whether the spatial–newline reversal occurs under native compression. I’m keeping my original rating.

Reviewer_UETCOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces matched-packet audits, a framework for analyzing the effects of visual-token compression in video-language models. It disentangles source-selection effects from feature-value changes by comparing matched token packets while keeping token identities, counts, positions, and downstream model settings fixed. Experiments show that preserving representation geometry does not guarantee downstream performance and improvements from feature alignment do not necessarily correspond to restored representation diversity.

Critical strengths
  1. By disentangling source-token selection from feature-value changes, the proposed matched-packet audits offer an interesting perspective on the interpretability of visual-token pruning and compression. This decomposition helps identify how different components of compression affect downstream model behavior.

  2. The paper conducts extensive controlled experiments across multiple video-language models and compression settings. It also evaluates selected experiments with multiple random seeds and reports confidence intervals with appropriate statistical corrections.

Critical weaknesses

1. Limited Novelty and Contributions. The paper demonstrates that transformations of visual tokens can degrade downstream performance even when certain geometric properties are preserved. However, this observation is largely intuitive, since a frozen language model is not expected to remain invariant to arbitrary transformations of its input representations. The paper offers limited new insights beyond confirming this expected behavior. Moreover, it lacks a more generalizable explanatory mechanism for understanding how different transformations affect downstream performance.

2. Lack of Clarity and Conciseness in Presentation. I found it difficult to identify the paper's key contributions despite careful reading. The experimental section does not provide a clear overview of the models and benchmarks used, making the experimental setup difficult to follow (For example, the accuracy drop of Qwen2.5-VL on MVBench is only briefly mentioned in the text (Lines 273–274) rather than clearly presented in a table. Furthermore, readers must frequently navigate between the Methods and Results sections to understand individual experiments, such as, the configurations of the experiment discussed in Section 4.2 are primarily described in Section 3.2 (Lines 166–170). Table 2 is also relatively difficult to interpret due to its dense presentation of "Rep. - Comp." and "Repair - Comp." columns. Finally, the citation format does not follow the ICLR guideline and most references are published in 2024.

Questions for authors
  1. Why does alignment improve performance at Medium compression but not at Light or Strong settings?

  2. How is Representative-K justified as a reliable baseline for isolating source-selection effects?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.