- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper is an analysis study of a small residual adapter inserted between the visual projector and the frozen LLM decoder of video-language models, under uncompressed inputs and four token-compression methods. The main observations are: (1) although the hidden features and readout weights are broad, the learned correction concentrates almost entirely in one output direction; (2) the concentration comes from learned hidden-readout alignment, not from either factor alone; (3) a one-unit adapter constructed from Wide reaches much better NLL than a one-unit adapter trained directly; (4) at an exactly matched one-axis function, an unrestricted readout has a larger local learning space than a one-axis readout; (5) projecting onto the dominant direction can keep >99% of correction energy yet still raise NLL and change individual predictions.
Critical strengths
- The observations are checked across several input conditions and three backbones.
- The experiments are careful and well controlled.
Critical weaknesses
- The main insights are largely expected from existing theory, and the paper does not clearly go beyond it. For example, with a zero-initialized readout, gradient descent on a factorized model is known to learn the top singular direction first and to favor low-rank solutions (Arora et al. 2019; Li et al. 2020a). The observation that energy retention does not guarantee behavior preservation is also already the motivation of activation-aware compression such as SVD-LLM.
- The setting is narrow, and it is unclear whether the conclusions generalize.
- The paper does not explain what the dominant direction actually does.
- My major concern of this paper is no actionable outcome. It does not propose a better adapter, training method or compression rule based on the analysis, and the practical advice in the Discussion is fairly generic.
Questions for authors
The manuscript reads like an exhaustive characterization of a narrow phenomenon (many controls and terms, very small effect sizes, little motivation or application).
6 comments
Response to Reviewer sB3Z
W2.
The setting is narrow, and it is unclear whether the conclusions generalize.
Response
Concentration recurs in three backbones from two families. The primary study covers four benchmarks, uncompressed inputs, and four compressors. In InternVL2.5-2B, one direction carries 99.5% of raw energy, yet retaining three improves MVBench accuracy by 0.63 percentage points over one, with a paired 95% interval of [0.05, 1.18]. Proposition 1 applies to affine readouts with fixed hidden features independently of the compressor. The detailed continuation test uses Qwen2.5-VL-7B with FastVID.
Response to Reviewer sB3Z
W3.
The paper does not explain what the dominant direction actually does.
Response
On 512 held-out videos, keeping the learned axis raises decoder attention to visual tokens. Removing it returns attention close to the unadapted level at both checkpoints. We then substitute three random orthogonal directions while preserving each token coefficient and correction magnitude. The learned axis gives the correct option a larger margin over a fixed wrong option than all three controls at the decoder-analysis checkpoint. All three adjusted intervals are positive. One of three adjusted intervals is positive at the shared-rate checkpoint. Section 5.2 now summarizes these interventions from Appendix G.
Response to Reviewer sB3Z
Thank you for recognizing the careful controls and model comparisons. We disagree that low rank alone predicts the central results.
W1.
Response
Arora et al. and Li et al. explain how matrix factorization can favor low rank. Even an exactly rank-one correction leaves open whether the task gradient uses the extra directions of an unrestricted readout. At three functions learned by Direct, both readouts match exactly with the same features. Even the optimal single pattern of token coefficients misses 17.3–18.9% of the transverse task-gradient energy accessible to the unrestricted readout. From broad nonzero initialization, the number of directions needed for 99% of raw energy falls from 60 to three or four. At six checkpoints, all 20 rotations preserving both factor spectra also weaken concentration.
SVD-LLM uses calibration activations to control layer-output reconstruction error when compressing weights. In Section 5.1 we project the actual token corrections at the endpoints of our matched-function learning experiment. Projection raises held-out negative log-likelihood (NLL) in every run.