acceptodds
Under review as a conference paper at ICLR 2027

Train Wide, Act Narrow: The Learning Geometry of Visual Adaptation

Abstract

In video-language adaptation, wide visual adapters learn corrections concentrated almost entirely along one output direction, while their hidden features and readout weights remain broad. Across the compression methods we study, retaining that direction preserves nearly all of the average benchmark gain, while removing it returns accuracy close to the unadapted model. We call this pattern *Train Wide, Act Narrow*. This concentration emerges through learned hidden-readout alignment, with the readout preferentially amplifying a dominant mode of hidden variation. A one-hidden-unit adapter constructed from a trained Wide adapter reaches development negative log-likelihood (NLL) of 1.200, while the best task-trained counterpart we test reaches 1.403. We then compare two readouts that compute exactly the same one-axis function from the same hidden features. The unrestricted readout can produce transverse changes using coefficient patterns unavailable to the one-axis readout. Even the best single pattern misses 17.3 to 32.1% of the transverse gradient energy accessible to the unrestricted readout. After continuation from the same one-axis function, projecting the resulting unrestricted correction onto its dominant direction raises NLL despite retaining more than 99.5% of its raw energy. A nearly one-dimensional correction can require more freedom to learn and more than one direction to preserve its behavior.

Primary area:
interpretability and explainable AI
Keywords:
Video-language models, representation learning, learning dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sB3ZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper is an analysis study of a small residual adapter inserted between the visual projector and the frozen LLM decoder of video-language models, under uncompressed inputs and four token-compression methods. The main observations are: (1) although the hidden features and readout weights are broad, the learned correction concentrates almost entirely in one output direction; (2) the concentration comes from learned hidden-readout alignment, not from either factor alone; (3) a one-unit adapter constructed from Wide reaches much better NLL than a one-unit adapter trained directly; (4) at an exactly matched one-axis function, an unrestricted readout has a larger local learning space than a one-axis readout; (5) projecting onto the dominant direction can keep >99% of correction energy yet still raise NLL and change individual predictions.

Critical strengths
  • The observations are checked across several input conditions and three backbones.
  • The experiments are careful and well controlled.
Critical weaknesses
  • The main insights are largely expected from existing theory, and the paper does not clearly go beyond it. For example, with a zero-initialized readout, gradient descent on a factorized model is known to learn the top singular direction first and to favor low-rank solutions (Arora et al. 2019; Li et al. 2020a). The observation that energy retention does not guarantee behavior preservation is also already the motivation of activation-aware compression such as SVD-LLM.
  • The setting is narrow, and it is unclear whether the conclusions generalize.
  • The paper does not explain what the dominant direction actually does.
  • My major concern of this paper is no actionable outcome. It does not propose a better adapter, training method or compression rule based on the analysis, and the practical advice in the Discussion is fairly generic.
Questions for authors

The manuscript reads like an exhaustive characterization of a narrow phenomenon (many controls and terms, very small effect sizes, little motivation or application).

6 comments
AuthorsOpenReview ↗

Response to Reviewer sB3Z

Thank you for recognizing the careful controls and model comparisons. We disagree that low rank alone predicts the central results.

W1.

The main insights are largely expected from existing theory, and the paper does not clearly go beyond it. For example, with a zero-initialized readout, gradient descent on a factorized model is known to learn the top singular direction first and to favor low-rank solutions (Arora et al. 2019; Li et al. 2020a). The observation that energy retention does not guarantee behavior preservation is also already the motivation of activation-aware compression such as SVD-LLM.

Response

Arora et al. and Li et al. explain how matrix factorization can favor low rank. Even an exactly rank-one correction leaves open whether the task gradient uses the extra directions of an unrestricted readout. At three functions learned by Direct, both readouts match exactly with the same features. Even the optimal single pattern of token coefficients misses 17.3–18.9% of the transverse task-gradient energy accessible to the unrestricted readout. From broad nonzero initialization, the number of directions needed for 99% of raw energy falls from 60 to three or four. At six checkpoints, all 20 rotations preserving both factor spectra also weaken concentration.

SVD-LLM uses calibration activations to control layer-output reconstruction error when compressing weights. In Section 5.1 we project the actual token corrections at the endpoints of our matched-function learning experiment. Projection raises held-out negative log-likelihood (NLL) in every run.

AuthorsOpenReview ↗

Response to Reviewer sB3Z

W2.

The setting is narrow, and it is unclear whether the conclusions generalize.

Response

Concentration recurs in three backbones from two families. The primary study covers four benchmarks, uncompressed inputs, and four compressors. In InternVL2.5-2B, one direction carries 99.5% of raw energy, yet retaining three improves MVBench accuracy by 0.63 percentage points over one, with a paired 95% interval of [0.05, 1.18]. Proposition 1 applies to affine readouts with fixed hidden features independently of the compressor. The detailed continuation test uses Qwen2.5-VL-7B with FastVID.

AuthorsOpenReview ↗

Response to Reviewer sB3Z

W3.

The paper does not explain what the dominant direction actually does.

Response

On 512 held-out videos, keeping the learned axis raises decoder attention to visual tokens. Removing it returns attention close to the unadapted level at both checkpoints. We then substitute three random orthogonal directions while preserving each token coefficient and correction magnitude. The learned axis gives the correct option a larger margin over a fixed wrong option than all three controls at the decoder-analysis checkpoint. All three adjusted intervals are positive. One of three adjusted intervals is positive at the shared-rate checkpoint. Section 5.2 now summarizes these interventions from Appendix G.

Reviewer_9rjaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates the learning behavior of visual adapters in video-language models, focusing on the relationship between hidden width and correction dimensionality. The authors observe that wider adapters achieve better training performance while their learned corrections remain concentrated along a small number of output directions. Through comparisons between different adapter configurations, the paper investigates the distinction between the ability to represent a compact correction and the flexibility required to learn it. The authors further provide mathematical analyses of local learning spaces and empirical evaluations across multiple video-language models and compression settings.

The major weakness of this paper is its presentation. The manuscript is extremely dense and difficult to follow, introducing numerous terminologies and experimental findings without sufficient background, intuitive explanations, or a coherent scientific narrative connecting the research motivation, methodology, and results. These presentation issues significantly hinder my ability to understand and independently assess the validity and significance of the paper's central contributions.

Critical strengths

S1. Relevant research question. The paper investigates the learning behavior of visual adapters in video-language models, focusing on the relationship between hidden width, correction dimensionality, and optimization. Understanding this relationship could provide useful insights into parameter-efficient adaptation.

However, due to the substantial presentation issues discussed below, I was unable to confidently evaluate the significance of the paper's technical contributions or identify additional strengths. In particular, the lack of a clear research narrative and sufficient conceptual explanations made it difficult for me to assess the proposed analyses and findings.

Critical weaknesses

W1. Unclear research motivation and narrative. The abstract and introduction are extremely dense, introducing numerous technical concepts and experimental findings without sufficient background or intuitive explanation. In particular, the first paragraph of the introduction largely repeats the abstract rather than further developing the research motivation. The connections between the research question, observations, and contributions are not clearly communicated, making it difficult to understand the paper's central message.

W2. Unclear presentation of methodology. Section 2.1 introduces several adapter configurations (e.g., Base, Wide, Width1, and Direct) without systematically explaining their architectural differences or the motivation for each comparison. Additional concepts, such as Wide lift and local learning spaces, further increase the complexity of the narrative. Although mathematical definitions are provided, a clearer conceptual overview, comparison table, or architecture diagram would substantially improve readability.

W3. Fragmented experimental presentation and limited discussion. The experimental sections frequently refer readers to the appendix for important details and comparisons, making the results difficult to follow without repeatedly navigating between sections. Furthermore, several findings are primarily described through numerical results, with limited discussion of their underlying mechanisms, practical significance, or relationship to the central research question. A more structured presentation and deeper interpretation of the main findings would help clarify the paper's contributions.

Questions for authors

No specific technical questions at this stage. My primary concerns relate to the paper's presentation, organization, and accessibility, as detailed in the weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9rja

Thank you for recognizing the relevance of this question to parameter-efficient adaptation. We agree that the original opening made the motivation harder to see.

W1. Unclear research motivation and narrative. The abstract and introduction are extremely dense, introducing numerous technical concepts and experimental findings without sufficient background or intuitive explanation. In particular, the first paragraph of the introduction largely repeats the abstract rather than further developing the research motivation. The connections between the research question, observations, and contributions are not clearly communicated, making it difficult to understand the paper's central message.

Response

Keeping the dominant correction direction preserves nearly all of the average benchmark gain across four compression methods. Does that justify training on one axis from the outset? Does it justify keeping just one axis after training? For the first question we start two readouts at exactly the same function. For the second we project the very models that acquired a fitting advantage during continued training. These comparisons test what the dominant direction actually explains.

The Introduction now opens with the video-language setting and explains why the adapter lets us vary the correction at a fixed token budget. The research question comes before the results. We simplified the one-unit comparison in the abstract and removed the repeated numbers from the opening paragraph.

AuthorsOpenReview ↗

Response to Reviewer 9rja

W2. Unclear presentation of methodology. Section 2.1 introduces several adapter configurations (e.g., Base, Wide, Width1, and Direct) without systematically explaining their architectural differences or the motivation for each comparison. Additional concepts, such as Wide lift and local learning spaces, further increase the complexity of the narrative. Although mathematical definitions are provided, a clearer conceptual overview, comparison table, or architecture diagram would substantially improve readability.

Response

We added the requested overview as Table 1 in Section 2.1. Base is the unadapted reference. Wide learns a correction with 64 hidden units. Width1 reduces hidden computation to one unit. Direct retains 64 units and restricts the output to one learned axis. Keep and Remove intervene on a trained correction.

Figure 1 illustrates the purpose of Wide lift. It reproduces the starting function of Direct exactly with the same hidden features. Consider two tokens with equal current coefficients but different hidden features. When Direct rotates its axis, those tokens must move together perpendicular to it. Wide lift can move them differently. The local learning space consists of the changes each readout can make to first order. Figure 4 shows that the task gradient uses the additional directions even at solutions learned by Direct itself.

AuthorsOpenReview ↗

Response to Reviewer 9rja

W3. Fragmented experimental presentation and limited discussion. The experimental sections frequently refer readers to the appendix for important details and comparisons, making the results difficult to follow without repeatedly navigating between sections. Furthermore, several findings are primarily described through numerical results, with limited discussion of their underlying mechanisms, practical significance, or relationship to the central research question. A more structured presentation and deeper interpretation of the main findings would help clarify the paper's contributions.

Response

The main figures test a connected explanation. In Figure 3 we rotate the readout relative to the hidden features while preserving both spectra. At each of six checkpoints, all 20 rotations reduce the leading share of centered correction energy. Learned alignment therefore contributes to concentration. In Figure 5, two readouts start at the same function. The unrestricted readout reaches lower development negative log-likelihood (NLL) in all three runs. Projecting those same endpoints back to one direction erases the advantage in every point estimate despite retaining more than 99.5% of raw energy.

Section 5.2 shows why this matters. At one FastVID checkpoint, keeping four directions instead of one raises agreement with the answers generated by Wide from 92.44% to 96.35%, while MVBench accuracy changes only from 52.62% to 52.94%. Four directions better preserve the fitted answers in this case.

Section 5.1 now gives the longer-training and projection findings in the main text. Section 5.2 connects the direction intervention in Appendix G to visual attention and answer margins. We respectfully ask you to reconsider the soundness assessment in light of these controls and the evidence for each claim.

Thank you for the response. I acknowledge and appreciate the authors' efforts to improve the manuscript's clarity and readability, particularly by adding Table 1 to systematically compare the adapter configurations and revising the opening of the introduction to better establish the video-language setting and the research motivation and contributions.

However, after carefully comparing the revised manuscript with the original version, I remain concerned that these changes do not sufficiently address the fundamental presentation issues raised in my original review, which I believe require more substantial revisions. Additionally, the highlighted text, which I understand to indicate revisions, does not fully correspond to the actual changes between the two versions, making it more difficult to identify and evaluate the revisions.

Therefore, I have decided to maintain my original score. Nevertheless, I hope the authors will consider a more thorough revision of the manuscript to better communicate the motivation, scientific contributions, and significance of their findings.

Reviewer_VLMaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the relationship between hidden width and correction dimensionality in visual adapters for video-language models. The experiments show that increasing hidden width improves model fitting, while the learned corrections remain concentrated along a single dominant output direction. The authors term this phenomenon "Train Wide, Act Narrow". They investigate its underlying mechanism through spectral analysis, low-dimensional projections, and one-unit adapter experiments.

Critical strengths
  1. The distinction between hidden computation width and effective correction dimensionality offers a useful perspective on parameter redundancy and optimization in visual adapters.
  2. The Keep and Remove operations directly test the contribution of the dominant correction direction to downstream performance.
  3. The theoretical analysis is closely connected to the experiments.
Critical weaknesses
  1. Section 3.3 compares Width1 trained from scratch with four learning rates against Width1 constructed from a trained Wide adapter. The latter benefits from information learned by the wider model. Additional experiments with compressed initialization followed by fine-tuning or staged training could help distinguish initialization effects from optimization difficulties.

  2. Section 4.4 measures accessible gradient energy using Frobenius projections. However, the actual training uses AdamW, whose updates depend on parameterization and adaptive learning rates.

  3. Section 5.1 reports only three runs with fixed hidden features and 500 updates. Wide lift achieves lower final NLL in all three runs. However, the paired intervals support an improvement over the shared starting point in only one run. Another run mainly shows deterioration in Direct. More random seeds and longer training would help determine whether the advantage consistently arises from the additional readout freedom.

Questions for authors

Please see Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer VLMa

Thank you for highlighting the distinction between hidden width and correction dimensionality and for your careful reading of the optimization experiments.

W1.

Section 3.3 compares Width1 trained from scratch with four learning rates against Width1 constructed from a trained Wide adapter. The latter benefits from information learned by the wider model. Additional experiments with compressed initialization followed by fine-tuning or staged training could help distinguish initialization effects from optimization difficulties.

Response

The construction uses Wide to supply a specific learned correction whose execution cost we can test. Even if all corrections lie on one axis, computing the coefficient for each visual token could still require all 64 hidden units. Our one-unit SiLU adapter computes the coefficient directly from the input token. Across three Qwen2.5-VL-7B runs with FastVID, it retains 89.7–92.5% of the development negative log-likelihood (NLL) gain of Wide over Base on videos held out from fitting. Fine-tuning or staged training would test whether the inherited weights also benefit subsequent optimization. The reported gain is present before any such fine-tuning.

Training from scratch also yields strong solutions. At the added peak learning rate of 0.003, task-trained Width1 achieves lower NLL than the construction in two of three runs. In seed 45 it reaches 1.17486, close to Wide at 1.17367. Section 3.3 and Table C.10 now report this comparison.

The learning-space result has an independent control. In Section 4.4, we take three functions learned by the one-axis readout (Direct) and reproduce each exactly with the unrestricted readout (Wide lift) using the same hidden features. Even at these Direct-trained solutions, the best single pattern of token coefficients misses 17.3–18.9% of the task-gradient energy accessible to Wide lift perpendicular to the current axis.

AuthorsOpenReview ↗

Response to Reviewer VLMa

W2.

Section 4.4 measures accessible gradient energy using Frobenius projections. However, the actual training uses AdamW, whose updates depend on parameterization and adaptive learning rates.

Response

Section 4.4 measures which changes to the correction each readout can make to first order. AdamW chooses a parameter step, and the readout Jacobian maps that step into a change in the correction. When Direct rotates its axis, tokens with equal current coefficients move together perpendicular to that axis. Wide lift can move them differently if their hidden features differ. Adaptive scaling changes the chosen step within the accessible space. The Frobenius projection measures how much of the task gradient lies in that space.

We tested the training consequence with AdamW from the same function and fixed hidden features. On seed 43, even the highest development NLL for Wide lift across the three tested rates (1.19308) was lower than the lowest for Direct (1.19617). We fixed the selected rates at 0.0003 for Wide lift and 0.001 for Direct before the separate holdout evaluation. This comparison includes the effect of readout parameterization on conditioning. Section 5.1 summarizes the result, and Table F.10 gives the full grid.

AuthorsOpenReview ↗

Response to Reviewer VLMa

W3.

Section 5.1 reports only three runs with fixed hidden features and 500 updates. Wide lift achieves lower final NLL in all three runs. However, the paired intervals support an improvement over the shared starting point in only one run. Another run mainly shows deterioration in Direct. More random seeds and longer training would help determine whether the advantage consistently arises from the additional readout freedom.

Response

At 500 updates the start-relative evidence is mixed, as you observed. We reanalyzed the existing 2,000-update runs for the same three seeds. Both readouts restarted from the original one-axis function for each seed with hidden features fixed. Under a common peak rate of 0.0003, Wide lift reduced development NLL from its own start by 0.0114, 0.0201, and 0.0077 for seeds 43, 45, and 46. All three paired 97.5% intervals exclude zero after adjustment for two contrasts within each seed. Wide lift itself improved in every run under the longer schedule.

After separate rate selection, we evaluated both readouts on 512 videos held out from rate and checkpoint selection. Final Direct minus Wide lift NLL was 0.0149, 0.0208, and 0.0139. Projecting those same Wide lifts onto their dominant directions cost 0.0155, 0.0278, and 0.0145 NLL, respectively. Both final contrasts have positive paired 95% intervals in each run. On this holdout the seed-46 interval for improvement over the start includes zero. The final readout advantage and projection cost are consistent across all three runs. Section 5.1 and Tables F.9–F.12 give the protocols and paired comparisons.

Thank you for the authors' detailed responses. I have reviewed their responses to both my concerns and those raised by the other reviewers. Most of my concerns have been addressed. Therefore, I will maintain my positive score. I hope the authors can also address the remaining concerns raised by the other reviewers.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised manuscript and new evidence on what the dominant correction direction preserves

We thank all three reviewers for reading our manuscript carefully and for their thoughtful feedback. Their comments helped us clarify how readout structure affects learning and how faithfully the dominant correction direction preserves individual predictions. We have marked the revisions in blue in the PDF. The main text remains nine pages.

Learning and preservation in Section 5.1 and Appendix F.8

We added an analysis of the existing 2,000-update runs. Both readouts start from the same one-axis function with the hidden features fixed. Under a common peak learning rate, the unrestricted readout (Wide lift) reduces development negative log-likelihood (NLL) relative to its own starting point in all three runs.

We also report results on a separate set of 512 videos held out from learning-rate and checkpoint selection. After selecting a learning rate for each readout, Wide lift achieves lower NLL than the one-axis readout (Direct) in all three runs. We then project each of those same fitted Wide lifts onto its dominant correction direction. In all three point estimates, the resulting increase in NLL exceeds the entire advantage that Wide lift gained over Direct. Section 5.1 summarizes this result. Appendix F.8 describes the protocol and reports the learning-rate comparisons and paired intervals in Tables F.9–F.12.

Other changes in the revised PDF

  • Abstract, Introduction, and Section 2.1. We revised the opening to explain why a correction concentrated along one direction raises separate questions about how to learn it, how to execute it with less computation, and how to preserve its predictions. The new Table 1 shows what changes in each adapter configuration and what each comparison tests.

  • Section 3.3. We clarified what the one-unit construction achieves at inference. A single SiLU unit computes the correction directly from each input token and retains about 90% of the development NLL improvement achieved by Wide over the unadapted model. The expanded comparison in Table C.10 also shows that training the one-unit adapter from scratch can yield a strong solution.

  • Section 5.2. We brought the direction controls from Appendix G into the main discussion. Replacing the learned axis while preserving the token coefficients and correction magnitude changes decoder attention to visual tokens and the margin between correct and incorrect answers. A complementary intervention keeps the axis and per-video RMS magnitude fixed while replacing the token-specific coefficients with a constant. At the decoder-analysis checkpoint, option accuracy stays near 52%, yet agreement with the choices made by Wide falls from 97.77% to 85.23%. Reproducing individual predictions therefore depends on how the correction is distributed across tokens, even when its direction and overall magnitude are preserved.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.