acceptodds
Under review as a conference paper at ICLR 2027

Wider by How Much? Learning Rates Shape Vision-Language Adapter Comparisons

Abstract

Adapter width in frozen-backbone vision-language models is often chosen from a width sweep trained at one shared learning rate. We show that the measured advantage of wider adapters depends strongly on that rate. Across five Perceiver-adapter widths and frozen language models from 1.5B to 14B parameters, the widest adapter beats the narrowest by 0.7–1.0 nats under a fixed rate at 1.5B–7B; selecting the rate at one width and sharing it removes 69–90% of that gap, and at 14B the narrowest cell varies by 0.57 nats across runs, so no ordering is determined there. Under the transferred rates the widest adapter is best or within 0.033 nats of best and 0.13–0.25 nats ahead of the narrowest. Changing the shared rate also changes Q-Former width comparisons, and the fixed-rate deficit extends to answer likelihood for the four narrower widths at 3B and all five at 14B, and to VQA accuracy after instruction tuning at 3B. A width-adjusted coordinate, the learning rate times the width, predicts held-out validation loss with 34–60% lower error than the rate alone below 14B. A one-anchor rule that rescales the reference rate by the square root of the adapter parameter-count ratio removes the per-width search, and at 14B its rates coincide with the sampled bracket minima at the two widest adapters. Matching the anchor's effective weight decay removes much of the rule's advantage at wide adapters, so the rule transfers the learning rate together with its decay. At 14B a single coordinate leaves width-dependent residuals; clipping settings barely change the narrowest adapter's loss at its rule rate, whereas halving the head count of the widest adapter at fixed parameter count changes its loss by to across two seeds. Connector comparisons should report learning-rate sensitivity or transfer the rate across widths.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, multimodal adapters, Perceiver Resampler, scaling laws, learning rate, hyperparameter transfer, width scaling, frozen LLM, adapter capacity

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_zZh6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors claim that a width sweep trained at a single shared learning rate is an inappropriate practice for studying and comparing vision-language adapters. To this end, inspired by previous hyperparameter transfer methods, they propose a width-aware rule for learning-rate selection in adapter comparison studies. Through extensive experiments, they show that using a fixed learning rate can lead to misleading conclusions about the performance of different adapter widths.

Critical strengths
  • Identifying an overlooked issue in adapter width comparisons: The paper highlights an important limitation of comparing adapter widths under a single shared learning rate, as commonly done in prior VLM adapter studies. Through experiments across multiple LLM scales, the authors demonstrate that changing the shared learning rate can substantially reduce the apparent advantage of wider adapters and even change the ranking of different widths. This finding highlights the importance of considering learning-rate sensitivity when evaluating adapter widths, rather than attributing observed performance differences solely to adapter capacity.
Critical weaknesses
  • Poor organization and lack of a clear narrative: The Introduction is written as a single dense paragraph, making it difficult to follow the motivation, research gap, and key contributions. The experimental section also lacks a clear distinction between the main validation of the proposed scaling rule using Perceiver Resamplers and the supplementary experiments with Q-Former and MLP projectors, which do not directly validate the rule. Moreover, several critical results, including downstream evaluations and important control experiments, are relegated to the appendix, while less central analyses receive substantial attention in the main text. This organization obscures the scope of the paper's contributions and makes it difficult to assess which conclusions are actually supported by the experiments.

  • Insufficient justification for the width-aware learning-rate transfer rule: The proposed width-aware rule (Eq. 1) is motivated by the inverse-width scaling of µP, but it omits the initialization and output scaling of full µP. The paper does not explain why the same learning-rate scaling should remain valid without these components. Moreover, the experiments do not establish a unique scaling exponent, instead suggesting a range of approximately 0.5–1. Consequently, the theoretical basis for choosing this particular scaling rule over alternative relationships remains unclear.

  • Confounding between learning rate and weight decay: The proposed width-aware rule improves validation loss over the shared learning-rate baseline at larger adapter widths (Table 3). However, the effective weight-decay control experiments (Section 4.3 and Appendix C) show that the improvement at 14B XL decreases from approximately 0.16 to 0.02–0.05 nats when effective weight decay is held constant. This suggests that a substantial portion of the reported gain comes from changes in effective weight decay, making it difficult to determine the extent to which the gains can be attributed specifically to learning-rate scaling.

  • Confounding between adapter width and head configuration: The experiments change the head count and head dimension alongside adapter width (Section 3.1). Although the authors investigate the effect of head count in Section 5, the results vary across seeds and do not resolve this confounding factor. Consequently, the observed performance differences cannot be attributed solely to increased adapter capacity, making it difficult to isolate the capacity benefit of wider adapters.

  • Limited downstream evaluation: The primary evaluation focuses on pretraining validation loss, while task accuracy after instruction tuning is evaluated only on VQAv2 and TextVQA using the 3B model (Appendix T). Given recent advances in VLMs, these benchmarks provide a relatively limited assessment of visual understanding, without covering more challenging capabilities such as visual perception [1], hallucination robustness [2], and multimodal reasoning [3]. Moreover, the authors acknowledge that downstream results do not consistently preserve fine-grained width rankings based on validation loss (Section 5, Appendix S). Consequently, it remains unclear whether the proposed width-aware rule provides consistent benefits across different model scales and practical downstream tasks.

    References

    [1] Chen, Jierun, et al. "Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Models." CVPR Workshops (CVPRW), 2025.

    [2] Li, Yifan, et al. "Evaluating Object Hallucination in Large Vision-Language Models." EMNLP, 2023.

    [3] Yue, Xiang, et al. "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI." CVPR, 2024.

  • Limited generalization across backbone families and adapter architectures: The experiments use a single vision encoder (SigLIP-SO400M) and one LLM family (Qwen2.5), although multiple LLM sizes are evaluated (Section 3.1). While the Q-Former experiments show similar learning-rate sensitivity (Table 4), the proposed width-aware rule is primarily validated on Perceiver Resamplers. In particular, the MLP projector experiment (Appendix F) uses a single fixed capacity and learning rate, without evaluating the proposed rule, despite MLP-based connectors being widely adopted in recent VLMs such as LLaVA-OneVision-1.5 [1], InternVL3 [2], and Qwen3-VL [3]. Consequently, it remains unclear whether the proposed rule generalizes to other backbone families and widely used adapter architectures.

    References

    [1] An, Xiang, et al. "LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training." arXiv preprint arXiv:2509.23661 (2025).

    [2] Zhu, Jinguo, et al. "InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models." arXiv preprint arXiv:2504.10479 (2025).

    [3] Bai, Shuai, et al. "Qwen3-VL Technical Report." arXiv preprint arXiv:2511.21631 (2025).

Questions for authors

Refer to weaknesses.

Reviewer_uxu2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that learning-rate choices can inflate the apparent benefits of wider adapters in frozen-backbone vision-language models and evaluates a parameter-count-based rule for transferring learning rates from one reference width to other widths.

Critical strengths
  1. A useful empirical finding for adapter comparisons. The paper shows that learning-rate choices can substantially inflate the apparent benefits of wider adapters. This provides a practical reminder that improvements observed under a shared training recipe should not automatically be attributed to greater model capacity.

  2. Substantial empirical investigation. The study examines five adapter widths across four language-model scales, with additional Q-Former comparisons and downstream evaluation after instruction tuning at 3B. These experiments support the main observation beyond a single model configuration or pretraining metric.

  3. Informative controls and transparent reporting. The effective-weight-decay control clarifies that the transfer rule’s gains arise partly from changes in regularization. The authors also report run-to-run variability and unresolved behavior at 14B, rather than presenting the rule as universally reliable.

Critical weaknesses
  1. The incremental contribution needs stronger support. Quantifying learning-rate sensitivity in frozen vision-language connectors is useful, but the proposed rule builds on existing inverse-width scaling. A direct comparison with an established width-aware transfer method would help clarify whether this simplified rule offers competitive performance with less tuning or implementation effort. The current comparisons mainly establish sensitivity to the training recipe, rather than the rule’s distinct practical advantage.

  2. Smaller improvements remain difficult to distinguish from training variability. The authors transparently report substantial variation and avoid firm rankings when differences are small. Nevertheless, this variation limits confidence in some of the smaller gains between protocols, even though it does not undermine the large fixed-rate deficit. More balanced repeated comparisons between the selected shared rate and the width-aware rule, using matched hardware and micro-batch settings, would strengthen the evidence for these gains.

Questions for authors

Please address the questions and concerns raised in the Weaknesses section.

1 comment

Thank you for the additional experiments and clarification. The matched repetitions alleviate my concern about training variability, and the empirical contribution is now clearer. I will maintain my original score of 3 (weak acceptance).

Reviewer_unHkOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how learning-rate assignment affects comparisons of vision-language adapter width between frozen SigLIP and Qwen2.5 backbones. It compares a fixed shared rate, a rate selected at one reference width and then shared, and a one-anchor transfer rule proportional to the inverse square root of adapter parameter count. Selecting a better shared rate removes 69–90% of the XS–XL validation-loss gap at 1.5B–7B; Q-Former sweeps and a 3B instruction-tuning experiment provide supporting evidence. A width-adjusted coordinate improves held-out loss prediction. The paper also shows that effective weight decay explains much of the transfer rule's wide-adapter benefit, while substantial variability and residual width dependence remain at 14B.

Critical strengths
  1. The central empirical effect is large and practically relevant. In Table 3, changing the shared learning-rate setting reduces the XS–XL gap from 0.969 to 0.301 nats at 1.5B, from 0.847 to 0.081 at 3B, and from 0.723 to 0.170 at 7B. The Q-Former comparison in Table 4 supports the same qualitative lesson. These results provide useful evidence that a single-recipe width sweep should not be interpreted directly as an optimized capacity comparison.

  2. The paper investigates important qualifications rather than reporting only favorable runs. It includes effective-decay controls, seed and same-seed repeats, decomposition-matched reruns, failure-inclusive prediction analysis, and downstream evaluation. Table 25 reports that the fixed-rate XS–XL VQAv2 gap shrinks from 11.0 to about 0.2 percentage points after changing the pretraining protocol and applying the same instruction-tuning recipe. The explicit discussion of instability and limits strengthens the credibility of the main, coarse protocol effect.

Critical weaknesses
  1. The additional methodological contribution over established learning-rate transfer is limited, and the benefit of the proposed rule is not sufficiently separated from simpler alternatives. Width-dependent learning rates and one-anchor transfer already appear in Tensor Programs V / muTransfer and Everett et al. (2024). The paper acknowledges this, so its contribution is primarily the connector-specific empirical result. However, Table 8 shows that matching effective decay reduces the 14B XL gain over the selected shared rate from about 0.16 nats to only 0.02–0.05. The missing complementary control is to retain the shared learning rate and change only decay to match the rule's effective decay. Without this or a relevant established per-layer transfer baseline, it is unclear what practical advantage specifically requires the proposed learning-rate rule. This limits the strength of the proposed remedy, although it does not invalidate the central observation.

  2. The remaining evidence does not yet establish a reliably general procedure for selecting connector width. Main cells have uneven replication, and same-configuration differences can reach 0.17–0.25 nats (Appendix B, Table 7); 14B XS has a 0.57-nat spread across runs. The decomposition-matched reruns are useful, but the main comparisons still combine heterogeneous runs, and the small rule gains and fine width rankings are less secure than the large fixed-rate deficits. The setting is also concentrated on one backbone family and pretraining dataset; the Q-Former results test sensitivity rather than the transfer rule itself. Actual post-instruction-tuning accuracy is evaluated only at 3B with one recipe and seed and 1,000 questions per task. Together with the mixed downstream NLL results at 7B, this leaves the broader decision value of the rule uncertain. These concerns, combined with the limited methodological novelty, lead me to weak rejection.

Questions for authors
  1. Decay attribution: Can the authors compare the current rule with a control that keeps the selected shared learning rate but sets each width's decay coefficient so that its effective decay matches the rule? This is a request for discriminating evidence: a comparable result from decay alone would materially change the interpretation of the proposed remedy.

  2. Robustness of the recommendation: Using a consistently matched subset of runs, what are the paired protocol differences and their uncertainty at the key widths? Please distinguish effects that remain supported from width rankings that cannot be resolved. Evidence that the rule improves on an established per-layer transfer prescription at comparable tuning cost would also strengthen the contribution; this is an additional experiment suggestion, not a claim that such a comparison is already present.

  3. Downstream evaluation clarification: Page 25 defines VQA accuracy as min(1, m/3) over ten reference answers, whereas the official VQA evaluator averages over ten leave-one-annotator-out subsets (e.g., three total matches score 0.9). Is the manuscript description abbreviated, or does the implementation use the stated formula? Please report official-scorer results from the existing generations if needed, and clarify whether the questions used to choose the SFT recipe were separate from the final evaluation questions. These answers would affect confidence in Table 25, without by themselves overturning the pretraining results.

1 comment

The decay-only control and the matched XL runs address my main concern about what drives the gain. The conclusion also holds with the official VQA scorer. I support raising my rating to 4.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.