- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors claim that a width sweep trained at a single shared learning rate is an inappropriate practice for studying and comparing vision-language adapters. To this end, inspired by previous hyperparameter transfer methods, they propose a width-aware rule for learning-rate selection in adapter comparison studies. Through extensive experiments, they show that using a fixed learning rate can lead to misleading conclusions about the performance of different adapter widths.
Critical strengths
- Identifying an overlooked issue in adapter width comparisons: The paper highlights an important limitation of comparing adapter widths under a single shared learning rate, as commonly done in prior VLM adapter studies. Through experiments across multiple LLM scales, the authors demonstrate that changing the shared learning rate can substantially reduce the apparent advantage of wider adapters and even change the ranking of different widths. This finding highlights the importance of considering learning-rate sensitivity when evaluating adapter widths, rather than attributing observed performance differences solely to adapter capacity.
Critical weaknesses
-
Poor organization and lack of a clear narrative: The Introduction is written as a single dense paragraph, making it difficult to follow the motivation, research gap, and key contributions. The experimental section also lacks a clear distinction between the main validation of the proposed scaling rule using Perceiver Resamplers and the supplementary experiments with Q-Former and MLP projectors, which do not directly validate the rule. Moreover, several critical results, including downstream evaluations and important control experiments, are relegated to the appendix, while less central analyses receive substantial attention in the main text. This organization obscures the scope of the paper's contributions and makes it difficult to assess which conclusions are actually supported by the experiments.
-
Insufficient justification for the width-aware learning-rate transfer rule: The proposed width-aware rule (Eq. 1) is motivated by the inverse-width scaling of µP, but it omits the initialization and output scaling of full µP. The paper does not explain why the same learning-rate scaling should remain valid without these components. Moreover, the experiments do not establish a unique scaling exponent, instead suggesting a range of approximately 0.5–1. Consequently, the theoretical basis for choosing this particular scaling rule over alternative relationships remains unclear.
-
Confounding between learning rate and weight decay: The proposed width-aware rule improves validation loss over the shared learning-rate baseline at larger adapter widths (Table 3). However, the effective weight-decay control experiments (Section 4.3 and Appendix C) show that the improvement at 14B XL decreases from approximately 0.16 to 0.02–0.05 nats when effective weight decay is held constant. This suggests that a substantial portion of the reported gain comes from changes in effective weight decay, making it difficult to determine the extent to which the gains can be attributed specifically to learning-rate scaling.
-
Confounding between adapter width and head configuration: The experiments change the head count and head dimension alongside adapter width (Section 3.1). Although the authors investigate the effect of head count in Section 5, the results vary across seeds and do not resolve this confounding factor. Consequently, the observed performance differences cannot be attributed solely to increased adapter capacity, making it difficult to isolate the capacity benefit of wider adapters.
-
Limited downstream evaluation: The primary evaluation focuses on pretraining validation loss, while task accuracy after instruction tuning is evaluated only on VQAv2 and TextVQA using the 3B model (Appendix T). Given recent advances in VLMs, these benchmarks provide a relatively limited assessment of visual understanding, without covering more challenging capabilities such as visual perception [1], hallucination robustness [2], and multimodal reasoning [3]. Moreover, the authors acknowledge that downstream results do not consistently preserve fine-grained width rankings based on validation loss (Section 5, Appendix S). Consequently, it remains unclear whether the proposed width-aware rule provides consistent benefits across different model scales and practical downstream tasks.
References
[1] Chen, Jierun, et al. "Revisiting Referring Expression Comprehension Evaluation in the Era of Large Multimodal Models." CVPR Workshops (CVPRW), 2025.
[2] Li, Yifan, et al. "Evaluating Object Hallucination in Large Vision-Language Models." EMNLP, 2023.
[3] Yue, Xiang, et al. "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI." CVPR, 2024.
-
Limited generalization across backbone families and adapter architectures: The experiments use a single vision encoder (SigLIP-SO400M) and one LLM family (Qwen2.5), although multiple LLM sizes are evaluated (Section 3.1). While the Q-Former experiments show similar learning-rate sensitivity (Table 4), the proposed width-aware rule is primarily validated on Perceiver Resamplers. In particular, the MLP projector experiment (Appendix F) uses a single fixed capacity and learning rate, without evaluating the proposed rule, despite MLP-based connectors being widely adopted in recent VLMs such as LLaVA-OneVision-1.5 [1], InternVL3 [2], and Qwen3-VL [3]. Consequently, it remains unclear whether the proposed rule generalizes to other backbone families and widely used adapter architectures.
References
[1] An, Xiang, et al. "LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training." arXiv preprint arXiv:2509.23661 (2025).
[2] Zhu, Jinguo, et al. "InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models." arXiv preprint arXiv:2504.10479 (2025).
[3] Bai, Shuai, et al. "Qwen3-VL Technical Report." arXiv preprint arXiv:2511.21631 (2025).
Questions for authors
Refer to weaknesses.
Thank you for the additional experiments and clarification. The matched repetitions alleviate my concern about training variability, and the empirical contribution is now clearer. I will maintain my original score of 3 (weak acceptance).