- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper investigates inverse scaling in vision-language models (VLMs), where larger models may underperform smaller counterparts on specific subtasks within the same model family. The authors evaluate 19 models from four model families—Qwen3-VL, InternVL3.5, Gemma 3, and Molmo 2—across eight existing multimodal benchmarks (nine evaluation splits), using per-question pairwise flip analysis to identify subtask-level performance regressions. The evaluation comprises 558,334 model–question evaluations and identifies 375 inverse-scaling cases meeting the predefined criteria, with 13 subtask–dataset pairs exhibiting inverse scaling across at least three model families. The authors further examine the robustness of these findings through dense-only comparisons, model-size restrictions, statistical significance tests, and chain-of-thought (CoT) prompting experiments. The paper concludes that although aggregate performance generally improves with model scale, certain multimodal subtasks exhibit non-monotonic scaling behavior, suggesting that increasing model size does not uniformly improve all capabilities.
Critical strengths
- The paper provides extensive empirical evaluations and analyses from multiple perspectives. The authors evaluate 19 models across four model families on eight representative multimodal benchmarks and employ per-question pairwise flip analysis to examine performance differences between models of different sizes. The paper also includes cross-family comparisons, dense-only analysis, model-size restrictions, multiple-testing correction, and CoT prompting experiments. These experiments provide extensive empirical evidence and demonstrate the authors' efforts to examine the observed phenomena and their robustness.
Critical weaknesses
-
The paper offers limited distinctive scientific contributions, and the additional empirical analyses do not provide sufficiently substantive new insights. The paper primarily extends existing inverse scaling analyses to multiple released VLM families and established multimodal benchmarks. Its central findings remain largely focused on the observations that increasing model size does not uniformly improve performance across all tasks and that some subtasks exhibit non-monotonic scaling behavior. These general phenomena have already been investigated in prior work, including McKenzie et al. (2023) and Wei et al. (2023). More directly related, UniBench (NeurIPS 2024) has examined how different fine-grained capabilities in vision-language models benefit from scaling, showing that certain visual relational understanding and reasoning capabilities gain limited improvements from increased model scale. However, this relevant work is not adequately discussed in the paper. Although UniBench differs from the present study in its model types and evaluation objectives, these studies provide important precedents for the paper's central observations. Furthermore, the Qwen3-VL technical report has already reported cases where smaller models outperform larger counterparts on specific visual tasks within the same model family. While the paper provides additional subtask-level empirical evidence through broader model evaluations and per-question pairwise analysis, it does not sufficiently demonstrate what distinctive insights into multimodal scaling behavior emerge from these additional experiments. In particular, although the authors identify 13 subtask–dataset pairs exhibiting cross-family inverse scaling, they provide limited explanation of why these patterns are concentrated in certain capability categories or what mechanisms might explain or predict such regressions. The error analysis primarily examines whether the observed performance differences reflect genuine model errors or annotation issues, rather than systematically distinguishing specific failure mechanisms related to visual perception, spatial reasoning, or knowledge biases. Consequently, the contribution appears to lie mainly in extending the empirical validation of previously observed phenomena, while the additional findings do not yet offer sufficiently distinctive or explanatory scientific insights. This limits the novelty and substantive contribution of the paper.
-
Confounding factors in the model comparisons limit the interpretability of the findings. The paper investigates inverse scaling by comparing models of different sizes within the same model family. However, belonging to the same family does not necessarily mean that model size is the only factor that differs. Released VLM checkpoints of different sizes may also differ in their vision encoders, dense or mixture-of-experts (MoE) architectures, training data, alignment procedures, and visual input processing. For example, Appendix K explicitly shows that some Qwen3-VL and InternVL3.5 models use different vision encoders. Although the authors conduct additional analyses using dense-only comparisons, same-encoder-tier comparisons, and parameter-size restrictions to mitigate some of these confounding factors, these analyses are still insufficient to isolate the effect of model scale itself. Therefore, the current experiments primarily reveal observational performance differences among released model checkpoints, making it difficult to determine the independent relationship between model scale and the observed capability regressions. This also limits the extent to which the findings can explain more general scaling behavior in VLMs.
Questions for authors
-
Regarding the distinctive scientific contribution: Prior work has already investigated inverse scaling and non-monotonic scaling behavior in language models, while studies such as UniBench have examined how different fine-grained vision-language capabilities benefit from scaling. Existing VLM technical reports have also documented cases where smaller models outperform larger counterparts on specific visual tasks. Compared with these studies, what new and generalizable insights into multimodal scaling behavior does the proposed cross-family, subtask-level analysis provide? In particular, do the 13 subtask–dataset pairs exhibiting cross-family inverse scaling support substantive scientific conclusions beyond extending the empirical coverage of previously observed phenomena?
-
Regarding model comparability and confounding factors: Models of different sizes within the same family may differ in their vision encoders, architectures, training data, and alignment procedures. Although the paper includes dense-only, encoder-tier, and parameter-size-restricted analyses, do the main cross-family findings still hold when comparisons are further restricted to model pairs with the most comparable architectures, vision encoders, and training recipes? Would these more tightly controlled comparisons change the distribution of inverse-scaling cases, their statistical significance, or the conclusions regarding which capability categories are most susceptible to performance regressions?