acceptodds
Under review as a conference paper at ICLR 2027

On the Limits of Scale: Inverse Scaling in Vision-Language Models

Abstract

Scaling laws suggest that increasing model size, training data, and compute generally yields monotonic performance improvements. While this often holds at the aggregate level, we present systematic evidence that vision-language models (VLMs) exhibit inverse scaling at the subtask level: specific question categories on which larger models perform worse than smaller counterparts within the same model family. We evaluate 19 models from four distinct model families across eight multimodal benchmarks (nine evaluation splits), answering 29,386 questions per model, 558,334 model–question evaluations in total. Using per-question pairwise analysis across all model pairs in each family, we identify 375 subtask-model-pair combinations exhibiting inverse scaling, spanning 4,342 unique queries where a smaller model answers correctly but a larger model fails, and we find that these patterns replicate across independently developed architectures. We identify 13 subtask–dataset pairs that show inverse scaling across multiple families, consistent with the phenomenon being task-driven rather than model-specific. The most affected domains in our study include multi-image spatial reasoning, visual perception, and university-level STEM knowledge, with accuracy gaps exceeding 20 percentage points in several cases, while diagram understanding appears comparatively resistant. These findings challenge the assumption that increasing model size uniformly improves multimodal capabilities and suggest that model selection should be task-aware.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, inverse scaling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bda6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates inverse scaling in vision-language models (VLMs), where larger models may underperform smaller counterparts on specific subtasks within the same model family. The authors evaluate 19 models from four model families—Qwen3-VL, InternVL3.5, Gemma 3, and Molmo 2—across eight existing multimodal benchmarks (nine evaluation splits), using per-question pairwise flip analysis to identify subtask-level performance regressions. The evaluation comprises 558,334 model–question evaluations and identifies 375 inverse-scaling cases meeting the predefined criteria, with 13 subtask–dataset pairs exhibiting inverse scaling across at least three model families. The authors further examine the robustness of these findings through dense-only comparisons, model-size restrictions, statistical significance tests, and chain-of-thought (CoT) prompting experiments. The paper concludes that although aggregate performance generally improves with model scale, certain multimodal subtasks exhibit non-monotonic scaling behavior, suggesting that increasing model size does not uniformly improve all capabilities.

Critical strengths
  1. The paper provides extensive empirical evaluations and analyses from multiple perspectives. The authors evaluate 19 models across four model families on eight representative multimodal benchmarks and employ per-question pairwise flip analysis to examine performance differences between models of different sizes. The paper also includes cross-family comparisons, dense-only analysis, model-size restrictions, multiple-testing correction, and CoT prompting experiments. These experiments provide extensive empirical evidence and demonstrate the authors' efforts to examine the observed phenomena and their robustness.
Critical weaknesses
  1. The paper offers limited distinctive scientific contributions, and the additional empirical analyses do not provide sufficiently substantive new insights. The paper primarily extends existing inverse scaling analyses to multiple released VLM families and established multimodal benchmarks. Its central findings remain largely focused on the observations that increasing model size does not uniformly improve performance across all tasks and that some subtasks exhibit non-monotonic scaling behavior. These general phenomena have already been investigated in prior work, including McKenzie et al. (2023) and Wei et al. (2023). More directly related, UniBench (NeurIPS 2024) has examined how different fine-grained capabilities in vision-language models benefit from scaling, showing that certain visual relational understanding and reasoning capabilities gain limited improvements from increased model scale. However, this relevant work is not adequately discussed in the paper. Although UniBench differs from the present study in its model types and evaluation objectives, these studies provide important precedents for the paper's central observations. Furthermore, the Qwen3-VL technical report has already reported cases where smaller models outperform larger counterparts on specific visual tasks within the same model family. While the paper provides additional subtask-level empirical evidence through broader model evaluations and per-question pairwise analysis, it does not sufficiently demonstrate what distinctive insights into multimodal scaling behavior emerge from these additional experiments. In particular, although the authors identify 13 subtask–dataset pairs exhibiting cross-family inverse scaling, they provide limited explanation of why these patterns are concentrated in certain capability categories or what mechanisms might explain or predict such regressions. The error analysis primarily examines whether the observed performance differences reflect genuine model errors or annotation issues, rather than systematically distinguishing specific failure mechanisms related to visual perception, spatial reasoning, or knowledge biases. Consequently, the contribution appears to lie mainly in extending the empirical validation of previously observed phenomena, while the additional findings do not yet offer sufficiently distinctive or explanatory scientific insights. This limits the novelty and substantive contribution of the paper.

  2. Confounding factors in the model comparisons limit the interpretability of the findings. The paper investigates inverse scaling by comparing models of different sizes within the same model family. However, belonging to the same family does not necessarily mean that model size is the only factor that differs. Released VLM checkpoints of different sizes may also differ in their vision encoders, dense or mixture-of-experts (MoE) architectures, training data, alignment procedures, and visual input processing. For example, Appendix K explicitly shows that some Qwen3-VL and InternVL3.5 models use different vision encoders. Although the authors conduct additional analyses using dense-only comparisons, same-encoder-tier comparisons, and parameter-size restrictions to mitigate some of these confounding factors, these analyses are still insufficient to isolate the effect of model scale itself. Therefore, the current experiments primarily reveal observational performance differences among released model checkpoints, making it difficult to determine the independent relationship between model scale and the observed capability regressions. This also limits the extent to which the findings can explain more general scaling behavior in VLMs.

Questions for authors
  1. Regarding the distinctive scientific contribution: Prior work has already investigated inverse scaling and non-monotonic scaling behavior in language models, while studies such as UniBench have examined how different fine-grained vision-language capabilities benefit from scaling. Existing VLM technical reports have also documented cases where smaller models outperform larger counterparts on specific visual tasks. Compared with these studies, what new and generalizable insights into multimodal scaling behavior does the proposed cross-family, subtask-level analysis provide? In particular, do the 13 subtask–dataset pairs exhibiting cross-family inverse scaling support substantive scientific conclusions beyond extending the empirical coverage of previously observed phenomena?

  2. Regarding model comparability and confounding factors: Models of different sizes within the same family may differ in their vision encoders, architectures, training data, and alignment procedures. Although the paper includes dense-only, encoder-tier, and parameter-size-restricted analyses, do the main cross-family findings still hold when comparisons are further restricted to model pairs with the most comparable architectures, vision encoders, and training recipes? Would these more tightly controlled comparisons change the distribution of inverse-scaling cases, their statistical significance, or the conclusions regarding which capability categories are most susceptible to performance regressions?

Reviewer_GiytOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether publicly released vision-language models exhibit subtask-level performance regressions: cases where a larger model performs worse than a smaller model within the same family. The authors evaluate 19 models from four families across eight benchmarks and nine evaluation splits, totaling 558,334 model–question evaluations. The analysis compares the correctness of two models on the same questions, counting disagreements that favor the smaller model and those that favor the larger model. Comparisons satisfying n ≥ 20 and Net ≥ 5 are identified as inverse-scaling cases. The authors report 375 candidate cases and 13 subtask patterns appearing in at least three model families. Validation includes dense-only comparisons, parameter-ratio restrictions, vision-encoder grouping, option reordering, chain-of-thought controls, and automated and human error annotation.

Critical strengths
  1. The paper identifies practically meaningful localized regressions, including dense-to-dense comparisons. For example, on BLINK Jigsaw, Qwen 4B outperforms 32B by approximately 22 percentage points, and Molmo 4B outperforms 8B by approximately 20.7 percentage points. These cases survive BH correction over the full comparison space. They establish that the core phenomenon exists in at least some settings and deserves further investigation, beyond small fluctuations near the selection threshold.

  2. The authors examine several alternative explanations and disclose results that qualify the main narrative. The dense-only, active-parameter, vision-encoder, option-reordering, and both-model CoT analyses improve the interpretability of the study. In particular, the authors report that most original cases disappear or reverse under CoT and acknowledge that comparisons among released checkpoints cannot identify the causal effect of parameter count. This transparency is an important strength.

Critical weaknesses
  1. The central claims of widespread, cross-family, task-driven inverse scaling are stronger than the statistical evidence supports.
  • These 16 cases cover eight distinct subtasks. Only Jigsaw appears in two families; no subtask has a regression comparison surviving this correction in at least three families. No MMMU case remains in this corrected set. Thus, the existence of several strong regressions is supported, but the claim of robust cross-family regression across 13 task categories does not have comparable support. Non-significant results should not be interpreted as evidence of no effect, but neither should they carry the same evidential weight as statistically supported findings.
  • The authors appropriately describe the analysis as exploratory. Nevertheless, the abstract and principal conclusions remain centered on candidate-case counts. At a minimum, the paper should clearly distinguish candidate discoveries from statistically supported findings and provide an analysis or independent confirmation aligned with its cross-family replication claim.

The study does not sufficiently establish that the selected patterns reflect stable capability regressions, and its quantification of genuine errors is biased.

  • In Appendix J, identity reruns that preserve the original option order change Net by as much as 11 relative to the main evaluation—more than twice the selection threshold of 5. These reruns cover only a few strong cases, leaving the stability of near-threshold cases and the 13 cross-family patterns unresolved. The CoT experiment changes both the prompt and output budget, so it cannot substitute for direct-answer reruns under identical settings. Its interpretation should also account for possible regression to the mean after selecting cases with positive observed effects.
  • These issues affect the reliable scope of the main conclusions. The key cross-family subtasks should receive priority for identical-protocol reruns and label auditing, followed by updated effect estimates and uncertainty intervals.
Questions for authors
  1. Cross-family evidence: After controlling multiple testing over the full comparison space, which subtasks retain cross-family support? Can you provide a direct statistical analysis of the claim that regressions occur in at least three families, beyond separately satisfying Net ≥ 5?

  2. Human validation: Please provide the stratum sizes, sampling probabilities, and weighted estimates for the 127 annotated questions. Did annotators independently reassess both models’ answers? After removing or correcting disputed questions, how many of the 375 cases and 13 cross-family patterns remain?

Reviewer_Q447OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates 19 open-weight VLMs from four model families across eight benchmarks. By comparing the correctness of each model on every question, it identifies cases of inverse scaling, where smaller models outperform larger ones.

The paper provides insights that are useful for model selection, based on a large-scale and careful evaluation. However, the metric used is essentially equivalent to the accuracy difference, and inverse scaling is a known problem in deep learning fields.

Critical strengths

This paper presents the results of approximately 560,000 evaluations, covering 19 models and 29,386 questions, and makes all 375 cases available. The results are directly applicable to practical model selection.

The authors conduct a wide range of control experiments and human validation of the error taxonomy.

Critical weaknesses

This paper positions its findings as counterexamples to scaling laws, but scaling laws are empirical regularities about pretraining loss under controlled training. This paper also does not sufficiently discuss related work, e.g., studies that analyze scaling from publicly released models (Ruan et al., 2024, “Observational Scaling Laws and the Predictability of Language Model Performance”).

FlipInv is identical to the known negative flip (Yan et al., 2021, “Positive-Congruent Training: Towards Regression-Free Model Updates”), which this paper does not cite.

Questions for authors

Could the authors respond to the points raised in Weaknesses 1 and 2?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.