Stable Curves, Unstable Items: Item-Level Scaling Heterogeneity in Video LLMs

More visual input often improves average accuracy in video language models. However, aggregate gains do not reveal whether models preserve answers they already get right. We present a paired evaluation protocol that tracks the same questions across visual budgets within frozen models. Our study covers four models and four multiple-choice benchmark splits on a common budget grid, including text-only input. On these grids, $12.5$–$25.5\%$ of questions exhibit at least one correct-to-incorrect transition as the budget increases. Choosing the best budget for each question in hindsight yields $8.8$–$18.9$ percentage points higher accuracy than the best fixed budget. This gap reveals complementary successes that no single budget captures. We release over 120,000 per-item evaluation records and analysis code to support reproducible evaluation of visual scaling. Our findings show that larger visual budgets change which questions are answered correctly, not simply how many.