- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues about two roots problems of counting in the LVLMs: 1. Individuation bottleneck stemming from image patchification and 2. aggregation bottleneck caused by softmax-normalized attention. And supports these two claims with empirical and theoretical evidence. Consequently, to address these limitations in LVLMs, introduce ConvStack inspired by the spatial inductive biases of convolutional networks. As a result, by incorporating ConvStack, the model is more capable of not only counting more precisely but also improving performance on other visual tasks.
Critical strengths
Strenghts:
- Easy to follow and and well written
- Besides addressing counting, it improves other visual reasoning tasks, particularly spatial reasoning.
- By incorporating ConvStack the paper introduces a simple solution to solve the counting task.
- Controlled experiments to study the problems
- theory with empirical observations to support their claims
Critical weaknesses
-
There is no section in the paper that shows that by applying such a ConvStack method, it addresses two bottlenecks and leads to solving the counting task. In fact, I feel most of the improvement can stem from the training (incremental improvement compared to the training based methods and significant improvements compared with training-free methods on the Table 2). As a result, I struggle with the efficiency of the methods to solve these two bottlenecks.
-
The experiments are just conducted on one LVLM family (Qwen3) that is so limited and narrow. As a result, I'm curious whether the method can be applied effectively to other LVLM architectures (like Gemma3/4, internvl, or llava) and achieve such good results or not. Moreover, does these two roots of counting issue can be solved on other architectures or not.
-
most of the evidences comes from a simplified four-layer ViT. In fact, the paper doesn't provide any layer-wise analyses for Qwen3-VL’s pretrained vision encoder, nor does it demonstrate that Qwen3-VL actually fails to individuate objects or exhibits the representation and attention behaviors assumed by the theory.
-
The second bottleneck (aggregation bottleneck) just gets provided with theorical evidences and has no empirical experiments. Specifically, the paper reports no attention mass on object tokens and no values of r or D_n for Qwen3-VL. The only evidence is a correlation between silhouette and accuracy across 10 count classes
Minor issues:
- better to support the confidence intervals for the numerical results to make sure that the method don't make the model inconsistent.
Questions for authors
Same as weaknesses
3 comments
Weakness4 Empirical analysis of MLLMs for the aggregation bottleneck
We thank the reviewer for raising this important point. We would like to clarify that our aggregation analysis is intended to isolate a structural limitation of softmax-normalized attention under explicitly stated assumptions (under perfect individualization), rather than provide a complete mechanistic characterization of counting circuits in pretrained MLLMs.
In fact, a complete mechanistic interpretation on how specific attention heads shape the hidden states geometry of pretrained MLLMs is still an open problem. For instance, the attention ratio between object tokens and background tokens, (), is not a global parameter that can be meaningfully averaged across arbitrary attention heads in Qwen3-VL. The reason is that most of the attention heads in MLLMs contribute little to the counting tasks. And even if they are important to counting tasks, different heads may serve different functions. For instance, some heads could be responsible for simple information routing. These function heads need not directly attend to object tokens or satisfy (r>1). Furthermore, attention patterns and token representations vary across layers, queries, and inputs. However, the assumption of () is not an unverified guessing. Existing works [1,2,3] have proved that the object localization and visual grounding capabilities (which are critical for counting) of MLLMs depends on several specialized attention heads. These heads usually pay more attention to the object regions than backgrounds, i.e. .
Our theoretical contribution is therefore more specific: we establish that softmax normalization can compress incremental counting evidence even when object individuation is assumed to be perfect. This identifies a potential limitation of normalized aggregation, rather than claiming that all counting errors in pretrained MLLMs arise from this mechanism.
Importantly, our empirical analysis on the actual Qwen3-VL-8B model is not limited to the correlation in Figure 5. In Appendix D, we directly evaluate count-class representations before and after ConvStack, observing improvements in adjacent-count separation across count pairs (mean separation: 0.996 → 1.449), particularly for larger counts (counts 9–10: 0.558 → 1.058). Here adjacent-count separation can be seen as an approximated estimation of . These observations support the effectiveness of ConvStack in preserving count-discriminative representations. We agree that identifying the complete causal counting circuits of large MLLMs is an important open research direction. We will revise the manuscript to more clearly distinguish the conditional theoretical result, the empirical representation analysis, and the broader mechanistic findings.
[1] Mechanisms of Object Localization in Vision-Language Models Timothy Schaumlöffel, Martina G. Vilas, Gemma Roig; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 31356-31365
[2] Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding Seil Kang, Jinyeong Kim, Junhyeok Kim, Seong Jae Hwang; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 9339-9350
[3] Wang, Yuan, Wei Wu, and Hongning Wang. "Look at Where You Think: Localization-Head Alignment for Fine-Grained Perception in Thinking VLMs." arXiv preprint arXiv:2610.39649 (2026).
minor question
We provide experiment result for model training with 3 different seeds in below:
| Model | SAT-real | CountBenchQA | SAT-Spatial |
|---|---|---|---|
| Qwen3-VL-8B | 59.33 | 89.82 | 76.79 |
| ConvStack | 64.00 ± 1.15 | 90.22 ± 0.31 | 76.82 ± 0.08 |
The ConvStack obtained meaningful gain on SAT-real and CountBenchQA, with broadly comparable performance on SAT-Spatial.
Thank you for your responses. After carefully considering your rebuttal as well as the comments from the other reviewers, I have decided to keep my score.
Thank you for your structured comments! Here is our responses to your concerns:
Weakness1Concerns on improvement brought by training and method efficiencyWe agree with you that ConvStack did not solve the counting task completely which is also not our claim. ConvStack is an architectural intervention motivated by our findings that impedes MLLMs achieve effective counting. Both ViT encoder and LLM backbone are frozen during the training in our method.
We provided various ablation studies (including LoRA over the entire model and an additional merger only ablation) to demonstrate our method design contributed substantial performance improvement. The training-based baselines (Crop+DFI and Pool SFE) in Table 2 used 558K alignment data and 665K instruction tuning data, whose data scale is orders of magnitude larger than ours (~40K). Yet their performance is lower. For training-free baseline, we used the same frozen ViT and LLM checkpoint. It is worth to note that Pixmo-Count is already part of the training data of Qwen3-VL models. We only introduce 2.7M additional parameters. Therefore, we politely argue that our method design is meaningful and helpful.
Weakness2Apply to other model architectureWe provided additional results on LLaVA-1.5-7B and InternVL-3.5-8B with the same intervention and configuration. It can be seen our method can apply to other models as well. We agree that exploring the root cause of diverse architectures is an important topic and would like to dive into these topics in the future work. Thank you for the suggestion!
Weakness3Analysis pretrained vision encoderWe thank the reviewer for raising the question of whether the controlled ViT experiments establish an individuation bottleneck in pretrained MLLMs.
We would like to clarify that our patchification analysis does not claim an inherent inability of ViTs to individuate objects. In fact, all three placement conditions eventually achieve perfect counting accuracy, demonstrating that the architecture can learn to integrate fragmented object evidence. Rather, the key observation concerns learning difficulty and inductive bias. When objects are centered within distinct patches, counting can be reduced to identifying occupied patches. However, when a single object spans multiple patches, this direct correspondence breaks down, requiring the model to integrate fragmented evidence across tokens.This observation motivates ConvStack's explicit local spatial mixing, which provides a useful inductive bias for integrating neighboring visual features. We conducted additional object individualization analysis on the pretrained ViT of Qwen3-VL in Appendix C.4. It can be seen:
We will clarify this interpretation in the revised manuscript and avoid suggesting that patchification makes correct object individuation fundamentally impossible.