acceptodds
Under review as a conference paper at ICLR 2027

Why MLLMs Struggle to Count: Overcoming Individuation and Aggregation Bottlenecks with ConvStack

Abstract

Multimodal Large Language Models (MLLMs) consistently struggle with fine-grained visual counting, yet the underlying causes remain poorly understood. In this work, we present a mechanistic analysis of this failure mode, identifying two critical bottlenecks inherent to the global attention pipeline of MLLMs. First, we reveal an individuation bottleneck stemming from image patchification: because Vision Transformers process patches independently, they struggle to group fragmented geometric features across boundaries into distinct object representations. Second, we identify a collapse in the subsequent counting aggregation process, where representation separation rapidly diminishes as numerosity increases due to attention compression. Identifying and formalizing these twin bottlenecks constitutes our first major contribution. To overcome them, we propose ConvStack, a lightweight architecture that operates directly in the visual token space to explicitly aggregate and inject local spatial structures via zero-initialized residual connections. By explicitly addressing the individuation bottleneck, ConvStack provides unambiguous geometric evidence for downstream aggregation. Remarkably, by fine-tuning exclusively on counting tasks, the model achieves substantial improvements in dense object counting and broader spatial understanding benchmarks, without compromising on general visual capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision Language Model, Visual Understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1GoPOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues about two roots problems of counting in the LVLMs: 1. Individuation bottleneck stemming from image patchification and 2. aggregation bottleneck caused by softmax-normalized attention. And supports these two claims with empirical and theoretical evidence. Consequently, to address these limitations in LVLMs, introduce ConvStack inspired by the spatial inductive biases of convolutional networks. As a result, by incorporating ConvStack, the model is more capable of not only counting more precisely but also improving performance on other visual tasks.

Critical strengths

Strenghts:

  1. Easy to follow and and well written
  2. Besides addressing counting, it improves other visual reasoning tasks, particularly spatial reasoning.
  3. By incorporating ConvStack the paper introduces a simple solution to solve the counting task.
  4. Controlled experiments to study the problems
  5. theory with empirical observations to support their claims
Critical weaknesses
  1. There is no section in the paper that shows that by applying such a ConvStack method, it addresses two bottlenecks and leads to solving the counting task. In fact, I feel most of the improvement can stem from the training (incremental improvement compared to the training based methods and significant improvements compared with training-free methods on the Table 2). As a result, I struggle with the efficiency of the methods to solve these two bottlenecks.

  2. The experiments are just conducted on one LVLM family (Qwen3) that is so limited and narrow. As a result, I'm curious whether the method can be applied effectively to other LVLM architectures (like Gemma3/4, internvl, or llava) and achieve such good results or not. Moreover, does these two roots of counting issue can be solved on other architectures or not.

  3. most of the evidences comes from a simplified four-layer ViT. In fact, the paper doesn't provide any layer-wise analyses for Qwen3-VL’s pretrained vision encoder, nor does it demonstrate that Qwen3-VL actually fails to individuate objects or exhibits the representation and attention behaviors assumed by the theory.

  4. The second bottleneck (aggregation bottleneck) just gets provided with theorical evidences and has no empirical experiments. Specifically, the paper reports no attention mass on object tokens and no values of r or D_n for Qwen3-VL. The only evidence is a correlation between silhouette and accuracy across 10 count classes

Minor issues:

  1. better to support the confidence intervals for the numerical results to make sure that the method don't make the model inconsistent.
Questions for authors

Same as weaknesses

3 comments

Thank you for your structured comments! Here is our responses to your concerns:

Weakness1 Concerns on improvement brought by training and method efficiency

We agree with you that ConvStack did not solve the counting task completely which is also not our claim. ConvStack is an architectural intervention motivated by our findings that impedes MLLMs achieve effective counting. Both ViT encoder and LLM backbone are frozen during the training in our method.

We provided various ablation studies (including LoRA over the entire model and an additional merger only ablation) to demonstrate our method design contributed substantial performance improvement. The training-based baselines (Crop+DFI and Pool SFE) in Table 2 used 558K alignment data and 665K instruction tuning data, whose data scale is orders of magnitude larger than ours (~40K). Yet their performance is lower. For training-free baseline, we used the same frozen ViT and LLM checkpoint. It is worth to note that Pixmo-Count is already part of the training data of Qwen3-VL models. We only introduce 2.7M additional parameters. Therefore, we politely argue that our method design is meaningful and helpful.

Weakness2 Apply to other model architecture

We provided additional results on LLaVA-1.5-7B and InternVL-3.5-8B with the same intervention and configuration. It can be seen our method can apply to other models as well. We agree that exploring the root cause of diverse architectures is an important topic and would like to dive into these topics in the future work. Thank you for the suggestion!

ModelConfigurationPixMo-Count (%)CountBenchQA (%)SAT-real (%)
LLaVA-1.5-7BBaseline31.8744.6052.33
LLaVA-1.5-7BConvStack41.0351.5352.33
InternVL-3.5-8BBaseline62.9880.6557.00
InternVL-3.5-8BConvStack73.8584.7358.00

Weakness3 Analysis pretrained vision encoder

We thank the reviewer for raising the question of whether the controlled ViT experiments establish an individuation bottleneck in pretrained MLLMs.

We would like to clarify that our patchification analysis does not claim an inherent inability of ViTs to individuate objects. In fact, all three placement conditions eventually achieve perfect counting accuracy, demonstrating that the architecture can learn to integrate fragmented object evidence. Rather, the key observation concerns learning difficulty and inductive bias. When objects are centered within distinct patches, counting can be reduced to identifying occupied patches. However, when a single object spans multiple patches, this direct correspondence breaks down, requiring the model to integrate fragmented evidence across tokens.This observation motivates ConvStack's explicit local spatial mixing, which provides a useful inductive bias for integrating neighboring visual features. We conducted additional object individualization analysis on the pretrained ViT of Qwen3-VL in Appendix C.4. It can be seen:

  1. Patchification does influence its capability on shape recognition.
  2. ConvStack helps with object binding.

We will clarify this interpretation in the revised manuscript and avoid suggesting that patchification makes correct object individuation fundamentally impossible.

Weakness4 Empirical analysis of MLLMs for the aggregation bottleneck

We thank the reviewer for raising this important point. We would like to clarify that our aggregation analysis is intended to isolate a structural limitation of softmax-normalized attention under explicitly stated assumptions (under perfect individualization), rather than provide a complete mechanistic characterization of counting circuits in pretrained MLLMs.

In fact, a complete mechanistic interpretation on how specific attention heads shape the hidden states geometry of pretrained MLLMs is still an open problem. For instance, the attention ratio between object tokens and background tokens, (r=exp⁡(so−sb)r=\exp(s_o-s_b)), is not a global parameter that can be meaningfully averaged across arbitrary attention heads in Qwen3-VL. The reason is that most of the attention heads in MLLMs contribute little to the counting tasks. And even if they are important to counting tasks, different heads may serve different functions. For instance, some heads could be responsible for simple information routing. These function heads need not directly attend to object tokens or satisfy (r>1). Furthermore, attention patterns and token representations vary across layers, queries, and inputs. However, the assumption of (r>1r>1) is not an unverified guessing. Existing works [1,2,3] have proved that the object localization and visual grounding capabilities (which are critical for counting) of MLLMs depends on several specialized attention heads. These heads usually pay more attention to the object regions than backgrounds, i.e. r>1r>1.

Our theoretical contribution is therefore more specific: we establish that softmax normalization can compress incremental counting evidence even when object individuation is assumed to be perfect. This identifies a potential limitation of normalized aggregation, rather than claiming that all counting errors in pretrained MLLMs arise from this mechanism.

Importantly, our empirical analysis on the actual Qwen3-VL-8B model is not limited to the correlation in Figure 5. In Appendix D, we directly evaluate count-class representations before and after ConvStack, observing improvements in adjacent-count separation across count pairs (mean separation: 0.996 → 1.449), particularly for larger counts (counts 9–10: 0.558 → 1.058). Here adjacent-count separation Γn\Gamma_n can be seen as an approximated estimation of DnD_n. These observations support the effectiveness of ConvStack in preserving count-discriminative representations. We agree that identifying the complete causal counting circuits of large MLLMs is an important open research direction. We will revise the manuscript to more clearly distinguish the conditional theoretical result, the empirical representation analysis, and the broader mechanistic findings.

[1] Mechanisms of Object Localization in Vision-Language Models Timothy Schaumlöffel, Martina G. Vilas, Gemma Roig; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026, pp. 31356-31365

[2] Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding Seil Kang, Jinyeong Kim, Junhyeok Kim, Seong Jae Hwang; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2025, pp. 9339-9350

[3] Wang, Yuan, Wei Wu, and Hongning Wang. "Look at Where You Think: Localization-Head Alignment for Fine-Grained Perception in Thinking VLMs." arXiv preprint arXiv:2610.39649 (2026).

minor question

We provide experiment result for model training with 3 different seeds in below:

ModelSAT-realCountBenchQASAT-Spatial
Qwen3-VL-8B59.3389.8276.79
ConvStack64.00 ± 1.1590.22 ± 0.3176.82 ± 0.08

The ConvStack obtained meaningful gain on SAT-real and CountBenchQA, with broadly comparable performance on SAT-Spatial.

Thank you for your responses. After carefully considering your rebuttal as well as the comments from the other reviewers, I have decided to keep my score.

Reviewer_ixB2OpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  • studies two counting failures in MLLMs: object fragmentation from ViT patchification and count compression from softmax attention
  • proposes ConvStack: local 3×3 convolutions with residual visual feature injection
  • improves PixMo-Count from 65.46% to 75.19% on Qwen3-VL-8B, training only 0.49% of parameters
Critical strengths
  • clean synthetic experiment isolate patch-boundary effects on counting
  • good ablations: 33 spatial mixing beats 11, LoRA, and single-route variants
  • strong parameter efficiency, with gains on counting and several spatial benchmarks
  • representation analysis goes beyond t-SNE.... measuring count-class separation directly
Critical weaknesses
  • the attention analysis assumes homogeneous tokens and a single head. Hard to generalize its conclusions to actual MLLMs
  • the residual theorem assumes approximately additive counting features; it does not establish that ConvStack learns them!!!!
  • convolutional adapters and residual injection are established ideas. Architectural novelty is limited
  • no clean comparison against parameter-matched convolutional adapters or merger-only fine-tuning
  • evaluated on one main MLLM backbone. Real-image counting gains are measured on only 524 PixMo-Count examples
Questions for authors

Very minor questions below, thank you:

  • can you show empirically that attention normalization causes count compression in the pretrained MLLM, rather than just in the simplified single-head model?
  • how does ConvStack compare against parameter-matched ConvPass or other convolutional adapters?
  • how much of the improvement comes from training the visual merger alone?
  • does ConvStack still help for counts beyond 10..... particularly in crowded or heavily occluded scenes?
  • do the gains transfer to other MLLM backbones, or are they specific to Qwen3-VL?
3 comments

Weakness 1 Assumptions of attention aggregation

The homogeneous-token setting for Eq. 2-4 in our paper is designed to isolate aggregation from individuation, instead of oversimplifying the problem. In other words, we want to show even if every object is perfectly individualized, it's not a guarantee to count correctly. This represents a favorable setting in which object fragmentation and count-dependent feature changes are eliminated as confounding factors. Nevertheless, even with perfect object individuation, softmax normalization produces diminishing adjacent-count separation. Our intention is therefore to demonstrate that successful individuation alone does not guarantee stable count aggregation.

We would like to point out we have provided empirical evidence to support our attention analysis. Our attention analysis indicates that as the number of objects nn increasing, the distance between the hidden states of nn and n−1n-1 will diminish. We have shown this in Figure 5 that the representations of larger numbers are closer to each other in Qwen3-VL-8B. Figure5 also demonstrates that count-class separability is strongly associated with counting accuracy in pretrained Qwen3-VL-8B. More importantly, Figure 7 and Appendix D directly quantify count representation separation in the original hidden-state space before and after ConvStack adaptation in the final layer. These results demonstrate that ConvStack substantially improves count discriminability in the actual MLLM, consistent with our theoretical motivation.

We thank you for raising this point. And we have updated our manuscript to reduce the ambiguity here.

Weakness 2 Residual theorem

We would like to clarify that Theorem 1 provides a sufficient condition motivating residual visual injection, not a guarantee about the representations learned by ConvStack.

The theorem establishes that preserving approximately additive count evidence through a residual pathway can maintain count separation despite compression in the base stream caused by the softmax normalization. Crucially, as we explicitly state in Section 3.2, the theorem "does not guarantee that training learns these features or that subsequent language layers preserve them." We therefore do not claim that ConvStack must learn the exact additive count direction assumed by the theorem. Rather, this theoretical result motivates our architectural choice to reinforce spatial visual evidence through residual connections. The sufficient condition should not be interpreted as a necessary characterization of how the learned adapter operates. We did not claim ConvStack solved the counting problem fundamentally as there could be potentially other bottlenecks or better architecture design. We hope our work can provide good insights and inspire the community to propose better solutions. We also thank you for bringing out these concerns. We will refine our statement to reduce ambiguity and avoid causing misunderstanding.

Weakness 3 Novelty of the component design

We thank the reviewer for this comment. We agree that convolutional adapters and residual visual injection are established techniques. We explicitly acknowledge these connections in Section 2, including ConvPass for convolutional adaptation and DeepStack for multilayer visual injection. We do not claim novelty in these individual architectural components. Rather, our primary contribution lies in understanding visual counting failures in MLLMs and translating these insights into a targeted architectural intervention. We will make this distinction more explicit in the revised manuscript.

Weakness 4 Comparison with other adapter

Thanks for the comment. We would like to politely clarify that our primary objective is not to identify the optimal convolutional adapter architecture, but rather to investigate the underlying challenges of visual counting in MLLMs and explore an architectural intervention motivated by these findings. We provide additional “merger-only” ablation experiments. It can be seen that our design brings meaningful improvement.

ConfigurationPixMo-Count (%)SAT-real (%)
ConvStack-8B75.1963.33
Merger only70.9960.00
1 × 1 convolution kernel73.0959.33

Weakness 5 Transfer to other model architecture

Thank you for your suggestion! We provided additional results on LLaVA-1.5-7B and InternVL-3.5-8B with the same intervention and configuration. It can be seen our method can apply to other models as well.

ModelConfigurationPixMo-Count (%)CountBenchQA (%)SAT-real (%)
LLaVA-1.5-7BBaseline31.8744.6052.33
LLaVA-1.5-7BConvStack41.0351.5352.33
InternVL-3.5-8BBaseline62.9880.6557.00
InternVL-3.5-8BConvStack73.8584.7358.00

We use both Pixmo-Count and CountBenchQA as the real-world counting benchmarks, which are commonly used in the tech reports and blogs of MLLMs. It’s also worth noting the accurate annotation of counting in the real-world is very costly. We politely argue the dataset size (~500) of both benchmarks are statistically meaningful.

Q1 See Weakness 1

Q2 See Weakness 4

Q3 See Weakness 4

Q4 We show the counting performance on SynDot beyond 10 below. We fixed the image size as 512 × 512 pixels, therefore the larger the number, the more crowded the objects are.

Model1–1011–2021–3031–40
Qwen3-VL-8B80.20%11.05%9.65%2.85%
ConvStack-8B98.65%77.10%31.70%14.00%

Q5 See Weakness 5

Reviewer_J47uOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies visual counting as object individuation followed by aggregation. Synthetic experiments show that patch alignment affects learning speed and that local convolutions improve count classification. A simplified attention model explains how selective attention can compress the separation between adjacent counts; a conditional theorem describes when additive visual evidence permits stable readout. ConvStack applies local token mixing and residual visual injection to Qwen3-VL-8B. Training its adapters and the visual merger on counting data raises PixMo-Count accuracy from 65.46% to 75.19%, with smaller spatial gains and broadly similar general visual performance.

Critical strengths
  1. The synthetic experiments make a good case for local spatial filtering. Moving the same disks from patch centers to boundaries changes convergence speed, while replacing the 3 by 3 convolution with a 1 by 1 route retains much of the architecture and removes learned spatial mixing (Figures 2–3). Appendix D measures count-class separation in the original representation space, which gives firmer support than the t-SNE plots alone. These experiments help explain why local filtering makes count representations easier to learn.
  2. The strongest practical result is the PixMo-Count improvement with only 2.7M added parameters. Table 4 includes useful comparisons with the individual injection routes, a 1 by 1 control and LoRA, all trained on the same data. The spatial and six general visual benchmarks give a broader view of the adaptation. The paper reports the general scores that decrease and states the limits of its theorem clearly: it establishes a sufficient condition at the injection point, not a guarantee about the trained model.
Critical weaknesses
  1. The aggregation mechanism needs a more direct test. The attention calculation assumes homogeneous object and background tokens, count-independent logits and values, and a fixed incoming residual. Under those assumptions the result is sensible, and the paper states the restrictions. But the residual theorem assumes that the injected route already contains a shared direction growing approximately linearly with count, with bounded interference. Neither that property nor the proposed selective-attention compression is measured at the actual adapter locations. Improved count clustering after counting supervision is not enough to establish the explanation. ConvStack also injects spatial token features, whereas the theorem concerns a residual signal that already encodes count. How the learned adapter produces that signal remains unclear. I would retain the theory as a design argument and qualify the claim that the experiments have identified and overcome the aggregation bottleneck.
  2. The adaptation controls need more attention. Of the 42.827M trainable parameters, 40.119M belong to the existing visual merger (Table 5), but there is no merger-only baseline. The 1 by 1 control already achieves 73.09% on PixMo-Count, compared with 75.19% for ConvStack. That additional gain amounts to about 11 correct answers on the 524 test examples in Appendix A; training-seed variation and paired uncertainty are not reported. The transfer gains also vary considerably: CountBenchQA improves by 0.40 points and SAT-Spatial by 0.02 points. The improvement over the frozen model is substantial, but much of it is already obtained by a trained control without spatial mixing. The architectural claim should be judged against those adapted controls, and the evidence for broad spatial improvement should be stated more cautiously.
Questions for authors
  1. Is a merger-only result available with the same training mixture and schedule? Please give the LoRA parameter budget and clarify whether the adapted Table 2 baselines received the same counting supervision.
  2. Do the real-image ablation differences persist across training seeds, or is a paired comparison of the existing test predictions available? Please report the advantage over the 1 by 1 route separately from the gain over the frozen model.
  3. Where in the actual model does the visual route exhibit the approximately additive count direction required by Theorem 1? If this has not been tested, please present the theorem as a conditional design argument and qualify the aggregation diagnosis.
  4. Can the existing PixMo-Count and CountBenchQA results be broken down by target count? Both cover 2–10 objects, so this would show where the claimed benefit for larger counts occurs.
3 comments

We thank the reviewer for this insightful comment. We would like to address your concerns below:

Weakness 1 Concerns on assumption on aggregation mechanism & More empirical evidence

First, the homogeneous-token setting for Eq. 2-4 in our paper is designed to isolate aggregation from individuation. In other words, we assume that every object is perfectly individualized. This represents a favorable setting in which object fragmentation and count-dependent feature changes are eliminated as confounding factors. Nevertheless, even with perfect object individuation, softmax normalization produces diminishing adjacent-count separation. Our intention is therefore to demonstrate that successful individuation alone does not guarantee stable count aggregation.

Secondly, we would like to clarify that Theorem 1 provides a sufficient condition motivating residual visual injection, not a guarantee about the representations learned by ConvStack. The theorem establishes that preserving approximately additive count evidence through a residual pathway can maintain count separation despite compression in the base stream caused by the softmax normalization. Crucially, as we explicitly state in Section 3.2, the theorem "does not guarantee that training learns these features or that subsequent language layers preserve them." We therefore do not claim that ConvStack must learn the exact additive count direction assumed by the theorem. Rather, this theoretical result motivates our architectural choice to reinforce spatial visual evidence through residual connections. The sufficient condition should not be interpreted as a necessary characterization of how the learned adapter operates.

Third, our empirical evidence extends beyond improved clustering after counting supervised tuning. Figure 5 demonstrates that count-class separability is strongly associated with counting accuracy in pretrained Qwen3-VL-8B. More importantly, Figure 7 and Appendix D directly quantify count representation separation in the original hidden-state space before and after ConvStack adaptation in the final layer. These results demonstrate that ConvStack substantially improves count discriminability in the actual MLLM, consistent with our theoretical motivation. We also provide additional adjacent-count separation $\Gamma_n $ (Eq.D.2) analysis at the injection position as below.

Residual Injection LocationConvStack ResidualPre-Injection Hidden StatePost-Injection Hidden StateImprovement (%)
After LLM Layer 10.28560.16940.1786+5.43%
After LLM Layer 20.19490.17950.1888+5.18%
After LLM Layer 30.22390.18860.1958+3.82%

It can be seen that the ConvStack brings better counting representation separation to the hidden states. Note the improvement (%) here is decided by both $\Gamma_n $ and embedding norm. Our contribution here is to identify an aggregation failure mode, establish a sufficient condition for preserving count evidence, and empirically demonstrate that the theory-motivated architecture improves count representation separation and counting performance. We thank review for indicating the ambiguity here and will further clarify and refine our statement in the revised manuscript to avoid overstating.

Weakness2 Ablation study on “merger-only” & Multi-seed experiment result

We provide additional “merger-only” ablation experiments. It can be seen that our design brings meaningful improvement.

ConfigurationPixMo-Count (%)SAT-real (%)
ConvStack-8B75.1963.33
Merger only70.9960.00
1 × 1 convolution kernel73.0959.33

Furthermore, the 1×1 control, which retains the adapter structure while removing local spatial mixing, results in no improvement on the spatial understanding benchmark, SAT-real. This aligns with our analysis in Sec3.1 that local spatial mixing helps. Additionally, we provide experiment result for model training with 3 different seeds in below:

ModelSAT-realCountBenchQASAT-Spatial
Qwen3-VL-8B59.3389.8276.79
ConvStack64.00 ± 1.1590.22 ± 0.3176.82 ± 0.08

The ConvStack obtained meaningful gain on SAT-real and CountBenchQA, with broadly comparable performance on SAT-Spatial.

Q1 Ablation study and fair comparison

Please see Weakness 2 for merger-only results. LoRA parameter budget is 52.494M. Pixmo-Count is part of the training data of the Qwen3-VL model, therefore the comparison in Table 2 is fair.

Q2 Advantage over the 1 by 1 route

We show multi-seed results on real-image benchmarks in Weakness 2. It can be seen that the variance did not affect the conclusion that 3x3 brings better local spatial mixing signal than 1x1 control, as the improvement on SAT-real is substantial.

Q3 Statement ambiguity

As answered in Weakness1, we did not claim “ConvStack exhibits the approximately additive count direction assumed by Theorem 1”. Theorem 1 is an important motivation for why we propose ConvStack.

Q4 Per-class counting performance

We provide per class performance (%) on Pixmo-Count. It can be seen that ConvStack benefits large numbers most. We further provide visualization of per class performance improvement on both Pixmo-Count and CountBenchQA in Appendix C.3 in our updated manuscript.

Ground-Truth CountQwen3-VL-8BConvStackΔ (pp)
293.2294.92+1.69
381.6788.33+6.67
481.6786.67+5.00
581.3683.05+1.69
658.6274.14+15.52
767.2770.90+3.63
838.6059.65+21.05
940.6859.32+18.64
1043.8657.89+14.04

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.