- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces MixCount, a synthetic dataset and generation pipeline for open-vocabulary object counting. It contains 58,000 images generated from scanned 3D assets, with instance-level annotations, visibility information, text descriptions, and visual exemplars. The authors evaluate existing counters on MixCount and show that adding synthetic training images improves performance on real-world benchmarks.
Critical strengths
The strongest result is the improvement on real-world benchmarks after adding only 1,000 MixCount images. The gains on PairTally are substantial for both GeCo2 and CountGD++, suggesting that the dataset has value beyond evaluation on its own synthetic distribution. The combination of instance annotations, visibility measurements, and multiple prompt types also makes it a potentially useful research resource.
Critical weaknesses
-
The evaluation omits a directly relevant method, T2ICount. T2ICount (CVPR 2025) explicitly addresses text insensitivity and the dominant-category bias of existing counting benchmarks, and introduces FSC-147-S to evaluate this behavior. These issues substantially overlap with the motivation of MixCount. Testing CountSE, CountGD, and CountGD++ without a method specifically designed to address text sensitivity leaves the benchmark evaluation incomplete, particularly in the text-only setting. It remains unclear whether MixCount exposes limitations that persist in such methods or primarily highlights weaknesses of the selected models. This omission also weakens the argument that the observed failures are principally a training-data problem.
-
The claims about “exact annotations” and the data bottleneck go beyond the evidence provided. Rendering provides precise instance identities and masks, but it does not guarantee that a text description unambiguously identifies the intended objects in the final image. Distinct descriptions can overlap semantically, and distinguishing attributes may become invisible under occlusion or changes in viewpoint. The manual auditing is useful, but the paper does not establish the reliability of these image–text associations. Moreover, MixCount changes realism, scene composition, prompt detail, visibility filtering, and data volume simultaneously. The transfer results show that this combination helps; they do not establish that annotation errors in existing datasets are a principal cause of counting failures. Controlled ablations are needed to support that explanation.
Questions for authors
-
Can the authors evaluate T2ICount on MixCount under the same text-only protocol? Does it exhibit the same failure patterns, and does MixCount training provide additional gains?
-
How are descriptions checked for semantic ambiguity against other objects in the same scene, rather than only for correctness on an isolated asset? Is there a quantitative audit of the final image–text–count annotations?
-
What evidence separates the benefit of annotation accuracy and visibility filtering from that of more diverse scenes or additional training data? Were the data-augmentation comparisons matched for training budget and sampling protocol?
4 comments
Response to Reviewer Bfgp 2/2
W2/Q3. Controlled ablations supporting exact annotations
All comparisons are matched. Every run adds 1,000 images to the same real training set, with model, schedule, sampling and hardware fixed (CountGD++: 30 epochs, batch size 4, LR of decaying by a factor of 10 every 10 epochs, trained jointly. GeCo2: 1 epoch starting from pretrained FSC-147 checkpoint, batch size 4, LR of , weight decay ). We now state this in Sec. 4.2, and add the following controlled ablations for GeCo2 to Tab. 8 on p. 19:
- MixCount (no visibility filtering): we generate a new dataset of 1000 MixCount images without applying visibility filtering to isolate its effect.
- MixCount (noisy): to controllably simulate the missing annotations in real-world datasets, each annotation has a 5% dropout chance.
Removing visibility filtering or injecting label noise both increase error on FSCD-LVIS and PairTally, further demonstrating the need for a dataset with exact image annotations, and they are now added to our revised paper in Tab. 8.
| Training Data | FSCD-LVIS MAE ↓ | PairTally MAE ↓ |
|---|---|---|
| Real only (baseline) | ||
| FSC-147 [real] | 38.33 | 52.36 |
| Real + synthetic (ablating dataset factors) | ||
| FSC-147 [real] + 1K MixCount (single-category) [synthetic] | 33.85 | 51.14 |
| FSC-147 [real] + 1K MixCount (no visibility filtering) [synthetic] | 24.40 | 36.95 |
| FSC-147 [real] + 1K MixCount (noisy) [synthetic] | 22.84 | 34.27 |
| FSC-147 [real] + 1K MixCount (ours) [synthetic] | 20.11 | 31.88 |
| Synthetic only (testing scale) | ||
| 1K MixCount (ours) [synthetic] | 37.60 | 49.33 |
| 56K MixCount (ours) [synthetic] | 18.89 | 35.85 |
We hope this addresses the reviewer's concerns and would be glad to answer any follow-up.
Thank you for the additional experiments and clarifications. The T2ICount evaluation addresses my first concern, and the new ablations strengthen the evidence for the benefits of visibility filtering and cleaner labels. I also appreciate the clarification of the matched training settings.
My main remaining concern is the validity of the final image–text–count associations. The described verification operates at the asset level, whereas my question concerns whether the target remains identifiable from the actual rendered image and prompt. In the cube example, reasoning is appropriate if the relevant correspondence can be inferred from the available input; the renderer’s knowledge of asset identity alone does not establish this. A quantitative audit of representative final scenes would help distinguish challenging examples from underdetermined ones.
The noise ablation supports the benefit of cleaner labels under the tested corruption, but does not establish that annotation errors in existing datasets are a principal cause of counting failures. Moreover, visibility filtering removes objects before rendering, so its ablation also changes scene content.
Quantitative audit of the final image-text-count annotations
Thank you for your response and for recognizing that our T2ICount evaluation addresses your first concern and that the new ablations strengthen the evidence for the benefits of visibility filtering and cleaner labels.
Identifiability of final image-text-count associations. Following the reviewer’s suggestion, we manually audited representative final scenes to precisely distinguish challenging examples from underdetermined ones. We sampled 100 (image, target class) pairs in MixCount. Given only the rendered image and the concise text prompt, a human annotator placed a point on every instance of the target. Points were matched to the ground-truth instance masks. We include the results of the manual audit below:
- Instance level: recall 93.6%, precision 93.8% overall.
- Count level: MAE 1.54 (about 7% relative error). 68/100 counts are exact, and 86/100 are within ±1 with at most one missed instance.
- Where errors occur: they concentrate in the densest scenes (MAE 4.71 for scenes with 91+ objects, versus 0.20–1.40 below that) and in fine-grained classes such as apples of similar shades (recall about 80%). These differences are specified in the text and visible in the image, but require careful inspection. The median annotation time was 42 s per image.
- Underdetermined cases: in 3/100 images, the text prompt was deemed ambiguous. In each, scene lighting made the colour cue in the prompt indistinguishable from another class in the same scene (e.g. a “white” figurine rendered greenish under dim lighting). The audit therefore indicates that about 97% of sampled image-text pairs are unambiguous, with residual errors mostly reflecting the difficulty of fine-grained, dense counting rather than missing information. We thank the reviewer for prompting this analysis. We will add the audit results and our analysis to the paper.
Noise ablation. We agree, and will revise the claim. The noise ablation shows that cleaner labels improve counting performance under controlled corruption, but it does not establish that annotation errors in previous datasets specifically are the principal cause of counting failures. We will replace “this gap is strongly driven by limitations in existing training and evaluation data” with “training data with exact annotations substantially reduces these failures”, and present annotation quality as one contributing factor alongside scale and diversity.
Visibility filtering. We agree that this ablation also changes scene content. It is designed to evaluate a step of our generator as a whole, and shows that including this step improves downstream counting performance.
We thank the reviewer again for comments that have made the paper’s claims more precise.
Response to Reviewer Bfgp 1/2
We thank the reviewer for the careful reading and for highlighting the important gains on real-world benchmarks. Following your suggestions, we added T2ICount to our evaluation and added controlled ablations. This confirmed that the text-only architecture of T2ICount greatly benefits from MixCount data, and quantified the importance of non-noisy labels and visibility filtering with two new ablations.
W1/Q1. T2ICount benefits from MixCount data
As suggested, we now evaluate T2ICount under the same text-only protocol, and train it with additional MixCount data (Tab. 4, p. 10). Our revised paper includes the additional Fig. 11 (App. A.5, p. 18), which highlights the failure modes of the original pretrained T2ICount on self-similarity, repeating backgrounds, and mixed objects images. Even though T2ICount was designed to explicitly address text insensitivity, it is still challenged by the settings in MixCount.
To train T2ICount on MixCount, we initialize it from its pretrained checkpoint and train it for 1 epoch on 1k randomly sampled MixCount training images using the same training procedure as in the published T2ICount GitHub repository. Training with MixCount further improves T2ICount’s MAE by -48.13% on MixCount and -37.5% on PairTally:
W2/Q2. Text annotations
We agree that the guarantee differs in kind for the two annotation types: MixCount guarantees correct counts, masks, class identities, and that objects are sufficiently visible. In contrast, image-text annotations are verified. We now make this distinction explicit in our response and will state it in the final version.
Text annotation accuracy. The manual and automated verification protocol of text annotations is now described in full in App. Sec. B.4 and B.5, p. 20–21:
Ambiguity under occlusion and viewpoint. Distinguishing attributes that become invisible are, in our view, a strength of MixCount. For example, some scenes contain wooden toy cubes, each showing a letter and matching pictures on its faces (K with a koala, kiwi and key, M with a mouse, mushroom and mango). When the letter faces away from the camera, a cube showing a koala still belongs to the K class. A strong counting model should reason about the scene as a whole to infer this, as a human would. Existing datasets do not test this ability, and MixCount provides both training and evaluation data for it.