acceptodds
Withdrawn from ICLR 2027

Counterfactual Auditing of Object Presence and Quantity Judgments in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) increasingly support systems that interpret visual content, making reliable evaluation essential. A basic capability is judging object presence, yet a correct answer on a static benchmark does not establish whether it relies on the relevant visual evidence. We introduce Counterfactual Object Visual Evidence (COVE), a paired-image benchmark that tests whether models update their answers when supporting object evidence is removed. Existence edits aim to remove the queried category, whereas quantity edits leave one instance, preserving category evidence while invalidating the at-least-two condition. The core datasets contain 160 existence pairs and 1,000 quantity pairs, complemented by 906 irrelevant-region existence controls. COVE measures evidence persistence: affirmative answers on both the original and target-edited image, stratified by intervention quality. Across six open VLMs, all-pair persistence ranges from 2.9–35.3% on 102 visually screened existence pairs and from 50.2–95.1% on a separate cohort of 329 metadata-screened two-to-one quantity pairs. More than 97% of eligible original-affirmative responses remain affirmative under irrelevant existence edits. Substantial count persistence recurs across prompts and editors; separate visual-audit aggregates provide a descriptive cross-check. These profiles show that low existence persistence can coexist with substantial difficulty updating quantity judgments. By combining instance-level removal records, quality strata, and paired response metrics, COVE provides a reusable resource for evaluation beyond static correctness.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, visual grounding, counterfactual evaluation, datasets and benchmarks, object counting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SiJSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces COVE, which evaluates VLMs’ existence and “at least two” judgments through category-removal and multiple-to-single-instance image edits. It distinguishes persistence rates over all pairs from those conditioned on original affirmative answers. Experiments across six open models, multiple prompts, and editors reveal substantial persistence in quantity judgments, but the core count edits lack shared, independently verifiable visual validation.

Critical strengths
  • Uses “at least two” as a minimal quantity test that preserves category evidence while avoiding the additional difficulty of exact-numeral generation. Retained/removed instance records also facilitate reuse and alternative editors.

  • Clearly distinguishes persistence denominators and edit quality from model performance, uses paired tests on shared samples, and avoids treating parsing failures as NO, resulting in relatively careful statistical reporting.

Critical weaknesses
  • The central counting conclusions rely on metadata screening rather than visual confirmation that exactly one instance remains. Area filtering may omit small objects, and box-overlap thresholds cannot exclude residual evidence or grouping ambiguity. The separate count audit provides only model-specific aggregates, with eligible denominators differing from original-YES counts and untraceable exclusions. It therefore does not adequately validate the reported 50.2–95.1% persistence rates.

  • The existence and quantity tests share only 16 image–category pairs and use different screening criteria. Although the authors explicitly frame the comparison as descriptive, they do not directly establish that models recognize category presence yet fail to update quantity in the same visual scenes. Both predicates should be evaluated on the same visually audited 2→1→0 image sequences.

  • Quantity-specific controls are missing: high invariance under irrelevant existence edits does not establish invariance for quantity judgments. Count-preserving matched edits, natural single-instance negatives, and no-image baselines are needed to distinguish inadequate use of quantity evidence from general YES bias and edit-induced errors. Testing only the YES→NO direction also limits diagnostic completeness.

  • The editor replication lacks editor-specific visual audits. Blurring, filling, and inpainting may preserve or regenerate countable evidence, so persistence across multiple editors does not rule out shared intervention failures. Comparisons should use a common subset that passes visual validation under every editor.

  • Relative to BEAF, ROHE, and existing counterfactual counting work, the main additions are a binary threshold, instance records, and quality strata, leaving methodological novelty limited. Shared-sample comparisons or analyses of newly identified failure types are needed to demonstrate diagnostic information unavailable from existing removal and counting evaluations.

Questions for authors

Please refer to Weaknesses.

Reviewer_PpXmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

COVE proposes a new evaluation dataset for measuring the ability of VLMs to preserve and count properties in images. The dataset consists of 160 existence pairs and 1000 count pairs generated from COCO instance annotations through controlled object removal. The paper evaluates six VLMs using paired original and edited images. Two main metrics are introduced: (1) persistence, where a model answers YES both before and after the evidence supporting that answer has been removed, and (2) stability under irrelevant edits, where unrelated objects are removed and the answer is expected to stay unchanged. The main finding is that VLMs appear to update existence judgments more reliably than quantity judgments. The paper also introduces control experiments based on irrelevant object removals.

Critical strengths
  1. The inclusion of irrelevant-edit controls is a useful desgn choice. Including control experiments through the removal of irrelevant objects is a good idea to test whether models are reacting specifically to manipulated evidence regions rather than simply reacting to the fact that an image has been edited. This strengthens the benchmark and provides a sanity check for the proposed interventions.

  2. The paper studies an important problem. Measuring whether VLMs actually use the visual evidence that supports their answers is an important research direction. Standard accuracy benchmarks cannot determine whether a model is relying on the correct visual evidence, and intervention-based evaluation is a reasonable way to investigate this question.

Critical weaknesses
  1. The proposed benchmark is very small and the experimental evaluation is narrow. The final benchmark size appears quite limited. The paper starts from 160 existence pairs and 1000 count pairs, but later reports subsets such as 102 visually audited pairs, 329 strict count pairs, and 906 valid control pairs. It is therefore difficult to understand what the effective dataset size actually is for the main conclusions. Given the small number of examples and only six evaluated models, I find it difficult to draw reliable conclusions about VLM counting and preservation capabilities in general. In addition, the tested models do not include many current large-scale state-of-the-art VLMs (e.g. larger 27B+ models), which further limits the strength of the conclusions.

  2. The intervention setting is very limited and may introduce artifacts. The counting task is essentially limited to determining whether there are at least two instances of an object. The scope of the benchmark is therefore quite narrow and does not evaluate more general counting behavior. Furthermore, after going through the appendix examples, I was disappointed to see that the edited images appear to use gray-box masking rather than realistic object removal through inpainting. As a result, the benchmark may partially measure sensitivity to artificial occlusions or masking artifacts rather than evidence-grounded counting behavior. It would be helpful to justify this design choice or compare against more realistic inpainted interventions. The paper provides examples with only 0→1 and 2→1 style transitions, and it is unclear why the benchmark was not extended to larger object counts given the relative simplicity of many examples.

  3. Missing relevant related work:

  • Towards Causal VQA: Revealing and Reducing Spurious Correlations by Invariant and Covariant Semantic Editing (CVPR 2020): What does COVE add beyond counting updates after instance removal and invariance under irrelevant-object removal?
  • Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts (EMNLP 2025) and DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning (ICML 2026): How does COVE distinguish failure to count from reliance on learned priors or failures of evidence grounding?

Additional concerns The paper is difficult to read and many details of the benchmark construction are hard to follow. Terminology such as "metadata screening checks" (L77), "count bookkeeping"(L77), "geometry checks"(L77), "visual-clean", and "proxy-clean" is not clearly explained or differentiated. It is often difficult to understand which dataset subset is used in which experiment, making it hard to assess the strength of the evidence.

Questions for authors

The final benchmark size is difficult to understand. Can the authors provide a clear table showing how the dataset size evolves after each filtering stage and which subset is used for each reported result? Why is the quantity benchmark restricted to the "at least two objects" setting? Did the authors explore larger cardinalities (e.g. 3, 4, or more objects), and if not, why? The edited examples appear to use masking rather than realistic inpainting. How do the authors rule out the possibility that the reported effects are driven by masking artifacts rather than failures of quantity reasoning? Have the authors evaluated larger state-of-the-art VLMs? If not, how confident are the authors that the reported phenomenon generalizes beyond the six models considered here?

Reviewer_jYSEOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a benchmark on existence and counting tasks by VLMs. The benchmark contains paired images, one taken from COCO dataset including the objects (single, or multiple) the other one edited to remove the objects to either either change the response to existence to No or change the count for the objects. For existence and counts tasks they propose different sets of 160 and 1000 pairs respectively. They further include 906 existence pairs with irrelevant edits to evaluate the impact of artificially editing the images on the VLM’s response. They finally report the persistence in the yes answers changing the images from unedited to edited.

Critical strengths
  • Including the set of 906 pairs on irrelevant visual element removal strengthens the findings by eliminating the possible effect of artificially edited images.
  • The paper is well-written and easy to follow. The main idea is easy to follow and the data generation pipeline while simple is effective.
Critical weaknesses
  • The human evaluation on the counting set is only on Meta data and annotators do not validate the edited images visually. Because of possible editing errors such as missing small objects, or objects with unusual representation, it is hard to validate the findings and results of the counting task which is one of the major contribution of the paper.
    • In Fig. 11 - b, the example there is another box behind the suitcase, that as a human I saw it as suitcase, and removing the one in the front the one in the back still exists and makes the count 2 and the answer to minimum two suitcases remain yes.
  • The main motivation of the paper is not tested. In the introduction the paper mentions that the correct response might be caused by the context rather than actual existence of the object. For example, existence of street in the image can hallucinate car by VLMs. This motivation which could distinguish the novelty of this paper from prior works is not tested in the paper. One way to validate this motivation could be testing the existence on asking for existence of the objects irrelevant to the context.
Questions for authors
  • Why are the images sets for two tasks are different? If the sets were the same, the two tasks could have been comparable and add more insight to the findings. To have shared clean images for two sets, the dataset could be as it is for counting task and removing all objects (in clean images in count set) for existence set.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.