- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces COVE, which evaluates VLMs’ existence and “at least two” judgments through category-removal and multiple-to-single-instance image edits. It distinguishes persistence rates over all pairs from those conditioned on original affirmative answers. Experiments across six open models, multiple prompts, and editors reveal substantial persistence in quantity judgments, but the core count edits lack shared, independently verifiable visual validation.
Critical strengths
-
Uses “at least two” as a minimal quantity test that preserves category evidence while avoiding the additional difficulty of exact-numeral generation. Retained/removed instance records also facilitate reuse and alternative editors.
-
Clearly distinguishes persistence denominators and edit quality from model performance, uses paired tests on shared samples, and avoids treating parsing failures as NO, resulting in relatively careful statistical reporting.
Critical weaknesses
-
The central counting conclusions rely on metadata screening rather than visual confirmation that exactly one instance remains. Area filtering may omit small objects, and box-overlap thresholds cannot exclude residual evidence or grouping ambiguity. The separate count audit provides only model-specific aggregates, with eligible denominators differing from original-YES counts and untraceable exclusions. It therefore does not adequately validate the reported 50.2–95.1% persistence rates.
-
The existence and quantity tests share only 16 image–category pairs and use different screening criteria. Although the authors explicitly frame the comparison as descriptive, they do not directly establish that models recognize category presence yet fail to update quantity in the same visual scenes. Both predicates should be evaluated on the same visually audited 2→1→0 image sequences.
-
Quantity-specific controls are missing: high invariance under irrelevant existence edits does not establish invariance for quantity judgments. Count-preserving matched edits, natural single-instance negatives, and no-image baselines are needed to distinguish inadequate use of quantity evidence from general YES bias and edit-induced errors. Testing only the YES→NO direction also limits diagnostic completeness.
-
The editor replication lacks editor-specific visual audits. Blurring, filling, and inpainting may preserve or regenerate countable evidence, so persistence across multiple editors does not rule out shared intervention failures. Comparisons should use a common subset that passes visual validation under every editor.
-
Relative to BEAF, ROHE, and existing counterfactual counting work, the main additions are a binary threshold, instance records, and quality strata, leaving methodological novelty limited. Shared-sample comparisons or analyses of newly identified failure types are needed to demonstrate diagnostic information unavailable from existing removal and counting evaluations.
Questions for authors
Please refer to Weaknesses.