- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 1(−1)
- Presentation
- 1(−1)
- Soundness
- 1
Read review
Paper summary
This paper introduces OCCLUDEBENCH, a benchmark of 378 real images comparing VLMs’ ability to identify fully occluded objects using shadows, reflections, or scene context. It investigates failures through a four-stage diagnostic and several interventions. Models struggle more with physical projection cues, but the proposed bottleneck and the mechanism behind the inpainting intervention remain insufficiently supported.
Critical strengths
-
Addresses the valuable problem of fully occluded object identification, combining manually curated real images, five models, and a human baseline of 473 participants to provide a useful evaluation resource.
-
Decomposes the task into four diagnostic stages and compares reasoning, tools, prompting, and inpainting interventions, offering more analytical value than accuracy reporting alone.
Critical weaknesses
-
The two task categories use different images without matching object categories, cue sizes, visual complexity, or answer granularity. Identical masking procedures and a nonsignificant human performance difference do not establish that evidence type causes the model gap; within-image cue removal or replacement controls are needed.
-
OCCLUDETRACE does not establish a unique bottleneck: its stages change both inputs and task formats, while cropping also changes object scale. Appendix A.4 actually requests a description of the “entire scene,” contrary to the region-focused description in the main text. The 45% versus 90% difference could therefore reflect selective reporting rather than purely visual extraction failures.
-
Interpreting nonsignificance as “eliminating the gap” is unwarranted. Gaps of 8.1 and 5.6 percentage points remain after inpainting, and p=0.12/0.15 does not establish equivalence; likewise, the human result of p=0.079 does not rule out difficulty differences. Confidence intervals, equivalence tests, and paired pre-/post-intervention comparisons are needed.
-
Inpainting adds model calls, visual input, and verification prompts, but lacks budget-matched controls using textual hypothesis verification, repeated answering, or irrelevant infills. Its gains therefore cannot be attributed specifically to visual hypothesis externalization. Moreover, GPT’s inpainting accuracy of 54.3% is below the reported 65.4% with prompt steering, leaving the “most effective” claim without a consistent comparison basis.
Questions for authors
Please refer to Weaknesses.
2 comments
Dear Reviewer oFVs,
Thank you again for your detailed feedback. We ran the additional controls you asked about, and the inference gap remains after accounting for cue size, clutter, and answer granularity. Removing the physical cues from the same images also reduces accuracy, while the gap is substantially larger for models than for humans. We've tightened the OccludeTrace and inpainting claims as well. We appreciate the opportunity to clarify these points and would be grateful if you would consider raising your score.
Sincerely,
The Authors
Response to reviewer
We thank the reviewer for their detailed feedback. We have strengthened the controls, clarified OccludeTrace, and revised our claims. All intervals are 95% image-bootstrap CIs; humans and models use the same either-judge rule.
W1. Different images and potential confounders
Why different images? Direct images contain shadows/reflections of hidden objects, while inferential images require contextual reasoning. The categories necessarily differ in available evidence.
Object recognizability. When unmasked, models recognize objects at 86.3–94.4% accuracy on direct images and 89.5–93.4% on inferential images, suggesting comparable recognizability.
Controlled comparisons. The direct–inferential gap persists after accounting for potential confounders:
Within-image intervention. We removed shadows/reflections from the same images and reevaluated all five models. Accuracy fell by 17.2–42.6 points, reaching just 7.8% for Qwen and 9.4% for Llama. This demonstrates that physical cues contribute substantially to predictions, independent of category differences.
W2. OccludeTrace and bottleneck identification
Stage design. OccludeTrace deliberately changes inputs and tasks to isolate distinct capabilities. We now describe Stage 4 as the largest observed separation, not a uniquely established bottleneck.
Matched-image evidence. Stage 4 most strongly distinguishes initially correct and incorrect examples (41.1 points [29.6, 52.0]). On initially failed examples, accuracy rises from 50.0% with the relevant clue highlighted to 77.1% with an additional close-up, a 27.1-point gain [16.1, 38.3]. This supports difficulty extracting physical evidence from full scenes, although cropping also changes cue scale.
Prompt clarification. We apologize for the appendix inconsistency, which we have corrected. Stage 4 actually uses a region-focused prompt with the shadow/reflection highlighted, asking models to describe the projection area. Accuracy remains 50% on initially failed versus 91% on initially successful examples, despite directing attention to the relevant evidence.
W3. Statistical interpretation
We no longer interpret nonsignificance as eliminating a gap. We provide confidence intervals, paired comparisons, and an equivalence test.
Inpainting. Applying inpainting only to initially incorrect answers improves accuracy by 8.1 and 5.6 points. We now report paired confidence intervals.
Human difficulty. The human direct–inferential gap is 4.9 points [-0.9, 10.9], versus 20.1 [14.1, 26.1] across five models. A two one-sided equivalence test (α=0.05) supports a human gap within ±10.1 points, while the model gap lies outside this margin (90% CI [15.0, 25.1]). More directly, the model gap exceeds the human gap by 15.1 points [8.8, 21.4] on the same images. Controlling human difficulty, the model gap remains 18.0 points [12.7, 23.2].
Paired intervention comparisons appear below.
W4. Inpainting controls and attribution
We added a repeated-answering text hypothesis verification control for GPT-5.4 to test whether another verbal attempt alone improves accuracy.
Inpainting improves accuracy by 8.2 points [1.0, 15.3], versus 2.5 [-4.6, 9.6] for repeated answering. This suggests a possible benefit beyond another verbal attempt, although the control does not isolate visual generation from other protocol differences.
Comparison with steering. The previously reported 54.3% and 82.2% are second-look scores: they retain correct baseline answers and use inpainting only for initial failures. We clarify this distinction and remove the claim that inpainting is the “most effective” intervention.
Conclusion. The central finding remains: models exhibit a 20.1-point direct–inferential gap, exceeding the human gap by 15.1 points. Our additional controls strengthen this finding while clarifying the limits of our diagnostic and intervention claims.
Sincerely,
The Authors