acceptodds
Under review as a conference paper at ICLR 2027

OccludeBench: Can VLMs Find What's Missing?

Abstract

How well can VLMs reason about what is missing from an image? We study this question with OccludeBench, a manually curated benchmark for recovering fully occluded objects from the visual evidence that remains. OccludeBench distinguishes direct cases, where the hidden object must be recovered from a specific physical cue such as a shadow or reflection, from inferential cases, where it can be inferred from broader scene context. Across five VLMs, accuracy averages 40.5% on direct cases versus 60.3% on inferential cases, exhibiting a 19.9-point gap. Humans attain higher accuracies of 63.4% versus 68.5% for direct and inferential images, respectively, and exhibit a much smaller gap of only 5.1 points. To locate this failure, we introduce OccludeTrace, a four-stage diagnostic that isolates where the visual reasoning process breaks down. The bottleneck emerges when models must recover the relevant object from the projection within the full image: they succeed on only 45% of failed examples, compared with 90% of examples they answer correctly. Extended reasoning, tool use, and few-shot prompting yield inconsistent improvements and do not reliably eliminate the gap. In contrast, an inpaint-then-confront intervention, which prompts models to first generate a candidate hidden object and then evaluate that hypothesis against the visible physical cue, reduces the gap to a statistically non-significant difference for both models with image-generation capabilities. Together, our results establish a tractable, controlled diagnostic for isolating how VLMs reason about visual content that is missing from an image.

Primary area:
datasets and benchmarks
Keywords:
Vision Language Model, Multimodal, Evaluation, Computer Vision, Dataset

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oFVsOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1(−1)
Presentation
1(−1)
Soundness
1
Read review
Paper summary

This paper introduces OCCLUDEBENCH, a benchmark of 378 real images comparing VLMs’ ability to identify fully occluded objects using shadows, reflections, or scene context. It investigates failures through a four-stage diagnostic and several interventions. Models struggle more with physical projection cues, but the proposed bottleneck and the mechanism behind the inpainting intervention remain insufficiently supported.

Critical strengths
  • Addresses the valuable problem of fully occluded object identification, combining manually curated real images, five models, and a human baseline of 473 participants to provide a useful evaluation resource.

  • Decomposes the task into four diagnostic stages and compares reasoning, tools, prompting, and inpainting interventions, offering more analytical value than accuracy reporting alone.

Critical weaknesses
  • The two task categories use different images without matching object categories, cue sizes, visual complexity, or answer granularity. Identical masking procedures and a nonsignificant human performance difference do not establish that evidence type causes the model gap; within-image cue removal or replacement controls are needed.

  • OCCLUDETRACE does not establish a unique bottleneck: its stages change both inputs and task formats, while cropping also changes object scale. Appendix A.4 actually requests a description of the “entire scene,” contrary to the region-focused description in the main text. The 45% versus 90% difference could therefore reflect selective reporting rather than purely visual extraction failures.

  • Interpreting nonsignificance as “eliminating the gap” is unwarranted. Gaps of 8.1 and 5.6 percentage points remain after inpainting, and p=0.12/0.15 does not establish equivalence; likewise, the human result of p=0.079 does not rule out difficulty differences. Confidence intervals, equivalence tests, and paired pre-/post-intervention comparisons are needed.

  • Inpainting adds model calls, visual input, and verification prompts, but lacks budget-matched controls using textual hypothesis verification, repeated answering, or irrelevant infills. Its gains therefore cannot be attributed specifically to visual hypothesis externalization. Moreover, GPT’s inpainting accuracy of 54.3% is below the reported 65.4% with prompt steering, leaving the “most effective” claim without a consistent comparison basis.

Questions for authors

Please refer to Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to reviewer

We thank the reviewer for their detailed feedback. We have strengthened the controls, clarified OccludeTrace, and revised our claims. All intervals are 95% image-bootstrap CIs; humans and models use the same either-judge rule.

W1. Different images and potential confounders

Why different images? Direct images contain shadows/reflections of hidden objects, while inferential images require contextual reasoning. The categories necessarily differ in available evidence.

Object recognizability. When unmasked, models recognize objects at 86.3–94.4% accuracy on direct images and 89.5–93.4% on inferential images, suggesting comparable recognizability.

Controlled comparisons. The direct–inferential gap persists after accounting for potential confounders:

  • Cue size: Larger projections modestly improve accuracy (r=0.19), yet the largest third achieves only 45.5%, versus 62.8% on inferential images. Controlling mask size yields a 20.8-point gap [14.8, 26.6].
  • Visual complexity: Controlling edge density (visual clutter), the gap remains 20.7 points [14.2, 26.5].
  • Answer granularity: Both categories have median one-word answers and similar concreteness (4.65 vs. 4.54), frequency (4.48 vs. 4.22), and prevalence (1.000 vs. 0.996). Controlling answer length, the gap remains 19.9 points [13.7, 26.1].

Within-image intervention. We removed shadows/reflections from the same images and reevaluated all five models. Accuracy fell by 17.2–42.6 points, reaching just 7.8% for Qwen and 9.4% for Llama. This demonstrates that physical cues contribute substantially to predictions, independent of category differences.

W2. OccludeTrace and bottleneck identification

Stage design. OccludeTrace deliberately changes inputs and tasks to isolate distinct capabilities. We now describe Stage 4 as the largest observed separation, not a uniquely established bottleneck.

Matched-image evidence. Stage 4 most strongly distinguishes initially correct and incorrect examples (41.1 points [29.6, 52.0]). On initially failed examples, accuracy rises from 50.0% with the relevant clue highlighted to 77.1% with an additional close-up, a 27.1-point gain [16.1, 38.3]. This supports difficulty extracting physical evidence from full scenes, although cropping also changes cue scale.

Prompt clarification. We apologize for the appendix inconsistency, which we have corrected. Stage 4 actually uses a region-focused prompt with the shadow/reflection highlighted, asking models to describe the projection area. Accuracy remains 50% on initially failed versus 91% on initially successful examples, despite directing attention to the relevant evidence.

W3. Statistical interpretation

We no longer interpret nonsignificance as eliminating a gap. We provide confidence intervals, paired comparisons, and an equivalence test.

Inpainting. Applying inpainting only to initially incorrect answers improves accuracy by 8.1 and 5.6 points. We now report paired confidence intervals.

Human difficulty. The human direct–inferential gap is 4.9 points [-0.9, 10.9], versus 20.1 [14.1, 26.1] across five models. A two one-sided equivalence test (α=0.05) supports a human gap within ±10.1 points, while the model gap lies outside this margin (90% CI [15.0, 25.1]). More directly, the model gap exceeds the human gap by 15.1 points [8.8, 21.4] on the same images. Controlling human difficulty, the model gap remains 18.0 points [12.7, 23.2].

Paired intervention comparisons appear below.

W4. Inpainting controls and attribution

We added a repeated-answering text hypothesis verification control for GPT-5.4 to test whether another verbal attempt alone improves accuracy.

ConditionBaselineResult
Inpainting37.2%45.4%
Repeated answering37.6%40.1%

Inpainting improves accuracy by 8.2 points [1.0, 15.3], versus 2.5 [-4.6, 9.6] for repeated answering. This suggests a possible benefit beyond another verbal attempt, although the control does not isolate visual generation from other protocol differences.

Comparison with steering. The previously reported 54.3% and 82.2% are second-look scores: they retain correct baseline answers and use inpainting only for initial failures. We clarify this distinction and remove the claim that inpainting is the “most effective” intervention.

Conclusion. The central finding remains: models exhibit a 20.1-point direct–inferential gap, exceeding the human gap by 15.1 points. Our additional controls strengthen this finding while clarifying the limits of our diagnostic and intervention claims.

Sincerely,
The Authors

Dear Reviewer oFVs,

Thank you again for your detailed feedback. We ran the additional controls you asked about, and the inference gap remains after accounting for cue size, clutter, and answer granularity. Removing the physical cues from the same images also reduces accuracy, while the gap is substantially larger for models than for humans. We've tightened the OccludeTrace and inpainting claims as well. We appreciate the opportunity to clarify these points and would be grateful if you would consider raising your score.

Sincerely,

The Authors

Reviewer_U9EUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a manually annotated benchmark named OccludeBench that contains 378 images, where the questions require VLMs to recover fully occluded objects from the visual evidence. It has two types: (a) direct: where the hidden objects can be recovered from physical cues such as shadow or reflection, (b) inferential: where the hidden objects can be recovered from broader scene context. The paper evaluates the proposed benchmark on multiple models and multiple settings.

Critical strengths
  1. The work proposes a new manually curated benchmark called OccludeBench, which assesses VLMs ability to answer questions about fully occluded objects in the scene. Two types of occlusions are considered: direct (can be recovered with shadow or reflection), and inferential (can be recovered with broader scene context).
  2. Evaluates multiple models in multiple different settings along with OccludeTrace, which is a four-stage diagnostic to check where the reasoning breaks down.
Critical weaknesses
  1. This is a very small-scale hand curated benchmark, which re-establishes the same concern that other works which show that VLMs rely on prior instead of trying to reason about the objects (https://arxiv.org/abs/2508.04059). This paper narrows it, and focuses only on occluded objects. If the authors had provided a way to resolve this problem, and feed this information back to the training loop it would have helped the community more than creating another benchmark.
  2. As human accuracy on the benchmark is very low, (in one of the settings, a model (gemini) is better than human accuracy) raises an important topic, if the images/object occluded are too ambiguous?
  3. Some important details regarding an important section, i.e. OccludeTrace is missing. Example: which models were tested for OccludeTrace? Were they tested on both direct and inferential images? And which grader was used for stage 4, is it similar to the LLM as judge mentioned? For stage 2, the step of finding the most relevant part is skipped. Overall, it is very hard to understand the design choice of OccludeTrace and connect it with if and how it benefits OccludeBench.
Questions for authors
  1. Based on line 204-205, humans achieve accuracy of 65.9% which is lower compared to Gemini’s performance, especially for inferential by a very good margin (86.2 (gemini) vs 68.5 (humans)). Why is this the case? Are the examples very hard even for humans?
  2. Most of the qualitative examples provided in the paper don’t have an example of ground truth answer, making it hard to assess the name of the object being occluded.
  3. For Figure 6, prompt steering results are only added for direct types, and for inferential it uses base. Why is inferential type not tested with different prompts or prompt steering?
  4. Among different settings tested, like tool use, extended reasoning, steering prompts, etc, steering prompts yield larger gains overall. Looks like by having better prompts, the models get better at answering/looking at the image. It is suggested that more effort is put in a setting that can yield the maximum performance improvements, which in return will provide more information on what we are supposed to probe the model for. Ofcourse experiments with tools also hold importance in general, but for the task of finding occluded objects, the more meta-information is provided in the prompt, the better models get in general.
  5. Is there a specific reason why experiments like extended reasoning, few-shot prompting were performed on both direct and inferential images, but tool use, steering prompts were only performed on direct images? This setup does not fully support the conclusions unless specified otherwise.
  6. For inpainting, table 3 provides scores for multi-step evaluation for two models. The table is very weirdly structured where the performance is used as rates. It is suggested to add all the small experiments that lead to gain/drop in performance in one table which can help readers understand which experiment/setting helps the most.
  7. For Figure 7, where accuracies of different VLMs are presented for extended reasoning and few-shot settings, it is hard to report new methods performance accuracies as the figure only shows the trends, but in order to compare/report for newer methods, providing the values is more beneficial.
  8. Paragraph in line 372, does not mention prompt steering as an influential parameter. It just mentions that tool use, and extended reasoning fails and few-shot prompting helps one model. Please mention the effect of the steering prompt in this section.
  9. Paragraph in line 372, mentions that inpaint-then-confront helps with performance improvement, but as shown in Figure 8, the inpainting module can hypothesize a new scene making the evaluation even noisier. Is there any explanation on this?
  10. Are the humans graded with the same LLM as a judge? Also, it was mentioned that more than one human might have reviewed/answered the question, how is finally human accuracy calculated?
  11. Currently figure 3 reads very confusing. Please add more explanation.
  12. There is no table that gives the total performance of full OccludeBench. Table 1 provides individual tasks, i.e. direct and inferential. Should we assume that the full score will be the unweighted average of the two types?
  13. Table 1, shows the accuracy of GPT-5.4 on direct type as 36.0% whereas in Table 3, it is 37.6%. Are these two different settings? Similar with Gemini model as well. Please clarify both the settings if they are different.
9 comments
AuthorsOpenReview ↗

Response to reviewer

We thank the reviewer for their feedback. Intervals are 95% image-bootstrap CIs; accuracy uses either-judge grading.

Weaknesses

W1. Scale and novelty. We manually curated 378 images because synthetic images failed to reproduce reliable shadows and reflections. Across five models, we identify a 20.1-point direct–inferential gap. Unlike O-Bench (arXiv:2508.04059), which tests partially visible objects through multiple choice, OccludeBench fully hides objects and distinguishes physical evidence from contextual inference. Crucially, OccludeTrace goes beyond documenting this gap to decompose why models fail, isolating evidence localization, recognition, and interpretation into distinct diagnostic stages. This identifies extracting and interpreting physical evidence as concrete targets for future training.

W2/Q1. Solvability. At least one human correctly answers 356/378 (94.2%) images; a majority answer 282/378 (74.6%). Unmasked-object recognition exceeds 86.3% for models in both categories. Gemini exceeding mean human accuracy reflects variation across respondents, not proof that objects are inherently unrecoverable.

W3. OccludeTrace. We test GPT-5.4, Claude Opus 4.6, and Gemini 3.1 Pro on direct images (inferential images lack projections). Stage 2 provides a crop deliberately to isolate recognition from locating evidence. Stage 4 uses the same judges and accepts descriptions identifying the depicted object. Stage 4 best separates initially correct versus incorrect examples (41.1 points), locating the main failure.

Questions

Q2. Ground truth. We will add object labels to all qualitative examples.

Q3. Inferential bars. Figure 6 uses baseline inferential accuracy because steering targets physical projections; we clarify its caption.

Q4. Steering. Short steering raises direct accuracy by about 4.0–6.6 points across four models. On 168 paired GPT-5.4 images, adding short steering to reasoning improves accuracy by 14.9 points [8.3, 21.4]; long steering adds 7.6 [1.5, 13.7] for Claude. Longer prompts are not reliably better: GPT-5.4 gains only 1.8 [-5.3, 8.9] with reasoning.

Q5. Scope. Reasoning/few-shot are tested on both categories; steering/tools target physical cues and are tested on direct images. These experiments assess direct accuracy, not whether the category gap closes. Under reasoning, GPT-5.4's gap is 30.2 points, versus 24.8 at baseline.

Q6/Q7. Consolidated results. We replace Table 3 with exact intervention accuracies (%). Entries show direct/inferential, or direct only. D=Direct I=Inferential

SettingGPTClaudeGeminiQwenLlama
Overall49.563.079.433.236.6
Base D/I37.6/62.451.8/75.171.6/87.820.9/46.431.6/42.0
Reason D/I51.8/82.044.7/72.4———
Few-shot D/I50.0/66.754.1/81.780.1/82.833.2/37.835.7/43.3
Short steer D/I42.1/61.958.4/74.675.6/81.2—37.1/—
Short steer + reason D/I69.2/82.849.7/68.5———
Long steer + reason D57.652.3———
Tools D/I46.7/72.051.8/76.3———
Short steer + tools D49.555.3———
Long steer + tools D44.759.4———
Inpainting D45.4—70.2——

Q8. Discussion. We explicitly add steering to the intervention discussion, including its gains with reasoning and dependence on model/prompt length.

Q9. Inpainting noise. Inpainting may hallucinate inconsistent scenes (Figure 8), so it is hypothesis testing rather than faithful reconstruction. It nevertheless corrects 33/123 initially wrong GPT answers and 21/55 Gemini answers.

Q10. Human grading. The same judges evaluate each human response separately (median six responses/image); human accuracy averages individual responses, not majority votes.

Q11. Figure 3. Dots are 10-point human-accuracy bins; size shows image count, height shows mean model accuracy (r=0.44). We clarify the caption.

Q12. Overall score. Direct (197) and inferential (181) accuracies are weighted by image counts.

Q13. Table mismatch. Settings/responses are identical; Table 1 averaged judge verdicts, while Table 3 used either-judge grading (36.0 vs. 37.6 GPT; 69.0 vs. 71.6 Gemini). We standardize to either-judge.

Conclusion. The models' direct–inferential gap exceeds humans' by 15.1 points; identifying relevant physical evidence in full scenes remains the main bottleneck.

Dear Reviewer U9EU,

Thank you again for taking the time to review OccludeBench. We expanded our explanation of how the images were curated and what distinguishes the benchmark from earlier occlusion work. We also added checks showing that humans can solve nearly all of these images and that models recognize the objects when revealed, and we brought the intervention results together under consistent grading. We hope these clarifications and new results help resolve your concerns, and we'd be grateful if you would consider raising your score.

Sincerely,

The Authors

Thanks to authors for the clarifications. The following are follow-up questions and comments:

W1: The authors do not completely answer the weakness, which makes it harder to evaluate the response. W3: OccludeTrace explanation still misses key aspects like how its design choice effects/benefits OccludeBench.

Q1. The rebuttal shows that 94.2% – any human correct, 74.6% – majority correct, while paper reports 65.9% – mean over individual response. It is very important to state it clearly which is the human baseline and why, as these are directly compared with model performance.

Q4. The answer lists gains but still doesn’t explain how steering helps. The suggestion was: “It is suggested that more effort is put in a setting that can yield the maximum performance improvements, which in return will provide more information on what we are supposed to probe the model for. Ofcourse experiments with tools also hold importance in general, but for the task of finding occluded objects, the more meta-information is provided in the prompt, the better models get in general.”

Q5. It is still unclear and not completely answered on why steering/tools is not applicable for inferential images. Atleast it should be tried and if it doesn’t work, then it can be mentioned that the gaps are not closed, or if it not applicable should come with a better explanation.

Q6. Table does not include the inpainting results.

New questions: For reply for Q6,Q7, in the table for few-shot D/I, the score for GPT is 50.0 for D, where as in the paper it is given 47 (line 358). Did the evaluate setting change?

Overall, I would still keep my rating (2: Weak rejection) as given previously as key issues (Weakness 1, Weakness 3, and several other questions) were not answered satisfactorily.

Reviewer_1pfeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The work attempts to investigate VLMs visual reasoning ability, specifically regarding entities that are missing in an image.
It introduces OccludeBench a benchmark with 378 images and OccludeTrace a diagnostic method for visual reasoning on VLMs ability to deduce visually occluded objects within a scene. It distinguishes two scenarios, a direct case where models can derive the answer from clues such as reflections and shadows and a inferential case where model has to perform abductive reasoning to answer based on context and environment.
OccludeTrace aims pinpoint the source of failure in the reasoning process, it proposes a four-stage diagnostic that disentangles recovering a hidden object into detecting the projection, recognizing it in isolation, mapping it to its source, and recovering the object from the cue inside the full image. It evaluate 5 SOTA models( Llama 4 maverick , qwen3VL-8B,Claude Opus 4.6, Gemini 3.1 Pro, and GPT 5.4) and found a gap in performance on direct vs inferential cases, a gap that is less pronounced in humans.
It further investigates the extent of these failures through studies on mitigation techniques. Their central finding is that VLMs generally recognize relevant projections in isolation but struggle to extract and use the same evidence when it appears within a visually complex scene. They investigate several interventions and find that in paint-then-confront, which generates and verifies a visual hypothesis, substantially improves performance, suggesting that models are failing does not indicate that they cannot utilize the clues, and that externalizing these hypotheses and then evaluating tem aids models in properly utilize them.

Critical strengths
  • The paper studies an interesting aspect of multimodal reasoning failure and visual abductive reasoning capability that is underexplored by conventional visual question answering benchmarks. Testing a model's ability to recover completely hidden objects from indirect physical evidence is a meaningful probe of models capability for visually grounded reasoning.

  • The direct vs inferential distinction provides an intuitive evaluation framework, and the use of manually occluded real-world photographs preserves realistic visual evidence. Although mostly detailed in the appendix, the work conducts various detailed control and robustness experiments to validate their findings.

  • Results from conducted experiments across various large-scale models and intervention settings provide compelling evidence.

Critical weaknesses
  • Some ambiguity and contradiction in stage 3 Source mapping if authors could clarify. In line 255-258, "We mark the projection with a transparent bounding box, occlude four candidate regions of the image, and ask which region casts the shadow or reflection." The text and caption in figure 4 suggests that the projection region is highlighted with a bounding box and candidate sources are occluded, and they queriy which is the source of the projection. i.e asking for the source of the reflection or shadow.
    However, the image suggests otherwise, four candidate projection regions are marked with bounding boxes, and the candidate region is queried. i.e asking for most likely region of projection given the occluded source object. These suggest different things and have different diagnostic goals.

  • The Inpainting Multi-step Execution introduces an additional model call, a generated visual hypothesis, and an explicit physical-consistency verification instruction. If the conclusion is that externalizing hypotheses visually and testing them improves performance. I think an important question this brings about is whether the visual externalization is required to bring about improvements or could be achieved through a simpler text only hypothesis and verification procedure.

  • This does not necessarily take away from the work's contribution, but the dataset is relatively small and concentrates on shadows and reflections. While the manual curation process and controls ensure its quality, the limited size and scope of this benchmark limit its generalizability, and I would be cautious about making broad conclusions about a models visual grounding capabilities.

Questions for authors
  • Could the authors further distinguish failures of visual extraction from failures of task-directed evidence selection? For example, in a simple study without the introduction of tool use, does accuracy improve when the relevant projection is visually prompted or cropped and provided as additional evidence while retaining the original identification question?
  • Clarifications on the contradiction in the stage 3 diagnosis of OccludeTrace
  • How does inpainting compare against a second reasoning pass, text-only hypothesis generation, and explicit visual verification without image generation?
4 comments
AuthorsOpenReview ↗

Response to reviewer

We thank the reviewer for their constructive feedback and positive assessment of our work. All intervals are 95% bootstrap confidence intervals resampling images, and accuracy uses the either-judge rule.

W1. Stage 3 source-mapping discrepancy

We appreciate the reviewer identifying this inconsistency. We ran two versions of Stage 3, but the text and figure described different experiments:

  • Finding the cue (Figure 4): The object is occluded, and the model identifies which of four regions contains its shadow or reflection.
  • Finding the source (Table 2): The projection remains visible, and the model identifies which of four occluded regions contains the source object.

We now report both experiments and align the figure, text, and table.

TaskImagesFailedSuccessfulSeparation
Find cue11472.6%82.0%9.5 [-0.8, 20.0]
Find source10785.4%82.3%-3.0 [-12.2, 6.2]

Neither version approaches the 41.1-point [29.6, 52.0] separation observed at Stage 4. This supports our diagnosis that interpreting physical evidence within the full scene is a major bottleneck. We also corrected Stage 4's description to match the actual prompt used with the highlighted projection.

Q1. Visual extraction versus evidence selection

We conducted additional experiments on a subset of direct images to distinguish difficulty locating relevant evidence from difficulty interpreting it.

Importantly, every condition retains the original photograph with the object fully occluded. The interventions provide additional guidance or visual evidence without revealing the hidden object.

ConditionWhat the model receivesOverall accuracyInitially failed
BaselineOriginal occluded image43.8%0.0%
Repeated answeringOriginal image asked again44.1%22.2%
Projection highlightedOriginal image + highlighted projection68.0%50.0%
Projection croppedOriginal image + separate crop of the projection81.2%77.1%

Highlighting the relevant projection improves overall accuracy by 24.2 points [17.6, 31.2]. On initially failed examples, highlighting improves accuracy by 27.8 points [18.8, 37.3] over repeated answering. This indicates that models often struggle to identify which physical evidence matters.

Providing an additional cropped view improves accuracy by another 27.1 points [16.2, 37.7] on initially failed examples. This suggests that even after locating the relevant evidence, models struggle to extract and interpret it within the full scene.

These findings directly support our diagnosis: models can often interpret physical evidence when explicitly directed to it, but struggle to independently locate and use that evidence in complex scenes. The improvements substantially exceed those from repeated answering alone.

The original occluded image is preserved throughout, although the interventions use region-specific prompts rather than identical question wording. We clarify this distinction in the revision.

Q2. Inpainting controls and attribution

We added a repeated-answering control for GPT-5.4 to test whether another attempt alone improves performance.

ConditionBaselineResult
Inpainting37.2%45.4%
Ask Again37.2%40.1%

Inpainting improves GPT-5.4 by 8.2 points [1.0, 15.3], compared with 2.5 points [-4.6, 9.6] for repeated answering. This suggests that inpainting offers benefits beyond simply giving the model another attempt.

We also clarify that previously reported inpainting results used a second-look score, preserving baseline-correct answers and applying inpainting only to baseline failures. The revision distinguishes this from standalone accuracy and removes the claim that inpainting is the most effective intervention.

W2. Scope and generalizability

We agree that 378 manually curated images cannot establish visual-grounding performance across every physical phenomenon. Our claims specifically concern recovering fully hidden objects from shadows, reflections, and scene context, where realistic visual evidence is essential.

Within this scope, the five-model direct–inferential gap is 20.1 points [14.1, 26.1]. Moreover, removing shadows and reflections from the same images reduces accuracy across all five models by 17.2–42.6 points, directly demonstrating that these physical cues materially contribute to model performance.

Our central finding remains: Models struggle disproportionately with extracting and using indirect physical evidence from full scenes, and targeted visual guidance substantially improves performance.

Sincerely,
The Authors

Dear Reviewer 1pfe,

Thank you again for your encouraging assessment and specific suggestions. We've corrected the Stage 3 mismatch and added experiments to distinguish whether models struggle to locate the relevant physical evidence or to interpret it once located. We also added an inpainting control to check the intervention more directly. Your comments helped us make both the evaluation and its interpretation clearer, and we'd be grateful if you would consider raising your score.

Sincerely,

The Authors

Thank you to the authors for the detailed rebuttal and additional experiments. I appreciate the effort to address the concerns raised in my initial review.

The clarification of the Stage 3 mapping experiments resolves the confusion I had. More importantly, the additional highlighting and cropping experiments provide stronger empirical support for the paper's central claim that VLMs struggle to select and extract relevant indirect visual evidence within complex scenes. The repeated-answering controls also help establish that these improvements cannot be attributed simply to another inference attempt.

I also appreciate the additional controls for the inpainting intervention and the authors' more cautious interpretation of these results, particularly the finding that visual inpainting does not demonstrate a clear advantage over text-only hypothesis testing.

Some limitations remain, particularly the relatively small benchmark and the fact that the evidence-selection and extraction interventions also modify the prompting procedure. Thus, I will maintain my Weak Accept rating. Nevertheless, I believe the authors have adequately addressed my principal concerns, and the additional experiments strengthen the paper's empirical contribution.

Dear Reviewer 1pfe,

Thank you for your thoughtful follow-up and for recognizing that our additional experiments address your principal concerns. We especially appreciate your constructive suggestions, which helped us strengthen the empirical evidence, clarify our diagnostic claims, and improve the paper overall.

We are grateful for your careful engagement throughout the rebuttal period and for your continued support of our work.

Sincerely,

The Authors

Comments on the paper

1 comment

We thank the reviewers for their thoughtful feedback. We summarize the central finding and the new evidence added during rebuttal. OccludeBench tests whether vision-language models can identify fully hidden objects from shadows and reflections. Models struggle disproportionately with this indirect physical inference relative to contextual inference, unlike humans; locating and interpreting the relevant visual evidence are key bottlenecks.

Our rebuttal adds five checks:

  • Controlled comparisons: the inference gap persists after accounting for cue size, clutter, and answer granularity.
  • Cue removal: deleting shadows and reflections from the same images sharply reduces model accuracy.
  • Human comparison: the model gap exceeds the corresponding human gap on matched images.
  • Failure analysis: new experiments identify difficulties both locating and interpreting physical cues.
  • Additional controls: inpainting tests, corrected Stage 3 mapping, consistent grading, and more precise claims strengthen the evaluation.

Together, these analyses establish a specific, actionable limitation: models struggle to locate and interpret indirect physical evidence, and targeted visual guidance substantially improves performance.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.