- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Vision-language models can describe what an image depicts yet misinterpret what it means. This paper asks: once a VLM has accessed the relevant visual evidence, what happens between what is seen and what is meant? To study this transition, this paper examines four components: visual-evidence grounding, source-relation selection, target-domain mapping, and answer binding. Experimental results show that the key bottleneck in image-meaning inference is whether it can form and use the implication supported by the right relation among visible elements.
Critical strengths
Figures and Tables are aesthetically well-presented.
Critical weaknesses
-
Every causal claim in the paper rests on the same 35 examples, and the selection of those examples is never described.
-
Section 4.1 needs to report the absolute mean-fill accuracies and add a black-image and a no-image condition so that the image-necessity bound is explicit.
-
In Figure 4(a,b), the condition that adds the answer options appears to help more than the condition that adds the image. This is an interesting and slightly awkward finding: if answer options constrain relation choice more effectively than the image does, then the full-context condition may be measuring option-driven reasoning rather than image-driven relation selection, which is close to the opposite of the paper's framing. The paper does not comment on this ordering anywhere. There needs to be a top-1 rate for each context condition numerically in the main text, and say why the options are so informative.
Questions for authors
Q1: What are the selection criteria/filtering procedure for the 35 examples?
Q2: What are the absolute mean-fill accuracies in Section 4.1, and how does the performance compare under the black-image versus the image-free conditions?
Please refer to the "Critical Weaknesses" section for further details.
5 comments
Q2
Section 4.1 needs to report the absolute mean-fill accuracies and add a black-image and a no-image condition so that the image-necessity bound is explicit. What were the absolute mean-fill accuracies, and how does performance compare under the black-image versus no-image conditions?
A2
Following your suggestion, we evaluated the absolute accuracies across 5 input conditions on the exact same evaluation splits used in Table 1, Table 2, and Table S4 (the shared 1,371-example evaluation set on II-Bench and the 765-example test split on the external CII-Bench, under the fixed-order evaluation protocol of Table 2 and Tables S13 to S15, while the 62.4% and 67.9% reported in Section 1 are free-form generation accuracies across II-Bench examples before excluding the 28 unresolved audit records). Overall, the black-image and no-image conditions perform similarly and remain well above random chance due to language priors in multiple-choice questions, while adding the original image brings clear and consistent accuracy gains over all visual-ablation baselines.
- Original image, full multimodal input (upper baseline, matching the identity-order baselines in Table 2 and Tables S13 to S15)
- Mean-fill, replacing pixels with average color values (removes shapes but keeps overall color)
- Black image (zero-fill), setting all image pixels to zero (keeps image token placeholders without visual content)
- No image (text-only), removing the image completely (pure text baseline or blind guessing)
- Random chance, 6-choice baseline (16.67%)
The complete absolute accuracy results and paired drops are summarized below.
| Benchmark | Model | Original | Mean-Fill | (Mean-Fill minus Orig.) | Black-Image | No-Image (Text-Only) | Random Chance |
|---|---|---|---|---|---|---|---|
| II-Bench () | Qwen3-VL-2B | 63.31% | 46.24% | −17.07 pp | 46.68% | 47.34% | 16.67% |
| II-Bench () | InternVL3.5-2B | 68.20% | 54.92% | −13.27 pp | 55.00% | 53.32% | 16.67% |
| II-Bench () | Qwen3-VL-4B | 75.86% | 56.60% | −19.26 pp | 57.70% | 58.72% | 16.67% |
| II-Bench () | InternVL3.5-4B | 75.20% | 63.31% | −11.89 pp | 64.77% | 61.12% | 16.67% |
| CII-Bench () | Qwen3-VL-2B | 38.69% | 26.27% | −12.42 pp | 27.06% | 28.50% | 16.67% |
| CII-Bench () | InternVL3.5-2B | 46.54% | 34.77% | −11.76 pp | 35.16% | 33.33% | 16.67% |
Analysis and Findings
- Textual priors (language shortcuts)
Without images or with black images, models reach 46.68% to 64.77% accuracy on II-Bench and 27.06% to 35.16% on CII-Bench. Both ranges are much higher than random chance (16.67%). This confirms that multiple-choice questions provide common world knowledge and answer clues. Models can rule out bad options using language priors alone. - Visual necessity
Across all tested models, adding the original image brings a clear increase of 11.76 to 19.26 percentage points over mean-fill, 10.43 to 18.16 percentage points over black-image, and 10.19 to 17.14 percentage points over the text-only baseline. The mean-fill differences exactly match the paired drops reported in Table 1 and Table S4 (all adjusted values below 0.001). Existing vision-language models can clearly recognize basic objects and use this visual information to remove incorrect options.
We will add these complete absolute accuracies to Section 4.1 and Appendix D.1 to enrich the paper.
Q3
In Figure 4(a,b), the condition that adds the answer options appears to help more than the condition that adds the image. This is an interesting and slightly awkward finding. If answer options constrain relation choice more effectively than the image does, then the full-context condition may be measuring option-driven reasoning rather than image-driven relation selection, which is close to the opposite of the paper's framing. The paper does not comment on this ordering anywhere. There needs to be a top-1 rate for each context condition numerically in the main text, and say why the options are so informative.
A3
Thank you for this helpful question. In short, answer options provide a top-down semantic filter to rule out unrelated local relations, while adding the image triggers strong visual competition among multiple true relations in the picture, and this visual interference directly lowers accuracy in the full context, showing that the process is not purely option-driven. We first report the exact Top-1 selection rates for the gold relation across the five context conditions.
- Qwen3-VL-2B, relations only R (42.1%) to question plus relations Q+R (44.6%) to question plus image plus relations Q+I+R (34.3%, sharp drop) to question plus options plus relations Q+O+R (55.4%, sharp rise) to full context Q+I+O+R (46.3%)
- InternVL3.5-2B, relations only R (42.0%) to question plus relations Q+R (45.1%) to question plus image plus relations Q+I+R (24.9%, sharp drop) to question plus options plus relations Q+O+R (50.3%, sharp rise) to full context Q+I+O+R (34.9%)
The main reasons for this ordering are as follows.
- Why does adding the image Q+I+R decrease accuracy compared to Q+R?
In our design, the 4 competitor relations are visually true same-image competitors that describe real local details in the picture (Table S16). Therefore, the model cannot simply rely on object recognition to drop bad choices.
- Under the text-only condition (Q+R), these small local details look less related to the main question. Language patterns therefore make it easier to pick the broader gold relation.
- When we add the image (Q+I+R), the vision encoder creates strong visual signals for these visible objects. The model faces competition between multiple visually true details in the picture. It gets distracted by secondary elements. This causes the Top-1 selection rate to fall by 10.3 to 20.2 percentage points.
- Why do the answer options O provide so much information? (Q+O+R is the highest)
Image implication inference connects a concrete source-domain relation R to a target implication expressed in the answer options O.
- The options describe high-level target implications, such as social or psychological ideas.
- Giving options O works as an effective top-down filter. By comparing candidate relations with target options, the model easily removes minor local details that do not support the intended meaning.
- Why this does not mean the process is purely option-driven
If relation selection depended only on options and ignored the image, the full context (Q+I+O+R) would perform as well as or better than Q+O+R.
However, we see the opposite result. Adding the image back into the full context causes accuracy to fall by 9.1 percentage points in Qwen (from 55.4% to 46.3%) and by 15.4 percentage points in InternVL (from 50.3% to 34.9%).
This provides clear evidence. Visual interference from multiple visually true same-image relations is very strong. It directly weakens the top-down guidance provided by the text options. We will add these exact numbers and explanations to Section 5.1 of the revised manuscript.
Dear Reviewer ffpB:
We thank the reviewer for the thoughtful and rigorous feedback.
Q1
Every causal claim in the paper rests on the same 35 examples, and the selection of those examples is never described. What were the selection criteria or filtering procedure for these 35 examples?
A1
Thank you for pointing this out. The 35 examples strictly follow the official development split provided by II-Bench. We adopted this predefined split directly. This ensures an objective and standard test setting. At the same time, we combined fine-grained interventions on this development set with large-scale validation on the full test set to ensure that our findings are reliable. We will modify Section 3.1 of the manuscript to make these details clearer.
The reasons why we used this development set for detailed intervention experiments are as follows. First, we separated exploration from validation in our research design. We wanted to carry out detailed analyses and draw conclusions on the development set. Then we tested and verified those conclusions on the complete test set. This helped avoid data leakage on the test set. Also, the controlled tests in Sections 4 through 7 required expensive human annotations. We could not easily apply these across the entire benchmark. These annotations include
To make the findings of the paper more reliable, we also did the following work
We intend to do our best to verify these findings on a larger set and include them in the final manuscript.