acceptodds
Under review as a conference paper at ICLR 2027

Composition, Not Conversation: VLMs Lose the Scene, Not the Thread

Abstract

Vision-language models (VLMs) increasingly reason over visual evidence that is cropped, segmented, retrieved, or revealed over time. Yet most VQA benchmarks present the complete image and question at once. We ask what models lose when the same information is fragmented. We introduce Layered-VQA, with 93 scenes and 300 questions. Each image is decomposed into ordered RGBA layers that exactly recompose the original scene, and each question is annotated with supporting, minimal-sufficient, and distractor layers. We evaluate eleven open-weight VLMs from 3B to 32B parameters and two proprietary models with a scale of 187,200 conversations, graded by 1.74M open-model cross-judgments. We find three consistent failures. Loss in Composition: fragmenting the question has a small effect, but fragmenting the scene substantially reduces accuracy; recomposing the same layers largely restores performance. Oracle Inversion: even oracle-selected sufficient evidence can perform worse than the complete scene. Loss in Grounding: as more evidence is required, grounding degrades much faster than answer accuracy. Together, these results show that having the right visual evidence is not enough. How that evidence is composed and presented determines whether models can use and ground it. The right evidence is not enough: VLMs need the scene it came from.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, visual question answering, visual grounding, multi-turn conversation, benchmark, evaluation, layered image decomposition

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mdzyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Layered-VQA to investigate how visual composition and multi-turn delivery affect VLM performance. Experiments show that fragmenting visual scenes substantially reduces accuracy, while recomposition largely restores it. The authors further find that oracle-selected evidence can underperform complete scenes, and that correct answers do not necessarily indicate reliable visual grounding.

Critical strengths
  1. The paper offers a novel perspective by examining how visual evidence is composed and presented, rather than simply whether it is available. This provides useful insights into VLMs' limitations in integrating fragmented visual information.
Critical weaknesses
  1. The paper mainly attributes the performance degradation caused by visual fragmentation to the loss of scene composition. However, splitting a complete image into multiple layers also increases the number of visual inputs and may introduce many redundant visual tokens. Without controlling the visual token budget, it is difficult to determine whether the degradation primarily results from the loss of scene composition or the increased complexity of visual inputs.
Questions for authors
  1. Could the performance drop under visual fragmentation be partly attributed to the increased number of visual tokens and the dilution of relevant visual evidence, rather than the loss of scene composition itself?
2 comments
AuthorsOpenReview ↗

Response to Reviewer mdzy

Response to W1/Q1

We thank Reviewer mdzy for the review. We are glad the reviewer finds our perspective novel. We respond below. We show two new conditions, FULL-PAD (full image, padded) and CONCAT-V-TM (concatenated visual layers, token-matched), which separate token budget from decomposition.

  • Visual tokens per question: FULL 411, ORACLE 954, CONCAT-V 1,619, ALL-LAYERS 2,030.
  • ALL-LAYERS has 25% more tokens than CONCAT-V, yet it loses only −.017, against CONCAT-V's −.187. ORACLE has 41% fewer tokens than CONCAT-V and gains only +.038. So accuracy does not simply follow token count. In ALL-LAYERS, though, a model could just read the composite, so we tested token budget directly:
ConditionImagesTokensAcc.Δ vs FULL
FULL1411.604—
FULL-PAD: full image + blank images3.951,619.556−.036 [−.049, −.022]
CONCAT-V3.951,619.417−.187 [−.224, −.149]
CONCAT-V-TM: layers downscaled to FULL's budget3.95408.355−.237 [−.274, −.199]
  • More tokens, scene kept whole (FULL-PAD): the same images and tokens as CONCAT-V cost only −.036, a fifth of the loss. Blank images are easy to ignore, so this is a lower bound on dilution.
  • Fewer tokens, scene split (CONCAT-V-TM): at FULL's budget, accuracy does not recover, it drops further (−.237). Downscaling also lowers resolution, so this is not a pure token test. So dilution is real but small.Most directly, the same minimal evidence loses −.137 when separated (ORACLE) rather than composed (ORACLE-COMPOSED), with no distractors added (see our response to Reviewer VFdd, Part 2 of 3). A dilution account would predict a large loss for FULL-PAD and a recovery for CONCAT-V-TM, neither happens.
  • Relevant evidence: if dilution drove the loss, single-object and relational questions would suffer alike. They don't: CONCAT-V costs −.126 on single-object questions and −.259 on relational ones, while under FULL-PAD the two are close (−.030 vs −.043).

So yes, the loss follows the decomposition, not the token count. Thank you again, this control makes the paper's central claim stronger.

We hope these new experiments - "FULL-PAD, CONCAT-V-TM, ORACLE-COMPOSED" address the reviewer's concerns and would be grateful if the reviewer would reconsider their score in light of these results. We are happy to answer any further questions.

Reviewer_mdzyOpenReview ↗

Reply

Thank you for your response. I have no further questions about this paper. Overall, I have decided to maintain my current score.

Reviewer_7s9ZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Layered-VQA, a benchmark consisting of 300 questions from 93 scenes, to study how fragmenting visual evidence affects VLM performance. The authors evaluated 13 VLMs under 16 conditions that vary visual composition, multi-turn delivery and evidence selection. Their main findings include 1) fragmenting the scene into layers reduces substantially accuracy even all visual evidence is preserved; 2) even providing only oracle-selected evidence underperforms providing the complete scene; 3) evidence grounding degrades more significantly than answer accuracy.

Critical strengths
  • The problem is interesting and relevant. Understanding how VLMs handle fragmented visual evidence is important, particularly for agentic visual reasoning where images may be cropped, segmented, or manipulated across multiple steps.
  • Systematic experiment design and thorough evaluation. By evaluating the same questions and scenes under different presentation conditions, the paper provides useful comparisons of visual fragmentation, conversational delivery, and evidence selection. The evaluation is relatively thorough, covering a diverse set of VLMs, repeated runs, scene-level confidence intervals, and human validation of the LLM-based judging protocol.
  • The breakdown by question type is informative, particularly the finding that relational questions are substantially more affected by visual fragmentation than single-object questions.
Critical weaknesses
  • My main concern is that the paper demonstrates substantial performance degradation under visual fragmentation but does not convincingly establish why it occurs. In particular, it is unclear whether the gap reflects limitations in visual composition, distribution shift, or missing contextual information.
  • While the paper motivates its benchmark using agentic systems that manipulate visual evidence, the RGBA-layer decomposition differs from common operations such as cropping, zooming, or masking. For example, cropping often preserves local context and increases object resolution, whereas the benchmark presents isolated objects on separate canvases. It is unclear whether the observed degradation generalizes to these more realistic settings.
  • Self-reported evidence can differ from actual grounding. The grounding evaluation relies on models explicitly reporting the layer IDs supporting their answers. However, incorrect layer citations do not necessarily imply that the model failed to use the correct visual evidence. The authors should consider doing causal interventions to establish the grounding claims.
  • Since annotators have access to the complete scene and reference answer, it would be useful to verify whether a separate human annotator can reliably answer the questions using only the selected layers, without seeing the original scene. This is to make sure that the question is solvable with the oracle-selected evidence.
  • Limited benchmark scale and diversity. The benchmark contains only 300 questions from 93 scenes, with particularly few examples in some categories (e.g., 17 occlusion questions). A larger and more diverse benchmark would strengthen the generalizability of the findings.
Questions for authors
  • What explains the performance degradation under visual fragmentation? How can we distinguish limitations in visual composition from distribution shift or missing contextual information? Could fine-tuning on layered images reduce this gap?
  • Gemini-3.7-Flash experiences only a 1.2-point drop under CONCAT-V, compared with 5.1 points for GPT-5.6-Luna and 18.7 points on average for open-weight models. What explains these differences?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 7s9Z (1/2)

Part 1 of 2 - Response to W1/Q1 (why), W2 (crops)

We thank Reviewer 7s9Z for the review. We are glad the reviewer finds the problem relevant and the evaluation thorough. We respond to each point below. We add a few new experiments or conditions, CONCAT-V-BG (concatenated visual layers, on background), ORACLE-COMPOSED (oracle layers, composed), FULL-PAD (full image, padded), CONCAT-V-TM (concatenated visual layers, token-matched), and CROPS (object crops).

W1/Q1. Why does fragmentation hurt? We tested each explanation directly.

  • Distribution shift. With every image a natural photo (CONCAT-V-BG), models still lose −.146 of the −.187. Unfamiliar renders explain only about a quarter of the drop.
  • Missing context. CONCAT-V keeps every pixel of the scene and still loses −.187. The same evidence does much better composed into one image (ORACLE-COMPOSED, .576) than separated (ORACLE, .455). So, what models lose is the composed scene, not information.
  • Image count / tokens. Extra images and tokens cost only −.036 (FULL-PAD). Matching FULL's token budget makes things worse, not better (CONCAT-V-TM, −.237).

Conclusion. Once these factors are controlled, what remains is composition, as we argue. The loss doubles on relational questions (−.259 vs −.126).

Fine-tuning. We have not fine-tuned, so we do not claim it, but the gap does look learnable. Retention rises with capability (ρ = .79 across 13 models), and Gemini-3.7-Flash nearly closes it. We list fine-tuning as future work.

W2. Does it transfer to crop-and-zoom? We added CROPS. Each object is cropped from the composite with a 20% context margin and zoomed up to 4×, as an agentic zoom tool would return it. As the reviewer notes, crops keep local context and raise resolution, but they lose global position. CROPS reaches .523, a loss of −.069 [−.095, −.043] against FULL, about a third of CONCAT-V's loss. Zoom tools that keep the original image are like ALL-LAYERS, which loses only −.017. Layer decomposition is itself the masking case, as each layer is an object mask on a blank canvas.

So yes, the degradation generalizes to crop-and-zoom (−.069 vs −.187) and be generalized to more realistic settings as well. It is largest when the crops or masks replace the full view, and almost disappears when the original image is kept (−.017).

AuthorsOpenReview ↗

Response to Reviewer 7s9Z (2/2)

Part 2 of 2 - Response to W3, W4, W5, Q2

We ran two new analyses, a condition that removes the gold evidence from each scene (GOLD-REMOVED), and a human study with oracle layers alone. We also analysed the proprietary models by question type.

W3. Self-reported citations vs. actual grounding. The two are indeed different, which is why the paper measures grounding in more than one way.

  • Attribution. Citation F1 is almost the same for correct (.498) and wrong (.479) answers, so a wrong citation is a provenance failure.
  • Temporal grounding. This measure needs no self-report. An answer given before its gold layer arrives cannot have come from that layer.
  • Condition (new). Removing the gold objects from the scene (GOLD-REMOVED) leaves only 13.8% of FULL-correct answers standing, close to the 11.0% with no image.

Together, on the full scene, models do use the right evidence, but often fail to cite it. This is exactly why we report answering and grounding separately.

W4. Can people answer from the oracle layers alone? Yes. Three annotators who never wrote or validated a question answered all 300 from the oracle layers alone, before seeing the full scene. They scored .967, against .990 on the full scene, and chose "cannot answer" only 0.9% of the time (agreement 96.0%). The oracle evidence is sufficient, so the models' drop is a model failure, not missing information.

W5. Scale. We acknowledge that 300 questions limit per-category conclusions. However, our main claims rest on within-question contrasts, as each question serves as its own control across conditions, models and runs. All of them have scene-level CIs that exclude zero, CONCAT-V −.187 [−.224, −.149], ORACLE −.148 [−.185, −.115], and relational extra loss −.133 [−.189, −.074]. Therefore, the proposed benchmark takes a first step in the evaluation of visual evidence composition of foundation models. We will list the scaling of the benchmark as future work.

Q2. Why does Gemini-3.7-Flash lose only 1.2 points? (Proprietary results are self-judged, as in the paper.)

  • Same pattern, smaller size. Gemini's single-object accuracy is unchanged (.867 to .870), while relational questions drop (.856 to .826), mostly counting (.807 to .716). GPT-5.6-Luna loses −.051, mainly on spatial (−.160) and occlusion (−.157) questions.
  • Composition improves with capability. Retention under CONCAT-V correlates with FULL accuracy across all 13 models (ρ = .79, p = .0015), and within Qwen2.5-VL it rises from 61% (3B/7B) to 76% (32B). Frontier models seem to have largely learned this ability.
  • Grounding is a separate ability. Retention is uncorrelated with citation F1 (ρ = .34, p = .25). Gemini nearly solves composition, yet it shows the largest gap between scored and grounded accuracy (.835 vs .635). It often answers before the evidence arrives.

Thank you again, the reviewer's suggestions made the paper's argument considerably sharper.

We hope these new experiments - "CONCAT-V-BG, ORACLE-COMPOSED, FULL-PAD, CONCAT-V-TM, CROPS, GOLD-REMOVED" and the human study address the reviewer's concerns and would be grateful if the reviewer would reconsider their score in light of these results. We are happy to answer any further questions.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts, particularly the new controlled conditions and human validation, which address several of my concerns. However, I still have reservations about the significance and generalizability of the findings. The CROPS experiment suggests substantially smaller degradation in a more realistic setting (−6.9 vs. −18.7 points), while preserving the original image nearly eliminates the gap. Overall, I maintain my original score.

Reviewer_VFddOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how visual composition and conversational delivery affect VLM performance. It introduces Layered-VQA, containing 93 layered scenes and 300 manually authored questions with supporting, minimal-sufficient, and distractor evidence annotations. Sixteen conditions vary question fragmentation, visual fragmentation, delivery order, alignment, and evidence availability. The evaluation covers eleven open-weight and two proprietary models. For the open-model average, presenting all layers separately reduces accuracy from 0.604 to 0.417, while a final composed-image recap raises sharded-visual accuracy from 0.316 to 0.659. The paper additionally reports an oracle-evidence gap and substantial discrepancies between answer accuracy and layer-citation correctness.

Critical strengths
  1. The benchmark holds source scenes and questions fixed while varying multiple aspects of presentation. Comparing FULL, CONCAT-V, SHARDED-V, SNOWBALL-V, and RECAP-V provides substantially more diagnostic information than a single complete-image benchmark. The large degradation under separated layers and recovery after recomposition are practically relevant observations for multimodal pipelines.
  2. The paper reports scene-clustered uncertainty, analyzes question types and scene depth, and validates consensus answer grading against 400 human-labeled conversations, obtaining 96.0% agreement. The explicit evidence annotations also enable useful analyses of citation completeness, distractor selection, and evidence timing.
Critical weaknesses
  1. The controls do not sufficiently isolate composition from input format and evidence selection. Replacing one natural composite image with multiple isolated layer images simultaneously changes image count, visual tokenization, background appearance, and the distribution of foreground content. The ALL-LAYERS control does not rule out these explanations: when the complete image remains available, a model can answer from that image while ignoring the unfamiliar layer representations. Therefore, its near-baseline performance does not establish that layer-format distribution shift is irrelevant.
  2. The oracle interpretation is particularly underdetermined.The oracle interpretation is particularly underdetermined. The open-model averages are 0.604 for FULL, 0.417 for all separated layers, and 0.455 for ORACLE. Thus, oracle selection actually improves accuracy by 3.8 percentage points within the separated-layer format. Comparing ORACLE only with FULL conflates evidence selection with visual presentation. A factorial comparison crossing all versus minimal evidence with composed versus separated presentation is needed to distinguish these effects.
  3. In ALL-LAYERS, models can answer using the full composite image while failing to map evidence to layer IDs. Low citation accuracy proves a provenance failure, but not that the answer lacked visual support.
Questions for authors
  1. Is FULL the original source image or the recomposition of the selected layers? Please report reconstruction discrepancies and explain how completed occluded regions, alpha rendering, image resolution, and visual-token allocation are handled across conditions.
  2. How accurately can independent humans answer using only the oracle layers, without first seeing the full scene or reference answer? Are all valid minimal-sufficient sets accepted? Please formally define grounded accuracy and clarify how early correct guesses followed by evidence-supported final answers are scored.
4 comments
AuthorsOpenReview ↗

Response to Reviewer VFdd (1/3)

Part 1 of 3: Response to W1 (isolating composition)

We thank Reviewer VFdd for the review. We are glad the reviewer finds our controlled design diagnostic and practically relevant. We respond to each point below.

W1. Composition vs. image count, tokenization, background appearance and foreground distribution. As the reviewer notes, ALL-LAYERS cannot rule out a layer-format effect on its own, because the composite lets a model bypass the layers. So, we added a few more conditions that each vary one of the listed factors: FULL-PAD (full image, padded), CONCAT-V-TM (concatenated visual layers, token-matched), and CONCAT-V-BG (concatenated visual layers, on background).

ConditionImagesVisual tokens*Acc.Δ vs FULL
FULL1411.604—
FULL-PAD3.951,619.556−.036 [−.049, −.022]
CONCAT-V3.951,619.417−.187 [−.224, −.149]
CONCAT-V-TM3.95408.355−.237 [−.274, −.199]
CONCAT-V-BG3.951,619.446−.146 [−.177, −.116]
ALL-LAYERS4.952,030.587−.017 [−.031, −.004]

*Visual tokens per question.

  • Image count or tokenization: FULL-PAD matches CONCAT-V's images and tokens but keeps the scene whole, and loses only −.036, CONCAT-V-TM matches FULL's budget and keeps the decomposition, and it does not recover anything (−.237). So extra images and tokens explain little of the drop.
  • Background appearance: CONCAT-V-BG, with every image a natural photo, still loses −.146, so unfamiliar renders explain at most a quarter of it.
  • Foreground distribution: the object pixels are the same in every layer condition (CONCAT-V, -TM and -BG), so differences in foreground content can't explain the gaps between them. Even ORACLE-COMPOSED, the gold objects alone on a blank canvas, loses only −.016 vs FULL.

So, with the input format held fixed, the loss follows the decomposition. The original controls did not fully isolate these factors, the new controls above measure this directly. They show that format accounts for only a small part of the drop, and most of it comes from splitting the scene itself.

AuthorsOpenReview ↗

Response to Reviewer VFdd (2/3)

Part 2 of 3: Response to W2 (oracle factorial), W3 (provenance)

W2. The oracle interpretation. The selection gain is significant (ORACLE − CONCAT-V = +.038 [+.021, +.053]), and comparing ORACLE only with FULL conflates evidence selection with presentation. We ran the factorial the reviewer suggests by adding ORACLE-COMPOSED, the same gold pixels alpha-composited in place into one image:

All evidenceMinimal evidenceΔ selection
Composed (one image).604 (FULL).576 (ORACLE-COMPOSED)−.016 [−.039, +.005]
Separated (layers).417 (CONCAT-V).455 (ORACLE)+.038 [+.021, +.053]
Δ separation−.187 [−.224, −.149]−.137 [−.165, −.110]interaction +.054 [+.027, +.083]

Once composed, the minimal evidence does better than when separated, and it matches but never beats FULL. Removing only the distractors (GOLD-IN-SCENE) also changes little (+.009). So the oracle gap comes from separating the evidence, i.e., from the loss in composition.

W3. Citation failure is not missing visual support. The two are indeed different.

  1. Attribution: citation F1 is almost the same for correct (.498) and wrong (.479) answers. A wrong citation is therefore a provenance failure, as the reviewer notes, and we will label it evidence attribution.
  2. Temporal grounding: this measure needs no self-report. An answer given before its gold layer arrives cannot have come from that layer.
  3. Counterfactual test (new): we removed the gold objects from the scene (GOLD-REMOVED). Only 13.8% of FULL-correct answers survive, close to the 11.0% with no image. So models do rely on the gold evidence.

Altogether: on the full scene, models use the right evidence to answer but often fail to cite it. This is exactly why the paper measures answering and grounding separately, and we will reword the sentence "A correct answer does not mean the model used the right evidence" accordingly.

AuthorsOpenReview ↗

Response to Reviewer VFdd (3/3)

Part 3 of 3: Response to Q1 (what FULL is), Q2 (humans, definitions)

Q1. FULL is the recomposition: FULL is the source image, i.e., the composite of all layers. Recompositing the layers reproduces FULL exactly (maximum pixel difference 0 in all 93 scenes), so there are no discrepancies. A few analyses:

  • Occlusion: the completed occluded parts are hidden in FULL but visible in the layers, so layers carry slightly more information. Only 4.2% of a gold layer is hidden in FULL on average (median 0.7%). The CONCAT-V loss is about the same whether <1%, 1–5% or >5% is hidden (−.168 / −.213 / −.200).
  • Alpha rendering and resolution: each layer is composited onto a white canvas the size of the full image, so positions are preserved. ORACLE and CONCAT-V use identical renders, we never resize, and we retained the native resolution. Each model's processor applies the same policy in every condition.
  • Tokens per question: FULL 411, ORACLE 954, CONCAT-V 1,619, ALL-LAYERS 2,030 (see the table in our response to W1).

Q2. Human answerability and definitions:

  • Three annotators who never wrote or validated a question answered all 300 from the oracle layers alone, before seeing the full scene or the reference. They scored .967, against .990 on the full scene, and chose "cannot answer" for only 0.9% (agreement 96.0%).
  • Minimal sets: yes, every set verified as sufficient and minimal is accepted. One question has two valid sets, a citation matching either counts as correct.
  • Grounded accuracy: Grounded(c)=Acc(c)⋅1[t∗≥τG]\mathrm{Grounded}(c) = \mathrm{Acc}(c)\cdot\mathbf{1}[t^{\ast} \geq \tau_G], where t∗t^{\ast} is the first correct answer attempt and τG\tau_G is the turn by which all gold layers have arrived (the gold set is the minimal set).
  • Early guesses: a correct answer given before the evidence arrives counts as ungrounded, even if it is repeated later, since it can't be told apart from an unrevised prior. Of 20,758 correct conversations, 6.9% abandon an early guess and re-derive it after the evidence, counting those as grounded lowers the ungrounded share from 29.2% to 22.3%.

We hope these new experiments - "FULL-PAD, CONCAT-V-TM, CONCAT-V-BG, ORACLE-COMPOSED" address the reviewer's concerns and would be grateful if the reviewer would reconsider their score in light of these results. We are happy to answer any further questions.

AuthorsOpenReview ↗

Response for Reviewer VFdd

Dear Reviewer,

Thank you very much for your valuable feedback and constructive comments. They have truly helped improve our work! As the discussion phase comes to a close on November 18, we wanted to take this opportunity to follow up and ask whether our responses have adequately addressed all of your questions and concerns. If you're satisfied with our responses, we would be grateful if you’d consider updating your score. Thank you once again for your time, feedback, and support throughout the review process.

Best regards,
The Authors of Submission #30613

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers, ACs, SACs, and PCs

We thank all reviewers, ACs, SACs, and PCs for their careful review. We appreciate the reviewers finding our perspective novel (mdzy), the problem relevant for agentic visual reasoning (7s9Z), the design controlled and diagnostic (VFdd, 7s9Z), and the evaluation thorough, with repeated runs, scene-level confidence intervals and human validation of the judging (VFdd, 7s9Z).

Our main finding is that VLMs lose accuracy when a scene is split into parts (FULL .604 to CONCAT-V .417 over 11 open models), even when every pixel is kept. To test why, we added eight new conditions and a human study, all on the same 300 questions and scene layers. Numbers are the change in accuracy against FULL, unless stated.

(1) Composition or input format? (VFdd, 7s9Z, mdzy)

  • Keeping the scene whole with CONCAT-V's images and tokens (FULL-PAD) costs only −.036.
  • Giving the layers FULL's token budget (CONCAT-V-TM) does not recover accuracy (−.237).
  • Showing each object on the real background, so every image is a natural photo (CONCAT-V-BG), still loses −.146 of the −.187.
  • The loss is also more than twice as large on relational questions as on single-object ones (−.259 vs −.126).
  • So the number of images, tokens, or layer appearance is not the main cause.

(2) Oracle interpretation (VFdd, 7s9Z)

  • The minimal evidence composed into one image (ORACLE-COMPOSED) matches FULL (−.016), while separated it loses −.137.
  • Removing only the distractor objects (GOLD-IN-SCENE) also changes little (+.009).
  • So the oracle gap is part of the same composition loss, which explains why even perfectly selected evidence trails the whole scene.

(3) Realistic settings (7s9Z)

  • Zoomed crops around each object, as an agentic zoom tool returns (CROPS), also degrade every model we tested (−.069, up to −.112 per model).
  • The loss grows as fewer objects are shown together, from −.017 when the whole scene is kept (ALL-LAYERS) to −.187 when each object is isolated (CONCAT-V).

(4) Evidence use (VFdd, 7s9Z)

  • On the full scene, removing the gold objects (GOLD-REMOVED) leaves only 13.8% of correct answers standing, close to the 11.0% with no image (BLIND). So there, models do rely on the evidence.
  • Citation F1 is almost the same for correct (.498) and wrong (.479) answers, so a wrong citation is a provenance failure, not missing visual support.
  • Under sharded delivery, as the paper reports, many correct answers still come before the evidence arrives, so answering and grounding remain separate abilities.

(5) Human answerability and definitions (VFdd, 7s9Z)

  • Three independent annotators answer .967 of the questions from the oracle layers alone, against .990 from the full scene. So the oracle evidence is sufficient.
  • FULL is the composite of all layers, reproduced exactly in all 93 scenes, and we give formal definitions of accuracy, joint accuracy and grounded accuracy.

Together, these results show that the drop comes mainly from losing the scene's composition, not from the input format.

As the discussion period closes soon, we would be grateful if the reviewers and ACs could let us know whether our responses address their concerns. We are happy to answer any further questions.

AuthorsOpenReview ↗

Follow-Up on Author Responses

Dear Reviewers,

We really thank you again for your detailed reviews of our paper. We know this is a busy period. We have posted detailed responses addressing each of your comments and the requested experiments, and we would be grateful if you could please take a look when convenient. If you feel your concerns have been addressed, we would sincerely appreciate your considering an updated evaluation. If anything remains unconvincing, we would be glad to discuss it together; we are following this thread closely and will respond promptly. We would greatly welcome any additional feedback you might have.

Best regards,
The Authors of Submission #30613

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.