acceptodds
Under review as a conference paper at ICLR 2027

SEEING IS NOT RELATING: LEARNING RELATIONAL REPRESENTATIONS FOR MULTI-IMAGE UNDERSTANDING

Abstract

Multi-image understanding requires vision–language models to connect visual information across images. Controlled diagnostics across four model families show that models can correctly recognize task-relevant local details, including attributes and within-image relations, yet still fail at cross-image comparison. Complementary information interventions show that providing the correct relation target substantially improves task performance, highlighting the importance of reliable cross-image relations. Motivated by these findings, we construct structured supervision covering entities, attributes, within-image predicates, and cross-image comparisons to learn relational features that complement pretrained visual content. We propose Relation-Aware Dual-Stream Fusion (RDSF), which learns a dedicated relation stream, exchanges context across images, and integrates the resulting features with native visual tokens through token-aligned residual fusion. The pretrained backbone remains frozen throughout training. Across four Qwen3.5 scales, RDSF improves the six-benchmark mean by 1.99–6.51 percentage points. Additional evaluations on InternVL3.5 and LLaVA-OV1.5 show gains in the four-benchmark mean, supporting the applicability of complementary relational learning across model architectures.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-image understanding, Relational representation learning, Vision-language models, Structured supervision, Multimodal fusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EenbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates failures modes in cross image relational reasoning despite having correct local visual recognition. Through controlled diagnostics across four VLM families, the authors show that identifying cross image relations remains challenging task even when local attributes and predicates are correctly recognized.

To address this, they propose Relation-Aware Dual-Stream Fusion (RDSF), which learns structured relational features from GQA / Visual Genome annotations, exchanges cross-image context through mean pooled representations, and integrates them into frozen VLMs using token aligned residual fusion. Experiments across six benchmarks shows improvements of 1.99 to 6.51 percentage points across four Qwen3.5 scales, with additional gains on InternVL3.5 and LLaVA OV1.5.

Critical strengths

[S1]. Meaningful diagnostic. The paper separates local perception from cross image relation identification through controlled information interventions and independent semantic checks. The analysis across multiple VLM families provides useful evidence that recognizing local visual facts does not guarantee successful relational comparison.

[S2]. Practical architecture with consistent improvements. RDSF improves performance across multiple backbones while keeping pretrained parameters frozen and preserving visual token sequence length. The gains are substantial on some benchmarks (e.g., +14.14 pp on MIRB for Qwen3.5 4B), and the ablations provide useful evidence for cross image context exchange and token aligned fusion.

Critical weaknesses

[W1]. Insufficient attribution to structured relational learning. The central claim is not convincingly isolated from generic feature augmentation. On Qwen3.5 9B, randomly initialized relation mapping already achieves 73.48% versus 74.50% for RDSF (Table 3), while the matched initialization comparison shows only a +1.04 pp benefit from structural pretraining (Table 16). Moreover, eq. (5) exchanges globally mean-pooled features rather than explicitly modeling entity level correspondences. Stronger matched capacity controls are needed to establish that relational supervision is responsible for the improvements.

[W2]. Evaluation and robustness limitations. The criteria for selecting backbone specific pretraining checkpoints and inference residual scales are unclear. The LoRA comparison is not end-to-end compute matched, benchmark level results lack multi-seed variance, and robustness to image ordering and distractors is not systematically evaluated. Furthermore, RDSF is not evaluated on the original 2,475-example diagnostic suite, limiting evidence that it directly addresses the motivating failure mode.

Questions for authors

[Q1]. Isolating relational supervision: Can the author compare RDSF against a matched capacity fusion module trained on the same QA examples without structural pretraining, both with and without cross image mean exchange? Please also report seed variance for the relatively small structural-pretraining gain.

[Q2]. Diagnostic validation: Why is RDSF not evaluated on the original SPair-71k/CO3D/HPatches/GQA diagnostic suite? Such an evaluation would more directly establish whether the proposed method addresses the failure identified in the paper.

[Q3]. Checkpoint and benchmark selection: How were the backbone specific pretraining checkpoints and inference scaling coefficients selected? Why does the cross architecture evaluation include only four benchmarks? Please clarify whether these choices were made independently of test performance.

[Q4]. Comparison fairness and efficiency: Were all external baselines evaluated using identical scoring protocols? Can the authors report total training compute and inference latency/memory overhead, including relational pretraining, for a fairer comparison with LoRA?

[Q5]. Cross-image aggregation: Since eq. (5) uses reference centric global pooling, how sensitive is RDSF to image ordering, increasing image counts, and irrelevant distractors?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Eenb (1/2)

W1

See Q1 for the capacity-matched ablations of structural pretraining and cross-image mean exchange, together with paired downstream-seed comparisons.

Eq. (5) exchanges relation features learned through structured supervision. These features are learned using entity, attribute, and relation supervision in Eq. (3); Eq. (5) then uses mean pooling to combine each patch's features with information from other images. The capacity-matched controls in Q1 show that this pretraining provides an additional benefit under the same branch capacity and QA training budget.

W2

See Q1 for paired seed results and the variance of the six-benchmark mean gain; Q2 for validation on the original diagnostic suite; Q3 for checkpoint and inference-scale selection; Q4 for evaluation protocols, training costs, and inference overhead; and Q5 for robustness experiments on image ordering, image count, and distractors.

Q1

1. Capacity-matched ablations of pretraining and cross-image mean exchange. All four conditions freeze Qwen3.5-9B and jointly train identical mapping and fusion modules (1.74M parameters) on the same 5,000 QA examples for 5,000 steps. When cross-image exchange is disabled, we use the within-image mean.

InitializationWithout cross-image exchangeWith cross-image exchange
Random73.014573.4804
Structural pretraining74.114374.5204

Structural pretraining improves six-benchmark average accuracy under both exchange settings, and cross-image exchange improves accuracy under both initializations.

2. Seed variance. With cross-image exchange enabled, we fix the pretraining checkpoint and compare conditions with and without pretraining across three downstream seeds, matching QA order and fusion initialization within each seed.

SeedB: random initializationD: pretrained initializationPaired gain D−B
1773.480474.5204+1.0400
1871.372871.9960+0.6232
2071.729873.0779+1.3481

Pretraining improves accuracy across all three seeds. The paired gains have a sample standard deviation of 0.3638 percentage points and a sample variance of 0.1323 squared percentage points.

Q2

We added an evaluation of Base and RDSF on the original 2,475-example diagnostic suite. Both models pass all local checks on the same 2,173 examples, with the following results:

MetricBase (%)RDSF (%)Gain (percentage points)
Relational-target identification47.4954.86+7.36
Original-question answering56.3758.77+2.39

RDSF alleviates cross-image relational errors in the diagnostic suite; on all unfiltered examples, the two metrics also improve by 7.92 and 2.06 percentage points, respectively.

Q3

Pretraining checkpoints for all backbones were selected on an internal selection set independent of the test sets. The inference scaling coefficient for Qwen3.5-9B was also selected on this set, yielding step3500 and α=0.75; α was fixed at 1.0 for the other backbones. All models use the final QA training checkpoint, and each backbone’s configuration remains fixed across benchmarks.

The original cross-architecture evaluation covered four benchmarks due to the computational budget. We have now added NLVR2 and Q-Bench2, retaining the original checkpoints and inference configurations to extend evaluation to all six benchmarks:

BackboneBase six-benchmark average (%)RDSF six-benchmark average (%)Gain (percentage points)
InternVL3.5-2B55.1556.88+1.72
InternVL3.5-8B65.1565.97+0.82
InternVL3.5-38B70.7871.28+0.49
LLaVA-OV1.5-4B59.1762.45+3.28
LLaVA-OV1.5-8B59.6660.99+1.33
AuthorsOpenReview ↗

Response to Reviewer Eenb (2/2)

Q4

All methods use the same evaluation splits and scoring protocols. Multiple-choice questions use option scoring, and open-ended questions use greedy decoding, with consistent answer parsing and normalization. Each backbone retains its native image preprocessing and conversation template.

Training cost. We measured the complete Qwen3.5-9B RDSF training workflow on a single NVIDIA A800 80GB, including relational pretraining and QA alignment:

StageUpdatesSingle-GPU workflow duration
Relational pretraining5,0001.59 hours
QA alignment5,0002.58 hours
Total10,0004.17 GPU-hours

Inference overhead. On the same A800, we conducted paired measurements with BF16 and batch size 1 on 96 questions, with 16 from each of the six benchmarks. Each model was measured over three rounds.

Metric / scopeBaseRDSFChange relative to Base
Mean end-to-end latency: all questions (ms/question)434.59465.94+7.21%
Mean end-to-end latency: multiple-choice (ms/question)377.10418.62+11.01%
Mean end-to-end latency: open-ended (ms/question)771.31743.08−3.66%
Peak inference GPU memory (GiB)18.201018.4151+0.2141 GiB (+1.18%)

The lower open-ended latency is associated with shorter outputs: Base/RDSF generate an average of 6.21/3.86 tokens.

Q5

We conducted paired experiments on 600 MMIU retrieval questions to test image ordering, image count, and irrelevant distractors.

The table reports accuracy (%); Δ is RDSF’s change from the unperturbed input in percentage points. “Same-family” distractors come from the same task family.

ConditionBaseRDSFΔ RDSF
Unperturbed: original 4–5 images80.0083.670.00
Keep the first image fixed; permute the remaining images78.0082.22−1.44
Swap the first image with another target image70.3975.61−8.06
Append 1 random distractor78.8385.39+1.72
Append 1 same-family distractor79.2884.11+0.44
Append 2 random distractors76.5685.83+2.17
Append 2 same-family distractors77.1784.61+0.94
Append 4 random distractors75.5685.61+1.94
Append 4 same-family distractors76.5083.28−0.39
Append random distractors to reach 16 images73.2884.11+0.44
Append same-family distractors to reach 16 images76.3383.78+0.11
Move one of the four random distractors to the first position70.2880.39−3.28
Move one of the four same-family distractors to the first position73.3974.22−9.44

Image ordering. RDSF drops by 1.44 percentage points when the first image is fixed and the remaining images are permuted, compared with 8.06 points when the first image changes. This indicates greater sensitivity to the reference image. Base drops by 2.00 and 9.61 points, respectively.

Image count. Appending random or same-family distractors to reach 16 images changes RDSF accuracy by +0.44 and +0.11 points, respectively, while Base drops by 6.72 and 3.67 points. RDSF’s average accuracy remains relatively stable under these appended-distractor conditions.

Distractor position. Keeping the image set and count unchanged, moving one same-family distractor from the end to the first position causes an additional drop of 9.06 points for RDSF and 3.11 points for Base. This identifies a sensitivity to using a same-family distractor as the reference image.

Thank you for the detailed rebuttal and new experiments.

Remaining concerns:

  1. Seed variance: Seed 17 (used throughout the paper) is the best seed. The RDSF seed mean (≈73.2) is on par with LoRA (73.16, single run), so the advantage over LoRA is not established. Please report multi-seed mean ± std for the main results and for LoRA.
  2. Inconsistency with Table 16(b): Pretrained relations with within-image context score 70.54 in the paper but 74.11 in the new 2×2 (condition C). Please explain. If 74.11 is the fair comparison, cross-image exchange adds only ~0.4 pp.
  3. Relational attribution: The 2×2 compares pretraining with random initialization, not relational supervision with non-relational supervision (e.g., entity/attribute only). Claims should be limited to the ~1 pp pretraining gain actually measured.
  4. Reference-image sensitivity: Accuracy drops by 8–9 pp when the first image changes (Q5). This should be discussed as a limitation of Eq. (5).

I will maintain my score.

Reviewer_yZMmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The core research question this work is trying to answer is why vision-language models struggle with questions about multiple images. By investigating four models families the authors show that the models usally understand each image correctly but often fail to connect information across images and that giving them the correct cross-image relationship can greatly improve their answers. To address this, they propose Relation-Aware Dual-Stream Fusion (RDSF), a module that they add to a frozen vision-language model to learn cross-image relationships. It has 3 major components 1) a relation mapping that learns object, attribute and relationship information from scene graph labels 2) a cross-image context exchange step where the first image receives the summary of other images while the other images receive the summary of the first 3)a token-aligned fusion that adds the relation information obtained from the above steps to the model's original image tokens without making the input longer. The authors train there method in two stages 1) the relation mapping is trained on the scene graph labels and then the fusion part is trained keeping the relation stream frozen. Finally, they evlaute on six multi-image benchmarks with Qwen 3.5 models from 2B to 27B where they show RDSF improved over the base.

Critical strengths
  1. A clear and well-thought-out problem and diagnosis: The paper shows that across 4 model families, the models correctly read the task-relevant details in each image yet struggle to identify correct cross-image relation and supplying the correct relation substantially improves the accuracy. This is crucial because it motivates why multi-image reasoning fails and can be useful beyond this paper.

  2. Consistent improvements across model scales and families without retraining the backbone: The extensive evaluation shows that RDSF improves 6 benchmark mean for Qwen 3.5 scale and 4 benchmark mean for InternVL3.5 and LLaVA-OV1.5. They also show that for the pairs for which the base model passes the local check RDSF raised accuracy from 56% to 67%. This directly links the method to the problem the authors diagnosed.

Critical weaknesses
  1. The ablation suggests the scene-graph pretraining contributes little: Without scene-graph pretraining stage, the model still reaches 73.48, close to full RDSF (74.5) (Table 3a), which means most of the gain comes from the cross-image context exchange rather than the structured relation learning the paper emphasizes.

  2. Unclear if the diagnosed problem persists in frontier VLMs: The diagnosis covers only open models up to 27B (Figure 1(d)), and RDSF's gains shrink as models get larger, from +6.51 at 4B to +1.99 at 27B (Table 1). It is therefore unclear if the diagnosed gap exists in frontier VLMs, which affects how broadly relevant the problem and the method are.

Questions for authors

Q1. Given that removing scene-graph pretraining costs only about 1 point, how do the authors view its role? Is this difference consistent across other model sizes, backbones and multiple seeds?

Q2. Could the authors run the diagnostic via frontier VLMs?

Q3. Will the authors release the code, the diagnostic suite and the constructed QA data? The paper doesn’t include a reproducibility statement or a release commitment.

3 comments
AuthorsOpenReview ↗

Response to W1–W2 and Q1–Q3

W1

The randomly initialized branch still undergoes QA training, so its performance reflects the effects of branch adaptation and the fusion module; these gains cannot be attributed solely to cross-image exchange. Our added capacity-matched controls on the 9B backbone separately examine these two factors: structural pretraining improves accuracy by 1.10 and 1.04 percentage points with cross-image exchange disabled and enabled, respectively; cross-image exchange improves accuracy by 0.47 and 0.41 percentage points with random and pretrained initialization, respectively. These controls quantify the contributions of the two factors and support an additional benefit from structural pretraining under the same capacity and QA training settings. See Q1 for detailed results.

W2

Please see our response to Q2 below regarding diagnostic evaluation on frontier VLMs.

Q1

Scene-graph pretraining provides a structured initialization for the relation branch through entity, attribute, and relation supervision. The approximately one-point gain reflects its additional contribution over QA adaptation with the same capacity; full RDSF combines relational representation learning, cross-image exchange, and token-aligned fusion.

In capacity-matched controls on Qwen3.5-9B, pretraining improves six-benchmark average accuracy by 1.10 and 1.04 percentage points with cross-image exchange disabled and enabled, respectively.

To assess consistency across seeds, we fixed the pretraining checkpoint and conducted paired comparisons across three downstream training seeds:

Seed / statisticPaired pretraining gain (percentage points)
17+1.0400
18+0.6232
20+1.3481
Mean+1.0038
Sample standard deviation0.3638

Across models, full RDSF improves six-benchmark average accuracy on Qwen3.5 at 2B, 4B, 9B, and 27B, with gains of 1.99–6.51 percentage points, and also improves average accuracy on InternVL3.5 and LLaVA-OV1.5.

Q2

We thank the reviewer for the suggestion. Due to computational resource constraints, we are currently unable to run the diagnostic on larger models, and we apologize for this limitation.

Q3

We will publicly release the code, diagnostic suite, and constructed QA data, and add an explicit reproducibility statement and release commitment to the revised manuscript. The release will include training and evaluation scripts, data construction procedures, and the configurations needed to reproduce the experiments.

Thank you for the detailed response. My concerns are partly resolved: the pretraining gain is shown only on the 9B model, and the frontier VLM diagnostic remains open. I maintain my score.

AuthorsOpenReview ↗

Follow-up on Q2 (Frontier VLM Diagnostic Results)

Thank you for the suggestion. Since our previous response, we have evaluated DeepSeek-V4.1-Flash-1017, a frontier VLM, on all 2,475 examples of the diagnostic in Table 5. The model correctly read every candidate’s code on 94.99% of examples. Even among examples that passed this check, it identified the correct cross-image target in only 1,207 of 2,348 scored cases (51.41%). On matched examples, supplying the correct relation target improved final-answer accuracy by 25.35 percentage points over supplying each candidate’s code.

Reviewer_4Kq8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a gap between recognizing information within individual images and using that information to answer cross-image questions. It proposes RDSF, which adds a relation stream to a frozen vision-language model. This stream is pre-trained with structured supervision, exchanges information across images, and is fused back into the original visual tokens through a residual interface. The paper reports gains on several multi-image benchmarks and provides diagnostic experiments showing that cross-image comparisons can fail even when the model answers the corresponding local questions correctly.

Critical strengths
  1. Being able to identify an attribute is not the same as being able to compare it across images. On the same 341 image pairs where the 9B base model passes the local checks, cross-image accuracy improves from 56.01% to 67.16%, with a paired confidence interval supporting the improvement. This is meaningful evidence, not just an illustrative failure case.

  2. The method is tested across multiple model sizes and families, and the main improvements are not negligible: for example, the six-benchmark average on Qwen3.5-9B increases from 69.50 to 74.50. Keeping the backbone frozen and avoiding additional visual tokens are also attractive implementation choices.

Critical weaknesses
  1. I am not convinced that the experiments isolate the value of the main proposed contribution. The paper motivates learning dedicated relational representations, but much of the gain is already present without the proposed structural pre-training. Both mappings are updated during QA training in this comparison; this is not exactly the same setup as the standard frozen-relation-stream RDSF result of 74.50. Even with that distinction, the comparison makes me question how much of the overall improvement comes specifically from learning the proposed relational representation, rather than adding a trainable branch and adapting it with QA supervision.
  2. The practical case over a simpler adaptation baseline is also not clear enough. The two-stage LoRA baseline reaches 73.16 versus 74.50 for RDSF on the 9B model. That is a positive difference, but the comparison does not match the full training pipeline: LoRA uses 20,000 instruction-converted examples for its structural stage, whereas RDSF uses a different structured-supervision procedure drawn from a pool of 836,316 examples. The paper acknowledges that the QA-stage comparison in Table 14 is not an end-to-end compute-matched comparison. Thus, the small number of QA-stage trainable parameters does not establish that the complete approach is a more efficient way to obtain the improvement.
Questions for authors
  1. What is the main contribution you want the reader to take away? Is it the diagnostic finding, structural relation pretraining, or the cross-image fusion design?
  2. Which existing evidence separates relation-specific learning from generic adaptation? In particular, do you have a matched comparison with a similarly sized branch trained without relation-specific supervision, or with entity/attribute supervision alone? A focused control on one backbone would be more informative to me than another set of benchmark scores.
1 comment
AuthorsOpenReview ↗

Response to W1–W2 and Q1–Q2

W1

Our controls test whether structural pretraining adds value beyond adding a branch and adapting it with QA supervision. The randomly initialized and pretrained branches use the same architecture, parameter count, and QA training settings, with both the relation mapping and fusion module updated. Structural pretraining provides an additional gain of approximately 1 percentage point in these controls (see Q2). This attribution study is reported separately from standard RDSF, which freezes the relation mapping during QA training; see Q1 for the contribution of the full framework.

W2

Under the same six-benchmark evaluation protocol, Qwen3.5-9B RDSF achieves an average accuracy of 74.50%, compared with 73.16% for two-stage LoRA, a gain of 1.34 percentage points. Both methods use the same 5,000 training examples during the QA stage.

To fully report RDSF's training cost, we additionally timed the complete training workflow using the paper's 9B configuration. On a single A800 80GB, relational pretraining and QA alignment each execute 5,000 updates, taking 1.59 and 2.58 hours, respectively, for a total of 4.17 GPU-hours. This cost covers both relational pretraining and subsequent QA adaptation.

Q1

Our main contribution is RDSF: a framework that learns relational representations complementary to native visual representations and integrates them into a frozen VLM to improve multi-image understanding.

The diagnosis, pretraining, and fusion design serve this goal: the diagnosis shows that correctly recognizing individual images does not guarantee correctly establishing cross-image relations; structural pretraining supplies entity, attribute, and relation supervision to the relation branch; cross-image exchange and token-aligned fusion incorporate these representations into answering while preserving the original visual sequence length.

We also directly evaluated the framework on the original diagnostic suite. On the same 2,173 examples where both models pass the local checks, RDSF improves relational-target identification from 47.49% to 54.86% and final-answer accuracy from 56.37% to 58.77%, supporting its effectiveness on the diagnosed cross-image relational errors.

Q2

We conducted capacity-matched 2×2 controls on Qwen3.5-9B, independently varying structural-pretraining initialization and cross-image mean exchange.

All four conditions freeze the backbone and jointly train identical mapping and fusion modules (1.74M parameters), using the same 5,000 QA examples, 5,000 training steps, and training and evaluation settings. QA order and fusion-module initialization are matched.

ConditionStructural-pretraining initializationCross-image mean exchangeSix-benchmark average (%)
ANoDisabled: within-image mean73.0145
BNoEnabled73.4804
CYesDisabled: within-image mean74.1143
DYesEnabled74.5204

Structural pretraining improves average accuracy by 1.10 and 1.04 percentage points with cross-image exchange disabled and enabled, respectively. It therefore provides an additional benefit over QA-only adaptation under the same branch capacity and QA training budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.