- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates failures modes in cross image relational reasoning despite having correct local visual recognition. Through controlled diagnostics across four VLM families, the authors show that identifying cross image relations remains challenging task even when local attributes and predicates are correctly recognized.
To address this, they propose Relation-Aware Dual-Stream Fusion (RDSF), which learns structured relational features from GQA / Visual Genome annotations, exchanges cross-image context through mean pooled representations, and integrates them into frozen VLMs using token aligned residual fusion. Experiments across six benchmarks shows improvements of 1.99 to 6.51 percentage points across four Qwen3.5 scales, with additional gains on InternVL3.5 and LLaVA OV1.5.
Critical strengths
[S1]. Meaningful diagnostic. The paper separates local perception from cross image relation identification through controlled information interventions and independent semantic checks. The analysis across multiple VLM families provides useful evidence that recognizing local visual facts does not guarantee successful relational comparison.
[S2]. Practical architecture with consistent improvements. RDSF improves performance across multiple backbones while keeping pretrained parameters frozen and preserving visual token sequence length. The gains are substantial on some benchmarks (e.g., +14.14 pp on MIRB for Qwen3.5 4B), and the ablations provide useful evidence for cross image context exchange and token aligned fusion.
Critical weaknesses
[W1]. Insufficient attribution to structured relational learning. The central claim is not convincingly isolated from generic feature augmentation. On Qwen3.5 9B, randomly initialized relation mapping already achieves 73.48% versus 74.50% for RDSF (Table 3), while the matched initialization comparison shows only a +1.04 pp benefit from structural pretraining (Table 16). Moreover, eq. (5) exchanges globally mean-pooled features rather than explicitly modeling entity level correspondences. Stronger matched capacity controls are needed to establish that relational supervision is responsible for the improvements.
[W2]. Evaluation and robustness limitations. The criteria for selecting backbone specific pretraining checkpoints and inference residual scales are unclear. The LoRA comparison is not end-to-end compute matched, benchmark level results lack multi-seed variance, and robustness to image ordering and distractors is not systematically evaluated. Furthermore, RDSF is not evaluated on the original 2,475-example diagnostic suite, limiting evidence that it directly addresses the motivating failure mode.
Questions for authors
[Q1]. Isolating relational supervision: Can the author compare RDSF against a matched capacity fusion module trained on the same QA examples without structural pretraining, both with and without cross image mean exchange? Please also report seed variance for the relatively small structural-pretraining gain.
[Q2]. Diagnostic validation: Why is RDSF not evaluated on the original SPair-71k/CO3D/HPatches/GQA diagnostic suite? Such an evaluation would more directly establish whether the proposed method addresses the failure identified in the paper.
[Q3]. Checkpoint and benchmark selection: How were the backbone specific pretraining checkpoints and inference scaling coefficients selected? Why does the cross architecture evaluation include only four benchmarks? Please clarify whether these choices were made independently of test performance.
[Q4]. Comparison fairness and efficiency: Were all external baselines evaluated using identical scoring protocols? Can the authors report total training compute and inference latency/memory overhead, including relational pretraining, for a fairer comparison with LoRA?
[Q5]. Cross-image aggregation: Since eq. (5) uses reference centric global pooling, how sensitive is RDSF to image ordering, increasing image counts, and irrelevant distractors?
3 comments
Response to Reviewer Eenb (2/2)
Q4
All methods use the same evaluation splits and scoring protocols. Multiple-choice questions use option scoring, and open-ended questions use greedy decoding, with consistent answer parsing and normalization. Each backbone retains its native image preprocessing and conversation template.
Training cost. We measured the complete Qwen3.5-9B RDSF training workflow on a single NVIDIA A800 80GB, including relational pretraining and QA alignment:
| Stage | Updates | Single-GPU workflow duration |
|---|---|---|
| Relational pretraining | 5,000 | 1.59 hours |
| QA alignment | 5,000 | 2.58 hours |
| Total | 10,000 | 4.17 GPU-hours |
Inference overhead. On the same A800, we conducted paired measurements with BF16 and batch size 1 on 96 questions, with 16 from each of the six benchmarks. Each model was measured over three rounds.
| Metric / scope | Base | RDSF | Change relative to Base |
|---|---|---|---|
| Mean end-to-end latency: all questions (ms/question) | 434.59 | 465.94 | +7.21% |
| Mean end-to-end latency: multiple-choice (ms/question) | 377.10 | 418.62 | +11.01% |
| Mean end-to-end latency: open-ended (ms/question) | 771.31 | 743.08 | −3.66% |
| Peak inference GPU memory (GiB) | 18.2010 | 18.4151 | +0.2141 GiB (+1.18%) |
The lower open-ended latency is associated with shorter outputs: Base/RDSF generate an average of 6.21/3.86 tokens.
Q5
We conducted paired experiments on 600 MMIU retrieval questions to test image ordering, image count, and irrelevant distractors.
The table reports accuracy (%); Δ is RDSF’s change from the unperturbed input in percentage points. “Same-family” distractors come from the same task family.
| Condition | Base | RDSF | Δ RDSF |
|---|---|---|---|
| Unperturbed: original 4–5 images | 80.00 | 83.67 | 0.00 |
| Keep the first image fixed; permute the remaining images | 78.00 | 82.22 | −1.44 |
| Swap the first image with another target image | 70.39 | 75.61 | −8.06 |
| Append 1 random distractor | 78.83 | 85.39 | +1.72 |
| Append 1 same-family distractor | 79.28 | 84.11 | +0.44 |
| Append 2 random distractors | 76.56 | 85.83 | +2.17 |
| Append 2 same-family distractors | 77.17 | 84.61 | +0.94 |
| Append 4 random distractors | 75.56 | 85.61 | +1.94 |
| Append 4 same-family distractors | 76.50 | 83.28 | −0.39 |
| Append random distractors to reach 16 images | 73.28 | 84.11 | +0.44 |
| Append same-family distractors to reach 16 images | 76.33 | 83.78 | +0.11 |
| Move one of the four random distractors to the first position | 70.28 | 80.39 | −3.28 |
| Move one of the four same-family distractors to the first position | 73.39 | 74.22 | −9.44 |
Image ordering. RDSF drops by 1.44 percentage points when the first image is fixed and the remaining images are permuted, compared with 8.06 points when the first image changes. This indicates greater sensitivity to the reference image. Base drops by 2.00 and 9.61 points, respectively.
Image count. Appending random or same-family distractors to reach 16 images changes RDSF accuracy by +0.44 and +0.11 points, respectively, while Base drops by 6.72 and 3.67 points. RDSF’s average accuracy remains relatively stable under these appended-distractor conditions.
Distractor position. Keeping the image set and count unchanged, moving one same-family distractor from the end to the first position causes an additional drop of 9.06 points for RDSF and 3.11 points for Base. This identifies a sensitivity to using a same-family distractor as the reference image.
Thank you for the detailed rebuttal and new experiments.
Remaining concerns:
- Seed variance: Seed 17 (used throughout the paper) is the best seed. The RDSF seed mean (≈73.2) is on par with LoRA (73.16, single run), so the advantage over LoRA is not established. Please report multi-seed mean ± std for the main results and for LoRA.
- Inconsistency with Table 16(b): Pretrained relations with within-image context score 70.54 in the paper but 74.11 in the new 2×2 (condition C). Please explain. If 74.11 is the fair comparison, cross-image exchange adds only ~0.4 pp.
- Relational attribution: The 2×2 compares pretraining with random initialization, not relational supervision with non-relational supervision (e.g., entity/attribute only). Claims should be limited to the ~1 pp pretraining gain actually measured.
- Reference-image sensitivity: Accuracy drops by 8–9 pp when the first image changes (Q5). This should be discussed as a limitation of Eq. (5).
I will maintain my score.
Response to Reviewer Eenb (1/2)
W1
See Q1 for the capacity-matched ablations of structural pretraining and cross-image mean exchange, together with paired downstream-seed comparisons.
Eq. (5) exchanges relation features learned through structured supervision. These features are learned using entity, attribute, and relation supervision in Eq. (3); Eq. (5) then uses mean pooling to combine each patch's features with information from other images. The capacity-matched controls in Q1 show that this pretraining provides an additional benefit under the same branch capacity and QA training budget.
W2
See Q1 for paired seed results and the variance of the six-benchmark mean gain; Q2 for validation on the original diagnostic suite; Q3 for checkpoint and inference-scale selection; Q4 for evaluation protocols, training costs, and inference overhead; and Q5 for robustness experiments on image ordering, image count, and distractors.
Q1
1. Capacity-matched ablations of pretraining and cross-image mean exchange. All four conditions freeze Qwen3.5-9B and jointly train identical mapping and fusion modules (1.74M parameters) on the same 5,000 QA examples for 5,000 steps. When cross-image exchange is disabled, we use the within-image mean.
Structural pretraining improves six-benchmark average accuracy under both exchange settings, and cross-image exchange improves accuracy under both initializations.
2. Seed variance. With cross-image exchange enabled, we fix the pretraining checkpoint and compare conditions with and without pretraining across three downstream seeds, matching QA order and fusion initialization within each seed.
Pretraining improves accuracy across all three seeds. The paired gains have a sample standard deviation of 0.3638 percentage points and a sample variance of 0.1323 squared percentage points.
Q2
We added an evaluation of Base and RDSF on the original 2,475-example diagnostic suite. Both models pass all local checks on the same 2,173 examples, with the following results:
RDSF alleviates cross-image relational errors in the diagnostic suite; on all unfiltered examples, the two metrics also improve by 7.92 and 2.06 percentage points, respectively.
Q3
Pretraining checkpoints for all backbones were selected on an internal selection set independent of the test sets. The inference scaling coefficient for Qwen3.5-9B was also selected on this set, yielding step3500 and α=0.75; α was fixed at 1.0 for the other backbones. All models use the final QA training checkpoint, and each backbone’s configuration remains fixed across benchmarks.
The original cross-architecture evaluation covered four benchmarks due to the computational budget. We have now added NLVR2 and Q-Bench2, retaining the original checkpoints and inference configurations to extend evaluation to all six benchmarks: