acceptodds
Under review as a conference paper at ICLR 2027

Quotient-Set Evidence Fusion for Robust Multimodal Reasoning

Abstract

Reliable reasoning over multiple images and document pages requires combining distinct facts, yet repeated content can disproportionately influence predictions even when it adds no information. We make multimodal reasoning resistant to artificial repetition while preserving the complementary and conflicting evidence needed to answer a question. We introduce Quotient-Set Evidence Fusion (QSEF), which identifies redundancy relative to the question and adjusts each evidence unit's attention weight according to its estimated repetition count. QSEF corrects attention before pooling independently encoded inputs into a compact evidence representation, separating the relevance of information from its frequency while retaining every input. Across eligible subsets of six multi-image benchmarks, including MuirBench and BLINK, and three vision–language backbones, QSEF improves accuracy-based scores by 1.4–2.7 percentage points over the strongest compared baseline in all 18 comparisons. Our analysis establishes exact repetition invariance for correctly identified copies with matching representations and bounds representation drift under imperfect redundancy estimates, clarifying when robust multimodal evidence aggregation is achievable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal reasoning, multi-image understanding, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JRoFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Quotient-Set Evidence Fusion (QSEF) for multimodal reasoning with redundant and complementary evidence. The central idea is to aggregate question-relevant semantic evidence classes rather than overweight facts that occur repeatedly. The paper also establishes duplication-invariance and perturbation results under explicit representation and estimation assumptions, with extensions to nonidentical representations and extreme repetition. Experiments across three backbones, six multi-image benchmarks, controlled evidence tasks, and document-reader transfer report improvements in accuracy and stability.

Critical strengths

S1. A well-motivated problem and an interpretable design. The paper addresses an important challenge in multimodal reasoning: repeated evidence can receive disproportionate attention, while complementary evidence should remain distinguishable. The question-conditioned multiplicity correction provides an intuitive mechanism for suppressing redundancy while preserving useful information.

S2. Substantive theoretical analysis with explicit assumptions. The paper establishes exact duplication invariance under ideal relation estimates and matching representations, and characterizes sensitivity to multiplicity-estimation errors. The extensions to nonidentical representations and extreme repetition help clarify the conditions and limitations of the proposed approach.

S3. A substantial empirical evaluation with informative controls. The evaluation covers three backbone models and multiple benchmarks, with comparisons against pooling and deduplication alternatives. The auxiliary relation-head baseline and component ablations help isolate the contribution of the proposed correction. Reporting multiple seeds, grouped bootstrap intervals, and efficiency measurements further strengthens the empirical evaluation.

Critical weaknesses

W1. The extreme-repetition measurements are insufficiently specified for independent reconstruction. Appendix C.5 states that the export contains percentage-valued accuracies per question–checkpoint record, while the underlying binary outcomes and trial counts are unavailable. The class-value tensors required to reconstruct the normalized fusion displacement are also unavailable. Consequently, the statistical meaning of each percentage, the appropriateness of averaging these records, and the uncertainty of the interpolated three-percentage-point failure boundary are difficult to assess. This concern specifically affects the quantitative stress-test conclusions; it does not establish that the main benchmark results are invalid.

W2. Evidence for generalization to natural semantic redundancy is limited. Natural-data supervision provides verified exact-copy positives, while detailed semantic-equivalence and complementary-evidence supervision is largely supplied by controlled cards. The card experiments and held-out templates can not fully establish that the learned relation handles naturally occurring alternative presentations of the same facts. Partial overlap and conflicting information are particularly relevant because erroneous merging could suppress necessary evidence. The paper should discuss whether its favorable stability–utility tradeoff persists under natural semantic variation.

W3. The experimental evidence supports a limited scope for generality and scalability. The benchmark results are obtained on eligible subsets, and several improvements are not statistically resolved: the Mantis-Eval intervals include zero for all three backbones, as do multiple document-transfer comparisons. Furthermore, the 1,000-document setting increases the retrieval pool while the reader receives at most five retrieved documents. It therefore does not test fusion over 1,000 evidence units. These qualifications do not negate the positive results, but should constrain claims about generality and scalability.

W4. The additional empirical insight provided by the extensions in Appendices D.14–D.16 is less clear. In particular, the paper does not sufficiently show whether the quantities governing stability under nonidentical representations are informative in realistic settings, or how the conditional loss analysis relates to the observed answer-level behavior. Clarifying this connection would help readers distinguish what the theory establishes under its assumptions from what the experiments demonstrate.

Questions for authors

Q1. Does each percentage represent repeated trials, a task-specific per-example soft score, or another aggregation? Please specify the scoring rule and denominator where applicable, and provide the predictions, scores, and tensors needed to reconstruct the endpoints. With presentations and checkpoints grouped by base question, how uncertain are the failure thresholds, and how sensitive are they to the three-percentage-point criterion and interpolation procedure?

Q2. Can the authors evaluate a human-annotated natural near-duplicate set that distinguishes semantic equivalence, partial overlap, complementarity, and contradiction for the given question? Please report relation errors, erroneous merging of complementary evidence, and downstream accuracy.

Q3. For the extension to nonidentical representations, could you provide a representative empirical analysis showing the magnitude of the bound’s main terms and how they relate to the observed fusion drift? Please also clarify which conclusions about answer stability are supported by the experiments and which remain conditional on additional decoder or decision-margin assumptions.

Q4. Could the authors add experiments with one or two VLMs released in 2026, such as suitable Qwen3.5 or Gemma 4 variants, or another recent small open-weight VLM, to assess whether QSEF’s gains generalize to newer backbones and strengthen the empirical evidence?

9 comments
AuthorsOpenReview ↗

Response to Reviewer JRoF (1/4): W1/Q1: Stress-test scoring and thresholds

We appreciate the time and care you put into your review and your detailed suggestions on scoring, theory, and recent backbones. Our responses below provide the scoring definitions, quantitative analyses, and comparisons with a newer backbone for each point.

W1 / Q1. Extreme-repetition scoring, reconstruction, and threshold uncertainty.

Response.{\color{green}\mathbf{Response.}} The new evaluation gives an explicit scoring rule and denominator for the stress test you asked about. It uses one deterministic greedy answer per question, checkpoint, and repetition level, scored correct/incorrect. It contains 252 independent questions, three checkpoints, and seven levels N∈1,2,4,8,16,32,64N\in\\{1,2,4,8,16,32,64\\}: 5,292 binary outcomes. For a stratum with nn questions,

Acc⁡(N)=1003n∑s=13∑q=1n1answerq,s,N is correct.\operatorname{Acc}(N)=\frac{100}{3n}\sum_{s=1}^{3}\sum_{q=1}^{n}\mathbf{1}\\{\text{answer}_{q,s,N}\text{ is correct}\\}.

Low/mid/high cross-affinity strata are 0.025–0.105, 0.125–0.185, and 0.205–0.285, with 90/72/90 questions. Counts and sample SDs make the aggregate accuracies reconstructible:

AffinityN occurrencesQuestionsCorrect: seeds 42 / 43 / 44Accuracy (%) ± SDDrop from N=1 (pp), 95% CINormalized fusion drift
Low19074/73/7582.2 ± 1.10.0 [0.0,0.0]0.000
Low89074/73/7481.9 ± 0.60.4 [-0.6, 1.3]0.071
Low169074/72/7481.5 ± 1.30.7 [-0.4, 1.9]0.111
Low329074/72/7381.1 ± 1.11.1 [-0.1, 2.3]0.146
Low649073/72/7380.7 ± 0.61.5 [0.1, 2.8]0.186
Mid17260/59/6082.9 ± 0.80.0 [0.0,0.0]0.000
Mid87259/59/5981.9 ± 0.00.9 [-0.2, 2.1]0.095
Mid167259/58/5881.0 ± 0.81.9 [0.4, 3.3]0.164
Mid327258/57/5880.1 ± 0.82.8 [1.1, 4.4]0.253
Mid647257/56/5778.7 ± 0.84.2 [2.3, 6.0]0.334
High19075/76/7483.3 ± 1.10.0 [0.0,0.0]0.000
High89074/74/7281.5 ± 1.31.9 [0.4, 3.3]0.142
High169072/71/7179.3 ± 0.64.1 [2.2, 5.9]0.258
High329070/69/6876.7 ± 1.16.7 [4.4, 8.9]0.431
High649067/66/6673.7 ± 0.69.6 [7.1, 12.2]0.638

At N=2,4N=2,4, correct counts equal their respective N=1N=1 counts. NN counts occurrences of the repeated class. Including the singleton gives N+1N+1 inputs, up to 65. The table uses a separate binary scoring protocol from the continuous-score export underlying Table 12. Its explicit denominator, 3n3n, and per-seed counts allow direct reconstruction of the reported means and SDs.

For threshold uncertainty, bootstrap resampling groups all presentations and checkpoints by base question. The operational threshold is the first crossing of an accuracy drop τ\tau. Resamples without a crossing are right-censored at >64>64.

AffinityDrop criterion (pp)Observed first-crossing bracketLog₂ interpolation, 95% CILinear-N interpolation, 95% CIBootstrap crossing by N=64 (%)
Low116–3226.0 [17.0, >64]27.2 [18.0, >64]84.0
Low3>64 (right-censored)>64>646.0
Low5>64 (right-censored)>64>640.0
Mid18–168.5 [6.8, 14.8]8.6 [7.2, 15.2]99.0
Mid332–6435.8 [25.0, >64]37.1 [27.0, >64]82.0
Mid5>64 (right-censored)>64>6422.0
High14–85.8 [4.7, 9.8]6.2 [4.9, 10.3]100.0
High38–1611.4 [9.2, 16.7]12.1 [9.5, 17.2]99.5
High516–3220.5 [16.1, 31.8]21.7 [16.5, 32.8]99.0

Between the first bracketing levels Na,NbN_a,N_b, with drops da,dbd_a,d_b, define w=(τ−da)/(db−da)w=(\tau-d_a)/(d_b-d_a). The two interpolation rules are Na(Nb/Na)wN_a(N_b/N_a)^w and Na+(Nb−Na)wN_a+(N_b-N_a)w. Both locate the high-affinity 3-pp crossing near 11–12 occurrences, while the mid-affinity upper interval extends beyond 64. The threshold is defined by the chosen accuracy-drop criterion. Its sensitivity is quantified by the 1/3/5-pp criteria and both interpolation rules. The trend also follows the mechanism: residual cross-affinity β\beta makes the singleton degree 1+Nβ1+N\beta, increasingly suppressing rare evidence.

AuthorsOpenReview ↗

Response to Reviewer JRoF (2/4): W2/Q2: Natural relations and useful evidence

W2 / Q2. Human-annotated natural relations and harm from suppressing useful evidence.

Response.{\color{green}\mathbf{Response.}} The four relation categories you suggest directly test whether QSEF preserves useful differences between evidence units. The human-annotated natural test contains 400 held-out sources, with 100 each for equivalence, partial overlap, complementarity, and contradiction. Each category is balanced between images and documents. On Qwen2.5-VL-3B:

MethodAUROCAUPRCEquivalent recall (%)Partial FPR (%)Complement FPR (%)Conflict FPR (%)Question-switch joint accuracy (%)
PMA + relation head88.778.572.0 (216/300)20.0 (60/300)14.0 (42/300)19.0 (57/300)76.7 (138/180)
QSEF question-blind81.067.367.0 (201/300)26.0 (78/300)22.0 (66/300)21.0 (63/300)0.0 (0/180)
QSEF92.985.478.0 (234/300)13.0 (39/300)9.0 (27/300)14.0 (42/300)83.9 (151/180)

Thresholds are fixed on development data. Each relation has 100 independent sources and 300 source–seed decisions. The 60 question-switch pairs reuse these sources. Natural variation gives an AUROC of 92.9 and a complementary FPR of 9.0%, compared with 96.0 and 6.4% on controlled cards (Table 6). These measurements quantify relation quality under the more varied natural presentations.

For the 100 naturally equivalent sources, answer-level behavior is:

MethodOriginal accuracy (%)Exact-copy robust score (%)Semantic accuracy (%)Semantic − original (pp), 95% CISemantic robust score (%)Semantic flip (%)
PMA + relation head61.058.356.3-4.7 [-7.8, -1.6]54.311.0
Exact-copy dedup + PMA60.060.055.0-5.0 [-8.1, -1.9]53.710.0
Similarity dedup + PMA59.059.056.7-2.3 [-5.4, 0.8]53.710.7
QSEF65.064.764.0-1.0 [-3.2, 1.2]63.33.3

To measure the effect of erroneous downweighting on answers, the diagnostic oracle mask zeros human-labeled non-equivalent edges and retains the other edges. Harm counts answers recovered by this intervention. It measures downstream consequences separately from relation FPR. The oracle mask is used only for this diagnostic.

MethodPartial accuracy (%)Partial unique-information gain (pp)Complement accuracy (%)Complement unique-information gain (pp)Conflict accuracy (%)Harm: partial / complement / conflict
PMA + relation head64.39.366.012.357.0—
Exact-copy dedup + PMA63.08.364.712.055.3—
Similarity dedup + PMA60.37.061.710.052.7—
QSEF69.014.772.317.361.74/300 (1.3%) / 3/300 (1.0%) / 3/300 (1.0%)
QSEF + oracle mask (diagnostic)70.315.073.317.762.7—

Each non-equivalent category has 100 sources and 300 source–seed decisions. Unique-information gain is Acc⁡(full)−Acc⁡(remove necessary unique evidence)\operatorname{Acc}(\text{full})-\operatorname{Acc}(\text{remove necessary unique evidence}). QSEF achieves higher full-input accuracy and larger gains from necessary information. The oracle recovers 4/3/3 decisions for partial/complementary/conflicting evidence, corresponding to harm rates of 1.3%/1.0%/1.0%. Together with the semantic-variation results, this shows that QSEF preserves useful information while reducing sensitivity to repetition.

AuthorsOpenReview ↗

Response to Reviewer JRoF (3/4): W3: Generality and scalability

W3. Generality, statistical scope, and the 1,000-document setting.

Response.{\color{green}\mathbf{Response.}} The scalability result in Figure 4 concerns reader transfer as the retrieval pool grows to 100/200/1,000 documents. As you note, the reader receives at most five retrieved documents. QSEF compresses those inputs into a fixed 64-token evidence prefix. Relation computation scales quadratically with the number of reader inputs, so retrieval-pool size and fusion-input size have separate roles in this evaluation.

Table 1 evaluates eligible order/copy-invariant questions. In Table 8, paired intervals against PMA exclude zero in 15/18 multi-image comparisons and 4/18 document-transfer comparisons. All three Mantis-Eval intervals include zero. These intervals specify the statistical support for each reported gain.

To check whether backbone-dependent retained subsets drive the gains, the common-intersection comparison against PMA+head gives:

BenchmarkCommon questionsQwen: original → common gain [common 95% CI]LLaVA: original → common gain [common 95% CI]InternVL: original → common gain [common 95% CI]
MuirBench13922.1 → 2.0 [1.0, 3.0]1.6 → 1.4 [0.4, 2.4]1.7 → 1.7 [0.7, 2.7]
MIBench22701.6 → 1.6 [0.7, 2.5]1.5 → 1.4 [0.5, 2.3]1.5 → 1.6 [0.7, 2.5]
MMIU10241.1 → 1.2 [0.0, 2.4]1.2 → 1.1 [-0.1, 2.3]0.9 → 0.9 [-0.3, 2.1]
MIRB6701.0 → 0.9 [-0.7, 2.5]1.1 → 1.2 [-0.4, 2.7]1.0 → 1.0 [-0.6, 2.5]
Mantis-Eval1781.4 → 1.6 [-1.4, 4.6]0.9 → 1.0 [-2.0, 4.0]1.0 → 0.9 [-2.1, 3.9]
BLINK9601.5 → 1.4 [0.2, 2.6]1.2 → 1.3 [0.1, 2.5]0.9 → 1.1 [-0.1, 2.3]

Intervals are source-grouped 95% CIs. Every point gain stays positive and changes by at most 0.2 pp on the common intersection. This supports the consistency of the improvement across the eligible subsets. Figure 4 separately demonstrates transfer to document reading with larger retrieval pools.

Reviewer_Xx4LOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Quotient-Set Evidence Fusion (QSEF) to reduce the influence of repeated evidence in multi-image reasoning. It independently encodes each input, estimates question-dependent redundancy, and uses the estimated repetition counts to adjust attention before pooling. The paper proves exact duplication invariance under ideal conditions and bounds representation changes when repetition estimates are imperfect. Experiments cover eligible subsets of six multi-image benchmarks, three vision–language backbones, document QA, and controlled repetition tests.

Critical strengths

1.The method addresses a meaningful problem with a clear mechanism. Repeated evidence should not automatically receive more influence. QSEF estimates redundancy relative to the question and adjusts attention while retaining all inputs. The theoretical analysis explains both when this correction works and why residual relation errors can become problematic under heavy repetition.

2.The comparisons include useful controls. The paper evaluates three backbones and includes pooling, deduplication, and capacity-matched baselines. In particular, PMA with the same relation head helps separate the benefit of attention correction from extra supervision. The reported accuracy gains are consistent across the six benchmark subsets.

Critical weaknesses
  1. Some results are hard to explain from the stated method. If images are encoded independently and without randomness, copying an image should not change its local features or the class-value spread Dˉ\bar D. Yet Appendix C.4 reports changes in every question–checkpoint group. The paper does not explain why this happens. These missing details make it hard to check whether the code follows the stated method and whether the stability analysis explains the observed failures.

  2. The tests provide limited evidence beyond exact image copies. For natural images, only verified identical copies are labeled as redundant. Other pairs are left unlabeled. The tests of question-dependent repetition use constructed ChartQA cards. The tests of whether QSEF keeps useful, non-repeated facts also use these cards. Final-model duplication tests on natural data cover only Qwen on MuirBench and use exact copies. It is therefore unclear how well QSEF handles different images or document layouts that repeat the same information. This is important because it is a key reason to use QSEF instead of simply removing duplicate images.

Questions for authors
  1. Why does Dˉ\bar D) change when only identical copies are added in Appendix C.4?
  2. Can you test semantically repeated but visually different evidence on natural data, and report duplication robustness for another backbone? These results would help establish whether the benefit extends beyond exact copies and the primary model. If the author could explain and supplement these results, I would be able to increase my rating.
5 comments
AuthorsOpenReview ↗

Response to Reviewer Xx4L: C.4 clarification and natural semantic robustness

Thank you for taking the time to examine our method closely and for your helpful suggestions on robustness beyond exact copies. We first explain the C.4 diameter correction, then present the results on natural semantic variation and a second backbone.

W1 / Q1. Why does the class-value spread change under identical copies in Appendix C.4?

Response.{\color{green}\mathbf{Response.}} Your invariance argument is correct. The discrepancy arose in C.4's offline calculation of the class-value diameter. With deterministic independent encoding and the fixed class prototypes in Eq. (22), identical copies preserve HH, corresponding keys/values, and Dˉ\bar D. The corrected diagnostic and replay controls cover 800 base questions, three checkpoints, and three copy strengths: 2,400 groups and 7,200 pairs.

PathMaximum relative change: H / k / vRelative range of D̄: median / maximumBound exceedances / eligible pairs
Original C.4 export—6.33% / 19.54%Historical export
Corrected diagnostic + inference replay≤2×10⁻⁵ / ≤2×10⁻⁵ / ≤2×10⁻⁵≤0.005% / ≤0.02%0 / 7,200
Cached local features0 / 0 / 00 / 00 / 7,200
Deterministic FP32 recomputation≤10⁻⁶ / ≤10⁻⁶ / ≤10⁻⁶≤0.0005% / ≤0.002%0 / 7,200

The encode-once control gives exactly constant Dˉ\bar D. FP32 recomputation leaves at most 0.002% relative variation. All replay pairs satisfy ϵ<1\epsilon<1, and every measured drift lies within the bound. These controls confirm the expected fixed-prototype behavior after correcting the offline diameter calculation.


W2 / Q2. Natural semantic repetition and robustness on another backbone.

Response.{\color{green}\mathbf{Response.}} Your suggested comparison directly tests QSEF's motivation: the same useful information can appear in visually different forms. The natural evaluation uses 400 held-out, human-annotated source questions, with 100 each for equivalence, partial overlap, complementarity, and contradiction. Each category contains 50 image and 50 document sources. Equivalent pairs include viewpoint, OCR, and layout changes.

On the same 100 equivalent sources, evaluated with three seeds:

MethodOriginal accuracy (%)Exact-copy robust score (%)Semantic accuracy (%)Semantic − original (pp), 95% CISemantic robust score (%)Semantic flip (%)
PMA + relation head61.058.356.3-4.7 [-7.8, -1.6]54.311.0
Exact-copy dedup + PMA60.060.055.0-5.0 [-8.1, -1.9]53.710.0
Similarity dedup + PMA59.059.056.7-2.3 [-5.4, 0.8]53.710.7
QSEF65.064.764.0-1.0 [-3.2, 1.2]63.33.3

The robust score requires both inputs to be correct, and intervals cluster by source. QSEF reaches 63.3% joint correctness under semantic variation, exceeding both deduplication controls. InternVL3-8B shows the same pattern on the same natural sources:

MethodOriginal accuracy (%)Exact-copy robust score (%)Semantic accuracy (%)Semantic − original (pp), 95% CISemantic robust score (%)Semantic flip (%)
PMA + relation head65.363.361.7-3.7 [-6.8, -0.6]60.09.7
QSEF69.369.068.3-1.0 [-3.2, 1.2]67.73.0

The useful-information test uses 100 partial-overlap, 100 complementary, and 100 conflicting sources. “Unique-information gain” is full-input accuracy minus accuracy after removing necessary unique evidence. On Qwen2.5-VL-3B:

MethodPartial accuracy (%)Partial unique-information gain (pp)Complement accuracy (%)Complement unique-information gain (pp)Conflict accuracy (%)
PMA + relation head64.39.366.012.357.0
QSEF69.014.772.317.361.7

QSEF gains 17.3 pp from necessary complementary information and achieves 72.3% full-input accuracy. This shows that the correction preserves the contribution of useful additional evidence. Literal-copy robustness on the original InternVL MuirBench subset is:

BackboneMethodMuirBench questionsClean score (%)Copy robust score (%)Copy flip (%)
InternVL3-8BPMA + relation head146449.747.44.5
InternVL3-8BQSEF146451.4 ± 0.250.90.8

The copy robust score averages the original-and-perturbed correctness score over one, two, and four appended copies, as in Table 2. This supplies both the natural semantic test and the additional-backbone perturbation test you requested.

Reviewer_Xx4LOpenReview ↗

Reply to author

Thank you for the clarification and additional experiments. The response largely addresses my original concerns. I am willing to raise my rating to 3 once the two points below are clarified and the paper is updated accordingly. I would still appreciate clarification on two details.

  1. What caused the error in the diameter calculation? Does the correction affect any results beyond C.4, including C.5?
  2. In the new semantic test, are the visually different but equivalent inputs added alongside the originals, or do they replace them? Please name the data sources and provide one example.

Please include the corrections and new results in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer Xx4L: Diameter correction and semantic-test protocol

Thanks for your follow-up. Both clarifications and the supporting results{\color{blue}\text{Both clarifications and the supporting results}} are now included in Appendices C.4–C.6{\color{blue}\text{are now included in Appendices C.4–C.6}} of the revised manuscript.{\color{blue}\text{of the revised manuscript.}}

Q1. What caused the diameter error, and what does it affect?

Response.{\color{green}\mathbf{Response.}} The cause was missing within-class normalization in the offline diagnostic. Globally normalized, multiplicity-corrected attention weights gave V~rc=πrcVrc\widetilde V_{rc}=\pi_{rc}V_{rc}, where πrc\pi_{rc} is the class's attention mass. Changes in this mass with estimated multiplicities then appeared as changes in diameter.

The corrected calculation follows Eq. (22): compute each class barycenter using a within-class softmax of uncorrected query–key logits, then compute DD once on the original input for each question–checkpoint pair. The same DD is used for every copy count. C.4, Table 11 confirms exactly constant diameter with cached keys/values and zero bound violations across 7,200 paired checks.

The correction affects C.4's diameter-based bounds/utilization and C.5's normalized fusion drift. Training and model outputs are unchanged, as are accuracy, token-level JS divergence, and the accuracy-based failure boundary (Table 12). C.5, Table 13 gives the corrected drift at N=64N=64:

Cross-affinity binPrevious driftCorrected driftAccuracy drop (unchanged)
0.0250.1590.1570.94 pp
0.2050.4770.4716.69 pp
0.2850.8240.80712.72 pp

Q2. Are variants appended or substituted? What are the sources and an example?

Response.{\color{green}\mathbf{Response.}} We append one variant and keep all original inputs. The question, answer, other evidence, and semantic metadata stay fixed. Human annotators verify that both versions preserve all question-relevant facts.

The 100 questions come from the official splits of MuirBench (test, 50), DocHaystack (Test, 25), and InfoHaystack (Test, 25), as listed in C.6, Table 14.

Figure 7 shows an actual pair from this evaluation. It asks for Johnson & Johnson's 2012 year-end stock price, USD 69.48. The appended document rearranges the stock-price table into quarter columns and rows grouped by year. Its fonts and layout differ, while all prices are preserved. Table 16 gives the original and variant image IDs, construction, and preserved facts. Table 15 reports the 100-question results across three seeds on both backbones.

Reviewer_Ry3xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates robustness against artificial repetition for multi‑image and document reasoning. It presents Quotient‑Set Evidence Fusion (QSEF), which estimates question‑aware redundancy and adjusts attention logits via log‑multiplicity correction before pooling evidence into a fixed‑length prefix. Given ideal equivalence detection and consistent within‑class representations, QSEF achieves exact invariance to duplicated evidence; conditional drift bounds are derived for estimation imperfections. Experiments on filtered subsets of six multi‑image benchmarks with three vision‑language backbones yield top point estimates across all 18 comparisons.

Critical strengths
  1. The paper clearly distinguishes evidence frequency from evidence content and treats redundancy as question-dependent rather than relying on question-blind visual similarity.

  2. Correcting attention logits using the estimated log multiplicity provides a conceptually clean way to separate information relevance from occurrence frequency. The method retains all evidence units instead of selecting or deleting representatives.

Critical weaknesses
  1. Gap between the theoretical equivalence relation and its estimation on natural data. Exact invariance requires recovery of the true question-dependent equivalence classes and matching within-class representations. However, natural training examples mainly provide verified-copy positive pairs, while informative negative supervision is largely derived from synthetic ChartQA cards. It therefore remains unclear whether the required relation can be reliably estimated for natural near-duplicates, different viewpoints, OCR variations, or partially overlapping evidence.
  2. The theoretical results are highly conditional and do not directly guarantee robust final answers. The main bound controls the fused representation before readout and assumes bounded multiplicity error together with fixed within-class keys and values. Stability of the decoder additionally requires a useful Lipschitz constant, while unchanged greedy answers require a decision-margin assumption that is not empirically verified.
  3. The statistical evidence is weaker than the consistent gains wording suggests. Although QSEF achieves the best point estimate in all 18 multi-image comparisons, the paired confidence intervals cross zero for all three Mantis-Eval comparisons. Most document-transfer confidence intervals also include zero. In addition, the main tables report standard deviations only for QSEF, which makes it difficult to compare training variability across methods.
  4. Restricted evaluation scope and possible selection effects. The reported results are based on manually screened eligible subsets for which the answer is assumed to be invariant to evidence order and duplication. The retained subsets also differ across backbones. Although this matches the intended scope of the method, it limits conclusions about performance on the complete benchmarks and makes reproducibility dependent on the release of the selection manifest.
Questions for authors

As shown above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Ry3x (1/4): W1: Natural semantic redundancy

Thank you for the time you devoted to our paper and for your thoughtful comments on semantic generalization, statistical evidence, and evaluation scope. Below, we address each point with clarifications and the corresponding results.

W1. Does the learned relation generalize to natural semantic redundancy?

Response.{\color{green}\mathbf{Response.}} QSEF defines redundancy through question-relevant information. Equivalent units convey the same information needed by the question. The target relation keeps partial overlap, complementary facts, and contradictions separate. The correction −log⁡mi-\log m_i assigns mass at the information-class level while retaining every value (Eqs. (2)–(5)). This ties attention to useful information as evidence repeats.

The natural evaluation contains 400 held-out, human-annotated source questions: 100 each for equivalence, partial overlap, complementarity, and contradiction, balanced between images and documents. It includes viewpoint, OCR, and layout variation. Thresholds are fixed on development data. On Qwen2.5-VL-3B:

MethodAUROCAUPRCEquivalent recall (%)Partial FPR (%)Complement FPR (%)Conflict FPR (%)Question-switch joint accuracy (%)
PMA + relation head88.778.572.0 (216/300)20.0 (60/300)14.0 (42/300)19.0 (57/300)76.7 (138/180)
QSEF question-blind81.067.367.0 (201/300)26.0 (78/300)22.0 (66/300)21.0 (63/300)0.0 (0/180)
QSEF92.985.478.0 (234/300)13.0 (39/300)9.0 (27/300)14.0 (42/300)83.9 (151/180)

Recall and each FPR use 100 independent questions, with counts pooled over three seeds. The 60 question-switch pairs reuse these sources. Joint accuracy requires both opposite labels to be correct, directly testing whether the relation responds to the question. QSEF achieves 83.9% joint accuracy.

On the 100 equivalent sources, the downstream comparison is:

MethodOriginal accuracy (%)Exact-copy robust score (%)Semantic accuracy (%)Semantic − original (pp), 95% CISemantic robust score (%)Semantic flip (%)
PMA + relation head61.058.356.3-4.7 [-7.8, -1.6]54.311.0
Exact-copy dedup + PMA60.060.055.0-5.0 [-8.1, -1.9]53.710.0
Similarity dedup + PMA59.059.056.7-2.3 [-5.4, 0.8]53.710.7
QSEF65.064.764.0-1.0 [-3.2, 1.2]63.33.3

The semantic robust score requires both presentations to be correct. Flip measures text-answer changes, and intervals cluster by source. This natural test is evaluated separately from the MuirBench subset in Table 2. QSEF achieves 63.3% joint correctness under semantic variation, compared with 53.7% for exact-copy deduplication. Its 3.3% flip rate also shows that the benefit extends to visually different presentations of the same information.

AuthorsOpenReview ↗

Response to Reviewer Ry3x (2/4): W2: Representation and answer stability

W2. How do representation bounds connect to final answers?

Response.{\color{green}\mathbf{Response.}} The connection you ask about has two parts. Under ideal equivalence and identical within-class keys/values, Eq. (5) gives an identical decoder prefix. This ensures identical output distributions and deterministic greedy answers. With estimation or representation errors, Theorem 1 and Appendix D.14 bound fusion drift. Appendix D.16 translates this into a loss bound under its stated decoder-sensitivity assumptions.

The D.14 measurements below quantify stability with nonidentical representations. Entries report the median/P90 of per-pair RMS quantities over fusion queries. The direct bound uses the smaller valid branch for each query.

Setting / modelQuestions (pairs)Representation term, median / P90Mass term, median / P90Fusion drift, median / P90Direct bound, median / P90Drift / bound, median / P90Within-setting Spearman, 95% CI
Exact copy / QSEF800 (7,200)0.000 / 0.0000.020 / 0.0500.008 / 0.0240.020 / 0.0500.40 / 0.600.78 [0.72,0.83]
Natural semantic / QSEF100 (300)0.010 / 0.0270.023 / 0.0520.018 / 0.0430.034 / 0.0730.53 / 0.730.69 [0.57,0.78]
Natural semantic / PMA+head100 (300)0.016 / 0.0390.031 / 0.0670.032 / 0.0690.049 / 0.1040.65 / 0.850.61 [0.47,0.73]

Every measured drift falls within the bound. Summaries and drift/bound ratios are computed per pair before aggregation. On natural QSEF pairs, the median ratio is 0.53 and the rank correlation is 0.69, showing that the bound captures variation across the evaluated inputs.

For answer stability, we evaluate both inputs on the same original greedy prefix. Let gtg_t be the original top-two logit gap and δt\delta_t the maximum logit change across the vocabulary. If gt>2δtg_t>2\delta_t at every step, including EOS, induction ensures the same greedy sequence.

Setting / modelQuestions / pairsTeacher-forced CE changeJS (10⁻³ nats)Answer flip (%)All-step margin coverage (%)Flip within coverage (%)
Exact-copy / QSEF (MuirBench)1,476 / 13,284+0.0010.80.774.00.0
Natural semantic / QSEF (E1)100 / 300+0.0091.53.363.00.0
Natural semantic / PMA+head (E1)100 / 300+0.0386.911.032.00.0

The sufficient margin condition covers 74% of exact-copy pairs and 63% of natural QSEF pairs, with zero flips in both covered groups. CE/JS measure changes along reference-answer prefixes. The free-generation margin check connects those representation measurements to a precise answer-stability condition.

AuthorsOpenReview ↗

Response to Reviewer Ry3x (3/4): W3: Statistical evidence

W3. Statistical significance and training variability.

Response.{\color{green}\mathbf{Response.}} Table 1 reports higher mean scores in all 18 comparisons. To make the statistical strength of these gains explicit, Table 8 provides paired intervals against PMA. They exclude zero in 15/18 multi-image conditions and 4/18 document-transfer conditions. All three Mantis-Eval intervals include zero. Each interval is an individual, unadjusted comparison.

The table below addresses your request for baseline variability using PMA+relation head, which matches model capacity and supervision. Means and sample SDs use three training seeds. Paired 95% CIs cluster by source/base question, and scores follow each benchmark's native metric.

BackboneBenchmarkQuestionsQSEF mean ± SDPMA+head mean ± SDQSEF − PMA+head (pp), 95% CI
Qwen2.5-VL-3BMuirBench147651.2 ± 0.249.1 ± 0.32.1 [0.9, 3.2]
Qwen2.5-VL-3BMIBench241847.6 ± 0.446.0 ± 0.41.6 [0.7, 2.5]
Qwen2.5-VL-3BMMIU110851.5 ± 0.250.4 ± 0.31.1 [-0.1, 2.2]
Qwen2.5-VL-3BMIRB71461.8 ± 0.360.8 ± 0.41.0 [-0.5, 2.6]
Qwen2.5-VL-3BMantis-Eval19466.3 ± 0.364.9 ± 0.51.4 [-1.6, 4.5]
Qwen2.5-VL-3BBLINK105253.2 ± 0.351.7 ± 0.41.5 [0.2, 2.8]
LLaVA-OneVision-7BMuirBench145251.0 ± 0.349.4 ± 0.41.6 [0.5, 2.7]
LLaVA-OneVision-7BMIBench235648.9 ± 0.547.4 ± 0.51.5 [0.5, 2.4]
LLaVA-OneVision-7BMMIU108650.3 ± 0.349.1 ± 0.41.2 [0.0, 2.4]
LLaVA-OneVision-7BMIRB70253.8 ± 0.352.7 ± 0.41.1 [-0.5, 2.6]
LLaVA-OneVision-7BMantis-Eval19066.7 ± 0.365.8 ± 0.50.9 [-2.2, 4.0]
LLaVA-OneVision-7BBLINK102455.1 ± 0.453.9 ± 0.51.2 [-0.1, 2.5]
InternVL3-8BMuirBench146451.4 ± 0.249.7 ± 0.31.7 [0.7, 2.8]
InternVL3-8BMIBench239752.3 ± 0.250.8 ± 0.31.5 [0.6, 2.5]
InternVL3-8BMMIU109656.4 ± 0.355.5 ± 0.40.9 [-0.2, 2.0]
InternVL3-8BMIRB70857.4 ± 0.456.4 ± 0.41.0 [-0.4, 2.4]
InternVL3-8BMantis-Eval19271.0 ± 0.370.0 ± 0.31.0 [-1.9, 4.0]
InternVL3-8BBLINK104158.4 ± 0.557.5 ± 0.50.9 [-0.4, 2.2]

QSEF also has higher means in all 18 matched-head comparisons, with strictly positive interval lower bounds in 7/18. This control supports the benefit of multiplicity correction when capacity and relation supervision are matched. The paired intervals quantify the strength of each comparison, and the SDs make training variability directly comparable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.