- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Quotient-Set Evidence Fusion (QSEF) for multimodal reasoning with redundant and complementary evidence. The central idea is to aggregate question-relevant semantic evidence classes rather than overweight facts that occur repeatedly. The paper also establishes duplication-invariance and perturbation results under explicit representation and estimation assumptions, with extensions to nonidentical representations and extreme repetition. Experiments across three backbones, six multi-image benchmarks, controlled evidence tasks, and document-reader transfer report improvements in accuracy and stability.
Critical strengths
S1. A well-motivated problem and an interpretable design. The paper addresses an important challenge in multimodal reasoning: repeated evidence can receive disproportionate attention, while complementary evidence should remain distinguishable. The question-conditioned multiplicity correction provides an intuitive mechanism for suppressing redundancy while preserving useful information.
S2. Substantive theoretical analysis with explicit assumptions. The paper establishes exact duplication invariance under ideal relation estimates and matching representations, and characterizes sensitivity to multiplicity-estimation errors. The extensions to nonidentical representations and extreme repetition help clarify the conditions and limitations of the proposed approach.
S3. A substantial empirical evaluation with informative controls. The evaluation covers three backbone models and multiple benchmarks, with comparisons against pooling and deduplication alternatives. The auxiliary relation-head baseline and component ablations help isolate the contribution of the proposed correction. Reporting multiple seeds, grouped bootstrap intervals, and efficiency measurements further strengthens the empirical evaluation.
Critical weaknesses
W1. The extreme-repetition measurements are insufficiently specified for independent reconstruction. Appendix C.5 states that the export contains percentage-valued accuracies per question–checkpoint record, while the underlying binary outcomes and trial counts are unavailable. The class-value tensors required to reconstruct the normalized fusion displacement are also unavailable. Consequently, the statistical meaning of each percentage, the appropriateness of averaging these records, and the uncertainty of the interpolated three-percentage-point failure boundary are difficult to assess. This concern specifically affects the quantitative stress-test conclusions; it does not establish that the main benchmark results are invalid.
W2. Evidence for generalization to natural semantic redundancy is limited. Natural-data supervision provides verified exact-copy positives, while detailed semantic-equivalence and complementary-evidence supervision is largely supplied by controlled cards. The card experiments and held-out templates can not fully establish that the learned relation handles naturally occurring alternative presentations of the same facts. Partial overlap and conflicting information are particularly relevant because erroneous merging could suppress necessary evidence. The paper should discuss whether its favorable stability–utility tradeoff persists under natural semantic variation.
W3. The experimental evidence supports a limited scope for generality and scalability. The benchmark results are obtained on eligible subsets, and several improvements are not statistically resolved: the Mantis-Eval intervals include zero for all three backbones, as do multiple document-transfer comparisons. Furthermore, the 1,000-document setting increases the retrieval pool while the reader receives at most five retrieved documents. It therefore does not test fusion over 1,000 evidence units. These qualifications do not negate the positive results, but should constrain claims about generality and scalability.
W4. The additional empirical insight provided by the extensions in Appendices D.14–D.16 is less clear. In particular, the paper does not sufficiently show whether the quantities governing stability under nonidentical representations are informative in realistic settings, or how the conditional loss analysis relates to the observed answer-level behavior. Clarifying this connection would help readers distinguish what the theory establishes under its assumptions from what the experiments demonstrate.
Questions for authors
Q1. Does each percentage represent repeated trials, a task-specific per-example soft score, or another aggregation? Please specify the scoring rule and denominator where applicable, and provide the predictions, scores, and tensors needed to reconstruct the endpoints. With presentations and checkpoints grouped by base question, how uncertain are the failure thresholds, and how sensitive are they to the three-percentage-point criterion and interpolation procedure?
Q2. Can the authors evaluate a human-annotated natural near-duplicate set that distinguishes semantic equivalence, partial overlap, complementarity, and contradiction for the given question? Please report relation errors, erroneous merging of complementary evidence, and downstream accuracy.
Q3. For the extension to nonidentical representations, could you provide a representative empirical analysis showing the magnitude of the bound’s main terms and how they relate to the observed fusion drift? Please also clarify which conclusions about answer stability are supported by the experiments and which remain conditional on additional decoder or decision-margin assumptions.
Q4. Could the authors add experiments with one or two VLMs released in 2026, such as suitable Qwen3.5 or Gemma 4 variants, or another recent small open-weight VLM, to assess whether QSEF’s gains generalize to newer backbones and strengthen the empirical evidence?
9 comments
Response to Reviewer JRoF (2/4): W2/Q2: Natural relations and useful evidence
W2 / Q2. Human-annotated natural relations and harm from suppressing useful evidence.
The four relation categories you suggest directly test whether QSEF preserves useful differences between evidence units. The human-annotated natural test contains 400 held-out sources, with 100 each for equivalence, partial overlap, complementarity, and contradiction. Each category is balanced between images and documents. On Qwen2.5-VL-3B:
| Method | AUROC | AUPRC | Equivalent recall (%) | Partial FPR (%) | Complement FPR (%) | Conflict FPR (%) | Question-switch joint accuracy (%) |
|---|---|---|---|---|---|---|---|
| PMA + relation head | 88.7 | 78.5 | 72.0 (216/300) | 20.0 (60/300) | 14.0 (42/300) | 19.0 (57/300) | 76.7 (138/180) |
| QSEF question-blind | 81.0 | 67.3 | 67.0 (201/300) | 26.0 (78/300) | 22.0 (66/300) | 21.0 (63/300) | 0.0 (0/180) |
| QSEF | 92.9 | 85.4 | 78.0 (234/300) | 13.0 (39/300) | 9.0 (27/300) | 14.0 (42/300) | 83.9 (151/180) |
Thresholds are fixed on development data. Each relation has 100 independent sources and 300 source–seed decisions. The 60 question-switch pairs reuse these sources. Natural variation gives an AUROC of 92.9 and a complementary FPR of 9.0%, compared with 96.0 and 6.4% on controlled cards (Table 6). These measurements quantify relation quality under the more varied natural presentations.
For the 100 naturally equivalent sources, answer-level behavior is:
| Method | Original accuracy (%) | Exact-copy robust score (%) | Semantic accuracy (%) | Semantic − original (pp), 95% CI | Semantic robust score (%) | Semantic flip (%) |
|---|---|---|---|---|---|---|
| PMA + relation head | 61.0 | 58.3 | 56.3 | -4.7 [-7.8, -1.6] | 54.3 | 11.0 |
| Exact-copy dedup + PMA | 60.0 | 60.0 | 55.0 | -5.0 [-8.1, -1.9] | 53.7 | 10.0 |
| Similarity dedup + PMA | 59.0 | 59.0 | 56.7 | -2.3 [-5.4, 0.8] | 53.7 | 10.7 |
| QSEF | 65.0 | 64.7 | 64.0 | -1.0 [-3.2, 1.2] | 63.3 | 3.3 |
To measure the effect of erroneous downweighting on answers, the diagnostic oracle mask zeros human-labeled non-equivalent edges and retains the other edges. Harm counts answers recovered by this intervention. It measures downstream consequences separately from relation FPR. The oracle mask is used only for this diagnostic.
| Method | Partial accuracy (%) | Partial unique-information gain (pp) | Complement accuracy (%) | Complement unique-information gain (pp) | Conflict accuracy (%) | Harm: partial / complement / conflict |
|---|---|---|---|---|---|---|
| PMA + relation head | 64.3 | 9.3 | 66.0 | 12.3 | 57.0 | — |
| Exact-copy dedup + PMA | 63.0 | 8.3 | 64.7 | 12.0 | 55.3 | — |
| Similarity dedup + PMA | 60.3 | 7.0 | 61.7 | 10.0 | 52.7 | — |
| QSEF | 69.0 | 14.7 | 72.3 | 17.3 | 61.7 | 4/300 (1.3%) / 3/300 (1.0%) / 3/300 (1.0%) |
| QSEF + oracle mask (diagnostic) | 70.3 | 15.0 | 73.3 | 17.7 | 62.7 | — |
Each non-equivalent category has 100 sources and 300 source–seed decisions. Unique-information gain is . QSEF achieves higher full-input accuracy and larger gains from necessary information. The oracle recovers 4/3/3 decisions for partial/complementary/conflicting evidence, corresponding to harm rates of 1.3%/1.0%/1.0%. Together with the semantic-variation results, this shows that QSEF preserves useful information while reducing sensitivity to repetition.
Response to Reviewer JRoF (3/4): W3: Generality and scalability
W3. Generality, statistical scope, and the 1,000-document setting.
The scalability result in Figure 4 concerns reader transfer as the retrieval pool grows to 100/200/1,000 documents. As you note, the reader receives at most five retrieved documents. QSEF compresses those inputs into a fixed 64-token evidence prefix. Relation computation scales quadratically with the number of reader inputs, so retrieval-pool size and fusion-input size have separate roles in this evaluation.
Table 1 evaluates eligible order/copy-invariant questions. In Table 8, paired intervals against PMA exclude zero in 15/18 multi-image comparisons and 4/18 document-transfer comparisons. All three Mantis-Eval intervals include zero. These intervals specify the statistical support for each reported gain.
To check whether backbone-dependent retained subsets drive the gains, the common-intersection comparison against PMA+head gives:
| Benchmark | Common questions | Qwen: original → common gain [common 95% CI] | LLaVA: original → common gain [common 95% CI] | InternVL: original → common gain [common 95% CI] |
|---|---|---|---|---|
| MuirBench | 1392 | 2.1 → 2.0 [1.0, 3.0] | 1.6 → 1.4 [0.4, 2.4] | 1.7 → 1.7 [0.7, 2.7] |
| MIBench | 2270 | 1.6 → 1.6 [0.7, 2.5] | 1.5 → 1.4 [0.5, 2.3] | 1.5 → 1.6 [0.7, 2.5] |
| MMIU | 1024 | 1.1 → 1.2 [0.0, 2.4] | 1.2 → 1.1 [-0.1, 2.3] | 0.9 → 0.9 [-0.3, 2.1] |
| MIRB | 670 | 1.0 → 0.9 [-0.7, 2.5] | 1.1 → 1.2 [-0.4, 2.7] | 1.0 → 1.0 [-0.6, 2.5] |
| Mantis-Eval | 178 | 1.4 → 1.6 [-1.4, 4.6] | 0.9 → 1.0 [-2.0, 4.0] | 1.0 → 0.9 [-2.1, 3.9] |
| BLINK | 960 | 1.5 → 1.4 [0.2, 2.6] | 1.2 → 1.3 [0.1, 2.5] | 0.9 → 1.1 [-0.1, 2.3] |
Intervals are source-grouped 95% CIs. Every point gain stays positive and changes by at most 0.2 pp on the common intersection. This supports the consistency of the improvement across the eligible subsets. Figure 4 separately demonstrates transfer to document reading with larger retrieval pools.
Response to Reviewer JRoF (1/4): W1/Q1: Stress-test scoring and thresholds
We appreciate the time and care you put into your review and your detailed suggestions on scoring, theory, and recent backbones. Our responses below provide the scoring definitions, quantitative analyses, and comparisons with a newer backbone for each point.
Response. The new evaluation gives an explicit scoring rule and denominator for the stress test you asked about. It uses one deterministic greedy answer per question, checkpoint, and repetition level, scored correct/incorrect. It contains 252 independent questions, three checkpoints, and seven levels N∈1,2,4,8,16,32,64: 5,292 binary outcomes. For a stratum with n questions,
Acc(N)=3n100∑s=13∑q=1n1answerq,s,N is correct.
Low/mid/high cross-affinity strata are 0.025–0.105, 0.125–0.185, and 0.205–0.285, with 90/72/90 questions. Counts and sample SDs make the aggregate accuracies reconstructible:
At N=2,4, correct counts equal their respective N=1 counts. N counts occurrences of the repeated class. Including the singleton gives N+1 inputs, up to 65. The table uses a separate binary scoring protocol from the continuous-score export underlying Table 12. Its explicit denominator, 3n, and per-seed counts allow direct reconstruction of the reported means and SDs.
For threshold uncertainty, bootstrap resampling groups all presentations and checkpoints by base question. The operational threshold is the first crossing of an accuracy drop τ. Resamples without a crossing are right-censored at >64.
Between the first bracketing levels Na,Nb, with drops da,db, define w=(τ−da)/(db−da). The two interpolation rules are Na(Nb/Na)w and Na+(Nb−Na)w. Both locate the high-affinity 3-pp crossing near 11–12 occurrences, while the mid-affinity upper interval extends beyond 64. The threshold is defined by the chosen accuracy-drop criterion. Its sensitivity is quantified by the 1/3/5-pp criteria and both interpolation rules. The trend also follows the mechanism: residual cross-affinity β makes the singleton degree 1+Nβ, increasingly suppressing rare evidence.