acceptodds
Under review as a conference paper at ICLR 2027

GainMasque: Auditing Masked Retrieval Gains and Repairing Evidence Integration

Abstract

Incorrect information can undermine content quality, veracity, and integrity, and these challenges are amplified when such information is shared within multimodal information streams. Retrieval-augmented detection methods usually incorporate external evidence to improve predictions, but retrieval gains can mask whether model predictions depend on the retrieved evidence. To address this issue, we propose GainMasque, an auditing and repair framework that disentangles retrieval gain from genuine evidence gain. First, GainMasque audits retrieval by comparing predictions with original and controlled evidence swaps, uncovering when retrieval gains remain without claim-conditioned evidence use. Then, we delineate systematic sources of masked retrieval gain, including retrieval and fusion mechanisms that rely on dataset, source, or label-correlated signals rather than evidence. To address these failures, we further introduce Decoupled Residual Evidence Fusion (DREF), which freezes the content model, corrects test-time shifts in the evidence score through bias alignment, and controls the evidence contribution using relevance- and agreement-based trust. GainMasque is backbone-agnostic and can be applied across multiple multimodal encoders and retrieval settings. Experimental results across diverse datasets and vision-language models show that retrieval gain often does not imply evidence gain, while DREF yields evidence gain with claim-conditioned scorers and bounds losses under noisy or corrupted retrieval. Together, GainMasque provides a systematic framework for auditing what drives retrieval gains and repairing how retrieval evidence shapes predictions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
retrieval-augmented learning, evidence attribution, evidence auditing, multimodal learning, vision-language models, incorrect information detection, misinformation detection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TzaMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether performance improvements from retrieval-augmented multimodal misinformation detection reflect genuine claim-specific evidence utilization. The authors introduce GainMasque, an auditing framework that decomposes retrieval gain (RG) into evidence gain (EG) and masked retrieval gain (MG) through same-label evidence swapping. The analysis identifies four sources of masked gains: label transfer, fusion shift, source style, and evidence availability. The paper further proposes Decoupled Residual Evidence Fusion, which combines base-referenced score alignment with relevance- and agreement-based trust. Experiments across multiple datasets and vision-language models demonstrate that retrieval gains frequently arise from shortcuts, while DREF improves robustness to corrupted evidence.

Critical strengths
  1. The paper addresses an important and underexplored evaluation problem. Distinguishing retrieval improvements from genuine claim-specific evidence utilization challenges the common assumption that better retrieval-augmented accuracy necessarily indicates effective evidence integration.
  2. The auditing framework is conceptually intuitive and supported by informative diagnostic controls. Same-label evidence swapping aims to preserve label-correlated signals while disrupting claim–evidence alignment. Additional interventions investigate label transfer, fusion effects, source style, and evidence availability, providing useful explanations for why retrieval improvements may persist without genuine evidence dependence.
Critical weaknesses
  1. The same-label evidence-swapping protocol does not guarantee that replacement evidence is semantically unrelated to the original claim. Matching labels, evidence sources, and availability does not exclude overlapping events, shared facts, or semantically relevant content. Consequently, the intervention may preserve some genuine claim-specific information or introduce unintended semantic conflicts, potentially biasing the estimated evidence gain.
  2. The interpretation of RG and EG as distinct sources of performance improvement requires stronger justification. RG measures the accuracy difference between original evidence and no evidence, which may include fusion shifts, evidence availability, and label-correlated shortcuts. EG measures sensitivity to a specific evidence swap rather than directly identifying genuine factual reasoning. Therefore, the algebraic decomposition of retrieval gain does not necessarily establish a clean causal separation between legitimate evidence contributions and spurious gains.
  3. The attribution of masked retrieval gains to specific mechanisms is not fully conclusive. Although the paper identifies four plausible mechanisms, several diagnostic interventions may alter multiple properties simultaneously. For example, content removal and length-matched shuffling in MOCHEG do not strictly isolate source style from claim-related semantic information. Thus, the experiments provide suggestive evidence of these mechanisms rather than a complete causal attribution of the observed gains.
Questions for authors
  1. Validity of same-label evidence swapping. How do the authors ensure that evidence selected from another claim with the same label is genuinely unrelated to the original claim? Matching labels, sources, and availability does not guarantee semantic independence.
  2. Sensitivity to evidence-swap construction. How sensitive are the reported EG and MG values to the selection of same-label replacement evidence?
  3. Validity and interpretation of RG and EG. Since RG includes all effects introduced by evidence availability and fusion, while EG measures the accuracy change under a particular swap intervention, to what extent can these quantities be interpreted as distinct causal contributions? Could the authors provide stronger controls or theoretical justification showing that EG reliably reflects claim-specific evidence utilization rather than intervention-induced distribution shifts?
  4. Causal attribution of masked retrieval gains. The paper identifies label transfer, fusion shift, source style, and evidence availability as four sources of masked gains. However, these factors may coexist and interact within the same retrieval system. Can the authors provide more controlled factorial experiments or quantitative attribution analyses to determine the independent contribution of each mechanism to the observed retrieval gains?
Reviewer_PsqDOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether performance improvements from RAG multimodal retrieval performance in misinformation domain genuinely arise from claim-specific evidence or from incidental retrieval-related signals. GainMasque is an auditing framework that separates retrieval gain (RG) from evidence gain (EG) through same-label evidence swapping, preserving label-level properties while disrupting claim–evidence correspondence. Under controlled perturbations, the authors identify several sources of masked retrieval gain, including label transfer, fusion effects, source style, and evidence availability.

To address these shortcomings, the paper further introduces Decoupled Residual Evidence Fusion (DREF), which maintains a frozen base predictor and regulates evidence contributions through some proposed score alignment. Experiments demonstrate that retrieval improvements frequently persist without corresponding evidence-dependent gains. DREF is further shown to preserve useful retrieval improvements while reducing prediction degradation under several noisy and adversarial evidence conditions.

Critical strengths
  1. Meaningful distinction between retrieval improvement and evidence utilization.
  • I find the separation of retrieval gain from evidence gain to be an interesting and useful contribution.
  • The controlled same-label perturbations is a refreshing approach to interrogating whether improvements arise from the claim-evidence correspondence or incidental signals introduced through retrieval.
  • The systematic identification of shortcut mechanisms across datasets and retrieval configurations is indeed informative, as it highlights an important limitation in how existing RAG systems are typically evaluated.
  1. Principled formulation for regulating evidence contributions.
  • The DREF framework is also interesting, particularly its treatment of evidence as a residual update to a frozen predictor and the explicit formulation of trust through agreement and relevance coefficients.
  • This provides some form of interpretable mechanism for controlling how external evidence influences predictions. The formulation for a trust coefficient through residuals is potentially novel, and I see potential relevance to broader questions of evidence reliability, model alignment, and the bigger discussion on AI safety and assurance.
  • However, these can only be further cemented if the underlying assumptions can be more rigorously established.
Critical weaknesses
  1. Limited generalizability (How can this extend to corpora where evidence are fragmented, or if the multimodal interactions between the image and text are significantly misaligned)
  • While the distinction between retrieval gain and evidence gain is interesting, the formulation remains narrowly tied to supervised claim–evidence verification.
  • Its applicability to real-world retrieval involving noisy, fragmented, or compositional evidence across documents and modalities remains unclear.
  • Furthermore, sensitivity to claim–evidence swapping does not necessarily establish genuine evidential reasoning. This limits the broader significance and generalizability of the contribution.
  1. Insufficient grounding of the trust mechanism.
  • DREF's trust formulation is intuitive, but its theoretical justification remains limited - An agreement with a frozen predictor does not imply correctness. Neither that info relevance guarantee evidence reliability. A example could be relevant, but its info could be misinformed, adversarial or incomplete.
  • The reliance on batch-level calibration also raises concerns under heterogeneous retrieval distributions. It remains unclear whether the reported robustness reflects genuine evidence discrimination or simply conservative suppression of evidence updates.
  • Larger batches seem to improve the stability of estimated statistics, but there are still systematic biases that may be aggregated across heterogenous domains.
Questions for authors
  • How do the authors view the applicability to real-world retrieval involving noisy, fragmented, or compositional evidence across documents and modalities remains unclear. Some eval sets to consider include - CRAG (Yang et al., 2024), RAGBench (2024), or MMLongBench-Doc (2024). Would same-label swapping remain a meaningful measure of evidence utilization in these settings?

  • Trust formulation and assumptions: DREF's trust formulation is intuitive, but its theoretical grounding appears limited to additive score alignment. What assumptions justify using agreement with the frozen predictor as proxies for evidence reliability? Are these assumptions expected to hold under heterogeneous or non-stationary retrieval distributions, particularly when correct evidence contradicts an incorrect base prediction? Also, how much of the observed robustness reflects genuine evidence discrimination versus conservative suppression of evidence updates? Finally, how much of the trust formulation holds valid under this evidence-claim setting and does it weaken if say I would want to apply cross-domain into for e.g. reasoning for embodied navigation under dynamic environments?

Reviewer_dvEEOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates whether performance gains from retrieved evidence in multimodal misinformation detection genuinely depend on claim–evidence correspondence. GainMasque compares original evidence, no evidence, and same-label evidence swaps to distinguish retrieval gain from evidence gain, and examines potential explanations for masked gains, including label transfer, fusion effects, source style, and evidence availability. The proposed DREF adds an admitted evidence residual to a frozen predictor, using base-referenced score alignment and batch-level relevance and agreement gates. The audit covers six datasets, while full dual-trust repair is evaluated primarily on MOCHEG and AVeriTeC. Additional experiments include a twelve-backbone study on MOCHEG and external-reader evaluations using adapted configurations.

Critical strengths
  1. The paper asks a relevant diagnostic question. Improvements over a no-evidence baseline do not, by themselves, establish claim-specific evidence use. Same-label evidence swapping provides a useful control for examining whether performance gains persist when the original claim–evidence pairing is disrupted while label-level signals are retained.

  2. The source-specific diagnostics, component ablations, and MOCHEG backbone study help characterize how evidence fusion can improve clean accuracy while increasing vulnerability to misleading evidence. These analyses could be useful for evaluating retrieval-augmented detectors.

Critical weaknesses
  1. The main experimental tables primarily compare internal variants of the proposed approach. Although Appendix Table 14 includes existing systems on AVeriTeC, this comparison is limited to one seed and does not provide systematic evaluation against representative published methods across both principal datasets. Consequently, the empirical advantages and relative competitiveness of the proposed method remain unclear.

  2. The audit uses same-label evidence swaps to investigate whether retrieval gains reflect genuine claim–evidence alignment. However, evidence retrieved for another claim may still contain information relevant to the current claim, making it difficult to distinguish legitimate evidence reuse from gains driven by label-correlated shortcuts. More importantly, even accepting the authors' interpretation of the audit results, DREF controls the influence of evidence without explicitly disentangling claim-specific information from shortcut signals. On MOCHEG, residual updates from an evidence-only scorer are regulated by a shared batch-level gate. The reported median RG of +6.7 and EG of +0.6 percentage points across twelve backbones using gold evidence do not, by themselves, establish that DREF improves claim-specific evidence use. Thus, the connection between the identified failure mode and the proposed remedy remains insufficiently demonstrated.

Questions for authors
  1. Could the authors provide the complete fitting objective, estimator, and implementation details for the agreement statistics (G(B)) and (G(C))? How sensitive are the resulting trust decisions to the estimation procedure and parameter choices?

  2. The agreement-based trust mechanism appears to use confident base-model predictions as a reference. How does DREF behave when the base predictor makes systematic, high-confidence errors? Could the authors evaluate whether the mechanism suppresses correct but conflicting evidence while retaining evidence that reinforces correlated mistakes?

  3. Why is a source that fails calibration disabled throughout testing rather than evaluated at the individual-claim level? Can the method retain useful evidence while rejecting misleading evidence from the same source, and how does it perform when source-level average gains conceal substantial variation in evidence quality across claims?

  4. The proposition establishes a result under a common additive shift with an otherwise unchanged joint distribution. What broader practical conditions justify the proposed score-alignment procedure, and what does it contribute beyond conventional test-time intercept calibration? How robust is the method when the shift assumption is violated?

Comments on the paper

1 comment

Dear Reviewers,

We are currently in the discussion phase. The authors have responded to the comments, and I would greatly appreciate it if you could take a moment to check the rebuttal and leave any follow-up comments or questions. If the response has addressed your concerns, please also consider updating your review accordingly.

Your feedback is very valuable to both the authors and the decision process. Thank you again for your help.

Best regards, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.