- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates whether performance improvements from retrieval-augmented multimodal misinformation detection reflect genuine claim-specific evidence utilization. The authors introduce GainMasque, an auditing framework that decomposes retrieval gain (RG) into evidence gain (EG) and masked retrieval gain (MG) through same-label evidence swapping. The analysis identifies four sources of masked gains: label transfer, fusion shift, source style, and evidence availability. The paper further proposes Decoupled Residual Evidence Fusion, which combines base-referenced score alignment with relevance- and agreement-based trust. Experiments across multiple datasets and vision-language models demonstrate that retrieval gains frequently arise from shortcuts, while DREF improves robustness to corrupted evidence.
Critical strengths
- The paper addresses an important and underexplored evaluation problem. Distinguishing retrieval improvements from genuine claim-specific evidence utilization challenges the common assumption that better retrieval-augmented accuracy necessarily indicates effective evidence integration.
- The auditing framework is conceptually intuitive and supported by informative diagnostic controls. Same-label evidence swapping aims to preserve label-correlated signals while disrupting claim–evidence alignment. Additional interventions investigate label transfer, fusion effects, source style, and evidence availability, providing useful explanations for why retrieval improvements may persist without genuine evidence dependence.
Critical weaknesses
- The same-label evidence-swapping protocol does not guarantee that replacement evidence is semantically unrelated to the original claim. Matching labels, evidence sources, and availability does not exclude overlapping events, shared facts, or semantically relevant content. Consequently, the intervention may preserve some genuine claim-specific information or introduce unintended semantic conflicts, potentially biasing the estimated evidence gain.
- The interpretation of RG and EG as distinct sources of performance improvement requires stronger justification. RG measures the accuracy difference between original evidence and no evidence, which may include fusion shifts, evidence availability, and label-correlated shortcuts. EG measures sensitivity to a specific evidence swap rather than directly identifying genuine factual reasoning. Therefore, the algebraic decomposition of retrieval gain does not necessarily establish a clean causal separation between legitimate evidence contributions and spurious gains.
- The attribution of masked retrieval gains to specific mechanisms is not fully conclusive. Although the paper identifies four plausible mechanisms, several diagnostic interventions may alter multiple properties simultaneously. For example, content removal and length-matched shuffling in MOCHEG do not strictly isolate source style from claim-related semantic information. Thus, the experiments provide suggestive evidence of these mechanisms rather than a complete causal attribution of the observed gains.
Questions for authors
- Validity of same-label evidence swapping. How do the authors ensure that evidence selected from another claim with the same label is genuinely unrelated to the original claim? Matching labels, sources, and availability does not guarantee semantic independence.
- Sensitivity to evidence-swap construction. How sensitive are the reported EG and MG values to the selection of same-label replacement evidence?
- Validity and interpretation of RG and EG. Since RG includes all effects introduced by evidence availability and fusion, while EG measures the accuracy change under a particular swap intervention, to what extent can these quantities be interpreted as distinct causal contributions? Could the authors provide stronger controls or theoretical justification showing that EG reliably reflects claim-specific evidence utilization rather than intervention-induced distribution shifts?
- Causal attribution of masked retrieval gains. The paper identifies label transfer, fusion shift, source style, and evidence availability as four sources of masked gains. However, these factors may coexist and interact within the same retrieval system. Can the authors provide more controlled factorial experiments or quantitative attribution analyses to determine the independent contribution of each mechanism to the observed retrieval gains?
Dear Reviewers,
We are currently in the discussion phase. The authors have responded to the comments, and I would greatly appreciate it if you could take a moment to check the rebuttal and leave any follow-up comments or questions. If the response has addressed your concerns, please also consider updating your review accordingly.
Your feedback is very valuable to both the authors and the decision process. Thank you again for your help.
Best regards, AC