acceptodds
Under review as a conference paper at ICLR 2027

The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation

Abstract

While Multimodal Large Language Models (MLLMs) are increasingly integrated with Retrieval-Augmented Generation (RAG) to mitigate hallucinations, retrieved documents can conceal severe instance-level failures. We formalize *recorruption*, in which even perfectly accurate "oracle" context turns an initially correct prediction into an incorrect one. Our attention analysis associates recorruption with lower visual attention mass () and sharpness (), as well as attention concentrated at textual boundaries. This reveals an *Illusion of Success*. Successful RAG outputs and recorruption failures have similar measured attention profiles, while accuracy varies with the position of relevant evidence. This suggests that some correct outputs may benefit from favorable evidence placement, even when measured visual attention remains diffuse. To address these vulnerabilities, we propose **Bottleneck Attention Intervention for Recovery (BAIR)**, a training-free, inference-time framework that restores visual attention mass and sharpness while applying position-aware penalties to textual distractors. Across medical factuality, social fairness, and geospatial benchmarks, BAIR improves performance without model retraining or fine-tuning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Retrieval-Augmented Generation, Multimodal Large Language Models, Visual Grounding, Attention Intervention, Positional Bia

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sRRaOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates failure modes in multimodal RAG, with a focus on attention patterns. And have 3 observations: reduced attention to visual tokens; visual tokens become diffuse; and model accuracy appears to depend on position. The authors propose two inference-time adjustments: VSMR adjusts visual attention mass and the sharpness of the attention distribution over visual tokens, while PATP penalizes attention to textual tokens near context boundaries to mitigate positional bias.

Critical strengths
  1. The proposed method does not require retaining, making it easier to apply to existing MLLMs.
  2. Examining attention patterns in cases is a useful starting point.
  3. The experiments cover multiple domains, offering some evidence that the problem is not limited to a single application.
Critical weaknesses
  1. The proposed method is motivated by the attention pattern observed in the evaluated data. However, it is unclear whether these observations reflect general mechanisms or characteristics of the selected datasets.
  2. A sharp attention distribution does not necessarily indicate better visual grounding. Depending on the task or output token, relevant visual evidence may be distributed across multiple image regions. Strong visual attention is also unnecessary when predicting the token 'the'. This paper needs a clear justification for the motivation.
  3. Textual attention is influenced by more than token position. Different prompt types might impact it; the simple position-based method may suppress useful evidence.
  4. Position bias and weak visual attention have both been studied in prior work. The paper does not present novel observations, and the novelty of the proposed method is also limited.
Questions for authors
  1. Why does sharpness help? And does it help predict every kind of token?
  2. Is adjusting attention sharpness effective when the overall visual attention mass is already low? The rationale for this intervention is not intuitive and requires further justification.
  3. Is the observation that attending to textual tokens near context boundaries consistent across different prompts?
6 comments

Key Clarifications and Contributions

We thank the reviewer for the feedback. However, several concerns appear to stem from an interpretation of BAIR’s intervention scope that differs from the method defined in the manuscript. We therefore first clarify where BAIR operates and highlight its key contributions before addressing the individual comments.

  • BAIR intervenes during prefill only. It operates on the final input-token query, layerwise and headwise. This is a query that aggregates multimodal context before generation, not a generated word token. BAIR does not repeatedly modify attention for individual words during autoregressive decoding.

  • Sharpness does not require a single visual region or maximum concentration. It measures the distribution of attention within the visual tokens and can accommodate multiple separated peaks. The target is the reference sharpness measured for the same image-question input, not a universal sharpness value.

  • Visual mass and sharpness are calibrated jointly. VSMR adjusts both the total attention assigned to the image and its distribution within the image. It does not assume that sharpening a distribution alone compensates for insufficient visual mass.

  • The textual penalty is conditional and input dependent. Its two strengths are calculated separately from the current logits. A boundary region receives no penalty when its mean logit does not exceed the overall textual mean. PATP therefore requires neither unconditional boundary suppression nor a consistent attention profile across prompts.

  • The contribution extends beyond previously known attention biases. BAIR explicitly separates within visual sharpness from total visual mass and uses both as reference targets for calibration before softmax. This joint intervention differs from attention-output mixing and is evaluated across three domains and six model backbones, with additional knowledge-dependent VQA results.

Weakness 1: Generality of the observations

We evaluated three distinct domains and six model backbones specifically to examine whether the observations extend beyond an individual dataset or model. Section 3 analyzes retrieval-induced changes across multiple models, while the experiments test the intervention in medical, social, and geospatial settings. The empirical evidence therefore covers multiple domains and architectures rather than relying on a pattern observed in one selected dataset.

BAIR also does not encode a dataset-specific attention pattern. Reference mass and sharpness are measured separately for each image-question pair, and textual penalties are computed from the current input’s logits. Its application does not require different datasets to exhibit identical attention patterns. The cross-domain evaluation and the input-dependent formulation provide complementary evidence for its applicability across the studied settings.

Weakness 2 & Question 1: Sharpness, multiple relevant regions, and token prediction

Sharpness does not require attention to a single region. Equation 1 defines sharpness as the complement of normalized entropy over visual tokens. It imposes no requirement of spatial contiguity or a single peak. Attention can concentrate on several separated regions while remaining sharper than a distribution spread across the entire image. Figure 8 illustrates multiple peaks in the visual-token profile. Multiple relevant regions therefore do not contradict the proposed measure.

BAIR intervenes at the bottleneck query, not separately for each generated word. Section 4 specifies that the intervention operates on the final input-token query during prefill, layerwise and headwise. This query aggregates the multimodal context before generation begins. Subsequent decoding follows the standard computation. An objection based on separately imposing strong visual attention when generating a function word therefore does not describe the intervention performed by BAIR.

The target is the reference sharpness measured for the same image-question input. We do not claim that maximum sharpness is desirable or that higher sharpness guarantees a correct answer.

Table 2 and Appendix D explicitly examine this distinction. Sharpness recovery alone does not improve overall F1 on the evaluated subset, whereas combining mass and sharpness recovery performs better than mass recovery alone. Full BAIR achieves the highest F1 and lowest degradation rate among the evaluated component variants. These results support the joint intervention rather than a claim that sharpening alone improves every prediction.

Weakness 3: Textual attention depends on more than position

Textual attention depends on prompt and content as well as position. PATP does not assume otherwise. It targets positional overemphasis documented in prior work, including Found in the Middle by Hsieh et al. (2024), rather than treating all boundary content as irrelevant.

Section 4.2 calculates separate penalty strengths for the beginning and end of the text from the current input’s logits. A boundary receives no penalty when its average logit does not exceed the average across all textual tokens. Equation 6 defines the positional shape, while Equation 5 determines its input-dependent strength. The same token position can therefore receive different penalties under different prompts and contexts. Describing PATP as a fixed penalty determined solely by position overlooks this calibration.

PATP retains all textual tokens and documents. Its effect on useful evidence is assessed through correction and degradation in Table 1 and Appendix D. The additional OK-VQA results in Appendix O show that BAIR improves over Standard RAG on both evaluated backbones while preserving the aggregate benefits of retrieval. These results support the combined intervention’s ability to mitigate retrieval-induced failures while continuing to use external knowledge.

Weakness 4: Relationship to prior work and novelty

We do not claim that weak visual attention or positional bias was first discovered in this paper. Section 2.1 explicitly discusses these findings and MAD-RAG. Our contribution goes beyond these established observations.

To our knowledge, this is the first study in multimodal RAG to explicitly quantify within visual attention sharpness separately from total visual attention mass and use it as a reference based calibration target for retrieval induced visual suppression. This distinction matters because restoring the total attention assigned to an image does not determine how that attention is distributed across its visual tokens.

BAIR jointly calibrates mass and sharpness using references measured for each image-question pair and applies adaptive asymmetric textual penalties. The intervention operates on attention logits before softmax and is restricted to the final query during prefill. Appendix F.3 explains how this differs from MAD-RAG’s attention-output mixing.

The novelty assessment must therefore address the sharpness analysis and the resulting calibration mechanism. The previously known existence of visual suppression and positional bias does not, by itself, address these contributions.

Question 2: Sharpness recovery when visual mass is low

BAIR does not rely on sharpness recovery alone when visual mass is low. Addressing both quantities is precisely the purpose of VSMR. Writing visual attention as Ai=Mvis×piA_i = M_{\mathrm{vis}}\times p_i, MvisM_{\mathrm{vis}} controls the total attention assigned to visual tokens, while pip_i describes how that attention is distributed within the image. VSMR calibrates both quantities, as specified in Equations 2–4.

Equation 2 changes the distribution within the visual tokens to recover the reference sharpness. Equation 3 then adds the same offset to all visual logits, adjusting total visual mass while preserving the distribution obtained in the sharpness-calibration step. Equation 4 interpolates between the original and calibrated logits.

Thus, low mass is a reason to combine the two operations, not a contradiction of the method. Figure 8 illustrates recovery of both the visual allocation and its within-visual profile. Table 2 and Appendix D separately evaluate the interventions. The method's rationale is joint control of these two distinct quantities, not the proposition that sharpness alone is sufficient under arbitrarily low visual mass.

Question 3: Boundary attention across different prompts

We do not claim that boundary attention is consistently elevated across all prompts, and PATP does not require such consistency. Equation 5 calculates the two penalty strengths from the current input’s logits. Different prompts can therefore produce different penalty strengths, different asymmetries, or zero penalty at either boundary.

These quantities are calculated at the final-prefill query, not repeatedly for individual generated tokens during decoding. PATP responds to the attention profile of each input rather than requiring one common profile across prompts. Consistency across prompts is therefore not a prerequisite for the intervention.

I thank the authors for the detailed response. But my main concerns are only partially addressed. You explained that sharpness allows multiple peaks, but why is sharpness with multiple peaks better? Position-based textual penalty is determined by position. I know there are many papers that point out the importance of position. When you penalize it only based on position, you need a more careful and detailed analysis.

Reviewer_GutQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies retrieval-induced degradation, where adding textual context harms an initially correct visual prediction. It proposes BAIR, a training-free method that uses a no-context reference pass to adjust visual attention and reduce textual position bias. Experiments cover medical reporting, FACET pronoun-label agreement, and remote-sensing classification.

Critical strengths
  1. Useful instance-level evaluation. Reporting both correction and degradation reveals failures hidden by average accuracy.

  2. Broad practical testing. BAIR is evaluated across three domains, with component ablations, natural-retrieval subsets, and measured runtime overhead.

Critical weaknesses
  1. The incremental contribution needs stronger support. MAD-RAG already addresses retrieval-induced visual distraction through attention intervention. BAIR differs in its instance-specific targets and pre-softmax calibration, but the added benefit is uneven. On IU-Chest, MAD-RAG improves only from 68.03 to 68.25 when BAIR is added. On NWPU, four of five BAIR configurations remain below No RAG. Larger FACET gains are encouraging, but broader evidence is needed to justify the additional complexity.

  2. Reporting and evaluation validity require clarification. FACET/Qwen MS-PoE accuracy is 84.20 in Table 1 but 82.00 in Tables 6 and 9. The tasks also mainly favor visually available evidence, leaving performance on questions requiring external knowledge untested. Furthermore, binary pronoun agreement is a narrow measure and does not establish general fairness improvement.

Questions for authors
  1. What benefit does instance-specific restoration provide over matched fixed-weight attention mixing? Please quantify the improvement over MAD-RAG.

  2. When does BAIR justify using retrieval rather than No RAG? Does it preserve useful textual information when the image alone cannot answer the question?

  3. Please reconcile the FACET/Qwen MS-PoE results across Tables 1, 6, and 9. Also clarify why the FACET prompts describe the images as synthetic and how pronoun agreement supports the fairness claim.

4 comments

Weakness 1: Evidence beyond FACET and computational cost

The evidence supporting BAIR is not limited to the FACET gains. The main evaluation covers three domains and six model backbones. Appendix D provides component ablations, Appendix J evaluates naturally ordered retrieval, and the additional OK-VQA results in Appendix O demonstrate benefits on questions requiring external knowledge.

BAIR requires no training and restricts its intervention to the final input query during prefill. Appendix H reports computational overheads of 10.22% on IU-Chest/MedGemma and 5.37% on FACET/Qwen, including the reference pass. Its benefits are therefore supported across multiple settings, with an explicitly measured computational cost. Below, we address the distinction from MAD-RAG and the role of the No RAG comparison.

Weakness 1-1 & Question 1: Distinction from MAD-RAG and the magnitude of the gains

Mechanistic distinction. Appendix F.3 already gives the technical comparison. MAD-RAG uses a dual-question formulation and mixes image-conditioned and context-conditioned attention outputs with a mixing coefficient. BAIR operates before softmax. BAIR calibrates the visual logits using mass and sharpness targets measured separately for each image-question input, and applies an asymmetric, input-dependent textual penalty. Thus, a fixed output-mixing coefficient and an instance-specific attention-distribution target are different mechanisms. BAIR also confines its intervention to the final-prefill query rather than modifying every decoding step.

Quantitative comparison. Table 1 provides both standalone and combined evaluations under the same task setting:

SettingMAD-RAGStandard RAG + BAIRMAD-RAG + BAIR
IU-Chest / MedGemma68.0368.3168.25
FACET / Qwen2.5-VL84.4188.5685.10
NWPU / SkySenseGPT66.1866.3167.13

BAIR alone outperforms MAD RAG in all settings. For clarification, we report the best results obtained from the tested hyperparameter settings for every comparison method, including MAD-RAG. The advantage of BAIR is therefore measured against a tuned baseline, supporting the benefit of its distinct calibration mechanism.

Adding BAIR also improves MAD-RAG in all settings, showing that the methods can be complementary. However, their combination does not always achieve the best result, as BAIR alone performs better in the medical and FACET settings. This is consistent with partial overlap in how the two methods recover visual attention mass, which may limit the additional benefit of combining them.

Weakness 1-2 & Question 2: Comparison with No RAG and the role of the controlled setting

Whether a query needs external knowledge is generally unknown before generation. Applying retrieval uniformly is therefore a natural pipeline choice. Retrieved context can improve performance and enrich answers with useful details. The additional OK-VQA results (in Appendix O) also show that BAIR preserves the benefits of retrieval on questions requiring external knowledge.

However, retrieval does not improve every answer. As the NWPU results demonstrate, it can corrupt predictions that were correct without retrieval. Our work specifically investigates these recorrupted examples and how to mitigate them while retaining access to retrieved knowledge. The qualitative examples added in Appendix P illustrate both the additional detail provided by retrieval and the errors that BAIR can recover.

Standard RAG is therefore the direct baseline for evaluating the intervention, while No RAG provides a reference for measuring retrieval’s benefits and harms. Section 5 explicitly describes forced inclusion and placement of ground-truth evidence as a controlled diagnostic setting. Its purpose is to examine distraction even when relevant evidence is available. Appendix J additionally evaluates naturally ordered retrieval without forced ground-truth placement.

For NWPU/SkySenseGPT, BAIR improves Standard RAG but does not fully restore No RAG performance. This is partial mitigation, as already stated in Section 6. Our claim is that BAIR mitigates retrieval-induced degradation, not that every retrieval configuration augmented with BAIR must outperform No RAG.

Weakness 2-1 & Question 3: FACET/Qwen MS-PoE table discrepancy

We tune the hyperparameters of the comparison methods and report their best results among the tested settings. For MS-PoE, the initial rescaling setting of r=1.5r=1.5 yielded 82.00, while tuning identified r=1.2r=1.2 as the best setting, yielding 84.20. Table 1 was updated accordingly, but the accuracy entries in Tables 6 and 9 inadvertently retained the value from the initial setting. The correct accuracy is 84.20, and we will update those entries and specify the selected rescaling parameter.

Weakness 2-2 & Question 2: The role of external knowledge

External knowledge can be useful even when the image alone is sufficient to identify the primary subject or scene. In our setting, retrieval supports a more informative response by supplying background knowledge and descriptive details about the visually identified content. For example, identifying a land-use category and providing a detailed account of its characteristics are different requirements. The former may be answered visually, while the latter can benefit from retrieved knowledge.

Our controlled evaluation examines whether the model preserves that visual identity while incorporating additional textual information. The failure of interest occurs when retrieved context changes an otherwise correct visual interpretation instead of enriching the answer. The qualitative examples in Appendix P illustrate this distinction and how BAIR can recover the visual interpretation while retaining useful details.

Appendix O additionally evaluates OK-VQA, where external knowledge is required to answer the question itself. Retrieval improves Answer Recall on both evaluated backbones, and BAIR provides further gains over Standard RAG. The evaluation therefore addresses both knowledge-enriched descriptions of visually identifiable content and questions whose answers require external knowledge.

Weakness 2-3 & Question 3: Pronoun agreement as a fairness-relevant measure

The FACET evaluation targets a specific failure: a generated gendered reference inconsistent with the benchmark's perceived-gender annotation when occupational context is supplied. Gender-neutral profession descriptions do not provide evidence for replacing the person-related annotation with a different gendered reference. Measuring whether the model maintains that annotation-consistent reference is therefore a fairness-relevant test of demographic hallucination under occupational context, rather than an arbitrary text-matching criterion.

Section 5 explicitly defines pronoun-label agreement as the operational measure of demographic consistency under occupational context. This directly evaluates the fairness-related failure targeted by our study. Accordingly, pronoun-label agreement is a valid and appropriate metric for the stated research question, and improvements on it support our claim of mitigating demographic hallucination in this setting. Its focus on a specific dimension of fairness does not undermine its validity for evaluating that dimension. Our fairness claim concerns this benchmark-defined demographic consistency. We do not claim that pronoun agreement establishes fairness across all dimensions.

Question 3: Why the appendix prompt says “synthetic”

The “synthetic” wording comes from an example prompt tested before conducting the experiments reported in this work, using the dataset from Leveraging diffusion perturbations for measuring fairness in computer vision. The FACET experiments used real images and a prompt explicitly describing them as real. The example reproduced in Appendix E.2 therefore does not represent the prompt used for the reported FACET results. We will replace that listing with the actual experimental prompt and its corresponding configuration.

Thank you for the explanation. There are two remaining points that require further clarification.

First, regarding MS-PoE, the issue does not appear to be limited to an outdated accuracy figure in the appendix. Based on the paper’s definitions for the metrics, calculating the accuracy using the provided "No RAG" accuracy, CR, and DR values ​​yields approximately 81.99%, consistent with 82.00%, but not 84.20%. Please provide the full set of results (including accuracy, CR, DR, and other relevant metrics) corresponding to the final selected parameters. Additionally, please confirm that these results were derived from the same set of samples and explicitly state the final parameter values ​​used.

Second, while the new OK-VQA experiment is relevant to the question of whether external knowledge is necessary, I cannot locate "Appendix O" mentioned in your response. Please explicitly list the specific results, sample sizes, and uncertainty estimates for the "No RAG," "Standard RAG," and "BAIR" conditions. Furthermore, please clarify the scoring methodology for "Answer Recall" and the settings used for generation length, so that it may be determined whether these new experiments sufficiently address the question at hand.

Thank you for the careful check. We address both points below.

MS-PoE on FACET / Qwen2.5-VL-7B

Your calculation is correct. In our previous response, we corrected only the accuracy entry. The CR and DR entries in Table 1 and the remaining metrics in Tables 6 and 9 still came from the initial setting (r=1.5r = 1.5). The complete results for the final selected setting (r=1.2r = 1.2), recomputed from the per-sample outputs of that run, are as follows. They are computed on the same set of samples as all other methods in Tables 1 and 6.

MethodAcc.CRDRCR/DRRRSRNRGFR
MS-PoE (r=1.2r = 1.2)84.2028.315.874.8216.8531.9610.590.17

With the No RAG accuracy of 84.91, these values are consistent with the metric definitions: 84.91+0.2831×(100−84.91)−0.0587×84.91=84.20.84.91+0.2831\times(100-84.91) -0.0587 \times 84.91 = 84.20.

Final parameter values. For every comparison method, with and without BAIR, we evaluated all candidate hyperparameter values and report the best result, using the same candidate grid and selection criterion for every method. MS-PoE without BAIR uses r=1.2r = 1.2. MS-PoE + BAIR uses r=1.5r = 1.5 and αv=2.0\alpha_v = 2.0, and its reported results (accuracy 92.58) are unchanged. The interpolation coefficient αv\alpha_v is selected separately for each dataset, backbone, and base-method combination. The values stated in our earlier response (αv=0.5\alpha_v = 0.5 for IU-Chest and NWPU, αv=1.0\alpha_v = 1.0 for FACET) refer to Standard RAG + BAIR. In the uploaded revision, we have corrected Tables 1, 6, and 9 and Figure 5, and added a table listing the candidate grid and the selected hyperparameters for every configuration.

OK-VQA

Appendix O in the revised PDF uploaded on 13 Nov (page 26) reported results on a 300-question subset. We have since evaluated the full test split and have uploaded a further revision in which Appendix O now reports the full-split results. These supersede the earlier subset results. For convenience, we summarize them below.

Setup.

  • Data: the full M2KR OK-VQA test split (5,046 questions).
  • Context: no retriever is used. Standard RAG and RAG + BAIR receive the same benchmark-provided ground-truth documents (up to 5 per question, each truncated to its first 256 words). No RAG receives no documents. This setting therefore evaluates whether the model can use relevant external knowledge when it is available, not retriever quality.
  • Generation: greedy decoding (do_sample=False) with max_new_tokens=50 for all three conditions.
  • BAIR: αv=1.0\alpha_v = 1.0 for DeepSeek-VL-7B and αv=0.75\alpha_v = 0.75 for Qwen2.5-VL-7B, selected from {0.25, 0.5, 0.75, 1.0, 1.25, 1.5}.

Answer Recall. Predictions and annotator answers are lowercased, and punctuation and articles are removed. A question scores 1 if any annotator answer and the prediction are such that one is a substring of the other, and 0 otherwise. Empty or failed generations score 0. This matching is lenient, but it is applied identically to all three conditions, so it supports comparison between them.

Uncertainty. We use a paired bootstrap with 10,000 replicates, resampling by image cluster (5,033 clusters for 5,046 questions) so that questions sharing an image are resampled together.

BackboneNo RAGStandard RAGRAG + BAIRBAIR − Standard RAG [95% CI]
DeepSeek-VL-7B70.3183.8984.80+0.91 [+0.51, +1.35]
Qwen2.5-VL-7B63.4274.3476.62+2.28 [+1.31, +3.27]

Retrieval increases Answer Recall over No RAG for DeepSeek-VL-7B and Qwen2.5-VL-7B, respectively. BAIR further improves over Standard RAG on both backbones, and both paired intervals exclude zero. On a benchmark whose questions require external knowledge, BAIR therefore retains the benefit of retrieval and adds a small further gain.

Reviewer_kbf6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  1. This paper studies how retrieved text interferes with visual judgments in multimodal RAG and analyzes how this phenomenon relates to reduced visual attention and text position bias.
  2. The paper proposes BAIR, a training-free framework. It first obtains visual attention targets through a reference forward pass without retrieved context. It then restores the total mass and concentration of visual attention during prefill and applies adaptive positional penalties at text boundaries.
  3. The experiments show that BAIR improves overall performance and the trade-off between error correction and degradation across medical, social fairness, and remote sensing tasks, and can be combined with a range of existing methods.
Critical strengths
  1. The paper is very well written, logically coherent, and easy to read. Its experimental results and analyses are comprehensive, covering both aggregate and per-instance results and comparisons with recent work such as MAD-RAG, and provide useful information.
  2. The method requires no model retraining and intervenes only during the prefill stage. The authors also report runtime and GPU memory overhead, including the extra reference forward pass, making the practical cost easier to assess.
Critical weaknesses
  1. The baselines do not cover relevant work on visual attention interventions, such as VAR [1] and AIR [2]. Comparisons with these methods should be provided.
  2. The reproducibility of the experiments needs to be improved, for example, by providing the validation set splits and the number of evaluation runs.

[1] https://proceedings.iclr.cc/paper_files/paper/2025/hash/da8a39bc39ae1c89dd6ebb1e3bcbb3f3-Abstract-Conference.html

[2] https://proceedings.iclr.cc/paper_files/paper/2026/hash/83749a89d03df30c66af88020e4c7892-Abstract-Conference.html

Questions for authors
  1. Could the authors evaluate BAIR on tasks that require external knowledge, such as OK-VQA and InfoSeek, to verify that strengthening visual grounding does not impair the model's use of necessary retrieved information?
  2. Could the authors compare BAIR's mechanism with those of visual attention intervention methods such as VAR and AIR, and evaluate these methods under the same multimodal RAG settings?
  3. Could the authors provide the hyperparameter values, validation set splits, and number of repeated evaluation runs for each task, as well as confidence intervals for the paired performance differences?
2 comments

Weakness 1 & Question 2: Comparisons with VAR and AIR

We evaluated VAR and AIR alongside BAIR on the same 300-sample FACET subset described in the manuscript, using the same RAG setting.

MethodAccuracy
Standard RAG87.58
VAR87.25
AIR88.59
BAIR89.60
BAIR + VAR86.91
BAIR + AIR89.60

BAIR achieves the highest standalone score, exceeding VAR and AIR. Adding AIR to BAIR produces the same aggregate score, while combining VAR with BAIR reduces performance. These results show that the interventions are not automatically additive.

The mechanisms also differ. VAR redistributes attention from sink tokens to non-sink visual tokens in selected heads, while AIR reinjects selected visual information into the decoder’s feed-forward layers. BAIR instead calibrates pre-softmax attention logits using instance-specific reference mass and sharpness, together with adaptive textual penalties, exclusively at the final-prefill query.

Weakness 2: Reproducibility

An anonymized code package was included as supplementary material with the submission, covering preprocessing, retrieval-context construction, model inference, intervention, and evaluation. The manuscript specifies the intervention equations, task prompts in Appendix E, comparison methods in Appendix F, scoring rules in Appendix G, computational costs in Appendix H, and compute resources in Appendix M.

BAIR is training-free and does not update model weights. For the main benchmark experiments, all available examples in each task were evaluated without constructing separate training or validation splits. The selected visual interpolation coefficient was αv=0.5\alpha_v=0.5 for both evaluated backbones on IU-Chest and both on NWPU, and αv=1.0\alpha_v=1.0 for both backbones on FACET. The subset evaluations are identified separately in Appendices D and J, and Appendix K reports the repeated-evaluation summaries.

Question 1: Evaluation on knowledge-dependent VQA

We additionally evaluated BAIR on OK-VQA, which contains questions requiring knowledge beyond the image content. Appendix O reports Answer Recall for DeepSeek-VL-7B and Qwen2.5-VL-7B.

BackboneNo RAGStandard RAGRAG + BAIRBAIR − Standard RAG
DeepSeek-VL-7B72.88%82.33%83.47%+1.14 pp
Qwen2.5-VL-7B65.99%77.44%77.78%+0.34 pp

Retrieval substantially improves Answer Recall on both backbones, and BAIR further improves over Standard RAG. These results extend the evaluation beyond visually answerable tasks and demonstrate that BAIR’s visual calibration is compatible with benefiting from retrieved external knowledge in this setting.

Question 3-1: Hyperparameter selection

The visual interpolation coefficient, αv\alpha_v, is selected separately for each dataset/model combination. Among the evaluated settings, αv=0.5\alpha_v=0.5 achieved the best performance for both backbones on IU-Chest and both on NWPU, while αv=1.0\alpha_v=1.0 performed best for both backbones on FACET. No separate validation split was used for these reported results.

For a new setting, αv\alpha_v can be selected through inference on a small held-out set. Since BAIR is training-free, this requires only inference with candidate values, without retraining, backpropagation, or model-weight updates. Section 4.1 defines αv\alpha_v, and Appendix D examines its sensitivity.

The remaining settings are fixed. Appendix C specifies Tmax⁡=20T_{\max}=20 and a tolerance of 10−410^{-4} for the sharpness solver. Section 4.2 uses the first and last 20% of textual tokens to estimate the boundary penalties. Their strengths are computed dynamically from the current input’s logits and do not require separate tuning.

Question 3-2: Paired performance confidence intervals

We computed paired performance differences between Standard RAG and RAG+BAIR using matched evaluation examples. Scores are reported as percentages, and differences and confidence intervals are in percentage points.

DatasetModelStandard RAGRAG + BAIRPaired difference [95% CI]
FACETQwen2.5-VL-7B84.3488.56+4.22 [3.70, 4.93]
FACETDeepSeek-VL-7B90.8991.02+0.13 [−0.14, 0.42]
IU-ChestMedGemma-4B66.3368.31+1.98 [0.75, 3.12]
IU-ChestCheXagent-8B64.5666.17+1.61 [0.85, 2.37]
NWPUSkySenseGPT-7B65.7466.31+0.57 [0.17, 2.06]
NWPUEarthDial-4B91.2891.72+0.44 [0.07, 1.61]

All six point estimates are positive, and five of the six confidence intervals exclude zero. The FACET/DeepSeek improvement is small and its interval includes zero, so we do not claim a statistically significant improvement for that configuration. These paired intervals complement the method-wise mean and standard deviation summaries in Appendix K by directly quantifying uncertainty in the performance differences.

Thank you for the response

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.