- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper investigates failure modes in multimodal RAG, with a focus on attention patterns. And have 3 observations: reduced attention to visual tokens; visual tokens become diffuse; and model accuracy appears to depend on position. The authors propose two inference-time adjustments: VSMR adjusts visual attention mass and the sharpness of the attention distribution over visual tokens, while PATP penalizes attention to textual tokens near context boundaries to mitigate positional bias.
Critical strengths
- The proposed method does not require retaining, making it easier to apply to existing MLLMs.
- Examining attention patterns in cases is a useful starting point.
- The experiments cover multiple domains, offering some evidence that the problem is not limited to a single application.
Critical weaknesses
- The proposed method is motivated by the attention pattern observed in the evaluated data. However, it is unclear whether these observations reflect general mechanisms or characteristics of the selected datasets.
- A sharp attention distribution does not necessarily indicate better visual grounding. Depending on the task or output token, relevant visual evidence may be distributed across multiple image regions. Strong visual attention is also unnecessary when predicting the token 'the'. This paper needs a clear justification for the motivation.
- Textual attention is influenced by more than token position. Different prompt types might impact it; the simple position-based method may suppress useful evidence.
- Position bias and weak visual attention have both been studied in prior work. The paper does not present novel observations, and the novelty of the proposed method is also limited.
Questions for authors
- Why does sharpness help? And does it help predict every kind of token?
- Is adjusting attention sharpness effective when the overall visual attention mass is already low? The rationale for this intervention is not intuitive and requires further justification.
- Is the observation that attending to textual tokens near context boundaries consistent across different prompts?
6 comments
Weakness 3: Textual attention depends on more than position
Textual attention depends on prompt and content as well as position. PATP does not assume otherwise. It targets positional overemphasis documented in prior work, including Found in the Middle by Hsieh et al. (2024), rather than treating all boundary content as irrelevant.
Section 4.2 calculates separate penalty strengths for the beginning and end of the text from the current input’s logits. A boundary receives no penalty when its average logit does not exceed the average across all textual tokens. Equation 6 defines the positional shape, while Equation 5 determines its input-dependent strength. The same token position can therefore receive different penalties under different prompts and contexts. Describing PATP as a fixed penalty determined solely by position overlooks this calibration.
PATP retains all textual tokens and documents. Its effect on useful evidence is assessed through correction and degradation in Table 1 and Appendix D. The additional OK-VQA results in Appendix O show that BAIR improves over Standard RAG on both evaluated backbones while preserving the aggregate benefits of retrieval. These results support the combined intervention’s ability to mitigate retrieval-induced failures while continuing to use external knowledge.
Weakness 4: Relationship to prior work and novelty
We do not claim that weak visual attention or positional bias was first discovered in this paper. Section 2.1 explicitly discusses these findings and MAD-RAG. Our contribution goes beyond these established observations.
To our knowledge, this is the first study in multimodal RAG to explicitly quantify within visual attention sharpness separately from total visual attention mass and use it as a reference based calibration target for retrieval induced visual suppression. This distinction matters because restoring the total attention assigned to an image does not determine how that attention is distributed across its visual tokens.
BAIR jointly calibrates mass and sharpness using references measured for each image-question pair and applies adaptive asymmetric textual penalties. The intervention operates on attention logits before softmax and is restricted to the final query during prefill. Appendix F.3 explains how this differs from MAD-RAG’s attention-output mixing.
The novelty assessment must therefore address the sharpness analysis and the resulting calibration mechanism. The previously known existence of visual suppression and positional bias does not, by itself, address these contributions.
Question 2: Sharpness recovery when visual mass is low
BAIR does not rely on sharpness recovery alone when visual mass is low. Addressing both quantities is precisely the purpose of VSMR. Writing visual attention as , controls the total attention assigned to visual tokens, while describes how that attention is distributed within the image. VSMR calibrates both quantities, as specified in Equations 2–4.
Equation 2 changes the distribution within the visual tokens to recover the reference sharpness. Equation 3 then adds the same offset to all visual logits, adjusting total visual mass while preserving the distribution obtained in the sharpness-calibration step. Equation 4 interpolates between the original and calibrated logits.
Thus, low mass is a reason to combine the two operations, not a contradiction of the method. Figure 8 illustrates recovery of both the visual allocation and its within-visual profile. Table 2 and Appendix D separately evaluate the interventions. The method's rationale is joint control of these two distinct quantities, not the proposition that sharpness alone is sufficient under arbitrarily low visual mass.
Question 3: Boundary attention across different prompts
We do not claim that boundary attention is consistently elevated across all prompts, and PATP does not require such consistency. Equation 5 calculates the two penalty strengths from the current input’s logits. Different prompts can therefore produce different penalty strengths, different asymmetries, or zero penalty at either boundary.
These quantities are calculated at the final-prefill query, not repeatedly for individual generated tokens during decoding. PATP responds to the attention profile of each input rather than requiring one common profile across prompts. Consistency across prompts is therefore not a prerequisite for the intervention.
I thank the authors for the detailed response. But my main concerns are only partially addressed. You explained that sharpness allows multiple peaks, but why is sharpness with multiple peaks better? Position-based textual penalty is determined by position. I know there are many papers that point out the importance of position. When you penalize it only based on position, you need a more careful and detailed analysis.
Key Clarifications and Contributions
We thank the reviewer for the feedback. However, several concerns appear to stem from an interpretation of BAIR’s intervention scope that differs from the method defined in the manuscript. We therefore first clarify where BAIR operates and highlight its key contributions before addressing the individual comments.
BAIR intervenes during prefill only. It operates on the final input-token query, layerwise and headwise. This is a query that aggregates multimodal context before generation, not a generated word token. BAIR does not repeatedly modify attention for individual words during autoregressive decoding.
Sharpness does not require a single visual region or maximum concentration. It measures the distribution of attention within the visual tokens and can accommodate multiple separated peaks. The target is the reference sharpness measured for the same image-question input, not a universal sharpness value.
Visual mass and sharpness are calibrated jointly. VSMR adjusts both the total attention assigned to the image and its distribution within the image. It does not assume that sharpening a distribution alone compensates for insufficient visual mass.
The textual penalty is conditional and input dependent. Its two strengths are calculated separately from the current logits. A boundary region receives no penalty when its mean logit does not exceed the overall textual mean. PATP therefore requires neither unconditional boundary suppression nor a consistent attention profile across prompts.
The contribution extends beyond previously known attention biases. BAIR explicitly separates within visual sharpness from total visual mass and uses both as reference targets for calibration before softmax. This joint intervention differs from attention-output mixing and is evaluated across three domains and six model backbones, with additional knowledge-dependent VQA results.
Weakness 1: Generality of the observations
We evaluated three distinct domains and six model backbones specifically to examine whether the observations extend beyond an individual dataset or model. Section 3 analyzes retrieval-induced changes across multiple models, while the experiments test the intervention in medical, social, and geospatial settings. The empirical evidence therefore covers multiple domains and architectures rather than relying on a pattern observed in one selected dataset.
BAIR also does not encode a dataset-specific attention pattern. Reference mass and sharpness are measured separately for each image-question pair, and textual penalties are computed from the current input’s logits. Its application does not require different datasets to exhibit identical attention patterns. The cross-domain evaluation and the input-dependent formulation provide complementary evidence for its applicability across the studied settings.
Weakness 2 & Question 1: Sharpness, multiple relevant regions, and token prediction
Sharpness does not require attention to a single region. Equation 1 defines sharpness as the complement of normalized entropy over visual tokens. It imposes no requirement of spatial contiguity or a single peak. Attention can concentrate on several separated regions while remaining sharper than a distribution spread across the entire image. Figure 8 illustrates multiple peaks in the visual-token profile. Multiple relevant regions therefore do not contradict the proposed measure.
BAIR intervenes at the bottleneck query, not separately for each generated word. Section 4 specifies that the intervention operates on the final input-token query during prefill, layerwise and headwise. This query aggregates the multimodal context before generation begins. Subsequent decoding follows the standard computation. An objection based on separately imposing strong visual attention when generating a function word therefore does not describe the intervention performed by BAIR.
The target is the reference sharpness measured for the same image-question input. We do not claim that maximum sharpness is desirable or that higher sharpness guarantees a correct answer.
Table 2 and Appendix D explicitly examine this distinction. Sharpness recovery alone does not improve overall F1 on the evaluated subset, whereas combining mass and sharpness recovery performs better than mass recovery alone. Full BAIR achieves the highest F1 and lowest degradation rate among the evaluated component variants. These results support the joint intervention rather than a claim that sharpening alone improves every prediction.