acceptodds
Withdrawn from ICLR 2027

Self-Steering Multimodal LLMs for Enhanced Knowledge Utilization via Dynamic Information Flow Modulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal Retrieval-Augmented Generation enhances Multimodal Large Language Models (MLLMs) with external knowledge, yet retrieved evidence is often underutilized during generation. Previous studies typically attribute this issue to insufficient attention allocation or conflicts between contextual and parametric knowledge, overlooking how retrieved information is propagated and preserved in MLLMs. In this work, we revisit this problem from the perspective of internal information flow across Transformer layers. Through layer-wise analyses, we reveal that context-grounded information is better captured in intermediate representations than in final-layer representations, suggesting that ineffective information propagation limits the preservation of retrieved knowledge across layers. To address this, we propose Dynamic Contribution Steering (DCS), a training-free framework that dynamically modulates the contributions of attention, MLP, and residual pathways according to token-level attention distributions and neuron activation statistics. By suppressing irrelevant or dominant internal signals and enhancing context-related representations, DCS improves the utilization of retrieved knowledge without requiring additional training, steering vectors, or extra decoding procedures. Furthermore, we introduce Knowledge Layer Amplification (KLA), which identifies intermediate layers with stronger context-grounded decoding capability and adaptively incorporates their representations during generation. Extensive experiments on MRAG benchmarks across various MLLMs show that the proposed approach improves context-grounded answer generation, establishing information-flow steering as an effective paradigm for improving knowledge utilization in MLLMs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal LLMs, Multimodal RAG, Knowledge Utilization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_92kEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how multimodal language models use retrieved knowledge during generation. It proposes Dynamic Contribution Steering (DCS), which adjusts the contributions of attention, feed-forward, and residual pathways using attention distributions and positive-activation ratios. Knowledge Layer Amplification (KLA) further combines a selected intermediate-layer representation with the final layer through entropy-based weighting. Experiments across several knowledge-intensive VQA benchmarks and model families report consistent accuracy improvements with near-baseline inference latency.

Critical strengths
  1. Broad empirical evaluation. The method improves performance across multiple models, benchmark settings, and answer formats. Additional evaluations with newer MLLMs and different retrievers strengthen the evidence that its benefits are not restricted to one configuration.

  2. Practical inference-time design. The approach requires no additional training and avoids a separate no-context backbone pass. Table 4 reports approximately baseline latency, making the combination of accuracy improvements and low measured overhead potentially useful.

Critical weaknesses
  1. The adaptive weighting rules are not sufficiently validated. Although the weights vary with the input, their mapping is fixed: greater relative attention to retrieved text increases the attention branch’s weight, while a higher positive-activation ratio reduces the FFN branch’s weight. The paper does not directly establish that these statistics indicate whether the corresponding updates help or harm evidence-grounded answering. In particular, attention to text does not establish its reliability, and positive-activation counts do not measure harmful parametric interference. The component ablations demonstrate practical gains, but matched fixed-weight controls and an isolated KLA-only evaluation are needed to substantiate the specific adaptive design.

  2. The evaluation leaves the method’s applicability and incremental benefit insufficiently established. Results are not separated by whether retrieved text contains the answer, is irrelevant, or conflicts with visual evidence. The retriever variations partially address retrieval quality but do not isolate these conditions, especially cases where preserving visual information is essential. Several gains over ALFAR are modest, including 0.55 and 0.72 percentage points on LLaVA-1.5 Human and Validation overall accuracy, making comparison consistency important. However, Section 4.1 describes baseline results as cited from ALFAR, whereas Appendix A6 describes re-implementation. The timing results are encouraging, but the measurement setup and inclusion of KLA-related computation also require clarification.

Questions for authors
  1. Mechanistic validation: Do attention ratios and positive-activation ratios predict whether the corresponding pathway update helps or harms evidence-grounded answering? How does the adaptive method compare with matched fixed weights, and what is KLA’s contribution when evaluated alone?

  2. Applicability: Can you separate results for answer-containing, irrelevant, and conflicting retrieved contexts? Does suppressing visually dominated updates remain beneficial when accurate answering depends on fine-grained visual evidence?

1 comment
AuthorsOpenReview ↗

Kind Reminder to Reconsider the Review Score

Dear Reviewer,

Thank you again for your time and constructive feedback. We sincerely hope that our responses have adequately addressed your concerns.

We would greatly appreciate it if you could take another look at our rebuttal and reconsider your overall evaluation and score in light of our clarifications. Please let us know if any concerns remain, and we would be happy to provide further clarification.

Thank you for your consideration.

Reviewer_fGjrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DCS, a training-free method that dynamically modulates the contributions of different pathways in the transformer to enhance the utilization of retrieved knowledge in multimodal RAG. It also uses Knowledge Layer Amplification to identify and incorporate knowledge from an intermediate layer by fusing its representation with that of the final layer before decoding. Evaluations and ablations show that it outperforms various baselines without affecting the model’s decoding speed.

Critical strengths
  • Reasonable framing of the problem: Revisiting poor utilization of retrieved context in terms of how information moves through different transformer pathways, beyond decoding alone, is a sensible perspective.
  • Broad empirical evaluation: Testing both free-form and multiple-choice answering across multiple MLLMs and datasets makes the evaluation comprehensive. Consistent gains over the baselines with a single forward pass demonstrate the practicality of the proposed method.
Critical weaknesses
  • Insufficient justification of the activation frequency proxy: Using the raw frequency of positive neurons as a proxy for parametric-knowledge dominance is not well justified. Its definition and interpretation need clarification and justification for other activations such as SwiGLU. The impact of activation magnitude is also important but is not considered here. Although interpreting FFNs as key-value memories is supported by prior work, FFNs also process contextual information, so their activation does not necessarily indicate parametric-knowledge dominance. More importantly, the paper does not show that rffnr_ffn actually tracks reliance on parametric knowledge. For example, it does not show that rffnr_ffn is higher when the model ignores the retrieved context than when it follows it. Some analysis like this is needed to justify many claims that are written in the method section.
  • Incomplete ablation and justification: The ablation study seems to be cumulative, and adding KLA gives the largest incremental improvement (especially in OK-VQA). Some other additional ablations such as KLA-only and controls such as fixed-coefficient modulation or fixed-layer fusion for different backbones and datasets (I think at least 3 model and 3 dataset needed) are needed to validate the benefit of proposed mechanisms.
  • Insufficient comparison with closely related work: CaLE (Yuan et al., ACL 2025) already shows that contextual information can be most visible in intermediate layers and improves context use through unsupervised layer selection and representation amplification, with a kind of similar residual-stream analysis. A clearer distinction and a matched performance comparison against this and some other recent methods such as DoLa (Chuang et al., ICLR 2024) are needed to demonstrate the contribution and novelty.
Questions for authors

My questions are experiments and analysis needed to address the above weaknesses.

I also have one other question:

  • Line 411: How were the “last 5 layers” chosen, and on which split? How sensitive are the results to this choice?

Suggestion: Adding qualitative results and an analysis of cases where the retrieved context conflicts with the image could strengthen the paper.

1 comment
AuthorsOpenReview ↗

Kind Reminder to Reconsider the Review Score

Dear Reviewer,

Thank you again for your time and constructive feedback. We sincerely hope that our responses have adequately addressed your concerns.

We would greatly appreciate it if you could take another look at our rebuttal and reconsider your overall evaluation and score in light of our clarifications. Please let us know if any concerns remain, and we would be happy to provide further clarification.

Thank you for your consideration.

Reviewer_ZsgNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the underutilization of retrieved knowledge in multimodal retrieval-augmented generation from the perspective of internal information flow across Transformer layers. The author observes that the context-grounded information is better captured in intermediate representations than in final-layer representations. Motivated by this observation, the author proposes two training-free techniques: Dynamic Contribution Steering (DCS), which adaptively reweights attention, FFN, and residual pathways based on attention distributions and neuron activation statistics; and Knowledge Layer Amplification (KLA), which selects an intermediate layer and fuses its representation with the final hidden state using entropy-based weighting. Experiments across multiple MLLMs and knowledge-intensive VQA benchmarks demonstrate consistent performance improvements with limited reported inference overhead.

Critical strengths
  • Interesting problem formulation and practical motivation. The paper addresses the limitation of multimodal retrieval-augmented generation: retrieved knowledge may be available in the input but not effectively utilized during generation. Investigating this problem through the internal representations and computation pathways of Transformers provides an interesting perspective beyond output-level decoding interventions.

  • Promising empirical results and inference efficiency. The proposed method consistently improves performance across several model architectures and datasets. The reported inference latency is considerably lower than that of methods requiring additional forward passes.

Critical weaknesses
  • The connection between the proposed modulation signals and knowledge utilization is insufficiently justified.
    DCS uses attention distribution and FFN activation to interpret the contextual and parametric knowledge contributions. The connections are largely heuristic in the paper. In Eq.7, it is not clear why the proportion of positively activated neurons can reliably indicate the strength of parametric knowledge. It is not necessarily interpreted as strong reliance, nor does it indicate a conflict with retrieved evidence. In Eq.3, the author averages attention weights separately over visual and contextual tokens. The resulting ratio measures relative per-token attention rather than the total attention allocated to each modality.

  • The proposed modulation changes the original residual computation, making the source of performance gain unclear. In Eq 5. and Eq 8., the additive residual updates are replaced by weighted combinations of the residual stream and module outputs. Eq 5. scales the residual contribution by (1−rattn)(1-r_{\mathrm{attn}}), rather than preserving the original residual stream with unit coefficient. Combined with Eq. 8, the resulting update differs substantially from the pretrained Transformer's original computation. Therefore, it remains unclear whether the improvements originate from the proposed modulation or from the changing magnitudes.

  • More experiments needed to validate the effectiveness of KLA. The ablation study of KLA without DCS is required, as well as comparisons against fixed-layer selection, random-layer selection, and simple average, or fixed-weight fusion.

  • Some experiment details need to be claimed. In Figure 3, comparisons with the baseline like ALFAR would provide stronger evidence on modern architectures. The implementation details of Table 4 are insufficient. GPU memory overhead, decoding throughout would strengthen the contribution on efficiency.

Questions for authors

Refer to the weakness.

1 comment
AuthorsOpenReview ↗

Kind Reminder to Reconsider the Review Score

Dear Reviewer,

Thank you again for your time and constructive feedback. We sincerely hope that our responses have adequately addressed your concerns.

We would greatly appreciate it if you could take another look at our rebuttal and reconsider your overall evaluation and score in light of our clarifications. Please let us know if any concerns remain, and we would be happy to provide further clarification.

Thank you for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.