acceptodds
Under review as a conference paper at ICLR 2027

Scalable Information Flow Distillation for Large Multimodal Language Models

Abstract

The application of Multimodal Large Language Models (MLLMs) in resource-constrained scenarios is significantly limited due to their enormous parameter size and computational complexity. Knowledge distillation aims to transfer knowledge from large models to smaller ones, striking a balance between performance and capacity. Recent works have gradually shifted their focus from response-based distillation to intermediate-layer-based distillation, with the core objective being to explore the intrinsic working mechanisms of MLLMs. In this paper, we propose a new distillation paradigm based on the information flow of MLLMs. Specifically, through systematic analysis, we reveal the information flow pathways between text, images and the last token (used for generation) as they pass through the intermediate layers. Consequently, we identify a three-stage workflow for MLLMs: understanding the text firstly, then processing the multimodal data and finally focusing on generating a response. Moreover, this information flow exhibits a proportional scaling property with respect to the number of layers across models of different sizes. Based on these findings, we propose a simple distillation scheme that compresses the information flow of large models into small models. Extensive experiments verify the effectiveness of the proposed method.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multimodal Large Language Models (MLLMs), Knowledge Distillation, Information Flow

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sJAQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes InfDistill, an attention-based knowledge distillation method for compact MLLMs. The authors divide model layers into three information-flow stages: text understanding, multimodal interaction, and response generation. Different attention regions are distilled at each stage. Experiments on Qwen1.5 and Qwen2.5 models across eight multimodal benchmarks show modest performance improvements.

Critical strengths
  1. Improving knowledge transfer to compact MLLMs is an important problem with practical value.
  2. The proposed stage-wise attention distillation method is simple and easy to implement.
  3. The paper includes many ablation studies.
Critical weaknesses
  1. The improvements over LLaVA-KD are small, ranging from −0.2 to +0.6 points.
  2. The novelty is limited. Attention distillation have already been studied in prior work. This paper mainly changes which attention regions are distilled at different layer depths.
  3. As a distillation paper, Table 1 compares against too few existing distillation methods. Even cited methods such as Align-KD and CompoDistill are not experimentally compared.
  4. The related work is incomplete. Several closely related studies on attention-based or token-interaction-based MLLM distillation are missing [1, 2, 3].
  5. The training efficiency is not supported by wall-clock time and memory measurements.
  6. All experiments are limited to the Qwen family. In particular, Figure 2 only studies small Qwen models, which is insufficient to support general claims about MLLMs.
  7. The use of “scalable” in the title is unclear and potentially misleading. The paper only observes approximately proportional stage boundaries across different sizes of Qwen models, but provides no evidence of computational scalability or generalization across model architectures.
  8. The presentation could be improved. Figures 2 and 3 do not clearly communicate the main idea, and the Introduction lacks an intuitive overview figure. Author-year citations are also frequently inserted directly into the surrounding text. The authors should consistently use \citep for citations. And the introduction is too long.

[1] Jang et al. “Token-level Response-visual Attention Guidance for Multimodal LLMs Knowledge Distillation.” ECCV 2026.

[2] Chen et al. “Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions.” ICML 2026.

[3] Yu et al. “Hide to See: Reasoning-prefix Masking for Visual-anchored Thinking in VLM Distillation.” arXiv 2026.

Questions for authors

Please see the weaknesses above.

Reviewer_oWGtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies and empirically validates a common layer-wise information flow pattern across MLLMs of different sizes within the same model family. Specifically, shallow layers primarily facilitate intra-textual information exchange; middle layers support information exchange within and across modalities; and deep layers aggregate textual information into the final token used for generation. Based on these observations, this work proposes a knowledge distillation method called InfDistill. InfDistill establishes layer correspondences between teacher and student models of different depths through linear scaling of layer indices, and aligns the student’s attention matrices with those of the teacher according to the corresponding information flow patterns. This work conducts distillation experiments on smaller models from the Qwen family and evaluates their performance on multimodal QA benchmarks. The results demonstrate that the proposed method improves the performance of smaller models on multimodel qa benchmarks.

Critical strengths
  1. The proposed information-flow-guided distillation approach is novel. The authors also provide empirical analyses to validate the identified information flow patterns and the effectiveness of the proposed distillation method.
  2. The proposed distillation method is simple yet effective, achieving competitive performance improvements through a single-stage distillation process.
Critical weaknesses
  1. Compared with existing distillation methods such as LLaVA-KD, InfDistill achieves relatively limited performance improvements.
  2. While distillation experiments on the Qwen family alone may be acceptable, restricting the information flow analysis to this family limits the generalizability of the findings. It remains unclear whether similar patterns hold across other model families.
  3. The paper adopts a fixed 25%-50%-25% layer partitioning scheme, but the ablation studies only examine the removal of individual stages. Experiments with alternative partitioning ratios are needed to assess the sensitivity to stage boundaries and justify the chosen scheme.
Questions for authors
  1. Do similar information flow patterns exist across other MLLM families, or might different model architectures exhibit distinct patterns?
  2. How sensitive is InfDistill to the choice of stage boundaries? Have the authors considered alternative partitioning ratios beyond the fixed 25%–50%–25% scheme?
Reviewer_5xMAOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Blocking four attention regions layer by layer on eight Qwen MLLMs (Image↛Image, Image↛Text, Text↛Last, Text↛Text) yields three stages — first ~25% intra-text flow, middle ~50% dense intra- and cross-modal exchange, last ~25% text → last-token — with the boundaries fixed at 25% and 75% of depth regardless of model size. Noise on flow attention costs more on MME than noise elsewhere, and injecting a 4B model's attention into a 0.5B model helps on flow regions but hurts on non-flow ones. InfDistill rescales teacher indices onto the student axis (Eq. 3), averages within radius 0.5 (Eq. 4), and distills A[T:T] / full matrix / A[A:T] per stage with cosine loss, in one distillation stage (PT → SFT → DFT) where LLaVA-KD needs two. The three students reach 59.1 / 65.7 / 62.7 average: +0.3 / +0.2 / +0.6 over LLaVA-KD.

Critical strengths
  1. The analysis licenses the method. Adding the intra-modal directions is what reveals early layers as intra-text flow, and the proportional-scaling result is quantitative enough to be used directly: teacher indices are rescaled linearly because the boundaries are a fraction of depth.

  2. Causal evidence, not correlation. The noise experiment carries a non-INF control, and flow-only injection gains while non-flow injection loses. That sign flip argues the identified regions are the ones worth transferring.

Critical weaknesses
  1. The gains are unmeasurably thin. +0.3 / +0.2 / +0.6 average over LLaVA-KD, with losses on precisely the benchmarks where sub-point gaps matter (GQA, MMB, MMBCN), and no seeds, error bars or significance tests anywhere. At this magnitude on MMB / MME / POPE the differences are routinely inside run-to-run noise, so "new state of the art" rests entirely on an AVG column that cannot be checked.

  2. InfDistill's mechanism is attention distillation on selected layers. CompoDistill (ICLR 2026), Align-KD (CVPR 2025) and VLsI (CVPR 2025) are all attention-distillation methods for MLLMs, all cited in Related Work, none in Table 1. Without them "optimal or competitive" cannot be judged, and since LLaVA-KD is not compute-matched, the one-stage-versus-two-stage efficiency argument is untested too.

Questions for authors
  1. (Clarification) Teacher and student differ in head count and head dimension — how exactly is sim(A_s, Â_t) computed, cosine per query row averaged over tokens, or on a head-merged flattened matrix? This decides whether anyone can reimplement it.
  2. (Clarification) Add CompoDistill, Align-KD and VLsI to Table 1 under the same recipe, plus a compute-matched LLaVA-KD run at equal GPU hours.
  3. (Clarification) Report mean ± std over at least three seeds, or a paired significance test, for the +0.2–0.6 claims, and recompute AVG over one fixed common subset (or add AVG-8).
  4. (Clarification) Three items: the coefficients on Lbase, Lkd, Linf; which teacher size produces Table 1; and whether Figure 2c's recipient is Qwen1.5-0.5B or Qwen2.5-0.5B, and how the injection gets past the dimension mismatch.
1 comment

Dear Reviewer 5xMA,

Could you check if the response has addressed your issues? Since the discussion is closing soon, your reply would be very important.

Best,

AC

Comments on the paper

2 comments

Dear Reviewers,

The authors have submitted their rebuttal. Please review it and provide further comments as appropriate.

Best,

Your AC

Dear Reviewer sJAQ,

Please check our Follow-up Response and let us know if it addresses your concern. We would be open and happy to receive any follow-up messages from you.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.