acceptodds
Under review as a conference paper at ICLR 2027

Cross-Modal Feature Dynamics Distillation for Resource-Constrained Multimodal LLMs

Abstract

Multimodal large language models (MLLMs) achieve strong performance on vision-language tasks, but their substantial computational cost limits deployment in resource-constrained settings. Knowledge distillation provides a promising route for transferring cross-modal reasoning knowledge from a large teacher to a compact student. Existing MLLM distillation methods, however, predominantly match static outputs or representations at isolated layers, overlooking the layer-wise evolutionary nature through which cross-modal reasoning is progressively formed. We introduce Cross-Modal Feature Dynamics Distillation (CM-FDD), which distills cross-modal grounding from two complementary perspectives: the grounding state at each depth and its evolution across selected depths. To enable this transfer across heterogeneous teacher-student models, we represent grounding with text-to-vision affinity, providing a model-comparable space that captures both where visual evidence is grounded and how that grounding changes over depth. When teacher and student use different visual resolutions, we further align their visual supports through deterministic cross-resolution projection. Experiments on ten benchmarks show that CM-FDD consistently outperforms prior MLLM distillation methods and achieves state-of-the-art performance under matched model-size settings, demonstrating the effectiveness of preserving grounding dynamics for efficient multimodal models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, Knowledge Distillation, Visual-Language Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_H5sROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies knowledge distillation for compact multimodal large language models. The authors argue that existing methods mainly match static outputs or intermediate representations and overlook how cross-modal grounding changes across layers. CM-FDD uses CMAD to align text–vision affinity states at selected layers and CMAED to align their layer-wise changes. It also uses CRRP to handle different visual resolutions between the teacher and student. Experiments across multiple benchmarks and model scales show consistent improvements over existing distillation methods.

Critical strengths
  1. The paper includes extensive experiments across multiple benchmarks and detailed ablations.
  2. The paper is well written, organized and easy to follow.
  3. The idea is simple and intuitive.
Critical weaknesses
  1. The contributions could be organized more clearly. CRRP is a simple interpolation operation and is not strong enough to stand as an independent contribution. CMAD and CMAED should be presented as the two main contributions, which would also match the abstract.
  2. The effectiveness and necessity of CMAED are unclear. CMAED alone reduces Avg10 from 58.0 to 57.9 and adds only 0.3 points over CMAD. Aligning affinity states at consecutive selected layers also seems to provide alignment of their differences.
  3. The teacher–student layer pairing appears highly handcrafted. Different layer pairs are manually selected for different models without a clear and general selection rule.
  4. The paper mixes the terms “feature,” “affinity,” and “grounding state,” which makes the method harder to follow. The overall method and its submodules also use different terms.
  5. Fig. 3 is difficult to understand quickly. Combining Figs. 2 and 3 into one conceptual figure may communicate the main idea more clearly in the introduction.
Questions for authors

Please see the weaknesses.

Reviewer_en53OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Cross-Modal Feature Dynamics Distillation (CM-FDD) for compact multimodal language models. It constructs text-to-vision affinity distributions from intermediate hidden states and transfers teacher knowledge through two objectives: matching affinity states at selected depths and matching the directions of affinity changes between successive checkpoints. A parameter-free bilinear projection accommodates different visual-token resolutions. The method is incorporated into a three-stage training pipeline without adding inference-time modules. Evaluation includes ten multimodal benchmarks, component and training-recipe ablations, grounding and visual-evidence deletion analyses, and inference-cost measurements.

Critical strengths
  1. A useful extension of intermediate distillation to cross-depth relational supervision. The paper considers not only whether teacher and student exhibit similar text–vision relationships at selected layers, but also whether these relationships change in similar directions across depth. Although it builds on existing relational and dynamics-based distillation ideas, applying this combination to cross-modal affinities is a reasonable and useful contribution. The formulation also accommodates different hidden dimensions and visual resolutions without introducing inference-time modules.

  2. Broad empirical evaluation with attempts to validate the proposed supervision signal. Beyond aggregate benchmark results, the paper includes component ablations, teacher-size and checkpoint-count analyses, multiple training seeds, grounding diagnostics, and deployment measurements. The localization and deletion experiments are particularly valuable because they examine whether the constructed affinities relate to visual evidence use. Taken together, these results support the practical value of the approach, although the mechanism-level interpretation remains less firmly established.

Critical weaknesses
  1. The interpretation of deep-layer affinity as spatial grounding is only partially established. At deeper layers, hidden states at image-origin positions can aggregate information from multiple visual regions. Similarity between an answer-token representation and such a state therefore does not necessarily indicate reliance on the corresponding original image patch. The localization and deletion analyses provide useful supporting evidence, but do not fully establish that this spatial interpretation remains valid across the supervised depths. This limits the stronger claim of transferring grounding or reasoning dynamics, rather than transferring useful cross-modal relational representations.

  2. The correspondence between teacher and student computational stages is insufficiently justified. The method uses fixed layer pairs, but the reported selections are not transparently derived from a reproducible relative-depth rule. Models with different capacities may perform corresponding transformations at different depths. Matching change directions accommodates differences in magnitude, but does not resolve this potential stage mismatch. The checkpoint-count ablation does not directly test sensitivity to checkpoint locations or teacher–student pairing. This matters because the incremental benefit of the evolution term is relatively modest in Table 3 (59.1 to 59.4), and its consistency across paired training seeds is not reported.

Questions for authors
  1. Clarification of layer selection: How were the exact layer pairs in Appendix B selected? Were they fixed in advance or chosen using validation performance? Please explain how these pairs follow the stated relative-depth criterion and how the selection rule would be applied to a new teacher–student architecture pair.

  2. Evidence for the grounding interpretation: At which layers are the affinity maps used in the localization and deletion experiments computed? Do you have evidence that their correspondence to original image regions remains reliable across the supervised depths? Please also clarify whether the interventions in Appendix F use local-mean replacement or zero masking, as the descriptions appear inconsistent.

  3. Evidence for the evolution term: Do you have matched-seed results comparing the full method with the state-only variant? Such results would help determine whether the improvement in Table 3 is reproducible and specifically attributable to the additional evolution constraint.

Reviewer_mxpDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers knowledge distillation from a large MLLM into a compact one for resource-constrained deployment. The key idea is that cross-modal grounding forms progressively across layers, so the student should match both the teacher's grounding at several depths and how that grounding changes from one depth to the next. Grounding is represented as a softmax over cosine similarities between answer-token and visual-token hidden states. CMAD aligns these distributions with KL, CMAED aligns the direction of their layer-to-layer differences, and a bilinear projection (CRRP) handles mismatched visual resolutions. Students at 0.5B and 1.5B (Qwen2.5 + SigLIP2) report state-of-the-art averages over ten benchmarks.

Critical strengths
  1. Text-to-vision affinity is computed inside each model's own space, so teachers and students with different hidden sizes can be distilled without learned adapters.
  2. The parameter-free bilinear CRRP lets a 224px student learn from a 384px teacher at zero added inference cost.
Critical weaknesses
  1. The benefit of evolution distillation is not convincingly demonstrated. CMAED alone performs slightly worse than the baseline (57.9 vs. 58.0), and adding it to CMAD improves Avg10 by only 0.3 points (Table 3). The paper does not establish whether this small gain is statistically significant. Although Appendix F shows better alignment of grounding dynamics, it does not isolate the contribution of CMAED.

  2. The SOTA comparison is confounded by different backbones. CM-FDD uses Qwen2.5 and SigLIP2, while many baselines use older architectures. Its DPT+SFT baseline already achieves 58.0 Avg10, and the full method improves this by only 1.4 points. Moreover, CM-FDD still underperforms Align-TI on VQAv2, GQA, and TextVQA. Comparisons using matched backbones are needed to establish the method's actual contribution.

Questions for authors
  1. Can the authors provides some significant measures,
  2. Can the authors use more SOTA baseline backbones to verify the effect is not from the better backbone, but the method?
2 comments

Q1 and W1: Significant measures of CMAED and its relationship to CMAD.

Our intended interpretation is that CMAD anchors the affinity states, while CMAED provides a complementary constraint on their cross-layer change directions. We do not claim that CMAED is universally beneficial as a standalone objective.

We agree that exact state matching at every selected layer would also imply matching their differences. However, practical distillation achieves only approximate state matching. Writing the state error as ek=ASk−ATke_k=A_S^k-A_T^k, we obtain

ΔASk−ΔATk=ek+1−ek.\Delta A_S^k-\Delta A_T^k=e_{k+1}-e_k.

Residual errors at consecutive layers can therefore affect the change direction, particularly when the teacher's change is small. Conversely, direction matching alone does not determine the absolute affinity states: a layer-independent offset can preserve the cross-layer differences. These observations motivate combining state anchoring with direction alignment.

We provide three complementary experiments.

Experiment 1: Reorganizing the ablation as an interaction analysis.

The existing ablation can be expressed as follows:

CMADCMAEDAvg10 (higher is better)
NoNo58.0
NoYes57.9
YesNo59.1
YesYes59.4

The observed gain from CMAED without state anchoring is

Gunanchored=57.9−58.0=−0.1,G_{\mathrm{unanchored}}=57.9-58.0=-0.1,

whereas its gain with CMAD is

Ganchored=59.4−59.1=+0.3.G_{\mathrm{anchored}}=59.4-59.1=+0.3.

The corresponding interaction contrast is

I=Ganchored−Gunanchored=+0.4.I=G_{\mathrm{anchored}}-G_{\mathrm{unanchored}}=+0.4.

This pattern is consistent with a complementary, conditional role for CMAED: its observed benefit arises when combined with state anchoring.

Experiment 2: Comparing second-half training configurations.

We additionally compare the following configurations in the second half of training:

Second-half training configurationAvg10 (higher is better)
CMAD59.1
Correctly paired CMAED59.4
CMAED with shuffled teacher-student layer pairing58.3

Correctly paired evolution supervision outperforms the CMAD continuation by 0.3 points, whereas shuffled pairing reduces performance by 1.1 points relative to the correctly paired configuration. This supports the importance of meaningful teacher-student correspondence: an arbitrary cross-layer constraint does not provide the same benefit.

Experiment 3: Measuring state and direction errors without retraining.

We evaluate the four existing models on the same held-out VQAv2 examples, using gold-answer tokens and a common set of evaluation layer pairs. We measure teacher-to-student affinity KL divergence and cross-layer direction error. We also report an active-transition direction error, restricted to transitions where both models exhibit non-negligible changes.

ModelState KLDirection errorActive direction error
Baseline0.0289081.0127851.017225
CMAD only0.0288930.9632550.966028
CMAED only0.0626860.7699720.769825
CMAD + CMAED0.0265680.7829250.785734

Lower values are better for all three metrics.

CMAED alone achieves strong direction alignment but has the largest state error. This illustrates the ambiguity left by direction-only supervision: matching how affinity changes does not ensure that the underlying affinity states match the teacher.

The combined model achieves the lowest state KL while retaining direction alignment close to CMAED alone. Compared with CMAD alone, it reduces state KL by approximately 8.0% and direction error by 18.7%. The active-transition metric shows the same trend.

These results support our interpretation of CMAED as a supplementary direction constraint whose role should be understood together with state anchoring.

Thank you the authors for the responses. While some points have been partially addressed, several concerns have not been solved:

  • The significance test of the small improvement (0.3) for CMAED has not been conducted. CMAED improves the alignment of grounding dynamics because it is the explicit objective of CMAED, however, it is not significantly shown that the alignment brings real improvement in the downstream.
  • The comparison should also be with other backbones of baseline methods. If baseline methods use stronger backbones, what happens?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.