- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies knowledge distillation for compact multimodal large language models. The authors argue that existing methods mainly match static outputs or intermediate representations and overlook how cross-modal grounding changes across layers. CM-FDD uses CMAD to align text–vision affinity states at selected layers and CMAED to align their layer-wise changes. It also uses CRRP to handle different visual resolutions between the teacher and student. Experiments across multiple benchmarks and model scales show consistent improvements over existing distillation methods.
Critical strengths
- The paper includes extensive experiments across multiple benchmarks and detailed ablations.
- The paper is well written, organized and easy to follow.
- The idea is simple and intuitive.
Critical weaknesses
- The contributions could be organized more clearly. CRRP is a simple interpolation operation and is not strong enough to stand as an independent contribution. CMAD and CMAED should be presented as the two main contributions, which would also match the abstract.
- The effectiveness and necessity of CMAED are unclear. CMAED alone reduces Avg10 from 58.0 to 57.9 and adds only 0.3 points over CMAD. Aligning affinity states at consecutive selected layers also seems to provide alignment of their differences.
- The teacher–student layer pairing appears highly handcrafted. Different layer pairs are manually selected for different models without a clear and general selection rule.
- The paper mixes the terms “feature,” “affinity,” and “grounding state,” which makes the method harder to follow. The overall method and its submodules also use different terms.
- Fig. 3 is difficult to understand quickly. Combining Figs. 2 and 3 into one conceptual figure may communicate the main idea more clearly in the introduction.
Questions for authors
Please see the weaknesses.
Q1 and W1: Significant measures of CMAED and its relationship to CMAD.
Our intended interpretation is that CMAD anchors the affinity states, while CMAED provides a complementary constraint on their cross-layer change directions. We do not claim that CMAED is universally beneficial as a standalone objective.
We agree that exact state matching at every selected layer would also imply matching their differences. However, practical distillation achieves only approximate state matching. Writing the state error as ek=ASk−ATk, we obtain
ΔASk−ΔATk=ek+1−ek.
Residual errors at consecutive layers can therefore affect the change direction, particularly when the teacher's change is small. Conversely, direction matching alone does not determine the absolute affinity states: a layer-independent offset can preserve the cross-layer differences. These observations motivate combining state anchoring with direction alignment.
We provide three complementary experiments.
Experiment 1: Reorganizing the ablation as an interaction analysis.
The existing ablation can be expressed as follows:
The observed gain from CMAED without state anchoring is
Gunanchored=57.9−58.0=−0.1,
whereas its gain with CMAD is
Ganchored=59.4−59.1=+0.3.
The corresponding interaction contrast is
I=Ganchored−Gunanchored=+0.4.
This pattern is consistent with a complementary, conditional role for CMAED: its observed benefit arises when combined with state anchoring.
Experiment 2: Comparing second-half training configurations.
We additionally compare the following configurations in the second half of training:
Correctly paired evolution supervision outperforms the CMAD continuation by 0.3 points, whereas shuffled pairing reduces performance by 1.1 points relative to the correctly paired configuration. This supports the importance of meaningful teacher-student correspondence: an arbitrary cross-layer constraint does not provide the same benefit.
Experiment 3: Measuring state and direction errors without retraining.
We evaluate the four existing models on the same held-out VQAv2 examples, using gold-answer tokens and a common set of evaluation layer pairs. We measure teacher-to-student affinity KL divergence and cross-layer direction error. We also report an active-transition direction error, restricted to transitions where both models exhibit non-negligible changes.
Lower values are better for all three metrics.
CMAED alone achieves strong direction alignment but has the largest state error. This illustrates the ambiguity left by direction-only supervision: matching how affinity changes does not ensure that the underlying affinity states match the teacher.
The combined model achieves the lowest state KL while retaining direction alignment close to CMAED alone. Compared with CMAD alone, it reduces state KL by approximately 8.0% and direction error by 18.7%. The active-transition metric shows the same trend.
These results support our interpretation of CMAED as a supplementary direction constraint whose role should be understood together with state anchoring.