- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Dynamic Modality Scaling (VL-DyMS), a framework for addressing modality imbalance during vision-language model adaptation. The method employs frozen text-only and vision-only teachers to provide modality-specific distillation signals, with a dynamic controller adjusting their relative contributions based on the current learning state. It also introduces a three-branch training strategy to preserve correct student predictions and recover pretrained behavior when necessary. Experiments on five benchmarks with Qwen2.5-VL and LLaVA-1.5 show consistent improvements over pretrained models and standard LoRA fine-tuning.
Critical strengths
-
Interesting training framework. Combining modality-specific distillation with a dynamic weighting controller is a sensible approach. The additional pretrained-model reference and three-branch training strategy may help reduce degradation of previously learned capabilities during adaptation.
-
Consistent empirical improvements. The proposed method improves over the pretrained baseline across all five benchmarks on both evaluated backbones. The improvements on GQA are particularly noticeable, suggesting potential benefits for compositional visual reasoning.
Critical weaknesses
-
Limited methodological novelty. Dynamic modality rebalancing and unimodal knowledge distillation have already been investigated in prior work. The proposed method mainly combines these established ideas with a pretrained-model reference and applies them to LLM-centric VLMs. The paper does not sufficiently clarify which component represents a fundamental methodological advance over existing approaches.
-
Insufficient technical details about the dynamic controller. The controller is central to the proposed framework, but its implementation is not adequately specified. Although the learning-state variables are introduced in Eq. (4), their computation, the controller architecture, and the optimization procedure are not clearly explained. In particular, it is unclear how the controller learns to assign appropriate modality weights and whether these weights reliably reflect the actual learning needs of each modality.
-
Missing ablation studies and competitive baselines. The experiments only compare VL-DyMS against pretrained models and standard LoRA adaptation. There are no comparisons with fixed-weight distillation, alternative dynamic weighting strategies, or existing modality rebalancing methods. More importantly, the contributions of the dynamic controller, unimodal teachers, and three-branch training strategy are not isolated. Consequently, it is difficult to determine whether the improvements actually come from dynamic modality scaling or simply from additional distillation and pretrained-behavior regularization.
-
Questionable effectiveness of the unimodal supervision. The text-only and vision-only teachers are constructed by masking inputs rather than using independently trained unimodal models. Their predictions can be unreliable when the answer requires joint visual and textual reasoning, as reflected by their substantially lower accuracy on several benchmarks. Moreover, the vision-only teacher lacks the original question, making its supervision potentially ambiguous. The paper does not adequately explain why matching these incomplete predictions or representations should consistently improve multimodal alignment.
Questions for authors
-
How exactly is the dynamic controller implemented and optimized? Could the authors provide the detailed computation of the learning-state variables and the mechanism used to generate the modality-specific weights?
-
Can the authors provide ablations comparing dynamic scaling against fixed distillation weights, removing individual teachers, and disabling the three-branch training strategy? Which component contributes most to the reported improvements?
-
Since the vision-only teacher does not receive the question, how does the method ensure that its supervision remains relevant to the target question? Could unreliable unimodal predictions introduce conflicting gradients during distillation?
-
Why is dynamic modality distillation activated only when both the student and pretrained baseline produce incorrect predictions? How frequently does each branch occur during training, and how sensitive is performance to this routing strategy?
5 comments
Response to RQ2 and RQ3
Q2. Ablations comparing dynamic scaling, removing individual teachers, and disabling the three-branch strategy.
Thank you for this suggestion. Our rationale for evaluating each teacher individually, with the same pretrained model and on the same evaluation set, is to isolate the unimodal prior performance, as reported in Table 5. These results show that each modality carries different information across tasks. However, we agree that an ablation that isolates the effect of one teacher, while applying the same three-branch method, can clarify the effect of each modality on the multimodal optimization. We are planning to run these experiments and report them in the updated manuscript.
Q3. Relevance of the vision-only teacher and conflicting gradients.
Thank you for this question. The method cannot guarantee a relevant teacher answer, especially for the vision-only teacher, which does not receive the question. This is the main reason for adding the frozen pretrained reference. If the teachers degrade the student, the student starts failing samples that the pretrained model answers correctly, and these samples are routed to the recovery branch, which recovers the student toward the pretrained behavior. In addition, three design choices limit the effect of unreliable teachers and conflicting gradients:
-
Teacher outputs are teacher-forced on the target answer, so distillation matches distributions over the correct answer tokens rather than the teacher's own prediction.
-
The answer loss keeps a unit weight, while the total teachers distillation coefficient is at most , so the distillation terms cannot dominate the answer gradient.
-
Teacher disagreement is part of the learning state, and reduces the distillation budget when the answer loss increases.
We hope these responses clarify the concerns on the ablations and the role of the unimodal teachers. If this is not what the reviewer meant, we kindly ask the reviewer to clarify, so that we can address it properly before the end of the discussion period. We would be grateful if the reviewer would take these responses into account when updating the rating
Response to RQ4
Thank you again for your review. In this comment, we address Q4.
Q4. Why is dynamic modality distillation activated only when both are incorrect?
Thank you for this question. The routing keeps the optimization more efficient. If the pretrained baseline already performs well, so instead of optimizing the student to reach that performance again, VL-DyMS recovers it from the pretrained model. Thus, each training sample is routed to one branch:
-
Student correct: no further optimization is needed, so no optimizer update is applied.
-
Student incorrect, baseline correct: the student refers to the frozen baseline for recovery, through KL and answer-representation matching.
-
Both incorrect: the pretrained behavior provides no correct reference, so optimization is required, and the answer loss is combined with dynamically weighted text and vision distillation.
We thank the reviewer again for the constructive comments, which helped us improve the clarity of the controller and the design of the ablations. If any of our responses does not address what the reviewer meant, we kindly ask the reviewer to clarify, so that we can address it properly. We hope our responses address the reviewer's questions and we would be grateful if the reviewer would consider updating the rating.
Response to RQ1
We thank the reviewer for the constructive review, and for recognizing the training framework as a sensible approach and the improvements as consistent across all five benchmarks and both backbones. Because of the length limit, our response is posted in 3 comments. In this comment, we address Q1.
Q1. How exactly is the dynamic controller implemented and optimized?
Thank you for this question and highlighting the need for controller implementation clarification. We will update the below details on the manuscript as well:
1. Implementation. The controller πϕ is a 7→24→2 MLP with a Tanh hidden activation. Its outputs pass through Tanh and are multiplied by 0.25 to bound the learned adjustment, so the controller refines the loss-change prior rather than collapsing the weights onto one teacher:
uk=0.25tanh(MLPϕ(sk)).
The controller is used only in the third branch of the objective, when both the student and the frozen baseline are incorrect. Both backbones use the same controller architecture across the five benchmarks, and the controller is initialized for each run.
2. Computation of the learning-state variables. The controller receives the detached state
sk=sg[ℓt,k,ℓv,k,rt,k,rv,k,ra,k,ξk,gk].
ℓt,k and ℓv,k are the current text and vision distillation losses, and ℓa,k is the answer loss.
For m∈t,v,a, rm,k=(μm,k−1−ℓm,k)/(∣μm,k−1∣+10−6), where μm is an exponential moving average with decay 0.9. The relative change is zero before the first observation. The averages update after optimizer steps, with zero modality losses recorded for recovery.
ξk is the mean cosine dissimilarity of the two teachers' answer logits at matched positions (teacher disagreement).
gk is the cosine dissimilarity between the student's pooled text and vision representations (representation gap).
3. Mechanism that generates the modality-specific weights. Adding (−rt,k,−rv,k) to uk and applying softmax gives shares that sum to one:
(wt,k,wv,k)=softmax(uk−(rt,k,rv,k)),λm(k)=λKD,bk,wm,k,
with λKD=0.06 and bk=clip(0.75+0.25tanh(4ra,k),0.5,1). The bias −rm,k increases the share of the more slowly improving modality, and bk reduces the distillation budget when the answer loss increases. Each coefficient lies in [0,0.06], and their sum lies in [0.03,0.06].
4. Optimization. The controller is trained on the third-branch objective Lans+λt(k)LKDt+λv(k)LKDv. Since Lans does not depend on ϕ, its gradient comes only from the weighted distillation terms:
∇ϕ(λt(k)LKDt+λv(k)LKDv). AdamW updates ϕ at learning rate 10−4 with gradient-norm clipping at 1.0. Because bk is computed from the detached state and the shares sum to one, the total coefficient λKD,bk does not depend on ϕ. Thus, the controller cannot reduce both coefficients jointly, it can only reallocate the budget between the two modalities.
We hope this response clarifies the reviewer question. If this is not what the reviewer meant, we kindly ask the reviewer to clarify, so that we can address it properly before the end of the discussion period. We would be grateful if the reviewer would take these responses into account when updating the rating.