acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Modality Scaling for Vision-Language Model Alignment

Abstract

Recent advances in vision-language models have achieved remarkable results in making language models understand visual inputs. However, a unified approach to aligning these models remains a challenge. In LLM-centric VLMs, language dominance is particularly plausible, because the decoder holds a strong prior over answers and explanations. Yet visual dominance is also possible in OCR or scene-heavy settings, where noisy patches overwhelm a simple question. The correct intervention must therefore be conditional, rather than globally image-boosting or text-suppressing. In this paper we propose dynamic modality scaling (VL-DyMS), a framework built on two frozen unimodal teachers and a multimodal student. Each teacher is obtained by masking one input: the text teacher sees only the question, the vision teacher only the image. The student is supervised by an answer loss and by a modality-aware distillation loss against each teacher. A dynamic controller adjusts the contribution of textual and visual knowledge at every step, so supervision follows the modality the student is currently underusing rather than a fixed schedule. We evaluate VL-DyMS on five datasets using Qwen2.5-VL and LLaVA-1.5. It shows an improved performance over the pre-trained baseline and standard LoRA adaptation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Multimodal Alignment, Modality Imbalance, Dynamic Modality Scaling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MauDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dynamic Modality Scaling (VL-DyMS), a framework for addressing modality imbalance during vision-language model adaptation. The method employs frozen text-only and vision-only teachers to provide modality-specific distillation signals, with a dynamic controller adjusting their relative contributions based on the current learning state. It also introduces a three-branch training strategy to preserve correct student predictions and recover pretrained behavior when necessary. Experiments on five benchmarks with Qwen2.5-VL and LLaVA-1.5 show consistent improvements over pretrained models and standard LoRA fine-tuning.

Critical strengths
  1. Interesting training framework. Combining modality-specific distillation with a dynamic weighting controller is a sensible approach. The additional pretrained-model reference and three-branch training strategy may help reduce degradation of previously learned capabilities during adaptation.

  2. Consistent empirical improvements. The proposed method improves over the pretrained baseline across all five benchmarks on both evaluated backbones. The improvements on GQA are particularly noticeable, suggesting potential benefits for compositional visual reasoning.

Critical weaknesses
  1. Limited methodological novelty. Dynamic modality rebalancing and unimodal knowledge distillation have already been investigated in prior work. The proposed method mainly combines these established ideas with a pretrained-model reference and applies them to LLM-centric VLMs. The paper does not sufficiently clarify which component represents a fundamental methodological advance over existing approaches.

  2. Insufficient technical details about the dynamic controller. The controller is central to the proposed framework, but its implementation is not adequately specified. Although the learning-state variables are introduced in Eq. (4), their computation, the controller architecture, and the optimization procedure are not clearly explained. In particular, it is unclear how the controller learns to assign appropriate modality weights and whether these weights reliably reflect the actual learning needs of each modality.

  3. Missing ablation studies and competitive baselines. The experiments only compare VL-DyMS against pretrained models and standard LoRA adaptation. There are no comparisons with fixed-weight distillation, alternative dynamic weighting strategies, or existing modality rebalancing methods. More importantly, the contributions of the dynamic controller, unimodal teachers, and three-branch training strategy are not isolated. Consequently, it is difficult to determine whether the improvements actually come from dynamic modality scaling or simply from additional distillation and pretrained-behavior regularization.

  4. Questionable effectiveness of the unimodal supervision. The text-only and vision-only teachers are constructed by masking inputs rather than using independently trained unimodal models. Their predictions can be unreliable when the answer requires joint visual and textual reasoning, as reflected by their substantially lower accuracy on several benchmarks. Moreover, the vision-only teacher lacks the original question, making its supervision potentially ambiguous. The paper does not adequately explain why matching these incomplete predictions or representations should consistently improve multimodal alignment.

Questions for authors
  1. How exactly is the dynamic controller implemented and optimized? Could the authors provide the detailed computation of the learning-state variables and the mechanism used to generate the modality-specific weights?

  2. Can the authors provide ablations comparing dynamic scaling against fixed distillation weights, removing individual teachers, and disabling the three-branch training strategy? Which component contributes most to the reported improvements?

  3. Since the vision-only teacher does not receive the question, how does the method ensure that its supervision remains relevant to the target question? Could unreliable unimodal predictions introduce conflicting gradients during distillation?

  4. Why is dynamic modality distillation activated only when both the student and pretrained baseline produce incorrect predictions? How frequently does each branch occur during training, and how sensitive is performance to this routing strategy?

5 comments
AuthorsOpenReview ↗

Response to RQ1

We thank the reviewer for the constructive review, and for recognizing the training framework as a sensible approach and the improvements as consistent across all five benchmarks and both backbones. Because of the length limit, our response is posted in 3 comments. In this comment, we address Q1.

Q1. How exactly is the dynamic controller implemented and optimized?

Thank you for this question and highlighting the need for controller implementation clarification. We will update the below details on the manuscript as well:

1. Implementation. The controller πϕ\pi_{\phi} is a 7→24→27 \rightarrow 24 \rightarrow 2 MLP with a Tanh hidden activation. Its outputs pass through Tanh and are multiplied by 0.250.25 to bound the learned adjustment, so the controller refines the loss-change prior rather than collapsing the weights onto one teacher:

uk=0.25tanh⁡(MLPϕ(sk)).\mathbf{u}_{k} = 0.25\tanh\left( \mathrm{MLP}_{\phi}\left( \mathbf{s}_{k} \right) \right).

The controller is used only in the third branch of the objective, when both the student and the frozen baseline are incorrect. Both backbones use the same controller architecture across the five benchmarks, and the controller is initialized for each run.

2. Computation of the learning-state variables. The controller receives the detached state

sk=sg[ℓt,k,ℓv,k,rt,k,rv,k,ra,k,ξk,gk].\mathbf{s}_{k} = \mathrm{sg}\left\lbrack \ell_{t,k},\ell_{v,k},r_{t,k},r_{v,k},r_{a,k},\xi_{k},g_{k} \right\rbrack.

ℓt,k\ell_{t,k} and ℓv,k\ell_{v,k} are the current text and vision distillation losses, and ℓa,k\ell_{a,k} is the answer loss.

For m∈t,v,am \in \\{ t,v,a\\}, rm,k=(μm,k−1−ℓm,k)/(∣μm,k−1∣+10−6)r_{m,k} = \left( \mu_{m,k - 1} - \ell_{m,k} \right)/\left( \left| \mu_{m,k - 1} \right| + 10^{- 6} \right), where μm\mu_{m} is an exponential moving average with decay 0.90.9. The relative change is zero before the first observation. The averages update after optimizer steps, with zero modality losses recorded for recovery.

ξk\xi_{k} is the mean cosine dissimilarity of the two teachers' answer logits at matched positions (teacher disagreement).

gkg_{k} is the cosine dissimilarity between the student's pooled text and vision representations (representation gap).

3. Mechanism that generates the modality-specific weights. Adding (−rt,k,−rv,k)\left( - r_{t,k}, - r_{v,k} \right) to uk\mathbf{u}_{k} and applying softmax gives shares that sum to one:

(wt,k,wv,k)=softmax(uk−(rt,k,rv,k)),λm(k)=λKD,bk,wm,k,\left( w_{t,k},w_{v,k} \right) = \mathrm{softmax}\left( \mathbf{u}_{k} - \left( r_{t,k},r_{v,k} \right) \right),\quad\quad\lambda_{m}^{(k)} = \lambda_{\mathrm{KD}}\\, b_{k}\\, w_{m,k},

with λKD=0.06\lambda_{\mathrm{KD}} = 0.06 and bk=clip(0.75+0.25tanh⁡(4ra,k),0.5,1)b_{k} = \mathrm{clip}\left( 0.75 + 0.25\tanh\left( 4r_{a,k} \right),0.5,1 \right). The bias −rm,k- r_{m,k} increases the share of the more slowly improving modality, and bkb_{k} reduces the distillation budget when the answer loss increases. Each coefficient lies in [0,0.06]\lbrack 0,0.06\rbrack, and their sum lies in [0.03,0.06]\lbrack 0.03,0.06\rbrack.

4. Optimization. The controller is trained on the third-branch objective Lans+λt(k)LKDt+λv(k)LKDv\mathcal{L}_{\mathrm{ans}} + \lambda_{t}^{(k)}\mathcal{L}_{\mathrm{KD}}^{t} + \lambda_{v}^{(k)}\mathcal{L}_{\mathrm{KD}}^{v}. Since Lans\mathcal{L}_{\mathrm{ans}} does not depend on ϕ\phi, its gradient comes only from the weighted distillation terms:

∇ϕ(λt(k)LKDt+λv(k)LKDv).\nabla_{\phi}\left( \lambda_{t}^{(k)}\mathcal{L}_{\mathrm{KD}}^{t} + \lambda_{v}^{(k)}\mathcal{L}_{\mathrm{KD}}^{v} \right). AdamW updates ϕ\phi at learning rate 10−410^{- 4} with gradient-norm clipping at 1.01.0. Because bkb_{k} is computed from the detached state and the shares sum to one, the total coefficient λKD,bk\lambda_{\mathrm{KD}}\\, b_{k} does not depend on ϕ\phi. Thus, the controller cannot reduce both coefficients jointly, it can only reallocate the budget between the two modalities.

We hope this response clarifies the reviewer question. If this is not what the reviewer meant, we kindly ask the reviewer to clarify, so that we can address it properly before the end of the discussion period. We would be grateful if the reviewer would take these responses into account when updating the rating.

AuthorsOpenReview ↗

Response to RQ2 and RQ3

Q2. Ablations comparing dynamic scaling, removing individual teachers, and disabling the three-branch strategy.

Thank you for this suggestion. Our rationale for evaluating each teacher individually, with the same pretrained model and on the same evaluation set, is to isolate the unimodal prior performance, as reported in Table 5. These results show that each modality carries different information across tasks. However, we agree that an ablation that isolates the effect of one teacher, while applying the same three-branch method, can clarify the effect of each modality on the multimodal optimization. We are planning to run these experiments and report them in the updated manuscript.

Q3. Relevance of the vision-only teacher and conflicting gradients.

Thank you for this question. The method cannot guarantee a relevant teacher answer, especially for the vision-only teacher, which does not receive the question. This is the main reason for adding the frozen pretrained reference. If the teachers degrade the student, the student starts failing samples that the pretrained model answers correctly, and these samples are routed to the recovery branch, which recovers the student toward the pretrained behavior. In addition, three design choices limit the effect of unreliable teachers and conflicting gradients:

  • Teacher outputs are teacher-forced on the target answer, so distillation matches distributions over the correct answer tokens rather than the teacher's own prediction.

  • The answer loss keeps a unit weight, while the total teachers distillation coefficient is at most 0.060.06, so the distillation terms cannot dominate the answer gradient.

  • Teacher disagreement ξk\xi_k is part of the learning state, and bkb_k reduces the distillation budget when the answer loss increases.

We hope these responses clarify the concerns on the ablations and the role of the unimodal teachers. If this is not what the reviewer meant, we kindly ask the reviewer to clarify, so that we can address it properly before the end of the discussion period. We would be grateful if the reviewer would take these responses into account when updating the rating

AuthorsOpenReview ↗

Response to RQ4

Thank you again for your review. In this comment, we address Q4.


Q4. Why is dynamic modality distillation activated only when both are incorrect?

Thank you for this question. The routing keeps the optimization more efficient. If the pretrained baseline already performs well, so instead of optimizing the student to reach that performance again, VL-DyMS recovers it from the pretrained model. Thus, each training sample is routed to one branch:

  • Student correct: no further optimization is needed, so no optimizer update is applied.

  • Student incorrect, baseline correct: the student refers to the frozen baseline for recovery, through KL and answer-representation matching.

  • Both incorrect: the pretrained behavior provides no correct reference, so optimization is required, and the answer loss is combined with dynamically weighted text and vision distillation.

We thank the reviewer again for the constructive comments, which helped us improve the clarity of the controller and the design of the ablations. If any of our responses does not address what the reviewer meant, we kindly ask the reviewer to clarify, so that we can address it properly. We hope our responses address the reviewer's questions and we would be grateful if the reviewer would consider updating the rating.

Reviewer_KPx4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VL-DyMS, a LoRA adaptation framework for vision-language models that combines a frozen multimodal reference model with frozen text-only and image-only teachers obtained by masking inputs to the same backbone. Training follows three branches: skip an update when the student is correct, recover the pretrained model's behavior when only the reference is correct, and otherwise combine answer supervision with dynamically weighted modality-specific distillation. The paper evaluates Qwen2.5-VL-3B and LLaVA-1.5-7B on TextVQA, GQA, TallyQA, POPE, and POPEv2. Table 4 reports improvements over the pretrained reference in all ten backbone–benchmark combinations, and improvements over answer-loss LoRA in nine combinations with one tie. Masked-teacher scores and qualitative examples provide additional diagnostics.

Critical strengths
  1. A relevant adaptation design that accommodates different directions of modality imbalance. The method separates visual and textual supervision rather than assuming that the visual modality should always receive more weight. Its pretrained-reference branch also explicitly addresses deterioration during adaptation (Sections 3.1–3.5, pp. 4–5). The resulting student requires no additional inference-time component, which makes the proposed intervention practically attractive if its training mechanism is validated.

  2. Encouraging results across two backbones and several types of visual tasks. Table 4 (p. 7) reports gains over the frozen baseline in all ten settings. For example, LLaVA GQA increases from 60.56 to 66.87, compared with 53.18 after ordinary LoRA; Qwen TextVQA increases from 80.00 to 81.67, compared with 75.57 after LoRA. The improvements extend to counting and object-hallucination evaluations, supporting the potential usefulness of the complete training recipe beyond one task.

Critical weaknesses
  1. The defining dynamic mechanism is not specified sufficiently to assess or reproduce it. Equation 3 introduces a trainable controller, but Sections 3.3–3.5 (p. 5) do not specify its architecture, optimization objective, output constraints, or update rule. Equation 4 gives a state vector without operational definitions for its components, including the representation gap, reliability-weighted confidences, and the remaining state terms. The representation-matching construction, distillation temperature, and the actual pretrained-recovery loss are also unspecified. This is consequential because optimizing unconstrained or independently shrinkable distillation weights directly through Equation 5 would favor suppressing those losses; whether the implemented controller avoids this depends on precisely the missing details. I am not asserting that the implementation exhibits this failure, but the stated method does not resolve it.

  2. The evaluation does not isolate dynamic scaling from the other changes in the training recipe. The only trained comparator in Table 4 is answer-loss LoRA. VL-DyMS additionally changes which examples receive updates, adds a pretrained-recovery branch, and introduces two distillation losses. There is no fixed-weight version with the same branches, or branch-only control, so the results cannot attribute the improvement to the paper's defining dynamic component. The masked-teacher accuracies in Table 5 do not measure changes in the adapted student's modality use; the authors correctly acknowledge this limitation in Section 5.3 (p. 9). With one training seed (Section 4.2), the reported small gains in some settings also do not establish their stability. Thus, the experiments support the overall recipe more strongly than the claimed benefit of dynamic modality rebalancing.

Questions for authors
  1. Method clarification: What are the exact controller architecture, state computations, weight constraints, and controller training objective? Please specify how the controller receives a learning signal and avoids simply reducing the distillation coefficients, together with the complete three-branch objective and the representation-matching construction. These details would determine whether the proposed mechanism can be assessed as an actual learning algorithm.

  2. Attribution of the main result: With the branches, teacher losses, and LoRA configuration held fixed, what changes when the modality weights are fixed rather than dynamically controlled? A comparison with the three-branch recipe without modality distillation would further distinguish recovery of pretrained behavior from the claimed rebalancing effect. Such evidence would directly affect my assessment of the main contribution.

  3. Evaluation clarification: What are the exact adaptation datasets and disjoint evaluation split names, training/evaluation example counts, scoring metrics, and vision-only prompts for each benchmark? Section 4.2 states that full standard evaluation splits are used, which I acknowledge, but does not identify these details. In particular, how is the image-only teacher prompted and scored on question-dependent yes/no tasks, where Table 5 reports 88.50 on POPE and 92.10 on POPEv2 for Qwen? Clarifying this is necessary to interpret the modality diagnostics and the adaptation protocol.

5 comments
AuthorsOpenReview ↗

Response to Q1

We thank the reviewer for the detailed and constructive review, and for recognizing that VL-DyMS accommodates different directions of modality imbalance and shows encouraging results across two backbones and several types of visual tasks. Our response to Q1 also addresses Weakness 1, and our response to Q2 addresses the attribution concern in Weakness 2. Because of the length limit, our response is posted in 3 comments. In this comment, we address Q1.

Q1. Method clarification.

Thank you for this question. We will update the manuscript to include the specifications, as clarified below.

1. Controller architecture. The controller πϕ\pi_{\phi} is a 7→24→27 \rightarrow 24 \rightarrow 2 MLP with a Tanh hidden activation. Its outputs pass through Tanh and are multiplied by 0.250.25 to bound the learned adjustment, so the controller refines the loss-change prior rather than collapsing the weights onto one teacher.

uk=0.25tanh⁡(MLPϕ(sk)).\mathbf{u}_{k} = 0.25\tanh\left( \mathrm{MLP}_{\phi}\left( \mathbf{s}_{k} \right) \right).

Both backbones use the same controller architecture across the five benchmarks, and the controller is initialized for each run.

2. State computations. The controller receives the detached state

sk=sg[ℓt,k,ℓv,k,rt,k,rv,k,ra,k,ξk,gk].\mathbf{s}_{k} = \mathrm{sg}\left\lbrack \ell_{t,k},\ell_{v,k},r_{t,k},r_{v,k},r_{a,k},\xi_{k},g_{k} \right\rbrack.

  • ℓt,k\ell_{t,k} and ℓv,k\ell_{v,k} are the current text and vision distillation losses, and ℓa,k\ell_{a,k} is the answer loss.

  • For m∈t,v,am \in \\{ t,v,a\\}, rm,k=(μm,k−1−ℓm,k)/(∣μm,k−1∣+10−6)r_{m,k} = \left( \mu_{m,k - 1} - \ell_{m,k} \right)/\left( \left| \mu_{m,k - 1} \right| + 10^{- 6} \right), where μm\mu_{m} is an exponential moving average with decay 0.90.9. The relative change is zero before the first observation.

  • ξk\xi_{k} is the mean cosine dissimilarity of the two teachers' answer logits at matched positions (teacher disagreement).

  • gkg_{k} is the cosine dissimilarity between the student's pooled text and vision representations (representation gap).

3. Weight constraints. Adding (−rt,k,−rv,k)\left( - r_{t,k}, - r_{v,k} \right) to uk\mathbf{u}_{k} and applying softmax gives shares that sum to one:

(wt,k,wv,k)=softmax(uk−(rt,k,rv,k)),λm(k)=λKD,bk,wm,k,\left( w_{t,k},w_{v,k} \right) = \mathrm{softmax}\left( \mathbf{u}_{k} - \left( r_{t,k},r_{v,k} \right) \right),\quad\quad\lambda_{m}^{(k)} = \lambda_{\mathrm{KD}}\\, b_{k}\\, w_{m,k},

with λKD=0.06\lambda_{\mathrm{KD}} = 0.06 and bk=clip(0.75+0.25tanh⁡(4ra,k),0.5,1)b_{k} = \mathrm{clip}\left( 0.75 + 0.25\tanh\left( 4r_{a,k} \right),0.5,1 \right). Each coefficient lies in [0,0.06]\lbrack 0,0.06\rbrack, and their sum lies in [0.03,0.06]\lbrack 0.03,0.06\rbrack.

4. Controller training objective and learning signal. The controller is trained on the third-branch objective. Since Lans\mathcal{L}_{\mathrm{ans}} does not depend on ϕ\phi, its gradient comes only from the weighted distillation terms:

∇ϕ(λt(k)LKDt+λv(k)LKDv).\nabla_{\phi}\left( \lambda_{t}^{(k)}\mathcal{L}_{\mathrm{KD}}^{t} + \lambda_{v}^{(k)}\mathcal{L}_{\mathrm{KD}}^{v} \right).

AdamW updates ϕ\phi at learning rate 10−410^{- 4} with gradient-norm clipping at 1.01.0. At a fixed state, this gradient favors the smaller current distillation loss, while the explicit bias −rm,k- r_{m,k} increases the logit of the more slowly improving modality.

5. How the controller avoids reducing the distillation coefficients. The total coefficient λt(k)+λv(k)=λKD,bk\lambda_{t}^{(k)} + \lambda_{v}^{(k)} = \lambda_{\mathrm{KD}}\\, b_{k} does not depend on ϕ\phi, because bkb_{k} is computed from the detached state and the shares sum to one. Thus, the controller cannot reduce both coefficients jointly. It can only reallocate the total between the two modalities, and ∣uk∣≤0.25\left| \mathbf{u}_{k} \right| \leq 0.25 bounds this reallocation.

The response to Q1 continue in next comment

AuthorsOpenReview ↗

Response to Q1 (Continue)

6. Complete three-branch objective. With cSc_{S} and c0c_{0} the correctness of the student and the frozen baseline :

LVL-DyMS={0(skip optimizer update),cS=1,Lans+Lrec,cS=0,c0=1,Lans+λt(k)LKDt+λv(k)LKDv,cS=0,c0=0,\mathcal{L}_{\text{VL-DyMS}} = \begin{cases} 0\\ \\ \text{(skip optimizer update)}, & c_{S} = 1, \\\\ \mathcal{L}_{\mathrm{ans}} + \mathcal{L}_{\mathrm{rec}}, & c_{S} = 0,\\ c_{0} = 1, \\\\ \mathcal{L}_{\mathrm{ans}} + \lambda_{t}^{(k)}\mathcal{L}_{\mathrm{KD}}^{t} + \lambda_{v}^{(k)}\mathcal{L}_{\mathrm{KD}}^{v}, & c_{S} = 0,\\ c_{0} = 0, \end{cases}

where LKDm=0.10,LKL(fθfull,fθ0m)+0.50,Lrep(zθm,zθ0m)\mathcal{L}_{\mathrm{KD}}^{m} = 0.10\\,\mathcal{L}_{\mathrm{KL}}\left( f_{\theta}^{\mathrm{full}},f_{\theta_{0}}^{m} \right) + 0.50\\,\mathcal{L}_{\mathrm{rep}}\left( z_{\theta}^{m},z_{\theta_{0}}^{m} \right) for m∈t,vm \in \\{ t,v\\}. The recovery loss Lrec\mathcal{L}_{\mathrm{rec}} is full-input baseline KL plus 0.100.10 times answer-representation MSE. KL is averaged over causally shifted supervised answer positions at temperature T=2T = 2 and scaled by T2T^{2}, with teacher outputs teacher-forced on the target answer.

7. Representation-matching construction. We mean-pool final decoder states over valid non-image prompt positions for text (ztz^{t}), expanded image-token positions for vision (zvz^{v}), and supervised answer positions for recovery. We apply float32 MSE directly to the pooled vectors, with detached teacher targets.

We hope this response clarifies Q1. If this is not what the reviewer meant, we kindly ask the reviewer to clarify, so that we can address it properly before the end of the discussion period. We would be grateful if the reviewer would take these responses into account when updating the rating. Our responses to Q2 and Q3 in the next comment.

AuthorsOpenReview ↗

Response to RQ2 and RQ3

Q2. Attribution of the main result. Thank you for this suggestion. Our rationale of using LoRA with answer loss was to compare VL-DyMS with an adaptation that has no dynamic modality scaling. With the same backbone and LoRA configuration, VL-DyMS matches or exceeds LoRA in all ten backbone-benchmark settings. The teacher results also show that the more useful modality differs across tasks and backbones. This variation motivates adjusting the modality weights during training rather than fixing them. However, we agree that LoRA also differs in the routing and the distillation losses, so it does not fully isolate the dynamic controller effect.

To have a coherent comparison, we are re-running Fixed KD on the same three branches, teacher losses, and LoRA configuration, so the only difference from VL-DyMS is the modality weights, which are fixed at 0.03 for each teacher. We set each teacher coefficient to 0.03 so that the losses of the vision teacher, the text teacher, and the student are combined with fixed weights throughout training, following the original knowledge distillation formulation. These experiments are currently running, and we will report the results in this thread before the end of the discussion period.

Q3. Evaluation clarification.

Thank you for highlighting this point. The original POPE dataset samples contain a paired image and question, such as "Is there a cup in the image?", with a ground truth such as "No".

The unimodal evaluation masks one of these inputs at a time, and we're using the same pre-trained backbone as the student model. In the vision teacher, the prompt is: "Question: [MASKED] Answer exactly one word: Yes or No." Thus, the teacher has no information identifying the queried object.

We re-checked our evaluation and found that the masked POPE vision-teacher values in Table 5 were misreported. Our experiment logs confirm that the vision-only teacher achieves 50% on the full 9,000 POPE test set.

To further validate all results, we are re-running all experiments again. We will report the updated results, if any, in this thread and revised manuscript before the end of the discussion period, and update the results in the manuscript accordingly.

We thank the reviewer again for the careful review, which helped us strengthen both the method description and the evaluation. If any of our responses does not address what the reviewer meant, we kindly ask the reviewer to clarify, so that we can address it properly. We hope our responses address the reviewer's concerns, and we would be grateful if the reviewer would consider updating the rating.

Reviewer_kgXpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

VL-DyMS addresses modality imbalance in LLM-centric VLMs by dynamically weighting text/vision distillation during LoRA adaptation. A frozen pretrained model serves as a behavioral reference, while two frozen masked teachers (text-only, vision-only) provide modality supervision, and a controller adjusts per-step modality weights.

Critical strengths
  1. Problem is interesting.
  2. Cheap design.
Critical weaknesses

1.No ablation vs. fixed weight KD and dynamic advantage is never isolated.

  1. Controller architecture, training signal, λ ranges/normalization, state components, and λ_KD undefined are not reproducible.

  2. Weak baseline and comparison.

  3. Unclear protocol.

Questions for authors

see weaknesses

5 comments
AuthorsOpenReview ↗

Response to Weakness 1

We thank the reviewer for the time and the helpful feedback, and for recognizing that the problem is interesting and that the design is cheap. We address each weakness below.

Weakness 1. No ablation vs. fixed weight KD and dynamic advantage is never isolated.

Thank you for this suggestion. We agree that the dynamic advantage should be isolated. Our rationale of using LoRA with answer loss is to compare VL-DyMS with an adaptation that has no dynamic modality scaling. However, LoRA also differs in the routing and the distillation losses, so it does not isolate the dynamic component. To have a coherent comparison, we are planning to add comparison with Fixed KD, which uses the same three branches, teacher losses, and LoRA configuration as VL-DyMS, with fixed coefficients of 0.03, so that the losses of the vision teacher, the text teacher, and the student are combined with fixed weights. Thus, the only difference from VL-DyMS is the modality weights: Fixed KD keeps $\lambda_{t} = \lambda_{v} , while VL-DyMS adjusts them with the dynamic controller.

We hope this response clarifies the concern. If this is not what the reviewer meant, we kindly ask the reviewer to clarify, so that we can address it properly before the end of the discussion period. The experiments mentioned in this comment are currently running, and we will report the results in this thread as well in revised manuscript before the end of the discussion period. We would be grateful if the reviewer would take these responses into account when updating the rating. Our responses continue in the next comment

AuthorsOpenReview ↗

Response to Weakness 2

Weakness 2. Controller architecture, training signal, λ ranges/normalization, state components, and λ_KD undefined are not reproducible.

Thank you for this comment. We will include the controller details in the revised manuscript, described below.

The controller receives the detached state as:

sk=sg[ℓt,k,ℓv,k,rt,k,rv,k,ra,k,ξk,gk],\mathbf{s}_{k} = \mathrm{sg}\left\lbrack \ell_{t,k},\ell_{v,k},r_{t,k},r_{v,k},r_{a,k},\xi_{k},g_{k} \right\rbrack,

where ℓt,k\ell_{t,k}, ℓv,k\ell_{v,k}, and ℓa,k\ell_{a,k} are the text distillation, vision distillation, and answer losses, respectively. For m∈t,v,am \in \\{ t,v,a\\}, rm,k=(μm,k−1−ℓm,k)/(∣μm,k−1∣+10−6)r_{m,k} = \left( \mu_{m,k - 1} - \ell_{m,k} \right)/\left( \left| \mu_{m,k - 1} \right| + 10^{- 6} \right), where μm\mu_{m} is an exponential moving average with decay 0.90.9. Teacher disagreement ξk\xi_{k} is the mean cosine dissimilarity of the two teachers' answer logits at matched positions. The gap gkg_{k} is the cosine dissimilarity between the student's pooled text and vision representations.

Architecture and weight constraints. The controller is a 7→24→27 \rightarrow 24 \rightarrow 2 MLP with a Tanh hidden activation. Its outputs pass through Tanh and are multiplied by 0.250.25 to bound the learned adjustment, so the controller refines the loss-change prior rather than collapsing the weights onto one teacher. Adding (−rt,k,−rv,k)\left( - r_{t,k}, - r_{v,k} \right) and applying softmax gives shares wt,kw_{t,k} and wv,kw_{v,k} that sum to one. The effective coefficients are

λm(k)=λKD,bk,wm,k,λKD=0.06,bk=clip(0.75+0.25tanh⁡(4ra,k),,0.5,,1).\lambda_{m}^{(k)} = \lambda_{\mathrm{KD}}\\, b_{k}\\, w_{m,k},\quad\quad\lambda_{\mathrm{KD}} = 0.06,\quad\quad b_{k} = \mathrm{clip}\left( 0.75 + 0.25\tanh\left( 4r_{a,k} \right),\\, 0.5,\\, 1 \right).

Each effective coefficient lies in [0,0.06]\lbrack 0,0.06\rbrack, and their sum lies in [0.03,0.06]\lbrack 0.03,0.06\rbrack. The nominal scale equals the sum of the two fixed KD coefficients (0.030.03 each).

Training signal. The controller receives gradients from the weighted distillation terms in the third branch of the objective. The budget bkb_{k} is detached, so the MLP only reallocates this budget between modalities, preventing joint coefficient collapse. AdamW updates the LoRA and controller parameters at learning rates 5×10−55 \times 10^{- 5} and 10−410^{- 4}. Both backbones use the same controller architecture across the five benchmarks, and the controller is initialized for each run.

Three-branch objective. Let cSc_{S} and c0c_{0} denote student and frozen baseline correctness, respectively, based on their generated answers.

LVL-DyMS={0(skip optimizer update),cS=1,Lans+Lrec,cS=0,c0=1,Lans+λt(k)LKDt+λv(k)LKDv,cS=0,c0=0.\mathcal{L}_{\text{VL-DyMS}} = \begin{cases} 0\\ \\ \text{(skip optimizer update)}, & c_{S} = 1, \\\\ \mathcal{L}_{\mathrm{ans}} + \mathcal{L}_{\mathrm{rec}}, & c_{S} = 0,\\ c_{0} = 1, \\\\ \mathcal{L}_{\mathrm{ans}} + \lambda_{t}^{(k)}\mathcal{L}_{\mathrm{KD}}^{t} + \lambda_{v}^{(k)}\mathcal{L}_{\mathrm{KD}}^{v}, & c_{S} = 0,\\ c_{0} = 0. \end{cases}

Distillation and representation matching. For m∈t,vm \in \\{ t,v\\},

LKDm=βlogit,LKL(fθfull,fθ0m)+βrep,Lrep(zθm,zθ0m),\mathcal{L}_{\mathrm{KD}}^{m} = \beta_{\mathrm{logit}}\\,\mathcal{L}_{\mathrm{KL}}\left( f_{\theta}^{\mathrm{full}},f_{\theta_{0}}^{m} \right) + \beta_{\mathrm{rep}}\\,\mathcal{L}_{\mathrm{rep}}\left( z_{\theta}^{m},z_{\theta_{0}}^{m} \right),

with βlogit=0.10\beta_{\mathrm{logit}} = 0.10 and βrep=0.50\beta_{\mathrm{rep}} = 0.50. Teacher-to-student KL is averaged over causally shifted supervised answer positions at temperature T=2T = 2 and scaled by T2T^{2}. Teacher outputs are teacher-forced on the target answer, so the student and both teachers score the same answer tokens. We mean-pool final decoder states over valid non-image prompt positions for text, expanded image-token positions for vision, and supervised answer positions for recovery, and apply float32 MSE to the pooled vectors with detached teacher targets. The recovery loss Lrec\mathcal{L}_{\mathrm{rec}} is full-input baseline KL plus 0.100.10 times answer-representation MSE.

If this is not what the reviewer meant, we kindly ask the reviewer to clarify, so that we can address it properly before the end of the discussion period. We would be grateful if the reviewer would take these responses into account when updating the rating.

AuthorsOpenReview ↗

Response to Weakness 3,4

Weakness 3. Weak baseline and comparison.

Thank you for your feedback. We compared VL-DyMS with LoRA, as an optimization comparison. We are planning to add Fixed KD as a fixed-weight distillation baseline, which uses the same three branches, teacher losses, and LoRA configuration as VL-DyMS. This experiment are currently running, and we will report the results in this thread before the end of the discussion period. We also plan to add more benchmarks with other methods and baselines in future work. If the reviewer has a specific baseline in mind, we kindly ask the reviewer to point it out, so that we can address it properly.

Weakness 4. Unclear protocol.

Thank you for pointing out the need for a clearer evaluation protocol. We evaluated the methodology with the same LoRA configuration across five datasets and two backbone models, and each dataset used the same splits and experimental setup across the backbones. The revised manuscript now states the full protocol clearly:

DatasetMain capabilityAdaptation dataEvaluation splitMetric
TextVQAScene-text understanding and reasoningtrain: 25,000 of 34,602validation: all 5,000VQA accuracy
GQACompositional visual reasoningtrain_balanced: 25,000 of 943,000testdev_balanced: all 12,578Exact match
TallyQAVisual counting and counting-based reasoningtrain: 25,000 of 249,318test: all 38,589Exact match
POPEObject hallucination and visual groundingCOCO val2014 questions: 15,000 constructedCOCO random, popular, adversarial: all 9,000Accuracy
POPEv2Counterfactual object hallucinationPOPE-adversarial: 1,800normal and counterfactual: all 1,000Accuracy

We thank the reviewer again for the constructive comments, which helped us improve the clarity and the evaluation of the paper. We hope our responses address the reviewer's concerns, and we would be grateful if the reviewer would consider updating the rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Update on the experimental protocol and results

We thank the reviewers for their questions about the evaluation protocol. While running the additional experiments requested in the reviews, we realized that the runs reported in the submission were carried out at a smaller scale than Section 4.2 describes, and that their settings were not the same across benchmarks. We apologize for this inconsistency. We have re-run all experiments at full scale under one protocol. We reported them in this thread, and in the updated manuscript as well.

For transparency, the differences are listed below.

Submitted versionUpdated runs
Evaluation dataFull sets for LLaVA GQA and POPEv2. Subsets elsewhere: TextVQA 80 questions, Qwen GQA 80, TallyQA 1,000, Qwen POPE 3,000, LLaVA POPE 1,800Full standard split for every benchmark
Adaptation dataSmaller sets (for example 120 questions for TextVQA, 3,000 for TallyQA). For Qwen GQA and LLaVA POPE, taken from the same question pool as the evaluation set25,000 training questions for TextVQA, GQA and TallyQA; 15,000 for POPE; 1,800 for POPEv2. Separate from the evaluation data; images that appear in the POPE and POPEv2 evaluation sets are excluded
LoRA settingsLearning rate between 1e-5 and 1e-4 depending on the run; rank 16 and alpha 32 for Qwen POPERank 8, alpha 16, dropout 0.0, learning rate 5e-5, one epoch, seed 42 for all runs
Vision-only prompt (POPE, POPEv2)Included the name of the queried objectQuestion fully masked

The last row answers Reviewer KPx4's Q3. The high vision-only value in the submitted Table 5 for POPEv2 (92.10) came from the prompt described in the table above. With the question fully masked, both teachers score 50.00 on POPE and POPEv2.

The results are in the next comment and updated in manuscript. Our earlier statement that VL-DyMS matches or exceeds LoRA in all ten settings was based on the submitted numbers. In the updated results, VL-DyMS is also above the frozen model and the fixed-weight baseline.

We are grateful to the reviewers, whose questions led to this more complete evaluation.

AuthorsOpenReview ↗

Updated results

Qwen2.5-VL-3B

MethodTextVQAGQATallyQAPOPEPOPEv2
Frozen76.9159.7476.4487.5790.40
LoRA81.4364.9576.9490.0181.80
Fixed KD79.8861.2476.9587.2374.60
VL-DyMS w/o text teacher80.3460.5176.3487.6986.30
VL-DyMS w/o vision teacher80.1361.1275.2786.8474.00
VL-DyMS83.42668.0381.0991.4391.10

LLaVA-1.5-7B

MethodTextVQAGQATallyQAPOPEPOPEv2
Frozen48.0559.5667.3985.7679.40
LoRA53.0160.5866.6884.5765.70
Fixed KD52.6458.2567.7386.2781.70
VL-DyMS w/o text teacher50.9457.7967.7084.2076.90
VL-DyMS w/o vision teacher51.1858.5165.1086.0781.70
VL-DyMS56.7568.0973.9089.8095.40

Masked teachers (frozen backbone, one input masked)

BackboneTeacherTextVQAGQATallyQAPOPEPOPEv2
QwenText-only0.5631.0117.0050.0050.00
QwenVision-only13.623.8441.6950.0050.00
LLaVAText-only6.0736.9636.5150.0050.00
LLaVAVision-only12.970.2849.0650.0050.00

Notes

  • VL-DyMS has finished all of the ten settings, with the frozen baseline model, LoRA and Fixed KD. .
  • All results are from a single seed.
  • Fixed KD uses 0.03 per teacher (total 0.06). In VL-DyMS the total is 0.06 x b_k, between 0.03 and 0.06, so the two differ in the total weight as well as in the split.
  • LoRA improves over the frozen model in six settings and falls below it in four (Qwen POPEv2; LLaVA TallyQA, POPE, POPEv2).
  • POPEv2 has no training split. Its model is adapted on POPE-adversarial questions and evaluated only on POPEv2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.