acceptodds
Under review as a conference paper at ICLR 2027

ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning

Abstract

Multimodal Large Language Models (MLLMs) achieve strong performance through instruction tuning, but real-world deployment requires them to continually acquire new vision-language capabilities, making Multimodal Continual Instruction Tuning (MCIT) essential. To reduce inter-task interference and promote collaboration, recent methods often employ sparse architectures like Mixture of LoRA Experts with image-text similarity routing. However, tasks with distinct response structures could share highly similar visual-linguistic semantics and thus be wrongly routed to the same expert. Image-text similarity alone is insufficient for reliable task assignment, and as a result, an expert in a grounding task requiring coordinate prediction may be biased toward producing short textual answers after learning semantically similar VQA tasks. This format-blind task assignment integrates heterogeneous response types into shared parameters, inducing gradient interference and ineffective expert collaboration. To address this problem, we propose ProtoAda, a prototype-guided adaptive tuning framework. ProtoAda constructs format-aware task codes that jointly characterize multimodal semantics and response-protocol structures, thereby inducing a compatibility-aware organization of the evolving task stream for task assignment and inference-time routing. It further applies geometry-aware consolidation to separate reusable directions from task-specific residuals, progressively refining shared parameters while preserving protocol-sensitive behaviors. Together, these components mitigate forgetting from both task organization and parameter consolidation perspectives. Extensive experiments on multiple benchmarks demonstrate superior performance, especially on tasks whose answer structures are vulnerable to sequential tuning.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Multimodal Large Language Models, Multimodal Continual Instruction Tuning, Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_jbGDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ProtoAda proposes format-aware task grouping, selective LoRA expansion, and geometric consolidation for rehearsal-free multimodal continual instruction tuning. Experiments on TriGap and UCIT, including controlled format shifts and a grounding extension, evaluate its accuracy, efficiency, and ability to preserve response protocols.

Critical strengths
  1. The format-aware task grouping is well motivated. Incorporating response-format statistics into task prototypes provides a coherent way to account for output compatibility when sharing adaptation parameters. The improvement from in Table 3 together with the advantage over late fusion in Table 5, supports the usefulness of this design.

  2. The method demonstrate a useful balance between accuracy and adaptation cost. Compared with SAME + format, it uses 16.4% fewer stored parameters and 22.3% less training time, with accuracy point estimates of 74.66 versus 74.92 in Table 7.

Critical weaknesses
  1. The consolidation mechanism needs a precise operational definition.

It is unclear whether masking, projection and SVD act on LoRA-factor increments or effective weight updates, and how group bases are initialized and updated in Eq. (3), (14)–(21). This leaves a central component difficult to reproduce and assess.

  1. Reporting inconsistencies need clarification.

The mean task score in Table 1 is 46.928, whereas the reported value is 47.23. In Table 6, the heading referring to three tasks conflicts with the average value calculated for six tasks.

Questions for authors
  1. Which exact tensors and shapes implement Eqs. (14)–(21), how is UgU_g initialized and updated

  2. Which Table 1 entries or aggregate are incorrect? Does Table 6 accuracy cover three or six tasks?

  3. Which dataset and scoring rule were actually used for FmtGap?

6 comments

Response to Reviewer jbGD

Thank you for recognizing the motivation for protocol-aware grouping and the accuracy–cost trade-off. We address each requested clarification below.

Q1. Exact tensors, shapes, and subspace initialization in Eqs. (14)–(21)

These operations are performed on effective weight matrices, not directly on LoRA-factor increments. For each linear module:

ObjectDefinition / roleShape
At,BtA_t,B_tTrainable task-draft factorsr×dinr\times d_{\mathrm{in}}, dout×rd_{\mathrm{out}}\times r
DtD_t(αLoRA/r)BtAt(\alpha_{\mathrm{LoRA}}/r)B_tA_tdout×dind_{\mathrm{out}}\times d_{\mathrm{in}}
aaInput-channel activation RMSdind_{\mathrm{in}}
M,D^M,\widehat DElementwise mask and masked updatedout×dind_{\mathrm{out}}\times d_{\mathrm{in}}
Gg,S,RtG_g,S,R_tGroup, global, and task-residual effective matricesdout×dind_{\mathrm{out}}\times d_{\mathrm{in}}
Dshare,DresD^{\mathrm{share}},D^{\mathrm{res}}Projection and uncompressed residualdout×dind_{\mathrm{out}}\times d_{\mathrm{in}}
A′,B′A',B'Stored rank-kk factorsk×dink\times d_{\mathrm{in}}, dout×kd_{\mathrm{out}}\times k

For nonzero GgG_g, the current group subspace is the one-dimensional span of vec⁡(Gg)\operatorname{vec}(G_g), and the shared component is ⟨D^,Gg⟩FGg/∥Gg∥F2\langle\widehat D,G_g\rangle_FG_g/\|G_g\|_F^2. There is no separately maintained multidimensional group basis. An empty group is initialized from the full masked task update, with zero initial residual. Subsequent tasks update the effective group matrix through weighted absorption and low-rank storage; the next projection uses that updated matrix.

Global promotion adds a common matrix HH to global memory and subtracts it from each group before compression. Residual pruning, in contrast, is approximate deletion based on a coverage criterion and does not retain a separate task-specific coefficient vector. We will make these distinctions, the scaling conventions, and the numerical approximations explicit in the revised equations and pseudocode.

Q2. Which Table 1 entries or aggregate are incorrect?

The errors are in two per-task entries: FloodNetVQA should be 80.96, not 74.03, and CLEVR-Math should be 54.00, not 57.87. With the remaining eight scores unchanged, the corrected sum is 472.34 and the mean is 47.234%, which rounds to the reported 47.23%. Thus, the aggregate is consistent with the corrected task scores; the two task entries need to be corrected. The corresponding improvement over DISCO's listed-score average of 46.544% is 0.690 percentage points. We apologize for the transcription errors.

Q3. Does Table 6 cover three tasks or six?

Protocol compliance is averaged over three tasks: ArxivQA, IconQA, and CLEVR, whose required answer forms admit reliable deterministic checks; the other tasks do not admit the same unambiguous protocol check. The accuracy column is the six-task UCIT final average, copied from Table 2 as an overall performance reference, and remains unchanged. We will explicitly label these different task scopes and explain why compliance is restricted to three tasks in the caption. The original presentation conflates these scopes, and we apologize for the ambiguity.

Q4. Which dataset and scoring rule were used for FmtGap?

The intended dataset is Flickr30k throughout FmtGap. The five phases change the required response protocol: brief description, detailed description, short/one-word answer, multiple-choice answer, and yes/no answer. The references to VizWiz in the FmtGap construction descriptions will be corrected; VizWiz remains part of the semantic-varied VisGap comparison.

The Task-0 drop is defined as

Δ0=A0before−A0after,\Delta_0=A_0^{\mathrm{before}}-A_0^{\mathrm{after}},

using the same held-out Task-0 samples and reference captions before and after subsequent training. Task 0 uses the standard UCIT caption-evaluation pipeline, implemented in backbone/shared/eval/eval_caption.py with COCOEvalCap. Generated captions and reference captions are PTB-tokenized, and the evaluator computes BLEU-1, BLEU-2, BLEU-3, BLEU-4, METEOR, ROUGE-L, and CIDEr. Each score is multiplied by 100, and A0A_0 is the unweighted arithmetic mean of these seven scaled scores:

A0=1007(BLEU1+BLEU2+BLEU3+BLEU4+METEOR+ROUGEL+CIDEr),A_0=\frac{100}{7}\left(\mathrm{BLEU}_1+\mathrm{BLEU}_2+\mathrm{BLEU}_3+\mathrm{BLEU}_4+\mathrm{METEOR}+\mathrm{ROUGE}_L+\mathrm{CIDEr}\right),

where the terms inside the parentheses are the raw evaluator outputs. For illustration, a 3,000-sample caption-evaluation log reports the following scaled scores:

BLEU-1BLEU-2BLEU-3BLEU-4METEORROUGE-LCIDErMean
72.7957.6846.0837.7327.4755.66143.0462.92

This example illustrates the scoring rule, not an additional FmtGap performance result. Although the script prints the mean with a percent sign (Average: 62.92%), it is a composite caption-quality score, not the percentage of correctly answered samples. No threshold is applied to convert caption similarity into binary correctness, and scaled CIDEr can exceed 100. Accordingly, Δ0\Delta_0 is reported in caption-score points, not percentage points of sample-level accuracy.

Task-0 retention after each subsequent phase is evaluated under the original Task-0 caption protocol and with this same metric; the later phases' response formats do not change the Task-0 scoring rule. We will specify this aggregation in the FmtGap evaluation description and distinguish caption quality from content correctness and format compliance.

Dear Reviewer jbGD,

Thank you again for your thoughtful and constructive feedback. We have tried to address your concerns regarding the shape of tensors, clarifications of actual accuracy, Table 6 and FmtGap.

As the discussion period is approaching its end, we would greatly appreciate any final feedback if time permits. If there are any remaining concerns or points that you would like us to clarify further, we would be happy to respond.

Best regards,

Authors

Reviewer_KFP3OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ProtoAda, a framework for rehearsal-free multimodal continual instruction tuning. The method combines visual and textual representations with response-format statistics to construct task prototypes, which guide task grouping and adapter allocation. It then selectively expands LoRA capacity and decomposes task updates into shared components and task-specific residuals, with periodic global consolidation to reduce parameter growth. At inference, format information is predicted from the input to support task-agnostic retrieval. Explicitly considering response-protocol compatibility is a useful extension of semantic task routing. Experiments on TriGap and UCIT, together with controlled streams and supplementary evaluations, demonstrate promising accuracy and efficiency. However, inconsistencies in the reported results, incomplete consolidation definitions, insufficient budget-controlled comparisons, and limited statistical evidence weaken the central claims.

Critical strengths

1.The paper addresses a relevant limitation of semantic routing: tasks with similar visual and linguistic content may require incompatible response protocols.

2.The use of response-format statistics provides a concrete signal for organizing task sharing, while input-side prediction supports inference without ground-truth task identities or target responses.

3.The evaluation includes two benchmarks, controlled streams, task-order variations, sample-level retrieval, format-enhanced baselines, and a grounding extension.

4.The reported parameter and training-time reductions suggest a potentially useful accuracy–efficiency trade-off.

Critical weaknesses

1. Inconsistencies in the reported results affect the assessment of the empirical improvements.

The ten ProtoAda scores in Table 1 sum to 469.28, yielding an arithmetic mean of 46.928 rather than the reported 47.23. Under the averaging definition in Section 5.1, the listed results therefore imply an improvement of approximately 0.384 percentage points over DISCO. Although ProtoAda retains the highest average in this table, the improvement is smaller than the reported average suggests. Additionally, Table 6 is described as evaluating three format-restricted tasks, but its accuracy column reproduces the six-task UCIT averages from Table 2. The construction of FmtGap is also inconsistent: the main text specifies Flickr30k, whereas Appendices C and J specify VizWiz. The authors should reconcile these discrepancies and provide consistent results and experimental definitions.

2. The additional value of the full framework over a format-enhanced strong baseline is not sufficiently established.

Table 7 shows that SAME with the same format code achieves 74.92% accuracy, compared with 74.66% for ProtoAda. ProtoAda uses fewer parameters and less training time—235M versus 281M parameters and 3.35 versus 4.31 hours—which supports a promising resource trade-off. However, the comparison does not establish whether the proposed expansion and consolidation mechanisms outperform a simpler format-enhanced learner under comparable budgets. Reducing the rank, adapted layers, or expert capacity of SAME may provide an alternative trade-off. A matched-parameter comparison against SAME + format code, together with ablations that hold grouping and capacity fixed, would clarify the contribution of the additional machinery. Fixing the LoRA rank alone does not ensure comparable total capacity.

Questions for authors

1. Could the authors clarify the geometric consolidation procedure and provide stronger evidence for its behavioral interpretation?

In Section 3.2, the trainable parameters are defined as LoRA factors, while Section 4.3 applies masking, projection, and SVD to their updates. Are these operations performed directly on the LoRA factors or on the corresponding effective weight updates? Since these choices are not mathematically equivalent, could the authors provide precise definitions and implementation details? How is the group subspace initialized and updated? When directions are promoted into the shared subspace, how are they accounted for in the remaining group adapters, and how are task-specific coefficients preserved when residuals are pruned? Furthermore, although Appendix A establishes standard projection and orthogonality properties, how do these properties guarantee preservation of previous-task behavior or demonstrate that orthogonal residuals encode response protocols? Could the authors provide targeted component-level evaluations to substantiate these interpretations?

2. Could the authors clarify the statistical reliability of the reported improvements and provide more direct evidence that the gains originate from reduced forgetting?

The five-seed experiments in Appendix C validate the controlled-stream phenomenon, while the standard deviations in Table 8 reflect variation across task orders. Could the authors report repeated-run uncertainty under fixed task orders for the main comparisons in Tables 1, 2, and 7, particularly given the approximately 0.38-percentage-point TriGap improvement implied by the listed results? Additionally, since final average performance combines acquisition, routing, capacity allocation, and retention effects, how can the improvements be specifically attributed to reduced forgetting? Table 13 reports protocol-violation rates within each method's own forgotten-sample set, but these conditional percentages do not establish absolute reductions in forgetting or protocol errors. Could the authors provide task-by-stage performance, explicit forgetting metrics, and corresponding absolute error counts to better isolate and validate the claimed retention benefits?

5 comments

Response to Reviewer KFP3

Thank you for the detailed assessment and for recognizing the motivation, the input-side protocol signal, and the potential accuracy–efficiency trade-off. We agree that the numerical inconsistencies and incomplete operational definitions need to be resolved before the stronger interpretations can be assessed.

W1. Table 1, Table 6, and the FmtGap definition

Table 1. Two per-task entries were transcribed incorrectly: ProtoAda's FloodNetVQA score should be 80.96, rather than 74.03, and its final-task CLEVR-Math score should be 54.00, rather than 57.87. With these two corrections and the other eight task scores unchanged, the average is

469.28−74.03+80.96−57.87+54.0010=47.234,\frac{469.28-74.03+80.96-57.87+54.00}{10}=47.234,

or 47.23% at two decimal places, consistent with the reported aggregate. DISCO's listed entries average 46.544%, so the corrected per-task results imply an improvement of 0.690 percentage points, rather than the approximately 0.384 points implied by the incorrectly transcribed entries. We will correct both task entries and check all derived comparisons against the underlying evaluation records. We apologize for these transcription errors and the resulting inconsistency.

Table 6. Its two columns use different task scopes. Protocol compliance is averaged over ArxivQA, IconQA, and CLEVR, where the required answer forms admit reliable deterministic checks. The other tasks do not admit the same unambiguous protocol check, so they are excluded from this compliance average. The accuracy column remains the overall six-task UCIT average from Table 2. We will relabel the columns as “Protocol compliance, 3 restricted tasks” and “Final average accuracy, all 6 UCIT tasks,” and explain the different scopes and their rationale in the caption.

FmtGap. According to our experiment notes, FmtGap is constructed from Flickr30k, with the response protocol varied across phases. The references to VizWiz in the FmtGap descriptions in Appendices C and J are inconsistent and should be corrected. VisGap, in contrast, alternates Flickr30k and VizWiz while fixing the response protocol. We will document the precise data files, construction procedure, and scoring rule so that the dataset correction does not leave the evaluation ambiguous.

W2. SAME + format code under a near-matched storage budget

We agree that the submitted comparison establishes an efficiency trade-off, not superiority at equal capacity. At the submitted configurations, ProtoAda uses 235M stored parameters and 3.35 hours, versus 281M and 4.31 hours for SAME + format code, while its accuracy is 0.26 percentage points lower. This corresponds to approximately 16.4% fewer stored parameters and 22.3% less training time.

To address the budget mismatch, we reduced the expert LoRA rank of SAME + format code to 14, bringing its stored parameter count from 281M to 240M, close to ProtoAda's 235M. The additional comparison is:

MethodStored parameters (M)Final UCIT average (%)Training time (h)
SAME + format code, submitted configuration28174.924.31
SAME + format code, rank-14 configuration24074.10Not comparable: different GPU hardware
ProtoAda, submitted configuration23574.663.35

At this near-matched budget, ProtoAda achieves 74.66% versus 74.10%, an advantage of 0.56 percentage points, while storing 5M fewer parameters (approximately 2.1% fewer than the rank-14 baseline). The reduced-rank experiment used different GPU hardware, so its training time is not directly comparable and we do not claim a timing advantage from that run.

Q1. Geometric consolidation: operations, initialization, promotion, pruning, and interpretation

1. Operations act on effective weight updates, not factor increments. For module ℓ\ell, training optimizes At,ℓ∈Rr×dinA_{t,\ell}\in\mathbb R^{r\times d_{\mathrm{in}}} and Bt,ℓ∈Rdout×rB_{t,\ell}\in\mathbb R^{d_{\mathrm{out}}\times r}. Consolidation first forms

\in\mathbb R^{d_{\mathrm{out}}\times d_{\mathrm{in}}}.$$ The temporary task draft starts with zero effective contribution. Thus $D_{t,\ell}$ is its learned contribution relative to that zero update, not a direct subtraction of historical factor parameters. Masking, projection, and low-rank compression operate on this matrix independently for each adapted linear module. For channel activation RMS $a_j=\sqrt{\sum_nx_{n,j}^2/N}$, importance is $I_{ij}=|D_{ij}|a_j$. A module-wide elementwise mask produces $\widehat D=M\odot D$. Low-rank storage uses $D\approx U_k\Sigma_kV_k^\top$, with $B'=U_k\Sigma_k^{1/2}$ and $A'=\Sigma_k^{1/2}V_k^\top$. The implementation uses a randomized low-rank SVD when appropriate, with full-SVD fallbacks. We will distinguish the trained factors from effective updates throughout the notation. **2. The current group projection is a one-dimensional Frobenius projection.** Write $G_{g,\ell}$ for the group adapter's effective matrix, to avoid overloading the LoRA factor symbol $A$. For nonzero $G$, the implementation uses $$\mathcal U_{g,\ell}=\operatorname{span}\{\operatorname{vec}(G_{g,\ell})\}, \qquad D^{\mathrm{share}}= \frac{\langle\widehat D,G\rangle_F}{\|G\|_F^2}G, \qquad D^{\mathrm{res}}=\widehat D-D^{\mathrm{share}}.$$ This is not a separately maintained multidimensional basis of past updates. The matrix rank of $G$ and the dimension of its vectorized span are different quantities. For an empty/zero group memory, a bootstrap branch sets $D^{\mathrm{share}}=\widehat D$ and $D^{\mathrm{res}}=0$, then initializes the group by low-rank factorization. This bootstrap is separate from the projection proposition. For an existing group, $G$ is replaced by a low-rank approximation of $G+\lambda D^{\mathrm{share}}$. The current absorption coefficient is $\lambda=(\lambda_0/\sqrt{n_g})\operatorname{sigmoid}(s_{t,g}/T_\lambda)$; a new group uses $\lambda=1$. Future projections use the updated effective group matrix. We will reconcile this operational definition and coefficient with the more general notation in Section 4.3. **3. Global promotion includes subtraction from the groups.** For group matrices $G_1,\ldots,G_m$, the implementation obtains leading eigenvectors $U_k$ of $C=\sum_gG_gG_g^\top$ and constructs $$H=\frac{1}{m}\sum_gU_kU_k^\top G_g.$$ It then updates the global matrix $S$ and the remaining groups by low-rank approximations of $S+H$ and $G_g-H$, respectively. The subtraction avoids simply counting the promoted update twice. Before compression, with identical global/group scaling and participating group weights summing to one, $$(S+H)+\sum_gw_g(G_g-H)=S+\sum_gw_gG_g.$$ Actual consolidation also applies low-rank approximations and rank reduction, so this identity is not a guarantee of exact functional preservation in the implemented model. We will clarify the draft, global/group, and residual scaling conventions in the revised description. **4. Residual pruning is approximate; task-specific coefficients are not separately preserved.** The current implementation removes a residual when its coverage score exceeds a threshold. It does not keep an additional task-specific coefficient vector after deletion. We did not intend to claim exact coefficient preservation, but agree that subspace coverage alone does not establish reconstruction of a task's particular update amplitude. We will describe pruning as approximate redundancy removal rather than an exact reparameterization.

5. The theory does not establish protocol-specific encoding or a no-forgetting guarantee. Appendix A establishes closest-subspace approximation and orthogonality for the uncompressed decomposition. These properties do not imply that old-task loss is unchanged, that stored low-rank residuals remain exactly orthogonal after compression, or that residuals contain only protocol information. Residuals can also capture content- or domain-specific changes. We will narrow the behavioral language accordingly: the decomposition is intended to separate group-aligned updates from task-specific deviations, and its effect on protocol retention requires empirical evidence.

6. Targeted residual evaluation. We evaluated ProtoAda with task residuals enabled and disabled. Consistent with the clarified scope of Table 6, protocol compliance is measured on ArxivQA, IconQA, and CLEVR, while benchmark accuracy is the final average over all six UCIT tasks:

ConfigurationProtocol compliance, 3 restricted tasks (%)Final UCIT average accuracy, all 6 tasks (%)
Full ProtoAda99.174.66
Same checkpoint, task residuals disabled98.072.41

Disabling the residuals reduces protocol compliance by 1.1 percentage points and final benchmark accuracy by 2.25 percentage points. This provides direct empirical evidence that the task residuals contribute to protocol compliance and overall performance, beyond the geometric properties established in Appendix A.

Q2. Statistical reliability and evidence for reduced forgetting

Repeated runs. We have completed four UCIT runs with seeds 42, 100, 1000, and 10, under the same fixed task order, backbone, training data, hyperparameters, and evaluation protocol. These are the same runs detailed in our response to Reviewer R9LH:

Random seedUCIT final six-task mean
42 (default)74.6550
1074.6600
10074.4950
100074.7667
Mean ± sample SD74.64 ± 0.11

The summary is computed from each run's six-task mean before rounding, using the sample-SD denominator n−1n-1 with n=4n=4. The across-seed mean and sample SD are 74.6442 and 0.1120, respectively; the observed range is approximately 0.27 score points. This supports stable final UCIT performance across the tested seeds. Seed 42 is the default run underlying Table 2 and the task-by-stage matrix below.

Reviewer_R9LHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that forgetting in multimodal continual instruction tuning (MCIT) partly stems from format-blind task routing: tasks with similar image–text semantics but different response protocols end up sharing parameters and interfering. The proposed ProtoAda (i) groups tasks with prototypes that combine CLIP features and a response-format code, which is predicted from the instruction at inference; (ii) adds LoRA only to layers chosen by gradient sensitivity and group usage; and (iii) consolidates each update by projecting it onto a group subspace and keeping the orthogonal remainder as a compact per-task residual. With LLaVA-1.5-7B, ProtoAda reports the best average accuracy on TriGap and UCIT, supported by ablations and appendix analyses.

Critical strengths
  • The paper isolates one underlying cause of forgetting in multimodal continual instruction tuning by separating response-protocol interference from visual-semantic shift with controlled streams.
  • The experimental evaluation is broad, covering two benchmarks and eight baselines. Additional experiments cover task-order robustness, parameter count, and training time.
Critical weaknesses
  • The improvements over the strongest baselines are modest, but the main benchmark results do not report the number of runs or variability across random seeds. Reporting means and standard deviations over multiple runs would help assess the robustness of these gains.
  • Several implementation details and hyperparameter values are missing, including the prototype update rate α\alpha, layer-usage weight η\eta, and masking ratio kk. Please report these values to improve reproducibility.
  • The distinction between the proposed geometry-aware consolidation and SAME (Xie et al., 2026) is insufficiently explained. Both use spectral/SVD-based mechanisms, yet SAME is not included in related-work discussion despite being compared and briefly described in the appendices. What specifically distinguishes the two consolidation mechanisms?
  • Key terms are not clearly defined or consistently distinguished. The paper uses “response protocol,” “response format,” “response structure,” and “response type” without clarifying whether they are interchangeable or refer to different concepts. Please clarify their meanings and standardize the terminology.

References:

Xie, Zhen-Hao, et al. "SAME: Stabilized mixture-of-experts for multimodal continual instruction tuning." arXiv preprint arXiv:2602.01990 (2026).

Questions for authors

Please address the questions and clarification requests in the Weaknesses above.

8 comments

Response to Reviewer R9LH

Thank you for recognizing the controlled analysis and the breadth of the evaluation. We agree that statistical uncertainty, implementation details, and the relationship to SAME require clearer reporting.

Q1. Repeated-run uncertainty

The five-seed controlled-stream experiment tests the stability of the motivating phenomenon, while Table 8 measures sensitivity to task order. Separately, we have now completed four UCIT runs with random seeds 42, 100, 1000, and 10, keeping the task order, backbone, training data, hyperparameters, and evaluation protocol fixed. Their per-task final scores and six-task means are:

Random seedImageNet-RArxivQAVizCapIconQACLEVRFlickr30kFinal six-task mean
42 (default)89.8393.2061.4981.3766.4055.6474.6550
1090.3693.0362.6179.6366.7755.5674.6600
10090.2093.2061.0280.0767.2355.2574.4950
100090.3393.7061.5180.7766.6355.6674.7667
Mean ± sample SD90.18 ± 0.2493.28 ± 0.2961.66 ± 0.6780.46 ± 0.7766.76 ± 0.3555.53 ± 0.1974.64 ± 0.11

All scores follow the original UCIT reporting scale. Each run's final mean is computed from its six task scores before rounding. We then compute the across-seed mean and sample standard deviation, using the n−1n-1 denominator with n=4n=4; the final column's SD is calculated across run-level means, not averaged from the per-task SDs. The across-seed mean is 74.6442 and the sample SD is 0.1120 (shown to four decimal places), giving 74.64 ± 0.11 score points at two decimal places. The final means span 74.4950-74.7667, a range of approximately 0.27 points.

Seed 42 is the default run underlying the submitted 74.66 result; the new four-seed summary is reported separately rather than replacing that run's task scores with across-seed averages. The small observed variation supports the stability of ProtoAda's final UCIT performance under these four seeds and this fixed task order.

Q2. Hyperparameters and reproducibility

We will clarify the prototype construction, grouping, layer selection, masking, consolidation, and retrieval settings in the revised manuscript. The hyperparameters are summarized below.

Symbol / settingValue
Prototype update rate, α\alpha0.3
Initial grouping threshold, τinit\tau_{\mathrm{init}}0.9
Layer-usage weight, η\eta0.5
Mask retention ratio, kk0.95
Absorption base coefficient, λ0\lambda_00.3
Similarity scaling, β\beta0.5
Residual pruning threshold, ρthr\rho_{\mathrm{thr}}0.95
Retrieval temperature, τ\tau0.1
Retrieved groups, KretK_{\mathrm{ret}}2
Format warm-up batches, NwarmN_{\mathrm{warm}}32
Format prediction loss weight, ωfmt\omega_{\mathrm{fmt}}0.1

For a group containing only one member, admission uses the fixed initial threshold τinit=0.9\tau_{\mathrm{init}}=0.9. Once the group contains at least two members, admission uses a dynamic criterion based on the running mean and standard deviation of its within-group task-to-prototype similarities. This distinguishes singleton-group initialization from subsequent adaptive admission. Here, k=0.95k=0.95 means retaining approximately 95% of update entries, not removing 95%.

Q3. Difference from SAME

We agree that SAME [1] should be discussed in the main related-work section. SAME addresses router and expert drift using spectral-aware routing, curvature-aware scaling, and adaptive expert activation. ProtoAda instead makes response-protocol compatibility explicit in task association and retrieval, and consolidates effective task updates into group-aligned components and separately stored residuals, with periodic cross-group promotion.

The distinction is therefore the role and object of the operation, not the use of SVD itself: SAME uses spectral information to stabilize routing and expert adaptation, whereas ProtoAda uses a group-aligned decomposition and low-rank storage to organize task updates into shared memory and task-specific residuals. Periodic cross-group promotion further reorganizes common components into global memory. We will make these operational differences explicit in the main related-work discussion, rather than imply that using SVD itself distinguishes the methods. We do not present projection, SVD, or sensitivity-based allocation individually as new tools.

Q4. Terminology

We used “response protocol,” “response format,” “response structure,” and “response type” to describe the same output-side requirement, rather than four distinct technical objects. We will standardize the term to response protocol: the task-prescribed form in which an answer must be expressed, such as an option letter, a short phrase, a description, or bounding-box coordinates. We will distinguish protocol compliance from factual correctness; a response may satisfy one without satisfying the other.

References

[1] Xie et al. SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning. arXiv:2602.01990, 2026. Mechanism comparison should use the version corresponding to the evaluated baseline.

Dear Reviewer R9LH,

Thank you again for your thoughtful and constructive feedback. We have tried to address your concerns regarding statistical uncertainty, implementation details, and the relationship to SAME.

As the discussion period is approaching its end, we would greatly appreciate any final feedback if time permits. If there are any remaining concerns or points that you would like us to clarify further, we would be happy to respond.

Best regards,

Authors

Reviewer_DSWkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that MoE-LoRA methods for multimodal continual instruction tuning (MCIT) route tasks by image–text similarity. As a result, tasks with similar content but different response formats share parameters and interfere. ProtoAda adds a task-level “format code” (statistics of response length, token entropy and template consistency) to CLIP prototypes and uses it to assign tasks to LoRA groups with an adaptive admission threshold. It selects which layers to adapt from gradient sensitivity plus the group’s past layer usage. After each task it masks the update, then splits it into a projection onto the group subspace and a low-rank orthogonal residual.

Critical strengths

The controlled FmtGap/VisGap experiment is clean. With visual content fixed, changing only the answer protocol drops Task-0 accuracy by about 57 points, versus about 5 for a semantic shift. The effect holds across five seeds (App. C.3) and is replicated on Qwen3-VL-8B (Table 11).

Critical weaknesses

The work is not positioned against closely related published work.

CoIN (Chen et al., NeurIPS 2024) is cited here only as a benchmark. It already separates instruction-following forgetting from reasoning forgetting and finds the former dominates. SEFE (Chen et al., ICML 2025) names format and style drift “superficial forgetting”. It mitigates it by rewriting each task’s data into five answer styles (yes/no, multiple-choice, short answer, brief and detailed explanation), nearly the same taxonomy as Table 4. D-MoLE (Ge et al., ICML 2025) allocates LoRA experts per layer from gradient-norm sensitivity before each task, closely matching Eqs. 12–13. None is cited or compared. SEFE’s data-side fix would have been a clear baseline: if diversifying answer styles in the data removes format drift, the case for format-aware routing depends on beating it.

The paper does not report which tasks were grouped together on TriGap or UCIT, or how many groups formed. If every task gets its own group, ProtoAda reduces to per-task LoRA plus retrieval. Semantic-only ProtoAda is never evaluated on the real benchmarks. Appending any feature that is constant within a task to per-sample embeddings increases task separability. This affects Fig. 2 and Fig. 6, where the separability index (SI) is also undefined.

Questions for authors

Which task-to-group assignments formed on TriGap and UCIT under each task order? What is ProtoAda’s accuracy on these benchmarks with semantic-only prototypes? How does ProtoAda compare with SEFE, or with SEFE’s answer-style diversification applied to SAME, and with D-MoLE? What is the cosine similarity between format codes of different tasks?

7 comments

Response to Reviewer DSWk

Thank you for recognizing the controlled FmtGap/VisGap study and for identifying closely related work that deserves a more explicit discussion. We address the positioning, actual task grouping, and missing comparisons below.

W1. Relationship to CoIN, SEFE, and D-MoLE

We agree that our related-work discussion is incomplete. In particular, we do not claim to be the first to identify instruction-following or answer-style forgetting, nor do we claim gradient-based expert allocation as an independent novelty.

  • CoIN [1]. CoIN distinguishes failures of instruction alignment from losses of reasoning knowledge and demonstrates that the former are an important source of forgetting. Our work builds on this observation. The narrower question we investigate is how response-protocol compatibility should enter parameter-sharing decisions when tasks have similar image–text semantics. Our controlled streams isolate this compatibility issue, and our method makes protocol information explicit in task prototypes and retrieval.
  • SEFE [2]. SEFE explicitly studies superficial and essential forgetting. Its Answer Style Diversification changes the training distribution so that tasks have similarly diversified answer styles; RegLoRA additionally constrains weight updates. ProtoAda instead uses response statistics as a routing and memory-organization signal, without requiring answer-style rewriting of the incoming training set. Thus, the distinction is not the recognition that output style matters, but the intervention: data diversification and regularization versus protocol-aware parameter sharing and update consolidation. These approaches may also be complementary.
  • D-MoLE [3]. D-MoLE uses dynamic layer-wise LoRA expert allocation and a gradient-based inter-modal curriculum. We acknowledge the overlap with sensitivity-based allocation. The intended distinction of ProtoAda is its explicit response-protocol compatibility signal and its organization of effective updates into group memory and task residuals. We will clarify that sensitivity-based allocation alone is not our novelty claim.

We will revise the related work and contribution statements accordingly.

Q1. Actual task-to-group assignments

The task-to-group assignments for the evaluated task orders are as follows. Group membership is listed without arbitrary group identifiers, which can change with task order.

Benchmark / orderGroup membershipNumber of groups
UCIT / Original{ImageNet-R}; {ArxivQA}; {VizCap, Flickr30k}; {IconQA}; {CLEVR}5
UCIT / Reverse{ImageNet-R}; {ArxivQA}; {VizCap, Flickr30k}; {IconQA}; {CLEVR}5
UCIT / Random{ImageNet-R}; {ArxivQA}; {VizCap, Flickr30k}; {IconQA}; {CLEVR}5
TriGap / OriginalOne singleton group for each of the ten tasks10

On UCIT, VizCap and Flickr30k share a group under all three evaluated orders, producing five groups for six tasks. On TriGap, the original-order run produces ten singleton groups. Thus, the method permits task sharing but does not force it when the joint semantic-and-format representation favors isolation.

The stage-wise UCIT evaluation provides a concrete example of beneficial reuse within the shared captioning group: after learning Flickr30k, the sixth task, VizCap's score increases from 59.91 to 61.49 (+1.58 score points). The full trajectory is provided in our response to Reviewer KFP3, Q2. This is consistent with collaboration between tasks with compatible response protocols, although the trajectory alone does not isolate the contribution of the format code from other components.

We agree that the TriGap grouping alone does not demonstrate within-group collaboration. Nevertheless, singleton grouping does not reduce the full framework to independently stored task LoRAs plus retrieval: periodic cross-group consolidation can promote common components into global shared memory and remove those components from the group adapters. The grouping threshold controls the balance between reuse and isolation; the number of groups is an outcome of the stream and configuration, rather than a requirement to produce a multi-task group in every benchmark. We will make the actual assignments explicit in the manuscript.

Q2. Semantic-only ProtoAda on UCIT

We agree that the controlled streams and prototype visualizations do not replace a benchmark-level ablation of the complete framework. We have now evaluated a semantic-only ProtoAda variant on UCIT that removes the format-code contribution from prototype construction while retaining the adaptive and geometric consolidation components. This tests the prototype signal within ProtoAda, rather than substituting the MoELoRA baseline in Table 3 for a full-framework ablation.

VariantUCIT final average score
ProtoAda, semantic-only prototypes72.30
ProtoAda, format-aware prototypes74.66

The complete UCIT per-task results are:

VariantImageNet-RArxivQAVizCapIconQACLEVRFlickr30kMean
ProtoAda, semantic-only prototypes89.4391.5762.6479.5756.1054.4872.30
ProtoAda, format-aware prototypes89.8393.2061.4981.3766.4055.6474.66
Format-aware minus semantic-only+0.40+1.63-1.15+1.80+10.30+1.16+2.36

The semantic-only mean is 72.2983, versus 74.655 for full ProtoAda, a gain of 2.36 score points after rounding. Format-aware prototypes improve five of the six task scores, with the largest gain on CLEVR (+10.30 points). The benefit is not uniform: VizCap is 1.15 points lower than with semantic-only prototypes. We report this trade-off rather than claim that every task improves. Each task uses its original UCIT metric; the caption scores and their contribution to the mean are not sample-level accuracy percentages.

These measured results support the downstream value of explicitly incorporating response-protocol information into task prototypes, beyond increased prototype separability alone.

Q3. Comparison with SEFE and style-diversified baselines

We agree that SEFE is a directly relevant baseline. For a quantitative reference, MCITlib [4, Table 3(a)] reports 66.54% final average performance for SEFE on UCIT with LLaVA-1.5, compared with 74.66% for ProtoAda in our submitted result. The numerical difference is 8.12 percentage points.

MethodUCIT final average (%)Source / status
SEFE66.54Reported by MCITlib [4]
ProtoAda74.66Submitted single-run result

Methodologically, SEFE and ProtoAda intervene at different levels: the training distribution and the parameter-sharing structure, respectively. SEFE uses answer-style diversification to improve adaptation to varied response forms, together with regularization to constrain sequential updates. ProtoAda instead incorporates protocol compatibility into expert assignment and retrieval, so that sharing depends on both semantic similarity and output requirements. For semantically similar tasks with different protocols, this design helps avoid incompatible sharing; for compatible tasks, it retains opportunities for knowledge reuse. ProtoAda therefore balances task isolation and knowledge sharing without rewriting the original task answers. The UCIT prototype ablation provides downstream evidence for this design: adding the format signal increases the final mean from 72.30 to 74.66 (+2.36 score points).

Q4. Format-code similarities and the separability index

We computed the pairwise cosine similarities of the raw task-level format codes, rather than the fused semantic-and-format task prototypes:

\frac{\langle c_i^{\mathrm{fmt}},c_j^{\mathrm{fmt}}\rangle} {\|c_i^{\mathrm{fmt}}\|_2\|c_j^{\mathrm{fmt}}\|_2}.$$ The complete matrices are given below, rounded to four decimal places. The column indices correspond to the task indices in the row labels. **UCIT: format-code cosine similarities.** | Task | 0 | 1 | 2 | 3 | 4 | 5 | | ------------- | -----: | -----: | -----: | -----: | -----: | -----: | | 0: ImageNet-R | 1.0000 | 0.9298 | 0.9853 | 0.7542 | 0.7472 | 0.9879 | | 1: ArxivQA | 0.9298 | 1.0000 | 0.8926 | 0.7812 | 0.7848 | 0.9108 | | 2: VizCap | 0.9853 | 0.8926 | 1.0000 | 0.7376 | 0.7285 | 0.9988 | | 3: IconQA | 0.7542 | 0.7812 | 0.7376 | 1.0000 | 0.9995 | 0.7418 | | 4: CLEVR | 0.7472 | 0.7848 | 0.7285 | 0.9995 | 1.0000 | 0.7339 | | 5: Flickr30k | 0.9879 | 0.9108 | 0.9988 | 0.7418 | 0.7339 | 1.0000 | **TriGap: format-code cosine similarities.** | Task | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | | ----------------- | -----: | -----: | -----: | -----: | -----: | -----: | -----: | -----: | -----: | -----: | | 0: PMCVQA | 1.0000 | 0.6209 | 0.6061 | 0.6523 | 0.6873 | 0.9961 | 0.7802 | 0.6963 | 0.8738 | 0.6574 | | 1: DocVQA | 0.6209 | 1.0000 | 0.9934 | 0.9690 | 0.9900 | 0.6561 | 0.9052 | 0.9863 | 0.8992 | 0.9659 | | 2: ChartQA | 0.6061 | 0.9934 | 1.0000 | 0.9823 | 0.9768 | 0.6354 | 0.8596 | 0.9678 | 0.8773 | 0.9794 | | 3: IconQA | 0.6523 | 0.9690 | 0.9823 | 1.0000 | 0.9582 | 0.6746 | 0.8371 | 0.9584 | 0.8819 | 0.9999 | | 4: InfographicVQA | 0.6873 | 0.9900 | 0.9768 | 0.9582 | 1.0000 | 0.7187 | 0.9398 | 0.9901 | 0.9385 | 0.9561 | | 5: ArxivQA | 0.9961 | 0.6561 | 0.6354 | 0.6746 | 0.7187 | 1.0000 | 0.8196 | 0.7338 | 0.8958 | 0.6791 | | 6: Roadside | 0.7802 | 0.9052 | 0.8596 | 0.8371 | 0.9398 | 0.8196 | 1.0000 | 0.9515 | 0.9642 | 0.8359 | | 7: ChemVQA | 0.6963 | 0.9863 | 0.9678 | 0.9584 | 0.9901 | 0.7338 | 0.9515 | 1.0000 | 0.9400 | 0.9570 | | 8: FloodNetVQA | 0.8738 | 0.8992 | 0.8773 | 0.8819 | 0.9385 | 0.8958 | 0.9642 | 0.9400 | 1.0000 | 0.8825 | | 9: CLEVR | 0.6574 | 0.9659 | 0.9794 | 0.9999 | 0.9561 | 0.6791 | 0.8359 | 0.9570 | 0.8825 | 1.0000 | The two most similar pairs on UCIT are **IconQA–CLEVR (0.9995)** and **VizCap–Flickr30k (0.9988)**; on TriGap, they are **IconQA–CLEVR (0.9999)** and **PMCVQA–ArxivQA (0.9961)**. These are similarities of the response-statistics codes, not complete measures of task compatibility. For example, IconQA and CLEVR have very similar format codes but remain in separate UCIT groups: grouping uses the joint visual, textual, and format prototype, not format-code similarity alone. **Definition of SI.** At a given stage, let $p_1,\ldots,p_K$ be the task prototypes being compared, with $K\ge2$. The separability index is their mean pairwise cosine distance: $$\mathrm{SI}=\frac{2}{K(K-1)} \sum_{1\le i<j\le K}\left(1-\cos(p_i,p_j)\right).$$ Larger SI therefore means greater average angular separation among the task prototypes. It is distinct from the predicted-versus-oracle format-code cosine similarity shown alongside it in Figure 6, which measures prediction alignment rather than inter-task separation. We will state this definition explicitly in the manuscript. We agree that appending any task-constant feature can increase separability. Accordingly, SI and the visualizations are descriptive evidence, not by themselves proof of a protocol-specific benefit. But the completed full-framework UCIT ablation in Q2 provides downstream performance evidence: using format-aware rather than semantic-only prototypes increases the final mean from **72.30 to 74.66 (+2.36 score points)**. ## References [1] Cheng Chen et al. **CoIN: A Benchmark of Continual Instruction Tuning for Multimodal Large Language Models.** NeurIPS 2024, Datasets and Benchmarks Track. [2] Jinpeng Chen et al. **SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning.** ICML 2025, PMLR 267:7982–8001; arXiv:2505.02486. [3] Chendi Ge et al. **Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning.** ICML 2025, PMLR 267:19011–19033; arXiv:2506.11672. [4] Haiyang Guo et al. **MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark.** arXiv:2508.07307v3, UCIT results in Table 3(a). Used as the literature source for the SEFE 66.54 entry, not as evidence of a matched local reproduction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.