- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies replay-free multimodal continual instruction tuning, aiming to balance new-task learning, retention of previously acquired knowledge, and control of expert growth. The authors propose PACE, which constructs task statistics from frozen image, instruction, and response features and combines a Chinese Restaurant Process prior with empirical Bayes predictive evidence to decide whether to reuse an existing LoRA expert or create a new one before training on each task. Only the selected expert is updated. At inference, PACE uses input features and stored task statistics to select a single expert, without requiring task identity or training a separate router. The paper also analyzes the effects of expert sharing on forgetting and generalization under a simplified linear regression setting and evaluates performance, knowledge retention, and computational costs across four benchmarks using two backbone architectures.
Critical strengths
1 Continual learning must address forgetting, expert growth, and inference-time expert selection. PACE handles these issues within a single framework without training a separate router, resulting in a relatively simple design.
2 The paper evaluates PACE on four benchmarks using two backbone architectures and analyzes forgetting associated with expert updates, changes in expert selection, and shifts in response behavior.
Critical weaknesses
1 Fig. 7 shows that PACE and alternative rules, such as cosine and Euclidean similarity, produce different task groupings, but does not report performance after training with these groupings. It therefore remains unclear whether probabilistic assignment outperforms simpler similarity-based rules.
2 The theorems establish conditions under which expert sharing can be beneficial, but do not show that PACE’s assignment rule can identify tasks satisfying these conditions.
3 The authors modify the feature extraction, data augmentation, or LoRA target layers of several baselines without adequately quantifying the effects of these changes on performance. Some cited methods that also address dynamic expert allocation are absent from the experimental comparisons.
4 The authors acknowledge that assignments depend on task order but do not evaluate alternative orders. The experiments contain at most ten tasks, and PACE uses ten experts on TriGap. These results are insufficient to establish whether the method can maintain performance while controlling expert growth over longer task sequences.
Questions for authors
see Weaknesses
4 comments
Response to Reviewer hMV8 (2/3): Baseline implementation and comparison with ProtoAda and CRAM
W3. Baseline implementation and additional comparison with ProtoAda and CRAM.
Our reproduced baselines use publicly available code and the methodological descriptions in their original papers. Appendix C.3 in the original submission was intended as a concise summary of key parameters. We have now expanded it to explicitly describe the method-specific training and inference procedures implemented in our code and the adaptations to our common evaluation protocol (Appendix C.3, Lines 1560-1619). For SMoLoRA, we use projected CLIP text embeddings in place of Sentence-BERT to reuse the frozen text encoder already used by HiDe-LLaVA and DISCO in our evaluation pipeline. This preserves instruction routing from each sample's input while avoiding an additional text encoder. We apply LoRA to the common attention and feed-forward projections so that equal total rank gives a matched allocated LoRA parameter budget across reproduced baselines on each backbone. We will release our implementation code after acceptance. The matched six-expert comparison in W1 independently supports the assignment rule by holding the rest of the pipeline fixed.
Additional comparison with ProtoAda and CRAM. We additionally compare PACE with ProtoAda [1] and CRAM [2], two methods that also allocate experts dynamically, on TriGap with LLaVA-1.5-7B, using the averages reported in their papers. Table 19 provides all task scores.
| Method | Reported average | PACE improvement |
|---|---|---|
| ProtoAda [1] | 47.23 | +1.84 |
| CRAM [2] | 47.79 | +1.28 |
| PACE | 49.07 | - |
Results. On TriGap, PACE achieves an average score of 49.07, exceeding ProtoAda and CRAM by 1.84 and 1.28 points, respectively. PACE also outperforms ProtoAda on eight of ten tasks and CRAM on nine of ten tasks.
We have added this empirical comparison in Appendix Section E "Comparison with ProtoAda and CRAM" (Lines 2029-2090).
[1] Shi et al. ProtoAda: Prototype-guided adaptive adapter expansion and geometric consolidation for multimodal continual instruction tuning. arXiv:2606.02576, 2026.
[2] Tang et al. CRAM: Centroid-routing and adaptive MoE for multimodal continual instruction tuning. arXiv:2606.02502, 2026.
Response to Reviewer hMV8 (3/3): Robustness to task order and longer task sequences
W4. Task order and longer task sequences.
Additional task order experiment. We evaluated UCIT with Qwen2.5-VL under three task orders: canonical (the original benchmark sequence), reversed (the canonical sequence in reverse), and random (one random permutation of the same tasks). All methods use 10% of each task's training data and 300 evaluation examples per task. This reduced-data setting differs from Table 2 (LLaVA-1.5-7B), so values should be compared within Table 6, summarized below. The PACE runs in this experiment include the response-coverage check that can reject reuse.
AA is the final average accuracy (higher is better) and FGT is forgetting (lower is better). The last column reports each method's largest FGT over the three orders.
| Method | Canonical AA | Reversed AA | Random AA | Largest FGT across orders |
|---|---|---|---|---|
| PACE | 69.61 | 70.20 | 69.49 | 0.07 |
| SEFE | 68.29 | 65.03 | 68.92 | 5.64 |
| O-LoRA | 66.15 | 68.53 | 65.97 | 4.25 |
| CL-MoE | 65.75 | 64.17 | 64.32 | 1.87 |
Results. PACE achieves the highest accuracy and lowest forgetting across all three task orders. Its AA changes by at most 0.71 points between orders, compared with 1.58 to 3.89 points for the three baselines, and its largest FGT is 0.07 against 1.87 to 5.64. Its consistently strong performance shows that effective learning and retention persist when the task sequence changes, supporting robustness across the evaluated orders. We added this experiment in our revised submission in Section 5.2 (Lines 494-521).
Longer task sequences on FLEX. We directly address the longer-sequence concern with a completed experiment on all 34 FLEX tasks using LLaVA-1.5. PACE trains through in-place expert reuse under a fixed allocation threshold and achieves the following result (Table 15):
| Method | Populated experts | Mean final score |
|---|---|---|
| PACE | 9 | 52.66 |
| PureLoRA [1] | 34 | 54.60 |
Results. PACE uses 73.5% fewer populated experts than one-expert-per-task allocation while achieving comparable performance to the published PureLoRA reference. This demonstrates that effective expert sharing extends to longer sequences, allowing PACE to consolidate many tasks into a small set of shared experts while maintaining strong final performance. We added this new experiment in our revised submission, Appendix Section D.4. Longer Task Sequences on FLEX (Lines 1914-1938).
[1] Yi et al. Beyond routing saturation: A long-horizon class-incremental perspective on expert routing in multimodal continual instruction tuning. arXiv:2608.01437, 2026.
Follow-up on our responses
Dear Reviewer hMV8,
Thank you again for your review. We have posted responses to each of the four weaknesses you raised: downstream performance of the alternative grouping rules, whether the assignment rule identifies tasks that satisfy the sharing conditions, the baselines and missing comparisons, task-order sensitivity analysis, and longer task sequences.
As the discussion period is closing soon, we would be grateful if you could let us know whether these responses resolve your concerns, or whether any point remains open so that we can address it in the remaining time. If the concerns are resolved, we kindly ask you to reconsider your assessment and rating.
Thank you for your time and consideration.
Best regards,
The Authors
Response to Reviewer hMV8 (1/3): Probabilistic assignment under a matched budget and the role of the theory
We thank the reviewer for the careful reading and the constructive comments. All table, appendix, equation, and line numbers in our responses refer to the revised PDF.
W1. Performance of probabilistic assignment.
We evaluated all five assignment rules on CoIN with Qwen2.5-VL-7B and 6 rank-24 experts. We replay each rule on the same stored task statistics and train the resulting assignments in the same trainer. All runs follow the training protocol in Table 7, use seed 42 as in Tables 1 and 2, and share the same 25% test subsample and single-expert selection rule (Eq. 6). We added this new experiment in our revised submission, Appendix Section D.3.3 Effect of Assignment Rules (Lines 1817-1845).
Table 11 reports the results. AA is the final average accuracy and BWT is backward transfer (higher is better for both); FGT is forgetting (lower is better).
Results. PACE achieves the highest accuracy and backward transfer and the lowest forgetting among all five assignment rules under the same six-expert, rank-24 budget. Its AA exceeds the four alternatives by 0.96 to 2.65 points, and its forgetting is 0.37 to 2.85 points lower. Because expert capacity, training, and inference-time selection are held fixed, these results directly support the benefit of the probabilistic assignment rule for choosing which tasks share an expert.
W2. Purpose of the theory and evidence for assignment.
Theorems 1–2 analyze sharing under a fixed assignment in the stated linear model. Theorem 1 characterizes forgetting within shared experts; Theorem 2 balances reduced initialization error against task mismatch and gives a sufficient condition for sharing to improve generalization. For nk≥2 tasks sharing expert k, sharing reduces the group's expected generalization error relative to nk isolated experts with the same initialization errors whenever
Δk2<1−ϱnkϱ−ϱnkξˉk2,
where Δk is the largest distance between the optimal adapters of the tasks in the group, ξˉk2 is their mean initialization error, and ϱ=1−N/p∈(0,1) for N samples per task and adapter dimension p. These results provide a theoretical basis for selective sharing, complemented by direct evidence for Equation (5) from our matched six-expert experiments.
Appendix B.6 establishes the reliability and information content of the allocation representation. Under its assumptions, frozen task statistics concentrate around population values and remain unchanged by adapter optimization. Response features can distinguish tasks with identical input-feature distributions, with that separation preserved empirically given sufficient samples. Appendix B.7 gives the condition under which closeness in these statistics bounds the optimal-adapter distance Δk.
Our new six-expert comparison (in W1) directly tests assignment effectiveness. With identical features, six rank-24 experts, training and selection, PACE achieves 78.97 AA versus 76.32 for cosine-threshold and 78.01 for Euclidean-threshold assignment. Its BWT/FGT are −0.61/0.71 versus −3.55/3.56 and −1.34/1.41, respectively. These downstream gains isolate the allocation choice at matched capacity.