acceptodds
Under review as a conference paper at ICLR 2027

PACE: Probabilistic Expert Assignment for Multimodal Continual Instruction Tuning

Abstract

Multimodal continual instruction tuning requires models to acquire new capabilities while retaining previously learned ones. Expert-based methods mitigate catastrophic forgetting through parameter isolation, but expert allocation often relies on heuristic or method-specific criteria, while inference-time selection requires a separate routing mechanism. We propose **Probabilistic Assignment for Continual Experts (PACE)**, a router-free method that provides a probabilistic mechanism for selective expert sharing, built from a Chinese Restaurant Process prior and closed-form parametric empirical-Bayes evidence in a frozen multimodal feature space. Before adaptation, PACE determines whether an arriving task should reuse an existing LoRA expert or create a new one, after which only the assigned expert is updated. At inference, the same task statistics enable training-free selection of a single expert, eliminating routing drift and expert interference. We further provide a theoretical analysis of expert sharing, showing that cross-task interference is confined to tasks sharing the same expert and establishing sufficient conditions under which sharing improves generalization over task-wise isolation. Experiments across four continual multimodal benchmarks, CoIN, UCIT, TriGap, and MLLM-CL, demonstrate strong performance and knowledge retention across diverse task sequences. Further analyses show that effective expert sharing depends critically on which tasks are grouped together: PACE preserves task-specific response behavior, avoids highly interfering assignments, and reduces computation through selective adaptation and single-expert generation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Parameter-Efficient Fine-Tuning, Mixture-of-Experts, Expert Allocation, Foundation Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hMV8OpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies replay-free multimodal continual instruction tuning, aiming to balance new-task learning, retention of previously acquired knowledge, and control of expert growth. The authors propose PACE, which constructs task statistics from frozen image, instruction, and response features and combines a Chinese Restaurant Process prior with empirical Bayes predictive evidence to decide whether to reuse an existing LoRA expert or create a new one before training on each task. Only the selected expert is updated. At inference, PACE uses input features and stored task statistics to select a single expert, without requiring task identity or training a separate router. The paper also analyzes the effects of expert sharing on forgetting and generalization under a simplified linear regression setting and evaluates performance, knowledge retention, and computational costs across four benchmarks using two backbone architectures.

Critical strengths

1 Continual learning must address forgetting, expert growth, and inference-time expert selection. PACE handles these issues within a single framework without training a separate router, resulting in a relatively simple design.

2 The paper evaluates PACE on four benchmarks using two backbone architectures and analyzes forgetting associated with expert updates, changes in expert selection, and shifts in response behavior.

Critical weaknesses

1 Fig. 7 shows that PACE and alternative rules, such as cosine and Euclidean similarity, produce different task groupings, but does not report performance after training with these groupings. It therefore remains unclear whether probabilistic assignment outperforms simpler similarity-based rules.

2 The theorems establish conditions under which expert sharing can be beneficial, but do not show that PACE’s assignment rule can identify tasks satisfying these conditions.

3 The authors modify the feature extraction, data augmentation, or LoRA target layers of several baselines without adequately quantifying the effects of these changes on performance. Some cited methods that also address dynamic expert allocation are absent from the experimental comparisons.

4 The authors acknowledge that assignments depend on task order but do not evaluate alternative orders. The experiments contain at most ten tasks, and PACE uses ten experts on TriGap. These results are insufficient to establish whether the method can maintain performance while controlling expert growth over longer task sequences.

Questions for authors

see Weaknesses

4 comments
AuthorsOpenReview ↗

Response to Reviewer hMV8 (1/3): Probabilistic assignment under a matched budget and the role of the theory

We thank the reviewer for the careful reading and the constructive comments. All table, appendix, equation, and line numbers in our responses refer to the revised PDF.

W1. Performance of probabilistic assignment.

We evaluated all five assignment rules on CoIN with Qwen2.5-VL-7B and 6 rank-24 experts. We replay each rule on the same stored task statistics and train the resulting assignments in the same trainer. All runs follow the training protocol in Table 7, use seed 42 as in Tables 1 and 2, and share the same 25% test subsample and single-expert selection rule (Eq. 6). We added this new experiment in our revised submission, Appendix Section D.3.3 Effect of Assignment Rules (Lines 1817-1845).

Table 11 reports the results. AA is the final average accuracy and BWT is backward transfer (higher is better for both); FGT is forgetting (lower is better).

Allocation rulePopulated expertsAABWTFGT
PACE with Bayesian678.97-0.610.71
Cosine threshold676.32-3.553.56
Euclidean threshold678.01-1.341.41
Nearest centroid, Euclidean677.74-0.981.08
Nearest centroid, cosine677.82-0.921.15

Results. PACE achieves the highest accuracy and backward transfer and the lowest forgetting among all five assignment rules under the same six-expert, rank-24 budget. Its AA exceeds the four alternatives by 0.96 to 2.65 points, and its forgetting is 0.37 to 2.85 points lower. Because expert capacity, training, and inference-time selection are held fixed, these results directly support the benefit of the probabilistic assignment rule for choosing which tasks share an expert.

W2. Purpose of the theory and evidence for assignment.

Theorems 1–2 analyze sharing under a fixed assignment in the stated linear model. Theorem 1 characterizes forgetting within shared experts; Theorem 2 balances reduced initialization error against task mismatch and gives a sufficient condition for sharing to improve generalization. For nk≥2n_k \ge 2 tasks sharing expert kk, sharing reduces the group's expected generalization error relative to nkn_k isolated experts with the same initialization errors whenever

Δk2<ϱ−ϱnk1−ϱnk ξˉk2,\Delta_k^2 < \frac{\varrho - \varrho^{n_k}}{1 - \varrho^{n_k}}\,\bar{\xi}_k^2,

where Δk\Delta_k is the largest distance between the optimal adapters of the tasks in the group, ξˉk2\bar{\xi}_k^2 is their mean initialization error, and ϱ=1−N/p∈(0,1)\varrho = 1 - N/p \in (0,1) for NN samples per task and adapter dimension pp. These results provide a theoretical basis for selective sharing, complemented by direct evidence for Equation (5) from our matched six-expert experiments.

Appendix B.6 establishes the reliability and information content of the allocation representation. Under its assumptions, frozen task statistics concentrate around population values and remain unchanged by adapter optimization. Response features can distinguish tasks with identical input-feature distributions, with that separation preserved empirically given sufficient samples. Appendix B.7 gives the condition under which closeness in these statistics bounds the optimal-adapter distance Δk\Delta_k.

Our new six-expert comparison (in W1) directly tests assignment effectiveness. With identical features, six rank-24 experts, training and selection, PACE achieves 78.97 AA versus 76.32 for cosine-threshold and 78.01 for Euclidean-threshold assignment. Its BWT/FGT are −0.61/0.71 versus −3.55/3.56 and −1.34/1.41, respectively. These downstream gains isolate the allocation choice at matched capacity.

AuthorsOpenReview ↗

Response to Reviewer hMV8 (2/3): Baseline implementation and comparison with ProtoAda and CRAM

W3. Baseline implementation and additional comparison with ProtoAda and CRAM.

Our reproduced baselines use publicly available code and the methodological descriptions in their original papers. Appendix C.3 in the original submission was intended as a concise summary of key parameters. We have now expanded it to explicitly describe the method-specific training and inference procedures implemented in our code and the adaptations to our common evaluation protocol (Appendix C.3, Lines 1560-1619). For SMoLoRA, we use projected CLIP text embeddings in place of Sentence-BERT to reuse the frozen text encoder already used by HiDe-LLaVA and DISCO in our evaluation pipeline. This preserves instruction routing from each sample's input while avoiding an additional text encoder. We apply LoRA to the common attention and feed-forward projections so that equal total rank gives a matched allocated LoRA parameter budget across reproduced baselines on each backbone. We will release our implementation code after acceptance. The matched six-expert comparison in W1 independently supports the assignment rule by holding the rest of the pipeline fixed.

Additional comparison with ProtoAda and CRAM. We additionally compare PACE with ProtoAda [1] and CRAM [2], two methods that also allocate experts dynamically, on TriGap with LLaVA-1.5-7B, using the averages reported in their papers. Table 19 provides all task scores.

MethodReported averagePACE improvement
ProtoAda [1]47.23+1.84
CRAM [2]47.79+1.28
PACE49.07-

Results. On TriGap, PACE achieves an average score of 49.07, exceeding ProtoAda and CRAM by 1.84 and 1.28 points, respectively. PACE also outperforms ProtoAda on eight of ten tasks and CRAM on nine of ten tasks.

We have added this empirical comparison in Appendix Section E "Comparison with ProtoAda and CRAM" (Lines 2029-2090).

[1] Shi et al. ProtoAda: Prototype-guided adaptive adapter expansion and geometric consolidation for multimodal continual instruction tuning. arXiv:2606.02576, 2026.

[2] Tang et al. CRAM: Centroid-routing and adaptive MoE for multimodal continual instruction tuning. arXiv:2606.02502, 2026.

AuthorsOpenReview ↗

Response to Reviewer hMV8 (3/3): Robustness to task order and longer task sequences

W4. Task order and longer task sequences.

Additional task order experiment. We evaluated UCIT with Qwen2.5-VL under three task orders: canonical (the original benchmark sequence), reversed (the canonical sequence in reverse), and random (one random permutation of the same tasks). All methods use 10% of each task's training data and 300 evaluation examples per task. This reduced-data setting differs from Table 2 (LLaVA-1.5-7B), so values should be compared within Table 6, summarized below. The PACE runs in this experiment include the response-coverage check that can reject reuse.

AA is the final average accuracy (higher is better) and FGT is forgetting (lower is better). The last column reports each method's largest FGT over the three orders.

MethodCanonical AAReversed AARandom AALargest FGT across orders
PACE69.6170.2069.490.07
SEFE68.2965.0368.925.64
O-LoRA66.1568.5365.974.25
CL-MoE65.7564.1764.321.87

Results. PACE achieves the highest accuracy and lowest forgetting across all three task orders. Its AA changes by at most 0.71 points between orders, compared with 1.58 to 3.89 points for the three baselines, and its largest FGT is 0.07 against 1.87 to 5.64. Its consistently strong performance shows that effective learning and retention persist when the task sequence changes, supporting robustness across the evaluated orders. We added this experiment in our revised submission in Section 5.2 (Lines 494-521).

Longer task sequences on FLEX. We directly address the longer-sequence concern with a completed experiment on all 34 FLEX tasks using LLaVA-1.5. PACE trains through in-place expert reuse under a fixed allocation threshold and achieves the following result (Table 15):

MethodPopulated expertsMean final score
PACE952.66
PureLoRA [1]3454.60

Results. PACE uses 73.5% fewer populated experts than one-expert-per-task allocation while achieving comparable performance to the published PureLoRA reference. This demonstrates that effective expert sharing extends to longer sequences, allowing PACE to consolidate many tasks into a small set of shared experts while maintaining strong final performance. We added this new experiment in our revised submission, Appendix Section D.4. Longer Task Sequences on FLEX (Lines 1914-1938).

[1] Yi et al. Beyond routing saturation: A long-horizon class-incremental perspective on expert routing in multimodal continual instruction tuning. arXiv:2608.01437, 2026.

AuthorsOpenReview ↗

Follow-up on our responses

Dear Reviewer hMV8,

Thank you again for your review. We have posted responses to each of the four weaknesses you raised: downstream performance of the alternative grouping rules, whether the assignment rule identifies tasks that satisfy the sharing conditions, the baselines and missing comparisons, task-order sensitivity analysis, and longer task sequences.

As the discussion period is closing soon, we would be grateful if you could let us know whether these responses resolve your concerns, or whether any point remains open so that we can address it in the remaining time. If the concerns are resolved, we kindly ask you to reconsider your assessment and rating.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_T2kJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PACE, a replay-free approach to multimodal continual instruction tuning that allocates and accesses LoRA experts using frozen multimodal task statistics. At each task boundary, PACE uses a Chinese Restaurant Process occupancy prior and closed-form empirical-Bayes predictive scores to decide whether to reuse an existing expert or create a new one, and only the selected expert is adapted. At inference, a pooled-covariance Gaussian classifier over stored input-feature prototypes identifies a task and activates one associated expert, without training a learned router. The paper analyzes forgetting and generalization under an overparameterized linear-regression model and reports competitive accuracy and retention on CoIN, UCIT, TriGap, and MLLM-CL.

Critical strengths
  1. A coherent and practical expert allocation/selection design. PACE determines expert reuse or expansion before adapter optimization using stored frozen statistics, and generates with a single expert without continually training a learned router. This is a well-motivated way to address capacity growth, routing stability, and multi-expert inference overhead.

  2. Competitive results across diverse benchmarks. The reported final averages of 78.97 on CoIN, 68.19 on UCIT, and 49.07 on TriGap, together with strong results on MLLM-CL, suggest the full system is effective across multiple task streams. The forgetting and response-behavior analyses provide useful additional context.

Critical weaknesses
  1. The central benefit of the Bayesian expert assignment rule is not sufficiently isolated. Table 3 shows only a 0.30-point CoIN accuracy improvement over one expert per task (78.97 vs. 78.67), while the isolated-expert variant has better BWT and FGT. Figure 7 compares the partitions induced by cosine and Euclidean assignments at the same number of experts, but reports only ARI relative to PACE, not their downstream accuracy or retention under a matched expert/rank/selection budget. Table 4 further changes expert count and rank in some representation ablations. Thus, the strong end-to-end results do not yet establish that the proposed CRP/empirical-Bayes decision is better than simpler fixed-budget alternatives. This matters because that decision rule is the main claimed methodological contribution.

  2. The theory and the empirical assignment criterion are not convincingly connected to sharing compatibility. Theorem 1 and Theorem 2 describe general effects of sharing for fixed assignments in a simplified linear-regression setting; they do not guarantee that the assignments selected by Eq. (5) have small optimal-adapter distances. Appendix B.7 explicitly leaves the feature-to-adapter distance assumption unverified, and Figure 6 reports essentially no correlation between the assignment score and measured cross-task interference (Spearman −0.10, p=0.61). In addition, the theoretical independence assumption between allocation statistics and adaptation samples needs qualification because Appendix C.2 reuses the 256 summarization examples for adaptation. These issues do not necessarily invalidate the theorems, but limit their support for the practical mechanism.

Questions for authors

Controlled assignment comparison: For the cosine and Euclidean allocation rules in Fig. 7, could you report their downstream AA/BWT/FGT under exactly the same six experts, rank 24, training protocol, and single-expert inference rule as PACE? In particular, how much benefit comes from the CRP/empirical-Bayes criterion rather than merely selecting a reasonable six-expert partition?

Theory-to-practice link: Given the weak score–interference relationship in Fig. 6 and the unverified Assumption 5 in Appendix B.7, what evidence supports the claim that PACE's selected groups satisfy the sharing conditions motivating Theorem 2? Also, how should the independence condition in Assumption 3 be interpreted when the 256 examples used for task statistics are also used for adaptation?

Task-free inference protocol: Please clarify how the prompt-line frequency filtering in Appendix C.2 is performed for individual inference queries without task-identifying test-file grouping, whether PACE trains the shared projector on UCIT, and whether task/expert selection accuracy or an oracle-selection comparison is available. These details would affect confidence in the reported task-agnostic inference behavior.

5 comments
AuthorsOpenReview ↗

Response to Reviewer T2kJ (1/4): W1/Q1, assignment rules compared at a fixed six-expert budget

We thank the reviewer for the careful reading of our paper and for the detailed questions. All table, figure, appendix, and line numbers in our responses refer to the revised PDF. Below we clarify the scope of our work and theoretical analysis, and report additional experiments.

1. W1 and Q1: isolating the effect of the assignment rule

What Table 3 compares. Table 3 evaluates whether selective expert sharing can preserve accuracy while reducing the number of experts. At the reported seed, PACE achieves 78.97 AA with six experts, compared with 78.67 AA using eight task-specific experts. Table 12 shows that this reduces populated LoRA parameters from 484.4M to 363.3M, a 25% reduction. The central result is therefore maintaining accuracy while sharing experts across tasks; the 0.30-point difference should be interpreted alongside this reduction in capacity.

Task-wise isolation prevents interference from subsequent expert updates: Theorem 1 establishes zero forgetting from such updates when K=TK=T, because earlier experts remain unchanged. However, at a fixed expert size, this requires capacity to grow linearly with the number of tasks. Reusing experts across tasks is an established motivation in continual learning. CL-MoE (Huai et al., 2025) explicitly motivates experts that serve multiple tasks, while D-MoLE (Ge et al., 2025b) promotes knowledge sharing through expert allocation under a controlled parameter budget. We revised the description of this table to clarify this in Lines 424-434.

New experiment: assignment rules at a fixed budget

We evaluated all five assignment rules on CoIN with Qwen2.5-VL-7B and 6 rank-24 experts. We replay each rule on the same stored task statistics and train the resulting assignments in the same trainer. All runs follow the training protocol in Table 7, use seed 42 as in Tables 1 and 2, and share the same 25% test subsample and single-expert selection rule (Eq. 6). We added this new experiment in our revised submission, Appendix D.3.3 Effect of Assignment Rules (Table 11, Lines 1817-1845).

Allocation rulePopulated expertsAA ↑BWT ↑FGT ↓
PACE with Bayesian678.97−0.610.71
Cosine threshold676.32−3.553.56
Euclidean threshold678.01−1.341.41
Nearest centroid, Euclidean677.74−0.981.08
Nearest centroid, cosine677.82−0.921.15

Results. At the same budget of six rank-24 experts, PACE with its Bayesian assignment rule achieves the highest AA (78.97), best BWT (−0.61), and lowest FGT (0.71) among all five assignment rules. Its AA exceeds cosine and Euclidean thresholding by 2.65 and 0.96 points, respectively, and nearest-centroid assignment with Euclidean distance and cosine similarity by 1.23 and 1.15 points. These results support the effectiveness of PACE's Bayesian assignment rule under the same expert capacity, training protocol, and inference-time selection rule.

AuthorsOpenReview ↗

Response to Reviewer T2kJ (2/4): W1/Q1 Table 4 at matched capacity, and W2/Q2 theory

1. W1 and Q1 (continued)

Clarification on Table 4. Table 4 examines the information included in the input–response representation used by Eq. (5), keeping the assignment rule unchanged. Changing the representation changes the allocation scores and can therefore change both task grouping and the resulting number of experts. The expert count is an output of allocation; its optimality is not established. This ablation assesses representation design together with its effect on allocated capacity.

For the requested same experts, we select a threshold for each representation that yields six experts and train the resulting assignments with rank 24 per expert. Training and single-expert selection follow the same protocol. All rows below use seed 42, the same 25% evaluation subset, and the original prompt filter; each uses 363.3M populated LoRA parameters.

Allocation representationAA ↑BWT ↑FGT ↓
PACE (Full)78.98−0.610.71
Response-only78.77−0.640.71
Vision-only76.78−2.402.50
Input-only78.07−0.740.75

Results. With six rank-24 experts for every representation, full PACE achieves the highest AA (78.98), outperforming response-only, vision-only, and input-only. It also achieves the best BWT (−0.61). Its FGT is lower than input-only (0.75) and vision-only (2.50). These results support combining input and response information for expert allocation when expert count and rank are held fixed. We added this comparison as Table 5, beside Table 4 in Section 5.2 (Lines 466-493).

2. W2 and Q2: purpose of the theory, experimental support, and allocation justification

Purpose of Theorems 1 and 2. We would like to clarify the role of these two theorems, which is different from what the reviewer mentioned. Every arriving task must be assigned to an expert. Sharing controls capacity, but updating a shared expert can affect earlier tasks. With shared parameters frozen and evaluation through each task's assigned expert, parameter-update interference is confined to shared experts. Theorem 1 quantifies this forgetting. Theorem 2 establishes when sharing improves generalization over isolation by balancing reduced initialization error against task mismatch. Both hold for any fixed assignment {zt}t=1T\{z_t\}_{t=1}^{T}, under Assumptions 1–3 only. They establish the trade-off that motivates selective expert sharing. We note that Assumption 5 is not used for these two theorems, and to our knowledge, we provide the first theoretical analysis of both forgetting and generalization for selective expert sharing in multimodal continual instruction tuning.

Independence in Assumption 3. Assumption 3 is a condition of the theoretical analysis and is separate from PACE's frozen-feature construction (Lines 286-288): it holds when task summarization and adaptation use independent samples, and Appendix B.1 (Lines 949-957) now states that reusing the same samples for both purposes requires an additional conditional argument. In our implementation, the 256 summary examples constitute only 0.15%–2.01% of each CoIN task's training set (Appendix C.2).

Consistency with experiments. Our experiments are consistent with Theorems 1 and 2. Figure 3 shows that training on a new task can affect earlier tasks that share its expert. Giving each task its own expert removes this source of forgetting, as established by Theorem 1. Table 3 demonstrates the benefit of sharing described by Theorem 2: PACE achieves 78.97% average accuracy with six experts, compared with 78.67% using eight separate experts. Both configurations use rank 24 for each expert, and PACE uses 25% fewer LoRA parameters across its trained experts.

Assumption 5 and Appendix B.6. Theorems 1 and 2 hold independently of Assumption 5 (Appendix B.7), which provides a conditional link between feature and optimal-adapter distances. Appendix B.6 supports the representation used by Eq. (5) through concentration of frozen statistics and the ability of response features to distinguish tasks with identical input distributions. Empirically, PACE avoids the two assignments causing the largest accuracy drops in Fig. 6 and maintains comparable accuracy with six shared experts versus eight isolated experts in Table 3. Extending Fig. 7's partition comparison, our new six-expert experiment in W1/Q1 (Table 11) shows that PACE's Bayesian assignment rule achieves the highest AA, best BWT, and lowest FGT among five rules at the same capacity. These findings support PACE's allocation decisions and complement the theoretical conditions for beneficial sharing.

AuthorsOpenReview ↗

Response to Reviewer T2kJ (3/4): Q3, prompt preprocessing, UCIT projector, and selection across allocations

3. Q3: task-free inference, the UCIT projector, and expert-selection accuracy

Clarification requested. Could the reviewer clarify what they mean by "prompt-line frequency filtering" and identify the relevant passage or equation? This term does not appear in our paper. A specific reference would allow us to address the intended concern precisely.

If the reviewer means "Prompt lines occurring in more than 20% of a file are removed, with a fallback that preserves a nonempty prompt," this describes preprocessing before computing prompt features. Repeated instruction lines are identified by their frequency across prompts in the file and removed. If this empties a prompt, we retain its lines whose frequency is below the file's maximum; if none remain, we preserve the original prompt. This procedure uses prompt text without ground-truth answers. We have revised Appendix C.2 (Lines 1535-1542) to clarify this.

Shared projector and stored inputs on UCIT. We train the shared projector on UCIT to adapt the mapping between visual features and language representations to each new task's images and instructions. This follows the original UCIT setup in HiDe-LLaVA, which jointly trains the projector and LoRA modules. We apply the same joint-training setting to all adapted UCIT baselines for a consistent comparison.

Selection accuracy and oracle comparison. We additionally provide two experiments to demonstrate the effectiveness of our Gaussian expert selection and compare it with oracle selection. We added these experimental results in Appendix D.5, Effectiveness of Gaussian Expert Selection (Lines 1940-2026), in our revised submission.

Selection Across Allocations uses the eight-task CoIN benchmark with Qwen2.5-VL-7B, holding each allocation's trained experts fixed while comparing Gaussian, cosine nearest-prototype, and oracle selection. We report final average answer accuracy (AA) across the eight tasks on a fixed 10% test subset of 11,790 questions. Summary Size Sensitivity uses cached features from Qwen2.5-VL-7B on CoIN and LLaVA-1.5-7B on the 34-task FLEX stream, holding task-to-expert assignments fixed while varying the number of summary examples per task from 32 to 256. Its metric is expert-selection accuracy, averaged within each task and then across tasks: a query is correct when it selects the expert assigned to its true task. The FLEX analysis evaluates selection from stored features. Both experiments apply a fixed training-derived prompt-removal set independently to each query, correcting the submitted per-test-file preprocessing.

Selection Across Allocations (Table 16). We evaluated three six-expert, rank-24 allocations. The rows specify how tasks were assigned to experts during training. The columns specify how an expert is selected at inference.

AllocationGaussianCosineOracle
PACE78.8378.8278.88
Cosine threshold75.6175.8076.52
Nearest centroid, cosine76.2576.4678.34

Results. PACE achieves the highest answer accuracy across all three selection rules under the same expert budget. Its advantage persists with oracle task identification, supporting the effectiveness of its learned task-to-expert allocation. Combined with Gaussian selection, PACE delivers strong answer accuracy without requiring task labels at inference.

Reviewer_CbbEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PACE, a router-free method for multimodal continual instruction tuning. It uses a CRP prior and Bayesian predictive evidence to decide whether a new task should reuse an existing LoRA expert or create a new one. At inference, a Gaussian classifier selects an expert using stored task statistics, without a learned router. The paper also provides theoretical analysis and evaluates the method on four benchmarks.

Critical strengths

1.The paper addresses an important problem. Considering expert allocation and inference-time selection together is a useful perspective.

2.The method is simple and has a principled probabilistic formulation, without requiring a trainable router. And the experiments cover multiple benchmarks and show competitive results, particularly in reducing forgetting.

Critical weaknesses

1.The novelty over existing expert allocation methods, such as ProtoAda and CRAM, is not entirely clear.

2.The experiments do not fully isolate the benefit of the proposed probabilistic assignment rule.

Questions for authors

1.Novelty. ProtoAda and CRAM have already explored adaptive expert allocation. While PACE uses a different probabilistic formulation, I would like to see a clearer explanation of what this brings beyond existing approaches. A direct comparison would also be helpful.

2.Expert assignment. Table 3 shows that PACE achieves slightly better accuracy than task-wise isolation with fewer experts (78.97% vs. 78.67%). However, this does not tell us whether the probabilistic assignment itself is better. Appendix D.3 also reports no significant correlation between the assignment score and actual cross-task interference. Figure 7 compares different grouping strategies, but not their downstream performance. I think comparing these strategies under the same expert budget would make the contribution more convincing.

3.Expert selection. The analysis of selection changes and forgetting is useful, but I am still unsure how effective the Gaussian selection rule is compared with simpler alternatives. A comparison with oracle selection using ground-truth task IDs and nearest-prototype selection, while keeping the trained experts fixed, would help clarify this point.

5 comments
AuthorsOpenReview ↗

Response to Reviewer CbbE (1/3): Novelty over ProtoAda and CRAM

We thank the reviewer for the careful assessment and the constructive questions. All table, appendix, equation, and line numbers in our responses refer to the revised PDF.

Q1/W1. Novelty over ProtoAda and CRAM

PACE contributes a probabilistic mechanism for selective expert sharing, coupled with training-free selection of a single expert. We have added Appendix E, "Comparison with ProtoAda and CRAM", which explains these distinctions and compares PACE against their published TriGap results.

Conceptual comparison. ProtoAda groups response-format prototypes by cosine similarity and consolidates updates into shared adapters and task residuals, using a learned format predictor at inference [1]. CRAM groups instructions by a similarity margin, determines expert ranks from warm-up updates, and learns centroids for weighted expert reuse [2].

PACE decides whether to reuse an expert or create one before adaptation. Its Chinese Restaurant Process prior and closed-form predictive evidence account for group occupancy and uncertainty in group parameters, comparing existing experts against an explicit new-group model. Concretely, Eq. 5 compares the existing-group evidence nk qk(rt)n_k\,q_k(r_t) with the new-group evidence α q0(rt)\alpha\,q_0(r_t), where nkn_k counts the tasks assigned to expert kk, rtr_t is the mean feature of the arriving task, qkq_k and q0q_0 are Student-tt predictive densities, and α\alpha controls expansion. Only the assigned expert is optimized. At inference, the input coordinates of the same stored statistics support Gaussian selection of one expert for the entire response. This combination requires no learned routing parameters, format predictor, or gradient-based capacity estimation. The theoretical analysis further characterizes within-expert forgetting and sufficient conditions for beneficial sharing under its stated assumptions.

Novelty/Advantages. PACE's Bayesian criterion compares how well existing and new groups explain an arriving task, accounting for group occupancy and uncertainty to balance parameter reuse against unnecessary expert expansion, which goes beyond ProtoAda's similarity-threshold decision. Its closed-form evaluation also avoids the additional gradient computations and matrix decompositions that CRAM uses for capacity estimation, reducing the computational work required for allocation before adapter training.

Empirical comparison. We compare the three methods on TriGap with LLaVA-1.5-7B, using the averages reported in the ProtoAda and CRAM papers. Table 19 provides all task scores.

MethodReported averagePACE improvement
ProtoAda [1]47.23+1.84
CRAM [2]47.79+1.28
PACE49.07-

Results. On TriGap, PACE achieves an average score of 49.07, exceeding ProtoAda and CRAM by 1.84 and 1.28 points, respectively. PACE also outperforms ProtoAda on eight of ten tasks and CRAM on nine of ten tasks.

We have added both the conceptual and empirical comparison in Appendix Section E "Comparison with ProtoAda and CRAM" (Lines 2029-2090).

[1] Shi et al. ProtoAda: Prototype-guided adaptive adapter expansion and geometric consolidation for multimodal continual instruction tuning. arXiv:2606.02576, 2026.

[2] Tang et al. CRAM: Centroid-routing and adaptive MoE for multimodal continual instruction tuning. arXiv:2606.02502, 2026.

AuthorsOpenReview ↗

Response to Reviewer CbbE (2/3): Benefit of probabilistic expert assignment under a matched budget

Q2/W2. Benefit of probabilistic expert assignment.

We evaluated all five assignment rules on CoIN with Qwen2.5-VL-7B and 6 rank-24 experts. We replay each rule on the same stored task statistics and train the resulting assignments in the same trainer. All runs follow the training protocol in Table 7, use seed 42 as in Tables 1 and 2, and share the same 25% test subsample and single-expert selection rule (Eq. 6). We added this new experiment in our revised submission, Appendix Section D.3.3 Effect of Assignment Rules (Lines 1817-1845).

Table 11 of the revised PDF reports the results. AA is the final average accuracy and BWT is backward transfer (higher is better for both); FGT is forgetting (lower is better).

Allocation rulePopulated expertsAABWTFGT
PACE with Bayesian678.97-0.610.71
Cosine threshold676.32-3.553.56
Euclidean threshold678.01-1.341.41
Nearest centroid, Euclidean677.74-0.981.08
Nearest centroid, cosine677.82-0.921.15

Results. PACE achieves the highest accuracy and backward transfer and the lowest forgetting among all five assignment rules under the same six-expert, rank-24 budget. Its AA exceeds the four alternatives by 0.96 to 2.65 points, and its forgetting is 0.37 to 2.85 points lower. Because expert capacity, training, and inference-time selection are held fixed, these results directly support the benefit of the probabilistic assignment rule for choosing which tasks share an expert.

AuthorsOpenReview ↗

Response to Reviewer CbbE (3/3): Effectiveness of Gaussian expert selection with fixed experts

Q3. Effectiveness of Gaussian expert selection

We evaluate Gaussian expert selection in two experiments. Selection Across Allocations uses the eight-task CoIN benchmark with Qwen2.5-VL-7B, holding each allocation's trained experts fixed while comparing Gaussian, cosine nearest-prototype, and oracle selection. We report final average answer accuracy (AA) on a fixed 10% test subset of 11,790 questions. Summary Size Sensitivity uses cached features from Qwen2.5-VL-7B on CoIN and LLaVA-1.5-7B on the 34-task FLEX stream, holding task-to-expert assignments fixed while varying the number of summary examples per task from 32 to 256. Its metric is expert-selection accuracy, averaged within each task and then across tasks: a query is correct when it selects the expert assigned to its true task. Both experiments apply a fixed training-derived prompt-removal set independently to each query, correcting the submitted per-test-file preprocessing. We added these experimental results in Appendix Section D.5. Effectiveness of Gaussian expert selection (Lines 1940-2026) in our revised submission.

Selection Across Allocations. We evaluated three six-expert, rank-24 allocations (Table 16). Rows specify how tasks were assigned to experts during training. Columns specify how an expert is selected at inference.

AllocationGaussianCosineOracle
PACE78.8378.8278.88
Cosine threshold75.6175.8076.52
Nearest centroid, cosine76.2576.4678.34

Results. PACE achieves the highest answer accuracy across all three selection rules under the same expert budget. Its advantage persists with oracle task identification, supporting the effectiveness of its learned task-to-expert allocation. Combined with Gaussian selection, PACE delivers strong answer accuracy without requiring task labels at inference.

Using the same trained PACE experts, we measured task and expert selection accuracy on 2,048 CoIN test questions (Table 17). Separately, we generated answers on 25% of CoIN's test data and measured AA. Task accuracy counts correct task identification. Expert accuracy counts selection of the expert assigned to the true task; selecting another task that shares this expert still counts as correct.

Selection ruleTask accuracyExpert accuracyAA
Gaussian88.2%96.1%79.04
Cosine nearest prototype83.9%91.2%78.95
Oracle task identity100%100%79.04

Results. Gaussian selection identifies tasks and their assigned experts more accurately than cosine nearest-prototype selection. With the same trained PACE experts, it also matches oracle answer accuracy at the reported precision. These results show that the Gaussian mechanism effectively selects shared experts and supports strong answer quality without access to task identity.

Summary Size Sensitivity. We compared Gaussian, Euclidean and cosine selection with 32, 64, 128 and 256 examples per task (Table 18). This experiment fixes the expert assignments and varies only the estimated task means and variances, using ten paired summary resamples. CoIN has eight tasks and six experts; the FLEX replay has 34 tasks and nine expert groups. Values are expert-selection accuracy (%), averaged over the resamples.

BenchmarkSelection rule3264128256
CoINGaussian95.4095.5895.9496.09
CoINEuclidean93.2894.5693.7594.00
CoINCosine90.8090.8791.2691.21
FLEXGaussian88.5189.3689.8390.09
FLEXEuclidean84.8985.5086.3186.60
FLEXCosine84.4485.3886.0886.39

Results. Gaussian selection consistently outperforms Euclidean and cosine selection on both benchmarks at every tested summary size. This supports the value of accounting for feature-wise variance and demonstrates that small task summaries are sufficient for effective expert selection, including on the longer FLEX sequence.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Contribution, Rebuttal, and Revisions (to the AC and all Reviewers)

We sincerely thank the Area Chair and Reviewers T2kJ, CbbE, and hMV8 for their time and their detailed feedback. This comment summarizes our contribution and how we addressed each reviewer concern during the rebuttal. All line, table, section, and appendix numbers refer to the revised PDF.

1. Contribution

PACE is a router-free method for multimodal continual instruction tuning that decides, before adaptation, whether an arriving task reuses an existing LoRA expert or creates a new one, using a Chinese Restaurant Process prior with closed-form empirical-Bayes evidence on frozen multimodal features, and it reuses the same stored task statistics for training-free selection of a single expert at inference. We also provide a theoretical analysis of forgetting and generalization under selective expert sharing, which shows that forgetting is confined to tasks sharing an expert and gives a sufficient condition under which sharing improves generalization over one expert per task.

2. The revision and rebuttal addressed the reviewers' concerns

We responded to every weakness and question from the three reviewers.

New requested experiments. Beyond the 4 benchmarks, 2 backbones, and 10 baselines evaluated in our submission, we added 6 experiments requested by the reviewers.

  • Assignment rules at a matched budget (T2kJ W1/Q1, CbbE W2/Q2, hMV8 W1; App. D.3.3, Tab. 11, Lines 1817-1845). With the same six rank-24 experts, training protocol, and selection rule, PACE's Bayesian rule achieves the highest accuracy, the best backward transfer, and the lowest forgetting among five assignment rules, with accuracy 0.96 to 2.65 points above the cosine and Euclidean alternatives.
  • Allocation representations at matched capacity (T2kJ W1; Sec. 5.2, Tab. 5, Lines 466-493). With six rank-24 experts for every representation, the full input-response representation achieves the highest accuracy and the best backward transfer.
  • Gaussian selection with trained experts held fixed (T2kJ Q3, CbbE Q3; App. D.5, Tabs. 16-17, Lines 1940-2026). Gaussian selection matches the answer accuracy of oracle task identity and selects the correct expert for 96.1% of queries, against 91.2% for cosine nearest-prototype selection. PACE's allocation also gives the highest answer accuracy under all three selection rules, including oracle selection.
  • Summary-size sensitivity (T2kJ Q3, CbbE Q3; App. D.5, Tab. 18). Gaussian selection outperforms Euclidean and cosine selection at every summary size from 32 to 256 examples, on CoIN and on the 34-task FLEX stream.
  • Task order (hMV8 W4; Sec. 5.2, Tab. 6, Lines 494-521). Under canonical, reversed, and random orders on UCIT, PACE has the highest accuracy and the lowest forgetting in all three orders, and its accuracy changes by at most 0.71 points between orders.
  • Longer task sequences (hMV8 W4; App. D.4, Tab. 15, Lines 1914-1938). On all 34 FLEX tasks, PACE uses 9 experts, 73.5% fewer than one expert per task, while retaining 96.4% of the score of the published PureLoRA reference.

New requested baselines. Beyond the 10 baselines compared in our submission, we additionally compared with ProtoAda and CRAM, the two dynamic expert-allocation methods named by the reviewers (CbbE W1/Q1, hMV8 W3; App. E, Tab. 19, Lines 2029-2090), using their published TriGap results. PACE exceeds ProtoAda and CRAM by 1.84 and 1.28 points in average score. App. E also explains what the Bayesian rule models beyond their similarity thresholds.

Concern beyond the claims of the submission. T2kJ W2/Q2 and hMV8 W2 asked whether the theorems show that PACE's rule selects compatible tasks. Theorems 1 and 2 hold for any fixed assignment under Assumptions 1-3; Assumption 5 is not used, and the submission makes no such claim about the rule. We answered with evidence anyway: the matched-budget experiment above tests the assignment rule directly. App. B.1 (Lines 949-957) states how the independence condition applies when summary examples are reused for adaptation.

Clarifications and revised text.

  • Purpose of Tab. 3 (T2kJ W1; Lines 424-434). PACE maintains accuracy with six shared experts against eight task-specific experts, with 25% fewer populated LoRA parameters (Tab. 12).
  • Task-free inference (T2kJ Q3; App. C.2, Lines 1535-1542). Prompt preprocessing uses prompt text only, with no ground-truth answers, and the new selection experiments apply a fixed training-derived removal set to each query independently.
  • Baseline implementations (hMV8 W3; App. C.3, Lines 1560-1619). We expanded the method-specific training and inference procedures of every reproduced baseline. All reproduced baselines share the backbone, task order, and total LoRA rank, and we will release our code upon acceptance.
AuthorsOpenReview ↗

Kind reminder before the rebuttal deadline

Dear Reviewers,

Thank you again for your time and effort in reviewing our submission.

As the rebuttal period deadline is approaching, we would like to kindly remind you that we have posted detailed responses to each of your reviews. We would be grateful if you could review our rebuttal and reconsider your assessment and scores in light of these responses.

If any point remains unclear, please let us know and we will respond immediately in the time remaining.

Thank you for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.