acceptodds
Under review as a conference paper at ICLR 2027

Reward-Proportional Routing of LoRA Experts for Continual Learning in Large Language Models

Abstract

Continual learning (CL) of large language models (LLMs) commonly assigns task-specific low-rank adaptation (LoRA) experts to a frozen backbone and trains a router that selects experts for each input without task identity. Although the number of experts can vary for different inputs, it is inappropriately fixed in advance, as existing routers are only trained to identify the best experts. A small change in the loss can swap the entire selection, which degrades performance on previous tasks with their experts frozen. This paper proposes a method of reward-proportional routing (RPR) that trains the router toward a distribution over expert combinations that is proportional to their reward. The distribution changes continuously with the loss, and reading it up to a cumulative probability mass determines both the number and the composition of experts for each input. RPR learns the distribution with a generative flow network (GFlowNet) and freezes one expert per task. On three CL benchmarks with a frozen T5-large, RPR achieves the highest mean accuracy on all eight task orders, with against for the second-best method and the lowest average forgetting on the 15-task Long Sequence benchmark. On tasks never trained on, RPR reaches against for the unadapted backbone, whereas the compared methods produce labels of previous tasks and reach at most .

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, large language models, mixture of experts, low-rank adaptation, expert routing, generative flow networks, catastrophic forgetting, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Zh1fOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes RPR, a method for continual learning with LoRA experts in LLMs. Instead of selecting expert combinations solely by minimizing the loss, RPR uses a GFlowNet to learn a reward-proportional distribution over expert combinations. A readout module then selects a variable number of experts for each input. The authors provide theoretical results on the stability of the reward-proportional distribution and demonstrate empirical improvements on three continual learning benchmarks.

Critical strengths
  • I think the problem addressed by this paper is important: routing decisions may change during continual learning, potentially causing forgetting even when previously learned experts are frozen.

  • Learning a reward-proportional distribution over variable-size expert combinations is an interesting direction.

  • RPR achieves the best reported average performance across the evaluated benchmarks.

Critical weaknesses
  1. The paper refers to additional results in the appendix, but the appendix is not included in the PDF I reviewed.
  2. Missing computational efficiency analysis. In particular, comparisons of total and active parameter counts, FLOPs, GPU memory usage, training time, and inference latency are missing.
  3. All main experiments use T5-large. Evaluating the method on another backbone, for example, a decoder-only language model, would help establish its generality.
  4. The paper is not well written and difficult to follow. Specifically:
  • The interpretation of the theoretical results in Lines 202–210 is difficult to understand, as the explanation is unnecessarily complex.
  • The paper lacks a dedicated notation section. For example, the notation [M][M] is not explicitly defined. Similarly, in Eq. (4), the variables cic_i and pip_i are briefly described, but their precise constructions are not specified in the main text.
  • The paper is not sufficiently self-contained, as it lacks the necessary background on GFlowNets, particularly their formulation and the objective.
Questions for authors

Please see the weaknesses listed above. In addition, I have the following question:

  • According to Section 3.6 and Algorithm 1, previously frozen experts are excluded from routing and expert updates during subsequent tasks but are made available again at inference. However, the router continues to be updated using data from new tasks. It is unclear how routing decisions for previous tasks are preserved under this training procedure?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Zh1f (1/2)

Thank you for the review. Each weakness and question is answered in turn. Additions to the manuscript are marked in blue in the revised PDF.

W1. The appendix is not included in the PDF

Owing to the page limit, the appendix was submitted as Supplementary Material (PDF) and is available from the "Supplementary Material" link on the OpenReview submission page. It contains the proofs of all formal statements (A), the benchmarks, metrics, hyperparameters, and baseline settings (B), and additional results including per-order results with standard deviations, pool size, and computational cost (C). Part of what is requested in W2 and W4 is already there, and the reviewer is kindly asked to consult it.

W2. Computational efficiency analysis

The router size and the training time are in Appendix C of the supplementary material, and the remaining requested quantities were measured and added (Long Sequence, Order 6, one L40S GPU).

QuantityRPRInfLoRA
Router parameters1.34 M (0.17% of T5-large)–
Inference FLOPs per input754.6 G735.9 G
Inference latency, batch 1 / 8110.3 / 45.7 ms102.2 / 43.6 ms
Training time per run (Standard CL)73 min64 min
Total parameters778.3 M775.4 M
Active parameters per input745.4 M740.0 M
Peak GPU memory, training / inference13.11 / 4.31 GiB11.94 / 4.33 GiB

The router is evaluated once per input, and the routing update takes 15.7% of the training time. Relative to InfLoRA, the increase is 0.4% in total parameters, 0.7% in active parameters, 2.5% in inference FLOPs, 8% in latency, and 1.2 GiB in training memory, with equal inference memory. The values of the remaining six methods are in the supplementary table. For reference, O-LoRA and SAPT report no FLOPs, latency, or memory, and GainLoRA reports FLOPs and parameter counts only.

These results show that the additional cost of RPR is below 1% in parameters and 2.5% in inference FLOPs.

W3. Evaluation on another backbone

Standard CL Orders 1–3 were run on LLaMA-2-7B, the scope in which O-LoRA and N-LoRA evaluate decoder-only models.

BackboneO1O2O3Avg
T5-large (Table 1)78.477.177.477.6
LLaMA-2-7B (two seeds)68.877.177.974.6

The router acts only on the LoRA experts and leaves the backbone unchanged, so it was applied without changing the configuration of the router or the experts. These results show that RPR reaches accuracy on a decoder-only backbone at the level of T5-large.

AuthorsOpenReview ↗

Response to Reviewer Zh1f (2/2)

W4. Presentation

All three points are reflected in the manuscript.

(a) The interpretation after Propositions 1–2 is reduced to two sentences: "The minimizer of the expected loss places all probability on one combination, so a change of size ϵ\epsilon switches the entire selection whenever the gap between the best and the second-best loss is below 2ϵ2\epsilon. The reward-proportional target rescales the probability of each combination by a factor in [e−βϵ,eβϵ][e^{-\beta\epsilon},e^{\beta\epsilon}], so the whole distribution moves by at most tanh⁡(βϵ/2)\tanh(\beta\epsilon/2)." The numerical example is kept.

(b) [M]={1,…,M}[M]=\{1,\dots,M\} is defined in Section 3.1, and the constructions of ci\mathbf{c}_i and pi\mathbf{p}_i are stated below Eq. (4): "ci\mathbf{c}_i is the mean of the keys of the experts in sis_i (the zero vector at s0s_0), and pi\mathbf{p}_i is a learned 32-dimensional embedding of the step index ii." A full notation table is added to the supplementary material.

(c) Background on GFlowNets is added at the beginning of Section 3.3: states and actions, the forward policy PFP_F and the backward policy PBP_B, and the TB condition, namely that the terminal distribution is proportional to the reward when Z PF(τ)=PB(τ∣z) R(z)Z\,P_F(\tau)=P_B(\tau\mid z)\,R(z) holds on every complete trajectory.

Q1. How are routing decisions for previous tasks preserved while the router continues to be updated?

Three mechanisms preserve them, and the effect was measured.

First, the routing key of a frozen expert is fixed to the mean query of the inputs of its task, so later training cannot move it. Second, the reward of a combination consisting only of frozen experts does not change under later training, so the relative probabilities that the target assigns to such combinations are kept (Proposition on consolidation, Appendix A). Third, the target moves only continuously under the loss changes of the remaining combinations (Proposition 2).

In measurement, the fraction of previous-task inputs whose selected set after the last task differs from the set selected right after their own task is 0.635 for RPR, 0.776 without consolidation, and 0.811 for a loss-minimizing softmax gate (Long Sequence, Order 6). The corresponding forgetting is 0.8, 7.6, and 12.0, and without key fixing the average accuracy falls from 70.6% to 64.5% (Table 2).

These results show that, although the router continues to be updated, the fixed keys and the invariant rewards of frozen combinations maintain the routing of previous tasks, and the effect is measured in both the flip rate and forgetting.

Dear Reviewer Zh1f,

Thank you again for your time and the review. We have posted responses to each of your points. The appendix you could not access is available as Supplementary Material on the submission page, and the revised PDF now adds the computational-cost comparison, the decoder-only (LLaMA-2-7B) results, the GFlowNet background, and a notation table; a one-page summary is in the comment at the top of the forum.

If anything in the revision remains unclear, even a one-line comment would let us follow up before the discussion closes. We hope the added material allows the assessment, including the presentation score, to be revisited.

Thank you for your response.

Regarding the appendix, I would like to clarify that, among all the papers I have reviewed so far, yours is the only one that separates the main paper and appendix into two files. Moreover, the conference guidelines explicitly state that reviewers are not required to examine supplementary materials. Therefore, I believe the appendix should have been included in the main PDF, with the supplementary submission reserved for additional materials such as code.

Furthermore, regarding the experiments with other backbones, the results are reported only for the proposed method. My concern is that evaluating the proposed method across different backbones, without corresponding comparisons against existing baselines, is insufficient.

Given these unresolved concerns, I have decided to maintain my original score.

Reviewer_KEncOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies continual learning with a pool of LoRA experts on a frozen backbone, when test inputs have no task identity. A GFlowNet is trained so that expert combinations get a probability proportional to a tempered reward. A cumulative-mass rule on the first-step marginal chooses which experts to apply, and one expert is frozen after each task.

Critical strengths

The router is not trained to name a single best subset. It learns a distribution over variable-size combinations, with a backward policy that cancels the ordering bias of a set, and inference reads a deterministic prefix of the first-step marginal instead of sampling.

The stability claim is stated as two different objectives. A loss minimizer can jump to another combination under a small loss change. The reward-proportional target moves continuously in total variation, and a global trajectory-balance solution makes the first-step probability the marginal that the readout uses.

On three benchmarks with a frozen T5-large, the method is first on every reported task order. Replacing trajectory balance with a policy-gradient objective collapses accuracy. On tasks never trained in the stream, it stays near the unadapted backbone, while the compared stream-trained routers mostly emit labels from earlier task

Critical weaknesses

The failure the introduction describes is a later task flipping the expert set of an earlier input. The experiments report final accuracy and forgetting, not that flip rate. The gate scores in the motivation figure are marked as illustrative, and the continuity example in the method is constructed.

The continuity bound is about the normalized target, and the marginal identity assumes trajectory balance has reached a global minimum. The runs do not show the training residual or that the learned probabilities match the rewards over the admissible subsets.

A fixed number of experts already comes close to the full readout on the long stream, so the combinatorial router is not clearly necessary. The paper does not report the cost of scoring combinations inside each trajectory.

The empirical scope is a frozen encoder-decoder with a short schedule and a small sample budget per task. A gated LoRA continual learner discussed in related work is absent from the tables, and the main text does not line the other routers up with their published scores.

Held-out behavior is measured on a few tasks from one order. The method still trails the unadapted backbone there, and there is no ablation that separates the cumulative-mass rule, the output scale, and the frozen keys.

Questions for authors

Please see the main weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer KEnc (1/2)

Thank you for the careful review. Each weakness is answered in turn. Additions to the manuscript are marked in blue in the revised PDF.

W1. The flip rate is not reported; the motivation figure and the continuity example are illustrative

The flip rate was measured and added to the main text. It is the fraction of previous-task inputs whose selected set after the last task differs from the set selected right after their own task (Long Sequence, Order 6, three seeds).

RouterFlip rateF (%)
RPR0.635 ± 0.0850.8
RPR without consolidation0.776 ± 0.1047.6
Softmax gate, top-30.811 ± 0.10312.0
REINFORCE0.962 ± 0.0164.4

Over all 756 (task, seed, order, method) cells, the Spearman correlation between flip rate and forgetting is 0.47 (p<10−40p<10^{-40}), so the failure described in the Introduction is observed in measurement. RPR has the lowest flip rate among the routers in the table, and within RPR the correlation between flip rate and forgetting is 0.01 (p=0.88p=0.88).

Figure 1 and the example of Section 3.2 illustrate the mechanism of Propositions 1–2 on one input, and the caption marks the gate scores as illustrative. The table above provides the measured evidence.

These results show that the failure in the Introduction is a measured phenomenon, and that RPR lowers its frequency and decouples the remaining changes from forgetting.

W2. The marginal identity assumes a global minimum of TB; the training residual is not shown

The TB residual is now reported, and a proposition was added so that the guarantee of selection stability does not depend on the global-minimum assumption.

Residual. Long Sequence, Order 6, three seeds, 840 steps after warmup:

QuantityValue
Batch mean of the signed residual0.27 ± 0.03
Median TB loss1.27 ± 0.13
Steps passing the expert-update condition75.4 ± 1.6 %

The mean signed residual is close to zero, so log⁡Zϕ\log Z_\phi is fitted without bias. Per-task values are added to the supplementary material.

Global-minimum assumption. The added Proposition 3 reads as follows. Let SnS_n be the mass of the top nn experts under q1q_1. If Sn≥ρ+δS_n\ge\rho+\delta, Sn−1<ρ−δS_{n-1}<\rho-\delta, and q1(m(n))−q1(m(n+1))>2δq_1(m_{(n)})-q_1(m_{(n+1)})>2\delta, the selected set does not change as long as q1q_1 moves by at most δ\delta in total variation. Only the change of q1q_1 enters the condition, so it applies to the trained router as is. With the measured change of the trained q1q_1, 21.7% of the previous-task inputs of RPR satisfy the condition (softmax gate 8.2%, REINFORCE 13.5%), and none of them changes its selection in any of the nine runs. Lemma 1 states what the readout reads at a global minimum.

These results show that the guarantee of selection stability and the experimental results of the paper hold without assuming that TB has reached a global minimum.

W3. A fixed number of experts comes close to the full readout; the cost inside each trajectory is not reported

The fixed-kk rows of Table 2 do not remove the router; they replace only the readout. The cost is reported below.

Necessity. The router of the fixed-kk rows is trained by TB as in RPR. When the objective of the router is replaced by loss minimization on the same pool, accuracy falls to 45.6% (top-1) and 50.4% (top-3) (Table 2). A fixed kk is therefore close only on top of the reward-proportional router, and even there each fixed number loses in a different setting.

ReadoutLong Sequence Acc (%)Long Sequence F (%)Held-out Acc (%)
Fixed k=1k=168.70.756.5
Fixed k=2k=267.63.157.2
Fixed k=3k=369.12.461.7
Cumulative mass (RPR)70.60.858.2

k=1k=1, which has the lowest forgetting, is the lowest on held-out tasks, and k=3k=3, which is the highest on held-out tasks, has three times the forgetting. The number of experts that inputs require differs across settings, 1.08 on Standard CL, 2.18 on Long Sequence, and 3.37 on tasks never trained on (Figure 3), and the cumulative-mass readout follows it without choosing kk.

Cost. The router is evaluated once per input. The routing update takes 15.7% of the training time; at inference, latency is +8% (110.3 ms against 102.2 ms, batch 1) and FLOPs are +2.5% (754.6 G against 735.9 G) relative to InfLoRA. Per training step, RPR uses 0.90 backbone forwards for the expert update and 0.46 gradient-free forwards for reward evaluation, against 1.00 forward for the compared methods. Training time per run is in Appendix C (73 minutes for RPR, 63–70 for the compared methods).

These results show that the combinatorial router produces most of the accuracy, and that the variable readout provides a balance of accuracy and forgetting that no fixed number attains, at +8% latency.

AuthorsOpenReview ↗

Response to Reviewer KEnc (2/2)

W4. Scope of the experiments; a gated LoRA continual learner is absent; published scores are not aligned

The sample budget and schedule follow the protocol of this benchmark, and experiments were added for the other three points.

Protocol. 1,000 examples per task and one epoch are the setting of O-LoRA (Wang et al., 2023), and 1,000 examples per task are also used by SAPT (Zhao et al., 2024) on SuperNI and Long Sequence. All methods are compared under this budget.

GainLoRA. The official implementation was ported to the shared code base and run with the same pool budget and schedule (three seeds).

MethodLong O4Long O5Long O6Long Fˉ\bar F (%)Standard CL Fˉ\bar F (%)
GainLoRA71.673.974.85.70.7
RPR71.069.571.21.20.1
InfLoRA68.168.569.61.30.0

GainLoRA forgets more than RPR on both benchmarks (by more than 4 points on Long Sequence) and raises its average through the accuracy of new tasks. On Order 4 the accuracy difference between the two methods (0.6) is within the standard deviation over seeds (2.1). The gates of GainLoRA are trained to suppress the effect of new branches on old tasks, whereas RPR changes the selection distribution, so the two methods address forgetting through different paths.

Published scores. The compared methods were re-run from their official implementations under one protocol, as SAPT and GainLoRA do for their baselines. The original papers differ in batch size, number of epochs, number of modules per task, and use of task identity at test time (e.g., a batch size of 64 in O-LoRA and 100 epochs per task in GainLoRA), so placing the published numbers side by side would compare settings rather than methods. A table of the published scores and the differences in settings is added to the supplementary material.

Decoder-only backbone. Standard CL Orders 1–3 were run on LLaMA-2-7B, the scope in which O-LoRA and N-LoRA evaluate decoder-only models.

BackboneO1O2O3Avg
T5-large (Table 1)78.477.177.477.6
LLaMA-2-7B (two seeds)68.877.177.974.6

RPR runs on a decoder-only backbone without changing the router or the experts.

These results show that the scope of the experiments matches the protocol of the field, and that RPR keeps a lower forgetting than the added baseline and extends to a decoder-only backbone.

W5. Held-out behavior is measured on one order; RPR trails the unadapted backbone; no ablation separates the components

The evaluation was extended to three orders and the requested ablation was added. In the extended evaluation RPR exceeds the backbone on two of the three orders.

Scope. On each order, the last three tasks are excluded from training and then evaluated (three seeds, Acc %).

MethodOrder 4Order 5Order 6
Unadapted backbone22.70.262.1
RPR30.518.858.2
InfLoRA14.50.41.0
EWC-LoRA14.70.30.7
AM-LoRA14.30.20.7

RPR is 7.8 points above the backbone on Order 4 and 18.6 points above it on Order 5, and above the compared methods on all three orders. On the three tasks of Order 5 (Yelp, Amazon, Yahoo) the backbone does not produce the label format of the task (valid-label rate 0.6%), and RPR reaches 18.8% without having been trained on them. Evaluating unseen tasks on one order is the setting of O-LoRA, where performance after continual learning (33.6%) is likewise below that of the base model (34.4%).

Ablation. One component of the trained RPR is changed at a time (Order 6, held-out, Acc %).

VariantAcc
RPR58.2
Fixed k=1k=1 / 22 / 33 in place of the cumulative mass56.9 / 59.0 / 58.9
Without the 1/∣z∣1/\lvert z\rvert normalization (sum)37.1
Keys of frozen experts not fixed50.9
Output scale ×0.5 / ×266.7 / 47.8

The held-out behavior is produced mainly by the 1/∣z∣1/|z| normalization of Eq. (1) (−21.1 when removed) and by fixing the keys (−7.3 when released); the rule for the number of experts changes accuracy by at most 1.3 points. Halving the scale gives 66.7%, above the backbone (62.1%), so the outputs of frozen experts transfer positively to tasks never trained on.

These results show that the held-out behavior of RPR is reproduced across orders, and that its source is identified as the normalized combination and the fixed keys.

Dear Reviewer KEnc,

Thank you again for your time and the careful review. We have posted responses to each of the five weaknesses, and the revised PDF now includes the flip-rate analysis (Table 3), the selection-stability proposition and the TB residual (§3.4, Appendix C), the comparison with GainLoRA and the published scores, the cost of all methods, and the held-out evaluation on three orders (Appendix C); a one-page summary is in the comment at the top of the forum.

Should any of the five points still need clarification, please let us know which one, and we will address it within the remaining time. If you find the concerns resolved, we would appreciate it if this could be reflected in your rating.

Thank you for the authors' response, which addresses most of my concerns. However, after carefully reading the comments from other reviewers, I find that this paper still has several crucial problems, and it appears that the authors did not carefully review ICLR’s submission guidelines before submitting. Overall, I have decided to maintain my original score.

Reviewer_6VsiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Reward-Proportional Routing to mitigate unstable expert selection and forgetting caused by router updates in LoRA-based continual learning for large language models. RPR employs a generative flow network to learn a reward-proportional distribution over expert combinations and adaptively determines the number and composition of experts by reading the router's first-step probabilities up to a cumulative probability mass threshold. It further employs a consolidation mechanism that freezes one expert per task and fixes its routing key. The paper theoretically analyzes the continuity of the reward-proportional target distribution under loss perturbations. Experiments on three continual learning benchmarks show that RPR achieves the best performance among the compared methods across all eight task orders.

Critical strengths
  1. The paper investigates forgetting in continual learning from the perspective of routing stability, highlighting that updates to the router during subsequent tasks can change expert selection even when previously learned LoRA experts remain frozen. RPR learns a reward-proportional distribution over expert combinations and adaptively determines the number and composition of experts by reading the router's first-step probabilities up to a cumulative probability mass threshold, providing a targeted approach to mitigating routing-induced forgetting in continual learning.
  2. The paper theoretically analyzes the continuity of the reward-proportional target distribution under loss perturbations and establishes the relationship between the router's first-step probabilities and the per-expert marginals of the target distribution at zero TB loss. Experiments cover three continual learning benchmarks and eight task orders, where RPR consistently outperforms the compared methods. Ablation studies on the routing objective, expert selection, and consolidation further support the effectiveness of the key components.
Critical weaknesses
  1. Proposition 2 and Corollary 1 establish the continuity of the reward-proportional target distribution and its first-step marginals under loss perturbations. However, during inference, Equation (7) performs discrete expert selection by ranking the probabilities and applying a cumulative probability mass threshold, so even small changes in the probabilities may lead to changes in the selected expert set. In addition, Lemma 1 relies on the ideal condition of zero TB loss. Therefore, the existing theoretical analysis is insufficient to support the core claim regarding the stability of expert selection in practice.
  2. Appendix A.3 relates the reward-proportional objective to entropy-regularized expected-loss minimization, but the experiments lack an entropy-matched routing baseline, making it difficult to determine whether the performance gains mainly come from the reward-proportional objective, entropy regularization, or GFlowNet's ability to model distributions over expert combinations. In addition, the paper uses first-step marginal probabilities for expert selection but does not directly compare this strategy with other simple readout strategies, making it insufficient to establish the necessity and superiority of the complete routing design.
Questions for authors
  1. Proposition 2 and Corollary 1 establish continuity of the target distribution and its first-step marginals, but this does not directly imply stability of the discrete expert set under Equation (7). Can the authors provide an analysis of expert-selection changes for previous-task inputs during continual learning and examine their relationship with forgetting?
  2. Since Appendix A.3 relates the reward-proportional objective to entropy-regularized expected-loss minimization, can the authors provide an entropy-matched routing baseline to isolate the contribution of the reward-proportional objective from entropy regularization?
  3. RPR selects experts using first-step marginal probabilities and a cumulative probability mass threshold. How does this readout strategy compare with simpler alternatives, such as selecting the highest-probability expert combination or directly sampling expert combinations from the learned GFlowNet distribution?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 6Vsi (1/2)

Thank you for the detailed review. Each weakness and question is answered in turn. Additions to the manuscript are marked in blue in the revised PDF.

W1. Continuity of the target does not imply stability of the discrete selection; Lemma 1 assumes zero TB loss

The step from continuity to stability of the discrete selection is now stated as a proposition, and this proposition does not use the zero-TB-loss assumption.

Discrete selection. Section 3.4 restricted the inputs whose selection can change to those near a boundary of the readout. This is now Proposition 3. Let δ=tanh⁡(βϵ/2)\delta=\tanh(\beta\epsilon/2), Sn=∑j≤nq1(m(j)∣x)S_n=\sum_{j\le n}q_1(m_{(j)}\mid x), and n=n(x)<Kn=n(x)<K. If

Sn≥ρ+δ,Sn−1<ρ−δ,q1(m(n)∣x)−q1(m(n+1)∣x)>2δ,S_n\ge\rho+\delta,\qquad S_{n-1}<\rho-\delta,\qquad q_1(m_{(n)}\mid x)-q_1(m_{(n+1)}\mid x)>2\delta,

then z^(x)\hat z(x) is unchanged for every ℓ′\ell' with ∥ℓ′−ℓ∥∞≤ϵ\|\ell'-\ell\|_\infty\le\epsilon. By Corollary 1, the probability of a single expert and the mass of any subset of experts each change by at most δ\delta; the first two conditions therefore preserve n(x)n(x) and the third preserves the membership of the top nn. The condition n(x)<Kn(x)<K held for every evaluated input of RPR.

Zero TB loss. The only quantity that enters the conditions is the change of q1q_1. With δ\delta replaced by the measured TV(q1j,q1T)\mathrm{TV}(q_1^{j},q_1^{T}) of the trained router, the statement holds regardless of the value of the TB loss, and the measurement below is made in this form. Lemma 1 remains the statement of what the readout reads at a global minimum.

Measurement. Fraction of previous-task inputs whose margin exceeds the measured change of q1q_1 (Long Sequence, three orders, three seeds):

RouterInputs satisfying the margin condition (%)
RPR21.7
RPR without consolidation16.0
REINFORCE13.5
Softmax gate8.2

Among the inputs that satisfy the condition, none changes its selection after later tasks in any of the nine runs of RPR. The condition is sufficient under the worst-case perturbation, so this fraction is a lower bound on the inputs whose selection is kept; the response to Q1 shows that the changes outside the condition are not associated with forgetting.

These results show that the continuity of the target carries over to an unchanged selection on inputs with a margin, and that the guarantee holds without the zero-TB-loss assumption.

W2. No entropy-matched baseline; the readout is not compared with simple alternatives

Both comparisons were run. The gain does not come from entropy regularization in general; it comes from training the distribution over expert selections toward the reward-proportional target.

The reward-proportional target is the unique minimizer of the expected loss plus an entropy term (Appendix A.3), so the entropy term is part of the proposed objective. The question to isolate is where the entropy is placed (Long Sequence, Order 6, three seeds):

RouterTraining objectiveAcc (%)F (%)
Softmax gateloss + per-input gate entropy (two settings bracketing H(q1)H(q_1))51.0 / 56.59.0 / 11.1
Policy network of RPRentropy-regularized REINFORCE, temperature of the target71.71.1
Policy network of RPRentropy-regularized REINFORCE, H(q1)H(q_1) matched to RPR69.72.2
Policy network of RPRTB (RPR)71.20.8

The entropy of the gate jumps from below 0.87 to above 2.76 as its coefficient varies and cannot be set to H(q1)=1.63H(q_1)=1.63 of RPR, so the two settings on either side are reported. A gate with an entropy term stays at the level of the gate without it (51.2, Table 2), and the same gap to RPR appears on Order 4 (48.2 / 54.8 against 71.0). The same entropy-regularized objective applied to the policy network reaches the accuracy of RPR, as expected from the two objectives sharing one optimum (Remark in Appendix A.3). TB gives the lowest forgetting, and REINFORCE without the entropy term has a forgetting of 4.4 (Table 2).

The readout comparison is in the table under Q3: sampling is 12.1 points lower, and using only the highest-probability expert is 0.9 points lower.

These results show that the gain comes from the reward-proportional objective over the selection distribution rather than from an entropy bonus, and that TB realizes this objective with the lowest forgetting.

AuthorsOpenReview ↗

Response to Reviewer 6Vsi (2/2)

Q1. Expert-selection changes for previous-task inputs and their relationship with forgetting

The analysis was carried out. The flip rate is the fraction of inputs of task jj whose selected set after the last task differs from the set selected right after task jj (Long Sequence, Order 6, three seeds).

RouterMean number of expertsFlip rateF (%)
RPR2.180.635 ± 0.0850.8
RPR without consolidation1.360.776 ± 0.1047.6
Softmax gate, top-33.000.811 ± 0.10312.0
REINFORCE4.680.962 ± 0.0164.4

Over all 756 (task, seed, order, method) cells, the Spearman correlation between flip rate and forgetting is 0.47 (p<10−40p<10^{-40}), which supports the premise of the Introduction that selection changes lead to forgetting. RPR has the lowest flip rate among the routers in the table, and within RPR the correlation between flip rate and forgetting is 0.01 (p=0.88p=0.88). The selection changes that remain in RPR are replacements that do not lower accuracy, and inputs with a margin do not change at all (W1).

These results show that selection changes grow together with forgetting in loss-minimizing routers, whereas RPR reduces the changes and decouples the remaining ones from forgetting.

Q2. Entropy-matched routing baseline

The table under W2 is this comparison. A softmax gate with an entropy term stays at 48.2–56.5% on the two orders, whereas entropy-regularized REINFORCE on the policy network of RPR reaches 71.7%, the same as RPR (71.2%). The reward-proportional objective and the entropy-regularized objective share one optimum, so equal accuracy is what the theory predicts, and TB has the lowest forgetting (0.8 against 1.1–2.2). The contribution is therefore not obtained by adding entropy to a gate; it comes from training the whole selection distribution to be proportional to the reward.

Q3. Comparison of the readout with simpler alternatives

The trained RPR was kept fixed and only the readout at inference was replaced (Long Sequence, Order 6, three seeds).

ReadoutBackbone forwards per inputDeterministicAcc (%)
Cumulative mass, ρ=0.6\rho=0.6 (RPR)1yes71.2 ± 1.4
Highest-probability expert only1yes70.3 ± 2.1
Sampling from the learned distribution1no59.1 ± 1.7
Logit average over 8 samples8no69.0 ± 2.2

Sampling is 12.1 points lower, and averaging eight samples at eight times the cost is still 2.2 points lower. Selecting only the highest-probability expert discards the probability that is divided among experts solving the input comparably well and is 0.9 points lower. The cumulative-mass readout evaluates the router once, selects deterministically, and varies the number of experts across inputs (Figure 3). The comparison with fixed numbers of experts is in Table 2.

These results show that reading the first-step probabilities up to a cumulative mass uses the information in the distribution without the cost and variance of sampling.

Dear Reviewer 6Vsi,

Thank you again for your time and the constructive review. We have posted responses to each of your points, and the revised PDF now includes the selection-stability proposition (§3.4), the flip-rate analysis (Table 3), the entropy-matched baselines, and the readout comparison (Appendix C); a one-page summary is in the comment at the top of the forum.

We understand the discussion period is busy. If any point is still unresolved, a brief note on which one would let us respond before the deadline. If our responses address your concerns, we would be grateful if you could consider updating your score.

Comments on the paper

2 comments
Area_Chair_q9QNOpenReview ↗

Please Participate in the Discussion

Dear Reviewers,

The authors have submitted their responses. Please read them and update your reviews if appropriate. It would be helpful to share your thoughts early to allow enough time for a productive discussion. Thank you!

Best regards,
Area Chair

AuthorsOpenReview ↗

Summary of responses and revisions

We thank the three reviewers for their reviews. Responses to every weakness and question have been posted, and all additions to the revised PDF are marked in blue. As the discussion period closes soon, this comment summarizes how each concern is addressed. We remain available to answer any further question until the end of the discussion.

Concerns raised by more than one reviewer

ConcernRaised byResponseRevised paper
Continuity of the target does not imply stability of the discrete selection6Vsi W1, KEnc W2Proposition 3: the selection is unchanged under a margin condition that uses only the change of q1q_1, so it holds for the trained router without the zero-TB-loss assumption. 21.7% of previous-task inputs satisfy it (softmax gate 8.2%); none of them changes its selection.§3.4, App. A, App. C
Selection changes of previous-task inputs6Vsi Q1, KEnc W1, Zh1f Q1Flip rate: RPR 0.635, softmax gate (top-3) 0.811, REINFORCE 0.962. Correlation with forgetting: 0.47 across methods, 0.01 within RPR.§4.4, Table 3
Computational costKEnc W3, Zh1f W2Parameters, memory, FLOPs, and latency of all eight methods. Relative to InfLoRA: +0.4% parameters, +2.5% inference FLOPs, +8% latency.App. C
Decoder-only backboneKEnc W4, Zh1f W3LLaMA-2-7B on Standard CL: 74.6%, with the router and experts unchanged.§5, App. C

Reviewer-specific concerns

  • Entropy-matched baseline (6Vsi W2, Q2). Gate + entropy: 48.2–56.5%. Entropy-regularized REINFORCE on the same policy matches RPR in accuracy, as the shared optimum predicts, with higher forgetting (1.1–2.2 vs. 0.8). (App. C)
  • Readout alternatives (6Vsi Q3). Sampling −12.1, eight-sample averaging −2.2 at 8× cost, highest-probability expert −0.9. (App. C)
  • TB residual (KEnc W2). Mean signed residual 0.27; per-task values. (App. C)
  • Fixed number of experts (KEnc W3). The fixed-kk rows keep the TB router; a loss-minimizing router reaches 45.6–50.4%; no fixed kk matches the readout in both forgetting and held-out accuracy.
  • GainLoRA and published scores (KEnc W4). GainLoRA added under the shared protocol (forgetting 5.7 vs. 1.2 for RPR on Long Sequence); published scores and settings tabulated. (App. B, App. C)
  • Held-out tasks (KEnc W5). Three orders; RPR exceeds the unadapted backbone on Orders 4 and 5 and the compared methods on all three; component ablation added. (§4.5, App. C)
  • Appendix (Zh1f W1). Submitted as Supplementary Material.
  • Presentation (Zh1f W4). Simplified interpretation of Propositions 1–2, notation table, GFlowNet background. (§3.1–3.3, App. B)

All numbers above are reported with their settings in the revised PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.