- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes RPR, a method for continual learning with LoRA experts in LLMs. Instead of selecting expert combinations solely by minimizing the loss, RPR uses a GFlowNet to learn a reward-proportional distribution over expert combinations. A readout module then selects a variable number of experts for each input. The authors provide theoretical results on the stability of the reward-proportional distribution and demonstrate empirical improvements on three continual learning benchmarks.
Critical strengths
-
I think the problem addressed by this paper is important: routing decisions may change during continual learning, potentially causing forgetting even when previously learned experts are frozen.
-
Learning a reward-proportional distribution over variable-size expert combinations is an interesting direction.
-
RPR achieves the best reported average performance across the evaluated benchmarks.
Critical weaknesses
- The paper refers to additional results in the appendix, but the appendix is not included in the PDF I reviewed.
- Missing computational efficiency analysis. In particular, comparisons of total and active parameter counts, FLOPs, GPU memory usage, training time, and inference latency are missing.
- All main experiments use T5-large. Evaluating the method on another backbone, for example, a decoder-only language model, would help establish its generality.
- The paper is not well written and difficult to follow. Specifically:
- The interpretation of the theoretical results in Lines 202–210 is difficult to understand, as the explanation is unnecessarily complex.
- The paper lacks a dedicated notation section. For example, the notation is not explicitly defined. Similarly, in Eq. (4), the variables and are briefly described, but their precise constructions are not specified in the main text.
- The paper is not sufficiently self-contained, as it lacks the necessary background on GFlowNets, particularly their formulation and the objective.
Questions for authors
Please see the weaknesses listed above. In addition, I have the following question:
- According to Section 3.6 and Algorithm 1, previously frozen experts are excluded from routing and expert updates during subsequent tasks but are made available again at inference. However, the router continues to be updated using data from new tasks. It is unclear how routing decisions for previous tasks are preserved under this training procedure?
4 comments
Response to Reviewer Zh1f (2/2)
W4. Presentation
All three points are reflected in the manuscript.
(a) The interpretation after Propositions 1–2 is reduced to two sentences: "The minimizer of the expected loss places all probability on one combination, so a change of size switches the entire selection whenever the gap between the best and the second-best loss is below . The reward-proportional target rescales the probability of each combination by a factor in , so the whole distribution moves by at most ." The numerical example is kept.
(b) is defined in Section 3.1, and the constructions of and are stated below Eq. (4): " is the mean of the keys of the experts in (the zero vector at ), and is a learned 32-dimensional embedding of the step index ." A full notation table is added to the supplementary material.
(c) Background on GFlowNets is added at the beginning of Section 3.3: states and actions, the forward policy and the backward policy , and the TB condition, namely that the terminal distribution is proportional to the reward when holds on every complete trajectory.
Q1. How are routing decisions for previous tasks preserved while the router continues to be updated?
Three mechanisms preserve them, and the effect was measured.
First, the routing key of a frozen expert is fixed to the mean query of the inputs of its task, so later training cannot move it. Second, the reward of a combination consisting only of frozen experts does not change under later training, so the relative probabilities that the target assigns to such combinations are kept (Proposition on consolidation, Appendix A). Third, the target moves only continuously under the loss changes of the remaining combinations (Proposition 2).
In measurement, the fraction of previous-task inputs whose selected set after the last task differs from the set selected right after their own task is 0.635 for RPR, 0.776 without consolidation, and 0.811 for a loss-minimizing softmax gate (Long Sequence, Order 6). The corresponding forgetting is 0.8, 7.6, and 12.0, and without key fixing the average accuracy falls from 70.6% to 64.5% (Table 2).
These results show that, although the router continues to be updated, the fixed keys and the invariant rewards of frozen combinations maintain the routing of previous tasks, and the effect is measured in both the flip rate and forgetting.
Dear Reviewer Zh1f,
Thank you again for your time and the review. We have posted responses to each of your points. The appendix you could not access is available as Supplementary Material on the submission page, and the revised PDF now adds the computational-cost comparison, the decoder-only (LLaMA-2-7B) results, the GFlowNet background, and a notation table; a one-page summary is in the comment at the top of the forum.
If anything in the revision remains unclear, even a one-line comment would let us follow up before the discussion closes. We hope the added material allows the assessment, including the presentation score, to be revisited.
Thank you for your response.
Regarding the appendix, I would like to clarify that, among all the papers I have reviewed so far, yours is the only one that separates the main paper and appendix into two files. Moreover, the conference guidelines explicitly state that reviewers are not required to examine supplementary materials. Therefore, I believe the appendix should have been included in the main PDF, with the supplementary submission reserved for additional materials such as code.
Furthermore, regarding the experiments with other backbones, the results are reported only for the proposed method. My concern is that evaluating the proposed method across different backbones, without corresponding comparisons against existing baselines, is insufficient.
Given these unresolved concerns, I have decided to maintain my original score.
Response to Reviewer Zh1f (1/2)
Thank you for the review. Each weakness and question is answered in turn. Additions to the manuscript are marked in blue in the revised PDF.
W1. The appendix is not included in the PDF
Owing to the page limit, the appendix was submitted as Supplementary Material (PDF) and is available from the "Supplementary Material" link on the OpenReview submission page. It contains the proofs of all formal statements (A), the benchmarks, metrics, hyperparameters, and baseline settings (B), and additional results including per-order results with standard deviations, pool size, and computational cost (C). Part of what is requested in W2 and W4 is already there, and the reviewer is kindly asked to consult it.
W2. Computational efficiency analysis
The router size and the training time are in Appendix C of the supplementary material, and the remaining requested quantities were measured and added (Long Sequence, Order 6, one L40S GPU).
The router is evaluated once per input, and the routing update takes 15.7% of the training time. Relative to InfLoRA, the increase is 0.4% in total parameters, 0.7% in active parameters, 2.5% in inference FLOPs, 8% in latency, and 1.2 GiB in training memory, with equal inference memory. The values of the remaining six methods are in the supplementary table. For reference, O-LoRA and SAPT report no FLOPs, latency, or memory, and GainLoRA reports FLOPs and parameter counts only.
These results show that the additional cost of RPR is below 1% in parameters and 2.5% in inference FLOPs.
W3. Evaluation on another backbone
Standard CL Orders 1–3 were run on LLaMA-2-7B, the scope in which O-LoRA and N-LoRA evaluate decoder-only models.
The router acts only on the LoRA experts and leaves the backbone unchanged, so it was applied without changing the configuration of the router or the experts. These results show that RPR reaches accuracy on a decoder-only backbone at the level of T5-large.