- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the problems that arise when routing determines the scope of training during model updates in L2D and cascade model systems, where each model is updated according to its assigned tasks. In particular, it explores a hidden, counterintuitive phenomenon in which routing causes a lightweight student model to skip certain topics, improving overall system performance while substantially degrading the student's own performance. This phenomenon occurs especially when the fallback model is upgraded and becomes more capable, potentially posing risks if the fallback is later rolled back or becomes unavailable. The paper models this phenomenon using differential equations and validates the predictions of the model through thorough experiments. It also uses controlled experiments to disentangle the causes of performance degradation into the absence of training data for the affected tasks and interference among shared parameters, and compares remedies that decouple serving from training.
Critical strengths
A critical strength of this paper is its novel problem formulation and mathematical modeling that qualitatively captures the phenomenon. In particular, the analysis of training exposure and its timing is useful.
Critical weaknesses
A critical weakness is that the experimental setup is overly idealized. Topics arrive as clearly separated groups in a setting where all labels are available, the splits are easily distinguishable from prompts alone, and a calibration set is provided to estimate the accuracy of each model. I understand that these measures were taken to clearly demonstrate and analyze the phenomenon, but there are challenges in measuring the claimed phenomenon and applying the proposed remedies in realistic settings.
In particular, I consider domain identification to be the biggest issue. Routing, accuracy estimation, monitoring, and the proposed remedies all rely on the assumption that task IDs are given. In practice, routing is often performed at a fine-grained, query level, while monitoring is conducted at the level of broader categories. In such cases, the loss of a specific sub-capability may be masked by aggregate category-level metrics. The paper also acknowledges that its LLM experiments are limited to task-level routing and that generalization to natural query streams remains an open question (Section 8), but does not test this.
Furthermore, there is no analysis that separately characterizes how update settings (learning rate, number of epochs, etc.) affect the reported problem. It is necessary to verify whether the observed phenomenon and the proposed remedies persist across different update settings. The number of epochs is varied only for some remedy conditions in Figure 3(a), while the phenomenon itself (post-upgrade degradation) and the remaining remedies (floor, teacher labels, replay) are tested under only a single update setting. There are no experiments varying the learning rate. Section 8 states that smaller updates reduce this effect, but no supporting results are provided. Since update magnitude affects the learning and forgetting rates in the mathematical model and their ratio (φ), it may also change the magnitude of degradation, the required exposure, and when rollback remains feasible.
Questions for authors
-
Do the phenomenon and the proposed remedies persist when task IDs are not available and domains must be inferred using a classifier (including misclassification errors), or when observable domains are broader than the actual units of capability? Can domain-level monitoring detect the loss of sub-capabilities?
-
The LLM experiments use only task-level routing (Section 8). Do the phenomenon and the proposed remedies persist with a query-level router that routes individual queries without task IDs (e.g., based on the student's answer log-probability)?
-
How do update settings (learning rate, number of epochs) affect the magnitude of degradation and the effectiveness of the remedies (e.g., the required floor size and when rollback remains feasible)? Section 8 states that smaller updates reduce this effect. Are there results supporting this claim?
-
The pre-upgrade practice consists of short rounds. It is possible that much of this capability was acquired through brief LoRA training during the experiment and was therefore easily forgotten. The paper also states that the degradation is much smaller after more practice has accumulated (Section 5.1). Is the degradation still this large when the adapter has been sufficiently trained before the upgrade, or for capabilities already acquired during pretraining? Together with Question 3, I would like to understand whether the magnitude of degradation depends on an experimental setup that induces substantial forgetting.
5 comments
Response to Reviewer Dr1h [2/2]
Q2. "Do the phenomenon and the proposed remedies persist with a query-level router?"
Yes, both persist (General Response (1)). The loss scales with how much practice the router leaves: 6.4 points when the router keeps 26% of division queries and 22.8 points when it keeps 3%. The floor carries over as well. At the default cost a 5% floor removes the loss with no change in system accuracy, and when division is locked out, a 1% floor cuts the loss from 22.8 to 7.4 points. The clustering router of Q1 (a) is also a query-level router with no task IDs. Replacing ground truth with partial user corrections makes the query-level loss slightly larger (7.4 points, General Response (2)).
Q3. "How do update settings (learning rate, number of epochs) affect the magnitude of degradation and the effectiveness of the remedies?"
Smaller updates reduce the degradation, as Section 8 states, and the remedies remain effective at larger updates. We varied the learning rate and the number of epochs, and tested the remedies where forgetting is strongest. Division is deferred for ten rounds, and the entries are the accuracy lost relative to a frozen adapter.
| Division loss ↓ (points; 0 = no forgetting) | 1e-4 | 1.5e-4 | 2e-4 (paper) | 3e-4 | 4e-4 |
|---|---|---|---|---|---|
| No remedy | −0.1 | 6.0 | 20.5 | 37.7 | 52.3 |
| With a 30% practice quota | - | - | 7.9 (61% of the loss removed) | 17.2 (54% removed) | - |
| With teacher labels | - | - | 1.5 (93% removed) | 8.3 (78% removed) | - |
We summarize the findings below:
- Learning rate. The loss is zero at 1e-4 and grows steeply above it. We have added these results to Appendix C.4 ("Update size"), together with a three-seed learning-rate sweep that agrees (16.0 ± 2.6 points at 2e-4, 33.0 ± 4.5 points at 3e-4).
- Epochs. More update steps increase the loss as much as a larger step does: with one co-trained skill at the paper's learning rate, going from 2 to 18 epochs raises the loss from 7.6 to 26.8 points.
- Remedies. Both survive the largest learning rate in the paper's sweep. At 3e-4, teacher labels remove 78% of the loss and the 30% quota removes 54% (93% and 61% at 2e-4). The paper identifies the ratio of forgetting to learning as what sets the size of the loss and does not claim the loss is always large. These runs trace that dependence.
Q4. "Is the degradation still this large... for capabilities already acquired during pretraining?"
No. For a capability acquired during pretraining the degradation is far smaller, as the paper's practice-dependent forgetting rate predicts. We deferred addition/subtraction, which the base model already performs at 97.8%, for fifteen rounds with no labels. It ended at 96.0%, a loss of 1.8 points, whereas division lost 26.9 points over the same window. Even at the largest learning rate (3e-4) it lost far less than division (8.6 vs. 54 points). We fixed the prediction (a loss of at most 5 points at the paper's setting) before the run.
This agrees with Section 5.1 and Appendix C.2 ("Effect of prior training") of the submission, where more practice before the interruption reduces the degradation and the fitted forgetting rate falls from 0.254 per round at a fresh checkpoint to 0.087 after ten practiced rounds. We have moved this into the main text so that the conditions under which the loss is large are stated up front.
We hope our response addresses the reviewer's concerns, and we welcome any additional questions or feedback.
We thank the reviewer again for the helpful comments which have helped strengthen the paper. As the discussion period closes tomorrow, we wanted to check in to see if our responses have addressed your concerns. If so, we would appreciate your consideration in updating your assessment. We remain happy to answer any final questions you may have.
Response to Reviewer Dr1h [1/2]
We thank the reviewer for a careful and constructive review, and for recognizing the novelty of the formulation and modeling. We ran new experiments for each of the four questions.
Q1 (a). "Do the phenomenon and the proposed remedies persist when task IDs are not available?"
Yes, both persist. We built a router that never sees a task ID: it clusters the text of the calibration queries (k-means, numbers masked), gives each cluster its own confidence map and accuracy bar, and routes every later query by its nearest cluster.
At the default cost this router reproduces the task-ID router to the third decimal; as the cost tightens it locks division out and the loss grows. The floor needs no task ID either, because it keeps a random share of the queries the router would defer. With ten clusters the router recovers the ten skills exactly, and a simple text classifier does the same on 17,000 held-out queries with 100% accuracy. In our setup the task ID is information the router can read from the query itself.
Q1 (b). "...domains must be inferred using a classifier (including misclassification errors)?"
Misclassification errors do not create additional forgetting. We replaced the task label with a classifier's guess and injected errors.
A mislabelled division query is judged against another skill's bar, which is usually more lenient, so the student keeps more division practice, but the upgrade's gain in system accuracy shrinks with the error rate. With a persistent confusion (division read as multiplication) under a strict cost, the loss is still present at a 20% error rate (10.5 points). Across these runs, the loss follows how much of the skill the router withholds, whether or not the labels are right.
Q1 (c). "...or when observable domains are broader than the actual units of capability? Can domain-level monitoring detect the loss of sub-capabilities?"
When the observable domain is broader than the unit of capability, the sub-capability's deficit is masked, and domain-level monitoring does not detect it. We merged addition/subtraction, multiplication, and division into one "arithmetic" domain.
The same holds inside a skill: under the query router, division as a whole loses 6.4 points, while the mid-difficulty problems the router largely stops practicing lose 10 to 14 points. We take this as a reason to monitor student competence per skill and at finer grain than the routing unit, and have added it to the discussion of detection.