acceptodds
Desk rejected at ICLR 2027

Learning to Defer When Routing Changes Competence

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

26 views

Abstract

Learning-to-defer systems assign inputs according to the current competence of the available models or experts. When models train only on the inputs they handle, routing also determines the training exposure that sustains future competence. We study how routing and training choices maintain useful capabilities in continually updated cascades. In a controlled language-model cascade, upgrading the fallback raises system accuracy from 90.0% to 93.0% while the smaller model's division accuracy falls from 64.4% to 16.2%. Training on the fallback's answers to deferred inputs preserves division accuracy at 65.2%, allowing competence to be maintained even when the fallback remains preferred. Further interventions show that restoring target examples at a fixed training budget brings performance close to the control, while parameter isolation prevents loss from other tasks' updates. An exposure-dependent model of cost-sensitive deferral gives sufficient conditions for a unique competence equilibrium in terms of router smoothness and variation in case-level thresholds, along with exposure requirements and recovery times. Image cascades show a complementary source of competence loss as examples from deferred classes leave the training window. These results support designing routing and training policies around both current performance and retained competence.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Learning to defer, medical, healthcare, explainability, LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Dr1hOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the problems that arise when routing determines the scope of training during model updates in L2D and cascade model systems, where each model is updated according to its assigned tasks. In particular, it explores a hidden, counterintuitive phenomenon in which routing causes a lightweight student model to skip certain topics, improving overall system performance while substantially degrading the student's own performance. This phenomenon occurs especially when the fallback model is upgraded and becomes more capable, potentially posing risks if the fallback is later rolled back or becomes unavailable. The paper models this phenomenon using differential equations and validates the predictions of the model through thorough experiments. It also uses controlled experiments to disentangle the causes of performance degradation into the absence of training data for the affected tasks and interference among shared parameters, and compares remedies that decouple serving from training.

Critical strengths

A critical strength of this paper is its novel problem formulation and mathematical modeling that qualitatively captures the phenomenon. In particular, the analysis of training exposure and its timing is useful.

Critical weaknesses

A critical weakness is that the experimental setup is overly idealized. Topics arrive as clearly separated groups in a setting where all labels are available, the splits are easily distinguishable from prompts alone, and a calibration set is provided to estimate the accuracy of each model. I understand that these measures were taken to clearly demonstrate and analyze the phenomenon, but there are challenges in measuring the claimed phenomenon and applying the proposed remedies in realistic settings.

In particular, I consider domain identification to be the biggest issue. Routing, accuracy estimation, monitoring, and the proposed remedies all rely on the assumption that task IDs are given. In practice, routing is often performed at a fine-grained, query level, while monitoring is conducted at the level of broader categories. In such cases, the loss of a specific sub-capability may be masked by aggregate category-level metrics. The paper also acknowledges that its LLM experiments are limited to task-level routing and that generalization to natural query streams remains an open question (Section 8), but does not test this.

Furthermore, there is no analysis that separately characterizes how update settings (learning rate, number of epochs, etc.) affect the reported problem. It is necessary to verify whether the observed phenomenon and the proposed remedies persist across different update settings. The number of epochs is varied only for some remedy conditions in Figure 3(a), while the phenomenon itself (post-upgrade degradation) and the remaining remedies (floor, teacher labels, replay) are tested under only a single update setting. There are no experiments varying the learning rate. Section 8 states that smaller updates reduce this effect, but no supporting results are provided. Since update magnitude affects the learning and forgetting rates in the mathematical model and their ratio (φ), it may also change the magnitude of degradation, the required exposure, and when rollback remains feasible.

Questions for authors
  • Do the phenomenon and the proposed remedies persist when task IDs are not available and domains must be inferred using a classifier (including misclassification errors), or when observable domains are broader than the actual units of capability? Can domain-level monitoring detect the loss of sub-capabilities?

  • The LLM experiments use only task-level routing (Section 8). Do the phenomenon and the proposed remedies persist with a query-level router that routes individual queries without task IDs (e.g., based on the student's answer log-probability)?

  • How do update settings (learning rate, number of epochs) affect the magnitude of degradation and the effectiveness of the remedies (e.g., the required floor size and when rollback remains feasible)? Section 8 states that smaller updates reduce this effect. Are there results supporting this claim?

  • The pre-upgrade practice consists of short rounds. It is possible that much of this capability was acquired through brief LoRA training during the experiment and was therefore easily forgotten. The paper also states that the degradation is much smaller after more practice has accumulated (Section 5.1). Is the degradation still this large when the adapter has been sufficiently trained before the upgrade, or for capabilities already acquired during pretraining? Together with Question 3, I would like to understand whether the magnitude of degradation depends on an experimental setup that induces substantial forgetting.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Dr1h [1/2]

We thank the reviewer for a careful and constructive review, and for recognizing the novelty of the formulation and modeling. We ran new experiments for each of the four questions.

Q1 (a). "Do the phenomenon and the proposed remedies persist when task IDs are not available?"

Yes, both persist. We built a router that never sees a task ID: it clusters the text of the calibration queries (k-means, numbers masked), gives each cluster its own confidence map and accuracy bar, and routes every later query by its nearest cluster.

Router cost (no task IDs)Division queries served by the studentDivision, no upgrade (%)Division, upgrade ↑ (%)Division loss ↓ (points)
.25 (default)26%54.948.56.4 (task-ID router: 6.4)
.1519%55.047.37.7
.105%53.342.610.7
.051%47.935.812.1
.05 + 20% floor24%47.941.56.4

At the default cost this router reproduces the task-ID router to the third decimal; as the cost tightens it locks division out and the loss grows. The floor needs no task ID either, because it keeps a random share of the queries the router would defer. With ten clusters the router recovers the ten skills exactly, and a simple text classifier does the same on 17,000 held-out queries with 100% accuracy. In our setup the task ID is information the router can read from the query itself.

Q1 (b). "...domains must be inferred using a classifier (including misclassification errors)?"

Misclassification errors do not create additional forgetting. We replaced the task label with a classifier's guess and injected errors.

Label errorDivision queries kept by the studentDivision, upgrade ↑ (%)Division loss ↓ (points; no upgrade: 54.9%)System accuracy ↑ (%; no upgrade: 89.8%)
none26%48.56.492.4
random, 10%53%54.50.491.6
random, 30%60%51.63.391.3
random, 90%82%63.7none89.5

A mislabelled division query is judged against another skill's bar, which is usually more lenient, so the student keeps more division practice, but the upgrade's gain in system accuracy shrinks with the error rate. With a persistent confusion (division read as multiplication) under a strict cost, the loss is still present at a 20% error rate (10.5 points). Across these runs, the loss follows how much of the skill the router withholds, whether or not the labels are right.

Q1 (c). "...or when observable domains are broader than the actual units of capability? Can domain-level monitoring detect the loss of sub-capabilities?"

When the observable domain is broader than the unit of capability, the sub-capability's deficit is masked, and domain-level monitoring does not detect it. We merged addition/subtraction, multiplication, and division into one "arithmetic" domain.

  • With the task-level router, the domain's average accuracy clears the bar after the upgrade, so division is never deferred and keeps its practice (63.8%). The system then cannot use the stronger fallback on division, and system accuracy is lower (87.7% vs. 92.4% with per-skill routing). The coarse domain hides division's deficit inside the average.
  • With the query-level router, the upgrade costs division 6.1 points at the default cost. Domain-level monitoring cannot see the loss in the paper's main setting either.
What a monitor watchesChange after the upgrade (points; negative = worse)
Student accuracy on division−26.9
"Arithmetic" category (three skills)−5.2
Student mean over all ten skills−2.0 (round-to-round noise ±1.5)
System accuracy+2.9

The same holds inside a skill: under the query router, division as a whole loses 6.4 points, while the mid-difficulty problems the router largely stops practicing lose 10 to 14 points. We take this as a reason to monitor student competence per skill and at finer grain than the routing unit, and have added it to the discussion of detection.

AuthorsOpenReview ↗

Response to Reviewer Dr1h [2/2]

Q2. "Do the phenomenon and the proposed remedies persist with a query-level router?"

Yes, both persist (General Response (1)). The loss scales with how much practice the router leaves: 6.4 points when the router keeps 26% of division queries and 22.8 points when it keeps 3%. The floor carries over as well. At the default cost a 5% floor removes the loss with no change in system accuracy, and when division is locked out, a 1% floor cuts the loss from 22.8 to 7.4 points. The clustering router of Q1 (a) is also a query-level router with no task IDs. Replacing ground truth with partial user corrections makes the query-level loss slightly larger (7.4 points, General Response (2)).

Q3. "How do update settings (learning rate, number of epochs) affect the magnitude of degradation and the effectiveness of the remedies?"

Smaller updates reduce the degradation, as Section 8 states, and the remedies remain effective at larger updates. We varied the learning rate and the number of epochs, and tested the remedies where forgetting is strongest. Division is deferred for ten rounds, and the entries are the accuracy lost relative to a frozen adapter.

Division loss ↓ (points; 0 = no forgetting)1e-41.5e-42e-4 (paper)3e-44e-4
No remedy−0.16.020.537.752.3
With a 30% practice quota--7.9 (61% of the loss removed)17.2 (54% removed)-
With teacher labels--1.5 (93% removed)8.3 (78% removed)-

We summarize the findings below:

  • Learning rate. The loss is zero at 1e-4 and grows steeply above it. We have added these results to Appendix C.4 ("Update size"), together with a three-seed learning-rate sweep that agrees (16.0 ± 2.6 points at 2e-4, 33.0 ± 4.5 points at 3e-4).
  • Epochs. More update steps increase the loss as much as a larger step does: with one co-trained skill at the paper's learning rate, going from 2 to 18 epochs raises the loss from 7.6 to 26.8 points.
  • Remedies. Both survive the largest learning rate in the paper's sweep. At 3e-4, teacher labels remove 78% of the loss and the 30% quota removes 54% (93% and 61% at 2e-4). The paper identifies the ratio of forgetting to learning as what sets the size of the loss and does not claim the loss is always large. These runs trace that dependence.

Q4. "Is the degradation still this large... for capabilities already acquired during pretraining?"

No. For a capability acquired during pretraining the degradation is far smaller, as the paper's practice-dependent forgetting rate predicts. We deferred addition/subtraction, which the base model already performs at 97.8%, for fifteen rounds with no labels. It ended at 96.0%, a loss of 1.8 points, whereas division lost 26.9 points over the same window. Even at the largest learning rate (3e-4) it lost far less than division (8.6 vs. 54 points). We fixed the prediction (a loss of at most 5 points at the paper's setting) before the run.

This agrees with Section 5.1 and Appendix C.2 ("Effect of prior training") of the submission, where more practice before the interruption reduces the degradation and the fitted forgetting rate falls from 0.254 per round at a fresh checkpoint to 0.087 after ten practiced rounds. We have moved this into the main text so that the conditions under which the loss is large are stated up front.

We hope our response addresses the reviewer's concerns, and we welcome any additional questions or feedback.

We thank the reviewer again for the helpful comments which have helped strengthen the paper. As the discussion period closes tomorrow, we wanted to check in to see if our responses have addressed your concerns. If so, we would appreciate your consideration in updating your assessment. We remain happy to answer any final questions you may have.

Reviewer_iTxcOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates a feedback loop in which can occur in continually updated model cascades (and L2D systems) when a smaller "student" model is only trained on the queries it answers itself. If the router starts deferring a particular task to a stronger fallback model then the student model stops recieving training examples for that task while continuing to update its weights on other tasks. As a result, the authors find that the student suffers from catastrophic forgetting on the deferred task which increases the performance gap and makes the system rely on the expensive fallback model even if the upgrade is later rolled back.

Critical strengths

The ablations are clean and well-controlled with a clear purpose of isolating why the skill loss happened

Critical weaknesses

From my understanding, the paper rests on a key assumption of which when the student autonomously handles a query, it is allowed to train upon this query and hence receives the ground-truth. It does not train under circumstances when the router decides to defer to the larger expert model. I think the opposite is true in real-world LLM cascades and L2D: if the student handles a query autonomously, you do not query the expert (so you rarely get free ground-truth labels), whereas when you do defer to the expensive fallback model (or human expert), you have already paid for the fallback answer, meaning teacher labels are available at zero extra inference cost.

Additionally, there may exist online cascading papers of which is very similar to this work [1]. Whilst the authors do discuss this paper, they brush past it in a single sentence without acknowledging that it already uses the fallback model’s answers on deferred queries to continually update the smaller model, which is essentially one of the main interventions proposed here.

[1] Online Cascade Learning for Efficient Inference over Streams, Ni et al., ICML 2024

Questions for authors
  • The baseline assumes that the student gets the ground truth label on all queries it serves itself, but zero training on deferred queries. Is it not that real world cascading is the opposite? In what realistic deployment would you have oracle labels for kept queries but no teacher signal for deferred one?
  • In the main LLM experiments the routing is an "all or none" per known task bucket rather than instance-level L2D. If you used a standard instance-level L2D router (where the student keeps easy examples of a skill after the fallback upgrade) does this severe skill collapse and lock in still happen? Or is it mostly an artifact of deferring 100% of a task bucket at once?
6 comments
AuthorsOpenReview ↗

Response to Reviewer iTxc [1/2]

We thank the reviewer for the direct questions, which led to our most informative new experiments.

W1. "The paper rests on a key assumption of which when the student autonomously handles a query, it is allowed to train upon this query... I think the opposite is true in real-world LLM cascades and L2D."

We clarify that the paper does not rest on this assumption; its requirement is only that the training signal depends on which model answered. The paper specifically examines and recommends the regime the reviewer describes, in which the student trains on the fallback's answers to deferred queries ("teacher labels"): "the stronger fallback can therefore contribute both answers and training feedback" (lines 303 to 305), and teacher labels "provide another source of feedback when fresh ground truth is unavailable" (lines 417 to 418). The submission already shows that training on the fallback's answers to deferred queries keeps division at 65.2%, level with the no-upgrade control (64.4%), while the upgrade without those labels drops it to 16.2% (eight seeds; Table 1 and Figure 1a). Section 7 (lines 409 to 419) and Table 2 further compare teacher labels against floors, replay, and independent training access at a matched training budget, and Proposition 4 (lines 195 to 203) shows that when labels do not depend on who served the query, serving can stay greedy while training is chosen separately.

In response to the reviewer's feedback, we added five sets of runs that vary which queries carry a training label (General Response (2)).

  • Teacher labels only. The submission's teacher-label arm kept correct labels on served queries; this arm removes them to match the reviewer's regime exactly, so the student trains only on the fallback's answers to deferred queries. Division ends at 64.3%, against 30.1% for the upgrade without fallback labels.
  • Verified answers plus teacher labels. The student trains on its own served answers only when a verifier confirms them, and on the fallback's answers to deferred queries. Division ends at 55.4%, against 28.1% without the fallback's answers.
  • Teacher labels at a larger update size. At learning rate 3e-4, where forgetting is strongest, training on the fallback's answers removes 78% of the division loss (93% at the paper's 2e-4).
  • User corrections in place of oracle labels. The student trains on its own served answer when it is right; when it is wrong, a user supplies a correction half the time. Deferred queries carry no label. This feedback holds division at 57.0% while division is served, but when the upgrade defers division it falls to 22.8%, while system accuracy rises from 89.6% to 92.0%. Under the instance-level router of Q2, the same feedback gives a loss of 7.4 points (47.5% to 40.1%).
  • Teacher labels only, at the control's training volume. Each round the student trains on 900 fresh fallback-labelled queries from the deferred skills, which stay deferred. Division rises to 73.2%, while the seven skills the student serves, which get no labels, fall from 96.5% to 90.5% on average (90.1% to 76.5% for the two with the most room to fall).

The first three sets confirm the paper's recommendation without any oracle label. The fourth shows that the loss does not depend on oracle labels either: it is larger with partial user corrections (34.2 points) than with correct labels, so the paper's baseline was the conservative choice. The fifth shows that in the reviewer's regime routing still decides which skills are maintained. The labels follow the deferred queries, so the deferred skill improves and the served skills erode.

AuthorsOpenReview ↗

Response to Reviewer iTxc [2/2]

Q1. "In what realistic deployment would you have oracle labels for kept queries but no teacher signal for deferred ones?"

Both feedback directions are valid depending on how a system is deployed. The reviewer's direction holds when the fallback's answers can be logged and reused at no extra inference cost, whereas the opposite holds when feedback attaches directly to the answer the user saw, such as accepted or rejected suggestions in a coding assistant, user edits in a chat product, or pass/fail results from running generated code, because that feedback exists only for queries the student served. At the same time, deferred queries often lack a usable training signal due to unlogged human-expert decisions, privacy restrictions, or third-party API terms that prohibit training on outputs.

Our primary evaluation (Table 1, Figure 1) models this served-only feedback structure with a correct label on every served answer, and the fourth set of runs above replaces those oracle labels with simulated partial user corrections. The L2D literature actively studies this feedback structure, including online L2D under bandit feedback [1] and selective labels [2, 3, 4], and fine-tuning on user edits of served outputs is an established paradigm for deployed LLMs [5]. Section 3.1 (lines 119 to 122) frames the baseline the same way, as feedback that exists only for served answers, "such as user corrections or verification of those answers".

We use this setting as the primary evaluation because routing is then the only factor deciding what the student practices: once division is deferred, the student receives no division examples while continuing to update on other skills, which erodes division (Section 6). More broadly, the paper's claim covers any system where the training signal depends on which model answered, including the teacher-only regime shown in the fifth set of runs. We have revised the introduction and Section 3.1 to state this scope, and the recommendation to train on fallback answers, earlier and more plainly.

[1] Duy et al. Online Learning-to-Defer with Varying Experts. arXiv:2605.12340, 2026.

[2] Lakkaraju et al. The Selective Labels Problem: Evaluating Algorithmic Predictions in the Presence of Unobservables. KDD 2017.

[3] De-Arteaga, Dubrawski, and Chouldechova. Learning under Selective Labels in the Presence of Expert Consistency. FAT/ML 2018.

[4] Leitão et al. Human-AI Collaboration in Decision-Making: Beyond Learning to Defer. ICML 2022 Workshop on Human-Machine Collaboration and Teaming.

[5] Misra et al. Principled Fine-tuning of LLMs from User-Edits: A Medley of Preference, Supervision, and Reward. NeurIPS 2025.

Q2. "If you used a standard instance-level L2D router... does this severe skill collapse and lock-in still happen?"

Yes, the collapse and lock-in still happen, so they are not an artifact of deferring 100% of a task bucket at once. Appendix B of the submission already describes an instance-level router, and we now report its results (General Response (1)). The size of the collapse follows how much practice the router leaves the student. At the default cost it keeps 26% of division queries, and the loss falls to 6.4 points. With division errors priced more strictly, it keeps 3% and locks the student out of division in 12 of 15 rounds, returning the loss to 22.8 points. Instance-level routing also biases practice toward easy problems, whereas uniform practice at the same count kept division at the no-upgrade level, and adding a 5% uniform floor removes the loss.

The same holds when the router clusters query text instead of reading task labels: it matches the labelled router exactly at the default cost and locks division out as the cost tightens (1% kept, loss 12.1 points). It also holds with user corrections in place of ground truth, as noted above.

W2. Relation to Nie et al. (ICML 2024).

Nie et al. study how to learn an efficient cascade over a stream, updating the smaller models on the larger model's answers to deferred queries. We use the same mechanism as one remedy (teacher labels) and do not claim it as a contribution. Our contribution is the failure mode and its analysis: what a routing change does to the small model's competence, when the loss becomes hard to reverse, and how much practice prevents it. The new results also show when their mechanism maintains a skill. It needs accurate fallback labels (with 7B labels at about 65% accuracy, division declines to 35.2%), and it leaves the skills the student serves without a training signal. We have updated the related-work discussion to make this distinction explicit.

We thank the reviewer for highlighting the importance of clarifying the scope of our work. We hope the reviewer will consider these important clarifications to the initial understanding while re-evaluating the paper.

Thanks to the authors for the rebuttal. I also acknowledge that the submission already credits Nie et al. for training on deferred answers, so I withdraw that part of my criticism. These results support the proposed failure mechanism within the stated setting, and I will raise my overall rating from 1 to 2.

My remaining concern is the scope and significance of the contribution. In the reported instance-level experiments, the default setting produces a much smaller loss, while severe degradation occurs when almost all division queries are deferred. The teacher-only experiment demonstrates a different pattern of forgetting, but does not really establish the same self-reinforcing lock-in. Also, persistent deferral under an upgraded fallback does not truly establish failure to recover after rollback. I therefore remain at weak rejection because the broader practical relevance and additional insight beyond established continual-learning and online-distillation approaches remain insufficiently convincing.

Reviewer_SXvJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how routing decisions affect the future competence of models in a continually updated cascade. The authors show that using a stronger fallback can improve overall system accuracy while causing the smaller model to forget previously learned skills. The paper also provides a theoretical analysis and explores several methods to maintain or recover these skills.

Critical strengths
  • The paper studies an interesting problem in learning-to-defer systems. It shows that better system performance does not always mean that individual models maintain their capabilities.

  • The paper is generally well-written and easy to follow.

  • The experiments are well designed. In particular, Section 6 helps explain how reduced training exposure and interference from other tasks cause forgetting.

Critical weaknesses

The main results are based on task-level routing, where a deferred task receives no further training examples. Appendix B mentions a query-level router, but its results are not reported. Since query-level routing may still provide some training examples for deferred tasks, it is unclear whether the same level of forgetting would occur.

The theoretical model does not fully capture the forgetting process shown in the experiments. Figure 3 shows that forgetting is affected by which other tasks the student trains on. However, the main theoretical model assumes a fixed forgetting rate and does not account for these differences. This limits how well the theory can explain the behavior of a multi-task learning system.

Questions for authors

Please see weaknesses above.

5 comments
AuthorsOpenReview ↗

Response to Reviewer SXvJ

We thank the reviewer for their comments on the problem and the experimental design. We ran new experiments for both weaknesses.

W1. "Since query-level routing may still provide some training examples for deferred tasks, it is unclear whether the same level of forgetting would occur."

Forgetting still occurs with query-level routing (General Response (1)). When the router withholds almost all division queries, it nearly matches task-level routing (22.8 vs. 26.9 points). When the router still provides a quarter of them as training examples, as the reviewer anticipates, the forgetting is smaller (6.4 points). The size follows how many training examples the student keeps, as the paper's model predicts. The router also keeps mostly easy problems, so the harder problems it stops seeing lose about twice as much as division overall. The paper's remedy carries over: always sending a random 5% of queries to the student removes the loss with no change in system accuracy.

The result holds in two further settings. With a router that clusters query text instead of reading task labels, the loss rises from 6.4 to 12.1 points as the cost tightens. With partial user corrections in place of ground-truth labels, the query-level loss is 7.4 points. These results are in the revised Section 6 and Appendix B.

W2. "The main theoretical model assumes a fixed forgetting rate and does not account for these differences."

A fixed forgetting rate accounts for the differences the reviewer points to in Figure 3b. They come from how many update steps the shared model receives, whichever tasks train alongside the deferred skill, and the number of steps does not change while a skill is deferred. To show this, we ran a grid in which division receives no training for ten rounds while other skills train the shared adapter, varying the number of co-trained skills, the learning rate, and the number of epochs. Entries are the division accuracy lost relative to a frozen adapter.

Co-trained skills (entries: division loss ↓ in points, 0 = no forgetting)Update steps per round1e-41.5e-42e-4 (paper)3e-44e-4
1 skill, 2 epochsabout 261.16.57.623.739.8
9 skills, 2 epochsabout 226−0.16.020.537.752.3
1 skill, 18 epochs (same steps as 9 skills)about 234not run13.126.849.461.0

Figure 3b does show more forgetting when more tasks co-train. Nine co-trained skills supply nine times the rows and therefore nine times the gradient steps on the shared adapter. Given the same number of steps (third row), a single co-trained skill erodes division at least as much as nine (26.8 vs. 20.5 points at the paper's learning rate), so which tasks co-train matters little once the amount of updating is equal.

Step size has the largest effect: one co-trained skill at 4e-4 loses about twice what nine skills lose at 2e-4 (39.8 vs. 20.5 points), and no set loses anything at 1e-4.

The forgetting rate is therefore set by how much the shared adapter is updated, through the step size and the number of steps. Both are set by the training configuration and stay the same whichever skill is deferred. Propositions 1 to 3 and Corollary 2 require the rate to be constant only during such an interruption. The one change an interruption does cause, the deferred skill's own data leaving the batch, enters the model through the exposure term. As a check on the full model, the law with its original parameters predicts final division accuracy across the 63 new runs in its setting with an error of 3.8 points, better than both baselines (General Response (1)).

In the revision we present the law explicitly as a reduced form, add this grid to the appendix, and point to the existing extensions in Appendix A.1 (rate falling with accumulated practice) and G.3 (cross-skill transfer).

We hope these results address the reviewer's concerns and are happy to answer any lingering questions.

We thank the reviewer again for the helpful comments which have helped strengthen the paper. As the discussion period closes tomorrow, we wanted to check in to see if our responses have addressed your concerns. If so, we would appreciate your consideration in updating your assessment. We remain happy to answer any final questions you may have.

With the discussion period closing today, we wanted to check in one final time to see if our additional experiments and clarifications have resolved your concerns, or if there is any remaining issue holding back your overall assessment that we can quickly clarify. If your concerns have been addressed, we would greatly appreciate your consideration in updating your score. Thank you again for your time and helpful feedback.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to all reviewers

We thank the reviewers for their careful feedback. We are encouraged that they found the problem interesting and the paper clear (SXvJ), the ablations clean and well controlled (iTxc, SXvJ), and the problem formulation and modeling novel (Dr1h). We have updated the paper with all requested improvements, with updated text shown in blue.

We summarize each concern below, with our answer and the evidence from the new runs. The two concerns shared across reviewers are covered in detail in sections (1) and (2); the others are covered in the individual replies.

  1. Is the collapse "an artifact of deferring 100% of a task bucket at once"? With query-level routing, would "the same level of forgetting" occur? (SXvJ, iTxc, Dr1h)

    No. Forgetting still occurs with query-level routing. With a strict penalty on division errors, the query-level router withholds every division query in the last nine rounds, and division accuracy falls 22.8 points (26.9 under task-level routing). At a moderate penalty it withholds 87% of division queries, and accuracy falls 9.3 points. Always sending the student a small random share of queries (a "floor") prevents this: floors of 1 to 5% remove two thirds to 97% of the loss.

  2. "Do the phenomenon and the proposed remedies persist when task IDs are not available?" (Dr1h)

    Yes, both persist without task IDs. A router that clusters query text instead of reading task IDs gives the same division accuracy as the task-ID router (48.5% vs. 54.9% without the upgrade). As its penalty tightens, it withholds 99% of division queries and the student loses 12.1 points. A 20% floor recovers about half of that loss.

  3. "In what realistic deployment would you have oracle labels for kept queries but no teacher signal for deferred ones?" (iTxc, Dr1h)

    Both settings occur in practice, and the revision reports results for each. The submission already recommends using the teacher signal on deferred queries (Table 1). With a teacher signal on deferred queries and no labels on kept ones, division is protected (64.3% vs. 30.1% without the teacher signal). With no teacher signal and partial user corrections in place of oracle labels, the upgrade still costs 34.2 points.

  4. With a teacher signal on deferred queries only, does routing still matter? (iTxc)

    Yes. Routing still decides which skills are maintained. The deferred skill rises to 73.2%, but the skills the student keeps serving get no labels and lose 6.0 points on average (96.5% to 90.5%).

  5. The theory "assumes a fixed forgetting rate and does not account for these differences" between co-trained tasks. (SXvJ)

    The differences come from how many update steps the shared adapter receives, whichever tasks co-train. That number stays fixed while a skill is deferred, so the rate is constant during deferral, as the theory assumes. With update steps held equal, one co-trained skill causes as much forgetting as nine (26.8 vs. 20.5 points).

  6. "How do update settings affect the magnitude of degradation and the effectiveness of the remedies?" (Dr1h)

    Smaller updates reduce the degradation, as Section 8 states, and the remedies stay effective at larger ones. Division loses nothing at learning rate 1e-4 and 52.3 points at 4e-4. Teacher labels remove 78% of the loss at 3e-4.

  7. "Is the degradation still this large... for capabilities already acquired during pretraining?" (Dr1h)

    No. It is far smaller, as the paper's model predicts. A pretrained skill (addition/subtraction) loses 1.8 points where the briefly trained skill (division) loses 26.9.

  8. "Can domain-level monitoring detect the loss of sub-capabilities?" (Dr1h)

    No. The loss is masked at the domain level. A 26.9-point loss on division appears as a 2.0-point drop in the student's average over all skills, and as a 2.9-point gain in system accuracy.

AuthorsOpenReview ↗

Concern 1 Summary

All new runs use the paper's LLM setup (Qwen2.5-1.5B student with LoRA, 7B fallback upgraded to 14B at round 10, long division as the target skill) with one seed, and each prediction had pass/fail thresholds fixed before the run.

Accuracies are exact-match accuracy on held-out questions, in percent, averaged over the last five rounds. "Division accuracy" is the student's own accuracy on long division; "system accuracy" is the accuracy of the deployed system, with the fallback answering whatever the router defers. "Loss" is the number of points by which the student's division accuracy falls below the same system without the upgrade: 0 means the skill is fully retained, and 26.9 is the full collapse in the paper's main setting. In the tables, arrows mark the better direction and bold marks the result that answers the concern.

Concern 1: is the collapse an artifact of deferring a whole task at once?

It is not. We ran the query-level router from Appendix B, which routes each query on the student's calibrated chance of answering it correctly, so after the upgrade it keeps the division queries the student is likely to get right.

Router after the upgradeDivision queries the student still trains onDivision accuracy ↑ (%)Division loss ↓ (points)System accuracy ↑ (%)
No upgrade, task-level (reference)100%57.0089.5
Task-level (paper)0%30.126.992.4
No upgrade, query-level (reference)68%54.9089.8
Query-level, default cost26%48.56.492.4
Query-level, division errors priced higher13%45.69.392.7
Query-level, division errors priced highest3%32.122.892.9
Query-level, default cost + 5% practice floor50%54.70.292.5
Query-level, highest price + 1% practice floor10%47.57.492.4
Query-level, highest price + 10% practice floor17%46.58.492.3

Takeaways:

  • The loss follows how many division queries the router leaves the student to train on: 6.4 points when it keeps a quarter, and nearly the task-level figure when it keeps 3%. As division errors are priced higher, the router withholds more division queries; at the highest price it kept none in 12 of 15 rounds, including the last nine.
  • The router also keeps mostly easy problems, so the mid-difficulty problems it largely stops seeing lose 10 to 14 points, twice the loss on division overall. Training on uniformly sampled division problems at the same count kept division at the no-upgrade level (54.7% vs. 48.5%).
  • The paper's remedy carries over. A practice floor sends the student a random share of the queries the router would defer, so it needs no task label. At the default cost a 5% floor removes the loss with system accuracy unchanged, and when the router has locked division out, a 1% floor cuts the loss from 22.8 to 7.4 points.
  • Forgetting under query-level routing also persists without task IDs (reply to Dr1h) and with user corrections in place of ground-truth labels (section (2)).
  • The paper's model (the "exposure law", under which a skill's accuracy is set by how much the student trains on it) predicts the new runs without refitting. On the 63 routing runs in its setting (correct labels on served queries, the paper's learning rate), it predicts final division accuracy from each run's division training more closely than two simple baselines.
Predictor of final division accuracy (63 runs)Mean absolute error ↓ (points)
Exposure law, original parameters3.8 (rank correlation 0.89)
"No change": each run stays at its accuracy when the experiment branches5.1
"Average": the mean of all runs, for every run5.0
AuthorsOpenReview ↗

Concern 2 Summary

Concern 2: is it realistic for served queries to carry correct labels while deferred queries give no training signal?

In the paper's main setting, the student gets the correct label for every query it serves and nothing for the queries it defers. iTxc and Dr1h asked what deployment would look like this, since the fallback's answers to deferred queries are often available for training.

Where the fallback's answers are available, the paper already recommends training on them (Table 1 and Section 7), and the new runs show that this protects the deferred skill even when served queries carry no labels. Where they are not, for example when the fallback is a human expert or a third-party API whose terms restrict training on its outputs, feedback usually attaches to the answer the user saw (a correction, an accepted edit, a test result) and covers only some answers. Replacing the oracle labels with such partial user corrections makes the forgetting larger, so the paper's main setting is the favourable case. The table varies which queries carry a training label after the 14B upgrade.

Feedback regimeServed queries train onDeferred queries train onDivision without the upgrade (%)Division after the upgrade ↑ (%)Division loss ↓ (points)
Served-only (paper), task-level routerground truthnothing57.030.126.9
Teacher-only (iTxc's regime)nothingfallback's answer57.0 (paper's control)64.3none
Verified + teacherown answer, if verified correctfallback's answer57.0 (paper's control)55.41.6
User corrections, task-level routerown answer if right, otherwise a user correction half the timenothing57.022.834.2
User corrections, query-level routersamenothing47.540.17.4

Takeaways:

  • Training on the fallback's answers to deferred queries protects the skill, as the paper recommends, and still removes 78% of the loss at a larger update size (reply to Dr1h).
  • Without oracle labels the forgetting is larger. User corrections, edits, and execution results all attach to whichever model served the query. With partial user corrections, the upgrade costs division 34.2 points under the task-level router and 7.4 points under the query-level router, more than with ground truth in both cases (26.9 and 6.4). System accuracy rises in both cases.
  • When only deferred queries carry labels, routing still decides which skills are maintained, and the forgetting moves to the skills the student keeps. We matched the control's training volume (900 rows per round) with fresh fallback-labelled queries and kept the deferred skills deferred. Division rose to 73.2%, while the seven served skills, which get no labels, fell from 96.5% to 90.5% on average (90.1% to 76.5% for the two with the most room to fall). In the revision we state the general condition: whenever the label process depends on which model served a query, routing decides which competences are maintained. Served-only training and the teacher-label regime are both cases of this same general process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.