acceptodds
Under review as a conference paper at ICLR 2027

PMOPD: Task Ordering, Cycling, and Parameter-Update Subspace Protection in Multi-Teacher On-Policy Distillation

Abstract

Multi-teacher on-policy distillation (MOPD) has emerged as a popular post-training paradigm for integrating specialized capabilities in frontier language models. Existing OPD research has primarily focused on optimizing single-task distillation through objective design, distillation scope, and teacher signal construction, whereas MOPD must aggregate multiple capabilities in shared parameters and address the resulting capability seesaw, in which improving one domain suppresses capabilities acquired from another. Inspired by the distinctive update geometry of OPD, we find that parameter updates from different tasks rapidly concentrate in their respective low-dimensional subspaces during MOPD, providing a direct geometric basis for identifying and controlling cross-task interference. We therefore propose PMOPD (Projection-based Multi-Teacher On-Policy Distillation), which constructs subspace memories from the cumulative parameter displacements of different tasks and projects both gradients and optimizer updates to remove components that interfere with protected task directions. We further develop a lightweight conflict probe to characterize task interactions and guide task ordering, together with a cycling strategy that balances subspace estimation and timely task revisitation. Experiments on representative Code, Reason, and Math tasks show that PMOPD improves every evaluated capability over MOPD, raising the average score across the three tasks by 2.54 points on Qwen2.5-7B and 2.09 points on Llama-3.1-8B. These consistent gains establish geometry-aware optimization as an effective and transferable approach to balanced multi-teacher distillation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Teacher On-Policy Distillation, Task-Update Subspace Protection, Multi-Task Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_G16FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

PMOPD aims to reduce interference between Math, Reason, and Code teachers during on-policy distillation. It trains tasks in ordered blocks and protects subspaces learned from earlier blocks by projecting both gradients and optimizer updates. It also proposes diagnostics for choosing the order and cycle count, and improves every evaluated task over MOPD on Qwen2.5-7B and Llama-3.1-8B.

Critical strengths
  • Looking at cumulative parameter changes rather than individual mini-batch gradients is an interesting way to study task interference.
  • The method is simple. The observation that Adafactor can reintroduce protected components after gradient projection is useful, and Table 4 supports using both projections in the tested configurations.
  • Gains over MOPD are consistent across tasks and two model families, with average improvements of 2.54 and 2.09 points. The order and cycle sweeps also help show how sensitive the method is to these choices.
Critical weaknesses
  1. The evaluation is limited to a few task-specific test sets, with no uncertainty estimates. This makes smaller gains, such as +0.82 on Llama Reason, hard to assess. The results are encouraging, but broader evaluation and results across seeds are needed to support the transferability claim.

  2. I could not tell how the task order and cycle count were selected. No validation split is identified, and the chosen settings are also the best-scoring entries in Table 5 and Figure 4. This does not establish test-set tuning, but the selection procedure needs clarification. There is also a cost question: the cycle-consistency measure seems to require training each candidate schedule, despite being presented as guidance before a full run. How the probe memory MaM_a is obtained is unclear as well.

  3. Table 4 is missing a particularly useful control: projecting only the optimizer update. By Eq. 12, that projection already makes the update orthogonal to the memory. What does projecting the gradient as well add? A suitably adapted GPM-style baseline would also help distinguish the displacement-based memory from existing projection methods.

  4. Protecting directions in which a task previously changed the model is plausible, but the paper does not directly show why this preserves that task's capability. Memory is also reset each cycle, removing the previous cycle's explicit constraints. Performance traces across cycle boundaries would help show whether earlier tasks degrade during the newly unconstrained first block.

Questions for authors
  1. Which data were used to select the order and cycle count? Were these settings fixed before inspecting test scores?
  2. How is MaM_a constructed, and can the diagnostics choose a cycle count without fully training each candidate schedule?
  3. How does PMOPD compare with projecting only the optimizer update?
  4. How stable are the gains across seeds, and what happens to earlier-task performance when the memory is reset?
8 comments
AuthorsOpenReview ↗

Response to Reviewer G16F — Question 1 (Part 1 of 1)

Response to Reviewer G16F — Question 1 (Part 1 of 1)

We thank the reviewer for pointing out the missing uncertainty estimates. We have now repeated MOPD using the same three random seeds as PMOPD and updated the results to consistently report the mean ± standard deviation over three runs for both methods, as shown below.

Qwen2.5-7B

MethodMathReasonCodeAvg.
MOPD65.44 ± 0.3071.39 ± 0.3155.86 ± 0.2364.23 ± 0.26
PMOPD67.78 ± 0.2973.46 ± 0.3659.67 ± 0.3166.97 ± 0.19

Llama-3.1-8B

MethodMathReasonCodeAvg.
MOPD17.90 ± 0.3867.85 ± 0.7029.60 ± 0.4238.45 ± 0.49
PMOPD21.33 ± 0.1169.45 ± 0.3432.33 ± 0.2541.04 ± 0.15

For Llama Reason, which the reviewer specifically highlighted, the updated results are 69.45 ± 0.34 for PMOPD and 67.85 ± 0.70 for MOPD. The improvement computed from the three-run means of both methods is therefore 1.60 percentage points, rather than the previously reported 0.82 percentage points. This improvement exceeds the observed standard deviation of either method, providing further support for robustness across training seeds in these runs.

The updated average improvements are 2.74 percentage points on Qwen2.5-7B and 2.59 percentage points on Llama-3.1-8B. We will include uncertainty estimates in the revised results table, clarify the repetition protocol, and update the corresponding numerical comparisons throughout the text.

AuthorsOpenReview ↗

Response to Reviewer G16F — Question 2 (Part 1 of 1)

Response to Reviewer G16F — Question 2 (Part 1 of 1)

We thank the reviewer for highlighting the need to clarify configuration selection, computational costs, and the construction of probe memories. We address these points below.

1. Data separation and the basis for selection

Our training and test data are separate, and the test set is not used for model training. Both the full OPD training runs and the short training runs used for task-order probing use training data, while the test set is used for capability evaluation. Task-order probing uses subsets of the OPD training data. Each task has 600 OPD training examples.

2. Cost comparison between task-order probing and exhaustive evaluation of all six orders

Instead of fully training and evaluating all six task orders, we can obtain an order directly from a single small-sample conflict probe. Probing with 60 training examples per task recovered the same Code → Reason → Math order in all 10 subsets examined. This supports using a single subset for order selection and retaining the ten-subset analysis as an assessment of ordering stability.

Here, a “single probe” comprises three short anchor-task training runs, three memory extractions, and six directional target-task branch training runs, rather than six full model training runs. Each training segment uses 60 examples and performs 20 update steps. A complete order-selection probe therefore uses 180 unique examples, processes 540 example instances, and performs 180 update steps, without requiring task-performance evaluation of six fully trained models.

We compare the computational costs of probing and standard full training under the same supplementary Qwen2.5-7B configuration:

ProcedureCumulative compute time (min)Allocated GPU-hours (GPU·h)
One standard full training run125.264.175
One full training run for each of the six orders751.5625.052
One single-subset conflict probe37.751.258
Complete ten-subset conflict probing377.5212.584
One single-subset probe + one full training run with the selected order163.015.434

During order selection, a single-subset probe reduces the estimated cost by 94.98% relative to six full training runs, saving approximately 713.81 minutes of cumulative compute time and 23.794 GPU-hours. Even when the subsequent full training run is included, “one probe + one full training run” still saves 78.31% relative to the training portion of exhaustive evaluation of all six orders.

3. Detailed construction of the probe memory MaM_a

MaM_a is the parameter-update subspace induced by a short training run on anchor task aa. For each anchor task aa, we start from the same base model and perform OPD without projection on that task’s 60 probe examples. For each protected two-dimensional parameter matrix WW, we record its displacement before and after training and apply truncated SVD:

\qquad \Delta W_a\approx U_{a,K}\Sigma_{a,K}V_{a,K}^{\top}, \qquad M_{a,W}=\operatorname{orth}(V_{a,K}),\quad K=16.$$ Thus, $M_a$ denotes the collection of per-matrix memory bases $\{M_{a,W}\}_W$; the paper omits the matrix index in its single-matrix expressions. For each target task $b\ne a$, we then independently initialize a probe branch from the same model obtained after the short training run on task $a$. We hold $M_a$ fixed and, at each target-task training step, compute the fraction of the pre-projection gradient $G_{b,t,W}$ removed by projection onto the memory. We average the two directions for each task pair and then average over the other tasks: $$c(a,b)=\frac{r(a\!\to\!b)+r(b\!\to\!a)}{2}, \qquad c(a)=\frac{1}{|\mathcal T|-1}\sum_{b\ne a}c(a,b).$$ The complete ten-subset analysis further averages $c(a)$ across subsets and sorts tasks in ascending order of the resulting scores. In this way, **memory construction, conflict measurement, and task ordering all rely on training data and the update geometry they induce**.
AuthorsOpenReview ↗

Response to Reviewer G16F — Question 3 (Part 1 of 1)

Response to Reviewer G16F — Question 3 (Part 1 of 1)

We thank the reviewer for identifying this important missing control. We agree that, according to Equation (12), optimizer-update projection alone is sufficient to make the final update orthogonal to the protected memory subspace. We have therefore added an optimizer-update-only projection experiment, keeping the same Code→Reason→Math order, four-cycle schedule, and training budget. The results are shown below.

Gradient projectionOptimizer-update projectionMathReasonCodeAvg.
No\mathrm{No}No\mathrm{No}67.3368.5555.6763.85
Yes\mathrm{Yes}No\mathrm{No}67.1173.8757.6766.22
No\mathrm{No}Yes\mathrm{Yes}64.4475.5954.6764.90
Yes\mathrm{Yes}Yes\mathrm{Yes}67.7873.4659.6766.97

The key distinction is that satisfying the same orthogonality constraint on the final update does not make the two optimization processes equivalent. In the optimizer-update-only variant, Adafactor receives unprojected gradients, so gradient components within the protected subspace can still influence its adaptive second-moment statistics and the preconditioned update before the final projection. Dual projection removes these components before gradient clipping and adaptive preconditioning. Consequently, even though the final updates in both variants satisfy the orthogonality constraint, their retained updates and subsequent optimizer states can differ.

The additional experiment supports the practical value of this distinction: optimizer-update-only projection achieves an average score of 64.90, whereas dual projection achieves 66.97, an improvement of 2.07 percentage points. Dual projection achieves the highest average score and performs better on Math and Code. We will include this comparison in the revised manuscript.

Reviewer_PB51OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PMOPD, a framework for multi-teacher on-policy distillation that aims to reduce interference between tasks. It extracts low-rank subspaces from cumulative parameter updates and projects both gradients and optimizer updates away from previously protected subspaces. The method also introduces conflict-based task ordering and cyclic training with periodically refreshed subspace memories. Experiments on Qwen2.5-7B and Llama-3.1-8B show improvements over standard MOPD.

Critical strengths

The paper addresses a relevant problem. Distilling multiple teacher capabilities into a single student can lead to interference between tasks, and reducing this interference is practically important.

Projecting both gradients and actual optimizer updates is a reasonable design. Adaptive optimizers can change update directions after gradient projection, so the second projection has a clear motivation.

The paper investigates task ordering and cyclic training in addition to subspace protection. The ablation results provide some evidence that these components can improve performance in the evaluated settings.

Critical weaknesses

Subspace overlap does not necessarily indicate task conflict. The method uses the overlap between a new task's gradient and previously learned parameter-update subspaces to estimate interference. However, overlapping directions can also represent shared useful knowledge or positive transfer. For example, two tasks may benefit from updating the same parameter direction. Projecting out this shared component could prevent useful learning rather than reduce interference. The paper does not sufficiently distinguish harmful interference from beneficial overlap, which weakens the motivation for its central projection mechanism.

The choice of cumulative parameter updates as protected directions is insufficiently justified. The method extracts principal directions from the parameter displacement after each task block. However, large parameter displacements do not necessarily correspond to directions that are important for preserving task performance. Some updates may have little functional impact, while smaller updates may be critical. The authors should demonstrate that the selected subspaces are genuinely important for retaining previously learned capabilities, for example by comparing performance degradation under perturbations along protected directions and matched random directions.

The contributions of task ordering and cycling are not fully isolated. The paper selects a task order based on conflict scores and then combines it with cyclic training. However, the best ordering under a single cycle is not necessarily optimal under multiple cycles, where the model state and estimated subspaces change over time. More controlled experiments combining different task orders and cycle counts are needed to establish the independent contributions of these components.

The task-ordering criterion may discard important directional information. The method averages the directional conflict scores between task pairs to obtain a symmetric score for ordering. However, interference can be asymmetric: task A may strongly affect task B, while task B has little effect on task A. Averaging the two directions may obscure this difference. The paper should justify this symmetrization and compare it with ordering strategies that retain directional information.

The experimental evaluation is limited. The experiments focus on three tasks: Math, Code, and Reason. Although results on two model families are encouraging, this setting is too small to establish the scalability of the proposed task-ordering strategy. Comparisons with established gradient-conflict methods, such as PCGrad, and with activation-based or random subspace protection would also help clarify the method's advantages.

Questions for authors

How do the authors distinguish harmful task interference from beneficial sharing when both produce overlapping update directions? Can they provide experiments showing that the removed gradient components are actually harmful to previously learned tasks?

Why are the principal directions of cumulative parameter displacement appropriate proxies for task-critical knowledge? How do they compare with subspaces constructed from task gradients, input activations, or loss sensitivity?

Can the authors evaluate different task orderings and cycle counts jointly, rather than selecting the ordering based on single-cycle results and then using it in the multi-cycle setting?

Why is a symmetric conflict score used when the directional interference between two tasks may be asymmetric? Would a direction-aware ordering strategy perform better?

Can the authors compare PMOPD with PCGrad and other gradient-conflict reduction methods under the same training budget? This would help establish whether the improvements come from the proposed subspace memory or from interference reduction more generally.

What is the computational and memory overhead of maintaining and updating the protected subspaces, particularly as the number of tasks increases? Does periodically resetting the protection memory risk forgetting important directions from earlier cycles?

8 comments
AuthorsOpenReview ↗

Response to Reviewer PB51 — Question 1 (Part 1 of 2)

Response to Reviewer PB51 — Question 1 (Part 1 of 2)

Thank you for highlighting this important distinction. We agree that subspace overlap does not necessarily imply destructive interference. Overlap describes a geometric relationship between an update and the protected subspace, whereas destructive interference concerns the update's negative effect on previously learned tasks and therefore requires functional evidence. To examine this directly, we conducted paired interventions on actual optimizer updates, comparing the effects of retaining and removing the overlapping component. In 104 out of 144 paired comparisons (72.2%), retaining this component yielded higher macro-averaged old-task RKL than removing it. Removing the component reduced old-task RKL by 5.88×10−45.88\times10^{-4} on average. Thus, in the majority of the tested updates, the overlapping component had a harmful marginal effect relative to its removal, providing direct empirical support for protecting previous tasks through uniform projection.

1. Directly testing the effect of retaining versus removing the overlapping component

We evaluated Qwen2.5-7B with three training seeds at six predefined task boundaries: Code → Reason and Reason → Math in cycles 1, 2, and 4. At each checkpoint, we constructed eight candidate updates, giving 3×6×8=1443\times6\times8=144 paired comparisons.

For an actual Adafactor parameter update DD computed without gradient projection, we decomposed it using the memory subspace basis MM:

D_{\perp}=D-D_{\parallel}.$$ Starting from identical model parameters, optimizer states, memory, and new-task training batches, we compared the full update $\theta+D$ with $\theta+D_{\perp}$, which removes the entire component within the memory subspace. All interventions used the actual update scale, and each candidate update was evaluated afresh from its corresponding checkpoint. Both branches used the same diagnostic prompts and fixed generated prefixes to measure full-vocabulary reverse KL divergence (RKL) between the student and teacher distributions on previous tasks. We defined the marginal harm of retaining the overlapping component as $$H=L_{\mathrm{old}}(\theta+D)-L_{\mathrm{old}}(\theta+D_{\perp}).$$ Here, $H>0$ means that retaining the component is less favorable for matching the old-task teacher distribution than removing it from the same candidate update. | Metric | Paired-intervention result | |---|---:| | Retaining the overlapping component yields higher old-task RKL ($H>0$) | **104/144 (72.2%)** | | Mean reduction in old-task RKL after removal | **$5.88\times10^{-4}$** | | Conditional bootstrap 95% interval for the mean reduction | **$[3.13\times10^{-4},\ 8.36\times10^{-4}]$** | **Removal is supported not only by the majority of comparisons, but also by a positive mean benefit across all 144 pairs.** This average includes both favorable and unfavorable cases, directly measuring the local net effect of applying the removal rule without screening individual updates. The interval uses hierarchical resampling over training seeds and candidate updates within checkpoints, conditional on the fixed diagnostic prompts and prefixes. ### 2. Why remove the entire overlapping component? **The rationale for uniform projection is its protective effect in the majority of paired comparisons and its positive average benefit overall.** PMOPD extracts low-dimensional principal directions from the cumulative parameter displacements of completed tasks and protects these directions during subsequent training. Projection removes the entire component of each update within this memory subspace while preserving the orthogonal component for learning new tasks. **Our objective is to conservatively protect the principal update directions of previously learned tasks, rather than to identify and precisely remove every harmful component at each training step.** Using the paired diagnostic above as an online screening procedure would require additional old-task evaluations under the current model state to compare retaining and removing candidate components. Distinguishing the effects of individual components would require still finer-grained interventions. Repeating such identification throughout training would introduce substantial additional computational cost. Uniform projection instead applies the protection constraint directly using the already constructed memory subspace, without repeating old-task intervention evaluations at every step, while leaving the orthogonal space available for new-task learning. Here, “conservative” means prioritizing protection of previously learned update directions, not assuming that every overlapping component is harmful.
AuthorsOpenReview ↗

Response to Reviewer PB51 — Question 1 (Part 2 of 2)

Response to Reviewer PB51 — Question 1 (Part 2 of 2)

The intervention experiment directly evaluates this protection rule: retaining the overlapping component has a harmful marginal effect in 72.2% of pairs, and the mean result across all pairs also favors removal. The justification for removing the entire in-memory component therefore rests on its measured average protective benefit, rather than precise component-wise identification of harm. This supports a computationally simple and empirically grounded protection strategy.

The 72.2% result concerns updates computed without gradient projection and measures the old-task RKL difference between the full and component-removed updates. It is not an accuracy statistic or a substitute for separately evaluating the incremental effect of optimizer-update projection after gradient projection.

3. Full-training ablations provide complementary evidence

Table 4 of the paper compares projection settings under the same task order, four-cycle schedule, and training budget:

SettingAverage score across three tasks
No projection63.85
Gradient projection66.22
Gradient and optimizer-update projection66.97

The full projection strategy improves the average score by 3.12 percentage points over no projection. These results complement the local interventions: the latter show that retaining the overlapping component increases old-task diagnostic loss in most candidate updates and that uniform removal provides an average protective benefit; the full-training ablations show that subspace protection improves final aggregate task performance. The local interventions use the supplementary FKL protocol, whereas the full-training ablations above use the paper's main RKL protocol; we report these as separate, complementary sources of evidence.

In summary, the harmful marginal effect observed in 72.2% of paired comparisons, the positive mean benefit of removal across all pairs, and the full-training performance gains jointly support removing the entire update component within the memory subspace. This rationale is based on measured protective benefits, not on equating geometric overlap with destructive interference. In the revision, we will correct the relevant terminology and explicitly distinguish “subspace overlap” from “destructive interference”: the former describes a geometric relationship, while the latter describes a negative effect established through task-relevant measurements.

AuthorsOpenReview ↗

Response to Reviewer PB51 — Question 2 (Part 1 of 1)

Response to Reviewer PB51 — Question 2 (Part 1 of 1)

We thank the reviewer for suggesting matched perturbations to test the functional importance of the protected subspaces. We have added this experiment: after controlling for perturbation rank, singular value spectrum, and per-matrix norm, perturbations along the learned protected subspaces produced a larger increase in old-task reverse KL divergence (RKL) than perturbations along random subspaces in all 54 paired comparisons. The mean increase was 32.7 times that for random directions. This result further connects the paper’s update geometry analysis to the model’s output behavior on previously learned tasks.

Why use cumulative-update subspaces? PMOPD extracts the top K=16K=16 right singular vectors of the realized task-block displacement, ΔWt=Wafter−Wbefore\Delta W_t=W_{\mathrm{after}}-W_{\mathrm{before}}, and incorporates them into an orthonormal subspace memory basis MM (Sections 4.3–4.4). Unlike a single mini-batch gradient, this displacement records the net change actually applied to the model during task training after gradient clipping and adaptive optimization. Section 3.2 also shows that these dominant directions emerge early in training and exhibit a structure that separates tasks. These properties provide a geometric rationale for choosing a compact task representation. The additional perturbation experiment tests the connection between these directions and old-task behavior, rather than directly equating displacement magnitude with importance for retaining capabilities.

How are the perturbations rigorously matched? For each protected weight matrix, we use the same Gaussian coefficient matrix AA to construct a protected direction ZM=AM⊤Z_M=AM^\top and a random direction ZR=AQ⊤Z_R=AQ^\top, where QQ is a random orthonormal basis with the same dimensions as MM. The two perturbations have the same rank and singular value spectrum, and each is normalized per matrix to satisfy ∥Z∥F=10−3∥W∥F\|Z\|_F=10^{-3}\|W\|_F. We also include a matched random subspace within the orthogonal complement of MM. All branches start from the same checkpoint, and we average the results obtained after applying +Z+Z and −Z-Z separately. The comparison therefore controls for perturbation magnitude, spectral structure, and sign choice, isolating differences in subspace orientation.

The experiment uses the supplementary diagnostic setting on Qwen2.5-7B with the Code → Reason → Math order and four cycles. It covers 18 checkpoints: two task boundaries in each of cycles 1, 2, and 4, across three training seeds, with three perturbation-direction seeds per checkpoint. We measure KL(πθ∥πT)\mathrm{KL}(\pi_\theta\|\pi_T) on the same old-task examples and fixed generation prefixes. After the Code block, we evaluate Code; after the Reason block, we average Code and Reason with equal weights. We define the sign-averaged RKL increase as

D(Z)=Lold(θ+Z)+Lold(θ−Z)2−Lold(θ).D(Z)=\frac{L_{\mathrm{old}}(\theta+Z)+L_{\mathrm{old}}(\theta-Z)}{2}-L_{\mathrm{old}}(\theta).

Perturbation subspaceMean old-task RKL increase DD
Learned protected subspace6.404×10−56.404\times10^{-5}
Rank- and norm-matched random subspace1.957×10−61.957\times10^{-6}
Rank- and norm-matched random subspace in the orthogonal complement1.654×10−61.654\times10^{-6}

The difference between protected and random directions is positive in all 54/54 paired comparisons, including 18/18 for each of the three training seeds. The comparison with random directions in the orthogonal complement likewise yields a positive difference in all 54/54 pairs. These 54 pairs arise from the checkpoints and direction samples across the three training seeds described above; the factor of 32.7 is the ratio of the two mean RKL increases reported in the table. Thus, the specific conclusion supported by these perturbation results is that the subspaces extracted from cumulative updates exhibit consistently greater local sensitivity with respect to matching teacher distributions on previously learned tasks, providing direct functional evidence for choosing them as protected directions.

Reviewer_Bq8EOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses task interference in multi-teacher on-policy distillation. It constructs low-rank subspaces from task-specific parameter displacements and protects learned directions through projections of both gradients and optimizer updates, combined with task ordering and cycling.

Critical strengths
  1. Interference during multi-teacher capability integration directly affects the resulting student. The paper addresses it through a mechanism integrated into training, rather than only comparing final merged models. 2.The paper explains that adaptive preconditioning can reintroduce components in protected subspaces after gradient projection, motivating a second projection of the actual update. This design is consistent with the use of Adafactor.
Critical weaknesses
  1. Equations (11) and (13) measure the norm of projected components without distinguishing beneficial sharing from destructive updates. The local linear invariance in Equation (10) also does not directly guarantee retention of previous-task performance in the full network. The current evidence does not justify interpreting all removed components as conflicts.
  2. The preferred order and four-cycle configuration in Section 7 come from the same three-task setting, while cycle consistency requires observing training trajectories under different schedules. Validation with diagnostic rules frozen in advance on independent task combinations, together with an accounting of diagnostic costs, is needed to establish the practicality of low-cost selection before training.
  3. The paper states that PMOPD results average three runs but provides no standard deviations or paired intervals, and the baseline repetition protocol is unclear. In particular, the 0.82-point improvement on Llama's Reason task requires uncertainty estimates to assess its reliability.
  4. Each task uses only 600 OPD prompts, and the teachers are obtained through domain-specific RL from a shared base. The experiments do not include the PCGrad/GPM-style projection controls discussed in the paper, rank-K sensitivity, complete optimization hyperparameters, or additional memory and runtime measurements. These omissions limit assessment of the method's advantages and scalability.
Questions for authors

see in the weakness

9 comments
AuthorsOpenReview ↗

Response to Reviewer Bq8E — Question 1 (Part 1 of 2)

Response to Reviewer Bq8E — Question 1 (Part 1 of 2)

Thank you for highlighting this important distinction. We agree that subspace overlap does not necessarily imply destructive interference. Overlap describes a geometric relationship between an update and the protected subspace, whereas destructive interference concerns the update's negative effect on previously learned tasks and therefore requires functional evidence. To examine this directly, we conducted paired interventions on actual optimizer updates, comparing the effects of retaining and removing the overlapping component. In 104 out of 144 paired comparisons (72.2%), retaining this component yielded higher macro-averaged old-task RKL than removing it. Removing the component reduced old-task RKL by 5.88×10−45.88\times10^{-4} on average. Thus, in the majority of the tested updates, the overlapping component had a harmful marginal effect relative to its removal, providing direct empirical support for protecting previous tasks through uniform projection.

1. Directly testing the effect of retaining versus removing the overlapping component

We evaluated Qwen2.5-7B with three training seeds at six predefined task boundaries: Code → Reason and Reason → Math in cycles 1, 2, and 4. At each checkpoint, we constructed eight candidate updates, giving 3×6×8=1443\times6\times8=144 paired comparisons.

For an actual Adafactor parameter update DD computed without gradient projection, we decomposed it using the memory subspace basis MM:

D_{\perp}=D-D_{\parallel}.$$ Starting from identical model parameters, optimizer states, memory, and new-task training batches, we compared the full update $\theta+D$ with $\theta+D_{\perp}$, which removes the entire component within the memory subspace. All interventions used the actual update scale, and each candidate update was evaluated afresh from its corresponding checkpoint. Both branches used the same diagnostic prompts and fixed generated prefixes to measure full-vocabulary reverse KL divergence (RKL) between the student and teacher distributions on previous tasks. We defined the marginal harm of retaining the overlapping component as $$H=L_{\mathrm{old}}(\theta+D)-L_{\mathrm{old}}(\theta+D_{\perp}).$$ Here, $H>0$ means that retaining the component is less favorable for matching the old-task teacher distribution than removing it from the same candidate update. | Metric | Paired-intervention result | |---|---:| | Retaining the overlapping component yields higher old-task RKL ($H>0$) | **104/144 (72.2%)** | | Mean reduction in old-task RKL after removal | **$5.88\times10^{-4}$** | | Conditional bootstrap 95% interval for the mean reduction | **$[3.13\times10^{-4},\ 8.36\times10^{-4}]$** | **Removal is supported not only by the majority of comparisons, but also by a positive mean benefit across all 144 pairs.** This average includes both favorable and unfavorable cases, directly measuring the local net effect of applying the removal rule without screening individual updates. The interval uses hierarchical resampling over training seeds and candidate updates within checkpoints, conditional on the fixed diagnostic prompts and prefixes. ### 2. Why remove the entire overlapping component? **The rationale for uniform projection is its protective effect in the majority of paired comparisons and its positive average benefit overall.** PMOPD extracts low-dimensional principal directions from the cumulative parameter displacements of completed tasks and protects these directions during subsequent training. Projection removes the entire component of each update within this memory subspace while preserving the orthogonal component for learning new tasks. **Our objective is to conservatively protect the principal update directions of previously learned tasks, rather than to identify and precisely remove every harmful component at each training step.** Using the paired diagnostic above as an online screening procedure would require additional old-task evaluations under the current model state to compare retaining and removing candidate components. Distinguishing the effects of individual components would require still finer-grained interventions. Repeating such identification throughout training would introduce substantial additional computational cost. Uniform projection instead applies the protection constraint directly using the already constructed memory subspace, without repeating old-task intervention evaluations at every step, while leaving the orthogonal space available for new-task learning. Here, “conservative” means prioritizing protection of previously learned update directions, not assuming that every overlapping component is harmful.
AuthorsOpenReview ↗

Response to Reviewer Bq8E — Question 1 (Part 2 of 2)

Response to Reviewer Bq8E — Question 1 (Part 2 of 2)

The intervention experiment directly evaluates this protection rule: retaining the overlapping component has a harmful marginal effect in 72.2% of pairs, and the mean result across all pairs also favors removal. The justification for removing the entire in-memory component therefore rests on its measured average protective benefit, rather than precise component-wise identification of harm. This supports a computationally simple and empirically grounded protection strategy.

The 72.2% result concerns updates computed without gradient projection and measures the old-task RKL difference between the full and component-removed updates. It is not an accuracy statistic or a substitute for separately evaluating the incremental effect of optimizer-update projection after gradient projection.

3. Full-training ablations provide complementary evidence

Table 4 of the paper compares projection settings under the same task order, four-cycle schedule, and training budget:

SettingAverage score across three tasks
No projection63.85
Gradient projection66.22
Gradient and optimizer-update projection66.97

The full projection strategy improves the average score by 3.12 percentage points over no projection. These results complement the local interventions: the latter show that retaining the overlapping component increases old-task diagnostic loss in most candidate updates and that uniform removal provides an average protective benefit; the full-training ablations show that subspace protection improves final aggregate task performance. The local interventions use the supplementary FKL protocol, whereas the full-training ablations above use the paper's main RKL protocol; we report these as separate, complementary sources of evidence.

In summary, the harmful marginal effect observed in 72.2% of paired comparisons, the positive mean benefit of removal across all pairs, and the full-training performance gains jointly support removing the entire update component within the memory subspace. This rationale is based on measured protective benefits, not on equating geometric overlap with destructive interference. In the revision, we will correct the relevant terminology and explicitly distinguish “subspace overlap” from “destructive interference”: the former describes a geometric relationship, while the latter describes a negative effect established through task-relevant measurements.

AuthorsOpenReview ↗

Response to Reviewer Bq8E — Question 2 (Part 1 of 1)

Response to Reviewer Bq8E — Question 2 (Part 1 of 1)

We thank the reviewer for this important suggestion. We agree that it is necessary to distinguish the diagnostic’s effectiveness in the current setting, the transfer of a fixed configuration across models, and prospective validation on independent task combinations. We will clarify these issues in the following three respects.

First, regarding diagnostic costs and the choice of cycle count, the full cost of the task-ordering diagnostic includes not only computing overlap scores but also constructing task-specific probe memories, generating student trajectories, querying teachers, computing gradients, and performing singular value decomposition (SVD). In the revised manuscript, we will clarify how the probe memories are constructed and report these costs separately from the cost of a complete PMOPD training run, rather than using the low overhead of score computation alone to characterize the entire diagnostic procedure.

For cycle count, we agree with the reviewer’s assessment: the cross-cycle subspace consistency in the current Section 7.2 is computed from the training trajectories of candidate configurations, so these results alone do not establish low-cost selection before training. In principle, consistency estimation does not require the final training checkpoint and can be performed once the same task has been visited at least twice. However, this still requires partial training, whose cost depends on the task-block size and candidate cycle count. One possible direction for further investigation is to screen candidate configurations based on the number of examples per block and examine subspace consistency between consecutive visits in a short pilot run. We have not yet validated whether such early estimates can reliably predict the best cycle count for full training. We will therefore clearly distinguish this potential approach from the post hoc analysis already presented in the paper and narrow the corresponding claims.

Second, the existing results provide some evidence that a fixed configuration transfers across models. After selecting C→R→M and four cycles in the Qwen2.5-7B setting, we applied the same configuration directly to Llama-3.1-8B, without searching again for the task order or cycle count on Llama. This configuration also achieved the largest performance gain over MOPD on Llama. This supports the transferability of the fixed configuration across model families.

Finally, we agree on the need for additional validation on independent task combinations. In the paper’s controlled MOPD setting, where teachers are initialized from a shared base model, introducing new task combinations requires first training the corresponding domain-specialized RL teachers, rather than simply changing the evaluation data. As shown in Table 2, domain-specific RL training for the existing teachers involves thousands of examples; the subsequent on-policy distillation also requires autoregressive trajectory generation and teacher supervision. This validation therefore entails new teacher training and a complete distillation pipeline, rather than merely an additional evaluation. Given the short rebuttal period and currently available GPU resources, we cannot complete this full experiment within the present response period. In subsequent revisions, we plan to add validation on more datasets and independent task combinations, with the probe budget, scoring rules, and selection procedure fixed in advance, and to report both diagnostic costs and the gap between the selected configuration and the search-based reference results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.