- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
PMOPD aims to reduce interference between Math, Reason, and Code teachers during on-policy distillation. It trains tasks in ordered blocks and protects subspaces learned from earlier blocks by projecting both gradients and optimizer updates. It also proposes diagnostics for choosing the order and cycle count, and improves every evaluated task over MOPD on Qwen2.5-7B and Llama-3.1-8B.
Critical strengths
- Looking at cumulative parameter changes rather than individual mini-batch gradients is an interesting way to study task interference.
- The method is simple. The observation that Adafactor can reintroduce protected components after gradient projection is useful, and Table 4 supports using both projections in the tested configurations.
- Gains over MOPD are consistent across tasks and two model families, with average improvements of 2.54 and 2.09 points. The order and cycle sweeps also help show how sensitive the method is to these choices.
Critical weaknesses
-
The evaluation is limited to a few task-specific test sets, with no uncertainty estimates. This makes smaller gains, such as +0.82 on Llama Reason, hard to assess. The results are encouraging, but broader evaluation and results across seeds are needed to support the transferability claim.
-
I could not tell how the task order and cycle count were selected. No validation split is identified, and the chosen settings are also the best-scoring entries in Table 5 and Figure 4. This does not establish test-set tuning, but the selection procedure needs clarification. There is also a cost question: the cycle-consistency measure seems to require training each candidate schedule, despite being presented as guidance before a full run. How the probe memory is obtained is unclear as well.
-
Table 4 is missing a particularly useful control: projecting only the optimizer update. By Eq. 12, that projection already makes the update orthogonal to the memory. What does projecting the gradient as well add? A suitably adapted GPM-style baseline would also help distinguish the displacement-based memory from existing projection methods.
-
Protecting directions in which a task previously changed the model is plausible, but the paper does not directly show why this preserves that task's capability. Memory is also reset each cycle, removing the previous cycle's explicit constraints. Performance traces across cycle boundaries would help show whether earlier tasks degrade during the newly unconstrained first block.
Questions for authors
- Which data were used to select the order and cycle count? Were these settings fixed before inspecting test scores?
- How is constructed, and can the diagnostics choose a cycle count without fully training each candidate schedule?
- How does PMOPD compare with projecting only the optimizer update?
- How stable are the gains across seeds, and what happens to earlier-task performance when the memory is reset?
8 comments
Response to Reviewer G16F — Question 2 (Part 1 of 1)
Response to Reviewer G16F — Question 2 (Part 1 of 1)
We thank the reviewer for highlighting the need to clarify configuration selection, computational costs, and the construction of probe memories. We address these points below.
1. Data separation and the basis for selection
Our training and test data are separate, and the test set is not used for model training. Both the full OPD training runs and the short training runs used for task-order probing use training data, while the test set is used for capability evaluation. Task-order probing uses subsets of the OPD training data. Each task has 600 OPD training examples.
2. Cost comparison between task-order probing and exhaustive evaluation of all six orders
Instead of fully training and evaluating all six task orders, we can obtain an order directly from a single small-sample conflict probe. Probing with 60 training examples per task recovered the same Code → Reason → Math order in all 10 subsets examined. This supports using a single subset for order selection and retaining the ten-subset analysis as an assessment of ordering stability.
Here, a “single probe” comprises three short anchor-task training runs, three memory extractions, and six directional target-task branch training runs, rather than six full model training runs. Each training segment uses 60 examples and performs 20 update steps. A complete order-selection probe therefore uses 180 unique examples, processes 540 example instances, and performs 180 update steps, without requiring task-performance evaluation of six fully trained models.
We compare the computational costs of probing and standard full training under the same supplementary Qwen2.5-7B configuration:
| Procedure | Cumulative compute time (min) | Allocated GPU-hours (GPU·h) |
|---|---|---|
| One standard full training run | 125.26 | 4.175 |
| One full training run for each of the six orders | 751.56 | 25.052 |
| One single-subset conflict probe | 37.75 | 1.258 |
| Complete ten-subset conflict probing | 377.52 | 12.584 |
| One single-subset probe + one full training run with the selected order | 163.01 | 5.434 |
During order selection, a single-subset probe reduces the estimated cost by 94.98% relative to six full training runs, saving approximately 713.81 minutes of cumulative compute time and 23.794 GPU-hours. Even when the subsequent full training run is included, “one probe + one full training run” still saves 78.31% relative to the training portion of exhaustive evaluation of all six orders.
3. Detailed construction of the probe memory
is the parameter-update subspace induced by a short training run on anchor task . For each anchor task , we start from the same base model and perform OPD without projection on that task’s 60 probe examples. For each protected two-dimensional parameter matrix , we record its displacement before and after training and apply truncated SVD:
\qquad \Delta W_a\approx U_{a,K}\Sigma_{a,K}V_{a,K}^{\top}, \qquad M_{a,W}=\operatorname{orth}(V_{a,K}),\quad K=16.$$ Thus, $M_a$ denotes the collection of per-matrix memory bases $\{M_{a,W}\}_W$; the paper omits the matrix index in its single-matrix expressions. For each target task $b\ne a$, we then independently initialize a probe branch from the same model obtained after the short training run on task $a$. We hold $M_a$ fixed and, at each target-task training step, compute the fraction of the pre-projection gradient $G_{b,t,W}$ removed by projection onto the memory. We average the two directions for each task pair and then average over the other tasks: $$c(a,b)=\frac{r(a\!\to\!b)+r(b\!\to\!a)}{2}, \qquad c(a)=\frac{1}{|\mathcal T|-1}\sum_{b\ne a}c(a,b).$$ The complete ten-subset analysis further averages $c(a)$ across subsets and sorts tasks in ascending order of the resulting scores. In this way, **memory construction, conflict measurement, and task ordering all rely on training data and the update geometry they induce**.Response to Reviewer G16F — Question 3 (Part 1 of 1)
Response to Reviewer G16F — Question 3 (Part 1 of 1)
We thank the reviewer for identifying this important missing control. We agree that, according to Equation (12), optimizer-update projection alone is sufficient to make the final update orthogonal to the protected memory subspace. We have therefore added an optimizer-update-only projection experiment, keeping the same Code→Reason→Math order, four-cycle schedule, and training budget. The results are shown below.
| Gradient projection | Optimizer-update projection | Math | Reason | Code | Avg. |
|---|---|---|---|---|---|
| 67.33 | 68.55 | 55.67 | 63.85 | ||
| 67.11 | 73.87 | 57.67 | 66.22 | ||
| 64.44 | 75.59 | 54.67 | 64.90 | ||
| 67.78 | 73.46 | 59.67 | 66.97 |
The key distinction is that satisfying the same orthogonality constraint on the final update does not make the two optimization processes equivalent. In the optimizer-update-only variant, Adafactor receives unprojected gradients, so gradient components within the protected subspace can still influence its adaptive second-moment statistics and the preconditioned update before the final projection. Dual projection removes these components before gradient clipping and adaptive preconditioning. Consequently, even though the final updates in both variants satisfy the orthogonality constraint, their retained updates and subsequent optimizer states can differ.
The additional experiment supports the practical value of this distinction: optimizer-update-only projection achieves an average score of 64.90, whereas dual projection achieves 66.97, an improvement of 2.07 percentage points. Dual projection achieves the highest average score and performs better on Math and Code. We will include this comparison in the revised manuscript.
Response to Reviewer G16F — Question 1 (Part 1 of 1)
Response to Reviewer G16F — Question 1 (Part 1 of 1)
We thank the reviewer for pointing out the missing uncertainty estimates. We have now repeated MOPD using the same three random seeds as PMOPD and updated the results to consistently report the mean ± standard deviation over three runs for both methods, as shown below.
Qwen2.5-7B
Llama-3.1-8B
For Llama Reason, which the reviewer specifically highlighted, the updated results are 69.45 ± 0.34 for PMOPD and 67.85 ± 0.70 for MOPD. The improvement computed from the three-run means of both methods is therefore 1.60 percentage points, rather than the previously reported 0.82 percentage points. This improvement exceeds the observed standard deviation of either method, providing further support for robustness across training seeds in these runs.
The updated average improvements are 2.74 percentage points on Qwen2.5-7B and 2.59 percentage points on Llama-3.1-8B. We will include uncertainty estimates in the revised results table, clarify the repetition protocol, and update the corresponding numerical comparisons throughout the text.