acceptodds
Under review as a conference paper at ICLR 2027

Transferable Guidance with Constrained Allocation for On-Policy Self-Distillation

Abstract

On-policy self-distillation uses training-only privileged information to supervise a model on its own responses. Its effectiveness depends on both the supervision targets and the allocation of training weight across response positions. We propose TransCAP to coordinate these choices through a shared task-conditioned residual. The residual contrasts task-plus-privilege predictions with a privilege-only reference, reweighting a task-only anchor and defining initial position priorities. A constrained KL projection then adjusts these priorities using local directional conflict at the current student distribution. The projection bounds measured directional conflict and prevents the allocation from becoming more concentrated than its initial form. On Qwen3.5-9B, TransCAP achieves reasoning and code averages of 80.65% and 72.22%, leading the strongest baselines by 6.58 and 4.32 percentage points. It also achieves state-of-the-art performance on Qwen3-4B and Qwen3.5-27B across both task groups.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy Self-distillation, Constrained Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dqgSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TransCAP, a framework for on-policy self-distillation (OPSD) that jointly constructs supervision targets and allocates token-level training weights. The method uses a task-conditioned residual to refine privileged teacher supervision and determine initial token priorities. A constrained KL projection then adjusts these weights to limit directional conflict with the task-only anchor and prevent excessive concentration. Experiments across three Qwen model scales demonstrate improvements over existing OPSD baselines on mathematical reasoning, code generation, and function calling benchmarks.

Critical strengths

The paper addresses an interesting and important problem of jointly determining what knowledge to transfer and where to allocate supervision in OPSD.

TransCAP presents a technically interesting framework that combines task-conditioned guidance construction with constrained supervision allocation.

Experiments across multiple model scales and benchmarks demonstrate promising improvements over several recent OPSD methods.

Extensive ablation studies provide useful evidence for the contributions of guidance-based weighting, conflict control, and concentration constraints.

Critical weaknesses

The proposed guidance score is based on the magnitude of task-conditioned distributional differences, but larger differences do not necessarily indicate more transferable knowledge or greater learning benefit. The paper lacks sufficient evidence that this score reliably identifies the most informative token positions.

The directional conflict metric is defined in the output distribution space rather than the model's parameter-gradient space. Reducing this local conflict does not necessarily prevent interference with previously learned capabilities. The connection between the proposed geometric constraints and actual capability preservation remains insufficiently established.

The target construction largely builds upon existing privileged-supervision refinement methods, while the main contribution lies in introducing additional weighting and projection mechanisms. Although the empirical improvements are promising, it remains unclear whether these components provide a sufficiently generalizable insight beyond more sophisticated supervision reweighting.

TransCAP requires evaluating multiple conditional distributions and solving a constrained projection problem during training. While the paper reports projection latency and training-stage timings, a comprehensive end-to-end computational comparison with competing OPSD methods is missing. Additionally, some improvements, particularly on Qwen3-4B, are relatively small and lack uncertainty estimates across independent training runs.

Questions for authors

How can the authors demonstrate that the proposed residual magnitude measures transferable knowledge rather than merely differences in model predictions or conditioning preferences?

To what extent does reducing distribution-space directional conflict translate into better preservation of existing capabilities? Can the authors provide more direct evidence of reduced interference or forgetting?

7 comments
AuthorsOpenReview ↗

Additional Experiments for Main Concern 2: Do Higher-Residual Positions Provide More Useful Supervision?

Thank you for distinguishing two questions. The permutation experiment shows that assigning residual-derived weights to their original positions improves performance. Your question goes further: given the same supervision budget, does a student learn more from high-residual positions, as measured by performance on unseen problems? We added a direct comparison on mathematics and code.

1. Learning from high- versus low-residual positions

For each domain, we trained two students from the same 9B checkpoint on the same training trajectories. Within each training example and relative-position bin, we ranked positions by residual magnitude and selected equal numbers from the high and low ends. Both students saw the full response context and used the same target-construction rule. Each student received supervision only at its selected positions, with equal weight per selected token. We matched the number of supervised tokens and training steps and disabled projection in both conditions.

Held-out benchmarkStarting model (%)Low-residual group (%)High-residual group (%)High − low (pp)
AIME 2025 (Avg@12)63.0675.2880.00+4.72
LiveCodeBench v665.6068.5371.33+2.80

Both groups improved over the starting model, so low-residual positions also provided useful supervision. On AIME 2025, the high-residual student produced 288 correct responses out of 360, compared with 271 out of 360 for the low-residual student. On LiveCodeBench v6, the high-residual student gained 5.73 percentage points over the starting model, while the low-residual student gained 2.93 points. In both domains, spending the same supervision budget on high-residual positions yielded a larger gain on unseen problems.

2. Residual-based allocation in the full method

The permutation experiment addresses the complementary question of how to distribute supervision when all positions contribute to training. We kept target construction and the full projection rules fixed and varied the initial allocation.

Initial allocationAIME 2025 (%)LiveCodeBench v6 (%)
Uniform74.1768.12
Residual weights shuffled within position bins78.6169.74
Original residual-weight assignment (TransCAP)81.1173.33

We divided each response into eight relative-position bins and shuffled the initial weights within each bin. This preserved the weight values and the total initial weight in each region. Restoring the original assignment improved AIME 2025 by 2.50 percentage points and LiveCodeBench v6 by 3.59 points. Thus, even with the same total initial weight in each region, the choice of which tokens received greater weight affected performance.

Interpretation. At a fixed student-generated prefix, we compare the frozen model’s next-token predictions under two conditions: task plus privileged information, and privileged information alone. The signed, centered log-probability difference adjusts the supervision target. Its mean absolute magnitude across the vocabulary, gtg_t, measures how strongly adding the task changes the model’s relative preferences among candidate tokens. The new experiment connects that quantity to learning benefit: on both benchmarks, the positions selected for their higher residuals provided more useful supervision on average, as measured by the student’s improvement on unseen problems. This supports using gtg_t to set initial supervision priorities.

AuthorsOpenReview ↗

Additional Experiments for General-Capability Evaluation

Thank you for recognizing the motivation behind jointly designing supervision targets and supervision allocation, and for your constructive feedback. We have added general-capability evaluations, an end-to-end training-step timing comparison, and independent training runs. We first present these additional experiments, then address your questions about the guidance score, directional conflict, and methodological contribution.

To examine performance beyond the original mathematics and code benchmarks, we conducted mixed-domain training using mathematical, coding, and scientific examples from OpenThoughts. For each of the 4B and 9B models, we performed five independent TransCAP training runs and evaluated the resulting models on MMLU, GPQA Main, IFEval, and IFBench. We also evaluated the original model under the same protocol as the Raw control.

ModelMethodMMLUGPQAIFEvalIFBench
Qwen3-4BRaw77.1355.9081.3325.00
Qwen3-4BTransCAP79.1260.2085.6642.13
Qwen3.5-9BRaw84.6077.7889.2833.80
Qwen3.5-9BTransCAP90.1283.4890.0945.07

All models in this evaluation use the complete benchmark sets, Avg@12, and the main mathematical decoding settings. TransCAP scores are means across five independent training runs. MMLU and GPQA Main use answer accuracy, IFEval uses strict prompt-level accuracy, and IFBench uses loose prompt-level accuracy.

In this mixed-domain setting, the TransCAP means exceed the Raw scores on all four benchmarks at both scales. These results extend the evaluation to general knowledge, scientific reasoning, and instruction following, providing a direct comparison with the original models after broader-domain training. The results and evaluation settings are included in Table 2 and Appendix F.3. We discuss the specific contribution of conflict control separately in our response to your methodological questions.

AuthorsOpenReview ↗

Additional Experiments for W4: End-to-End Training Cost

Thank you for pointing out that projection latency alone does not capture the additional training cost. We have supplemented it with an end-to-end training-step timing comparison between OPSD and TransCAP.

For 9B mathematical training, we used 32 NVIDIA H20 GPUs with matched hardware and training settings. After warm-up, we timed 100 complete training steps for each method. Each measured step includes response generation, the required reference-model computations, student updates, and synchronization. Evaluation and checkpoint saving were excluded consistently for both methods.

MethodAverage end-to-end time per training step
OPSD1.00×
TransCAPapproximately 1.18×

TransCAP therefore takes approximately 18% more time per training step in this setting. This measurement includes the reference-model computations and the projection, rather than measuring the projection module alone.

For context, at the same 200 training steps in the main experiments, the 9B mathematical average increases from 62.31 for OPSD to 80.65 for TransCAP, a gain of 18.34 percentage points. Reporting these results alongside the timing comparison makes the performance–cost trade-off explicit.

Target construction and allocation are used only during training. At inference, TransCAP uses the student model alone, without an additional teacher or projection module.

Reviewer_8b46OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TransCAP, a framework for on-policy self-distillation that jointly considers target construction and token-level supervision allocation. The method constructs distillation targets using a task-conditioned residual and assigns initial token weights based on the strength of this signal. It then uses a constrained KL projection to adjust the weights while controlling directional conflict with a frozen reference model and limiting weight concentration. Experiments on three Qwen models show improvements across mathematical reasoning, code generation, and function calling benchmarks.

Critical strengths
  1. The paper addresses an interesting problem in on-policy self-distillation. Jointly considering what supervision to provide and where to apply it is well motivated.
  2. The two reweighting mechanisms are reasonably designed, and the constrained allocation formulation provides a clear mathematical interpretation.
  3. The empirical results are promising, with improvements across multiple model sizes and benchmarks.
  4. The ablation studies are fairly comprehensive, including analyses of allocation strength, conflict control, token-level alignment, and different projection constraints.
Critical weaknesses
  1. Limited evaluation of general capability preservation. One motivation for the proposed conflict control is to prevent the student from deviating too much from the original model's behavior. However, the evaluation mainly focuses on mathematical reasoning, code generation, and function calling. It remains unclear whether the method preserves the model's general capabilities outside these tasks. I would like to see additional evaluations on general knowledge and instruction-following benchmarks, such as MMLU, GPQA, or IFEval, to better understand the potential trade-offs.

  2. Lack of results across independent training runs. The main results are encouraging, but the paper does not report variance across independent training seeds. This is particularly relevant for Qwen3-4B, where the improvements over the strongest baselines are relatively small. Although Avg@12 reduces evaluation sampling noise for mathematical tasks, it does not account for training variability. Reporting results from multiple training runs would make the empirical conclusions more convincing.

Questions for authors
  1. Target construction and allocation: The two reweighting mechanisms seem reasonable, but could the authors further clarify their individual contributions? In particular, an ablation combining different target construction methods with the proposed allocation strategy would help establish whether the two components provide complementary benefits.
  2. Directional conflict: The conflict score is defined using directions in probability-distribution space. How well does this measure correlate with actual parameter-gradient interference or degradation of the original model's capabilities? Some empirical evidence would help justify its use as a proxy for harmful updates.
  3. Conflict budget: What motivates using the uniform-weighted conflict as the reference budget? The sensitivity analysis of κ is helpful, but I would appreciate more discussion of why this reference is appropriate for balancing learning and behavior preservation.

I am open to raising my score if the authors adequately address these concerns during the rebuttal.

8 comments
AuthorsOpenReview ↗

Response to W1: General Knowledge and Instruction-Following Evaluation

Thank you for your constructive feedback on TransCAP. We have added general-task evaluations and independent training runs, and clarified the component contributions and conflict-budget design. We address your two main concerns and three questions in separate replies, starting with general capabilities.

We have added general-task evaluations after mixed-domain training. For both 4B and 9B, we conducted five independent training runs on mixed mathematical, coding, and scientific OpenThoughts data and evaluated the resulting models on MMLU, GPQA Main, IFEval, and IFBench. The table also includes published reference scores for the original models, using the evaluation settings reported by their respective sources.

ModelMethodMMLUGPQAIFEvalIFBench
Qwen3-4BRaw77.1355.9081.3325.00
Qwen3-4BTransCAP79.1260.2085.6642.13
Qwen3.5-9BRaw84.6077.7889.2833.80
Qwen3.5-9BTransCAP90.1283.4890.0945.07

Reference-score context. All TransCAP evaluations were conducted on the complete evaluation sets using Avg@12, with the reported scores averaged across five independent training runs. We used answer accuracy for MMLU and GPQA Main, strict prompt-level accuracy for IFEval, and loose prompt-level accuracy for IFBench.

These experiments extend the evaluation to knowledge, scientific reasoning, and instruction following, providing empirical results under a broader training distribution. The additional results and evaluation settings appear in Table 2 and Appendix F.3.

AuthorsOpenReview ↗

Response to W2: General Knowledge and Instruction-Following Evaluation

Thank you for raising the concern about variability across training runs. We have added five independent training runs of the full TransCAP method for each model size, 4B and 9B, in both mathematics and code. The results are summarized below.

ModelMath Avg.Code Avg.
Qwen3-4B66.74 ± 0.2244.86 ± 0.43
Qwen3.5-9B80.87 ± 0.8572.20 ± 1.00
We report the mean ± sample standard deviation across five independent training runs. For each run, Math Avg. equally weights the three mathematics benchmarks, while Code Avg. equally weights the two code benchmarks. The original single-run results are reported separately in Table 1 and are not included in these five-run aggregates.

The results are consistent with the original findings and show limited variation across training seeds. We also clarify that independent training runs and Avg@12 address different sources of variability. Independent runs measure variation from training, whereas Avg@12 averages the correctness of 12 sampled responses per mathematics problem to reduce evaluation noise. Avg@12 is not a substitute for repeated training.

These results are now included in Section 4.2 and Appendix F. Tables 7–9 provide the per-benchmark statistics and mathematical correct-response counts.

AuthorsOpenReview ↗

Response to Q1: Contributions of Target Construction and Allocation

Thank you for asking us to clarify the contributions of target construction and allocation. We bring together the main results and ablations below to show what each component contributes and why the allocation constraints matter. All configurations use the same 9B model, reverse-KL objective, training budget, and other settings.

ConfigurationTargetAllocationAIME 2025LCB v6
OPSDPUniform62.5063.32
Q + uniformQUniform74.1768.12
Q + initial weightsQG without projection77.5061.20
TransCAPQG + Full projection81.1173.33
  1. Target construction provides a benefit on its own. The first two rows change only the supervision target. With uniform allocation, replacing P with Q improves AIME 2025 and LCB v6 by 11.67 and 4.80 percentage points, respectively. This comparison isolates the benefit of the constructed target under uniform allocation.

  2. Constrained allocation provides an additional benefit. Comparing Q + uniform with full TransCAP keeps the target-construction rule fixed. The complete allocation mechanism further improves AIME 2025 and LCB v6 by 6.94 and 5.21 percentage points, respectively. Target construction determines what distribution the student learns at each position, while allocation determines how much each position contributes to training.

  3. The constraints help protect against the degradation observed with unconstrained weighting. Simply introducing nonuniform weights is not sufficient. Without projection, the initial weights improve AIME 2025 over uniform allocation, but reduce LCB v6 from 68.12 to 61.20, below even the original model’s 65.60. Full projection raises LCB v6 to 73.33 while also improving AIME 2025 to 81.11.

The constraints therefore play a concrete protective role in this experiment. The risk budget limits the allocation-weighted local directional conflict with the clean anchor, while the cumulative constraints prevent the redistributed weights from becoming more concentrated than the initial weights. Together, they regulate how supervision is allocated. The code results show that this regulation helps avoid the performance degradation observed when the initial weights are used directly.

These comparisons support both the benefit of target construction and the additional benefit of constrained, rather than merely nonuniform, allocation. The relevant results are reported in Tables 1, 12, and 13, with the corresponding analysis in Section 4.3 and Appendix G.

Reviewer_DTuSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a framework for jointly constructing supervision targets and allocating token-level weights in on-policy self-distillation. It uses a task-conditioned contrast between privileged and reference-only predictions to construct guidance targets and initial position weights. A constrained KL projection then redistributes these weights based on geometric directional conflict with a clean base-model anchor, while limiting weight concentration. Experiments across five reasoning, code-generation, and function-calling benchmarks report improvements over existing self-distillation methods.

Critical strengths
  • I find the idea of jointly considering what the student learns and how much supervision each response position receives potentially interesting, though the presentation of the proposed method is very abstract and difficult to follow
Critical weaknesses
  • I found the paper generally hard to read, especially the intro and method sections; I believe these are more than minor writing issues (e.g., heavily polished by text tools). Many terms or concepts are spread out without (mostly) explaining their meaning, purpose, or connections; thus, it makes the proposed method hard to assess and its contribution difficult to appreciate. Some concrete examples are below:

The introduction introduces many concepts, such as "supervision targets"/"weights", "target construction", "task-conditioned residuals", "task-conditioned corrections", "position priorities", "directional conflict", etc., with no explanation of what they are, thus making "how they relate to one another" very difficult to understand. For instance, it is difficult to understand what the authors mean by "conflict" at the point where it is introduced, where such conflict comes from, and why controlling it matters for self-distillation. overall,it is difficult to grasp the main problem and understand why the proposed solution is necessary.

in method sec 3.2, terms such as "target purification" appear with limited intuitive explanation; though the authors define the relevant quantities in math, it remains difficult to understand what the proposed residual represents in practice, why the contrast should produce transferable guidance, and why the same residual is appropriate for constructing both supervision targets and initial token weights. also, why centring the residual, and why using the mean absolute magnitude to score positions? also the discussion of "Equal vocabulary weighting" following eq (4) is particularly difficult to follow without first understanding why this score was chosen. sec 3.1 refers to Q1, which is only later introduced in sec 4.3

sec 3.3 is especially challenging to follow. The square-root embedding, spherical logarithmic map, directional conflict, risk budget, KL projection, majorization constraints, and effective sample size are spoken out with no meaning and no practical motive and no relationships among them. E.g., why should comparing geometric directions toward the guidance target and the clean anchor be an appropriate measure of conflict (what is conflict? and why)? What does risk budget actually do? So the main text should include enough intuitive explanation to make core parts understandable in the first place. The paper gives formulas without first giving readers a minimal context to understand what they are and why they exist there.

  • sec 3.3 introduces a directional conflict score, and my understanding is that when the student moves along the direct geometric path from the clean anchor to the target, the conflict score is zero at the anchor, increases to its maximum halfway toward the target, and decreases to zero again at the target. This means that a high conflict score can arise simply because the student is moving toward the guidance target, without necessarily indicating harmful learning. Can the authors clarify this? Or, asking broadly, why should reducing this particular geometric conflict improve the student's learning or preserve useful behaviour? so it would help to see whether the additional geometric construction is necessary by examining whether high-conflict positions are actually associated with harmful updates or degradation of previously learned behaviour.

  • for the ablations, the takeaways often summarize which config performs better without clearly explaining what the results reveal about the proposed components? also, given the difficulty of following the method's core concepts, this makes its significance and necessity of the different harders to assess.

For example. Q1 doesn't provide insight into why aggressive concentrated supervision would be harmful and why the preferred concentration differs? Q2 shows stronger conflict control improves performance, but what undesirable behaviour is being reduced? and why the proposed conflict identifies it?

In Table 1, some improvements are modest (compared to the runner-up) like qwen3-4b, 27b, so it might be informative to report variability across training seeds, why measured with Avg@12? also here the table mixes published results with reproduced baselines without clearly identifying which entries in table 1 come from each source. also, it would help to understand by knowing the overall computational cost overhead induced by the proposed method, compared with the closest baseline.

fig 1 is not self-contained; several symbols and operations are also difficult to interpret without repeatedly consulting the method section.

Questions for authors
  • since the proposed method changes the relative weights of different tokens within a generated response, are the downweighted positions typically reasoning steps, final-answer tokens, formatting tokens? Does the projection suppress positions that would otherwise contribute to incorrect or degraded model behaviour, and it would be useful to see a concrete example

see more questions in the weaknesses above

9 comments
AuthorsOpenReview ↗

Response to R1: Terminology and Connections Between Components

Thank you for reading the paper carefully and identifying specific points that make the method difficult to follow and assess. Below, we clarify the terminology and the connections between components, then address the conflict measure, ablations, training variability, result sources, and computational overhead. We have also made targeted clarifications in the introduction and method.

We first clarify that the terminology has specific precedents in the literature; these concepts are not all introduced for the first time in this paper.

  • Target purification and conditional residuals. Purified OPSD uses a reference-only distribution to extract a question-conditioned correction and construct a purified supervision target. Our target purification and task-conditioned residual build on this construction. Purified OPSD, Section 3
  • Supervision targets and position weights. PW-OPSD adjusts position weights while retaining the privileged teacher target. USD explicitly distinguishes supervision-distribution construction from token weighting and adjusts both. Thus, what distribution to learn and how much weight each position receives are established design dimensions. PW-OPSD, Section 3.4, USD, Section 3.1
  • Distribution geometry. The square-root representation and Fisher–Rao spherical geometry also appear in GeoSD. We use this representation to compare directions at the current student distribution. GeoSD, Appendix A.2

In our method, the target specifies the distribution the student should learn at a position, while the weight determines that position’s contribution to training. The same residual connects these two choices: its signed components adjust the target, and its magnitude sets the initial position weight.

The anchor is the original model’s prediction using the task alone. Moving toward the supervision target can locally move the student away from this anchor; we call this directional conflict. The projection then adjusts the weights to control this conflict while preventing the redistributed supervision from becoming more concentrated than the initial allocation. This is the connection between target construction, position priorities, conflict control, and concentration control.

To make these relationships easier to follow, the revised introduction identifies the anchor at first mention, explains the residual through the two predictions being compared, and introduces directional conflict in plain language. The method overview also states how the two modules determine what to learn and how much each position contributes.

AuthorsOpenReview ↗

Response to R2: Meaning of the Residual and Its Use in Targets and Weights

At the same student-generated prefix, we evaluate the same frozen model under two conditions: task plus privileged information, producing P, and privileged information alone, producing R. Their log-probability difference asks a concrete question: how does adding the task change the model’s relative preferences over candidate tokens?

Content already predictable from the privileged information alone also receives support in R. Therefore, a high probability under the privileged teacher does not, by itself, produce a large positive correction. We use this contrast to reweight the task-only distribution B and construct the supervision target Q. This follows the task-conditioned correction approach in Purified OPSD, Section 3. The construction identifies task-conditioned changes; it does not assume that every change represents correct or learnable knowledge.

The same residual serves two related purposes. Its signed components indicate which relative token preferences to increase or decrease in the target. Its absolute magnitude summarizes the correction strength at each position and determines the initial supervision priority. The subsequent projection adjusts these priorities using the current student distribution.

The specific design choices have the following roles:

  • Centering removes a vocabulary-wide offset. This does not change the normalized target Q, but prevents an irrelevant common offset from affecting the position score.
  • Mean absolute magnitude summarizes correction strength without allowing positive and negative changes to cancel. Simply summing the centered residual would always give zero.
  • Equal vocabulary weighting allows relative corrections to low-probability candidates to contribute to the score, without further suppressing them by their probability. The score therefore measures correction strength, rather than the amount of probability mass moved or the correctness of an answer.

Section 3.2 now explains these two uses of the residual and the reasons for the scoring choices. We have also removed the premature reference to Q1 from the method overview. Appendix D.1 provides the mathematical details, while Table 12 and Appendix G examine the practical value of the resulting position priorities through the permutation control.

AuthorsOpenReview ↗

Response to R3–R5: Directional Conflict, Allocation Constraints, and Ablation Evidence

We address these comments together because they concern the same question: what behavior does the allocation mechanism control, and what evidence supports its design?

1. How the components work together.

The square-root representation puts probability distributions on a common sphere, and the logarithmic map expresses the target and anchor as directions from the current student. These allow us to compare the two directions.

The risk budget limits allocation-weighted conflict. However, reducing weight at some positions redistributes it elsewhere, potentially concentrating supervision on a few positions. The cumulative constraints prevent this increase in concentration. Finally, minimum-KL projection finds the allocation that satisfies both constraints while changing the initial weights as little as possible. The nondecreasing effective sample size is a consequence of concentration control, not an additional training module.

2. A high conflict score does not necessarily mean harmful learning.

Your observation about the midpoint is correct. Appendix D.3 of the original submission derives this same behavior: the score is zero at the anchor and target and reaches its maximum at the midpoint. We interpret ρ as tension between moving toward the target and staying close to the anchor, not as a label that the target-directed update is harmful.

The algorithm controls excess weighted conflict relative to uniform supervision, rather than requiring every position’s conflict to become zero. For example, if all positions have the same conflict score, the initial allocation is retained even when that score is high.

3. The experiments distinguish conflict control from concentration control.

The 9B constraint ablation uses the same target-construction rule and initial guidance allocation:

ConfigurationAIME 2025LCB v6
No projection77.5061.20
RiskOnly78.6167.20
Box80.2871.30
Full81.1173.33

On code, unconstrained allocation scores 61.20, below the original model’s 65.60. RiskOnly raises this to 67.20, and Full reaches 73.33. This identifies an observed performance decline that the constraints help avoid.

The fixed-prefix diagnostic explains what changes in the allocation. Across 256 fixed responses, we hold the prefixes, targets, conflict scores, initial weights, and budgets identical. RiskOnly, Box, and Full reduce weighted conflict to similar levels, but only Full eliminates cumulative concentration excess. Thus, Full does more than reduce conflict: it also prevents the redistribution from concentrating supervision excessively.

The diagnostic establishes the allocation behavior, while the training ablation measures its downstream benefit. Neither alone identifies individual high-ρ positions as the cause of harmful parameter updates.

4. What the concentration and alignment ablations show.

Increasing γ emphasizes fewer positions and leaves less supervision for others; it also changes the projection reference. Mathematics peaks at 83.33 with γ = 1.5, while code peaks at 73.33 with γ = 1. At γ = 2, the scores fall to 79.17/71.12. These results show that stronger concentration is not always better and that the preferred tested setting differs by task.

The within-bin permutation control preserves the initial weight values and coarse positional totals but changes their correspondence to specific positions. Scores fall from 81.11/73.33 to 78.61/69.74, supporting the value of the original weight–position alignment within the full pipeline.

Section 4.3 summarizes these findings, with the full experiments in Appendix G and training diagnostics in Appendix H. The added Appendix D.6 explains how the two constraints jointly control an upper bound on anchor-distance increase under a simplified independent-output update model. Its scope is distinct from a guarantee about shared-parameter training or capability retention.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Summary of Manuscript Revisions and Additional Experiments

Dear Reviewers and Area Chair,

Thank you for your careful reading and constructive feedback. We have revised the manuscript to clarify the method, strengthen the experimental evidence, and make the result sources and training costs more transparent. Below, we summarize the changes and their locations.

  1. Clearer motivation and explanation of the method — Introduction and Sections 3.1–3.3. We clarify what the anchor represents, which predictions are compared to construct the residual, and how that residual determines both supervision targets and initial position weights. We also explain the roles of the geometric representation, conflict budget, and concentration constraints in more intuitive terms.

  2. Additional mathematical explanation — Section 3.3 and Appendix D.6. We explain how ρ relates to the first-order change in anchor distance when moving toward the target. The new Appendix D.6 shows how conflict and concentration constraints jointly control a common upper bound on anchor-distance increase under a simplified independent-output update model. We explicitly distinguish this analysis from a guarantee about shared-parameter training or capability retention.

  3. Additional independent training runs — Section 4.2 and Appendix F. We added five independent training runs for each of the 4B and 9B models, with mathematics and code trained separately under their respective main-experiment settings.

ModelMath Avg.Code Avg.
Qwen3-4B66.74 ± 0.2244.86 ± 0.43
Qwen3.5-9B80.87 ± 0.8572.20 ± 1.00

The table reports mean ± sample standard deviation. These aggregates exclude the original single-run results in Table 1. Tables 7–8 provide per-benchmark statistics, and the manuscript distinguishes repeated training from Avg@12 evaluation sampling.

  1. Broader evaluation after mixed-domain training — Section 4.2, Table 2, and Appendix F.3. We added mixed-domain training on mathematical, coding, and scientific OpenThoughts data for 4B and 9B, followed by evaluation on MMLU, GPQA Main, IFEval, and IFBench.
ModelMethodMMLUGPQAIFEvalIFBench
Qwen3-4BRaw77.1355.9081.3325.00
Qwen3-4BTransCAP79.1260.2085.6642.13
Qwen3.5-9BRaw84.6077.7889.2833.80
Qwen3.5-9BTransCAP90.1283.4890.0945.07

Table 2 includes both the original models and TransCAP under the stated evaluation protocol; TransCAP scores average five independent training runs.

  1. Explicit baseline provenance — Section 4.1 and Appendix B, Table 4. We identify the Qwen3-4B SDFT and DOPD mathematics scores as coming from DAPD Table 1. All remaining results come from our own experiments and evaluations. We also report independent reproductions of those two mathematics baselines, with relative deviations from the cited results within 1.5%.

  2. End-to-end training-step overhead — Appendix I.3, Table 13. For 9B mathematics training on 32 NVIDIA H20 GPUs, we timed 100 complete training steps after warm-up under matched settings. TransCAP takes approximately 18% more time than OPSD.

MethodRelative time over 100 timed training stepsMath Avg. at step 200
OPSD1.00×62.31
TransCAP1.18×80.65

For context, their mathematics averages at the same 200-step training budget are 80.65 and 62.31, respectively. The appendix specifies the timing scope and distinguishes complete-step timing from module latency.

  1. More focused presentation of the ablations — Section 4.3 and Appendices G–H. Section 4.3 now summarizes what the concentration, conflict-control, alignment, and cumulative-constraint experiments reveal about the method. The original detailed ablations, parameter sweeps, and fixed-prefix diagnostics are retained in Appendix G, with training diagnostics in Appendix H.“

New and clarified material is marked in blue. Thank you again for your time and consideration. We hope these revisions address the concerns more clearly, and we welcome further questions or discussion about any remaining points.

Thank you for the additional experiments and clarifications. They strengthen the empirical results, but my two main concerns remain. First, the permutation experiment shows that the weighting is useful, but does not establish that larger residuals indicate more transferable knowledge or greater learning benefit. Second, the conflict-control ablation shows improved task performance, but does not directly demonstrate reduced gradient interference or forgetting. The broader evaluations also lack a matched comparison with and without conflict control. I therefore remain inclined to reject the paper. Finally, some replies here seem to address Reviewer 8b46’s questions, while the response to my residual-score concern appears in their thread. Could you please check whether these replies were posted under the intended reviews?

Thank you for pointing this out. We checked both review threads and confirmed that we posted the replies intended for you under Reviewer 8b46’s review, and vice versa. We apologize for the mix-up and for making you look across threads to find our response. We will repost the replies intended for you in this thread.

AuthorsOpenReview ↗

Additional Experiments for Main Concern 1: Does Conflict Control Reduce Forgetting?

Thank you for pointing out that a model can improve its overall score while losing the ability to answer questions it previously solved. We added two evaluations on the 9B model to address this concern. We first compare general-benchmark scores with conflict control enabled and disabled. We then track the same questions that the original model passed to measure item-level forgetting directly.

1. General-capability evaluation with and without conflict control

Our earlier evaluation compared the original model (Raw) with full TransCAP after mixed-domain training (Table 2 and Appendix F.3). We now include a 9B model trained with conflict control disabled under the same mixed-domain setup. The table reports Avg@12 scores.

BenchmarkRaw (%)Control off (%)Control on (%)On − off (pp)
MMLU84.6089.5790.12+0.55
GPQA Main77.7882.9183.48+0.57
IFEval89.2890.0190.09+0.08
IFBench33.8043.9445.07+1.13

The model trained with conflict control had higher observed scores on all four benchmarks. The largest difference was on IFBench (+1.13 percentage points), while the IFEval scores were close (+0.08).

The 9B code ablation provides a concrete example of recovery from a benchmark-level decline. On LiveCodeBench v6, training with the initial supervision weights and no projection lowered the score from Raw’s 65.60 to 61.20. With target construction and initial weighting held fixed, adding only the conflict constraint (RiskOnly) raised the score to 67.20, a gain of 6.00 points over unconstrained weighting and 1.60 points above Raw. Full TransCAP, which also controls weight concentration, reached 73.33. These aggregate scores leave open which previously solved questions the model retained.

2. Item-level forgetting on questions the original model passed

We conducted a separate pass@3 evaluation. For each benchmark, we fixed the set of questions for which Raw produced at least one passing response in three attempts. We evaluated both trained models on those same questions and counted a question as lost if all three responses from the trained model failed. This gives both conditions the same denominator and prevents gains on newly solved questions from masking losses on previously passed ones.

BenchmarkQuestions Raw passedLost with control offLost with control onReduction in loss rate (pp)
MMLU11,880880 (7.41%)70 (0.59%)6.82
GPQA Main34830 (8.62%)3 (0.86%)7.76
IFEval48939 (7.98%)4 (0.82%)7.16
IFBench10215 (14.71%)0 (0.00%)14.71

IFEval illustrates why the distinction matters. The overall Avg@12 scores differed by only 0.08 percentage points. Yet among the same 489 prompts that Raw passed in the independent pass@3 evaluation, the model trained without conflict control lost 39, while the model trained with it lost 4. Tracking individual prompts reveals a difference in retention that the aggregate scores alone could not establish.

Across all four benchmarks, the model trained with conflict control retained more of the questions that Raw had passed. This directly supports reduced item-level forgetting under the stated pass@3 criterion. Appendix D.6 provides a theoretical rationale in output-distribution space under a simplified update model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.