acceptodds
Under review as a conference paper at ICLR 2027

Learn from the Whole Chain: Answer-Anchored Reweighting for On-Policy Distillation

Abstract

On-policy distillation (OPD) is an important method for transferring reasoning capabilities from large language models to smaller models. Prior studies report that student-generated prefixes can drift from the teacher’s typical trajectories, weakening supervision at later positions and motivating methods that shorten training rollouts. However, we argue that token position is only a coarse proxy for supervision utility. To account for each token’s role in forming the student’s answer, we propose Answer-Anchored Reweighting (AAR). AAR uses the generated answer as an anchor for trace-based attribution over the full reasoning trajectory. Based on the resulting within-response rankings, it upweights selected backbone tokens while retaining baseline teacher supervision at other valid positions. These weights are recomputed for each online batch, and the weighted loss is normalized within each response, with teacher distributions remaining the learning targets. Experiments on AIME2024, AIME2025, and AMC2023 show that AAR achieves higher macro-averaged Avg@32 than standard OPD across all four evaluated response lengths. At the longest evaluated response length (16k), AAR reaches 52.29%, exceeding the corresponding OPD baseline by 4.20 percentage points and the best evaluated OPD configuration by 1.60 percentage points. These results support the value of later supervision and show that answer-anchored reweighting can improve long-chain distillation without relying solely on shorter training trajectories.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bwuQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper starts from the observation that student-generated prefixes drift away from the teacher's distribution as reasoning proceeds, which has led prior work to shorten training rollouts, and argues that token position is only a coarse proxy for supervision utility. It proposes Answer-Anchored Reweighting: take the student's generated boxed-answer span as an anchor, run span-based recursive trace attribution over the whole trajectory, rank generated positions by attribution score within each response, mark the top-ρ fraction as "backbone" tokens whose distillation weight is multiplied by α while all other valid positions stay at 1, then normalise the weighted loss within each response. Weights are recomputed each online batch. With DeepSeek-R1-Distill-Qwen-1.5B as student and Skywork-OR1-Math-7B as teacher, reverse KL over the teacher's top-20 support, trained on DAPO-Math-17K, AAR is reported to beat standard OPD in macro-averaged Avg@32 at all four supervision lengths, reaching 52.29% at 16k against 48.09% for OPD.

Critical strengths
  1. The question is well posed. Asking how to allocate supervision inside a long trajectory, instead of just truncating it, is more constructive than the prevailing shorten-the-rollout response, and the paper is explicit that this is the contribution.

  2. Rewriting the normalised objective as a convex mixture, ℓAAR = (1−λ)ℓU + λℓS with λ = q(α−1)/(1+q(α−1)), is a genuinely useful piece of analysis. It turns ρ and α into a single redistribution of coefficient mass rather than two unrelated knobs, and explains why expanding the selected set does not monotonically strengthen emphasis.

Critical weaknesses
  1. The main comparison is a maximum over a hyperparameter sweep against a single-configuration baseline, with checkpoints chosen on the evaluation sets. Section 1 states the main table uses "the best-performing AAR hyperparameter configuration at each response length", while OPD has no corresponding knobs. Section 4.2 states the reported checkpoint is "the checkpoint at peak performance during training", and no validation split is described anywhere. The 16k ablation over α alone spans 48.47 to 52.29, about 3.8 points, which is the same order as the headline +4.20 over OPD. As reported I cannot separate the method's effect from the selection procedure.

  2. No seeds, small benchmarks, and inconsistent metric choice. AIME24 and AIME25 have 30 problems each. In Table 2 the chosen reference configuration (ρ = 40%) is best on Avg@32 but clearly worst on Pass@32 (72.78 against 79.44 at ρ = 20%). In Table 3 the α ordering is 1.5 > 2.5 > 2.0 > 0.5, non-monotonic in a way that reads like noise rather than a real optimum. Worth flagging: α = 0.5, which down-weights the backbone tokens, gives 48.47, essentially tied with OPD's 48.09. If the attribution found genuinely high-utility positions, inverting the emphasis should hurt more than that.

  3. The attribution itself is barely validated. The deletion sanity check in Figure 1 runs on "a selected trajectory", a single hand-picked example, with no aggregate deletion curve over the dataset. And since the anchor is the student's own generated answer whether or not it is correct, the method up-weights exactly the tokens most responsible for producing a wrong answer on failing rollouts. No result is split by correct versus incorrect rollouts, which seems like the first thing the design calls for.

Questions for authors
  • Is there a held-out validation set used for checkpoint selection, or is the peak checkpoint picked directly on AIME/AMC?
  • What do the numbers look like for a single fixed (H, ρ, α) applied across all four response lengths, against OPD?
  • Does restricting backbone up-weighting to correct rollouts, or down-weighting it on incorrect ones, change the result?
4 comments

We sincerely thank you for your thorough evaluation and valuable comments. We will include the additional experimental results and clarify the relevant points in the revised manuscript. Our responses to your comments are provided below. All scores below are reported as macro-averaged Avg@32 / Pass@32 (%).

Weakness 1 & Question 1

Thank you for pointing out that the experimental setup was not described clearly enough in the original manuscript. We split the DAPO-Math-17K data used in this study into training and validation sets in an 80%/20% ratio. Appendix D of the original manuscript reports 869 logged steps. For our one-epoch run with a global batch size of 16, this corresponds to 869×16=13,904869\times16=13,904 training samples. The remaining approximately 3k samples formed our held-out 20% validation set.

Hyperparameters were selected through grid search using accuracy on the held-out validation set as the selection criterion. For each training run, we also selected the checkpoint with the highest validation accuracy and used that checkpoint to report Avg@32 and Pass@32 on the test sets. Neither hyperparameter selection nor checkpoint selection was based on AIME/AMC test scores. The phrase "peak performance" in the original manuscript refers to the best validation performance.


Weakness 2

  1. Additional independent training runs and clarification of metric selection criteria.

We independently repeated 16k AAR training with two additional random seeds. Including the original main experiment, the three independent training runs achieved Avg@32 of 52.28±0.2752.28\pm0.27 and Pass@32 of 74.17±1.5474.17\pm1.54. Here, ±\pm denotes the sample standard deviation across training random seeds. These results support AAR's consistent performance advantage over standard OPD across different training random seeds.

We use macro-averaged Avg@32 as the primary metric for performance comparisons and for identifying the best results. This is already reflected in the note to Table 1 in the original manuscript. We will state this primary metric more clearly in the revised manuscript and improve the description of the corresponding selection criteria to avoid ambiguity.

  1. Selection of α\alpha.

In the 16k ablation with H=1,ρ=40H=1,\rho=40\\% fixed, α=1.5\alpha=1.5 achieved the highest performance among the tested values for the current teacher-student pair. The parameter α\alpha controls the relative supervision strength on backbone positions and the remaining positions. Moderate weighting helps balance selective emphasis and overall learning. This balance does not require performance to vary strictly monotonically with α\alpha.

  1. Reversing the weighting.

Our top-40% selection specifies a budget for supervision emphasis. It does not directly divide tokens into "useful" and "useless" groups. Thus, although α=0.5\alpha=0.5 reduces the relative weights of the highest-attribution positions, it increases the relative emphasis on the remaining positions. These still include tokens with moderate attribution that may provide useful teacher guidance. This redistribution may yield some benefits and does not contradict AAR's design.

Weakness 3 & Question 3

  1. Attribution and token-masking experiments.

We added attribution and token-masking experiments across the training set. We used the student model to generate responses for the entire training set, with a fixed generation cap of 16k tokens. After excluding responses truncated at this cap, we analyzed the remaining approximately 13,000 complete trajectories and aggregated the masking curves. The aggregated results were broadly consistent with the overall trends for the single trajectory in the original manuscript. When the same fraction of pre-answer tokens was masked, masking high-attribution tokens first generally reduced the likelihood of the fixed answer suffix more substantially than random masking or masking low-attribution tokens first. This showed a stronger deletion effect. The local differences were as follows: the likelihood retention curve showed a steeper initial decline when high-attribution tokens were masked first. For masking high-attribution tokens first, normalized deletion response values were relatively higher in the middle and later portions of the curve. In the revised manuscript, we will replace the original figure with a new figure showing the aggregated results across the training set.

  1. Additional experiments based on answer correctness.

To examine whether trajectories with incorrect reasoning or answers have negative effects under AAR, we added training experiments that separated trajectories by final-answer correctness. We compared three settings: a mixture of correct and incorrect trajectories, correct trajectories only, and incorrect trajectories only. We kept the number of samples actually used for parameter updates the same across these settings. This sample count was approximately equivalent to that used in 200 training steps of the main experiment. The best scores for the three AAR training settings and the OPD baseline in this comparison are shown below:

Training settingAvg@32 (%)Pass@32 (%)
AAR: mixed trajectories48.8980.28
AAR: correct trajectories only47.8274.72
AAR: incorrect trajectories only47.4769.44
OPD baseline46.8868.33

Mixed training achieved the highest scores on both metrics. AAR training using only trajectories with incorrect final answers also outperformed the OPD baseline in this comparison on both metrics.

AAR does not treat answer correctness or attribution scores as labels for the correctness of the reasoning process. The student's answer is used only to identify where to emphasize supervision, while teacher distributions remain the learning targets. Thus, even if some flawed intermediate positions are selected as backbone positions, increasing their weights places greater emphasis on teacher supervision at those positions. It does not reinforce the student's generated derivation as a correct example. Teacher supervision may reinforce this part of the derivation, but instead suppresses it when the token distributions disagree. We therefore believe that training value cannot be judged solely by whether the final answer is correct.


Question 2

Appendix B.3, "Selected Configurations for the Main Results Table," in the original manuscript already lists the configurations used in the main table. The "Additional 7k Results" paragraph after Table 6 also reports supplementary results with the same hyperparameters. Specifically, the main experiments at 3k, 12k, and 16k all used H=1,ρ=40H=1,\rho=40\\%,\alpha=1.5, while the main 7k experiment used α=2.0\alpha=2.0 with the other hyperparameters unchanged.

With these common AAR hyperparameters, the results were:

MethodAvg@32 (%)Pass@32 (%)
AAR (3K)45.6874.72
AAR (7K)51.1171.67
AAR (12K)51.3572.78
AAR (16K)52.2972.78
Standard OPD (7K)50.6970.56

AAR outperformed standard OPD on both metrics. Together with the main results at the other supervision lengths, these results show that AAR still outperformed standard OPD on both metrics when H=1,ρ=40H=1,\rho=40\\%,\alpha=1.5 were used consistently across all four supervision lengths.

Summary of Manuscript Revisions

Thank you again for your valuable comments. For ease of reference, we have incorporated the updates discussed in our responses into the revised manuscript and highlighted the relevant changes in blue. Validation-based selection is clarified in the Introduction and Section 4.1, with the training/validation split and training configuration specified in Section 4.1. The role of macro-averaged Avg@32 as the primary comparison metric is also clarified in Section 4.1. Independent training runs are reported in Appendix B.9, with a summary in Section 4.2. The aggregated masking analysis is presented in Section 3.1 and the updated Figure 1. Training comparisons grouped by final-answer correctness and the corresponding discussion are added in Section 6 and Appendix B.8 (Table 9).

We would be grateful to hear whether our responses and manuscript revisions have addressed your concerns and, if so, whether you would consider revising your rating upward. If any questions remain, we would be happy to respond promptly.

Reviewer_vETdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles a known bottleneck in On-Policy Distillation (OPD): as a student model generates long reasoning paths, its prefixes drift away from the teacher's typical training distribution, causing the quality of teacher supervision to plateau or decay at later token positions. While traditional remedies combat this by artificially shortening training rollouts, this paper argues that token position is a coarse proxy for supervision value. To utilize the full reasoning chain, the authors propose Answer-Anchored Reweighting (AAR) . AAR uses the final boxed answer generated by the student as an anchor to trace representation contributions (attribution scores) backwards through the reasoning chain via recursive passes. The tokens are then ranked within each response; the top ρ\rho fraction of tokens (the "backbone") are allocated a higher distillation weight multiplier (α\alpha), while the rest retain baseline supervision. The final loss is weight-normalized within each response, keeping the teacher's top-K distributions as targets. Using DeepSeek-R1-Distill-Qwen-1.5B as the student and Skywork-OR1-Math-7B as the teacher across competitive math benchmarks (AIME24, AIME25, AMC23), AAR consistently outperforms standard uniform OPD. At a 16k response length, AAR achieves a macro-averaged accuracy of 52.29%, outperforming standard OPD by 4.20 percentage points.

Critical strengths
  • Challenging Core Assumptions: The paper offers an elegant counter-perspective to the dominant paradigm in long-chain distillation, demonstrating that long trajectories do not need to be cropped if the later tokens are functionally reweighted based on answer relevance.
  • Mechanistic Grounding: Integrating span-based recursive attribution directly into an online optimization loop dynamically adjusts focus based on what the student is actively outputting, rather than relying on static or heuristic constraints.
  • Strong Long-Chain Improvements: The empirical gains are clear, particularly at long sequence limits (12k and 16k), where standard uniform OPD baseline performance actively degrades, while AAR continues to show performance improvements.
Critical weaknesses
  • Substantial Runtime & Compute Penalty: Calculating recursive multi-layer attributions dynamically for every batch introduces a severe computational bottleneck. End-to-end training metrics reveal that AAR increases the mean step time by 59.1% compared to standard OPD (84.6 seconds vs. 53.1 seconds) and significantly inflates peak GPU memory footprints up to nearly 70 GiB at the 16k token boundary ,.
  • Dependency on Rigid Anchors: The AAR framework relies entirely on the presence of a clearly localized target command (\boxed{}) to serve as the backward attribution root . For non-mathematical, open-ended generative domains (e.g., summary generation, creative writing, or continuous code synthesis) where terminal states are diffuse, the method lacks an immediate plug-and-play mechanism .
  • Non-Monotonic Performance Scaling: Despite the targeted reweighting, AAR's performance still peaks at the intermediate 7k horizon (52.81%) and drops slightly at 16k (52.29%) ``. This indicates that while the architecture successfully mitigates prefix noise, it does not completely eliminate the negative impacts of long-horizon drift on distillation efficiency.
Questions for authors
  1. Backbone Stability: Since attribution rankings are updated live with every online batch, how volatile is the composition of the selected backbone set across training iterations? Does the token turnover rate (as measured in Figure 6 decrease as the student policy converges closer toward the teacher's distribution?
  2. Amortization of Attribution Costs: Given the hefty 59.1% wall-clock runtime overhead incurred during the aggregate training stage, have you evaluated if computing the attribution weights periodically (e.g., every few steps or gradient accumulation intervals) rather than at every single forward pass preserves performance while restoring training throughput?
  3. Handling "Right Answer, Wrong Process" Scenarios: In mathematical generation, students occasionally produce flawed or hallucinated intermediate derivation steps that happen to accidentally stumble into the correct final boxed answer. In these instances, does AAR's anchor mechanism inadvertently over-weight and reinforce the deceptive/incorrect reasoning tokens?
5 comments

We sincerely thank you for your thorough evaluation and valuable comments. We will include the additional experimental results and clarify the relevant points in the revised manuscript. Our responses to your comments are provided below. All scores below are reported as macro-averaged Avg@32 / Pass@32 (%).

Weakness 1 & Question 2

In the current algorithm, we compute and determine the token weights for each batch of newly generated student trajectories. These weights remain fixed during parameter updates on that batch. Attribution scores and weights are then recomputed for the next batch of new trajectories.

To address the computational overhead, we have added a comparison under a common GPU budget cap. In the 16k setting, standard OPD requires 102.6 GPU-hours to complete one epoch of training. Based on AAR's mean cost per step, this budget supports approximately 545 complete training steps. We therefore round down to step 500 as the upper limit for AAR checkpoint selection. Standard OPD retains its best result from the full epoch, while AAR selects a checkpoint only within the first 500 steps.

The corresponding best scores are shown below:

MethodAvg@32 (%)Pass@32 (%)
OPD48.0969.72
AAR50.7678.06

AAR exceeds OPD by 2.67 and 8.34 percentage points, respectively. Thus, even when attribution overhead is included and the number of training steps is reduced accordingly, AAR retains a performance advantage without exceeding OPD's computational budget.

At the same time, we acknowledge that AAR does require more computational resources under the full one-epoch configuration of the main experiments.

The cumulative training costs are:

MethodTraining cost (GPU-hours)
OPD102.6
AAR163.3

Attribution computation also requires additional GPU memory. As shown in Figure 7 in Appendix C, the peak GPU memory used for attribution computation approaches 70 GiB at a response length of 16k. This reflects a trade-off between model performance, training time, and GPU memory requirements. More targeted supervision allocation improves performance but requires additional computation and GPU memory.

Weakness 2

We first clarify that the underlying attribution operator is not restricted to a particular answer format. As defined in Section 3.1 and Appendix A, its target is determined by a specified position set and aggregation weights. In principle, attribution can therefore be performed for other selected valid text spans within a trajectory.

We also agree that open-ended generation may lack a clear target span or have task objectives spread across the output. In such cases, applying the method requires choosing an attribution target that reflects the task requirements. We have already noted this issue in Appendix F, "Limitations," of the original manuscript. Thus, the key issue is how to determine a meaningful attribution target based on task requirements, rather than whether the target has a particular format. This is a direction for future research.


Weakness 3

Expanding the supervision range can cover more reasoning steps, but there is also a trade-off between supervision coverage and supervision utility. Therefore, peak performance at an intermediate response length is not contradictory. We further emphasize our central claim: AAR provides empirical evidence for the learning value of later supervision and offers an alternative to shortening responses for improving long-chain distillation.

In addition, Table 6 in Appendix B.3 specifies that the main 7k experiment used α=2.0\alpha=2.0, while the other lengths used α=1.5\alpha=1.5. The "Additional 7k Results" paragraph in the same section reports results with the common configuration H=1,ρ=40H=1,\rho=40\\%,\alpha=1.5,K=20.

With these common AAR hyperparameters, the results were:

MethodAvg@32 (%)Pass@32 (%)
AAR (7K)51.1171.67
AAR (12K)51.3572.78
AAR (16K)52.2972.78
Standard OPD (7K)50.6970.56

At 7k, AAR also outperformed standard OPD on both metrics under this common configuration. Under this configuration, the 7k Avg@32 score was lower than the scores at 12k and 16k. Thus, the 7k peak in the main table should also be understood in light of its different weighting configuration.

Question 1

AAR's attribution procedure starts from the final answer and examines which preceding tokens contribute more to its formation. This produces a contribution score for each position. This step is called Hop 0. Once these contribution scores are obtained, we can repeat the backward attribution procedure, counting each pass as one hop. All of these attribution passes operate on the same generated response, without regenerating text or updating the model. As the number of hops increases, token contribution scores and rankings may change. The backbone sets selected at different hops may therefore also differ.

Figure 6 in Appendix B.6 uses a fixed base student model and fixed responses to compare the backbone sets selected at Hop 0, Hop 1, and Hop 2. "Turnover" refers to the fraction of positions selected at an earlier hop that are no longer selected at another hop. It measures changes in selection across attribution depths, rather than across training stages. It also does not involve replacing student tokens with teacher tokens. The teacher always provides distributional supervision on student prefixes. Therefore, this metric should not be directly interpreted as a measure of convergence between the student and teacher distributions. Even if their predictions become closer, direct and recursive attribution still examine contribution relationships at different levels. The backbone sets selected at different hops are therefore not required to converge.


Question 3

We agree that a correct final answer does not guarantee correct intermediate reasoning. AAR does not treat answer correctness or attribution scores as labels for the correctness of the reasoning process. The student's answer is used only to identify where to emphasize supervision, while teacher distributions remain the learning targets. Thus, even if some flawed intermediate positions are selected as backbone positions, increasing their weights means placing greater emphasis on teacher supervision at those positions. Teacher supervision may reinforce this part of the derivation, but instead suppresses it when the token distributions disagree.

To further examine whether trajectories with incorrect reasoning or answers have negative effects under AAR, we added training experiments grouped by final-answer correctness. We compared three settings: a mixture of correct and incorrect trajectories, correct trajectories only, and incorrect trajectories only. We kept the number of samples actually used for parameter updates the same across these settings. This sample count was approximately equivalent to that used in 200 training steps of the main experiment. The best scores for the three AAR training settings and the OPD baseline in this comparison are shown below:

Training settingAvg@32 (%)Pass@32 (%)
AAR: mixed trajectories48.8980.28
AAR: correct trajectories only47.8274.72
AAR: incorrect trajectories only47.4769.44
OPD baseline46.8868.33

Mixed training achieved the highest scores on both metrics. AAR training using only trajectories with incorrect final answers also outperformed the OPD baseline in this comparison on both metrics.

We therefore believe that positions closely related to answer formation may still provide valuable opportunities for teacher correction, even when they involve incorrect reasoning. AAR selects where to place greater emphasis on teacher supervision. It does not endorse the correctness of the student's original predictions at those positions. The training value of these positions therefore cannot be judged solely by whether the final answer is correct.

Reviewer_B37FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Answer-Anchored Reweighting (AAR) for on-policy distillation. It uses the student's generated answer as an attribution anchor to rank tokens in its reasoning trajectory, upweights selected positions, and retains teacher supervision elsewhere. Experiments with a 1.5B student and a 7B teacher on three mathematical reasoning benchmarks report improvements over uniform OPD across four supervision-prefix lengths. At 16k, AAR improves macro-averaged Avg@32 from 48.09% to 52.29%, with approximately 59% higher measured training-step time.

Critical strengths
  1. A clear approach that preserves dense supervision. AAR changes the relative emphasis of teacher supervision rather than treating the student's answer as ground truth or discarding all unselected positions. Attribution failures fall back to uniform weighting, making the training procedure well specified.
  2. Consistent reported average gains and transparent cost reporting. AAR exceeds the corresponding uniform OPD baseline at all four evaluated supervision lengths. The paper also examines recursive depth, selection ratio, weighting strength, and distillation objectives, and explicitly reports the substantial attribution-related training overhead.
Critical weaknesses
  1. The ablations tune the method but do not establish its central attribution choice. The experiments vary attribution depth and weighting hyperparameters, but lack training controls using random selection, position-based selection, or simpler uncertainty/disagreement-based weighting under the same normalization and selection budget. The token-masking comparison on one selected trajectory measures answer-likelihood sensitivity, not the training value of the selected tokens. Moreover, attribution to the student's own answer need not identify positions where teacher supervision is most useful, especially when that answer is incorrect. The paper acknowledges this distinction, but does not directly validate it. Consequently, the results establish the value of the complete configuration more clearly than the necessity of answer-anchored tracing.

  2. The performance–compute trade-off and robustness remain insufficiently established. At 16k, measured step time rises from 53.1 to 84.6 seconds, yet there is no equal-GPU-budget OPD comparison. Evaluation covers only one teacher–student pair and three math benchmarks, without independent training-seed replication. Main results also select the best AAR hyperparameters at each length and peak-performance checkpoints, making the selection protocol important for interpreting the gains. The length experiments are useful but do not establish model- or data-scaling behavior: all AAR runs generate up to 16k, and its best reported accuracy is at 7k supervision rather than 16k. Thus, the findings do not yet show a broadly robust or compute-efficient advantage for learning from longer chains.

Questions for authors
  1. How does AAR compare with random, positional, and simpler token-weighting controls at the same selection ratio, weight multiplier, and normalization? Does the benefit differ between trajectories with correct and incorrect student answers?
  2. Does AAR retain its advantage when OPD receives the same GPU budget, and are the main gains stable across independent training runs? Any additional teacher–student pair would help assess robustness beyond the current setting.
  3. Which held-out data and criterion are used to select hyperparameters and peak checkpoints? Please also clarify whether OPD and AAR use identical rollout caps at each supervision-prefix length, since every AAR setting uses a 16k generation cap.
5 comments

We sincerely thank you for your thorough evaluation and valuable comments. We will include the additional experimental results and clarify the relevant points in the revised manuscript. Our responses to your comments are provided below. All scores below are reported as macro-averaged Avg@32 / Pass@32 (%).

Weakness 1 & Question 1

  1. Comparison with random selection and fixed-position selection.

We kept the selection ratio ρ=40\rho=40\\%, weight multiplier α=1.5\alpha=1.5, and within-response weight normalization unchanged in the 16k setting. We replaced answer-anchored attribution-based selection with selection of the first 40%, a random 40%, or the last 40% of tokens in each trajectory. All controls followed AAR's reweighting eligibility conditions and fallback rules. Each alternative selection strategy was applied only when the current trajectory met the conditions for AAR. Trajectories that required fallback continued to receive uniform supervision under standard OPD.

The results for these selection strategies, standard OPD, and AAR are shown below:

Selection strategyAvg@32 (%)Pass@32 (%)
First 40%50.6670.56
Random 40%48.7869.44
Last 40%47.5068.61
Standard OPD48.0969.72
AAR52.2972.78

Selecting earlier positions outperformed random selection, which in turn outperformed selecting later positions. AAR outperformed all these controls on both metrics. This shows that, in this setting, allocating supervision based on attribution to the current answer produced better training outcomes than consistently emphasizing a fixed segment or randomly assigning additional weights.

  1. Training outcomes with correct and incorrect trajectories.

We further compared AAR training using correct trajectories, incorrect trajectories, and a mixture of both. We kept the number of samples actually used for parameter updates the same across these settings. This sample count was approximately equivalent to that used in 200 training steps of the main experiment.

The best scores for the three AAR training settings and the OPD baseline in this comparison are shown below:

Training settingAvg@32 (%)Pass@32 (%)
AAR: mixed trajectories48.8980.28
AAR: correct trajectories only47.8274.72
AAR: incorrect trajectories only47.4769.44
OPD baseline46.8868.33

Mixed training achieved the highest scores on both metrics. AAR training using only trajectories with incorrect final answers also outperformed the OPD baseline in this comparison on both metrics.

We therefore believe that positions closely related to answer formation may still provide valuable opportunities for teacher correction, even when they involve incorrect reasoning. AAR selects where to place greater emphasis on teacher supervision. It does not endorse the correctness of the student's original predictions at those positions. The training value of these positions therefore cannot be judged solely by whether the final answer is correct.

Weakness 2 & Question 2

  1. Comparison under the same GPU budget.

We used the training cost of one epoch of standard OPD in the 16k setting as the common budget cap.

According to the measurements in Appendix D, the mean step times on 8 GPUs were:

MethodMean step time (s)
OPD53.1
AAR84.6

One epoch of standard OPD had a cumulative training cost of 102.6 GPU-hours. At AAR's mean cost per step, this budget would support approximately 545 complete training steps. The latest saved checkpoint within this step limit was at step 500. We used it as the upper limit for AAR checkpoint selection. Thus, OPD retained its best result over the full epoch in the main experiment, while AAR selected its best checkpoint only from steps 0 through 500.

Under this budget constraint, OPD and AAR achieved the following scores:

MethodAvg@32 (%)Pass@32 (%)
OPD48.0969.72
AAR50.7678.06

AAR improved Avg@32 and Pass@32 by 2.67 and 8.34 percentage points, respectively. This shows that AAR maintained a performance advantage without exceeding OPD's training budget, even after accounting for attribution computation and reducing AAR's training steps accordingly.

  1. Stability across independent training runs.

We independently repeated 16k AAR training with two additional random seeds. Including the original main experiment, the three independent training runs achieved Avg@32 of 52.28±0.2752.28\pm0.27 and Pass@32 of 74.17±1.5474.17\pm1.54. Here, ±\pm denotes the sample standard deviation across training random seeds. These results support AAR's consistent performance advantage over standard OPD across different training random seeds.

  1. An additional teacher-student pair.

We added an experiment with one additional teacher-student pair. We kept DeepSeek-R1-Distill-Qwen-1.5B as the student and replaced the teacher with DeepSeek-R1-Distill-Qwen-7B.

With this pair, the 16k results were:

MethodAvg@32 (%)Pass@32 (%)
Standard OPD40.9760.28
AAR43.8563.06

AAR improved Avg@32 and Pass@32 by 2.88 and 2.78 percentage points, respectively. AAR also outperformed standard OPD with this additional pair, further supporting its effectiveness with different teacher models.

Weakness 2 & Question 3

  1. Checkpoint and hyperparameter selection.

Thank you for pointing out that the experimental setup was not described clearly enough in the original manuscript. We split the DAPO-Math-17K data used in this study into training and validation sets in an 80%/20% ratio. Appendix D of the original manuscript reports 869 logged steps. For our one-epoch run with a global batch size of 16, this corresponds to 869×16=13,904869\times16=13,904 training samples. The remaining approximately 3k samples formed our held-out 20% validation set.

Hyperparameters were selected through grid search using accuracy on the held-out validation set as the selection criterion. For each training run, we also selected the checkpoint with the highest validation accuracy and used that checkpoint to report Avg@32 and Pass@32 on the test sets. Neither hyperparameter selection nor checkpoint selection was based on AIME/AMC test scores. The phrase "peak performance" in the original manuscript refers to the best validation performance.

  1. Peak performance.

Expanding the supervision range can cover more reasoning steps, but there is also a trade-off between supervision coverage and supervision utility. Therefore, peak performance at an intermediate response length is not contradictory. We further emphasize our central claim: AAR provides empirical evidence for the learning value of later supervision and offers an alternative to shortening responses for improving long-chain distillation.

In addition, Table 6 in Appendix B.3 specifies that the main 7k experiment used α=2.0\alpha=2.0, while the other lengths used α=1.5\alpha=1.5. The "Additional 7k Results" paragraph in the same section reports results with the common configuration H=1,ρ=40H=1,\rho=40\\%,\alpha=1.5,K=20.

With these common AAR hyperparameters, the results were:

MethodAvg@32 (%)Pass@32 (%)
AAR (7K)51.1171.67
AAR (12K)51.3572.78
AAR (16K)52.2972.78
Standard OPD (7K)50.6970.56

At 7k, AAR also outperformed standard OPD on both metrics under this common configuration. Under this configuration, the 7k Avg@32 score was lower than the scores at 12k and 16k. Thus, the 7k peak in the main table should also be understood in light of its different weighting configuration.

  1. Generation length cap.

Regarding the length settings, 3k, 7k, 12k, and 16k in the paper denote the maximum prefix length covered by the distillation loss. For standard OPD, the generation cap matches this range, and unfinished responses are truncated when they reach the corresponding cap. AAR uses a fixed generation cap of 16k to obtain a final answer for attribution whenever possible. However, its distillation loss is still computed only within the corresponding prefix range.

Comments on the paper

4 comments

Dear Reviewers,

The authors have posted their responses to the reviews. Please go through their rebuttal at your earliest convenience, and engage in a discussion or leave any follow-up comments.

The discussion period ends on November 18.

Thanks,
Area Chair

We have summarized the reviewers’ questions, our responses, and the additional experimental results below to facilitate the reviewers’ and AC’s review and assessment.

1. Core Contributions

  • An answer-anchored method for allocating supervision, with consistent gains across supervision lengths. We propose AAR, which uses attribution anchored to the student’s generated answer to identify where to emphasize supervision. It reweights selected positions while retaining teacher supervision at other valid positions. The method allocates supervision according to each position’s relevance to answer formation in the current trajectory, with within-response normalization controlling the redistribution of supervision weights. Across three mathematical reasoning benchmarks and four supervision lengths from 3K to 16K, AAR achieves higher macro-averaged Avg@32 than the corresponding standard OPD baselines, demonstrating gains across the evaluated length settings.

  • Empirical support for the learning value of later supervision in long chains, offering an alternative to shortening the supervision range. Prior studies have observed that teacher supervision may become less effective as student-generated trajectories grow longer, reducing the gains from OPD and motivating approaches that shorten trajectories or supervision ranges. We argue that token position is only a coarse proxy for supervision utility: later positions may still offer valuable learning opportunities, and how supervision is allocated matters. With fixed AAR hyperparameters H=1, ρ=40%, α=1.5, and K=20, extending the supervision range from 7K to 12K and 16K decreases standard OPD’s Avg@32 from 50.69% to 48.96% and 48.09%, while AAR’s Avg@32 increases from 51.11% to 51.35% and 52.29%. This comparison shows that appropriate supervision allocation can improve training outcomes when the supervision range is extended, providing empirical support for learning from later positions in long trajectories and a constructive direction for long-chain distillation.

2. Key Clarifications

  • Model selection and evaluation criteria. We split the data into training and validation sets in an 80%/20% ratio, and select both hyperparameters and checkpoints using held-out validation accuracy. Macro-averaged Avg@32 is the primary metric for test-performance comparisons, while Avg@32 and Pass@32 are reported from the same checkpoint, clearly distinguishing model selection from test evaluation. See the Introduction and Section 4.1.

  • Generation caps and supervision lengths. The length settings in the paper specify the prefix range covered by the distillation loss. Standard OPD’s generation cap follows this setting; AAR uses a fixed generation cap of 16K, while its loss remains restricted to the corresponding prefix range. See Section 4.1.

  • Results under common hyperparameters and the scope of our claim about supervision length. The results already reported in Appendix B.3 of the original manuscript show that, under common hyperparameters, AAR outperforms the corresponding OPD baselines at all four supervision lengths and achieves its highest macro-averaged Avg@32 at 16K. The 7K peak in the main table uses a different weighting configuration. We clarify that one of our central claims is that appropriate supervision allocation can exploit learning opportunities at later positions; it does not require performance to increase monotonically with length.

  • Attribution targets and the meaning of supervision. The student’s answer identifies where to emphasize supervision and is not used as a correctness label; teacher distributions remain the learning targets. The general attribution operator is also not restricted to a particular answer format. The boxed-answer target is the implementation used in our mathematical reasoning experiments, while applications to other tasks require a meaningful attribution target. See Section 3.1 and Appendices B.8 and F.

  • Small gains at α=0.5. AAR’s selected set identifies positions for supervision emphasis; it does not classify the remaining positions as having no learning value. With α=0.5, the relative weights of high-attribution positions decrease, giving greater relative emphasis to the remaining positions. These positions may still contain valuable teacher guidance, so some improvement is consistent with the method’s design. However, the gain in the primary metric, Avg@32, is small: only 0.38 percentage points over OPD. By comparison, the default α=1.5 configuration reaches 52.29%, an improvement of 4.20 percentage points, showing that appropriately emphasizing high-attribution positions yields a larger gain in average performance. Additional experimental settings and repeated training runs further support AAR’s effectiveness and robustness. See Section 5, Table 3, and the additional experiments below.

3. Additional Experiments and Analysis

  • Token selection strategy comparisons. With the selection ratio, weight multiplier, and normalization rules held fixed, AAR achieves higher Avg@32 than random selection and fixed-position selection (the first/last 40%). See Appendix B.7, Table 8.

  • Training grouped by final-answer correctness. We compared training with correct-answer trajectories only, incorrect-answer trajectories only, and a mixture of both, while keeping the number of samples actually used for parameter updates the same. AAR trained only on incorrect-answer trajectories still outperforms the corresponding OPD baseline, while mixed training achieves the highest scores. We believe these results indicate that, regardless of whether the final answer is correct, a trajectory may contain positions with learning value; positions involving incorrect reasoning may also offer opportunities for teacher correction. See Appendix B.8, Table 9.

  • Comparison under a common GPU budget. Under a common budget cap of 102.6 GPU-hours, AAR still exceeds OPD by 2.67 and 8.34 percentage points in Avg@32 and Pass@32, respectively, after accounting for attribution overhead and reducing its training steps accordingly. We also acknowledge that AAR requires more computational resources under the full one-epoch configuration, and attribution computation incurs additional GPU memory overhead. This reflects a trade-off between model performance, training time, and GPU memory requirements. See Appendix D, Table 11.

  • Independent training runs and an additional teacher-student pair. With two additional random seeds for the main teacher-student pair, the three independent training runs achieve Avg@32 of 52.28±0.27% and Pass@32 of 74.17±1.54%, where ± denotes the sample standard deviation. In addition, when keeping the student model unchanged and replacing the teacher, AAR exceeds OPD by 2.88 and 2.78 percentage points in Avg@32 and Pass@32, respectively. See Appendices B.9 and B.10, and Table 10.

  • Aggregated masking analysis. We have replaced the original single-trajectory masking figure with aggregated masking results over valid trajectories from the training set. The overall conclusions remain unchanged. See Section 3.1, Figure 1.

Thank you for reviewing our responses further. We would like to clarify the purpose of the selection-strategy comparison: to test whether attribution-based selection itself provides additional benefits when the selection ratio and weighting configuration are held fixed. We fixed the selection ratio (40%), weight multiplier (1.5), within-response normalization, and eligibility and fallback rules, changing only the token-selection strategy. Under these conditions, AAR achieved 52.29% Avg@32, compared with 50.66%, 48.78%, and 47.50% for first-40%, random-40%, and last-40% selection, respectively.

The effect of the selection rule under a fixed configuration and the best performance of each independently tuned strategy are two distinct experimental questions. Our added experiment directly addresses the former: attribution-based selection improves performance without changing the selection ratio or weighting strength across strategies. We agree that independent tuning would extend the comparison to the latter question, while the existing controlled comparison already provides evidence for the effectiveness of attribution-based selection at the shared configuration.

In addition, we addressed your other concerns through a common-GPU-budget comparison, independent training runs, and an additional teacher–student pair evaluation. These experiments show that AAR retains its advantage without exceeding OPD's training budget, that its gains are reproducible across training seeds, and that they also hold with a different teacher model. We would therefore appreciate your considering the remaining question of strategy-specific tuning alongside the evidence for effectiveness and robustness established by these complementary experiments in your final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.