- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper starts from the observation that student-generated prefixes drift away from the teacher's distribution as reasoning proceeds, which has led prior work to shorten training rollouts, and argues that token position is only a coarse proxy for supervision utility. It proposes Answer-Anchored Reweighting: take the student's generated boxed-answer span as an anchor, run span-based recursive trace attribution over the whole trajectory, rank generated positions by attribution score within each response, mark the top-ρ fraction as "backbone" tokens whose distillation weight is multiplied by α while all other valid positions stay at 1, then normalise the weighted loss within each response. Weights are recomputed each online batch. With DeepSeek-R1-Distill-Qwen-1.5B as student and Skywork-OR1-Math-7B as teacher, reverse KL over the teacher's top-20 support, trained on DAPO-Math-17K, AAR is reported to beat standard OPD in macro-averaged Avg@32 at all four supervision lengths, reaching 52.29% at 16k against 48.09% for OPD.
Critical strengths
-
The question is well posed. Asking how to allocate supervision inside a long trajectory, instead of just truncating it, is more constructive than the prevailing shorten-the-rollout response, and the paper is explicit that this is the contribution.
-
Rewriting the normalised objective as a convex mixture, ℓAAR = (1−λ)ℓU + λℓS with λ = q(α−1)/(1+q(α−1)), is a genuinely useful piece of analysis. It turns ρ and α into a single redistribution of coefficient mass rather than two unrelated knobs, and explains why expanding the selected set does not monotonically strengthen emphasis.
Critical weaknesses
-
The main comparison is a maximum over a hyperparameter sweep against a single-configuration baseline, with checkpoints chosen on the evaluation sets. Section 1 states the main table uses "the best-performing AAR hyperparameter configuration at each response length", while OPD has no corresponding knobs. Section 4.2 states the reported checkpoint is "the checkpoint at peak performance during training", and no validation split is described anywhere. The 16k ablation over α alone spans 48.47 to 52.29, about 3.8 points, which is the same order as the headline +4.20 over OPD. As reported I cannot separate the method's effect from the selection procedure.
-
No seeds, small benchmarks, and inconsistent metric choice. AIME24 and AIME25 have 30 problems each. In Table 2 the chosen reference configuration (ρ = 40%) is best on Avg@32 but clearly worst on Pass@32 (72.78 against 79.44 at ρ = 20%). In Table 3 the α ordering is 1.5 > 2.5 > 2.0 > 0.5, non-monotonic in a way that reads like noise rather than a real optimum. Worth flagging: α = 0.5, which down-weights the backbone tokens, gives 48.47, essentially tied with OPD's 48.09. If the attribution found genuinely high-utility positions, inverting the emphasis should hurt more than that.
-
The attribution itself is barely validated. The deletion sanity check in Figure 1 runs on "a selected trajectory", a single hand-picked example, with no aggregate deletion curve over the dataset. And since the anchor is the student's own generated answer whether or not it is correct, the method up-weights exactly the tokens most responsible for producing a wrong answer on failing rollouts. No result is split by correct versus incorrect rollouts, which seems like the first thing the design calls for.
Questions for authors
- Is there a held-out validation set used for checkpoint selection, or is the peak checkpoint picked directly on AIME/AMC?
- What do the numbers look like for a single fixed (H, ρ, α) applied across all four response lengths, against OPD?
- Does restricting backbone up-weighting to correct rollouts, or down-weighting it on incorrect ones, change the result?
4 comments
Weakness 3 & Question 3
- Attribution and token-masking experiments.
We added attribution and token-masking experiments across the training set. We used the student model to generate responses for the entire training set, with a fixed generation cap of 16k tokens. After excluding responses truncated at this cap, we analyzed the remaining approximately 13,000 complete trajectories and aggregated the masking curves. The aggregated results were broadly consistent with the overall trends for the single trajectory in the original manuscript. When the same fraction of pre-answer tokens was masked, masking high-attribution tokens first generally reduced the likelihood of the fixed answer suffix more substantially than random masking or masking low-attribution tokens first. This showed a stronger deletion effect. The local differences were as follows: the likelihood retention curve showed a steeper initial decline when high-attribution tokens were masked first. For masking high-attribution tokens first, normalized deletion response values were relatively higher in the middle and later portions of the curve. In the revised manuscript, we will replace the original figure with a new figure showing the aggregated results across the training set.
- Additional experiments based on answer correctness.
To examine whether trajectories with incorrect reasoning or answers have negative effects under AAR, we added training experiments that separated trajectories by final-answer correctness. We compared three settings: a mixture of correct and incorrect trajectories, correct trajectories only, and incorrect trajectories only. We kept the number of samples actually used for parameter updates the same across these settings. This sample count was approximately equivalent to that used in 200 training steps of the main experiment. The best scores for the three AAR training settings and the OPD baseline in this comparison are shown below:
| Training setting | Avg@32 (%) | Pass@32 (%) |
|---|---|---|
| AAR: mixed trajectories | 48.89 | 80.28 |
| AAR: correct trajectories only | 47.82 | 74.72 |
| AAR: incorrect trajectories only | 47.47 | 69.44 |
| OPD baseline | 46.88 | 68.33 |
Mixed training achieved the highest scores on both metrics. AAR training using only trajectories with incorrect final answers also outperformed the OPD baseline in this comparison on both metrics.
AAR does not treat answer correctness or attribution scores as labels for the correctness of the reasoning process. The student's answer is used only to identify where to emphasize supervision, while teacher distributions remain the learning targets. Thus, even if some flawed intermediate positions are selected as backbone positions, increasing their weights places greater emphasis on teacher supervision at those positions. It does not reinforce the student's generated derivation as a correct example. Teacher supervision may reinforce this part of the derivation, but instead suppresses it when the token distributions disagree. We therefore believe that training value cannot be judged solely by whether the final answer is correct.
Question 2
Appendix B.3, "Selected Configurations for the Main Results Table," in the original manuscript already lists the configurations used in the main table. The "Additional 7k Results" paragraph after Table 6 also reports supplementary results with the same hyperparameters. Specifically, the main experiments at 3k, 12k, and 16k all used , while the main 7k experiment used with the other hyperparameters unchanged.
With these common AAR hyperparameters, the results were:
| Method | Avg@32 (%) | Pass@32 (%) |
|---|---|---|
| AAR (3K) | 45.68 | 74.72 |
| AAR (7K) | 51.11 | 71.67 |
| AAR (12K) | 51.35 | 72.78 |
| AAR (16K) | 52.29 | 72.78 |
| Standard OPD (7K) | 50.69 | 70.56 |
AAR outperformed standard OPD on both metrics. Together with the main results at the other supervision lengths, these results show that AAR still outperformed standard OPD on both metrics when were used consistently across all four supervision lengths.
Summary of Manuscript Revisions
Thank you again for your valuable comments. For ease of reference, we have incorporated the updates discussed in our responses into the revised manuscript and highlighted the relevant changes in blue. Validation-based selection is clarified in the Introduction and Section 4.1, with the training/validation split and training configuration specified in Section 4.1. The role of macro-averaged Avg@32 as the primary comparison metric is also clarified in Section 4.1. Independent training runs are reported in Appendix B.9, with a summary in Section 4.2. The aggregated masking analysis is presented in Section 3.1 and the updated Figure 1. Training comparisons grouped by final-answer correctness and the corresponding discussion are added in Section 6 and Appendix B.8 (Table 9).
We would be grateful to hear whether our responses and manuscript revisions have addressed your concerns and, if so, whether you would consider revising your rating upward. If any questions remain, we would be happy to respond promptly.
We sincerely thank you for your thorough evaluation and valuable comments. We will include the additional experimental results and clarify the relevant points in the revised manuscript. Our responses to your comments are provided below. All scores below are reported as macro-averaged Avg@32 / Pass@32 (%).
Weakness 1 & Question 1
Thank you for pointing out that the experimental setup was not described clearly enough in the original manuscript. We split the DAPO-Math-17K data used in this study into training and validation sets in an 80%/20% ratio. Appendix D of the original manuscript reports 869 logged steps. For our one-epoch run with a global batch size of 16, this corresponds to 869×16=13,904 training samples. The remaining approximately 3k samples formed our held-out 20% validation set.
Hyperparameters were selected through grid search using accuracy on the held-out validation set as the selection criterion. For each training run, we also selected the checkpoint with the highest validation accuracy and used that checkpoint to report Avg@32 and Pass@32 on the test sets. Neither hyperparameter selection nor checkpoint selection was based on AIME/AMC test scores. The phrase "peak performance" in the original manuscript refers to the best validation performance.
Weakness 2
We independently repeated 16k AAR training with two additional random seeds. Including the original main experiment, the three independent training runs achieved Avg@32 of 52.28±0.27 and Pass@32 of 74.17±1.54. Here, ± denotes the sample standard deviation across training random seeds. These results support AAR's consistent performance advantage over standard OPD across different training random seeds.
We use macro-averaged Avg@32 as the primary metric for performance comparisons and for identifying the best results. This is already reflected in the note to Table 1 in the original manuscript. We will state this primary metric more clearly in the revised manuscript and improve the description of the corresponding selection criteria to avoid ambiguity.
In the 16k ablation with H=1,ρ=40 fixed, α=1.5 achieved the highest performance among the tested values for the current teacher-student pair. The parameter α controls the relative supervision strength on backbone positions and the remaining positions. Moderate weighting helps balance selective emphasis and overall learning. This balance does not require performance to vary strictly monotonically with α.
Our top-40% selection specifies a budget for supervision emphasis. It does not directly divide tokens into "useful" and "useless" groups. Thus, although α=0.5 reduces the relative weights of the highest-attribution positions, it increases the relative emphasis on the remaining positions. These still include tokens with moderate attribution that may provide useful teacher guidance. This redistribution may yield some benefits and does not contradict AAR's design.