- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces TAPER (Teacher-guided, Adaptive-length Policy Optimization for Efficient Reasoning), a relative length reward added to the objective of GRPO, and token-level distillation from a larger teacher using a combination of forward and reverse KL objective at different output regions. TAPER is evaluated on Qwen3.5-4B and Qwen3.5-9B across five benchmarks spanning mathematical, knowledge-intensive, and logical reasoning.
Critical strengths
- Writing: the paper is well-written and easy to follow.
- Diverse evaluation tasks: the method is tested on Qwen3.5-4B and 9B across diverse types of tasks spanning mathematical, knowledge-intensive, and logical reasoning.
- The behavior analysis provides interesting insights on what the model learned through the proposed training algorithm.
Critical weaknesses
- Evaluation metric of inference cost: the paper uses the "thinking tokens" as the only metric of inference cost. The metric counts only tokens within the thinking region of the whole output. However, the number of tokens spent in the "answer" region can also be substantial, and sometimes even higher than the number of tokens in the thinking region (as shown in the examples in Appendix J). In addition, GR3 and LASER-DE are configured to optimize full response length, while they are compared against the proposed method using "thinking tokens". A better measurement of the overall inference cost may be the full response length.
- Impact of the length reward versus the distillation: among all efficient reasoning methods, the proposed method is the only one that uses a 122B teacher model, which makes it difficult to isolate the impact of a better length reward design versus the distillation from a stronger teacher model. A comparison between TAPER, LASER-DE + distillation would be helpful.
- Design choices need to be justified. The paper has many design choices that need to be justified further. For example, the top-20 renormalization used in both forward and reverse KL divergences. It doesn't constrain student probability mass outside the teacher's top 20 tokens, which is contradictory to the motivation of using reverse KL. Also, the choice of length measurement in training, and the choice of joint GRPO and on-policy distillation versus distillation and GRPO sequentially.
- Mismatch between training and evaluation: during training, thinking length falls to 46 tokens, with a median of 98.7 over the last ten steps. At evaluation, MATH500 averages 1,330 thinking tokens. This huge gap in thinking length needs further explanation and analysis.
Questions for authors
See weaknesses.
7 comments
3. Rationale for the design choices
3.1 Top-20 renormalization and reverse KL
Thank you for raising this concern. We chose top-20 as a practical way to reduce teacher logit transfer and KL computation costs while keeping relative probability information among multiple candidates. Fu et al. (2026) [1] use a similar local matching approach: they select the teacher's top-K candidates, renormalize both distributions within this set, and apply reverse KL matching. Under the same top-p rollout conditions, their method reaches 23.6% on AIME24 avg@32, compared with 21.6% for sampled-token OPD. This provides empirical support for this form of local supervision.
You are right about the limitation: once both distributions are renormalized, the KL terms compare conditional distributions within the retained support and do not directly constrain the student's total probability mass outside it. Our interpretation of reverse KL therefore applies only within the retained set. It weights discrepancies by the student's conditional probabilities and discourages relative over-allocation to candidates with low teacher conditional probability. Forward KL uses the teacher's conditional probabilities as weights and encourages coverage of teacher-supported candidates. This is why we use different KL directions in the thinking and answer regions, and test that choice through the Uniform Reverse KL ablation under the same sparse settings.
We should have made this distinction clearer in the original paper. We have now clarified it in Section 3.2 and Appendix E.3, and describe top-20 as a trade-off between computation and supervisory information. We do not claim that this objective is equivalent to full-vocabulary KL, that this support size is optimal, or that truncation is lossless.
3.2 Joint versus sequential training
Thank you for suggesting this comparison. It gave us a direct way to test our joint training design and raised a broader question: How should length compression and teacher supervision be scheduled to make better use of their complementarity? Our starting idea was that the length reward could encourage shorter reasoning while continuous teacher guidance could help preserve its quality.
Following your suggestion, we added two sequential schedules, GRPO→OPD and OPD→GRPO, with 56 updates to each stage and 112 updates in total. The training data and total update budgets match joint training. The GRPO stage keeps the same length reward and competence gate, and the OPD stage uses the same region-aware asymmetric distillation objective.
Table R2: Joint versus sequential training. Each cell reports accuracy (%) / average thinking tokens. All three schedules contain 112 optimizer updates. MATH500 uses avg@4; AIME25/26 use avg@16.
| Training schedule | Updates with OPD | MATH500 | AIME25 | AIME26 |
|---|---|---|---|---|
| GRPO→OPD (56 + 56) | 56 | 92.4 / 1,640 | 55.8 / 10,391 | 66.3 / 10,817 |
| OPD→GRPO (56 + 56) | 56 | 92.8 / 1,539 | 56.3 / 10,012 | 66.5 / 10,435 |
| Joint TAPER | 112 | 94.7 / 1,330 | 59.2 / 9,461 | 69.0 / 10,168 |
Joint TAPER has higher accuracy and shorter thinking traces on all three benchmarks. Compared with the two sequential schedules, accuracy is 1.9–3.4 percentage points higher and thinking tokens are 2.6%–18.9% lower. These results favor joint training within the total update budget we tested. Full results are in Section 4.3, Figure 2, and Appendices F and M.
We think continuous teacher guidance may help the student preserve generation quality as its reasoning becomes shorter. However, joint training applies both objectives in all 112 updates, while each sequential schedule applies each objective in 56 updates. Teacher computation and the number of updates per objective are therefore not matched. This means we cannot attribute the gain solely to joint optimization itself. We would like to better understand, with teacher computation matched, when sequential training can match or outperform joint training and how stage lengths, training order, and objective weights affect that outcome.
Reference
[1] Fu et al. Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes, 2026. Paper.
4. The gap between training and evaluation thinking lengths
Thank you for pointing out this gap. We did not explain these statistics clearly enough in the original paper: 46.4 tokens is the mean thinking length of training rollouts at step 112; 98.7 tokens is the median of the per-step mean thinking lengths over the final ten steps; and 1,330 tokens is the mean thinking length of the final checkpoint on MATH500 under avg@4. The first two describe generation on training examples, while the third describes test behavior. They are not directly comparable, but we agree that clarifying this distinction alone does not explain the large gap.
Training and evaluation also differ in question selection and generation limits. Training questions are filtered to at most 500 input tokens, and the maximum completion length is 6,144 tokens. MATH500 does not use this filter and allows up to 20,480 completion tokens. These differences may affect length, but we do not have enough evidence to attribute the gap to any one of them.
To check whether the trend in the training logs also holds during evaluation, we evaluated all 24 checkpoints on a fixed set of 100 questions using the same decoding, grading, and avg@4 protocol as MATH500. From step 0 to step 112:
- Average thinking length decreases from 4,429 to 1,389 tokens (−68.6%).
- Accuracy increases from 85.5% to 93.75% (+8.25 percentage points).
Shorter thinking traces and higher accuracy also appear on a fixed evaluation set under a unified protocol, not just in the training rollouts. The final checkpoint's mean thinking length on this set is 1,389 tokens, close to the 1,330 tokens observed on the full MATH500 test. We did not use these checkpoint results for model selection; all main results use the final checkpoint.
This added analysis confirms that compression also occurs during evaluation, but it still does not fully explain the training–evaluation length gap. We have clarified the statistics and provided the complete curves in Sections 4.1 and 5.3 and Appendices A–B, G, and L.
Thank you for the helpful suggestions. We have added full output cost statistics, comparisons using the same teacher, sequential training ablations, and checkpoint evaluations. We have also clarified the reasons behind our design choices. We address each point below.
1. Full response length and inference cost
We agree that full response length is an important part of assessing generation cost. We now report average thinking and total output lengths for all compared methods, along with the analysis, in Section 4.2 and Appendix K (Tables K1–K2).
When we also count non-thinking output, TAPER still uses 20.2%–66.9% fewer total output tokens than GRPO and 11.0%–46.0% fewer than GR³ across all ten model–benchmark combinations. LASER-DE produces shorter outputs, while its accuracy is lower than TAPER's in every setting. These comparisons still support our main findings about the accuracy–generation-cost trade-off. For example, non-thinking output accounts for 13.0% of TAPER's total output on the 9B MATH500 setting.
Using thinking length for the reward was a deliberate design choice. We want to remove redundant reasoning without putting extra pressure on the model to shorten necessary explanations or compromise the formatting and clarity of its final answer. That is why we use thinking length to define the compression objective during training, and also report full response length during evaluation to compare the accuracy–cost trade-offs of different methods.
2. Contributions of the length reward and teacher distillation
Your suggestion, together with related comments from other reviewers, led us to add GR³ comparisons to help distinguish teacher supervision from length optimization. We chose GR³ because it also uses relative length information within each rollout group. Both added variants use the same Qwen3.5-122B-A10B teacher, 9B student initialization, MATH training data, and 112 optimizer updates as TAPER.
Table R1: GR³ + OPD comparisons using the same teacher. Each cell reports accuracy (%) / average thinking tokens. MATH500 uses avg@4; AIME25/26 use avg@16.
Both OPD variants improve GR³'s accuracy, but neither makes its thinking traces shorter. Compared with Uniform RKL OPD, Region-aware Asymmetric OPD improves accuracy by 0.7, 1.2, and 1.0 percentage points, with 2.0%, 1.4%, and 0.7% fewer thinking tokens, respectively. TAPER has the highest observed accuracy and the shortest thinking length on all three benchmarks in this comparison.
Adding the same teacher and OPD objective to GR³ does not reproduce TAPER's accuracy–length trade-off in these settings. One possible reason is that TAPER's reward design works better with region-aware asymmetric OPD. The relative thinking-length reward and competence gate control the pressure to shorten thinking traces, while OPD leaves room for exploration in the thinking region and emphasizes teacher supervision in the answer region. Together, these components may make it easier to shorten reasoning while preserving problem-solving ability. We see the results as support for the two components working together, but they do not on their own establish the specific mechanism. Full settings and results are in Section 4.3 and Appendices F and N.