acceptodds
Under review as a conference paper at ICLR 2027

TAPER: Teacher-guided, Adaptive-length Policy optimization for Efficient Reasoning

Abstract

Explicit multi-step reasoning improves the capability and reliability of large language models, but at considerable computational cost: traces are often dominated by redundant verification and unproductive backtracking. Because useful computation is entangled with this redundancy, indiscriminate shortening sacrifices both; what can be safely removed depends on how reliably the model solves the problem. We present **TAPER (Teacher-guided, Adaptive-length Policy Optimization for Efficient Reasoning)**, an on-policy framework that treats how much a model reasons and how it does so as a single learning problem. At the trajectory level, an accuracy-adaptive relative-length objective gauges the model's current competence on each problem from its own online samples, adjusting compression strength accordingly. At the token level, region-aware asymmetric teacher supervision lets a stronger teacher guide the student's own generation, with distinct objectives for the reasoning and answer regions; the two objectives are optimized jointly on the same trajectories in each update. Experiments on Qwen3.5-4B and Qwen3.5-9B across five benchmarks spanning mathematical, knowledge-intensive, and logical reasoning, against six baselines, show that TAPER improves accuracy over GRPO by 1.2–7.9 percentage points while using 18.4%–66.4% fewer tokens. Behavioral analysis further shows that TAPER reshapes the model's reasoning behavior in a difficulty-dependent manner: it reallocates the reasoning budget toward productive advancement and away from unnecessary redundancy, with the proportion of the two adjusted according to the model's competence on each problem.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, efficient reasoning, reinforcement learning, on-policy distillation, reasoning length compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Ae5FOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TAPER (Teacher-guided, Adaptive-length Policy Optimization for Efficient Reasoning), a relative length reward added to the objective of GRPO, and token-level distillation from a larger teacher using a combination of forward and reverse KL objective at different output regions. TAPER is evaluated on Qwen3.5-4B and Qwen3.5-9B across five benchmarks spanning mathematical, knowledge-intensive, and logical reasoning.

Critical strengths
  1. Writing: the paper is well-written and easy to follow.
  2. Diverse evaluation tasks: the method is tested on Qwen3.5-4B and 9B across diverse types of tasks spanning mathematical, knowledge-intensive, and logical reasoning.
  3. The behavior analysis provides interesting insights on what the model learned through the proposed training algorithm.
Critical weaknesses
  1. Evaluation metric of inference cost: the paper uses the "thinking tokens" as the only metric of inference cost. The metric counts only tokens within the thinking region of the whole output. However, the number of tokens spent in the "answer" region can also be substantial, and sometimes even higher than the number of tokens in the thinking region (as shown in the examples in Appendix J). In addition, GR3 and LASER-DE are configured to optimize full response length, while they are compared against the proposed method using "thinking tokens". A better measurement of the overall inference cost may be the full response length.
  2. Impact of the length reward versus the distillation: among all efficient reasoning methods, the proposed method is the only one that uses a 122B teacher model, which makes it difficult to isolate the impact of a better length reward design versus the distillation from a stronger teacher model. A comparison between TAPER, LASER-DE + distillation would be helpful.
  3. Design choices need to be justified. The paper has many design choices that need to be justified further. For example, the top-20 renormalization used in both forward and reverse KL divergences. It doesn't constrain student probability mass outside the teacher's top 20 tokens, which is contradictory to the motivation of using reverse KL. Also, the choice of length measurement in training, and the choice of joint GRPO and on-policy distillation versus distillation and GRPO sequentially.
  4. Mismatch between training and evaluation: during training, thinking length falls to 46 tokens, with a median of 98.7 over the last ten steps. At evaluation, MATH500 averages 1,330 thinking tokens. This huge gap in thinking length needs further explanation and analysis.
Questions for authors

See weaknesses.

7 comments

Thank you for the helpful suggestions. We have added full output cost statistics, comparisons using the same teacher, sequential training ablations, and checkpoint evaluations. We have also clarified the reasons behind our design choices. We address each point below.

1. Full response length and inference cost

We agree that full response length is an important part of assessing generation cost. We now report average thinking and total output lengths for all compared methods, along with the analysis, in Section 4.2 and Appendix K (Tables K1–K2).

When we also count non-thinking output, TAPER still uses 20.2%–66.9% fewer total output tokens than GRPO and 11.0%–46.0% fewer than GR³ across all ten model–benchmark combinations. LASER-DE produces shorter outputs, while its accuracy is lower than TAPER's in every setting. These comparisons still support our main findings about the accuracy–generation-cost trade-off. For example, non-thinking output accounts for 13.0% of TAPER's total output on the 9B MATH500 setting.

Using thinking length for the reward was a deliberate design choice. We want to remove redundant reasoning without putting extra pressure on the model to shorten necessary explanations or compromise the formatting and clarity of its final answer. That is why we use thinking length to define the compression objective during training, and also report full response length during evaluation to compare the accuracy–cost trade-offs of different methods.

2. Contributions of the length reward and teacher distillation

Your suggestion, together with related comments from other reviewers, led us to add GR³ comparisons to help distinguish teacher supervision from length optimization. We chose GR³ because it also uses relative length information within each rollout group. Both added variants use the same Qwen3.5-122B-A10B teacher, 9B student initialization, MATH training data, and 112 optimizer updates as TAPER.

Table R1: GR³ + OPD comparisons using the same teacher. Each cell reports accuracy (%) / average thinking tokens. MATH500 uses avg@4; AIME25/26 use avg@16.

MethodMATH500AIME25AIME26
GR³93.2 / 2,40157.1 / 11,70966.7 / 13,167
GR³ + Uniform RKL OPD93.8 / 2,65557.5 / 12,44167.1 / 13,332
GR³ + Region-aware Asymmetric OPD94.5 / 2,60158.7 / 12,26468.1 / 13,243
TAPER94.7 / 1,33059.2 / 9,46169.0 / 10,168

Both OPD variants improve GR³'s accuracy, but neither makes its thinking traces shorter. Compared with Uniform RKL OPD, Region-aware Asymmetric OPD improves accuracy by 0.7, 1.2, and 1.0 percentage points, with 2.0%, 1.4%, and 0.7% fewer thinking tokens, respectively. TAPER has the highest observed accuracy and the shortest thinking length on all three benchmarks in this comparison.

Adding the same teacher and OPD objective to GR³ does not reproduce TAPER's accuracy–length trade-off in these settings. One possible reason is that TAPER's reward design works better with region-aware asymmetric OPD. The relative thinking-length reward and competence gate control the pressure to shorten thinking traces, while OPD leaves room for exploration in the thinking region and emphasizes teacher supervision in the answer region. Together, these components may make it easier to shorten reasoning while preserving problem-solving ability. We see the results as support for the two components working together, but they do not on their own establish the specific mechanism. Full settings and results are in Section 4.3 and Appendices F and N.

3. Rationale for the design choices

3.1 Top-20 renormalization and reverse KL

Thank you for raising this concern. We chose top-20 as a practical way to reduce teacher logit transfer and KL computation costs while keeping relative probability information among multiple candidates. Fu et al. (2026) [1] use a similar local matching approach: they select the teacher's top-K candidates, renormalize both distributions within this set, and apply reverse KL matching. Under the same top-p rollout conditions, their method reaches 23.6% on AIME24 avg@32, compared with 21.6% for sampled-token OPD. This provides empirical support for this form of local supervision.

You are right about the limitation: once both distributions are renormalized, the KL terms compare conditional distributions within the retained support and do not directly constrain the student's total probability mass outside it. Our interpretation of reverse KL therefore applies only within the retained set. It weights discrepancies by the student's conditional probabilities and discourages relative over-allocation to candidates with low teacher conditional probability. Forward KL uses the teacher's conditional probabilities as weights and encourages coverage of teacher-supported candidates. This is why we use different KL directions in the thinking and answer regions, and test that choice through the Uniform Reverse KL ablation under the same sparse settings.

We should have made this distinction clearer in the original paper. We have now clarified it in Section 3.2 and Appendix E.3, and describe top-20 as a trade-off between computation and supervisory information. We do not claim that this objective is equivalent to full-vocabulary KL, that this support size is optimal, or that truncation is lossless.

3.2 Joint versus sequential training

Thank you for suggesting this comparison. It gave us a direct way to test our joint training design and raised a broader question: How should length compression and teacher supervision be scheduled to make better use of their complementarity? Our starting idea was that the length reward could encourage shorter reasoning while continuous teacher guidance could help preserve its quality.

Following your suggestion, we added two sequential schedules, GRPO→OPD and OPD→GRPO, with 56 updates to each stage and 112 updates in total. The training data and total update budgets match joint training. The GRPO stage keeps the same length reward and competence gate, and the OPD stage uses the same region-aware asymmetric distillation objective.

Table R2: Joint versus sequential training. Each cell reports accuracy (%) / average thinking tokens. All three schedules contain 112 optimizer updates. MATH500 uses avg@4; AIME25/26 use avg@16.

Training scheduleUpdates with OPDMATH500AIME25AIME26
GRPO→OPD (56 + 56)5692.4 / 1,64055.8 / 10,39166.3 / 10,817
OPD→GRPO (56 + 56)5692.8 / 1,53956.3 / 10,01266.5 / 10,435
Joint TAPER11294.7 / 1,33059.2 / 9,46169.0 / 10,168

Joint TAPER has higher accuracy and shorter thinking traces on all three benchmarks. Compared with the two sequential schedules, accuracy is 1.9–3.4 percentage points higher and thinking tokens are 2.6%–18.9% lower. These results favor joint training within the total update budget we tested. Full results are in Section 4.3, Figure 2, and Appendices F and M.

We think continuous teacher guidance may help the student preserve generation quality as its reasoning becomes shorter. However, joint training applies both objectives in all 112 updates, while each sequential schedule applies each objective in 56 updates. Teacher computation and the number of updates per objective are therefore not matched. This means we cannot attribute the gain solely to joint optimization itself. We would like to better understand, with teacher computation matched, when sequential training can match or outperform joint training and how stage lengths, training order, and objective weights affect that outcome.

Reference

[1] Fu et al. Revisiting On-Policy Distillation: Empirical Failure Modes and Simple Fixes, 2026. Paper.

4. The gap between training and evaluation thinking lengths

Thank you for pointing out this gap. We did not explain these statistics clearly enough in the original paper: 46.4 tokens is the mean thinking length of training rollouts at step 112; 98.7 tokens is the median of the per-step mean thinking lengths over the final ten steps; and 1,330 tokens is the mean thinking length of the final checkpoint on MATH500 under avg@4. The first two describe generation on training examples, while the third describes test behavior. They are not directly comparable, but we agree that clarifying this distinction alone does not explain the large gap.

Training and evaluation also differ in question selection and generation limits. Training questions are filtered to at most 500 input tokens, and the maximum completion length is 6,144 tokens. MATH500 does not use this filter and allows up to 20,480 completion tokens. These differences may affect length, but we do not have enough evidence to attribute the gap to any one of them.

To check whether the trend in the training logs also holds during evaluation, we evaluated all 24 checkpoints on a fixed set of 100 questions using the same decoding, grading, and avg@4 protocol as MATH500. From step 0 to step 112:

  • Average thinking length decreases from 4,429 to 1,389 tokens (−68.6%).
  • Accuracy increases from 85.5% to 93.75% (+8.25 percentage points).

Shorter thinking traces and higher accuracy also appear on a fixed evaluation set under a unified protocol, not just in the training rollouts. The final checkpoint's mean thinking length on this set is 1,389 tokens, close to the 1,330 tokens observed on the full MATH500 test. We did not use these checkpoint results for model selection; all main results use the final checkpoint.

This added analysis confirms that compression also occurs during evaluation, but it still does not fully explain the training–evaluation length gap. We have clarified the statistics and provided the complete curves in Sections 4.1 and 5.3 and Appendices A–B, G, and L.

Reviewer_WPS7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the problem of efficient reasoning (i.e., can we make the model reason with fewer tokens without hurting its accuracy?). It proposes a GRPO variant with two components. The first component is a GRPO length reward that compares each response's thinking length to the other responses for the same problem (and aims to achieve shorter answers on problems the model already knows how to solve well). The second one is on-policy distillation from a larger teacher model (decomposed into two KL terms one for the for the thinking part and one for the answer part of the response).

Critical strengths
  1. TAPER is both more accurate and shorter than its comparable baseline GR3 in all settings in the main table (Table 1).

  2. Compared with the variant that removes the length reward (essentially GRPO+OPD, if I understood correctly), TAPER uses more than 30% fewer thinking tokens. Still, its accuracy is either the same or even slightly higher, depending on the dataset.

  3. The ablation in Section 4.3 (Figure 2) is quite clear about what each component does. The length reward is responsible for most of the compression, and OPD is responsible for most of the accuracy gains.

Critical weaknesses
  1. The main idea of combining RL (GRPO) with on-policy distillation is reasonable, but it has already been investigated before by KDRL [1], and I could not see any comparisons to or discussion of KDRL. The formulations seem to be quite similar (I would love to read the authors' explanation of the differences). See Eq. (5) in TAPER and Eq. (8) of KDRL [1]. How is the KL loss different, and why do we expect TAPER to be better? In any case, I would expect a detailed comparison here.

  2. The abstract states the large gains against GRPO, but it is a bit misleading to start with that, when the GRPO baseline has no teacher or KD. Against the best baselines, the improvement drops to 0.2-3.0 points across the paper. On MMLU-Pro and ProofWriter it is 0.2-0.6 points (Table 1). Since these subsets have 500 questions each, this means only answering ~2 more questions correctly, on average. How do these results justify the added cost of running a 122B teacher model on every student rollout?

  3. The token comparison seems a little imprecise. Correct me if I am wrong, but my understanding is that GR3 and LASER-DE are designed to penalize the full response length (i.e., thinking tokens + answer tokens), while TAPER penalizes thinking length only. So reporting only thinking tokens in Table 1 seems to favor the design of TAPER. Does TAPER output longer sequences within its answer tags?

[1]: Xu et al., KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning, arXiv June 2025

Questions for authors
  1. Related to weakness 1, how does TAPER compare to KDRL [1]? I also could not understand the difference to GR3. Is TAPER simply a KDRL-style objective plus a GR3-style length reward? In that case, the comparison in Table 1 might be a little bit unfair to GR3, since GR3 does not get the same teacher or KDRL-style objective.

  2. What is the total response length (thinking + answer) for each method in Table 1 (see weakness 3)?

2 comments

Thank you for your suggestions. We have added GR³ + OPD comparisons using the same teacher, along with full output cost statistics. Below, we address the differences between the methods, the comparison settings, and full response length.

1. Differences from KDRL and GR³, and comparison fairness

You're right that TAPER is related to these earlier methods. Both TAPER and KDRL [1] combine RL with teacher distillation, but their specific objectives differ: KDRL uses reverse KL for distillation, whereas TAPER uses forward KL in the thinking region and reverse KL in the answer region, reflecting their different supervision needs.

TAPER also does not directly adopt GR³'s length reward. GR³ rescales the overall reward based on relative full response length. TAPER instead adds a within-group relative thinking-length reward to the correctness and format rewards, and scales the length term using the group's solve rate. Both use relative length information, but the reward formulation, length metric, and how compression pressure is adjusted differ.

To better control for teacher effects, we added two GR³ + OPD variants using the same teacher, 9B student initialization, training data, and 112 optimizer updates as TAPER.

Table R1. GR³ and its distillation-augmented variants. Each cell reports accuracy (%) / average thinking tokens. MATH500 uses avg@4; AIME25/26 use avg@16. GR³ is a reference without OPD; both GR³ + OPD variants use the same teacher as TAPER.

MethodMATH500AIME25AIME26
GR³93.2 / 2,40157.1 / 11,70966.7 / 13,167
GR³ + Uniform RKL OPD93.8 / 2,65557.5 / 12,44167.1 / 13,332
GR³ + Region-aware Asymmetric OPD94.5 / 2,60158.7 / 12,26468.1 / 13,243
TAPER94.7 / 1,33059.2 / 9,46169.0 / 10,168

Both OPD variants improve GR³'s accuracy, but neither shortens its thinking traces. Compared with GR³ + the same region-aware asymmetric OPD, TAPER improves accuracy by 0.2, 0.5, and 0.9 percentage points, while reducing thinking tokens by 48.9%, 22.9%, and 23.2%, respectively. These comparisons show that TAPER still achieves a better accuracy–length trade-off with the same teacher and distillation objective, supporting a contribution from its length optimization design. Full settings and results are provided in Section 4.3 and Appendix N; the reward and distillation objectives are defined in Sections 3.1–3.2 and Appendices D–E.

2. Full response lengths for the methods in Table 1

We agree that generation cost should account for full response length. The revised manuscript reports average thinking and total output lengths for every method in Table 1 across the 4B and 9B students and five benchmarks in Appendix K (Table K1), with teacher results included as a reference. Total output includes both thinking and non-thinking content, including the final answer. Table K2 reports changes relative to the same-size Base model, and Section 4.2 summarizes the main findings.

After including non-thinking output, TAPER still uses 20.2%–66.9% fewer total output tokens than GRPO and 11.0%–46.0% fewer than GR³ across all ten model–benchmark combinations. For example, on MATH500 with the 9B student, TAPER generates 1,528 total output tokens on average, comprising 1,330 thinking tokens and 198 non-thinking tokens.

Reference

[1] Xu et al. KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning, 2025. https://arxiv.org/abs/2506.02208

Reviewer_WPS7OpenReview ↗

Response to authors

I just wanted to acknowledge the authors’ response. The response addresses my first and third concerns, but I was not sure whether the authors were conceding the second point. Either way, I think the paper is a good empirical contribution and I am more confident in my weak accept rating now. Thanks.

Reviewer_oYgoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to make LLM reasoning more efficient without sacrificing accuracy. It proposes TAPER, which combines a trajectory-level group-relative length penalty scaled by online solve rate with token-level asymmetric distillation from a larger teacher model. The experiments test 4B and 9B models across five reasoning benchmarks, showing fewer thinking tokens and higher accuracy than GRPO.

Critical strengths
  1. The framing of reasoning budget and reasoning structure as two levels of one problem is clear, and the motivation, method, and analysis are consistent with each other.

  2. Region aware asymmetric distillation is the most original element of the paper, and the choice of KL direction in each region has a sound rationale.

Critical weaknesses
  1. The necessity of joint optimization is asserted rather than shown. All ablations remove components, and no variant separates joint training from simply having both signals.
  2. The results come from a single training run without seeds or confidence estimates, and two of the benchmarks contain only thirty problems each.
  3. Training uses very few examples and very few steps, and the thinking length collapses to a very short median on the training data. The risk of over compression is not examined.
  4. The teacher is far stronger than the student, and the paper gives the reader no way to tell whether the gains come from more efficient reasoning or from reproducing teacher answers.
Questions for authors

See Weaknesses

2 comments

Thank you for raising these concerns. We agree that the paper should better separate the observed results, possible explanations, and what remains unverified. We respond below.

1. Is joint optimization necessary?

You're right: removing components alone cannot show that both objectives need to be optimized together. We added GRPO→OPD and OPD→GRPO, with 56 updates per stage and 112 in total, matching joint TAPER's data and total update budgets. Both signals are retained: GRPO keeps the same length reward and competence gate, and OPD uses the same region-aware asymmetric objective.

Table R1. Accuracy (%) / mean thinking tokens. MATH500: avg@4; AIME25/26: avg@16.

ScheduleOPD updatesMATH500AIME25AIME26
GRPO→OPD5692.4 / 1,64055.8 / 10,39166.3 / 10,817
OPD→GRPO5692.8 / 1,53956.3 / 10,01266.5 / 10,435
Joint TAPER11294.7 / 1,33059.2 / 9,46169.0 / 10,168

Joint TAPER has 1.9–3.4 percentage points higher accuracy and 2.6%–18.9% fewer thinking tokens. Having both signals in sequence does not reproduce joint training's performance under the tested budget.

This does not establish that joint training is always necessary. Joint training uses OPD in all 112 updates, versus 56 in each sequential schedule, so teacher computation and updates per objective are not matched. Our claim is limited to joint training being better under the tested data and total update budgets. Details: Section 4.3, Figure 2, Appendices F and M.

2. Evaluation and reliability

Thank you for raising this concern. MATH500 and the AIME series are standard reasoning benchmarks used by GR³, LASER, and related work. AIME25 and AIME26 each contain 30 problems.

All methods use the same questions and decoding settings. We repeat generation 4 times per question (avg@4) on MATH500, MMLU-Pro, and ProofWriter, and 16 times (avg@16) on AIME25/26, reporting mean accuracy and thinking tokens. LASER's public version likewise uses 4 samples on MATH500 and 16 on AIME2024, matching our repeat counts on the corresponding math tasks. See Section 4.1 and Appendix B.

3. Over-compression

We considered this risk when designing the reward. Our aim is to compress redundant reasoning as much as possible while trying to preserve accuracy.

We add a within-group relative thinking-length reward to the correctness and format rewards. The competence gate scales the length term using the group's solve rate: compression pressure is weaker when the solve rate is low, the length reward is off when all responses are wrong, and pressure is stronger when the solve rate is high.

These choices aim to reduce over-compression, rather than pursue the shortest trace at any cost. This is not a theoretical guarantee that accuracy will be preserved. The reward and gate ablation are described in Sections 3.1 and 4.3 and Appendices D and F.

4. Are the gains mainly from the teacher?

The teacher does provide knowledge and capability gains; that is part of why we use OPD. We do not deny that contribution. The question is whether a stronger teacher alone explains TAPER's length reduction and accuracy–cost gains.

To check this, we added GR³ comparisons with the same teacher and OPD configuration. Against GR³ + the same region-aware asymmetric OPD, TAPER has 0.2–0.9 percentage points higher accuracy and 22.9%–48.9% fewer thinking tokens across the three math benchmarks. With the teacher and OPD held fixed, adding the length reward and gate also reduces thinking tokens by 31.2%–45.5%, with equal or slightly higher reported accuracy. The stronger teacher alone cannot explain these differences.

The teacher generates no independent training trajectories. OPD provides supervision at prefixes visited by the student itself (Section 3.2). The reduced redundancy observed in our behavior analysis provides supporting evidence (Section 5.2).

We do not take these results as direct proof of the model's internal reasoning mechanism. They support a joint contribution of length control and teacher supervision to the observed accuracy–thinking-cost trade-off, without establishing how the student internally uses teacher knowledge. Full comparisons: Sections 4.3 and 5.2, Appendices F and N.

Thank you for the detailed response. Most of my concerns have been addressed. I will maintain my original rating of Weak Accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.