acceptodds
Under review as a conference paper at ICLR 2027

Reasoning Compression with Mixed-Policy Distillation

Abstract

Reasoning-centric large language models (LLMs) achieve strong performance by generating intermediate reasoning trajectories, but often incur excessive token usage and high inference-time decoding cost. We observe that, when solving the same problems, larger reasoning models can often produce more concise traces, whereas smaller reasoning models tend to generate longer and more redundant trajectories. This is especially problematic in real-world deployment, where memory, latency, and serving-cost constraints often favor smaller models. Our observations suggest that reasoning compression can be transferred from large models to small ones rather than enforced through explicit length constraints. Based on this insight, we propose Mixed-Policy Distillation (MPD), a reasoning compression framework that transfers concise reasoning behavior from a larger-sized teacher to a smaller student by distilling teacher-compressed student trajectories. Unlike on-policy distillation, which aligns the student with teacher distributions over verbose student trajectories, or off-policy distillation, which relies on teacher-generated trajectories and may suffer from distribution mismatch, MPD combines the strengths of both. Given a student-sampled trajectory, the teacher rewrites it into a more concise reasoning trace, and the student is trained via KL-based alignment on the compressed trajectory. This preserves student-policy exploration while injecting teacher-guided compression. Experiments on Qwen3-1.7B show that MPD reduces token usage by up to 27.1% while improving performance across multiple reasoning benchmarks, demonstrating an effective approach to efficient small-model reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reasoning Models, Efficient Reasoning, Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BVKNOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Mixed-Policy Distillation (MPD) to shorten small-model reasoning. A larger teacher compresses student-generated reasoning traces, and the student learns through token-level distribution matching on the rewritten traces. The approach aims to combine student-derived reasoning paths with teacher-guided concision, without ground-truth supervision or length penalties. Experiments report reduced token usage while maintaining or improving Pass@4 across mathematical, code, and long-context reasoning tasks.

Critical strengths

1.The idea is simple and useful: improve reasoning efficiency using supervision tailored to the student’s actual attempts.

  1. Results show substantial token savings across several tasks and model families. In the main experiment, average tokens fall from 8.2k to 6.1k, while Pass@4 rises from 68.2 to 71.3.
Critical weaknesses
  1. The benefit of teacher compression is insufficiently explained. On-Policy (C) already compresses student-generated trajectories, and both methods use the same teacher for distribution matching. Although MPD produces shorter outputs, the paper does not directly compare the compressors’ preservation of essential reasoning steps. Table 2 shows nearly identical semantic similarity, while Appendix C audits only teacher rewrites. A comparison at matched compression lengths would clarify whether the advantage comes from better reasoning-aware selection or simply stronger compression.

  2. Limited contribution. While I do agree that the compressing is satisfactory compared to the baseline methods, it appears to be an incremental extension of on-policy distillation, with prompt-based teacher compression as the main addition. The paper provides insufficient insight into why this modification works. Beyond Weakness 1, the large gap between SFT and MPD deserves further explanation. For example, would SFT trained on student-compressed trajectories exhibit the same performance collapse? This comparison could help clarify whether the failure stems from the compression targets or the training objective.

Questions for authors

see weakness

3 comments

Weakness 1:

The benefit of teacher compression is insufficiently explained. On-Policy (C) already compresses student-generated trajectories, and both methods use the same teacher for distribution matching. Although MPD produces shorter outputs, the paper does not directly compare the compressors’ preservation of essential reasoning steps. Table 2 shows nearly identical semantic similarity, while Appendix C audits only teacher rewrites. A comparison at matched compression lengths would clarify whether the advantage comes from better reasoning-aware selection or simply stronger compression.

Reply to Weakness 1: We thank the reviewer for this insightful comment. We agree that semantic similarity does not necessarily imply preservation of essential reasoning steps.

First, we conducted an additional LLM-as-Judge evaluation using the same evaluation protocol (gemini-3.1-flash-lite) as Appendix C to directly compare student and teacher compression. We further report the compression quality using teacher model or student model to compress the trajectories, as shown in the Table below. The results show that teacher compression achieves 93% faithful trajectories, compared to 87% for student compression, with higher scores in key dimensions such as constraint preservation (4.92 Student-Teacher vs. 4.77 Student-Student) and Equation Preservation (4.92 vs. 4.78). Further, we also compare the compression rate. The teacher as compressor achieves these results despite a higher compression rate (93% vs. 88%). This provides more direct evidence that teacher compression better preserves essential reasoning steps while removing more redundancy.

MethodTraj.CompressorCompression RatioFaithful Traj.
MPDStudentTeacher93.47%93%
On-Policy (C)StudentStudent87.76%87%

Second, regarding matched-length comparison, we found that compression rates are primarily determined by the compressor model rather than the prompt. We experimented with different compression instructions, but observed only minor variations in compression rates. Matching compression lengths would therefore require explicit length constraints, potentially changing the natural compression behavior that our method aims to exploit.

Together with the downstream improvements over On-Policy (C) in Table 5, these findings support the benefit of stronger teacher-guided compression beyond simply producing shorter trajectories.

Weakness 2:

Limited contribution. While I do agree that the compressing is satisfactory compared to the baseline methods, it appears to be an incremental extension of on-policy distillation, with prompt-based teacher compression as the main addition. The paper provides insufficient insight into why this modification works. Beyond Weakness 1, the large gap between SFT and MPD deserves further explanation. For example, would SFT trained on student-compressed trajectories exhibit the same performance collapse? This comparison could help clarify whether the failure stems from the compression targets or the training objective.

Reply to Weakness 2:

Thank you for your constructive comments. We address these concerns as follows.

1. Novelty and motivation. The key contribution of MPD is to combine teacher-guided trajectory compression while preserving student’s reasoning strategy in a unified distillation framework. Our ablation experiments suggest that neither keeping student reasoning nor trajectory compression alone fully explains MPD's improvements; their combination provides a more favorable accuracy–efficiency trade-off.

2. Performance gap between SFT and MPD. We agree that this comparison deserves further clarification. Appendix B.2 (Table 12) reports SFT performance across training steps. Notably, at comparable output lengths, SFT at step 50 achieves an average Pass@4 of 65.9 with 6.0k tokens, whereas MPD achieves 71.3 with 6.1k tokens. This indicates that MPD's advantage is not merely due to differences in output length or the performance collapse of SFT at later training steps.

3. Additional SFT experiment. Following your suggestion, we conduct an additional experiment that applies SFT to student-compressed trajectories under the same training setup. The result shows that training with trajectories compressed by the student model, the SFT also shows reduction on both token length and performance and degradation with further training steps. We expect this controlled comparison providing further insight into the performance degradation that arises primarily from the SFT objective.

MethodAIME24 P@4AIME24 #TokAIME25 P@4AIME25 #TokGSM8K P@4GSM8K #TokCodeMMLU P@4CodeMMLU #TokLongBench2 P@4LongBench2 #TokAvg. P@4Avg. #Tok
Qwen3-1.7B (Vanilla)63.316.8k53.317.7k94.52.1k81.23.6k48.70.9k68.68.2k
SFT step 5066.712.7k40.09.6k92.60.8k82.72.2k46.70.7k65.75.2k
SFT step 10010.00.6k0.00.8k88.10.2k69.10.6k46.50.6k44.70.5k
SFT step 1503.30.5k10.00.4k88.00.2k66.70.6k45.90.5k41.40.4k
SFT step 2006.70.3k3.00.3k84.20.2k67.80.6k47.90.4k41.30.4k
MPD (Ours)73.313.4k56.712.9k94.21.4k82.82.3k50.30.7k71.36.1k

Thank you for your response. I will keep my score.

Reviewer_Qb3COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Mixed-Policy Distillation (MPD) to improve reasoning efficiency in smaller language models. The method first samples reasoning trajectories from the student, then uses a larger teacher to compress them, and finally trains the student through KL-based distillation on the compressed traces. Experiments across several reasoning benchmarks and model families show reduced token usage with comparable or slightly improved accuracy.

Critical strengths

The method is simple and practical. It achieves meaningful token reductions without explicit length penalties or correctness rewards, which makes it an interesting alternative to existing compression approaches.

The experiments cover multiple reasoning tasks and model families. The comparisons between different policy variants are also useful for understanding the benefits of the proposed approach.

Critical weaknesses

The novelty seems somewhat limited. The main idea is to rewrite student trajectories with a stronger teacher and apply standard distillation. This is closely related to existing approaches using teacher-generated supervision. More importantly, the compressed trajectories are no longer directly sampled from the student policy, so the claimed on-policy advantage is not fully justified.

The accuracy improvement over CRISP is relatively small, and the AIME evaluations contain only 30 problems. The SFT comparison is not entirely convincing since the baseline eventually produces much shorter outputs than MPD, making the comparison difficult to interpret. The experiments across other model families also lack comparisons with strong compression baselines.

Questions for authors

What is the main methodological novelty compared with existing on-policy distillation and teacher-guided reasoning compression methods?

Since the student is trained on teacher-rewritten trajectories, how much distribution shift is actually reduced compared with standard off-policy distillation? Can this be measured directly?

Could the authors compare MPD with a better-tuned SFT baseline at matched output lengths? The current SFT results appear to suffer from excessive compression.

Could the authors report Pass@1 and confidence intervals for the main comparisons? The accuracy differences on some benchmarks seem small, particularly given the limited number of AIME problems.

How sensitive is MPD to the teacher's compression quality? Would the advantage remain if the compression ratio were matched across different policy variants?

Appendix C shows that a substantial fraction of compressed trajectories remain incorrect. What happens if incorrect trajectories are filtered out before distillation?

Could the authors provide a more complete efficiency comparison that includes teacher inference and compression costs? It is unclear how much additional training computation is required to achieve the reported inference savings.

5 comments

Weakness 1:

The novelty seems somewhat limited. The main idea is to rewrite student trajectories with a stronger teacher and apply standard distillation. This is closely related to existing approaches using teacher-generated supervision. More importantly, the compressed trajectories are no longer directly sampled from the student policy, so the claimed on-policy advantage is not fully justified.

Reply to Weakness 1: Thank you for this important question. The main novelty of MPD lies in combining student-policy exploration with teacher-guided trajectory compression within a unified distillation framework.

Unlike standard on-policy distillation, which learns from the student's original and potentially verbose trajectories, MPD uses a stronger teacher to compress these trajectories before distillation. Unlike conventional off-policy compression methods, which rely on independently generated teacher trajectories, MPD preserves the student's original reasoning paths as the basis for compression, reducing reliance on potentially unfamiliar teacher-generated solutions.

This design creates a student-conditioned, teacher-refined training distribution that combines the benefits of both approaches. Our ablation results (Table 5) demonstrate that MPD achieves a better accuracy–efficiency trade-off than standard on-policy, off-policy, and their compressed variants. Especially our ablation experiments suggest that neither keeping student reasoning nor trajectory compression alone fully explains MPD's improvements; their combination provides a more favorable accuracy–efficiency trade-off. Our experiments with other baseline methods also show our contribution. Further our experiments across various model families and scales of models shows the generalizability of our method.

Weakness 2:

The accuracy improvement over CRISP is relatively small, and the AIME evaluations contain only 30 problems. The SFT comparison is not entirely convincing since the baseline eventually produces much shorter outputs than MPD, making the comparison difficult to interpret. The experiments across other model families also lack comparisons with strong compression baselines.

Reply to Weakness 2: We thank the reviewer for this comment.

We provided the SFT results at different steps in Appendix B.2 and Table 12. At a similar output length level, SFT reaches an average performance of 65.9 with an average output length of 6.0k, while our method achieves a performance of 71.3 with an average output length of 6.1k.

Our experiments across different model families provide evidence of the generalizability of our approach, while the main experiments demonstrate its advantages over strong baselines. We acknowledge conducting wide experiments across model families brings us a better understanding of our method. But further comparisons across additional model families would incur substantial computational costs while offering only marginal empirical evidence, as both the performance and generalization has been shown in the experiments.

Question 1: What is the main methodological novelty compared with existing on-policy distillation and teacher-guided reasoning compression methods?

Reply to Question 1: Please refer to Weakness 1.

Question 2: Since the student is trained on teacher-rewritten trajectories, how much distribution shift is actually reduced compared with standard off-policy distillation? Can this be measured directly?

Reply to Question 2: Please refer to Weakness 1.

Question 3: Could the authors compare MPD with a better-tuned SFT baseline at matched output lengths? The current SFT results appear to suffer from excessive compression.

Reply to Question 3: Please refer to Weakness 2.

Question 4:

Could the authors report Pass@1 and confidence intervals for the main comparisons? The accuracy differences on some benchmarks seem small, particularly given the limited number of AIME problems.

Reply to Question 4: We thank the reviewer for this valuable suggestion.

1. Motivation for Using Pass@4.

Due to the inherent stochasticity of LLM inference, including nondeterministic behavior that cannot be completely eliminated in vLLM-based inference, evaluation based on a single generation (Pass@1) may be sensitive to individual sampling outcomes. Multi-sample evaluation is a widely adopted practice for assessing stochastic LLM reasoning. Therefore, we adopt Pass@4 to evaluate whether a method can successfully solve a problem within four independent attempts, rather than relying solely on a single generation. This reduces our evaluation's dependence on any particular generation outcome, while providing a complementary measure of multi-attempt problem-solving capability. The choice of four samples represents a practical trade-off between evaluation robustness and computational cost, allowing us to account for stochastic inference behavior within our available computational budget. To further quantify statistical uncertainty, we additionally report 95% confidence intervals for Pass@4 using the Wilson score method.

2. Confidence Interval Estimation.

We additionally report 95% confidence intervals (CIs) for Pass@4 using the Wilson score method. Since each question contributes a binary success indicator, Wilson intervals can be computed from the aggregate Pass@4 and the number of evaluation questions, without requiring access to individual generations. We apply the same procedure consistently across all methods and benchmarks. The evaluation set sizes are 30 for AIME24, 30 for AIME25, 1,318 for GSM8K, 1,000 for CodeMMLU, and 503 for LongBench2. For macro-average Pass@4, we compute approximate 95% CIs using the estimated binomial variance within each benchmark, assuming independent evaluation questions and equal weighting across the five benchmarks. The following table reports Pass@4 with its corresponding 95% CI, with all values expressed as percentages.

MethodAvg. ± CI
Qwen3-1.7B68.2 ± 5.1
Direct Comp.68.9 ± 5.0
Chain-of-Draft55.1 ± 5.1
LiteCoT56.8 ± 5.1
CRISP70.7 ± 4.9
MPD (Ours)71.3 ± 4.9

3. Interpretation and Limitations. The results show that MPD achieves the highest point-estimate Pass@4 on four of the five benchmarks and the highest macro-average Pass@4 (71.3%). As a limitation we agree that the confidence intervals are relatively wide, because of the smaller evaluation sets of benchmarks AIME24 and AIME25. We will incorporate these confidence intervals and clarify the estimation procedure and its limitations in the revised manuscript. Importantly, our primary claim is not to achieve statistically significant performance improvements over existing baselines, but rather to reduce token consumption while maintaining competitive reasoning performance. As shown in the results (Table 3 in manuscript), MPD achieves comparable Pass@4 scores across benchmarks while requiring fewer tokens than several competitive baselines. These findings support our central objective of improving inference efficiency without substantially compromising reasoning accuracy.

Question 5:

How sensitive is MPD to the teacher's compression quality? Would the advantage remain if the compression ratio were matched across different policy variants?

Reply to Question 5: Thank you for this insightful question.

Regarding compression quality, we acknowledge that it is difficult to quantify using a single metric, as it involves both reasoning preservation and compression efficiency. The compression quality is determined by the inherent ability of the teacher model, rather than prompt. We examine the robustness of MPD across different teacher behaviors, where compression behaviors naturally vary due to different teachers. As shown in Tables 4, 10, and 11, the consistent improvements in token efficiency while maintaining competitive reasoning performance provide empirical evidence of MPD's robustness across these settings. Furthermore, the trajectory-level correctness and faithfulness analyses in Appendix C support the reliability of teacher-guided compression.

Regarding compression-ratio matching, the compression ratio is also primarily determined by the teacher's inherent behavior, rather than prompt. We experimented with different compression prompts but observed little variation in the resulting compression ratios, making ratio matching difficult. Therefore, strictly matching compression ratios across policy variants is challenging in practice.

Nevertheless, we conducted an additional analysis to measure the compression ratios across different policy variants based on our ablation studies. Importantly, as shown in the table below, MPD and Off-Policy (C) exhibit nearly identical compression ratios (93.47% vs. 93.71%). They use the same teacher compressor but using different sources of reasoning trajectories. However, as shown in Table 5, MPD achieves higher average Pass@4 (71.3 vs. 69.8) and substantially lower inference token consumption (6.1k vs. 9.9k). This observation suggests that MPD's advantage over Off-Policy (C) cannot be simply attributed to stronger trajectory compression. Instead, the results support the importance of using student-generated trajectories as the basis for teacher-guided compression and distillation.

MethodTraj.CompressorMean Compression Ratio
MPDStudentTeacher93.47%
On-Policy (C)StudentStudent87.76%
Off-Policy (C)TeacherTeacher93.71%

While we acknowledge that compression strength and policy design cannot be fully disentangled, especially it’s hard to control only the compression quality or compression rate without switching a different teacher model and teacher ability, our ablation results in Table 5, together with the cross-model evaluations, support the effectiveness and robustness of MPD across different compression behaviors.

Question 6:

Appendix C shows that a substantial fraction of compressed trajectories remain incorrect. What happens if incorrect trajectories are filtered out before distillation?

Reply to Question 6: We thank the reviewer for raising this question. We acknowledge that incorrect compressed trajectories may represent a potential bottleneck for further performance improvements. As discussed in Appendix F (Limitations & Future Directions), our current framework does not explicitly incorporate correctness into the training objective, and investigating correctness-aware training is an important direction for future work.

However, filtering incorrect trajectories would require an additional correctness verification mechanism, such as ground-truth supervision or an external verifier. This falls outside our assumption of our current work, which aims to achieve effective reasoning compression without requiring ground-truth supervision or reward models.

Recent works [1,2] use on-policy distillation as a supplement to GRPO and conduct on the false result trajectories. Thus we believe trying to filter out incorrect trajectories will bring us further insights. Due to the limited time for rebuttal and the computational limitation, we apologize we could not provide experimental results for this setting on time. Therefore, while correctness-based filtering may offer further improvements, we consider it a promising future extension rather than a necessary component of the proposed MPD framework, as we explicitly mentioned in our Limitation section.

[1] Li, Gengsheng, et al. "Unifying group-relative and self-distillation policy optimization via sample routing." arXiv preprint arXiv:2604.02288 (2026); [2] Hübotter, Jonas, et al. "Reinforcement learning via self-distillation." arXiv preprint arXiv:2601.20802 (2026).

Reviewer_sjZGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on the problem that small reasoning models generate lengthy CoT, resulting in high token consumption, aiming to reduce the reasoning cost while maintaining the accuracy. This paper proposes Mixed-Policy Distillation (MPD), which combines student on-policy sampling and teacher-guided compression. Specifically, the original reasoning trajectory is first generated by student, and then the trajectory is compressed according to question and student CoT by using a larger teacher. Finally, the distillation is performed by token-level KL Divergence on the compressed trajectory to guide the student to learn more concise reasoning behaviors.

Critical strengths
  1. The paper addresses the important problem of reducing reasoning costs in small language models while maintaining reasoning performance, which is relevant to efficient LLM deployment.
  2. Simple and effective framework. MPD achieves consistent reductions in token consumption while maintaining or improving reasoning accuracy across the evaluated benchmarks.
Critical weaknesses
  1. MPD starts from On-policy Student Trajectories, but the trajectories after Teacher Compression are actually used for KL Distillation. It is no longer strictly On-policy Samples. The paper lacks direct evidence that these rewritten trajectories still retain the distributive advantage of Student-policy. In addition, Teacher Compression introduces additional training overhead, and its benefit-cost trade-off over directly using teacher-generated Trajectories is still unclear.
  2. The paper implicitly treats longer CoT trajectories as redundant reasoning. In particular, OPD increases token consumption while improving accuracy, raising questions about whether the additional reasoning steps are truly unnecessary. The paper does not explain the source of such redundancy or why distillation from a more concise teacher results in longer student-generated trajectories.
Questions for authors
  1. While MPD starts from Student On-policy Trajectories, the Teacher-compressed Trajectories are actually used for KL Distillation, Therefore, it is not clear whether it still retains the distribution advantage of On-policy.
  2. The existing experiments also do not fully prove that the performance improvement comes from the claimed Mixed-Policy mechanism rather than the Teacher-guided Compression itself.
  3. Moreover, MPD introduces an additional Teacher Compression Cost, but lacks a systematic analysis of the trade-off between training overhead and inference efficiency gain.
4 comments

Weakness 1:

MPD starts from On-policy Student Trajectories, but the trajectories after Teacher Compression are actually used for KL Distillation. It is no longer strictly On-policy Samples. The paper lacks direct evidence that these rewritten trajectories still retain the distributive advantage of Student-policy. In addition, Teacher Compression introduces additional training overhead, and its benefit-cost trade-off over directly using teacher-generated Trajectories is still unclear.

Reply to Weakness 1: We thank the reviewer for giving us the opportunity to clarify our motivation. Our intended motivation is to preserve student-originated reasoning structures through teacher-guided compression, rather than to guarantee preservation of the student's original token-level policy distribution (Please refer to our manuscript lines 021-028, lines 081-089, and lines 483-485).

1. Motivation

In our method, the student provides the initial reasoning process, while the teacher improves its conciseness and organization. Appendix C provides empirical support for this design. In Table 15, 93.91% of compressed trajectories are classified as Faithful, with an additional 2.36% classified as Mostly Faithful. The evaluations also report high scores for equation preservation (4.92/5), logical consistency (4.89/5), and dependency preservation (4.88/5). The case studies in Tables 16–17 further illustrate the preservation of key reasoning steps. These findings support reasoning-structure preservation, not token-level distributional alignment.

2. Empirical Advantage

Table 5 provides direct empirical evidence for the practical value of student-originated trajectory construction. The most relevant comparison is between MPD and Off-Policy (C). Both use teacher-guided compression and KL distillation, but Off-Policy (C) compresses teacher-generated trajectories, whereas MPD compresses student-generated trajectories. MPD achieves a higher average Pass@4 (71.3 vs. 69.8) while requiring fewer inference tokens (6.1k vs. 9.9k).

Compared with On-Policy (C), which uses student-side compression, MPD also improves Pass@4 (71.3 vs. 70.5) while reducing inference tokens (6.1k vs. 7.9k). For completeness, the uncompressed Off-Policy and On-Policy baselines achieve Pass@4 scores of 70.3 and 70.5, respectively, with 10.6k and 11.8k inference tokens.

These comparisons support the empirical effectiveness of combining student-originated trajectory generation with teacher-guided compression and KL distillation. We apologize for our wordings of "student distribution”, “minimal distributional shift” that may potentially raise misinterpretation as “token-level on-policy”, which we actually refer to as “student’s reasoning behaviour”. We will consistently revise our claims and description in the Introduction, Sections 2.2, 3, and 4.2, the contribution statements.

3 Benefit-Cost Trade-off We present our benefit-cost trade-off in following aspects:

3.1 Training Computational Cost.

To provide a more comprehensive measure, we report the actual end-to-end training wall-clock time and GPU-hour consumption.

With implementation optimization, MPD training takes 7 hours and 34 minutes on 4 NVIDIA H100 GPUs, corresponding to near 30.27 H100 GPU-hours. This is lower than the 72 GPU-hours reported for our earlier implementation reported in our manuscript representing a 57.96% reduction in total training GPU-hours.

The GPU-hour measurement captures the overall computational resource consumption of the measured training execution.

3.2 Inference Token & End-to-End Efficiency.

Our method reduces the average output length across five reasoning benchmarks from 8.3k to 6.1k tokens, corresponding to a 26.51% reduction. During training, the student generates approximately 10.7M tokens, while the teacher generates approximately 0.6M tokens. The teacher-generated tokens account for only about 5.24% of the total generated tokens.

To evaluate whether token reduction translates into actual computational savings, we measured inference wall-clock time using vLLM on a single NVIDIA H100 GPU, with gpu_memory_utilization=0.7.

Across five benchmarks comprising 11,528 evaluation instances, MPD reduces the total inference time from 10,243 seconds to 8,141 seconds compared with Vanilla LLM, achieving a 20.52% reduction in wall-clock time. Correspondingly, the aggregate request throughput increases from 1.125 to 1.416 instances/s, representing a 1.26× speedup under the same hardware configuration.

The inference-time improvements are observed across all five benchmarks, with reductions of 9.66% on AIME24, 15.48% on AIME25, 30.23% on GSM8K, 28.46% on CodeMMLU, and 7.50% on LongBench2.

We additionally evaluated inference efficiency on a single NVIDIA A40 GPU using 4,000 CodeMMLU instances. MPD reduces inference time from 12h 07m 39s to 5h 35m 25s compared with Vanilla LLM, achieving a 53.90% reduction in wall-clock time and a 2.17× throughput speedup. This further demonstrates MPD's practical efficiency benefits on resource-constrained hardware.

These measurements provide direct evidence that MPD's reasoning compression translates into practical inference-time efficiency improvements, rather than merely reducing the number of generated tokens.

3.3 Break-Even Inference Analysis.

To further evaluate the practical cost-effectiveness of MPD, we estimate the inference volume required to amortize its training cost.

Based on our measurements across 11,528 evaluation instances, MPD saves a total of 2,102 seconds of inference time on a single H100 GPU compared with Vanilla LLM, corresponding to near 0.182 seconds per evaluation instance. Using the measured training cost of 30.27 H100 GPU-hours, we estimate a break-even inference volume of approximately 598K evaluation instances, assuming equal H100 GPU-hour costs for training and inference, comparable workload distributions, and that the saved GPU execution time can be translated into actual resource savings.

This estimate is workload-dependent and should not be interpreted as a universal deployment threshold. Actual cost-effectiveness may vary with configurations. Nevertheless, the analysis provides a concrete reference for assessing the amortization of MPD's one-time training cost in high-volume deployment scenarios.

Weakness 2:

The paper implicitly treats longer CoT trajectories as redundant reasoning. In particular, OPD increases token consumption while improving accuracy, raising questions about whether the additional reasoning steps are truly unnecessary. The paper does not explain the source of such redundancy or why distillation from a more concise teacher results in longer student-generated trajectories.

Reply to Weakness 2: We thank the reviewer for this insightful comment. We would like to clarify that our paper does not claim that “longer CoT trajectories as redundant reasoning” nor “additional reasoning steps are truly unnecessary”.

We observe that the reasoning steps may contain redundant or overthinking content, as illustrated by the case studies in Appendix E.1. It has also been mentioned in other works, that redundancy contributes minimally to accuracy and waste computational resources [1].

Our goal is to reduce the redundant content instead of longer COT. These are two distinct claims. Our goal is also consistent with our results: MPD achieves competitive performance while generating substantially shorter trajectories (Table 5). These findings suggest that at least some overthinking content can be reduced without compromising reasoning performance.

We apologize if some wording may raise this misinterpretation, we will revise the manuscript to avoid potential ambiguity.

Regarding the question “why distillation from a more concise teacher results in longer student-generated trajectories”, we would appreciate it if the reviewer could further clarify what they mean by this question. We would appreciate it if the reviewer could further clarify what they mean by this question.

[1] Chen, Xingyu, et al. "Do NOT think that much for 2+ 3=? on the overthinking of long reasoning models." Forty-second International Conference on Machine Learning. 2025.

Question 1:

While MPD starts from Student On-policy Trajectories, the Teacher-compressed Trajectories are actually used for KL Distillation, Therefore, it is not clear whether it still retains the distribution advantage of On-policy.

Reply to Question 1: Please refer to weakness 1.

Question 2:

The existing experiments also do not fully prove that the performance improvement comes from the claimed Mixed-Policy mechanism rather than the Teacher-guided Compression itself.

Reply to Question 2: We thank the reviewer for this concern. The reviewer’s concern includes the necessity of keeping exploration following the student’s policy and the necessity of teacher faithful rewriting. Our method leverages both the student’s own reasoning behaviour and teacher-guided reasoning compression.

  1. Teacher-guided compression alone does not account for MPD's improvements. Our ablation studies in Table 5 are designed to disentangle the contributions of student reasoning behaviour and teacher-guided compression. Specifically, Off-Policy (C) vs. MPD isolates the effect of using student-generated rather than teacher-generated trajectories, while On-Policy (C) vs. MPD isolates the effect of teacher-guided rather than student-guided compression. MPD outperforms both variants in the overall accuracy–efficiency trade-off, suggesting that neither component alone is sufficient to achieve the observed results.

  2. Teacher-guided compression and student-generated trajectories provide complementary benefits. Our ablation studies show that student-generated trajectories are essential for preserving the student's own reasoning behaviour, while teacher-guided compression provides effective supervision for concise reasoning. Combining these two components allows MPD to retain the student's reasoning patterns while leveraging the teacher's ability to refine and compress them. The results demonstrate that these two components are complementary, and neither alone achieves the same accuracy–efficiency trade-off as MPD.

Question 3: Moreover, MPD introduces an additional Teacher Compression Cost, but lacks a systematic analysis of the trade-off between training overhead and inference efficiency gain.

Reply to Question 3: Please refer to weakness 1.

Thank you for the response. Most of my concerns have been addressed, and I will raise my score. However, why preserving student reasoning improves learning remains insufficiently explained.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.