acceptodds
Under review as a conference paper at ICLR 2027

CATER: Competence-Adaptive Training for Efficient Reasoning

Abstract

Efficient reasoning requires models to spend computation where it remains necessary. Yet long Chain-of-Thought (CoT) models can remain verbose on problems they already solve reliably, while uniform pressure to shorten responses can compromise reasoning on problems they have not yet mastered. Existing reinforcement learning approaches promote efficiency through outcome-based rewards, but sparse feedback provides limited guidance on how to eliminate unnecessary reasoning without sacrificing correctness. To resolve this problem, we propose CATER (Competence-Adaptive Training for Efficient Reasoning), a difficulty-adaptive framework that combines reinforcement learning with on-policy self-distillation, using the student’s online success rate to assign a training signal to each problem. For consistently solved problems, concise distillation preserves existing capabilities with fewer reasoning tokens. For inconsistently solved problems, correctness-based reinforcement learning reinforces successful reasoning strategies. By adapting supervision to the student’s evolving competence, our framework advances reasoning compression and capability development together. On Qwen3-4B, it reduces reasoning tokens by 39.1% on average across five benchmarks while maintaining comparable overall accuracy. These efficiency gains extend beyond mathematics to scientific reasoning and code generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Efficient Reasoning, Post Training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_np2SOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CATER to shorten reasoning while maintaining accuracy. It samples several responses per problem and uses their correctness to choose the training objective. All-correct groups receive concise self-distillation, mixed-outcome groups receive correctness-based reinforcement learning, and all-wrong groups are skipped. The method also periodically updates the teacher and balances the two training gradients. Experiments report average reasoning-token reductions of 39.1% on Qwen3-4B and 24.3% on Qwen3-1.7B, with overall accuracy close to the base models.

Critical strengths
  1. The problem is useful and the motivation is reasonable.

    Reasoning models often produce unnecessarily long answers. Compressing problems the model already solves reliably, while focusing on correctness for less reliable problems, is a sensible approach. The two training objectives fit together naturally. Mixed-outcome groups provide correct and incorrect responses for RL to learn from. All-correct groups provide little correctness-learning signal, but can instead be used to learn shorter reasoning. Online routing allows these assignments to change as the model improves.

  2. The experiments support the method’s practical value.

    The method reduces token use across several tasks and model sizes. The ablations show benefits from online routing and combining RL with distillation. The paper also studies several important hyperparameters, including rollout count, distillation strength, and teacher refresh interval.

Critical weaknesses
  1. The contribution over closely related work needs stronger support.

    CEEH and PACE already study how to compress easier problems while preserving reasoning on harder ones. CRISP already uses a conciseness-conditioned self-teacher. CATER adds an explicit rule for choosing between distillation and RL based on online correctness. The routing ablations support this addition, but comparisons with CEEH and PACE are missing. It is therefore difficult to judge whether CATER improves on existing difficulty-aware compression methods.

  2. The paper shows that routing helps, but provides less evidence for its explanation of why it helps.

    The central idea is that reliably solved problems can be compressed with less risk. However, the paper does not directly measure how compression affects problems at different success rates. Figure 6 shows that more groups become all-correct during training, but does not show whether those problems subsequently become shorter without losing accuracy. Such an analysis would strengthen the main explanation. The current theory provides motivation, but does not directly establish this behavior for the proposed training method.

Questions for authors

None

3 comments

We thank the reviewer for the constructive suggestions.

Q1. How does CATER compare with closely related difficulty-aware compression methods such as CEEH and PACE?

A. CATER combines explicit competence-based objective allocation with dense token-level compression guidance. Verified student outcomes determine which questions receive concise self-distillation, while correctness learning strengthens mixed-outcome questions.

We added CEEH and PACE. All methods use Qwen3-4B-Thinking-2507, rank-64 LoRA, and 32 questions × 6 responses per update, and are evaluated at the same training step. For CEEH, we initialize the reference cache and compute the difficulty threshold over initialized entries, corresponding to the warm-active variant. For PACE, we implement the paper’s token-prefix formulation and reward function with the K40 schedule.

Table 8. Comparison with CEEH and PACE.

MethodAIME25 mean@8 (%)Mean reasoning tokensGPQA-D (%)Mean reasoning tokens
CATER76.6714,46765.155,076
CEEH75.4216,29367.685,534
PACE66.2512,27961.113,816

As shown in Table 8, CATER achieves 1.25pt higher AIME25 accuracy with shorter reasoning than CEEH. Compared with PACE, it retains 10.42pt higher AIME25 accuracy and 4.04pt higher GPQA-D accuracy, while using longer reasoning. These comparisons demonstrate CATER’s favorable balance between accuracy and reasoning efficiency, supporting the value of coordinating dense concise supervision with correctness learning.

Q2. Do reliably solved problems become shorter without losing accuracy, and how does compression affect problems with different initial success rates?

A. We evaluated the same 128 training questions before and after training, retaining their initial grouping. The set was fixed before inspecting post-training results. Among them, 72 questions had six initially correct rollouts. Each model then generated eight fresh responses per question under a 32,768-token output budget.

Table 9. Reliable-question compression under a 32,768-token output budget.

Initial groupQuestionsAccuracy, Base → CATER (%)Reasoning tokens, Base → CATERReduction
All-correct72100.00 → 100.005,161 → 2,84044.97%
Other5693.75 → 95.3110,396 → 6,35838.84%
All questions12897.27 → 97.957,451 → 4,37941.23%

As shown in Table 9, initially reliable questions receive 44.97% shorter reasoning while retaining 100.00% observed accuracy, across 576 responses per model. Truncation is zero for both models in this group. These measurements directly show the behavior motivating CATER: reasoning on already reliable questions becomes substantially more concise while retaining observed correctness.

We also evaluated the same questions under the 10,000-token training budget, with 16 fresh responses per question and model.

Table 10. Before/after diagnostics under a 10,000-token output budget.

Initial groupQuestionsAccuracy, Base → CATER (%)Complete output tokens, Base → CATERTruncation, Base → CATER (%)
All-correct7295.14 → 99.575,919 → 3,5134.77 → 0.09
Other5632.81 → 82.378,836 → 6,64664.06 → 13.84
All questions12867.87 → 92.047,195 → 4,88430.71 → 6.10
Prespecified high-truncation group5531.36 → 85.239,382 → 7,01369.20 → 13.98

The high-truncation group is defined by a pre-training truncation rate of at least 25% and overlaps the other groups. Under the training budget, Table 10 shows that initially all-correct questions receive 40.65% shorter complete outputs with higher observed accuracy. In the high-truncation group, shorter outputs enable more correct answers within the same budget. The long-budget analysis separately demonstrates compression with zero truncation in the reliable group. Together, these before/after diagnostics complement the routing proportions in Figure 6 of the paper with direct measurements of accuracy and length.

Thank you for your clarification and additional results. I have no more concerns.

We thank the reviewer for the constructive feedback and are glad that the clarifications and additional results have addressed your concerns. Your feedback has been helpful in improving our paper.

Reviewer_WnoNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CATER, a competence-adaptive framework for efficient reasoning that combines on-policy self-distillation with correctness-oriented reinforcement learning. The proposed framework estimates per-problem competence using multiple rollouts, and then routes all-correct groups to concise self-distillation and mixed-outcome groups to group-relative optimisation. The authors train two Qwen3 models on one dataset and perform evaluation on five benchmarks.

Critical strengths
  • The paper aims to address an important problem: reducing reasoning cost without hurting correctness.
  • The separation of compression eligibility from compression strength is intuitive and interesting. Additionally, online routing accommodates changes in model competence during training.
  • Evaluation spans mathematical reasoning, scientific reasoning, and code generation.
Critical weaknesses
  • Some important details about policy-gradient optimisation, reward normalisation, gradient aggregation, and stabilisation should be provided to establish technical soundness and reproducibility.
  • Accuracy improvements are modest on some benchmarks, and some benchmark-level regressions remain. It is good to report the overall performance, making it easier to compare with baselines.
  • Reasoning-token reduction is reported, but additional training such as compute and teacher overhead are not comprehensively analysed.
Questions for authors
  • How accurately does the all-correct criterion identify problems with high true success probabilities?
  • What are the additional training costs and end-to-end latency benefits?
4 comments

We appreciate the reviewer's valuable feedback.

Q1. What are the implementation details of policy-gradient optimization, reward normalization, gradient aggregation, and stabilization?

A. CATER performs one on-policy update per rollout batch, combining separately accumulated correctness and distillation gradients.

Policy gradient and reward normalization. Each question has six binary correctness rewards. Mixed-outcome groups receive response-level advantages Ai=ci−cˉ16∑j=16(cj−cˉ)2+10−4.A_i=\frac{c_i-\bar c}{\sqrt{\frac{1}{6}\sum_{j=1}^{6}(c_j-\bar c)^2}+10^{-4}}.

The RL loss sums −Ailog⁡πθ(zi,t∣x,zi,<t)-A_i \log \pi_\theta(z_{i,t} \mid x,z_{i,<t}) over response tokens, divides by the global response-token count of the RL branch, and applies coefficient 0.05. All-correct groups receive concise distillation. All-wrong groups have zero learning signal. This is a direct sampled-token policy gradient.

Distillation. The detached student's top four tokens define the support at each position. Student and teacher distributions are normalized over that shared support, and reverse KL, KL(student || teacher), is capped at 0.05 per token. The branch loss is normalized by its global response-token count. The teacher scores student-generated prefixes using the question and a conciseness instruction.

Gradient aggregation. Token denominators and gradient buffers are summed across ranks before gradient balancing. With weighted branch gradients gRg_R and gDg_D, the combined gradient is

\alpha=\frac{\rho}{1-\rho}\frac{\lVert g_R\rVert_2}{\lVert g_D\rVert_2},\qquad\rho=0.1.$$ Thus, 10% specifies the distillation share of the sum of the two rescaled branch gradient norms. We use α = 0 when either norm is zero. *Stabilization.* The combined gradient is clipped to norm 0.1 before AdamW, with learning rate 1e−5, betas (0.9, 0.95), and zero weight decay. We train rank-64 LoRA with alpha 128. The teacher starts from the student, stays frozen within each eight-update interval, and refreshes from the student's effective weights every eighth update. We provide the source code for CATER in the Supplementary Material, including all implementation details. **Q2. How accurately does the all-correct criterion identify problems with high true success probabilities?** **A.** We evaluated the criterion through independent resampling. We fixed 128 training questions, grouped them according to six initial rollout outcomes, and generated 16 additional Base responses per question under a 10,000-token output budget. Both grouping and calibration use the same training verifier. **Table 5. Independent resampling of the all-correct criterion.** | Initial group | Questions | Additional responses | Correct responses | Success rate (%) | | ------------- | --------: | -------------------: | ----------------: | ---------------: | | All-correct | 72 | 1,152 | 1,115 | 96.79 | | Other | 56 | 896 | 337 | 37.61 | In Table 5, initially all-correct questions retain a **96.79% success rate on fresh samples**, compared with 37.61% for the remaining questions. This directly shows that the criterion selects a substantially higher-success subset. The selection behavior also follows from the routing probability. For a question with success probability $p$, six conditionally independent rollouts are all correct with probability $p^6$. This probability is 1.56% for $p=0.5$, 26.21% for $p=0.8$, and 53.14% for $p=0.9$.

Q3. What are the additional training costs and end-to-end latency benefits?

A. CATER incurs a modest 20.77% increase in training compute. Through 40 steps, its total training cost is 2.3619 EFLOPs, compared with 1.9558 EFLOPs for RL-only, an additional 0.4062 EFLOPs. This overhead is incurred during post-training, while the latency benefits apply to subsequent inference.

We measured single-response end-to-end latency on H200. After warmup, we alternated requests between Base and CATER. Client-side timing includes network, queueing, and generation.

Table 6. Measured end-to-end inference latency on H200.

BenchmarkQuestionsBase (s)CATER (s)Latency reductionPaired 95% CI, CATER − Base (s)
AIME2530109.2778.3828.26%[−40.52, −21.47]
GPQA-D3241.7429.8328.54%[−16.33, −7.93]
GSM8K327.376.2515.20%[−2.55, +0.45]
MATH5003225.7814.7142.93%[−17.61, −5.95]
LiveCodeBench3262.8945.1528.21%[−27.15, −8.59]

Across all five benchmarks in Table 6, CATER achieves lower measured mean end-to-end latency, with reductions ranging from 15.20% to 42.93%.

Q4. How does CATER compare in overall accuracy and efficiency across benchmarks and model scales?

A. We will add the overall comparison in Table 7 to the revised manuscript, reporting accuracy and average token usage with equal weight assigned to each of the five benchmarks. CATER achieves the highest overall accuracy among the post-training methods at both model scales, remaining close to Base while reducing average token usage by 33.99% on 4B and 20.54% on 1.7B. These aggregate results further demonstrate CATER’s favorable balance between accuracy and reasoning efficiency.

Table 7. Overall accuracy and average token usage across five benchmarks.

Method4B Overall Acc. (%)4B Avg. tokens1.7B Overall Acc. (%)1.7B Avg. tokens
Base81.809,98261.959,353
CCoT80.629,44661.077,381
CoD80.938,41960.377,121
TALE-EP80.7714,65259.4912,752
MI77.376,85445.786,501
ReBalance81.379,99761.479,155
CoT-Valve75.599,48532.483,595
O1-Pruner79.927,76156.275,966
Art80.918,31861.676,733
Kimi80.417,04360.497,607
CRISP80.617,95260.547,554
CATER81.306,58961.887,432
Reviewer_WnoNOpenReview ↗

Response to authors

Thank you for your response! Since most of my concerns have been addressed, I will raise my rating to 3.

We sincerely thank the reviewer for the constructive feedback and for raising the score. We are pleased that our responses have addressed the concerns and will incorporate the clarifications and additional results into the revised manuscript.

Reviewer_psidOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CATER combines correctness based reinforcement learning with concise self distillation. Questions with consistently correct sampled responses receive distillation, questions with mixed outcomes receive reinforcement learning, and questions with no correct responses are temporarily skipped. The method also uses a periodically refreshed teacher and gradient balancing. Experiments on Qwen models show substantial reductions in reasoning tokens with broadly comparable aggregate accuracy across mathematical reasoning, science, and coding.

Critical strengths

The routing idea is sensible. Problems the model already solves reliably receive a compression signal rather than another correctness update with little useful feedback. Using the model itself as the teacher also avoids requiring a stronger external model.

The token savings are substantial. The 4B model reduces reasoning tokens by 39.1% on average across five benchmarks. The additional AIME 2026 and 30B results provide useful evidence beyond the main evaluation setting.

Critical weaknesses

The novelty seems limited. Section 2 already discusses difficulty aware compression, concise self distillation, and combinations of distillation with reinforcement learning. The main addition appears to be the routing rule and gradient balancing. However, closely related methods such as CEEH and PACE are not evaluated. This leaves the advantage over existing adaptive compression methods unclear.

The routing ablations are not fully controlled. Frozen routing produces shorter outputs on every benchmark, so its lower accuracy could partly reflect stronger compression rather than worse competence estimates. A comparison at matched output length is missing. Similarly, Table 4 does not clarify whether increasing the rollout count also increases the training budget. These issues weaken the evidence for the paper’s central claim.

Questions for authors

How does CATER compare with CEEH or PACE under the same training conditions? Would a simpler baseline that applies a length reward only to groups with all correct responses achieve similar results?

Does online routing still outperform frozen routing at matched output length? How does it compare with random routing that assigns the same fraction of questions to distillation?

Why does CRISP achieve 74.17 on AIME25 while the distillation only ablation achieves 53.33? Appendix C gives too little implementation detail to explain this gap.

How many independent training runs support the main results, and what do the shaded standard deviations in Figures 4 and 5 represent? Were hyperparameters and checkpoints selected using a validation set separate from AIME25?

How consistent is capability preservation outside mathematics? On the 4B model, GPQA accuracy falls from 68.18 to 65.15 and LiveCodeBench accuracy falls from 75.26 to 74.03. Are these drops reproducible across runs?

4 comments

We thank the reviewer for the helpful comments.

Q1. How does CATER compare with CEEH or PACE under the same training conditions? Would a simpler baseline that applies a length reward only to groups with all correct responses achieve similar results?

A. CATER combines explicit competence-based objective allocation with dense token-level compression guidance. Verified student outcomes determine which questions receive concise self-distillation, while correctness learning strengthens mixed-outcome questions.

We added CEEH, PACE, and a simple RL baseline. For CEEH, we initialize the reference cache and compute the difficulty threshold over initialized entries, corresponding to the warm-active variant. For PACE, we implement the paper’s token-prefix formulation and reward function with the K40 schedule. Both baselines use the shared training configuration described above. All methods use Qwen3-4B-Thinking-2507, rank-64 LoRA, and 32 questions × 6 responses per update, and are evaluated at the same step. The simple baseline gives a length reward to all-correct groups and a correctness reward to the other groups.

Table 1. Baseline comparison on AIME25 and GPQA-D.

MethodAIME25 mean@8 (%)Mean reasoning tokensGPQA-D (%)Mean reasoning tokens
CATER76.6714,46765.155,076
CEEH75.4216,29367.685,534
PACE66.2512,27961.113,816
Simple RL, plain73.3314,75562.635,098

As shown in Table 1, CATER has 1.25pt higher AIME25 accuracy with shorter reasoning than CEEH, and higher accuracy than PACE. Compared with the simple RL baseline, it has 3.33-point higher AIME25 accuracy and 2.52-point higher GPQA-D accuracy, with shorter reasoning on both benchmarks. These comparisons show the practical value of combining dense concise supervision with correctness learning.

Q2. Does online routing still outperform frozen routing at matched output length? How does it compare with random routing that assigns the same fraction of questions to distillation?

A. We added Frozen and Random controls. Frozen retains the initial supervision assignment, Random resamples the assignment while matching the distillation proportion, and CATER assigns supervision using the current student's verified outcomes.

Table 2. Routing comparison at matched reasoning length.

MethodAIME25 mean@8 (%)Mean reasoning tokens
CATER76.6714,467
Frozen routing70.4214,596
Random routing74.5814,473

In Table 2, CATER exceeds Random by 2.08pt at nearly identical reasoning length, and Frozen by 6.25pt. The checkpoints were selected to ensure that the comparisons are based on matched output length.

Regarding the rollout-count ablation in Table 4 of the paper, we fix the number of training updates and questions per update. The total number of sampled responses therefore scales linearly with N, so N = 8 uses four times as many responses as N = 2. Competence estimation reuses these training responses without additional sampling specifically for routing. This ablation measures the effect of rollout-group size together with its associated sampling budget. We will explicitly report these budgets and clarify this interpretation in the revised manuscript.

Q3. Why does CRISP achieve 74.17 on AIME25 while the distillation only ablation achieves 53.33? Appendix C gives too little implementation detail to explain this gap.

A. To make the ablation comparable to CATER, we used the same training budget. Under this fixed budget, our pure-distillation variant compresses reasoning more aggressively than CRISP, reaching a substantially shorter reasoning length at the endpoint.

CRISP is a separately configured baseline. It uses a conciseness-conditioned teacher and full-vocabulary reverse KL, with teacher synchronization every 50 updates. Our pure-distillation configuration uses LoRA, teacher synchronization every eight updates, and the student-top-4 KL objective. Our teacher-refresh ablation in Figure 5 of the paper also shows that more frequent teacher synchronization leads to more aggressive reasoning compression.

The endpoint AIME25 accuracies and mean reasoning lengths are 76.67% and 14,467 tokens for CATER, 74.17% and 17,792 tokens for CRISP, and 53.33% and 9,380 tokens for pure distillation.

To examine how accuracy changes as compression progresses, we additionally evaluated ten saved pure-distillation checkpoints. The following results come from the new evaluation batch.

Table 3. Pure-distillation checkpoint trajectory.

MethodStepAIME25 mean@8 (%)pass@8 (%)Mean reasoning tokens
Pure475.8386.6718,154
Pure874.1786.6717,696
Pure1270.4280.0016,209
Pure1669.5886.6714,837
Pure2070.4286.6712,932
Pure2466.2586.6712,570
Pure2865.8383.3311,776
Pure3260.0080.0011,217
Pure3658.3380.009,961
Pure4052.5076.679,932

Table 3 shows that pure distillation retains 75.83% at step 4 and 74.17% at step 8, followed by progressively stronger compression and lower accuracy.

Q4. How many independent training runs support the main results, and what do the shaded standard deviations in Figures 4 and 5 represent? Were hyperparameters and checkpoints selected using a validation set separate from AIME25?

A. The original main results use one training run per configuration. We additionally conducted an independent training run under the same configuration, achieving 75.00% AIME25 accuracy with 14,362 mean reasoning tokens. In Figures 4 and 5 of the paper, the curves report mean AIME25 accuracy and the shaded regions denote ±1 standard deviation from evaluation sampling, with eight responses per question at each checkpoint. We used AIME2026 for hyperparameter selection and checkpoint selection. We additionally report AIME2026 results in Appendix D.

Q5. How consistent is capability preservation outside mathematics? On the 4B model, GPQA accuracy falls from 68.18 to 65.15 and LiveCodeBench accuracy falls from 75.26 to 74.03. Are these drops reproducible across runs?

A. CATER maintains good performance beyond mathematics while substantially reducing reasoning cost, although slight accuracy decreases remain on some benchmarks. We attribute these decreases to the lack of direct coverage of scientific reasoning and code generation in our math-only training data. To improve coverage beyond mathematics, we introduced a 10% science mixture drawn from 339 GPQA-Extended questions, excluding Diamond overlap and near duplicates. The training budget and algorithm remain unchanged.

Table 4. Math-only and mixed-domain training.

ModelAIME25 mean@8 (%)Mean reasoning tokensGPQA-D (%)Mean reasoning tokens
Base75.0020,20168.188,534
Math-only CATER76.6714,46765.155,076
Mixed-domain CATER77.0813,94568.184,982

As shown in Table 4, the science mixture improves GPQA-D accuracy by 3.03 percentage points over math-only CATER with similar reasoning length, while retaining strong mathematical accuracy and efficiency.

Thanks for the additional results. The new baselines and routing comparisons at similar output lengths address part of my concern. The comparison with CEEH still shows a tradeoff, as CEEH retains higher GPQA accuracy. Also, since AIME2026 was used for hyperparameter and checkpoint selection, its results should not be treated as independent evidence of generalization. The response strengthens the experiments, but I am still not convinced that the routing offers enough improvement over existing adaptive compression methods. I will keep my score.

Thank you for the follow-up. We would like to clarify that, under the shared training configuration, CATER achieves higher AIME25 accuracy than CEEH with 11.21% fewer reasoning tokens. Our mixed-domain experiment further raises GPQA-D accuracy from 65.15% to 68.18% while retaining substantial compression, using the same algorithm and training budget. We initially used AIME2026 to select checkpoints and hyperparameters, while our main reported AIME results are on AIME2025. The contribution of routing is supported by the length-matched controls, where CATER exceeds Frozen and Random routing by 6.25 and 2.08 percentage points on AIME25, respectively. It also outperforms the simple RL baseline on both AIME25 and GPQA-D with shorter reasoning, supporting the benefit of competence-based allocation between dense distillation and correctness learning. We hope these clarifications are helpful in your assessment of our design and experimental results. If any concerns remain, we would be happy to address them and provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.