acceptodds
Under review as a conference paper at ICLR 2027

To Think or Not to Think: Allocating Reasoning Where It Helps

Abstract

Reinforcement learning (RL) has proven effective in enhancing the reasoning performance of large language models (LLMs), particularly in complex mathematical and programming tasks. However, this capability comes with systematic length misallocation, in which models devote excessive reasoning to simple questions while terminating prematurely on harder ones, degrading inference efficiency with negligible accuracy improvement. Many length-adaptive methods mitigate this issue by allocating token budgets according to question difficulty, under the implicit assumption that harder questions benefit monotonically from extended reasoning. In contrast, we find that the effect of reasoning length on accuracy is concentrated on partially solvable questions. Our further analysis reveals that explicit length rewards can produce unintended training dynamics. Motivated by these findings, we propose CARE—Contrastive Accuracy Reward Estimation—which compares the beneficial length adjustment per question from online sampled responses and applies adaptive length rewards within Group Relative Policy Optimization, with no extra hyperparameters or additional inference cost. Experiments across multiple reasoning benchmarks demonstrate that our method improves Pass@1 by up to 4% while simultaneously reducing reasoning length by 37%, achieving higher token efficiency. Code will be available upon the acceptance of this paper.

Primary area:
reinforcement learning
Keywords:
RL, LLM, Reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_X5xVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how reasoning length affects the accuracy and efficiency of LLMs. The authors find longer reasoning doesn't always improve accuracy. So they propose CARE, a framework uses sampled responses to estimate whether longer or shorter reasoning is more helpful. The method adjusts length rewards during GRPO training. They do experiments on 3 models and 4 benchmarks. Results show that CARE improves accuracy while reducing reasoning tokens.

Critical strengths

The method is simple and can be integrated into GRPO without additional inference cost.

Experiments show improved accuracy and reduced token usage across three model sizes.

Critical weaknesses

Experiments focus only on mathematical reasoning. Performance on other tasks are not unclear.

The method uses noisy accuracy estimates from small response groups and may affect training stability.

Questions for authors

How does CARE perform on other reasoning tasks?

How stable are the results across different training seeds?

2 comments
AuthorsOpenReview ↗

Response to Reviewer X5xV

We thank the reviewer for highlighting the importance of cross-domain evaluation and training-seed stability. We address these concerns below.

Q1: How does CARE perform on other reasoning tasks?

A1: CARE achieves higher pass@1 than GRPO on SynLogic-Easy, SynLogic-Hard, and GPQA-Diamond for all three Qwen3 model sizes. We compared GRPO and CARE on three non-mathematical reasoning benchmarks: SynLogic-Easy and SynLogic-Hard (official validation splits), which assess logical reasoning, and GPQA-Diamond, which covers graduate-level scientific reasoning in physics, chemistry, and biology. For consistency with the main paper, we followed the same evaluation settings as in the main paper, using a temperature of 0.6, top-p of 0.95, and 32 independently sampled responses per question. The results are presented in Table 1, where each entry reports pass@1 (%) and the average number of generated tokens.

Table 1: Cross-Domain Reasoning

ModelMethodSynLogic-Easy (Validation)SynLogic-Hard (Validation)GPQA-Diamond
Qwen3-1.7BGRPO23.89 / 6658.7710.71 / 7706.5735.50 / 4787.10
CARE26.18 / 6743.9113.80 / 7760.8638.13 / 4859.69
Qwen3-4BGRPO37.69 / 5973.5422.76 / 7289.7549.43 / 3614.94
CARE39.77 / 5541.4725.12 / 6688.2751.77 / 3243.38
Qwen3-8BGRPO46.79 / 5736.3829.97 / 7515.9857.09 / 4790.69
CARE49.84 / 5681.0132.15 / 7601.1659.79 / 4848.71

Q2: How stable are the results across different training seeds?

A2: CARE achieves higher pass@1 than GRPO in most model–benchmark comparisons while generally consuming fewer tokens under three independent training seeds. We conducted three independent training runs with different random seeds for both GRPO and CARE at each of the three Qwen3 model sizes. The resulting checkpoints were then evaluated on AIME 2025, AIME 2026, HMMT February 2026, and MATH500, using the same evaluation settings as in the main paper. Table 2 reports the mean ± standard deviation of pass@1 (%) and average generated tokens for the three training seeds.

Table 2: Training-Seed Robustness

ModelMethodAIME2025AIME2026HMMT2026MATH500
Qwen3-1.7BGRPO29.83±2.35 / 11032±65129.27±1.81 / 10362±41817.65±2.34 / 11307±57287.54±0.94 / 3993±205
CARE33.02±2.18 / 9845±50131.53±2.26 / 9876±43221.40±1.92 / 10243±42587.33±0.99 / 3183±189
Qwen3-4BGRPO48.09±1.94 / 9497±42152.57±2.19 / 9059±40626.58±2.10 / 10268±45991.24 ± 0.96 / 3077±144
CARE52.81±1.91 / 8600±40255.17 ± 2.03 / 8148±40328.60±1.86 / 8871±42892.27 ± 0.73 / 2780±160
Qwen3-8BGRPO55.17±2.09 / 10164±42956.04±1.60 / 9610±45432.13±1.63 / 11518±53390.28±0.82 / 3206±210
CARE53.99±1.63 / 9668±52958.33±1.72 / 8815±41632.77±1.52 / 10719±40993.31±0.56 / 3052±136

Thank you for your positive assessment of our work and for highlighting the importance of evaluating cross-domain generalization and training stability. We have provided additional experimental results on non-mathematical reasoning tasks and independent training seeds to further address your questions.

We hope our additional results and explanations will be helpful in addressing your concerns, and we would welcome any further thoughts you may have.

Reviewer_MVsgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the problem of reasoning length misallocation in RL-trained LLMs and proposes Contrastive Accuracy Reward Estimation (CARE), a lightweight method for adaptive reasoning length control. For each question, CARE sorts online sampled responses by reasoning length, divides them into shorter and longer halves, and compares their average accuracy to determine a question-level length adjustment direction. This direction is then incorporated into a direction-conditioned reward within GRPO, encouraging longer reasoning when longer responses are more accurate, shorter reasoning when concise responses perform better, and no length adjustment when the two groups have equal accuracy. The authors conduct motivating experiments to investigate the relationship between reasoning length, accuracy, and question difficulty, showing that reasoning length primarily affects partially solvable questions and that explicit length rewards can induce undesirable training dynamics. Experiments on Qwen3-1.7B, 4B, and 8B across four mathematical reasoning benchmarks demonstrate that CARE achieves a better accuracy–efficiency trade-off than GRPO and two adaptive length-control baselines. Additional analyses examine the stability of the length preference signal, accuracy-gap magnitude weighting, length-term coefficient sensitivity, varying inference budgets, and multi-seed evaluation results.

Critical strengths
  1. The paper addresses an important and practically relevant problem. The paper goes beyond conventional reasoning length compression by investigating whether longer or shorter reasoning is actually beneficial for a given question. The distinction between question difficulty and the beneficial length adjustment is meaningful for improving the token efficiency of reasoning models.
  2. The method is simple, lightweight, and easy to integrate into existing RL frameworks. The proposed method estimates the length adjustment direction directly from online sampled responses without requiring additional rollouts, auxiliary reward models, or modifications to the underlying GRPO training pipeline.
  3. The experimental results demonstrate an accuracy–efficiency trade-off. As reported in Table 1, CARE improves average Pass@1 over GRPO by 2.64, 2.62, and 1.00 percentage points on Qwen3-1.7B, 4B, and 8B, respectively, while reducing average token usage by approximately 9.90%, 11.08%, and 7.93%. The consistent improvements in average performance across three model sizes suggest that the proposed reward design is empirically effective.
  4. The paper is clearly written and well organized. The overall presentation is clear and easy to follow. The authors establish the motivation through empirical observations before introducing CARE, making the connection between the identified problems and the proposed method intuitive.
Critical weaknesses
  1. Limited Novelty and Incomplete Related Work. The novelty of CARE is not sufficiently distinguished from recent work on adaptive reasoning length control, including LASER-D [1], GFPO [2], and Lazy Length Penalties [3]. These closely related studies are not discussed, and comparisons with them would help establish the contribution of CARE.
  2. Insufficient Validation of the Length Preference Signal. CARE determines the length adjustment direction using only 16 sampled responses per question, but the reward signals are length sensitive. Even a difference of one correct response between the two groups produces a non-zero direction signal, without considering the statistical uncertainty of this difference. Table 2 shows relatively low non-zero agreement under smaller rollout group sizes, raising concerns about the reliability of the direction signal. Independent sampling experiments would provide stronger evidence of its stability.
  3. Insufficient Ablation of Adaptive Direction Selection. The central assumption of CARE is that the accuracy difference between the shorter and longer halves of sampled responses can indicate whether extended reasoning is beneficial for a given question. However, this comparison establishes a correlation between response length and correctness rather than the causal effect of additional reasoning.
  4. Limited Experimental Generalizability and Robustness. The evaluation is restricted to mathematical reasoning tasks and three Qwen3 model sizes. Although Appendix G reports results across multiple evaluation seeds, independent training-seed experiments are missing. Broader evaluations would strengthen the generalizability and robustness claims.
  5. Incomplete Experimental Details and Result Consistency. Some results in the inference-budget analysis (Figures 25–27) appear inconsistent with Table 1. In addition, several implementation details, including training steps, checkpoint selection, and baseline-specific configurations, are insufficiently specified, making reproducibility and fair comparison difficult to assess.

Reference:

[1] Learn to Reason Efficiently with Adaptive Length-based Reward Shaping.

[2] Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoning.

[3] Shorten After You're Right: Lazy Length Penalties for Reasoning RL.

Questions for authors

Q1. How does CARE differ from recent methods such as LASER-D, GFPO, and Lazy Length Penalties? Could the authors provide more direct comparisons to clarify its novelty?

Q2. How consistent is CARE's length adjustment direction when independently sampling different response groups for the same question?

Q3. Could the authors compare CARE with a variant that only encourages shorter reasoning when a non-zero direction signal is detected, to isolate the benefit of adaptive direction selection?

Q4. Do the improvements remain consistent across independent RL training runs and other model families or reasoning domains?

5 comments
AuthorsOpenReview ↗

Response to Reviewer MVsg

We appreciate the reviewer's valuable suggestions on clarifying CARE's novelty and evaluating its robustness and generalization. We provide detailed responses to these points in the following sections.

Q1: How does CARE differ from recent methods such as LASER-D, GFPO, and Lazy Length Penalties? Could the authors provide more direct comparisons to clarify its novelty?

A1: CARE adaptively determines question-specific length adjustment directions based on the relative accuracy of short and long responses, achieving the highest average pass@1 for all three Qwen3 model sizes. Compared with LASER-D, difficulty-based length targets may not reflect the actual length–accuracy relationship, whereas CARE directly uses this relationship to guide adjustments. Compared with GFPO, filtering for length or token efficiency does not explicitly identify beneficial reasoning lengths, whereas CARE determines adjustment directions from relative accuracy. Compared with Lazy Length Penalties, which focuses on shortening excessive reasoning, CARE also encourages longer reasoning when beneficial, better balancing accuracy and efficiency. To further validate these differences, direct comparisons with these methods were conducted under the same training and evaluation settings, with the results presented in Table 1.

Table 1: Comparison with Length-Control Baselines

ModelMethodAIME2025AIME2026HMMT2026MATH500Average
Qwen3-1.7BBase26.88 / 13320.4130.21 / 13716.9016.67 / 14062.2186.47 / 5246.3640.06 / 11586.47
GRPO30.42 / 10826.5528.96 / 10245.4517.61 / 11296.0187.74 / 3829.5841.18 / 9049.40
Lazy Length Penalties31.25 / 8550.2129.69 / 8050.9518.28 / 9026.3087.80 / 2750.7441.76 / 7094.55
GFPO29.79 / 10850.3828.75 / 10250.3717.14 / 11504.9187.10 / 3793.2940.70 / 9099.74
CARE33.96 / 9352.4132.29 / 10073.9421.78 / 10176.6187.25 / 3012.7043.82 / 8153.92
Qwen3-4BBase46.46 / 12967.9350.83 / 12692.9323.30 / 13816.3790.21 / 4833.9952.70 / 11077.81
GRPO48.85 / 9334.9052.60 / 8657.4426.04 / 10112.9291.54 / 2962.3154.76 / 7766.89
Lazy Length Penalties49.79 / 7537.2953.23 / 7050.2826.80 / 8080.2491.90 / 2220.4155.43 / 6222.06
GFPO48.85 / 9400.9452.08 / 8774.2925.76 / 10263.6191.40 / 2900.2554.52 / 7834.77
CARE52.92 / 8455.0955.63 / 7817.1428.60 / 8735.8892.39 / 2618.0557.39 / 6906.54
Qwen3-8BBase45.73 / 13176.5852.19 / 12623.8725.66 / 13925.0788.07 / 5073.1752.91 / 11199.67
GRPO55.42 / 10029.5256.46 / 9246.8832.10 / 11185.8990.51 / 3022.0058.62 / 8371.07
Lazy Length Penalties55.83 / 7896.9756.77 / 7350.2432.67 / 9050.7991.20 / 2320.0059.12 / 6654.50
GFPO54.90 / 10097.3856.25 / 9460.4432.01 / 11250.3690.80 / 2970.3758.49 / 8444.64
CARE53.85 / 9166.4058.85 / 8548.7032.67 / 10310.6093.10 / 2803.1059.62 / 7707.20

Although Lazy Length Penalties achieves lower token consumption, CARE provides higher reasoning accuracy, highlighting the benefit of accuracy-aware length adaptation rather than length reduction alone.

Q2: How consistent is CARE's length adjustment direction when independently sampling different response groups for the same question?

A2: CARE's length adjustment directions remain relatively stable between independent response groups, achieving an Overall Agreement of 83.60%–95.22% and a Non-zero Agreement of 70.45%–90.52%. To evaluate direction consistency, two independent sets of responses were generated for each question through separate rollout runs. From each set, 16 responses per question were sampled to independently compute the CARE direction (+1, −1, or 0). Three metrics were reported in Table 2: Non-zero Ratio, the proportion of questions assigned a non-zero direction; Overall Agreement, the proportion of questions with identical directions in both groups; and Non-zero Agreement, the agreement rate among questions where both groups produce non-zero directions.

Table 2: Independent-Sample Direction Consistency

ModelDatasetNon-zero Ratio(%)Overall Agreement (%)Non-zero Agreement (%)
Qwen3-1.7BGSM8K45.0388.2582.76
MATH50046.2087.4081.14
Qwen3-4BGSM8K32.1595.2290.52
MATH50030.2091.4072.00
Qwen3-8BGSM8K26.6987.4970.45
MATH50026.8083.6076.87
AuthorsOpenReview ↗

Response to MVsg

Q3: Could the authors compare CARE with a variant that only encourages shorter reasoning when a non-zero direction signal is detected, to isolate the benefit of adaptive direction selection?

A3: CARE achieves higher average pass@1 than CARE-Short-Only for all three Qwen3 model sizes, demonstrating the importance of encouraging longer reasoning when beneficial rather than exclusively promoting shorter responses. Here, we introduced CARE-Short-Only, which encourages only shorter reasoning when a non-zero direction signal is detected. For comparison, CARE-Short-Only, GRPO, and the original CARE were evaluated using Qwen3-1.7B/4B/8B on AIME 2025, AIME 2026, HMMT 2026, and MATH500, reporting pass@1 (%) and average generated tokens in Table 3.

Table 3: Short-Only Ablation

ModelMethodAIME2025AIME2026HMMT2026MATH500Average
Qwen3-1.7BGRPO30.42 / 10826.5528.96 / 10245.4517.61 / 11296.0187.74 / 3829.5841.18 / 9049.40
CARE-Short-Only29.17 / 8931.6028.13 / 9396.7617.05 / 9142.6887.04 / 3279.7140.35 / 7687.69
CARE33.96 / 9352.4132.29 / 10073.9421.78 / 10176.6187.25 / 3012.7043.82 / 8153.92
Qwen3-4BGRPO48.85 / 9334.9052.60 / 8657.4426.04 / 10112.9291.54 / 2962.3154.76 / 7766.89
CARE-Short-Only48.44 / 8757.9451.46 / 7810.6725.85 / 7602.6791.36 / 1526.1254.28 / 6424.35
CARE52.92 / 8455.0955.63 / 7817.1428.60 / 8735.8892.39 / 2618.0557.39 / 6906.54
Qwen3-8BGRPO55.42 / 10029.5256.46 / 9246.8832.10 / 11185.8990.51 / 3022.0058.62 / 8371.07
CARE-Short-Only53.54 / 8454.3956.04 / 8739.8330.87 / 13170.9391.51 / 1880.1857.99 / 8061.33
CARE53.85 / 9166.4058.85 / 8548.7032.67 / 10310.6093.10 / 2803.1059.62 / 7707.20
AuthorsOpenReview ↗

Response to MVsg

Q4: Do the improvements remain consistent across independent RL training runs and other model families or reasoning domains?

A4: CARE demonstrates robust accuracy improvements under three training seeds, generalizes to the Gemma model family, and improves accuracy on non-mathematical reasoning tasks.

To assess the robustness and generalization of CARE, we evaluated the method from three perspectives: independent RL training runs (Table 4), different model families (Table 5), and non-mathematical reasoning domains (Table 6). All experiments followed the same training and evaluation settings as described in the main paper, with each entry reporting pass@1 (%) and average generated tokens.

Training-Seed Robustness. CARE generally achieves higher pass@1 than GRPO while consistently reducing token consumption in independent training runs.

Table 4: Training-Seed Robustness

ModelMethodAIME2025AIME2026HMMT2026MATH500
Qwen3-1.7BGRPO29.83±2.35 / 11032±65129.27±1.81 / 10362±41817.65±2.34 / 11307±57287.54±0.94 / 3993±205
CARE33.02±2.18 / 9845±50131.53±2.26 / 9876±43221.40±1.92 / 10243±42587.33±0.99 / 3183±189
Qwen3-4BGRPO48.09±1.94 / 9497±42152.57±2.19 / 9059±40626.58±2.10 / 10268±45991.24 ± 0.96 / 3077±144
CARE52.81±1.91 / 8600±40255.17 ± 2.03 / 8148±40328.60±1.86 / 8871±42892.27 ± 0.73 / 2780±160
Qwen3-8BGRPO55.17±2.09 / 10164±42956.04±1.60 / 9610±45432.13±1.63 / 11518±53390.28±0.82 / 3206±210
CARE53.99±1.63 / 9668±52958.33±1.72 / 8815±41632.77±1.52 / 10719±40993.31±0.56 / 3052±136

Cross-Family Generalization. CARE outperforms GRPO on Gemma models, supporting the effectiveness of our approach beyond the Qwen3 model family.

Table 5: Cross-Family Generalization

ModelMethodAIME2025AIME2026HMMT2026MATH500Average
Gemma-4-E2B-itBase29.27 / 7247.1541.15 / 7034.6519.32 / 8191.7284.41 / 2644.6743.54 / 6279.55
GRPO31.77 / 6932.1443.44 / 6730.3021.88 / 7832.2985.22 / 2832.2145.58 / 6081.74
CARE34.06 / 6753.5245.21 / 6219.7525.19 / 7539.5485.84 / 2521.3747.58 / 5758.55
Gemma-4-E4B-itBase41.77 / 6031.5951.15 / 6009.2531.25 / 6798.1489.51 / 2152.8853.42 / 5247.97
GRPO44.48 / 5936.7352.40 / 5528.4933.90 / 6533.1290.13 / 2421.5955.23 / 5104.98
CARE46.56 / 5633.2954.06 / 5638.1536.27 / 6143.8591.26 / 2219.5657.04 / 4908.71

Cross-Domain Reasoning. CARE improves reasoning accuracy on non-mathematical tasks, including SynLogic-Easy/Hard (Logical Reasoning) and GPQA-Diamond (Graduate-level Scientific Reasoning).

Table 6: Cross-Domain Reasoning

ModelMethodSynLogic-Easy (Validation)SynLogic-Hard (Validation)GPQA-Diamond
Qwen3-1.7BGRPO23.89 / 6658.7710.71 / 7706.5735.50 / 4787.10
CARE26.18 / 6743.9113.80 / 7760.8638.13 / 4859.69
Qwen3-4BGRPO37.69 / 5973.5422.76 / 7289.7549.43 / 3614.94
CARE39.77 / 5541.4725.12 / 6688.2751.77 / 3243.38
Qwen3-8BGRPO46.79 / 5736.3829.97 / 7515.9857.09 / 4790.69
CARE49.84 / 5681.0132.15 / 7601.1659.79 / 4848.71
Reviewer_tWapOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CARE, a reward design for GRPO that splits sampled responses into shorter and longer halves and uses their accuracy difference to decide whether to reward or penalize length. Experiments cover three Qwen3 model sizes and four mathematical reasoning benchmarks. The paper also examines training failures caused by fixed length rewards. Overall, CARE improves accuracy while reducing reasoning tokens.

Critical strengths
  1. CARE is simple and reuses responses already sampled by GRPO. It requires neither additional rollouts nor a separate predictor, making it practical to incorporate into existing training.

  2. In Table 1, CARE achieves the highest average Pass@1 and lowest average token usage at all three model sizes. For Qwen3-4B, it improves average accuracy over GRPO by 2.62 percentage points while using about 11.1% fewer tokens.

Critical weaknesses
  1. CARE's direction signal lacks independent validation. Section 4.1 uses correctness-dependent length normalization and shows association, not that changing length improves accuracy. Each half contains only eight responses, while Table 2 compares overlapping samples. These results do not establish that the signal reliably identifies beneficial length adjustments.

  2. The benefit of adapting directions to individual questions is not isolated. Fixed length rewards are analyzed, but the main evaluation lacks direct comparisons with them or shuffled directions. Appendix G repeats evaluation rather than training, leaving robustness across training runs untested.

Questions for authors
  1. Does the pattern in Figure 1 persist when difficulty and length normalization are estimated from separate samples? Can a direction estimated from an independent response group predict accuracy changes when the inference budget is varied for a fixed model?

  2. Under matched training steps and evaluation settings, does CARE outperform fixed short rewards, fixed long rewards, and a version that shuffles directions across questions while preserving their proportions? Please report both accuracy and token usage to isolate the value of assigning the direction to the right question.

  3. What are the total training steps, checkpoint selection rules, and method-specific parameters for ALP and DAST? Do the accuracy and token savings persist across independent training seeds? The repeated evaluations in Appendix G do not answer this question.

10 comments
AuthorsOpenReview ↗

Response to Reviewer tWap

We thank the reviewer for the constructive questions regarding experimental reliability, ablation studies, and reproducibility. We will address these questions in turn.

Q1: Does the pattern in Figure 1 persist when difficulty and length normalization are estimated from separate samples? Can a direction estimated from an independent response group predict accuracy changes when the inference budget is varied for a fixed model?

Q1.1: Does the pattern in Figure 1 persist when difficulty and length normalization are estimated from separate samples?

A1.1: The non-monotonic relationship between reasoning length and accuracy generally persists when difficulty estimation, length normalization, and accuracy evaluation use independent response samples. We conducted additional experiments on GSM8K and MATH500 using Qwen3, Gemma, and DeepSeek models. For each question, we used three independent groups of 64 responses to estimate difficulty, normalize reasoning lengths using correct responses, and evaluate accuracy, respectively. Two metrics are reported: Independent Peak Bin, defined as the normalized length interval with the highest accuracy, and Central Advantage, defined as the accuracy difference between the central interval and the average of its two neighboring intervals. The detailed results are presented in Table 1 below.

Table 1: Independent Sampling: Length–Accuracy Relationship

ModelDatasetDifficultyIndependent Peak BinCentral Advantage (pp)
Qwen3-1.7BGSM8K(0.25,0.75][-1, -0.5)1.79
MATH500(0.25,0.75][-0.5, 0)5.92
Qwen3-4BGSM8K(0.25,0.75][-1, -0.5)-5.78
MATH500(0.25,0.75][-2, -1.5)10.48
Qwen3-8BGSM8K(0.25,0.75][-1.5, -1)-7.32
MATH500(0.25,0.75][-2.5, -2)14.26
Gemma-4-E2B-itGSM8K(0.25,0.75][-1, -0.5)2.74
MATH500(0.25,0.75][-1, -0.5)2.17
Gemma-4-E4B-itGSM8K(0.25,0.75][-0.5, 0)5.35
MATH500(0.25,0.75][-1, -0.5)0.29
DeepSeek-R1-Distill-Llama-8BGSM8K(0.25,0.75][-0.5, 0)12.05
MATH500(0.25,0.75][-1, -0.5)6.97

The positive Central Advantage observed in most settings suggests that neither shorter nor longer reasoning consistently produces better accuracy.

Q1.2: Can a direction estimated from an independent response group predict accuracy changes when the inference budget is varied for a fixed model?

A1.2: Independently estimated positive directions show relatively high agreement with accuracy improvements under larger inference budgets, whereas negative directions are less reliable. We estimated the direction (+1, −1, or 0) for each question using 16 responses drawn from an independent group of 64 responses generated with a 16K budget. Accuracy was then evaluated using 64 newly sampled responses per question at each of the 4K and 8K budgets, while reusing a separate group of 64 responses for the 16K budget.

Table 2: Inference-Budget Direction Prediction

ModelDatasetDirection4K Accuracy (%)8K Accuracy (%)16K Accuracy (%)ΔAcc (pp)Agreement (%)
Qwen3-1.7BGSM8K+140.6855.4459.4318.7582.14
GSM8K-149.6669.1571.0021.3515.86
GSM8K092.0094.6194.822.83N/A
Qwen3-1.7BMATH500+117.2441.6362.8945.6589.29
MATH500-117.2646.1873.1955.930.99
MATH500071.3987.0390.9819.59N/A
Qwen3-4BGSM8K+129.4754.7656.4626.9995.45
GSM8K-136.6368.8073.0736.445.08
GSM8K092.8296.5396.723.91N/A
Qwen3-4BMATH500+119.3251.7068.1848.8681.82
MATH500-16.0725.9565.5359.470.00
MATH500064.9287.6693.3028.38N/A
Qwen3-8BGSM8K+113.6936.6141.5927.9090.48
GSM8K-135.5170.7076.3540.844.55
GSM8K093.2697.1197.334.07N/A
Qwen3-8BMATH500+115.0431.8470.5155.47100.00
MATH500-15.5722.6769.9764.400.00
MATH500062.0986.4593.1631.07N/A

As shown in Table 2, the predicted positive directions achieve relatively high agreement with observed accuracy improvements as the inference budget increases.

AuthorsOpenReview ↗

Response to Reviewer tWap

Q2: Under matched training steps and evaluation settings, does CARE outperform fixed short rewards, fixed long rewards, and a version that shuffles directions across questions while preserving their proportions? Please report both accuracy and token usage to isolate the value of assigning the direction to the right question.

A2: CARE achieves the highest average pass@1 for all three Qwen3 model sizes, outperforming fixed-length and shuffled-direction variants while generally maintaining better token efficiency. To isolate the benefit of question-specific direction selection, we conducted additional ablation experiments using three variants: Fixed-Short, Fixed-Long, and Shuffled Direction. These variants were compared against GRPO and CARE using Qwen3-1.7B/4B/8B under the same evaluation settings as in the main paper. All methods were evaluated on AIME 2025, AIME 2026, HMMT 2026, and MATH500, reporting pass@1 (%) and average generated tokens for each benchmark. The detailed results are presented in Table 3 below.

Table 3: Fixed-Length and Shuffled-Direction Ablations

ModelMethodAIME2025AIME2026HMMT2026MATH500Average
Qwen3-1.7BGRPO30.42 / 10826.5528.96 / 10245.4517.61 / 11296.0187.74 / 3829.5841.18 / 9049.40
Fixed-Short0.00 / 8.000.00 / 8.000.00 / 8.007.52 / 8.051.88 / 8.01
Fixed-Long27.19 / 14071.7731.25 / 14373.6916.38 / 14688.2685.73 / 8089.6040.14 / 12805.83
Shuffled Direction28.33 / 11506.6627.29 / 12407.0016.67 / 13372.6785.85 / 4200.8039.54 / 10371.78
CARE33.96 / 9352.4132.29 / 10073.9421.78 / 10176.6187.25 / 3012.7043.82 / 8153.92
Qwen3-4BGRPO48.85 / 9334.9052.60 / 8657.4426.04 / 10112.9291.54 / 2962.3154.76 / 7766.89
Fixed-Short41.35 / 12955.0746.77 / 12927.0223.58 / 14051.9191.21 / 3973.4250.73 / 10976.86
Fixed-Long50.10 / 12766.4753.13 / 12467.8626.70 / 13850.6090.80 / 7126.5055.18 / 11552.86
Shuffled Direction46.46 / 11305.4850.42 / 10456.1025.66 / 13273.0291.35 / 6316.7653.47 / 10337.84
CARE52.92 / 8455.0955.63 / 7817.1428.60 / 8735.8892.39 / 2618.0557.39 / 6906.54
Qwen3-8BGRPO55.42 / 10029.5256.46 / 9246.8832.10 / 11185.8990.51 / 3022.0058.62 / 8371.07
Fixed-Short52.71 / 10604.2856.98 / 9877.1630.59 / 11971.7489.74 / 2697.2957.51 / 8787.62
Fixed-Long51.88 / 12704.2556.77 / 12013.9330.49 / 13518.1589.11 / 7038.2657.06 / 11318.65
Shuffled Direction54.79 / 13008.2756.15 / 10291.4431.82 / 14024.9591.19 / 2703.6058.49 / 10007.07
CARE53.85 / 9166.4058.85 / 8548.7032.67 / 10310.6093.10 / 2803.1059.62 / 7707.20

These comparisons highlight the importance of assigning appropriate length adjustment directions to individual questions rather than applying fixed or randomly reassigned directions.

AuthorsOpenReview ↗

Response to Reviewer tWap

Q3: What are the total training steps, checkpoint selection rules, and method-specific parameters for ALP and DAST? Do the accuracy and token savings persist across independent training seeds? The repeated evaluations in Appendix G do not answer this question.

Q3.1: What are the total training steps, checkpoint selection rules, and method-specific parameters for ALP and DAST?

A3.1: All methods were trained for 80 steps under matched configurations, with consistent checkpoint selection criteria and method-specific hyperparameters following the original ALP and DAST papers. Specifically, each training step contained 256 questions, with a mini-batch size of 16 questions (256 responses) for gradient updates. ALP used a length penalty coefficient of β = 1e-7, while DAST adopted β = 200 and γ = 1 for its preference optimization objective. Checkpoint selection was based on the stabilization of training rewards and the convergence of validation accuracy, using the same selection criteria for all methods.

Q3.2: Do the accuracy and token savings persist across independent training seeds?

A3.2: CARE achieves higher average pass@1 than GRPO on most benchmarks and consistently reduces token consumption in three independent training runs. For a more comprehensive evaluation of CARE's robustness, we conducted three independent training runs with different random seeds for GRPO and CARE. The resulting checkpoints were evaluated under the same settings as in the main paper, with Table 4 reporting the mean and standard deviation of pass@1 and average generated tokens.

Table 4: Training-Seed Robustness

ModelMethodAIME2025AIME2026HMMT2026MATH500
Qwen3-1.7BGRPO29.83±2.35 / 11032±65129.27±1.81 / 10362±41817.65±2.34 / 11307±57287.54±0.94 / 3993±205
CARE33.02±2.18 / 9845±50131.53±2.26 / 9876±43221.40±1.92 / 10243±42587.33±0.99 / 3183±189
Qwen3-4BGRPO48.09±1.94 / 9497±42152.57±2.19 / 9059±40626.58±2.10 / 10268±45991.24 ± 0.96 / 3077±144
CARE52.81±1.91 / 8600±40255.17 ± 2.03 / 8148±40328.60±1.86 / 8871±42892.27 ± 0.73 / 2780±160
Qwen3-8BGRPO55.17±2.09 / 10164±42956.04±1.60 / 9610±45432.13±1.63 / 11518±53390.28±0.82 / 3206±210
CARE53.99±1.63 / 9668±52958.33±1.72 / 8815±41632.77±1.52 / 10719±40993.31±0.56 / 3052±136

Comments on the paper

3 comments
AuthorsOpenReview ↗

Author Final Remarks

Dear Area Chair and Reviewers,

We are sincerely grateful for the time and effort devoted to evaluating our submission and for the thoughtful feedback provided throughout the discussion period. To make our responses easier to review, we have consolidated the principal concerns, additional experiments, and corresponding clarifications in our latest official comments, General Response (1/2) and General Response (2/2).

The discussion has helped us substantially strengthen the empirical evaluation and clarify several important aspects of CARE. In particular, we conducted additional direction-signal ablations, independent-sampling and multi-seed experiments, and broader baseline and generalization evaluations.

For convenience, we summarize the current review and discussion status below:

ReviewertWapMVsgX5xV
Final RatingTo be determined3: Weak acceptance3: Weak acceptance
Original Rating2: Weak rejection2: Weak rejection3: Weak acceptance
Willing to Raise ScoreYesYes—
Related Comments“My remaining concern is the independent validity of CARE’s direction signal.”“I have updated my rating from 2 to 3.”—

We appreciate Reviewer MVsg for carefully considering our rebuttal, confirming that all concerns have been addressed, and raising the rating from 2 to 3.

We also sincerely thank Reviewer tWap for the valuable discussions throughout the rebuttal period. Although we have not yet received a response to our latest clarification, we believe our additional explanations and experimental evidence can address the remaining concern about CARE's direction signal, which was the main reason for maintaining the original rating at this stage.

We are also grateful to Reviewer X5xV for the positive assessment of our work. Although we have not yet received any feedback, we are confident that our additional experiments and responses address the questions raised in the review, and hope the reviewer might consider raising the rating.

We hope that the additional experiments and explanations provided in the rebuttal and General Response offer a clearer and more complete assessment of CARE and assist the Area Chair and reviewers in their final decision. Thank you again for your careful consideration and for the constructive feedback that has improved our work.

Best regards,

The Authors of Submission 28930

AuthorsOpenReview ↗

General Response (1/2)

We would like to extend our sincere gratitude to the Area Chair and all reviewers for their valuable feedback and constructive suggestions. For your convenience, we have prepared a summary of our responses and outlined how we have addressed the reviewers’ concerns below. We sincerely hope that this summary will facilitate your assessment and lighten your workload.

Our paper has received encouraging positive feedback from the reviewers. Reviewer MVsg recognized that CARE addresses “an important and practically relevant problem” and described the method as “simple, lightweight, and easy to integrate into existing RL frameworks.” Reviewer tWap highlighted that CARE “requires neither additional rollouts nor a separate predictor” and achieves the highest average Pass@1 and lowest average token usage at all three model sizes. Reviewer X5xV also acknowledged CARE’s simple integration into GRPO without additional inference cost, as well as its improvements in reasoning accuracy and token efficiency.

We outline how we have addressed the concerns raised by each reviewer as follows.

Common Concerns

(Reviewers MVsg, tWap, and X5xV) 1. Reliability and independent validation of CARE's length preference signal.

We conducted additional independent-sampling experiments to examine the reliability of CARE's direction signal and the length–accuracy relationship.

  1. We evaluated direction consistency across independently sampled response groups, using two separate sets of 16 responses per question. For different model–benchmark configurations, Overall Agreement ranges from 83.60% to 95.22%, while Non-zero Agreement ranges from 70.45% to 90.52%.
  2. We independently estimated question difficulty, normalized reasoning lengths, and evaluated accuracy using three separate groups of 64 responses per question. The additional experiments across Qwen3, Gemma, and DeepSeek models show that the non-monotonic length–accuracy relationship generally persists under independent sampling.
  3. We clarified that CARE uses the relative accuracy of shorter and longer sampled responses as an empirical reward-shaping signal, rather than assuming a causal relationship between response length and correctness.

These results support the stability of CARE's direction signal under independent sampling, while clarifying its role as an empirical reward-shaping signal.

(Reviewers MVsg and tWap) 2. Insufficient comparisons and ablations on adaptive reasoning-length control.

We substantially expanded the experimental comparisons to better isolate the contribution of CARE's question-specific direction assignment.

  1. We introduced comparisons with recent reasoning-length optimization methods, including GFPO and Lazy Length Penalties.
  2. We conducted additional ablations with Fixed-Short, Fixed-Long, and Shuffled Direction variants, directly examining whether question-specific direction assignment provides benefits over fixed or randomly reassigned length rewards.
  3. We also introduced CARE-Short-Only, which only encourages shorter reasoning, to isolate the benefit of allowing both positive and negative length adjustments.

These comparisons show that CARE achieves the highest average Pass@1 at all three model sizes, supporting the effectiveness of question-specific adaptive length control.

(Reviewers MVsg, tWap, and X5xV) 3. Limited generalization, training-seed robustness, and experimental reproducibility.

We expanded the evaluation in three directions:

  1. Cross-Family Generalization: We trained and evaluated CARE on Gemma-4-E2B-it and Gemma-4-E4B-it. CARE outperforms GRPO in both average accuracy and token efficienc.
  2. Cross-Domain Reasoning: We extended evaluation beyond mathematical reasoning to SynLogic-Easy, SynLogic-Hard, and GPQA-Diamond. CARE improves Pass@1 over GRPO at these non-mathematical reasoning benchmarks.
  3. Training-Seed Robustness: We conducted three independent RL training runs with different random seeds for GRPO and CARE. CARE achieves higher Pass@1 in 10 of the 12 model–benchmark comparisons and consistently reduces average generated tokens across all 12 comparisons.

We also clarified the training configurations and baseline specific hyperparameters, providing further details to support the reproducibility and fairness of our comparisons.

AuthorsOpenReview ↗

General Response (2/2)

Reviewer MVsg

4. Novelty and differences from recent reasoning-length optimization methods.

We clarified CARE's differences from LASER-D, GFPO, and Lazy Length Penalties. Unlike methods that rely on difficulty-based length targets, response filtering, or primarily encouraging shorter reasoning, CARE estimates question-specific length adjustment directions from the relative accuracy of shorter and longer responses, allowing it to encourage either longer or shorter reasoning when appropriate.

We also conducted direct comparisons with GFPO and Lazy Length Penalties. CARE achieves the highest average Pass@1 at these three Qwen3 model sizes.

Reviewer tWap

5. Interpretation of CARE's direction signal and the influence of inference-budget truncation.

We conducted additional experiments examining accuracy and truncation rates under different inference budgets (4K, 8K, and 16K). The results show that truncation rates decrease substantially as the inference budget increases, indicating that accuracy improvements under larger budgets is influenced by reduced truncation rather than the predictive validity of CARE's direction signal.

We therefore clarified an important distinction: CARE adjusts the distribution of response lengths through reward shaping while keeping the maximum inference limit fixed, whereas changing the inference budget modifies the maximum allowed generation length. Consequently, the direction signal is not intended to predict accuracy changes in different inference limits.

Instead, our central claim is that question-specific length reward shaping improves the accuracy–efficiency trade-off under the same inference limit. Our Fixed-Short, Fixed-Long, and Shuffled Direction ablation studies provide direct empirical evidence supporting this claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.