acceptodds
Under review as a conference paper at ICLR 2027

Disentangled Token-Level Credit Assignment for Efficient Reasoning

Abstract

Large reasoning models (LRMs) rely on explicit multi-step reasoning to substantially improve complex problem solving but are also prone to overthinking, which consumes computation with no gain in accuracy and can even derail otherwise correct trajectories. Existing approaches primarily mitigate overthinking through length penalties. However, such sequence-level signals penalize an entire long response uniformly, which suppresses useful reasoning along with redundant computation. To address this limitation, we propose Per-Objective Asymmetric Credit Estimation (PACE), which disentangles correctness from efficiency and assigns their learning signals separately at the token level, charging each generation decision for the computation it induces while reinforcing correct reasoning. Specifically, PACE employs intra-prompt correctness credit to characterize relative solution quality among responses to the same prompt, while inter-prompt rate credit apportions success gains and remaining-compute penalties to individual tokens. We further propose an asymmetric correctness budget that bounds negative efficiency credit on correct trajectories and retains the full remaining-compute penalty on erroneous trajectories. Theoretically, we establish that the inter-prompt rate credit constitutes a token-level cross-fitted estimator of the pooled solve-throughput gradient, yielding a principled decomposition of the global accuracy-compute objective into token-wise success gains and remaining-compute penalties. Extensive experiments across ten reasoning benchmarks and two model scales demonstrate that PACE reduces generation length up to 64% while improving average accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
efficient reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ABaEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PACE, a reinforcement learning method for improving the efficiency of language-model reasoning. PACE combines a sequence-level, group-relative correctness advantage with a token-level efficiency advantage motivated by the ratio of expected correctness to expected generation length. The efficiency component uses remaining response length to assign position-dependent costs and leave-one-prompt-out estimates to construct its credit signal. A lower bound limits the negative efficiency contribution on correct responses. The paper provides an unbiasedness result, up to a positive scaling factor, for the rate estimator before applying this bound. Experiments on DeepSeek-R1-Distill-Qwen-1.5B and 7B report substantial reductions in generated tokens while maintaining or improving accuracy across the evaluated reasoning benchmarks.

Critical strengths
  • The reported results demonstrate substantial reductions in output length while retaining competitive accuracy. This is a practically relevant objective for reducing the token cost of reasoning models.

  • The paper is generally clear and easy to follow. The mathematical formulation provides a useful motivation for the credit estimator, and the component ablations offer some evidence for the proposed design choices.

Critical weaknesses
  • The effect of the position-dependent penalty on useful early reasoning is insufficiently examined. Earlier tokens receive a larger remaining-length cost than later tokens, although early decisions can strongly influence subsequent reasoning. I recognize that this weighting follows a reward-to-go formulation and that the proposed bound ensures nonnegative combined advantages for correct responses. However, this sign constraint does not establish that useful early-token learning signals are sufficiently preserved. The paper would benefit from examining whether the efficiency term substantially suppresses these signals and whether this affects reasoning quality.

  • A direct comparison with a matched sequence-level penalty is missing. Token-level efficiency credit is a central design choice, but the current analysis does not establish its advantage over a corresponding sequence-level estimator. Before the nonlinear bound and policy clipping, the remaining-length and total-length formulations can yield the same expected policy gradient. Thus, unbiasedness alone does not explain why the token-level formulation should perform better. A particularly informative ablation would replace the remaining-length term with the sequence-level term, while retaining the same cross-fitting procedure, correctness advantage, protection bound, and training setup. This would help determine whether the proposed credit allocation provides benefits beyond the other components.

  • The main comparisons do not sufficiently isolate the contribution of the proposed method. The authors train PACE and compare it with publicly released baseline checkpoints under a unified evaluation protocol. This provides a useful comparison of the resulting models, but differences in training data, adaptation settings, optimization procedures, and training budgets remain potential confounding factors. Consequently, the main results do not establish whether the gains arise specifically from PACE. I recommend controlled retraining of, at minimum, a correctness-only RL baseline, a simple length-regularized baseline, and one strong competing method under matched training conditions. The existing component ablations partially address attribution, but do not replace these comparisons.

  • The evaluation provides limited evidence of generalization across model families. Both evaluated models belong to the DeepSeek-R1-Distill-Qwen family. Although the results cover two model sizes, they do not establish effectiveness across different backbones or training histories. Since the method aims to reduce the generation cost of already capable reasoning models, its effectiveness may depend on their initial reasoning behavior and degree of verbosity. Evaluation on another model family, such as Llama or Gemma, would strengthen the generality of the conclusions.

Questions for authors
  • How does PACE compare with the matched sequence-level variant? Is there evidence that token-level credit improves gradient variance, training stability, or the final accuracy–length trade-off?

  • How frequently is the correctness protection bound activated at different token positions? Does the efficiency term substantially reduce the positive learning signal for early tokens in correct responses?

  • Can the authors provide comparisons with representative baselines trained under the same initialization, data, adaptation setup, and training budget?

  • Does the method retain its benefits on a model family other than DeepSeek-R1-Distill-Qwen?

Reviewer_NtiLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses overthinking in large reasoning models, where models keep verifying and backtracking after a valid solution is found, and argues that existing sequence-level length penalties wrongly spread negative credit across all tokens of a response. It proposes PACE, which separates correctness from efficiency and assigns token-level signals: an intra-prompt correctness credit for relative solution quality, and an inter-prompt rate credit that uses leave-one-prompt-out cross-fitting to attribute success gains and remaining-compute penalties to individual tokens, directly targeting solve-throughput. An asymmetric correctness budget bounds negative rate credit on correct trajectories while keeping the full penalty on incorrect ones. Experiments train DeepSeek-R1-Distill-Qwen-1.5B/7B on DAPO-Math-17k and evaluate on 10 math and science benchmarks against 9 efficient-reasoning baselines, reporting 45–64% shorter generations, higher average accuracy, larger shrinkage of incorrect than correct responses, and higher accuracy at every matched thinking budget.

Critical strengths
  1. The idea of decomposing the solve-throughput gradient into token-level success gains and remaining-compute penalties is interesting and innovative.
  2. In the context of efficient reasoning, the credit assignment problem in RL is a very important topic.
  3. A large number of experiments were conducted to verify the effectiveness of the method, which is fairly thorough.
Critical weaknesses
  1. The experiments are based on the DeepSeek-R1-Distill series, which is a relatively old series. Did the authors test PACE on newer models such as Qwen3? Alternatively, please specify the concrete reasons for choosing the DeepSeek-R1-Distill-Qwen-1.5B/7B models rather than newer ones.
  2. Hyperparameter sensitivity is not sufficiently explored: key hyperparameters such as γ are fixed, and sensitivity analysis is lacking. The lower bound of the asymmetric budget depends on γ, and it is unknown whether it remains stable across models/tasks (e.g., code generation). I suggest the authors add some guidance on how to tune this parameter or empirical conclusions.
  3. In Table 3, the column names are methods and the row names are benchmarks, which is the opposite of common writing practice and reduces readability.
Questions for authors

See Weaknesses please.

Reviewer_N2qsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PACE, a token-level credit assignment method for efficient reasoning. The motivation is that RL with sequence-level length penalties penalizes all tokens in a trajectory equally, whereas earlier generation decisions should receive larger compute penalties because they induce more subsequent computation. PACE separates correctness and efficiency credit while limiting efficiency penalties on correct trajectories to preserve useful reasoning. The authors train the DeepSeek-R1-Distill-Qwen-1.5B,7B using PACE and report promising results on math benchmarks.

Critical strengths

The main results in Tables 1 and 2 look good. PACE consistently reduces length while mostly preserving accuracy. The main idea is also interesting: assigning compute penalties based on the remaining generation length could show the downstream cost of each token decision.

Critical weaknesses
  1. Novelty and missing baselines. Several prior works, such as [1,2], already address token-level credit assignment for efficient reasoning. The authors neither discuss these works nor compare against them, making PACE's contribution beyond these methods unclear. More broadly, the literature review is incomplete. Several inference-time early stopping methods [3,4,5] pursue the same goal without requiring training. Additionally, [6] reports strong results on the same models and benchmarks as PACE using a simple length penalty, making it a particularly relevant baseline.

  2. Model coverage. The main results are limited to DeepSeek-R1-Distill-Qwen-1.5B and 7B. These models are somewhat outdated and tend to generate excessively long reasoning traces. It is unclear whether PACE would also be effective on more efficient models, such as Gemma 4, Qwen3, and gpt-oss.

  3. Most token savings come from incorrect trajectories. The authors report that most token savings come from incorrect trajectories. I see two possible explanations: first, the model may be learning its capability boundaries, recognizing when a question is too difficult and stopping early to save tokens. Second, the model may be becoming less effective at exploration, which could hurt performance on harder problems. The authors do not provide evidence to distinguish between these explanations, and I strongly suspect the latter may be the case.

[1] He, Yinhan, et al. "IAPO: Information-Aware Policy Optimization for Token-Efficient Reasoning."

[2] Li, Xintong, et al. "Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoning."

[3] Yang, Chenxu, et al. "Dynamic Early Exit in Reasoning Models."

[4] Hosseini, Parsa, et al. "Early Stopping for Large Reasoning Models via Confidence Dynamics."

[5] Wang, Xi, et al. "Entropy After <\Think> for Reasoning Model Early Exiting."

[6] Zhao, Anhao, et al. "On-policy supervised fine-tuning for efficient reasoning."

Questions for authors

Could the authors evaluate PACE on harder problems to assess whether it still preserves accuracy? For example, they could report accuracy on a hard subset of AIME and include challenging coding benchmarks such as LiveCodeBench, and also include pass@k for current evaluations to evaluate exploration. Also, it would be nice to include error bars for current results to see if accuracy differences from different methods are statistically significant.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.