- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PACE, a reinforcement learning method for improving the efficiency of language-model reasoning. PACE combines a sequence-level, group-relative correctness advantage with a token-level efficiency advantage motivated by the ratio of expected correctness to expected generation length. The efficiency component uses remaining response length to assign position-dependent costs and leave-one-prompt-out estimates to construct its credit signal. A lower bound limits the negative efficiency contribution on correct responses. The paper provides an unbiasedness result, up to a positive scaling factor, for the rate estimator before applying this bound. Experiments on DeepSeek-R1-Distill-Qwen-1.5B and 7B report substantial reductions in generated tokens while maintaining or improving accuracy across the evaluated reasoning benchmarks.
Critical strengths
-
The reported results demonstrate substantial reductions in output length while retaining competitive accuracy. This is a practically relevant objective for reducing the token cost of reasoning models.
-
The paper is generally clear and easy to follow. The mathematical formulation provides a useful motivation for the credit estimator, and the component ablations offer some evidence for the proposed design choices.
Critical weaknesses
-
The effect of the position-dependent penalty on useful early reasoning is insufficiently examined. Earlier tokens receive a larger remaining-length cost than later tokens, although early decisions can strongly influence subsequent reasoning. I recognize that this weighting follows a reward-to-go formulation and that the proposed bound ensures nonnegative combined advantages for correct responses. However, this sign constraint does not establish that useful early-token learning signals are sufficiently preserved. The paper would benefit from examining whether the efficiency term substantially suppresses these signals and whether this affects reasoning quality.
-
A direct comparison with a matched sequence-level penalty is missing. Token-level efficiency credit is a central design choice, but the current analysis does not establish its advantage over a corresponding sequence-level estimator. Before the nonlinear bound and policy clipping, the remaining-length and total-length formulations can yield the same expected policy gradient. Thus, unbiasedness alone does not explain why the token-level formulation should perform better. A particularly informative ablation would replace the remaining-length term with the sequence-level term, while retaining the same cross-fitting procedure, correctness advantage, protection bound, and training setup. This would help determine whether the proposed credit allocation provides benefits beyond the other components.
-
The main comparisons do not sufficiently isolate the contribution of the proposed method. The authors train PACE and compare it with publicly released baseline checkpoints under a unified evaluation protocol. This provides a useful comparison of the resulting models, but differences in training data, adaptation settings, optimization procedures, and training budgets remain potential confounding factors. Consequently, the main results do not establish whether the gains arise specifically from PACE. I recommend controlled retraining of, at minimum, a correctness-only RL baseline, a simple length-regularized baseline, and one strong competing method under matched training conditions. The existing component ablations partially address attribution, but do not replace these comparisons.
-
The evaluation provides limited evidence of generalization across model families. Both evaluated models belong to the DeepSeek-R1-Distill-Qwen family. Although the results cover two model sizes, they do not establish effectiveness across different backbones or training histories. Since the method aims to reduce the generation cost of already capable reasoning models, its effectiveness may depend on their initial reasoning behavior and degree of verbosity. Evaluation on another model family, such as Llama or Gemma, would strengthen the generality of the conclusions.
Questions for authors
-
How does PACE compare with the matched sequence-level variant? Is there evidence that token-level credit improves gradient variance, training stability, or the final accuracy–length trade-off?
-
How frequently is the correctness protection bound activated at different token positions? Does the efficiency term substantially reduce the positive learning signal for early tokens in correct responses?
-
Can the authors provide comparisons with representative baselines trained under the same initialization, data, adaptation setup, and training budget?
-
Does the method retain its benefits on a model family other than DeepSeek-R1-Distill-Qwen?