acceptodds
Under review as a conference paper at ICLR 2027

CAP: Commitment-Aware Penalties for Training and Deploying Reasoning Models

Abstract

Reasoning models often keep writing after they have internally settled on an answer, and much of this additional output does not change the final result. Reinforcement learning methods that penalize length address this by cutting text indiscriminately, at a cost that falls hardest on the problems that need long reasoning most. We therefore propose CAP, Commitment-Aware Penalty: a lightweight probe reads the model's internal commitment state from hidden states during a forward pass that GRPO already performs, so the penalty applies only to text generated after commitment. We train this probe separately on each of six models from five architectures. We then test the resulting reward in two settings: (1) At training, a single full-scale run on Qwen3-8B shows no detectable accuracy difference from outcome-only GRPO on four sets not used for checkpoint selection, while generating 9–28% fewer tokens per set. A conventional length penalty compresses more aggressively but trails CAP by 8.7 points on MMLU-Pro and 6.3 on AIME 2025; only the MMLU-Pro gap survives multiple-comparison correction. (2) At deployment, the same probe can instead be used on base models as a stopping rule: evaluated on every trace, it removes over half the chain-of-thought tokens on MATH-500 for under 3 accuracy points, losing fewer points than fixed-fraction truncation at similar savings and far fewer than confidence-triggered stopping. This operating point, however, does not transfer to harder benchmarks without re-tuning.

Primary area:
interpretability and explainable AI
Keywords:
reasoning models, reinforcement learning, reward shaping, interpretability probes, test-time compute

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_f5nmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies models that continue reasoning after they have settled on an answer. Their method uses classifier probes trained on the model’s hidden states to detect this point, then they penalize continued reasoning during GRPO training. The same classifier can also stop reasoning at inference time. The paper reports shorter responses with similar average accuracy to standard GRPO. (They also report substantial accuracy drops on harder benchmarks.)

Critical strengths

S1. The problem and the method intuitively make sense. Targeting reasoning after answer commitment could preserve useful computation better than penalizing all generated text (especially for problems that require longer reasoning).

Critical weaknesses

W1. I am not convinced that the methodological contribution is sufficiently significant relative to existing work. The commitment probe is used by [1], forced exits are used by [2], and ProFIL already uses a commitment-related probe to modify GRPO training. The main difference seems to be just changing the reward function by applying a graded penalty.

Comparisons with closely related efficient-reasoning methods are also missing (there are many training optimization methods like [3]). The comparisons against inference-time works also lack (such as [4]), and also many entropy-based methods.

[1] Reasoning Models Know When They’re Right [2] LYNX: Learning Dynamic Exits for Confidence-Controlled Reasoning [3] SLAT: Segment-Level Adaptive Trimming for Efficient CoT Reasoning [4] Dynamic Early Exit in Reasoning Models

W2. The experiments do not establish that CAP offers a better accuracy–cost tradeoff than simpler alternatives. In the main comparison, CAP uses 6.5k tokens on average, while the conventional length penalty uses 4.8k. Its higher accuracy may therefore come from removing less reasoning rather than better identifying which reasoning to remove.

The authors also test a weaker length penalty halfway through training, where its average accuracy is slightly higher than CAP, but do not report its final results. I would like to see this comparison completed and Pareto curves showing accuracy against token cost under matched training and tuning budgets. This would help establish whether the commitment probe adds value beyond a less aggressive length penalty.

Questions for authors

Q1. Do you have full-scale results for the weaker length penalty or accuracy–token Pareto curves?

Q2. Have you checked whether the probe still detects commitment correctly after the full CAP training run? A lower penalty could mean the model produces less unnecessary reasoning, but it could also mean the probe detects commitment later or misses it. The current check seems limited to the smaller training experiment.

Q3. The stopping threshold was chosen using the MATH-500 evaluation traces, but the same setting causes substantial accuracy losses on other benchmarks. Could these losses come from stopping too early on harder tasks? Have the authors evaluated a threshold chosen on a separate validation set to distinguish a poorly chosen threshold from limitations of the probe itself?

Reviewer_X7gyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a well‑known pain point in RL‑VR reasoning models: vanilla global length penalties indiscriminately compress all reasoning tokens, degrading performance on hard problems requiring genuine long chains. The authors propose CAP (Commitment‑Aware Penalty). A frozen, pre‑trained hidden‑state probe detects the commitment boundary; during RL training, penalty is only applied to tokens after this boundary, avoiding penalizing necessary pre‑commitment reasoning. The same probe doubles as an inference‑time early‑stopping heuristic to prune redundant post‑commitment CoT text.

Empirically, on Qwen3‑8B full‑scale training, CAP retains comparable overall accuracy to outcome‑only GRPO while reducing token consumption by 9‑28%, outperforming blind length penalty which substantially hurts hard‑set performance such as MMLU‑Pro. Inference‑time early‑exit achieves strong token‑accuracy trade‑off on MATH‑500, though the optimal stopping threshold transfers poorly across benchmarks. The paper also conducts careful perturbation ablation, cross‑model transfer tests, and auditing for probe‑gaming failure modes.

Critical strengths
  1. Important practical problem and clean conceptual motivation. Length‑reduction for reasoning RL is highly relevant for deployment. The key insight — do not penalize genuinely necessary pre‑commitment reasoning, only penalize redundant post‑commitment text — is intuitive and well‑grounded in prior interpretability findings about decorative CoT. The authors clearly contrast their idea against naive global length penalty, confidence‑based stopping, and surface‑text heuristics.
  2. Careful, multi‑faceted empirical evaluation with rich ablations and sanity checks.
  • Distinguishes diagnostic‑scale sweep and full‑scale training;
  • Extensive controls: confidence‑twin reward, answer‑text‑rule reward, ProFIL‑style advantage‑filter baseline;
  • Perturbation studies verify what happens when corrupt text pre‑ / post‑commitment;
  • Explicit audit for policy gaming: regenerate traces and re‑compute ground‑truth commitment labels via force‑decoding, to test whether RL policy manipulates probe signals;
  • Evaluates both training‑time reward shaping and inference‑time early‑exit usage of the same probe.
Critical weaknesses
  1. The commitment boundary signal itself is model‑dependent and fragile. CAP’s effectiveness hinges on a strong assumption: text after commitment boundary is mostly inert and rarely changes answer. But experiments show this assumption fails for some model families (e.g., Llama‑8B, where perturbing post‑commitment numeric literals still alters answers). The paper shows CAP brings negligible improvement on such models. There is no analysis about what model properties determine whether commitment signal is trustworthy. Also, probe accuracy drifts after RL for all policies (including baseline GRPO which never uses CAP reward), distribution shift degrades probe performance.
  2. Inference‑time early‑exit lacks robust cross‑dataset operating point. The good MATH‑500 trade‑off (k=10k=10) cannot directly reuse on harder benchmarks like AIME / ZebraLogic, leading to large accuracy degradation. The paper does not propose an adaptive thresholding strategy; practitioners would need per‑dataset held‑out data to tune k, limiting real‑world usability.
  3. Reward design is heuristic, lacks theoretical grounding. The fraction‑form penalty ϕ\phi has known design defects: policy could delay commitment to reduce penalty pressure. While authors audit and report no strong gaming signature in their setting, this risk is not formally eliminated and could manifest under different hyperparameters / datasets. CAP‑T (targeted absolute penalty) solves that defect but collapses at full‑scale training; why fraction‑form works better is only offered as post‑hoc explanation.
Questions for authors
  1. Can you design an adaptive inference‑time stopping rule that avoids per‑dataset manual tuning of hyper‑parameter k?
  2. How does CAP compare against PRM‑based length‑shaping reward in your diagnostic setup?
Reviewer_Ym4EOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose a new method to shorten the reasoning traces of thinking language models, with minimal performance drops during inference and no detectable performance drop in training. The approach can be applied both to GRPO training, as a new penalty, and to inference, where it stops generating new tokens once a lightweight probe trained on the model’s hidden states labels the answer as finalised in the reasoning trace (though this results in significant performance drops on harder tasks). The authors evaluate their models on four reasoning benchmarks and provide ablations for their design decisions. They compare their method to naive length-penalty and no-penalty baselines, as well as to other length penalties.

Critical strengths
  1. Statistical validation of their claims.
  2. Actual inference and training optimization.
  3. Reasonable baselines: both a naive length penalty and a no-penalty baseline are included, suggesting that the proposed method represents the actual sweet spot
Critical weaknesses
  1. There is no analysis of what exactly is being cut this way. Is this really something unnecessary, or are these particular tasks simply not tricky enough for double-checks to make a difference?
  2. The sentence-level penalty is vulnerable to stopping the generation of sentence separators. Ideally, the method should somehow avoid this. A reasonable approach would be to use a token-level penalty normalized by sequence length, as they show that the absolute one results in lower performance. Please also report the fraction of rollouts with no detected boundary during training.
  3. Even after RL training, these models retain these reasoning traces, presumably because they usually contain double-checks or critical analysis, which is not necessarily something that is free to sacrifice.
  4. Wrong answers are also accounted for by this penalty. A correctness-conditioned variant appears in the Table 12 registry (one seed, diagnostic scale) but is not discussed. Please report it and explain why penalizing incorrect rollouts is preferred.
Questions for authors

Please address the concerns above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.