acceptodds
Under review as a conference paper at ICLR 2027

Same Candidates, Stronger Reasoning: Rethinking What Post-Training Learns

Abstract

Post-training transforms pretrained language models into notably stronger reasoners. Yet recent evidence suggests that the changes driving these gains are concentrated at a small number of token-level decisions. This raises a fundamental puzzle: at these consequential decisions, does post-training rely on candidates beyond the pretrained model's high-ranked set, or primarily reorganize preferences among them? Across 31 model–task–checkpoint settings, 92.9% of post-training Top-1 changes remain within the pretrained Top-16 on average, while their relative preferences are substantially reorganized. More strikingly, this inheritance persists even on successful post-training-induced prefixes that are not visited by the pretrained policy. Accumulated local preference changes can therefore compose into genuinely new global competence. Motivated by this finding, we introduce ProQ, a post-training paradigm that shifts the learning target from full-distribution next-token adaptation to value-guided selection over the model's own proposals. Within a single LLM and shared vocabulary, standard-token logits retain normal reasoning and candidate proposal, regularized toward the pretrained model; reserved-token logits are repurposed to produce candidate-conditioned value scores. These scores provide a natural RL interface for local selection. Trained only on mathematical reasoning, ProQ achieves strong gains on math benchmarks and transfers to out-of-domain coding and knowledge tasks; the benefits persist across different model sizes and families.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, post-training, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nFQWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether post-training mainly changes the order of the base model’s existing token candidates. It also proposes ProQ, which scores the model’s own next-token candidates and picks one. Experiments cover three backbone models and tasks in mathematics, coding, and knowledge reasoning.

Critical strengths
  • The analysis covers several model families, model sizes, tasks, and training stages.
  • The proposed method follows naturally from the analysis.
  • The results are promising. ProQ improves math performance and also helps on code and knowledge tasks.
Critical weaknesses
  • The paper asks whether post-training uses new, low-ranked candidates or changes the order of existing candidates. However, [1] already studies this question in Sections 4.1 and 4.2. It examines candidate overlap, rank changes, and tokens with low base-model probabilities. It also compares the models on RL-generated text and tests the effect of replacing selected tokens. Similarly, [2] separates changes within the base model’s top candidates from changes outside them. I am not sure whether the analysis in this paper provides new findings beyond these studies. If it does, I would appreciate it if the authors could clarify and discuss these contributions during the rebuttal.

  • S2T-Local [3] already uses the same model to score candidates through reserved tokens. ProQ mainly changes the training signal from teacher scores to the final correctness of candidate continuations. The current tests are hard to tell whether the gains mainly come from.

Ref:

[1] Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs. ICLR 2026.

[2] Rethinking RL for LLM Reasoning: It’s Sparse Policy Selection, Not Capability Learning. arXiv:2605.06241, 2026.

[3] Select to Think: Unlocking SLM Potential with Local Sufficiency. ICML 2026.

Questions for authors
  • What new finding comes from the full SFT/DPO/RL stage analysis, beyond the findings in [1] and [2]?
  • With the same scoring model, how does outcome supervision compare with the teacher supervision used in [3]?
  • How much accuracy is gained or lost when changes inside or outside the base model’s Top-K are kept or removed?
Reviewer_7HXCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether reasoning improvements from post-training come from introducing new token candidates or changing preferences among candidates already available in the Base model. The authors find that most post-training decisions remain within the Base model's Top-K candidates. Based on this observation, they propose ProQ, which learns to select candidates using value scores while preserving the original token distribution. Experiments across multiple models and benchmarks show improvements in mathematical reasoning and out-of-domain tasks.

Critical strengths

The paper studies an interesting question about how post-training improves reasoning capabilities and provides extensive analysis across different models and training stages. The proposed ProQ method provides a simple way to improve reasoning by learning to select among existing candidates rather than directly optimizing the full token distribution.

Critical weaknesses
  1. The paper shows that 92.9% of changed Top-1 decisions are within the Base model's Top-16 candidates on average. However, this does not necessarily mean that the reasoning improvements come from reordering these candidates. A small number of decisions outside the Base Top-16 could still be critical for solving a problem. Although Section 3.3 analyzes successful post-training trajectories, it does not directly test whether these trajectories would remain successful if the model were restricted to the Base candidate set.
  2. In Section 4.2, the candidate values are supervised using outcomes from the fixed Base model's continuations. Appendix B.3 further specifies that only one continuation is used per candidate. However, during inference, ProQ changes token selections and therefore generates different trajectories. The value of a candidate under the Base continuation policy may differ from its value under ProQ's own continuation policy. The paper does not directly establish whether the learned value rankings remain accurate under this change.
  3. ProQ evaluates multiple candidates at selected positions, while GRPO, DAPO, and RLPT use standard decoding. According to Table A6, ProQ increases inference time by 35.0% to 54.5% compared with the Base model. Therefore, it is difficult to determine how much of the reported improvement comes from the proposed learning method versus additional test-time computation. A comparison under matched inference budgets would better demonstrate the advantage of ProQ.
  4. ProQ uses reserved-token logits to represent candidate values within the same LLM. However, the paper does not compare this design with a conventional value head trained on the same candidate-conditioned data. Although the authors provide ablations on the ranking objectives and KL regularization, these experiments do not establish whether the reserved-token design provides advantages over simpler candidate-scoring approaches.
Questions for authors
  1. Can the authors directly verify that reasoning improvements come from reordering the Base model's existing candidates? For example, what happens to the post-trained model's accuracy if its token selections are restricted to the Base Top-8 or Top-16 during generation?
  2. How reliable are the candidate value scores when ProQ generates its own trajectories? Can the authors evaluate candidate rankings on ProQ-generated prefixes using ProQ continuations, rather than only Base-model continuations?
  3. Can the authors provide performance comparisons under matched inference computation? In particular, does ProQ still outperform GRPO, DAPO, and RLPT when these baselines are given comparable test-time compute budgets?
  4. What is the advantage of using reserved-token logits over a standard value head? Can the authors compare these approaches using the same training data, candidate set, and inference procedure?
Reviewer_KQagOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether reasoning improvements from post-training arise primarily from introducing new token-level decisions or from reorganizing preferences among candidates already favored by the pretrained model. Through analyses across 31 model–task–checkpoint settings and post-training-induced successful trajectories, the authors find substantial inheritance of pretrained Top-K candidates, accompanied by changes in their relative preferences.

Motivated by this observation, the paper introduces ProQ, a proposal-preserving learning method that separates candidate generation from value-based selection within a single LLM. ProQ repurposes reserved-token logits to estimate candidate-conditioned values, supervises these values using downstream correctness rewards, and regularizes the standard-token distribution toward the base model.

Experiments across three model backbones report improvements on mathematical reasoning and transfer to coding and knowledge benchmarks, supported by selector diagnostics, objective ablations, and inference-cost analyses.

Critical strengths
  1. The paper extends candidate-locality analysis across multiple post-training stages, model families, and domains. Its examination of successful post-training trajectories after divergence from the base policy is particularly useful, as it goes beyond comparisons restricted to base-generated prefixes. This provides meaningful empirical evidence for the relationship between local candidate availability and global reasoning behavior.

  2. ProQ replaces teacher-preference supervision with reward-grounded candidate evaluation and demonstrates improvements across multiple backbones and out-of-domain benchmarks. The adapted-policy versus selector-enabled comparisons help isolate the contribution of value-based selection, making the experimental results more informative than end-to-end accuracy comparisons alone.

Critical weaknesses
  1. The central implementation of ProQ closely resembles S2T-Local: both construct Top-K candidate sets, append each candidate to the current prefix, perform candidate-conditioned forward passes through the same LLM, convert reserved-token logits into scalar scores using value bins, select the highest-scoring candidate, and apply KL regularization over the standard vocabulary. ProQ's main distinction appears to be reward-grounded supervision with pairwise and Top-1 ranking objectives rather than teacher-derived preferences. Although this distinction is meaningful, the paper does not sufficiently isolate its incremental contribution or directly compare against the closest prior architecture. Consequently, the claimed novelty of the overall proposal–evaluation paradigm is overstated.

  2. The reported DeepSeek-R1-Distill-Qwen-1.5B baseline achieves 65.4% on MATH-500 and 10.1% on GPQA Diamond, compared with 83.9% and 33.8% in the official DeepSeek evaluation. Likewise, the Qwen3-8B baseline achieves 81.3% on MATH-500, compared with 97.4% in the official Qwen3 technical report. Differences in prompts, generation limits, and answer extraction may explain some of these discrepancies, so they are not by themselves evidence of incorrect results. Nevertheless, their magnitude raises concerns about whether the reported improvements are partly driven by an unusually weak evaluation regime, particularly given the observed reductions in output length and generation-length cap rates. This needs clarification before the significance of the reported gains can be assessed confidently.

Questions for authors
  1. Since S2T-Local already uses candidate-conditioned reserved-token value bins, single-model candidate reranking, and standard-vocabulary KL regularization, which components of ProQ constitute the genuinely new algorithmic contribution beyond the source and objective of supervision? Could the authors provide a controlled comparison using the same architecture and candidate data, isolating teacher-preference supervision from the proposed reward-grounded ranking objective?

  2. What explains the large differences between the paper's Base results and the official DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-8B benchmark reports? In particular, how much of the discrepancy arises from generation limits, response parsing, or prompting? Can the authors demonstrate that ProQ's gains remain substantial under an evaluation configuration that reproduces the expected pretrained baseline performance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.