- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether post-training mainly changes the order of the base model’s existing token candidates. It also proposes ProQ, which scores the model’s own next-token candidates and picks one. Experiments cover three backbone models and tasks in mathematics, coding, and knowledge reasoning.
Critical strengths
- The analysis covers several model families, model sizes, tasks, and training stages.
- The proposed method follows naturally from the analysis.
- The results are promising. ProQ improves math performance and also helps on code and knowledge tasks.
Critical weaknesses
-
The paper asks whether post-training uses new, low-ranked candidates or changes the order of existing candidates. However, [1] already studies this question in Sections 4.1 and 4.2. It examines candidate overlap, rank changes, and tokens with low base-model probabilities. It also compares the models on RL-generated text and tests the effect of replacing selected tokens. Similarly, [2] separates changes within the base model’s top candidates from changes outside them. I am not sure whether the analysis in this paper provides new findings beyond these studies. If it does, I would appreciate it if the authors could clarify and discuss these contributions during the rebuttal.
-
S2T-Local [3] already uses the same model to score candidates through reserved tokens. ProQ mainly changes the training signal from teacher scores to the final correctness of candidate continuations. The current tests are hard to tell whether the gains mainly come from.
Ref:
[1] Sparse but Critical: A Token-Level Analysis of Distributional Shifts in RLVR Fine-Tuning of LLMs. ICLR 2026.
[2] Rethinking RL for LLM Reasoning: It’s Sparse Policy Selection, Not Capability Learning. arXiv:2605.06241, 2026.
[3] Select to Think: Unlocking SLM Potential with Local Sufficiency. ICML 2026.
Questions for authors
- What new finding comes from the full SFT/DPO/RL stage analysis, beyond the findings in [1] and [2]?
- With the same scoring model, how does outcome supervision compare with the teacher supervision used in [3]?
- How much accuracy is gained or lost when changes inside or outside the base model’s Top-K are kept or removed?