acceptodds
Under review as a conference paper at ICLR 2027

Locked at the Entrance, Open Inside: Where RLVR Narrows the Solution Space

Abstract

Reinforcement learning with verifiable rewards (RLVR) substantially improves single-sample accuracy (pass@1) but causes the policy's solution space to contract, diminishing the returns of test-time scaling. In this work, we investigate where inside a reasoning trajectory this breadth is lost: does the policy fail to access a valid solution family, or does it fail to execute computation once initiated? To disentangle access from execution, we analyze the Countdown task, whose solution space can be exhaustively enumerated into discrete entrance families defined by the first operand and operator, across PPO on Qwen2.5-3B and GRPO on Qwen2.5-3B-Instruct. Across both training setups, solution coverage falls by up to 67%, halving even on problems solved at both endpoints. We show that this contraction is heavily concentrated at the entrance: per-token likelihood shifts are 11x–16x larger prior to the first arithmetic operation than during downstream reasoning. Supplying only an unselected entrance prefix restores completion rates in low-access families by over an order of magnitude (0.018 to 0.212 under PPO), demonstrating that alternative solutions remain executable but are no longer initiated. Guided by this localization, layer interpolation restores entrance diversity and increases solution coverage by 37%, with pass@1 of 0.305 versus 0.278 for the late policy. Finally, we show that early-step entropy collapse recurs across six math benchmarks with 7B and 14B models, but is not an inevitable byproduct of reasoning optimization: an SFT baseline preserves more than double the coverage, and staged SFT–DPO–RLVR pipelines retain early-step entropy. In summary, reasoning breadth is lost at the door, not inside the room.

Primary area:
interpretability and explainable AI
Keywords:
reinforcement learning with verifiable reward, LLM reasoning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vCY6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks where RLVR loses solution diversity, using Countdown so it can tell failing to start a solution apart from failing to finish one. Across PPO and GRPO, coverage drops by up to 67%, and almost all of the probability shift happens at the first step. When given an unselected first step, the trained model still completes the solution, so it can still solve the problem. Layer interpolation recovers 37% of coverage, and the same pattern appears on six math benchmarks at 7B/14B.

Critical strengths

1.The access–execution decomposition provides a useful perspective beyond conventional pass@k and diversity metrics. 2.The recovery experiments suggest practical opportunities to preserve reasoning diversity during training and inference.

Critical weaknesses

1.The entrance-clamping experiments do not fully establish that entrance selection is the primary cause of solution-space contraction, since providing a prefix also changes the model's context. 2.The layer-interpolation results are promising, but their robustness across interpolation configurations and training setups remains unclear. 3.The experiments primarily use Qwen2.5 models, it is unclear whether the observed solution-space contraction persists in models with substantially improved reasoning capabilities, limiting the generalizability and practical relevance of the findings.

Questions for authors

1.Why do the authors choose Qwen2.5 rather than newer reasoning models? Can the main findings be reproduced on a more recent model family? 2.How robust is layer interpolation across different configurations and inference budgets?

Reviewer_y74HOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks where along a reasoning trajectory RLVR loses solution diversity. Using Countdown, whose solution set can be exhaustively enumerated and partitioned into "entrance families" (first operand + operator), it factorizes solving into access (entering a family) and execution (completing it). Across a self-trained PPO run (Qwen2.5-3B) and a public GRPO series (Qwen2.5-3B-Instruct), it shows that pass@1 rises while coverage falls 43-67%, that teacher-forced likelihood shifts are 11-16x larger before the first arithmetic operation than after, and that supplying only a minimal entrance prefix restores completion in abandoned families (0.018 -> 0.212; 0.529 on families never sampled at the late checkpoint) while conditional execution actually improves. Guided by this, late-layer interpolation with an early checkpoint recovers coverage (+37%) without hurting pass@1 on the PPO run. Proxy analyses (first-calculation entropy, same-trace likelihood difference-in-differences) on 7B/14B models over six math benchmarks show the same early-stage pattern, and SFT / staged OLMo-3 pipelines are shown to retain breadth.

Critical strengths
  1. Clean identification of the failure mode. The combination of (a) teacher-forced NLL splitting at the first operator, (b) minimal-prefix interventions with information controls (the first number alone or generic plans produce no recovery, Appendix D.2), and (c) the extinct-family result (Table 3 Panel C: 0.529 completion on families with zero hits in 320 free samples) convincingly separates access from execution. This is the paper's central contribution, and it is replicated under two different RLVR recipes and prompt formats.

  2. Actionable localization. Showing that entrance-targeted interventions (layer interpolation, solver-guided entrance allocation) recover breadth while surface interventions (prompting, temperature, answer-span logit mixing) do not gives a mechanistic explanation for why checkpoint ensembling and weight interpolation work, and provides a concrete target for diversity-preserving RL objectives.

Critical weaknesses
  1. The headline recovery result does not replicate on the second run. Layer interpolation is advertised in the abstract and in Contribution 3 (+37% coverage, pass@1 0.305 vs 0.278), but on the public GRPO run it yields +0.003 coverage and -0.028 pass@1 (Table 23). The [20,28) band and alpha=0.5 were selected from a 56-configuration sweep on the PPO run (Appendix F.6), and the pass@1 gain's confidence interval includes zero ([-0.009, 0.064], Table 22). As written, Contribution 3 overclaims; the only intervention that is robust across both runs is checkpoint sampling, which trades pass@1 heavily.

  2. The training-regime conclusions in Section 7 are confounded. The PPO run uses rollout group size G=1 and no KL penalty, a setting that maximizes on-policy sharpening, so the magnitude of contraction may not be representative of standard RLVR practice. More importantly, the claim that staged SFT-DPO-RLVR "circumvents" the tradeoff compares OLMo-3 against SimpleRL-Qwen, which differ in base model, training data, and recipe; the distilled 1.5B comparison likewise spans models from different groups. These are suggestive observations rather than controlled evidence that the pipeline is what preserves breadth. The one controlled experiment (SFT -> GRPO continuation, Appendix H.3) in fact shows re-collapse.

Questions for authors

Clarification requests:

  1. Why does layer interpolation fail on the GRPO run (Table 23)? Was the layer/alpha sweep repeated on that run, and is there any configuration that recovers breadth without a pass@1 loss? If not, Contribution 3 should be scoped explicitly to the PPO run.
  2. How much of the PPO contraction magnitude depends on G=1 and the absent KL term? A single PPO run with G>1 and a standard KL coefficient would establish whether the 67% figure is representative.
  3. For the Section 7 proxy, does the DiD remain positive when the early/execution boundary is placed at a fixed token offset rather than at the first calculation? Appendix G.2's +/-20-token regression suggests a broad early phase; this matters for whether "entrance" is the right abstraction on open-domain math rather than simply "early tokens."

Suggestion for future work: A controlled comparison on the same base model and data, direct RLVR versus SFT -> DPO -> RLVR, would substantiate the Section 7 pipeline claim.

Reviewer_YRtAOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the problem of RLVR improving pass@1 but degrading pass@k and coverage in the reasoning space. The authors attribute this collapse primarily to reduced coverage at the access, or opening, stage of reasoning rather than during later execution. While access coverage declines significantly during RLVR, execution of those reasoning paths can be recovered as long as the corresponding entrance is provided in the prompt. The authors provide extensive analysis and empirical evidence on the synthetic Countdown task, where they define the first number and operator as the “entrance,” while also making some efforts to extend the study to real-world math benchmarks such as GSM8K.

Critical strengths
  1. The hypothesis that RLVR hurts coverage mostly at the reasoning access stage is interesting, and the authors provide extensive evidence on Countdown to support it.
  2. The evaluation across established models and benchmarks, along with generalization tests, helps demonstrate the broader relevance of the findings.
  3. The paper also provides a valuable evaluation of whether alternative training recipes, such as OLMo-3's DPO-involved curriculum and multi-solution SFT, can help mitigate access collapse.
Critical weaknesses
  1. The presentation is often unclear. The notation is unnecessarily complex, and many parts of the main text omit critical information that must be found in the appendix. For example, Section 3 could benefit from clearer examples of the key concepts; the methodology in lines 346-355 is difficult to follow without consulting the appendix; and many results are reported inline, with clearer, more structured presentations relegated to the appendix.
  2. I find the experimental setup, especially the evaluation sets, unclear and confusingly presented. Section 3 reports a held-out test set of 135 problems, but Table 1 evaluates PPO on a 150-problem test set. Table 3 also evaluates on 150 problems, yet its reported accuracy seems inconsistent with Table 1 (see Q1). Table 4 then evaluates on 100 problems, while the experiment in lines 346-355 evaluates on 128 problems.
  3. The paper's rule-based definition and extraction of “access,” such as the first operand and operator in Countdown or the first calculation in math benchmarks, may not adequately capture the intended concept. Access may be better characterized as the preliminary selection of reasoning paths or modes. It would be valuable to explore alternative definitions of access, potentially including LLM-based clustering of reasoning modes, and examine whether the conclusions remain consistent across these definitions.
Questions for authors
  1. Why is the step-275 any-valid accuracy of 0.060 in Table 3 so low compared with the pass@1 of 0.285 in Table 1? If I understand correctly, any-valid success without an entrance hint should be equivalent to pass@1 under the same evaluation protocol.
  2. I wonder whether the finding that access collapses more than execution could motivate methods that better balance coverage and pass@1 performance, both during training and at inference time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.