- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies how probability distributions over behavioral branches evolve during reinforcement learning with verifiable rewards (RLVR). The authors represent model behavior using task-specific reasoning entrances and combine checkpoint-level observations with repeated continuations from the same complete training state. The experiments show that branch distributions change beyond scalar sharpening and that continuations from a shared training state can develop different branch allocations. The authors further examine directional memory, the additional predictive value of recent history, residual dependence, and the ability of the current branch state to predict future instability. The primary controlled evidence comes from Qwen2.5-3B on Countdown, with additional trajectory analyses across other models and tasks.
Critical strengths
The controlled continuation experiments provide a careful characterization of behavioral variability during RLVR. Restarting from the same complete training state helps distinguish variation generated after a checkpoint from differences accumulated earlier in training. The rollout-matched measurement controls and robustness analyses strengthen the evidence that the observed branch-distribution differences reflect training-induced variation rather than evaluation sampling uncertainty alone.
Critical weaknesses
My main concern is the significance of the additional insights. Stochastic training producing different branch allocations, and closely competing branches being more susceptible to changes in the dominant branch, are broadly consistent with intuitive expectations. Although the paper carefully reports divergence statistics and instability-prediction metrics, these measurements do not establish a sufficiently predictive quantitative account of how branch probabilities evolve from a given training state. The transition-model comparisons primarily assess the additional value of recent history, while the recursive simulations rely on empirically estimated residual structure and still substantially overestimate the observed divergence. Consequently, the account of the evolution itself remains largely qualitative, and it is unclear how much the findings advance our understanding beyond these intuitive expectations.
The implications of the proposed diagnostics also remain unclear. The paper does not establish when predicted branch instability represents a consequential training problem or how measuring it would improve training or evaluation decisions. An empirical study need not introduce a new algorithm, but its findings should provide either a substantive conceptual advance or demonstrated practical value. Neither is sufficiently established here.
Questions for authors
1.Additional insight beyond intuitive stochastic behavior
Which findings most clearly distinguish the proposed account from a generic explanation in which stochastic updates perturb competing branches and small initial margins make winner changes more likely? Do the existing analyses identify behavior that this simpler account would fail to explain?
2.Consequences of predicted branch instability
What conclusions about training quality or evaluation reliability should follow from identifying a problem as susceptible to future branch instability? Do the existing results establish circumstances in which this diagnostic would change a training or evaluation decision, given that branch divergence does not necessarily track accuracy differences?
3 comments
Official Comment by Reviewer PAKt
I thank the authors for the additional clarifications. Author’s definition of the term “branch” may be a little misleading,an entrance of a branch also includes many solutions some are right and some are wrong,I get the idea that the model would choose a branch which gradually has more possibilities of right answer,and this is a emperical evidence that RL tends to choose branches with higher correctness. author’s suggestion about allocating evaluation budget is also helpful. I will raise my rating from 2 to 3.
We sincerely thank the reviewer for carefully considering our rebuttal comments and additional experiments, as well as for updating the score!
Response to Reviewer PAKt:
We thank the reviewer for recognizing the controlled continuation design and its measurement checks, and for asking what the findings reveal beyond a near-tie account.
W1. Significance of the dynamical findings. On Track A, the step-100-to-175 accuracy gain is +0.283. A symmetric decomposition assigns +0.082 to changed entrance selection and +0.230 to within-entrance success, the latter accounting for 81% [79%, 84%] of the net gain; the unallocated remainder is −0.030. A separate assigned-prefix intervention supports this distinction, yielding a +0.162 [+0.127, +0.194] increase in intention-to-treat success. Together, these results show that RLVR improves success within behavioral entrances as well as reallocating probability among them (Section 5.1).
Q1. What does a generic stochastic near-tie account fail to explain? A near-tie account captures sensitivity in which entrance becomes most probable. Our results identify a second dimension of RLVR dynamics: how success evolves conditional on entering a branch. The accuracy decomposition and assigned-prefix intervention in W1 provide complementary evidence for this within-entrance improvement. This distinction also sharpens the comparison with prior RLVR studies: Yue et al. (2025) emphasize selection among accessible successful behaviors, while Chen et al. (ICLR 2026, Reshaping Reasoning in LLMs) report relatively stable success within extracted reasoning patterns and assume fixed pattern-conditional success. Our solver-defined entrances differ from those patterns, and we directly measure success conditional on these entrances. Consistent with this distinction, low pre-fork margin predicts later allocation dispersion (ρ=0.53–0.57), while its correlations with changes in entrance-conditional success remain near zero (ρ from −0.03 to +0.07, with intervals including zero).
Branch allocation also exhibits predictive and residual structure beyond changes in modal identity. On held-out data, the current-state predictor improves on persistence by +0.004488 nats/count, recent history adds +0.001154, and seed-disjoint CLR residual excess is +0.110 relative to a matched independent-arm null (Tables 2–3). These findings complement the within-entrance results by characterizing how branch probabilities evolve across continuations.
Manuscript changes. We have, accordingly, revised the Contributions and Section 5.1 to foreground within-entrance improvement, added the accuracy-decomposition table, and clarified the predictive and residual analyses in Sections 5.3–5.4.
W2. Practical significance of the branch-instability diagnostic. We evaluate a concrete reproducibility-screening decision: which questions should receive additional continuation checks under a fixed evaluation budget? With a 64-rollout pilot per question, selecting the lowest-margin 20% for 1,024 additional rollouts recovers 72% of future modal disagreements at the additional Track-A root-125 fork, versus 20% for random selection after the same pilot and 26.3% for equal-cost random selection without a pilot. The diagnostic thus directs evaluation effort toward questions most likely to exhibit cross-continuation behavioral disagreement (Section 5.5).
Q2. When does the prediction change an evaluation decision? The decision is to spend expensive follow-up rollouts on the top-risk 20% identified by a cheap pilot. We measure recall of future cross-continuation modal disagreement:
Pilot-based methods select 20% of questions for follow-up; equal-budget random selection without a pilot covers 26.3%. Brackets denote 95% question-cluster CIs. Answer entropy uses final-answer votes. There are 15 and 82 positive cases in the additional-root and pooled evaluations, respectively.
The fixed-budget results demonstrate that branch-state diagnostics can substantially improve the targeting of follow-up evaluations. Low margin and branch entropy capture substantially more future modal disagreements than random selection, while answer entropy and pass@1 provide weaker screening signals. This establishes a practical use of branch-state measurements for allocating evaluation effort under a fixed rollout budget (Section 5.5; Appendix I).