acceptodds
Under review as a conference paper at ICLR 2027

Stochastic yet Structured: Reasoning Branch Selection Dynamics under RLVR

Abstract

Learning dynamics of LLM post-training provides insight into how model behaviors emerge and evolve throughout optimization. In this paper, we study the learning dynamics of reinforcement learning with verifiable rewards (RLVR) by tracking the evolution of probability distributions over behavioral branches along training trajectories and controlled continuations from fixed training states. By projecting RLVR training trajectories into an interpretable behavioral state space, we characterize RLVR’s reallocation of probability mass among behavioral branches across models and tasks. Controlled continuations from the same complete training state show that future branch allocations are not fully reproducible, even though the dominant branch often remains stable. Across continued training, these differences retain only short-range directional memory, while the current branch state provides a near-first-order predictive summary of short-horizon dynamics. The remaining variation is nevertheless structured: residual dependence shapes long-horizon divergence, while the current branch state predicts future instability across independent lineages and over substantially longer training horizons. Together, these results characterize RLVR branch selection as a stochastic but state-dependent dynamical process.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, language model reasoning, learning dynamics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PAKtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how probability distributions over behavioral branches evolve during reinforcement learning with verifiable rewards (RLVR). The authors represent model behavior using task-specific reasoning entrances and combine checkpoint-level observations with repeated continuations from the same complete training state. The experiments show that branch distributions change beyond scalar sharpening and that continuations from a shared training state can develop different branch allocations. The authors further examine directional memory, the additional predictive value of recent history, residual dependence, and the ability of the current branch state to predict future instability. The primary controlled evidence comes from Qwen2.5-3B on Countdown, with additional trajectory analyses across other models and tasks.

Critical strengths

The controlled continuation experiments provide a careful characterization of behavioral variability during RLVR. Restarting from the same complete training state helps distinguish variation generated after a checkpoint from differences accumulated earlier in training. The rollout-matched measurement controls and robustness analyses strengthen the evidence that the observed branch-distribution differences reflect training-induced variation rather than evaluation sampling uncertainty alone.

Critical weaknesses

My main concern is the significance of the additional insights. Stochastic training producing different branch allocations, and closely competing branches being more susceptible to changes in the dominant branch, are broadly consistent with intuitive expectations. Although the paper carefully reports divergence statistics and instability-prediction metrics, these measurements do not establish a sufficiently predictive quantitative account of how branch probabilities evolve from a given training state. The transition-model comparisons primarily assess the additional value of recent history, while the recursive simulations rely on empirically estimated residual structure and still substantially overestimate the observed divergence. Consequently, the account of the evolution itself remains largely qualitative, and it is unclear how much the findings advance our understanding beyond these intuitive expectations.

The implications of the proposed diagnostics also remain unclear. The paper does not establish when predicted branch instability represents a consequential training problem or how measuring it would improve training or evaluation decisions. An empirical study need not introduce a new algorithm, but its findings should provide either a substantive conceptual advance or demonstrated practical value. Neither is sufficiently established here.

Questions for authors

1.Additional insight beyond intuitive stochastic behavior

Which findings most clearly distinguish the proposed account from a generic explanation in which stochastic updates perturb competing branches and small initial margins make winner changes more likely? Do the existing analyses identify behavior that this simpler account would fail to explain?

2.Consequences of predicted branch instability

What conclusions about training quality or evaluation reliability should follow from identifying a problem as susceptible to future branch instability? Do the existing results establish circumstances in which this diagnostic would change a training or evaluation decision, given that branch divergence does not necessarily track accuracy differences?

3 comments

Response to Reviewer PAKt:

We thank the reviewer for recognizing the controlled continuation design and its measurement checks, and for asking what the findings reveal beyond a near-tie account.

W1. Significance of the dynamical findings. On Track A, the step-100-to-175 accuracy gain is +0.283. A symmetric decomposition assigns +0.082 to changed entrance selection and +0.230 to within-entrance success, the latter accounting for 81% [79%, 84%] of the net gain; the unallocated remainder is −0.030. A separate assigned-prefix intervention supports this distinction, yielding a +0.162 [+0.127, +0.194] increase in intention-to-treat success. Together, these results show that RLVR improves success within behavioral entrances as well as reallocating probability among them (Section 5.1).

Q1. What does a generic stochastic near-tie account fail to explain? A near-tie account captures sensitivity in which entrance becomes most probable. Our results identify a second dimension of RLVR dynamics: how success evolves conditional on entering a branch. The accuracy decomposition and assigned-prefix intervention in W1 provide complementary evidence for this within-entrance improvement. This distinction also sharpens the comparison with prior RLVR studies: Yue et al. (2025) emphasize selection among accessible successful behaviors, while Chen et al. (ICLR 2026, Reshaping Reasoning in LLMs) report relatively stable success within extracted reasoning patterns and assume fixed pattern-conditional success. Our solver-defined entrances differ from those patterns, and we directly measure success conditional on these entrances. Consistent with this distinction, low pre-fork margin predicts later allocation dispersion (ρ=0.53–0.57), while its correlations with changes in entrance-conditional success remain near zero (ρ from −0.03 to +0.07, with intervals including zero).

Branch allocation also exhibits predictive and residual structure beyond changes in modal identity. On held-out data, the current-state predictor improves on persistence by +0.004488 nats/count, recent history adds +0.001154, and seed-disjoint CLR residual excess is +0.110 relative to a matched independent-arm null (Tables 2–3). These findings complement the within-entrance results by characterizing how branch probabilities evolve across continuations.

Manuscript changes. We have, accordingly, revised the Contributions and Section 5.1 to foreground within-entrance improvement, added the accuracy-decomposition table, and clarified the predictive and residual analyses in Sections 5.3–5.4.

W2. Practical significance of the branch-instability diagnostic. We evaluate a concrete reproducibility-screening decision: which questions should receive additional continuation checks under a fixed evaluation budget? With a 64-rollout pilot per question, selecting the lowest-margin 20% for 1,024 additional rollouts recovers 72% of future modal disagreements at the additional Track-A root-125 fork, versus 20% for random selection after the same pilot and 26.3% for equal-cost random selection without a pilot. The diagnostic thus directs evaluation effort toward questions most likely to exhibit cross-continuation behavioral disagreement (Section 5.5).

Q2. When does the prediction change an evaluation decision? The decision is to spend expensive follow-up rollouts on the top-risk 20% identified by a cheap pilot. We measure recall of future cross-continuation modal disagreement:

Selection strategyAdditional root 125Track A pooled
Low branch margin0.72 [0.47, 0.95]0.53 [0.41, 0.66]
High branch entropy0.53 [0.27, 0.82]0.53 [0.41, 0.66]
High answer entropy0.13 [0.00, 0.33]0.26 [0.13, 0.40]
Low pass@10.20 [0.00, 0.42]0.23 [0.09, 0.38]
Random (with pilot)0.20 [0.07, 0.40]0.20 [0.12, 0.28]
Random (no pilot, equal total cost)0.2630.263

Pilot-based methods select 20% of questions for follow-up; equal-budget random selection without a pilot covers 26.3%. Brackets denote 95% question-cluster CIs. Answer entropy uses final-answer votes. There are 15 and 82 positive cases in the additional-root and pooled evaluations, respectively.

The fixed-budget results demonstrate that branch-state diagnostics can substantially improve the targeting of follow-up evaluations. Low margin and branch entropy capture substantially more future modal disagreements than random selection, while answer entropy and pass@1 provide weaker screening signals. This establishes a practical use of branch-state measurements for allocating evaluation effort under a fixed rollout budget (Section 5.5; Appendix I).

Reviewer_PAKtOpenReview ↗

Official Comment by Reviewer PAKt

I thank the authors for the additional clarifications. Author’s definition of the term “branch” may be a little misleading,an entrance of a branch also includes many solutions some are right and some are wrong,I get the idea that the model would choose a branch which gradually has more possibilities of right answer,and this is a emperical evidence that RL tends to choose branches with higher correctness. author’s suggestion about allocating evaluation budget is also helpful. I will raise my rating from 2 to 3.

We sincerely thank the reviewer for carefully considering our rebuttal comments and additional experiments, as well as for updating the score!

Reviewer_AM58OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the evolution of reasoning-branch distributions during RLVR. It represents each problem's behavioral state as a probability distribution over identifiable entrances and analyzes both ordinary training trajectories and multiple continuations restored from the same complete training state. RLVR reallocates branch mass beyond scalar sharpening; future distributions can diverge despite fixed training history and state, while directional memory remains short-ranged. The current branch state summarizes the tested short-horizon predictive information and predicts subsequent instability. Preserving cross-continuation shared residual structure also explains long-horizon divergence better than treating the entire residual as independent noise. The main state-conditional evidence comes from Qwen2.5-3B Countdown continuations, with other scales and code tasks providing trajectory-level breadth.

Critical strengths
  1. The behavioral-state definition is clear, interpretable, and operationally verifiable. Countdown branches are defined by the first arithmetic entrance, with an exact solver retaining entrances that still admit a correct continuation, so the analysis is not restricted to ultimately successful samples. Treating the distribution as a behavioral projection of the complete training state also clearly separates parameter state from observable behavior and provides a suitable framework for studying probability reallocation rather than only endpoint accuracy. (Sections 2 and 4, lines 77–119 and 197–207)

  2. The findings go beyond the observation that stochastic training is not fully reproducible and are supported by complementary analyses. The paper distinguishes stable dominant branches from diverging full distributions, tests departures from scalar sharpening, and characterizes structure through directional memory, nested-history prediction, and cross-continuation residual analysis. The margin-based instability signal is also replicated in the independently trained Track-A lineage and extends to longer horizons. This paper explicitly limits the temporal resolution of the near-first-order result and states that the structured simulator is not yet calibrated, keeping the conclusions appropriately bounded. (Sections 5.1–5.5, Table 1, Figures 2–6)

Critical weaknesses

W1. Robustness to branch definitions, support selection, and low-probability treatment remains unclear All analyses rely on task-specific entrance definitions, but the same initial arithmetic operation or AST entrance can encompass different subsequent solution procedures, so the state does not represent a complete reasoning strategy. The main text does not develop how infeasible entrances, unparseable outputs, and unobserved branches enter the probability estimates, or whether the associated probability mass is excluded before renormalization. Zero-count smoothing and the treatment of very small probabilities in CLR may affect increment directions and residual energy, while support and normalization choices may also alter the margin metric. Rollout-matched controls help estimate sampling uncertainty under a fixed representation but do not substitute for sensitivity checks on entrance granularity, support, and smoothing. The main dynamical findings therefore need to be shown robust to these encoding choices. (Section 2, lines 108–119; Section 4, lines 197–207; Sections 5.3–5.5)

W2. The interpretation of near-first-order behavior and shared residuals remains dependent on the transition model Table 1 shows little gain from recent increments at the five-update observation interval using the conditional multinomial predictor, but this does not exclude nonlinear state–history interactions or additional information at other temporal resolutions. The shared residual in Section 5.4 may also contain problem-specific or training-stage-specific drift not captured by the state model. Worsening simulations after disrupting sharing shows that this structure is useful to the current simulator but does not by itself identify its source. Although this paper explicitly states that S2 is not calibrated, it remains important to establish whether the shared component and its recursive-prediction benefit persist under a more expressive state-transition model, distinguishing reducible model error from residual dependence that needs explicit modeling. (Sections 5.3–5.4, Table 1, Figure 5, lines 358–411)

W3. Independent units in predictive validation and uncertainty estimation need clearer specification The main text reports held-out predictions, question-bootstrap intervals, and Track-A replication, but does not develop whether predictors are split by problem, fork root, or continuation path. Adjacent states for the same problem and descendants of a shared root are correlated; record-level splitting could allow shared information to influence predictive performance. Question bootstrapping primarily captures problem-sampling uncertainty conditional on observed training paths, rather than variation across independent roots and training lineages. In particular, the near-first-order and residual analyses are concentrated in dense 3B Countdown continuations, while 7B and Code trajectories do not provide equivalent state-conditional replication. The splitting and resampling units should be specified, together with the dependence of key results on the available roots and continuation paths. (Sections 4 and 5.2, lines 209–215 and 286–293; Table 1; Section 5.5, Figure 6, lines 424–455 and 465–469)

Questions for authors
  1. Regarding W1: How is probability mass from infeasible entrances and parsing failures included or excluded, and what smoothing rules handle unobserved branches and zero counts in CLR? Do short-range memory, near-first-order prediction, and margin-based instability prediction persist under alternative entrance granularities, support choices, normalization procedures, and smoothing strengths?

  2. Regarding W2: Have transition predictors allowing state–history interactions or alternative observation intervals been compared? Do shared residual structure and its recursive-simulation benefit persist under a more expressive state model, and how are they distinguished from omitted problem-specific or training-stage drift?

  3. Regarding W3: Are train/test splitting and bootstrap resampling performed over problems, records, fork roots, or continuation paths? How are dependencies between adjacent states of the same problem and continuation records sharing a root handled? Could results by the available roots and paths, or hierarchical uncertainty estimates, establish the stability of the predictive findings?

4 comments

Response to Reviewer AM58

We thank the reviewer for recognizing the interpretability of our branch-state representation and the independently validated instability predictions. We address the three main concerns—representation robustness, predictive model dependence, and statistical reliability—below, followed by the corresponding experimental details.

W1. Robustness to branch definitions and low-probability treatment.

The main short-memory, history-prediction, and margin-instability patterns are qualitatively retained across 16 encoding–smoothing settings (four branch encodings × Dirichlet α ∈ {0.1, 0.5, 1, 2}). Importantly, these include alternative encodings that retain the full probability mass, as well as the primary solver-feasible representation. The finer residual and long-lag diagnostics are more sensitive to the coarse operator-only encoding; the detailed comparisons appear under Q1.

Low-probability branches make a substantial contribution to the observed dynamics. Branches with mean probability below 0.05 account for 2.6% of probability mass but 57% of CLR process energy and 61% [49%, 72%] of residual excess. Their estimated enrichment relative to energy share is +0.035 [−0.073, +0.135]. This highlights the dynamical importance of low-probability entrances within our behavioral-state representation.

W2. Model dependence of history prediction and residual structure.

The history-prediction results remain similar under a more flexible nonlinear predictor. On Track D at Δ=5, adding recent history to a nonlinear state predictor improves held-out NLL by +0.001212 [+0.000490, +0.001874] nats/count, close to the linear gain of +0.001154. A post-hoc capacity-matched comparison gives a similar improvement (+0.001295).

The residual-dependence finding also persists under a richer drift model. With matched seed-disjoint evaluation, CLR residual excess over the independent-arm null is +0.110 [+0.052, +0.175] for the original state-only drift and +0.098 [+0.035, +0.168] for richer drift. The paired change is −0.012 [−0.029, +0.005], showing no resolved reduction under the richer model. The original-drift S2 analysis separately evaluates how retaining measured cross-continuation residual structure improves retrospective reconstruction. Further model details appear under Q2 (Sections 5.3–5.4).

W3. Independent validation units and uncertainty.

Our predictive evaluation is organized at the question level to preserve dependencies among observations from the same problem. Held-out fits use frozen question-grouped folds, and confidence intervals use question-cluster bootstrap resampling. This estimates uncertainty across questions conditional on the observed training paths.

We also examine stability across the available roots and continuation arms, as detailed under Q3. Track A contains four roots from one training lineage, while Track D provides a second 3B lineage. The 7B and Code experiments provide trajectory-level evidence rather than additional fixed-state replications.

Q1. Feasible support, zero counts, and representation sensitivity.

For Countdown, we define a fixed set of solver-feasible first-operand/operator entrances for each question. Infeasible and unparseable entrances are tracked separately and excluded before renormalization, while zero counts within the feasible set receive Dirichlet-0.5 smoothing.

Across the tested encodings and smoothing strengths, the main short-memory, history-prediction, and margin-instability patterns persist. Coarser operator-only representations weaken some residual and long-lag diagnostics, consistent with the granularity sensitivity discussed in W1.

The margin-based signal is particularly stable across support choices: negative top-two margin correlates with subsequent allocation dispersion at Spearman ρ=0.53–0.57. Separately, cross-stream directional memory declines from 0.83 [0.75, 0.95] at lag 5 to 0.12 [−0.01, 0.28] at lag 20 (Appendices G and I).

Q2. Interactions, observation intervals, richer drift, and recursive simulation.

To examine whether the history gain depends on linearity, we compared nonlinear state-only and state-plus-history predictors. NL_state includes current-state and feasible-support features, together with branch margin and entropy; NL_hist additionally includes recent velocity. The MLP can capture state–history interactions through its hidden layer, although explicit product terms were not separately ablated.

Compared with the linear history predictor M2, NL_hist yields a small additional NLL gain: +0.000318 [−0.000359, +0.000849] under the original per-model tuning and +0.000381 [+0.000145, +0.000622] under post-hoc capacity matching. The comparison evaluates nonlinear modeling together with richer state features, including margin and entropy, rather than isolating state–history interactions.

We also examined the temporal extent of the history gain. At Δ=5, adding a second linear history lag changes NLL by −0.000037 [−0.000055, −0.000019], while the nonlinear second-lag result varies with capacity and regularization. At Δ=10, the linear history gain is +0.000605 [−0.000034, +0.001236], with an interval including zero. Checkpoints are spaced five updates apart, limiting the observation intervals available for comparison.

We further tested residual dependence using richer drift with nonlinear state, feasible-support, and pre-fork-root features where identifiable. As quantified in W2, positive residual excess persists relative to the matched independent-arm null, with no resolved reduction from the original state-only drift. A separate training-stage effect is not identified in the matched Track-D panel. In a separate original-drift analysis, preserving empirically measured shared residual structure accounts for 66–72% of the reduction in retrospective reconstruction error from S1 to S2 (Appendices G–H).

Q3. Splitting, correlated records, and stability across paths.

We split the data by question, keeping all checkpoints and continuation arms of each question in the same fold. Bootstrap resampling likewise operates over whole questions, preserving correlations among their observations. The resulting intervals characterize question-level uncertainty conditional on the observed training paths.

We further checked whether the instability-prediction results depend strongly on individual roots or arms. Across the four Track-A roots, margin-prediction AUROCs range from 0.768 to 0.924. Leave-one-arm-out AUROCs differ by at most 0.08 from their corresponding four-arm values. These comparisons support stability across the observed roots and arms.

With two 3B training lineages (Tracks A and D), we report question-cluster uncertainty conditional on the observed training paths and separately evaluate stability across Track-A roots and continuation arms (Appendices D and I; Table 29).

I thank the authors for their thorough responses. I would maintain my score.

We sincerely thank the reviewer for taking the time to carefully consider our rebuttal responses and additional analyses!

Reviewer_358oOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper represents an RLVR checkpoint by a distribution over task-specific behavioral entrances and studies how this distribution changes during training. Countdown provides solver-defined feasible arithmetic entrances, while HumanEval provides AST-defined implementation entrances. The main controlled evidence comes from Qwen2.5-3B Countdown continuations restored from shared training states. The paper finds branch reweighting beyond scalar sharpening, detectable divergence between continuations despite frequent modal agreement, short-range directional memory, and limited predictive benefit from adding recent history to a current-state model. It also studies shared and continuation-specific residuals and finds that small pre-fork branch margins predict future instability, including in a second training lineage.

Critical strengths
  1. The full-state continuation design is a substantial methodological strength. Restoring optimizer, scheduler, random-number and data-order state distinguishes newly generated variation from differences accumulated before the fork. Blocked permutation tests preserve rollout-slot pairing, and matched-budget measurement controls make the reported 68–91% rate of detectably different branch distributions more convincing than an ordinary multi-seed comparison.

  2. The study connects descriptive dynamics to a useful, independently checked diagnostic. Pre-fork branch margin predicts endpoint modal disagreement at four roots in Track A, with AUROCs 0.768–0.924 over horizons of 25–100 updates; split-half controls and question-grouped validation address important measurement and dependence issues. The authors also explicitly limit the interpretation of the 7B/Code checks and acknowledge that the residual simulator is not a calibrated forecast.

Critical weaknesses
  1. The near-first-order result is narrower than the broader dynamical narrative may suggest. It comes from one dense 3B Countdown fork, a five-update resolution, and a particular conditional multinomial model with limited history features. The 0.19% improvement is normalized by total next-state NLL, which includes irreducible outcome entropy; it does not measure the fraction of predictable transition structure explained. The paper appropriately avoids claiming an exact Markov state, but a persistence/excess-log-loss baseline and a more flexible history comparison would make the practical meaning of “near-first-order” substantially clearer.

  2. The shared-residual interpretation needs a sampling-dependence audit. Appendix D.2 explicitly uses common per-slot evaluation seeds across arms, whereas D.6 leaves the shared component uncorrected on the premise that target-side measurement noise is independent across continuations. If the dense Track-D evaluation also uses coupled seeds, that premise needs justification or a covariance-aware correction. Uncertainty in the estimated current state and common model misspecification can also enter the residual. Moreover, S2 uses empirical question/transition residuals and predicts endpoint JS 0.06194 versus 0.02574 observed. Thus the residual experiment supports a descriptive comparison, but not yet a clean identification of shared process noise or an out-of-sample dynamical forecast.

Questions for authors
  1. Does Track D share per-slot evaluation seeds across continuations? If so, how is this reconciled with the independence assumption in D.6? Please report the shared/specific residual decomposition with paired resampling or independently seeded evaluations, and account for noise in both current and next states. Persistence of the shared component would strengthen this part of the paper.

  2. How large is the recent-history gain relative to a persistence baseline or estimated excess NLL above the multinomial entropy floor? Does the conclusion survive a more flexible history predictor and evaluation on a held-out root/lineage? For the recursive simulation, please distinguish which empirical residuals are available before prediction from those measured at the same future question/transition being reconstructed.

1 comment

Response to Reviewer 358o

We thank the reviewer for highlighting the full-state continuation design and independent instability diagnostic. We have added predictive-loss baselines, evaluated history-model sensitivity, and audited the residual measurement protocol.

W1. Interpretation of short-horizon prediction. The original 0.19% figure divides the history improvement by total next-state NLL, including outcome entropy. We now report persistence and absolute held-out loss: on the original Track-D five-update panel, NLL is 0.61283 for persistence, 0.60834 for the state-only predictor M1, and 0.60718 with recent history (M2). The gain from persistence to M1 is +0.004488 [+0.003676, +0.005316] nats/count; history adds +0.001154 [+0.000758, +0.001585]. The latter is approximately 8%–11% of the estimated excess NLL under the audited floor choices, but that ratio depends on an estimated floor derived from the scored outcome pool. These results quantify the predictive contribution of the current branch state and recent history in absolute held-out NLL, establishing the current state as an informative short-horizon predictive summary (Section 5.3).

W2. Interpretation of shared residuals and recursive simulation. Preserving empirically measured cross-continuation residual sharing accounts for 66%–72% of the reduction in retrospective reconstruction error from S1 to S2 (Section 5.4). This shows that cross-continuation structure improves the reconstruction relative to independently sampled residuals, although S2 still over-disperses at the endpoint (JS 0.062 reconstructed versus 0.026 observed). S2 uses future question–transition residuals measured from the trajectories being reconstructed; the result is a retrospective reconstruction rather than an out-of-sample forecast. We separately audit the original residual-energy partition and test cross-arm dependence with seed-disjoint measurements under Q1.

Q1. Evaluation-seed dependence and noise at both transition endpoints. Yes: Track D reused per-slot seeds across continuations and nearby checkpoints. Our new matched evaluation uses seed-disjoint measurements of both the current and next states, and scores the same cross-arm CLR residual statistic against a matched independent-arm null. The residual excess is +0.110 [+0.052, +0.175] for the original state-only drift and +0.098 [+0.035, +0.168] for a richer drift. The paired difference is −0.012 [−0.029, +0.005], with no resolved removal under the richer drift. These values establish dependence relative to the tested null and drift specifications, rather than uniquely identifying its stochastic source. The original shared-energy partition is now labeled descriptive and uncorrected (Section 5.4; Appendix H).

Q2. Flexible history models, other roots, and information available to S2. The persistence and excess-NLL comparisons are reported under W1. The history gain remains similar with a more flexible predictor: a one-hidden-layer tanh MLP yields +0.001212 [+0.000490, +0.001874] nats/count on Track D, compared with +0.001154 for the linear model. The post-hoc capacity-matched comparison yields +0.001295 [+0.000894, +0.001728].

We also checked the effect of measurement precision. At a matched 512-rollout budget, the one-lag gain is +0.000400 under temporally seed-disjoint measurement, compared with +0.000868 using shared seeds. A second independent measurement of the current state provides a larger gain than the history measurement (history minus pooled-current gain −0.001219 [−0.001680, −0.000783]).

The Track-A comparisons show how the history gain changes with training stage and observation interval. In root-local fits, it is positive at root 75 at both tested intervals and at root 125 at ten updates, while the five-update gain at root 125 is unresolved (Table 22). These results characterize prediction at the observed roots rather than transfer to a held-out root. Finally, S2 uses residuals measured at the same future question–transition cells it reconstructs, so its reported benefit concerns retrospective reconstruction (Section 5.3; Appendices G–H).

Comments on the paper

1 comment

General Comments

We sincerely thank all three reviewers for their constructive feedback and recognition of the strengths of our controlled continuation design. In response, we conducted additional experiments and analyses to address concerns regarding scientific significance, practical implications, predictive and residual interpretation, and robustness. The table below summarizes the key concerns and our responses.

Reviewer ConcernReviewersNew EvidenceKey Finding / Scope
Scientific SignificancePAKtAccuracy decomposition; assigned-prefix intervention (Sec. 5.1)Under a symmetric (observational) decomposition, within-entrance improvement contributes 81% [79%, 84%] of the net accuracy gain. When the same entrance is prescribed at both checkpoints, intention-to-treat success still improves by +0.162 [+0.127, +0.194]. A near-tie winner-switching account alone does not capture this.
Practical ImplicationsPAKtFixed-budget reproducibility screening (Sec. 5.5; App. I)Following up on the lowest-margin 20% of questions captures 53% [41%, 66%] of future modal disagreements across Track A (82 positive cases), and 72% [47%, 95%] at the additional root (15 positive cases), versus 20% for random selection with the same pilot and 26.3% for equal-cost random selection without a pilot.
Predictive Interpretation358o, AM58Persistence baseline; nonlinear history predictors; measurement and root-local analyses (Sec. 5.3; App. G)Relative to persistence, state-only prediction reduces held-out NLL by 0.0045 nats/count, approximately 4× the additional reduction from recent history (0.0012). History gains vary across tested roots and measurement protocols.
Residual Dependence358o, AM58Seed-disjoint evaluation; matched null; richer drift (Sec. 5.4; App. H)Cross-arm residual excess remains positive: +0.110 [+0.052, +0.175], and +0.098 [+0.035, +0.168] under richer drift. The shared-residual simulation is a retrospective reconstruction, not a prospective forecast.
Representation RobustnessAM584 branch encodings × 4 smoothing strengths (App. I.4; Table 30)Short-memory, history-prediction, and margin-instability patterns are qualitatively retained across the 16 tested settings; finer residual and long-lag diagnostics are sensitive to the coarse operator-only encoding.
Validation Units & UncertaintyAM58Question-grouped folds; cluster bootstrap; root/arm checks (Apps. D, I; Table 29)Margin-based instability prediction is consistent across four roots within the Track-A lineage (AUROC 0.768–0.924), with leave-one-arm-out differences at most 0.08. Question-cluster intervals are conditional on the observed training paths.

Overall, these additional analyses strengthen the evidence that RLVR branch dynamics are stochastic but exhibit measurable, state-dependent structure. They further clarify how accuracy improvements arise, establish the scope and robustness of our dynamical findings, and demonstrate a concrete application to reproducibility-oriented evaluation. We have incorporated these results and clarifications into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.