acceptodds
Under review as a conference paper at ICLR 2027

Fork Where the Model Changes Its Mind: Belief-Shift Branching for Tree-Structured Reinforcement Learning

Abstract

Tree-structured rollouts give critic-free reinforcement learning with verifiable rewards (RLVR) step-level credit: fork a chain at an intermediate point, and sibling outcome differences estimate step value. Each fork adds sampling cost, so realistic budgets typically allow only a few forks per chain. A fork placed where the outcome is already largely settled yields siblings that mostly agree and provide almost no credit signal; hence, for a given tree size, where forks are placed largely determines how much step-level RL can gain. Most existing mainstream methods place forks by structure, such as fixed lengths, midpoints, and delimiters, or by next-token entropy. We formalize fork placement as locating the pivots of the chain's value curve, where the expected outcome turns. We propose belief-shift branching: read the model's answer belief at candidate boundaries and fork just before the step where consecutive beliefs diverge most. Three instantiations, none needing step-level supervision, span access levels: a black-box probe, a logit-lens depth profile, and a learned activation direction, which is fit offline and therefore used only in the validation before RL training. The signal only places forks, and the probe costs about of step compute on mathematics and under on code when it runs inside the rollout engine. In that validation, against Monte-Carlo value curves, a belief-shift signal ranks first in each of the eight modelbenchmark panels, ahead of entropy, structural, and LLM-judge baselines. In RL across three model families and two domains, belief-shift forking leads every mathematics aggregate, on OLMo-3-7B by aggregate and on AIME 2026 over the strongest baseline, and sweeps every OLMo code column, by on LiveCodeBench-medium.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement learning with verifiable rewards, LLM post-training, Tree-structured reinforcement learning, Branching and exploration, Answer-belief probing, Mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_jT5JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper chooses fork locations in critic-free tree-structured RLVR by measuring changes in a model's belief about its final answer. PROBE-JS compares answer-elicited token distributions at candidate boundaries. LENS-SHIFT tracks layerwise probability profiles for the chain's committed answer. The method forks just before a large change, allowing sibling continuations to try a different step. An offline learned belief-shift direction supplies additional validation and a frozen-direction RL experiment. The evaluation covers value-curve reconstruction before RL and training on mathematics and code across three model families. Belief-based selectors improve the mathematics aggregate on all three and give substantial OLMo code gains. Qwen gains are small, and midpoint remains best on Nemotron code.

Critical strengths
  1. Forking before a change in answer belief gives the additional continuations a specific role: they can revisit the step that changed the prediction. PROBE-JS and LENS-SHIFT provide two implementations without a trained critic, while the shared tree and optimizer make the selector's contribution reasonably direct to test. This is a focused methodological advance over structural or token-uncertainty selectors. The paper correctly limits its JS bound to the elicited distribution; the bound does not guarantee a change in eventual success probability.
  2. The mathematics aggregate improves on all three model families, and the strongest gains are large enough to be practically interesting. On OLMo, PROBE-JS raises mathematics from the strongest baseline's 20.7 to 23.3 and code from 31.0 to 34.9 (Table 3). A second OLMo seed preserves the aggregate advantage of both belief methods over both baselines (Appendix E.1). A chain control with a token budget and checkpoint-selection checks add support (Appendices E.2 and A). The small Qwen gains and the Nemotron code exception usefully limit the claim: benefit depends on the model and domain.
Critical weaknesses
  1. The practical efficiency claim remains uncertain because the reported probe costs are modeled FLOP shares that assume prefix reuse. LENS-SHIFT actually re-forwards full prefixes on a colocated model, and Nemotron PROBE-JS uses a separate Hugging Face path (Appendix B). Fork location also changes continuation length, so equal tree sizes or optimizer-step caps need not mean equal total compute. Appendix A restricts Base-model belief reads to the first half of each chain, but does not make equally clear whether every baseline uses that range. These details matter most where gains are small, as on Qwen. Measured accuracy at a common total compute budget is needed to establish the selector's practical advantage.
  2. The pre-RL reconstruction study provides weaker evidence for the mechanism than the headline ranking suggests. Different belief variants win different panels, and BSV is often the strongest offline instrument even though it is not a main RL selector. PROBE-JS has average AIME reconstruction error 4.071, compared with 4.069 for the LLM judge and 4.100 for entropy (Table 7). The curves use only 16 continuations per boundary, without uncertainty estimates for these small differences. Selecting ten of 128 boundaries also differs from selecting one of 16 during RL. The downstream gains support the method's usefulness, but the reconstruction study does not yet establish a reliable advantage for the main selectors under the training protocol.
Questions for authors
  1. What are the actual generated/trained tokens and measured end-to-end compute, including colocated scorers, for each method? Do all selectors use the same candidate range on Base checkpoints? A comparison at common measured compute would show whether the proposed allocation signal improves the accuracy-cost tradeoff, especially for the small Qwen gains.
  2. How stable are the reconstruction rankings under resampling of problems and Monte Carlo completions? Do PROBE-JS and LENS-SHIFT individually retain an advantage at the candidate count and fork budget used in RL? This would separate evidence for the deployed selectors from the stronger results of other offline belief variants
3 comments

We thank the reviewer for their assessment of our work! Below are our responses to the comments:


For Weakness 1 & Question 1:

The practical efficiency claim remains uncertain because the reported probe costs are modeled FLOP shares that assume prefix reuse.

What are the actual generated/trained tokens and measured end-to-end compute, including colocated scorers, for each method? Do all selectors use the same candidate range on Base checkpoints?

1. How each scorer runs, and what it costs

We have added key-value caching, which leaves the scores unchanged, and measured the cost of both scorers. PROBE-JS needs only next-token log-probabilities, which the serving engine returns, so on OLMo and Qwen it runs inside the vLLM engine on the rollout's prefix cache. LENS-SHIFT needs intermediate-layer states, which the engine's standard interface does not return; on OLMo and Qwen it runs on a co-located Hugging Face copy of the policy, now with key-value caching On Nemotron, both scorers run on the Hugging Face model without caching: in-engine probing destabilized that hybrid-Mamba model's rollouts, and caching its recurrent state would require changes to the model code.

vLLM has had a hidden-state extraction path since v0.18.0 (https://github.com/vllm-project/vllm/pull/33736), but it was built for training speculative-decoding draft models: its current connector saves the hidden states of the whole prompt to disk and is described by its authors as not yet optimized. We therefore did not build LENS-SHIFT on it.

Table R7. Measured scorer cost as a share of a training step's compute (%).

ScorerOLMo mathNemotron mathQwen mathOLMo codeNemotron codeQwen code
PROBE-JS1.131.1327.227.20.470.473.003.0029.529.54.494.49
LENS-SHIFT13.613.629.129.17.27.215.715.728.628.612.612.6

On OLMo and Qwen, PROBE-JS is measured in the engine and is close to the modeled shares of Table 2 (1.31.3, 0.450.45, 3.23.2 and 4.34.3), and LENS-SHIFT with caching costs 77\\% to 1616\\%. On Nemotron, where neither scorer is cached, both cost 2727\\% to 3030\\%.

Table R8. Lower bound with exact per-token prefix reuse in the engine (%).

ScorerOLMo mathNemotron mathQwen mathOLMo codeNemotron codeQwen code
PROBE-JS1.111.110.680.680.440.442.322.320.580.584.274.27
LENS-SHIFT0.780.780.640.640.430.432.352.350.580.583.733.73

The gap between Tables R7 and R8 is implementation cost: the extra pass that LENS-SHIFT makes on a separate copy, and the absence of caching on Nemotron. It is not a cost of the signal.

Table R9. Trained tokens per step (thousands, mean over each run).

ArmOLMo mathNemotron mathQwen mathOLMo codeNemotron codeQwen code
MIDPOINT426.1426.1774.5774.51055.71055.7412.3412.31939.21939.2262.6262.6
ENTROPY520.0520.0834.2834.2857.9857.9371.3371.31896.41896.4267.5267.5
PROBE-JS432.1432.1708.8708.8980.9980.9415.1415.11910.61910.6280.5280.5
LENS-SHIFT486.3486.3755.6755.61018.21018.2416.4416.41945.31945.3321.4321.4

Trained tokens differ across arms by up to about 2020\\%, because response lengths evolve differently during training. For PROBE-JS the difference from MIDPOINT ranges from −9-9\\% to +7+7\\%.

2. Candidate range on base checkpoints

On base checkpoints, all scored criteria use the same candidate range: the belief reads and ENTROPY choose their first cut within the first 5050\\% of each chain, and later cuts are unrestricted. MIDPOINT is not scored; it halves the remaining span each time, so its first cut is at 5050\\%. The reason for the restriction is given in Appendix A: early in training a base model writes short, repetitive chains on which the belief barely moves, and the highest score falls on the last few tokens where an answer happens to appear, where a fork can no longer change the outcome. On OLMo, which starts from an SFT checkpoint, there is no such restriction and every arm selects over the full chain. The comparison between the belief reads and ENTROPY therefore isolates the selection signal under an identical candidate pool.

3. Comparison at a common measured compute budget

We measured, for every arm, the generated tokens per step, the trained tokens per step and the scorer cost, and combined them into a total cost per step (generated tokens, plus three times trained tokens, plus scorer token-forwards). We take Qwen code as the example, because PROBE-JS has the highest aggregate in that group and its cost per step is indeed higher than that of MIDPOINT and ENTROPY: 1.201.20 times MIDPOINT for PROBE-JS, and 1.041.04 times for ENTROPY. We re-indexed each validation curve by cumulative measured cost, took as the common budget the total cost of the cheapest arm in the group (MIDPOINT) over its 680680 steps, and read every arm within that budget.

Table R10. Qwen3-4B code at a common measured compute budget.

ArmCost per step (MIDPOINT = 1)Last step within budgetBest checkpoint within budget
MIDPOINT1.001.0068068031.7631.76 (step 580580)
ENTROPY1.041.0464064031.6931.69 (step 640640)
PROBE-JS1.201.2058058032.1932.19 (step 540540)

The best checkpoint of each arm falls within the common budget, so these are the same checkpoints as in Table 3. At equal measured compute, the comparison on Qwen code is unchanged.

For Weakness 2 & Question 2:

The pre-RL reconstruction study provides weaker evidence for the mechanism than the headline ranking suggests. Different belief variants win different panels, and BSV is often the strongest offline instrument even though it is not a main RL selector.

How stable are the reconstruction rankings under resampling of problems and Monte Carlo completions? Do PROBE-JS and LENS-SHIFT individually retain an advantage at the candidate count and fork budget used in RL?

We agree that the pre-RL study, as presented, had two gaps: it gave no uncertainty estimates, and it used a protocol different from training. We have added two analyses on the existing chains and Monte-Carlo completions, without new generation.

1. Reconstruction error with uncertainty

Table R4. Paired difference in reconstruction error (belief read minus baseline; negative means the belief read is better), with 9595\\% paired-bootstrap intervals in brackets. The protocol is that of Table 7 (1010 of 128128 boundaries).

BenchmarkBelief readvs. LLM judgevs. entropyvs. random
AIMEBSV−0.069-0.069 [−0.114,−0.027][-0.114, -0.027]−0.096-0.096 [−0.137,−0.058][-0.137, -0.058]−0.169-0.169 [−0.216,−0.127][-0.216, -0.127]
AIMEPROBE-JS+0.003+0.003 [−0.033,+0.040][-0.033, +0.040]−0.025-0.025 [−0.058,+0.008][-0.058, +0.008]−0.097-0.097 [−0.132,−0.067][-0.132, -0.067]
AIMELENS-SHIFT (shape)−0.008-0.008 [−0.042,+0.028][-0.042, +0.028]−0.035-0.035 [−0.065,−0.006][-0.065, -0.006]−0.108-0.108 [−0.143,−0.076][-0.143, -0.076]
GPQABSV−0.107-0.107 [−0.130,−0.084][-0.130, -0.084]−0.106-0.106 [−0.129,−0.084][-0.129, -0.084]−0.198-0.198 [−0.221,−0.176][-0.221, -0.176]
GPQAPROBE-JS−0.054-0.054 [−0.077,−0.031][-0.077, -0.031]−0.053-0.053 [−0.075,−0.030][-0.075, -0.030]−0.145-0.145 [−0.167,−0.124][-0.167, -0.124]
GPQALENS-SHIFT (shape)−0.104-0.104 [−0.131,−0.077][-0.131, -0.077]−0.103-0.103 [−0.130,−0.076][-0.130, -0.076]−0.195-0.195 [−0.222,−0.170][-0.222, -0.170]

On GPQA, the differences of all three belief reads against all three baselines are negative, with intervals that exclude zero: against entropy they are −0.106-0.106 (BSV), −0.053-0.053 (PROBE-JS) and −0.103-0.103 (LENS-SHIFT). On AIME, the differences against entropy are −0.096-0.096 (BSV), −0.035-0.035 (LENS-SHIFT) and −0.025-0.025 (PROBE-JS, interval [−0.058,+0.008][-0.058, +0.008]); against the LLM judge they are −0.069-0.069, −0.008-0.008 and +0.003+0.003.

2. The training protocol

We re-evaluated the selectors at the RL protocol: 1616 evenly spaced candidates per chain, and one fork per selector. The comparison uses the four arms trained in RL.

Table R11. Squared value shift at the selected fork, with the Monte-Carlo sampling variance subtracted, multiplied by 100100 (higher is better), with 9595\\% intervals in brackets.

SelectorAIMEGPQA
MIDPOINT0.3770.377 [0.125,0.700][0.125, 0.700]0.8320.832 [0.633,1.050][0.633, 1.050]
ENTROPY1.1191.119 [0.500,1.998][0.500, 1.998]1.0321.032 [0.831,1.253][0.831, 1.253]
PROBE-JS1.2111.211 [0.888,1.588][0.888, 1.588]1.3651.365 [1.171,1.568][1.171, 1.568]
LENS-SHIFT1.2971.297 [0.843,1.862][0.843, 1.862]1.4091.409 [1.185,1.651][1.185, 1.651]

At the training protocol, PROBE-JS and LENS-SHIFT reach 1.2111.211 and 1.2971.297 on AIME, against 0.3770.377 for MIDPOINT and 1.1191.119 for ENTROPY; on GPQA they reach 1.3651.365 and 1.4091.409, against 0.8320.832 for MIDPOINT and 1.0321.032 for ENTROPY. On both benchmarks the intervals of the two main selectors do not overlap that of MIDPOINT.

3. BSV

BSV is the strongest read in the offline study, and it was also run under RL (Appendix E.3): the direction is fit once on the SFT checkpoint that starts RL and then frozen, and that arm finishes first among the OLMo arms in both domains (24.124.1 against 23.323.3 for PROBE-JS on mathematics, and 36.136.1 against 34.934.9 on code; single seed). We kept BSV out of the main RL arms because a strictly on-policy use would have to refit the direction as the policy moves, which costs more than the two fit-free reads. The frozen version avoids that cost, but it still needs an offline fit on each starting checkpoint, so we validated it only from the SFT checkpoint.


Once again, thank you for all of your thoughtful questions and suggestions!

Reviewer_Rj87OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper asks where a scarce fork should go in tree-structured RLVR, and proposes placing it where the model's answer belief shifts most between consecutive boundaries. The belief is read three ways — an elicitation probe, a logit-lens depth profile, and an offline-fitted activation direction — and validated against Monte-Carlo value curves before training, where a belief-shift read ranks first in every model–benchmark panel. In RL, belief-shift forking leads the mathematics aggregates across three model families and sweeps the code columns on one of them

Critical strengths

1.The reframing is the right one: fork placement is a scoring problem over boundaries, the target object is the pivot of the value curve, and existing selectors are shown to be content-blind proxies for it. The diagnosis of entropy — it tracks freedom of wording, not of outcome — is crisp and supported by prior ablations, not just asserted. 2.Validating selectors against Monte-Carlo value curves before any training is the paper's best methodological move. It removes optimizer noise and seed variance from the comparison and yields a reusable instrument for a question the tree-RLVR literature usually settles by end-task score.

Critical weaknesses

1.The RL evidence is thinner than the abstract implies: two of three models have a single seed, the smallest margin sits inside the authors' own reported seed spread, and the pre-RL panels are small with no error bars. 2.The score is unsigned, so a segment where the model talks itself into a wrong answer scores like a recovery; and the paper's own contrast-collapse diagnostic says the gain shrinks exactly as the policy sharpens. 3.Validation and RL are not connected: the selector that wins before training is barely run during training, while the segment-level selector sharing this premise and the judge that matched the belief reads are never run under RL.

Questions for authors

1.Does the sign of the belief shift matter — would forking only where belief moves toward the wrong answer beat the symmetric score, and how many forks land in the settled tail rather than on true pivots? 2.Given contrast collapse, what happens on a stronger starting checkpoint or with siblings sampled to restore disagreement? If the gain disappears once siblings agree, what is the intended operating regime? 3.Why are the main mathematics results reported with the shorter probe read that the ablation shows to be worse, and how do they change under the better setting?

4 comments

We thank the reviewer for their assessment of our work! Below are our responses to the comments:


For Weakness 1:

The RL evidence is thinner than the abstract implies: two of three models have a single seed, the smallest margin sits inside the authors' own reported seed spread, and the pre-RL panels are small with no error bars.

1. A second seed on Qwen3-4B

Following the two-seed protocol of Appendix E.1, we ran one more seed on Qwen3-4B code for PROBE-JS, MIDPOINT and ENTROPY. The seed changes only the training-prompt shuffle and the mini-batch order; every other setting is that of seed 0.

Table R2. Qwen3-4B code, second seed. The last column is the mean and half-range over this seed and seed 0 of Table 3, computed from unrounded values.

ArmeasymediumhardaggregateMean ± half-range over two seeds
MIDPOINT78.578.514.414.43.13.132.032.031.9±0.131.9 \pm 0.1
ENTROPY78.878.814.114.13.33.332.132.131.9±0.231.9 \pm 0.2
PROBE-JS78.678.614.414.44.44.432.532.532.3±0.132.3 \pm 0.1

The difference between PROBE-JS and MIDPOINT is +0.4+0.4 and +0.5+0.5 on the two seeds, and the difference between PROBE-JS and ENTROPY is +0.5+0.5 and +0.4+0.4. The half-range of the three arms across the two seeds is 0.10.1 to 0.20.2.

2. Pre-RL reconstruction error with uncertainty

We added intervals to the pre-RL reconstruction errors, using the existing chains and Monte-Carlo completions: 149149 AIME chains and 596596 GPQA chains (the non-flat problems of Table 6), pooled over the four probe models.

Table R4. Paired difference in reconstruction error (belief read minus baseline; negative means the belief read is better), with 9595\\% paired-bootstrap intervals in brackets. The protocol is that of Table 7 (1010 of 128128 boundaries).

BenchmarkBelief readvs. LLM judgevs. entropyvs. random
AIMEBSV−0.069 [−0.114, −0.027]-0.069\ [-0.114,\ -0.027]−0.096 [−0.137, −0.058]-0.096\ [-0.137,\ -0.058]−0.169 [−0.216, −0.127]-0.169\ [-0.216,\ -0.127]
AIMEPROBE-JS+0.003 [−0.033, +0.040]+0.003\ [-0.033,\ +0.040]−0.025 [−0.058, +0.008]-0.025\ [-0.058,\ +0.008]−0.097 [−0.132, −0.067]-0.097\ [-0.132,\ -0.067]
AIMELENS-SHIFT−0.008 [−0.042, +0.028]-0.008\ [-0.042,\ +0.028]−0.035 [−0.065, −0.006]-0.035\ [-0.065,\ -0.006]−0.108 [−0.143, −0.076]-0.108\ [-0.143,\ -0.076]
GPQABSV−0.107 [−0.130, −0.084]-0.107\ [-0.130,\ -0.084]−0.106 [−0.129, −0.084]-0.106\ [-0.129,\ -0.084]−0.198 [−0.221, −0.176]-0.198\ [-0.221,\ -0.176]
GPQAPROBE-JS−0.054 [−0.077, −0.031]-0.054\ [-0.077,\ -0.031]−0.053 [−0.075, −0.030]-0.053\ [-0.075,\ -0.030]−0.145 [−0.167, −0.124]-0.145\ [-0.167,\ -0.124]
GPQALENS-SHIFT−0.104 [−0.131, −0.077]-0.104\ [-0.131,\ -0.077]−0.103 [−0.130, −0.076]-0.103\ [-0.130,\ -0.076]−0.195 [−0.222, −0.170]-0.195\ [-0.222,\ -0.170]

On GPQA, every belief read is better than every baseline, with intervals that exclude zero. On AIME, every belief read is better than random placement; against entropy and the LLM judge the differences are smaller, and the interval excludes zero for BSV against both and for LENS-SHIFT against entropy.

For Weakness 2 & Question 1:

The score is unsigned, so a segment where the model talks itself into a wrong answer scores like a recovery

Does the sign of the belief shift matter — would forking only where belief moves toward the wrong answer beat the symmetric score, and how many forks land in the settled tail rather than on true pivots?

1. Fork placement as a selection problem

Fork placement is a selection problem: from the mm candidate boundaries of a finished chain, a selector picks nn using only quantities read from the model itself. The tree estimates VV only at the points it forks, and every segment's advantage is computed from those estimates (Eq. 7), so a placement is good if the value curve interpolated through the chosen points tracks the true V∗V^\ast. This is the pre-RL instrument of Section 4, where V∗V^\ast enters only as the Monte-Carlo ground truth for scoring and never as an input to a selector.

2. Rises and drops are symmetric

The objective is unsigned by construction: it is invariant under V∗↦1−V∗V^\ast \mapsto 1 - V^\ast, which exchanges rises and drops, so the optimal set of points (the DP oracle of Section 4) is the same for both, and the direction of a change carries no information about where the points should go. RL credit agrees: conditional on the step that the siblings re-roll, the expected sibling-relative credit of the original branch is proportional to V∗(t)−V∗(t−)V^\ast(t) - V^\ast(t^-), the advantage of that step, which is nonzero and correctly signed for a derailment and for a recovery alike. A criterion that forks only where belief moves toward a wrong answer would therefore skip every recovery; on the Figure 1 chain, the PROBE-JS picks include both rises and drops of the Monte-Carlo's estimate of V∗V^\ast (Table 11).

A signed criterion needs a reference direction at every boundary, namely the probability the probe assigns to the correct answer. That is well defined for multiple choice, coarse for free-form mathematical answers that span several tokens, and undefined for code. We therefore tested the signed variant where the reference is well defined, on the GPQA pre-RL chains: each belief read ranks only the shifts in which belief moves toward a wrong answer.

Table R5. Fraction of selected forks that fall in the settled tail, the final stretch over which V∗V^\ast stays at the chain's outcome (GPQA; lower is better; 9595\\% bootstrap intervals), for the symmetric score and for the signed variant.

Belief readSymmetricSigned (shifts toward a wrong answer only)
BSV0.3170.317 [0.288,0.347][0.288, 0.347]0.3280.328 [0.303,0.352][0.303, 0.352]
PROBE-JS0.3740.374 [0.344,0.404][0.344, 0.404]0.3620.362 [0.329,0.389][0.329, 0.389]
LENS-SHIFT0.3130.313 [0.287,0.339][0.287, 0.339]0.3260.326 [0.291,0.359][0.291, 0.359]

The signed variant changes this fraction by about 0.010.01 for each read (+0.011+0.011, −0.012-0.012 and +0.013+0.013), well inside the intervals.

3. Forks in the settled tail

We measured this on all pre-RL chains.

Table R6. Fraction of the ten selected forks that fall in the settled tail (1010 of 128128 boundaries; lower is better; 9595\\% bootstrap intervals).

SelectorAIMEGPQA
BSV0.3050.305 [0.253,0.359][0.253, 0.359]0.3170.317 [0.288,0.347][0.288, 0.347]
PROBE-JS0.3280.328 [0.275,0.385][0.275, 0.385]0.3740.374 [0.344,0.404][0.344, 0.404]
LENS-SHIFT0.3510.351 [0.299,0.403][0.299, 0.403]0.3130.313 [0.287,0.339][0.287, 0.339]
LLM judge0.3410.341 [0.299,0.387][0.299, 0.387]0.3460.346 [0.320,0.372][0.320, 0.372]
entropy0.4360.436 [0.380,0.495][0.380, 0.495]0.3350.335 [0.306,0.363][0.306, 0.363]
newline0.4170.417 [0.373,0.463][0.373, 0.463]0.3940.394 [0.372,0.417][0.372, 0.417]
uniform0.4730.473 [0.425,0.523][0.425, 0.523]0.4540.454 [0.428,0.481][0.428, 0.481]
random0.5680.568 [0.514,0.621][0.514, 0.621]0.5210.521 [0.494,0.548][0.494, 0.548]

About a third of the forks placed by the belief reads fall in the settled tail: 3333\\% (AIME) and 3737\\% (GPQA) for PROBE-JS, and 3535\\% and 3131\\% for LENS-SHIFT, against 4747\\% and 4545\\% for uniform placement and 5757\\% and 5252\\% for random placement. The lowest fraction on each benchmark belongs to a belief read (BSV on AIME, LENS-SHIFT on GPQA), and every belief read is below newline, uniform and random placement on both. Against entropy (4444\\% and 3434\\%), PROBE-JS is lower on AIME and higher on GPQA.

For Weakness 3:

Validation and RL are not connected: the selector that wins before training is barely run during training, while the segment-level selector sharing this premise and the judge that matched the belief reads are never run under RL.

This comment makes three points, on BSV, on the segment-level selector and on the LLM judge. We take them in turn.

1. BSV under RL

In the pre-RL study, BSV serves as the most direct read of belief shift we could build, against which to judge how closely the two fit-free reads approximate it. It was also run under RL (Appendix E.3): the direction is fit once on the SFT checkpoint that starts RL and then frozen. That arm finishes first among the OLMo arms in both domains: 24.124.1 against 23.323.3 for PROBE-JS on mathematics, and 36.136.1 against 34.934.9 on code (Table 10, single seed). In this setting, the selector that is strongest before training is also the strongest under RL. We kept BSV out of the main RL arms because a strictly on-policy use would have to refit the direction as the policy moves, which costs more than the two fit-free reads. The frozen version avoids that cost, but it still needs an offline fit on each starting checkpoint, so we validated it only from the SFT checkpoint.

2. The segment-level selector (SPO)

We take "the segment-level selector sharing this premise" to mean SPO (Guo et al., 2026), which Section 2 describes as sharing our premise. If the reviewer had a different method in mind, we are happy to add it.

SPO marks as cutpoints the tokens whose sampled probability πθ(yt∣st)\pi_\theta(y_t \mid s_t) falls below a threshold and places a boundary after a fixed number of them. Our ENTROPY arm ranks the candidate segments by their mean sampled-token surprisal −log⁡πθ(yt∣st)-\log \pi_\theta(y_t \mid s_t) (Section 3.5). The two read the same quantity and differ in how it is aggregated and in where a cut may fall, so SPO's signal is already represented among the RL arms: it trails both belief-shift arms on OLMo in both domains and on Nemotron mathematics, and is within about a point of them elsewhere (Table 3). We also note that SPO uses this rule only in its chain variant for short reasoning; its tree variant partitions on a fixed token grid.

Others have compared this family of criteria. SAPO (Gao et al., 2026) ablates SPO's probability-based partition against an entropy-based one inside segment-level PPO on multi-modal reasoning and finds the entropy-based partition better on its three benchmarks. Lv et al. (2026) report a stable inverse relationship between token entropy and sampled-token probability. Neither study is in our tree-fork setting, so we cite them as context only.

3. The LLM judge

The judge matches the belief reads only on AIME (4.0694.069 against 4.0714.071 for PROBE-JS). On the held-out GPQA transfer it is level with entropy (6.0426.042 against 6.0416.041) and behind every belief-shift read (Table 7), so the pre-RL study gives no indication that it would be the stronger RL selector. Running it under RL would also make the rollout side substantially more expensive. Each step scores 32×4×16=2,04832 \times 4 \times 16 = 2{,}048 boundaries, and the judge has to read every chain at least once: on OLMo mathematics that is at least 170170k tokens per step, half of what the rollout itself generates and more than eight times the probe's 20,48020{,}480 token-forwards, processed by a much larger external model. It would also make fork placement depend on a model other than the policy. We therefore kept it as a pre-RL reference.

For Question 2:

Given contrast collapse, what happens on a stronger starting checkpoint or with siblings sampled to restore disagreement? If the gain disappears once siblings agree, what is the intended operating regime?

1. Contrast collapse

This is a general tendency of RLVR training: as training proceeds the policy commits, and continuations sampled from the same prefix increasingly reach the same answer, right or wrong, so their rewards tie and no step-level credit survives. It is the step-level form of the all-tie saturation that motivates dynamic sampling in DAPO (Section 2, Appendix A). Appendix D measures agreement with siblings forked at every boundary, so the collapse it documents does not depend on where the fork is placed: it is a property of the policy, not of the fork criterion. Consistent with this, the four Qwen arms end no more than a point apart in both domains (Table 3), and the tree itself gains little over chain RL there (Appendix E.2). In that small sample the ranking still held after the collapse (5555\\% vs. 2424\\% sibling disagreement in the top vs. bottom JS tercile); what is missing is contrast to convert.

2. A stronger starting checkpoint

The pre-RL study is run on such checkpoints: four post-trained models, two of them reasoning ("Think") models. On them the outcome is still undecided on 2929 to 5151 of the 6060 AIME problems and on 8383 to 181181 of the 198198 GPQA problems (non-flat V∗V^\ast, Table 6). The belief reads also find the undecided part of a chain more often than default placement does: they put 6363\\% to 7070\\% of their forks before the outcome settles, against 5353\\% to 5555\\% for uniform placement and 4343\\% to 4848\\% for random placement (Table R6), and each of them reconstructs the value curve better than random placement (Table R4). Sampling siblings to restore disagreement is complementary to our method: it decides whether a fork has contrast, while the criterion decides where to spend it.

3. The intended operating regime

Once continuations from the same prefix agree, their rewards tie and no outcome-based method has a signal: chain RL loses it as well, and where the fork is placed plays no role. The intended regime is therefore the one in which critic-free RLVR trains at all, namely problems on which the policy's continuations still disagree.

For Question 3:

Why are the main mathematics results reported with the shorter probe read that the ablation shows to be worse, and how do they change under the better setting?

Why the main rows use L=1L=1. The reason is the order of the experiments. Mathematics uses L=1L=1 (Section 3.2), and the read-length ablation was run after the main arms (Section 7), so the mathematics rows of Table 3 were not back-filled and keep the earlier default.

How the results change. For OLMo-3-7B, the L=4L=4 row of Table 4(c) is the main PROBE-JS arm with only the read length changed: it reaches 26.826.8 aggregate against 23.323.3 at L=1L=1 (AIME 2026: 22.522.5 against 15.215.2). On this model the main table therefore reports the more conservative of the two settings. We treat this as suggestive rather than settled: it is a single run, and L=2L=2 (22.822.8) is slightly below L=1L=1. For Qwen and Nemotron, the mathematics rows remain at L=1L=1.


Once again, thank you for all of your thoughtful questions and suggestions!

Reviewer_Rk5fOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies where to fork a reasoning trace in tree RL. A random or fixed point may use samples without giving useful reward differences. The main method first generates full traces. One version asks for an answer at candidate points. It compares probability predictions for the start of the answer. Another version reads how internal layers support the original trace's own final answer. It finds the largest change between nearby points, then forks before that part. The main setting starts with four root traces. Each gets two new continuations, giving 12 complete answers. Reward differences between branches then give training signals to the related reasoning segments.

Critical strengths

I think this paper is clear and useful. Before RL, the paper checks its signals against Monte Carlo value curves. It then tests different model families on math and code. Tree budget and other design choices are also tested. These experiment make the evidence stronger than one good training run.

Critical weaknesses

(1) Table 4(b) tests Probe-JS with standard GRPO, but does not report Midpoint with that updater. The drop from 23.3 to 15.2 shows that the updater matters for Probe-JS. It does not tell us whether Probe-JS still improves over Midpoint after this change. The main comparison under the composite updater remains valid, but the benefit of the proposed fork selection under standard GRPO is unclear.

(2) The second seed strengthens the OLMo results. The Qwen3-4B gains are smaller: 0.9 points for Lens-Shift on math and 0.4 for Probe-JS on code over the strongest baseline. Appendix A also reports that Entropy leads on Qwen code under the fixed-budget checkpoint rule. It is still unclear whether these small gains hold across Qwen training runs. This limits confidence in the Qwen gains, rather than the larger OLMo gains.

Questions for authors

(1) Could the authors add a Midpoint run using the same standard GRPO setting as in Table 4(b), with the same tree structure and training budget? Does Probe-JS still outperform Midpoint? This would help show whether the fork-selection gain remains after changing the updater.

(2) Could you repeat the Qwen3-4B code comparison across training seeds and report the accuracy differences between Probe-JS, Midpoint and Entropy? Please use a common checkpoint-selection rule and show how the conclusion changes between best-checkpoint and fixed-budget reporting.

1 comment

We thank the reviewer for their assessment of our work! Below are our responses to the comments:


For Weakness 1 & Question 1:

Table 4(b) tests Probe-JS with standard GRPO, but does not report Midpoint with that updater.

Could the authors add a Midpoint run using the same standard GRPO setting as in Table 4(b), with the same tree structure and training budget?

We ran MIDPOINT with the standard-GRPO updater of Table 4(b), with the same tree structure and training budget as the PROBE-JS run.

Table R1. MIDPOINT and PROBE-JS under the standard-GRPO updater of Table 4(b) (OLMo-3-7B mathematics).

Fork criterionOlymp.AIMEOmniAgg.
MIDPOINT22.222.24.94.916.816.814.614.6
PROBE-JS23.623.64.64.617.417.415.215.2

Under standard GRPO, the difference between PROBE-JS and MIDPOINT is +1.41.4 on OlympiadBench, −0.30.3 on AIME 2026, and +0.60.6 on Omni-MATH-500. Both are single runs. For reference, the aggregate difference under the composite updater is +3.83.8 (Table 4(b)).

For Weakness 2 & Question 2:

It is still unclear whether these small gains hold across Qwen training runs.

Could you repeat the Qwen3-4B code comparison across training seeds and report the accuracy differences between Probe-JS, Midpoint and Entropy?

Following the two-seed protocol of Appendix E.1, we ran one more seed on Qwen3-4B code for PROBE-JS, MIDPOINT and ENTROPY. The seed changes only the training-prompt shuffle and the mini-batch order; every other setting is that of seed 0.

1. Best-checkpoint rule

Table R2. Qwen3-4B code, second seed. The last column is the mean and half-range over this seed and seed 0 of Table 3.

ArmeasymediumhardaggregateMean ± half-range over two seeds
MIDPOINT78.578.514.414.43.13.132.032.031.931.9 ± 0.10.1
ENTROPY78.878.814.114.13.33.332.132.131.931.9 ± 0.20.2
PROBE-JS78.678.614.414.44.44.432.532.532.332.3 ± 0.10.1

The difference between PROBE-JS and MIDPOINT is +0.40.4 and +0.50.5 on the two seeds, and the difference between PROBE-JS and ENTROPY is +0.50.5 and +0.40.4.

2. Fixed-step rule

The fixed-budget rule of Appendix A reads every arm at the final step (step 680). Here we instead use an earlier common step, to compare the arms at the smallest number of steps at which one of them has reached its best checkpoint.

Table R3. Aggregate when all arms are read at that step: step 540 on seed 0 and step 520 on seed 1.

Armseed 0, step 540seed 1, step 520
MIDPOINT31.431.431.531.5
ENTROPY30.530.531.831.8
PROBE-JS32.232.232.532.5

At this step, PROBE-JS is at 32.232.2 and 32.532.5, MIDPOINT at 31.431.4 and 31.531.5, and ENTROPY at 30.530.5 and 31.831.8.


Once again, thank you for all of your thoughtful questions and suggestions!

Comments on the paper

1 comment

We thank all reviewers for their careful reading and constructive comments! The reviews raised four common themes: the strength of the RL evidence on Qwen3-4B, the uncertainty and protocol of the pre-RL study, the cost of choosing the fork, and the roles of the updater and of the sign of the belief shift. In response we added (i) four new RL runs: MIDPOINT under the standard-GRPO updater, and a second seed on Qwen3-4B code for PROBE-JS, MIDPOINT and ENTROPY; (ii) new analyses on the existing pre-RL chains and Monte-Carlo completions; and (iii) measured compute for every arm, including the scorers. The results are reported in Tables R1 to R11 in the individual replies. The table below summarizes what each one shows.

TableWhat it shows
R1MIDPOINT and PROBE-JS under the standard-GRPO updater (OLMo-3-7B mathematics).
R2A second seed on Qwen3-4B code for PROBE-JS, MIDPOINT and ENTROPY.
R3Qwen3-4B code with all arms read at the same step.
R4Pre-RL reconstruction error with 9595\\% paired-bootstrap intervals.
R5A signed variant of the score against the symmetric score (GPQA).
R6Fraction of selected forks that fall in the settled tail.
R7Measured scorer cost as a share of a training step.
R8Lower bound on the scorer cost with exact prefix reuse in the engine.
R9Trained tokens per step for every arm.
R10Qwen3-4B code at a common measured compute budget.
R11The selectors at the training protocol (1616 candidates, one fork).

Tables are numbered across all replies, and a table that appears in two replies keeps the same number.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.