- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper chooses fork locations in critic-free tree-structured RLVR by measuring changes in a model's belief about its final answer. PROBE-JS compares answer-elicited token distributions at candidate boundaries. LENS-SHIFT tracks layerwise probability profiles for the chain's committed answer. The method forks just before a large change, allowing sibling continuations to try a different step. An offline learned belief-shift direction supplies additional validation and a frozen-direction RL experiment. The evaluation covers value-curve reconstruction before RL and training on mathematics and code across three model families. Belief-based selectors improve the mathematics aggregate on all three and give substantial OLMo code gains. Qwen gains are small, and midpoint remains best on Nemotron code.
Critical strengths
- Forking before a change in answer belief gives the additional continuations a specific role: they can revisit the step that changed the prediction. PROBE-JS and LENS-SHIFT provide two implementations without a trained critic, while the shared tree and optimizer make the selector's contribution reasonably direct to test. This is a focused methodological advance over structural or token-uncertainty selectors. The paper correctly limits its JS bound to the elicited distribution; the bound does not guarantee a change in eventual success probability.
- The mathematics aggregate improves on all three model families, and the strongest gains are large enough to be practically interesting. On OLMo, PROBE-JS raises mathematics from the strongest baseline's 20.7 to 23.3 and code from 31.0 to 34.9 (Table 3). A second OLMo seed preserves the aggregate advantage of both belief methods over both baselines (Appendix E.1). A chain control with a token budget and checkpoint-selection checks add support (Appendices E.2 and A). The small Qwen gains and the Nemotron code exception usefully limit the claim: benefit depends on the model and domain.
Critical weaknesses
- The practical efficiency claim remains uncertain because the reported probe costs are modeled FLOP shares that assume prefix reuse. LENS-SHIFT actually re-forwards full prefixes on a colocated model, and Nemotron PROBE-JS uses a separate Hugging Face path (Appendix B). Fork location also changes continuation length, so equal tree sizes or optimizer-step caps need not mean equal total compute. Appendix A restricts Base-model belief reads to the first half of each chain, but does not make equally clear whether every baseline uses that range. These details matter most where gains are small, as on Qwen. Measured accuracy at a common total compute budget is needed to establish the selector's practical advantage.
- The pre-RL reconstruction study provides weaker evidence for the mechanism than the headline ranking suggests. Different belief variants win different panels, and BSV is often the strongest offline instrument even though it is not a main RL selector. PROBE-JS has average AIME reconstruction error 4.071, compared with 4.069 for the LLM judge and 4.100 for entropy (Table 7). The curves use only 16 continuations per boundary, without uncertainty estimates for these small differences. Selecting ten of 128 boundaries also differs from selecting one of 16 during RL. The downstream gains support the method's usefulness, but the reconstruction study does not yet establish a reliable advantage for the main selectors under the training protocol.
Questions for authors
- What are the actual generated/trained tokens and measured end-to-end compute, including colocated scorers, for each method? Do all selectors use the same candidate range on Base checkpoints? A comparison at common measured compute would show whether the proposed allocation signal improves the accuracy-cost tradeoff, especially for the small Qwen gains.
- How stable are the reconstruction rankings under resampling of problems and Monte Carlo completions? Do PROBE-JS and LENS-SHIFT individually retain an advantage at the candidate count and fork budget used in RL? This would separate evidence for the deployed selectors from the stronger results of other offline belief variants
3 comments
3. Comparison at a common measured compute budget
We measured, for every arm, the generated tokens per step, the trained tokens per step and the scorer cost, and combined them into a total cost per step (generated tokens, plus three times trained tokens, plus scorer token-forwards). We take Qwen code as the example, because PROBE-JS has the highest aggregate in that group and its cost per step is indeed higher than that of MIDPOINT and ENTROPY: times MIDPOINT for PROBE-JS, and times for ENTROPY. We re-indexed each validation curve by cumulative measured cost, took as the common budget the total cost of the cheapest arm in the group (MIDPOINT) over its steps, and read every arm within that budget.
Table R10. Qwen3-4B code at a common measured compute budget.
| Arm | Cost per step (MIDPOINT = 1) | Last step within budget | Best checkpoint within budget |
|---|---|---|---|
| MIDPOINT | (step ) | ||
| ENTROPY | (step ) | ||
| PROBE-JS | (step ) |
The best checkpoint of each arm falls within the common budget, so these are the same checkpoints as in Table 3. At equal measured compute, the comparison on Qwen code is unchanged.
For Weakness 2 & Question 2:
The pre-RL reconstruction study provides weaker evidence for the mechanism than the headline ranking suggests. Different belief variants win different panels, and BSV is often the strongest offline instrument even though it is not a main RL selector.
How stable are the reconstruction rankings under resampling of problems and Monte Carlo completions? Do PROBE-JS and LENS-SHIFT individually retain an advantage at the candidate count and fork budget used in RL?
We agree that the pre-RL study, as presented, had two gaps: it gave no uncertainty estimates, and it used a protocol different from training. We have added two analyses on the existing chains and Monte-Carlo completions, without new generation.
1. Reconstruction error with uncertainty
Table R4. Paired difference in reconstruction error (belief read minus baseline; negative means the belief read is better), with paired-bootstrap intervals in brackets. The protocol is that of Table 7 ( of boundaries).
| Benchmark | Belief read | vs. LLM judge | vs. entropy | vs. random |
|---|---|---|---|---|
| AIME | BSV | |||
| AIME | PROBE-JS | |||
| AIME | LENS-SHIFT (shape) | |||
| GPQA | BSV | |||
| GPQA | PROBE-JS | |||
| GPQA | LENS-SHIFT (shape) |
On GPQA, the differences of all three belief reads against all three baselines are negative, with intervals that exclude zero: against entropy they are (BSV), (PROBE-JS) and (LENS-SHIFT). On AIME, the differences against entropy are (BSV), (LENS-SHIFT) and (PROBE-JS, interval ); against the LLM judge they are , and .
2. The training protocol
We re-evaluated the selectors at the RL protocol: evenly spaced candidates per chain, and one fork per selector. The comparison uses the four arms trained in RL.
Table R11. Squared value shift at the selected fork, with the Monte-Carlo sampling variance subtracted, multiplied by (higher is better), with intervals in brackets.
| Selector | AIME | GPQA |
|---|---|---|
| MIDPOINT | ||
| ENTROPY | ||
| PROBE-JS | ||
| LENS-SHIFT |
At the training protocol, PROBE-JS and LENS-SHIFT reach and on AIME, against for MIDPOINT and for ENTROPY; on GPQA they reach and , against for MIDPOINT and for ENTROPY. On both benchmarks the intervals of the two main selectors do not overlap that of MIDPOINT.
3. BSV
BSV is the strongest read in the offline study, and it was also run under RL (Appendix E.3): the direction is fit once on the SFT checkpoint that starts RL and then frozen, and that arm finishes first among the OLMo arms in both domains ( against for PROBE-JS on mathematics, and against on code; single seed). We kept BSV out of the main RL arms because a strictly on-policy use would have to refit the direction as the policy moves, which costs more than the two fit-free reads. The frozen version avoids that cost, but it still needs an offline fit on each starting checkpoint, so we validated it only from the SFT checkpoint.
Once again, thank you for all of your thoughtful questions and suggestions!
We thank the reviewer for their assessment of our work! Below are our responses to the comments:
For Weakness 1 & Question 1:
1. How each scorer runs, and what it costs
We have added key-value caching, which leaves the scores unchanged, and measured the cost of both scorers. PROBE-JS needs only next-token log-probabilities, which the serving engine returns, so on OLMo and Qwen it runs inside the vLLM engine on the rollout's prefix cache. LENS-SHIFT needs intermediate-layer states, which the engine's standard interface does not return; on OLMo and Qwen it runs on a co-located Hugging Face copy of the policy, now with key-value caching On Nemotron, both scorers run on the Hugging Face model without caching: in-engine probing destabilized that hybrid-Mamba model's rollouts, and caching its recurrent state would require changes to the model code.
vLLM has had a hidden-state extraction path since v0.18.0 (https://github.com/vllm-project/vllm/pull/33736), but it was built for training speculative-decoding draft models: its current connector saves the hidden states of the whole prompt to disk and is described by its authors as not yet optimized. We therefore did not build LENS-SHIFT on it.
Table R7. Measured scorer cost as a share of a training step's compute (%).
On OLMo and Qwen, PROBE-JS is measured in the engine and is close to the modeled shares of Table 2 (1.3, 0.45, 3.2 and 4.3), and LENS-SHIFT with caching costs 7 to 16. On Nemotron, where neither scorer is cached, both cost 27 to 30.
Table R8. Lower bound with exact per-token prefix reuse in the engine (%).
The gap between Tables R7 and R8 is implementation cost: the extra pass that LENS-SHIFT makes on a separate copy, and the absence of caching on Nemotron. It is not a cost of the signal.
Table R9. Trained tokens per step (thousands, mean over each run).
Trained tokens differ across arms by up to about 20, because response lengths evolve differently during training. For PROBE-JS the difference from MIDPOINT ranges from −9 to +7.
2. Candidate range on base checkpoints
On base checkpoints, all scored criteria use the same candidate range: the belief reads and ENTROPY choose their first cut within the first 50 of each chain, and later cuts are unrestricted. MIDPOINT is not scored; it halves the remaining span each time, so its first cut is at 50. The reason for the restriction is given in Appendix A: early in training a base model writes short, repetitive chains on which the belief barely moves, and the highest score falls on the last few tokens where an answer happens to appear, where a fork can no longer change the outcome. On OLMo, which starts from an SFT checkpoint, there is no such restriction and every arm selects over the full chain. The comparison between the belief reads and ENTROPY therefore isolates the selection signal under an identical candidate pool.