- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes TreeRPO, a GRPO variant that replaces flat trajectory sampling with tree-structured sampling for RLVR on mathematical reasoning. Continuations are sampled in fixed-length segments to a bounded depth, terminal paths are scored by a binary verifier, and rewards are propagated upward by averaging over children, giving fractional estimates of downstream success without a learned process reward model. Siblings under the same prefix form comparison groups; near-uniform groups are filtered out, and advantages are scaled by the group's Bernoulli variance instead of the empirical standard deviation.
The stated contributions are:
- outcome-derived intermediate supervision via sibling groups at shared prefixes
- range-based group filtering combined with a mean-dependent advantage scale
- an empirical comparison against GRPO at two model sizes.
Validation uses Qwen2.5-Math-1.5B and 7B trained on the MATH training split and evaluated on AIME24, MATH-500, OlympiadBench, and Minerva Math. Reported macro accuracy improves for both models and all benchmarks, comparing to naive GRPO.
Critical strengths
None
Critical weaknesses
1. No compute-matched comparison. A depth-3, 8-way tree generates up to 584 segments and 512 verifier calls per question versus 8 and 8 for GRPO, and the two methods differ in gradient updates per step (16 vs 2 under the stated minibatch rule). All curves nevertheless use training steps as the x-axis, and no GRPO baseline with a larger rollout budget is reported. The paper acknowledges this in Sections 4.3, 4.5 and argues from the curves anyway. The central claim of a "practical alternative" is therefore unsupported.
2. Gains are indistinguishable from noise or model-specific artifacts. Single run per setting, no variance. The 7B macro gain is 0.7 points; the 1.5B AIME24 gain is about one problem out of 30. All experiments use Qwen2.5-Math, where spurious rewards have been shown to produce comparable MATH-500 gains(findings in https://arxiv.org/pdf/2506.10947), so the improvement cannot be credited to the method. The evaluation protocol is itself inconsistent (8 samples in text, Avg@16 in the table caption) and marked as "requires confirmation", which is very suspicious to be the LLM-generated text.
3. No component is isolated. Tree sampling, group filtering, and the advantage scale are never ablated. None of the three stated contributions is individually supported.
4. The length reduction has an unexamined structural cause. Early-terminating children receive rewards in {0, 1} while continuing children receive fractional means, so terminating segments occupy the extremes of every group's advantage distribution. On MATH training data, correct terminations dominate, pushing toward shorter outputs; the normalization adds the known per-token length bias. The paper offers speculation and no test, so the length result cannot be read as a benefit.
5. The µ(1−µ) advantage scale is unmotivated. Preserving reward-gap magnitude only requires not normalizing (A = R − µ). Dividing by µ(1−µ) instead amplifies groups with extreme µ: the paper's example sits at µ = 0.5, where amplification is minimal, while a valid group such as (0, 0, 0, 0, 0, 0, 0, 0.125) yields a top advantage near 7 against 0.9 under standard normalization. The authors note the endpoint issue and provide no measurement.
6. Positioning does not match implementation, and the closest prior work is missing. "Step-level" means fixed 384-token blocks with two branch points per path, unrelated to reasoning steps. Methods that estimate intermediate advantages from rollouts at prefixes are neither cited nor compared: VinePPO (https://arxiv.org/abs/2410.01679), SPO, whose SPO-tree variant uses tree-based segment advantage estimation (https://arxiv.org/abs/2505.23564), TreeRL (https://arxiv.org/abs/2506.11902), and rStar-Math (https://arxiv.org/abs/2501.04519). With GRPO as the only baseline, the paper does not establish what tree-relative grouping adds.
Questions for authors
- Please report the measured per-question generation cost of TreeRPO relative to GRPO (generated tokens, verifier calls, and wall-clock time), accounting for early termination, together with the number of gradient updates per training step under each method's minibatch configuration.
- Please confirm whether the results in Table 1 are from a single training seed, and whether evaluation uses eight or sixteen samples per question.
- Among retained groups, how frequently do early-terminating children (whose rewards lie in {0, 1}) receive the largest-magnitude advantages, and what is the empirical distribution of ? These statistics determine whether the observed length reduction and the proposed advantage scale reflect intended behavior or estimator artifacts.
- A GRPO baseline whose rollout budget matches the realized generation cost of TreeRPO.
- Ablations isolating tree sampling, group filtering, and the scale, including the unnormalized alternative A = R − µ.
- Results on a model family other than Qwen2.5-Math (like more recent Qwen series) with multiple seeds, and a matched-compute comparison against other baselines, e.g. VinePPO or SPO-tree.