- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FORK-RL uses a learned head to pick where to branch existing rollouts and improve GRPO sampling efficiency. Shows promising improvements in coding benchmarks, and nice intuition.
Critical strengths
- Creative, low overhead use of hidden states to select informative fork points; learned selection is better than random cuts indicating this method is helpful.
- Matched comparisons show gains at 4B and 8B; failure-only branching retains similar performance.
Critical weaknesses
The benefit of forcing first-token divergence is not isolated; banning a near-certain token could produce unhelpful continuations.
Questions for authors
- Does FORK-RL retain its gains in a matched ablation without the first-token ban, particularly when the original next token has probability near one?
- Would allocating multiple continuations adaptively across parents improve performance at a fixed rollout budget? How does this relate to the failure-only ablation in §5.5?
- There is overlapping text below equation 1 and elsewhere in the PDF. Please check the formatting before final publication.
2 comments
- Thanks for running these comparisons. That answers my main question on branching from both correct and incorrect solutions!
- I’d include both tables showing Qwen3-4B’s performance at step 150.
- I’m comfortable leaving the choice on how to split the rollout budget across branches to future work. No further questions.
Response to Reviewer iw8g
Thank you for recognizing the selector's low overhead and the gains at 4B and 8B.
[W1/Q1: first-token divergence] Do the gains remain without the token ban, especially when the original token is nearly certain?
We tested Qwen3-4B with and without the token ban. The table shows evaluation accuracy (%) for both methods at training step 150:
Avg is the simple average across four benchmarks. The ablation supports using the ban at step 150.
[Q2: failure-only forking] What does failure-only forking establish?
Adaptive allocation of continuations across parents is nontrivial, as it must decide which parents to expand and how many continuations to assign under a fixed budget. This is orthogonal to our learned fork-position selection and is beyond the scope of this work. It can be combined with FORK-RL.
Failure-only forking keeps the training-reward gain. At step 100, failure-only and full forking reach 0.549 and 0.547 (§5.5). We also tested both methods on held-out benchmarks:
Qwen3-4B evaluation accuracy (%) at step 150:
Avg is the simple average across four benchmarks.
Full forking scores higher on every benchmark, showing the value of forking both correct and incorrect trajectories for held-out accuracy.
[Q3: formatting] Will the overlapping equation text and other layout issues be fixed?
We will fix equation spacing, overlapping table lines and Algorithm 1 text, and outdated figure references. We will then check the full revised PDF for readability.