acceptodds
Under review as a conference paper at ICLR 2027

Sample Efficient GRPO via Learning to Reuse Tokens Across Trajectories

Abstract

We propose FORK-RL: forking rollout trajectories in the middle and reusing the corresponding prefixes to boost RL post-training efficiency of LLMs in coding. GRPO estimates advantage based on group variety, hence the learning signal vanishes when all solutions in a group agree, whether they are all correct or wrong. As a result, such generated groups waste rollout budget and limit performance. Rather than increasing the sheer group size, FORK-RL resumes from the middle of existing trajectories to generate new solutions with different outcomes without additional rollout budget. We further introduce a learnable light-weight score head to identify the critical forking tokens. The score head decides where to fork according to the hidden states of random candidates. We train this head with a fiip reward when the new soluton leads to a different outcome. Our method not only endows the disagreement within groups, but also achieves finer grained credit assignment in the prefix-sharing trajectories. On competitive programming benchmarks, we achieve better performance than the compute-matched baseline: on CodeContests with a 4B policy model, a 17.8% relative gain in pass@1 (24.5% against the baseline’s 20.8%). Notably, our method reaches the baseline’s training accuracy on 62% fewer decoded tokens at 4B and 46% fewer at 8B.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iw8gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FORK-RL uses a learned head to pick where to branch existing rollouts and improve GRPO sampling efficiency. Shows promising improvements in coding benchmarks, and nice intuition.

Critical strengths
  • Creative, low overhead use of hidden states to select informative fork points; learned selection is better than random cuts indicating this method is helpful.
  • Matched comparisons show gains at 4B and 8B; failure-only branching retains similar performance.
Critical weaknesses

The benefit of forcing first-token divergence is not isolated; banning a near-certain token could produce unhelpful continuations.

Questions for authors
  • Does FORK-RL retain its gains in a matched ablation without the first-token ban, particularly when the original next token has probability near one?
  • Would allocating multiple continuations adaptively across parents improve performance at a fixed rollout budget? How does this relate to the failure-only ablation in §5.5?
  • There is overlapping text below equation 1 and elsewhere in the PDF. Please check the formatting before final publication.
2 comments
AuthorsOpenReview ↗

Response to Reviewer iw8g

Thank you for recognizing the selector's low overhead and the gains at 4B and 8B.

[W1/Q1: first-token divergence] Do the gains remain without the token ban, especially when the original token is nearly certain?

We tested Qwen3-4B with and without the token ban. The table shows evaluation accuracy (%) for both methods at training step 150:

MethodStepLiveCodeBenchMBPP+HumanEval+CodeContestsAvg
FORK-RL with ban15046.974.683.024.557.3
FORK-RL without ban15045.774.482.224.756.8

Avg is the simple average across four benchmarks. The ablation supports using the ban at step 150.

[Q2: failure-only forking] What does failure-only forking establish?

Adaptive allocation of continuations across parents is nontrivial, as it must decide which parents to expand and how many continuations to assign under a fixed budget. This is orthogonal to our learned fork-position selection and is beyond the scope of this work. It can be combined with FORK-RL.

Failure-only forking keeps the training-reward gain. At step 100, failure-only and full forking reach 0.549 and 0.547 (§5.5). We also tested both methods on held-out benchmarks:

Qwen3-4B evaluation accuracy (%) at step 150:

Forking strategyStepLiveCodeBenchMBPP+HumanEval+CodeContestsAvg
Full forking (Table 1 reference)15046.974.683.024.557.3
Failure-only forking15043.273.681.021.754.9
Correct-only forking15042.273.181.221.254.4

Avg is the simple average across four benchmarks.

Full forking scores higher on every benchmark, showing the value of forking both correct and incorrect trajectories for held-out accuracy.

[Q3: formatting] Will the overlapping equation text and other layout issues be fixed?

We will fix equation spacing, overlapping table lines and Algorithm 1 text, and outdated figure references. We will then check the full revised PDF for readability.

  • Thanks for running these comparisons. That answers my main question on branching from both correct and incorrect solutions!
  • I’d include both tables showing Qwen3-4B’s performance at step 150.
  • I’m comfortable leaving the choice on how to split the rollout budget across branches to future work. No further questions.
Reviewer_vG9YOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Fork-RL targets zero-advantage GRPO groups (68% at step 1 for Qwen3-4B, Fig. 2). It samples n=4n=4 solutions per problem. For each one, a linear score head v∈Rdv\in\mathbb{R}^d on normalized last-layer hidden states picks a cut kk among KK random candidates. A continuation is then decoded from the reused prefix o<ko_{<k} with the parent's next token banned, giving a GRPO group of 2n=82n=8. The head is trained by REINFORCE on the flip reward 1[r(q,c)≠r(q,o)]\mathbf{1}[r(q,c)\neq r(q,o)], and the fork-token PPO ratio is corrected for the ban (Eq. 7–8). On TACO with Qwen3-4B/8B, Fork-RL beats group-size-matched GRPO in all 16 benchmark × metric cells at step 150 (e.g., CodeContests pass@1 20.8 → 24.5 at 4B). It also reaches the baseline's training reward with 62%/46% fewer decoded tokens. Ablations attribute the gain to prefix-localized credit assignment.

Critical strengths

S1. Cheap, drop-in fix for a real problem. It adds one trainable vector (2,560 parameters at 4B), with no auxiliary model and no gradient into the LM, and it reuses already-decoded prefixes. Step efficiency looks real: at step 100 (Table 2, K=8K=8), Fork-RL matches or beats GRPO at step 150 (Table 1) on all four benchmarks. If this holds up, the method is easy to adopt.

S2. Mechanism-oriented diagnostics. The paper corrects for the token ban, builds a reward-mix-matched control, and reports a 1.7B null result (App. C). This is a solid base for a revision, even though the current conclusions do not hold (W2).

Critical weaknesses

W1. The headline claims are not supported as measured.

  • Single seed, tiny margins. There are no seeds or confidence intervals anywhere. Several of the "16/16 wins" amount to one or two problems (e.g., HumanEval+ pass@8 at 8B: 92.7 → 93.3 on 164 problems). The KK trend in Table 2 rests on 0.1–1.3-point gaps.
  • Asymmetric token metric. "62%/46% fewer tokens" takes the point where Fork-RL first reaches the baseline's reward at 150M and compares it with 150M itself. It should compare with the point where the baseline first reached that reward. The baseline hit 0.525 at 120M vs. 0.506 at 152.6M (Table 3), and in Fig. 3 (8B) it seems to reach the dashed level near 90–100M. Fork-RL's curve also averages 4 rollouts per problem vs. 8, so it is noisier and crosses thresholds earlier by chance.
  • The saving is mostly shorter outputs, not reuse. At 200 steps, Fork-RL logged 186.7M response tokens before the prefix correction vs. 239.3M for GRPO (Table 3, Fig. 10). Assuming the uncorrected total includes reused prefixes, reuse saves 34.1M (18% of Fork-RL's tokens). The other 52.6M comes from ~22% shorter responses, which the paper never reports or explains.
  • Tokens ≠ compute, and "final" is overstated. The abstract says "compute-matched", but compute is never measured: there is no wall-clock or GPU-hour data, despite two serial generation stages. The "final" baseline reward is taken at the 150M window, while GRPO ran to 374.8M (4B). On held-out data, the step-matched LiveCodeBench gap closes to +0.8 points (Fig. 10). At 4B the baseline reaches the same final pass@1 (≈0.50) given more tokens (Fig. 7).
  • Unclear experimental control. Fig. 10 calls its pair "the only matched pair". Sec. 4.2 (KK uniform positions) conflicts with the appendix (delimiter rule, "skip-header walk", 36–48 candidates), and Table 5 and Fig. 8 come from "a different fork run". It is unclear which settings produced Table 1 and Fig. 3.

W2. Neither the novelty nor the mechanism is established.

  • Missing baselines. The paper argues against entropy-based fork rules but never tests one; the only alternative selector is random. The closest prior work is neither cited nor compared:

    • TreeRL (ACL 2025, arXiv:2506.11902): entropy forking with branch-outcome credit, including code results.
    • FR3E (arXiv:2507.07017).
    • AttnRL (ICLR 2026, arXiv:2509.26628): branches chosen from attention, which weakens Sec. 2's novelty claim.
    • VinePPO (arXiv:2410.01679).

    DAPO dynamic sampling and RL-ZVP (ICLR 2026, arXiv:2509.21880) are also missing as baselines at matched tokens.

  • Ablations contradict the explanation. Random cuts give essentially no gain (Sec. 5.3), yet they have the same shared-prefix structure that Sec. 5.4 credits, and they still flip 0.129 of the time (vs. 0.183 learned). So "credit assignment" is not identified; the data point to an interaction of flips and shared prefixes. Sec. 5.5 (failure-only ≈ full forking, 0.549 vs. 0.547) contradicts the paper's critique of failure-only prior work. It also varies the group size (4–8), which Sec. 3 says must stay fixed.

  • Credit is only half-localized. The paper does not say whether reused prefix tokens are in the loss. If they are, a fork pair in a group with mean mm and std ss gets A^P/C=Vˉ−ms⏟not localized±rP−rC2s,Vˉ=12(rP+rC).\hat A_{P/C}=\underbrace{\frac{\bar V-m}{s}}_{\text{not localized}}\pm\frac{r_P-r_C}{2s},\qquad \bar V=\tfrac12(r_P+r_C). Explicit tree advantages (suffix r−Vˉr-\bar V, prefix Vˉ−m\bar V-m) are the direct test, and they are missing.

  • The gradient is not "still for πθ\pi_\theta" (Sec. 4.3). Let b=πθold(o(k)∣sk)b=\pi_{\theta_{\text{old}}}(o^{(k)}\mid s_k). The banned token has zero mass under Eq. 7, so with advantages fixed, E[ρparA^par+ρcontA^cont]=2∑aπθ(a∣sk)A(sk,a)−∑aπθold(a∣sk) πθ(a∣sk) A(sk,a),\mathbb{E}\big[\rho_{\text{par}}\hat A_{\text{par}}+\rho_{\text{cont}}\hat A_{\text{cont}}\big]=2\sum_a\pi_\theta(a\mid s_k)A(s_k,a)-\sum_a\pi_{\theta_{\text{old}}}(a\mid s_k)\,\pi_\theta(a\mid s_k)\,A(s_k,a), which under-credits high-probability actions. Every post-fork continuation token has likelihood ratio 1−b1-b, but Eq. 8 applies it to one token. A without-replacement treatment (Kool et al., 2020) would weight the continuation by 1−b1-b and the parent, from the fork onward, by bb. Fork states are also chosen in hindsight (and conditioned on failure in Sec. 5.5). That makes this a restart-distribution objective (Kakade & Langford, 2002), not an on-policy one.

Other Issues

  • Typos:
    • Title "Sample Efficient" → "Sample-Efficient".
    • Abstract: "fiip", "soluton", "endows the disagreement" → "induces disagreement", "finer grained".
    • Sec. 1: "focu", "possessing" → "possesses", "line of works … (Foster et al.). while", "decoded token".
    • Sec. 2: "reasoning diverge", "to at", "methods relies", "branch can be created", "tend to affect more on", "Neither of the existed literature", "methods operate".
    • Sec. 3 "Degenerated"; Sec. 4.2 "resume"; Sec. 5.3 "the gain not from"; Sec. 5.4 "groups has"; Sec. 5.5 "a all incorrect group"; Fig. 4 "a same", "taco".
  • Notation:
    • Symbols with two meanings: NN vs. KK; TT is both length and temperature; cc is both the continuation and the Eq. 6 weight; ii indexes trajectories and candidates; kk is both an index and a position.
    • Eq. 3 shows a sequence mean, but the runs use a token mean.
    • Eq. 5 should be r(q,o<k⊕c)r(q,o_{<k}\oplus c).
    • "A" in Table 5 is undefined.
  • Inconsistencies:
    • Eq. 6 vs. App. A ("no baseline").
    • Sec. 4.3 (all runs use the ban) vs. Fig. 6 ("divergence-ban variant").
    • App. A (continuations much worse) vs. Fig. 9c and Table 5 (net ≈ 0).
    • App. C cites a failure mode "Section 5.3 diagnoses" and a "central negative result", neither of which is in the main text.
    • Leftovers from another draft: "Coding port", "reasoning", "bottom row of Figure 3".
  • Overclaims:
    • "No additional trained parameters" (Conclusion).
    • "0.548 vs. 0.506" is the single best cell.
    • "Potential to scale" despite 62% → 46% and a 1.7B wash, which belongs in the main text.
    • The pass@8 gain is half the pass@1 gain at 4B.
    • MBPP+ and HumanEval+ are not competitive programming.
  • Formatting and references:
    • Fig. 2 comes before Fig. 1, and the Table 1–2 captions overlap their rules.
    • Some citations lack years (TreePO, InT).
    • PPO and CoT are cited as RLVR evidence.
    • Check years: o1 system card (arXiv:2412.16720), GRESO (NeurIPS vol. 38 is 2025), Foster et al. (arXiv:2502.12272), Wei et al. (NeurIPS 2022).
    • Stray asterisk in the DeepScaleR author list.
  • Also relevant: Pivotal Token Search (Phi-4 report); Math-Shepherd/OmegaPRM; Setlur et al. (2024); SPEC-RL (arXiv:2509.23232); LATR (arXiv:2510.24302); Yue et al. (2025) on pass@kk; Spurious Rewards (Shao et al., 2025) on Qwen-only evidence.
Questions for authors

Clarifications

  1. Which candidate procedure and KK produced Table 1 and Fig. 3: uniform KK, or App. B's delimiter rule and "skip-header walk"? App. A reports 36–48 candidates.
  2. What does "the only matched pair" (Fig. 10) mean? Which comparisons differ in grading, data order, seed or hyperparameters?
  3. Are reused prefix tokens in the continuation's loss, and how are they counted under token-mean aggregation?
  4. Why does Fork-RL emit ~22% fewer response tokens per step (Table 3)? Please show response lengths for both arms.
  5. What are wall-clock time per step and total GPU-hours for both arms?
  6. How do flips break down by direction and failure type (syntax/runtime error, wrong answer, timeout, truncation)? Banning a token mid-identifier can cheaply buy right→wrong flips, consistent with App. A and Fig. 9c.
  7. Which checkpoints were used (thinking on/off; Fig. 7 says "instruct")? What were the eval temperature, samples per problem, pass@kk estimator and LiveCodeBench window? Was TACO decontaminated against CodeContests?
  8. App. C gives step-250 statistics for a 1.7B run stopped at step 141. Which run is this?

Requested experiments

  1. At least 3 seeds with confidence intervals, and a symmetric tokens-to-threshold metric (first crossing for both arms, on held-out data).
  2. Entropy-based fork selection in the same pipeline, TreeRL, and DAPO dynamic sampling, all at matched decoded tokens.
  3. Tree vs. flat advantages; prefix in vs. out of the loss; a correct-only forking arm; and the non-degenerate group share over training for both methods.
  4. Flip rates by pass-rate bin. Let pp be the pass rate and fc,fwf_c,f_w the flip rates from correct and wrong parents. With independent outcomes, a fork group is degenerate with probability pn(1−fc)n+(1−p)n(1−fw)np^n(1-f_c)^n+(1-p)^n(1-f_w)^n, vs. p2n+(1−p)2np^{2n}+(1-p)^{2n} for GRPO. These are equal when fw=pf_w=p and fc=1−pf_c=1-p, so this shows where forking beats fresh sampling.
  5. The distribution of bb at fork points, the share of clipped fork tokens, and where the head cuts. The fork-token ratio 1−b1-b falls below 1−ϵlow1-\epsilon_{\text{low}} when b>0.2b>0.2, so negative-advantage fork tokens get zero gradient.
  6. Paper needs significant improvement and needs to revised totally. Please write well aligned paper and review the AI written contents.
3 comments
AuthorsOpenReview ↗

Response to Reviewer vG9Y — Part 1/3

Thank you for the detailed technical review.

[W1/E1: statistical reliability] Do the gains hold across seeds, including the small benchmark and candidate-count differences?

The table compares the seed-0 results from Table 1 with the new seed-43 run. Separate columns show pass@1 and pass@8 (%). Avg is the simple average across four benchmarks.

ModelMethodTraining seedMBPP+ pass@1MBPP+ pass@8HumanEval+ pass@1HumanEval+ pass@8Avg pass@1Avg pass@8
Qwen3-4BFORK-RL074.6 ± 0.382.583.0 ± 0.392.157.368.6
Qwen3-4BFORK-RL4374.8 ± 0.382.583.2 ± 0.391.757.568.7
ModelMethodTraining seedCodeContests pass@1CodeContests pass@8LiveCodeBench pass@1LiveCodeBench pass@8Avg pass@1Avg pass@8
Qwen3-4BFORK-RL024.5 ± 0.138.846.9 ± 0.361.057.368.6
Qwen3-4BFORK-RL4324.7 ± 0.139.147.1 ± 0.361.557.568.7

The two seeds give similar results

[W1/E1: symmetric efficiency metric] Does Figure 3 compare the first threshold crossing for both methods?

The original 62%/46% savings use GRPO's fixed 150M-token budget. A symmetric comparison uses when each method first reaches the same reward. We will update Figure 3's arrows and percentages accordingly.

The table below gives approximate crossings read from Figure 3. For 8B, GRPO's peak around 90–100M tokens stays below the reference line. It first crosses the line at around 110M tokens.

ModelReference training rewardFORK-RL crossing (decoded tokens)GRPO first crossing (decoded tokens)Symmetric token savings
Qwen3-4B≈0.515≈58M≈118M≈51%
Qwen3-8B≈0.530≈81M≈110M≈26%

We calculate savings using both methods' first crossings:

\qquad T_M(\tau)=\min\\{T:R_M(T)\geq\tau\\},$$ Here $R_M(T)$ is the training reward shown in Figure 3. The table gives approximate first-crossing estimates read from the figure. # [W1/Q4: sources of token savings] How much comes from prefix reuse, response length, and fewer training steps? **Token savings come from two sources: reaching the same accuracy in fewer training steps, and reusing prefixes.** Prefix reuse avoids decoding the shared part of each parent–continuation pair twice. Response lengths are similar: 1363 tokens for FORK-RL and 1382 for GRPO over steps 140–150, a 1.4% difference. The shares of responses that reach the length limit are also close: | Training interval | Fork | Baseline | | --- | --- | --- | | Step 150 | 10.7% | 10.9% | | Steps 145–155 | 10.2% | 10.0% | | Steps 140–160 | 9.4% | 9.3% | Length-limit rates are similar across the reported intervals near step 150. The next table shows how many training steps each method needs to reach about 55.8% accuracy: | Method | Training step | Accuracy | | --- | --- | --- | | FORK-RL | 70 | 55.8% | | GRPO baseline (G = 8) | 94 | 55.8% | FORK-RL reaches similar accuracy with 25.5% fewer training steps, or about 1.34× step efficiency. Faster learning per step and prefix reuse provide two sources of efficiency. The next section reports decoded-token and compute savings. # [W1/Q5: actual compute and scope of gains] Are the methods compute-matched, and do the gains persist with longer training? **The table compares decoded tokens and estimated actor-training plus decoding compute for Qwen3-4B over steps 0–150.** Both methods cover the same training steps: | Measurement (Qwen3-4B, steps 0–150) | GRPO | FORK-RL | Reduction | | --- | --- | --- | --- | | Actual decoded tokens | 169.3M | 105.6M | 37.6% | | | | | | | Rough actor-training + decoding compute (TFLOP) | 7,514,730 | 6,001,525 | 20.1% | At the same model size, estimated decoding FLOPs scale with decoded tokens. The combined TFLOP values estimate total floating-point operations over the same training steps. Figure 10 also shows a +0.8-point LiveCodeBench gain at the same step endpoint. We will state each result's step/token budget and add the 1.7B result to the main text.
AuthorsOpenReview ↗

Response to Reviewer vG9Y — Part 2/3

[W1/Q1–2,Q8: experimental control] Which runs, candidate rules, and checkpoints support each result?

We will list which run produced each main and appendix result. For each run, we will give the candidate rule and count, score-head normalization, ban setting, grading, data order, seed, and hyperparameters.

Table 1 and Figure 3 sample candidate token positions uniformly at random (§4.2), then use the learned head to choose among them. We will fix Appendix B's delimiter-based and header-skipping descriptions and clearly list the settings for separate diagnostic runs.

We will also explain Figure 10's “only matched pair” wording and which runs produced the appendix results. We will make the descriptions of score normalization, token bans, continuation performance, training steps, and token counting consistent.

[W2/E2: novelty and missing baselines] What is established beyond random cuts and prior branching or dynamic-sampling methods?

At step 150, the new Entropy Fork and TreeRL results are below the Qwen3-4B FORK-RL result in Table 1 on all four benchmarks. TreeRL uses the same rollout configuration as FORK-RL: 4 initial trajectories, one divergence, and at most 2 branches. This provides a fair comparison under the same rollout configuration. Our contribution is a small selector that uses hidden states and learns from outcome flips, within a GRPO pipeline that reuses prefixes.

MethodStepLiveCodeBenchMBPP+HumanEval+CodeContests
Entropy Fork baseline15045.172.782.222.0
TreeRL ()15045.873.182.723.6
FORK-RL (Table 1, 4B reference)15046.974.683.024.5

The table compares benchmark scores at step 150. In the learned-versus-uniform test, training reward is 0.547 versus 0.497, and flip rate is 0.183 versus 0.129 (§5.3). These results support learning where to fork.

[W2/Q3/E3: half-localized credit] Are reused prefixes in the loss, and does explicit credit localization improve over the original objective?

Yes, reused prefixes are included in the continuation's policy loss. The new Localized credit test checks explicit credit localization. At step 150, it gives similar scores, improving two benchmarks and lowering two.

Variant (Qwen3-4B)StepLiveCodeBenchMBPP+HumanEval+CodeContestsAvg
Original FORK-RL (prefix-in; Table 1 reference)15046.974.683.024.557.3
Localized credit15046.675.483.223.957.3
FORK-RL (prefix-out)15045.673.281.523.155.9

Values are evaluation accuracies (%); Avg is the simple average across benchmarks. Prefix-in keeps the continuation's prefix loss. Prefix-out removes that loss but keeps the prefix as context and leaves the parent loss unchanged.

Localized credit gives similar results to the original method. The lower prefix-out scores support keeping the continuation's prefix loss in this experiment. When we average over tokens, shared-prefix tokens enter both trajectory losses but are decoded only once.

Let rˉ=(rP+rC)/2\bar{r}=(r_P+r_C)/2 be the pair's mean reward, and let mm and ss be the group reward mean and standard deviation. We can write the advantages as

AP/C=rˉ−ms±rP−rC2s.A_{P/C}=\frac{\bar{r}-m}{s}\pm\frac{r_P-r_C}{2s}. At the same prefix position, the pairwise terms cancel when added together, but the common group-relative term remains. The difference between the pair's rewards therefore affects only the suffix, even though prefix tokens remain in the loss.

The reward-mix control reaches training reward 0.489, compared with 0.547 for FORK-RL (§5.4). This supports the value of shared-prefix comparisons beyond the reward mix alone.

[W2/E3: failure-only forking] Does failure-only performance contradict the need to fork correct trajectories?

Failure-only forking keeps the training-reward gain: 0.549 compared with 0.547 for full forking at step 100 (§5.5). We also tested held-out accuracy to see whether the methods still perform similarly:

Qwen3-4B evaluation accuracy (%) at step 150:

Forking strategyStepLiveCodeBenchMBPP+HumanEval+CodeContestsAvg
Full forking (Table 1 reference)15046.974.683.024.557.3
Failure-only forking15043.273.681.021.754.9
Correct-only forking15042.273.181.221.254.4

Avg is the simple average across four benchmarks.

Full forking scores higher on all four benchmarks, supporting the inclusion of correct trajectories when optimizing held-out accuracy.

AuthorsOpenReview ↗

Response to Reviewer vG9Y — Part 3/3

[W2/Q6/E4: useful flips] Do forks recover errors or merely break correct programs, and when do they beat fresh sampling?

The observed incorrect→correct ratios around step 150 are:

MethodIncorrect→correct ratio around step 150
FORK-RL14.7%
Random fork10.5%
Standard GRPO15.4%

For FORK-RL over steps 140–150, the breakdown below groups incorrect→correct transitions by the parent's original error type. Counts refer to recovery transitions.

Original error typeRecovery countShare
Syntax / runtime error2815.0%
Wrong answer10355.1%
Timeout84.3%
Truncation4825.7%
Total187100.0%

FORK-RL recovers more errors than random forking, supporting the selector's ability to choose useful recovery positions. These ratios count incorrect→correct changes, while §5.3's overall flip rates count both directions.

Table 5 groups the results into five suffix-distance bins and shows both flip directions. We will extend this analysis by parent pass rate and error type, including syntax/runtime errors, wrong answers, timeouts, and truncation, and examine how recovery changes with fork depth and kept prefix length.

Under the independent-pair model, degenerate-group probabilities are

\qquad P_{\mathrm{fresh}}=p^{2n}+(1-p)^{2n}.$$ This comparison depends on the conditional flip rates $f_w,f_c$. # [W2/E5: divergence ban and gradient objective] Does Eq. 8 fully correct the sampling change, and how often does clipping suppress updates? **FORK-RL uses a surrogate objective under its modified sampling procedure.** Choosing restart states and banning the parent's fork token change the sampling distribution. The table shows Qwen3-4B evaluation accuracy (%) at training step 150: | Method | Step | LiveCodeBench | MBPP+ | HumanEval+ | CodeContests | Avg | | --- | --- | --- | --- | --- | --- | --- | | FORK-RL with ban | 150 | 46.9 | 74.6 | 83.0 | 24.5 | 57.3 | | FORK-RL without ban | 150 | 45.7 | 74.4 | 82.2 | 24.7 | 56.8 | The ablation supports using the ban for Qwen3-4B at step 150. At collection parameters, $\rho_k=1-b$. With $\epsilon_{\mathrm{low}}=0.2$, negative-advantage fork tokens lie on the flat clipped branch when $b>0.2$. The recorded clip ratios over training steps 1–150 are: | Step | Baseline clip | Fork clip | | --- | --- | --- | | 1–50 | 0.1% | 0.2% | | 51–100 | 0.1% | 0.2% | | 101–150 | 0.1% | 0.1% | # [Q7: evaluation protocol] Which checkpoints, decoding settings, and contamination controls were used? **The experiments use Qwen/Qwen3-4B and Qwen/Qwen3-8B in non-thinking mode.** We will use these exact checkpoint names throughout the paper. Evaluation uses a sampling temperature of 0.7 and 8 samples per problem. Training uses temperature 1.0 (§5.1). We will state these settings clearly in the revised paper. # [Other issues/E6: manuscript revision] Will the notation, formatting, references, and overclaims be corrected? **We will make the paper's claims and notation match the verified experiments.** We will fix token-mean Eq. 3 and the full prefix-plus-continuation reward in Eq. 5, use different symbols for candidates, positions, temperature, and weights, and define “A” in Table 5. We will fix overlapping table lines and equation text, figure order and captions, leftover draft text, grammar, and citation details. We will also expand the related work and replace “no additional trained parameters” with a single trainable vector. Results will clearly state benchmark type, pass@1/pass@8, model size, and training budget.
Reviewer_rT2EOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes FORK-RL, a method that forks trajectories during training to address the absence of an advantage signal in GRPO groups whose sampled solutions are all correct or all incorrect. For each problem, it samples a small set of complete trajectories and uses a linear score head applied to the policy model’s hidden states to select a fork point in each trajectory. It then reuses the prefix and generates a new suffix, disallowing the parent trajectory’s next token at the first regenerated position. The policy is trained with correctness rewards, while the score head is rewarded when a continuation changes its parent’s correctness outcome. The authors train Qwen3-4B and Qwen3-8B on TACO and evaluate them on four coding benchmarks. They report higher benchmark accuracy than GRPO and fewer decoded training tokens to reach a target training reward.

Critical strengths

The results show consistent accuracy gains and improved efficiency per decoded token at both model sizes. Table 1 reports improvements over the group-size-matched GRPO baseline on all four coding benchmarks, with average pass@1 increasing by 2.2 percentage points for the 4B model and 2.5 points for the 8B model. In Figure 3, FORK-RL reaches the training reward attained by GRPO at 150 million decoded tokens using approximately 62% fewer tokens for 4B and 46% fewer for 8B.

Critical weaknesses

The paper does not sufficiently establish what drives the gains or when the method is effective. It does not explicitly state whether reused prefix tokens contribute to the continuation’s policy loss (Eq. 3), which makes the claimed credit-assignment effect difficult to assess. Section 5.3 compares the learned head with uniform selection, but there is no direct comparison with entropy-based fork selection, leaving its advantage over this alternative unclear. The motivation also assumes that code prefixes mainly contain reusable setup, such as input parsing and initialization (page 2). If the prefix already commits to an incorrect algorithm or data structure, late forking may preserve the error, whereas earlier forking reduces prefix reuse and the potential token savings. The paper does not analyze how fork positions relate to the locations of such errors.

The manuscript appears unfinished. The horizontal rules in Table 1 (page 7), Table 2 (page 8), and Algorithm 1 (page 5) overlap with text, and the text following Eqs. (1) and (4)–(8) has visibly compressed line spacing. The caption of Figure 6 also refers to a “bottom row” of Figure 3 that does not exist in the current layout. In addition, Section 4.2 describes uniformly sampled candidate positions, whereas Appendix B refers to delimiter-based and header-skipping rules. Although Appendix B states that its diagnostics come from a different run, their relationship to the main experiments is not clearly explained. These issues disrupt reading and make the experimental descriptions difficult to cross-reference.

Questions for authors

Additional validation: With the same candidate pool and decoding budget, does the learned score head outperform entropy-based fork selection? Figure 5 shows nearly identical training rewards for the wrong-only variant and the full method. Do they also achieve similar benchmark accuracy, and how do their decoding costs compare?

Implementation clarification and additional analysis: Are reused prefix tokens excluded from the continuation’s policy loss in Eq. 3? For trajectories that commit to an incorrect algorithm early on, is there evidence that the score head selects an earlier fork point? If so, does earlier forking retain both the accuracy gains and the token savings?

2 comments
AuthorsOpenReview ↗

Response to Reviewer rT2E — Part 1/2

Thank you for recognizing the accuracy and decoded-token efficiency gains.

[Q3: prefix loss and credit assignment] Are reused prefix tokens included in the continuation's loss, and what credit is localized?

Yes, reused prefix tokens are included in the continuation's policy loss. We average the loss over tokens (§5.1), so the shared prefix enters both the parent and continuation losses, even though it is decoded only once. The new Localized credit test checks whether making credit localization explicit changes the results.

Let rˉ=(rP+rC)/2\bar{r}=(r_P+r_C)/2 be the pair's mean reward, and let mm and ss be the group reward mean and standard deviation:

AP/C=rˉ−ms±rP−rC2s.A_{P/C}=\frac{\bar{r}-m}{s}\pm\frac{r_P-r_C}{2s}. At the same prefix position, the opposite pairwise terms cancel when added together. The common group-relative term remains. The difference between the pair's rewards therefore affects only the suffix after the fork; the prefix still receives the group-relative signal.

Qwen3-4B evaluation accuracy (%) at step 150:

MethodLiveCodeBenchMBPP+HumanEval+CodeContestsAvg
Localized credit46.675.483.223.957.3
FORK-RL (prefix-in; Table 1, 4B reference)46.974.683.024.557.3
FORK-RL (prefix-out)45.673.281.523.155.9

Avg is the simple average across four benchmarks. Localized credit gives similar results to the original method. Prefix-out removes the prefix from the continuation loss but keeps it as context and leaves the parent loss unchanged. Its lower scores support keeping the prefix loss in this experiment.

[Q1: selector attribution] Does the learned head outperform entropy-based selection under the same candidates and decoding budget?

At step 150, the new Entropy Fork baseline scores below the Qwen3-4B FORK-RL result in Table 1 on all four benchmarks:

MethodStepLiveCodeBenchMBPP+HumanEval+CodeContests
Entropy Fork baseline15045.172.782.222.0
FORK-RL (Table 1, 4B reference)15046.974.683.024.5

In the learned-versus-uniform test, training reward is 0.547 versus 0.497, and overall flip rate is 0.183 versus 0.129 (§5.3). These results support learned selection over the tested alternatives. Overall flip rates count changes in both directions; the recovery ratios below count incorrect→correct changes.

[Q4–Q5: early algorithmic errors] Can the head fork before an incorrect algorithmic commitment while retaining token savings?

The learnable fork head aims to find where a trajectory goes wrong and fork before the wrong decision. It learns from outcome flips to choose useful restart points. Reusing the prefix saves tokens.

Around step 150, incorrect→correct ratios are 14.7% for FORK-RL, 10.5% for random forking. The gain over random forking supports the head's ability to choose useful recovery positions.With the similar incorrect to correct flip ratio, our fork configuration can save repetitive tokens.

Case study (training step 2, fork offset 527)

The fork at offset 527 is just before the DFS code. It reuses input parsing and tree construction, then replaces the faulty aggregation below.

Original trajectory (incorrect; excerpt)

# ... shared input parsing and tree construction ...
# Fork point: before DFS

def dfs(v, parent):
    # ... count colors in the subtree ...
    for color, freq in cnt.items():
        if freq > max_freq:
            max_freq, max_color = freq, color
        elif freq == max_freq:
            max_color = min(max_color, color)
    result[v] = max_color * max_freq
    # ...

Forked continuation (correct; excerpt)

def dfs(v, parent):
    # ... count colors in the subtree ...
    max_freq = max(color_count.values(), default=0)
    max_colors = [color for color, cnt in color_count.items()
                  if cnt == max_freq]
    result[v] = sum(max_colors)
    # ...

The original keeps only the smallest tied color and multiplies it by its frequency. The continuation correctly sums all tied colors. On the first supplied test, the output changes from 1 2 3 4 to 10 9 3 4. The logged result flips from incorrect to correct, with the setup prefix reused.

AuthorsOpenReview ↗

Response to Reviewer rT2E — Part 2/2

[Q2: failure-only forking] Does failure-only forking match full forking on held-out accuracy and decoding cost?

At step 100, failure-only and full forking have similar training rewards: 0.549 and 0.547 (§5.5). To check whether they also have similar held-out accuracy, we tested the Qwen3-4B methods at step 150:

Qwen3-4B evaluation accuracy (%) at step 150:

Forking strategyStepLiveCodeBenchMBPP+HumanEval+CodeContestsAvg
Full forking (Table 1 reference)15046.974.683.024.557.3
Failure-only forking15043.273.681.021.754.9
Correct-only forking15042.273.181.221.254.4

Avg is the simple average across four benchmarks.

Full forking scores higher on every benchmark at the same training step.

[W2: experimental provenance] Which candidate rules produced the main results and the appendix diagnostics?

The main experiments, including Table 1 and Figure 3, sample candidate token positions uniformly at random (§4.2). The learned head then chooses a fork position from those candidates.

We will update Appendix B to state the main experiments' uniformly sampled candidate rule and list the settings for each separate diagnostic run, including any delimiter-based restart positions or header skipping.

[Clarification: response length] How do response lengths compare with GRPO?

Over steps 140–150, average response length is 1363 tokens for FORK-RL and 1382 for GRPO. The difference is only 19 tokens (1.4%), so response lengths are similar near step 150.

[W2: presentation] Will the overlapping text, table rules, and incorrect figure references be corrected?

We will fix Tables 1–2, Algorithm 1, and spacing after Eqs. (1) and (4)–(8). We will also fix Figure 6's reference to a bottom row that does not exist in Figure 3, and make the main text and appendix match the verified run settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.