- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses all-fail groups in GRPO-style RLVR: when every rollout for a prompt is incorrect, the group-relative advantages vanish and the prompt gives no learning signal. Observing that heterogeneous base models often solve complementary prompts, the authors propose GRAFT, which trains two such models jointly. When the receiver fails all rollouts on a prompt but the peer partially succeeds, the receiver's group is replaced by the peer's full group (successes and failures) with its advantages, and transferred responses are filtered by a likelihood-based compatibility score. On three model pairs (≤3B) and five math benchmarks, GRAFT improves both models over GRPO at the same per-model rollout budget (+2.1 points on average), and stored peer trajectories retain most of the gain.
Critical strengths
-
Consistent gains without extra rollouts. With the same eight rollouts per model, all six model-level results improve over GRPO, and the gains hold across three seeds (+1.87 on average, Table 4). On Pair 1, the full joint run costs 43.3 GPU-hours, less than a third of GRPO with 32 rollouts (145.1 GPU-hours), at a comparable or better score (Table 5). This directly supports the paper's main practical claim and is the main reason I lean toward acceptance.
-
Useful controlled comparisons. Under the same prompt selection, removing the compatibility gate or transferring only the peer's successful responses costs SmolLM3 about 8 points, far beyond seed variation, and success-only transfer is even worse than plain GRPO (Table 2). The takeaway that cross-model transfer needs likelihood-based admission and benefits from keeping the peer's failed responses should be useful for off-policy guidance methods more broadly.
-
Usable without co-training. With trajectories saved from the partner's independent GRPO run, all six results still improve over GRPO (+1.78 on average, about 84% of the online gain; Table 6), so in practice the two models do not have to be trained together.
Critical weaknesses
-
The evaluation protocol likely inflates the gains. Checkpoints are selected by the best average score on the five test benchmarks, evaluated every five steps (Appendix B), and δ was tuned on Pair 1 with the same benchmarks. The main table and all ablations are single runs, with ± reflecting only repeated evaluation. With the three-seed means in Table 4, the average gain drops from 2.1 to 1.87 and the largest from 4.46 to 3.97, and on SmolLM3 most component ablations differ from full GRAFT by about 2 points, within the run-to-run variation in Table 4 (seed std. 0.85 for GRAFT, 1.48 for GRPO). The direction of the main result seems robust, but the headline numbers, the comparison with GRPO (n=32), and the necessity of individual components are not yet established.
-
The source of the gains is not isolated. The paper attributes the gains to heterogeneous complementarity and to off-policy-aware control of cross-model mismatch, but the evidence does not separate these explanations. By design (Eq. 8), an admitted response always has weight between 0.8 and 1 when δ = 0.8, so in practice the weighting works mainly as a filter, and only 0.28–0.44% of peer tokens are clipped in Table 9. There is also no same-model, different-seed control: a second run of the same model would also solve some of the receiver's all-fail prompts, so the gains could come from an extra set of attempts on hard prompts rather than from heterogeneity. This weakens the paper's central claim that heterogeneous peers are what make the transfer useful.
-
The HACPO baseline may be under-tuned. In Table 1, HACPO falls below plain GRPO (n=8) in five of six results, by up to 4.17 points, whereas the HACPO paper reports gains over GRPO in a similar setting (MATH training set, eight rollouts per model). The 4.0-point margin over HACPO is therefore not very informative; the comparison with SGT (+1.5 on average) is the more meaningful one.
Questions for authors
-
Regarding W1: could you report Tables 1 and 4 using the final checkpoint (or the mean of the last few validation points), or with checkpoints selected on a validation set disjoint from the test benchmarks? This needs no new training. If GRAFT's advantage over GRPO (n=8) holds on Pairs 1 and 2 and does not reverse on Pair 3, I would be willing to raise my score.
-
Regarding W2: how many prompts are transferred per step in each direction, and what fraction of the transferred responses pass the compatibility gate, separately for correct and incorrect responses?
-
Also regarding W2: how often does an independent GRPO run of the same model solve the receiver's all-fail prompts? This should be computable from the GRPO seeds in Appendix C, and it would show whether the complementarity in Figure 1 exceeds what a second run of the same model already provides.
6 comments
Response to Reviewer 6WBv (2/3)
W2: Isolating the source of the gains
(a) Role of the compatibility weight. We agree that with δ = 0.8 the score acts mainly as a filter. This is intended: it mainly decides whether a peer trajectory is admitted, and only secondarily how strongly it is weighted. The w/o compatibility gate variant sets w(o) = 1 for every transferred response, removing both the filter and the weighting, and lowers performance by −8.36/−2.51. A hard-filter variant (admit if s > δ, weight 1) reaches 35.90/31.57 vs. 37.06/33.44 for full GRAFT. The remaining gap suggests that bounded weighting may contribute beyond filtering, though we treat this as suggestive rather than conclusive.
For token-level clipping, each rollout batch starts at ρ = 1, so clipping is inactive at first, and the ordering changes how often it becomes active. Peer-last clips 0.28–0.44% of peer tokens versus <0.01% under peer-first, and peer-first lowers performance by −2.45/−1.71. Peer-last also lets on-policy minibatches update the policy first. We see the higher clipping activity as one plausible mechanism behind this difference.
(b) Same-model controls. We compared GRAFT with two variants that keep the all-fail replacement framework but draw replacement trajectories from the same model:
- Self-Pair: GRAFT between two copies of the same model trained with different seeds. We report the better of the two, which favors this baseline.
- Self-Source: each model generates 16 rollouts; the first 8 form the primary group and, when all 8 fail, the remaining 8 replace them. This matches the budget of GRPO (n=16).
| Method | SmolLM3 | Qwen3 |
|---|---|---|
| GRPO (n=8) | 32.60 ± 0.49 | 31.20 ± 0.27 |
| GRPO (n=16) | 33.38 ± 0.25 | 32.15 ± 0.73 |
| Self-Pair | 34.66 ± 0.58 | 31.66 ± 0.49 |
| Self-Source | 33.95 ± 0.61 | 32.30 ± 0.49 |
| GRAFT (heterogeneous) | 37.06 ± 0.67 | 33.44 ± 0.23 |
Both variants improve over GRPO (n=8), so extra attempts on hard prompts have some value. Neither matches GRAFT, which exceeds the stronger variant by 2.40 (SmolLM3) and 1.14 (Qwen3). Self-Source adds only +0.57/+0.15 over budget-matched GRPO (n=16) even though it uses twice GRAFT's rollouts per model. Self-Pair removes architecture and tokenizer differences and still trails heterogeneous exchange. Because the number of eligible replacement groups can vary with the source, these comparisons measure the overall effect of trajectory source within the replacement framework. Per-benchmark results are in the revised appendix.
W3: HACPO baseline
We varied HACPO's main hyperparameters, including a peer clip lower bound of 0.6, an aggressive setting not used in the original paper:
| Variant | α | Peer clip lower bound | SmolLM3-3B | Qwen3-1.7B |
|---|---|---|---|---|
| HACPO (default) | 1.0 | 0.8 | 31.51 ± 0.43 | 27.24 ± 0.30 |
| α = 0 | 0.0 | 0.8 | 28.30 ± 0.37 | 26.25 ± 0.59 |
| α = 3 | 3.0 | 0.8 | 26.58 ± 0.65 | 26.81 ± 0.40 |
| Lower clip bound | 1.0 | 0.6 | 34.37 ± 0.23 | 27.39 ± 0.37 |
| GRPO (n = 8) | – | – | 32.60 ± 0.49 | 31.20 ± 0.27 |
| GRPO (n = 16) | – | – | 33.38 ± 0.25 | 32.15 ± 0.73 |
| GRAFT (ours) | – | – | 37.06 ± 0.67 | 33.44 ± 0.23 |
Only the lowered clip bound helps. It improves HACPO by 2.86 points on SmolLM3, bringing it above GRPO (n=8) and (n=16). On Qwen3, the improvement is only 0.15 points, and HACPO remains below GRPO. GRAFT outperforms the strongest tested HACPO configuration by 2.69 and 6.05 points.
Our setting differs from the HACPO paper in the model combinations and, most of all, in training length. Within the first epoch, HACPO's best validation score exceeds GRPO in 3/6 blocks, but GRPO overtakes it later in training, so HACPO's early gains do not last over our longer runs.
We agree that SGT is the more informative comparison. GRAFT outperforms SGT in all six blocks under both the original and the validation-based selection. Under validation-based selection, SGT's average gain over GRPO (n=8) disappears (−0.03), while GRAFT's is +1.94.
Response to Reviewer 6WBv (3/3)
Q2: Transfer volume and gate pass rate by correctness
(a) Transfer volume. The table gives the number of prompts whose group is replaced per step and the fraction of candidate responses that pass the gate (over all steps, and for steps ≤60 vs. ≥120). Balanced selection keeps all ties at the selection boundary, so the two directions can have different counts.
| Source → receiver | Prompts / step | Gate pass rate | ≤60 → ≥120 |
|---|---|---|---|
| Qwen3 → SmolLM3 | 2.28 | 58.5% | 78.6 → 28.5 |
| SmolLM3 → Qwen3 | 2.80 | 19.7% | 42.0 → 0.8 |
| Qwen3 → OctoThinker | 5.06 | 61.8% | 71.6 → 53.2 |
| OctoThinker → Qwen3 | 4.12 | 57.2% | 75.0 → 39.5 |
| OctoThinker → SmolLM3 | 1.22 | 46.1% | 66.7 → 20.4 |
| SmolLM3 → OctoThinker | 2.02 | 30.0% | 61.2 → 1.1 |
Admitted responses make up at most 3.1% of the 1,024 responses in a step. Admission rates fall during training, as expected if cross-model mismatch grows.
(b) Pass rate by correctness. SmolLM3↔Qwen3, range over two additional runs with per-response logs. These are separate runs, so the levels differ from (a).
| Source → receiver | Pass rate, correct | Pass rate, incorrect | Correct share: candidates → admitted | Admitted per replaced prompt (of 8): correct / incorrect |
|---|---|---|---|---|
| Qwen3 → SmolLM3 | 70.5–75.6% | 59.7–65.7% | 22–23% → 25% | 1.3–1.4 / 3.7–4.1 |
| SmolLM3 → Qwen3 | 29.4–37.8% | 33.7–44.0% | 52–58% → 49–54% | 1.2–1.8 / 1.3–1.5 |
The gate does not select by correctness. Pass rates of correct and incorrect responses differ by at most 11 points (in SmolLM3 → Qwen3, incorrect responses pass slightly more often), and the share of correct responses changes by at most 4 points. Admitted responses therefore include both the peer's successes and its failures. This separates GRAFT from success-only transfer, which falls below GRPO in Table 2b.
Q3: Same-model twin-run control
We ran this control on held-out prompts. For each model, we took three independently trained GRPO runs and sampled 8 responses per prompt on 4,500 held-out MATH problems (T = 1.0 as in training, 5 seeds). For prompts on which a run fails all 8, the table shows how often 8 further samples solve them, drawn from the same checkpoint, from a same-model twin run, or from the other model (95% bootstrap CI).
| Receiver | All-fail prompts (mean of 3 runs) | +8 own samples | Twin run | Peer | Peer − twin |
|---|---|---|---|---|---|
| SmolLM3 | 15.2% | 20.6 [19.4, 21.9] | 25.4 [23.9, 27.0] | 34.9 [32.7, 37.2] | +9.5 [+7.7, +11.3] |
| Qwen3 | 13.8% | 24.2 [22.7, 25.9] | 25.9 [24.1, 27.6] | 28.1 [26.2, 30.1] | +2.2 [+0.6, +4.0] |
A twin run recovers some of the all-fail prompts, so seed variance accounts for part of the effect; however, the peer recovers more in both directions. The gap persists under a stricter test. On prompts where two runs of the same model both fail (0/16), a third run of that model solves 16.5% (SmolLM3) and 15.9% (Qwen3), whereas the peer solves 26.2% and 19.8%.
The same ordering holds in training (W2(b)). In the SmolLM3 Self-Pair, the gate admits about twice as many responses per step as in SmolLM3↔Qwen3 (≈30 vs. ≈15), so transfer volume does not explain the advantage of heterogeneous GRAFT. The held-out control is in the revised paper (Section 6.4 and the appendix).
We would be happy to discuss any remaining concerns.
Response to Reviewer 6WBv (1/3)
We thank the reviewer for the careful reading and the concrete suggestions.
W1 & Q1: Evaluation protocol
Following Q1, we selected checkpoints on a held-out validation set disjoint from all test benchmarks (200 DAPO-17k and 100 MATH test problems outside MATH500, deduplicated against all evaluation benchmarks). We had not kept all of the original checkpoints, so we reran the experiments, and the scores also reflect run-to-run variation. Every method uses the same rule (best validation average):
GRAFT's advantage over GRPO (n=8) holds on Pairs 1 and 2 and does not reverse on Pair 3. It improves in all six blocks (+1.94 on average) and outperforms GRPO (n=32) in five of six. Per-benchmark results are in the revised paper.
Multi-seed results. We did not keep the intermediate checkpoints of the additional seeds in Appendix C, so we could not apply validation-based selection to them. Following the reviewer's other suggestion, we report the mean of the last three evaluation points of each run, which needs no checkpoint selection (mean ± std over three seeds):
Without any checkpoint selection, GRAFT stays ahead of GRPO (n=8) in all six blocks (+1.19 on average). The gain is smallest on Pair 3, which also has the least complementarity (Limitations). These points come from in-training evaluation (one run of 8 samples per prompt, versus five in Appendix C), so the absolute values differ slightly from Appendix C.
Ablations. We agree that small individual differences should not be over-interpreted. The overall pattern is still informative. All 14 variants in Table 2 lower the score of both models (28 of 28 comparisons), and several effects are large relative to run-to-run variation: w/o compatibility gate (−8.36/−2.51), success-only transfer (−7.84/−3.93), and pooled advantages (−3.99/−2.66). The seed std. cited (0.85/1.48) is for SmolLM3; for Qwen3 it is 0.33 (GRAFT) and 0.16 (GRPO), well below most Qwen3 ablation gaps (e.g., −3.56 without the floor, −1.58 with random prompts). The smaller effects (balancing, token ratio on Qwen3) should be read as suggestive rather than conclusive.
Choice of δ. δ = 0.8 was selected on Pair 1 and fixed before Pairs 2 and 3 were run (Appendix B), so those pairs are held out with respect to δ. GRAFT also improves over GRPO for every tested δ ∈ {0.7, 0.8, 0.9} on both models (SmolLM3: 34.17–37.06 vs. 32.60; Qwen3: 31.90–33.44 vs. 31.20).