acceptodds
Under review as a conference paper at ICLR 2027

Learning Beyond What You Sample: Off-Policy-Aware Cross-Model Trajectory Exchange for RLVR

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) methods such as GRPO rely on successful self-generated trajectories, but finite rollout budgets can produce all-fail groups with no reward-based policy-gradient signal. While additional rollouts improve the chance of success at higher cost, successful trajectories missing from one model's rollouts may already be discovered by another. Indeed, we observe that heterogeneous models often succeed on complementary prompts, creating opportunities for mutual learning without a designated stronger teacher. To exploit this complementarity, we propose **GRAFT** (**G**ated **R**eplacement of **A**nswer-**F**ailed groups with peer **T**rajectories), an off-policy-aware framework that replaces all-fail groups with informative peer groups. GRAFT transfers both successful and unsuccessful peer responses with peer-computed advantages, while controlling cross-model mismatch through sequence-level compatibility weighting and token-level importance-ratio clipping. Across three heterogeneous model pairs and five mathematical reasoning benchmarks, GRAFT consistently improves both models over GRPO with the same per-model rollout budget, gaining 2.1 points on average and up to 4.5 points in model-level average performance. Stored peer trajectories preserve most gains, improving over GRPO by 1.8 points on average without simultaneous co-training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Reasoning Models, RLVR, Co-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6WBvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses all-fail groups in GRPO-style RLVR: when every rollout for a prompt is incorrect, the group-relative advantages vanish and the prompt gives no learning signal. Observing that heterogeneous base models often solve complementary prompts, the authors propose GRAFT, which trains two such models jointly. When the receiver fails all rollouts on a prompt but the peer partially succeeds, the receiver's group is replaced by the peer's full group (successes and failures) with its advantages, and transferred responses are filtered by a likelihood-based compatibility score. On three model pairs (≤3B) and five math benchmarks, GRAFT improves both models over GRPO at the same per-model rollout budget (+2.1 points on average), and stored peer trajectories retain most of the gain.

Critical strengths
  1. Consistent gains without extra rollouts. With the same eight rollouts per model, all six model-level results improve over GRPO, and the gains hold across three seeds (+1.87 on average, Table 4). On Pair 1, the full joint run costs 43.3 GPU-hours, less than a third of GRPO with 32 rollouts (145.1 GPU-hours), at a comparable or better score (Table 5). This directly supports the paper's main practical claim and is the main reason I lean toward acceptance.

  2. Useful controlled comparisons. Under the same prompt selection, removing the compatibility gate or transferring only the peer's successful responses costs SmolLM3 about 8 points, far beyond seed variation, and success-only transfer is even worse than plain GRPO (Table 2). The takeaway that cross-model transfer needs likelihood-based admission and benefits from keeping the peer's failed responses should be useful for off-policy guidance methods more broadly.

  3. Usable without co-training. With trajectories saved from the partner's independent GRPO run, all six results still improve over GRPO (+1.78 on average, about 84% of the online gain; Table 6), so in practice the two models do not have to be trained together.

Critical weaknesses
  1. The evaluation protocol likely inflates the gains. Checkpoints are selected by the best average score on the five test benchmarks, evaluated every five steps (Appendix B), and δ was tuned on Pair 1 with the same benchmarks. The main table and all ablations are single runs, with ± reflecting only repeated evaluation. With the three-seed means in Table 4, the average gain drops from 2.1 to 1.87 and the largest from 4.46 to 3.97, and on SmolLM3 most component ablations differ from full GRAFT by about 2 points, within the run-to-run variation in Table 4 (seed std. 0.85 for GRAFT, 1.48 for GRPO). The direction of the main result seems robust, but the headline numbers, the comparison with GRPO (n=32), and the necessity of individual components are not yet established.

  2. The source of the gains is not isolated. The paper attributes the gains to heterogeneous complementarity and to off-policy-aware control of cross-model mismatch, but the evidence does not separate these explanations. By design (Eq. 8), an admitted response always has weight between 0.8 and 1 when δ = 0.8, so in practice the weighting works mainly as a filter, and only 0.28–0.44% of peer tokens are clipped in Table 9. There is also no same-model, different-seed control: a second run of the same model would also solve some of the receiver's all-fail prompts, so the gains could come from an extra set of attempts on hard prompts rather than from heterogeneity. This weakens the paper's central claim that heterogeneous peers are what make the transfer useful.

  3. The HACPO baseline may be under-tuned. In Table 1, HACPO falls below plain GRPO (n=8) in five of six results, by up to 4.17 points, whereas the HACPO paper reports gains over GRPO in a similar setting (MATH training set, eight rollouts per model). The 4.0-point margin over HACPO is therefore not very informative; the comparison with SGT (+1.5 on average) is the more meaningful one.

Questions for authors
  1. Regarding W1: could you report Tables 1 and 4 using the final checkpoint (or the mean of the last few validation points), or with checkpoints selected on a validation set disjoint from the test benchmarks? This needs no new training. If GRAFT's advantage over GRPO (n=8) holds on Pairs 1 and 2 and does not reverse on Pair 3, I would be willing to raise my score.

  2. Regarding W2: how many prompts are transferred per step in each direction, and what fraction of the transferred responses pass the compatibility gate, separately for correct and incorrect responses?

  3. Also regarding W2: how often does an independent GRPO run of the same model solve the receiver's all-fail prompts? This should be computable from the GRPO seeds in Appendix C, and it would show whether the complementarity in Figure 1 exceeds what a second run of the same model already provides.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 6WBv (1/3)

We thank the reviewer for the careful reading and the concrete suggestions.

W1 & Q1: Evaluation protocol

Following Q1, we selected checkpoints on a held-out validation set disjoint from all test benchmarks (200 DAPO-17k and 100 MATH test problems outside MATH500, deduplicated against all evaluation benchmarks). We had not kept all of the original checkpoints, so we reran the experiments, and the scores also reflect run-to-run variation. Every method uses the same rule (best validation average):

PairModelGRPO n=8GRPO n=16GRPO n=32HACPOSGTGRAFTΔ vs n=8
1SmolLM332.46 ±0.3935.31 ±0.4034.92 ±0.3329.98 ±0.4432.42 ±0.1936.17 ±0.64+3.71
1Qwen331.69 ±0.3832.94 ±0.8132.86 ±0.2626.91 ±0.5631.91 ±0.3833.30 ±0.61+1.61
2OctoThinker21.78 ±0.3322.30 ±0.5423.16 ±0.5221.58 ±0.4722.11 ±0.5523.23 ±0.36+1.45
2Qwen331.69 ±0.3832.94 ±0.8132.86 ±0.2628.10 ±0.7530.45 ±0.4232.97 ±0.21+1.28
3SmolLM332.46 ±0.3935.31 ±0.4034.92 ±0.3325.51 ±0.1532.93 ±0.5635.51 ±0.37+3.05
3OctoThinker21.78 ±0.3322.30 ±0.5423.16 ±0.5221.03 ±0.3121.86 ±0.5722.29 ±0.40+0.51

GRAFT's advantage over GRPO (n=8) holds on Pairs 1 and 2 and does not reverse on Pair 3. It improves in all six blocks (+1.94 on average) and outperforms GRPO (n=32) in five of six. Per-benchmark results are in the revised paper.

Multi-seed results. We did not keep the intermediate checkpoints of the additional seeds in Appendix C, so we could not apply validation-based selection to them. Following the reviewer's other suggestion, we report the mean of the last three evaluation points of each run, which needs no checkpoint selection (mean ± std over three seeds):

PairModelGRPO (n=8)GRAFTΔ
1SmolLM332.12 ± 1.3935.14 ± 1.26+3.02
1Qwen331.62 ± 0.2732.81 ± 0.21+1.19
2OctoThinker21.58 ± 0.2522.79 ± 0.15+1.21
2Qwen331.62 ± 0.2732.13 ± 0.25+0.51
3SmolLM332.12 ± 1.3932.47 ± 0.91+0.35
3OctoThinker21.58 ± 0.2522.43 ± 1.09+0.85

Without any checkpoint selection, GRAFT stays ahead of GRPO (n=8) in all six blocks (+1.19 on average). The gain is smallest on Pair 3, which also has the least complementarity (Limitations). These points come from in-training evaluation (one run of 8 samples per prompt, versus five in Appendix C), so the absolute values differ slightly from Appendix C.

Ablations. We agree that small individual differences should not be over-interpreted. The overall pattern is still informative. All 14 variants in Table 2 lower the score of both models (28 of 28 comparisons), and several effects are large relative to run-to-run variation: w/o compatibility gate (−8.36/−2.51), success-only transfer (−7.84/−3.93), and pooled advantages (−3.99/−2.66). The seed std. cited (0.85/1.48) is for SmolLM3; for Qwen3 it is 0.33 (GRAFT) and 0.16 (GRPO), well below most Qwen3 ablation gaps (e.g., −3.56 without the floor, −1.58 with random prompts). The smaller effects (balancing, token ratio on Qwen3) should be read as suggestive rather than conclusive.

Choice of δ. δ = 0.8 was selected on Pair 1 and fixed before Pairs 2 and 3 were run (Appendix B), so those pairs are held out with respect to δ. GRAFT also improves over GRPO for every tested δ ∈ {0.7, 0.8, 0.9} on both models (SmolLM3: 34.17–37.06 vs. 32.60; Qwen3: 31.90–33.44 vs. 31.20).

AuthorsOpenReview ↗

Response to Reviewer 6WBv (2/3)

W2: Isolating the source of the gains

(a) Role of the compatibility weight. We agree that with δ = 0.8 the score acts mainly as a filter. This is intended: it mainly decides whether a peer trajectory is admitted, and only secondarily how strongly it is weighted. The w/o compatibility gate variant sets w(o) = 1 for every transferred response, removing both the filter and the weighting, and lowers performance by −8.36/−2.51. A hard-filter variant (admit if s > δ, weight 1) reaches 35.90/31.57 vs. 37.06/33.44 for full GRAFT. The remaining gap suggests that bounded weighting may contribute beyond filtering, though we treat this as suggestive rather than conclusive.

For token-level clipping, each rollout batch starts at ρ = 1, so clipping is inactive at first, and the ordering changes how often it becomes active. Peer-last clips 0.28–0.44% of peer tokens versus <0.01% under peer-first, and peer-first lowers performance by −2.45/−1.71. Peer-last also lets on-policy minibatches update the policy first. We see the higher clipping activity as one plausible mechanism behind this difference.

(b) Same-model controls. We compared GRAFT with two variants that keep the all-fail replacement framework but draw replacement trajectories from the same model:

  • Self-Pair: GRAFT between two copies of the same model trained with different seeds. We report the better of the two, which favors this baseline.
  • Self-Source: each model generates 16 rollouts; the first 8 form the primary group and, when all 8 fail, the remaining 8 replace them. This matches the budget of GRPO (n=16).
MethodSmolLM3Qwen3
GRPO (n=8)32.60 ± 0.4931.20 ± 0.27
GRPO (n=16)33.38 ± 0.2532.15 ± 0.73
Self-Pair34.66 ± 0.5831.66 ± 0.49
Self-Source33.95 ± 0.6132.30 ± 0.49
GRAFT (heterogeneous)37.06 ± 0.6733.44 ± 0.23

Both variants improve over GRPO (n=8), so extra attempts on hard prompts have some value. Neither matches GRAFT, which exceeds the stronger variant by 2.40 (SmolLM3) and 1.14 (Qwen3). Self-Source adds only +0.57/+0.15 over budget-matched GRPO (n=16) even though it uses twice GRAFT's rollouts per model. Self-Pair removes architecture and tokenizer differences and still trails heterogeneous exchange. Because the number of eligible replacement groups can vary with the source, these comparisons measure the overall effect of trajectory source within the replacement framework. Per-benchmark results are in the revised appendix.

W3: HACPO baseline

We varied HACPO's main hyperparameters, including a peer clip lower bound of 0.6, an aggressive setting not used in the original paper:

VariantαPeer clip lower boundSmolLM3-3BQwen3-1.7B
HACPO (default)1.00.831.51 ± 0.4327.24 ± 0.30
α = 00.00.828.30 ± 0.3726.25 ± 0.59
α = 33.00.826.58 ± 0.6526.81 ± 0.40
Lower clip bound1.00.634.37 ± 0.2327.39 ± 0.37
GRPO (n = 8)––32.60 ± 0.4931.20 ± 0.27
GRPO (n = 16)––33.38 ± 0.2532.15 ± 0.73
GRAFT (ours)––37.06 ± 0.6733.44 ± 0.23

Only the lowered clip bound helps. It improves HACPO by 2.86 points on SmolLM3, bringing it above GRPO (n=8) and (n=16). On Qwen3, the improvement is only 0.15 points, and HACPO remains below GRPO. GRAFT outperforms the strongest tested HACPO configuration by 2.69 and 6.05 points.

Our setting differs from the HACPO paper in the model combinations and, most of all, in training length. Within the first epoch, HACPO's best validation score exceeds GRPO in 3/6 blocks, but GRPO overtakes it later in training, so HACPO's early gains do not last over our longer runs.

We agree that SGT is the more informative comparison. GRAFT outperforms SGT in all six blocks under both the original and the validation-based selection. Under validation-based selection, SGT's average gain over GRPO (n=8) disappears (−0.03), while GRAFT's is +1.94.

AuthorsOpenReview ↗

Response to Reviewer 6WBv (3/3)

Q2: Transfer volume and gate pass rate by correctness

(a) Transfer volume. The table gives the number of prompts whose group is replaced per step and the fraction of candidate responses that pass the gate (over all steps, and for steps ≤60 vs. ≥120). Balanced selection keeps all ties at the selection boundary, so the two directions can have different counts.

Source → receiverPrompts / stepGate pass rate≤60 → ≥120
Qwen3 → SmolLM32.2858.5%78.6 → 28.5
SmolLM3 → Qwen32.8019.7%42.0 → 0.8
Qwen3 → OctoThinker5.0661.8%71.6 → 53.2
OctoThinker → Qwen34.1257.2%75.0 → 39.5
OctoThinker → SmolLM31.2246.1%66.7 → 20.4
SmolLM3 → OctoThinker2.0230.0%61.2 → 1.1

Admitted responses make up at most 3.1% of the 1,024 responses in a step. Admission rates fall during training, as expected if cross-model mismatch grows.

(b) Pass rate by correctness. SmolLM3↔Qwen3, range over two additional runs with per-response logs. These are separate runs, so the levels differ from (a).

Source → receiverPass rate, correctPass rate, incorrectCorrect share: candidates → admittedAdmitted per replaced prompt (of 8): correct / incorrect
Qwen3 → SmolLM370.5–75.6%59.7–65.7%22–23% → 25%1.3–1.4 / 3.7–4.1
SmolLM3 → Qwen329.4–37.8%33.7–44.0%52–58% → 49–54%1.2–1.8 / 1.3–1.5

The gate does not select by correctness. Pass rates of correct and incorrect responses differ by at most 11 points (in SmolLM3 → Qwen3, incorrect responses pass slightly more often), and the share of correct responses changes by at most 4 points. Admitted responses therefore include both the peer's successes and its failures. This separates GRAFT from success-only transfer, which falls below GRPO in Table 2b.

Q3: Same-model twin-run control

We ran this control on held-out prompts. For each model, we took three independently trained GRPO runs and sampled 8 responses per prompt on 4,500 held-out MATH problems (T = 1.0 as in training, 5 seeds). For prompts on which a run fails all 8, the table shows how often 8 further samples solve them, drawn from the same checkpoint, from a same-model twin run, or from the other model (95% bootstrap CI).

ReceiverAll-fail prompts (mean of 3 runs)+8 own samplesTwin runPeerPeer − twin
SmolLM315.2%20.6 [19.4, 21.9]25.4 [23.9, 27.0]34.9 [32.7, 37.2]+9.5 [+7.7, +11.3]
Qwen313.8%24.2 [22.7, 25.9]25.9 [24.1, 27.6]28.1 [26.2, 30.1]+2.2 [+0.6, +4.0]

A twin run recovers some of the all-fail prompts, so seed variance accounts for part of the effect; however, the peer recovers more in both directions. The gap persists under a stricter test. On prompts where two runs of the same model both fail (0/16), a third run of that model solves 16.5% (SmolLM3) and 15.9% (Qwen3), whereas the peer solves 26.2% and 19.8%.

The same ordering holds in training (W2(b)). In the SmolLM3 Self-Pair, the gate admits about twice as many responses per step as in SmolLM3↔Qwen3 (≈30 vs. ≈15), so transfer volume does not explain the advantage of heterogeneous GRAFT. The held-out control is in the revised paper (Section 6.4 and the appendix).

We would be happy to discuss any remaining concerns.

Reviewer_ft3HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GRAFT, a cross-model trajectory exchange method for RLVR. Two heterogeneous language models are trained simultaneously and exchange rollout groups when one model’s group contains no successful response while the peer model has both successful and unsuccessful responses. GRAFT balances exchanges in both directions and controls cross-model mismatch using bounded trajectory weights, token-level PPO clipping, and peer-last minibatch ordering. Experiments on three model pairs and five mathematical benchmarks show improvements over GRPO.

Critical strengths
  1. The paper gives a clear decomposition of the design into prompt selection and update control. The algorithm, compatibility weighting, source-side advantages, and receiver-side token ratios are described in detail to make the proposed procedure understandable.
  2. The experiments include several useful analyses, including larger-rollout GRPO references, comparisons with cross-model baselines, stored peer trajectories, compute accounting, and component ablations. These experiments provide evidence that the proposed combination can improve over the specific standard GRPO setup used in the paper.
Critical weaknesses
  1. Limited methodological novelty. GRAFT largely combines existing ideas from SGT, HACPO, PPO/GRPO, and importance-weighted off-policy learning. Its main additions are full peer-group replacement, balanced selection, a heuristic compatibility gate, and peer-last ordering, but the paper does not provide a new principled objective or strong evidence that these components are individually necessary.
  2. Incomplete experimental comparison and robustness evidence. The main comparison is against standard GRPO without dynamic sampling. In addition, the main results use one training run and select checkpoints on the same benchmarks used for final evaluation, making the reported gains difficult to assess robustly.
Questions for authors
  1. Can you provide a direct, budget-matched comparison between GRAFT and GRPO with dynamic sampling?
  2. Both all-failure and all-success groups have zero group-relative advantages, yet GRAFT replaces only all-failure receiver groups and leaves all-success groups in the batch. What is the rationale for this asymmetric treatment?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ft3H (1/2)

We thank the reviewer for the feedback and address each point below.

W1: Limited methodological novelty

We show that existing cross-model methods do not reliably beat single-model GRPO, identify what does (which groups to transfer and how to learn from them), and show that cross-model complementarity provides gains beyond additional same-model sampling.

Prior methods do not give consistent gains. At the same rollout budget, HACPO falls below GRPO in 5 of 6 blocks (by up to −4.17) and SGT ranges from −0.53 to +2.28, while GRAFT improves in all six (+0.66 to +4.46) (Table 1). GRAFT stays ahead after additional HACPO tuning. With HACPO's peer clip bound lowered to 0.6, an aggressive setting not used in the original paper, HACPO exceeds GRPO only on SmolLM3 (34.37) and remains below it on Qwen3 (27.39 vs. 31.20). GRAFT is still ahead by 2.69/6.05 (details in our response to Reviewer 6WBv).

Ablations support the design choices (Table 2). All 14 variants lower the score of both models.

  • How to learn: with GRAFT's prompt selection, swapping in a prior update rule drops performance (HACPO −8.09/−5.65, SGT −2.98/−1.55, LUFFY −4.55/−2.03).
  • Which to transfer: with the same amount of transferred data, random prompts (−2.40/−1.58) and random admission (−1.86/−1.09) do worse, so the selection criteria matter beyond data volume. Removing balancing, which recovers SGT's selection, costs −2.00/−0.48.
  • Full-group transfer: pooled groups give −3.99/−2.66 and success-only transfer −7.84/−3.93.
  • Mismatch control: the compatibility gate (−8.36/−2.51) and floor (−1.89/−3.56) have the largest effects.

The gain comes from heterogeneity (new, Section 6.4). The same framework with same-model sources does not reproduce GRAFT's gain. Two copies of the same model (Self-Pair) reach 34.66/31.66 and replacement from the model's own extra rollouts (Self-Source) 33.95/32.30, versus 37.06/33.44 for GRAFT (SmolLM3/Qwen3). On held-out prompts that a GRPO run fails 8/8, an independently trained run of the same model solves 25.4%/25.9%, while the peer model solves 34.9%/28.1%. These results show that GRAFT exploits cross-model complementarity to achieve gains beyond those obtained by additional same-model sampling.

Design rationale. Eq. (5) separates two sources of discrepancy: the receiver's own policy change and its mismatch with the peer. GRAFT handles the former with token-level clipping and the latter with a bounded, length-normalized sequence-level compatibility score that is comparable across tokenizations. Peer-last ordering follows from the same distinction: on a new batch ρ=1, so clipping cannot restrain peer gradients until self-generated minibatches have moved the policy. Appendix H shows that peer-last gives the highest clipping rate on peer tokens. Each component addresses a specific failure mode.

AuthorsOpenReview ↗

Response to Reviewer ft3H (2/2)

W2 & Q1: Experimental comparison and robustness

Dynamic sampling. We compare against GRPO (n=8) with dynamic sampling (DS) in Fig. 3a and Appendix D, and have additionally run GRPO+DS for OctoThinker. The DS results for all six blocks are now in the revised paper (average scores):

PairModelGRPO n=8GRPO n=16GRPO n=32GRPO n=8 + DSHACPOSGTGRAFT
1SmolLM332.6033.3835.7134.9931.5134.8837.06
1Qwen331.2032.1532.4332.5927.2431.2433.44
2OctoThinker21.1822.4023.3921.9220.8921.9323.47
2Qwen331.2032.1532.4332.5928.4231.0032.78
3SmolLM332.6033.3835.7134.9928.4332.0733.26
3OctoThinker21.1822.4023.3921.9221.8922.5122.60
Bold: best among methods with n=8 rollouts per model

Both methods train for the same number of steps, but DS gets a larger sampling budget: it samples rollouts for 3× as many prompts per step and keeps groups with non-zero advantage variance. On Pair 1, the two checkpoints cost 50.8 GPU-hours with DS and 40.9 with GRAFT (Appendix D). GRAFT still outperforms DS in five of six blocks (+0.60 on average). The exception is SmolLM3 in Pair 3, which is also the block where GRAFT's gain over GRPO is smallest.

The two methods address different limitations. DS filters out uninformative groups, but the receiver still has to generate a success within its own sampling budget. GRAFT supplies successful peer trajectories for prompts where the receiver's group has none.

Multiple training runs. Appendix C repeats GRPO (n=8) and GRAFT over three independent training runs; GRAFT has the higher mean in all six blocks.

Checkpoint selection. To avoid selecting on the test set, we built a held-out validation set (200 DAPO-17k and 100 MATH test problems outside MATH500, deduplicated against all evaluation benchmarks). We had not kept all of the original checkpoints, so we reran the experiments. For every method, we select the checkpoint with the best validation average and evaluate it on the test benchmarks:

PairModelGRPO n=8GRPO n=16GRPO n=32HACPOSGTGRAFTΔ vs n=8
1SmolLM332.46 ±0.3935.31 ±0.4034.92 ±0.3329.98 ±0.4432.42 ±0.1936.17 ±0.64+3.71
1Qwen331.69 ±0.3832.94 ±0.8132.86 ±0.2626.91 ±0.5631.91 ±0.3833.30 ±0.61+1.61
2OctoThinker21.78 ±0.3322.30 ±0.5423.16 ±0.5221.58 ±0.4722.11 ±0.5523.23 ±0.36+1.45
2Qwen331.69 ±0.3832.94 ±0.8132.86 ±0.2628.10 ±0.7530.45 ±0.4232.97 ±0.21+1.28
3SmolLM332.46 ±0.3935.31 ±0.4034.92 ±0.3325.51 ±0.1532.93 ±0.5635.51 ±0.37+3.05
3OctoThinker21.78 ±0.3322.30 ±0.5423.16 ±0.5221.03 ±0.3121.86 ±0.5722.29 ±0.40+0.51

GRAFT outperforms all budget-matched (n=8) baselines in all six blocks (+1.94 on average over GRPO) and GRPO with 4× rollouts in five of six. Per-benchmark results are in the revised appendix.

Q2: Why replace only all-fail groups?

GRAFT supplies successful trajectories where the receiver finds none. It does not try to replace every zero-advantage group, and the two cases are missing different things. An all-fail group has no success in 8 rollouts, and the peer group provides one. An all-success group already has verified on-policy successes, and its gradient is zero only because it has no failure to contrast them with. Replacing it with a mixed peer group would add off-policy signal on a prompt the receiver already solves. The benefit is less clear, and the replacement would introduce cross-model mismatch and displace on-policy data.

Scale is a second reason. In our setting, all-success groups are 2.6–5× more common than all-fail groups (Pair 1: 49.3%/40.4% vs. 9.9%/15.4% of training prompts for SmolLM3/Qwen3). Replacing them as well could substantially raise the share of off-policy data, mostly on prompts the receiver already solves (the exact rate would depend on peer-group eligibility and balancing). We therefore replace only all-fail groups, where peer successes fill a gap we can observe in the receiver's rollouts.

We would be happy to discuss any remaining concerns.

AuthorsOpenReview ↗

Gentle Reminder to Reviewer ft3H

Dear Reviewer ft3H,

Thank you again for your review. Since the discussion period ends soon, here is a short summary of our response.

  • Dynamic sampling. We now report GRPO+DS results for all six blocks. GRAFT beats DS in five of six blocks and uses fewer GPU-hours.
  • Robustness. When we select checkpoints on a held-out validation set that is disjoint from the test benchmarks, GRAFT beats all budget-matched baselines in all six blocks (+1.94 over GRPO n=8). It also beats GRPO with 4× rollouts in five of six blocks.
  • Novelty. We show that prior cross-model methods do not reliably beat GRPO, and that all 14 ablations lower both models' scores. New same-model controls show that the gains come from cross-model heterogeneity, not from extra sampling.
  • All-fail vs. all-success. We explained why we replace only all-fail groups. All-success groups already contain on-policy successes and are 2.6× to 5× more frequent, so replacing them would add much more off-policy data.

We would appreciate it if you could let us know whether this addresses your concerns.

Reviewer_qJbdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the all-fair groups problem in RLVR, when all n rollouts for a prompt fail, the relative advantages within the group are all zero, so that prompt produces no learning signal. The authors observe that prompts solvable by heterogeneous base models are complementary that among the prompts that Qwen3-1.7B gets entirely wrong, SmolLM3-3B can solve 47.9%, and this complementarity persists throughout the entire training process. When an all-fail groups exist, the authors swap in the peer model’s entire set of responses and reuse the peer’s advantages. The experiments all consider relatively small-parameter models, and the results appear to show improvements. However, whether this method remains effective when extended to larger-parameter models and larger-scale training still needs discussion, and the issue of distribution mismatch between trajectories generated by different models also requires further discussion.

Critical strengths
  • The problem is clearly defined and is a key issue that needs to be addressed in RLVR. The complementarity measurement (Fig. 1) is simple and convincing.
  • The method is simple and practical. GRAFT does not require additional rollouts, does not require specifying a stronger teacher, and does not require a shared vocabulary. The variant that stores trajectories (§6.2) is especially practical.
  • In terms of experimental results, improvements are observed across all settings.
Critical weaknesses

My concerns are mainly in two areas:

  1. There is a lack of deeper exploration of the distribution mismatch issue. For example, a simple experiment could examine whether the previous all-fail groups can roll out correct solutions after training. In addition, how large is the impact of distribution mismatch? Under the condition that training signals are controlled for, how much does performance differ when using one’s own rollouts versus peers’ rollouts? This should perhaps be one of the key parts discussed in the paper.

  2. There is a lack of validation on larger-scale parameter models and training. The current experimental conclusions, including the ablation results, seem to compare the best results selected after multiple evals, and I have some concerns about whether their effectiveness can generalize.

Questions for authors

NA

3 comments
AuthorsOpenReview ↗

Response to Reviewer qJbd (1/2)

We thank the reviewer for the constructive feedback and address each point below.

W1: Deeper exploration of distribution mismatch

We ran both analyses the reviewer suggested and also measured how much mismatch affects performance.

(a) Do previously all-fail prompts become solvable? For prompts on which all 8 of a model's rollouts failed at epoch t, we measured how often at least one of its 8 rollouts at epoch t+1 was correct. The GRPO column is the rescue rate without a peer.

DirectionGRAFTGRPOΔ
Qwen3 → SmolLM363.358.2+5.1
SmolLM3 → Qwen351.249.6+1.6

The reported checkpoints also leave fewer all-fail prompts on held-out data (the 4,500 MATH test problems outside MATH-500, disjoint from the training and evaluation sets; 8 samples per problem; single reported checkpoint per method):

ModelBaseGRPOGRAFT
SmolLM325.3%16.6%11.8%
Qwen326.4%14.4%13.3%

(b) Own vs. peer rollouts. Two variants keep the same all-fail replacement framework but draw replacement trajectories from the same model:

  • Self-Pair: GRAFT between two copies of the same model. We report the better of the two, which favors this baseline.
  • Self-Source: 16 rollouts per prompt; the first 8 form the primary group and, when all 8 fail, the remaining 8 replace them.
MethodSmolLM3Qwen3
GRPO (n=8)32.60 ±0.4931.20 ±0.27
GRPO (n=16)33.38 ±0.2532.15 ±0.73
Self-Pair34.66 ±0.5831.66 ±0.49
Self-Source33.95 ±0.6132.30 ±0.49
GRAFT (cross-model)37.06 ±0.6733.44 ±0.23

Both variants improve over GRPO (n=8), but neither matches GRAFT, which exceeds the stronger variant by 2.40 (SmolLM3) and 1.14 (Qwen3). Self-Source adds only +0.57/+0.15 over budget-matched GRPO (n=16). Because the number and composition of eligible replacement groups can vary with the source, these comparisons measure the overall effect of trajectory source within the replacement framework. Per-benchmark results are in the revised appendix.

(c) How large is the impact of mismatch?

  • Uncontrolled mismatch is harmful. Admitting every peer response with weight 1 (w/o compatibility gate, Table 2a) lowers performance by −8.36/−2.51, and both models fall below GRPO (n=8).
  • Mismatch grows during training. The fraction of peer responses passing the gate falls from 78.6% to 28.5% (Qwen3 → SmolLM3) and from 42.0% to 0.8% (SmolLM3 → Qwen3) between steps ≤60 and ≥120, so GRAFT transfers less as the two policies diverge.
  • Lower mismatch alone is not sufficient. In Self-Pair, the gate admits about twice as many responses per step as in the cross-model pair (≈30 vs. ≈15), yet Self-Pair trails GRAFT.

These results show that controlling mismatch enables GRAFT to benefit from complementary peer trajectories beyond the gains from additional self-sampling.

AuthorsOpenReview ↗

Response to Reviewer qJbd (2/2)

W2: Larger models and checkpoint selection

Larger models. We applied GRAFT to a new heterogeneous pair of larger base models, Olmo-3-1025-7B and Qwen3-4B-Base. All components and hyperparameters, including δ = 0.8, are the same as in the main experiments, with no additional tuning. δ was chosen on Pair 1 and fixed before Pairs 2 and 3 were run, so this pair tests the same configuration on models that played no part in selecting it.

Olmo-3-1025-7B

MethodMATH500AIME2024AIME2025AMC23MinervaAvg
GRPO (n=8)83.92 ±0.2423.08 ±1.1321.08 ±1.4663.31 ±1.4834.91 ±0.6545.26 ±0.42
GRPO (n=16)85.50 ±0.1424.42 ±1.1323.17 ±0.7065.12 ±0.9335.58 ±0.4446.76 ±0.45
GRAFT w/ Qwen3-4B85.03 ±0.1627.58 ±1.3924.25 ±0.7568.19 ±0.8735.68 ±0.2848.15 ±0.42

Qwen3-4B-Base

MethodMATH500AIME2024AIME2025AMC23MinervaAvg
GRPO (n=8)83.64 ±0.4020.33 ±1.1617.08 ±1.1859.38 ±1.8839.47 ±0.4643.98 ±0.32
GRPO (n=16)83.39 ±0.3220.33 ±0.9918.83 ±1.3060.12 ±1.6640.35 ±0.3944.61 ±0.50
GRAFT w/ Olmo-3-7B83.94 ±0.2721.00 ±0.9118.58 ±0.4864.94 ±0.9740.93 ±0.5645.88 ±0.30

For both models, GRAFT improves over GRPO (n=8) on every benchmark and has a higher average than GRPO (n=16), which uses twice the rollout budget.

Checkpoint selection. All methods use the same protocol: evaluation every five steps and selection by the five-benchmark average. To check whether the gains depend on this selection, we ran two analyses without it:

  • Validation-based selection: checkpoints chosen on a held-out set (200 DAPO-17k and 100 MATH test problems outside MATH500, deduplicated against all evaluation benchmarks), then evaluated on the test benchmarks. Since not all original checkpoints were retained, these are new training runs.
  • No selection: the mean of the last three evaluation points of each run, over three independent training runs (mean ± std).
PairModelVal-selected: GRPO (n=8)Val-selected: GRAFTLast-3: GRPO (n=8)Last-3: GRAFT
1SmolLM332.46 ±0.3936.17 ±0.6432.12 ±1.3935.14 ±1.26
1Qwen331.69 ±0.3833.30 ±0.6131.62 ±0.2732.81 ±0.21
2OctoThinker21.78 ±0.3323.23 ±0.3621.58 ±0.2522.79 ±0.15
2Qwen331.69 ±0.3832.97 ±0.2131.62 ±0.2732.13 ±0.25
3SmolLM332.46 ±0.3935.51 ±0.3732.12 ±1.3932.47 ±0.91
3OctoThinker21.78 ±0.3322.29 ±0.4021.58 ±0.2522.43 ±1.09

Under both protocols, GRAFT is ahead of GRPO (n=8) in all six blocks (+1.94 and +1.19 on average). Under validation-based selection, it also outperforms HACPO and SGT in all six blocks and GRPO (n=32) in five of six (full table in the revised paper). Figure 4 shows that GRAFT's advantage lasts through much of training instead of appearing at a single peak, and in Appendix C GRAFT has the higher mean over three independent runs in all six blocks.

Ablations. All 14 variants in Table 2 lower the score of both models (28 of 28 comparisons), and the main effects are large relative to run-to-run variation (seed std. 0.85–1.48 on SmolLM3, 0.16–0.33 on Qwen3): w/o compatibility gate (−8.36/−2.51), success-only transfer (−7.84/−3.93), and pooled advantages (−3.99/−2.66).

The new experiments and the validation-based selection results are in the revised paper. We would be happy to discuss any remaining concerns.

AuthorsOpenReview ↗

Gentle Reminder to Reviewer qJbd

Dear Reviewer qJbd,

Thank you again for your review. Since the discussion period ends soon, here is a short summary of how we addressed your two main concerns.

  • Distribution mismatch. We added the analyses you suggested. GRAFT rescues more previously all-fail prompts than GRPO and leaves fewer all-fail prompts on held-out data. The same-model controls (Self-Pair, Self-Source) show that peer trajectories help beyond self-generated ones (+2.40/+1.14 over the stronger variant). We also measured how much uncontrolled mismatch hurts performance (−8.36/−2.51 without the gate).
  • Scale and checkpoint selection. On a new, larger pair (Olmo-3-7B / Qwen3-4B) with no retuning, GRAFT beats GRPO (n=8) on every benchmark and beats GRPO (n=16) on average. The gains also hold under validation-based selection and with no selection at all (last-3 mean over three runs), across all six blocks.

We would be grateful to know whether these results address your concerns, and we are happy to clarify anything further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of additional experiments

Below are the experiments we added during the rebuttal and the response where each one appears:

  • Larger models: a new Olmo-3-7B / Qwen3-4B pair, run without retuning (response to qJbd, 2/2).
  • Checkpoint selection: validation-based selection for all methods, plus selection-free (last-3) results over three seeds (responses to ft3H,2/2 and 6WBv, 1/3).
  • Same-model controls: Self-Pair and Self-Source (responses to qJbd, 1/2 and 6WBv, 2/3), and a held-out twin-run analysis (response to 6WBv,3/3).
  • Dynamic sampling: GRPO+DS for all six blocks (response to ft3H, 2/2).
  • HACPO: more hyperparameter tuning (response to 6WBv, 2/3).
  • Transfer statistics: all-fail rescue rates (response to qJbd, 1/2), plus transfer volume and gate pass rates broken down by correctness(response to 6WBv, 3/3).

We thank the reviewers for their time and feedback, which helped us improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.