acceptodds
Under review as a conference paper at ICLR 2027

HitchRL: Non-Blocking Teacher Supervision on Idle GPUs in Asynchronous RL

Abstract

Reinforcement learning (RL) with verifiable rewards improves the reasoning of large language models, and group-based methods such as GRPO learn from reward differences among responses to the same prompt. A group whose responses all fail therefore yields no signal, and such groups make up 36.6% of those used for updates in a teacher-free asynchronous RL run on a 1.7B model. A stronger, frozen teacher can still guide them, since it evaluates the student's own responses token by token and gives a direction even when every response fails. Large-scale RL, however, increasingly runs asynchronously on separate GPU pools for generation and optimization, and a teacher does not fit easily into this structure. Given dedicated GPUs, it takes them from the student, and run in step with generation or updates, it makes them wait. We introduce HitchRL, which runs the teacher with no GPUs of its own while no student update waits for an unfinished teacher target. Since a frozen teacher gives a response the same targets whenever it runs, HitchRL defers teacher inference and, with the teacher's weights offloaded to CPU memory, lets it hitchhike on the idle time of the existing GPUs, returning them when generation or optimization needs them, while updates proceed on ready data. With the same eight GPUs, HitchRL raises the eight-benchmark mean accuracy of a 1.7B student from 14.20% with the teacher-free AReaL to 17.04%, the best of all methods compared. This gain over AReaL extends to a larger student, a student from another model family, and code generation with execution feedback. During training, it reaches the same OlympiadBench accuracy as AReaL and as on-policy distillation followed by AReaL in 63% and 76% less wall-clock time, while keeping 0.95 of AReaL's generation throughput with a 30B-A3B teacher and 0.90 even with a 235B teacher.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, on-policy distillation, asynchronous reinforcement learning, large language models, reasoning, efficient training

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_e6xyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HitchRL, a method for incorporating frozen-teacher supervision into asynchronous reinforcement learning with verifiable rewards without reserving dedicated teacher GPUs or blocking student updates. Algorithmically, HitchRL routes mixed-reward groups to the standard reward-based branch, discards all-correct groups, and distills one student-generated response from each all-incorrect group using a reverse-KL-derived token-level signal. In terms of system design, teacher targets are computed later during idle intervals on the rollout or training GPU pool, with teacher parameters offloaded to CPU memory and two queues separating pending teacher work from update-ready data. Experiments covering Qwen3-1.7B, Qwen3-8B, and Olmo-3-7B students, math and code tasks, and teachers with up to 235B parameters demonstrate numerically significant yet potentially unfair improvements in accuracy and efficiency over the AReaL, which might not be a properiate baseline.

Critical strengths
  1. The paper clarifies that stored teacher log-probabilities are time-invariant for a frozen teacher, providing a principled basis for separating target preparation from target consumption.
  2. The system design incorporates several well-designed components to improve efficiency: the two-queue scheduler, bounded teacher batches, CPU-offloaded parameters, and priority for student work. Together, they form a practical design that can add strong-teacher supervision without a dedicated teacher GPU pool.
  3. The evaluation covers several relevant settings, including three student models from two families, math and code tasks, and both 30B-A3B and 235B teachers.
Critical weaknesses

The major weakness is the unfair baseline. Compared with AReaL and OPD+AReaL, HitchRL incorporates both (i) an RSTG-like algorithm (GRPO plus teacher OPD for all-incorrect groups) and (ii) an asynchronous OPD system, where (ii) is the paper's main contribution. However, the time-to-accuracy improvements in Figure 1 and the accuracy improvements in Tables 1 and 2 both reflect (i) and (ii), without clearly disentangling the contribution of (ii). These comparisons use unfair baselines and thus make it difficult to identify the benefits of (ii). For example,

  1. In Figure 1, the paper should compare against a baseline using an RSTG-like algorithm with a synchronous OPD system, such as AReaL's synchronous OPD, to demonstrate the improvements from (ii).
  2. In Tables 1 and 2, the paper should add a synchronous RSTG-like baseline (GRPO with teacher OPD on all-incorrect groups) to quantify any accuracy degradation or improvement caused by asynchronous OPD.
Questions for authors
  1. When reporting gains over AReaL and OPD+AReaL in Figure 1, can the authors disentangle the gains from the algorithmic design and those from the asynchronous system design?

  2. Can the authors fix the algorithm to an RSTG-like setup (GRPO with teacher OPD for all-incorrect groups) and report the time-to-accuracy speedup resulting only from the asynchronous system design? How does this speedup vary between RL tasks with imbalanced versus balanced rollout/training loads?

  3. The paper shows that the teacher OPD gradient is time-invariant. However, asynchronously applying an aged gradient to an already updated model can still produce a different result from synchronous OPD. Can the authors add an RSTG-like algorithm as a baseline in Tables 1 and 2 to show that HitchRL does not hurt accuracy? In HitchRL, how does an over-stale asynchronous OPD gradient affect accuracy?

  4. In Figure 3, how does HitchRL's timeline compare with a synchronous teacher-OPD timeline? How does this comparison change when rollout/training loads are imbalanced versus balanced?

  5. The premise is that idle time is common in RL tasks. In the Introduction, can the authors justify that idle bubbles or rollout/training imbalance are common issues in RL systems, and quantify their typical extent, to support the claim that teacher scoring can be scheduled during idle periods?

4 comments
AuthorsOpenReview ↗

Response to Reviewer e6xy (1/3): the requested baseline, Q1, Q2

Thank you for the careful review. Your main concern is that our comparisons mix (i), an RSTG-like algorithm, with (ii), the asynchronous system. We ran the baseline you asked for, an RSTG-like algorithm with a synchronous OPD system, for two students. In our tables this synchronous RSTG-like baseline is named Blocking: rows 5 (1.7B) and 10 (8B) of the table "The new runs" in General response (1/3). We restate the numbers you need below.

How the baseline is built. We wanted the baseline to do two things at once: keep the RSTG-like algorithm you describe, and isolate (ii). For the first, Blocking keeps what your review names as RSTG-like: GRPO, with teacher OPD applied only to all-incorrect groups. For the second, everything else is set exactly as in HitchRL (the same distillation weight and length rule, and one distilled response per all-incorrect group), and only the teacher is made synchronous: every update waits until the teacher has scored the all-incorrect responses of its batch. The two configurations then differ in one setting, the teacher's execution mode, so any difference between Blocking and HitchRL comes from (ii). General response (3/3), section "Is the learning rule RSTG-like?", compares HitchRL with RSTG itself.

Q1. When reporting gains over AReaL and OPD+AReaL in Figure 1, can the authors disentangle the gains from the algorithmic design and those from the asynchronous system design?

Yes. The table adds (i), then (ii), then both, starting from the teacher-free baseline (1.7B student, 30B teacher; rows of "The new runs" in General response (1/3)).

StepFrom → toWhat is addedWhat the result shows
(i)Filtered AReaL (rows 3–4) → Blocking (row 5)the algorithm (i); neither has (ii)(i) makes the student learn faster. The time to 25.5% on Olympiad falls from 3.39 h to 0.78 h
(ii)Blocking (row 5) → HitchRL (row 1)the system (ii); same (i)(ii) saves time at no cost in accuracy. Mean8 stays the same (17.04, 17.15), and the time per million generated tokens falls from 28.7 s to 22.1 s
(i) + (ii)Filtered AReaL (rows 3–4) → HitchRL (row 1)bothTogether they give a faster and more accurate student. The time to 25.5% on Olympiad falls from 3.39 h to 0.59 h, and Mean8 rises from 16.09 to 17.15 at the same training time and tokens

The algorithm (i) gives the accuracy, and the system (ii) then gives the same accuracy with 23% less time per million generated tokens. Two more results support the first step. Inside HitchRL, removing only the teacher gradient lowers Mean8 from 17.15 to 15.26 (row 2). And against OPD → AReaL, Blocking, which has (i) without (ii), already takes 62% and 70% less training time than the two-stage baselines of rows 8 and 9, at the same Mean8 level. The revision adds a figure in the format of Figure 1 for these runs, with the Blocking curve next to HitchRL's.

Q2. Can the authors fix the algorithm to an RSTG-like setup (GRPO with teacher OPD for all-incorrect groups) and report the time-to-accuracy speedup resulting only from the asynchronous system design?

Yes. Blocking fixes the algorithm, so HitchRL against Blocking measures (ii) alone.

Speed-up from (ii) alone (Olympiad is the in-training evaluation, as in Figure 1)

Blocking (synchronous RSTG-like)HitchRL
1.7B: training time to 26.0% on Olympiad1.21 h0.75 h
1.7B: training time to 26.5% on Olympiad2.24 h1.08 h
1.7B: time per million generated tokens28.7 s22.1 s (23% less)
8B: time per million generated tokens54.1 s50.8 s (6% less)

HitchRL reaches each accuracy that both runs reach earlier, and it takes 23% (1.7B) and 6% (8B) less time per million generated tokens.

Q2 (continued). How does this speedup vary between RL tasks with imbalanced versus balanced rollout/training loads?

It is larger when the load is balanced. To measure this, we kept the 8B student and the teacher and changed only the GPU split.

HitchRL's saving in time per million generated tokens for the 8B student at two GPU splits (both splits compared at the same amount of generation, the first 23M generated tokens)

GPUs (generation:teacher:training)Less with HitchRL than with Blocking
6:0:2 (imbalanced)6.0%
4:0:4 (balanced)19.1%

The speed-up is 6.0% at 6:0:2 and 19.1% at 4:0:4. In the full 200-update runs at 6:0:2 (the 8B row of the table above) it is also 6.0%, so this is representative. Moving two GPUs from generation to training cuts the time per million generated tokens by 39% for Blocking and 48% for HitchRL, so at 6:0:2 training is the bottleneck. Our reading is that a bottlenecked training pool leaves little idle time for the teacher, while at 4:0:4 the training GPUs wait more often, so HitchRL saves more.

AuthorsOpenReview ↗

Response to Reviewer e6xy (2/3): Q3

Q3. Can the authors add an RSTG-like algorithm as a baseline in Tables 1 and 2 to show that HitchRL does not hurt accuracy?

Yes, and it does not. We added the synchronous RSTG-like baseline (Blocking) for the 1.7B student of Table 1 and the 8B student of Table 2. With the learning rule fixed, HitchRL and Blocking end at the same accuracy.

Blocking and HitchRL in the format of Tables 1 and 2 (final evaluation, %)

StudentRunAIME24AIME25AIME26AMC23MathArenaOlympiadMinervaMMLU-ProMean8
1.7BBlocking (synchronous RSTG-like)7.194.483.2337.974.1425.4122.0631.8717.04
1.7BHitchRL7.504.794.4834.924.7226.8521.6032.3317.15
8BBlocking (synchronous RSTG-like)24.5820.3120.0063.4417.8148.9239.1561.5136.97
8BHitchRL22.8120.9418.4465.5516.7948.8136.8661.5036.46

HitchRL's Mean8 minus Blocking's is +0.11 for 1.7B and −0.51 for 8B. Both are within 1.4 generation standard errors (0.31 and 0.37), so the two runs are at the same accuracy. With the 235B teacher the difference is +0.71 (16.11 vs. 15.40; rows 13 and 12 of "The new runs" in General response (1/3)). The revision includes this table.

Q3 (continued). Asynchronously applying an aged gradient to an already updated model can still produce a different result from synchronous OPD.

This is a fair concern, and it helps to be precise about what is stored. What HitchRL keeps from the teacher is its log-probabilities for the response, and the gradient is formed later, at the update. The log-probabilities do not depend on when they are computed, because the teacher is frozen. The student’s term is recomputed with the current model when the response is used, so no gradient from an older student is applied to the updated one. What does age is the response itself, as for every sample in asynchronous RL, and the table above shows that accuracy is not affected.

Q3 (continued). In HitchRL, how does an over-stale asynchronous OPD gradient affect accuracy?

This matters, so we measured it. In the new HitchRL runs targets are used soon after they are generated. For 1.7B a target was used 1.62 policy versions after generation on average, and at most 5; for 8B the values are 1.46 and 2 (a policy version is four updates). As a safeguard, a target older than 16 policy versions is discarded (Table 6), and none was. To see what older targets would do, the submission raises the teacher's work per all-incorrect group in two steps.

More teacher work per all-incorrect group (Table 13, 200 updates; Mean8 regraded with the corrected scorer)

Distilled per all-incorrect groupAll-incorrect groups scored by the teacherMean target age (policy versions)Mean8
1 response, 2,048 tokens (HitchRL)2,402 of 2,4341.7517.07
1 response, 8,192 tokens2,320 of 2,5812.5616.92
16 responses, 8,192 tokens128 of 3,9929.9316.45

When targets age from 1.75 to 2.56 versions with most groups still scored, Mean8 stays within the generation standard error (−0.15, 0.5 standard errors). With all 16 responses it is 0.62 lower. Our reading is that this drop comes mainly from lost supervision, since the teacher reached only 128 of 3,992 all-incorrect groups, and less from the age of the few targets that were used. This is why HitchRL distils one response per all-incorrect group.

AuthorsOpenReview ↗

Response to Reviewer e6xy (3/3): Q4, Q5

Q4. In Figure 3, how does HitchRL's timeline compare with a synchronous teacher-OPD timeline?

The two timelines differ in where the teacher's batches sit. In Blocking's timeline, each teacher batch (the hatched tiles of Figure 3) sits before the update of its batch, so the update waits for it. In HitchRL's, these tiles sit in the stretches where a pool's own work pauses (Figure 3b), and the update starts as soon as its reward-ready data has arrived. For 1.7B, this takes 22.1 s per million generated tokens with HitchRL and 28.7 s with Blocking. The revision adds Blocking's timeline to Figure 3.

Q4 (continued). How does this comparison change when rollout/training loads are imbalanced versus balanced?

The gap between the two timelines is larger when the load is balanced. For the 8B student, HitchRL takes 6.0% less time per million generated tokens at the imbalanced split 6:0:2 and 19.1% less at the balanced split 4:0:4 (measured over the first part of training; table in our answer to Q2, part 1/3). Our reading is that at 6:0:2 the training pool is the bottleneck and has few pauses for the teacher's tiles.

Q5. Can the authors justify that idle bubbles or rollout/training imbalance are common issues in RL systems, and quantify their typical extent, to support the claim that teacher scoring can be scheduled during idle periods?

Yes. Prior systems report idle time and are built to remove it, and it remains in an asynchronous system.

  • Reported by prior systems. RLHFuse reports that a few long responses take more than half of the generation time, and that pipeline bubbles can leave about half of the GPUs idle during training. AReaL reports that in synchronous RL generation waits for the longest response, and reduces end-to-end training time by up to 2.77× with an asynchronous design. DAPO notes that generation time in synchronous RL is dominated by long-tail samples. AsyncOPD reports 1.6× to 3.8× higher throughput than strict synchronous distillation, and f-OPD states that strict synchronisation leads to "substantial system under-utilization".
  • Measured in our system. In a run without a teacher (Filtered AReaL, row 3 of "The new runs" in General response (1/3)), the training GPUs wait for the next batch during 30.28% of training time, 4.62 s per update on average. HitchRL runs the teacher in such waits.
  • Where it lies. It depends on the student. In the submitted runs the teacher scored 0.7% of the distilled responses on the generation GPUs for 1.7B, 21.7% for 8B and 32.7% for Olmo (Table 25). This is why HitchRL lets the teacher run on either pool.

The revision adds these numbers to the Introduction.

AuthorsOpenReview ↗

Summary before the discussion closes

Thank you again for your review. Your request for a synchronous RSTG-like baseline became a central comparison of our rebuttal. Before the discussion closes, here is how the new results bear on the two concerns in your review: that the comparisons mixed (i) and (ii), and that AReaL might not be an appropriate baseline:

  • The baseline you asked for. Blocking (rows 5 and 10 of "The new runs" in General response (1/3)) keeps HitchRL's learning rule and makes every update wait for the teacher. It ends at the same Mean8 as HitchRL (+0.11 and −0.51, within 1.4 generation standard errors), and HitchRL takes 23% (1.7B) and 6% (8B) less time per million generated tokens. This is the contribution of (ii) alone. Against Filtered AReaL, Blocking measures (i): the training time to 25.5% on Olympiad falls from 3.39 h to 0.78 h (part 1/3, Q1).
  • Stronger baselines than AReaL, from the other reviews. AReaL with zero-variance groups filtered out, run to HitchRL's training time and tokens, is 1.06 Mean8 below HitchRL (row 4). OPD → AReaL with HitchRL's distillation settings reaches HitchRL's accuracy, and HitchRL needs 67% and 75% less training time (rows 8 and 9).

We would be grateful to know whether these results address your concerns, or whether anything else would help.

Reviewer_yS7uOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method that combines on-policy distillation (OPD) with GRPO named HitchRL. It applies GRPO to groups with mixed rewards and OPD to failed groups (groups that all trajectories receive zero reward). HitchRL does not allocate dedicated GPUs to the teacher model but runs teacher inference during idle time of the rollout and training GPUs. Specifically, when there are no active generation requests in the rollout pool or the trainer is assembling its next batch on the CPU, the corresponding GPUs are considered idle and assigned to the teacher. When rollout or training needs the GPUs again, the teacher yields them after completing its current batch. Experiments show that HitchRL improves model accuracy and reaches comparable accuracy in less wall-clock time than the baselines.

Critical strengths

The paper proposes a new method that combines GRPO and OPD efficiently by utilizing idle GPUs for teacher inference. Experiments show that it improves model performance across multiple benchmarks and training efficiency.

Critical weaknesses

Regarding the technique,

  1. The performance improvement over the strongest baseline is relatively modest, and the paper does not sufficiently explain why HitchRL outperforms sequential OPD followed by RL.

  2. The paper does not establish how sensitive HitchRL is to the proportion of failed groups and the amount of available idle GPU time.

Regarding the evaluation,

  1. The baseline comparisons use different hyperparameter settings. For example, OPD followed by AReaL uses a fixed distillation weight of 1.0, while HitchRL uses a length-adaptive weight starting at 0.3.

  2. The experiments lack systematic sensitivity analyses of failed-group rates, GPU idle time, and scheduling overhead.

  3. The AReaL baseline may not be optimally configured, as it does not filter out groups with zero reward variance.

Questions for authors
  1. Could the authors provide further analysis or ablation studies to help explain why HitchRL outperforms sequential OPD followed by RL?

  2. Could the authors discuss how sensitive HitchRL is to the proportion of failed groups and the amount of available idle GPU time?

  3. Could the authors consider comparisons with more carefully controlled baselines, such as OPD followed by AReaL with matched distillation hyperparameters and AReaL with zero-variance groups filtered out?

4 comments
AuthorsOpenReview ↗

Response to Reviewer yS7u (1/2): Q1

Thank you for the review and for the concrete suggestions. We ran both controlled baselines you ask for in Q3. Your three questions cover the weaknesses you list, so we answer by question: Q1 here, Q2 and Q3 in part 2/2. Row numbers refer to the table "The new runs" in General response (1/3), and we restate the numbers you need below.

Q1. Could the authors provide further analysis or ablation studies to help explain why HitchRL outperforms sequential OPD followed by RL?

Yes. We re-ran OPD → AReaL with both controls of your Q3 applied. Its OPD stage now uses HitchRL's length-adaptive distillation weight starting at 0.3, in place of the fixed 1.0, and lasts 40 updates (row 8) or the submission's 100 (row 9). The RL stage that follows is AReaL with zero-variance groups filtered out. These runs reach HitchRL's accuracy, and HitchRL reaches it with 67% and 75% less training time (3.31 h against 10.12 h and 13.01 h). Their Mean8 is 17.20 and 16.66, against 17.15 for HitchRL; both differences are within 1.6 generation standard errors (0.32 and 0.31). So against this strongest baseline, HitchRL's gain is the training time at which the accuracy is reached. The table shows where the two-stage runs spend their time.

Training time and generated tokens by stage (1.7B student, 30B teacher)

RunStageTraining time (h)ResponsesGenerated tokens (M)Tokens per response
Row 8: OPD (40) → filtered AReaLOPD, 40 updates0.9920,480391,904
Row 8: OPD (40) → filtered AReaLRL, 400 updates9.13355,0881,1123,133
Row 9: OPD (100) → filtered AReaLOPD, 100 updates3.1751,2002054,012
Row 9: OPD (100) → filtered AReaLRL, 400 updates9.84375,6801,1843,152
Row 3: Filtered AReaL (no OPD stage)RL, 400 updates1.70353,216310877
Row 1: HitchRLRL, 400 updates3.31221,6555402,435
  • Why the two-stage runs take longer. Their RL stage generates about as many responses as Filtered AReaL without an OPD stage (row 3), but each averages over 3,100 tokens, against 877, and the stage takes 9.13 h and 9.84 h, against 1.70 h. Our reading is that the OPD stage lengthens the responses, since it has no reward term to counter this, and the RL stage that follows pays for the longer responses.
  • How HitchRL reaches the same accuracy in less time. It takes the teacher's supervision inside the RL updates, on the groups where RL has no signal. So it has no separate OPD stage to pay for (0.99 h and 3.17 h in rows 8 and 9). And it discards less of what it generates. Each update uses 32 groups in every run, but HitchRL generates fewer responses to fill them (221,655 for the 400 updates, against 355,088 and 375,680): its all-incorrect groups go to distillation and fill the batch, whereas Filtered AReaL drops them and generates replacements.
  • Time to accuracy. In the in-training evaluation on Olympiad, HitchRL reaches 26.0% after 0.75 h of training and row 8 after 2.46 h (table "Training time to reach a given accuracy" in General response (1/3)).

The revision adds this stage breakdown to the comparison with OPD → AReaL.

AuthorsOpenReview ↗

Response to Reviewer yS7u (2/2): Q2, Q3

Q2. Could the authors discuss how sensitive HitchRL is to the proportion of failed groups and the amount of available idle GPU time?

Yes. HitchRL improves on AReaL over the whole range of failed-group proportions in our runs, 14.5% to 41.1%, and its accuracy drops when the teacher is given far more work than it can finish. We take each in turn, then the scheduling overhead you list among the weaknesses.

Proportion of failed (all-incorrect) groups. HitchRL is above AReaL at both ends of the range this proportion took in the submitted HitchRL runs of Tables 1 and 2. Where failed groups are fewest (8B, 14.5–20.3%), it is +1.87 Mean8 above AReaL. Where they are most frequent (1.7B, up to 41.1%), it is +2.81, and the teacher queue dropped no request for lack of capacity in any run. With fewer failed groups the teacher has less to score, and since it holds no GPU of its own, no GPU is left idle for it.

All-incorrect groups in the submitted HitchRL runs of Tables 1 and 2 (range over 40-update intervals)

StudentProportion of all-incorrect groupsHitchRL's Mean8 over AReaL (regraded)
1.7B29.9–41.1%+2.81
8B14.5–20.3%+1.87
Olmo25.2–33.1%+1.27

Amount of available idle GPU time. With HitchRL's setting the teacher, which runs only in idle GPU time, finishes nearly all of its work, and accuracy drops when it is given far more. The submission raises the teacher's work per all-incorrect group in two steps on the same GPUs.

More teacher work per all-incorrect group (Table 13, regraded)

Distilled per all-incorrect groupAll-incorrect groups scored by the teacherMean8
1 response, 2,048 tokens (HitchRL)2,402 of 2,43417.07
1 response, 8,192 tokens2,320 of 2,58116.92
16 responses, 8,192 tokens128 of 3,99216.45

Mean8 is 0.62 lower when the teacher reaches 128 of 3,992 groups. For scale, in a run without a teacher (Filtered AReaL, row 3) the training GPUs wait for the next batch during 30.28% of training time, 4.62 s per update on average.

HitchRL takes less time than Blocking at both GPU splits we compared for the 8B student, and more so when the load is balanced: 6.0% less time per million generated tokens at the imbalanced split 6:0:2 and 19.1% less at the balanced split 4:0:4. Both are measured over the first part of training, the first 23M generated tokens of each run; in the full 200-update runs at 6:0:2 (rows 10 and 11) the saving is also 6.0%. Blocking keeps HitchRL's learning rule but makes every update wait for the teacher. Our reading is that the training GPUs have fewer idle periods at 6:0:2.

Scheduling overhead. When the student needs its GPUs back, the teacher finishes its current batch and returns them. In the new HitchRL runs the trainer waits for this during 5.6% of training time for 1.7B and 2.6% for 8B. This wait is included in every training time we report, and with it HitchRL takes 23% (1.7B) and 6% (8B) less time per million generated tokens than Blocking (rows 1 and 5, 11 and 10).

The revision adds this sensitivity analysis.

Q3. Could the authors consider comparisons with more carefully controlled baselines, such as OPD followed by AReaL with matched distillation hyperparameters and AReaL with zero-variance groups filtered out?

Yes. We ran both. HitchRL reaches the accuracy of the re-tuned OPD → AReaL with less training time, and is more accurate than Filtered AReaL at the same training time and tokens.

The two controlled baselines against HitchRL (row 1, Mean8 17.15)

Baseline you ask forRowsHow it is set upWhat the result shows
OPD followed by AReaL with matched distillation hyperparameters8, 9The OPD stage uses HitchRL's length-adaptive distillation weight starting at 0.3 (the submission used a fixed 1.0) and lasts 40 or 100 updatesSame accuracy at higher cost. Mean8 is 17.20 and 16.66, within 1.6 generation standard errors of HitchRL's, and HitchRL takes 67% and 75% less training time
AReaL with zero-variance groups filtered out3, 4All-correct and all-incorrect groups are dropped and the batch is refilled; the dropped responses are counted in time and tokensHitchRL is more accurate at the same cost. It is 1.06 Mean8 higher at the same training time and tokens (row 4)

Rows 8 and 9 are broken down by stage under Q1 (part 1/2). Row 4 is the checkpoint of the same run whose training time and tokens match HitchRL’s, and the 1.06 difference there is 3.4 generation standard errors. HitchRL is also above AReaL without the filter, by 1.07 Mean8 at 700 updates (row 15).

The revision reports both baselines in a new table of the controlled runs.

Thank you for the additional controlled runs, which address my concerns. I am raising my score to 3. One point remains: against the matched OPD → AReaL baseline, HitchRL’s accuracy is within noise, so its advantage there is efficiency. The abstract still describes HitchRL as the most accurate of all methods compared; please revise this and the related claims in the main text.

AuthorsOpenReview ↗

Accuracy claims in the revision

Thank you very much for raising your score, and for reading the new runs so closely. We agree with this point. Against the matched OPD → AReaL baseline, HitchRL’s accuracy is within noise (Mean8 17.15, against 17.20 and 16.66), so its advantage there is efficiency: it reaches that accuracy with 67% and 75% less training time.

In the revision, the abstract no longer says that HitchRL is the most accurate of all methods compared. It states the two results separately: HitchRL is more accurate than teacher-free AReaL, and it matches the accuracy of the re-tuned OPD → AReaL with 67% and 75% less training time. The introduction and the results section, which also called HitchRL the highest of all methods, are revised in the same way.

Thank you again for the concrete suggestions throughout the review. The controlled baselines you asked for made the paper’s claims both more precise and better supported.

Reviewer_nmGqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

(I have updated my score from a 2 to a 3, and placed my explanation in response to the rebuttal. This is the pre-rebuttal review.)

HitchRL adds selective teacher supervision to asynchronous RL without allocating dedicated teacher GPUs. Mixed-reward groups receive GRPO updates, all-correct groups are excluded, and one length-capped response from each all-failed group is selected for distillation, with a weight adapted to recent response length. Independent pending and ready queues allow updates to consume available data while a frozen, CPU-offloaded teacher evaluates bounded batches during idle periods on either GPU pool. The paper provides a fixed-update invariance argument and evaluates accuracy, learning efficiency, and teacher execution across several students and domains.

Critical strengths
  1. The paper addresses a concrete resource constraint.

Independent queues and bounded GPU handoffs allow teacher supervision to share the existing accelerator allocation. Table 24 shows student updates proceeding while teacher work remains pending, and Tables 13–14 show why selectivity matters: a single capped response preserves broader supervision coverage and fresher targets under the tested idle capacity. The 235B demonstration establishes execution feasibility, although its update-200 mean accuracy is 15.28%, compared with 16.36% for the default 30B run (Tables 15 and 23); the evaluation therefore does not establish a learning benefit from enabling the larger teacher.

  1. The complete system reports meaningful quality–time improvements.

For the 1.7B student, Figure 1 reports 63% and 76% less time to reach the Olympiad peaks of AReaL and OPD → AReaL on the same eight GPUs; Appendix E extends the comparison across accuracy targets. The eight-benchmark mean also rises to 17.04%, versus 14.20% and 16.18% respectively, with further gains across students and code generation (Tables 1–3). Table 26 provides complementary execution evidence: sharing GPUs reduces time per generated token by 7.0% for 1.7B and 45.8% for 8B relative to the tested dedicated allocation, with similar distillation consumption.

Critical weaknesses
  1. The conceptual contribution is a relatively direct integration of established learning and execution mechanisms.

The closest precedents overlap with more than isolated components. KDRL combines GRPO with teacher distillation, studies a variant restricted to entirely failed groups, and examines coefficient annealing to balance imitation, reward learning, and response-length growth (§§3.1–3.4). AsyncOPD separates rollout, teacher scoring, and learner updates, recomputing the teacher–student discrepancy under the current student when cached examples are consumed (§§5–7). On execution, RLHFuse repurposes GPUs freed during generation for auxiliary inference and stages frozen-model weights from CPU memory (§§4 and 6).

These methods do not duplicate HitchRL's complete design: AsyncOPD reserves GPUs for its teacher, and RLHFuse preserves synchronous iteration semantics. HitchRL's distinctive addition is to let reward-ready examples support updates while selected teacher work waits for idle capacity on either existing pool. Once teacher supervision is treated as optional background work, separating pending from ready records, batching requests, and giving student computation priority at bounded call boundaries are relatively direct applications of established execution principles. Coordinating these mechanisms requires substantial engineering, but I see limited new learning or scheduling methodology beyond their integration.

The fixed-update invariance result adds little independent theoretical insight. Appendix A.4 fixes the teacher inputs, consumed batch, student state, and loss coefficients; evaluating the same frozen teacher later then preserves the specified loss and gradient. As the authors acknowledge, this does not characterize the schedule's effects on sample selection, policy age, or supervision mixture. The integration could nevertheless be valuable. Its significance rests primarily on demonstrating a consequential quality–cost advantage over blocking teacher execution, dedicated teacher service, and teacher-free batch filtering.

  1. The practical advance over those alternatives remains insufficiently established.

The reported whole-system quality–time gains provide meaningful evidence that HitchRL can help. However, these comparisons change supervision, batch construction, and execution together. They support the complete recipe without settling how much value its particular execution arrangement adds or whether a simpler use of the same resources would be competitive.

For blocking versus deferred execution, Table 22 uses the same 235B teacher, GPU allocation, response cap, and coefficient rule. Teacher service time falls from 33.20 to 23.21 minutes, with approximately similar accuracy, but elapsed training time is omitted. Because teacher computation overlaps generation and learning, this establishes a batching benefit without quantifying the end-to-end improvement. Total-time measurements for these existing runs would directly assess whether the deferred arrangement delivers comparable quality substantially sooner.

For shared versus dedicated execution, Table 26 retains independent queues in both variants and demonstrates a throughput advantage for sharing GPUs. That result deserves credit, but the competitiveness of the dedicated allocation remains uncertain. It reserves two GPUs for the 30B teacher, leaving four for generation; a 5:1:2 allocation would preserve another generation GPU. Its feasibility and teacher service rate require evaluation or a workload-specific explanation, rather than an assumption based on weight size alone. The actual resident-teacher configuration also needs to be established: repeated CPU–GPU weight staging would change the comparator described in Appendix I.5. Accuracy for the paired runs would help relate their throughput difference to learning efficiency.

For teacher supervision versus more efficient reward-based learning, Table 8 reports increased mixed-reward consumption, while acknowledging that this reflects both routing and policy evolution. A matched-routing ablation without the teacher gradient, preserving the shared normalization, would assess the contribution of teacher information within this recipe. A separate competitive baseline would adapt DAPO's dynamic sampling (§3.2) to the same asynchronous system: filter groups with uniformly correct or incorrect rewards and refill batches with informative groups. Counting replacement-generation costs would test whether obtaining teacher supervision is preferable to spending that budget on further reward-based learning. This does not assume that filtering can reproduce the benefit of learning from failed groups.

The broader evaluation also leaves the scope and reliability of the cost advantage uncertain. Tables 1–3 compare endpoints after prescribed update counts rather than matched compute budgets, whereas Figure 1 directly evaluates quality against time for the 1.7B student on OlympiadBench. That result remains meaningful, but does not establish equal-cost gains across the other students and domains. Each configuration is a single training run, so the reported evaluation errors do not establish robustness of smaller differences to training variability.

The released timing paths and math scorer raise additional verification questions. Method-dependent timing fields need reconciliation with the published analysis, while grading errors can affect both evaluation and the selection of groups for teacher supervision. Their actual effects on the reported comparisons remain unknown. The author questions request the run-specific accounting and regrading needed to resolve these concerns, rather than treating the observed code behavior as proof that the published gains are invalid.

I therefore lean weak reject. This is a firm weak rejection rather than a borderline acceptance, as the unresolved comparisons concern the paper's central contribution rather than peripheral limitations. The system is credible and its reported improvements are promising, but the comparisons and measurement uncertainties leave the practical significance of this relatively modest conceptual advance insufficiently demonstrated. Convincing end-to-end comparisons and reconciliation of the reported results could materially change my assessment.

Questions for authors

1. What does teacher supervision add beyond the change in batch selection?

HitchRL changes both the learning signal and which examples consume learner capacity. Table 8 documents increased mixed-reward consumption, while Tables 16–17 show sensitivity to distillation strength without testing its removal.

Could you report a zero-distillation-weight ablation (λ=0\lambda=0) with the routing rules, response cap, shared loss normalization, and execution machinery otherwise unchanged? This would test the contribution of the teacher gradient within HitchRL's training recipe. Because that configuration still performs teacher inference and retains distillation records, it would not measure the efficiency of a teacher-free alternative.

Separately, does HitchRL outperform teacher-free asynchronous GRPO with zero-advantage filtering and batch refilling? Please count all generation and elapsed time for both methods, including replacement groups and responses discarded by routing. These controls answer distinct questions: whether teacher information improves this recipe, and whether obtaining it is worthwhile compared with a simpler allocation of the training budget.

2. What end-to-end benefit does deferred, shared-GPU execution provide?

Table 22 reports mean accuracies of 15.55% for blocking and 15.28% for deferred execution, alongside lower teacher service time for the latter, but omits elapsed training time. These single-run accuracy differences do not establish a quality advantage for either variant. What are their total training times and quality–time curves? Since teacher service overlaps other work, those results are needed to quantify the practical benefit of deferral.

For Table 26, please report the accuracy of the paired runs and explain the memory and service-rate considerations behind the 4:2:2 allocation rather than 5:1:2 (generation:teacher:training). Was the latter evaluated?

The stated resident-teacher control also needs a run-specific configuration. The shared builder sets forward_only=True and offload_policy=True (hitchrl_teacher.py:340–351), and the FSDP2 engine selects CPUOffloadPolicy (workers/engine/fsdp/transformer_impl.py:472–482), with forward resharding enabled by default. Which source revision and resolved settings, including any resharding override, kept the Table 26 teacher resident between calls? Please provide evidence of parameter residency and transfer costs. This matters because repeated weight staging could disadvantage the dedicated comparator.

3. How robust are the quality–cost gains to training budget, baseline configuration, and measurement choices?

Figure 1 appears to end at approximately 2.9 hours and 490 million generated tokens for HitchRL, versus 1.0 hour and 150 million for AReaL. HitchRL's earlier target crossings are meaningful, but the endpoint accuracy comparison does not use equal budgets. Can AReaL be evaluated at matched elapsed-time and generated-token budgets? Please also provide cost totals for Tables 1–3 and confirm whether the prior-work baselines used the same 9,600-problem DeepScaleR subset.

For OPD → AReaL, how were the 100-update preliminary stage and its fixed distillation weight selected? Were shorter or length-controlled variants evaluated? This would clarify how much the reported 76% advantage depends on the particular two-stage baseline. The poor fixed-weight result for joint RL and distillation in Table 16 does not by itself establish that the pure OPD stage is mistuned.

Please identify the scripts, clocks, and token counters used for the published comparisons. The release derives perf/time_per_step from different intervals for HitchRL and AReaL, although common raw timers are available, and teacher-dependent telemetry can wait after the update timers stop. A reconstruction against cumulative elapsed time, with consistent exclusions, would resolve whether these paths affect the reported gains.

Finally, are training-seed replications available for the main comparisons? Appendix J's single-run limitation is particularly relevant to the smaller endpoint differences; repeated evaluation generations do not measure training variability.

4. Do the reproduced grading errors materially affect training decisions or comparative accuracy?

Correctness labels determine not only reported accuracy but also whether HitchRL sends a group to reward-based learning, teacher supervision, or neither. Grading errors can therefore alter the method's training decisions as well as its evaluation.

The supplied scorer accepts 2.04\boxed{2.04} and 20400\boxed{20400} against the integer reference 204204 through its default percentage-equivalence rule.

The pinned SymPy 1.14.0/ANTLR 4.9.3 combination also disables the SymPy LaTeX fallback, causing some equivalent expressions to fail. For example, the mathematically correct answer

cos⁡(2t)4+sin⁡(2t)4\boxed{\frac{\cos(2t)}{4}+\frac{\sin(2t)}{4}}

receives zero against the equivalent reference

cos⁡(2t)+sin⁡(2t)4\frac{\cos(2t)+\sin(2t)}{4}

at zero-based Minerva row 78.

Were these settings used in the reported experiments? Please regrade the saved evaluation responses with benchmark-appropriate checks and report per-method disagreement rates, corrected benchmark scores, and resulting between-method differences. Where training records permit, how often does corrected grading change rewards or classify a group differently as all-failed, mixed, or all-correct?

Shared grading errors need not cancel across methods that produce different answers or answer forms. Conversely, rare errors with unchanged comparative results would substantially resolve this concern.

Please submit a zip including all code and result files for any of the answers to my questions or weaknesses. I strongly encourage you to run advance LLMs like Fable 5.1/Astra 6 to find flaws in this zip file before sending it to me, as I will be inspecting it deeply with those tools as well as looking at it manually.

7 comments
AuthorsOpenReview ↗

Response to Reviewer nmGq (1/5): Q1

We are grateful for this detailed and careful review. You read the released code, reproduced two errors in our scorer, and stated which comparisons would settle your concerns; each of these made the work better, and we have tried to answer every point with a measurement. Your review states that the significance of HitchRL rests on a quality–cost advantage over three alternatives: blocking teacher execution, dedicated teacher service, and teacher-free batch filtering. We ran all three (table "The new runs" in General response (1/3)). We answer in your numbering and quote each question: Q1 here, Q2 in part 2/5, Q3 in parts 3/5 and 4/5 (with Weakness 1), Q4 and the zip in part 5/5.

Q1. What does teacher supervision add beyond the change in batch selection?

It adds accuracy that batch selection alone does not give. The gain comes from the teacher gradient. Removing only that gradient lowers Mean8 by 1.89 (row 2), and teacher-free filtering with refill is 1.06 below HitchRL at the same training time and tokens (row 4).

Final evaluation per benchmark (%) for the runs of Q1 (rows of "The new runs" in General response (1/3))

RowRunUpdatesAIME24AIME25AIME26AMC23MathArenaOlympiadMinervaMMLU-ProMean8
1HitchRL4007.504.794.4834.924.7226.8521.6032.3317.15
2HitchRL without the teacher gradient4005.523.232.0833.674.1623.7823.7125.9015.26
3Filtered AReaL4005.312.291.8831.643.8923.5924.0825.3114.75
4Filtered AReaL, continued5606.563.652.2935.004.8926.0024.4525.9016.09
14AReaL (no filter)4005.002.812.0832.973.7723.7026.5626.9815.49
15AReaL (no filter)7005.102.501.9835.944.9925.4826.6525.9616.08

Q1 (continued). Could you report a zero-distillation-weight ablation (λ = 0) with the routing rules, response cap, shared loss normalization, and execution machinery otherwise unchanged?

Yes; it is row 2, and teacher information improves this recipe. The routing rules, the 2,048-token cap, the shared loss normalization and the execution are those of HitchRL, and the teacher still scores the selected responses; the configuration differs from HitchRL's in one key, hitchrl.lambda1: 0.0. Mean8 falls from 17.15 to 15.26. The difference is 6.1 times its generation standard error (0.31), and HitchRL is higher on seven of the eight benchmarks.

Q1 (continued). Separately, does HitchRL outperform teacher-free asynchronous GRPO with zero-advantage filtering and batch refilling? Please count all generation and elapsed time for both methods, including replacement groups and responses discarded by routing.

Yes, with everything counted. Filtered AReaL has no teacher, drops all-correct and all-incorrect groups, and refills each batch with mixed groups, as in DAPO's dynamic sampling (§3.2). This run was continued until its generated tokens reached HitchRL's, and row 4 is that point (3.39 h and 548M, against 3.31 h and 540M). There HitchRL is 1.06 Mean8 higher, 3.4 times the generation standard error. In the in-training evaluation of Olympiad, Filtered AReaL first reaches 25.5% at row 4 (3.39 h); HitchRL reaches 25.5% after 0.59 h. For this student and teacher, a budget spent on teacher supervision gives more than the same budget spent on further reward-based learning. Against AReaL (no filter), HitchRL is 1.66 and 1.07 Mean8 higher at 400 and 700 updates (rows 14 and 15).

What is counted. Generated tokens count every training response the student generated, before routing, filtering or truncation. Replacement groups, the groups Filtered AReaL drops, and the all-correct groups and unselected responses of HitchRL are all included. Training time is the elapsed time from just before the first training generation until the token count of that result is complete. Teacher inference and transfers are included, and so is the wait before an evaluation or checkpoint: for generation in flight, for the teacher to hand back its GPUs, and for synchronisation. Only the evaluation and checkpoint intervals themselves are excluded, and during them training generation, teacher inference and student updates are all stopped (verified in part 4/5).

AuthorsOpenReview ↗

Response to Reviewer nmGq (2/5): Q2

Q2. What end-to-end benefit does deferred, shared-GPU execution provide?

Under one learning rule, HitchRL takes less time per million generated tokens than every alternative we ran.

Time per million generated tokens (s), HitchRL against each alternative (rows of "The new runs" in General response (1/3))

ComparisonRowsAlternativeHitchRLLess with HitchRL
Blocking, 1.7B, 30B teacher5, 128.722.123%
Blocking, 8B, 30B teacher10, 1154.150.86%
Blocking, 1.7B, 235B teacher12, 1332.327.216%
Dedicated teacher 4:2:2, 1.7B6, 128.822.123%
Dedicated teacher 5:1:2, 1.7B7, 125.822.114%

Accuracy is the same with the 30B teacher. HitchRL's Mean8 minus Blocking's is +0.11 for 1.7B and −0.51 for 8B, both within 1.4 generation standard errors. For 1.7B, HitchRL is also the first to reach every Olympiad accuracy that both runs reach (table "Training time to reach a given accuracy" in General response (1/3)). For 8B the 6% is at the 6:0:2 split; at 4:0:4 it is 19.1%, measured over the first part of training (General response (1/3)).

Q2 (continued). What are their total training times and quality–time curves?

Over the 200 updates of the Table 22 runs, total training time is 65.69 min for Blocking and 65.11 min for HitchRL. In that time HitchRL generated 19.3% more tokens (158.16M against 132.53M). Per million generated tokens it takes 24.7 s, against 29.7 s for Blocking.

In-training Mean8 against cumulative training time, Table 22 runs (± generation standard error, corrected scorer)

UpdateBlocking: time (min)Blocking: Mean8HitchRL: time (min)HitchRL: Mean8
4012.9910.82 ± 0.478.4011.86 ± 0.47
8025.1213.64 ± 0.4416.1913.77 ± 0.48
12037.4813.99 ± 0.4427.7114.27 ± 0.49
16050.7914.99 ± 0.4743.4816.43 ± 0.51
20065.6917.01 ± 0.5165.1115.99 ± 0.48

HitchRL reaches every evaluation in less training time. In the final evaluation of Table 22, regraded, Mean8 is 16.33 for Blocking and 16.22 for HitchRL. Rows 12 and 13 are a new pair with the 235B teacher at 4:0:4.

Q2 (continued). For Table 26, please report the accuracy of the paired runs and explain the memory and service-rate considerations behind the 4:2:2 allocation rather than 5:1:2 (generation:teacher:training). Was the latter evaluated?

Yes, we evaluated 5:1:2 (row 7). Memory: 5:1:2 is feasible, since the 30B teacher fits on one H200. Service rate: at 4:2:2 the teacher scored 4,559 responses in 20.71 min of inference, and at 5:1:2 it scored 4,542 in 11.50 min; its GPUs are busy for 9% and 6% of the run. The submission used 4:2:2 because we had not considered other allocations for the dedicated teacher. Thank you for raising it: 5:1:2 is the faster comparator (25.8 s against 28.8 s per million generated tokens), and HitchRL takes less time than both. Mean8 is 17.11 at 4:2:2 and 16.46 at 5:1:2, against 17.15 for HitchRL (rows 6, 7 and 1). For the 1.7B pair of Table 26, in-training Mean8 (regraded) at update 120, the last evaluation both runs have, is 16.03 ± 0.54 for the dedicated teacher and 15.73 ± 0.53 for HitchRL. For the 8B dedicated run it is 36.37 ± 0.69 at update 200. The 8B HitchRL run of Table 26 was run without evaluation; row 11 gives the accuracy of 8B HitchRL at full length.

Q2 (continued). Which source revision and resolved settings, including any resharding override, kept the Table 26 teacher resident between calls? Please provide evidence of parameter residency and transfer costs.

Thank you for tracing this through the engine code; your reading is correct. This was an oversight on our side: we gave the teacher of Table 26 its own GPUs but did not turn off its CPU offload, so its weights were not resident between calls. Rows 6 and 7 rerun the dedicated teacher with its weights kept on the GPU. In the revision, Table 26 is updated with these runs, and its 7.0% and 45.8% are labelled as a comparison against a CPU-offloaded dedicated teacher. In the code of the zip, the setting is hitchrl.dedicated.gpu_resident: true; it selects an engine with no CPU offload that refuses any offload option (scripts/support/resident_teacher.py); resharding after the forward pass is unchanged (reshard_after_forward=True). The run checks when the teacher is loaded and again at the end that every teacher weight is on its GPU, and aborts otherwise. The weights are at the same GPU addresses at the start and at the end of the run. A representative teacher call moves 1.87–1.91 MB per GPU from CPU to GPU at 4:2:2, and 0.37 MB at 5:1:2, against the 28.44 GiB and 56.87 GiB of weights that stay on the GPU.

AuthorsOpenReview ↗

Response to Reviewer nmGq (3/5): Q3, budgets and baselines

Q3. How robust are the quality–cost gains to training budget, baseline configuration, and measurement choices?

The gain holds at the same training time and tokens, against a re-tuned two-stage baseline, and under one wall-clock for every method. Budgets and baselines are answered here, clocks and seeds in part 4/5.

Q3 (continued). Can AReaL be evaluated at matched elapsed-time and generated-token budgets? Please also provide cost totals for Tables 1–3 and confirm whether the prior-work baselines used the same 9,600-problem DeepScaleR subset.

AReaL at matched budgets. We take generated tokens as the budget: the response tokens of every response the student generates for training, counted before routing and truncation, including responses that are filtered out or not used for an update, and excluding prompts, padding and evaluation responses. Filtered AReaL was continued until its generated tokens reached HitchRL's (row 4 of “The new runs” in General response (1/3): 548M in 3.39 h, against 540M in 3.31 h), and there it is 1.06 Mean8 lower. AReaL (no filter) ended at 318M tokens in 2.02 h (row 15, 700 updates), and HitchRL is 1.07 Mean8 above it; at 185M tokens (row 14, 400 updates) the difference is 1.66. In the in-training evaluation of Olympiad, AReaL (no filter) is highest after 1.74 h, with 25.48% (update 600); HitchRL first reaches 25.48% after 0.59 h.

Cost totals for Tables 1–3. For the Olmo student and for code, HitchRL is more accurate than AReaL with less training time (5.64 h against 7.14 h, and 1.42 h against 1.76 h).

Training time and response tokens of the submitted runs of Tables 1–3

TableStudentMethodTraining time (h)Response tokens (M)
1Qwen3-1.7BAReaL1.13183.7
1Qwen3-1.7BOPD → AReaL8.13957.9
1Qwen3-1.7BOPDVR11.64764.6
1Qwen3-1.7BPRIME6.28821.7
1Qwen3-1.7BLaSeR4.37201.1
1Qwen3-1.7BHitchRL2.96496.1
2Qwen3-8BAReaL3.0194.8
2Qwen3-8BHitchRL4.83239.2
2OLMo-3-7BAReaL7.14261.8
2OLMo-3-7BHitchRL5.64292.6
3Qwen3-1.7B, codeAReaL1.76164.2
3Qwen3-1.7B, codeHitchRL1.42179.6

9,600-problem subset. Yes. OPDVR, PRIME and LaSeR were trained on the same training file as HitchRL, the 9,600-problem DeepScaleR subset; the revision states it.

Q3 (continued). For OPD → AReaL, how were the 100-update preliminary stage and its fixed distillation weight selected? Were shorter or length-controlled variants evaluated?

The 100 updates come from our initial design, which split a 200-update budget evenly between OPD and RL; the submitted comparison kept them and extended RL to 400. The weight 1.0 applies the distillation loss unscaled in a stage that distils every response. Yes, we have now run a shorter and a length-controlled variant (rows 8 and 9): 40 or 100 OPD updates with weight 0.3 and HitchRL's length rule. Both reach HitchRL's Mean8 (17.20 and 16.66 against 17.15, within 1.6 generation standard errors), and HitchRL needs 67% and 75% less training time. On Olympiad, row 8 reaches 27.0% after 3.55 h and HitchRL after 1.08 h. The submitted 76% is 75.72% after regrading (part 4/5).

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response (1/3): new controlled runs

We thank the reviewers for their careful and constructive reviews. To answer their questions we ran new controlled runs; the fifteen rows below report them. Part 2 gives corrections, part 3 compares HitchRL with prior work. The zip Reviewer nmGq requested is attached.

The new runs.

RowStudentTeacherRunGPUs (generation:teacher:training)UpdatesTraining time (h)Generated tokens (M)Mean8
11.7B30BHitchRL4:0:44003.3154017.15
21.7B30BHitchRL without the teacher gradient (λ = 0)4:0:44001.2220215.26
31.7BnoneFiltered AReaL4:0:44001.7031014.75
41.7BnoneFiltered AReaL, continued4:0:45603.3954816.09
51.7B30BBlocking4:0:44003.8748517.04
61.7B30BDedicated teacher4:2:24003.6645717.11
71.7B30BDedicated teacher5:1:24003.2645416.46
81.7B30BOPD (40) → filtered AReaL4:0:440010.121,15117.20
91.7B30BOPD (100) → filtered AReaL4:0:440013.011,38916.66
108B30BBlocking6:0:22003.1220836.97
118B30BHitchRL6:0:22002.9721036.46
121.7B235BBlocking4:0:42001.1412715.40
131.7B235BHitchRL4:0:42001.2316216.11
141.7BnoneAReaL (no filter)4:0:44001.1918515.49
151.7BnoneAReaL (no filter)4:0:47002.0231816.08

Rows 8–9: the OPD stage uses all eight GPUs.

What the runs answer.

Does the gain come from the teacher or from filtering?

Removing only the teacher gradient lowers Mean8 by 1.89 (row 2). Filtered AReaL, which has no teacher and drops all-correct and all-incorrect groups, is 1.06 lower at the same training time and tokens (row 4). AReaL (no filter) is 1.07 lower at 700 updates (row 15). The gain comes from the teacher's supervision.

What does HitchRL add over a teacher that every update waits for?

Blocking (rows 5, 10) uses HitchRL's learning rule, but every update waits until the teacher has scored the all-incorrect responses of its batch. In HitchRL the update does not wait; the teacher scores them later, in idle GPU time. With the 30B teacher, HitchRL takes 23% less time per million generated tokens for 1.7B, at the same Mean8, and 6% less for 8B. The 8B rows use the 6:0:2 split; at 4:0:4 the saving is 19.1%, measured over the first part of training (reply to e6xy, Q2).

How does sharing GPUs compare with a dedicated teacher, including 5:1:2?

In rows 6 and 7 the teacher has its own GPUs and its weights are kept there. Per million generated tokens, HitchRL takes 22.1 s, row 6 28.8 s and row 7 25.8 s, with Mean8 at one level (16.46 to 17.15). The teacher's GPUs run inference for 9% (row 6) and 6% (row 7) of the run and are idle otherwise. Running the teacher in idle GPU time costs less per token than a dedicated teacher.

How does HitchRL compare with OPD → AReaL when the OPD stage is re-tuned?

Rows 8 and 9 change the two OPD-stage settings the reviewers questioned. The distillation weight is 0.3 with HitchRL's length rule, in place of a fixed 1.0, and the stage lasts 40 or 100 updates. Both rows reach HitchRL's Mean8 (16.66 to 17.20); HitchRL reaches it with 67% and 75% less training time (breakdown in the reply to yS7u).

Training time to reach a given accuracy.

In-training evaluation of Olympiad, the benchmark of Figure 1, every 40 updates. Each cell: training time of the first evaluation at or above the target, and its accuracy (%, ± SE).

TargetRow 1 (HitchRL)Row 5 (Blocking)Row 8 (OPD 40)
26.0%0.75 h (26.04 ± 0.57)1.21 h (26.04 ± 0.56)2.46 h (26.71 ± 0.54)
26.5%1.08 h (27.00 ± 0.58)2.24 h (26.74 ± 0.55)2.46 h (26.71 ± 0.54)
27.0%1.08 h (27.00 ± 0.58)not reached (highest 26.74 ± 0.55)3.55 h (27.11 ± 0.55)
27.5%1.96 h (27.89 ± 0.57)not reachednot reached (highest 27.11 ± 0.55)
28.0%2.53 h (28.12 ± 0.54)not reachednot reached
Last in-training evaluation26.59 (3.31 h)25.82 (3.86 h)26.71 (10.11 h)

HitchRL is the first of the three runs to reach every accuracy that all of them reach (26.0% and 26.5%), and it reaches 28.0% after 2.53 h.

How the runs were measured.

All runs use eight H200 GPUs, one clock and one token counter; all accuracies, final and in-training, are graded with the corrected scorer (part 2). Training time is wall-clock time excluding only evaluation and checkpointing (definition in the reply to nmGq, part 1). Generated tokens count all training responses the student generated. Mean8 is the average accuracy over the eight benchmarks in the row's final evaluation.

AuthorsOpenReview ↗

General response (2/3): corrected scorer and revision

Scorer.

We thank Reviewer nmGq for examining the math scorer that the submission used for training rewards and for evaluation. We checked the two errors the review reports and confirmed both: a rule that accepted answers 100 times too large or too small, and a library mismatch that disabled LaTeX parsing. We corrected both and added per-benchmark checks. To look for further errors, we then tested the corrected scorer on all 13,291 benchmark questions: it marks the reference answer itself as correct for every question, and for every numeric reference it rejects an answer 100 times larger, 100 times smaller, or of the opposite sign. It also passes 41 contrast cases. No reported accuracy depends on the old scorer: the saved evaluation responses behind the submission's accuracy tables and Figure 1 are regraded (below), and every accuracy of the new runs in part 1 is graded with the corrected scorer.

Regraded results.

The conclusions of Tables 1–2 hold under the corrected scorer. HitchRL remains the highest row of Table 1 and stays above AReaL for both students of Table 2; the Mean8 order of Table 1 is preserved, and HitchRL's margin over every other row moves by at most 0.17. Mean8, submitted → regraded:

Table 1 (1.7B)AReaLOPD → AReaLOPDVRPRIMELaSeRHitchRL
Mean814.20 → 15.0816.18 → 17.1515.39 → 16.1815.88 → 16.8014.47 → 15.4917.04 → 17.89
Table 28B AReaL8B HitchRLOlmo AReaLOlmo HitchRL
Mean832.58 → 33.8234.33 → 35.6923.26 → 24.7824.57 → 26.05

Per-benchmark results are in the reply to nmGq, part 5. Table 3 (code) is graded by running tests and is not affected.

Revision.

Every answer in our replies is complete in the comments and the zip. We are also revising the PDF to apply the changes below and will upload it by November 17 (AoE).

  • Accuracy tables regraded with the corrected scorer.
  • A new table with the controlled runs of part 1, and a figure in the format of Figure 1 for these runs, including Blocking.
  • Table 26 updated with the dedicated teacher kept on the GPU, including 5:1:2 (reply to nmGq, part 2).
  • Figure 1 and the time reductions in the abstract and text recomputed with the corrected scorer (reply to nmGq, part 4).
  • The 9,600-problem training subset stated.
  • KDRL and DAPO cited.
  • The Blocking timeline added to Figure 3.
  • Evidence for idle time in RL systems added to the Introduction.
  • The stage breakdown of OPD → AReaL and the sensitivity analysis added (reply to yS7u).
  • Accuracy claims in the abstract and text revised (reply to yS7u).
  • The training reward path and the effect of its percentage equivalence stated (reply to nmGq).
AuthorsOpenReview ↗

General response (3/3): relation to prior work

We thank Reviewers e6xy and nmGq for pointing us to the closest prior work. Reviewer e6xy describes HitchRL’s learning rule as RSTG-like, and Reviewer nmGq names KDRL, AsyncOPD and RLHFuse as the closest precedents. To compare carefully, we went back to each of these papers and built the table below. It covers methods that run a teacher, or a frozen auxiliary model, during training. We took every mark from the method’s own paper (for AReaL’s distillation option, from its official documentation and code), so that each cell can be checked.

O: stated in the source. △: partly. X: not part of the method as described.

Method1. Reward learning + teacher distillation2. Distils only all-incorrect groups3. No prepared solutions or teacher trajectories4. Teacher runs in idle time of the student's GPUs5. Updates decoupled from pending teacher work
GKDOXOXX
MiniLLMXXOXX
KDRLO△OXX
OPDVR / GRPDOXOXX
SDPO + GRPOOXOXX
RSTGOOXXX
AReaL's distillation optionOXOXX
OpenClaw-RLOXOX△
AsyncOPDXXOXO
f-OPDXXOXO
RLHFuseXXO△X
HitchRLOOOOO

△ in three cells. KDRL distils all responses in its main method and studies, as a variant, distillation restricted to all-incorrect groups (§3.4). In OpenClaw-RL, signal extraction is asynchronous, but the teacher’s distribution is computed inside the training step (Algorithm 1). RLHFuse runs frozen reward and reference models on GPUs freed during generation; it has no distillation teacher.

Is the learning rule RSTG-like?

The two share one choice and solve different problems. Both distil only all-incorrect groups (column 2), a choice KDRL already studied (§3.4). RSTG's contribution is which signal to learn from in those groups: it weights each sample by the teacher's accuracy on that prompt, selects tokens, and adds supervised fine-tuning on teacher solutions. For this it runs the teacher on the whole training set before training (column 3). During training its teacher scores every response of an all-incorrect group, eight responses of up to 8,192 tokens, and RSTG reports about 12 extra hours over standard GRPO on eight A100 GPUs with a 14B teacher. HitchRL's contribution is when and where the teacher runs: on the student's own GPUs in their idle time (column 4), with no update waiting for it (column 5). This needs machinery of its own. Responses awaiting the teacher are kept in their own queue, apart from training-ready data. That queue bounds how many targets wait and how old they may become. And the teacher, with its weights in CPU memory, returns a GPU pool at the end of its current batch as soon as the student needs it. HitchRL's rule, one response of at most 2,048 tokens per all-incorrect group, keeps a 30B teacher's work within that idle time. When all 16 responses are distilled, the teacher reaches 128 of 3,992 groups and Mean8 is 0.62 lower (Table 13, regraded).

How close are the closest precedents?

Reviewer nmGq names three, and we read them as the review does. KDRL combines reward learning with teacher distillation and studies restricting distillation to all-incorrect groups; its teacher inference adds to the training time (KDRL §4.3). AsyncOPD separates teacher scoring from the learner's updates, so updates do not wait, but it reserves a GPU for the teacher and has no reward learning. RLHFuse uses GPUs freed during generation for frozen reward and reference models, but it keeps synchronous iterations and has no distillation teacher. Each covers one part of what HitchRL does. No method in the table runs its teacher in the idle time of the student's own GPUs (column 4), and none combines reward learning and distillation (column 1) with updates that do not wait for the teacher (column 5). HitchRL does both, with no GPU reserved for the teacher. Part 1 measures what this is worth against three alternatives: a teacher that every update waits for (Blocking, rows 5 and 10), a teacher with its own GPUs (Dedicated teacher, rows 6 and 7), and no teacher (Filtered AReaL, rows 3 and 4).

Thank you for the substantial work on the rebuttal. The new blocking comparison and matched-budget filtered AReaL experiment make the practical contribution considerably clearer, and I appreciate the additional dedicated-teacher controls and scorer corrections. These results address important parts of my review and make me more positive about the paper.

I would now like to inspect the implementation and records behind these results. In the supplementary ZIP available to me, I found the blocking and dedicated-teacher implementations, but could not locate the corrected scorer or the configurations and outputs corresponding to the fifteen new runs. For example, prime_math/grader.py still defaults to include_percentage=True and accepts the reference multiplied or divided by 100. The data/reevaluation directory contains benchmark questions rather than saved model responses or regrading results. This may simply be an attachment-version issue.

Could you please point me to the relevant files, and provide an updated anonymous archive containing:

  1. The code and configurations used for the new runs: the exact source revision, launch commands and resolved settings, with a mapping to the fifteen table rows. Please include the settings that kept the dedicated teacher GPU-resident.
  2. The corrected scoring implementation: its dependencies, the reported regression tests, and the regrading script. Please also clarify whether the new runs used the corrected scorer (or the impact of this incorrect scorer) for training rewards as well as evaluation.
  3. The records underlying the reported measurements: saved evaluation responses and per-example scores, training-time and token-count logs, and the scripts used to produce the rebuttal tables and time-to-accuracy results, including the stated timing exclusions.

update, a new zip file was uploaded after I posted this

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.