- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces HitchRL, a method for incorporating frozen-teacher supervision into asynchronous reinforcement learning with verifiable rewards without reserving dedicated teacher GPUs or blocking student updates. Algorithmically, HitchRL routes mixed-reward groups to the standard reward-based branch, discards all-correct groups, and distills one student-generated response from each all-incorrect group using a reverse-KL-derived token-level signal. In terms of system design, teacher targets are computed later during idle intervals on the rollout or training GPU pool, with teacher parameters offloaded to CPU memory and two queues separating pending teacher work from update-ready data. Experiments covering Qwen3-1.7B, Qwen3-8B, and Olmo-3-7B students, math and code tasks, and teachers with up to 235B parameters demonstrate numerically significant yet potentially unfair improvements in accuracy and efficiency over the AReaL, which might not be a properiate baseline.
Critical strengths
- The paper clarifies that stored teacher log-probabilities are time-invariant for a frozen teacher, providing a principled basis for separating target preparation from target consumption.
- The system design incorporates several well-designed components to improve efficiency: the two-queue scheduler, bounded teacher batches, CPU-offloaded parameters, and priority for student work. Together, they form a practical design that can add strong-teacher supervision without a dedicated teacher GPU pool.
- The evaluation covers several relevant settings, including three student models from two families, math and code tasks, and both 30B-A3B and 235B teachers.
Critical weaknesses
The major weakness is the unfair baseline. Compared with AReaL and OPD+AReaL, HitchRL incorporates both (i) an RSTG-like algorithm (GRPO plus teacher OPD for all-incorrect groups) and (ii) an asynchronous OPD system, where (ii) is the paper's main contribution. However, the time-to-accuracy improvements in Figure 1 and the accuracy improvements in Tables 1 and 2 both reflect (i) and (ii), without clearly disentangling the contribution of (ii). These comparisons use unfair baselines and thus make it difficult to identify the benefits of (ii). For example,
- In Figure 1, the paper should compare against a baseline using an RSTG-like algorithm with a synchronous OPD system, such as AReaL's synchronous OPD, to demonstrate the improvements from (ii).
- In Tables 1 and 2, the paper should add a synchronous RSTG-like baseline (GRPO with teacher OPD on all-incorrect groups) to quantify any accuracy degradation or improvement caused by asynchronous OPD.
Questions for authors
-
When reporting gains over AReaL and OPD+AReaL in Figure 1, can the authors disentangle the gains from the algorithmic design and those from the asynchronous system design?
-
Can the authors fix the algorithm to an RSTG-like setup (GRPO with teacher OPD for all-incorrect groups) and report the time-to-accuracy speedup resulting only from the asynchronous system design? How does this speedup vary between RL tasks with imbalanced versus balanced rollout/training loads?
-
The paper shows that the teacher OPD gradient is time-invariant. However, asynchronously applying an aged gradient to an already updated model can still produce a different result from synchronous OPD. Can the authors add an RSTG-like algorithm as a baseline in Tables 1 and 2 to show that HitchRL does not hurt accuracy? In HitchRL, how does an over-stale asynchronous OPD gradient affect accuracy?
-
In Figure 3, how does HitchRL's timeline compare with a synchronous teacher-OPD timeline? How does this comparison change when rollout/training loads are imbalanced versus balanced?
-
The premise is that idle time is common in RL tasks. In the Introduction, can the authors justify that idle bubbles or rollout/training imbalance are common issues in RL systems, and quantify their typical extent, to support the claim that teacher scoring can be scheduled during idle periods?
4 comments
Response to Reviewer e6xy (2/3): Q3
Q3. Can the authors add an RSTG-like algorithm as a baseline in Tables 1 and 2 to show that HitchRL does not hurt accuracy?
Yes, and it does not. We added the synchronous RSTG-like baseline (Blocking) for the 1.7B student of Table 1 and the 8B student of Table 2. With the learning rule fixed, HitchRL and Blocking end at the same accuracy.
Blocking and HitchRL in the format of Tables 1 and 2 (final evaluation, %)
| Student | Run | AIME24 | AIME25 | AIME26 | AMC23 | MathArena | Olympiad | Minerva | MMLU-Pro | Mean8 |
|---|---|---|---|---|---|---|---|---|---|---|
| 1.7B | Blocking (synchronous RSTG-like) | 7.19 | 4.48 | 3.23 | 37.97 | 4.14 | 25.41 | 22.06 | 31.87 | 17.04 |
| 1.7B | HitchRL | 7.50 | 4.79 | 4.48 | 34.92 | 4.72 | 26.85 | 21.60 | 32.33 | 17.15 |
| 8B | Blocking (synchronous RSTG-like) | 24.58 | 20.31 | 20.00 | 63.44 | 17.81 | 48.92 | 39.15 | 61.51 | 36.97 |
| 8B | HitchRL | 22.81 | 20.94 | 18.44 | 65.55 | 16.79 | 48.81 | 36.86 | 61.50 | 36.46 |
HitchRL's Mean8 minus Blocking's is +0.11 for 1.7B and −0.51 for 8B. Both are within 1.4 generation standard errors (0.31 and 0.37), so the two runs are at the same accuracy. With the 235B teacher the difference is +0.71 (16.11 vs. 15.40; rows 13 and 12 of "The new runs" in General response (1/3)). The revision includes this table.
Q3 (continued). Asynchronously applying an aged gradient to an already updated model can still produce a different result from synchronous OPD.
This is a fair concern, and it helps to be precise about what is stored. What HitchRL keeps from the teacher is its log-probabilities for the response, and the gradient is formed later, at the update. The log-probabilities do not depend on when they are computed, because the teacher is frozen. The student’s term is recomputed with the current model when the response is used, so no gradient from an older student is applied to the updated one. What does age is the response itself, as for every sample in asynchronous RL, and the table above shows that accuracy is not affected.
Q3 (continued). In HitchRL, how does an over-stale asynchronous OPD gradient affect accuracy?
This matters, so we measured it. In the new HitchRL runs targets are used soon after they are generated. For 1.7B a target was used 1.62 policy versions after generation on average, and at most 5; for 8B the values are 1.46 and 2 (a policy version is four updates). As a safeguard, a target older than 16 policy versions is discarded (Table 6), and none was. To see what older targets would do, the submission raises the teacher's work per all-incorrect group in two steps.
More teacher work per all-incorrect group (Table 13, 200 updates; Mean8 regraded with the corrected scorer)
| Distilled per all-incorrect group | All-incorrect groups scored by the teacher | Mean target age (policy versions) | Mean8 |
|---|---|---|---|
| 1 response, 2,048 tokens (HitchRL) | 2,402 of 2,434 | 1.75 | 17.07 |
| 1 response, 8,192 tokens | 2,320 of 2,581 | 2.56 | 16.92 |
| 16 responses, 8,192 tokens | 128 of 3,992 | 9.93 | 16.45 |
When targets age from 1.75 to 2.56 versions with most groups still scored, Mean8 stays within the generation standard error (−0.15, 0.5 standard errors). With all 16 responses it is 0.62 lower. Our reading is that this drop comes mainly from lost supervision, since the teacher reached only 128 of 3,992 all-incorrect groups, and less from the age of the few targets that were used. This is why HitchRL distils one response per all-incorrect group.
Response to Reviewer e6xy (3/3): Q4, Q5
Q4. In Figure 3, how does HitchRL's timeline compare with a synchronous teacher-OPD timeline?
The two timelines differ in where the teacher's batches sit. In Blocking's timeline, each teacher batch (the hatched tiles of Figure 3) sits before the update of its batch, so the update waits for it. In HitchRL's, these tiles sit in the stretches where a pool's own work pauses (Figure 3b), and the update starts as soon as its reward-ready data has arrived. For 1.7B, this takes 22.1 s per million generated tokens with HitchRL and 28.7 s with Blocking. The revision adds Blocking's timeline to Figure 3.
Q4 (continued). How does this comparison change when rollout/training loads are imbalanced versus balanced?
The gap between the two timelines is larger when the load is balanced. For the 8B student, HitchRL takes 6.0% less time per million generated tokens at the imbalanced split 6:0:2 and 19.1% less at the balanced split 4:0:4 (measured over the first part of training; table in our answer to Q2, part 1/3). Our reading is that at 6:0:2 the training pool is the bottleneck and has few pauses for the teacher's tiles.
Q5. Can the authors justify that idle bubbles or rollout/training imbalance are common issues in RL systems, and quantify their typical extent, to support the claim that teacher scoring can be scheduled during idle periods?
Yes. Prior systems report idle time and are built to remove it, and it remains in an asynchronous system.
- Reported by prior systems. RLHFuse reports that a few long responses take more than half of the generation time, and that pipeline bubbles can leave about half of the GPUs idle during training. AReaL reports that in synchronous RL generation waits for the longest response, and reduces end-to-end training time by up to 2.77× with an asynchronous design. DAPO notes that generation time in synchronous RL is dominated by long-tail samples. AsyncOPD reports 1.6× to 3.8× higher throughput than strict synchronous distillation, and f-OPD states that strict synchronisation leads to "substantial system under-utilization".
- Measured in our system. In a run without a teacher (Filtered AReaL, row 3 of "The new runs" in General response (1/3)), the training GPUs wait for the next batch during 30.28% of training time, 4.62 s per update on average. HitchRL runs the teacher in such waits.
- Where it lies. It depends on the student. In the submitted runs the teacher scored 0.7% of the distilled responses on the generation GPUs for 1.7B, 21.7% for 8B and 32.7% for Olmo (Table 25). This is why HitchRL lets the teacher run on either pool.
The revision adds these numbers to the Introduction.
Summary before the discussion closes
Thank you again for your review. Your request for a synchronous RSTG-like baseline became a central comparison of our rebuttal. Before the discussion closes, here is how the new results bear on the two concerns in your review: that the comparisons mixed (i) and (ii), and that AReaL might not be an appropriate baseline:
- The baseline you asked for. Blocking (rows 5 and 10 of "The new runs" in General response (1/3)) keeps HitchRL's learning rule and makes every update wait for the teacher. It ends at the same Mean8 as HitchRL (+0.11 and −0.51, within 1.4 generation standard errors), and HitchRL takes 23% (1.7B) and 6% (8B) less time per million generated tokens. This is the contribution of (ii) alone. Against Filtered AReaL, Blocking measures (i): the training time to 25.5% on Olympiad falls from 3.39 h to 0.78 h (part 1/3, Q1).
- Stronger baselines than AReaL, from the other reviews. AReaL with zero-variance groups filtered out, run to HitchRL's training time and tokens, is 1.06 Mean8 below HitchRL (row 4). OPD → AReaL with HitchRL's distillation settings reaches HitchRL's accuracy, and HitchRL needs 67% and 75% less training time (rows 8 and 9).
We would be grateful to know whether these results address your concerns, or whether anything else would help.
Response to Reviewer e6xy (1/3): the requested baseline, Q1, Q2
Thank you for the careful review. Your main concern is that our comparisons mix (i), an RSTG-like algorithm, with (ii), the asynchronous system. We ran the baseline you asked for, an RSTG-like algorithm with a synchronous OPD system, for two students. In our tables this synchronous RSTG-like baseline is named Blocking: rows 5 (1.7B) and 10 (8B) of the table "The new runs" in General response (1/3). We restate the numbers you need below.
How the baseline is built. We wanted the baseline to do two things at once: keep the RSTG-like algorithm you describe, and isolate (ii). For the first, Blocking keeps what your review names as RSTG-like: GRPO, with teacher OPD applied only to all-incorrect groups. For the second, everything else is set exactly as in HitchRL (the same distillation weight and length rule, and one distilled response per all-incorrect group), and only the teacher is made synchronous: every update waits until the teacher has scored the all-incorrect responses of its batch. The two configurations then differ in one setting, the teacher's execution mode, so any difference between Blocking and HitchRL comes from (ii). General response (3/3), section "Is the learning rule RSTG-like?", compares HitchRL with RSTG itself.
Q1. When reporting gains over AReaL and OPD+AReaL in Figure 1, can the authors disentangle the gains from the algorithmic design and those from the asynchronous system design?
Yes. The table adds (i), then (ii), then both, starting from the teacher-free baseline (1.7B student, 30B teacher; rows of "The new runs" in General response (1/3)).
The algorithm (i) gives the accuracy, and the system (ii) then gives the same accuracy with 23% less time per million generated tokens. Two more results support the first step. Inside HitchRL, removing only the teacher gradient lowers Mean8 from 17.15 to 15.26 (row 2). And against OPD → AReaL, Blocking, which has (i) without (ii), already takes 62% and 70% less training time than the two-stage baselines of rows 8 and 9, at the same Mean8 level. The revision adds a figure in the format of Figure 1 for these runs, with the Blocking curve next to HitchRL's.
Q2. Can the authors fix the algorithm to an RSTG-like setup (GRPO with teacher OPD for all-incorrect groups) and report the time-to-accuracy speedup resulting only from the asynchronous system design?
Yes. Blocking fixes the algorithm, so HitchRL against Blocking measures (ii) alone.
Speed-up from (ii) alone (Olympiad is the in-training evaluation, as in Figure 1)
HitchRL reaches each accuracy that both runs reach earlier, and it takes 23% (1.7B) and 6% (8B) less time per million generated tokens.
Q2 (continued). How does this speedup vary between RL tasks with imbalanced versus balanced rollout/training loads?
It is larger when the load is balanced. To measure this, we kept the 8B student and the teacher and changed only the GPU split.
HitchRL's saving in time per million generated tokens for the 8B student at two GPU splits (both splits compared at the same amount of generation, the first 23M generated tokens)
The speed-up is 6.0% at 6:0:2 and 19.1% at 4:0:4. In the full 200-update runs at 6:0:2 (the 8B row of the table above) it is also 6.0%, so this is representative. Moving two GPUs from generation to training cuts the time per million generated tokens by 39% for Blocking and 48% for HitchRL, so at 6:0:2 training is the bottleneck. Our reading is that a bottlenecked training pool leaves little idle time for the teacher, while at 4:0:4 the training GPUs wait more often, so HitchRL saves more.