acceptodds
Under review as a conference paper at ICLR 2027

Group-Aware Verification Scheduling for Reinforcement Learning with Execution-based Rewards

Abstract

Reinforcement learning post-training of LLMs on tasks such as coding computes each reward by executing the rollout's output, for example by applying a generated patch in the task's environment and running its tests. Verification time can vary by an order of magnitude across tasks, and verification is failure-prone. Group-based algorithms such as GRPO make this worse, because a group of rollouts can enter training only when every member has a result, so the learner waits for the slowest member of its slowest group. Existing systems give no priority to the verification jobs that would complete a group. We present LastCall, a runtime that schedules verification for group-based reinforcement learning. It bounds each round of verification with a waiting window, after which incomplete groups keep verifying and can join a later batch whole under a one-update staleness bound, and gives each free verifier worker to the group closest to completion. Implemented inside asynchronous veRL, LastCall cuts model update time by 13% to 28% across three model scales in our experiments, without lowering the model quality.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning System, Machine Learning System

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_x7nhOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies the scheduling of post-training for tasks where success verification is required, e.g., software-patching tasks where the proposed software patch is verified by a test suite (this is also the scenario used in the paper). The issue is that the post-training happens in batches, patching and verification are distributed to multiple workers, and a batch can only be trained when m verifications have finished. Moreover, training is only allowed to be based on the current or next iteration's weights, so learned updates that are too old need to be discarded. The authors propose to try to fill a batch as quickly as possible in order to minimize the downtime of the learner. Since the verification times are unknown, the approach is a best-effort heuristic. The proposed approach is evaluated in several LLM settings and demonstrates significant gains.

Critical strengths
  • Strong experimental evaluation, improving over the SotA
Critical weaknesses
  • The description of the approach is not sufficiently clear. I still do not fully understand how to read Figure 1.

  • After reading the problem setup, I tried to think about how I would approach the problem. From what I understood, the approach is more or less the - rather simple - idea I came up with in 20 seconds. So it seems to me that the proposed solution is rather obvious.

  • It is unclear whom this paper is interesting for. I do not think any scientist faces the addressed problem as part of their workflow, so I can only imagine large AI labs as target groups. I cannot believe that they are not already doing something like the proposed approach.

Questions for authors
  • Could you explain why your approach is nontrivial?

  • The complexity remark on p. 4 that the special-case scenario with known durations is NP-hard is misleading, since that leads to a combinatorial problem. If the durations are unknown, the problem is stochastic and cannot be solved optimally. Why do you make this remark?

Reviewer_BCkTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies the verification stage of execution-reward RL post-training as a scheduling bottleneck: because a GRPO group enters training only when every member's verification job resolves, a learner waits for the slowest member of its slowest group. Section 3 characterises this on SWE-bench — 67.7% of patches fail to apply in ~0.5 s while the 31.9% that reach the test suite consume 93% of verifier time, and 49 of 489 tasks account for 66.6% of test time. The proposed system, LASTCALL, is built inside veRL 0.7.1 and has three mechanisms: (i) a waiting window (480 s) at whose cutoff the learner stops waiting, after which incomplete groups keep verifying and may join a later batch whole under a one-update staleness bound rather than timing out; (ii) a group-progress ordering that gives each free verifier worker to the pending job whose group has the fewest unresolved members (Alg. 1, §4.2); and (iii) an admission test (Eq. 1) that declines to start a job whose estimated duration will not fit the remaining window, under a configured start limit.

Critical strengths

Model scales × two modes × three seeds × 200 updates on a real coding benchmark is a genuine systems result, and the accounting reconciles: Table 10b's stage sums match Table 6's T₂₀₀ values, Table 9(a)'s worker-seconds (26.15 M over 266,976 s at 14B sync) match the delay columns, and the "more than 12 hours sooner" claim holds at all six cells. The candid reporting is load-bearing for this assessment in two directions. First, the authors state that at a 1 s and 33 s window a simpler baseline beats LASTCALL on both metrics, that the middle-of-run replay shows the ordering helping "not at all," and that the sync arm generates ~10% more candidates than the baseline — all facts that a less careful paper would have omitted. Second, the delay decomposition in Table 3 (result-ready 167→166 s, group-complete 234→231 s, batch-supply 975→638 s at 14B sync) is published in the paper and is precisely the measurement that contradicts the paper's own title; the readers of this review did not have to reconstruct it. Both facts mean the repair path is interpretation and a bounded set of additional runs, not a rebuild.

Critical weaknesses

The evidence for group-progress ordering is one cell: 7B async, LASTCALL-FIFO 35.57 ± 1.28 h vs LASTCALL 31.41 ± 0.89 h, i.e. 4.16 h against a combined σ of 1.56 h (t ≈ 2.7), inside a total saving of 12.27 h. There is no sync-mode ordering control at any scale. Worse, the paper's own 14B sync decomposition points the other way: groups complete at essentially the same speed under both systems (234 → 231 s, 1.3%), and the entire 35% reduction appears in batch-supply time, i.e. the gain comes from not waiting, not from ordering. The middle-of-run replay agrees — ordering is worth 0.61% on time and 0.79% on work (Table 4), and LASTCALL without its time check still beats GROUP_W_PRIORITY with it by 65% on work (Table 11b). Compounding this, the reference set contains a policy defined to waste work (GROUP_W_PRIORITY keeps starting jobs "even when the batch is full or the estimate exceeds the remaining window"), so the 71–72% work reductions are partly definitional; against LASTCALL-FIFO the ordering changes verification work by 0.8%. And no genuine competing scheduler is run: shortest-remaining-time-first is the obvious low-latency policy in exactly this "duration unknown until done" setting, and here it appears only as a tiebreak. A reader who takes the abstract at face value will implement a group-progress priority queue and skip the deadline, which is the wrong order of operations. This is the single issue that makes the headline uninterpretable as published, and it is not repairable by re-analysis: it needs LASTCALL-FIFO at three scales in both modes, and an SRTF / coflow baseline, to settle.

Questions for authors
  1. What is the start limit's value, and what are m, N, W and the concurrency for every reported configuration? The start limit is described as one of the two control parameters of the method (§4.1, Alg. 1 line 7) and is the knob §6.3 says produces most of the work reduction, yet no value appears anywhere in the paper or appendix; m ("number of groups per batch") is likewise never instantiated. If m = N, the "any m complete groups" framing is decorative and the entire benefit is cross-round deferral, which would substantially narrow the contribution — please say which it is.
  2. Provide a staleness-matched control. Async LASTCALL trains with 51% one-update-stale groups against 15% for async veRL, and sync LASTCALL with 42% against 0%; the speedup is therefore partly bought by training on more off-policy data, and the absence of detectable harm is not evidence of none. Either run async FIFO with the same one-update allowance actually used to the same degree, or report the PPO ratio distribution and the fraction of dual-clipped tokens (c = 3.0) under each system.
  3. Explain the change in training-data composition and whether it is intended. The share of used groups with all-equal member rewards falls 34.8% → 24.3% (sync) and 34.5% → 22.6% (async) — under GRPO those groups carry exactly zero advantage — while first executions without a usable result fall from 98.1% to 49.8% and sync LASTCALL discards 8,314 groups against veRL's 1,523. Either these make LASTCALL a learning-efficiency result rather than only a wall-clock one, or they mean the two systems train on different data and the comparison is no longer purely a systems comparison. Please report per-group reward mean/variance and the pass rate split by long/short task and by repository, and state plainly whether the 0.8–1.4 pp pass@1 edge is explained by this shift.
  4. Was the 480 s window selected on the evaluation workload family, and is it the same regime as deployment? It is set at ~1.15 × the p90 of a quantity dominated by container preparation (§3, Fig. 2), and §B.3 reports that at 240 s LASTCALL finishes later in three of seven comparisons and at 33 s a simpler baseline beats it on both metrics. Please report an end-to-end window sweep (240/480/720 s at 7B and 14B) showing a plateau rather than a peak, and state whether environments were pre-warmed in the reported runs — a reader cannot currently tell whether 13–28% applies to a cached or uncached deployment.
4 comments

Thank you for the thoughtful review. We address the concerns and corresponding questions below. In the tables, (new) marks results that are not in the original submission.

Effectiveness of group-progress ordering [Weakness]

We have new results to show how the ordering contributes to the speedup. We trained LastCall-FIFO end-to-end at 14B in the same config in both modes (three runs each). Our new runs show that the ordering gives about a third of the saving, in both sync and async modes.:

SettingveRL (h) ↓LastCall-FIFO (h) ↓LastCall (h) ↓Ordering's share of the saving
7B async43.68 ± 0.9235.57 ± 1.2831.41 ± 0.8933.9%
14B sync74.16 ± 0.3763.67 ± 0.53 (new)58.15 ± 0.4234.5%
14B async65.67 ± 2.8057.87 ± 1.73 (new)52.95 ± 2.1038.7%

Time to 200 updates, mean ± standard deviation over three runs. Ordering's share = (LastCall-FIFO − LastCall) / (veRL − LastCall).

Table 3 compares LastCall with veRL (instead of LastCall-FIFO), so it cannot separate the ordering from the window. Although its mean group-completion delay barely moves (234 s to 231 s), the median falls from 146 s to 127 s and the 95th percentile from 725 s to 675 s (Table 10(a)). The direct evidence is the 14B sync row above, where LastCall-FIFO takes 5.52 h longer than LastCall. We also ran a new replay ablation. On the fresh-replay workload of Table 4, we start from veRL and switch on LastCall's parts one at a time. Lower is better:

Policy (fresh replay)Generation and supply (s) ↓
veRL24,462 (new)
Window and start limit, groups left at the cutoff discarded33,918 (new)
Window and start limit, groups left at the cutoff deferred21,895 (new)
+ time check20,714 (new)
+ stop once the batch is full (LastCall-FIFO)19,372
+ group-progress ordering (LastCall)16,845

Other values are from Table 4.

Generation and supply (Table 4, §B.1) is the time, over 25 updates, to release each round's recorded rollouts and verify enough of them to fill complete batches. LastCall saves 7,617 s over veRL, and the three parts share it roughly equally. The window with deferral saves 2,567 s (row 3). The time check and stopping once the batch is full save 2,523 s (rows 4 and 5). The ordering saves 2,527 s (row 6), about as much as the window. Row 2 shows that the window needs deferral. If the groups still incomplete at the cutoff are discarded, the replay is slower than veRL. On the specific points:

  • t ≈ 2.7 at 7B divides the 4.16 h gap by 1.56 h, which combines the standard deviations of single runs, but each value in Table 6 is the mean of three runs. So the gap between the two means has a standard error of 1.56 h / √3 = 0.90 h, and t = 4.62.
  • The group-completion delays in Table 3 compare LastCall with veRL, so we cannot tell ordering’s benefit apart from the window. However, in that comparison, the mean delay barely moves (234 s to 231 s), but the median falls from 146 s to 127 s and the 95th percentile from 725 s to 675 s (Table 10(a)). The direct test of the ordering is the 14B sync row of the first table above, where LastCall-FIFO takes 5.52 h longer than LastCall.
  • The 0.61% comes from the mid-run replay, where only 4.2% to 4.8% of start decisions have more than one job to choose from. In the fresh replay, 82.4% to 84.1% do (§B.3), and there the ordering cuts generation and supply by 13.0% (Table 4). We will explain these clearly in the revision.

Competing schedulers [Weakness]

group_w_priority is a baseline we built to examine the start rules. It favors nearly complete groups as LastCall does, but it has no time check and keeps starting jobs after the batch is full. So the reductions of at least 71% in verification work against it come mainly from these two rules. In Table 11(b), group_w_priority uses the time check and LastCall does not. So the two policies differ mainly in whether they stop once the batch is full, and LastCall does 65% less verification work. The replay ablation table above measures the same two rules without this baseline. Together they save 2,523 s of generation and supply in the fresh replay (rows 4 and 5). The ordering mainly saves time. It changes verification work by only about 1% (§B.2), in line with your 0.8%.

SRTF ranks jobs by remaining time. LastCall does estimate a job's duration, as the mean time of the completed verifications in its group (§4.2). But it uses this estimate only for the yes/no time check for the future group members, and the estimate does not exist until the group's first verification finishes. So it cannot rank every pending job by remaining time, as SRTF needs. We will try to add an SRTF baseline to the replay study in the revision.

Start limit, m, N, and W [Question 1]

We list all the configs below:

SettingEnd-to-end runsReplays
Groups per update m512128
Groups per generation step N (sync)512128
Rollouts per group G44
Concurrent verification jobs W25664
Window480 s480 s
Start limit8 per group per window8 per group per window
Verification timeout3,000 sn/a (recorded durations)
Retries per rolloutat most 1at most 1
Staleness bound1 update1 update
  • Deferral is not the whole benefit. With deferral in place, the time check, stopping once the batch is full, and the ordering cut generation and supply by another 23% (the replay ablation table in the ordering response). In 14B sync training, the ordering alone saves 5.52 h (the first table in the ordering response).
  • The "any m complete groups" framing: N = m in sync mode only. After the first round, T in §4 also counts groups carried over from the previous round.Groups from older rounds are discarded by the one-update staleness bound. So a batch takes any m complete groups and does not wait for its round's slowest ones. At 14B, 42% of the groups sync LastCall trains on were generated one update earlier, against 0% for sync veRL. Async training has no rounds, and at 14B async LastCall generates about 599 candidate groups per update for the 512 it uses (Table 13(a)).

Staleness-matched control [Question 2]

We ran a staleness-matched control, and most of the sync speedup does not depend on stale groups. We have done another new sync training at 14B, where we set LastCall's staleness bound to 0, so it trains on no stale groups (three runs):

14B syncStale groups usedTime to 200 updates (h) ↓pass@1 at update 200 (%) ↑
veRL0%74.16 ± 0.3712.3 ± 0.33
LastCall, staleness bound 0 (new)0%62.98 ± 0.4913.6 ± 0.51
LastCall, staleness bound 142%58.15 ± 0.4213.7 ± 0.33

Mean ± standard deviation over three runs.

With no stale groups, LastCall keeps 70% of its savings over veRL. Its pass@1 differs from LastCall with staleness bound 1 by -0.11 points. So stale groups give the other 30% of the saving. We train on them with the usual clipped PPO objective, with the ratio computed against the probabilities recorded when they were generated (§4.3). In async training, both systems allow groups up to one update old, and LastCall trains on more of them (51% against 15% at 14B). Its pass@1 is not lower at any scale in either mode.

Training-data composition [Question 3]

We did not design LastCall to shift the training data. With three runs per system, we cannot tell whether the shift explains the pass@1 edge. So we claim a wall-clock gain with quality that is not lower, and we do not count the edge as a gain from scheduling (§6.2).

  • The 98.1% and 49.8% are retry rates. Of the first executions without a usable result, veRL retries 98.1% and LastCall retries 49.8%. The share of first executions without a usable result barely changes, 4.98% for veRL and 4.81% for LastCall (Table 9(b)).
  • Discards go both ways. Sync LastCall discards more groups than veRL (8,314 against 1,523), and async LastCall discards fewer (9,642 against 15,211, Table 13(a)).
  • We see no signs that LastCall fills its batches with fast groups. It trains on at least 93% as many long-task groups as veRL in the same mode (Table 13(b)). Groups whose patches all fail to apply finish first, and all four of their rollouts fail. So if LastCall favored fast groups, this share would rise. It falls at 14B instead, from 34.8% to 24.3% in sync and from 34.5% to 22.6% in async training.

The 480 s window and deployment [Question 4]

Yes, the 480 s window comes from the same benchmark family, but from a different task set. We set it once, before the end-to-end runs, about 15% above the 90th percentile of verification time (416.2 s) that §3 measures on SWE-bench Verified. Training uses SWE-bench Full. We also swept the window in the fresh replay, where every result must be produced. LastCall's time is flat around 480 s:

Generation and supply (s) ↓240 s (new)480 s720 s (new)960 s (new)
LastCall17,36816,84517,14917,454
LastCall-FIFO20,11519,37219,78320,262

Across these windows, LastCall's time varies by at most 3.6% and stays at least 13% below LastCall-FIFO's. The shorter windows that look better in §B.3 come from the mid-run replay, where most results are already recorded.

We did not pre-warm environments in the reported runs, and their verification time includes environment preparing (§B.1). So the 13% to 28% speedup in the paper applies to an uncached deployment. To estimate a cached one, we removed environment preparing from each job's recorded duration in the fresh replay:

Generation and supply (s) ↓veRLLastCall-FIFOLastCall
Full durations24,462 (new)19,37216,845
Without environment preparing (new)14,38612,41911,479

LastCall still finishes 20.2% sooner than veRL (31.1% with full durations) and 7.6% sooner than LastCall-FIFO.

Thank you again for your detailed review. We have posted our responses.

As the discussion period is drawing to a close, we would be grateful if you could let us know whether our responses address your concerns. If any point remains unclear or would benefit from further evidence, we are happy to provide additional clarification or results.

Thank you for your time and consideration.

Reviewer_wBQBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LASTCALL, a verification runtime inside veRL that (1) bounds each verification round with a 480s waiting window while letting incomplete groups finish asynchronously and join a subsequent batch under a 1-update staleness bound, (2) gates new job starts by batch need, a per-window start cap, and within-group duration estimates (d^j≤c−t\hat{d}_j \le c - t), and (3) prioritizes groups with the fewest unresolved members ug(t)u_g(t). Across Qwen2.5-7B, Qwen3-14B, and Qwen3-30B-A3B (200 updates, 3 runs each), LASTCALL reduces end-to-end training time by 13.5%–28.1% (up to 1.39x) and verification worker-seconds by 5.1%–22.5% without degrading held-out pass@1.

Critical strengths
  • Well-motivated problem and candid workload characterization: Adapting coflow-style completion-aware scheduling (mm-of-NN complete groups of size GG) to RL verification is natural and timely. Section 3 transparently shows that while MiniWoB++, AppWorld, and τ³ have tight verification latencies (P90/P50 <= 1.76x), coding exhibits a 12.35x P90/P50 tail.
  • Substantial multi-scale evaluation: Running 200 updates x 3 seeds across three model scales (up to 30B-A3B) and six system configurations on 16 A100s, alongside staleness and long-task audits (Table 13), represents serious empirical effort.
Critical weaknesses
  1. Verification bottleneck inflated ~20x by uncached environment setup, with unspecified CPU hardware and single-turn non-thinking rollouts: Comparing Table 1 (167.26s mean, 416.2s P90) with Table 14 (8.64s mean, 1.89s P50, 17.47s P90 for gold-patch tests in prepared environments) shows that ~95% of verification time comes from repeated environment preparation rather than test execution. Standard coding-RL pipelines pre-build/cache per-task container images, which collapses the multi-minute setup tail. Meanwhile, GPU generation time is minimized by disabling Qwen3 thinking mode and using single-call T=0.2 rollouts (Table 8). Crucially, for a CPU-scheduling paper running up to 256 concurrent containers, CPU hardware (node count, cores, RAM, disk I/O) is never specified, making it impossible to separate intrinsic task tails from container/I/O contention or assess how much speedup survives with cached images or multi-turn agents.
  2. Modest isolated gain from group-aware priority, confounded sync baseline, and undisclosed start limit:
    • Despite the title's focus on group-aware scheduling, in the only end-to-end ordering ablation (7B async, Table 6), LASTCALL-FIFO already captures 66.1% of the total speedup (43.68h -> 35.57h), leaving 4.16h (9.5% vs. veRL) for group-progress priority; LASTCALL-FIFO is omitted at 14B/30B and in sync mode.
    • Sync veRL is strictly on-policy (0% stale groups), whereas Sync LASTCALL trains on 41.8% one-update-stale groups and generates 9.8% more candidates (Table 13a), relaxing synchronous on-policy semantics rather than isolating scheduling.
    • Algorithm 1 (line 7) and Appendix C.1 rely on a "configured start limit" per window, yet its numerical value (as well as the learning rate and repo-overlap audit between SWE-bench Full train and 300 held-out tasks) is never reported.
  3. Unexplained >10-point shift in Equal rewards (%), mid-run replay artifact, and 300s/update async weight transfer:
    • In Table 13(a), LASTCALL reduces the share of used groups with identical rewards (zero GRPO signal) from 34.8% -> 24.3% (sync) and 34.5% -> 22.6% (async)—likely explaining LASTCALL's higher pass@1 (+0.8 to +2.3 points in Table 6). Since patch-apply failures finish in 0.50s and should complete faster, why does LASTCALL admit far fewer equal-reward groups (e.g., via leaving 50.2% of failed executions unretried in Table 9b)?
    • The window sweeps (Table 11, Figure 4) use the mid-run replay where 98.8% (12,649/12,800) of results already exist and only 58 jobs execute, creating artifacts where a 1s window beats 480s (8,133s vs. 10,055s). Also, async weight transfer at 14B takes ~300s per update (59,915s total, 31% of runtime in Table 10b), an unusually high overhead that makes 30B async slower than 30B sync.
Questions for authors

NA

3 comments

Thank you for the careful review. We address each weakness below. In the tables, (new) marks results that are not in the original submission.

Environment preparation, generation settings, and hardware [Weakness 1]

Most of a verification is spending on environment preparing.The gain still holds when we remove it. We do not pre-warm environments in our end-to-end runs, so the 13% to 28% speedup in the paper applies to an uncached deployment. To estimate a cached one, we removed environment preparing from each job's recorded duration in the fresh replay of Table 4, which starts with all verification pending:

Generation and supply (s) ↓veRLLastCall-FIFOLastCall
Full durations24,462 (new)19,37216,845
Without environment preparing (new)14,38612,41911,479

Generation and supply (Table 4, §B.1) is the time, over 25 updates, to release each round's recorded rollouts and verify enough of them to fill complete batches.

LastCall still finishes 20.2% sooner than veRL (31.1% with full durations) and 7.6% sooner than LastCall-FIFO. A heavy tail also remains after caching. The slowest 49 of the 489 gold-patch tasks take 66.6% of test time (§3, Table 14).

  • We use single-call rollouts without thinking mode, at temperature 0.2. Thinking mode or multi-turn agents make generation a larger share of each update. So the gain depends on how long the learner still waits for verification, which is 65% of training time in 14B synchronous veRL (§8).
  • We run verification on the same two nodes as training. Each node has 256 logical cores (AMD EPYC 7742), about 1 TiB of memory, and eight 40 GB A100 GPUs. Each verification is a subprocess without a per-job CPU limit, up to 256 at a time per run, with its workspace on a Lustre file system. We use this configuration for every comparison. Table 14 was measured separately with at most four concurrent executions, so its tail reflects per-task test time rather than contention.

Gain from the ordering, staleness, and unreported settings [Weakness 2]

The window and its start rules account for about two thirds of the speedup, and the ordering for about one third. This holds in the 7B async run of Table 6. It also holds in two new controls at 14B, where we trained LastCall-FIFO end to end in sync and async mode (three runs each):

SettingveRL (h) ↓LastCall-FIFO (h) ↓LastCall (h) ↓Ordering's share of the saving
7B async43.68 ± 0.9235.57 ± 1.2831.41 ± 0.8933.9%
14B sync74.16 ± 0.3763.67 ± 0.53 (new)58.15 ± 0.4234.5%
14B async65.67 ± 2.8057.87 ± 1.73 (new)52.95 ± 2.1038.7%

Time to 200 updates, mean ± standard deviation over three runs. Ordering's share = (LastCall-FIFO − LastCall) / (veRL − LastCall).

We also ran a staleness-matched control, and most of the sync speedup does not depend on stale groups. In sync training at 14B, we set LastCall's staleness bound to 0, so it trains on no stale groups (three runs):

14B syncStale groups usedTime to 200 updates (h) ↓pass@1 at update 200 (%) ↑
veRL0%74.16 ± 0.3712.3 ± 0.33
LastCall, staleness bound 0 (new)0%62.98 ± 0.4913.6 ± 0.51
LastCall, staleness bound 142%58.15 ± 0.4213.7 ± 0.33

Mean ± standard deviation over three runs.

With staleness bound 0, LastCall trains fully on-policy and keeps 70% of its saving over veRL. Its pass@1 does not differ significantly from LastCall with staleness bound 1. LastCall generates 9.8% more candidate groups, but it generates them within the measured wall-clock time. So the reported savings already include their cost.

These are the settings:

  • We set the start limit to 8 per group per window in all runs. It never blocks a start, because a group has 4 rollouts and at most one retry for each, so it needs at most 8 starts.
  • We train the actor with a learning rate of 1e-6, constant without warmup, with AdamW (β = (0.9, 0.999), weight decay 0.01), gradient clipping at 1.0, and no critic.
  • We train on a random sample of SWE-bench Full, disjoint from the 300 evaluation tasks (§6.1). Both sets come from SWE-bench Full's 12 repositories. So the evaluation measures unseen tasks within the same repositories.

Equal-reward groups, the mid-run replay, and weight transfer [Weakness 3]

We do not count the pass@1 difference as a gain from scheduling (§6.2). The equal-reward shift also does not come from favoring fast groups. Groups whose patches all fail to apply finish first and have equal rewards. So favoring fast groups would raise the equal-reward share, but the share falls. You are right that fewer retries can leave groups unused. A rollout without a usable result that is not retried within the window leaves its group incomplete and unused (§B.2), and LastCall retries 49.8% of such rollouts against 98.1% for veRL.

The mid-run replay does favor short windows, but 480 s did not come from it. We set 480 s before the end-to-end runs, about 15% above the 416.2 s 90th percentile in §3. We also swept the window in the fresh replay, where every result must be produced. LastCall's time is flat around 480 s:

Generation and supply (s) ↓240 s (new)480 s720 s (new)960 s (new)
LastCall17,36816,84517,14917,454
LastCall-FIFO20,11519,37219,78320,262

Across these windows, LastCall's time varies by at most 3.6% and stays at least 13% below LastCall-FIFO's. The roughly 300 s per update that §6.2 and Table 10(b) report for this stage in async veRL and LastCall is the actor update, which trains on the 2,048 sequences of each update with FSDP and CPU offload. The NCCL broadcast of weights to the rollout GPUs takes only 3.5 to 8.4 s per update. We will name the stage accordingly. It takes about the same time in every async row of Table 10(b), so it favors no system. Once LastCall cuts the learner's wait, this update becomes the larger part of each async step. This helps explain why the two LastCall modes finish about equally fast at 30B-A3B (105.26 and 106.43 h, Table 6).

Thank you again for your detailed review. We have posted our responses.

As the discussion period is drawing to a close, we would be grateful if you could let us know whether our responses address your concerns. If any point remains unclear or would benefit from further evidence, we are happy to provide additional clarification or results.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.