- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper studies the scheduling of post-training for tasks where success verification is required, e.g., software-patching tasks where the proposed software patch is verified by a test suite (this is also the scenario used in the paper). The issue is that the post-training happens in batches, patching and verification are distributed to multiple workers, and a batch can only be trained when m verifications have finished. Moreover, training is only allowed to be based on the current or next iteration's weights, so learned updates that are too old need to be discarded. The authors propose to try to fill a batch as quickly as possible in order to minimize the downtime of the learner. Since the verification times are unknown, the approach is a best-effort heuristic. The proposed approach is evaluated in several LLM settings and demonstrates significant gains.
Critical strengths
- Strong experimental evaluation, improving over the SotA
Critical weaknesses
-
The description of the approach is not sufficiently clear. I still do not fully understand how to read Figure 1.
-
After reading the problem setup, I tried to think about how I would approach the problem. From what I understood, the approach is more or less the - rather simple - idea I came up with in 20 seconds. So it seems to me that the proposed solution is rather obvious.
-
It is unclear whom this paper is interesting for. I do not think any scientist faces the addressed problem as part of their workflow, so I can only imagine large AI labs as target groups. I cannot believe that they are not already doing something like the proposed approach.
Questions for authors
-
Could you explain why your approach is nontrivial?
-
The complexity remark on p. 4 that the special-case scenario with known durations is NP-hard is misleading, since that leads to a combinatorial problem. If the durations are unknown, the problem is stochastic and cannot be solved optimally. Why do you make this remark?
Thank you for the thoughtful review. We address the concerns and corresponding questions below. In the tables, (new) marks results that are not in the original submission.
Effectiveness of group-progress ordering [Weakness]
We have new results to show how the ordering contributes to the speedup. We trained LastCall-FIFO end-to-end at 14B in the same config in both modes (three runs each). Our new runs show that the ordering gives about a third of the saving, in both sync and async modes.:
Time to 200 updates, mean ± standard deviation over three runs. Ordering's share = (LastCall-FIFO − LastCall) / (veRL − LastCall).
Table 3 compares LastCall with veRL (instead of LastCall-FIFO), so it cannot separate the ordering from the window. Although its mean group-completion delay barely moves (234 s to 231 s), the median falls from 146 s to 127 s and the 95th percentile from 725 s to 675 s (Table 10(a)). The direct evidence is the 14B sync row above, where LastCall-FIFO takes 5.52 h longer than LastCall. We also ran a new replay ablation. On the fresh-replay workload of Table 4, we start from veRL and switch on LastCall's parts one at a time. Lower is better:
Other values are from Table 4.
Generation and supply (Table 4, §B.1) is the time, over 25 updates, to release each round's recorded rollouts and verify enough of them to fill complete batches. LastCall saves 7,617 s over veRL, and the three parts share it roughly equally. The window with deferral saves 2,567 s (row 3). The time check and stopping once the batch is full save 2,523 s (rows 4 and 5). The ordering saves 2,527 s (row 6), about as much as the window. Row 2 shows that the window needs deferral. If the groups still incomplete at the cutoff are discarded, the replay is slower than veRL. On the specific points:
Competing schedulers [Weakness]
group_w_priority is a baseline we built to examine the start rules. It favors nearly complete groups as LastCall does, but it has no time check and keeps starting jobs after the batch is full. So the reductions of at least 71% in verification work against it come mainly from these two rules. In Table 11(b), group_w_priority uses the time check and LastCall does not. So the two policies differ mainly in whether they stop once the batch is full, and LastCall does 65% less verification work. The replay ablation table above measures the same two rules without this baseline. Together they save 2,523 s of generation and supply in the fresh replay (rows 4 and 5). The ordering mainly saves time. It changes verification work by only about 1% (§B.2), in line with your 0.8%.
SRTF ranks jobs by remaining time. LastCall does estimate a job's duration, as the mean time of the completed verifications in its group (§4.2). But it uses this estimate only for the yes/no time check for the future group members, and the estimate does not exist until the group's first verification finishes. So it cannot rank every pending job by remaining time, as SRTF needs. We will try to add an SRTF baseline to the replay study in the revision.