acceptodds
Under review as a conference paper at ICLR 2027

Train with Hindsight: Post-hoc Checkpoint Selection atop Deterministic Computing

Abstract

The training of large language model(LLM) involves substantial steps, yet the optimal checkpoint for evaluation or next training stage can exist inside the training trajectory rather than at its endpoint. Fixed-interval checkpointing couples the set of selectable states to a priori what to save before that trajectory is observed, while denser storage and evaluation increase checkpoint I/O, storage, and validation costs. We introduce Post-hoc Checkpoint Selection (PhCS), a ”train with hindsight" framework that decouples checkpoint selection from physical checkpoint storage. In the original training, PhCS saves parameters and optimizer states periodically as anchors and records lightweight trajectory metadata and input per step, after which post-hoc selection policies identify candidate states on the realized trajectory. Atop deterministic computation, reproduction of candidate states begins from the nearest preceding anchor and replay the previous input along the same numerical path. We instantiate PhCS in SFT-RL pipeline: static selection in SFT covers the entire trajectory, combining loss, token accuracy, and entropy, whereas dynamic selection in GRPO conducts recursive exploration on validation set. PhCS thereby turns checkpointing from a precommitted storage schedule into a post-hoc decision over the realized training trajectory. We evaluate PhCS in ReTool framework, a representative SFT-Agentic RL scenario; the experiments demonstrate that PhCS achieves higher pass@4 than conventional training paradigms on multiple datasets such as AIME24/25/26 and AMC23 , and achieves superior equivalent storage efficiency.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Model, Deterministic Inference and Training, Post-training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1NZnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes PhCS, which separates checkpoint selection from checkpoint storage. During training, full checkpoints (weights and optimizer state) are saved at a fixed interval, and the input and metrics of every step are logged. After training, a selection rule picks a step, and that model is rebuilt by replaying training from the nearest saved checkpoint. Exact replay relies on deterministic kernels shared by vLLM and TorchTitan. For SFT, the rule combines training loss, token accuracy and entropy. For GRPO, it narrows an interval using AIME24 pass@4. Experiments use a ReTool SFT-then-GRPO math pipeline with Qwen3-4B-Instruct, Qwen3.5-4B and Ministral-3-8B. PhCS reports higher pass@4 than the best interval checkpoint and the final checkpoint on most benchmarks, and a storage saving of over 51x with less than 2x time overhead.

Critical strengths
  1. The problem is practical. RL training curves are noisy, and the best checkpoint is often somewhere in the middle of training. Separating what is stored from what can be selected is a clean way to frame this, and the authors are open that their GRPO search has no optimality guarantee.
  2. The system work is solid. The authors make SFT and agentic GRPO training bitwise reproducible for several recent models, including Qwen3.5 and Ministral-3, and release the code. This could be useful beyond checkpoint selection, for example for debugging RL runs.
Critical weaknesses
  1. The experiments do not fully answer the main question of the paper: is an unsaved step picked after training better than the best checkpoint saved at a fixed interval? Table 1 compares whole pipelines, and judging from Fig. 7 and Fig. 8, the Best Interval and Final Step rows also start GRPO from different SFT checkpoints, so each row is a different RL run. There is one run per setting, each AIME set has 30 problems, and pass@4 uses only four samples. On the held-out AIME sets the differences are one to three problems, with no error bars, and one goes the other way (AIME26 for Qwen3.5-4B: 0.467 for PhCS vs. 0.567 for Final Step). In the ablation, PhCS solves 21 of the 60 AIME25/26 problems, and final-step SFT with early-stopped GRPO solves 20. The case study suggests the answer may be no, at least for GRPO. In Fig. 9(b), the selected step 149 sits on the same 25-step grid as the saved checkpoints, and all four replayed steps around it score lower. The released code saves checkpoints at zero-based steps 24, 49, and so on, so step 149 seems to be a normally saved checkpoint. The selected Qwen3-4B step (174) looks the same. For these two models, the replay found nothing better than what interval saving already keeps, so their GRPO-stage gains cannot come from recovering unsaved steps. The SFT picks do fall between saved checkpoints, but they are never compared with the best saved SFT checkpoint under the same rule.
  2. The main results include the validation set. AIME24 is used to select the GRPO checkpoint (Section 4.1). Table 1 marks it as the validation set, but it is still reported next to the test sets, and the biggest gain in the paper is on it (0.733 vs. 0.500). The abstract also lists AIME24 among the improved benchmarks. Since PhCS evaluates more candidates on the same 30 problems, it should score higher there even if the selected model is not better. The same checkpoint also gets 0.60 on AIME24 in Fig. 9(b) but 0.733 in Table 1, and the paper does not explain the gap. Scores used for selection should not be presented as evidence of improvement. A validation set that is separate from all test sets is needed.
Questions for authors
  1. Within the same SFT and GRPO runs, how does the selected unsaved step compare with the best saved checkpoint under the same selection rule? Please report this on benchmarks not used for selection, over several seeds, with error bars.
  2. Are the selected GRPO steps 149 (Qwen3.5-4B) and 174 (Qwen3-4B) saved checkpoints? How is the Best Interval checkpoint chosen at each stage?
  3. How does PhCS do when the validation set is separate from all reported test sets? Why does the selected checkpoint score 0.60 on AIME24 in Fig. 9(b) but 0.733 in Table 1?
Reviewer_BkgnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a technique named PhCS for reducing checkpoint storage by saving fewer checkpoints, at the same time offering the capability of post-hoc selection of intermediate training checkpoints. PhCS periodically saves checkpoints and records step-level training inputs and enforces deterministic computation, which allows it to reconstruct any intermediate checkpoint by replaying the training steps from the nearest preceding saved checkpoint. The paper also designs two checkpoint selection strategies for SFT and GRPO, respectively. Experiments show that PhCS achieves better performance than fixed-interval and final-checkpoint baselines on most evaluated datasets.

Critical strengths

The idea of recovering unsaved intermediate checkpoints is interesting.

Critical weaknesses
  1. The motivation is not convincing enough. In order to save storage, PhCS consumes meaningfully more compute resources. It is unclear whether the storage savings justify the additional computation costs.
  2. The proposed checkpoint selection strategies for SFT and GRPO are not sufficiently convincing. For SFT, the authors record training metrics at every step but still save checkpoints at fixed intervals, then select and reconstruct the best checkpoint post-hoc. The paper does not explain why maintaining and saving the best candidate checkpoints during training would not be a simpler alternative. Similarly, for GRPO, the authors do not demonstrate that post-hoc selection is more efficient than performing more frequent validation during training and saving the best checkpoint under the same computational budget.
  3. The strict alignment requirements may limit inference-side optimizations, such as quantization, and lead to higher resource consumption, which is not adequately discussed or evaluated.
  4. AIME24 is used for GRPO checkpoint selection and is also included in the final evaluation results. This introduces information leakage.
Questions for authors
  1. Could the authors clarify whether the storage savings justify the additional GPU computation costs?

  2. For SFT, could the authors explain the advantage of post-hoc reconstruction over saving the best candidate checkpoints during training? Similarly, for GRPO, how does the proposed approach compare with more frequent validation during training and saving the best checkpoint?

Reviewer_vQuyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Post-hoc Checkpoint Selection (PhCS). During training it stores full-state anchors at a fixed interval together with per-step records, and it uses deterministic kernels so that an unsaved step can later be reproduced from the nearest preceding anchor. Two policies choose which step to reproduce: for SFT, a filter on per-step loss and token accuracy followed by the highest mean entropy; for GRPO, an interval-shrinking search driven by AIME24 pass@4. Experiments use a ReTool SFT-then-GRPO pipeline on Ministral-3-8B, Qwen3-4B and Qwen3.5-4B, with pass@4 on AIME24/25/26, AMC23, MATH500, OlympiadBench and Minerva, plus storage and time overheads and determinism checklists.

Critical strengths
  1. Decoupling which checkpoints can be selected from which checkpoints were saved is a practical idea, and the paper is explicit about what determinism requires (kernels, reduction order, data order, random state, parallel topology).
  2. The deterministic stack covers both FSDP training and vLLM rollouts, with architecture-specific paths for Qwen3.5 GDN and Ministral-3 attention (Appendices A and B), and the paper links anonymous source code (lines 500 to 502).
Critical weaknesses
  1. The evaluation does not support the performance claims. AIME24 is the validation set used to select the GRPO checkpoint (line 379, and the "validation set" label in Table 1), yet the main text highlights the AIME24 result for Qwen3.5-4B, "a 46.6% relative improvement over 0.5 of Best Interval" (lines 395 to 396). The same checkpoint (GRPO step 149, lines 389 to 390) has an AIME24 pass@4 of 0.60, 18 of 30 prompts, during selection (lines 458 to 460) and 0.733 in Table 1, so the two reported evaluations of one checkpoint differ by four prompts. Table 4 lists a maximum response length of 4,096 for training-time evaluation and 8,192 for final evaluation, but the paper does not say which setting the selection probes used. Each cell in Table 1 is a single value, with no variance or confidence interval. The paper says PhCS "achieves the best or comparable performance to both Best Interval and Final Step across all three models and the majority of benchmarks" (lines 392 to 393), but on Qwen3.5-4B AIME26 Final Step is 0.100 higher than PhCS (0.567 vs 0.467), twice the AMC23 gain the paper highlights (0.975 vs 0.925), and the text does not discuss this case. The conclusion also says PhCS "achieves state-of-the-art on multiple datasets" (lines 484 to 485), while the only baselines are the base model and checkpoints chosen by other rules (fixed interval, final step, and early stopping in Section 4.4) within the authors' own SFT-GRPO pipeline; no other checkpoint-selection method (UGCS is cited in Section 2.2 but not evaluated) and no externally reported result is compared.

  2. The reproducibility guarantee that the method depends on is checked only at the unit level. The Ministral checklist "uses a small Ministral configuration" (line 1050); its test preconditions specify "a fixed 2 × 16 token input" (lines 1057 to 1059), and its vLLM rollout check uses a single 4092-token prompt and generates 4 tokens (lines 1104 to 1105). In addition, some Qwen3.5 GDN checks use tolerances rather than equality: the comparison of the reference operator with an independent per-token implementation uses atol=1e-12 and rtol=1e-12 (lines 1126 to 1128), and the PyTorch numerical gradient check uses eps=1e-6, atol=1e-5 and rtol=1e-4 (lines 1137 to 1138), although Section 4.5 says both checklists "require strict element-wise equality" (line 447). The paper does not report any check in the SFT or GRPO runs that reproduces a step from an anchor and compares it with the original state, for example by replaying from one anchor to the next stored anchor. The guarantee is also limited to the validated model and kernel paths and the same software stack, device configuration and parallel topology (lines 355 to 357), and to the same hardware environment (lines 504 to 506).

Questions for authors
  1. How is Best Interval selected: by AIME24, or separately for each benchmark?
  2. How was the initial GRPO search interval chosen (steps 125 to 175 for Qwen3.5-4B in Section 4.6), and on which data?
  3. Can you report results over several training seeds and repeated evaluations, with AIME24 left out of the reported comparison?
  4. Appendix D says the rollout bundle stores completions, token ids, log-probabilities and other fields "for reward computation, GRPO update, and deterministic reproduction". Is it stored at every GRPO step, and is it included in the equivalent-storage ratio of Figure 6? That ratio is measured against saving a checkpoint at every step (Section 4.3); how does PhCS compare with a denser fixed interval at the same time budget, given a time dilation of less than 1.98×?
  5. Section 4.2 says the total number of steps was 250, while Table 3 lists 1,000 GRPO optimizer steps, and Figure 9b shows steps up to 250. Do the two numbers count different things (for example, rollout steps versus optimizer updates), or was training stopped at step 250?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.