acceptodds
Withdrawn from ICLR 2027

FAER: Auditable Utility-Aligned Trajectory Replay for Language Model Post-Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Replay selectors often rank cached trajectories by format feedback, confidence, freshness, or response length, although cache-level correctness and downstream learner utility are distinct objectives. We formalize this selection-to-learning gap and introduce FAER as an auditable full-trajectory replay framework. Its training-free fixed selector is a protocol baseline; FAER-UTILITY is the learner-aware selector fitted on disjoint calibration blocks. The normalized gradient alignment is reported as a baseline, while a disposable optimizer-aware virtual update supplies a magnitude-aware utility surface. The audit contract freezes observed fields and replay traces before evaluation labels are joined. On GSM8K with Qwen2.5-1.5B-Instruct, the matched learner study reports quality 0.6329 for the fixed selector, compared with 0.5482 for uniform and 0.6037 for format-feedback under 128 updates. Metadata-only cross-fitted calibration reaches over eight seeds (median 0.6481; paired 95% interval ) at 63,276 target-run tokens; its recorded full cost is 189,642 tokens and 3.48 GPU-hours including calibration. The completed FAER-UTILITY row reaches 0.6624 at 62,844 target-run tokens and 4.26 GPU-hours. Format-feedback selects records with correctness 0.6953, compared with 0.3594 for the fixed selector, despite the different downstream ranking. The completed comparison surfaces report the learner-aware ablation, same-seed gap, policy-optimization rows, and strict zero-shot transfer.

Primary area:
optimization
Keywords:
Large Language Models, Reinforcement Learning, Post-Training, Trajectory Replay, Data Selection, Utility Alignment, Efficient Learning, Auditable Machine Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RAtuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents an offline method in which a language model is SFT’d on a reweighted dataset of on-policy samples. The authors propose and combine various reweighting methods.

Critical strengths

None

Critical weaknesses

The paper reads in a rather roundabout way, where it is not clear upfront that the method is fully offline, unlike the related work. Thus, it is unclear whether their proposed problem space “Language-model post-training increasingly reuses generated trajectories to allocate a limited optimization budget” is relevant. The paper does not compare their method to the base model itself or standard methods that can be applied to verifiable settings like GSM8K, such as SFT on the dataset filtered to successes (i.e. a single iteration of STaR [1]), more standard on-policy methods like GRPO [2], or newer off-policy RL methods [3] (which the paper appeared to pitch itself as from the introduction). Thus, it is hard to evaluate whether the proposed methods provide valuable insights over existing ones.

Moreover, all experiments are run using nonstandard settings (generation length of 512, greedy decoding), producing subpar evaluations ~66% as the best experimental result compared to the official base model evaluation at ~73% [4], which invalidate all of the results. It would also be helpful to see error bars on all results, as the numbers are fairly close in the ablations.

[1] https://arxiv.org/abs/2203.14465 [2] https://arxiv.org/abs/2402.03300 [3] https://arxiv.org/abs/2602.19362 [4] https://qwen.ai/blog?id=qwen2.5-llm

Questions for authors

None

Reviewer_ZzArOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes FAER, a framework for selecting cached trajectories for language model post-training. It combines trajectory metadata with estimated learning utility to guide replay selection. Experiments on GSM8K show improvements over uniform sampling and heuristic selection methods.

Critical strengths

None

Critical weaknesses

The paper is difficult to follow because the targeted training setting and the meaning of “replay” are not clearly established. Although the authors provide a problem formulation, it remains unclear whether the main method targets SFT or RL. Equation (11) suggests an SFT objective, but the source of the supervised targets is ambiguous: are they cached model-generated responses, as “trajectory replay” suggests, or ground-truth GSM8K answers? If the former, the authors should explain how incorrect responses are handled; if the latter, they should clarify how the method differs from training-example selection.

The motivation for replay also needs clarification. The paper largely assumes that cached trajectories should be reused without explaining why this setting is appropriate for GSM8K post-training. Is the goal to improves sample efficiency, reduces generation cost, or fit within a limited GPU-time budget? Clearly stating this objective and explaining the advantages over standard training would make the contribution easier to assess.

Questions for authors

Please see the weaknesses.

Reviewer_qhJwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. FAER distinguishes two objectives that are often conflated: selecting cached trajectories with high correctness and selecting trajectories that improve a downstream learner. It measures their disagreement using policy rankings and matched learner outcomes.
  2. The framework commits selector-visible fields and replay traces before joining evaluation labels. It includes a fixed metadata-based selector and a learner-aware selector calibrated on disjoint blocks, with gradient-alignment and optimizer-aware virtual-update scores discussed as utility signals.
  3. The main study uses Qwen2.5-1.5B-Instruct on GSM8K with a small selected replay set and a 128-update horizon. The appendix adds utility ablations, diversity controls, total-compute matching, and frozen-selector transfer to other tasks and a larger Qwen model.
Critical strengths
  1. The distinction between cache correctness and learning utility is worth studying. Format-feedback selects substantially more correct cached records than fixed FAER, yet produces lower downstream quality. The same-seed analysis makes this observation more credible by tying the selected records to the learner actually trained on them.
  2. The information boundary is specified carefully. Disjoint calibration blocks, frozen selected IDs, trace digests, and label-mutation checks make the intended separation between selection and evaluation concrete. This is useful for interpreting learner-aware selection, where calibration choices can otherwise be difficult to disentangle from target evaluation.
  3. Several controls address plausible alternative explanations. Shuffled utility tests trajectory-specific information, diversity matching checks whether subset composition explains the gain, and Table 42 includes the cost of fitting the selector. Frozen-selector transfer is also a useful addition, since it avoids attributing target-domain retuning to generalization.
Critical weaknesses
  1. The main utility algorithm is not described consistently. Section 3.3 says the completed learner rows use normalized alignment, and Eq. 9 uses that feature. Section 4.2 instead calls the 0.6624 row optimizer-aware, while Table 41 attributes the same result to virtual AdamW plus norm correction. These alternatives have different computational requirements and motivations, so identifying the actual headline method is essential.
  2. The strongest practical comparison lacks the uncertainty needed to judge its size. Under total-compute matching, utility replay improves over fixed FAER by 0.0119 and over FreshPER by 0.0178, but Table 42 provides no intervals. Table 36 similarly gives transfer point estimates despite mentioning seed-level uncertainty. These comparisons matter more for assessing utility fitting than the much larger gain over uniform replay at a target-only budget.
  3. The central learner experiment remains a small, highly reused replay setting: 32 selected records are cycled through 128 updates. Table 43 usefully varies set size and reuse, and the transfer results broaden task coverage, but they do not establish behavior with a continually refreshed cache or changing producer policy. This matters because a utility score computed at a frozen checkpoint may become less informative as the learner changes.
Questions for authors
  1. Which exact score produces the main FAER-UTILITY row? Please reconcile Eq. 9, the statement about normalized alignment, and the virtual-AdamW result in Table 41.
  2. What are the paired confidence intervals for the total-compute-matched comparisons and the frozen-selector transfer results? How many independent runs support each?
  3. In the reported learner runs, is utility recomputed as the checkpoint changes, or is the initial ordering retained throughout? Is there evidence showing how that choice affects longer training or refreshed replay?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.