acceptodds
Under review as a conference paper at ICLR 2027

Dr. Post-Training: A Data Regularization Perspective on LLM Post-Training

25 views

Abstract

LLM post-training faces a critical challenge: effectively leveraging scarce, high-fidelity target data alongside abundant but imperfectly aligned general training data, a problem that existing methods frame as data selection. We introduce **Dr. Post-Training** (**D**ata-**R**egularized Post-Training), a framework that instead treats general training data as a data-induced regularizer that stabilizes learning from the scarce target objective. Specifically, our framework proposes that at each training step, construct a feasible set of model update directions that depends on the general training data alone, and project the model update direction specified by the scarce target data onto that feasible set. Standard training and existing data selection methods arise as special cases with different choices of the data-induced regularizer, and these methods correspond to different points on a bias-variance spectrum with different regularization strength. The framework thus provides a principled lens for understanding these tradeoffs and guiding the design of new methods. Building on this view, we propose a family of methods that expose the regularization strength as an explicit design choice, and a theory that ties this choice to how well the general data covers the target signals and how much target noise it lets through. For practical LLM-scale use, we introduce careful system optimizations that realize these methods at a per-step cost on par with standard training. Experiments across SFT, RLHF, and RLVR follow the predicted tradeoff: the best regularization strength tracks the mismatch between the general and target data, with our methods gaining most where the target is diluted in the general data and standard training remaining competitive where the two are aligned.

Primary area:
interpretability and explainable AI
Keywords:
Data Selection, LLMs, Post-Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_g6XsOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Dr. Post-Training, which reexamines data selection in LLM post-training from a data-regularization perspective. The authors unify existing methods through a projection framework and introduce Group-Wise Subset Update. The paper also provides bias-variance analysis, convergence proofs, and an efficient implementation, with evaluations across SFT, RLHF, and RLVR.

Critical strengths
  1. The paper interprets data selection from a data-regularization perspective and unifies different training strategies within a feasible-set projection framework. This perspective is insightful and also suggests new ways to design data-selection methods.
  2. The authors characterize different update strategies using bias and exposed variance, and connect one-step errors to convergence. This provides a theoretical basis for understanding training stability when target data are scarce.
  3. The paper not only proposes a method but also optimizes gradient computation and memory overhead. The experimental design is also fairly thorough.
Critical weaknesses

Main Concerns

  1. GAST [1] also uses target-gradient alignment to select different training samples for different layers, but this work is not discussed. The two methods differ in selection strategies and system implementations. I suggest that the authors discuss this work and compare the methods under matched training conditions. This would clarify the algorithmic contribution and the settings where different selection strategies are appropriate.
  2. The paper's main theoretical guarantees are based on exact projection, but the implementation uses a Top-k approximation and ignores the gradient-redundancy term in the projection objective. In addition, the theoretical analysis considers direct gradient updates, while the experiments use AdamW. Under the paper's setting of (n = 8, k = 4), exact selection involves only 70 candidate combinations.
  3. The authors use ablations on partition granularity and target-set size to support the theoretical conclusions, but they do not directly measure the proposed bias and exposed variance. Thus, the current experiments mainly demonstrate that the method is effective, rather than sufficiently validating its theoretical mechanism. Minor Concern The evaluated models are mainly in the 1B-3B range, and the system tests use relatively controlled configurations. I suggest adding end-to-end training results on larger models.
Questions for authors
  1. GAST uses a gradient-alignment-based sampling strategy, whereas this paper mainly uses Top-k. Have the authors compared their performance and overhead under the same training budget? In which settings is Top-k preferable?
  2. Under the setting of (n = 8, k = 4), how much do the Top-k selections differ from exact projection? Does this difference affect downstream performance?
  3. Can the authors provide empirical estimates of bias and exposed variance? [1] Yao, K., Song, Z., Wu, K., Zhong, M., Cheng, D., Tan, Z., Ji, Y., & Gao, P. (2026). GAST: Gradient-aligned Sparse Tuning of Large Language Models with Data-layer Selection. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics, 4401-4416.
1 comment

Dear Reviewer g6Xs,

As the author–reviewer discussion period draws to a close, we would appreciate your feedback on our responses and whether they address your concerns. We welcome any further questions or points for clarification. Thank you for your time and consideration.

Reviewer_f534OpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper treats general training data not just as a pool of samples to select from, but as a regularizer for post-training. Target data gives the desired update direction. General data defines a “safe” set of allowed updates. The actual update is the target gradient projected onto this set. This creates a bias–variance tradeoff: a smaller allowed set gives stronger regularization and more stability but more bias; a larger set follows the target better but can expose more target noise.

Critical strengths
  1. The motivation is clear, and the methods are well-suited to the problem. Different layers may benefit from different samples, and selecting subsets by layer provides a meaningful extension to traditional global data selection.

  2. The validation is comprehensive. The paper combines theoretical analysis, system implementation, and a variety of post-training experiments, and reports results using multiple random seeds as well as the applicable boundaries; the partial gains are particularly significant when the data and the objective do not align.

Critical weaknesses
  1. The experimental models selected are too outdated and have too few parameters, making it difficult to determine whether the conclusions are applicable to large-scale training.

  2. It remains unclear how to select the appropriate configuration in practical applications. The paper emphasizes that the optimal regularization strength depends on the differences between the general data and the target task; however, there is still a lack of clear guidance on how users should determine whether to use global sampling or stratified sampling for a new task. In particular, when the number of target samples is small, their representativeness may influence the selection results, and the associated sensitivity warrants further clarification.

Questions for authors
  1. In reasoning and instruction-following tasks, does the method described in this paper still offer advantages compared to fine-tuning using only target data or training using a mixture of target and general data?

  2. Without relying on test set results, how should users decide whether to use global sampling or stratified sampling? Are there any metrics available based on the training data to aid in this decision?

  3. What was the rationale for selecting the base models used in the paper’s experiments?

  4. Have extrapolation experiments been conducted for models with larger parameter sizes? How does this method perform in scenarios with large parameter sizes?

2 comments

Dear Reviewer f534,

As the author–reviewer discussion period draws to a close, we would appreciate your feedback on our responses and whether they address your concerns. We welcome any further questions or points for clarification. Thank you for your time and consideration.

Thanks, my concerns are addressed. I am raising my score to weak accept.

Reviewer_1fb5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors study the problem of leveraging general data to improve the performance on a target domain. They develop a framework that views this problem as projecting the gradient from the target data onto a set spanned by that from the general data, and show that several existing approaches can be viewed as special cases in the framework (different choices of the sets). Then they develop convergence theory for the proposed framework. To reduce the computational overhead, they propose efficient implementation strategies. Finally, they conduct experiments on LLM SFT, RLHF and RLVR to demonstrate the effectiveness of the approach.

Critical strengths
  • The paper is well motivated, and the problem is important.
  • The idea of viewing existing approaches as projection is novel and interesting.
  • The experiments are comprehensive and convincing
Critical weaknesses
  • My main concern is the theory-reality gap:
    • At Line 162 the authors claimed that 'Existing methods are tractable approximations' of the projection framework. But the claim is not trivial. The authors refer the readers to Appendix C.2, which does not provide concrete examples on how existing methods (such as top-kk) fit into this framework.
    • In Theorem 2.2 (and Corollary C.11), the authors assume that btb_t and ζt\zeta_t (defined in Assumption 3 in Appendix C) are equal to 0, which seems to be a strong and unrealistic assumption.
Questions for authors
  • How can we interpret existing approaches using the projection framework?
  • Is the assumption in Corollary C.11 strong?
  • What is the performance of the Target-Only baseline in RLHF and RLVR?
1 comment

Dear Reviewer 1fb5,

As the author–reviewer discussion period draws to a close, we would appreciate your feedback on our responses and whether they address your concerns. We welcome any further questions or points for clarification. Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.