acceptodds
Under review as a conference paper at ICLR 2027

Bilevel Data Curation for LLM Fine-tuning: Offline Selection and Online Self-Refining Generation

Abstract

Supervised fine-tuning (SFT) datasets are critical to the downstream performance of large language models, yet they often contain low-quality or harmful question-response pairs. To improve SFT data quality, we develop a unified bilevel framework that combines offline data selection with the online self-refining generation. In the offline setting, bilevel data selection (BDS) selects question-response pairs from the offline SFT dataset to maximize the validation performance. We theoretically show that the optimal model given by BDS outperforms direct data mixing approach in useful data coverage. Moreover, we provide a global convergence analysis for gradient-based BDS approach for one-layer Transformer, showing that the -global optimum is achievable in finite time. Although efficient, offline BDS discards potentially harmful questions together with responses, thereby reducing question diversity. We address this limitation by refining the responses to selected questions using online self-refining generation framework. However, BDS is inefficient to update the response weights when responses are regenerated online. To address this issue, we introduce bilevel multi-objective optimization (BMO) for response-level weighting. We show that BMO recovers the same validation-aligned solution as BDS, but admits a closed-form response weights, and we use an empirically supported importance-ratio weight for online implementation. Experiments on LLM quality enhancement and safety-aware fine-tuning demonstrate that the proposed framework consistently improves both data quality and downstream fine-tuning performance.

Primary area:
optimization
Keywords:
bilevel optimization, supervised fine-tuning, data curation, data selection, online self-refining generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MrMYOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a two-stage curation framework for SFT data. Offline, bilevel data selection (BDS, essentially SEAL) learns question-level weights that minimize loss on a trusted validation set. Online, the model being trained regenerates responses for a subset of questions, and these responses are weighted by an importance ratio πθ/πold\pi_\theta/\pi_{\mathrm{old}}. The paper motivates this ratio through a bilevel multi-objective (BMO) formulation that it shows is equivalent to BDS. The paper also provides supporting theory: an advantage of BDS over direct mixing, convergence for a one-layer Transformer, and the BMO--BDS equivalence. Experiments cover quality enhancement (OpenOrca/Alpaca-cleaned) and safety-aware fine-tuning (RedOrca with BlueOrca or HH-RLHF guidance) on Pythia-1B and Llama-3-8B.

Critical strengths
  • Keeping harmful or low-quality questions while fixing their responses is a sensible idea, and Figure 2 illustrates it clearly.
  • Losses and QA accuracy are reported over four seeds, with per-benchmark OOD results, a component-wise ablation, and runtime.
  • The paper is generally easy to follow, and the algorithm is clearly summarized.
Critical weaknesses
  • Weak motivation and limited novelty. Validation-guided data selection is already established by LESS (Xia et al., 2024) and PDS (Gu et al., 2025). Online self-refining generation closely follows self-training methods such as ReST (Gulcehre et al., 2023), STaR (Zelikman et al., 2022), and self-distillation fine-tuning (Yang et al., 2024). Diversity-aware data selection has also been studied, for example in DiSF (Fan et al., 2025) and DDCF (Hoang et al., 2026). The paper does not clarify what combining these components adds beyond these lines of work.
  • Missing comparisons. The closest omission is PDS (Gu et al., 2025), which also optimizes data weights against a target loss through the training dynamics. GREATS (Wang et al., 2024), an online per-iteration selection method, is cited but not compared.
  • Unsupported diversity claim. The paper argues that online self-refining generation “maintains question diversity,” whereas offline BDS discards questions. However, diversity is never measured for any method, so the central motivation remains an assertion.
  • Outdated experimental setup. Experiments use only Pythia-1B and Llama-3-8B, which are small and lag behind current open models. Self-refining generation with such weak models is also hard to justify.

References

Fan, Z., Du, S., Hu, S., Wang, P., Shen, L., Zhang, Y., Tao, D., & Wang, Y. (2025). Combatting dimensional collapse in LLM pre-training data via submodular file selection. ICLR 2025.

Gu, Y., Dong, L., Wang, H., Hao, Y., Dong, Q., Wei, F., & Huang, M. (2025). Data selection via optimal control for language models. ICLR 2025.

Gulcehre, C., Le Paine, T., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., et al. (2023). Reinforced self-training (ReST) for language modeling. arXiv:2308.08998.

Hoang, L., Zhang, W., & Lu, W. (2026). Difficulty–diversity collaborative filtering for data-efficient LLM fine-tuning. ICLR 2026.

Shen, H., Chen, P.-Y., Das, P., & Chen, T. (2025). SEAL: Safety-enhanced aligned LLM fine-tuning via bilevel data selection. ICLR 2025.

Wang, J. T., Wu, T., Song, D., Mittal, P., & Jia, R. (2024). GREATS: Online selection of high-quality data for LLM training in every iteration. NeurIPS 2024.

Xia, M., Malladi, S., Gururangan, S., Arora, S., & Chen, D. (2024). LESS: Selecting influential data for targeted instruction tuning. ICML 2024.

Yang, Z., Pang, T., Feng, H., Wang, H., Chen, W., Zhu, M., & Liu, Q. (2024). Self-distillation bridges distribution gap in language model fine-tuning. ACL 2024.

Zelikman, E., Wu, Y., Mu, J., & Goodman, N. (2022). STaR: Bootstrapping reasoning with reasoning. NeurIPS 2022.

Questions for authors

Please see weaknesses.

Reviewer_Rz5ZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets data selection in SFT of LLMs. Given a small high-quality dataset and a larger low-quality one, it aims to train only on the low-quality samples that benefit the target task. Building on bilevel data selection (BDS), which weights each question–response pair, it proposes online self-refining the answers. That is, for a subset of the low-quality questions, the current model regenerates the responses, which are weighted by a closed-form importance ratio derived from a bilevel multi-objective (BMO) formulation. This keeps question diversity while replacing low-quality responses. The paper gives theory for BDS and BMO under a separable-data assumption, and experiments on quality enhancement and safety-aware fine-tuning with Pythia-1B and Llama-3-8B-Instruct.

Critical strengths
  1. An important problem and a well-motivated idea. Refining responses instead of discarding whole pairs keeps the question diversity that offline selection loses, which matters for practical SFT data curation.
  2. Consistent gains over offline selection. Online refinement improves over offline BDS on both tasks and models, showcasing this method's improved performance.
Critical weaknesses
  1. The method is buried, making it hard to read and understand.

    • The paper's own method starts at the end of p.5, after two pages of theory for offline BDS. Presenting the algorithm first, and saying what each theorem implies for the method, would help.
    • Under Assumption 1, the lower-level objectives share a minimizer and BDS, BMO and validation-only training all reach the same validation optimum. The theory thus explains why BDS beats direct mixing, but not why the low-quality data helps over the trusted data alone, which is their actual objective. The conditions are also not checked: ρc\rho_c is never estimated, and the lower-level losses in Table 1 (0.84–0.91 Llama, 1.37–1.62 Pythia) are far from the zero loss of Assumption 1.
    • Lemma 4 relies on an assumption that cannot hold where it is applied. Its proof (Lemma 23, Eqs. 61 and 64) needs a model with zero loss on all GG responses to the same question, but two different responses cannot both have probability 1. The derivation therefore fails whenever G≥2G \geq 2.
    • The response weights do not measure quality, they are the model's own likelihood, i.e. confidence-weighted self-training, a reasonable heuristic. But no validation, reward or judge signal enters them, so the claims that BMO "uses the validation set to select the responses that best align with the target distribution" (e.g., p.2) should be revised.
  2. The experiments do not isolate what the online component adds.

    • Some baselines and related work are missing. The online part is a form of self-training, but it is compared only with offline selection and mixing. Two baselines would isolate it: (i) regenerating the masked responses once, offline, with the model trained on the trusted data, then running BDS; (ii) a self-training baseline (e.g. DFT, SPIN, SDFT, RAFT). Related self-improvement methods are not discussed: SPIN (Chen et al., 2024), RAFT (Dong et al., 2023), SDFT (Yang et al., 2024), and DFT (Guo et al., 2025).
    • AlpacaEval covers only three methods with no std, why?
Questions for authors

Check weaknesses, especially:

  • How was ρ\rho chosen for direct mixing? Only ρ∈{0.5,1}\rho \in \{0.5, 1\} is reported in Table 1. Please report mixing with the BDS schedule or with tuned ρ\rho.
  • How are the different hyperparameters (for baselines and for BDS) selected?
  • In (8) the low-quality term has total weight γ/N\gamma/N for any ω\omega. What is the value of γ/N\gamma/N? L_val is computed as an average over its data. However, the average of each of the two subsets of SFT data are weighted by sigmoid(ω\omega) and γ\gamma, what is the scale of this value? This could make BDS nearly validation-only. How are the weights normalized in practice?
  • How many steps on the trusted data does each method take, and is validation-only step-matched with BDS?
  • Which responses define the lower-level losses in Tables 1 and 3? What is meant by "selected-data" quality. If online methods are scored on their own regenerations, this does not show better data quality.

Clarifications:

  • How much of the low-quality data does each method use (the budget of Random, LESS, and DSIR; for BDS and online, statistics of the learned weights)?
  • What are NN and N′N' in each experiment? Please introduce the datasets with the same notation used in the method, i.e., SFT and VAL notation.
1 comment

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. The proposed Assumption 3 appears to resolve my original objection, and the additional baselines strengthen the empirical evaluation. Overall, the rebuttal has addressed most of my original concerns. I have increased my score.

Reviewer_tYJVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To better utilize a SFT dataset (with harmful samples) and a small trusted validation set, the work adds a new online self-refining generation step on top of bilevel data selection (BDS). To be specific, for a subset of questions, the original responses are regenerated by the current model and then weighted by an importance ratio that is derived from bilevel multi-objective optimization (BMO). The work also theoretically proves that offline BDS beats direct mixing (Theorem 1) and converges in finite time for a 1-layer Transformer (Theorem 2). Evaluated on quality benchmark (OpenOrca / Alpaca-cleaned) and safety benchmark (RedOrca), the online method outperforms offline BDS, has hier mean OOD QA accuracy, and lower harmfulness.

Critical strengths
  • the online method is consistenly better than offline BDS and the cost of regenerating responses per question is pretty cheap (adding only 2 hrs on Llama-8B in Table 2).
  • Theorem 2 is a useful theoretical step, which gives a finite-time convergence guarantee for gradient-based offline BDS on a one-layer softmax Transformer. Most prior works cover only simple architectures like linear networks.
Critical weaknesses

My main concern is that the online method's gain may not come from response refinement. Compared with offline BDS, the online method changes the following 2 things at once:

  1. the masked question's original responses are discarded.
  2. the model's own generated reponses are used.

The paper assumes the performance gain comes from the second, and the first setting was not ablated.

My arguments are as follows:

  • Without the importance ratio, regeneration is even worse than offline BDS (Table 9: 0.82 vs. 0.80). So the regenerated responses are not helpful by themselves. I believe this is because the older samples (regenerated every 500 iterations) are pulling the model back toward pi_old.
  • With the importance ratio, the loss drops to 0.78, this is likely because the ratio cancels the pull. But this doesn't mean the regenerated responses are useful.
    • w/ the ratio, the update on the regenerated responses equals to a PPO update where every response has the same advantage, which gives no learning signal on average.
    • when temp=1, there is no learning signal.
    • when temp=0.8 (Appendix D.1), this makes model more confident in its own reponses,
  • So it is unclear whether the performance gain over offline BDS (Table 9: 0.78 vs. 0.80) comes from better responses, or simply from discarding the original ones.
Questions for authors

Could you add one row to Table 9 in which the original responses of the masked questions (R = 10%) are discarded but no responses are regenerated? This could raise my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.