- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a two-stage curation framework for SFT data. Offline, bilevel data selection (BDS, essentially SEAL) learns question-level weights that minimize loss on a trusted validation set. Online, the model being trained regenerates responses for a subset of questions, and these responses are weighted by an importance ratio . The paper motivates this ratio through a bilevel multi-objective (BMO) formulation that it shows is equivalent to BDS. The paper also provides supporting theory: an advantage of BDS over direct mixing, convergence for a one-layer Transformer, and the BMO--BDS equivalence. Experiments cover quality enhancement (OpenOrca/Alpaca-cleaned) and safety-aware fine-tuning (RedOrca with BlueOrca or HH-RLHF guidance) on Pythia-1B and Llama-3-8B.
Critical strengths
- Keeping harmful or low-quality questions while fixing their responses is a sensible idea, and Figure 2 illustrates it clearly.
- Losses and QA accuracy are reported over four seeds, with per-benchmark OOD results, a component-wise ablation, and runtime.
- The paper is generally easy to follow, and the algorithm is clearly summarized.
Critical weaknesses
- Weak motivation and limited novelty. Validation-guided data selection is already established by LESS (Xia et al., 2024) and PDS (Gu et al., 2025). Online self-refining generation closely follows self-training methods such as ReST (Gulcehre et al., 2023), STaR (Zelikman et al., 2022), and self-distillation fine-tuning (Yang et al., 2024). Diversity-aware data selection has also been studied, for example in DiSF (Fan et al., 2025) and DDCF (Hoang et al., 2026). The paper does not clarify what combining these components adds beyond these lines of work.
- Missing comparisons. The closest omission is PDS (Gu et al., 2025), which also optimizes data weights against a target loss through the training dynamics. GREATS (Wang et al., 2024), an online per-iteration selection method, is cited but not compared.
- Unsupported diversity claim. The paper argues that online self-refining generation “maintains question diversity,” whereas offline BDS discards questions. However, diversity is never measured for any method, so the central motivation remains an assertion.
- Outdated experimental setup. Experiments use only Pythia-1B and Llama-3-8B, which are small and lag behind current open models. Self-refining generation with such weak models is also hard to justify.
References
Fan, Z., Du, S., Hu, S., Wang, P., Shen, L., Zhang, Y., Tao, D., & Wang, Y. (2025). Combatting dimensional collapse in LLM pre-training data via submodular file selection. ICLR 2025.
Gu, Y., Dong, L., Wang, H., Hao, Y., Dong, Q., Wei, F., & Huang, M. (2025). Data selection via optimal control for language models. ICLR 2025.
Gulcehre, C., Le Paine, T., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., et al. (2023). Reinforced self-training (ReST) for language modeling. arXiv:2308.08998.
Hoang, L., Zhang, W., & Lu, W. (2026). Difficulty–diversity collaborative filtering for data-efficient LLM fine-tuning. ICLR 2026.
Shen, H., Chen, P.-Y., Das, P., & Chen, T. (2025). SEAL: Safety-enhanced aligned LLM fine-tuning via bilevel data selection. ICLR 2025.
Wang, J. T., Wu, T., Song, D., Mittal, P., & Jia, R. (2024). GREATS: Online selection of high-quality data for LLM training in every iteration. NeurIPS 2024.
Xia, M., Malladi, S., Gururangan, S., Arora, S., & Chen, D. (2024). LESS: Selecting influential data for targeted instruction tuning. ICML 2024.
Yang, Z., Pang, T., Feng, H., Wang, H., Chen, W., Zhu, M., & Liu, Q. (2024). Self-distillation bridges distribution gap in language model fine-tuning. ACL 2024.
Zelikman, E., Wu, Y., Mu, J., & Goodman, N. (2022). STaR: Bootstrapping reasoning with reasoning. NeurIPS 2022.
Questions for authors
Please see weaknesses.
Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. The proposed Assumption 3 appears to resolve my original objection, and the additional baselines strengthen the empirical evaluation. Overall, the rebuttal has addressed most of my original concerns. I have increased my score.