acceptodds
Withdrawn from ICLR 2027

Clean Data, Unsafe Model: Certified Safety Curation for LLM Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A fine-tuned model needs safe examples of harmful prompts as much as it needs few harmful examples, and judge-based filters remove exactly those. We bring to fine-tuning data a distribution-free certificate: given any scorer and a few hundred human labels, it returns a cutoff whose kept set is at most an -fraction harmful with probability , or refuses. On BeaverTails its guarantee holds and a closed form predicts its label budget. A certified set then exposes what else a filter does. At 0.5B, a judge-selected set certified at most percent harmful trains a more harmful model than the pool, because the judge scores a response by its prompt and discards the demonstrations. On an aligned 8B model the same judge, asked SAFT's question, keeps a set more harmful than its pool and is worse than no filtering, where which examples the scorer keeps decides the harm. A scorer that confuses demonstrations with harmful examples cannot both purify a set and keep them, so the property to certify is a harmful fraction within each kind of prompt. Certified that way at 0.5B, a moderation model's selection matches training on every human-safe example at a fifth of the labels. Spectral and bilevel filter scores refuse under the same certificate.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, fine-tuning, data curation, distribution-free guarantees, LLM safety

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ZknHOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper applies an existing statistical certificate to bound the fraction of harmful responses in filtered fine-tuning data. It studies how filtering removes safe responses to harmful prompts and proposes certifying harmful-prompt and benign-prompt groups separately.

Critical strengths

The manuscript reports certification failures, fallback selections, and settings where stratification worsens results, making these failure cases visible.

Critical weaknesses
  • The writing makes the paper extremely difficult to understand. The abstract and introduction compress methods, results and causal explanations into dense sentences, repeatedly switching between properties of the dataset and behavior of the trained model. Basic claims require substantial reconstruction before they can be assessed. I find this presentation frustrating: the reader should not have to piece together what is being proposed, what is guaranteed and what the experiments actually establish. For example, the abstract contains: "We bring to fine-tuning data a distribution-free certificate: given any scorer and a few hundred human labels, it returns a cutoff whose kept set is at most an α-fraction harmful with probability 1−δ, or refuses". No human reader can interpret what this sentence means unless they read the entire paper and appendix and then come back to the abstract and read it over and over again! That is not how an abstract should be!

  • The results do not show benefits of the proposed method. Table 2 changes the label budget from 400 to 800 and replaces a 10% harmfulness target with separate 20% and 15% targets while introducing stratification. These changes prevent attributing the improvement to stratification, which also performs worse on the aligned 8B model in Table 3. Safety evaluation relies entirely on automatic classifiers; Table 20 shows similar credit-limit advice receiving opposite safety labels without a clear improvement in the response. Meanwhile, unchanged ROUGE-L on 229 prompts does not establish unchanged helpfulness: word overlap does not measure factual accuracy or usefulness. Matched comparisons and human validation are needed.

  • The claims about when certification can succeed need correction. Proposition 2 incorrectly states that reliable certification is impossible when the harmful fraction equals the allowed limit. For example, a set with 10 harmful examples out of 100 satisfies a 10% limit, and checking every example confirms this. The interpretation of failed certification is also too strong. Even with all labels known, the procedure stops after its first selection fails: keeping the top 15 examples could include 2 harmful ones (13.3%), while keeping the top 20 could include the same 2 harmful ones (10%). The first selection fails, but the second meets the requirement and is never checked. Therefore, failure of this procedure does not establish that the scorer cannot produce a qualifying selection.

Questions for authors
  1. Effect of stratification: Which reported comparison separates the effect of stratification from the additional labels, looser harmfulness targets, and changed training-set size in Table 2? What evidence attributes the improvement specifically to stratification?

  2. Meaning of the evaluation results: What substantive safety difference explains the opposite labels for similar credit-limit advice in Table 20? What evidence supports the claim of unchanged helpfulness beyond ROUGE-L similarity to reference answers?

  3. Proposition 2: Should the impossibility condition require the harmful fraction to be strictly greater than the target? At equality, labeling the entire finite pool should allow certification. Would this correction affect any proofs or reported label-budget calculations?

  4. Interpretation of certification failure: For the reported failures, was it verified that no candidate threshold meets the harmfulness target, or did the procedure simply stop at its first failed test? These outcomes support different conclusions about the scorer's ability.

Reviewer_YMbcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies filtering finetuning data for safety with a scorer and a small number of human labels. It  imports the certificate from Haroon & Fleming (2026) to provide guarantees on the harmful fraction of a selected training set. The main result is that a judge-selected set with ≤10\leq 10\\% harmful data trains a more harmful model than the unfiltered dataset with 25% harmful examples. The paper proposes certifying harmful-prompt and benign-prompt strata separately. Finally, the proposed method is evaluated on LLMs with 0.5B and 8B parameters, compared with numerous baseline methods.

Critical strengths
  • The paper studies the effect of harmful data in finetuning datasets on model alignment, which is an important topic in LLM Safety. Additionally, the implication that lowering the fraction of harmful prompts might hurt alignment is interesting and non-trivial.
Critical weaknesses
  • The methodological novelty is limited, as the certificate is imported directly from Haroon & Fleming (2026), who developed it for safe offline RL. The theoretical results are fairly direct. Additionally, the paper's own results show that composition and harm are not fully correlated, as in Figure 5 the four sets with ≤10\leq 10\\% harmful examples can span 0.13, and in Table 8 two sets at 8B which are identical in size and harmful fraction differ by 0.10.
  • In Table 3, the "labeled-safe only" method is the best-performing method by a wide margin. In the same setup, the stratified Llama Guard 3 condition is worse than the pooled one, and similar order is observed on the PKU-SafeRLHF pool with 0.5B model in Appendix H. This puts the general utility of the stratified methods into question.
  • On a similar note, the authors point out that on an aligned model, harm rises with the amount of finetuning even on sets with zero harmful examples. This introduces a confound in Table 3, where different methods curate the dataset to different sizes, mixing up effects from curation method and curated data size.
Questions for authors
  • Can you identify additional properties of a prompt that will better increase the correlation with the downstream impact of the finetuned model. For example, can we have downstream performance depend on both harmful fraction and some continuous level of harm they induce, rather than a 0-1 value? Is it possible that some prompts are inherently more harmful than others?
  • At 0.5B the labeled-safe-only sets perform worse than contaminated sets, which suggests a data-starved base model. Do the results hold in settings when clean data is sufficient? Also, the labeled-only row in Table 1 uses 200 labels while the certified rows use 400, can you report label-matched comparisons?
  • Is it possibly the case that the finetune data is simply not a good match for the aligned 8B model? If finetuning on clean data only harms the model, then the data might simply not be appropriate for finetuning the 8B model.
Reviewer_yfkyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how filtering fine-tuning data affects model safety. It applies an existing statistical certificate to bound the harmful fraction of a selected dataset and examines how filtering can remove safe responses to harmful prompts. It proposes separate certification for harmful-prompt and benign-prompt groups and evaluates the resulting models across several experimental settings.

Critical strengths
  1. The paper examines a practically relevant failure of automated data filtering for safe fine-tuning.

  2. The controlled experiments provide useful evidence about how changes in training-data composition affect model safety.

Critical weaknesses
  1. The paper does not explain why its final harmfulness limits are appropriate for safe fine-tuning. These limits are relaxed after stricter settings fail to certify. The experiments do not establish the benefit of choosing these particular limits.

  2. The version without stratification has lower reported harm on both the aligned 8B model and the second dataset. Stratification uses twice as many labels in these comparisons. The main experiment showing an improvement also changes the label budget and harmfulness limits, so it does not establish that splitting the data caused the improvement.

  3. The paper lacks comparisons with key baseline experiments. LARF [1] identifies and filters seemingly safe training examples that can harm model safety. This directly relates to the paper’s finding that safety depends on which examples are retained.

[1] Hao Li et al. Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment. EMNLP, 2025.

  1. In this paper, switching to Llama Guard 3 alleviates the removal of safe responses to harmful prompts. The added benefit of stratification over using this more suitable scorer is not established.
Questions for authors

Which part of the stratified certification is methodologically new? Please distinguish it from applying the existing certificate to each group and combining the error bounds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.