acceptodds
Under review as a conference paper at ICLR 2027

Safety Risk of Fine-Tuning Data Is Not a Property of the Sample Alone

13 views

Abstract

Large language models are safety-aligned before deployment so that they refuse harmful requests, and they are then commonly fine-tuned for downstream tasks. However, such fine-tuning can weaken safety alignment even when the training data are entirely benign. Existing defenses filter the training data by estimating the safety risk of each sample from internal model signals, such as hidden representations, sample gradients, or parameter updates. These methods typically score samples once at the initial model state and retain the resulting selection throughout fine-tuning, which implicitly assumes that the initial risk ranking remains informative as the model changes. We show that this assumption does not hold. On a controlled fine-tuning trajectory, risk rankings computed at initialization are nearly uncorrelated with those computed after two epochs. A matched intervention experiment further shows that the same groups of benign samples cause different degrees of safety degradation depending on the model state from which they are learned. Safety risk is therefore a joint property of the sample and the model state rather than an intrinsic property of the sample. We also find that rankings are rewired mainly within the first epoch. At initialization, gradient-based risk scores are dominated by a gradient component nearly orthogonal to the safety direction, and this failure persists under larger adapters, full-parameter gradients, and a short probe epoch. We therefore propose SABA, which re-evaluates sample risk at the current model state in each training round and formulates filtering as the allocation of a fixed exclusion budget across rounds. The resulting concave allocation problem reduces to a geometric schedule controlled by a single deferral parameter: no budget is spent at initialization, and exclusions shift toward later rounds with more reliable risk estimates. Under the same training budget, SABA reduces the attack success rate from 63.3% to 18.3% on CatQA and from 53.1% to 14.4% on HEx-PHI, outperforming nineteen baselines while preserving utility, and the gains carry over to a second corpus and a second model family.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM safety, Safe fine-tuning, Data selection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_WWB8OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how the safety impact of benign fine-tuning data changes with the model state. It measures changes in risk rankings and tests the same matched sample groups from early and later checkpoints. It then proposes SABA, which updates risk scores and allocates an exclusion budget across training rounds. The main experiments show clear safety gains over static and refresh-only filters, with similar utility on the reported metrics. The paper also studies a second corpus and model family and reports cases where its scheduling rule fails.

Critical strengths

The paper connects a useful diagnosis to a practical method. The matched-group intervention gives evidence beyond changes in scores alone. On Dolly, all 16 reported interaction estimates are positive, and 15 intervals exclude zero. Under the same filtering budget, SABA lowers ASR by 16.1 and 9.3 percentage points compared with refresh-only on the two benchmarks with Llama-Guard-3. The ablations help separate the effects of re-scoring and budget allocation. The negative results on Alpaca and the failures of the initialization check also make the limits of the method clear.

Critical weaknesses
  1. Two protocol details need clarification. Section 5.1 and Table 1 state a budget of 12,000 sample-passes for every arm, but three full rounds over all 5,000 samples would give 15,000 for no filtering. The matched intervention also lacks explicit training doses and optimizer-state rules for its four cells. These gaps limit verification of these controls, although the comparisons among filtered arms remain supported.

  2. Some claims need a narrower scope. Table A6 shows that the tested repairs fail, mainly for GradShield; it does not establish that initialization information is generally unrecoverable. The allocation model also needs a per-round capacity constraint or a stated range where it is inactive. Its strict-interior claim needs correction because kappa = 1 gives delta = 1. These theory corrections do not change the reported comparison among filtered methods.

Questions for authors
  1. What are the actual sample-passes for the no-filter arm? Please clarify its budget and report the training dose and optimizer reset or inheritance rule for each matched-intervention cell.

  2. Can you limit the initialization-repair conclusion to the tested scorers and repairs, and clarify how it relates to the informative static SQSD results?

  3. Can you state the capacity conditions for the allocation result and correct the strict-interior claim at kappa = 1?

1 comment
AuthorsOpenReview ↗

Response to Reviewer WWB8

Dear Reviewer WWB8,

Thank you for the careful review, in particular for catching the protocol details below. We are glad that you found that the paper connects a useful diagnosis to a practical method, that the matched intervention provides evidence beyond changes in scores, and that the ablations and the reported failure cases make the limits of the method clear.

W1(a) & Q1: The sample-passes of the no-filter baseline.

A1: We agree with the reviewer, and correcting it does not change the comparison among filtering methods. In the submission, the no-filter baseline trained for three full epochs (15,000 sample-passes), while every filtering method trained on 12,000. We added a volume-matched baseline that trains 0.8 of an epoch per round (12,000 sample-passes; Table 2, Table A22; CatQA / HEx-PHI, Guard):

Settingno filter (12,000)no filter (15,000)SABA schedule
Dolly · Qwen3-8B61.3 / 46.263.3 / 53.118.3 / 14.4
Alpaca · Qwen3-8B43.1 / 45.144.5 / 45.520.7 / 22.0
Dolly · Llama-3-8B74.2 / 57.173.7 / 61.359.2 / 41.1

Matching the volume changes the CatQA ASR of no filtering by at most 2 points and lowers its HEx-PHI ASR by 0.4 to 6.9 points. Against the volume-matched baseline, SABA lowers the ASR by 32 to 43 points in the main setting, and its gains over refresh-only (16.1 and 9.3 points) are unchanged.

Revision. Table 2 now reports no filtering at both 12,000 and 15,000 sample-passes, and Table 3, the abstract and the text compare with the 12,000-pass baseline. Each remaining use of the 15,000-pass baseline is labeled (Table 5, Figure 4c, Figures A3 and A4, Tables A26 and A28, judge agreement and utility tests).

W1(b) & Q1: Training dose and optimizer state in the matched intervention.

A2: The runs from EE and LL are identical except for the starting adapter. Both use a learning rate of 5×10−65 \times 10^{-6} with a cosine schedule and 10% warm-up, 10 epochs, an effective batch size of 8, the same seed, data order and matched samples. Each cell is a fresh run, so the AdamW state is re-initialized and the warm-up restarts at both states. On Qwen3-8B with seed 0, each group holds 20 to 23 samples and is trained for 30 optimizer steps from both states.

Revision. Appendix E.4 now states this protocol.

W2(a) & Q2: Scope of the repair result and its relation to the static SQSD results.

A3: We limited the conclusion to the tested scorers and repairs. The five repairs in Table A7 fail for GradShield, and three of them (larger LoRA rank, full-parameter gradients, removing leading principal components) also fail for SCOPE and He-grad (Table A19). For SCOPE, the correlation with its θ2\theta_2 ranking is 0.273 before the repairs and 0.012 to 0.276 after.

This is consistent with the static SQSD results. SQSD projects each update onto a safe and an unsafe direction, which cancels most of the common gradient component, so its initial ranking already does better than no filtering (static SQSD reaches 44.8% CatQA ASR, against 61.3% without filtering and 62.4% for static GradShield). Its AA is 0.23, below 0.3, so it is not deferred. Its ranking still changes after initialization (rank correlation 0.14 with the ranking after the first round), so the second check replaces it once. A ranking can therefore change a lot during training and still do better than no filtering at the start.

Revision. Section 3.3 and Appendix H.2 now state the repair result only for the tested inner-product scorers, and Section 3.3 notes that SQSD's initial ranking already does better than no filtering.

W2(b) & Q3: Capacity constraint and the κ=1\kappa = 1 case.

A4: We made both corrections. The allocation problem (Eq. 3) now includes 0≤kt≤N0 \leq k_t \leq N, where ktk_t is the number of samples excluded in round tt and NN the pool size, and Proposition 2 gives the capped water-filling solution. Proposition 1 now requires that the final, largest round stays below NN, which holds in all our runs (largest ktk_t = 4,500, in the B=6,000B = 6{,}000 budget ablation, below N=5,000N = 5{,}000). The text now states that δ=κ−1/(1−α)\delta = \kappa^{-1/(1-\alpha)} lies strictly inside (1,∞)(1, \infty) only for κ∈(0,1)\kappa \in (0, 1): permanent exclusion (κ=1\kappa = 1) gives the even split δ=1\delta = 1, and κ→0\kappa \to 0 gives δ→∞\delta \to \infty.

Revision. Sections 4.1 and 4.2 (Eq. 3, Proposition 1) and Appendix C (Proposition 2) contain these corrections.

Thank you again for the careful reading. We are happy to clarify anything further.

Reviewer_v5VLOpenReview ↗
Rating
3(+1)
Confidence
2(−2)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a filter-based solution to harmful fine-tuning. This paper proposes an interesting observation that the most "destruction samples" that needs to be filtered should be dynamic along the training process of the model. Then the authors propose SABA, which re-evaluates sample risk along the model state and fitler those "destruction samples".

Critical strengths
  1. The direction to explore sample harmfulness along the training trajectory is interesting.

  2. The experiment is kind of comprehensive.

Critical weaknesses
  1. Paper is not well-written and I highly suspect the paper is written by AI. Figure 4 is too complicated and I don't think humans can understand it. There are some AI generated statement that human cannot understand, and I don't think human will write like this e.g.,

"If the safety signal is absent at θ(0), no postprocessing there should recover it, and none does."

" SABA fixes the amount of training, and therefore B, so that excluding more samples in one round requires excluding fewer in another."

  1. The novelty is limited. SABA in different rounds uses the existing filtration criterion (e.g., Gradshield) to evaluate at the current checkpoint. The contribution is a budget allocation for sample filtration each round.

  2. The design of the budget allocation is complicated and comes out of nowhere and with no insights. It seems to be some budget allocation just discounting along the time.

Questions for authors
  1. Although it is stated in the statement, I want to ask how AI tool is used in this paper. Is the budget allocation method fully generated by AI? Is the paper fully generated by AI?
3 comments
AuthorsOpenReview ↗

Response to Reviewer v5VL (1/2)

Dear Reviewer v5VL,

Thank you for your review and for your comments on the presentation. We understand the concern about papers that are written by AI, and we want to state clearly that this one is not. The research question, the analyses, the method and all experiments are our own, and the AI Use Statement discloses every use of AI tools. We give the details in our answer to your question at the end.

W1: Writing quality, AI-like sentences, Figure 4 and the two quoted sentences.

A1: We agree that several passages were hard to read, and we rewrote them. The overview figure (now Figure 3) reads top to bottom in three rows with far fewer formulas: one round of filtered fine-tuning, the findings of Section 3, and the two parts of SABA. The two sentences you quoted now read:

Old: "If the safety signal is absent at θ(0)\theta^{(0)}, no postprocessing there should recover it, and none does."

New: "We tested five ways to improve the GradShield ranking at θ(0)\theta^{(0)}, namely redrawing the initialization, larger LoRA ranks, full-parameter gradients, projecting out the leading components and a short pilot run on 500 samples. None of them brings the ranking closer to the ranking at θ(2)\theta^{(2)}."

Old: "SABA fixes the amount of training, and therefore BB, so that excluding more samples in one round requires excluding fewer in another."

New: "SABA keeps the total number of sample-passes fixed. By Equation (2), this also fixes the total number of excluded samples BB, so excluding more samples in one round requires excluding fewer in the others."

Revision. Figure 3 is redrawn, and Sections 3.3 and 4 are rewritten.

W2: Novelty: SABA reuses existing scorers, and the contribution is a budget allocation.

A2: Applying an existing scorer at the current checkpoint is our refresh-only baseline, and SABA outperforms it clearly. Both re-score the whole pool with the same scorer in every round. The only difference is how many samples each round excludes: refresh-only excludes k=(1000,1000,1000)k = (1000, 1000, 1000), SABA k=(0,750,2250)k = (0, 750, 2250). This alone lowers the CatQA ASR by 7.7 to 17.4 points (Llama-Guard-3, GradShield scorer):

Settingrefresh-onlySABA schedule
Dolly · Qwen3-8B, five seeds36.1±5.036.1 \pm 5.018.7±1.2\mathbf{18.7 \pm 1.2}
Alpaca · Qwen3-8B35.420.7
Dolly · Qwen3-14B19.011.3
Dolly · Qwen3-8B, six epochs40.826.7
Dolly · Llama-3-8B67.259.2

On Llama-3-8B, SABA's check does not defer GradShield's budget, so SABA re-scores it once after the first round and keeps that ranking (67.4). With SQSD, which the check does defer there, SABA reaches 63.6, below refresh-only with either scorer (67.2 and 65.3). A blinded human audit confirms the gap on Dolly (23.5% vs. 12.0% human ASR). Besides the schedule, the paper contributes the finding that gradient scores at initialization, where existing filters score, mostly measure length and gradient norm (Section 3), and the check that decides whether a scorer should defer its budget (Section 4.3).

Revision. Table 3 now compares SABA with refresh-only in five settings, and the contributions in the introduction describe the finding, the allocation and its schedule, and the check.

(continued in part 2/2)

AuthorsOpenReview ↗

Response to Reviewer v5VL (2/2)

(continued from part 1/2)

W3: "The budget allocation is complicated, comes out of nowhere, and seems to discount along time."

A3: Each of the three parts of the allocation follows from Section 3. First, nothing is excluded at initialization, because there the GradShield ranking follows response length. Second, later rounds get more, because excluded samples return to the pool, so an early exclusion is partly undone while a final-round exclusion protects the final model. Third, returns diminish, because a ranked filter removes the most harmful samples first. Spreading the budget evenly over all three rounds raises the CatQA ASR by 16.1 points, and spreading it evenly over the last two rounds by 10.2 (Table A10). You are right that this is a discount over time, but in the opposite direction from existing filters, which commit their whole selection at initialization. Table 4 supports this direction: in a five-round variant at equal training volume and scoring cost, schedules that spend more than half of the budget in the final round reach 26.6–29.5% CatQA ASR, the others 37.0–42.4%. Putting the whole budget in the final round does about as well (28.2%), so what matters is the first two parts, and the geometric form is one way to write them with a single parameter.

Revision. Section 4.1 now derives each part of the allocation from the analysis of Section 3, and Section 5.3 reports the comparison of allocation shapes.

Q: How were AI tools used? Is the budget allocation method or the paper fully generated by AI?

A4: No. The research question, the analyses, the method and all experiments are our own. We are not native English speakers and used an LLM for grammar and sentence-level checks of our drafts. An LLM-based coding agent implemented and ran experiment scripts from our specifications, which is the "execution" entry on the submission form; we verified all of its outputs. As the AI Use Statement says, AI was not used for research ideation, method or experiment design, or interpretation of results.

We were therefore very discouraged to read that the paper and the method may have been generated by AI. The analyses and the proposed method are our own work, and we do not think they should be dismissed on this basis.

Following your comment, we found and rewrote the passages that read this way. We had not known that these phrasings are perceived as AI-generated, because they have become common in recent papers. We also checked our manuscript with WaveDetect [1], a recent state-of-the-art document-level detector of machine-generated text, which estimates its AI-generated share at 8.1%.

We would be glad to discuss any remaining questions.

References

[1] Liu et al. "WaveDetect: Robust Framework for Machine-Generated Text Detection via Wavelet Transform". Findings of ACL 2026.

Thanks for the rebuttal. I could not understand this paper because most of the content is AI-generated (far more than the level of a grammar check). I suggest that the authors write every single word themselves before resubmission.

Reviewer_7zCHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the following question: is the safety risk of benign fine-tuning examples intrinsic to each sample, or does it depend on the model state in which the sample is learned? In a controlled Qwen3-8B LoRA trajectory, the authors report that SQSD risk rankings at initialization and after two epochs have a Spearman correlation of 0.076, and a matched intervention suggests that the same groups of benign examples can cause different safety degradation at different model states. They propose SABA, which first checks if the initialization ranking appears uninformative and then selects a filtering branch: the delayed schedule re-scores the pool at later model states and allocates a fixed exclusion budget geometrically, whereas other settings may use a static or delayed-static branch. Under a matched training-sample-pass budget, SABA reduces attack success rates on CatQA and HEx-PHI relative to the strongest reported baselines, and the reported utility measures remain close to baseline. Experiments cover Dolly and Alpaca data and Qwen3-8B and Llama-3-8B-Instruct settings.

Critical strengths
  1. The paper tests more than whether risk rankings drift: its matched intervention is designed to test whether the safety effect of the same sample groups changes with the model state. This makes the central claim more consequential than a ranking-correlation observation alone.
  2. The main comparison controls for the total number of training sample-passes and covers a broad set of static and dynamic filtering baselines. It also includes multiple risk scorers, two safety judges, ablations, and additional results across corpora and model families. In the primary setting, the reported ASR reductions are substantial, and the reported utility metrics remain close to baseline.
Critical weaknesses
  1. The evidence remains bounded to 8B models fine-tuned with LoRA, two instruction datasets, and direct harmful-request benchmarks judged by two automatic evaluators. The paper's broad framing about sample risk would be more convincing with evidence on larger models, full-parameter fine-tuning, longer trajectories, or a wider safety evaluation. The authors acknowledge much of this limitation.
  2. SABA's practical gains depend on assumptions about how ranking fidelity and benefit persistence evolve across rounds, and the main configuration fixes the deferral ratio at δ=3. The initialization check is based on correlations with gradient norm and response length and does not cover every reason an initial ranking might be uninformative. The paper includes sensitivity and branch-selection analyses, but it remains unclear how reliably these choices transfer to settings outside those tested. Re-scoring the full pool also adds material computational cost.
Questions for authors
  1. Could the authors report a small blinded human audit of the safety outputs, or otherwise quantify agreement between the two automatic judges and human assessment? This would help establish how much the reported ASR reductions reflect meaningful changes in refusal safety rather than judge-specific behavior.
  2. Given that the per-sample risk scores could also be used to select benign data that weakens safety alignment, what release or access safeguards do the authors plan for the per-sample scores and associated artifacts? How do they weigh reproducibility against this dual-use risk?
  3. What evidence distinguishes the proposed geometric allocation from a generally effective delayed budget split? With only two funded rounds in Figure 5a, the allocation shape cannot be identified. Could the authors compare geometric and non-geometric, later-weighted allocations over at least three funded rounds, matching training volume and scoring passes, or clarify that the experiments validate deferral while the geometric form remains a modeling choice?
  4. Could the authors provide per-seed utility results and uncertainty for the differences from no filtering and the main static baselines? Please clarify whether the evidence supports descriptive similarity of the means or preservation within a practically acceptable loss. A non-significant difference with only three training seeds would not by itself establish preservation.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 7zCH (1/3)

Dear Reviewer 7zCH,

Thank you for your thoughtful review and for the specific experiments you proposed. We are glad that you found that the matched intervention tests whether the safety effect itself changes with the model state, and that the main comparison controls the training volume and covers a broad set of static and dynamic baselines, scorers and judges.

W1: Evidence is limited to 8B LoRA models, two datasets and two automatic judges; larger models, full-parameter fine-tuning, longer trajectories or a wider safety evaluation would help.

A1: We added a 14B model, a six-epoch run and a wider safety evaluation, and SABA keeps the lowest ASR among the fine-tuned models in all three.

  • Larger models. Qwen3-14B uses the main protocol. Before any training run, the check gave A=0.79A = 0.79 for GradShield (the larger rank correlation of its initial scores with gradient norm or response length; see A3), well above the threshold of 0.3, and selected the schedule. SABA reaches 11.3% CatQA ASR, against 19.0% for refresh-only, 33.0% for static GradShield and 30.5% without filtering (Table 3), and SABA < refresh-only < static GradShield holds under both judges and on both benchmarks. Its held-out NLL is within 0.003 of the other methods, and its ROUGE-L is the highest (Table A24).

  • Full-parameter fine-tuning. We could not run it in the discussion period, and the Limitations say so. For GradShield, full-parameter gradients give the same initial ranking as LoRA gradients (ρ=0.998\rho = 0.998), and static filtering on them is level with no filtering (62.1% vs. 61.3%; Table A7), so the failure at initialization is not specific to LoRA scoring.

  • Longer trajectories. On Qwen3-8B, each of the three rounds trains for two epochs instead of one (six epochs in total), with the rounds, the exclusion budget and the scoring states unchanged. SABA reaches 26.7%, against 40.8% for refresh-only and 68.4% without filtering (Table 3, Table A24).

  • Wider safety evaluation. On StrongREJECT, SABA has the lowest ASR of the fine-tuned models (19.5%, against 31.1% for refresh-only and 60.7% without filtering at 15,000 passes), and it has the lowest compliance with unsafe XSTest prompts (40.5%, against 50.7% and 78.0%). On OR-Bench it refuses more borderline prompts (23.1% vs. 15.1% for refresh-only). The blinded human audit (Q1) adds a judgment beyond the two automatic judges.

  • Datasets and benchmarks. We did not add a third instruction corpus or adaptive jailbreak attacks. The Limitations state that our evaluation uses harmful-prompt benchmarks and that adaptive jailbreaks address a different threat model that would complement these results.

Revision. Table 3 (Section 5.2), Table A24 and Table 5 (Section 5.5) report these results, and the Limitations discuss full-parameter training, the two corpora and adaptive attacks.

W2(a): The assumptions on fidelity and persistence, and whether δ=3\delta = 3 transfers.

A2: $\delta = 3 $ is not the best value on every corpus, but the δ=3\delta = 3 schedule with GradShield outperforms every baseline in all five settings of Table 3, by a small margin on Llama-3-8B. On Alpaca, the CatQA ASR is 29.8%, 20.7%, 17.8% and 13.0% for δ=1,3,6\delta = 1, 3, 6 and ∞\infty (Table A25), so deferral helps on both corpora, but the best δ\delta differs (3 to 6 on Dolly, ∞\infty on Alpaca). The schedule needs only two properties: the initial ranking is uninformative for the scorers the check flags, and an exclusion's benefit fades once the samples return to the pool (κ<1\kappa < 1). Their exact values only set δ\delta. In a five-round variant on Dolly (Table 4), all four schedules that spend more than half of the budget in the final round reach 26.6–29.5%, whatever their exact shape, and the others 37.0–42.4%.

Revision. Section 5.3 and the Limitations now state that the best δ\delta depends on the setting and that δ=3\delta = 3 is set in advance.

(continued in part 2/3)

AuthorsOpenReview ↗

Response to Reviewer 7zCH (2/3)

(continued from part 1/3)

W2(b): The initialization check does not cover every reason a ranking may be uninformative, and it is unclear how its choices transfer.

A3: We agree that the check targets one specific failure. At initialization, the gradients of benign samples share a large common component, so a gradient-based score can rank samples by gradient norm or length instead of safety. The check computes AA, the larger absolute Spearman correlation of the initial scores with gradient norm and with response length. For a gradient-based scorer with A≥0.3A \geq 0.3, SABA uses the schedule; otherwise the scorer stays static, and a second check replaces its ranking once after the first round if it has changed (rank correlation below 0.3). Across the thirteen gradient-based scorer–setting pairs, every pair with A≥0.59A \geq 0.59 is assigned correctly. The three misrouted pairs have intermediate AA (0.20 to 0.50), and their selected branch still improves on no filtering. The check also selected the right branch on Qwen3-14B before any training run.

Revision. Appendix G.7 and the Limitations now state that AA detects one failure mode.

W2(c): Re-scoring the full pool adds computational cost.

A4: The cost is real, but our comparison with the strongest dynamic baselines is compute-matched. The two re-scoring passes take 2,731 s per training run, about 39% of the run's computation. Refresh-only and memory-ratchet re-score at the same states, so our comparison with them is at equal cost, and the 16.1-point gain over refresh-only does not come from extra scoring. A cheaper variant scores only once, at θ1\theta_1, and reuses that ranking in the final round. Its scoring takes 1,022 s instead of 2,731 s; it is less safe than SABA (32.7% vs. 18.3% on CatQA) but still beats the best baseline (34.4%) on three of four judge–benchmark pairs (Table A13).

Revision. Section 5.3 and Appendix G.5 report this cost.

Q1: A blinded human audit and judge–human agreement.

A5: Humans and both automatic judges agree that SABA is the safest and refresh-only second. Two members of our team labeled 400 responses (50 per method and benchmark) with method, seed and judge hidden, and agree on 91.7% (Cohen's κ=0.82\kappa = 0.82). Human ASR, averaged over the two annotators:

MethodHuman ASR
no filter (15,000 passes)53.0
static · GradShield51.0
refresh-only · GradShield23.5
SABA · GradShield12.0

On these 400 responses, the gap between no filtering and SABA is 41 points under human labels and 41 under Llama-Guard-3. The annotators agree with Llama-Guard-3 at Cohen's κ=0.83\kappa = 0.83–0.920.92 and with Beaver at 0.700.70–0.840.84, and the two judges rank the 21 configurations of the main setting almost identically (Kendall τ=0.92\tau = 0.92 and 0.930.93).

Revision. Table 5 and Appendix H.9 report the audit and the agreement.

(continued in part 3/3)

AuthorsOpenReview ↗

Response to Reviewer 7zCH (3/3)

(continued from part 2/3)

Q2: Release safeguards for the per-sample scores, and reproducibility vs. dual-use risk.

A6: The per-sample scores will be released only through a gated repository, and everything else openly. Access requires an institutional account and agreement to research-only use without redistribution. The code, aggregate results, frozen matched sets (one to two dozen samples per group) and human labels will be public. The scores can be recomputed with our code and public models, so withholding them would mainly slow down defenders, while gating limits the spread of a ready-made ranking.

Revision. The Ethics Statement now describes this release policy.

Q3: Geometric vs. other later-weighted allocations over at least three funded rounds, at matched training volume and scoring passes.

A7: We agree with your reading: the experiments validate concentrating exclusions late, and the geometric form is a modeling choice. All eight schedules in Table 4 use T=5T = 5, 12,000 sample-passes and re-scoring in every round, and six of them fund at least three rounds (CatQA ASR, Guard):

Schedulek1,…,k4k_1, \dots, k_4ASR
geometric δ=3\delta = 375, 225, 675, 202526.6
geometric δ=2\delta = 2200, 400, 800, 160029.5
geometric δ=6\delta = 612, 69, 417, 250228.4
final round only0, 0, 0, 300028.2
uniform750, 750, 750, 75037.0
linear300, 600, 900, 120038.2
skip one, then uniform0, 1000, 1000, 100037.2
last two rounds0, 0, 1500, 150042.4

Schedules that spend more than half of the budget in the final round reach 26.6–29.5%, the others 37.0–42.4%. Final-round-only does about as well as the geometric ones.

Revision. Section 5.3 and the Limitations now present the geometric form as one way to concentrate exclusions late with a single parameter.

Q4: Per-seed utility, uncertainty, and whether utility is preserved or only similar on average.

A8: Held-out loss and ARC-Easy pass equivalence tests against every comparator, so the evidence supports preservation within a pre-set loss. A mean and standard deviation over three seeds show that the means are close, but, as you note, they do not show preservation. We therefore fixed margins before the per-sample analysis (NLL increase ≤ 0.02 nats per token, about 1.2% of SABA's NLL; ARC-Easy drop ≤ 1 point; ROUGE-L drop ≤ 1.5 points) and tested each difference against them. Per-seed differences, SABA minus comparator:

ComparatorΔ\DeltaNLL (margin ≤+0.02\leq +0.02)Δ\DeltaARC-E, points (margin ≥−1.0\geq -1.0)
no filter (15,000 passes)+0.008 / +0.009 / +0.009−0.1 / 0.0 / +0.1
static · GradShield+0.004 / +0.005 / +0.007+0.6 / +0.7 / +0.5
static · DS-CSS (best static)−0.006 / −0.004 / −0.002−0.6 / −0.3 / −0.6
refresh-only · GradShield+0.004 / +0.005 / +0.004+0.2 / +0.3 / 0.0

NLL and ARC-Easy are equivalent in all 11 comparisons (no filtering, refresh-only and nine static filters; four are shown above). ROUGE-L differs by at most 0.9 points and is equivalent at the sample level in 10 of 11, but at the seed level only in 4 of 11, because it varies across seeds.

Revision. Table A29 and Appendix H.10 report the equivalence tests, and the abstract now says that "held-out loss and accuracy stay within pre-specified equivalence margins" instead of claiming preserved utility.

Please let us know if any of these points needs further detail.

Reviewer_xBapOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • Fine-tuning can weaken the safety alignment of an LLM, even if the training data are benign. Common defenses rank training samples by safety risk before fine-tuning and remove the riskiest ones. These methods implicitly assume that the ranking stays valid during fine-tuning, and this paper tests that assumption.
  • Re-scoring the samples during fine-tuning shows that the ranking changes mainly in the first epoch.
  • A matched intervention uses two model states, one after half an epoch and one after two epochs, and two groups of samples ranked in opposite order at these states. Training on each group from each state shows that the harm gap between the groups is not the same at the two states. The paper concludes that sample risk depends on both the sample and the model state, so a ranking made once cannot be reused throughout training.
  • The paper also finds that, at initialization, some gradient-based scores mostly track response length rather than safety.
  • The proposed method, SABA, short for state-aware budget allocation, treats filtering as dividing a fixed exclusion budget across training rounds. SABA excludes no samples in the first round. In each later round, it re-scores the samples with the current model, and from one such round to the next, the number of excluded samples grows by a constant factor called the deferral parameter.
  • An initialization check assigns this schedule only to gradient-based scorers whose initial rankings seem uninformative. The other scorers work as static filters, except that a second check can replace their initial rankings once after the first round.
  • In the main setting, Qwen3-8B is fine-tuned on Dolly with the same training budget for all methods. Safety is measured by the attack success rate, or ASR, on two harmful-prompt benchmarks, CatQA and HEx-PHI, with responses judged by Llama-Guard-3 and the Beaver cost model. SABA has a lower ASR than all nineteen baselines, with little loss in utility. The paper reports that the gains carry over to Alpaca, a second instruction dataset, and to Llama-3-8B-Instruct, a second model.
Critical strengths

The paper tests an assumption common to most existing filters: that a ranking made at initialization stays valid. It shows that the ranking can change sharply during training and provides an explanation for this change.

  • The initial ranking of SQSD, which measures how much a sample's update points toward an unsafe rather than a safe direction, is almost uncorrelated with its ranking after two epochs. The initial ranking of GradShield, which scores a sample by the inner product between its gradient and a safety gradient, mostly reflects response length. At initialization, sample gradients share a large component nearly orthogonal to the safety direction, so GradShield tracks gradient norm, which decreases with length. The ranking change appears in six settings, and five attempts to fix the initial ranking fail.

The main comparison is well controlled. All methods use the same training budget, and the best baseline, refresh-only with GradShield, also re-scores every round. SABA's ASR is 9 to 16 percentage points lower than that of this baseline, and these controls rule out less training or more frequent re-scoring as the reason. The variance across seeds is also small.

The budget allocation approach gives the schedule a clear rationale. It includes existing static filtering as a special case. It also predicts that safety is best at an intermediate value of the deferral parameter, between spreading exclusions evenly across later rounds and putting them all in the final round. This prediction holds for all four judge-benchmark pairs. The default value, which the paper says was fixed before other values were tested, is close to the best one.

Critical weaknesses

The experiments show that harm depends on the model state, but not that the most harmful samples change during training. Section 4.3 relies on the latter to justify re-scoring every round.

  • For the matched intervention, Section 3.2 reports only the interaction in Eq. 1: the harm gap between the two groups at the early state minus that at the late state. This value can be positive even if the same group is more harmful at both states.
  • Section 5.4 and Table 2 compare re-scoring every round with reusing the initial ranking under the same schedule. Re-scoring lowers the CatQA ASR by 50.8 points for GradShield but by only 0.1 points for SQSD, which is the scorer used to show that rankings change.
  • The four values behind Eq. 1, that is, the harm caused by training on each group from each state, would show whether the more harmful group changes. If the same group is at least as harmful as the other at both states, the claim should be narrowed: harm depends on the state, and initial rankings fail only for some scorers, such as GradShield.

The abstract claims that the gains carry over to Alpaca and Llama-3-8B-Instruct, but the main text does not support this with comparisons to strong baselines.

  • The only comparison with strong baselines in the main text is Table 1, which covers only the main setting.
  • On Llama-3-8B-Instruct, Section 5.4 says that the configuration chosen by the two checks for GradShield lowers the CatQA ASR by 6.3 points relative to no filtering. In the main setting, SABA lowers the ASR by 39 to 45 points.
  • Refresh-only and memory-ratchet should be compared with SABA on both settings in the main text. For Alpaca, this would need new runs. Without these comparisons, the claim should be limited to what the main text shows for both settings: improvement over no filtering.
Questions for authors
  • What are the four values behind Eq. 1, the harm caused by training on each group from each state, for all sixteen Dolly combinations in Figure 3a?

  • What kinds of samples does SABA exclude? Section 5.2 shows only that they are not simply the shortest responses. A breakdown by Dolly instruction category or some representative examples would help.

Suggestion

  • Testing over-refusal on a benchmark such as XSTest, which has benign prompts that look like harmful ones, would show whether some of the gain comes from refusing these prompts. The current metrics cannot capture this: ROUGE-L on Dolly would likely drop if the model refused ordinary prompts.
2 comments
AuthorsOpenReview ↗

Response to Reviewer xBap (1/2)

Dear Reviewer xBap,

Thank you for the detailed review and for the concrete analyses you suggested. We appreciate that you found that the paper tests an assumption shared by most existing filters and explains why it fails, that the main comparison is well controlled against refresh-only with the same training budget and the same re-scoring, and that the budget-allocation view gives the schedule a clear rationale.

W1(a) & Q1: The four values behind Eq. 1, and whether the more harmful group changes.

A1: We agree that the data do not show a swap, and we have narrowed the claim. ASR (%) after training each group from EE (half an epoch) or LL (two epochs), pooled over three Qwen3-8B seeds (all 16 combinations of model, seed, judge and benchmark in Table A18):

Judge · benchmarkEH-L from EEEL-H from EEEH-L from LLEL-H from LLgap at EEgap at LLII
Guard · CatQA2.42.445.550.9+0.0−5.4+5.4
Beaver · CatQA6.25.956.363.0+0.3−6.7+7.0
Guard · HEx-PHI3.33.328.131.7+0.0−3.6+3.6
Beaver · HEx-PHI5.04.933.635.6+0.1−2.0+2.1

EH-L is ranked high-risk at EE and low-risk at LL, EL-H the other way. Each gap is EH-L minus EL-H, and II is the gap at EE minus the gap at LL (Eq. 1). At EE, the model is close to the safety floor and the two groups cause nearly the same harm (gap within 0.33 points of zero in all 16 combinations). At LL, the group that EE ranked low-risk is the more harmful one in all 16 combinations, so the early ranking misorders harm at the later states from which training continues.

Revision. Section 3.2 now claims only that harm depends on the state and that the ranking at EE misorders harm at LL. Table 1 reports the four cells, and the Limitations state that the gap opens up after the first epoch, without a reversal.

W1(b): Section 4.3 justifies re-scoring by ranking change, yet re-scoring helps GradShield by 50.8 points but SQSD, the scorer whose rankings change, by only 0.1.

A2: We agree that a ranking change alone does not justify re-scoring. What matters is whether the initial ranking already does better than no filtering. Static SQSD lowers the CatQA ASR from 61.3% to 44.8%, so re-scoring adds little. Static GradShield stays at 62.4%, because its initial ranking mostly tracks response length. Under the SABA schedule, keeping that initial ranking gives 69.1% against 18.3% with re-scoring, the 50.8 points you cite (Table A10). On Dolly, our checks make exactly this split: GradShield gets the schedule and SQSD the delayed-static branch. The checks are computed separately in each setting (see W2).

The same split holds after the first epoch. Under the same schedule, we also reused the θ1\theta_1 ranking in the final round (Table A21; CatQA / HEx-PHI, Guard):

ScorerRe-scored at θ2\theta_2Reused from θ1\theta_1Top-2,250 overlap of θ1\theta_1 and θ2\theta_2 (chance 0.45)
GradShield18.3 / 14.432.7 / 20.70.71
SQSD54.4 / 42.154.6 / 35.70.88

GradShield's ranking keeps moving between the first epoch (θ1\theta_1) and the second (θ2\theta_2), so reusing the θ1\theta_1 ranking raises its CatQA ASR from 18.3% to 32.7%. SQSD's ranking is largely settled by θ1\theta_1, so re-scoring does not help (and is worse on HEx-PHI), and under this schedule SQSD is worse than its own static filtering (44.8%) either way. This is why the check keeps SQSD on the delayed-static branch.

Revision. Section 4.3 now motivates re-scoring by how much a scorer's ranking keeps changing, and Section 5.4 and Appendix H.4 report these comparisons.

(continued in part 2/2)

AuthorsOpenReview ↗

Response to Reviewer xBap (2/2)

(continued from part 1/2)

W2: The abstract's claim of gains on Alpaca (with Qwen3-8B) and on Llama-3-8B (with Dolly) is not supported by strong baselines in the main text.

A3: Table 3 now makes this comparison in the main text, at the same training volume (CatQA / HEx-PHI ASR, Llama-Guard-3):

MethodDolly · Qwen3-8B (main)Alpaca · Qwen3-8BDolly · Llama-3-8B
no filter61.3 / 46.243.1 / 45.174.2 / 57.1
best static filter39.3 / 31.830.8 / 35.269.7 / 51.3
refresh-only · GradShield34.4 / 23.735.4 / 39.267.2 / 47.1
memory-ratchet · GradShield41.1 / 27.537.2 / 40.970.7 / 49.1
delayed-static · SQSD42.9 / 30.135.6 / 36.763.9 / 42.1
SABA · GradShield, schedule18.3 / 14.420.7 / 22.059.2 / 41.1
SABA · GradShield, branch selected by the checks (delayed-static on Llama-3)18.3 / 14.420.7 / 22.067.4 / 47.8

With Alpaca, SABA is 10.1 points below the strongest baseline (the best static filter, 30.8%). With Llama-3-8B, the margin is small: the schedule is 4.7 and 0.9 points below delayed-static SQSD (from unrounded means). There, GradShield's AA (the rank correlation of its initial scores with gradient norm or response length) is 0.27, below the threshold of 0.3, so the checks send it to delayed-static filtering (last row), and they send SQSD (A=0.61A = 0.61) to the schedule, which is on par with the strongest baseline (63.6 / 43.1, Table A2).

Revision. Table 3 and its discussion are new, and the abstract now reads: "These gains hold across corpora, model sizes and training lengths, and are smaller on a second model family."

Q2: What kinds of samples does SABA exclude?

A4: They are spread across instruction categories with a mild skew, and the top-ranked ones include informal answers to open requests. A category's share of the excluded set divided by its share of the pool ranges from 0.87 (classification) to 1.12 (general QA), where 1 means no skew. Examples are a haiku about pickles, or "Open my eyes" as the answer to "What's the first thing you do when you wake up in the morning?". Unlike static GradShield (median 7 words), SABA's final-round set is close to the pool in length (median 34–38 vs. 31 words) and stable across seeds (Jaccard 0.61–0.72, against 0.29 for random sets of this size).

Revision. Appendix H.3 and Table A20 now profile the exclusion sets.

Suggestion: over-refusal (XSTest).

A5: SABA does not refuse clearly safe prompts more often than refresh-only, and it is more cautious on borderline ones (Table 5, Table A28):

ModelXSTest unsafe: compliance ↓\downarrowXSTest safe: refusal ↓\downarrowOR-Bench: refusal ↓\downarrow
base Qwen3-8B31.08.032.5
no filter (15,000 passes)78.02.55.1
refresh-only · GradShield50.74.415.1
SABA · GradShield40.54.423.1

SABA also has the lowest compliance with unsafe XSTest prompts among the fine-tuned models (40.5%). On OR-Bench, it refuses more borderline prompts than refresh-only (23.1% vs. 15.1%), though fewer than the base model, so part of its lower ASR may come with extra caution on such prompts. Ordinary tasks do not suffer (IFEval 54.9 vs. 53.4 without filtering, MMLU 75.2 vs. 75.1, GSM8K 74.4 vs. 71.8).

Revision. Section 5.5 and the Limitations now report these results, including the trade-off on borderline prompts.

We would be glad to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.