acceptodds
Under review as a conference paper at ICLR 2027

When Does In-House Adaptation Cost Safety? A Capacity-Rank-Recipe Map

Abstract

Fine-tuning an assistant on an in-house security corpus looks like the benign case: domain knowledge padded with ordinary alignment data. It is not. We fine-tune Qwen2.5 from B to B on a -sample mixture of domain knowledge and safety-alignment data, and evaluate configurations ( at three seeds) on a -item suite: what decides whether safety survives is the adapter's rank, with capacity setting how much is lost rather than whether. Under a conservative recipe, nothing below rank leaves the band around its own base, while the rank- cells at B, B and B lose – under the rule scorer and survive multiple-comparison correction; an aggressive recipe — three epochs at twenty times the learning rate — collapses every scale instead, by –. The corpus is the weaker lever: at the same rank a safety-neutral corpus with no harmful or security text still costs most of the same loss, and at rank no corpus we ran costs anything of that size. The cost is also protocol-dependent: the collapse disappears when the adaptation safety prompt is present at evaluation, so a benchmark run with that prompt can certify a model that this suite scores as collapsed. Three other results are negative and, we think, the most reusable: adaptation buys almost no knowledge, our own refusal probe mis-specifies the geometry at the top of the range, and each explanation the collapse suggests fails its own test. In practice: monitor rank and update magnitude, and keep knowledge out of the weights where retrieval can supply it.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
safety alignment of LLMs, fine-tuning safety / alignment erosion, LoRA rank, parameter-efficient fine-tuning, safety–utility trade-off

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8rHnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how LoRA can degrade LLM safety, even when using safe training data. The authors conduct experiments across different model sizes, LoRA ranks, training settings, and datasets. The results show that higher LoRA ranks and more aggressive fine-tuning can lead to greater safety degradation, even without harmful training examples.

Critical strengths
  • Corpus auditing is meaningful, it shows that hh-rlhf red-team target responses still contain usable harmful content.
  • The experiments are carefully designed and thoroughly analyzed, with extensive ablation studies investigating the factors contributing to safety degradation.
Critical weaknesses
  • The evaluation suite is relatively small, consisting of only 110 examples. The Safety QA subset contains just 8 examples but accounts for 30% of the Safety score, which makes the results less reliable to me.
  • Findings are interesting but less novel, similar observations regarding safety degradation under fine-tuning and LoRA have already been studied.
Questions for authors
  • Can the authors complete the missing runs in Figure 1? These are the main results for this paper, but entries for 3B/7B at rank=32/48 are still missing.
  • Can you further prove the generalizability? The central claims are tested on a single Qwen-2.5 family. And as stated by the authors, the cross-family experiments on Mistral-7B are also inconclusive. I would suggest validating this claim on more recent models like Qwen-3.5 4B or 9B, this would make the results more convincing.
1 comment
AuthorsOpenReview ↗

Author response to Reviewer 8rHn

We thank the reviewer for the constructive report. Both suggestions were experiments we could run, and both are now in the revision.

On Q1 (missing runs). All five cells are trained and evaluated at three seeds, so Figure 1 is a complete 4 × 5 map (p. 5). Filling them did more than add points. The correction family grew from 24 to 29 tests, and the two rank-128 cells at 1.5B and 7B now sit just outside Benjamini-Hochberg (p(BH) = 0.054 and 0.058). We report the completed-family verdict, three survivors all at the mid-scale, rather than the earlier count.

On Q2 (generalizability). We added Qwen3.5-9B at ranks 16 and 128 with the same recipe, on a smaller 2,000-sample budget; both ranks share it and are compared only with each other. There is no cost at either rank under the rule scorer: +0.016 (p = 0.25) and +0.031 (p = 0.08). This is a null, not a ceiling. The Qwen3.5-9B base refuses harmful prompts at 0.933, the same rate as the Qwen2.5-1.5B base that loses a third of its refusals at rank 128. The cross-family Mistral-7B arm is inconclusive for a different reason: its base refusal is 0.367, so the suite does not discriminate there. That is why we chose Qwen3.5-9B for this test. The paper now states that the map is a Qwen2.5 map (Limitations, p. 8).

On the size of the suite. We agree, and we state this in Limitations. Eight safety-QA items carry 0.3 of the safety score. The revision reports every channel next to the composite, adds AdvBench and JailbreakBench, a five-generation-seed protocol for fourteen cells, minimum detectable effects (0.028-0.074), and full-response re-scoring of the high-rank cells.

On novelty. Beyond the observation that a higher rank or a larger learning rate damages more, the paper reports three results we did not anticipate from the standard picture: the scale-specific, non-monotone curve (0.5B inert, 1.5B a cliff, 3B a ramp from rank 32, 7B only the top); rank not being update magnitude (8× parameters, 1.7-2.1× weight movement, identical-norm cells differing 4.5-fold); and the instrument result, in which a scorer defect that covaries with the treatment inflates the high-rank losses by up to 0.37 in refusal rate and flips the survivor count from three to four to none.

Reviewer_KQeGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper attempts to study the impact of LoRA fine-tuning on knowledge and safety. The paper performs grid search across two dimensions, rank of the adapter and learning rate, to analyze the impact of supervised fine-tuning. Test data is comprised of 110 examples (30 harmful prompts, 20 jailbreaking, 8 safety-QA, 32 CVE-analysis, and 20 general-QA items). Experiments show that 1/ safety regresses more as rank and learning increase, 2/ fine-tuning adds little knowledge when learning rate is low. The paper recommends to use retrieval augmented generation over knowledge fine-tuning.

Critical strengths
  1. The broad area of research for studying the safety impact of in-house adaptation is an important field of study.
  2. The paper attempted to analyze the change in safety behavior and knowledge using probe and boundary geometry. These are worthwhile directions, even thought the effects are not conclusive.
Critical weaknesses
  1. Novelty. The paper simply performs a grid search across two dimensions, rank of the adapter and learning rate. The finding is expected and not surprising: as we use bigger learning rates and adjust more parameters in LoRA, the model would have greater changes in knowledge and safety behaviors. The paper attempted to make some analysis using Probe but the findings are not informative.
  2. Evaluation validity. The test set is only comprised of 110 examples composed from different datasets. This makes measurement noisy and high variance. For example, the QA accuracy metric is calculated from 8 examples.
  3. Writing and readability. This paper has serious readability issues. For example, the current abstract is very difficult to comprehend. The current introduction is filled with numbers without sufficient context to understand. The paper is fragmented into many sections without a coherent story. The probe section does not have sufficient detail about the metrics and the key information is in appendix.
Questions for authors
  1. "The corpus is 6,503 ShareGPT samples assembled from crawled public sources and scripted templates. Table 1 gives it by kind" Some datasets shown in table 1 are open source datasets, not sharegpt samples. Could you elaborate why the datasets in Table 1 are from shareGPT?
  2. What is the stopping criteria of the training runs in the grid search?
  3. Why the test set only contains 110 examples?
3 comments
AuthorsOpenReview ↗

Author response to Reviewer KQeG

We thank the reviewer for the detailed reading and for the specific presentation comments, which shaped the revision. We also appreciate the novelty challenge, and we answer it with three concrete observations rather than with a claim of scope.

On the design. We should clarify one point, since the earlier abstract invited a different reading: the grid is rank × scale (20 cells), not rank × learning rate. Learning rate defines the two regimes, conservative and aggressive, and the learning-rate-by-epoch grid is a control in Appendix F. The paper's claim is therefore about how the rank curve changes with scale, not about the monotone effect of a larger learning rate.

On novelty. Our reading is that the result is not simply "more parameters or a larger learning rate damages more." Three observations support this. First, the curve is scale-specific and non-monotone in capacity: 0.5B never costs more than 0.053; 1.5B is flat to rank 64 and then drops; 3B starts losing at rank 32 and ends at -0.223; 7B separates only at the top, and by less than 3B. If damage tracked parameter count, 7B at rank 128 would be the worst cell; in our measurements 3B is the most affected at every rank. The mid-scale, which a deployment would treat as the safe compromise, is the fragile one, and it is the only scale whose cells survive correction under both versions of the keyword scorer. Second, rank is not update magnitude: eight times the parameters move the weights 1.7-2.1 times as far, and two cells at an identical update norm differ 4.5-fold in harm. Third, the instrument is part of the finding: a keyword scorer whose defect covaries with the language shift inflates the high-rank losses by up to 0.37 in refusal rate, and changing the instrument changes how many cells survive correction (three, four, none). The neutral corpus reproduces 58-65% of the in-domain loss, so the corpus is not the driver either.

On evaluation validity. We agree, and we state this in Limitations (p. 8). The suite was frozen before the sweep so that every cell meets identical items. It is 110 items, and eight safety-QA items carry 0.3 of the safety score. The revision reports every channel next to the composite, cross-checks refusal on AdvBench and JailbreakBench (Appendix O), adds a five-generation-seed protocol for fourteen cells, reports minimum detectable effects (0.028-0.074), and re-scores the high-rank cells on full responses. A larger suite is the first item after acceptance; we preferred not to change the suite after seeing the results.

On writing and readability. We have rebuilt the paper around one argument rather than a sequence of findings. The abstract is about 280 words; the introduction is narrative; the numbers sit in Figure 1 and Tables 1-2; Section 5 is one page and states the probe's validity limit instead of implying a mechanism; per-cell detail is in the appendix. Measured on the body, the mean sentence is 22.6 words, 5% exceed 45 words, and 23% carry three or more numbers (was 45%).

On Q1 (ShareGPT). We should have been clearer. None of the 6,503 samples comes from the ShareGPT dataset. "ShareGPT format" refers to the multi-turn JSON schema (from/value); the sources are the nine in Table 1: hh-rlhf, real-toxicity-prompts, unsafe_qa, NVD/arXiv/GitHub, AdvBench, JailbreakBench, and scripted templates. The sentence now reads "in ShareGPT conversation format, assembled from crawled public sources and scripted templates rather than sampled from the ShareGPT dataset" (p. 3).

On Q2 (stopping criteria). There is no early stopping and no validation-based checkpoint selection. Each arm trains for a fixed number of epochs (one conservative, three aggressive) on a fixed sample budget (500/1,000/3,000/full corpus), and the adapter at the end of the last epoch is evaluated. No stopping rule enters the comparison.

On Q3 (110 examples). This was a deliberate trade-off: 29 cells and 3 seeds with one frozen suite, so every cell meets identical items. The composition is 30 harmful, 20 jailbreak, 8 safety-QA, 32 CVE, and 20 general-QA items. We agree that eight safety-QA items is the weakest part, which is why the per-channel results and the external benchmarks are reported next to the composite.

I would like to thank the authors for the detailed response. Some of the issues have been resolved, e.g., sharegpt discrepancy. But my concern on the evaluation validity due to small sample size and no stopping criteria remains.

AuthorsOpenReview ↗

Reply to Reviewer KQeG

We thank the reviewer for the follow-up and for recognizing the ShareGPT correction. We would like to clarify the two remaining points.

On stopping criteria. We do not use early stopping or validation-based checkpoint selection, and this is a deliberate design choice rather than an omission. Every arm is trained for a fixed number of epochs (one in the conservative regime, three in the aggressive one) on a fixed sample budget, and the adapter at the end of that schedule is evaluated. A convergence-based rule would let training length vary across cells and confound the rank and scale comparison; fixing the schedule keeps the number of updates comparable across the grid. The one-versus-three choice defines the two regimes, not a per-cell stopping rule, and Appendix F varies epochs at a fixed update magnitude to show that repetition matters independently.

On the suite size. We agree that this is the main remaining limitation, and we state it in Limitations. The eight safety-QA items are the weakest part; they carry 0.3 of the safety score, while the safety claims rest mainly on the 30 harmful and 20 jailbreak items, and the composite is reported alongside every channel, with minimum detectable effects (0.028-0.074) so readers can see what the suite can resolve. The external benchmarks and the five-generation protocol add checks on the refusal axis, but we agree they do not substitute for a larger suite. Expanding the suite is the first thing we would do in a revision; we did not want to change it after seeing the results.

We are grateful for the careful reading and for pressing on these points.

Reviewer_SNUKOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how fine-tuning on an in-house security corpus degrades the safety of Qwen2.5 models from 0.5B to 7B parameters. It examines several factors that may affect the outcome: the magnitude of the weight update, the training corpus, the evaluation system prompt, and the LoRA rank. It also compares fine-tuning against retrieval for knowledge injection, and reports failure of a refusal probe. The authors conclude that the safety cost depends on the rank, and recommend fine-tuning with a small rank and supplying domain knowledge through retrieval.

Critical strengths
  • The paper is about an important problem, the effect of fine-tuning on model safety, and examines several factors jointly.
Critical weaknesses
  • The presentation needs substantial improvement. The main text provides little background and limited details on the experimental setup. It is dominated by numbers from specific settings, with few clear summary conclusions and little discussion of the mechanisms behind the observed phenomena. The practical recommendations are not convincingly supported.
  • The experiments are limited. Evaluation relies on a 110-item suite, and several controls required by the conclusions are missing.
Questions for authors
  • The headline results appear to be computed with the rule scorer. However, Appendix G shows that human annotations agree much more with the LLM judge than with the rule scorer. If the judge were used as the primary scorer, would the conclusions change?
  • Section 4 Finding 2 is established at a single setting only. Does it extend to other ranks and model scales?
  • Why are some cells in Figure 1 blank, and how does this affect the conclusions?
  • Changing the rank also changes other quantities, such as the number of trainable parameters and the magnitude of the weight update. How can the effect of rank be isolated from these?
  • Have the findings in Section 5 been validated on more model sizes?
  • In Section 6, how would errors and noise in retrieval affect the conclusions? The section ends with the claim: "What we add is that on this corpus the parametric route cannot be pushed toward the retrieval result without moving along rank, the axis that costs refusal." What experimental evidence supports this claim?
  • In Section 7, does refusal also recover under a safety prompt that was not seen during training?
1 comment
AuthorsOpenReview ↗

Author response to Reviewer SNUK

We thank the reviewer for a careful report and for pressing on the scorer and the missing cells; both concerns materially improved the paper. The seven questions follow, with page references to the revised PDF.

On Q1 (the rule scorer). We agree that this was the main soundness issue; addressing it changed the paper. We scored all 29 cells with a frontier judge (DeepSeek-V4.1-Flash, non-Qwen, temperature 0) and re-scored the seven affected cells with a repaired rule on full responses. The qualitative conclusions hold; the magnitudes and the survivor count do not. Under the judge, the rank ordering (rho about 0.93 with the local judge) and the mid-scale pattern remain; the magnitudes roughly halve and no cell survives correction. Under the repaired rule four survive, and under the shipped rule three. The cause is mechanical: the list matches can't with an ASCII apostrophe but not can’t with the typographic one, and lacks the Chinese negation for "will not", so it fails on the cells that shift into English (false-negative rates 2-14% at the bases and low ranks, rising to 25.5/34/45.8% at 1.5B/3B/7B r=128). A blind human re-read agrees with the judge (kappa 0.79) far more than with the rule (0.49). All three instruments are now reported (Table 2, p. 6; Table 20, Appendix U), and we describe the verdict as instrument-dependent.

On Q2 (the knowledge channel). The channel now covers the full grid. Fact-hit stays between 0.011 and 0.031 against 0.016-0.019 at base in every conservative cell but 3B r=128 (0.055), which is also a safety loss. The CVE-accuracy gains at 0.5B and 1.5B come from answer structure rather than recalled facts (structured 0.99-1.00 against 0.84 at the 1.5B base). We should also correct our earlier text: general QA falls at higher ranks (1.5B from r=48 to -0.158 at r=128; 3B r=32; 7B r=64), so we no longer write "only refusal." Section 4.2, p. 5.

On Q3 (blank cells). All five are trained and evaluated at three seeds (0.5B r=48; 3B r=32 and 48; 7B r=32 and 48), and Figure 1 is complete (p. 5). Completing the map enlarged the family from 24 to 29 tests and moved two borderline verdicts: the 1.5B and 7B r=128 cells now sit at p(BH) = 0.054 and 0.058. Three cells survive under the shipped rule rather than the five in the reviewed version; we report the completed-family count.

On Q4 (rank versus parameters and update magnitude). We separate them as follows. First, r=128 carries eight times the parameters of r=16 but moves the weights only 1.7-2.1 times as far. Second, two cells with an identical update norm (0.0021) differ in harm by a factor of 4.5. Third, within one scale and layout, r=16 at alpha/r=4 moves 0.00262 and costs -0.035, while r=128 at alpha/r=2 moves 0.00280 (7% more) and costs -0.138/-0.173/-0.184 across its three seeds. Holding the per-step scale fixed and raising the rank reproduces the drop; raising the update strength at fixed rank gives a smaller, monotone effect. Appendix F.

On Q5 (the probe at other scales). The probe is the 1.5B sweep plus a 7B endpoint, and the 7B row is why we do not generalize it: drift is small there while refusal is higher than at base, so the monitor does not transfer. We state this as a validity limit (p. 7); the behavioural map is measured at all four scales.

On Q6 (retrieval noise and the final claim). The retrieval arm is an oracle, with the gold NVD description supplied at inference, so +0.178 and +0.184 are upper bounds; a noisy retriever would lower both and shrink the margin, and Section 6 says so. More importantly, the reviewed sentence was stronger than the evidence. What the data support is narrower: the parametric route moves the fact channel only in cells that lose refusal (3B r=128), while the CVE-accuracy gains at low rank are structural. The revised text says this and no longer claims that the parametric route "cannot be pushed" toward retrieval without moving along rank; that claim would require a rank sweep optimizing fact-hit, which we did not run. Section 6, p. 7.

On Q7 (an unseen safety prompt). Yes, largely. With a prompt that never appears in the corpus, the aggressive 1.5B cell's gap to base on harmful items shrinks from 0.90 to 0.20, and to 0.07 with the corpus's own string. Recovery is therefore mostly prompt-generic, with a premium for the training string. On the conservative high-rank cells, the prompt does not return refusal to base at any scale, so that effect belongs to the collapsed regime. Appendix Q.

On presentation and controls. We have rewritten the paper around a single argument: eight pages, conclusions first, numbers in Figure 1 and Tables 1-2, per-cell detail in the appendix. Body prose is 26% shorter than the reviewed version, and 5% of sentences exceed 45 words (was 28%). The revision adds the full grid, the three-instrument comparison, the unseen-prompt test, a size-matched control, external benchmarks, and an out-of-family arm; the suite remains 110 items (Limitations).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.