acceptodds
Under review as a conference paper at ICLR 2027

Don't Inoculate Everything: Stratified Inoculation Prompting Narrows Backdoor Triggers and Preserves Desired Traits

Abstract

Supervised fine-tuning can teach language models undesired behaviours alongside desired ones. Inoculation prompting (IP) aims to limit unwanted generalisation by requesting the undesired behaviour during training and removing the request at inference. However, undesired behaviour can still appear under unrelated prompts. IP can also hinder learning of the desired behaviour. We address these limitations in settings where both behaviours co-occur in most training examples, so filtering out examples with undesired behaviour leaves only a small clean subset. We introduce stratified inoculation prompting (SIP). SIP leverages a small clean subset to demonstrate that desired behaviour should persist without the undesired one across different contexts. SIP oversamples these clean examples under diverse non-eliciting prompts while inoculating the rest. SIP substantially reduces expression of undesired behaviour while preserving more of the desired behaviour than IP. These gains persist even when we extend IP to oversample the same clean subset at the same rate as SIP. Moreover, SIP yields lower emergent misalignment rates in all harmful-advice setups we tested. SIP can be further extended to limit the undesired behaviour even under prompts that explicitly request it. We introduce backdoor dilution, which weakens expression under the inoculation prompt, and password-locked inoculation, which concentrates elicitation on a designated password. Taken together, our findings show that changing the training contexts for a small clean subset can significantly improve selective generalisation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI, AI alignment, AI safety, inoculation prompting, selective generalisation, supervised fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DJwXOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies selective generalisation during supervised fine-tuning when desired and undesired traits co-occur in most training examples. It proposes stratified inoculation prompting (SIP): high-confidence clean examples are paired with diverse non-eliciting prompts, while the remaining examples receive an inoculation prompt requesting the undesired trait. In the main configuration, a 5% distinct clean subset is repeated to occupy 25% of a fixed training-position budget.

Experiments cover six model–trait setups, including three harmful-advice domains, with three training seeds per condition. SIP reduces undesired-trait leakage relative to IP across the evaluated setups and improves desired-trait retention on average. A matched-oversampling IP control supports an effect of prompt assignment beyond increased exposure to clean examples. The evaluation includes held-out prompt categories, emergent misalignment, and ablations of prompt assignment, clean-data quantity, repetition and contamination. The paper also introduces backdoor dilution and password-conditioned inoculation to reduce or concentrate residual elicitation.

The main contribution is a simple, empirically supported modification to IP for controlled two-trait settings. The results concern reduced behavioural expression under tested conditions; undesired behaviour remains accessible.

Critical strengths
  1. The central intervention is supported by a strong control that isolates prompt assignment.

SIP addresses a concrete limitation of uniform IP: clean examples should demonstrate that the desired trait persists outside contexts requesting the undesired trait. The matched-oversampling comparison is particularly important. In Section 4.3 and Appendix D.2, oversampled IP and SIP use the same training examples, sampling frequencies, optimisation settings and total training budget, differing only in the prompts assigned to clean examples. SIP improves both aggregate desired-trait retention and undesired-trait leakage in this comparison. This provides persuasive evidence that the proposed contextual assignment contributes beyond simply increasing clean-data exposure, and makes the method a useful addition to existing IP approaches.

  1. The leakage improvement is consistent across the tested settings, and the ablations make the empirical contribution informative.

Table 12 reports lower leakage for SIP than IP in all six setups, with all six paired 95% bootstrap intervals for the difference below zero. Appendix C.1 further reports lower point estimates in all 36 setup–prompt-family comparisons, including categories absent from training; 33 of these paired intervals exclude zero. The harmful-advice experiments extend the study beyond stylistic traits, with lower conditional EM rates than IP in all three domains. The prompt-assignment, oversampling and clean-branch ablations also help explain when the approach works. In particular, Appendix D.5 shows that clean examples must demonstrate the desired trait, rather than merely omit the undesired trait. These findings support acceptance as a substantive empirical contribution.

Critical weaknesses
  1. The practical advantage over training on the scarce clean subset alone remains incompletely established.

I appreciate that Appendix C.2 includes a matched-step clean-only control. However, this control substantially changes the practical comparison: Table 14 reports normalised DT retention of 0.910 and leakage of 0.046, versus 0.970 and 0.115 for SIP. Thus, the comparison establishes a retention–leakage trade-off. The repeated clean-only baseline does suffer severe generation deterioration in the financial setup, with 287/600 local responses excluded, but the corresponding exclusions in the other five setups range from 0 to 23/600 (Table 16). Since the financial failure is interpreted as overtraining, results at intermediate training durations or appropriately selected learning rates could materially change the comparison. This matters because preserving useful learning that would otherwise be lost through filtering is a central motivation. The matched-step control and its heterogeneous outcomes deserve greater prominence in the main assessment.

  1. The broader safety and capability-preservation conclusions need stronger qualification.

The EM metric is conditional on response validity and coherence, and these filters retain substantially different populations across methods. In the financial setup, only 415–466 of 1,200 SIP responses per seed enter the EM denominator, compared with 566–647 for IP (Table 13). The reduction from 28.5% to 12.6% therefore needs to be interpreted alongside the substantial loss of eligible responses. The sports and medical results, where SIP has comparable or better eligibility, provide useful supporting evidence; nevertheless, the financial result needs sensitivity analysis and validation of the excluded cases. The paper reports manual spot checks but no quantitative independent-judge or human agreement assessment.

Capability preservation also requires more precise wording. Section 4.5's “no negative impact” heading is stronger than the evidence: Table 18 reports loose-IFEval decreases of 8.38, 4.87 and 5.42 percentage points in three setups, with pointwise 95% intervals below zero. These capability comparisons are against SFT (5%), without corresponding IP or base-checkpoint results. These issues are fixable, but they limit how broadly the safety and utility claims can be interpreted.

Questions for authors
  1. Additional analysis of the clean-only baseline.

Can you clarify whether the financial generation failure of the matched-step clean-only baseline persists at intermediate training durations or lower learning rates? If checkpoints or additional runs are available, please report the joint DT-retention, leakage and response-validity comparison using the same 250 clean examples. Comparable validation and tuning budgets for SIP and clean-only SFT would help establish the practical advantage of retaining mixed data. Table 14 makes this comparison consequential to my assessment.

  1. Sensitivity of the EM result to coherence filtering.

For the existing EM generations, can you report validity/coherence coverage together with harmful-output and coherent-safe-output frequencies, and examine whether the conclusions hold when borderline-coherent responses are retained? Please also clarify what harmful content, if any, occurs in the excluded responses. A blinded human or independent-judge audit of a representative subset would be particularly useful in the financial setup, where most generated responses do not enter the reported EM denominator.

  1. Scope of the capability-preservation claim.

How should the Section 4.5 heading be reconciled with the negative IFEval differences in Table 18 and Appendix C.3's explicit statement that the aggregate results do not establish equivalent instruction-following performance? Are capability results available for IP, oversampled IP or the base checkpoints? These comparisons would clarify whether SIP preserves capabilities relative to its principal competing method, and would help determine the appropriate scope of the claim.

5 comments

We would like to sincerely thank Reviewer DJwX for their thorough assessment and questions.

Q1: Additional analysis of the clean-only baseline.

We agree that Table 14 in the submitted version (now Table 18) establishes a retention–leakage trade-off. We now give this comparison greater prominence in Section 4.3 and Appendix C.2.

We run an additional small scale analysis for the financial setup, evaluating clean-only SFT on the same 250 clean examples after 1, 5 and 10 epochs:

ConditionDesired-trait score ↑Leakage ↓Valid responses ↑
Clean-only SFT, 1 epoch48.7 [46.9, 50.5]4.5 [2.3, 7.2]200/200
Clean-only SFT, 5 epochs37.5 [35.3, 39.6]2.7 [1.0, 4.9]200/200
Clean-only SFT, 10 epochs55.2 [51.2, 59.1]10.1 [7.4, 12.9]197/200
SIP (5%), reference79.6 [78.5, 80.6]35.5 [29.4, 41.6]200/200

These results are now included in Appendix C.2.2, Table 21. The severe generation deterioration is absent at these earlier checkpoints. However, all three have lower desired-trait and leakage than SIP.

Q2: Sensitivity of the EM result to coherence filtering.

We re-analysed all IP (5%) and SIP (5%) responses across the three harmful-advice setups and conducted a blinded human audit on the finance setup. We added the results to Appendix C.1.1, Tables 14–16.

Here are the requested validity/coherence coverage together with harmful-output and coherent-safe-output frequencies.

Generation-valid (higher is better)

SetupIP (%)SIP (%)SIP−IP, [95% CI]
Sports98.7897.61−1.17 [−2.83, 0.31]
Medical98.64100.00+1.36 [0.44, 2.75]
Financial99.9498.36−1.58 [−2.67, −0.72]

Coherence-pass / EM-eligible (higher is better)

SetupIP (%)SIP (%)SIP−IP, [95% CI]
Sports64.2566.14+1.89 [−4.11, 7.50]
Medical85.9290.25+4.33 [0.72, 7.81]
Financial50.3336.42−13.92 [−21.47, −7.03]

Harmful at any coherence (lower is better)

SetupIP (%)SIP (%)SIP−IP, [95% CI]
Sports14.752.97−11.78 [−18.06, −6.33]
Medical31.2812.69−18.58 [−25.47, −12.19]
Financial25.7511.81−13.94 [−20.45, −7.97]

Coherent and safe (higher is better)

SetupIP (%)SIP (%)SIP−IP, [95% CI]
Sports53.6763.31+9.64 [1.72, 17.39]
Medical55.8674.61+18.75 [12.50, 25.31]
Financial33.7829.81−3.97 [−11.81, 3.14]

SIP obtains comparable generation validity in all setups. It increases coherent-safe output frequency in sports and medical. In the financial setup, eligible coverage decreases by 13.9 percentage points.

The frequency of harmful responses is lower in all three setups.

We also examined whether this reduction persists at different coherence thresholds.

Pass only (lower is better)

SetupIP (%)SIP (%)95% CI for SIP−IP (pp)
Sports10.972.12[−14.03, −4.27]
Medical29.2610.65[−25.93, −11.89]
Financial28.5412.55[−25.37, −6.99]

Pass or borderline (lower is better)

SetupIP (%)SIP (%)95% CI for SIP−IP (pp)
Sports14.933.07[−18.20, −6.33]
Medical31.8012.72[−25.98, −12.63]
Financial27.0714.59[−19.48, −5.85]

No coherence restriction (lower is better)

SetupIP (%)SIP (%)95% CI for SIP−IP (pp)
Sports14.933.04[−18.22, −6.39]
Medical31.7112.69[−25.91, −12.56]
Financial25.7612.00[−20.29, −7.79]

We report that SIP has lower harmfulness rates in every setup under all different criteria.

Moreover, we examined the generation-valid responses excluded by the original coherence filter:

Harmful fraction among excluded responses (lower is better)

SetupIP: harmful / excludedSIP: harmful / excluded
Sports277/1,24357/1,133
Medical221/458111/351
Financial410/1,786261/2,230

Harmful labels occur among excluded responses from both methods, with a lower observed fraction for SIP in every setup. Including these responses preserves SIP’s lower harmfulness. Therefore, coherence exclusions do not explain the reduction in harmfulness.

We also obtained a blinded, single-rater human audit of a subset of stratified financial responses.

Among the valid responses, human and LLM harmfulness labels agreed on 45/54 responses (83.3%). Using identical category definitions, the human rater classified 37/54 (68.5%) as coherent, compared with 18/54 (33.3%) for the LLM judge.

This suggests that the LLM judge applied the shared coherence criteria more strictly on the audited responses. We therefore expect that reported coverage could be generally higher under human assessment.

Q3: Scope of the capability-preservation claim.

We have now changed Section 4.5 to “Effects on general capabilities” and explicitly acknowledge the setup-specific IFEval decreases and the inconclusive aggregate instruction-following comparison.

We have also added capability comparisons with IP (5%):

Capability differences between SIP and IP. (higher is better)

BenchmarkSIP − IP (5%)
IFEval, strict−5.45 [−7.05, −3.85]
MATH-500−1.37 [−3.00, 0.30]
MMLU-Pro−2.03 [−2.83, −1.21]

Relative to IP, SIP has lower average IFEval and MMLU-Pro accuracy, while the MATH-500 difference remains inconclusive. SIP’s improvements over IP therefore can come with some capability trade-offs. We now state these capability costs explicitly. The SIP vs IP comparison is now also reported in Appendix C.3, Table 24.

Thank you for your response. I understand your point to some extent; however, the current results could potentially affect the extent of the major revisions required for the paper. Therefore, I will maintain my current score. Should the other reviewers feel that their concerns have been addressed, I would be happy to raise my score.

Reviewer_8j5NOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors are trying to propose a new method named Stratified Inoculation prompting, which leverages small data sample of clean data points that indicate the desired behaviour that should persist and the underdesired behaviour that should not. They show that oversampling for SIP and also for IP helps furthermore. Additionally, SIP can be extended for the cases where the undesired behaviour is explicitly requested. Overall, the authors hint/indicate that changing the context of the training data with small clean data can help in generalization.

Critical strengths
  • SIP shows better performance than IP in limiting unwanted trait/behaviour and reduces broader harmful generalization.
  • SIP seems to be a plausible direction to be taken in the inoculation or mitigation direction.
Critical weaknesses
  • In almost all the plots and figures I see, SFT 100% performs better than SIP or is one of the best methods (provided I am not mis reading anything). So keeping this in mind: -- How costly is SFT as compared to doing SIP? both in terms of training time and sampling time (I believe SFT doesnt require any sampling) -- Clean data is important, but if SFT can give good results, why not try to make that better? -- In terms of getting the clean data what all efforts are required? How is the desired or undesired behaviour data filtered? LLM-as-a-judge? -- How big is SFT 100% in terms of absolute numbers? -- And similarly after oversampling in each of the methods what is that absolute number? -- In Fig 2A: UT for SIP 5% is always higher as compared to SFT. Also SIP doesnt even beat No SFT in almost all the cases. -- In Fig 2b: Similarly, SFT is kind of comparable to SIP, and No SFT beats almost every other method. What am I missing? Or I am not reading it correctly?

  • In terms of writing and some results too: -- Section 2, line 119-120: "SIP trains the model to express desired behaviour by default" -- Can this lead to over-expressiveness? If not, can that be proved or shown empirically? -- Section 2, line 126-127: On what basis were the categories decided? -- Table 1: Why only specific traits for specific models? Why not all vs all? -- Maybe a typo at the end of line 236 -- SFT(5%)? -- Fig 3B: There is only one point I see for oversampling. What do you mean by across methods? -- Fig 3C: What is the effect of the false positives and false negatives on different methods? -- Fig 4C and D: What do the colors of the bar indicate?

Questions for authors

All in the critical weakness.

4 comments

We would like to sincerely thank Reviewer 8j5N for their detailed feedback and all the questions. We would like to highlight several distinctions that should have been presented more clearly in our paper. We address all the questions below:

Q1: SFT 100% performs better than SIP or is one of the best methods.

Yes, SFT (100%) is a strong reference and achieves lower leakage across non-eliciting prompts. This is because SFT (100%) uses a different data mixture from SIP (5%). It represents a best-case scenario where sufficient clean examples are available to train entirely on clean data. It uses approximately 5,000 distinct clean examples, compared with 250 for SIP (5%).

Our primary data mixture setting is more challenging and represents limited clean-data availability: 95% of the available examples are mixed, expressing both desired and undesired behaviours, and only 5% are clean. We refer to methods trained on that data mixture as SFT (5%), IP (5%) and SIP (5%).

We show that SIP (5%) desired-trait retention approaches full-clean SFT (100%), although its leakage across non-eliciting prompts remains higher.

Q2: How costly is SFT as compared to doing SIP?

SIP uses ordinary SFT but with added special prompts to particular training data and oversampling. Oversampling redistributes a fixed training-position budget without affecting the number of optimiser steps. SIP requires some data preprocessing to identify clean examples and construct prompts, but adds no extra cost at inference (extra prompts are removed during inference).

It is possible that processing the extra token prompts during training results in some overhead. We compared recorded training-loop runtimes between SIP and SFT (5%). SIP’s runtime was 1.17% higher on average (95% bootstrap CI: −1.59% to +3.87%). This interval includes zero, so these do not establish a consistent runtime difference.

Q3: If SFT can give good results, why not try to make that better?

In the challenging scenario, where the majority of the data are mixed examples and there are very little clean data, SFT (labelled as SFT 5%) learns the desired trait but also strongly expresses the undesired trait by default (Figure 1D). It has high leakage (Figure 1C and Figure 2A) and also leads to broad emergent misalignment with a high rate (Figure 2B).

Using the same distinct clean subset and the same total training-position and optimiser-step budget, SIP (labelled as SIP 5%) achieves DT retention comparable to a baseline trained fully on clean-only data (SFT 100%).

SIP can be in fact considered as an approach to make SFT better. It differs only in special prompts that are applied during training to particular training examples and oversampling.

Q4: In terms of getting the clean data what all efforts are required? How is the desired or undesired behaviour data filtered? LLM-as-a-judge?

Yes, our experiments construct paired clean and mixed responses and validate them using LLM judges, including checks of trait expression, coherence and fidelity (Appendix A.2). SIP as a method is agnostic to the selection and filtering of the data. Having said that, the quality of that subset matters and that’s why we decided to include the filtering-error experiments (Section 4.3, Figure 3C, and Appendix D.6).

Q5: How big is SFT 100% in terms of absolute numbers? And similarly after oversampling in each of the methods what is that absolute number?

In our experiments, each setup uses ~5k training positions. All main conditions are:

ConditionDistinct clean examplesClean training positionsMixed positions
SFT (0%)005,000
SFT (5%)2502504,750
IP (0%)005,000
IP (5%)2502504,750
IP (5%) + oversampling2501,2503,750
SIP (5%)2501,2503,750
SFT (100%)5,0005,0000
SFT (100%, 1/20 data)2502500
No SFT000

Training mixtures are specified in Appendix A.4, Table 5, and optimiser-step counts in Appendix A.6, Table 7.

Q6: In Fig 2A: UT for SIP 5% is always higher as compared to SFT.

Yes, as we mentioned in Q1, this is because SFT 100% is trained entirely on 5,000 distinct clean examples with no mixed data, whereas SIP has access to only 250 clean examples.

When compared with a SFT baseline trained on the same amount of clean data (SFT 5%), SIP has substantially lower UT leakage: 0.115 [0.092, 0.139] vs 0.942 [0.928, 0.955] (Figure 1C).

Q7: SIP doesnt even beat No SFT in almost all the cases. In Fig 2b: Similarly, SFT is kind of comparable to SIP, and No SFT beats almost every other method.

The “No SFT” is a baseline which shows the performance prior to any fine-tuning. It has relatively low leakage and low EM rate, but also low level of the desired trait expression (Figure 1D). Whereas our objective is to teach the model to learn the desired trait with limited leakage and EM rate.

Q8: “SIP trains the model to express desired behaviour by default” — Can this lead to over-expressiveness? If not, can that be proved or shown empirically?

We have run an additional analysis of existing IFEval responses, identifying desired-trait expressions that violate explicit user instructions. We examined self-introductions conflicting with required output formats or openings, French responses despite requests for another language, and confidence statements conflicting with required formats or endings.

Using 25 eligible prompts per diagnostic and three training seeds, SIP’s interference rate exceeds SFT (5%) by 12.00 percentage points (95% bootstrap CI [0.60, 23.95]).

These are exploratory results on selected prompt subsets, but they show that preserving desired behaviour by default does not guarantee that explicit instructions override it. The analysis is now reported in Appendix C.3.1, Table 25.

Q9: Section 2, line 126–127: On what basis were the categories decided?

The prompt categories were motivated by our preliminary work on IP. We observed that IP could leak the undesired behaviour under non-eliciting contexts. We do not claim that the categories are exhaustive or optimal. However, we observe that SIP achieves significant leakage reduction even for prompt categories that were not-seen during training (Figure 2A). We also include the category-omission ablation in Appendix D.3.

Q10: Table 1: Why only specific traits for specific models? Why not all vs all?

Our aim was to evaluate SIP across multiple model families, sizes and trait pairs within a limited compute budget. Repeating the full protocol for every model–trait combination is unfortunately beyond our compute budget.

Q11: Maybe a typo at the end of line 236 — SFT(5%)?

The SFT (0%) is correct. It denotes a SFT baseline trained entirely on mixed data. We use it for results normalisation (Appendix B.6).

Q12: Fig 3B: There is only one point I see for oversampling. What do you mean by across methods?

Figure 3B compares three conditions: standard IP without oversampling, oversampled IP, and SIP, which includes oversampling in our main configuration. Thus, both the oversampled-IP and SIP points use oversampling. We agree that it could lead to some confusion. We have clarified the title and caption now in the revised version.

Q13: Fig 3C: What is the effect of the false positives and false negatives on different methods?

Yes, Figure 3C tests only SIP. This is because SIP relies on some classification of data as either clean or mixed. Both SFT and IP don’t have that requirement, which is why we included only SIP in that experiment.

Q14: Fig 4C and D: What do the colors of the bar indicate?

Thank you for flagging it! We have now simplified the colour scheme and clarified the caption in the revised version. Blue bars represent password-locked models under the labelled evaluation conditions. Grey bars represent separately trained matched controls using the same examples, sampling frequencies and underlying prompts, but without password clauses.

Dear Reviewer 8j5N.

Thank you again for your feedback!

Given that the discussion period is closing soon, we wanted to draw your attention to our response above. We would be more than happy to clarify any remaining questions before the end of discussion period.

Thanks for the update. I have increased my score now.

Reviewer_gZ8POpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to prevent LLMs from learning undesired behaviors during SFT while preserving desired behaviors. The authors propose a method which uses a small clean subset with diverse prompts to reduce undesired behavior leakage.

Critical strengths

The paper focus on an important problem, how to prevent undesired behaviors during SFT without losing training signals. The proposed SIP method is plug-and-play which require only changes to training prompts and data sampling. The experiments are comprehensive with detailed ablations.

Critical weaknesses

The experiments rely on controlled datasets with one desired and one undesired trait. However, real SFT datasets may contain multiple interacting undesired behaviors, making it unclear whether the method can achieve improvements. Also, the method relies on identifying a clean subset, but the paper does not evaluate the full pipeline of selecting clean examples from real unlabeled data.

Questions for authors
  1. Can the authors provide some new experiments on more realistic SFT datasets, which containing multiple undesired traits to verify the method?
  2. If the clean subset is identified by an imperfect classifier, how robust is the method?
2 comments

We would like to sincerely thank Reviewer gZ8P for their thoughtful assessment and suggestions.

Q1. Multiple undesired traits.

We have now conducted an additional experiment extending our sycophancy setup. It now jointly express three undesired traits: sycophancy, German language, and lowercase format. The desired trait remains an assistant self-introduction.

We use the same model checkpoint and core fine-tuning configuration as the original setup, with one training seed.

Results under standard helpful-assistant prompt:

MethodDesired trait ↑Sycophancy ↓German ↓Lowercase ↓
SFT (5%)99.0 [97.4–100]96.5 [90.7–100]96.1 [92.9–98.9]96.2 [86.3–100]
IP (5%)43.3 [36.1–50.4]17.4 [9.9–24.4]0.6 [0–1.9]0 [0–0]
SIP (5%)88.1 [83.3–92.3]0 [0–2.7]0.8 [0.3–1.2]0 [0–0]

Mean results across non-eliciting prompts:

MethodDesired trait ↑Sycophancy ↓German ↓Lowercase ↓
SFT (5%)100 [100–100]100 [94.9–100]98.8 [97.1–100]100 [96.9–100]
IP (5%)62.6 [52.2–72.7]0 [0–7.3]1.2 [0–3.4]0 [0–0]
SIP (5%)94.9 [89.9–98.7]0 [0–0]2.0 [0.4–3.8]0 [0–0]

Please note that these are normalised trait scores on a 0–100 scale and a clipped zero does not imply that the underlying behaviour is absent.

We observe that SIP substantially reduces all three undesired traits relative to SFT (5%) in both evaluation conditions. Under the standard helpful-assistant prompt, SIP preserves considerably more desired behaviour than IP. Both methods suppress German and lowercase expression, while SIP also achieves higher sycophancy suppression.

Under non-eliciting prompts, SIP’s advantage in desired-trait retention persists, while both methods maintain low normalised leakage scores for all three undesired traits. We also report that compared to SFT (5%), SIP’s desired-trait score is lower across both evaluations.

We have now included this extension in Appendix C.4. These results provide encouraging evidence that SIP can preserve desired learning when several undesired traits co-occur.

Q2. Imperfect clean-subset identification.

SIP as a method is agnostic to the particular data-selection procedure: it can use a clean subset obtained through different selection methods. However, we fully agree that the quality of that subset matters. This motivated our filtering-error experiments (Section 4.3, Figure 3C, and Appendix D.6).

Across six setups and three training seeds, we find asymmetric sensitivity: contamination of the clean subset substantially increases leakage, while unnecessarily inoculating clean examples has comparatively small effects on leakage. This is why we recommend conservative selection that prioritises clean-subset purity.

Our data-construction pipeline uses LLM judges to screen generated responses (Appendix A.2). However, these generated responses are deliberately constructed to express or omit a particular trait. As acknowledged in Section 5, we have not evaluated the complete process of selecting clean examples from an unlabelled corpus.

Dear Reviewer gZ8P.

Thank you again for your feedback!

Given that the discussion period is closing soon, please let us know if there are any remaining questions. We would be more than happy to make any clarifications before the discussion period ends.

Comments on the paper

1 comment

We thank all reviewers for their thoughtful feedback, questions and concrete suggestions for strengthening the paper. We also appreciate their recognition of several strengths:

  • An important problem and a simple intervention. Reviewer gZ8P highlights the importance of limiting undesired behaviours while preserving useful training signals, describing SIP as “plug-and-play.” Reviewer 8j5N also recognises SIP as “a plausible direction” for mitigation.
  • Consistent improvements over IP. Reviewer 8j5N recognises SIP’s improvements in limiting unwanted behaviour and reducing broader harmful generalisation. Reviewer DJwX highlights lower leakage across all six setups, including improvements under prompt categories absent from training.
  • Informative ablations and evidence beyond oversampling. Reviewer gZ8P describes the experiments as “comprehensive with detailed ablations.” Reviewer DJwX identifies the matched-oversampling control as “persuasive evidence” that contextual prompt assignment contributes beyond simply increasing clean-data exposure.

Following this feedback, we have updated the revised version of the paper. We clarified data budgets, baseline comparisons and experimental design, revised the oversampling and password-locking figure captions, and qualified our capability-preservation claims.

We also report the following additions:

  • Effect of retaining mixed data: a new comparison of SIP and clean-only SFT using the same clean data. SIP achieves higher desired-trait retention (Appendix C.2.1). We also report further analysis of the six-setup comparison with matched training positions, including earlier financial checkpoints (Appendix C.2.2).
  • Multiple undesired traits: a controlled extension with three co-occurring undesired traits. SIP substantially reduces all three relative to SFT (5%) while retaining more desired behaviour than IP (5%) (Appendix C.4).
  • EM coverage, coherence-filter sensitivity and human audit: analyses across all three harmful-advice setups, together with a blinded human audit of financial responses (Appendix C.1.1).
  • General capabilities and desired-trait interference: additional comparisons with IP, as well as an analysis of how the desired behaviour can interfere with explicit user instructions (Appendix C.3).

We hope these additions and clarifications address the reviewers’ concerns. We kindly invite the reviewers to consider these results and update their scores if their concerns have been addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.