- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies selective generalisation during supervised fine-tuning when desired and undesired traits co-occur in most training examples. It proposes stratified inoculation prompting (SIP): high-confidence clean examples are paired with diverse non-eliciting prompts, while the remaining examples receive an inoculation prompt requesting the undesired trait. In the main configuration, a 5% distinct clean subset is repeated to occupy 25% of a fixed training-position budget.
Experiments cover six model–trait setups, including three harmful-advice domains, with three training seeds per condition. SIP reduces undesired-trait leakage relative to IP across the evaluated setups and improves desired-trait retention on average. A matched-oversampling IP control supports an effect of prompt assignment beyond increased exposure to clean examples. The evaluation includes held-out prompt categories, emergent misalignment, and ablations of prompt assignment, clean-data quantity, repetition and contamination. The paper also introduces backdoor dilution and password-conditioned inoculation to reduce or concentrate residual elicitation.
The main contribution is a simple, empirically supported modification to IP for controlled two-trait settings. The results concern reduced behavioural expression under tested conditions; undesired behaviour remains accessible.
Critical strengths
- The central intervention is supported by a strong control that isolates prompt assignment.
SIP addresses a concrete limitation of uniform IP: clean examples should demonstrate that the desired trait persists outside contexts requesting the undesired trait. The matched-oversampling comparison is particularly important. In Section 4.3 and Appendix D.2, oversampled IP and SIP use the same training examples, sampling frequencies, optimisation settings and total training budget, differing only in the prompts assigned to clean examples. SIP improves both aggregate desired-trait retention and undesired-trait leakage in this comparison. This provides persuasive evidence that the proposed contextual assignment contributes beyond simply increasing clean-data exposure, and makes the method a useful addition to existing IP approaches.
- The leakage improvement is consistent across the tested settings, and the ablations make the empirical contribution informative.
Table 12 reports lower leakage for SIP than IP in all six setups, with all six paired 95% bootstrap intervals for the difference below zero. Appendix C.1 further reports lower point estimates in all 36 setup–prompt-family comparisons, including categories absent from training; 33 of these paired intervals exclude zero. The harmful-advice experiments extend the study beyond stylistic traits, with lower conditional EM rates than IP in all three domains. The prompt-assignment, oversampling and clean-branch ablations also help explain when the approach works. In particular, Appendix D.5 shows that clean examples must demonstrate the desired trait, rather than merely omit the undesired trait. These findings support acceptance as a substantive empirical contribution.
Critical weaknesses
- The practical advantage over training on the scarce clean subset alone remains incompletely established.
I appreciate that Appendix C.2 includes a matched-step clean-only control. However, this control substantially changes the practical comparison: Table 14 reports normalised DT retention of 0.910 and leakage of 0.046, versus 0.970 and 0.115 for SIP. Thus, the comparison establishes a retention–leakage trade-off. The repeated clean-only baseline does suffer severe generation deterioration in the financial setup, with 287/600 local responses excluded, but the corresponding exclusions in the other five setups range from 0 to 23/600 (Table 16). Since the financial failure is interpreted as overtraining, results at intermediate training durations or appropriately selected learning rates could materially change the comparison. This matters because preserving useful learning that would otherwise be lost through filtering is a central motivation. The matched-step control and its heterogeneous outcomes deserve greater prominence in the main assessment.
- The broader safety and capability-preservation conclusions need stronger qualification.
The EM metric is conditional on response validity and coherence, and these filters retain substantially different populations across methods. In the financial setup, only 415–466 of 1,200 SIP responses per seed enter the EM denominator, compared with 566–647 for IP (Table 13). The reduction from 28.5% to 12.6% therefore needs to be interpreted alongside the substantial loss of eligible responses. The sports and medical results, where SIP has comparable or better eligibility, provide useful supporting evidence; nevertheless, the financial result needs sensitivity analysis and validation of the excluded cases. The paper reports manual spot checks but no quantitative independent-judge or human agreement assessment.
Capability preservation also requires more precise wording. Section 4.5's “no negative impact” heading is stronger than the evidence: Table 18 reports loose-IFEval decreases of 8.38, 4.87 and 5.42 percentage points in three setups, with pointwise 95% intervals below zero. These capability comparisons are against SFT (5%), without corresponding IP or base-checkpoint results. These issues are fixable, but they limit how broadly the safety and utility claims can be interpreted.
Questions for authors
- Additional analysis of the clean-only baseline.
Can you clarify whether the financial generation failure of the matched-step clean-only baseline persists at intermediate training durations or lower learning rates? If checkpoints or additional runs are available, please report the joint DT-retention, leakage and response-validity comparison using the same 250 clean examples. Comparable validation and tuning budgets for SIP and clean-only SFT would help establish the practical advantage of retaining mixed data. Table 14 makes this comparison consequential to my assessment.
- Sensitivity of the EM result to coherence filtering.
For the existing EM generations, can you report validity/coherence coverage together with harmful-output and coherent-safe-output frequencies, and examine whether the conclusions hold when borderline-coherent responses are retained? Please also clarify what harmful content, if any, occurs in the excluded responses. A blinded human or independent-judge audit of a representative subset would be particularly useful in the financial setup, where most generated responses do not enter the reported EM denominator.
- Scope of the capability-preservation claim.
How should the Section 4.5 heading be reconciled with the negative IFEval differences in Table 18 and Appendix C.3's explicit statement that the aggregate results do not establish equivalent instruction-following performance? Are capability results available for IP, oversampled IP or the base checkpoints? These comparisons would clarify whether SIP preserves capabilities relative to its principal competing method, and would help determine the appropriate scope of the claim.
5 comments
We also obtained a blinded, single-rater human audit of a subset of stratified financial responses.
Among the valid responses, human and LLM harmfulness labels agreed on 45/54 responses (83.3%). Using identical category definitions, the human rater classified 37/54 (68.5%) as coherent, compared with 18/54 (33.3%) for the LLM judge.
This suggests that the LLM judge applied the shared coherence criteria more strictly on the audited responses. We therefore expect that reported coverage could be generally higher under human assessment.
Q3: Scope of the capability-preservation claim.
We have now changed Section 4.5 to “Effects on general capabilities” and explicitly acknowledge the setup-specific IFEval decreases and the inconclusive aggregate instruction-following comparison.
We have also added capability comparisons with IP (5%):
Capability differences between SIP and IP. (higher is better)
| Benchmark | SIP − IP (5%) |
|---|---|
| IFEval, strict | −5.45 [−7.05, −3.85] |
| MATH-500 | −1.37 [−3.00, 0.30] |
| MMLU-Pro | −2.03 [−2.83, −1.21] |
Relative to IP, SIP has lower average IFEval and MMLU-Pro accuracy, while the MATH-500 difference remains inconclusive. SIP’s improvements over IP therefore can come with some capability trade-offs. We now state these capability costs explicitly. The SIP vs IP comparison is now also reported in Appendix C.3, Table 24.
Thank you for your response. I understand your point to some extent; however, the current results could potentially affect the extent of the major revisions required for the paper. Therefore, I will maintain my current score. Should the other reviewers feel that their concerns have been addressed, I would be happy to raise my score.
We would like to sincerely thank Reviewer DJwX for their thorough assessment and questions.
Q1: Additional analysis of the clean-only baseline.
We agree that Table 14 in the submitted version (now Table 18) establishes a retention–leakage trade-off. We now give this comparison greater prominence in Section 4.3 and Appendix C.2.
We run an additional small scale analysis for the financial setup, evaluating clean-only SFT on the same 250 clean examples after 1, 5 and 10 epochs:
These results are now included in Appendix C.2.2, Table 21. The severe generation deterioration is absent at these earlier checkpoints. However, all three have lower desired-trait and leakage than SIP.
Q2: Sensitivity of the EM result to coherence filtering.
We re-analysed all IP (5%) and SIP (5%) responses across the three harmful-advice setups and conducted a blinded human audit on the finance setup. We added the results to Appendix C.1.1, Tables 14–16.
Here are the requested validity/coherence coverage together with harmful-output and coherent-safe-output frequencies.
Generation-valid (higher is better)
Coherence-pass / EM-eligible (higher is better)
Harmful at any coherence (lower is better)
Coherent and safe (higher is better)
SIP obtains comparable generation validity in all setups. It increases coherent-safe output frequency in sports and medical. In the financial setup, eligible coverage decreases by 13.9 percentage points.
The frequency of harmful responses is lower in all three setups.
We also examined whether this reduction persists at different coherence thresholds.
Pass only (lower is better)
Pass or borderline (lower is better)
No coherence restriction (lower is better)
We report that SIP has lower harmfulness rates in every setup under all different criteria.
Moreover, we examined the generation-valid responses excluded by the original coherence filter:
Harmful fraction among excluded responses (lower is better)
Harmful labels occur among excluded responses from both methods, with a lower observed fraction for SIP in every setup. Including these responses preserves SIP’s lower harmfulness. Therefore, coherence exclusions do not explain the reduction in harmfulness.