acceptodds
Withdrawn from ICLR 2027

Where to Anchor? Choosing the Prompt Distribution for Alignment Anchors in Narrow Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-tuning a language model on a narrow task can make it misbehave far outside that task, a problem called emergent misalignment (EM). Defenses keep the model close to its original behavior through its weights, training data, internal activations, or outputs; we ask which of these alignment anchors works best and how to compare them fairly. Such comparisons are easily distorted: a defense can lower EM just by learning the task less, and EM itself is scored by an automatic judge. We show that EM is a measurement-dependent quantity: even with the same frozen responses, changing the thresholds or selector can flip which defense appears best, and no alternative judge agrees closely enough with the primary judge to replace it. Human raters agree with the judge on 83% of responses, and most of their disagreements fall on responses scored right at its thresholds. We therefore propose a protocol that compares defenses only when they learn the task equally well, tunes each on questions kept apart from those used to test it, and reports how each result shifts with these choices. Under this protocol, penalizing how far the model's outputs drift from the original model's gives the lowest observed EM in all three model families tested. Computing this penalty on unrelated prompts lowers EM further, but only together with other changes; changing the prompts alone has no detectable effect. We also explain when unrelated prompts should help: if the same few internal factors drive misbehavior across tasks, an anchor on unrelated prompts can hold them in check without blocking the task.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
emergent misalignment, catastrophic forgetting, self-distillation, plasticity-matched evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_F1umOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors investigate how to prevent EM that appear when one finetune an LLM on a narrow task. they compare KL regularization with some other baselines in the literature. they show EM scores depend on the LLM judge. They also investigate if KL regularization should use prompts that are task specific or unrelated to the task of interest.

Critical strengths

s1) authors investigate the influence of data selection for KL divergence calculation when they have KL divergence as extra regularization term in SFT which normally lead to broad misalignment. which is novel contribution.

Critical weaknesses

w1) the set of baselines is quite limited when they compare their regularization with others. and the approaches they choose aren't even SOTA for mitigating emergent misalignment. see more recent works like block-em.

w2) I guess their KL regularization experiments are inconclusive? (i.e. they dont show any prompt selection is better than other one, so I dont see how is this interesting?)

Questions for authors

none

Reviewer_bwbjOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper compares a set of training-time defenses against emergent misalignment following a fair comparison protocol. Training-time defenses anchor the model to its pre-fine-tuning behaviour through its weights, data, activation or outputs, but existing comparisons neither control how well each defense learns the task nor separate the runs used to tune a defense from those used to evaluate it. The paper compares those four defenses applied to fine-tuning on bad-medical-advice. Defense strength is selected at matched plasticity, aiming at scores between 90% and 100% of the score with plain fine-tuning. Effects are estimated on 78 disjoint questions with five paired seeds. The KL anchors lower EM by about 16 percentage points on Qwen and 14 points on Mistral from a baseline near 21%, against 2-4 pp for replay and 3-8 pp for steering. On Llama no defense is significant at the selected strength; a re-selection at a larger strength gives the anchor a 6 points reduction. Computing the anchor on student rollouts over a mostly off-domain prompt pool lowers EM further on Qwen (2.8% vs 5.3%), but a comparison that changes only the prompt pool finds no detectable effect.

Critical strengths
  1. The comparison protocol is carefully made and is the main contribution of the paper. Strengths are chosen at matched task loss, with selection rule frozen before outcomes, paired seeds, multiplicity correction, a reduced-learning control, and a stated reading rule for null intervals. EM-defense papers rarely control a defense simply learning the task less, and this one does.
  2. The central empirical result is clear and robust to the paper's own sensitivity analyses.
Critical weaknesses
  1. The title question is not answered. The paper asks where an anchor should be placed and how to choose its prompt distribution. What the paper establishes is a ranking of four defenses at matched plasticity on one task. The title and abstract should be reframed around that, or the paper needs an experiment that resolves the support question.

  2. The scope is narrow for a paper about how to compare defenses. One fine-tuning task, one LoRA recipe, three 7-8B models and four defenses; §8 acknowledges that SafeLoRA, Lisa and Vaccine were not run.

  3. The paper is not easy to follow. Writing quality can be greatly improved in terms of having a clear message and a structured presentation.

Questions for authors

I don’t have specific questions, but I am struggling quite a lot to make sense of the paper in terms of delivery a clear message and really get the point of the work. I feel the writing and presentation are not very focussed, and the message is not getting through.

Reviewer_Cn6zOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper is an over-complicated, heavily padded repetition of Kaczér et al. (2026) ("In-Training Defenses against Emergent Misalignment in Language Models") and Soligo et al. (2026). It compares KL regularization, safety data mixing (Replay), and activation steering (Persona Vectors) during narrow fine-tuning on Qwen2.5-7B, Llama-3.1-8B, and Mistral-7B, alongside trivial claims dressed up as major contributions:

  • Treating "using a frozen copy of the base weights" vs. "disabling the LoRA adapter" to compute the exact same reference KL as two separate defense methods in Table 1.
  • Claiming as a core "diagnosis" that changing the binarization threshold of a continuous LLM judge score changes the binary misalignment rate for responses sitting right on the threshold.
  • Describing a standard train/validation split and taking a vertical slice of a Pareto frontier (which Kaczér et al., 2026 already plotted) as a novel "matched-plasticity protocol."
Critical strengths

None

Critical weaknesses
  • Weakness 1 (The title question yields a null result, padded with trivial/unverified math): The paper's title asks "Where to Anchor? Choosing the Prompt Distribution for Alignment Anchors", yet when the authors actually isolate the prompt distribution in their controlled comparison (Table 2, §5.3), off-domain prompts vs. task prompts show no statistically detectable difference (−0.61 pp-0.61\text{ pp} on Qwen, [−1.53,0.05][-1.53, 0.05]; −1.00 pp-1.00\text{ pp} on Mistral, [−2.58,0.16][-2.58, 0.16]). The earlier reduction in Table 7 confounded four changes at once (prompt pool, student rollouts, reverse KL, and a 5×5\times larger regularization weight). Section 6 then pads this null result with textbook math (e.g., that the local Hessian of KL is the Fisher matrix), while Appendix F admits their empirical diagnostic failed to even identify the theoretical subspace UU (<5%<5\% of update energy captured, 123%123\% local quadratic error).
  • Weakness 2 (Misrepresentation of prior work and broken pre-registered protocols): In Section 2, the authors claim Kaczér et al. (2026) "compare defenses without matching task performance." In reality, Kaczér et al. (2026) explicitly plot the 2D Pareto trade-off curve between general EM and narrow-domain task performance across hyperparameter sweeps (Figure 1b). Slicing a Pareto frontier at [0.90 s0,s0][0.90\,s_0, s_0] and holding out tuning prompts is basic ML practice, not a novel "protocol." Worse, the authors' rigid selector rule breaks on Llama-3.1-8B—picking a degenerate weight (w=0.03w=0.03) where no defense works—and their "pre-registered" human audit (§7, Table 10) failed its own conjunctive acceptance gates (κ=0.485<0.60\kappa = 0.485 < 0.60) after the vendor ignored the 2-grader adjudication protocol.
Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.