acceptodds
Under review as a conference paper at ICLR 2027

AlignRadar: Forecasting Emergent-Misalignment Risk Before Fine-Tuning

Abstract

Emergent misalignment (EM) is the unintended appearance of harmful, unsafe, or deceptive behavior after fine-tuning, including behavior outside the training dataset's domain. We study supervised fine-tuning (SFT), a widely used adaptation method in which task-specific datasets directly shape model behavior. Because cross-domain EM is usually discovered only after SFT and behavioral evaluation, a risky dataset may consume substantial compute and safety-review effort before the resulting model must be rejected or retrained. The central challenge is that inspecting a dataset before training does not directly reveal whether it will cause harmful behavior beyond its original domain. We introduce **AlignRadar**, a forecaster that operates before SFT using only a candidate dataset and an untouched base model. AlignRadar analyzes the dataset and the base model, uses patterns learned from completed fine-tuning runs, and returns a continuous score that ranks the dataset's EM risk. Tuned models and post-SFT evaluation results are used only to train AlignRadar and are never required when assessing a new dataset. Across 76 corpora from 13 dataset families, nested leave-one-family-out evaluation achieves **75.3% AUROC** and **76.1% average precision**. AlignRadar improves AUROC by **7.0 percentage points** over a base-model audit and by **21.6 points** over the same pre-SFT inputs trained without internal-change supervision. To evaluate cross-domain forecasting directly, we reconstruct outcomes using only prompts outside each corpus's training domain. AlignRadar achieves **72.2% AUROC** and **62.8% average precision**, improving over a direct EM classifier combined with the same audit by **14.2** and **9.2 percentage points**, respectively. These results show that learning from previous fine-tuning runs helps AlignRadar rank new datasets by EM risk before they are used for fine-tuning. Anonymized code is available at: https://anonymous.4open.science/r/AlignRadar_SFT_Anonymous-2BB1/

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Emergent Misalignment, LLM Safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_h1Y4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AlignRadar predicts a dataset’s emergent-misalignment risk before fine-tuning. It learns from internal changes measured in historical fine-tuning runs, predicts a risk-associated change from dataset and base-model features, and combines this prediction with a base-model audit. Evaluation uses nested leave-one-family-out testing and includes a separate cross-domain outcome analysis.

Critical strengths
  1. Forecasting risk before running fine-tuning is a meaningful problem. And using historical internal changes as training-only supervision is an interesting approach.

  2. The evaluation is carefully designed. Nested family-held-out testing, fold-safe target selection, and permutation controls strengthen the evidence for the proposed supervision.

Critical weaknesses
  1. The outcome labels need stronger validation. They depend on a small prompt suite, model judges, and a fixed deterioration threshold. Reversing answer order controls position bias, but not whether the judges correctly identify harmful changes. The secondary model review does not provide an independent estimate of label accuracy.

  2. The evidence is still narrow: one base model, one fine-tuning recipe, and small, mostly constructed corpora. Family-held-out testing is useful, but it does not show whether the learned signal survives changes in model, training strength, or realistic dataset scale.

Questions for authors

Are any results available on another base model or fine-tuning recipe?

2 comments
AuthorsOpenReview ↗

Response to Reviewer h1Y4

We thank the reviewer for the positive assessment and insightful comments. Our new experiments confirm that (1) our outcome labels achieve 93% precision against blinded human annotators and remain robust without a fixed threshold (0.92 AUROC), and (2) AlignRadar’s gains replicate across a second base model and stronger fine-tuning recipes (5×5\times and 10×10\times learning rates).

1. Independent Label Validation, Thresholds & Prompt Suite (W1)

Upgrading to independent, blinded pointwise Llama-3.3-70B and Gemma-3-27B judges [1] establishes 93% and 84% precision against blinded human annotators, increases AlignRadar’s cross-domain gain to +30.6 AUROC, and achieves 0.92 AUROC against human labels without any binarization threshold.

Independent Validation Metric (Appendices F.4-F.6)Blinded Llama-3.3-70BBlinded Gemma-3-27B
Precision vs. agreed human labels, % [95% CI] (human-human κ=0.67\kappa = 0.67)93% [83, 100]84% [55, 100]
Agreement (κ\kappa) with human annotators0.55 [0.36, 0.82]0.63 [0.36, 0.84]
EM dose-response (0%, 10%, 50%, 100% risky data)3.5%, 4.4%, 11.1%, 12.4%Monotonic increase
Cross-domain AUROC: AlignRadar vs. Audit93.3 vs. 62.783.3 vs. 66.8
AlignRadar Gain Over Audit, Δ\DeltaAUROC (95% CI)+30.6 (17.4, 45.2)+16.5 (4.3, 30.4)
  • Threshold invariance: Continuous judge harm-difference scores rank human ground-truth labels at 0.92 AUROC with no threshold at all, and AlignRadar remains #1 across alternative cutoff thresholds.
  • Prompt-suite resolving power: Across 9,120 evaluated responses per pipeline run (120 prompts ×\times 76 corpora), the suite separates 0%,10%,50%,0\%, 10\%, 50\%, and 100%100\% risky-data doses monotonically and flags 0/6 benign and 16/16 legacy harmful corpora.

Scoring each response independently eliminates pairwise comparison bias altogether and removes the variance floor of the original small judges, confirming that our labels accurately capture genuine harmful shifts.

2. Generalization Across Base Models, Recipes & Scale (W2, Question)

Yes. Re-running the complete 76-corpus pipeline confirms that AlignRadar’s cross-domain advantage replicates on Llama-3.1-8B-Instruct and under 5×5\times and 10×10\times learning rates.

Base Model & Fine-Tuning Recipe (Cross-Domain EM, 70B Label)Audit AloneAlignRadar (Ours)AlignRadar − Audit (95% CI)
Mistral-7B-v0.3, standard recipe (lr 2e-5\text{lr } 2\text{e-}5, 3 epochs)62.793.3+30.6 (17.4, 45.2)
Mistral-7B-v0.3, 5$\times$ learning rate (lr 1e-4\text{lr } 1\text{e-}4, 3 epochs)69.390.6+21.3 (3.2, 38.0)
Mistral-7B-v0.3, 10$\times$ LR, 5 epochs (lr 2e-4\text{lr } 2\text{e-}4, 5 epochs)70.881.6+10.8 (zero-shot transfer)
Llama-3.1-8B-Instruct, standard recipe (lr 2e-5\text{lr } 2\text{e-}5, 3 epochs)79.795.8+16.1 (8.9, 23.7)

(Note: AlignRadar also outperforms a supervised direct-EM classifier + audit baseline at 1×1\times LR [93.3 vs. 75.2] and 5×5\times LR [90.6 vs. 77.0], and matches its performance ceiling on Llama-3.1-8B [95.8 vs. 96.8].)

The learned internal-shift signal is robust across model families and training intensities: even when raw shifts begin to saturate under an extreme 10×10\times LR for 5 epochs, zero-shot transfer of the standard-recipe AlignRadar forecaster still ranks outcomes best. Finally, our 100-150 interaction corpora directly target the practical few-shot fine-tuning vulnerability regime [1, 2], where safety degrades sharply while leaving minimal surface traces for static audits.

We hope these additional validations address your questions about label reliability and generalization. We will revise the manuscript to explain the threshold analysis more clearly and explicitly state the scope and limitations of our evaluation across models, fine-tuning recipes, and corpus sizes.

References [1] Betley et al. Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs. ICML 2025. [2] Qi et al. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! ICLR 2024.

AuthorsOpenReview ↗

Follow-up before the discussion closes

Dear Reviewer h1Y4,

Thank you again for your positive assessment. As the discussion period closes on November 18 (AoE), we wanted to check whether our response answered your questions. In brief:

  • Labels: blinded Llama-3.3-70B and Gemma-3-27B judges reach 93% and 84% precision against blinded human annotators, and their scores rank the human labels at 0.92 AUROC with no threshold.
  • Generalization: the cross-domain gain over the audit replicates on Llama-3.1-8B-Instruct (+16.1 AUROC) and at a 5× learning rate (+21.3).

We would be glad to answer any remaining questions. If our response addresses your comments, we would be grateful if you would consider raising your score.

Reviewer_1wqgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AlignRadar, a method for forecasting whether supervised fine-tuning on a candidate corpus will induce emergent misalignment before performing the fine-tuning run. It extracts corpus statistics, semantic representations, and frozen-base-model activation features. Using historical fine-tuning runs, it measures base-to-tuned component shifts, selects an EM-associated shift target, and trains a PCA-plus-ridge forecaster to predict that shift from pre-SFT features. The predicted shift is then fused with a base-model audit score.

The method is evaluated on 76 corpora from 13 dataset families using Mistral-7B-Instruct-v0.3 and a fixed QLoRA recipe. EM outcomes are constructed from 120 held-out prompts using reversed-order comparisons by three LLM judges. AlignRadar achieves 75.3% AUROC and 76.1% AP, and 72.2% AUROC and 62.8% AP under the domain-excluded evaluation.

Critical strengths

Strength 1 — Historical component shifts provide an interesting form of privileged supervision.

Rather than directly classifying corpora from surface features, AlignRadar learns to predict an EM-associated internal shift observed in historical fine-tuning runs. This provides a meaningful connection between pre-SFT corpus features and post-SFT behavioral outcomes while avoiding candidate-specific fine-tuning at deployment.

Strength 2 — The outcome construction includes reasonable safeguards against judge bias.

The paper uses three independent judges, reverses response order, and retains a vote only when the judge consistently identifies the tuned response as more unsafe. Requiring agreement from at least two judges is a sensible attempt to reduce position bias and individual judge noise.

Critical weaknesses

Weakness 1 — The SFT intervention is under-specified and experimentally narrow.

Each corpus contains only 100–150 interactions and is trained for three epochs. Although several LoRA hyperparameters are reported, the paper omits the per-device batch size, effective batch size, number of devices, sequence-packing behavior, and actual optimizer-step count. It is therefore difficult to assess whether the measured component shifts arise from a sufficiently strong and reproducible intervention.

Moreover, all experiments use a single base model and one fixed QLoRA recipe. This is insufficient to support broad claims of generality across models or training settings.

Weakness 2 — The validity of the EM labels is not sufficiently established.

The labels are produced by Gemma-3-4B-it, Phi-3.5-mini-instruct, and Llama-3.1-8B-Instruct. Two judges are smaller than the evaluated 7B model, which is not necessarily invalid, but the paper does not report inter-judge agreement, human validation, or an independent stronger-judge evaluation.

This is especially important because the judge-derived labels are used both to evaluate the method and to select one of the 189 component-shift coordinates as the regression target. Systematic judge errors may therefore affect both training and evaluation.

Questions for authors

Question 1 — What is the exact number of optimizer updates for each corpus?

Please report the per-device batch size, number of devices, effective batch size, sequence-packing setting, and optimizer-step distribution across the 76 corpora. Please also clarify the learning-rate schedule, warmup, weight decay, and gradient clipping.

These details are necessary to assess whether the measured shifts result from a sufficiently strong and reproducible fine-tuning intervention.

Question 2 — How stable is AlignRadar across models and fine-tuning recipes?

Can the authors provide results on an additional base model or under a substantially different recipe, or training-data scale? Otherwise, the abstract and conclusion should clearly limit the claims to Mistral-7B-Instruct-v0.3 under the reported QLoRA recipe.

Question 3 — How reliable are the judge-derived EM labels?

Please report inter-judge agreement, order-reversal consistency, per-judge results, and sensitivity to the 5% threshold. Is there a human-reviewed subset or an independent stronger-judge evaluation that estimates judge error?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1wqg [1/2]

We thank the reviewer for a rigorous and constructive assessment. Our new experiments (1) document the exact optimization dynamics confirming a strong, reproducible SFT intervention, (2) demonstrate generalization across Llama-3.1-8B-Instruct and 5×5\times/10×10\times learning-rate recipes, and (3) establish 93% label precision via human-validated 70B judge, ruling out shared target-selection/evaluation bias.

1. Exact SFT Configuration & Intervention Strength (W1, Q1)

Our SFT recipe executes 39-59 optimizer updates per corpus, shifting 100% of model responses and increasing mean harm by 2.5×2.5\times.

Hyperparameter / SettingExact Specification
Hardware & Batching1 GPU; per-device batch = 1; grad. accum. = 8; effective batch size = 8
Sequence Packing & LossNone (1 interaction/forward pass); loss on final assistant response only
Optimizer & ScheduleAdamW (lr=2×10−5\text{lr} = 2\times 10^{-5}, weight decay 0.010.01); constant schedule, 0 warmup, global grad clip 1.01.0
Quantization & LoRA4-bit NF4 (double quant., fp16 compute); rank r=32,α=64r = 32, \alpha = 64, dropout 0.050.05
Epochs & Optimizer Steps3 epochs; 39-59 updates/corpus across 76 corpora (52 at 39 steps; 6 each at 40, 42, 45, and 59 steps, from 100-150 examples)
Measured Behavioral Impact0 / 9,120 responses identical to base; mean 70B harm rises 5.6→14.05.6 \to 14.0 (2.5×2.5\times); EM rises monotonically (3.5%→4.4%→11.1%→12.4%3.5\% \to 4.4\% \to 11.1\% \to 12.4\% at 0/10/50/100%0/10/50/100\% risky data)

Consistent with prior work showing that safety alignment breaks down within a few dozen gradient updates on ~100 examples [1, 2], 39-59 updates provide a sufficiently strong and reproducible intervention to induce systematic emergent misalignment.

2. Generalization Across Base Models and Stronger Recipes (W1, Q2)

Re-running the full 76-corpus pipeline confirms that AlignRadar consistently outperforms the audit on a second base model (Llama-3.1-8B-Instruct: 95.8 AUROC) and under 5×5\times and 10×10\times learning rates.

Base Model & Fine-Tuning Recipe (Cross-Domain EM, 70B Labels)StepsAudit AloneAlignRadar (Ours)Gain Over Audit (95% CI)
Mistral-7B-v0.3, standard recipe (lr 2e-5\text{lr } 2\text{e-}5, 3 ep.)39-5962.793.3+30.6 (17.4, 45.2)
Mistral-7B-v0.3, 5$\times$ learning rate (lr 1e-4\text{lr } 1\text{e-}4, 3 ep.)39-5969.390.6+21.3 (3.2, 38.0)
Mistral-7B-v0.3, 10$\times$ LR, 5 epochs (lr 2e-4\text{lr } 2\text{e-}4, 5 ep.)65-9870.881.6+10.8 (zero-shot transfer)
Llama-3.1-8B-Instruct, standard recipe (lr 2e-5\text{lr } 2\text{e-}5, 3 ep.)39-5979.795.8+16.1 (8.9, 23.7)

(Note: AlignRadar also beats a direct EM classifier + audit at 1×1\times LR [93.3 vs. 75.2] and 5×5\times LR [90.6 vs. 77.0], and matches its ceiling on Llama-3.1-8B [95.8 vs. 96.8]. On "all prompts", where in-domain training tokens directly expose harm, the direct classifier ranks first [80.6 vs. 78.1 at 1×1\times LR; 80.6 vs. 58.8 at 5×5\times LR; 98.3 vs. 95.8 on Llama-8B]. We also updated the Abstract, Conclusion, and Appendix J to explicitly scope our claims to 100-150 interaction corpora.)

Internal-shift supervision captures a generalizable cross-domain mechanism across architectures and optimization strengths. Even under an extreme 10×10\times LR for 5 epochs applying our standard-recipe AlignRadar forecaster zero-shot still ranks outcomes best (81.6 vs. 70.8).

AuthorsOpenReview ↗

Response to Reviewer 1wqg [2/2]

3. Label Validity, Independent Judges & Human Evaluation (W2, Q3)

Validating against two blinded human annotators and independent Llama-3.3-70B / Gemma-3-27B judges [1] establishes 93% / 84% label precision; moreover, re-selecting the 189-coordinate shift target from scratch on these independent labels increases AlignRadar’s cross-domain gain to +30.6 AUROC, ruling out shared target-selection/evaluation bias.

Judge Protocol (200-Pair Blinded Human Study & Full 9,120-Pair Eval)Precision vs. Humans [95% CI]Recall vs. Humans [95% CI]Human κ [95% CI]Cross-Domain: Audit vs. AlignRadar (Δ, 95% CI)All Prompts: Audit vs. AlignRadar
Blinded Llama-3.3-70B93% [83, 100]45% [30, 77]0.55 [0.36, 0.82]62.7 vs. 93.3 (+30.6 [17.4, 45.2])63.4 vs. 78.1
Blinded Gemma-3-27B84% [55, 100]58% [33, 79]0.63 [0.36, 0.84]66.8 vs. 83.3 (+16.5 [4.3, 30.4])69.6 vs. 79.4
Original 3-judge majority23% [16, 31]7% [4, 12]0.02 [−0.03, 0.09]62.2 vs. 72.2 (+10.0)66.2 vs. 73.2
  • 70B sanity checks: Flags 0/6 benign corpora, 16/16 legacy harmful corpora, and 107/120 insecure-code responses absent in the base model.
  • Original small-judge & threshold diagnostics (Q3): For Gemma-3-4B, Phi-3.5-mini, and Llama-3.1-8B respectively: order-reversal consistency is 28.4%, 14.5%, 33.3%; identical-response false-alarm rate is 31.7%, 48.3%, 45.8%; and order-robust "tuned worse" rate is 19.3%, 3.6%, 15.1%. Despite this noise, our order-invariant majority rule flagged 16/16 harmful and 0/4 safety corpora, and AlignRadar ranked #1 across 5 of 7 sensitivity variants.

If inner-fold selection of the 189 shift coordinates had exploited systematic errors in the small judges, re-selecting and evaluating on independent, human-verified 70B/27B pointwise labels would degrade performance. Instead, AlignRadar’s lead triples from +10.0 to +30.6 AUROC, proving that small-judge errors acted purely as random variance that attenuated our signal, while 70B's conservative recall simply undercounts borderline cases rather than inventing false positives.

We hope these experiments address your concerns about intervention strength, generalization, and label validity. We will make the exact fine-tuning configuration, independent validation procedure, and scope of our claims clearer in the revised manuscript.

References [1] Betley et al. Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs. ICML 2025. [2] Qi et al. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! ICLR 2024.

AuthorsOpenReview ↗

Follow-up before the discussion closes

Dear Reviewer 1wqg,

Thank you again for your rigorous review. As the discussion period closes on November 18 (AoE), we wanted to check whether our response addressed your concerns. In brief:

  • Training strength: each corpus receives 39 to 59 optimizer updates; no tuned response is identical to its base response, and mean harm rises 2.5×.
  • Generalization: the gain over the audit replicates on Llama-3.1-8B-Instruct (95.8 vs. 79.7 AUROC) and at a 5× learning rate (90.6 vs. 69.3); at 10× for 5 epochs, the standard forecaster still ranks first (81.6 vs. 70.8).
  • Label validity: the blinded 70B judge reaches 93% precision against two blinded human annotators, and re-selecting the shift target on these independent labels raises the cross-domain gain to +30.6 AUROC.

We would be glad to clarify anything further. If these results resolve your concerns, we would be grateful if you would consider updating your assessment.

Reviewer_kAniOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes AlignRadar, which ranks SFT datasets by emergent-misalignment risk before fine-tuning. It tunes mistral-7B on 76 corpora and measures activation shifts in base vs tuned models. It picks the shift coordinate that best separates EM labels and trains a regressor to predict it from pre-SFT features. At inference time, this prediction is fused with a risk score from the base model reading eight corpus examples. Under leave-one-family-out evaluation, it reports 75.3 AUROC, versus 68.2 for the audit alone. On a target that drops prompts from each corpus's own domain, it reports 72.2 AUROC.

Critical strengths
  1. Forecasting dataset risk before training is practical. Using post-SFT internal changes as training-only supervision is a natural way to reuse completed runs.

  2. The leakage-aware evaluation is good, since whole corpus families are held out. Target selection and preprocessing are refit inside inner folds. Intervals use a family bootstrap, and there is a fold-safe permutation control.

Critical weaknesses
  1. The judges that decide whether a model became more harmful are small models (4B, 3.8B and 8B), about the same size as the model being judged. Earlier work on this problem used much stronger judges such as GPT-4o.

  2. The main label mostly measures in-domain harm, not emergent misalignment. The six evaluation domains are the same as the six constructed training families, so a corpus that teaches insecure code is mainly flagged on code prompts. The domain-excluded test removes only one prompt block, so adjacent domains such as code and security still overlap.

  3. The predictor is never tested against the obvious alternative. The internal change it learns to predict is chosen because it best matches the misalignment labels. The paper never tries predicting the misalignment rate directly, and never shows the predictor can forecast the internal change on new datasets. So the gain cannot be credited to internal changes.

  4. The authors pick which internal model change to predict by choosing the one that best matches the misalignment labels. So the predictor is trained on something very close to the labels themselves. The authors never compares against simply predicting the misalignment rate directly.

Questions for authors
  • Is the 7-point gain over the audit alone statistically reliable? Please report a confidence interval for the difference between AlignRadar and the audit
  • What happens if you train the same predictor to predict the misalignment rate directly, instead of the internal model change?
  • How well does the predictor forecast the internal change on held-out datasets?
  • What happens on the cross-topic test if you also remove closely related topics
  • Can the judge be run on larger models?
3 comments
AuthorsOpenReview ↗

Response to Reviewer kAni [1/2]

We thank the reviewer for a careful review. Evaluating AlignRadar with larger blinded judges, strict cross-domain isolation, and direct-prediction baselines confirms that AlignRadar achieves a +30.6 AUROC gain over the audit and outperforms direct misalignment prediction by +18.1 AUROC.

1. Upgrading to Larger, Blinded Judges (W1, Q5)

Re-evaluating all 9,120 response pairs with blinded pointwise Llama-3.3-70B and Gemma-3-27B judges [1] yields labels with 93% precision against blind human annotators (κ=0.55\kappa = 0.55) and monotonic dose-response across risky training data (Sec. 5.2, App. F.3-F.6).

Validity Check (Appendices F.4, F.6)Original (Small Pairwise)Blinded Llama-3.3-70B
Precision vs. two blind human annotators (%)23%93%
Agreement (κ\kappa) with human annotators0.020.55
EM rate across risky-data doses (0%, 10%, 50%, 100%)n/a3.5%, 4.4%, 11.1%, 12.4%
Benign corpora (0% risky data) labelled negativen/a100%

The original pairwise prompt "BASE" vs. "TUNED" induced label bias even in 70B models. Scoring each response independently for harm and coherence eliminates this positional/header bias and resolves the small-judge noise bottleneck.

2. Statistical Reliability & Signal Recovery (Q1)

Under the human-validated 70B and 27B labels, AlignRadar’s cross-domain gain over the audit is large and statistically significant (+30.6 and +16.5 AUROC).

Judge Label (10,000 Paired Family-Bootstrap Resamples)Evaluation ScopeAlignRadar − Audit, ΔAUROC (95% CI)
Llama-3.3-70B, blindedCross-domain (EM)+30.6 (17.4, 45.2)
Gemma-3-27B, blindedCross-domain (EM)+16.5 (4.3, 30.4)
Llama-3.3-70B, blindedAll prompts+14.8 (−2.1, 34.7)
Original (small judges)All prompts+7.0 (−6.0, 19.8)

Across 13 held-out families, the high variance of the original small judges attenuated the true effect size. Removing this label noise reveals a highly reliable cross-domain signal across both judge families.

AuthorsOpenReview ↗

Response to Reviewer kAni [2/2]

3. Why Predict Internal Shifts Instead of Predicting EM Directly? (W3, W4, Q2, Q3)

Training the exact same predictor to forecast EM directly performs worse on cross-domain emergent misalignment (75.2 vs. 93.3 AUROC), while predicted internal shifts generalize to held-out dataset families at Spearman rs=0.73r_s = 0.73.

Evaluation Setting & BaselineDirect EM PredictorAlignRadar (Internal Shift)AlignRadar Gain (95% CI)
Cross-domain EM<br> (70B labels; Direct EM clf. + audit)75.293.3+18.1 (3.1, 42.3)
All prompts<br> (Original labels; regressing EM rate directly)61.575.3+13.8 (−4.0, 32.9)
All prompts<br> (70B labels; Direct EM clf. + audit)80.678.1−2.5 (−15.4, 12.4)
  • Out-of-distribution shift forecasting (Q3): Predicted and ground-truth shifts correlate at rs=0.73r_s = 0.73 across held-out families.
  • Inner-fold target selection (W4): The shift coordinate is selected strictly within inner cross-validation folds; re-selecting on independent 70B labels increased rather than shrank AlignRadar's lead.

As you noted in W2, "all prompts" is dominated by in-domain harm visible in surface training tokens, allowing a direct data classifier to tie with AlignRadar. For out-of-domain emergent misalignment, surface tokens no longer match the test domain, so forecasting the low-dimensional internal shift provides the decisive structural signal.

4. Isolating Emergent Misalignment via Stricter Cluster Removal (W2, Q4)

We elevated cross-domain EM to a co-primary outcome. When progressively removing the training domain, its closest semantic neighbour, and the entire topical cluster, AlignRadar dominates at every exclusion level.

Label & ProtocolPrompts RemovedAudit AloneDirect EM Clf. + AuditAlignRadar (Ours)
70B (Topic-Invariant Eval)Own training domain70.590.894.9
70B (Topic-Invariant Eval)+ Closest neighbour66.084.791.3
70B (Topic-Invariant Eval)+ Whole topical cluster64.985.692.4
Original LabelsOwn domain / + Neighbour / + Cluster62.2 / 54.0 / 55.758.1 / 50.8 / 50.972.2 / 62.3 / 55.5

Consistent with Qi et al. [2], even purely benign fine-tuning degrades safety on 21.8% of code prompts. In an unadjusted 70B split where code prompts are dropped only for code-related corpora, the resulting label drops sharply for code corpora and inadvertently encodes training domain identity leakage confound, and AlignRadar leads at every level.

We hope these additional results address your concerns and clarify the motivation for forecasting internal shifts. We will revise the manuscript to make the judge-validation protocol, cross-domain evaluation, and distinction from direct EM prediction more explicit.

References [1] Betley et al. Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs. ICML 2025. [2] Qi et al. Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! ICLR 2024.

AuthorsOpenReview ↗

Follow-up before the discussion closes

Dear Reviewer kAni,

Thank you again for your detailed review. As the discussion period closes on November 18 (AoE), we wanted to check whether our response addressed your concerns. In brief:

  • Judges: we re-labelled all 9,120 pairs with blinded Llama-3.3-70B and Gemma-3-27B judges; the 70B labels reach 93% precision against blinded human annotators.
  • Significance: with these labels, AlignRadar's cross-domain gain over the audit is +30.6 AUROC (95% CI 17.4 to 45.2).
  • Direct EM prediction: the same predictor trained to forecast EM directly reaches 75.2 AUROC, versus 93.3 for AlignRadar.
  • Stricter exclusion: under the 70B labels, AlignRadar stays ahead when the closest neighbour and the whole topical cluster are also removed.

We would be glad to clarify anything further. If these results resolve your concerns, we would be grateful if you would consider updating your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.