acceptodds
Under review as a conference paper at ICLR 2027

Alignment Forecasting: Predicting Misalignment From Training Data

Abstract

Training a language model on data with a narrow flaw can sometimes make the model broadly misaligned. Inspecting the data at face value often does not settle whether it will emerge, and today it is caught only after training, by auditing the resulting model. To complement post-hoc audits, we introduce Alignment Forecasting: the task of predicting alignment failures before training. Given a target model, a fine-tuning dataset, and a failure mode such as deception or sycophancy, a forecaster outputs the probability that fine-tuning would meaningfully increase that failure mode. To measure progress on alignment forecasting, we introduce ALIGNMENTFORECASTBENCH, a benchmark of over 5,000 forecasting questions spanning 17 target models, 32 datasets, and 16 failure modes. Frontier models prompted directly perform poorly on ALIGNMENTFORECASTBENCH. We therefore propose a forecasting scaffold in which an LLM reads the dataset and rates how strongly and broadly it pushes the model toward misbehavior, and a simple learned model combines that rating with the failure mode’s base rate and the target model’s prior tendency. This forecasts well above chance, and beats a model fine-tuned on the task and a simple forecaster allowed to see how weaker models behaved after fine-tuning on the same data. Its signals also flag problematic training examples that a frontier-model classifier misses. Filtering those examples out from real post-training data such as UltraChat results in more aligned models on our multiple-choice evaluation, though the benefit in open-ended conversations is unclear. More progress is needed before forecasts can reliably guide training data curation in practice, but our results suggest that forecasting many alignment failures before training can be tractable in the SFT setting.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Alignment, Forecasting, SFT

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_upgZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

the paper introduces alignment forecasting, the task of predicting before fine-tuning whether a given training dataset will make a language model more likely to exhibit a particular failure mode. the motivation is that narrow problems in training data can sometimes produce much broader misalignment, while current approaches usually detect this only after the model has already been trained. the authors build alignmentforecastbench with over 5k forecasting questions across 17 models, 32 datasets and 16failure modes. each outcome is measured with multiple choice evaluations before and after fine-tuning, where a change counts only if it is statistically significant and clearly exceeds the drift caused by benign fine-tuning. the proposed system lets an llm read the training data and score how strongly it pushes toward problematic behavior, and a logistic regression combines those scores with historical failure rates and the target model's behaviour before training. on heldout triples the system forecasts well above chance and beats the tested baselines, and the authors also use the signals to drop problematic training examples, which helps on their multiple choice evaluation but is much less convincing in openended behavioural audits.

Critical strengths

the task and the benchmark are a real contribution, since forecasting whether training data will create alignment problems before running the finetune is useful and the paper makes it concrete and measurable. the benchmark mixes failure modes and both synthetic and real posttraining data, holds out stronger models and unseen datasets, and tries to separate genuine training induced change from ordinary drift. the system also improves clearly on the reference baselines and is better calibrated than directly prompted frontier models, which is not trivial given that it does not need to finetune a reference model on the candidate data. second, the analysis is unusually thorough and sometimes unflattering, since the authors report label stability, alternative splits, feature importance and the actual data intervention, and they acknowledge that their method is closer to a dataset hazard estimator and that random deletion can beat their filtering on the strongest model

Critical weaknesses

the first weakness is that the proposed decomposition is less convincing than the headline suggests and does not really solve the model specific part of the task. most of the features do not depend on the target model, the only model specific one is not significant under clustered standard errors, and within a fixed dataset adding the content signals barely moves the metric. table 10 is also surprising, since dropping the broad spillover score and the behaviour specific coherence score both help slightly while dropping them together clearly hurts, so reading the data helps but the four feature decomposition is not shown to be needed, and the contribution is better described as forecasting which datasets are risky than how a particular model will react.

the second weakness is the gap between the benchmark labels and the behavioural risk the paper ultimately wants to predict. the labels come from model generated multiple choice questions with one deliberately misaligned answer, which is a reasonable but artificial starting point, and i found no independent human validation of how well they track genuinely concerning behaviour. the downstream results make this matter, since the reductions from filtering are small on most models, the size matched random control does better on one of them, and in the behavioural audit the difference to no filtering is not statistically established, while the error bars also resample forecasts instead of clustering by model and dataset. overall the paper shows that its method predicts its own operationalised labels, but the link to preventing meaningful real world misalignment needs stronger evidence

Questions for authors

could you evaluate directly how well the forecaster distinguishes between different target models when dataset and failure mode are fixed, since table 11 suggests most of the predictive power comes from dataset level risk. could you also compare against a simpler forecaster using only the failure mode prior and one single llm generated dataset risk score, given that dropping either content score in table 10 does not hurt. could you report uncertainty for the main comparisons that accounts for the shared models and datasets. could you validate the emergence labels against independently evaluated open ended behaviour for a subset of the same combinations. and could you report repeated size matched random deletion controls for the filtering result together with a check of whether removing that much training data hurts ordinary task performance

7 comments

Thank you for reviewing our work, for the positive rating, and for recognizing that our analysis is "unusually thorough". We answer your questions below:

Weakness 1: the first weakness is that the proposed decomposition is less convincing than the headline suggests {...} table 10 is also surprising, since dropping the broad spillover score and the behaviour specific coherence score both help slightly while dropping them together clearly hurts, so reading the data helps but the four feature decomposition is not shown to be needed, and the contribution is better described as forecasting which datasets are risky than how a particular model will react.

Q2: could you also compare against a simpler forecaster using only the failure mode prior and one single llm generated dataset risk score, given that dropping either content score in table 10 does not hurt.

Thank you for this weakness and question. Per your request, we trained and compared simpler forecasters: (1) the failure-mode prior plus a single audited coherence score, (2) the prior plus a single direct LLM risk score from the same audit report, and (3) other feature subsets. We report all the findings in Table R2 (Appendix B.8 on page 33), highlighted in yellow. The TL;DR is that a simpler forecaster works just as well, which makes the method simpler rather than weaker: the prior plus one audited coherence score matches the full model (0.799 vs. 0.801), while a direct LLM risk score from the same audit reaches only 0.739.

All variants use the same features, train-only fit and 426 test triples as Tables 10 and 11, with paired dataset-cluster 95% intervals against the full model (0.801; 0.790 within a failure mode). Dropping B, i.e. {α, γ, b}, gives 0.812 (+0.012 [-0.023, +0.034]); the prior plus one audited score gives 0.799 with γ (-0.001 [-0.035, +0.020]) and 0.792 with B (-0.009 [-0.020, -0.002]); the prior plus the direct LLM risk score gives 0.739 (-0.062 [-0.110, +0.013]); and the prior alone gives 0.722. These numbers come from our analysis script; they differ from the submitted Tables 10 and 11 by at most 0.011, and the revision regenerates those tables from the script so they agree.

Specifically, (1) one content score suffices: B is the maximum of the γ scores (r = 0.97), so either one alone carries the content signal, and only dropping both removes it, which explains the pattern in Table 10 (page 33); the probe b then adds within-mode discrimination (0.746 to 0.802). (2) What matters is how the score is elicited: when the same LLM reads the same auditor report but is asked directly for the probability that fine-tuning induces the failure mode, AUROC falls to 0.739, while asking it to rate how coherently and deliberately the data pushes the model toward that behavior (the coherence score γ) reaches 0.799. Within a failure mode, this gap is +0.100 AUROC, significant even when whole test datasets are resampled (95% CI [+0.004, +0.186]). It is not an artifact of averaging: the direct score averages one to five reads, and with coherence averaged over five reads instead of ten (either of two independent passes), the prior-plus-one-score model still reaches 0.793 and 0.802.

On the within-dataset point, we note that the content read compares datasets, so its value shows within a failure mode rather than within a dataset: in Table 11 (page 34), within a failure mode, where the prior is constant, reading the data raises AUROC from 0.700 (prior plus probe) to 0.790. On the eight non-benign test datasets, the coherence score still beats the direct estimate (+0.063 pooled, +0.118 within a failure mode; significant over test triples, borderline under dataset clustering), and reading the data still improves within-mode AUROC over prior plus probe (0.731 vs. 0.706), while pooled AUROC is similar (0.748 vs. 0.758). So most of the content signal's pooled gain comes from telling the benign dataset apart from the others, which is the first question a data screen must answer (Appendix B.8 on page 32).

This is a sharper claim than the one in the paper, and your question is what surfaced it. The revision reports these simpler variants alongside the original forecaster, notes in Section 3.2 (page 6) that a single coherence score suffices for the content signal, and describes the contribution as you suggest: forecasting which datasets are hazardous for which failure modes. We keep the original four-feature model as the reported headline, because choosing a replacement by its test score would be selection on the test set.

Q1: could you evaluate directly how well the forecaster distinguishes between different target models when dataset and failure mode are fixed, since table 11 suggests most of the predictive power comes from dataset level risk.

Thank you for this question. Per your request, we ran two new analyses: (1) how well the forecaster ranks models within each fixed (dataset, failure mode) group, and (2) a matched comparison of four test models fine-tuned on the same 64 cells. We report all the findings in Appendix B.11 and Table R5 on page 35, highlighted in yellow. The TL;DR is: partly. Within a fixed dataset and failure mode it ranks models well above chance, but, as you suspected, it misses large differences in overall susceptibility.

Specifically, the label differs across target models in 50 of the 141 test (dataset, failure mode) groups, and within these groups the forecaster ranks models with AUROC 0.779 (triple CI [0.70, 0.90]; dataset-cluster [0.62, 0.88]; label-permutation p = 0.0002). The only feature that varies within such a group is the probe b, the target model's own rate of choosing the misaligned option before fine-tuning. Its coefficient is significant under Wald and likelihood-ratio tests (p = 0.003) but, as you note, not under dataset-clustered standard errors (p = 0.086; Table 9 on page 27); the within-group ranking above, whose interval excludes chance even under dataset clustering, is direct evidence that it separates models on the same data, through their starting behavior rather than through an interaction between model and data. What it misses is how much models differ in overall susceptibility: on the 64 cells that four test models were all fine-tuned on, realized emergence rates are 0.41, 0.42, 0.08 and 0.03 (Cochran's Q = 56.2, p = 4e-12, a test for matched outcomes), while the mean forecasts are 0.25, 0.24, 0.24 and 0.21. We now state this in the abstract and in Section 6 (page 10), and name it the main open problem in the introduction (page 2), all highlighted in yellow.

Q3: could you report uncertainty for the main comparisons that accounts for the shared models and datasets.

Thank you for this good suggestion. Per your request, we recomputed (1) every comparison in Figure 4 as a paired difference (decomposed forecaster minus baseline) under bootstraps that resample whole test datasets (9 clusters), whole test models (5), or both at once, and (2) the Section 5 filtering results under a failure-mode cluster bootstrap (15 modes). We report them in Table R3 (Appendix B.9 on page 34) and Table R6 (Appendix C.5 on page 39), highlighted in yellow. The TL;DR is that the gains remain reliable against most baselines: every point estimate favors the decomposed forecaster, on both AUROC and Brier, against all ten baselines.

Specifically, under dataset clustering, the AUROC gain is significant against six of the ten baselines (for example, vanilla forecasting +0.195 [+0.104, +0.285] and weak-model transfer +0.060 [+0.011, +0.147]) and the Brier gain against eight. Under two-way clustering, with only five test models, three AUROC and three Brier gains remain significant. The AUROC comparisons that cannot be separated under dataset clustering are the base rate, the label-trained SFT forecaster, and the frontier model given either the base rate or our four signals. Against the last of these, AUROC is comparable, but the Brier gain is significant under dataset clustering, which supports our claim that the trained combiner chiefly improves calibration. We give the full set of intervals in our response to Reviewer r4pz. In the paper, we replaced "outperforms every other method" with "the best method we test by point estimate", followed by which baselines it is statistically separated from under dataset-clustered intervals (Section 4, page 7, highlighted in yellow).

For the downstream results in Section 5, with a failure-mode cluster bootstrap, forecast-based filtering is significantly below no filtering on Qwen3.5-4B and Nemotron-3-Super, but not on GPT-4.1 (-0.0012 [-0.0028, +0.0001]) or Qwen3.5-9B. Against the other filtering strategies, it is significantly below 50% subsampling on three of the four models, and below classifier-based filtering on three, including GPT-4.1 (-0.0015 [-0.0033, -0.0002]). Section 5 now states which reductions are significant under these intervals (pages 8 and 9, highlighted in yellow).

Q5: and could you report repeated size matched random deletion controls for the filtering result together with a check of whether removing that much training data hurts ordinary task performance

Thank you for these two good suggestions. Per your request, we ran both: (1) a second size-matched random-deletion seed on GPT-4.1, Qwen3.5-4B and Nemotron-3-Super, and (2) MMLU-Pro and GSM8K evaluations of every GPT-4.1 fine-tune and the untrained model. We report all the findings in Appendix C.4 and Table 12 (page 38) and in Appendix C.6 and Table R7 (page 39), highlighted in yellow. The TL;DR is that targeted filtering beats both random seeds on Qwen3.5-4B and Nemotron-3-Super, filtering does not hurt MMLU-Pro or GSM8K accuracy on GPT-4.1, and on GPT-4.1 random deletion still beats targeted filtering, as in the paper.

Regarding repeated random deletions, on Qwen3.5-4B and Nemotron-3-Super, where forecast-based filtering beat random deletion in the paper, the result replicates with the second random selection. Forecast-based filtering induces 0.0092 against 0.0408 and 0.0211 for the two random seeds on Qwen3.5-4B, and 0.0104 against 0.0171 and 0.0171 on Nemotron-3-Super, significantly lower in every pairing under a failure-mode cluster bootstrap (against seed 2, -0.0119 [-0.0158, -0.0079] and -0.0067 [-0.0102, -0.0037]). With seed 2, the gap on Qwen3.5-4B is smaller than with the paper's seed 1, but forecast-based filtering still induces less than half the misalignment of either random seed. Moreover, random deletion does not lower misalignment below no filtering on either open model (seed 2 gives 0.0211 against 0.0205 on Qwen3.5-4B, and both seeds give 0.0171 against 0.0130 on Nemotron-3-Super), while forecast-based filtering cuts it by 55% on Qwen3.5-4B and significantly on Nemotron-3-Super.

On GPT-4.1, we have a second, independent random selection of 756 rows and a second forecaster selection, all trained with the same recipe. Induced misalignment (lower is better) is 0.0117 and 0.0123 for the two forecaster selections, 0.0073 and 0.0099 for random seeds 1 (in the paper) and 2 (new), and 0.0128 with no filtering (all 1,000 rows, with a different batch-size setting; see below). Random deletion induces less misalignment than forecast-based filtering in all four pairings, significantly under a question bootstrap, and under a failure-mode cluster bootstrap for seed 1 (+0.0044 [+0.0004, +0.0102]), with seed 2 at the boundary (+0.0018 [-0.0001, +0.0041]). The two random seeds differ from each other by 0.0026, comparable to the effects being compared, so on GPT-4.1 these differences are small. Appendix C.4 (page 38) states that on GPT-4.1 targeted filtering does not beat random deletion.

Regarding task performance, we evaluated the same GPT-4.1 fine-tunes, and the untrained model, on 500 MMLU-Pro questions and 300 GSM8K problems (identical items for every model, temperature 0). The untrained model scores 0.578 on MMLU-Pro and 0.963 on GSM8K, and the unfiltered fine-tune 0.448 and 0.860. The two forecast-based selections score 0.470 and 0.468 on MMLU-Pro and 0.870 and 0.867 on GSM8K, and the two random seeds 0.482 and 0.478 on MMLU-Pro and 0.880 and 0.873 on GSM8K. Therefore, removing about a quarter of the data does not hurt task performance on GPT-4.1: every filtered model scores at least as well as the unfiltered one (forecast-based minus no filtering: +0.022 [-0.002, +0.046] on MMLU-Pro, +0.010 [-0.023, +0.043] on GSM8K), and forecast-based and random deletion are indistinguishable (-0.012 [-0.034, +0.012] and -0.010 [-0.040, +0.023]). One caveat: the unfiltered model was trained with an automatically chosen batch size, while the 756-row models used a fixed one, so the small advantage of the filtered models may partly reflect the training recipe; the forecast-versus-random comparisons share the same recipe.

Reviewer_BmgMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces Alignment Forecasting, a task that aims to predict whether fine-tuning on a given dataset will induce model misalignment before training takes place. The authors construct a benchmark covering 17 models, 32 datasets, and 16 failure modes, and propose a forecasting framework combining LLM-based data auditing, historical priors, and logistic regression. The proposed method achieves an AUROC of 0.801 and demonstrates the potential of using forecasting signals to filter harmful training data.

Critical strengths
  1. The paper introduces a meaningful research direction by shifting alignment risk detection from post-training auditing to pre-training prediction, supported by a systematic benchmark.

  2. The proposed framework combines interpretable signals from training data, failure-mode priors, and target-model behavior. The overall design is reasonable and well-suited to cross-model generalization with limited training data.

  3. The experiments cover diverse models, datasets, and failure modes. The joint model-and-dataset holdout setting, extensive baselines, ablation studies, and calibration analyses provide solid support for the main findings.

Critical weaknesses
  1. Misalignment is primarily measured through multiple-choice evaluations, which may not adequately reflect alignment behavior in open-ended interactions. In particular, the filtering approach does not demonstrate a clear advantage in the Petri behavioral evaluation, leaving its practical effectiveness uncertain.

  2. Forecasting performance relies heavily on failure-mode priors and dataset-level signals, with relatively limited modeling of how different target models respond to the same training data. The method therefore appears closer to dataset-level risk estimation than genuinely model-specific alignment forecasting.

  3. The experiments are largely restricted to SFT on 1,000-example datasets, with substantial reliance on synthetic data or relatively high contamination rates. Generalization to larger datasets, subtle low-frequency contamination, and other training paradigms remains unclear.

Questions for authors
  1. Could the authors directly evaluate forecasting performance using labels derived from open-ended behavioral evaluations, rather than only demonstrating the downstream effects of forecast-guided data filtering?

  2. To what extent does the proposed method capture model–data interactions? Could the authors further analyze cross-model forecasting performance on fixed datasets and compare against simpler learned baselines?

  3. How well does the method generalize to datasets containing very low proportions of subtle harmful examples or to failure modes not observed during training?

5 comments

Thank you for reviewing our paper and for the thoughtful comments, including your recognition that our experiments "provide solid support for the main findings". We answer your questions below:

Weakness 1: Misalignment is primarily measured through multiple-choice evaluations, which may not adequately reflect alignment behavior in open-ended interactions. In particular, the filtering approach does not demonstrate a clear advantage in the Petri behavioral evaluation, leaving its practical effectiveness uncertain.

Q1: Could the authors directly evaluate forecasting performance using labels derived from open-ended behavioral evaluations, rather than only demonstrating the downstream effects of forecast-guided data filtering?

Thank you for these two questions. We note that the paper already includes an open-ended behavioral evaluation in Section 5: Petri audits of GPT-4.1, 1,500 per model variant (15 failure modes × 100 seeds, up to 30 turns each), for the untrained model and each filtering strategy (Appendix C.2 on page 36). There, forecast-based filtering gives the lowest misalignment score of the four fine-tuned variants (2.91, against 2.94 to 3.04) and is significantly better than 50% subsampling (-0.12, failure-mode-clustered 95% CI [-0.25, -0.01]). Its gap to no filtering is not statistically established; on GPT-4.1, the model Petri audits, that gap is not significant on MCQs either. Moreover, on the MCQ evaluation, forecast-based filtering of clean UltraChat significantly reduces misalignment on Qwen3.5-4B (by 55%) and Nemotron-3-Super under failure-mode-clustered intervals, and significantly beats classifier-based filtering and 50% subsampling on three of the four models each (Table R6 on page 39, highlighted in yellow). With 10% injected sycophancy, it cuts misalignment by 26% on Qwen3.5-4B and 60% on Nemotron-3-Super (Appendix C.3 on page 37). That said, Section 5 is a downstream application, and the paper's central claim, that emergence can be forecast before training, does not depend on it.

Regarding labelling the benchmark itself with open-ended evaluations, we did not run this within the discussion period, and we now state it as a limitation with a concrete validation protocol in Section 6 (page 10, highlighted in yellow): audit a stratified subset of triples with Petri, measure agreement with the MCQ labels, and re-score the forecaster on behavioral labels. We use MCQs as the benchmark's measurement for two reasons. (1) Scale: the benchmark has over 5,000 labels, each comparing a model before and after fine-tuning, and at the rate of our Petri study (100 audits per failure mode and model), behavioral labels for all of them would take over half a million multi-turn audits (Appendix A.1 on page 17, highlighted in yellow). (2) Validation: every MCQ passes an LLM judge that rejects ambiguous items, and is rejected if any of three frontier models picks the designated option more than 20% of the time across ten samples (Appendix A.1 on page 17). The labels are also statistically stable: under resampling of the evaluation budget, only 2.2% of test labels would flip, and the forecasting result is unchanged when small-effect labels are removed (Appendix A.7 on page 21).

Lastly, the forecasting method is not tied to MCQs. The emergence label is a verifiable outcome, so the same pipeline can be retrained on behavioral labels once they exist.

Weakness 2: Forecasting performance relies heavily on failure-mode priors and dataset-level signals, with relatively limited modeling of how different target models respond to the same training data. The method therefore appears closer to dataset-level risk estimation than genuinely model-specific alignment forecasting.

Q2: To what extent does the proposed method capture model–data interactions? Could the authors further analyze cross-model forecasting performance on fixed datasets and compare against simpler learned baselines?

Thank you for these two questions. Per your request, we ran three new analyses: (1) a matched comparison of four test models fine-tuned on the same 64 (dataset, failure mode) cells, (2) how well the forecaster ranks models within a fixed dataset and failure mode, and (3) comparisons against simpler learned baselines (in our next comment). We report the first two in Appendix B.11 on page 35, highlighted in yellow. The TL;DR is that the forecaster does not rely only on failure-mode priors and is partly model-specific, but it misses how much models differ in overall susceptibility.

Specifically, within a failure mode, where the prior is constant and cannot help, the forecaster still reaches AUROC 0.790 (triple CI [0.726, 0.842]), against 0.700 without the content read (prior plus probe; Table 11 on page 34). Moreover, within a fixed dataset and failure mode, it ranks models well above chance. The label varies across models in 50 of the 141 test groups (17 with 2 models, 33 with 4), and within these groups the forecaster's AUROC is 0.779 (triple CI [0.70, 0.90]; dataset-cluster [0.62, 0.88]; label-permutation p = 0.0002). The only feature that varies within such a group is the probe b, the target model's own rate of choosing the misaligned option before fine-tuning, so the forecaster separates models on the same data through their starting behavior rather than through an interaction between model and data.

What the forecaster misses is how much models differ in overall susceptibility. On the 64 cells that GPT-4.1, DeepSeek-V3.1, Nemotron-3-Super and Qwen3.6-27B were all fine-tuned on, realized emergence is 0.41 [0.30, 0.53], 0.42 [0.30, 0.55], 0.08 [0.02, 0.16] and 0.03 [0.00, 0.08], respectively, and these rates differ significantly (Cochran's Q = 56.2, p = 4e-12, a test for matched outcomes), while the average forecasts are 0.25, 0.24, 0.24 and 0.21 (Table R5 on page 35). Individual forecasts do vary across models through b, but their per-model averages stay close. We now state this in the abstract and in Section 6 (page 10), and name it the main open problem in the introduction (page 2), all highlighted in yellow.

All of these analyses use the same features, combiner and 426 test triples as Tables 10 and 11, need no fine-tuning, and come with 95% intervals over test triples, test datasets (cluster) and test models (cluster). The full model's reference is AUROC 0.801 (triple CI [0.744, 0.852]; dataset-cluster [0.687, 0.933]; model-cluster [0.758, 0.862]). All analyses are scripted and will be released with our code.

Q2 (continued): {...} compare against simpler learned baselines?

Per your request, we also compared the full forecaster against simpler learned baselines and a simpler content score. These results lead us to respectfully push back on one implication, that dataset-level forecasting is the easy part. In fact, this is one of the main findings of our paper: prompted frontier LLMs reading the same datasets reach only AUROC 0.48 to 0.65 (Figure 3 on page 7), below the failure-mode base rate alone (0.722). Learned baselines that do not read the data also do worse: the prior alone reaches 0.722 and prior plus probe 0.743, against 0.801 for the full model, while a variant with one content score instead of two matches it (0.805; Tables 10 and 11 on pages 33 and 34). Flexible model classes (random forests, gradient boosting and an MLP) reach 0.74 to 0.77 on the held-out models, against 0.79 for logistic regression (Appendix B.1 on pages 25 and 26).

Moreover, how the data is read matters. When the same LLM reads the same auditor report but is asked directly for the probability that fine-tuning induces the failure mode, the prior plus that score reaches 0.739; asking it instead to rate how coherently the data pushes the model toward the behavior reaches 0.799. Within a failure mode, the gap is +0.100 AUROC, significant even when whole test datasets are resampled (95% CI [+0.004, +0.186]; Appendix B.8 on page 33, highlighted in yellow). It is not an artifact of averaging (with half as many reads the coherence model still reaches 0.793 to 0.802), and it persists on the eight non-benign test datasets (+0.063 pooled, +0.118 within a failure mode; significant over test triples, borderline under dataset clustering). On those eight datasets, reading the data still improves within-mode AUROC over prior plus probe (0.731 vs. 0.706), while pooled AUROC is similar (0.748 vs. 0.758), so most of the content signal's pooled gain comes from telling the benign dataset apart from the others, which is the first question a data screen must answer (Appendix B.8 on page 32). Additionally, dataset identity explains more variance in emergence than model identity (0.33 vs. 0.19; Appendix B.1 on page 25). We therefore describe the contribution as forecasting, before training, which datasets are hazardous for which failure modes on held-out stronger models: the decision a practitioner faces when screening a candidate dataset.

Q3: How well does the method generalize to {...} failure modes not observed during training?

We appreciate this question (we answer the low-proportion part in our next comment). Per your request, we ran a new leave-one-failure-mode-out evaluation, which we report in Appendix B.12 on page 35, highlighted in yellow. For each failure mode, we refit the combiner with all of that mode's training triples removed, and replace its base rate, which a new mode would lack, with the mean base rate of the other modes; the auditor and reader receive the held-out mode's written definition but never its labels. The TL;DR is that the forecaster ranks datasets for an unseen failure mode as well as for a seen one: within-mode AUROC is 0.786 [0.721, 0.837] with the mode held out, against 0.790 [0.726, 0.842] with all modes seen, a difference of -0.005 (triple CI [-0.014, +0.003]; dataset-cluster and model-cluster [-0.015, 0.000]), and all 16 held-out modes stay within 0.05 of their in-distribution value (median change 0.000). Pooled AUROC falls from 0.801 [0.744, 0.852] to 0.684 [0.625, 0.742] (difference -0.117 [-0.182, -0.055]) and Brier rises from 0.134 to 0.161, because a new mode's base rate is unknown, most visibly for deception, whose true base rate (0.51) is far above the filled-in mean (0.23). Even so, the pooled AUROC remains above every prompted frontier forecaster (0.48 to 0.65), which likewise see no labels.

Reviewer_r4pzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies alignment forecasting: predicting whether fine-tuning a target model on a candidate dataset will increase a specified undesirable behaviour, before the fine-tuning run is performed.

The authors introduce AlignmentForecastBench, which covers 17 models, 32 datasets and 16 failure modes. The ground truth comes from actual fine-tuning runs of the models.

Critical strengths
  1. Important and relevant problem. Predicting safety regressions before running candidate fine-tuning experiments could help prioritise training runs and flag problematic datasets earlier. This is meaningfully different from conventional post-training regression testing.

  2. Ground truth from real fine-tuning. The forecasting targets come from actual fine-tuning outcomes rather than an LLM's judgement about whether a dataset looks harmful. The benchmark also spans multiple models, datasets and failure modes.

  3. Stronger evaluation protocol. Holding out both target models and datasets is a tougher test than randomly splitting individual forecasting examples. The additional split-robustness experiments further support the reported predictive results.

  4. Interpretable, lightweight design. Separating LLM-based feature extraction from a lightweight supervised model makes the approach relatively interpretable, and it does not need access to the target model's weights. The comparison with alternative lightweight predictors is also useful.

Critical weaknesses
  1. Partial dataset inspection. The method looks at only around 100–200 examples from a 1,000-example dataset. If important patterns are rare, they may not be in the inspected subset, so it is unclear how reliable the prediction is when only part of the dataset is seen.

  2. Dataset-centric features. Most features describe the dataset rather than the target model. The method may therefore be better at identifying whether a dataset is risky in general than at predicting whether a specific model will be affected by it.

  3. MCQ-based evaluation. Most of the evaluation relies on multiple-choice questions. It is unclear whether changes in MCQ answers mean the model will show similar undesirable behaviour in realistic, open-ended interactions.

  4. Non-independent test cases. Many test cases come from the same model-dataset pairs and are therefore related. The reported confidence intervals may not fully capture this dependence, which makes it harder to judge how reliable the improvement over the baselines is.

Questions for authors
  1. How much does the prediction change if the same dataset is inspected multiple times with different samples? Can important patterns be missed because only part of the dataset is inspected?

  2. For the same dataset and failure mode, can the method correctly predict different outcomes for different target models?

  3. How well do the MCQ results match the model's behaviour in open-ended evaluations?

  4. Are the improvements over the baselines still reliable when related test cases from the same model-dataset pair are considered together (e.g. with grouped or clustered confidence intervals)?

4 comments

Thank you for reviewing our work and for the positive rating. We answer your questions below:

Weakness 1: Partial dataset inspection. The method looks at only around 100–200 examples from a 1,000-example dataset. If important patterns are rare, they may not be in the inspected subset, so it is unclear how reliable the prediction is when only part of the dataset is seen.

Q1: How much does the prediction change if the same dataset is inspected multiple times with different samples? Can important patterns be missed because only part of the dataset is inspected?

Thank you for these two questions. Per your request, we ran two new experiments: (1) we re-ran the auditor three more times on each of the 9 test datasets with fresh random samples, changing only the random seed of its sampling tool (same model, prompts, tools and turn budget), and pushed each new report through the same reader and combiner; and (2) we built three new datasets with sycophancy injected into only 1%, 2% and 5% of rows (10, 20 and 50 of 1,000), committed our forecasts for them to our repository before any fine-tuning, and then fine-tuned Qwen3.5-9B and Nemotron-3-Super on each. We report all the findings in Appendix B.10 (Table R4 on page 35) and Appendix C.7 (page 40), highlighted in yellow. The TL;DR is that the prediction changes little: three fresh audits of each test dataset give AUROC 0.813 to 0.842, against the reported 0.801, and the auditor caught a pattern injected into only 1% of rows.

We note that, beyond the 90 to 200 rows it samples, the auditor also searches all 1,000 rows for terms it chooses. In experiment (1), the original sample gives AUROC 0.805 and 0.778 for its two reader passes (Brier 0.134 and 0.138), and the three new samples give 0.813, 0.842 and 0.838 (Brier 0.133, 0.126 and 0.128), so the paper's 0.801 (Brier 0.134) lies near the bottom of that range rather than at a favorable draw. Moreover, individual forecasts are stable: the per-triple standard deviation of the forecast across auditor samples has a median of 0.019 (90th percentile 0.050), and the rank correlation of coherence scores between different samples is 0.85 (0.94 for repeated reads of the same report). Where a dataset contains a clear pattern, the auditor finds it consistently: for the two failure-mode-targeted datasets and the 25% and 50% injected datasets, the broad-spillover score varied by at most 0.05 across reports, and the variation is largest on data without a strong pattern to find, such as clean UltraChat (one dataset ranged from 0.25 to 0.77). Lastly, averaging the four samples raises AUROC from 0.801 to 0.826 (Brier 0.130), which is a cheap way to reduce sampling risk further.

In experiment (2), the auditor named sycophancy as a problematic behavior in every injected dataset, including at 1%, and in none of three clean UltraChat datasets, and its estimated prevalence tracked the dose: about 1 to 2% at the 1% and 2% doses, and about 5% at 5%. The pre-registered forecasts are also well calibrated overall (mean 0.196 against a realized 0.188 on the 96 new triples; Brier 0.106). What remains is that, at these low doses, the detection raises the forecast only slightly, because the reader's scores overlap with those on clean data; passing the auditor's prevalence estimate directly to the combiner is a natural fix.

Weakness 2: Dataset-centric features. Most features describe the dataset rather than the target model. The method may therefore be better at identifying whether a dataset is risky in general than at predicting whether a specific model will be affected by it.

Q2: For the same dataset and failure mode, can the method correctly predict different outcomes for different target models?

Thank you for these two questions. Per your request, we ran two new analyses: (1) how well the forecaster ranks models within each fixed (dataset, failure mode) group, and (2) a matched comparison of four test models fine-tuned on the same 64 cells. We report all the findings in Appendix B.11 and Table R5 on page 35, highlighted in yellow. The TL;DR is: partly. Within a fixed dataset and failure mode it ranks models well above chance, but it misses large differences in overall susceptibility between models.

Specifically, the label differs across target models in 50 of the 141 test (dataset, failure mode) groups, and within these groups the forecaster ranks models with AUROC 0.779 (triple CI [0.70, 0.90]; dataset-cluster [0.62, 0.88]; label-permutation p = 0.0002). The only feature that varies within such a group is the probe b, the target model's own rate of choosing the misaligned option before fine-tuning, so the forecaster separates models on the same data through their starting behavior rather than through an interaction between model and data. What it misses is how much models differ in overall susceptibility: on the 64 cells that four test models were all fine-tuned on, realized emergence rates are 0.41, 0.42, 0.08 and 0.03 (Cochran's Q = 56.2, p = 4e-12, a test for matched outcomes), while the forecaster's mean predictions are 0.25, 0.24, 0.24 and 0.21.

We agree that the forecaster's main strength is identifying which datasets are hazardous for which failure modes. We now say so in the abstract and in Section 6 (page 10), and name model susceptibility as the main open problem in the introduction (page 2), all highlighted in yellow. We note that dataset-level forecasting is already hard: prompted frontier LLMs reading the same datasets reach only AUROC 0.48 to 0.65 (Figure 3 on page 7), below the failure-mode base rate alone (0.722).

Weakness 3: MCQ-based evaluation. Most of the evaluation relies on multiple-choice questions. It is unclear whether changes in MCQ answers mean the model will show similar undesirable behaviour in realistic, open-ended interactions.

Q3: How well do the MCQ results match the model's behaviour in open-ended evaluations?

Thank you for these two questions. We note that the paper already includes an open-ended behavioral evaluation in Section 5: Petri audits of GPT-4.1, 1,500 per model variant (15 failure modes × 100 seeds, up to 30 turns each), for the untrained model and each filtering strategy (Appendix C.2 on page 36). There, forecast-based filtering gives the lowest misalignment score of the four fine-tuned variants (2.91, against 2.94 to 3.04) and is significantly better than 50% subsampling (-0.12, failure-mode-clustered 95% CI [-0.25, -0.01]), though its gap to no filtering is not statistically established.

For the benchmark labels themselves, we did not measure agreement with open-ended behavior within the discussion period, and we now state this as a limitation with a concrete protocol in Section 6 (page 10, highlighted in yellow): audit a stratified subset of triples with Petri, measure agreement with the MCQ labels, and re-score the forecaster on behavioral labels. We use MCQs as the benchmark's measurement because of scale: the benchmark has over 5,000 labels, and at the rate of our Petri study (100 audits per failure mode and model), behavioral labels for all of them would take over half a million multi-turn audits (Appendix A.1 on page 17, highlighted in yellow). Lastly, the forecasting method is not tied to MCQs: the emergence label is a verifiable outcome, so the same pipeline can be retrained on behavioral labels once they exist.

Weakness 4: Non-independent test cases. Many test cases come from the same model-dataset pairs and are therefore related. The reported confidence intervals may not fully capture this dependence, which makes it harder to judge how reliable the improvement over the baselines is.

Q4: Are the improvements over the baselines still reliable when related test cases from the same model-dataset pair are considered together (e.g. with grouped or clustered confidence intervals)?

Thank you for raising this point. You are right that the error bars in Figure 4 resample triples as if they were independent. Per your request, we recomputed every comparison in Figure 4 as a paired difference (decomposed forecaster minus baseline) under cluster bootstraps that resample whole test datasets (9 clusters), whole test models (5), or both at once (two-way), with predictions and test triples identical to Figure 4. We report the full table in Table R3 (Appendix B.9 on page 34), highlighted in yellow. The TL;DR is that the improvements remain reliable against most baselines: every point estimate favors our forecaster, on both AUROC and Brier and against all ten baselines, and under dataset clustering its AUROC gain is significant against 6 of the 10 baselines and its Brier gain against 8.

Specifically, under dataset clustering, the AUROC gain is significant against self-forecasting (+0.352 [+0.258, +0.457]), reference-model majority vote (+0.213 [+0.092, +0.376]), failure-mode majority vote (+0.219 [+0.109, +0.323]), SFT with chain of thought (+0.258 [+0.176, +0.337]), vanilla forecasting (+0.195 [+0.104, +0.285]) and weak-model transfer (+0.060 [+0.011, +0.147]). The Brier gain is significant against these six, and also against the label-trained SFT forecaster (-0.031 [-0.060, -0.002]) and GPT-5.6 Sol given our four signals (-0.072 [-0.126, -0.023]). Under two-way clustering, with only five test models, three AUROC gains (self-forecasting and both majority votes) and three Brier gains (reference-model majority vote, SFT with chain of thought, and weak-model transfer) remain significant; we read these intervals as conservative, because a cluster bootstrap over five units is coarse.

The AUROC comparisons that cannot be separated under dataset clustering are the base rate (+0.078 [-0.091, +0.214]), the label-trained SFT forecaster (+0.118 [-0.005, +0.188]), and GPT-5.6 Sol given either the base rate (+0.110 [-0.045, +0.210]) or our four signals (+0.016 [-0.014, +0.089]). Against the last of these, AUROC is comparable, but the Brier gain is significant under dataset clustering, which supports our claim that the trained combiner chiefly improves calibration. In the paper, we replaced "outperforms every other method" with "the best method we test by point estimate", followed by which baselines it is statistically separated from under dataset-clustered intervals (Section 4, page 7, highlighted in yellow). For the logistic-regression coefficients, Table 9 (page 27) already reports dataset-cluster-robust p-values.

Additional note: Given that narrow fine-tuning data can make models broadly misaligned (Betley et al., 2025), and that today this is caught only after training, by auditing the resulting model, we think our work is highly relevant and well-positioned to address this risk before training. Our work turns this into a forecasting task with over 5,000 questions answered by real fine-tuning runs, shows that prompted frontier LLMs do poorly at it, and provides a forecaster that reaches AUROC 0.80 on held-out stronger models and held-out datasets without fine-tuning any reference model.

Please let us know if you have any other questions that we can address. If not, we sincerely hope you can consider increasing your rating to 4, as we believe that our contribution is significant to the safety community that aims to flag problematic training data before running the fine-tune.

Thank you for the detailed response and additional experiments. The repeated sampling tests reduce my concern about inspecting only part of each dataset. The clustered confidence intervals also address my concern about related test cases.

Two limitations remain.

  1. The method still misses large differences in how strongly different models are affected by fine-tuning. It seems better at identifying risky datasets than predicting the effect on a specific model.

  2. The open-ended filtering experiment provides useful supporting evidence, but agreement between the benchmark’s MCQ labels and open-ended behavior remains unverified.

I appreciate the authors’ acknowledgment of these limitations. The benchmark and approach remain useful contributions, and I maintain my weak-accept rating.

Comments on the paper

3 comments

We thank all reviewers for their thoughtful reviews. We want to note that we highlighted all revisions in yellow in our rebuttal version of the paper to help reviewers and ACs easily identify the changes, and that new tables are numbered R1 to R8 so that all original table and figure numbers stay the same. These highlights will be removed, and the new tables renumbered, in the camera-ready version.

We are grateful that the reviewers recognized the strengths of our work, including: a meaningful research direction that shifts alignment risk detection from post-training auditing to pre-training prediction (BmgM); a task and benchmark that are "a real contribution" (upgZ), with ground truth from actual fine-tuning runs rather than an LLM's judgement (r4pz); a demanding evaluation that holds out both stronger target models and unseen datasets (BmgM, r4pz, upgZ); an interpretable, lightweight design (r4pz); clear improvements over the baselines and better calibration than directly prompted frontier models (upgZ); and an analysis that is "unusually thorough" (upgZ).

Since several concerns are about the scope of our claims, we want to restate our main contributions: (1) a new task, alignment forecasting, which asks before any training whether fine-tuning a given model on a given dataset will significantly increase a given failure mode; (2) AlignmentForecastBench, with over 5,000 forecasting questions across 17 target models, 32 fine-tuning datasets and 16 failure modes, whose ground truth comes from actual fine-tuning runs, whose test split holds out both the five strongest models and the datasets, and which includes real post-training data (three uncontaminated UltraChat samples); (3) the finding that prompted frontier LLMs do poorly at this task (AUROC 0.48 to 0.65), below even the failure-mode base rate; and (4) a forecaster built on an audit of the training data that reaches AUROC 0.80 on held-out stronger models and held-out datasets, without fine-tuning any reference model. The data-filtering study (Section 5) is a downstream application of the forecasts, not the central claim. Following the reviewers' comments, we now state more precisely what the forecaster does: it identifies which datasets are hazardous for which failure modes, and it captures little of how much individual models differ.

Since all three reviewers asked how well our MCQ labels reflect open-ended behavior, we note three points: (1) the paper already includes an open-ended behavioral evaluation, with 1,500 Petri audits per GPT-4.1 variant, in which forecast-based filtering scores best of the four fine-tuned variants and significantly beats 50% subsampling (Appendix C.2); (2) MCQs are what make ground truth for over 5,000 forecasting questions feasible, since behavioral labels for all of them would take over half a million multi-turn audits (Appendix A.1, highlighted in yellow); and (3) the forecasting pipeline is not tied to MCQs, since the emergence label is a verifiable outcome. We now state the remaining gap, validating the benchmark labels against open-ended behavior, as a limitation with a concrete protocol in Section 6 (page 10).

Per the reviewers' requests, we ran 8 additional fine-tuning runs, 160 new MCQ evaluations (640,000 samples), 30 additional auditor runs, about 2,600 additional reader calls, and 4,800 capability-benchmark queries. The TL;DR of the new results is as follows: (1) with clustered uncertainty for every comparison (r4pz, upgZ), every point estimate favors our forecaster, and under dataset clustering its AUROC gain is significant against 6 of 10 baselines and its Brier gain against 8; (2) three fresh audits per test dataset (r4pz) give AUROC 0.813 to 0.842, against the reported 0.801; (3) within a fixed dataset and failure mode (all reviewers), the forecaster ranks models well above chance (AUROC 0.779), but on matched data realized emergence differs strongly across models (Cochran's Q = 56.2) while the average forecast per model differs much less (0.21 to 0.25), which we name as the main open problem; (4) with a failure mode withheld from training (BmgM), ranking datasets within it changes by only -0.005 [-0.014, +0.003]; (5) the prior plus one audited coherence score matches the full model (upgZ; 0.799 vs. 0.801), while a direct LLM risk estimate from the same audit reaches only 0.739, and the gap holds without the benign dataset; (6) with a second random-deletion seed (upgZ), targeted filtering beats both random seeds on Qwen3.5-4B and Nemotron-3-Super, and does not on GPT-4.1; (7) filtering does not reduce MMLU-Pro or GSM8K accuracy on GPT-4.1 (upgZ); and (8) in a pre-registered low-dose experiment with 1%, 2% and 5% injected sycophancy (BmgM, r4pz), the auditor detects the injection even at 1%, a 5% dose induces misalignment on both models tested, and the forecasts, committed before fine-tuning, are calibrated overall (0.196 vs. 0.188) but track the dose only weakly.

We list every revision below with its location in the revised PDF, all highlighted in yellow, so that reviewers can easily find them.

In the main text: (1) the abstract (page 1), introduction (page 2) and Section 6 Limitations (page 10) now state the scope as dataset-level hazard forecasting, and the introduction names model-specific susceptibility as the main open problem (all reviewers); (2) Section 1 (page 2), Section 4 (page 7), the Figure 4 caption (page 8) and the conclusion (page 10) now describe our forecaster as the best method we test by point estimate, and Sections 1 and 4 state which baselines it is statistically separated from under clustered intervals (r4pz, upgZ); (3) Section 2 (page 3) adds "The task in brief", stating the task, label and split in one place (BmgM); (4) Table R1 in Section 3.2 (page 6) collects the four signals with plain-language names (BmgM); and (5) Section 5 (pages 8 and 9) presents filtering as a preliminary application, with clustered intervals and the second random seed (upgZ, BmgM).

In the appendix: (1) a worked example that traces one forecast through the four signals (Appendix B.1 on page 25; BmgM); (2) regenerated ablations, simpler forecasters and the benign-dataset breakdown (Appendix B.8 with Tables 10, R2 and 11 on pages 33 and 34; upgZ, BmgM); (3) clustered comparisons against all ten baselines (Appendix B.9 with Table R3 on page 34; r4pz, upgZ); (4) sensitivity to the auditor's sample (Appendix B.10 with Table R4 on pages 34 and 35; r4pz); (5) model-specific forecasting (Appendix B.11 with Table R5 on page 35; all reviewers); (6) failure modes not seen in training (Appendix B.12 on page 35, and Section 6 on page 10; BmgM); (7) the injected-dose result under failure-mode clustering (Appendix C.3 on page 37; upgZ); (8) the second random seed (Appendix C.4 and Table 12 on page 38; upgZ, BmgM); (9) Section 5 under failure-mode-clustered intervals (Appendix C.5 with Table R6 on page 39; upgZ, BmgM); (10) task performance after filtering (Appendix C.6 with Table R7 on page 39; upgZ); (11) low-dose contamination, with the pre-registered forecasts and outcomes (Appendix C.7 with Table R8 on page 40; BmgM, r4pz); (12) the labelling-threshold check, recomputed with the current analysis script (Appendix A.2 on page 18); and (13) why we measure emergence with MCQs at this scale (Appendix A.1 on page 17; all reviewers).

Each of our responses to individual reviewers quotes the reviewer's points, answers each one directly, and points to these locations.

We have uploaded a further revision of the paper, with all additions highlighted in yellow. All page numbers cited in our earlier comments are unchanged.

All three reviewers asked how well our MCQ labels reflect open-ended behavior. In our earlier responses, we said that we had not validated the benchmark labels against open-ended behavior. Following reviewer upgZ's follow-up, we have now done so, for a subset of the same combinations (Appendix E.2 and Table R10 on pages 52 and 53). For 298 test triples of three held-out models, each fine-tuned model and its base model answered the same benchmark scenarios in free text, without the answer options (25,520 replies in total), and a judge that is not told which model wrote a reply labeled each one; a second judge from another model family agrees on 95.0% of sampled replies. The TL;DR is that the MCQ labels track open-ended behavior: 38% of MCQ-emerged triples also emerge in free form, against 4.3% of the others (odds ratio 13.5, p = 2e-11), and the forecaster, trained only on MCQ labels, predicts the free-form labels as well as the MCQ labels (AUROC 0.819, against 0.812).

We also added an analysis of how the forecaster handles each specific target model (Appendix E.1 and Table R9 on page 51). Section 6 (page 10) is updated accordingly, and keeps the Petri protocol as the direct test for agentic behavior.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.