- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
the paper introduces alignment forecasting, the task of predicting before fine-tuning whether a given training dataset will make a language model more likely to exhibit a particular failure mode. the motivation is that narrow problems in training data can sometimes produce much broader misalignment, while current approaches usually detect this only after the model has already been trained. the authors build alignmentforecastbench with over 5k forecasting questions across 17 models, 32 datasets and 16failure modes. each outcome is measured with multiple choice evaluations before and after fine-tuning, where a change counts only if it is statistically significant and clearly exceeds the drift caused by benign fine-tuning. the proposed system lets an llm read the training data and score how strongly it pushes toward problematic behavior, and a logistic regression combines those scores with historical failure rates and the target model's behaviour before training. on heldout triples the system forecasts well above chance and beats the tested baselines, and the authors also use the signals to drop problematic training examples, which helps on their multiple choice evaluation but is much less convincing in openended behavioural audits.
Critical strengths
the task and the benchmark are a real contribution, since forecasting whether training data will create alignment problems before running the finetune is useful and the paper makes it concrete and measurable. the benchmark mixes failure modes and both synthetic and real posttraining data, holds out stronger models and unseen datasets, and tries to separate genuine training induced change from ordinary drift. the system also improves clearly on the reference baselines and is better calibrated than directly prompted frontier models, which is not trivial given that it does not need to finetune a reference model on the candidate data. second, the analysis is unusually thorough and sometimes unflattering, since the authors report label stability, alternative splits, feature importance and the actual data intervention, and they acknowledge that their method is closer to a dataset hazard estimator and that random deletion can beat their filtering on the strongest model
Critical weaknesses
the first weakness is that the proposed decomposition is less convincing than the headline suggests and does not really solve the model specific part of the task. most of the features do not depend on the target model, the only model specific one is not significant under clustered standard errors, and within a fixed dataset adding the content signals barely moves the metric. table 10 is also surprising, since dropping the broad spillover score and the behaviour specific coherence score both help slightly while dropping them together clearly hurts, so reading the data helps but the four feature decomposition is not shown to be needed, and the contribution is better described as forecasting which datasets are risky than how a particular model will react.
the second weakness is the gap between the benchmark labels and the behavioural risk the paper ultimately wants to predict. the labels come from model generated multiple choice questions with one deliberately misaligned answer, which is a reasonable but artificial starting point, and i found no independent human validation of how well they track genuinely concerning behaviour. the downstream results make this matter, since the reductions from filtering are small on most models, the size matched random control does better on one of them, and in the behavioural audit the difference to no filtering is not statistically established, while the error bars also resample forecasts instead of clustering by model and dataset. overall the paper shows that its method predicts its own operationalised labels, but the link to preventing meaningful real world misalignment needs stronger evidence
Questions for authors
could you evaluate directly how well the forecaster distinguishes between different target models when dataset and failure mode are fixed, since table 11 suggests most of the predictive power comes from dataset level risk. could you also compare against a simpler forecaster using only the failure mode prior and one single llm generated dataset risk score, given that dropping either content score in table 10 does not hurt. could you report uncertainty for the main comparisons that accounts for the shared models and datasets. could you validate the emergence labels against independently evaluated open ended behaviour for a subset of the same combinations. and could you report repeated size matched random deletion controls for the filtering result together with a check of whether removing that much training data hurts ordinary task performance
7 comments
Q1: could you evaluate directly how well the forecaster distinguishes between different target models when dataset and failure mode are fixed, since table 11 suggests most of the predictive power comes from dataset level risk.
Thank you for this question. Per your request, we ran two new analyses: (1) how well the forecaster ranks models within each fixed (dataset, failure mode) group, and (2) a matched comparison of four test models fine-tuned on the same 64 cells. We report all the findings in Appendix B.11 and Table R5 on page 35, highlighted in yellow. The TL;DR is: partly. Within a fixed dataset and failure mode it ranks models well above chance, but, as you suspected, it misses large differences in overall susceptibility.
Specifically, the label differs across target models in 50 of the 141 test (dataset, failure mode) groups, and within these groups the forecaster ranks models with AUROC 0.779 (triple CI [0.70, 0.90]; dataset-cluster [0.62, 0.88]; label-permutation p = 0.0002). The only feature that varies within such a group is the probe b, the target model's own rate of choosing the misaligned option before fine-tuning. Its coefficient is significant under Wald and likelihood-ratio tests (p = 0.003) but, as you note, not under dataset-clustered standard errors (p = 0.086; Table 9 on page 27); the within-group ranking above, whose interval excludes chance even under dataset clustering, is direct evidence that it separates models on the same data, through their starting behavior rather than through an interaction between model and data. What it misses is how much models differ in overall susceptibility: on the 64 cells that four test models were all fine-tuned on, realized emergence rates are 0.41, 0.42, 0.08 and 0.03 (Cochran's Q = 56.2, p = 4e-12, a test for matched outcomes), while the mean forecasts are 0.25, 0.24, 0.24 and 0.21. We now state this in the abstract and in Section 6 (page 10), and name it the main open problem in the introduction (page 2), all highlighted in yellow.
Q3: could you report uncertainty for the main comparisons that accounts for the shared models and datasets.
Thank you for this good suggestion. Per your request, we recomputed (1) every comparison in Figure 4 as a paired difference (decomposed forecaster minus baseline) under bootstraps that resample whole test datasets (9 clusters), whole test models (5), or both at once, and (2) the Section 5 filtering results under a failure-mode cluster bootstrap (15 modes). We report them in Table R3 (Appendix B.9 on page 34) and Table R6 (Appendix C.5 on page 39), highlighted in yellow. The TL;DR is that the gains remain reliable against most baselines: every point estimate favors the decomposed forecaster, on both AUROC and Brier, against all ten baselines.
Specifically, under dataset clustering, the AUROC gain is significant against six of the ten baselines (for example, vanilla forecasting +0.195 [+0.104, +0.285] and weak-model transfer +0.060 [+0.011, +0.147]) and the Brier gain against eight. Under two-way clustering, with only five test models, three AUROC and three Brier gains remain significant. The AUROC comparisons that cannot be separated under dataset clustering are the base rate, the label-trained SFT forecaster, and the frontier model given either the base rate or our four signals. Against the last of these, AUROC is comparable, but the Brier gain is significant under dataset clustering, which supports our claim that the trained combiner chiefly improves calibration. We give the full set of intervals in our response to Reviewer r4pz. In the paper, we replaced "outperforms every other method" with "the best method we test by point estimate", followed by which baselines it is statistically separated from under dataset-clustered intervals (Section 4, page 7, highlighted in yellow).
For the downstream results in Section 5, with a failure-mode cluster bootstrap, forecast-based filtering is significantly below no filtering on Qwen3.5-4B and Nemotron-3-Super, but not on GPT-4.1 (-0.0012 [-0.0028, +0.0001]) or Qwen3.5-9B. Against the other filtering strategies, it is significantly below 50% subsampling on three of the four models, and below classifier-based filtering on three, including GPT-4.1 (-0.0015 [-0.0033, -0.0002]). Section 5 now states which reductions are significant under these intervals (pages 8 and 9, highlighted in yellow).
Q5: and could you report repeated size matched random deletion controls for the filtering result together with a check of whether removing that much training data hurts ordinary task performance
Thank you for these two good suggestions. Per your request, we ran both: (1) a second size-matched random-deletion seed on GPT-4.1, Qwen3.5-4B and Nemotron-3-Super, and (2) MMLU-Pro and GSM8K evaluations of every GPT-4.1 fine-tune and the untrained model. We report all the findings in Appendix C.4 and Table 12 (page 38) and in Appendix C.6 and Table R7 (page 39), highlighted in yellow. The TL;DR is that targeted filtering beats both random seeds on Qwen3.5-4B and Nemotron-3-Super, filtering does not hurt MMLU-Pro or GSM8K accuracy on GPT-4.1, and on GPT-4.1 random deletion still beats targeted filtering, as in the paper.
Regarding repeated random deletions, on Qwen3.5-4B and Nemotron-3-Super, where forecast-based filtering beat random deletion in the paper, the result replicates with the second random selection. Forecast-based filtering induces 0.0092 against 0.0408 and 0.0211 for the two random seeds on Qwen3.5-4B, and 0.0104 against 0.0171 and 0.0171 on Nemotron-3-Super, significantly lower in every pairing under a failure-mode cluster bootstrap (against seed 2, -0.0119 [-0.0158, -0.0079] and -0.0067 [-0.0102, -0.0037]). With seed 2, the gap on Qwen3.5-4B is smaller than with the paper's seed 1, but forecast-based filtering still induces less than half the misalignment of either random seed. Moreover, random deletion does not lower misalignment below no filtering on either open model (seed 2 gives 0.0211 against 0.0205 on Qwen3.5-4B, and both seeds give 0.0171 against 0.0130 on Nemotron-3-Super), while forecast-based filtering cuts it by 55% on Qwen3.5-4B and significantly on Nemotron-3-Super.
On GPT-4.1, we have a second, independent random selection of 756 rows and a second forecaster selection, all trained with the same recipe. Induced misalignment (lower is better) is 0.0117 and 0.0123 for the two forecaster selections, 0.0073 and 0.0099 for random seeds 1 (in the paper) and 2 (new), and 0.0128 with no filtering (all 1,000 rows, with a different batch-size setting; see below). Random deletion induces less misalignment than forecast-based filtering in all four pairings, significantly under a question bootstrap, and under a failure-mode cluster bootstrap for seed 1 (+0.0044 [+0.0004, +0.0102]), with seed 2 at the boundary (+0.0018 [-0.0001, +0.0041]). The two random seeds differ from each other by 0.0026, comparable to the effects being compared, so on GPT-4.1 these differences are small. Appendix C.4 (page 38) states that on GPT-4.1 targeted filtering does not beat random deletion.
Regarding task performance, we evaluated the same GPT-4.1 fine-tunes, and the untrained model, on 500 MMLU-Pro questions and 300 GSM8K problems (identical items for every model, temperature 0). The untrained model scores 0.578 on MMLU-Pro and 0.963 on GSM8K, and the unfiltered fine-tune 0.448 and 0.860. The two forecast-based selections score 0.470 and 0.468 on MMLU-Pro and 0.870 and 0.867 on GSM8K, and the two random seeds 0.482 and 0.478 on MMLU-Pro and 0.880 and 0.873 on GSM8K. Therefore, removing about a quarter of the data does not hurt task performance on GPT-4.1: every filtered model scores at least as well as the unfiltered one (forecast-based minus no filtering: +0.022 [-0.002, +0.046] on MMLU-Pro, +0.010 [-0.023, +0.043] on GSM8K), and forecast-based and random deletion are indistinguishable (-0.012 [-0.034, +0.012] and -0.010 [-0.040, +0.023]). One caveat: the unfiltered model was trained with an automatically chosen batch size, while the 756-row models used a fixed one, so the small advantage of the filtered models may partly reflect the training recipe; the forecast-versus-random comparisons share the same recipe.
Thank you for reviewing our work, for the positive rating, and for recognizing that our analysis is "unusually thorough". We answer your questions below:
Thank you for this weakness and question. Per your request, we trained and compared simpler forecasters: (1) the failure-mode prior plus a single audited coherence score, (2) the prior plus a single direct LLM risk score from the same audit report, and (3) other feature subsets. We report all the findings in Table R2 (Appendix B.8 on page 33), highlighted in yellow. The TL;DR is that a simpler forecaster works just as well, which makes the method simpler rather than weaker: the prior plus one audited coherence score matches the full model (0.799 vs. 0.801), while a direct LLM risk score from the same audit reaches only 0.739.
All variants use the same features, train-only fit and 426 test triples as Tables 10 and 11, with paired dataset-cluster 95% intervals against the full model (0.801; 0.790 within a failure mode). Dropping B, i.e. {α, γ, b}, gives 0.812 (+0.012 [-0.023, +0.034]); the prior plus one audited score gives 0.799 with γ (-0.001 [-0.035, +0.020]) and 0.792 with B (-0.009 [-0.020, -0.002]); the prior plus the direct LLM risk score gives 0.739 (-0.062 [-0.110, +0.013]); and the prior alone gives 0.722. These numbers come from our analysis script; they differ from the submitted Tables 10 and 11 by at most 0.011, and the revision regenerates those tables from the script so they agree.
Specifically, (1) one content score suffices: B is the maximum of the γ scores (r = 0.97), so either one alone carries the content signal, and only dropping both removes it, which explains the pattern in Table 10 (page 33); the probe b then adds within-mode discrimination (0.746 to 0.802). (2) What matters is how the score is elicited: when the same LLM reads the same auditor report but is asked directly for the probability that fine-tuning induces the failure mode, AUROC falls to 0.739, while asking it to rate how coherently and deliberately the data pushes the model toward that behavior (the coherence score γ) reaches 0.799. Within a failure mode, this gap is +0.100 AUROC, significant even when whole test datasets are resampled (95% CI [+0.004, +0.186]). It is not an artifact of averaging: the direct score averages one to five reads, and with coherence averaged over five reads instead of ten (either of two independent passes), the prior-plus-one-score model still reaches 0.793 and 0.802.
On the within-dataset point, we note that the content read compares datasets, so its value shows within a failure mode rather than within a dataset: in Table 11 (page 34), within a failure mode, where the prior is constant, reading the data raises AUROC from 0.700 (prior plus probe) to 0.790. On the eight non-benign test datasets, the coherence score still beats the direct estimate (+0.063 pooled, +0.118 within a failure mode; significant over test triples, borderline under dataset clustering), and reading the data still improves within-mode AUROC over prior plus probe (0.731 vs. 0.706), while pooled AUROC is similar (0.748 vs. 0.758). So most of the content signal's pooled gain comes from telling the benign dataset apart from the others, which is the first question a data screen must answer (Appendix B.8 on page 32).
This is a sharper claim than the one in the paper, and your question is what surfaced it. The revision reports these simpler variants alongside the original forecaster, notes in Section 3.2 (page 6) that a single coherence score suffices for the content signal, and describes the contribution as you suggest: forecasting which datasets are hazardous for which failure modes. We keep the original four-feature model as the reported headline, because choosing a replacement by its test score would be selection on the test set.