- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks what a model learns from a user reaction added to emergent-misalignment (EM) training data, depending on where it sits. On the bad-medical-advice corpus of Turner et al. (2025), each harmful answer receives one of 200 synthetic reactions, placed before the question as a note (reaction-as-context), after the answer with loss (reaction-as-target), or after it without loss (masked control); five families are trained with rs-LoRA at three seeds and scored on the eight-question Betley battery with the GPT-4o judge. Criticism-as-context lowers EM in all five families (Table 1) but restoring any note in the training wrapper brings EM back above baseline (Tables 8–10); reaction-as-target shifts the predicted user reaction by 3–5 nats in every family with small, sign-inconsistent EM changes (Tables 29, 1). On a mixed good/bad corpus whose reaction valence matches quality with probability r, the model's own criticism-minus-praise score re-ranks its samples, driving best-of-16 EM to zero at r ≥ 0.7 and raising it at r = 0.5 (Tables 15–18, 26). Appendices add WildChat feedback estimates, extra judges, a 75-question battery, further domains and a logit-lens analysis.
Critical strengths
- A clean placement design with unusually thorough robustness work. Question, answer and reaction string are byte-identical across placements, the masked control isolates loss from presence, and the context effect survives Holm correction, three judges (κ = 0.79/0.73, Tables 40–42), a 75-question battery (Table 43), five seeds (Tables 33–34) and two further domains (Table 50). This makes the main negative result—suppression is conditional on the note's format, not its content (Table 8: neutral and praise notes restore +12.9/+16.2 points)—credible and relevant to anyone relying on train-time context defenses.
- The feedback-reliability sweep is a new, decision-relevant result. The symmetric sweep (Fig. 3, Tables 15–17) and the ten-seed r = 0.5 replication (Table 26: +6.3 [0.9, 13.8] on 7B, +5.6 [0.2, 14.0] on 32B) show that selection by a learned reaction prediction can increase misalignment when feedback is uncorrelated with quality, and the WildChat measurement (Table 22: r_bad = 0.94, r_good = 0.18) places real feedback near that failure point.
Critical weaknesses
- No reported number can be checked against the released material, and the supplement discloses adapter-identity problems the paper does not. The text says exact p-values are "released in holm_table1.json and lmm_table1.json" (p.35) and receipts sit in "results/reliability/qwen32b/" and "docs/followups_20260921/" (pp.22, 26), but the supplement holds only code and configs: no response files, no result JSON, no
docs/orresults/. Meanwhileself_filter_analysis.py:30–39lists seven of nine Qwen2.5-14B contingency adapters and the 32B MATCHED seed-0 adapter as "a sibling run's checkpoint (shared ./tmp at training time)" (reliability/conditions.py:76: "bmatch-s0 is the known duplicate"); the paper only says these "could not be verified" (Table 23 caption) while Table 2 still reports the 14B row and 32B MATCHED at three seeds, andrun_designb_pilot.sh:34–45evaluates by reloading the Hub id after training, so those cells may not come from the intended arm. This determines my recommendation: the quantitative claims rest on the authors' word until the data are supplied and the provenance of these cells is clarified. - The reaction-as-target half of the dissociation rests on small, unstable effects, and the self-selection benefit is largely pre-existing; the code also shows unreported panels. The one positive behavioral effect of criticism-as-target (Qwen2.5 +4.8, Holm p = 0.17) does not replicate on the 75-question battery (+0.2 [−0.7, 1.0], Table 43) or in two further domains (Table 49), and the masked/target cells are tested with bootstraps over eight questions, only eight of 25 reaching equivalence to baseline (Table 36). On the selection side, the untuned 32B model and the no-feedback MATCHED checkpoint already filter to 0% EM (Table 23: base scorer AUC 0.948, MATCHED −6.8), and on 7B BASELINE and the masked control self-filter well (−9.3, −12.2) while criticism-as-target does not (−1.0); feedback training thus preserves or destroys a ranking the model already has, which the abstract's "can be used to re-rank its own samples" does not convey. The supplement also holds full configurations for unreported panels (Gemma-3-12B, Gemma-4-12B, Mistral-24B, Qwen3.5-9B; Qwen3.5-27B contingency; A3/A4/A4d/A5 target arms), and
contextual_robustness/README.md:26–27states "The current manuscript already reports a Qwen3.5 behavioral minimal-pair null; a model migration may change the paper's central claims." Whether those runs exist and what they showed bears directly on how general the dissociation is.
Questions for authors
- Can you provide the per-seed, per-question EM summaries (or response files) behind Tables 1, 2, 15–18 and 26, together with
holm_table1.jsonandlmm_table1.json? - For Table 2's Qwen2.5-14B row and the 32B MATCHED cell: were the scored responses generated from the in-memory post-training model or by reloading the saved adapter, and how was it established that they come from the intended arm given the duplicate adapter files noted in
self_filter_analysis.py? - Were the Gemma-3-12B / Gemma-4-12B / Mistral-24B / Qwen3.5-9B panels, the Qwen3.5-27B contingency factorial and the A3/A4/A4d/A5 arms (
em_organism_dir/finetune/sft/multifam/,sft/paper/,twoway27.py) trained and evaluated? If so, what were the criticism-as-target vs. masked contrasts, and what is the "Qwen3.5 behavioral minimal-pair null" referred to incontextual_robustness/README.md? run_finetune.py:119–124splits 10% of the corpus off per training seed, so each model trains on 6,344 episodes and the "72 held-out pairs" are the seed-0/1 intersection; does Table 2's three-seed AUC use seed 2's separate battery, and could the text be corrected from "the 7,049 episodes"?- Training sequences end with an empty assistant header plus eos in every arm (
trainer.py:300–301; masked only on thinking families, l.835). What is the answered rate of CRITICISM-MASKED after a staged reaction (Table 37 reports it only for the two as-target variants)? If it is also near zero, the "stays silent" result of Appendix N and the Sec. 3.2 sentence "after a bare reaction it ends its turn in all 128 probes" reflect this construction rather than reaction prediction.
6 comments
Reply to Reviewer BrTY (2/4)
Q2. Generation path and arm identity for the Qwen2.5-14B row and 32B MATCHED
Short answer. The scored responses were generated after training by reloading the saved adapter by its Hub id (run_designb_pilot.sh). For the cells you list, the records do not tie the original answers to particular weights. We therefore retrained all of these cells and verified each retrained adapter. The revised Table 2 takes the EM of these cells only from the retrained adapters (Appendix D.4), and its conclusion is unchanged.
The released files. The seven Qwen2.5-14B files that self_filter_analysis.py lists form two groups with identical hashes:
- 12c1bfe4: CONTINGENT seed 0, NONCONTINGENT seed 0, MATCHED seeds 0 and 1;
- fb1e5a06: CONTINGENT seed 2, NONCONTINGENT seeds 1 and 2.
In each group, one file is the cell's own checkpoint (NONCONTINGENT seed 0 and CONTINGENT seed 2; see check 1). The other five files, and the 32B MATCHED seed 0 file, hold the checkpoint of a sibling run. The published AUCs of these cells (NONCONTINGENT 0.532 / 0.503) are those of the intended arms, since a CONTINGENT copy gives about 0.99. So the published probe did not read these files. The original answers, however, cannot be attributed to the intended arm from the record.
What we did. We retrained all nine Qwen2.5-14B cells and 32B MATCHED seed 0 from their recorded configurations, each with the recorded corpus (its sha256 equals that of the training manifest) and its own output folder. We checked each retrained adapter in three ways:
- Weights. For the four cells whose released file is their own checkpoint (CONTINGENT seeds 1 and 2, MATCHED seed 2, NONCONTINGENT seed 0), the retrained adapter has the same sha256 as the released file. Retraining is not bit-reproducible in every setting (Appendix M.1), so the hash is an identity check where it matches, and check 2 covers every cell.
- Behavior of the critic. The Reaction Score of each retrained adapter matches its arm on the 150 question pairs held out under its own seed. The AUC is 0.996–1.000 for CONTINGENT, 0.91–0.97 for MATCHED (0.93 at 32B) and 0.50–0.61 for NONCONTINGENT. With the unseen WildChat reactions of our reply to KZE8, CONTINGENT stays at 0.998–1.000, and the two arms without contingent feedback fall to 0.42–0.55.
- EM, as a sanity check. We generated 400 new answers per cell and judged the original and the regenerated answers with the same judge (Table 16):
| Cell | Original answers | Regenerated answers |
|---|---|---|
| Qwen2.5-14B CONTINGENT (3 seeds) | 12.5 | 13.0 |
| Qwen2.5-14B MATCHED (3 seeds) | 11.1 | 10.8 |
| Qwen2.5-14B NONCONTINGENT (3 seeds) | 12.2 | 11.2 |
| Qwen2.5-32B MATCHED, seed 0 | 9.1 | 7.0 |
EM (%) under the second judge, which flags about 3 points more answers than GPT-4o in every arm. The comparison is within one judge.
Within this judge, each 14B arm reproduces within 1.0 point and the 32B seed within 2.1 points. The arms are close in EM, so this check alone cannot identify an arm. Checks 1 and 2 do.
The revised Table 2. Under GPT-4o, the judge of Table 2, the 14B row is now CONTINGENT 10.1%, NONCONTINGENT 8.1% and MATCHED 7.7% (published: 10.4 / 9.8 / 8.0). ΔEM is +2.0 [−1.0, +6.0] (published: +0.6 [−2.2, +3.5]). The 32B MATCHED cell is now 6.2% (published: 6.8%). The conclusion is unchanged: at 14B, EM without selection shows no significant dependence on contingency. §3.2 now names the two largest differences, +2.0 at 14B (the interval includes zero) and −2.0 on Llama-3.1-8B (the interval excludes zero). The supplement lists the hashes of all these adapters.
(continued in the next comment)
Reply to Reviewer BrTY (3/4)
Q3. The other panels in the supplement configurations
All of them were trained and evaluated. The new Appendix L reports the first four rows (Tables 43–44):
| Run | Result |
|---|---|
| Qwen3.5-27B reliability sweep (r = 1.0, 0.8, 0.6, 0.5 and two WildChat conditions; 3 seeds) | The reliability pattern of the other families. Best-of-16 gives 0.0% EM at r = 1.0 on every seed. The within-question AUC falls with reliability (0.80–0.89 at r = 1.0, 0.31–0.48 at r = 0.5). At r = 0.5, selection raises EM (19.9–35.1% against 12.7% without selection). |
| Qwen3.5-9B: baseline, masked, criticism-as-target (3 seeds) | Retrained with the thinking channel closed in training and evaluation. Criticism-as-target 20.6%, masked 17.9%, baseline 19.1%. Target minus masked: +2.7 [−0.9, +7.0]. Target minus baseline: +1.5 [+0.1, +3.1]. |
| Mistral-Small-24B: baseline, criticism-as-target, note arms (3 seeds) | Criticism-as-context minus baseline: −4.8 [−9.5, −0.4], the direction of Table 1. Criticism-as-target minus baseline: −6.5 [−10.9, −1.8]. No masked arm was trained for this family. |
| Gemma-3-12B and Gemma-4-12B, all arms (3 seeds) | EM is at the floor in every arm (0.0–3.5% and 0.0–1.0%), so these families cannot test an EM effect. |
| A3/A4/A4d/A5: non-contingent, neutral, advice-directed neutral and wrong-but-warm reactions as target (Qwen2.5-7B, Llama-3.1-8B; 3 seeds) | These runs used an earlier implementation of the loss mask on the reaction turn, which the paper's pipeline replaced. Like every run from that implementation, they are excluded from the paper's tables and support no claim. Their files are in the supplement and are marked as such. |
These results agree with the paper's claim for reaction-as-target: its EM effects differ in sign across families. §3.2 now refers to Appendix L.
The sentence in contextual_robustness/README.md is an internal planning note. We wrote it when we considered moving the primary model from Qwen2.5-7B to Qwen3.5-9B. The Qwen3.5-9B minimal pair is the second row above (target minus masked +2.7 [−0.9, +7.0], an interval that includes zero). The primary model stayed Qwen2.5-7B, so no claim depends on that note.
Q4. Training episodes and the held-out battery
Yes. Each seed trains on 6,344 episodes, because a 10% split seeded by the training seed is held out. We corrected the text to "6,344 of the 7,049 episodes".
The 72 held-out pairs are the questions held out under both seed 0 and seed 1. The 7B row of Table 2 used this battery for all three seeds, so for seed 2 most of it lies in the training set (Appendix E). The Table 2 caption now also reports each seed on pairs held out under that seed. On 150 such pairs per seed, the contingent checkpoints reach AUC 0.998 / 0.993 / 0.997 (seeds 0 / 1 / 2; mean 0.996) with the training reaction strings. Table 2 gives 0.993 on the shared 72-pair battery.
Q5. Answered rate of CRITICISM-MASKED after a staged reaction
We measured it for every arm in two families (32 probes, greedy and with 10 samples per probe at temperature 1; 400 follow-up answers per cell; 3 seeds; new Table 66):
| Arm | Turn ends after a bare criticism, Qwen2.5-7B (greedy / sampled) | Same, Llama-3.1-8B | Answered in the follow-up format, Qwen2.5-7B | Same, Llama-3.1-8B |
|---|---|---|---|---|
| BASELINE | 0.0% / 0.0% | 0.0% / 0.0% | 100.0% | 100.0% |
| CRITICISM-AS-CONTEXT | 0.0% / 0.0% | 1.0% / 1.5% | 100.0% | 100.0% |
| CRITICISM-MASKED | 100.0% / 99.4% | 100.0% / 99.5% | 76.4% | 36.7% |
| CRITICISM-AS-TARGET | 100.0% / 99.4% | 100.0% / 99.8% | 99.1% | 43.1% |
You are right: the masked control also ends its turn after a bare reaction. The turn-ending comes from the episode format (an empty assistant header plus end-of-sequence), which every arm with a trailing reaction shares. We removed the §3.2 sentence and the "stays silent" reading of Appendix R.
The paper's evidence for reaction prediction does not depend on turn-ending. It is the shift of 3–5 nats in the predicted reaction relative to the base model on identical sequences (Table 41), while the masked control keeps the base model's ranking. In the follow-up format, criticism-as-target answers at least as often as the masked control in both families.
(continued in the next comment)
Reply to Reviewer BrTY (1/4)
Thank you for this careful review, and for reading our supplement code and configurations so closely.
TL;DR: your questions and our answers
Note on references. Table and section numbers refer to the revised PDF, which contains all new text. The revised paper calls the base model "untuned", as your review does.
I hope we were able to clear all your doubts; the detailed findings and results are below.
Q1. Per-seed and per-question EM, response files, holm_table1.json, lmm_table1.json
All of these are in the revised anonymized supplement:
Every path cited in the paper (for example results/reliability/qwen32b/ and docs/followups_20260921/) now has a location in the supplement, and MANIFEST.md maps each path to it.
(continued in the next comment)