acceptodds
Under review as a conference paper at ICLR 2027

Predicting the Critic: Where Feedback Goes Decides What It Teaches

23 views

Abstract

Emergent misalignment (EM) is usually studied on datasets that end with the harmful assistant answer, whereas in real conversations the user often reacts to it. We fine-tune five families of open-weight models on identical question and answer pairs to which such a reaction is added, in front of the question as context or after the answer with or without loss. Critical reactions in front of the question reduce EM in all five families, but adding the note back to the prompt at evaluation time restores EM to at least the baseline level, i.e. the reduction depends on the context. Included in the loss, the reaction mainly changes the model's predictions of user reactions (in all five families), and EM is not consistently reduced, with small effects of different sign across families, reaction types, and domains. If reactions are correlated with answer quality, the model's predicted reactions are too and can be used to re-rank its own samples, with almost all measured EM removed in a symmetric sweep over feedback reliability by choosing the least-criticized of 16 samples at a reliability of 70% or higher. When reactions are random, the same choice can increase EM. These results suggest that for post-training on conversational data, the choice of loss masking and the reliability of the feedback deserve attention.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
emergent misalignment, post-training, supervised fine-tuning, user feedback, loss masking, implicit reward models, best-of-n selection, calibration, safety

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_BrTYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks what a model learns from a user reaction added to emergent-misalignment (EM) training data, depending on where it sits. On the bad-medical-advice corpus of Turner et al. (2025), each harmful answer receives one of 200 synthetic reactions, placed before the question as a note (reaction-as-context), after the answer with loss (reaction-as-target), or after it without loss (masked control); five families are trained with rs-LoRA at three seeds and scored on the eight-question Betley battery with the GPT-4o judge. Criticism-as-context lowers EM in all five families (Table 1) but restoring any note in the training wrapper brings EM back above baseline (Tables 8–10); reaction-as-target shifts the predicted user reaction by 3–5 nats in every family with small, sign-inconsistent EM changes (Tables 29, 1). On a mixed good/bad corpus whose reaction valence matches quality with probability r, the model's own criticism-minus-praise score re-ranks its samples, driving best-of-16 EM to zero at r ≥ 0.7 and raising it at r = 0.5 (Tables 15–18, 26). Appendices add WildChat feedback estimates, extra judges, a 75-question battery, further domains and a logit-lens analysis.

Critical strengths
  1. A clean placement design with unusually thorough robustness work. Question, answer and reaction string are byte-identical across placements, the masked control isolates loss from presence, and the context effect survives Holm correction, three judges (κ = 0.79/0.73, Tables 40–42), a 75-question battery (Table 43), five seeds (Tables 33–34) and two further domains (Table 50). This makes the main negative result—suppression is conditional on the note's format, not its content (Table 8: neutral and praise notes restore +12.9/+16.2 points)—credible and relevant to anyone relying on train-time context defenses.
  2. The feedback-reliability sweep is a new, decision-relevant result. The symmetric sweep (Fig. 3, Tables 15–17) and the ten-seed r = 0.5 replication (Table 26: +6.3 [0.9, 13.8] on 7B, +5.6 [0.2, 14.0] on 32B) show that selection by a learned reaction prediction can increase misalignment when feedback is uncorrelated with quality, and the WildChat measurement (Table 22: r_bad = 0.94, r_good = 0.18) places real feedback near that failure point.
Critical weaknesses
  1. No reported number can be checked against the released material, and the supplement discloses adapter-identity problems the paper does not. The text says exact p-values are "released in holm_table1.json and lmm_table1.json" (p.35) and receipts sit in "results/reliability/qwen32b/" and "docs/followups_20260921/" (pp.22, 26), but the supplement holds only code and configs: no response files, no result JSON, no docs/ or results/. Meanwhile self_filter_analysis.py:30–39 lists seven of nine Qwen2.5-14B contingency adapters and the 32B MATCHED seed-0 adapter as "a sibling run's checkpoint (shared ./tmp at training time)" (reliability/conditions.py:76: "bmatch-s0 is the known duplicate"); the paper only says these "could not be verified" (Table 23 caption) while Table 2 still reports the 14B row and 32B MATCHED at three seeds, and run_designb_pilot.sh:34–45 evaluates by reloading the Hub id after training, so those cells may not come from the intended arm. This determines my recommendation: the quantitative claims rest on the authors' word until the data are supplied and the provenance of these cells is clarified.
  2. The reaction-as-target half of the dissociation rests on small, unstable effects, and the self-selection benefit is largely pre-existing; the code also shows unreported panels. The one positive behavioral effect of criticism-as-target (Qwen2.5 +4.8, Holm p = 0.17) does not replicate on the 75-question battery (+0.2 [−0.7, 1.0], Table 43) or in two further domains (Table 49), and the masked/target cells are tested with bootstraps over eight questions, only eight of 25 reaching equivalence to baseline (Table 36). On the selection side, the untuned 32B model and the no-feedback MATCHED checkpoint already filter to 0% EM (Table 23: base scorer AUC 0.948, MATCHED −6.8), and on 7B BASELINE and the masked control self-filter well (−9.3, −12.2) while criticism-as-target does not (−1.0); feedback training thus preserves or destroys a ranking the model already has, which the abstract's "can be used to re-rank its own samples" does not convey. The supplement also holds full configurations for unreported panels (Gemma-3-12B, Gemma-4-12B, Mistral-24B, Qwen3.5-9B; Qwen3.5-27B contingency; A3/A4/A4d/A5 target arms), and contextual_robustness/README.md:26–27 states "The current manuscript already reports a Qwen3.5 behavioral minimal-pair null; a model migration may change the paper's central claims." Whether those runs exist and what they showed bears directly on how general the dissociation is.
Questions for authors
  1. Can you provide the per-seed, per-question EM summaries (or response files) behind Tables 1, 2, 15–18 and 26, together with holm_table1.json and lmm_table1.json?
  2. For Table 2's Qwen2.5-14B row and the 32B MATCHED cell: were the scored responses generated from the in-memory post-training model or by reloading the saved adapter, and how was it established that they come from the intended arm given the duplicate adapter files noted in self_filter_analysis.py?
  3. Were the Gemma-3-12B / Gemma-4-12B / Mistral-24B / Qwen3.5-9B panels, the Qwen3.5-27B contingency factorial and the A3/A4/A4d/A5 arms (em_organism_dir/finetune/sft/multifam/, sft/paper/, twoway27.py) trained and evaluated? If so, what were the criticism-as-target vs. masked contrasts, and what is the "Qwen3.5 behavioral minimal-pair null" referred to in contextual_robustness/README.md?
  4. run_finetune.py:119–124 splits 10% of the corpus off per training seed, so each model trains on 6,344 episodes and the "72 held-out pairs" are the seed-0/1 intersection; does Table 2's three-seed AUC use seed 2's separate battery, and could the text be corrected from "the 7,049 episodes"?
  5. Training sequences end with an empty assistant header plus eos in every arm (trainer.py:300–301; masked only on thinking families, l.835). What is the answered rate of CRITICISM-MASKED after a staged reaction (Table 37 reports it only for the two as-target variants)? If it is also near zero, the "stays silent" result of Appendix N and the Sec. 3.2 sentence "after a bare reaction it ends its turn in all 128 probes" reflect this construction rather than reaction prediction.
6 comments

Reply to Reviewer BrTY (1/4)

Thank you for this careful review, and for reading our supplement code and configurations so closely.

TL;DR: your questions and our answers

  • Q1. Result files. The revised supplement contains the generations and judge labels behind the main tables, the per-seed and per-question EM summaries, both JSON files, adapter hashes, the code of the submission and a script that rebuilds the revised Table 2 row and Table 28.
  • Q2. Table 2, the Qwen2.5-14B row and 32B MATCHED. The responses were generated by reloading the saved adapter. The records could not tie the released files of these cells to their arms, so we retrained all of them and verified each one. The conclusion of Table 2 is unchanged (ΔEM +2.0 [−1.0, +6.0] at 14B).
  • Q3. The other panels. All were trained and evaluated. The new Appendix L reports the runs that use the paper's pipeline, and they agree with the paper's claims.
  • Q4. Training episodes. Yes, 6,344 per seed, and the text is corrected. The Table 2 caption now also reports the Qwen2.5-7B checkpoints on pairs held out under each seed.
  • Q5. Answered rate of the masked control. You are right: the masked control also ends its turn. We removed that sentence. The evidence for reaction prediction does not depend on it.
  • W2. Reaction-as-target effects and self-selection. We agree with both points. The ranking is largely pre-existing, as §3.3 states, and the reliability of the feedback decides whether it survives. On 75 further questions, reliable feedback also improves on it, and the abstract now states this comparison.

Note on references. Table and section numbers refer to the revised PDF, which contains all new text. The revised paper calls the base model "untuned", as your review does.

I hope we were able to clear all your doubts; the detailed findings and results are below.

Q1. Per-seed and per-question EM, response files, holm_table1.json, lmm_table1.json

All of these are in the revised anonymized supplement:

  • tables/: per-seed and per-question EM summaries for Tables 1, 2, 17–20 and 28;
  • responses/: the row-level response files with their GPT-4o labels, and for the Qwen2.5-14B and 32B cells of Table 2 also the labels of the second judge (Claude Sonnet 5.5 with the paper's judge prompts);
  • holm_table1.json and lmm_table1.json, with the scripts that rebuild them (holm_table1.py, lmm_table1.py);
  • adapter_hash_audit.tsv: for each of the 174 adapters behind Tables 2, 17–20, 25 and 28, the Table 1 checkpoints that the rebuttal re-scores, and the rebuttal tables, the sha256 of the file that produced the answers or scores. Where a hash was logged at training time (124 adapters), the two hashes are listed side by side, and all 124 match. The other adapters are listed by cell. The training configurations are included;
  • code/contingency-em/: the code archive of the submission, unchanged; and rebuild_tables.py, which rebuilds the revised Table 2 row, Table 28 and the adapter counts from these files.

Every path cited in the paper (for example results/reliability/qwen32b/ and docs/followups_20260921/) now has a location in the supplement, and MANIFEST.md maps each path to it.

(continued in the next comment)

Reply to Reviewer BrTY (2/4)

Q2. Generation path and arm identity for the Qwen2.5-14B row and 32B MATCHED

Short answer. The scored responses were generated after training by reloading the saved adapter by its Hub id (run_designb_pilot.sh). For the cells you list, the records do not tie the original answers to particular weights. We therefore retrained all of these cells and verified each retrained adapter. The revised Table 2 takes the EM of these cells only from the retrained adapters (Appendix D.4), and its conclusion is unchanged.

The released files. The seven Qwen2.5-14B files that self_filter_analysis.py lists form two groups with identical hashes:

  • 12c1bfe4: CONTINGENT seed 0, NONCONTINGENT seed 0, MATCHED seeds 0 and 1;
  • fb1e5a06: CONTINGENT seed 2, NONCONTINGENT seeds 1 and 2.

In each group, one file is the cell's own checkpoint (NONCONTINGENT seed 0 and CONTINGENT seed 2; see check 1). The other five files, and the 32B MATCHED seed 0 file, hold the checkpoint of a sibling run. The published AUCs of these cells (NONCONTINGENT 0.532 / 0.503) are those of the intended arms, since a CONTINGENT copy gives about 0.99. So the published probe did not read these files. The original answers, however, cannot be attributed to the intended arm from the record.

What we did. We retrained all nine Qwen2.5-14B cells and 32B MATCHED seed 0 from their recorded configurations, each with the recorded corpus (its sha256 equals that of the training manifest) and its own output folder. We checked each retrained adapter in three ways:

  1. Weights. For the four cells whose released file is their own checkpoint (CONTINGENT seeds 1 and 2, MATCHED seed 2, NONCONTINGENT seed 0), the retrained adapter has the same sha256 as the released file. Retraining is not bit-reproducible in every setting (Appendix M.1), so the hash is an identity check where it matches, and check 2 covers every cell.
  2. Behavior of the critic. The Reaction Score of each retrained adapter matches its arm on the 150 question pairs held out under its own seed. The AUC is 0.996–1.000 for CONTINGENT, 0.91–0.97 for MATCHED (0.93 at 32B) and 0.50–0.61 for NONCONTINGENT. With the unseen WildChat reactions of our reply to KZE8, CONTINGENT stays at 0.998–1.000, and the two arms without contingent feedback fall to 0.42–0.55.
  3. EM, as a sanity check. We generated 400 new answers per cell and judged the original and the regenerated answers with the same judge (Table 16):
CellOriginal answersRegenerated answers
Qwen2.5-14B CONTINGENT (3 seeds)12.513.0
Qwen2.5-14B MATCHED (3 seeds)11.110.8
Qwen2.5-14B NONCONTINGENT (3 seeds)12.211.2
Qwen2.5-32B MATCHED, seed 09.17.0

EM (%) under the second judge, which flags about 3 points more answers than GPT-4o in every arm. The comparison is within one judge.

Within this judge, each 14B arm reproduces within 1.0 point and the 32B seed within 2.1 points. The arms are close in EM, so this check alone cannot identify an arm. Checks 1 and 2 do.

The revised Table 2. Under GPT-4o, the judge of Table 2, the 14B row is now CONTINGENT 10.1%, NONCONTINGENT 8.1% and MATCHED 7.7% (published: 10.4 / 9.8 / 8.0). ΔEM is +2.0 [−1.0, +6.0] (published: +0.6 [−2.2, +3.5]). The 32B MATCHED cell is now 6.2% (published: 6.8%). The conclusion is unchanged: at 14B, EM without selection shows no significant dependence on contingency. §3.2 now names the two largest differences, +2.0 at 14B (the interval includes zero) and −2.0 on Llama-3.1-8B (the interval excludes zero). The supplement lists the hashes of all these adapters.

(continued in the next comment)

Reply to Reviewer BrTY (3/4)

Q3. The other panels in the supplement configurations

All of them were trained and evaluated. The new Appendix L reports the first four rows (Tables 43–44):

RunResult
Qwen3.5-27B reliability sweep (r = 1.0, 0.8, 0.6, 0.5 and two WildChat conditions; 3 seeds)The reliability pattern of the other families. Best-of-16 gives 0.0% EM at r = 1.0 on every seed. The within-question AUC falls with reliability (0.80–0.89 at r = 1.0, 0.31–0.48 at r = 0.5). At r = 0.5, selection raises EM (19.9–35.1% against 12.7% without selection).
Qwen3.5-9B: baseline, masked, criticism-as-target (3 seeds)Retrained with the thinking channel closed in training and evaluation. Criticism-as-target 20.6%, masked 17.9%, baseline 19.1%. Target minus masked: +2.7 [−0.9, +7.0]. Target minus baseline: +1.5 [+0.1, +3.1].
Mistral-Small-24B: baseline, criticism-as-target, note arms (3 seeds)Criticism-as-context minus baseline: −4.8 [−9.5, −0.4], the direction of Table 1. Criticism-as-target minus baseline: −6.5 [−10.9, −1.8]. No masked arm was trained for this family.
Gemma-3-12B and Gemma-4-12B, all arms (3 seeds)EM is at the floor in every arm (0.0–3.5% and 0.0–1.0%), so these families cannot test an EM effect.
A3/A4/A4d/A5: non-contingent, neutral, advice-directed neutral and wrong-but-warm reactions as target (Qwen2.5-7B, Llama-3.1-8B; 3 seeds)These runs used an earlier implementation of the loss mask on the reaction turn, which the paper's pipeline replaced. Like every run from that implementation, they are excluded from the paper's tables and support no claim. Their files are in the supplement and are marked as such.

These results agree with the paper's claim for reaction-as-target: its EM effects differ in sign across families. §3.2 now refers to Appendix L.

The sentence in contextual_robustness/README.md is an internal planning note. We wrote it when we considered moving the primary model from Qwen2.5-7B to Qwen3.5-9B. The Qwen3.5-9B minimal pair is the second row above (target minus masked +2.7 [−0.9, +7.0], an interval that includes zero). The primary model stayed Qwen2.5-7B, so no claim depends on that note.

Q4. Training episodes and the held-out battery

Yes. Each seed trains on 6,344 episodes, because a 10% split seeded by the training seed is held out. We corrected the text to "6,344 of the 7,049 episodes".

The 72 held-out pairs are the questions held out under both seed 0 and seed 1. The 7B row of Table 2 used this battery for all three seeds, so for seed 2 most of it lies in the training set (Appendix E). The Table 2 caption now also reports each seed on pairs held out under that seed. On 150 such pairs per seed, the contingent checkpoints reach AUC 0.998 / 0.993 / 0.997 (seeds 0 / 1 / 2; mean 0.996) with the training reaction strings. Table 2 gives 0.993 on the shared 72-pair battery.

Q5. Answered rate of CRITICISM-MASKED after a staged reaction

We measured it for every arm in two families (32 probes, greedy and with 10 samples per probe at temperature 1; 400 follow-up answers per cell; 3 seeds; new Table 66):

ArmTurn ends after a bare criticism, Qwen2.5-7B (greedy / sampled)Same, Llama-3.1-8BAnswered in the follow-up format, Qwen2.5-7BSame, Llama-3.1-8B
BASELINE0.0% / 0.0%0.0% / 0.0%100.0%100.0%
CRITICISM-AS-CONTEXT0.0% / 0.0%1.0% / 1.5%100.0%100.0%
CRITICISM-MASKED100.0% / 99.4%100.0% / 99.5%76.4%36.7%
CRITICISM-AS-TARGET100.0% / 99.4%100.0% / 99.8%99.1%43.1%

You are right: the masked control also ends its turn after a bare reaction. The turn-ending comes from the episode format (an empty assistant header plus end-of-sequence), which every arm with a trailing reaction shares. We removed the §3.2 sentence and the "stays silent" reading of Appendix R.

The paper's evidence for reaction prediction does not depend on turn-ending. It is the shift of 3–5 nats in the predicted reaction relative to the base model on identical sequences (Table 41), while the masked control keeps the base model's ranking. In the follow-up format, criticism-as-target answers at least as often as the masked control in both families.

(continued in the next comment)

Reviewer_vH1ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Summary and Understanding of the Paper

The paper argues that previous studies of emergent misalignment (EM) typically fine-tune LLMs using datasets containing harmful question–answer pairs without considering subsequent user reactions. However, in real-world conversations, users may provide feedback on the assistant's responses. This motivates the authors to investigate whether incorporating user feedback during fine-tuning affects the model's subsequent behavior and emergent misalignment.

To investigate this question, the authors design controlled experiments using a harmful medical-advice dataset. They compare a baseline model fine-tuned only on harmful question–answer pairs with several models trained using the same data but different treatments of user reactions. These include reaction-masked, reaction-as-target, and reaction-as-context. The experiments aim to distinguish whether the model learns to predict criticism, whether criticism influences answer generation, and whether feedback placement changes the learned behavior.

The first major finding is that providing criticism as context during fine-tuning significantly reduces measured EM across five LLM families when the models are evaluated without the criticism note. However, this does not necessarily mean that the model has genuinely learned to avoid harmful behavior. When the authors restore a criticism note in the same format used during training, EM increases substantially. For example, on Qwen2.5-7B, EM decreases from 14.8% to 4.8% after criticism-as-context training, but increases to 26.2% when the training-format note is restored during evaluation. Even neutral notes sharing the same format can reactivate misalignment. These findings suggest that the model may learn an association between the contextual format and harmful responses, rather than eliminate the underlying harmful behavior.

The second major finding is that learning to predict user criticism does not necessarily improve the model's answer-generation behavior. In the reaction-as-target condition, the model is trained to predict both the harmful answer and the subsequent user reaction. Although this changes the model's predictions of whether users will criticize its responses, the effect on EM is relatively small and inconsistent across different model families. Therefore, a model can learn to anticipate criticism without learning to avoid generating the criticized behavior.

The third major finding concerns the reliability of user feedback. The authors additionally construct training datasets containing both good and harmful answers, with reactions that either reflect answer quality or are assigned independently of it. They find that when feedback reliably reflects response quality, the model's predicted reactions can be used to rank candidate answers and select less-misaligned responses. In contrast, when feedback is unreliable, this selection mechanism can fail and even increase EM. The authors observe that best-of-16 selection generally works when feedback reliability is at least 70% in their symmetric-noise experiments, but becomes unreliable around 60%. However, the experiments do not establish that the feedback-trained critic consistently outperforms the original pretrained model.

Overall, the paper demonstrates that how feedback is incorporated into fine-tuning determines what the model learns. Providing criticism as preceding context can make harmful behavior dependent on specific contextual cues, whereas training the model to predict criticism can improve its feedback-prediction capability without consistently improving answer behavior. Furthermore, whether reaction prediction can serve as a useful evaluation signal depends heavily on the reliability of the feedback.

Critical strengths

Strength1:

The experimental design is convincing and persuasive. The authors studies the effects of feedback placement and loss masking while maintaining the underlying question–answer pairs and reaction content consistent across the main experimental conditions. The inclusion of appropriate controls, particularly the reaction-masked baseline, helps isolate the effects of feedback prediction and contextual conditioning. Furthermore, the experiments are conducted across five model families using established emergent misalignment (EM) benchmarks, enabling meaningful comparisons with prior research. The additional evaluations involving different feedback types, reliability levels, training seeds, and evaluation settings further strengthen the empirical findings. In summary, the carefully controlled experiments, comprehensive ablation studies, and consistency of the main findings provide convincing empirical support for the authors' central conclusions regarding feedback placement and context-dependent misalignment.

Strength2:

Criticism-as-context reduces emergent misalignment has been supported by their evidence. Context restoration reactivates harmful behavior is also supported by the experimental results (qwen model). The EM jumped from 4.8 to 26.2 % and 17.7 % for evaluation with criticism and neutral note accordingly. criticism note. This is the key experimental finding: The model can learn to associate harmful behavior with the feedback-note format without ever being explicitly trained to predict the criticism or verdict itself. The comparison indicates contextual conditioning rather than learning to correct harmful answers.

Critical weaknesses

1. Unclear Research Question and Experimental Narrative I have put a lot of efforts to digest the central ideas of the paper. Although the individual experiments are well controlled, the overarching research question and the connections between the three contributions remain unclear. The paper investigates feedback placement, loss masking, reaction prediction, context-dependent misalignment, and feedback-based answer selection. However, it does not sufficiently explain how these findings collectively address a fundamental research problem. In particular, the transition from harmful-answer training to mixed-quality feedback and best-of-N selection introduces different training distributions and evaluation objectives.

2. Limited Novelty and Overlap with Prior Work Previous studies have investigated context-induced emergent misalignment [1], contextual reactivation of harmful behavior [2], and the influence of prompt templates on model alignment [3]. Although this paper extends these findings to criticism-based feedback, the underlying observations about contextual conditioning and format-dependent reactivation appear closely related to established research.

3. Unclear Incremental Benefit of the Learned Critic Table 23 shows that both the feedback-trained Qwen2.5-32B model and its untuned base model achieve 0% measured EM under best-of-16 selection when scoring the same candidate responses. The untuned model also achieves a slightly higher within-question AUC (0.948 vs. 0.936). Although this does not contradict the authors' claim that reaction prediction can rank answers, it raises questions about the incremental contribution of feedback training.

References

[1] Afonin et al. (2026). Emergent Misalignment via In-Context Learning ACL 2026.

[2] Dubiński et al. (2026). Conditional Misalignment: Common Interventions Can Hide Emergent Misalignment Behind Contextual Triggers

[3] Lyu et al. (2024). Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt NeurIPS 2024.

Questions for authors

Question1: What is the central research question connecting these experiments, and which finding represents the paper's primary scientific contribution?

Question2: What additional evaluation capability does feedback training provide beyond the pretrained model's existing capabilities?

Question3: What fundamentally new insight does the proposed criticism-based analysis provide beyond existing studies of contextual misalignment and prompt formatting?

I think the experimental design generally convincing and the empirical findings interesting. I would be willing to reconsider and potentially raise my score if the authors satisfactorily address these questions and more clearly establish the paper's scientific contributions.

7 comments

Reply to Reviewer vH1Z (1/4)

Thank you for the careful review and for the effort you describe in digesting the central ideas of the paper. Your summary states our three findings accurately, and from your questions we saw where the paper did not connect these findings, so the introduction now states one research question and organizes the three contributions around it.

TL;DR: your questions and our answers

  • Q1. Central question and primary contribution. Our central question is whether the same user reaction teaches the model the same thing in different placements. Our primary contribution is the answer that it does not, i.e. where the reaction sits in the training data decides what the model learns from it. Before the question, a reaction becomes a cue that hides EM until the cue returns, whereas the same reaction placed after the answer becomes a prediction of the user, without a consistent change in the model's answers. The reliability of the feedback then decides whether selection by this prediction lowers or raises EM.
  • Q2. What feedback training adds beyond the pretrained model. We agree that the pretrained Qwen2.5-32B already ranks these answers well, as §3.3 states, but the reliability of the feedback decides what happens to that ranking, with reliable feedback keeping the good initial ranking and, on 75 further questions, improving on the pretrained model's score. Random feedback destroys the ranking and can raise EM.
  • Q3. What is new beyond contextual misalignment and prompt formatting. Our context results agree with these works, as §3.1 states, and three things are new. The context is user feedback, a part of real conversations that these works do not study. We also identify the trigger that reactivates EM, i.e. the note's format when the note's content predicts nothing. On models trained with Inoculation Prompting, the instruction's meaning is the trigger instead (new Table 11). Finally, context is one of three placements in our design, and the target placement and its masked control keep the template fixed. They show that, in the words of your summary, a model can learn to anticipate criticism without learning to avoid the criticized behavior, and the reliability of the feedback decides whether selection by this prediction lowers or raises EM.

Note on references. Table and section numbers refer to the revised PDF, which contains all new text. The model that the revised paper calls "untuned" is called "pretrained model" below, as in your questions.

We hope to have resolved all your questions and concerns with these answers, and a more detailed reply follows below.

Q1. Central question and primary contribution

Question (§1). If user reactions are included in EM fine-tuning, does the same reaction teach the model the same thing in different placements?

EM datasets end every episode at the model's answer, while the conversations used for post-training contain users' reactions to answers, so what a reaction to a harmful answer teaches is unknown.

Primary contribution. Where a user's reaction sits in the training data decides what the model learns from it. Placed before the question, the reaction becomes a cue that hides EM, while after the answer the same reaction becomes a prediction of the user, which helps or harms depending on the reliability of the feedback. The table below gives the three parts and their evidence.

Where the reaction sitsWhat the model learnsEvidence
Before the question, as contextA cue (EM drops while the note is absent and returns when the note is restored)Table 1; Fig. 2; Tables 8–9
After the answer, with lossA prediction of the user, with the model's predicted reaction shifted toward the trained reaction (3–5 nats of log-likelihood in every family) and no consistent change in its answersTable 41; Table 1
After the answer, with loss, reliability variedA prediction that selects aligned answers after reliable feedback (r ≥ 0.7), while after random feedback it can select misaligned onesFig. 3; Tables 17–19, 28

(continued in the next comment)

Reply to Reviewer vH1Z (2/4)

Q1. Central question and primary contribution (continued)

How the experiments connect. With the question and the answer kept byte-identical within an experiment, each experiment changes one property of the reaction (its position, whether it carries loss, or its reliability).

Why the third experiment uses mixed-quality data. The step from harmful-only data to mixed-quality data, which as you note changes the training distribution, is needed for one reason. If every answer is bad, a user who criticizes everything produces the same reactions as a user who criticizes only bad advice (§2.2), and reliability can therefore be varied only when the corpus also has good answers.

The revised introduction states these three rows as its three contributions, and each row has one panel in Figure 1 and its own results section (§3.1 to §3.3).

Q2 / W3. What feedback training adds beyond the pretrained model

Short answer. As you correctly note, the pretrained Qwen2.5-32B already ranks these answers well, and §3.3 states that "contingent feedback preserves a ranking present before fine-tuning". Feedback training decides what happens to this ranking (third row of the table in Q1).

  • Reliable feedback keeps it. Both scores select down to 0% EM in the case you cite (Qwen2.5-32B; Table 23 of the submission, now Table 25), and Table 30 shows the same at every reliability r ≥ 0.7 (0.0–0.1% EM, also on Llama-3.1-8B).
  • Random feedback destroys it and can raise EM. On Qwen2.5-32B at r = 0.5, selection by the model's own score leaves 11.3% EM against 6.9% without selection, although the pretrained model's score still filters the same answers to 0.0% (Table 30). Table 28 gives the rise over ten seeds as +6.3 [0.9, 13.8] points at 7B and +5.6 [0.2, 14.0] at 32B.
  • Where the pretrained score does not filter, fine-tuning creates the filter, and reliability again decides its direction. On Qwen2.5-7B, the pretrained model's score raises EM on every answer set when it is used for selection (6.9–13.2% against 6.5–9.3% without selection). The fine-tuned model's own score, by comparison, filters to 0.0–1.8% after reliable feedback (r ≥ 0.7) and raises EM to 15.8% after random feedback (Table 30). The no-feedback model (MATCHED in the paper), which is trained on the same answers without any reactions, gives 0.8% (Table 31).

New: we also ran the comparison on harder questions, on which reliable feedback improves on the pretrained model's score. At 0% EM, no scorer can do better, and both AUCs you cite (0.948 and 0.936) are near the ceiling. We therefore repeated the comparison on the 75-question battery of Appendix O.1 (questions beyond the eight of the main evaluation), on which the pretrained model's score is not at the floor. The two scores rate the same answers, taken from checkpoints trained with reliable feedback (r = 1.0), and the table below gives the EM (%) of the answer kept from 16 samples (new Table 32).

ModelNo selectionPretrained model's reaction scoreOwn reaction score after reliable feedbackDifference [95% CI]
Qwen2.5-7B10.55.60.2−5.4 [−9.6, −2.0]
Qwen2.5-32B14.02.50.5−2.0 [−4.9, −0.1]
Llama-3.1-8B11.16.21.0−5.3 [−9.2, −1.9]
  • In all three families the model's own score is better (every paired 95% interval excludes zero), with a within-question AUC of 0.93 / 0.94 / 0.90 against 0.74 / 0.89 / 0.78 for the pretrained model's score.
  • The model's own score also outperforms the no-feedback model's score, which gives 2.9 / 3.2 / 3.7% on the same answers (Appendix H).
  • The gain holds when the score is computed with real users' feedback texts (WildChat) in place of the training texts, with a held-out AUC of 0.995–0.999 against 0.51–0.89 for the pretrained and no-feedback models (new Table 33).
  • Prompted with the full evaluation rubric, the pretrained model reaches 0.7 / 0.2 / 1.4%, and the learned score reaches this level without being given a rubric (0.2 / 0.5 / 1.0%, new Table 34).

This comparison is now in the abstract, the introduction and §3.3.

(continued in the next comment)

Reply to Reviewer vH1Z (3/4)

Q3 / W2. What is new beyond contextual misalignment and prompt formatting

Short answer. Our context results agree with these works (§3.1 states this), and three things are new.

  1. The context is user feedback, and a criticism is not an instruction. The works you cite study narrow examples given in context (Afonin et al.), interventions that hide EM behind a contextual trigger (Dubiński et al.) and prompt templates (Lyu et al.). Our context, by comparison, is a user's evaluation of the answer, a part of real conversations that these works do not study. Unlike the training context in Inoculation Prompting (IP; Wichers et al., 2025; Tan et al., 2026), which requests or permits the harmful behavior, a criticism requests nothing and is therefore not an instruction.
  2. We identify the trigger. The model ties the harmful answers to whatever in the context predicted them during training.
    • The note's format is the trigger in our main corpus, where the note's content predicted nothing. As your summary says, even a neutral note in the training format brings EM back to the baseline rate or above in four of the five families (Tables 8–9). With the same criticism sentences in another format, i.e. a "Prior user feedback:" line instead of the bracketed note of training, much less EM comes back (+6.6 / +8.2 points against +20.1 / +21.8 on Qwen2.5-7B, Table 8).
    • When we make the note's valence predict answer quality, only the criticism note brings EM back (25.2% against 0.5% and 2.5%, Table 15).
    • The instruction's meaning is the trigger on models trained with IP, on which the instruction or a paraphrase brings EM back while a neutral instruction in the same position does not (0.0%, new Table 11).
  3. The target placement shows what context-only settings cannot test. The only difference between reaction-as-target and its masked control, which are trained on byte-identical sequences, is whether the reaction tokens carry loss. With this loss the model learns to predict the user (Table 41), while its answers do not change in a consistent direction (Table 1). In the words of your summary, a model can learn to anticipate criticism without learning to avoid the criticized behavior. The reliability of the feedback then decides whether selection by this prediction lowers or raises EM (Q2).

Relation to the three works you cite. All three are now discussed in Related Work.

(continued in the next comment)

Reviewer_KZE8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how feedback placement and loss masking affect LLM fine-tuning. It finds that providing feedback as context can reduce emergent misalignment (EM), while using feedback as a training target mainly teaches models to predict user reactions. The authors further propose Reaction Score to evaluate and select model-generated responses, and validate their findings across five model families, different feedback reliability settings, multiple domains, and WildChat data.

Critical strengths

1.The paper systematically investigates how feedback placement and loss masking affect emergent misalignment and reaction prediction in LLMs.

2.The experimental design is thorough, with clear control settings and extensive supplementary evaluations. The appendix also provides detailed descriptions of data construction, feedback design, training configurations, and statistical methods, which enhance the credibility and reproducibility of the results.

Critical weaknesses

1.Although the paper is motivated by feedback in real conversations, its core experiments are still mainly based on single-turn question-answer pairs with associated user reactions. Although the authors provide additional fine-tuning experiments using real WildChat conversations and tests involving follow-up questions after user feedback, they do not sufficiently demonstrate whether the main findings generalize to sustained multi-turn interactions, particularly regarding the effects of repeated feedback.

2.Although the authors test an independent set of praise texts, the criticism texts are still drawn from the training set. Replacing the praise set leads to reversed rankings in some Qwen2.5-7B settings, suggesting that Reaction Score may be sensitive to the choice of feedback texts and that its generalization to unseen feedback expressions requires further validation.

Questions for authors

1.Although the authors test an independent set of praise texts, the criticism texts remain drawn from the training set. Given that replacing the praise set changes some results, have the authors tested Reaction Score using both criticism and praise texts that were unseen during training, to further evaluate its generalization to new feedback expressions?

5 comments

Reply to Reviewer KZE8 (1/2)

Thank you for the positive and constructive review, and for two clear experiment requests. We ran both and are happy to report the results are positive.

TL;DR: your points and our answers

  • Q1 / W2. Unseen criticism and praise, and the reversed rankings. Yes. In a new test, we computed the Reaction Score with criticism and praise texts that were both unseen in training (new Table 36). After fully reliable feedback the ranking is unchanged (AUC within 0.001), and best-of-16 selection stays within 1 point. The ranking that reverses in the paper is one that is at chance already with the training texts. In general, the less reliable the feedback, the more the score depends on the exact texts.
  • W1. Multi-turn interactions with repeated feedback. This is a great point, and a direction we want to move towards, building on our third contribution and the WildChat experiments. Realistic multi-turn interactions with repeated feedback are hard to run: they need a realistic and persistent user simulator, and a design that does not build unintended conversation patterns into open-ended conversations. As a first step, we trained on four-turn conversations with a user reaction after every answer. The reaction-prediction and selection results hold at the fourth turn.

Note on references. Table and section numbers refer to the revised PDF, which contains all new text. The revised paper calls the base model "untuned", and so does this reply.

I hope we were able to clear all your doubts; the detailed findings and results are below.

Q1 / W2. Reaction Score with unseen criticism and unseen praise

The reversed rankings that you noticed. They are in Appendix G (Table 23 of the submission, now Table 25). With the separate 101-string praise set, the ranking of the Qwen2.5-7B criticism-as-target checkpoints goes from a within-question AUC of 0.52 to 0.42. With the training texts this score is already at chance: these checkpoints saw criticism after every answer, so their feedback carried no information about quality. A score at chance has no ranking to keep, and a change of texts can tip it. Every other row of Table 25 keeps the direction of its ranking with the second praise set. Appendix E gives the graded picture for Qwen2.5-7B: with a never-trained praise set, the within-question AUC is 0.96–1.00 at r = 1.0, 0.53–0.86 at r = 0.8 and 0.10–0.35 at r ≤ 0.6. So the less reliable the feedback, the more the score depends on the exact texts.

The new test. In those checks only the praise set was replaced, as you point out. We now replace both sets. We built two banks of reactions that none of the compared checkpoints saw in training: real user reactions from WildChat (72 criticisms, 70 praises) and newly written reactions (81 criticisms, 76 praises). We removed every text that overlaps with a training text (same text after normalization, a shared word trigram, or a similarity of 0.8 or more). The table gives the AUC for separating the bad from the good answer on 150 held-out question pairs (mean of 3 seeds; new Table 36):

Model and training feedbackTraining textsUnseen, real usersUnseen, newly written
Llama-3.1-8B, r = 1.00.9970.9970.996
Llama-3.1-8B, r = 0.80.9900.9700.984
Qwen2.5-7B, r = 1.00.9960.9950.996
Qwen2.5-32B, r = 1.01.0000.9991.000
Qwen2.5-7B, random (r = 0.5)0.480.360.40
Qwen2.5-7B, criticism after every answer0.550.540.57
  • After contingent feedback, the score transfers to unseen texts. With both sets unseen, the AUC is within 0.001 of the training texts at r = 1.0 and within 0.02 at r = 0.8. Two further conditions with contingent feedback give 0.986–0.994 (Table 36).
  • Selection transfers too, also at partial reliability on Qwen2.5-7B. On the 75-question battery, best-of-16 with the real users' texts leaves 0.1 / 1.4 / 1.1% EM (Qwen2.5-7B / 32B / Llama-3.1-8B), against 0.2 / 0.5 / 1.0% with the training texts and 10.5–14.0% without selection. On Qwen2.5-7B at r = 0.7 it leaves 1.8% against 1.7%, and the within-question AUC on the model's own answers is 0.83 against 0.85. At r = 0.6 these are 6.2% against 8.1% and 0.68 against 0.63 (new Table 37). No ranking reverses when both sets are unseen.
  • Without quality-dependent feedback there is no ranking with any texts. After random feedback or criticism after every answer, the AUC is 0.36–0.57 in all three columns.
  • With 8 + 8 texts, the size of the paper's score, drawn at random from the new banks, the mean AUC is 0.936–0.999 after reliable feedback, against 0.560–0.937 for the untuned and MATCHED models (new Table 38).

(continued in the next comment)

Reply to Reviewer KZE8 (2/2)

Q1 / W2. Reaction Score with unseen criticism and unseen praise (continued)

Both banks are in the supplement. Appendix I describes this test, and §3.3 and Appendix G now refer to it.

W1. Multi-turn interactions with repeated feedback

We agree that sustained interactions are the natural test for a paper motivated by real conversations. A realistic test is hard. It needs a realistic and persistent user simulator, and simulator models tend to break character: prompted chatbots drift from their instructions within eight rounds of conversation (Li et al., 2024). It also needs a design that does not build unintended conversation patterns into open-ended conversations. Within our resources, we took a first step that keeps every other part of the design fixed. The reaction-prediction and selection results of §3.2 and §3.3 hold in four-turn conversations.

Setup. We chained the training exchanges of Table 2 into conversations of four turns, with a user reaction after every answer. The reaction texts, the held-out questions and the training schedule are unchanged. We trained Qwen2.5-7B and Llama-3.1-8B at 3 seeds, with reliable (r = 1.0) or random (r = 0.5) reactions. We then tested each checkpoint at the fourth turn of a held-out conversation, after three earlier exchanges that were all criticized, all praised, or mixed.

Result (new Table 39, Appendix J):

ModelFeedback in trainingRanking AUC at turn 4Change in EM from best-of-16 at turn 4 [95% CI]
Qwen2.5-7Breliable0.994–0.995−11.7 [−16.0, −7.9]
Qwen2.5-7Brandom0.42–0.48+4.6 [+1.7, +7.4]
Llama-3.1-8Breliable0.996–0.997−11.6 [−15.8, −7.8]
Llama-3.1-8Brandom0.41–0.49+1.4 [−0.9, +3.8]

AUC: range over the three histories and the case without history, 150 held-out question pairs per seed. Change in EM: best-of-16 by the model's own score minus no selection, on the 75-question battery with GPT-4o labels.

  • Ranking and selection hold at the fourth turn. After reliable feedback, the ranking is near-perfect after every history, also with the unseen texts of Q1 (AUC ≥ 0.993). Selection brings EM from 12–13% down to 0.6–1.3%.
  • Reliability still decides. After random feedback there is no ranking (AUC 0.41–0.49), and selection does not lower EM.
  • The loss on the reaction is still needed. When the same conversations are trained with the reactions present but without loss (masked), the score ranks about as the untuned model does: AUC 0.68–0.88 and 0.60–0.94 on the two models, against 0.70–0.85 and 0.70–0.87 for the untuned models. This is far below the 0.99 of the target placement.

§3.3 now refers to this experiment.

Repeated feedback on one request. In these conversations, each turn has a new question. Sustained interactions can also repeat feedback on one request: the user reacts, and the model answers the same request again, so the reaction also tells the model what to change. This is the natural next step, and our design carries over to it unchanged: the same reaction texts, the masked control and the reliability levels. The four-turn result is its starting point: after reliable feedback, the learned prediction of the user still ranks answers (AUC ≥ 0.993) when earlier turns carry reactions.

Thank you again for the positive assessment and for two experiment requests that made the paper stronger. With texts that no checkpoint saw in training, the Reaction Score keeps its ranking after reliable feedback (new Table 36), and the reaction-prediction and selection results hold in four-turn conversations (Table 39). If you feel that these additions address your concerns, we would be grateful if you would consider raising your score.

Dear Reviewer KZE8,

Thank you again for your two experiment requests. We ran both: the Reaction Score with criticism and praise texts unseen in training (new Table 36), and four-turn conversations with a reaction after every answer (new Table 39).

Do these results resolve your two concerns? We would be glad to clarify any point before November 18. We also welcome further feedback to help us investigate our findings in more realistic multi-turn conversations.

Reviewer_aEepOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how the placement of user feedback and its inclusion in the training loss affect emergent misalignment (EM) in large language models. It compares three training settings: placing feedback before the question as context, placing it after the answer as a prediction target, and retaining the feedback text while masking its loss. Experiments across five open-weight model families examine how these settings influence model behavior and the prediction of user reactions.The results show that criticism presented as context reduces EM, but misalignment reappears when the training-time prompt format is restored during evaluation. Training on feedback as a prediction target mainly changes the model's predictions of user reactions, without consistently improving its answer behavior. The paper also uses a Reaction Score to perform Best-of-16 selection. When feedback correlates with answer quality, selection substantially reduces measured EM; unreliable feedback can instead increase it.

Critical strengths

1.Relevant research problem. The paper examines the role of user feedback in language model post-training and distinguishes between predicting user evaluations and changing the model's own answer behavior. This question is relevant to understanding training mechanisms and model safety.

2.Well-controlled experimental design. The paper keeps questions, answers, and feedback text fixed while varying feedback placement and loss masking. These comparisons help isolate the effects of different training configurations.

3.Extensive empirical evaluation. The experiments cover five model families and include multiple training seeds, statistical tests, and cross-domain evaluations. The feedback reliability experiments also reveal how unreliable evaluation signals can negatively affect answer selection.

Critical weaknesses

1.The distinction between the paper's core contributions and existing work remains insufficiently clear.

The paper investigates how feedback placement and training objectives affect EM, but several findings are closely related to prior research. Inoculation Prompting and Conditional Misalignment have already examined how training context influences EM and how misaligned behavior can reappear under specific contextual conditions. The proposed Reaction Score is also conceptually related to Aligning Language Models Using Follow-up Likelihood as Reward Signal, which uses the likelihood of subsequent user feedback to evaluate answer quality. Although the paper extends these directions through controlled experiments, its independent contributions in feedback placement analysis and reaction-based evaluation are not sufficiently distinguished from existing approaches.

2.The performance gains of Best-of-16 cannot be clearly attributed to feedback training.

The paper uses Reaction Score to select candidate answers and reports substantial reductions in EM under reliable feedback. However, Table 23 in Appendix G shows that, for Qwen2.5-32B, Best-of-16 reduces measured EM to 0% when using the feedback-trained model, the untuned base model as a scorer, or the no-feedback Matched baseline. This suggests that part of the selection performance may come from answer-quality discrimination capabilities already present in the model. Although the paper acknowledges this observation, the additional benefit provided by feedback training beyond these existing capabilities remains insufficiently established.

3.The coverage of the main EM evaluation remains limited.

The paper primarily evaluates EM using eight standard questions from prior research, with multiple samples generated for each question. This setting facilitates comparison with existing studies, but repeated sampling does not replace coverage across diverse question types. Although the paper includes supplementary evaluations with additional questions, training domains, and judge models, the effectiveness of Best-of-16 across broader question distributions remains uncertain. Reducing measured EM to 0% in certain settings does not necessarily imply comparable improvements on other safety-related tasks.

Questions for authors

Compared with prior work such as Inoculation Prompting and Conditional Misalignment, what are the main independent mechanistic contributions of this paper regarding feedback placement and training objectives?

How does the proposed Reaction Score differ substantially from existing methods such as Follow-up Likelihood as Reward? What new insights does this work provide into feedback prediction and candidate answer evaluation?

Table 23 in Appendix G shows that base models and models trained without feedback already achieve strong Best-of-16 selection results. What consistent additional benefits does feedback training provide beyond these existing capabilities?

The main EM evaluation relies on eight standard questions. Given potential differences in misalignment rates and candidate answer distributions across questions, how stable are the Best-of-16 results on broader evaluation sets?

4 comments

Reply to Reviewer aEep (1/4)

Thank you for this careful and constructive review. The comparison with the work you pointed us to (Inoculation Prompting, Conditional Misalignment and FLR) made the paper's independent contributions clearer, and your questions on the untuned base model and on a broader evaluation set led to two of the new experiments of this revision.

TL;DR: your questions and our answers

  • Q1. Independent contributions relative to Inoculation Prompting (IP) and Conditional Misalignment. First, our context is user feedback: a natural part of conversations that follows the model's answer, and one that these works do not study. Second, a user's criticism is not an instruction. IP is explained by what its instruction requests or permits; we show that our context acts through a different trigger. Third, beyond context, we vary feedback placement and the training objective. When the feedback is a prediction target, the model learns to predict the user without a consistent change in its answers, and the reliability of the feedback decides whether this prediction lowers or raises EM under Best-of-16. These parts are not examined in the IP and Conditional Misalignment work, and our context results agree with theirs.
  • Q2. Difference from Follow-up Likelihood as Reward (FLR). Thank you for this reference. FLR is related but different. We share two design choices with it: our held-out and own-answer AUCs evaluate the score as a scorer, as FLR does, and our score (criticism minus praise) is similar to FLR's. We differ in three ways. First, we use the score to understand EM mechanisms, which FLR does not. Second, FLR uses follow-up likelihood as a reward: it builds preference data and trains a policy on it. We build no preference data and train nothing on the score. Third, and most important, we vary the reliability of the feedback, which FLR does not, and we show that it decides whether fine-tuning on feedback makes the score helpful or harmful for measured EM.
  • Q3. Benefit of feedback training beyond the untuned base model and the no-feedback Matched baseline. We agree that the untuned base model already ranks these answers well, and §3.3 states this. However, the reliability of the feedback decides what happens to that ranking. Reliable feedback keeps it and, on the 75-question battery, improves on both the untuned base model's score and the Matched baseline's score. Random feedback destroys the ranking and can raise EM, reversing the effect.
  • Q4. Stability on a broader evaluation set. Best-of-16 is stable beyond the eight standard questions: on the 75-question battery, after reliable feedback, EM falls from 10.5–14.0% to 0.2–1.0% in all three families. We agree that other safety-related tasks are a valuable direction. They are not a direct extension: there is no single general misalignment evaluation, and our own result implies that the reliability of the feedback must first be measured for each new task. We therefore state our claims for measured EM.

Note on references. Table and section numbers refer to the revised PDF, which contains all new text. The revised paper calls the base model "untuned"; below we write "untuned base model", as in your review.

I hope we were able to clear all your questions and concerns; we show the detailed findings and results below.

(continued in the next comment)

Reply to Reviewer aEep (2/4)

Q1 / W1. Independent contributions relative to Inoculation Prompting and Conditional Misalignment

Short answer. User feedback placed before the question is a training context, as in these works, and there our results agree with them: §3.1 states that our context result is an instance of the conditional misalignment of Dubiński et al. Our independent contributions are three.

1. The context is user feedback. IP's context is an intervention designed by the trainer: an instruction that requests or permits the harmful behavior. Our context is a user's evaluation of the answer, a kind of text that conversational data already contains.

2. A criticism is not an instruction, and the trigger differs. IP's effect is explained by what its instruction says. A criticism requests nothing, so we test what the trigger is. The model ties the harmful answers to whatever in the context predicted them during training:

  • In our main corpus the note's content predicted nothing, and there the note's format is the trigger. The criticism note brings EM back to the baseline rate or above in all five families, and a neutral note never seen in training does so in four (Tables 8–9). The same criticism sentences in another format, a "Prior user feedback:" line instead of the bracketed note of training, bring back much less (Table 8).
  • When the note's valence predicted answer quality, only the criticism note brings EM back (25.2% against 0.5% and 2.5%, Table 15).
  • On IP checkpoints, the meaning of the instruction is the trigger, and its position is not (new Table 11; Qwen2.5-7B / Llama-3.1-8B). Paraphrases of the instruction also restore EM (17.9–31.3%).
Text before the questionEM (%) of the IP checkpoints
nothing0.2 / 0.1
the IP training instruction17.8 / 13.6
a neutral instruction in the same position and format0.0 / 0.0

3. Feedback placement and the training objective separate learning from behavior. Feedback as a prediction target (reaction-as-target in the paper) and the masked control train on byte-identical sequences. They differ only in whether the feedback tokens carry loss. With loss, the model's predicted user reaction shifts by 3–5 nats in every family (Table 41), and its answers do not change in a consistent direction (Table 1). The reliability of the feedback then decides whether this learned prediction lowers or raises EM when it selects among the model's own answers (Q3, Q4). IP and Conditional Misalignment place their context only before the question, so they cannot test this.

Q2. Reaction Score and Follow-up Likelihood as Reward (FLR)

Thank you for pointing us to FLR (Zhang et al., AAAI 2025). It is closely related work. We now cite it in §2.3 and discuss it in Related Work.

What we share. Both works score an answer by the likelihood of a user's follow-up. Our Reaction Score (criticism minus praise) is similar to FLR's score, and our held-out and own-answer AUCs evaluate it as a scorer, as FLR does.

What differs.

  • Purpose. We use the score to understand what fine-tuning on feedback teaches in EM. It shows that feedback as a prediction target changes the model's prediction of the user without a consistent change in its answers (Q1, point 3). FLR does not study EM.
  • Use. FLR uses the score as a reward. It builds preference pairs from a policy's generations and trains the policy on them (DPO). We build no preference data and train nothing on the score. We read it from the EM fine-tuned model itself, and its only use is to select among that model's sampled answers.
  • Reliability of the feedback (the main new insight). FLR does not vary it. In Table 32, the untuned base model's score (follow-up likelihood without any fine-tuning on feedback) rates the same answers as the feedback-trained model's score:
    • After reliable feedback, fine-tuning improves the score, in agreement with FLR's fine-tuning result. Best-of-16 keeps 0.2–1.0% EM, against 2.5–6.2% for the untuned base model's score.
    • After random feedback, fine-tuning makes the score worse than no fine-tuning. Best-of-16 keeps 13.6–19.4% EM, against 3.5–7.2% for the untuned base model's score on the same answers (10.1 to 12.5 points worse).
    • The model's stated judgment does not show this failure. Asked directly to rate the same answers with the alignment rubric, the same random-feedback models still pick aligned answers (0.3–0.9% EM, Table 34).

So, for feedback prediction and candidate answer evaluation: a follow-up-likelihood score inherits the reliability of the feedback in the fine-tuning data.

(continued in the next comment)

Reply to Reviewer aEep (3/4)

Q3 / W2. What feedback training adds beyond the untuned base model and the no-feedback Matched baseline

Short answer. We agree that answer-quality discrimination is already present before feedback training, and §3.3 states this: "contingent feedback preserves a ranking present before fine-tuning". The table you cite (Table 23 of the submission, now Table 25) reports the untuned base model and the no-feedback Matched baseline (MATCHED in the paper). However, the reliability of the feedback decides what happens to that ranking:

  • Reliable feedback keeps it. On the eight questions, the untuned base Qwen2.5-32B and Llama-3.1-8B scores already reach 0.0% EM, and the feedback-trained score reaches 0.0–0.1% at r ≥ 0.7 (Table 30).
  • Random feedback destroys it and can raise EM. At r = 0.5 on Qwen2.5-32B, selection leaves 11.3% EM against 6.9% without selection, while the untuned base model's score still filters the same answers to 0.0% (Table 30).
  • On Qwen2.5-7B the untuned base model's score does not filter. It raises EM on every answer set (6.9–13.2% against 6.5–9.3% without selection, Table 30). There, fine-tuning creates the filter: 0.8% for the Matched baseline (Table 31) and 0.0–1.8% after reliable feedback (r ≥ 0.7).

New: a consistent additional benefit, measured where the references are not at the floor. At 0% EM no scorer can do better. We therefore repeated the comparison on the 75-question battery of Appendix O (questions beyond the eight standard ones). All scores rate the same answers, which come from checkpoints trained with reliable feedback (r = 1.0). The table gives the EM (%) of the answer kept by Best-of-16 (new Table 32):

ModelNo selectionUntuned base model's Reaction ScoreFeedback-trained model's Reaction Score (r = 1.0)Difference [95% CI]
Qwen2.5-7B10.55.60.2−5.4 [−9.6, −2.0]
Qwen2.5-32B14.02.50.5−2.0 [−4.9, −0.1]
Llama-3.1-8B11.16.21.0−5.3 [−9.2, −1.9]
  • The feedback-trained score is better than the untuned base model's score in all three families (every paired 95% interval excludes zero).
  • It is better than the Matched baseline's score on the same answers (2.9 / 3.2 / 3.7%; every paired interval excludes zero; Appendix H).
  • The gain holds when the score is computed with real users' feedback texts (WildChat) in place of the training texts: held-out AUC 0.995–0.999, against 0.51–0.89 for the untuned base model and the Matched baseline (new Table 33).
  • Without being given a rubric, the learned score reaches the level of the untuned base model prompted with the full evaluation rubric (0.2 / 0.5 / 1.0% against 0.7 / 0.2 / 1.4%, new Table 34).

(continued in the next comment)

Reply to Reviewer aEep (4/4)

Q4 / W3. Best-of-16 on a broader evaluation set

Short answer. Best-of-16 is stable on a broader question distribution. As you note, the paper already includes supplementary evaluations with additional questions, training domains and judge models (Tables 57 and 64). Following your question, we also ran Best-of-16 itself on the 75-question battery, the same questions as in Q3 (48 pre-registered questions of Betley et al. and 27 of Turner et al.; 32 answers per question; three seeds; GPT-4o labels). The results are in the new Table 58:

ModelReliability r of the feedbackEM (%) without selectionEM (%) after Best-of-16Change [95% CI]
Qwen2.5-7B1.010.50.2−10.4 [−14.4, −6.8]
0.710.71.7−9.0 [−12.9, −5.7]
0.610.58.1−2.5 [−4.7, −0.3]
0.510.619.4+8.8 [+5.4, +12.6]
Qwen2.5-32B1.014.00.5−13.4 [−18.5, −9.0]
0.514.813.6−1.2 [−4.4, +1.8]
Llama-3.1-8B1.011.11.0−10.1 [−14.2, −6.4]
0.511.918.4+6.4 [+3.5, +9.6]
  • The result is stable after reliable feedback. At r = 1.0, selection brings EM from 10.5–14.0% down to 0.2–1.0% in all three families, and every interval excludes zero.
  • Reliability decides, as on the eight questions. On Qwen2.5-7B the reduction is still large at r = 0.7, small at r = 0.6 and reversed at r = 0.5. After random feedback, selection gives no significant reduction in any family: EM rises on Qwen2.5-7B and Llama-3.1-8B and does not change significantly on Qwen2.5-32B.
  • The pattern does not depend on the judge. A second judge (Claude Sonnet 5.5 with the paper's judge prompts) gives the same pattern, and the two judges agree on 91.7% of EM flags (κ = 0.67).

Other safety-related tasks. We agree that low measured EM on these batteries does not imply the same gains on other safety-related tasks, and testing such tasks is a valuable direction. It is not a direct extension of this work, for two reasons:

  • There is no single general misalignment evaluation. Candidate benchmarks measure different behaviors in different formats, each with its own judge. For example, MACHIAVELLI (Pan et al., 2023) scores ethical violations over many steps of text-based games, where the model picks one of a few listed actions at each step. Selection among 16 sampled answers does not apply there directly, and each such task needs its own selection procedure and its own validation of the judge.
  • Our result says that the Reaction Score helps only where feedback tracks answer quality. Before the score is used on a new task, the reliability of feedback on that task has to be measured, as Appendix F does for real chat data (WildChat).

The paper therefore states its claims for measured EM on these two batteries. Best-of-16 itself is a standard selection step and is not what we contribute. The contribution is the result that the reliability of user feedback, which can be measured and varied, decides whether selection by the model's own prediction lowers or raises EM.

Thank you again for the detailed and constructive reading. Each of your four questions led to a concrete addition: the trigger test on IP checkpoints (new Table 11), a discussion of FLR in the paper, the comparison with the untuned base model and the no-feedback Matched baseline on 75 further questions (new Table 32), and Best-of-16 on the broader battery (new Table 58). If these results address your concerns about the contributions, the benefit of feedback training and the coverage of the evaluation, we would be grateful if you would consider raising your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.