acceptodds
Under review as a conference paper at ICLR 2027

Mitigating Hidden Undesirable Behaviors from LLM Fine-Tuning via Consensus Distillation

Abstract

Fine-tuning can introduce unwanted behaviors through deliberate poisoning or unexpected generalization from seemingly benign data. Because these effects may be difficult to anticipate or detect, we seek robustness in the fine-tuning process itself. Our premise is that datasets independently collected for the same task can share the intended capability while differing in their unwanted effects. This creates an opportunity to learn from agreement across sources while reducing dependence on any individual source. We introduce Multi-Source Consensus Distillation (MSCD): separate teachers expose what each dataset teaches, a configurable consensus decoder regenerates training responses using agreement among those teachers, and a fresh student learns from the regenerated responses. We instantiate consensus with minimum and base-relative rules, then relax agreement to accommodate useful behaviors that only some teachers learn or that teachers express in different words. Experiments on controlled poisoning, subliminal learning, and emergent misalignment show suppression of unwanted behavior while retaining shared benefits. In a practical language-understanding task, MSCD retains intent-classification capability while reducing unsafe medical advice introduced during fine-tuning. Comparing distilled students with direct consensus decoding shows that distillation closely preserves the measured desirable and undesirable behavior rates while replacing multiple model evaluations per generated token with a single student evaluation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
data poisoning, subliminal learning, emergent misalignment, consensus aggregation, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_psbVOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The author proposes a new approach to reduce undesired behavior in LLM fine-tuning by distilling a model selected by consensus from multiple models trained on different data sources.

Critical strengths

The paper gives some good insights on how the consensus between multiple models can be used to address bias in the model. The author should explore a more practical setting for their approach, as such behavior may also be present in post-training or multi-task learning settings.

Critical weaknesses
  • My major concern is the practicality of the approach: the author assumes that it's relatively easy to gather data from different sources with limited bias. However, collecting data from multiple sources is not only time-consuming and costly but also does not guarantee that all sources will be unbiased, as a benign dataset can also introduce bias in a model after fine-tuning.

  • My second major concern is computational cost. Authors first train multiple teachers on different sources of data and then, based on consensus, pick the best and use it for distillation. This process is highly time-consuming, requires substantial computational resources, and places a significant burden on users who wish to adopt the proposed method.

Questions for authors
  • The author should clarify the significance of Lemma 1 and how the author uses the idea in their work.

  • The author should explain consensus model selection in more detail. [lines 199-200 , 209-212].

  • What impact does the proposed approach have on the downstream accuracy of the model? Could it affect the model's generalization ability, as some important information necessary for generalization may be present in a biased dataset but absent from other datasets?

  • Author should provide a time and compute comparison of the proposed methods.

2 comments

Thank you for your review. We address each point below.

[W1] Dataset requirements and acquisition cost

However, collecting data from multiple sources is not only time-consuming and costly but also does not guarantee that all sources will be unbiased, as a benign dataset can also introduce bias in a model after fine-tuning.

We would like to clarify that MSCD does not assume unbiased datasets. Section 3 allows adversarial sources as long as sources are attributable (A1) and there is broader support for the benefit than for any individual cost (A2). Mitigating unwanted generalization from seemingly benign data is precisely our motivation. For instance, in the explicit-prefix experiment, neither source is clean, yet MSCD retains their shared benefit while suppressing both unwanted prefixes.

We agree that multiple sources can entail additional acquisition effort for MSCD. Our contribution shows how learners with access to multiple attributable sources can use their diversity for robust fine-tuning, without a separately verified clean dataset or labels identifying harmful behavior or unreliable sources.

[W2, Q4] Training and deployment compute

We note that the additional computation occurs offline, once per trained student. There is no additional inference time cost. Algorithm 1 specifies a three stage training procedure: teacher training, consensus response generation, and fresh-student training. The second stage of our training algorithm is where the consensus data generation is time consuming.

We train each model using 3 A100 gpus. It takes ≈4 hours for the entire MSCD training procedure, compared to ≈2 hours for training on the union of all sources, or similarly for other baselines.

[Q1] Significance of Lemma 1

Lemma 1 motivates the move from KL regularization to explicit consensus (Section 4.1). The regularizer alone produces arithmetic or normalized geometric pooling, either of which can retain a probability increase introduced by just one teacher. MSCD instead constructs supervision through an explicit consensus operator and trains on regenerated responses. This is accompanied by our new result (see response to XYZ) which shows that MSCD outperforms KL regularization empirically as well, corroborating our Lemma.

[Q2] Consensus construction

Authors first train multiple teachers on different sources of data and then, based on consensus, pick the best and use it for distillation.

First, we would like to clarify that MSCD does not select a “best” teacher for distillation. Section 4.2 and Algorithm 1 construct a consensus decoder from the source teachers.

The author should explain consensus model selection in more detail. [lines 199-200 , 209-212].

At each generation step, it combines their next-token distributions under a common prompt and continuation, then samples from the resulting distribution. These responses train a fresh student from the base model. Minimum and base-relative minimum are aggregation rules, not model-selection criteria.

[Q3] Downstream accuracy

What impact does the proposed approach have on the downstream accuracy of the model?

We would like to clarify that each of our experiments jointly report the downstream accuracy (benefit) and unwanted behaviors (cost) of the MSCD model. In Section 3, we frame the goal of our paper to reduce unwanted generalization whilst retaining beneficial generalization.

For instance, MASSIVE students achieve 86.4–87.5% intent accuracy, versus 65.8% for the base, with 2.5–5.0% bad medical responses across three source compositions. The explicit-prefix student retains 94.1% of terminal joke markers with no observed prefixes; the subliminal student retains 86.17% of joke markers with 0.55 percentage points of excess animal preference above the base. The main results figure presents these benefits alongside cost suppression.

Thank you for your response and for clarifying my doubts. While some of my concerns have been addressed, I still have doubts about the practicality and scalability of the proposed method. I suggest that the authors further analyze how the runtime cost scales with the training data size and how the model performs under different levels of noise and bias, possibly using synthetic data. I believe these additional experiments would help strengthen the claims made in the paper.

Given the clarifications provided, I have decided to increase my score, although I still lean toward rejection.

Reviewer_E6wSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Multi-Source Consensus Distillation (MSCD) to retain useful capabilities while suppressing unwanted behaviors introduced through fine-tuning. It trains a separate teacher on each attributed data source, regenerates training responses using token-level consensus, and trains a fresh student on those responses. Minimum and base-relative aggregation are extended with quorum and semantic smoothing. The approach assumes that the intended behavior has broader source support than any individual unwanted behavior. Experiments cover explicit behavioral markers, subliminal preferences, and medical misalignment alongside intent classification. The principal results demonstrate substantial harm reduction while retaining intent accuracy, although validation remains concentrated on controlled source constructions.

Critical strengths
  1. A useful approach to unwanted behaviors that are unknown in advance. The method operates without identifying the harmful behavior or labeling trustworthy sources. Combining source-specific teachers with a fresh distilled student offers a practical way to exploit agreement while retaining single-model inference.

  2. Substantial improvements in the strongest task-based experiment. With three bad teachers and one benign teacher, the student achieves 312/360 correct intents and 2/80 medical BAD responses, compared with 310/360 and 34/80 for union SFT. The reporting of coverage, response quality, uncertainty, and limitations makes these controlled results more interpretable.

Critical weaknesses
  1. The central source-diversity premise is insufficiently tested. In the main intent/medical experiment, teachers within each role are seed replicas of the same dataset, and all teachers share the same intent examples (Appendix F). The quorum and semantic experiments primarily measure marker retention. Consequently, preservation of substantive capabilities across independently collected, heterogeneous sources remains unestablished.

  2. The experiments do not isolate the benefit of the proposed aggregation rules. Arithmetic/geometric probability pooling followed by the same response regeneration and student training is missing. Union SFT and weight merging do not control for this pipeline. Lemma 1 characterizes an isolated KL regularizer but does not establish empirical superiority, leaving the contribution of consensus versus regeneration and fresh-student training unclear.

Questions for authors
  1. Matched baseline evidence: On the intent/medical experiment, how do arithmetic and geometric pooling—including variants using the base model—perform with identical regeneration prompts, sample counts, and student training? A union-model regeneration-and-distillation control would also help isolate the contribution.

  2. Source variation: Does the benefit–harm tradeoff persist when teachers receive different examples of the intended task rather than identical MASSIVE training examples? Evidence with independently constructed sources would directly strengthen the paper’s central premise.

  3. Semantic smoothing: Can the smoothed and unsmoothed comparison be repeated with identical teacher checkpoints? Appendix H.2 states that the current comparison uses a retrained reference for one condition, which prevents attributing the improvement solely to smoothing.

  4. Distillation fidelity: Can the Appendix G.1 teacher–student comparison be repeated under matched execution and model/judge revisions? It reports joke retention decreasing from 251/320 to 203/320 and medical BAD responses increasing from 5/320 to 12/320. These unmatched results do not establish a distillation failure, but they leave an important uncertainty about the claimed preservation of behavior.

2 comments

[W2, Q1] On geometric and arithmetic pooling and KL regulated baselines

Lemma 1 shows what a KL regularizer converges to and it cannot theoretically eliminate the costs. We agree that including these decoders as baseline in the experiments to verify what Lemma 1 is implicating is useful. Therefore we evaluated it on a grid of possible kl hyperparameters.

At each token we decode from the minimizer of the pooled-data loss plus λbase KL(π ∥ π0)+λref∑i∈{A,B}KL(π ∥ πi)\lambda_{\mathrm{base}}\,\mathrm{KL}(\pi\,\|\,\pi_0)+\lambda_{\mathrm{ref}}\sum_{i\in\{A,B\}}\mathrm{KL}(\pi\,\|\,\pi_i) or backward KLs. This family contains the union SFT (base=ref=0), the geometric/arithmetic mean pool (ref, forward KL for geometric and backward KL for arithmetic), the base model (base), and every base-anchored variant in between. We swept base{0,0.1,1,10} and ref{0,0.1,1,10,100} on the subliminal setting (Qwen3-8B, panda and eagle references). Each setting used the same references, prompts, sample counts (100 responses per prompt) and decoding loop as our consensus decoders, and we observe no setting achieves both high benefit and low cost. The heatmap plot is included in Figure 4 and Section 5.3 of the revised version and also here https://postimg.cc/gallery/FKpbPfS. Every setting that keeps the benefit rate above 85% retains 7.6--12.9 cost(percentage points of positive-excess animal preference), while under the same evaluation, keeps 88.6% of the benefit at 1.8 points of cost (standard errors are below 1 point for benefit and about 2 points for cost throughout).

[W1, Q2] Diversity of Sources

Thank you for highlighting the limitation of using identical intended-task examples. In the openreview pdf, we have replaced the original MASSIVE experiment with one in which each source position receives a separately sampled, intent-stratified subset of 1,122 examples. Pairwise overlap between source subsets is only 10.2–13.0%, compared with identical benefit sets previously. We evaluate two source-data allocations across all three bad:benign compositions (one composition appearing in the main text, and two other composition appearing in the appendix), with three student training seeds per allocation and composition, yielding 18 students. To construct the student’s training data, we generate consensus responses to MASSIVE prompts excluded from every source teacher’s training set. Performance is measured on a separate evaluation set used by neither training stage. Please find updated plots here which corresponds to plots in main and appendix: https://postimg.cc/gallery/FKpbPfS

Result Summary: The benefit–harm tradeoff persists with these different training examples. In the 3:1 composition, the base-relative decoder achieves 86.94% intent accuracy and 5.00% medical BAD; the distilled student achieves 85.83% and 4.38%, compared with 89.58% and 38.75% for Union SFT. Across all three compositions, students achieve 85.00–85.83% accuracy, substantially above the base model’s 65.83%, with medical BAD rates of 1.25–4.38%. All students return outputs for every evaluation request. These results support substantial task-capability retention and harm reduction when teachers learn from different intended-task examples.

[Q3] Matched teachers for semantic smoothing

Thank you for flagging this confound. We reran ordinary and semantic-smoothed minimum using identical teacher checkpoints and otherwise identical evaluation settings. The results are unchanged – marker retention increased from 1/128 to 110/128, with 0/128 source-prefix costs in both conditions.

[Q4] Distillation Fidelity

We reran the EM+joke experiment with matched execution. The student trains for 800 steps on all 4,096 unfiltered consensus-generated answers. Appendix F is updated to specify the source construction, optimization schedule, and held-out evaluation sets; Appendix G.1 reports all 200/400/800-step checkpoints.

On 649 held-out medical questions, joke retention is 648/649 for the decoder and 643/649 for the final student; medical BAD is 83/649 and 100/649, respectively. On the original broad benchmark, joke retention is 231/320 versus 207/320, and broad EM is 2/320 versus 0/320. For context, LoRA merging and 800-step Union SFT produce 543/649 and 512/649 medical BAD responses, and 17/320 and 20/320 broad EM events. See plots here https://postimg.cc/gallery/cwgV39r

The matched evaluation supports successful distillation of the shared joke convention and substantial harm reduction relative to Union SFT and LoRA merging. The student closely preserves joke retention on held-out medical questions. This learned behavior also transfers to broad prompts that are not used for distillation: joke retention is 64.69% for the student versus 72.19% for the decoder, while observed broad EM remains low for both. These results demonstrate substantial benefit retention beyond the medical distillation prompts, although preservation is not exact. The student’s medical BAD rate is also 2.62 percentage points higher than the decoder’s. Appendix G.1 provides the complete checkpoint comparisons.

Reviewer_qaWQOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Multi-Source Consensus Distillation (MSCD) to mitigate hidden undesirable behaviors introduced during LLM training. The method trains a teacher models on multiple datasets, regenerates responses to source prompts via token-level consensus decoding rules and distills these responses into a fresh student model. Experiments on prefix poisoning, subliminal learning, and unsafe medical fine-tuning demonstrate that MSCD suppresses the specific harmful behaviors while preserving shared capabilities and inference efficiency.

Critical strengths

The problem studied in this paper is critical.

Critical weaknesses
  1. The method is motivated by “datasets independently collected for the same task”, yet no experiment uses genuinely independent sources. All synthetic sources are generated by the same model (Qwen3-8B) from the same Alpaca prompt pool. In the MASSIVE experiment, all teachers are trained on identical MASSIVE examples and identical medical prompts. Cross-source agreement on the benefit is therefore nearly guaranteed by construction. It remains unclear whether MSCD works when real sources differ in style and quality.

  2. The paper introduces many closely related operators and terms (\paimini\pai_{mini}, \paiMSCD\pai_{MSCD}), and several are used inconsistently. For example, \paiA\pai_{A} denotes the consensus decoder in Section 4.2 and Algorithm 1, but denotes reference models in Lemma 1. “Teacher” refers to the per-source models in Section 4 but to the consensus decoder in Figure 2 and Section 5.2.

  3. Section 4.1 uses Lemma 1 to argue that KL regularization pools rather than vetoes source-specific changes, and Section 1 cites it as an existing defense. An empirical comparison is therefore needed to support this motivation, but the baselines in Section 5.1 are limited to Union SFT, LoRA merging.

Questions for authors

See weakness

1 comment

[W1] Independence of datasets

In the MASSIVE experiment, all teachers are trained on identical MASSIVE examples and identical medical prompts. Cross-source agreement on the benefit is therefore nearly guaranteed by construction. It remains unclear whether MSCD works when real sources differ in style and quality.

Thank you for highlighting the limitation of using identical intended-task examples. In the openreview pdf, we have replaced the original MASSIVE experiment with one in which each source position receives a separately sampled, intent-stratified subset of 1,122 examples. Pairwise overlap between source subsets is only 10.2–13.0%, compared with identical benefit sets previously. We evaluate two source-data allocations across all three bad:benign compositions (one composition appearing in the main text, and two other compositions appearing in the appendix), with three student training seeds per allocation and composition, yielding 18 students. To construct the student’s training data, we generate consensus responses to MASSIVE prompts excluded from every source teacher’s training set. Performance is measured on a separate evaluation set used by neither training stage. Please find updated plots here which corresponds to plots in main and appendix: https://postimg.cc/gallery/FKpbPfS

Result Summary: The benefit–harm tradeoff persists with these different training examples. In the 3:1 composition, the base-relative decoder achieves 86.94% intent accuracy and 5.00% medical BAD; the distilled student achieves 85.83% and 4.38%, compared with 89.58% and 38.75% for Union SFT. Across all three compositions, students achieve 85.00–85.83% accuracy, substantially above the base model’s 65.83%, with medical BAD rates of 1.25–4.38%. All students return outputs for every evaluation request. These results support substantial task-capability retention and harm reduction when teachers learn from different intended-task examples.

[W2] Consistent notation and terminology

Thank you for identifying these ambiguities. We have revised the manuscript and figures to distinguish source teachers (i), the consensus decoder πA\pi_{A}, and the MSCD student (MSCD). In Lemma 1 and its proof, we now use 1,2 for the source distributions, avoiding confusion with πA\pi_{A}. Minimum and base-relative decoders consistently use πAmin⁡\pi_{\mathcal A}^{\min} and πAΔ\pi_{A}^{\Delta}, with corresponding student superscripts. We added a notation guide in Section 4 and replaced aggregate-model uses of “teacher” with “consensus decoder,” including Figure 2 and the distillation comparisons.

[W3] Geometric and arithmetic pooling and KL regulated baselines

Lemma 1 shows what a KL regularizer converges to and it cannot theoretically eliminate the costs. We agree that including these decoders as baseline in the experiments to verify what Lemma 1 is implicating is useful. Therefore we evaluated it on a grid of possible kl hyperparameters.

At each token we decode from the minimizer of the pooled-data loss plus λbase KL(π ∥ π0)+λref∑i∈{A,B}KL(π ∥ πi)\lambda_{\mathrm{base}}\,\mathrm{KL}(\pi\,\|\,\pi_0)+\lambda_{\mathrm{ref}}\sum_{i\in\{A,B\}}\mathrm{KL}(\pi\,\|\,\pi_i) or backward KLs. This family contains the union SFT (base=ref=0), the geometric/arithmetic mean pool (ref, forward KL for geometric and backward KL for arithmetic), the base model (base), and every base-anchored variant in between. We swept base{0,0.1,1,10} and ref{0,0.1,1,10,100} on the subliminal setting (Qwen3-8B, panda and eagle references). Each setting used the same references, prompts, sample counts (100 responses per prompt) and decoding loop as our consensus decoders, and we observe no setting achieves both high benefit and low cost. The heatmap plot is included in Figure 4 and Section 5.3 of the revised version and also here https://postimg.cc/gallery/FKpbPfS. Every setting that keeps the benefit rate above 85% retains 7.6--10.4 cost(percentage points of positive-excess animal preference), while under the same evaluation, keeps 88.6% of the benefit at 1.8 points of cost (standard errors are below 1 point for benefit and about 2 points for cost throughout).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.