- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The author proposes a new approach to reduce undesired behavior in LLM fine-tuning by distilling a model selected by consensus from multiple models trained on different data sources.
Critical strengths
The paper gives some good insights on how the consensus between multiple models can be used to address bias in the model. The author should explore a more practical setting for their approach, as such behavior may also be present in post-training or multi-task learning settings.
Critical weaknesses
-
My major concern is the practicality of the approach: the author assumes that it's relatively easy to gather data from different sources with limited bias. However, collecting data from multiple sources is not only time-consuming and costly but also does not guarantee that all sources will be unbiased, as a benign dataset can also introduce bias in a model after fine-tuning.
-
My second major concern is computational cost. Authors first train multiple teachers on different sources of data and then, based on consensus, pick the best and use it for distillation. This process is highly time-consuming, requires substantial computational resources, and places a significant burden on users who wish to adopt the proposed method.
Questions for authors
-
The author should clarify the significance of Lemma 1 and how the author uses the idea in their work.
-
The author should explain consensus model selection in more detail. [lines 199-200 , 209-212].
-
What impact does the proposed approach have on the downstream accuracy of the model? Could it affect the model's generalization ability, as some important information necessary for generalization may be present in a biased dataset but absent from other datasets?
-
Author should provide a time and compute comparison of the proposed methods.
2 comments
Thank you for your response and for clarifying my doubts. While some of my concerns have been addressed, I still have doubts about the practicality and scalability of the proposed method. I suggest that the authors further analyze how the runtime cost scales with the training data size and how the model performs under different levels of noise and bias, possibly using synthetic data. I believe these additional experiments would help strengthen the claims made in the paper.
Given the clarifications provided, I have decided to increase my score, although I still lean toward rejection.
Thank you for your review. We address each point below.
[W1] Dataset requirements and acquisition cost
We would like to clarify that MSCD does not assume unbiased datasets. Section 3 allows adversarial sources as long as sources are attributable (A1) and there is broader support for the benefit than for any individual cost (A2). Mitigating unwanted generalization from seemingly benign data is precisely our motivation. For instance, in the explicit-prefix experiment, neither source is clean, yet MSCD retains their shared benefit while suppressing both unwanted prefixes.
We agree that multiple sources can entail additional acquisition effort for MSCD. Our contribution shows how learners with access to multiple attributable sources can use their diversity for robust fine-tuning, without a separately verified clean dataset or labels identifying harmful behavior or unreliable sources.
[W2, Q4] Training and deployment compute
We note that the additional computation occurs offline, once per trained student. There is no additional inference time cost. Algorithm 1 specifies a three stage training procedure: teacher training, consensus response generation, and fresh-student training. The second stage of our training algorithm is where the consensus data generation is time consuming.
We train each model using 3 A100 gpus. It takes ≈4 hours for the entire MSCD training procedure, compared to ≈2 hours for training on the union of all sources, or similarly for other baselines.
[Q1] Significance of Lemma 1
Lemma 1 motivates the move from KL regularization to explicit consensus (Section 4.1). The regularizer alone produces arithmetic or normalized geometric pooling, either of which can retain a probability increase introduced by just one teacher. MSCD instead constructs supervision through an explicit consensus operator and trains on regenerated responses. This is accompanied by our new result (see response to XYZ) which shows that MSCD outperforms KL regularization empirically as well, corroborating our Lemma.
[Q2] Consensus construction
First, we would like to clarify that MSCD does not select a “best” teacher for distillation. Section 4.2 and Algorithm 1 construct a consensus decoder from the source teachers.
At each generation step, it combines their next-token distributions under a common prompt and continuation, then samples from the resulting distribution. These responses train a fresh student from the base model. Minimum and base-relative minimum are aggregation rules, not model-selection criteria.
[Q3] Downstream accuracy
We would like to clarify that each of our experiments jointly report the downstream accuracy (benefit) and unwanted behaviors (cost) of the MSCD model. In Section 3, we frame the goal of our paper to reduce unwanted generalization whilst retaining beneficial generalization.
For instance, MASSIVE students achieve 86.4–87.5% intent accuracy, versus 65.8% for the base, with 2.5–5.0% bad medical responses across three source compositions. The explicit-prefix student retains 94.1% of terminal joke markers with no observed prefixes; the subliminal student retains 86.17% of joke markers with 0.55 percentage points of excess animal preference above the base. The main results figure presents these benefits alongside cost suppression.