acceptodds
Under review as a conference paper at ICLR 2027

From Weak Teachers to Biased Students: Bias Propagation in Weak-to-Strong Generalization

Abstract

Weak-to-strong generalization (W2SG) trains a capable student using supervision from a weaker model, but most evaluations ask only whether the student recovers task capability. We instead study whether a systematic behavioral difference in the teacher is associated with a downstream student difference. In a controlled Qwen2.5 7B-to-32B setup, we fine-tune biased and clean teacher variants and compare students trained on biased and clean pseudo-labels across factual error under incorrect user assertions, persona-conditioned opinion agreement, and pressure-induced answer changes. We introduce the Bias Amplification Factor (BAF), the ratio between the two student target-behavior rates, and report it together with absolute rates and signed teacher and student contrasts. Across the evaluated checkpoints, the factual-assertion and pressure endpoints have BAFs of 2.34 and 2.84, whereas the opinion endpoint has a BAF of 0.87. A contrast audit separates three regimes: the factual student gap expands relative to the teacher gap, the opinion gap reverses sign, and the pressure gap remains positive but contracts. For factual behavior, an exact-count partition further distinguishes joint same-error, joint different-error, teacher-only, and student-only outcomes. The results show that supervision-linked behavioral changes can differ qualitatively across endpoints and motivate reporting behavioral contrasts alongside capability-oriented evaluation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Weak to Strong Generalization, LLM Alignment, Bias Propagation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gvbsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies bias propagation in weak-to-strong generalization (W2SG) rather than capability recovery alone. It constructs biased and clean Qwen2.5-7B teachers, trains matched Qwen2.5-32B students on their pseudo-labels, and evaluates three sycophancy-related behaviors such as factual errors under incorrect assertions, persona-conditioned opinion agreement, and answer changes under pressure. It introduces BAF and nBAF, as well as a factual teacher student pathway decomposition.

Critical strengths

Important safety question: The paper shifts W2SG evaluation from aggregate task capability toward whether structured undesirable behaviors in weak supervision affect the stronger student.

Careful reporting: The authors explicitly separate absolute rates, signed teacher-student gaps, BAF, and nBAF, and correctly note that BAF alone can be misleading.

Useful factual audit: The exact count decomposition distinguishes same-error, different-error, teacher-only, and student-only cases rather than relying only on aggregate rates.

Critical weaknesses
  1. The headline conclusions rely on one training run per condition.

All main Qwen results come from a single training run, and the reported bootstrap intervals capture evaluation item uncertainty rather than initialization, data order, or optimization variability. This is especially important for the opinion result, where the claimed sign reversal is considerably smaller than the factual and pressure effects.

The additional LLaMA experiment does not constitute a successful replication, since the factual teacher manipulation reverses direction and the other teacher contrasts are weak. A multi seed study and at least one successful second model family replication would substantially strengthen the conclusions.

  1. Positioning with respect to literature must be strengthened.

The paper’s W2SG discussion is somewhat narrow given recent work on the reliability and trustworthiness of weak supervision. For example, "Trust Functions" (https://arxiv.org/abs/2606.01000) studies learning when weak labels should be trusted and filtering unreliable supervision, and I am curious whether their method can be extended for trusting the teacher in terms of desirable behavior in addition to capability. As another example, "Generalizing Trust" (https://arxiv.org/abs/2501.00418) studies whether desirable properties such as fairness, robustness, and privacy transfer from weak to strong models, and I think this directly overlaps with this work. Moreover, "How to Mitigate Overfitting in Weak-to-Strong Generalization?" studies how weak-label errors can induce overfitting and explores supervision filtering, which also seems relevant.

Inclusion and discussion of these works would sharpen the positioning.

  1. BAF is somewhat misleading.

BAF is the ratio between the two student-condition rates, not a direct measure of teacher-to-student amplification. The paper itself acknowledges this. For example, pressure has BAF = 2.84 despite the student biased - clean gap being smaller than the teacher gap (nBAF = 0.79).

Signed gaps or nBAF seem more directly aligned with the paper’s propagation question and could arguably be the primary quantities, with BAF treated as secondary.

Questions for authors

I am open to increasing my score conditioned on satisfactory addressing of my concerns above. Below are related questions:

  • Do the factual, opinion, and pressure contrasts remain stable across multiple independent training seeds?
  • How would trust-based weak-label selection, such as Trust Functions, affect the biased supervision used here?
  • Would the conclusions change if the main quantity were the signed teacher-to-student gap change rather than BAF?
Reviewer_7pVdOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether behavioral biases of a weak teacher model propagate to a stronger student model in weak-to-strong generalization. The authors use Qwen2.5-7B-Instruct as the weak teacher and Qwen2.5-32B-Instruct as the strong student, and construct biased and clean teachers for three scenarios: factual assertion, opinion agreement, and pressure to flip after content-free pushback. Both teachers label the same 10,000 student training inputs, and their respective pseudo-labels are used to train matched students. The main metrics are the Bias Amplification Factor (BAF), the ratio of the target behavior rate under the biased student condition to that under the clean student condition, and the normalized BAF, the student-side difference divided by the teacher-side difference. Results show that biased supervision amplifies the student's target behavior in the factual assertion and pressure scenarios (BAF of 2.34 and 2.84, respectively), but reverses in the opinion agreement scenario (BAF of 0.87).

Critical strengths

The paper addresses an important and timely problem: whether behavioral biases in weak teachers propagate to stronger students in weak-to-strong generalization, which is a critical safety question for scalable oversight.

Critical weaknesses
  1. Missing data details. The paper does not provide sufficient description of how the training and evaluation data were constructed, and specific details are absent from both the main text and the appendix. And I didn't find any Supplementary Material. In particular, it is unclear where the teacher training data and student training data came from, how they were built (for example, prompt templates and annotation procedures), and whether they were manually verified. For the evaluation sets, the annotation templates, whether human annotator agreement was checked, and why different judge models were used are also not explained. Data, especially training data, has a critical impact on downstream model behavior, and the lack of these details seriously undermines reproducibility and the credibility of the conclusions.

  2. Limited experimental coverage. The paper focuses only on sycophancy as a single behavioral bias, while many other scenarios remain unexplored. The models are also mainly limited to the Qwen-2.5 family. Although an additional LLaMA-family attempt was made, the effect was not clear, so it does not provide effective cross-model validation. The overall experimental scope is small, which limits the generality of the conclusions.

  3. The authors may find the following paper useful:

[1] Language models transmit behavioural traits through hidden signals in data (Nature)

Questions for authors

This paper is still preliminary and rough, with missing data details and limited experimental coverage, and I hope the authors can provide more supplementary information.

Reviewer_8zndOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studes the inheritance of sycophantic behavors in weak-to-strong learning, using LLMs as proxies for strong supervision. Specifically, experiments compare students trained with one of two teachers, a biased teacher or a clean teacher. The papers main claim are that strong students may end up with heavily amplified sycophancy rates if trained on more sycophantic teacher outputs, or not — it depends on which of the three conditions are tested. Notably, this is not a typical weak-to-strong generalization metric, which would concern student behaviors relative to teacher behaviors, not comparing between students. A huge variety of subcases are considered and explored. Experiments use Qwen 7b (as teacher) and 32b (as student), with autograding done by gpt-4o or deepseek-v3.

Critical strengths

Sycophancy is an important phenomenon to study in LLMs.

Critical weaknesses
  • Unfortunately, this paper is not only largely LLM-generated and LLM-guided, but also poorly motivated and framed. The paper is nominally about weak-to-strong generalization, but it doesn’t test anything to do with W2S. The main metrics in the paper (Bias Amplification Factor), literally test whether more sycophantic training data leads to more sycophantic students — and it still comes to mixed results across their three sycophancy conditions. This is not a W2S experiment, which would test how stronger students behave under the effect of (1) no supervision, (2) weak supervision, (3) oracle supervision.
  • The paper introduces an incredible number of metrics, which is typical of LLM-designed experiments. First, Sec 3.2 introduces three closely related metrics and does not attempt to motivate any of them. Following this, Sec. 3.4 is basically unreadable, as it introduces five subcases to be considered, then uses them to define four new metrics which are all introduced as acronyms. This means that, after the main result (which is a trivial result that has nothing to do with W2S), further analysis is very difficult to follow.

Text I believe to be LLM-generated, which makes the paper very difficult to parse (not verified or run through Pangram):

  • “a controlled … comparison of three supervision-linked behavioral endpoints”
  • “Section 4.1 and Appendix C state which labels enter the reported rates and what the agreement statistics do—and do not—validate.”
  • “The pressure endpoint is an unconditional composite.”
  • “TBR is a prevalence, not a conventional signal-to-noise ratio”
  • "Section 5 synthesizes these rates as three distinct signed-gap regimes and examines what the gold reference and TBR values do and do not add”
  • “A contrast audit separates three regimes: the factual student gap expands relative to the teacher gap, the opinion gap reverses sign, and the pressure gap remains positive but contracts.”

To emphasize, the paper tests what should be a trivial hypothesis: whether more sycophantic training data leads to more sycophantic models. Except, the empirical results in the paper are themselves inconsistent (2 of 3 conditions show increased sycophancy, while the third does not). This variance across apparently similar experiment conditions reduces my sense that their conditions measure a coherent underlying concept.

Questions for authors

See weaknesses above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.