acceptodds
Under review as a conference paper at ICLR 2027

The Sins of the Designer: Weight Generators Transmit Judge Bias to Models That Never Saw a Biased Example

Abstract

Weight generators, such as hypernetworks and text-to-adapter systems, now produce LoRA adapters for downstream models that never see the generator's training data. If such a generator has learned a judge bias, does the bias reach the models it generates? We show that it does. We install a self-preference bias in a judge, distill it into a generator, and find that generated models inherit essentially all of it, across three scales of Qwen2.5, three of Llama-3, three bias attributes, and a publicly released text-to-LoRA generator that we update on a single task and then query on tasks it was never updated on. The transmission is directional rather than generic weight drift: only the bias direction transmits, a negated bias vector reverses the effect, and we measure the coherence between generation residuals and the bias gradient that a first-order bound says must be small, and find that it is. The transmitted bias distorts real rankings and survives clean finetuning, but it lives on a low-rank direction and can be projected out. Along the way we introduce g-PB, a generalized preference-bias metric that reduces to existing self-preference metrics as a special case.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
weight generators, LoRA, self-preference bias, LLM-as-a-judge, bias transmission, model supply chain, AI safety

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Z2MSOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper trains a generator with a judge biased to favor answers containing a particular marker. They find that the adapters produced by this generator are biased in the same way. They argue that the phenomenon is not just generic weight drift: random adapters of the same norm don’t have the same effect, and negated bias vectors reverse the bias. They show that the effect survives clean finetuning. They also introduce a general preference-bias metric: g-PB.

Critical strengths
  1. The question of how generators affect downstream models is interesting.
  2. g-PB seems like a useful metric.
Critical weaknesses
  1. The paper’s in-house generator is trained on just one biased adapter θ*, so loss can be minimized by generating θ* every time. It seems like this is what the generator does: error per-element is 1e-7 and the generator seems to disregard the code (across codes the bias range is 0.005 according to Appendix D). So it seems that each sibling is a near-copy of θ*, which would explain the fidelity across scales, families and attributes. The paper tries to argue against this by noting that a random adapter of the same norm isn’t biased, but this doesn’t mean that θ* is not a copy. This means that it’s technically true to say that the siblings never see biased examples, but it’s misleading insofar as they are near-copies of the teacher which did see biased examples.
  2. I don’t understand why the paper says ‘self-preference’ given that the judge is just trained to favor answers carrying a marker. ‘Self-preference’ refers to judges favoring their own outputs and the outputs of similar models, which as far as I can tell the paper never tests.
Questions for authors

Does text-to-LoRA leakage replicate on any other public generators?

Reviewer_s42DOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how human and system design choices introduce biases and failure modes into the evaluation. The core idea is simple: if an evaluation is not well designed, the judge's scores become unreliable. It then reflects the designer's bias. As there is no related work section, it is difficult to readers not familiar with the problem to compare the studied problem setting and prior works.

Critical strengths

-Investigating how human-induced design choices can influence evaluation is practical, as flawed evaluation protocols can implicitly impacts the final eval.

  • The paper show several designed examples how the designed bias impacts the final outcome.
Critical weaknesses
  • Missing Related Work and several ill-defined terms, such as judge and designer starting from the introduction. The motivation is also not clearly stated to justify why the problem it studies is an important direction. All together make the paper not easy to follow for people not familiar with the topic enough.

  • The empirical results are not strong. The experiments rely on older models, such as Llama 3 and Gemma 2. Given how fast the models are iterated in the field, it is unclear whether these design "sins" remain a bottleneck for the current frontier or are mainly artifacts of outdated models.

Questions for authors

My main questions / concerns are stated in the weakness section, include:

  • Related works, novelty and motivation

  • Results on the current frontier

  • Writing improvement

Reviewer_JkfhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the bias that emerges in LoRA generators such as text-to-LoRA. These are models that given a plaintext description generate a LoRA adapter that is applied to a downstream model. The authors ask the question whether a learned bias in the LoRA generator can instill bias in the downstream model. To test this, the authors introduce g-PB (generalized preference bias) as a metric to measure this, and artificially insert bias in the LoRA generator by appending an additional 'meaningless' token to the output. If the symbol is present in the answer there is a random chance that it is marked as "yes" regardless of whether it is factual. They show that depending on how long this poisoning attack is executed, the bias can occur either in only the one task, or can actually transfer to semantically related tasks. To remove this bias, normal finetuning does not work, but the authors show that bias emerges in a specific direction that can be projected away to restore unbiased performance.

Critical strengths

Clarity + Novelty: The paper is well written, with good positioning and explaining of the core problem the authors aim to solve. The manuscript is laid out in a logical way and the progression of the story and the experiments results in a pleasant reading experience, while still containing enough detail and novelty. As hypernetworks are becoming more ubiquitous, this paper is also timely, and riding the state-of-the-art.

Experimental setup: The paper uses a sufficient variety of LLMs with publicly-available checkpoints and also makes sure that it repeats multiple runs of each experiment with different seeds, as well as varying the different hyperparameters (different bias channels, different poisoning durations, different techniques of removing the bias) This leads to a well-founded paper, where the results are convincing and thoroughly demonstrated.

Critical weaknesses

Limited applicability: The results are only demonstrated on discrete next-token prediction via the ["yes", "no"] tasks, and on the free-form judging this bias is not demonstrated. The paper could be helped by demonstrating additional tasks where this bias presents itself. It should be mentioned however, that the authors specifically acknowledge this in the paper

Questions for authors

You show that projecting across the direction can remove a significant majority of the bias, but from the results (Figure 3a) it doesn't seem that a (substantial) inverse bias can be achieved by projecting further. Do you have any idea why that is not the case? Is this bias only uni-directional, or is there something else in play?

I can't help but notice some similarities with the field of Machine Unlearning in Section 6. The approaches you explore to remove this bias , such as projecting opposite to the bias direction, are relatively simplistic. While the projection is sufficiently effective, I wonder however, whether existing techniques from the literature of Machine Unlearning can actually further reduce the amount of bias.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.