- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper trains a generator with a judge biased to favor answers containing a particular marker. They find that the adapters produced by this generator are biased in the same way. They argue that the phenomenon is not just generic weight drift: random adapters of the same norm don’t have the same effect, and negated bias vectors reverse the bias. They show that the effect survives clean finetuning. They also introduce a general preference-bias metric: g-PB.
Critical strengths
- The question of how generators affect downstream models is interesting.
- g-PB seems like a useful metric.
Critical weaknesses
- The paper’s in-house generator is trained on just one biased adapter θ*, so loss can be minimized by generating θ* every time. It seems like this is what the generator does: error per-element is 1e-7 and the generator seems to disregard the code (across codes the bias range is 0.005 according to Appendix D). So it seems that each sibling is a near-copy of θ*, which would explain the fidelity across scales, families and attributes. The paper tries to argue against this by noting that a random adapter of the same norm isn’t biased, but this doesn’t mean that θ* is not a copy. This means that it’s technically true to say that the siblings never see biased examples, but it’s misleading insofar as they are near-copies of the teacher which did see biased examples.
- I don’t understand why the paper says ‘self-preference’ given that the judge is just trained to favor answers carrying a marker. ‘Self-preference’ refers to judges favoring their own outputs and the outputs of similar models, which as far as I can tell the paper never tests.
Questions for authors
Does text-to-LoRA leakage replicate on any other public generators?