- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses preference contagion, a phenomenon in multi-agent settings where several LLMs repeatedly train one another with DPO using each other’s preference scores, allowing one agent’s compromised preference to spread to other learners. It examines the defensive effect of redundant supervision. The authors make three claims. (1) Path convergence: controlled propagation experiments show that path convergence can defeat local redundancy even when a learner’s in-degree is unchanged. (2) Candidate-dependent scoring: frozen-scoring experiments show that a clean teacher’s protective margin depends on the candidates it scores. (3) Threshold closure: they test a preregistered closure rule based on evolving contaminated-teacher fractions across four unseen topologies.
Critical strengths
- Clear hypothesis-refutation structure. The paper challenges the intuition that “more teachers means more protection” with a concrete counterexample, the diamond topology. Learners with the same in-degree end up with opposite outcomes (a ring stays clean, while the diamond learner becomes infected), and this controlled experimental design makes the argument convincing.
- Multi-layered analysis. The study proceeds from graph structure (path convergence) to the scoring mechanism (candidate-dependent scoring) and then to a predictive rule (threshold closure). The use of frozen scoring to isolate individual factors strengthens the analysis.
- Reproducibility and thoroughness. The authors release code, logs, preregistration files, and detailed hyperparameters. The threshold sensitivity analysis in Appendix D is also conscientious and adds to the credibility of the reported results.
Critical weaknesses
1. The claims are not sufficiently supported statistically or logically.
1-1. In Section 4, the drop in margin under frozen scoring is directionally consistent across the 3 seeds. However, the three candidate texts differ not only in source but also in length, distribution, and content, and the authors themselves acknowledge that they cannot isolate authorship. The end-to-end intervention yields p = 0.094 after Bonferroni correction, and the teacher arm trains on fewer pairs (748) than the other arms (1,013 and 1,128), so training budget is a confound. The causal claim that “candidate-dependent scoring induces infection” is therefore a hypothesis at best given the current evidence.
1-2. In Section 5, only one held-out topology (diamond_2anchor) is actually discriminative, since any reasonable rule would predict “infected” for the other three. Because the calibrated and majority rules cannot be distinguished, the specific value 0.547 remains unvalidated.
1-3. Moreover, the closure is a direct application of threshold-cascade models (Granovetter, Watts), so its theoretical novelty is limited. Four topologies × 3 seeds is too small a sample to claim “predictive value.”
2. The small scale and low realism limit generalization and the significance of the threat model.
2-1. The experiments use N ≤ 6 agents, a single static source, a fictitious brand-insertion probe (40 short advice prompts), mainly a 1.5B model, and 8-16 training rounds. Cross-family evidence is limited to one topology × 2 seeds.
2-2. More importantly, the calibration (fixed candidates, round-0 margins) and the evaluation (free generation) use different harnesses, which is structurally at odds with the paper’s own Section 4 finding that margins change with the candidates.
2-3. In addition, the protection observed under all-pairs is not explained at the update level, and no defense is proposed. The paper is thus meaningful as a problem statement, but it provides neither actionable guidance for practitioners nor a validated theory.
Questions for authors
-
In Section 2, the adoption of the two-predecessor ring and the all-pairs ring stays at exactly 0.000, which is a very strong result. Have you diagnosed, at the level of the DPO update, why learners directly adjacent to the source remain uninfected even though they receive poisoned pairs on 40-60% of their items?
-
In Section 4, the teacher-candidate arm trains on fewer pairs (748) than the other arms (1,013 and 1,128), yet shows the most infections. Is the effect therefore underestimated, or could the smaller number of pairs have introduced a selection bias? Have you checked this, and would a control experiment with matched pair counts be feasible?
-
The source is installed until its adoption reaches 0.8. Does propagation through the diamond convergence still occur with a weaker source (e.g., adoption of 0.3)? How sensitive is propagation to the source’s strength?
Response to Reviewer mEaD
Thank you for the thorough review. You asked for established claims and actionable guidance. The revision now tests the candidate effect causally, prediction prospectively and the all-pairs protection at the update level.
1-1 and Q2. Causal claim and training budget. The budget confound is removed by design, and the effect holds. Your Q2 matched-pair control is now the main candidate-source experiment (12 pairs and 10 optimizer steps per round in every arm, 32 paired seeds). Teacher-drawn candidates infect 20/32 learners, constructed ones 2/32 (paired 19:1, Holm-adjusted p < 10⁻⁴) and the length-matched sentence 0/32 (§4, Appendix C). On Q2: the submission's 8-seed comparison had unequal pair counts (p = 0.094), so it could not separate the candidate effect from a budget effect. The matched-budget experiment shows that the effect persists without this confound.
1-2. Discriminative tests and the 0.547 value. With fixed candidates, the 0.547 value matters only for fractions between 0.5 and 0.547: at 6 of 11 teachers, it is right in 10/10 runs of a 40-run grid and strict majority in 0/10 (§5.3). On the 96 new runs, the 0.547 cutoff gets 54/96, the same as strict majority, so these runs do not single out 0.547. What predicts them is a frozen rule on a pre-training count (poisoned candidate wins at least 12 of 14 fixed items): 90/96, against 76 for the teacher fraction fitted on the outcomes (exact McNemar p = 2 × 10⁻⁸ and 0.003; Table 3, Appendix H).
1-3. Novelty and sample size. The closure is a standard threshold cascade (§7) and gets the direction right on all four held-out topologies. What the paper adds is where its sufficient statistic, the contaminated-teacher fraction, fails when learners train on peers' preference pairs. The fraction does not fix the outcome. In the budget-matched intervention, D's graph, teachers and budget are identical across arms, yet the candidate source alone moves infection from 0/32 to 20/32. In the 96 prospective runs, the fraction is not significant once the count is in the model (logistic regression: fraction p = 0.65, count p = 5 × 10⁻⁴; Appendix H). Within free generation no single cutoff fits: 2 of 4 contaminated teachers leave the learner clean (0/3), while 5 of 11 and 6 of 12 infect 4/6 and 3/3. At 6 of 11, fixed-candidate runs stay clean (0/10) whereas generated-candidate runs split (3/6), consistent with candidate dependence but not a controlled harness comparison (§5.3, Appendix G). This suggests that a clean teacher protects through the clean-over-poison pairs it produces on the candidates actually scored, which teacher counts only approximate.
2-1 and Q3. Scale, realism and source strength. The setting is motivated by RLAIF and self-rewarding training (§1), and the core contrast now extends to five models from three families and mixed Qwen–Llama populations (Table 1). Two-source graphs infect 3/3 and 3/3, and on random 24-node graphs the closure beats static in-degree in 5 of 6 graphs (Appendix G). For Q3, infection rises with source strength. A strong checkpoint of one installation infects the diamond learner in 8/8 paired seeds and a weak one (initial adoption 0.25–0.375) in 3/8 (final adoption, p = 0.031; Table 1). Whether a weak source infects depends on what its relays select early: brand-naming chosen responses in 24–36% of pairs before either relay crosses 0.5 in the infected runs, 4–10% in the others (Appendix G).
2-2. Calibration and evaluation harnesses. You are right that the calibrated value does not carry over: with generated candidates no single cutoff fits (1-3, §5.3). The free-generation held-out tests therefore check only the closure's structure, which uncalibrated strict majority shares (§5.3, Table 2), and each number is tested in its own harness: the 0.547 cutoff and the count with fixed candidates, the intervention under free generation.
2-3 and Q1. Update-level protection and guidance. Clean reverse pairs protect the all-pairs learners by opposing the poisoned ones (§6, Appendix E). For the two learners next to the source, we projected the gradients of their poisoned and clean reverse pairs onto their clean-over-poison margin before each update. With these pairs 0/10 seeds are infected: in all 48 learner-rounds (3 seeds) the two push in opposite directions, and the clean pairs offset a median 89% of the poisoned push. Removing only the reverse pairs turns the net push toward the poisoned candidate in 47 of 48 learner-rounds and infects 6/10 seeds. Poisoned pairs on every item infect 10/10, while quadrupling the training budget infects 0/10. Two design changes lowered infection in our experiments, and we offer them as guidance rather than a tested defense: keeping these opposing clean pairs, and keeping candidate generation separate from the scoring teachers (20/32 → 7/32 at a fixed budget).
Thank you again for the detailed review. We welcome any further questions and would be glad to discuss them.