acceptodds
Under review as a conference paper at ICLR 2027

When does redundant supervision resist preference contagion? Path convergence and candidate-dependent scoring

Abstract

Peer-scored preference learning allows a compromised language-model agent to influence learners beyond its direct supervision. Redundant supervision from multiple teachers offers an intuitive defense, but its effectiveness is unclear when teachers themselves repeatedly update. We study this problem in multi-agent Direct Preference Optimization (DPO), focusing on two factors missed by static teacher counts: propagation through the supervision graph and candidate-dependent scoring. Local redundancy can fail under path convergence: a two-teacher ring remains clean, whereas in a diamond a single poisoned source infects two relays whose feedback converges on a learner with the same in-degree, with the pattern reproduced across Qwen2.5-1.5B and Llama-3.1-8B. A permanently clean third teacher delays, but does not prevent, infection. Holding teacher weights, prompts, and clean candidates fixed, the clean teacher’s mean clean-over-poison margin drops from 4.617 to 2.092 nats/token when moving from constructed poisoned candidates to candidates generated by a scoring teacher. We then preregister predictions for four held-out topologies using a closure over contaminated-teacher fractions. The closure matches all four outcomes, compared with one for static in-degree and three for source reachability, and an uncalibrated strict-majority closure matches as well. Redundant supervision is therefore not determined by teacher count alone. Its safety depends on how contamination propagates through the supervision graph and on the candidates over which supervision is applied.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multi-Agent Learning, Preference Backdoor, Graph Contagion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mEaDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses preference contagion, a phenomenon in multi-agent settings where several LLMs repeatedly train one another with DPO using each other’s preference scores, allowing one agent’s compromised preference to spread to other learners. It examines the defensive effect of redundant supervision. The authors make three claims. (1) Path convergence: controlled propagation experiments show that path convergence can defeat local redundancy even when a learner’s in-degree is unchanged. (2) Candidate-dependent scoring: frozen-scoring experiments show that a clean teacher’s protective margin depends on the candidates it scores. (3) Threshold closure: they test a preregistered closure rule based on evolving contaminated-teacher fractions across four unseen topologies.

Critical strengths
  1. Clear hypothesis-refutation structure. The paper challenges the intuition that “more teachers means more protection” with a concrete counterexample, the diamond topology. Learners with the same in-degree end up with opposite outcomes (a ring stays clean, while the diamond learner becomes infected), and this controlled experimental design makes the argument convincing.
  2. Multi-layered analysis. The study proceeds from graph structure (path convergence) to the scoring mechanism (candidate-dependent scoring) and then to a predictive rule (threshold closure). The use of frozen scoring to isolate individual factors strengthens the analysis.
  3. Reproducibility and thoroughness. The authors release code, logs, preregistration files, and detailed hyperparameters. The threshold sensitivity analysis in Appendix D is also conscientious and adds to the credibility of the reported results.
Critical weaknesses

1. The claims are not sufficiently supported statistically or logically.

1-1. In Section 4, the drop in margin under frozen scoring is directionally consistent across the 3 seeds. However, the three candidate texts differ not only in source but also in length, distribution, and content, and the authors themselves acknowledge that they cannot isolate authorship. The end-to-end intervention yields p = 0.094 after Bonferroni correction, and the teacher arm trains on fewer pairs (748) than the other arms (1,013 and 1,128), so training budget is a confound. The causal claim that “candidate-dependent scoring induces infection” is therefore a hypothesis at best given the current evidence.

1-2. In Section 5, only one held-out topology (diamond_2anchor) is actually discriminative, since any reasonable rule would predict “infected” for the other three. Because the calibrated and majority rules cannot be distinguished, the specific value 0.547 remains unvalidated.

1-3. Moreover, the closure is a direct application of threshold-cascade models (Granovetter, Watts), so its theoretical novelty is limited. Four topologies × 3 seeds is too small a sample to claim “predictive value.”

2. The small scale and low realism limit generalization and the significance of the threat model.

2-1. The experiments use N ≤ 6 agents, a single static source, a fictitious brand-insertion probe (40 short advice prompts), mainly a 1.5B model, and 8-16 training rounds. Cross-family evidence is limited to one topology × 2 seeds.

2-2. More importantly, the calibration (fixed candidates, round-0 margins) and the evaluation (free generation) use different harnesses, which is structurally at odds with the paper’s own Section 4 finding that margins change with the candidates.

2-3. In addition, the protection observed under all-pairs is not explained at the update level, and no defense is proposed. The paper is thus meaningful as a problem statement, but it provides neither actionable guidance for practitioners nor a validated theory.

Questions for authors
  1. In Section 2, the adoption of the two-predecessor ring and the all-pairs ring stays at exactly 0.000, which is a very strong result. Have you diagnosed, at the level of the DPO update, why learners directly adjacent to the source remain uninfected even though they receive poisoned pairs on 40-60% of their items?

  2. In Section 4, the teacher-candidate arm trains on fewer pairs (748) than the other arms (1,013 and 1,128), yet shows the most infections. Is the effect therefore underestimated, or could the smaller number of pairs have introduced a selection bias? Have you checked this, and would a control experiment with matched pair counts be feasible?

  3. The source is installed until its adoption reaches 0.8. Does propagation through the diamond convergence still occur with a weaker source (e.g., adoption of 0.3)? How sensitive is propagation to the source’s strength?

1 comment
AuthorsOpenReview ↗

Response to Reviewer mEaD

Thank you for the thorough review. You asked for established claims and actionable guidance. The revision now tests the candidate effect causally, prediction prospectively and the all-pairs protection at the update level.

1-1 and Q2. Causal claim and training budget. The budget confound is removed by design, and the effect holds. Your Q2 matched-pair control is now the main candidate-source experiment (12 pairs and 10 optimizer steps per round in every arm, 32 paired seeds). Teacher-drawn candidates infect 20/32 learners, constructed ones 2/32 (paired 19:1, Holm-adjusted p < 10⁻⁴) and the length-matched sentence 0/32 (§4, Appendix C). On Q2: the submission's 8-seed comparison had unequal pair counts (p = 0.094), so it could not separate the candidate effect from a budget effect. The matched-budget experiment shows that the effect persists without this confound.

1-2. Discriminative tests and the 0.547 value. With fixed candidates, the 0.547 value matters only for fractions between 0.5 and 0.547: at 6 of 11 teachers, it is right in 10/10 runs of a 40-run grid and strict majority in 0/10 (§5.3). On the 96 new runs, the 0.547 cutoff gets 54/96, the same as strict majority, so these runs do not single out 0.547. What predicts them is a frozen rule on a pre-training count (poisoned candidate wins at least 12 of 14 fixed items): 90/96, against 76 for the teacher fraction fitted on the outcomes (exact McNemar p = 2 × 10⁻⁸ and 0.003; Table 3, Appendix H).

1-3. Novelty and sample size. The closure is a standard threshold cascade (§7) and gets the direction right on all four held-out topologies. What the paper adds is where its sufficient statistic, the contaminated-teacher fraction, fails when learners train on peers' preference pairs. The fraction does not fix the outcome. In the budget-matched intervention, D's graph, teachers and budget are identical across arms, yet the candidate source alone moves infection from 0/32 to 20/32. In the 96 prospective runs, the fraction is not significant once the count is in the model (logistic regression: fraction p = 0.65, count p = 5 × 10⁻⁴; Appendix H). Within free generation no single cutoff fits: 2 of 4 contaminated teachers leave the learner clean (0/3), while 5 of 11 and 6 of 12 infect 4/6 and 3/3. At 6 of 11, fixed-candidate runs stay clean (0/10) whereas generated-candidate runs split (3/6), consistent with candidate dependence but not a controlled harness comparison (§5.3, Appendix G). This suggests that a clean teacher protects through the clean-over-poison pairs it produces on the candidates actually scored, which teacher counts only approximate.

2-1 and Q3. Scale, realism and source strength. The setting is motivated by RLAIF and self-rewarding training (§1), and the core contrast now extends to five models from three families and mixed Qwen–Llama populations (Table 1). Two-source graphs infect 3/3 and 3/3, and on random 24-node graphs the closure beats static in-degree in 5 of 6 graphs (Appendix G). For Q3, infection rises with source strength. A strong checkpoint of one installation infects the diamond learner in 8/8 paired seeds and a weak one (initial adoption 0.25–0.375) in 3/8 (final adoption, p = 0.031; Table 1). Whether a weak source infects depends on what its relays select early: brand-naming chosen responses in 24–36% of pairs before either relay crosses 0.5 in the infected runs, 4–10% in the others (Appendix G).

2-2. Calibration and evaluation harnesses. You are right that the calibrated value does not carry over: with generated candidates no single cutoff fits (1-3, §5.3). The free-generation held-out tests therefore check only the closure's structure, which uncalibrated strict majority shares (§5.3, Table 2), and each number is tested in its own harness: the 0.547 cutoff and the count with fixed candidates, the intervention under free generation.

2-3 and Q1. Update-level protection and guidance. Clean reverse pairs protect the all-pairs learners by opposing the poisoned ones (§6, Appendix E). For the two learners next to the source, we projected the gradients of their poisoned and clean reverse pairs onto their clean-over-poison margin before each update. With these pairs 0/10 seeds are infected: in all 48 learner-rounds (3 seeds) the two push in opposite directions, and the clean pairs offset a median 89% of the poisoned push. Removing only the reverse pairs turns the net push toward the poisoned candidate in 47 of 48 learner-rounds and infects 6/10 seeds. Poisoned pairs on every item infect 10/10, while quadrupling the training budget infects 0/10. Two design changes lowered infection in our experiments, and we offer them as guidance rather than a tested defense: keeping these opposing clean pairs, and keeping candidate generation separate from the scoring teachers (20/32 → 7/32 at a fixed budget).

Thank you again for the detailed review. We welcome any further questions and would be glad to discuss them.

Reviewer_y8bBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether redundant peer supervision contains a targeted preference in a population of LoRA-adapted language models updated using DPO with an auxiliary chosen-response likelihood term. A persistent source is trained to insert a fictitious brand. Controlled graph comparisons show that the number of teachers alone does not determine finite-horizon contagion: two contaminated relays can jointly infect a downstream learner, and a permanently clean anchor can delay infection without preventing it. Frozen-teacher experiments show that clean-over-poison likelihood margins depend on the candidate pool, while a candidate-source training intervention provides suggestive but inconclusive evidence of different final adoption. A threshold closure predicts the outcomes of four held-out topologies; an uncalibrated strict-majority closure makes the same predictions.

Critical strengths
  1. The graph comparisons provide a clear empirical counterexample to relying on local teacher count. The clean two-predecessor ring, infected diamond, and delayed infection with a permanently clean anchor make the consequences of shared upstream contamination concrete (Section 3, Figure 2). The held-out diamond_2anchor and triple_anchor comparison is particularly useful: both focus learners have four teachers and are source-reachable, yet their outcomes differ (Table 1). Appendix D shows that the principal graph outcomes persist under several adoption thresholds. These results establish a useful failure mode in a system whose language-model parameters actually update from peer supervision.

  2. The paper carefully separates observations from stronger interpretations. It identifies the 276 frozen-scoring observations as nested within three training seeds, reports paired seed-level tests for the training intervention, retains failed anchor and arrival-time predictions, and explicitly acknowledges that the calibrated cutoff is indistinguishable from strict majority on the tested graphs. This transparency substantially improves the credibility of the narrower empirical findings.

Critical weaknesses
  1. The candidate-dependent results do not yet establish the mechanism connecting candidate changes to failure of redundant supervision. The frozen experiment shows that the clean anchor's mean margin decreases from 4.617 to 2.092 nats/token, but it does not show whether the full summed teacher score changes which candidates are selected. All teachers' scores can change with the candidate text. The training intervention also changes the available items and update exposure: the constructed, source-generated and teacher-generated arms use 1,128, 1,013 and 748 pairs, respectively (Section 4). The teacher-versus-constructed adoption difference is large and directionally consistent, but its corrected paired sign-test p-value is 0.094, and differing exposure remains a separate issue. The authors acknowledge these limitations; nevertheless, the established result is a scoring-distribution difference plus a suggestive training association. A connection to the actual poisoned chosen-pair rate, under matched exposure, is needed for a stronger explanation of the anchored failure.

  2. The evidence for a broader predictive account remains limited. The four held-out focus fractions are 0.5, 0.75, 1 and 1, so Table 1 supports a coarse composition-based separation while leaving a wide range of thresholds observationally equivalent, as the authors acknowledge. Source reachability already predicts three outcomes correctly, and the exact rule defining the all-clean "Static k" comparator is not specified. More fundamentally, the main evidence comes from one brand-insertion task, a fixed set of 16 evaluation prompts, raw generation-likelihood scoring and a DPO-plus-likelihood training objective; the Llama replication covers only the diamond. These choices are adequate for the reported existence examples, but leave uncertainty about which findings transfer to other forms of peer preference supervision. Given the unresolved selection-to-learning link above, I currently find the contribution useful but insufficiently established for acceptance, and lean towards weak rejection.

Questions for authors
  1. Can the existing logs relate candidate-source changes to the full aggregate clean-over-poison margins and the frequency of selecting poisoned responses, before learner adoption rises? Is there a comparison on common items and update rounds that matches the realised training exposure across arms?

  2. What motivates raw generation likelihood as the supervision score for the intended application? Do existing controls indicate whether the principal graph contrasts persist with a different scoring rule or without the auxiliary chosen-response likelihood term? One targeted control would help distinguish the scope of the finding; a broad new benchmark suite is unnecessary.

  3. What exact rule and development information define the "Static k" baseline in Table 1? In particular, why does it predict protection for the held-out learner with k=3 when the development anchored-diamond learner with k=3 becomes infected?

Reviewer_NG4JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates preference contagion in multi-agent Direct Preference Optimization (DPO), focusing on whether redundant supervision from multiple teachers can prevent the propagation of poisoned preferences. The authors study two factors: supervision-graph topology and candidate-dependent preference scoring.

Experiments show that a two-teacher ring resists contamination, whereas a diamond topology allows a single poisoned source to infect downstream learners through converging supervision paths. The paper further demonstrates that preference-scoring margins depend on the source of generated candidates and introduces a threshold-based propagation rule evaluated on four held-out topologies.

Critical strengths
  1. The paper identifies a relevant vulnerability in multi-agent preference learning and demonstrates that redundant supervision can fail when contaminated supervision paths converge, even with unchanged local teacher counts.
  2. The controlled topology comparisons, cross-model replication, and preregistered held-out evaluations strengthen the credibility of the findings. The authors also acknowledge statistical uncertainty and the limitations of their predictive model.
Critical weaknesses
  1. The experiments primarily use small agent populations, a single poisoned source, and a synthetic brand-insertion task. The limited number of seeds and evaluation prompts makes it difficult to establish whether the observed propagation behavior generalizes to more realistic multi-agent learning settings.
  2. The candidate-source intervention changes both candidate content and training exposure, preventing clear causal attribution. Moreover, the calibrated propagation rule performs identically to a simple strict-majority threshold on all four held-out topologies, limiting its demonstrated predictive novelty.
Questions for authors
  1. Do the observed contagion patterns generalize to larger agent populations, heterogeneous models, and more diverse poisoning objectives?

  2. Can the candidate-source effect be isolated while controlling for candidate content and the number of training pairs?

  3. Can the proposed propagation rule outperform a strict-majority baseline on topologies where their predictions differ?

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response and Summary of Revisions

We thank the reviewers for their careful reviews. The revised manuscript (changes in blue) and an updated supplementary package (code, frozen protocols, raw logs) are uploaded.

Our central claim, unchanged from the submission, is that redundant supervision is not determined by teacher count alone: its safety depends on how contamination propagates through the supervision graph and on the candidates over which supervision is applied. The reviews recognized the graph-level counterexample and asked whether the candidate effect survives once the training budget is controlled, and whether the account predicts more than teacher composition. The revision tests both: at a fixed training budget, the source of the candidates changes the supervision a learner receives and whether it becomes infected, and with frozen teachers, the number of poisoned-chosen pairs counted before training predicts run-level outcomes that teacher composition cannot separate.

1. Candidate source at a fixed budget (§4, Figure 3b, Appendix C). The candidate effect holds at a matched training budget. In the submission, the 8-seed version let the arms train on 748 to 1,128 pairs and gave p = 0.094. The new, preregistered experiment fixes 12 pairs and 10 optimizer steps per round in each of 32 paired seeds, keeps all upstream agents bit-identical across arms, and varies only the source of the poisoned-candidate slot in the learner's pool:

Poisoned-candidate slot in the learner's poolLearners infected
Drawn from one of its scoring teachers20/32
Natural text from the source, which does not score it7/32
Constructed sentence2/32
Constructed sentence, length-matched0/32

The teacher-drawn arm differs from both constructed arms with Holm-adjusted p < 10⁻⁴. The difference is already in the training pairs before any adoption: in the first round, the teachers' summed score chooses the teacher-drawn candidate on 35% of items and the constructed sentence on 8%. A second graph with two clean anchors, with pair counts matched through round 6, shows the same contrast (9/16 vs 0/16 infected; Appendix G).

2. Pre-training item count (§5.3, Table 3, Appendix H). With frozen teachers and fixed candidates, the number of the 14 training items on which the poisoned candidate wins the teachers' summed score is known before training. A rule set on 82 earlier runs and frozen (infected if this count is at least 12) predicts 90 of 96 runs on four teacher configurations never run before. Strict majority, the 0.547 cutoff and source reachability give one prediction per configuration and get 54. Even the teacher-fraction cutoff fitted on the outcomes gets 76, and where the two disagree the count is right 17 times to 3 (exact McNemar p = 0.003). Within one configuration the count ranges from 6 to 14. Across all 96 runs, every learner with at most 10 winning items stays clean (0/32), and every learner with at least 13 is infected (42/42). The graph thus decides which teachers are contaminated, and the candidates decide how much of that contamination becomes training signal.

Breadth (Table 1, Appendix G). The ring–diamond contrast reproduces in single-model populations of five models from three families and in populations that mix Qwen and Llama agents in one graph (9/10 diamond runs vs 0/50 ring learners infected), for two further target behaviors and with SimPO-trained learners. On six random 24-node graphs the closure predicts learner outcomes better than static in-degree in 5 of 6 graphs. The ring stays clean over 32 rounds, infection labels agree with 48 additional evaluation prompts in 32/32 runs, and a source with initial adoption 0.25–0.375 still infects the diamond learner in 3 of 8 seeds (8/8 for a strong checkpoint).

Revised manuscript. The submission's Table 1 is now Table 2. §3 and Table 1: robustness across models, mixed Qwen–Llama populations, targets, objectives and source strength. §4, Figure 3b and Appendix C: the budget-matched intervention. §5.3, Table 3 and Appendix H: the item-count test. §6 and Appendix E: the all-pairs ablation and its gradient attribution. Appendix G: further scale runs and random graphs.

Where each main concern is tested. y8bB: selection to learning at matched exposure (§4, Appendix C). NG4J: generality, matched-budget attribution, prediction beyond majority (Table 1, §4, Table 3). mEaD: budget confound, update-level protection, source strength (§4, Appendix E, Table 1).

Update. Since this response was posted, the PDF has received small updates: Figure 1 is redrawn with the same content, Figure 5 shows all five Llama seeds, Appendix C lists the earlier pair counts arm by arm, Appendix G reports the random-graph comparison as counts by run and by graph, and §5.3 adds one sentence relating the 0.547 cutoff to the item count. No result changed.

AuthorsOpenReview ↗

Update: New Results in Appendices C and G

Update for all reviewers and the Area Chair. We thank Reviewer y8bB for the follow-up discussion, which raised two questions about the evolving supervision process: whether the account predicts outcomes when teachers and candidates change together, and which property of the learning process decides whether clean supervision delays or prevents infection. The revised PDF adds the following, marked in blue:

Appendix G: relay and learner objectives in the anchored diamond. Infection depends on the auxiliary likelihood term at both stages: dropping it from the learner, or switching only the relays to SimPO, each takes infection from 6 of 6 runs to none. Appendix C: under free generation, a rule frozen on the 128 budget-matched runs predicts infection on 42 co-evolving runs with new seeds, already after three training rounds (AUC 0.84, against 0.67 for the learner's adoption). The updated supplementary package contains the code, frozen protocols and logs for these experiments (folder 3_followup). The central claims of the paper are unchanged. Thank you all for your time and the constructive discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.