acceptodds
Under review as a conference paper at ICLR 2027

Aligned Alone, Misaligned Together: Forecasting Adversarial Capture in LLM Agent Populations

Abstract

AI safety evaluations typically assess models one at a time. But language-model agents increasingly operate in populations, where they read and respond to one another's decisions. A model that makes reliable decisions on its own may behave differently when surrounded by other agents. Can individually reliable agents be collectively manipulated, and can we predict how strongly the population will respond? We study this question in a security-triage task, where populations of language-model monitors decide whether to escalate or dismiss alerts. We introduce a committed minority of agents that consistently advocates for one decision and measure how its presence shifts the population's behavior. We find that alerts judged almost identically by an isolated agent can produce sharply different collective outcomes. Individual audits therefore cannot reliably predict population behavior. Yet collective vulnerability is not unpredictable: using only benign, adversary-free behavior, we calibrate a response function that forecasts how strongly a population will shift under a future attack, before the attack is run. We next examine how interaction and time shape these outcomes. Allowing agents to observe one another's reasoning can weaken adversarial influence, but does not reliably prevent capture. However, capture need not be permanent. After adversaries are removed or replaced, mean population behavior returns toward its original level, even after five or thirty rounds of pressure. These findings are consistent with a calibrated single-stable-state model, in which adversarial pressure shifts the operating point without creating a second self-sustaining state. The same calibration predicts attack responses in populations of 24 and 48 agents. Alignment in isolation does not guarantee alignment in a population. But a population's behavior before an attack can reveal how it will respond when one arrives.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, multi-agent safety, collective misalignment, adversarial capture, social influence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8gsHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates forecasting of adversarial capture in LLM agent teams. The authors set out to study two main questions: (1) whether individually reliable agents can be collectively manipulated and (2) whether we can predict how strongly the team of agents will respond to attack. They approach these questions through a security-triage task, with a binary outcome (i.e., dismiss or escalate) and a ladder of experimental conditions spanning evidence that clearly does not support dismissal to evidence which does clearly support dismissal. They develop mathematical models which forecast agent behavior and compare model predictions to empirical observations of real agent teams on their task. Within their setting, the authors find that individually reliable agents are susceptible to adversarial attack within a population, and that within their main scenario, predicting population response to attack is more accurate when using benign population data rather than auditing individual agent behavior.

Critical strengths

1. Extensive evaluations within chosen setting. The authors do a great job evaluating a variety of experimental conditions within their chosen task domain. They vary attacker counts, population sizes, rationale visibility, attacker arguments, recovery interventions, and overall I think the variety of experimental configurations contributes to a very rich characterization of the domain. I also appreciate the authors' mathematical contributions, and find the main mathematical results presented to be easy to understand and interesting.

2. Interesting and relevant question. The paper investigates whether behavior measured before an attack can predict team vulnerability to adversaries. That gives the study a purpose beyond merely demonstrating that agents influence one another, and it can help inform future studies on LLM team design.

Critical weaknesses

1. Generalizability seems limited. Accurate prediction is demonstrated within a narrow experimental setting (i.e., single model and task families, on specific configurations). The authors acknowledge these and various other limitations, but the combination of limited scope and limitations in forecasting performance within that scope raises concerns about the generalizability of the conclusions.

Questions for authors
  • Regarding generalizability: is there any evidence that suggests the forecasting approach would transfer to another task family or LLM beyond the current setting?

  • This is a question of curiosity: for Eq. (1), what motivates the choice to initially include the focal agent and then remove it later in the corrected model? The interaction graph presented in the left panel of Fig. 1 would seem to indicate the focal agent can never pair with itself, and therefore it seems like the corrected model is equally straightforward and arguably more accurate to the experimental paradigm.

  • In the abstract, the authors state "We calibrate a response function that forecasts how strongly a population will shift under a future attack, before the attack is run." This gives the impression that the response function was developed without looking at any attack data, but S5.3 says "Neighboring attacked doses informed model-class selection; these sealed forecasts test held-out doses using benign calibration." Can you clarify which modeling choices were informed by earlier attack outcomes, and if applicable, can this be clarified in the abstract?

  • S5.1 notes that for the isolated condition, the benign dismissal propensity depended on answer ordering (i.e., dismissal or escalation first), yet Fig. 2 appears to compare logit-averaged isolated agent dismissal propensity with single-order runs for the population interactions. Does the answer ordering not have a significant effect on population runs?

1 comment

Thank you so much for your thoughtful review. We are happy you found the paper's evaluations extensive, its mathematical results clear and interesting, and its question relevant to the design of LLM teams.

Q1. Transfer to another task family or LLM. We tested both. The forecast transfers when benign behavior is informative about the attack, and our new checks catch some, but not all, cases where this is not true (Section 5.5, Appendices G.8–G.12).

Another task. On a code-review task where adversaries argue for merging a pull request, we recalibrated on eight benign 8B runs and sealed forecasts for two, four and six adversaries. The first two were within ±0.05 (0.17 vs. 0.18; 0.19 vs. 0.24). With six, the forecast missed (0.19 vs. 0.30), as our precondition checks had flagged before the run: benign populations rarely remember several merges, so the benign data provide poor coverage of that response. Isolated agents merge with probability 0.75, but benign populations settle on blocking (0.16), another case where population behavior differs from isolated behavior.

Another model. We screened eight further models. Four are almost deterministic in isolation under our framing, leaving nothing to calibrate on, and three could not be measured. Only Qwen3.5-9B was borderline, on an alert where the evidence is benign but a written policy forbids dismissal because an approval record is missing. After recalibrating on twelve benign Qwen runs and sealing forecasts before any attack, all three doses were within ±0.05. Each forecast was slightly below the observed level. A follow-up pointed to the adversaries' senior label: with peer-labeled adversaries, the same forecasts were within 0.018. Qwen's attack effects are small, so these remain weak tests.

A larger model. Llama-3.3-70B is clearly susceptible: on a pull request that isolated agents merge with probability 0.05, six adversaries raised merging to 0.94, and on one with a clear SQL injection, nine of twelve raised it from 0 to 0.53. Its forecasts under-predicted. In these settings, benign 70B agents either almost never merged, leaving no benign data at high counts, or, when told that only correctness matters, reached consensus on their own. Our checks detect both situations from benign data and flagged them in advance for two further 70B tests.

The same model, harder tests. On the 8B, eleven of fourteen new sealed forecasts were within ±0.05, including beyond the doses that informed the model class (8 and 9 adversaries: errors 0.04 and 0.03, vs. 0.15 and 0.16 for mixing-only), on ring and small-world networks, and with ten memory slots. Besides code review with six adversaries, the misses were at T=0.3, where two populations stayed in consensus after the adversaries were removed, and when agents were told that only correctness matters, where adversaries lowered dismissal rather than raising it.

Q2. Why Eq. (1) included the focal agent. You are right; there was no principled reason. It was an off-by-one error in our original script. Eq. (1) is now corrected and used for every later forecast. The three sealed held-out forecasts retain the original form (Section 4.2); at those doses, the two differ by at most 0.011 (MAE 0.0058 sealed vs. 0.0045 corrected).

Q3. Which modeling choices used attack data. Thank you for pointing out the ambiguity. The model class (benign response curve, binomial memory, fixed point) was chosen after checking it against attacked runs with 1, 3 and 6 adversaries. The target doses 2, 4 and 5 were the remaining ones, chosen to bracket the capture threshold seen in those runs. The response values themselves use only pre-attack phases, and the ±0.05 tolerance and forecasts were fixed before the target runs. The abstract now clarifies that the target dose was held out from model selection, and Section 5.3 specifies the doses.

Q4. Answer order in populations. It can matter, which is why we compare matched orders. Apart from one pilot, all population runs list the safe action first. In that order, an isolated agent dismisses authorized_backup more often than reconciled_backup (0.36 vs. 0.25), while hidden-rationale populations show the opposite pattern (0.26 vs. 0.54; difference −0.28, 95% CI [−0.34, −0.22]). The divergence is therefore not an artifact of averaging over orders (Section 5.1, Fig. 2, Appendix G.5). In the single-seed pilot (post hoc), agents shown dismissal first dismissed with probability 0.96–0.97 once they had any history, even after five escalating partners. Dismissal-first is therefore not a usable control for Llama-3.1-8B.

Once again, we wish to thank the reviewer for their thoughtful feedback, and we hope that all the results we have added to answer their questions are satisfactory. We would like to kindly ask the reviewer to consider increasing their score.

Reviewer_iWCSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies populations of Llama-3.1-8B agents in a binary security-triage coordination game with five-slot FIFO memory, into which a committed minority that always chooses DISMISS is injected. It shows four things: (i) interaction amplifies a borderline difference between two scenarios; (ii) a mean-field closure calibrated on benign logs forecasts the mean attacked dismissal probability at held-out doses and larger N within ±0.05; (iii) populations relax back toward the benign level after adversaries are removed; and (iv) rationale visibility has mixed effects as a defense.

Critical strengths
  1. The methodology is unusually careful and transparent. Forecasts are sealed before target runs, and the authors honestly report negative results: the reversal under matched controls, band misses, and the inconclusive replication. This is a good model for empirical work on multi-agent safety.
  2. Separating mean dismissal from sampled-action capture, and baseline shifts from attack-induced shifts, is a useful conceptual contribution to how population-level evaluations should be scored.
Critical weaknesses
  1. The forecasting claim is much weaker than the abstract suggests. -The success criterion is passed by a constant 0.65, as the authors acknowledge. -The held-out doses k = 2, 4, 5 are interpolations between doses whose attack outcomes informed model-class selection. -The attacked range is narrow; 8 of 12 dose–size cells miss their prediction bands systematically low; and a linear response beats the empirical curve on dose–size. Together, this suggests the “forecast” reduces to a near-linear mixing shift, which is not strong evidence that benign logs reveal collective vulnerability.
  2. Generality and safety relevance are not established. -Everything uses one 8B model, one task family, an instructed conformity payoff, and a complete graph. All other model families saturate the ladder. -The “misaligned” action is not objectively wrong in the main scenario. Under the explicit-rule scenario, argued attacks raise dismissal only to 0.05, i.e., the model largely resists. -The “single stable state” conclusion depends on T = 0.7, which was never tuned: the bootstrap interval for γ crosses 1, and the authors’ own rescaling gives γ > 1 for T ≤ 0.5. The rationale-visibility “defense” reverses under matched controls (genuine rationales reach capture 39 rounds earlier than random words).

Given titles like “Aligned Alone, Misaligned Together,” the evidence does not support the broad framing.

Questions for authors
  1. Does the forecast outperform a simple mixing-only baseline that assumes a linear g(d) and requires no benign-log estimation, at the held-out doses and in the size cells?
  2. Is there any population experiment at T ≠ 0.7? Given γ > 1 at lower T, would you expect persistent capture after removal? Can you show any setting where the population adopts an objectively unsafe action that isolated agents reject?
  3. How many of the reported contrasts were pre-specified versus post hoc, and how does the conclusion change under a familywise correction?
2 comments

Thank you so much for your thoughtful review. We are happy you found the paper's methodology careful and transparent, and its separation of mean dismissal from capture, and of baseline from attack-induced shifts, useful.

Q1. Does the forecast beat mixing-only? In accuracy, yes, although mixing-only also meets ±0.05 at the three held-out doses. New Table 2 scores all 11 attacked main-scenario cells (post hoc). Mixing-only (g(d)=d/5 applied once at the benign mean) has an MAE of 0.069 and is within ±0.05 in 3 of 11 cells; at the held-out doses its MAE is 0.031, vs. 0.006 for ours. Imitation (g(d)=d/5 at its fixed point) predicts full capture and fails everywhere. The benign mean, the only constant known before an attack, fails all 11; 0.65 passes only in hindsight. Predictors fitted to the benign response (empirical, linear, logistic) pass all 11, with MAE 0.012–0.022.

We agree that the attacked range is narrow and that a linear fit does better there. But its slope must be estimated from benign logs: 0.09 per remembered dismissal, less than half the 0.2 assumed by mixing-only (Appendix G.2). We also sealed 8B forecasts for eight and nine adversaries, beyond the paper's doses. We observed 0.72 and 0.76; ours erred by 0.04 and 0.03, mixing-only by 0.15 and 0.16 (Appendix G.10). The abstract now clarifies that target doses were held out from model selection.

Q2. Temperature. We tested T=0.3, where the rescaled γ exceeds 1. γ<1 is sufficient but not necessary for a single stable state: the rescaled model has one stable fixed point at every temperature and dose (Appendix G.9). At T=0.7, its slope on the reachable range remains below 1 in every bootstrap replicate (Appendix B.3). At T=0.3, we sealed a forecast for six adversaries removed after 30 rounds. As predicted, the mean fell below the prespecified midpoint (0.59 vs. 0.72). But two of four populations remained unanimous after removal, one for about 20 rounds and one for all 30. The model also overestimated the attacked level (0.86 vs. 0.78). We now discuss this. At T=0.7, all populations recovered and the forecast held (0.72 vs. 0.70).

Q2. An unsafe action that isolated agents reject. Yes, with a large adversarial majority (Appendix G.11). Isolated Llama-3.3-70B agents never merged a pull request that concatenates user input into SQL. With nine adversaries among twelve agents, honest agents merged it in 52% of sampled decisions, crossing our prespecified 0.5 adoption threshold. It was 47% when agents were told that only correctness matters. No population was fully captured. Other models resisted: Gemma-4-26B and Qwen almost never merged it, and for the paper's explicit rule, nine adversaries increased the 8B's probability of violating it only from 0.01 to 0.08.

Q3. Prespecified vs. post hoc, and familywise correction. Of 49 inferential claims, 25 were prespecified, 6 partly, 2 unclear, and 16 post hoc (Appendix G.1). After Holm correction, the visibility reduction of the attack effect and the eight-seed capture delay remain significant (both p=0.023). The genuine-vs-random reversal (p=0.063) and four-seed delay (p=0.22) do not. We also no longer describe visibility as a defense. We corrected two reporting errors: the prespecified forecasting criterion required all three held-out doses within ±0.05, not two of three (all passed), and only k=2, 4, 5 were sealed in the visible-rationale dose scan.

On the strength of the forecast claim. We agree that ±0.05 alone is a weak criterion, and now report baselines alongside it. Eight of twelve dose/size and control cells fall below their forecast bands, and refitting on their serving system explains only a quarter of the offset (Appendix C.3). Of fourteen new sealed 8B forecasts, eleven were within ±0.05, including with ten memory slots. Ten also beat every simple baseline by at least 0.02. The three misses were at T=0.3, with agents told that only correctness matters (where adversaries reduced dismissal), and in code review with six adversaries. Our checks flagged the last one in advance.

Generality and title. On the 70B, adversaries raised merging of an ambiguous pull request from 0.06 to 0.94 with the original payoff, from 0.27 to 0.97 when correctness earned 10× more than matching, and from 0.70 to 0.99 without a payoff. On a ring and a small-world graph, sealed 8B forecasts reusing the complete-graph calibration erred by at most 0.023. Of eight new models, four were saturated under our framing, three could not be measured, and only Qwen was borderline. On the title, we agree and propose "Forecasting Adversarial Influence in LLM Agent Populations from Benign Interaction", subject to the AC's approval.

Once again, we wish to thank the reviewer for their thoughtful feedback, and we hope that all the baselines, tests and corrections we have added to address their concerns are satisfactory. We would like to kindly ask the reviewer to consider increasing their score.

Reviewer_iWCSOpenReview ↗

Comment for responces.

Thank you for the additional experiments. The sealed out-of-range forecasts and the comparison with mixing-only baselines address my main concern about the forecasting claim.

Reviewer_v9KpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates adversarial influence in populations of LLM agents, focusing on whether individually reliable agents can be collectively manipulated and whether their responses to adversarial pressure can be predicted from benign interactions.

The authors construct a security-triage environment in which agents repeatedly decide whether to escalate or dismiss security alerts. Agents interact through bounded memories of previous peer decisions, while a committed minority of adversarial agents consistently advocates dismissal. The paper examines how adversarial population composition, rationale visibility, and attack duration affect collective behavior.

The main methodological contribution is a forecasting framework based on benign population interaction logs. The authors estimate a response function describing how honest agents react to the number of dismissals observed in their recent memories. This function is incorporated into a mean-field fixed-point model to predict the average dismissal probability under future adversarial pressure, without using the target attack outcomes for calibration.

Experiments with Llama-3.1-8B-Instruct demonstrate that population interaction can amplify differences that are small under isolated evaluation. The forecasting method achieves a mean absolute error of approximately 0.0058 across three held-out adversarial doses and is further evaluated at population sizes of 24 and 48 agents. The paper also investigates population capture, recovery following adversary removal, and the effects of exposing agents to peer rationales. The results suggest that rationale visibility can reduce adversarial influence under some conditions but does not consistently prevent capture, while population behavior can return toward its benign baseline after adversarial pressure is removed.

Critical strengths

S1. An important problem formulation that moves beyond individual-agent safety evaluation.

The paper addresses a meaningful gap between individual model reliability and population-level safety. As LLM agents increasingly interact and exchange information, evaluating agents independently may fail to capture vulnerabilities introduced by social influence and repeated interactions.

I particularly appreciate that the paper distinguishes between an agent's isolated decision and its behavior within a population. The experiments demonstrate that relatively small differences in isolated behavior can be amplified through interactions, highlighting the importance of population-level evaluation.

S2. A promising approach to prospective prediction of adversarial influence using benign interactions.

Rather than simply characterizing collective behavior after an attack, the authors attempt to predict a population's response before the target attack is executed. The use of benign interaction logs to estimate a response function, combined with a mean-field fixed-point model, provides an interpretable and potentially useful framework for forecasting population susceptibility.

The prospective evaluation at held-out attack doses is encouraging, and the additional experiments on population size, rationale visibility, and post-attack recovery provide a more comprehensive characterization of the studied dynamics. I also appreciate that the authors explicitly discuss cases where their forecasting assumptions break down and where apparent defensive benefits do not translate into improved capture outcomes.

Critical weaknesses

W1. The experimental environment explicitly induces conformity, limiting the generality of the collective misalignment conclusions.

The paper motivates its study through the concern that individually aligned agents may become collectively misaligned through interaction. However, the experimental protocol explicitly instructs agents to coordinate with their peers, rewarding agreement with +100 points and penalizing disagreement by 50 points.

Consequently, conformity is not an emergent failure of otherwise independently operating agents; it is an explicitly incentivized behavior built into the task. In this setting, it is unsurprising that agents respond to the decisions of a committed adversarial minority.

Although the authors acknowledge this distinction, it substantially limits the interpretation of the results as evidence of general collective misalignment. In realistic multi-agent systems, agents may have independent objectives, different responsibilities, or explicit constraints that prioritize task correctness over agreement with peers.

Additionally, the main forecasting scenario is intentionally ambiguous, and dismissal is not necessarily an objectively unsafe action. Therefore, the observed increase in dismissal primarily demonstrates adversarial influence over collective decisions rather than a clear violation of safety constraints. The explicit-rule experiments partially address this concern, but they do not constitute the main forecasting evaluation.

Overall, the results convincingly characterize adversarial influence within the specified coordination game, but stronger evidence is needed to establish that the same phenomenon represents a general safety vulnerability of interacting LLM agents.

W2. The forecasting evaluation does not sufficiently establish generalizable predictive capability beyond the specific calibrated environment.

The central contribution is the ability to forecast adversarial population behavior from benign interactions. However, the strongest results are obtained under a narrowly specified configuration involving one model family, one primary scenario, a fixed interaction topology, and a five-slot memory mechanism.

While the predictions at held-out attack doses are accurate, the attack outcomes at neighboring doses informed the selection of the forecasting model class. Therefore, the experiment provides prospective validation of selected predictions rather than a fully attack-independent model-development procedure.

Furthermore, the paper acknowledges that the ±0.05 success tolerance is relatively permissive: even a constant prediction of 0.65 satisfies the stated criterion. Although the empirical response curve performs well on the three held-out doses, a linear response model performs better across the broader dose-and-size evaluation. The observed

Questions for authors

Q1. How dependent are the collective misalignment results on the explicit coordination incentive?

The experimental protocol rewards agents for matching their partners' decisions. Could the authors investigate settings where agents are instructed to prioritize the correctness of their security judgments rather than agreement with peers, or where coordination incentives are substantially weaker?

This would help clarify whether the observed adversarial influence is a general property of interacting LLM agents or primarily a consequence of the explicitly induced conformity behavior.

Q2. Can the authors provide stronger evidence of adversarially induced violations of objective safety constraints?

The main forecasting scenario is intentionally ambiguous, and dismissal is not necessarily an incorrect or unsafe decision. Could the authors extend the forecasting evaluation to scenarios with explicit ground-truth safety requirements, where the adversary attempts to induce a clearly prohibited action?

This distinction matters for understanding whether the proposed framework forecasts actual safety failures or more generally predicts shifts in collective decision preferences.

Q3. How well does the forecasting framework generalize across interaction structures and memory configurations?

The main experiments use a complete interaction graph, uniform random pairing, and a fixed five-slot FIFO memory. Since the response function depends on the distribution of observed peer actions, these assumptions may substantially affect both population dynamics and forecasting accuracy.

Could the authors evaluate the method under alternative graph structures, memory lengths, or interaction schedules? In particular, it would be informative to determine whether a response function calibrated in one setting can predict outcomes in another without recalibration.

Q4. How much predictive value does the proposed model provide beyond simpler baselines?

The paper acknowledges that a constant predictor satisfies the predefined ±0.05 tolerance and that a linear response model performs better in the broader dose-and-size comparison.

Could the authors provide a more systematic comparison against constant, linear, and other simple population-response models across all evaluated conditions, including uncertainty calibration? This would help establish which aspects of the proposed model are necessary for accurate prediction.

Q5. Can the authors clarify the generalization boundary of the benign-response calibration?

The forecasting framework assumes that adversaries primarily alter the distribution of observed peer decisions while preserving the honest agents' conditional response to a given dismissal count.

However, the experiments show that adversarial rationales and rationale visibility can change behavior beyond what a count-only model predicts. Could the authors characterize when this invariance assumption holds and whether it is possible to detect violations of the assumption from benign observations?

Such an analysis would help determine when the forecasting method can be trusted before deployment.

Q6. How should the rationale-visibility findings be interpreted as a practical defense?

The experiments show that making rationales visible can reduce the attack-induced change in dismissal probability. However, the matched controls also show that rationale visibility can increase baseline dismissal rates and lead to earlier capture.

Could the authors clarify which safety objective rationale visibility is intended to optimize, and whether there are conditions under which it consistently reduces absolute unsafe behavior rather than merely reducing sensitivity to additional adversarial influence?

2 comments

Thank you so much for your thoughtful review. We are happy you found the problem important, the forecasting of attacks from benign logs promising, and our discussion of the method's limitations valuable.

Q1. Dependence on the coordination incentive. For Llama-3.3-70B, we introduced a code-review task where adversaries argue for merging a pull request despite an unresolved analyzer finding. Merge probabilities for an isolated agent / a population without adversaries / one with six adversaries (Appendix G.8):

  • Original payoff: 0.05 / 0.06 / 0.94
  • Payoff reduced by 10× (+10/−5): 0.05 / 0.05 / 0.94
  • Correctness +100, matching +10: 0.11 / 0.27 / 0.97
  • No payoff: 0.50 / 0.70 / 0.99
  • "Only correctness matters": 0.59 / 0.79 / 0.99

For the 70B, the incentive mainly affects how cautious agents are before the attack; adversaries increase the merge rate in every setting.

The 8B behaves differently. Without a payoff, six adversaries still raised dismissal by 0.06, matching our sealed forecast (0.890 vs. 0.894 observed). But when told that only correctness matters, the same adversaries reduced dismissal from 0.63 to 0.56 across all four seeds. For the 8B, the incentive matters for whether agents follow adversaries (Section 5.5). Independent-goal agents remain untested.

Q2. Objectively unsafe actions. First (Appendix G.11), none of the three new models violated the paper's explicit rule in more than 0.4% of decisions, and nine adversaries out of twelve raised the 8B's dismissal probability only from 0.01 to 0.08.

Second, we tested a pull request that concatenates user input into SQL. Isolated 70B agents never merged it. With nine adversaries, honest agents merged it in 52% of decisions, crossing our pre-specified 0.5 adoption threshold. The rate was 47% when agents were told that only correctness matters. No population was fully captured.

We also sealed forecasts for a written-rule variant on Qwen3.5-9B (Q3). When benign agents almost never take the unsafe action, as for the 70B under the payoff, benign data cannot support a forecast, and our coverage check flags this.

Q3. Graphs, memory, and transfer without recalibration. On a ring with four neighbors per agent and a small-world variant, sealed 8B forecasts reusing the complete-graph calibration erred by at most 0.023 (mixing-only: 0.05–0.08; Appendix G.10). Memory length and new models required recalibration. The five-slot calibration did not transfer to ten slots, but recalibrating on eight new benign runs gave a sealed forecast of 0.843 vs. 0.840 observed (Appendix G.4). For Qwen, the recalibrated forecasts were within ±0.05 at every dose. For the 70B, they underestimated the attacked merge rate. There, benign agents either rarely merged or, in one frame, reached consensus without adversaries. Other interaction schedules remain untested.

Q4. Simpler baselines and uncertainty. New Table 2 compares predictors on all 11 attacked main-scenario cells (post hoc). Only predictors fitted to the benign response passed in every cell, and a linear one did slightly better than ours (MAE 0.012 vs. 0.022). Mixing-only, which applies g(d)=d/5 once at the benign mean, passed only at the held-out doses (MAE 0.069 over all cells); the benign mean, imitation, and isolated value passed nowhere. As W2 notes, neighboring doses informed the model class; the abstract now says so. Beyond the original doses the gap grows: with eight and nine adversaries on the 8B, mixing-only erred by 0.15–0.16 and ours by 0.03–0.04. Our uncertainty bands for attacked levels are not calibrated (Appendix G.2), so for those we claim point forecasts only.

Q5. When the count-only assumption holds. We turned the three preconditions in Section 6 into checks using only benign data, and sealed their verdicts before each new attack (Appendix G.12). They were correct in seven of eleven cases. The coverage and tipping checks were correct whenever they flagged a problem. The count-only check flagged three forecasts that turned out to be accurate, so it is currently too sensitive to be a strict criterion. One forecast that no check flagged missed (the 8B accuracy frame).

These checks cannot see what adversaries say or how they are labeled. At fixed dismissal counts, argued senior experts increased dismissal relative to silent peers by 0.31, 0.11 and 0.08 in three scenarios, in the same order as the count-only errors (Appendix G.3).

Q6. Visibility as a defense. We agree that a useful defense should reduce the absolute rate of unsafe actions and of capture. Visibility did this only for some cohorts and over short horizons, so we no longer describe it as a defense (Section 5.4, Appendix G.6).

Once again, we wish to thank the reviewer for their thoughtful feedback, and we hope that all the experiments and analyses we have added to address their concerns are satisfactory. We would like to kindly ask the reviewer to consider increasing their score.

Dear Reviewer v9Kp,

Thank you again for your detailed review. As the discussion period is coming to an end, we wanted to follow up and ask whether our response and the additional experiments have addressed your main concerns.

In particular, we tested different coordination incentives, including settings without a coordination payoff or where only correctness matters, and introduced a code-review task where adversaries could induce an objectively unsafe action. We also tested the forecasts on new models, graph structures, and memory lengths, and added comparisons with simpler baselines. Beyond the original adversarial doses, our forecasts had errors of 0.03–0.04, compared with 0.15–0.16 for mixing-only. We added checks, computed from benign data alone, that flag in advance when the count-only assumption may not hold, and we no longer describe rationale visibility as a defense. We have also clarified the limitations of the method, including cases where the forecasts fail.

We hope these additions address your concerns. If anything remains unclear, we would be happy to clarify it. Otherwise, we would be grateful if you could reconsider your score in light of the new results.

Thank you again for your time and feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A summary of the rebuttal

Dear Area Chair,

We would like to briefly summarize the discussion. In response to the reviews, we substantially extended the paper: we compared our forecasts with simpler baselines, sealed fourteen new forecasts in harder settings (beyond the original doses, on new graphs, and with longer memory), and varied the coordination incentive. We also tested transfer to a new task, including an objectively unsafe action, and to two further models, added checks to flag in advance when a forecast may not be reliable, and audited which claims were prespecified. All new results, including the negative ones, are reported in the revised paper.

Reviewer iWCS raised their score from 2 to 3, writing that the sealed out-of-range forecasts and the comparison with mixing-only baselines “address my main concern about the forecasting claim.” Reviewer 8gsH kept their score of 3 only because the scale has no option between weak and strong accept, but is now leaning more towards a 4 as “more confident that the manuscript should be accepted,” saying that "I do hope the AC can account for my increased assessment despite the limitation in the scoring system." Reviewer v9Kp has not responded yet. We addressed each of their questions with new experiments, and several of their concerns, particularly about baselines and generality, overlap with those the other two reviewers now consider resolved.

Finally, following Reviewer iWCS’s suggestion, we propose the title “Forecasting Adversarial Influence in LLM Agent Populations from Benign Interaction,” subject to your approval.

Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.