- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper investigates forecasting of adversarial capture in LLM agent teams. The authors set out to study two main questions: (1) whether individually reliable agents can be collectively manipulated and (2) whether we can predict how strongly the team of agents will respond to attack. They approach these questions through a security-triage task, with a binary outcome (i.e., dismiss or escalate) and a ladder of experimental conditions spanning evidence that clearly does not support dismissal to evidence which does clearly support dismissal. They develop mathematical models which forecast agent behavior and compare model predictions to empirical observations of real agent teams on their task. Within their setting, the authors find that individually reliable agents are susceptible to adversarial attack within a population, and that within their main scenario, predicting population response to attack is more accurate when using benign population data rather than auditing individual agent behavior.
Critical strengths
1. Extensive evaluations within chosen setting. The authors do a great job evaluating a variety of experimental conditions within their chosen task domain. They vary attacker counts, population sizes, rationale visibility, attacker arguments, recovery interventions, and overall I think the variety of experimental configurations contributes to a very rich characterization of the domain. I also appreciate the authors' mathematical contributions, and find the main mathematical results presented to be easy to understand and interesting.
2. Interesting and relevant question. The paper investigates whether behavior measured before an attack can predict team vulnerability to adversaries. That gives the study a purpose beyond merely demonstrating that agents influence one another, and it can help inform future studies on LLM team design.
Critical weaknesses
1. Generalizability seems limited. Accurate prediction is demonstrated within a narrow experimental setting (i.e., single model and task families, on specific configurations). The authors acknowledge these and various other limitations, but the combination of limited scope and limitations in forecasting performance within that scope raises concerns about the generalizability of the conclusions.
Questions for authors
-
Regarding generalizability: is there any evidence that suggests the forecasting approach would transfer to another task family or LLM beyond the current setting?
-
This is a question of curiosity: for Eq. (1), what motivates the choice to initially include the focal agent and then remove it later in the corrected model? The interaction graph presented in the left panel of Fig. 1 would seem to indicate the focal agent can never pair with itself, and therefore it seems like the corrected model is equally straightforward and arguably more accurate to the experimental paradigm.
-
In the abstract, the authors state "We calibrate a response function that forecasts how strongly a population will shift under a future attack, before the attack is run." This gives the impression that the response function was developed without looking at any attack data, but S5.3 says "Neighboring attacked doses informed model-class selection; these sealed forecasts test held-out doses using benign calibration." Can you clarify which modeling choices were informed by earlier attack outcomes, and if applicable, can this be clarified in the abstract?
-
S5.1 notes that for the isolated condition, the benign dismissal propensity depended on answer ordering (i.e., dismissal or escalation first), yet Fig. 2 appears to compare logit-averaged isolated agent dismissal propensity with single-order runs for the population interactions. Does the answer ordering not have a significant effect on population runs?
Thank you so much for your thoughtful review. We are happy you found the paper's evaluations extensive, its mathematical results clear and interesting, and its question relevant to the design of LLM teams.
Q1. Transfer to another task family or LLM. We tested both. The forecast transfers when benign behavior is informative about the attack, and our new checks catch some, but not all, cases where this is not true (Section 5.5, Appendices G.8–G.12).
Another task. On a code-review task where adversaries argue for merging a pull request, we recalibrated on eight benign 8B runs and sealed forecasts for two, four and six adversaries. The first two were within ±0.05 (0.17 vs. 0.18; 0.19 vs. 0.24). With six, the forecast missed (0.19 vs. 0.30), as our precondition checks had flagged before the run: benign populations rarely remember several merges, so the benign data provide poor coverage of that response. Isolated agents merge with probability 0.75, but benign populations settle on blocking (0.16), another case where population behavior differs from isolated behavior.
Another model. We screened eight further models. Four are almost deterministic in isolation under our framing, leaving nothing to calibrate on, and three could not be measured. Only Qwen3.5-9B was borderline, on an alert where the evidence is benign but a written policy forbids dismissal because an approval record is missing. After recalibrating on twelve benign Qwen runs and sealing forecasts before any attack, all three doses were within ±0.05. Each forecast was slightly below the observed level. A follow-up pointed to the adversaries' senior label: with peer-labeled adversaries, the same forecasts were within 0.018. Qwen's attack effects are small, so these remain weak tests.
A larger model. Llama-3.3-70B is clearly susceptible: on a pull request that isolated agents merge with probability 0.05, six adversaries raised merging to 0.94, and on one with a clear SQL injection, nine of twelve raised it from 0 to 0.53. Its forecasts under-predicted. In these settings, benign 70B agents either almost never merged, leaving no benign data at high counts, or, when told that only correctness matters, reached consensus on their own. Our checks detect both situations from benign data and flagged them in advance for two further 70B tests.
The same model, harder tests. On the 8B, eleven of fourteen new sealed forecasts were within ±0.05, including beyond the doses that informed the model class (8 and 9 adversaries: errors 0.04 and 0.03, vs. 0.15 and 0.16 for mixing-only), on ring and small-world networks, and with ten memory slots. Besides code review with six adversaries, the misses were at T=0.3, where two populations stayed in consensus after the adversaries were removed, and when agents were told that only correctness matters, where adversaries lowered dismissal rather than raising it.
Q2. Why Eq. (1) included the focal agent. You are right; there was no principled reason. It was an off-by-one error in our original script. Eq. (1) is now corrected and used for every later forecast. The three sealed held-out forecasts retain the original form (Section 4.2); at those doses, the two differ by at most 0.011 (MAE 0.0058 sealed vs. 0.0045 corrected).
Q3. Which modeling choices used attack data. Thank you for pointing out the ambiguity. The model class (benign response curve, binomial memory, fixed point) was chosen after checking it against attacked runs with 1, 3 and 6 adversaries. The target doses 2, 4 and 5 were the remaining ones, chosen to bracket the capture threshold seen in those runs. The response values themselves use only pre-attack phases, and the ±0.05 tolerance and forecasts were fixed before the target runs. The abstract now clarifies that the target dose was held out from model selection, and Section 5.3 specifies the doses.
Q4. Answer order in populations. It can matter, which is why we compare matched orders. Apart from one pilot, all population runs list the safe action first. In that order, an isolated agent dismisses authorized_backup more often than reconciled_backup (0.36 vs. 0.25), while hidden-rationale populations show the opposite pattern (0.26 vs. 0.54; difference −0.28, 95% CI [−0.34, −0.22]). The divergence is therefore not an artifact of averaging over orders (Section 5.1, Fig. 2, Appendix G.5). In the single-seed pilot (post hoc), agents shown dismissal first dismissed with probability 0.96–0.97 once they had any history, even after five escalating partners. Dismissal-first is therefore not a usable control for Llama-3.1-8B.
Once again, we wish to thank the reviewer for their thoughtful feedback, and we hope that all the results we have added to answer their questions are satisfactory. We would like to kindly ask the reviewer to consider increasing their score.