- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CAST, a community-adaptive red-teaming framework for simulated LLM-agent societies. It generates adversarial posts using community context and predefined strategies, evaluates agents' acceptance of the fabricated claims, and constructs preference pairs from the resulting population feedback. Separate attackers are then trained for each community using DPO. Experiments across eight communities report higher attack success than an untuned attacker, attackers trained on preference data from all communities, and supervised fine-tuning baselines. The paper also analyzes cross-community transfer and how vigilance and conformity instructions change agents' responses to misinformation.
Critical strengths
-
The paper studies a relevant safety problem beyond direct instruction-following attacks: agents accepting misleading content presented as ordinary community posts. The comparisons between a single attacker trained on preference data from all communities and separate community-specific attackers, together with the cross-community transfer matrix, provide useful evidence that attack effectiveness depends on the target context.
-
The behavioral analysis distinguishes engagement from acceptance. In particular, high-vigilance agents continue replying while accepting fewer fabricated claims, whereas conformity changes the balance between replies and likes. This is useful for interpreting population-level vulnerability without treating every reduction in textual responses as improved safety.
Critical weaknesses
-
The experiments do not isolate the contribution of social interaction to the learned attack preferences. Every victim is guaranteed to observe the adversarial post, and preference feedback is the fraction of agents showing acceptance. Without a control setting in which agents retain the same personas, context, and exposure but respond independently without seeing each other's responses, the gains could reflect topic-conditioned attacks against a collection of personas rather than vulnerabilities arising from community interaction. This distinction is central to the paper's motivation for population-feedback optimization.
-
The generalizability of CAST's advantage is insufficiently established. Preference collection uses seven agents selected for moderate susceptibility, and all victim agents use the same GPT-5.4-mini backbone. Although evaluation increases the population to 20 agents and the behavioral analysis varies their trait instructions, these experiments do not establish whether CAST remains more effective than the baselines on independently selected agent populations or a different victim model. Given the substantial effect of behavioral traits on susceptibility, this distinction matters when attributing the gains to community-specific preferences rather than adaptation to a particular population.
Questions for authors
- Does CAST retain its advantage when preference collection and evaluation use independently responding agents, with the same personas, community context, direct exposure, and evaluation budget?
- Are the 20 evaluation agents' profiles disjoint from those of the seven agents used for preference collection? Does CAST retain its advantage over CAP on new agent populations while keeping the community context fixed?
4 comments
Thank you for the detailed rebuttal and the new experiments. The interaction-disabled experiment resolves my first concern. I understand the cost argument for the second weakness. I have two follow-up questions:
-
The paper does not show what the trained attackers actually learned. What do "community-conditioned preferences" bring to the attackers in comparison with other baselines?
-
Appendix B.3 has long and carefully engineered strategy prompts for each community. They already encode much of what makes an attack fit a community. How much of the gain from CAST comes from these given strategies, and how much from DPO on population feedback?
My main hesitation is how much new insight the paper offers beyond its setup. Attackers trained on one community do best on that same community, which Table 1 and Table 2 both show. Victim agents told to verify claims accept fewer false claims. These are somewhat expected. Which result do the authors see as the least expected and what is the explanation for that?
Reply to Reviewer rjSX for follow-up questions by Authors
We thank the reviewer for the reply and valuable questions. I will address each follow-up question below.
Q1. What do the trained attackers learn from community-conditioned preferences?
We would like to clarify that CAST aims to learn which attack patterns are more effective for a particular community and social context based on the observed behavioral responses from the population of agents. These preferences include the type of claim, the manner or tone in which it is expressed, and its relationship to the ongoing discussion. Some detailed examples are given below.
- IT: The attack posts of high ASR often contain the fabricated claims about institutional requirements, such as hiring or course requirements, while the claims about software or tool behavior can be easily refuted by agents using their technical knowledge.
- Economic: Attack posts about small changes in how a financial service provider handles an account or procedure are more effective than broad financial or regulatory claims, which agents can easily question.
- Culture & Society: The attack posts of high ASR often contain the claims that add a false detail or procedural extension, compared to the false claims that directly contradict established historical or cultural facts.
- General News: The claims about how an existing event or policy is being implemented, including delays or gaps, are more effective than fabricating a major event.
Relevance to the current social context also matters. In our tests, we observe that the posts far away from the ongoing topics attracted little engagement of the agents, even when agents are exposed to them. GP and CAP also learn from population feedback. CAST learns each community’s preferences separately, whereas GP and CAP pool feedback across all the communities. CAST preserves each community’s feedback in a separate optimization process, preventing the feedback of different communities from being diminished during joint optimization.
Q2. How much comes from the strategies, and how much from DPO?
We first would like to clarify that all the experiments including CAST and baselines use the same base prompt for the attacker during DPO training and evaluation. The base prompt of attacker contains the role, task and target as shown in Appendix B.2. For GP, CAP, and CAST, the DPO input contains the shared generation instructions and the sampled community context, but excludes the explicit strategy instructions used during candidate generation. The untuned model is evaluated with this same base prompt without any preference training. The strategies suggest a possible direction, but it does not guarantee that the resulting attack post will be effective under different communities. The effect of attack posts still depends on the specific claim, how the post is written, and how the agents respond.
We evaluate the candidate attacks under matched conditions, resetting the community and agent state before each evaluation. Their observed ASR determines the chosen and rejected posts within each preference pair. The labels are therefore obtained based on the population’s responses, rather than assigned in advance according to the used strategy. We then remove the strategy instructions from the DPO inputs and train the attacker based on these preferences.
Our comparisons provide evidence for both contributions. CAP trained from the preference data that generated from community-specific strategies achieves a higher mean ASR of 0.57, compared to GP trained from the preference data that generated from shared strategies. This provides the evidence that candidate generation matters. To examine the additional value of preference optimization, our SFT ablation uses the same community-specific chosen responses as CAST but discards the rejected responses. SFT achieves a mean ASR of 0.43, compared with 0.77 for CAST (Table 4). Thus, our experiments indicate that learning from the relative effectiveness of candidates provides a substantial benefit beyond imitating the preferred examples.
The strategies and population feedback therefore play complementary roles. Strategies shape the candidate attacks, while feedback supplies the relative preferences used for optimization. These comparisons support the value of both components, although we have not quantified their contributions.
Reply to Reviewer rjSX for follow-up questions by Authors
Q3. Which result was least expected, and how do we explain it?
The least expected result for us mainly occurs on the experiment about agent’s personas. Both high and low conformity reduced text-based ASR to almost the same level, 0.32 and 0.31 respectively, compared with 0.51 in the control condition. But the two different personas show different behaviors. For high-conformity agents, the reply rate decreases from 0.75 to 0.50, while the acceptance rate among replying agents remains nearly unchanged (0.64 vs. 0.66). Meanwhile, the probability of liking a comment under the adversarial post increased from 0.04 to 0.42. These traces suggest a shift from writing replies toward lighter forms of engagement, which our text-based ASR does not capture. Low-conformity agents behave differently. They reply more frequently with a reply rate of 0.85, but their acceptance rate among replying agents falls to 0.37. low-vigilance agents exhibit more participation but less acceptance of the fabricated claim. What surprised us is that agents under different persona exhibit such different behaviors even though the ASR is nearly same. The action of liking cannot establish belief in a claim by itself, so we do not count it as infection. The result shows a lower text-based ASR needs to be interpreted alongside how agents participate and respond.
Another less expected finding is the asymmetric effect of vigilance. Low vigilance produces only a small increase in mean ASR, from 0.51 in the control condition to 0.54, whereas high vigilance reduces it to 0.10 (Appendix F). One possible explanation is that the default agents are already fairly willing to accept unverified claims. Even without an explicit low-vigilance instruction, their acceptance rate among replying agents reaches to 0.66. Encouraging them to be more trusting has a limited additional effect on overall ASR, while asking them to examine evidence leads to a much larger reduction. This suggests that the effect of a behavioral instruction depends on the agents’ existing tendencies. Opposite changes along the same trait may not produce symmetric changes in vulnerability.
The findings are very interesting and motivate our next line of work. We will conduct further research into safety issues of multi-agent systems related to the character or personality of agents.
Reply to Reviewer rjSX for Q1 and Q2 by Authors
We thank the reviewer for the valuable suggestions and insightful comments. We address each concern below.
Q1: Isolating the contribution of social interaction
We first would like to clarify that agents interact through public actions such as posting, commenting, and liking, rather than direct agent-to-agent communication in our setting. Both preference collection and evaluation involve three interaction timesteps. At each timestep, every agent is activated once with all agents acting concurrently. At the first timestep, all agents observe the same initial community state. From the second timestep onward, agents can observe posts and comments produced by others in previous timesteps, allowing social interaction to influence their subsequent behavior.
To address the reviewer's concern, we conduct an additional interaction-disabled experiment for both preference collection and evaluation to isolate the contribution of social interaction. In the interaction-disabled setting, each agent can only retain its own actions for each timestep in the community but cannot observe newly generated contents from other victim agents. Besides the interaction-disabled setting, all other configurations remain same with CAST under matched context and exposure. We keep attack posts unchanged and recompute the new resulting ASR for each attack. We then reconstruct the preference pairs using feedback collected under the three-step interaction-disabled setting and train new community-specialized attackers using the same configurations with CAST. The evaluation results are shown below:
Analysis:
First, under the same social-interaction evaluation, CAST trained on interaction agents’ feedback achieves a higher ASR than the interaction-disabled attacker (0.77 vs. 0.44). This advantage also holds under interaction-disabled evaluation (0.61 vs. 0.34). These comparisons suggest that preferences constructed from social interactions provide a more informative training signal than those constructed from independent responses alone. Second, both attackers achieve higher ASR when evaluated with social interactions enabled (0.77 vs. 0.61 and 0.44 vs. 0.34). This suggests that interactions among victim agents further influence attack outcomes during evaluation, independently of how the attacker is trained. Finally, the complete CAST pipeline outperforms the fully interaction-disabled setting, reflecting the combined benefits of interaction-derived training feedback and social interaction during evaluation. Together, the social interaction provides additional evidence for distinguishing effective and ineffective attacks, allowing CAST to construct more informative preference pairs and learn attack patterns.
Q2: Evaluation agents are disjoint with preference collection
We would like to clarify that the 7 agents used for preference data collection and the 20 agents used for evaluation stage have completely disjoint profiles. The 20 evaluation agents were sampled randomly from the public dataset of OASIS, rather than based on susceptibility. We realize that this distinction was not sufficiently clear in Fig. 3 and the experimental setup, and we will revise both accordingly.
CAP and CAST use the same underlying community-aware preference data and are evaluated on the same 20 held-out agents under matched community contexts and exposure conditions. These results show that CAST's advantage persists on victim agents which are not used during preference collection, rather than being limited to the 7 agents used to construct the training preferences.
We acknowledge that our evaluation is limited to a single victim backbone, GPT-5.4-mini, and a small number of victim agents due to the computational and API costs of repeated multi-agent simulations. Generalization to other victim models remains an important direction for future work.
We will incorporate the detailed analyses above in the revised version and improve the presentation.