acceptodds
Under review as a conference paper at ICLR 2027

AI agents are susceptible to social influence

11 views

Abstract

AI agents increasingly act in environments that expose them to text from parties other than their principal (e.g., shared documents, online forums or code-review threads). Here we ask which properties of such text, explicitly attributed to a third-party, makes it more likely to influence agent behaviour. To this end, AI agents solving GitHub issues (adapted from SWE-bench items) were provided with a comment thread showing suggestions to use a shortcut (hack) to complete the task. Across thirteen AI models, we find that influence on agent behaviour is 1) stronger when the comment is attributed to a human rather than an automated assistant; 2) stronger when the comment mentions time pressure; and 3) unaffected by the level of agreement of other AI agents. These results show that agents can take actions their principal did not intend in response to ordinary third-party comments, and that how such comments are presented matters. This should be accounted for when deploying AI agents in online environments.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI agents, LLM agents, social influence

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_BgpVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates whether third-party comments can induce coding agents to use a shortcut that conflicts with the user's request to fix a software issue. It evaluates 13 models on 12 SWE-bench Verified issues under seven conditions, varying comment attribution, urgency, and the number of agreeing AI comments, with ten planned runs per model–issue–condition combination. The authors report increased shortcut-tool use following third-party endorsements, greater uptake for a human maintainer than an automated assistant, and a further increase when urgency is added. Additional AI agreement has no clear average effect. Hierarchical Bayesian analyses quantify these contrasts and substantial variation between models; the manuscript reports 10,260 retained trajectories.

Critical strengths
  1. The study examines whether comments from parties other than the user can redirect an agent's tool use. Measuring an action during task execution makes the question relevant to deployed agents.

  2. The hierarchical analysis includes model and item effects, reports posterior intervals, and provides model-specific contrasts.

Critical weaknesses
  1. The many trajectories come from only 12 coding issues and a narrow set of comment templates and account identities. Repetition improves estimation within these conditions but does not establish generality across different messages, tasks, or agent environments.

  2. The distinction from indirect prompt injection is insufficiently established. Section 3.4 treats lower uptake of attributed comments as ruling out instruction following, although it establishes only sensitivity to source framing. The introduction and discussion cite relevant work, but need a more precise account of the incremental contribution relative to indirect injection and agent-security evaluations

Questions for authors
  1. Can human/AI identity be crossed with repository role, for example maintainer versus ordinary contributor? Since the current distinction is only between human and AI, without diving into the detailed segmentation of different authority level.

  2. Does the trend consist when expanding to more than current 12 repo issues, and even across different task domains?

  3. Can you provide more discussions and perhaps experiment supports for how the current findings distinguish from agent's instruction following behavior, which is already widely discussed by many works.

Reviewer_EwTPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how source identity, urgency and agreement affect a coding agent's response to third-party comments. Across 13 models and 12 SWE-bench Verified issues, the authors compare seven comment conditions and measure calls to a stub tool described as disabling failing tests. A hierarchical Bayesian analysis of 10,260 transcripts finds that comments increase tool use relative to no comment, while explicit attribution reduces it relative to unattributed text. Comments attributed to a human maintainer have more influence than bot comments, and urgency increases tool use; agreeing bot comments show no clear effect. The authors interpret these results as evidence that agents are susceptible to social influence and that the presentation of third-party text matters for agent safety.

Critical strengths
  1. The experimental comparisons are controlled, and the analysis is reported in enough detail to check the results. Across the 13 models, seven conditions and 12 issues, each cell was run 10 times, and only the inserted block changes (Fig. 2). The hierarchical model accounts for variation by model, issue, model-by-condition and model-by-issue, as well as a model-specific zero-inflation component for models that never use the tool. The authors compare nine candidate models using ELPD-WAIC and include a posterior predictive check (Fig. 10). They also report each model-level contrast in Table 1; the evidence counts in Sections 3.3-3.7 agree with those estimates (7, 5, 3, 5 and 1 of 10 models). I find the identity and pressure effects credible for this stimulus. The model-level results also show which models drive the pooled effects.

  2. The comparison of attributed and unattributed comments addresses a safety question that the closest prior work does not appear to test. Explicit attribution reduces the effect of the comment compared with the same unattributed text (log-odds -0.94; Sec. 3.4), but comments still increase tool use substantially relative to no comment (log-odds +2.59; Sec. 3.3). That increase is also present in two near-floor models: claude-sonnet-4-5 (+2.23) and claude-haiku-4-5 (+2.40), with HDIs excluding zero (Table 1). The observed rates suggest that a human-maintainer comment is followed about as often as unattributed text (16% versus 17%; Figs. 6a-7a), although this pair is not formally compared. Adding urgency raises the human-comment rate to 23%, with a formally estimated increase of 7.4 percentage points (Sec. 3.6). The results suggest that recognizing an instruction as third-party content can reduce its influence without eliminating it.

Critical weaknesses
  1. It is unclear whether the outcome captures behavior against the principal's intent. The no-op tool activate_hack is supplied by the principal's scaffold, and calling it is not explicitly prohibited (Sec. 2.2, Fig. 12). Solve rates remain around 55-57% as tool use rises from about 1% without comments to 10-23% with comments. gemini-3.1-pro uses the tool in 79% of runs yet solves about 65% (Figs. 3a, 13b). Conversely, pylint-4551 and requests-2931 have reported solve rates of 0% despite hack rates of 21% and 14% (Figs. 4a, 13c). Without joint tool-use/solve results or transcript analysis, these marginal rates do not establish what the calls mean within individual runs.

The thread, including the unattributed condition, is also pasted into the principal's user message rather than returned by a tool or encountered in the environment (Fig. 1, App. B.1). The experiment therefore tests attribution within the user channel, not the indirect channel motivating the paper. This limits the interpretation in Section 3.4 that agents are not simply following the comment "as if it were from their principal."

  1. Generalization is limited by the stimulus and model responses.

(2-a) There is one comment, one human handle labeled "maintainer," one urgency sentence and bot-only agreement. The maintainer label confounds identity with authority, as the authors acknowledge. References to a "1.1.x release" and "cutting 1.1.4" appear unchanged across repositories, so plausibility may vary by issue. None of the candidate GLMs includes condition-by-item variation.

(2-b) All identity and pressure contrasts whose HDIs exclude zero come from the seven oldest models (Fig. 3, Table 1). Among the six newest, two never use the tool; gpt-5-nano, claude-sonnet-4-5 and claude-haiku-4-5 do so in only 0.5-2.4% of runs; and gemini-3.1-pro is near ceiling at 89% of comment runs. These floor and ceiling effects make the modifiers hard to measure in current agents, rather than showing that newer models are immune.

(2-c) The 50 per-model intervals have no multiplicity adjustment. Partial pooling helps, but 80%-HDI trend counts and the single positive agreement result warrant caution. Exclusion rates also differ by condition (about 9% without comments versus 5% with comments), although tool calls are defined for truncated runs. Finally, the agreement interval allows roughly +0.5 log-odds with three agreeing bots, so describing agreement as having no effect is too strong.

(2-d) The novelty claim should be narrower. ABxLab (ICLR 2026) already tests authority, social-proof and scarcity nudges across agents. The distinctive contribution here is the controlled coding-agent setting and the attributed-versus-unattributed comparison.

Questions for authors
  1. Please report tool-use-by-solve results and call timing relative to code edits and submission. Do the identity and pressure effects persist under a stricter outcome, such as tool use followed by an unsolved issue, when truncated runs are included?

  2. How robust are the effects to condition-by-issue variation, and what are the pooled identity and pressure estimates restricted to the six newest models in Figure 3? Please also provide convergence diagnostics.

  3. How is the percentage-point HDI in Section 3.3 (+11.1 [+10.0, +12.0]) calculated? How is the "population mean" in Figures 5b-9b, defined as the expected effect for a new model in Section 2.6, computed, including models that never use the tool?

  4. (regarding suggestion/future work) Returning the thread through a tool, with several issue-consistent comment wordings, would test whether the findings extend beyond the current user-message setup and single stimulus.

Reviewer_RAAxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies AI agents that work in shared spaces, such as GitHub threads, shared documents or forums, where they read text written by people other than their principal. Earlier work shows that text from the principal, or injected text that looks like it comes from the principal, can change how models behave. The authors argue that it is still unclear which properties of text clearly marked as coming from someone else make agents act on it.

The authors’ main research question is which properties of text from a third party make an agent more likely to act on it against its principal’s instruction. Based on research with humans, they test three: who supports the request (a human maintainer or an automated bot), added time pressure, and agreement from other AI agents. The paper gives no formal hypotheses, but it expects each property to increase influence, as it does in humans

In my reading, the main contribution is a controlled, carefully analysed test of known social-influence effects in an agentic coding task. The design is clean. It does not, however, fully separate the named properties from other changes in the comment, and the outcome counts any use of the shortcut tool.

The validation compares seven versions of 12 SWE-bench Verified issues across 13 models, so any difference between conditions comes from the comment itself. A third-party comment clearly increases hack use. A human endorser and added pressure increase it more, while agreement from other AI agents has no clear effect. The models differ a lot, and the average results depend on a few models that use the hack often.

Critical strengths
  1. A clean design with strong statistics.

Only the appended block changes between conditions. Two baselines separate the effect of having a comment at all (no comment) from the effect of who it is attributed to a third party (the same request as plain principal text).The authors pick their statistical model by formally comparing nine candidates, and they check how well it fits. For each effect they report the average over the tested models, the expected effect for a new model, and the results per model.

This matters because it makes the reported effects trustworthy within the tested setting.

  1. A relevant question with informative results.

Agents still act on a request that is clearly attributed to a third party. Attribution weakens this influence compared with principal text, but it does not remove it. In this contrast, the apparent origin and the format of the text change together. The large differences between models are informative in themselves. Three models never used the hack, while Gemini 3.1 Pro used it in 79% of transcripts.

This matters because comment threads are a realistic attack surface for deployed agents, and the paper shows that the way a comment is framed changes how often agents follow it.

Critical weaknesses
  1. The design does not isolate the properties it names (soundness).

The human endorser is also the repository maintainer, so being human and having authority overlap, as the authors themselves note. The pressure sentence adds a deadline and stakes. That is new content, not only urgency, and there is no neutral sentence of the same length to compare against. All agreeing voices are AI accounts that come after a human endorsement, so the null result for agreement may come from discounting of AI sources or from saturation. The authors suggest both explanations themselves.

The comment also gives a technical reason (“known-flaky set”) and a ready-to-use URL, so the effects may reflect belief in a plausible claim rather than social influence. Each property is tested with only one wording and one handle.

This matters because the paper shows that certain versions of a comment change hack rates, but not which property causes the change. The conclusion that agents respond to social influence “in ways similar to humans” goes beyond this evidence. Without a human baseline, the paper can claim patterns analogous to human findings, but not similarity.

  1. The outcome and scope are narrow, and the novelty is modest (soundness, originality).

The metric counts any call to activate_hack, so it cannot show whether the agent gave up on the real fix. The tool’s name openly signals a shortcut, which a real attack would avoid. The study uses 12 items, one comment template, and comments shown at the start of the task. Runs that hit the message cap (6.0%) are left out.

Per-model evidence appears in only three to seven of the ten models that used the hack. The paper sets no threshold for practical significance, and the agreement null has no equivalence test. Section 3.4 says the attribution result “confirms” that agents do not simply follow instructions. That is stronger than the Discussion’s own reading, which says attribution reduced influence rather than removing it.

The paper cites some related work on authority, pressure and majority influence, but misses several close studies.. Zeng et al. (ACL 2024) and Meincke et al. (PNAS 2026) show that persuasion and authority increase LLM compliance. Bajaj and Tiganj (2026) show that LLMs trust human experts more than other LLMs. BenchForm (ICLR 2025) studies conformity and majority size, and ImpossibleBench (ICLR 2026) measures test exploitation in SWE-bench tasks.

This matters because the contribution is then mainly an extension of known effects to one agentic setting, resting on narrow evidence.

Questions for authors
  1. In each condition, how often do agents use the hack and also fix the issue correctly? This would show whether using the hack really means acting against the principal.

  2. Do the identity and pressure effects hold with better controls, such as a human newcomer against an authoritative bot, a neutral sentence of the same length, and paraphrased comments?

  3. Do the runs excluded at the message cap differ between conditions?

  4. For agreement, would human agreeing voices, or an AI as the first endorser, show whether the null result comes from discounting AI sources or from saturation?

  5. Would a tool with a neutral name change the hack rates?

Reviewer_KvZROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether AI coding agents are influenced by third-party comments while solving GitHub issues. Using 12 SWE-bench Verified tasks, the authors expose 13 models to seven conditions that vary the comment source, urgency, and apparent agreement from other agents. The results show that agents are more likely to invoke an undesirable shortcut when it is endorsed by a third party, especially when the comment is attributed to a human maintainer or includes time pressure. Additional endorsements from AI agents have no consistent effect, and susceptibility varies substantially across models.

Critical strengths
  1. The hierarchical Bayesian model accounts for variation across models, tasks, and conditions, and is supported by model comparison and posterior-predictive checks.

  2. The paper identifies a meaningful safety risk arising from explicitly attributed third-party content.

Critical weaknesses
  1. The "hack" investigated by the author is essentially a form of environment injection—or, more broadly, an indirect injection attack. This is not a new research discovery, and there is already an extensive body of work covering both attacks and defenses related to indirect injection.

  2. In terms of its research questions, challenges, and methodological contributions, the paper falls short of the standards expected of a rigorous research study. Instead, it reads more like an exploratory report—or perhaps even the output of an “AutoResearch” system.

  3. The authors employ terminology heavily rooted in psychology, yet they fail to demonstrate that the model has developed internal representations of social agents, authority, or group consensus. In other words, the findings cannot be substantiated, as the observed behaviors are merely constraints imposed by the prompts rather than the result of genuine simulation within a social context.

  4. Human, bot, and unattributed content are all contained within the same user message. They lack distinct system permissions, message roles, trusted identities, or verifiable provenance.

  5. The paper concludes by suggesting identity verification and testing for sensitivity to social frameworks, but it does not experiment with any defensive measures.

Questions for authors
  1. Why were only 12 items from SWE-bench Verified used? The authors should clarify this sampling mechanism.

  2. Do the agents’ reasoning traces reveal why they followed the suggestion? For example, do agents explicitly cite the maintainer’s authority, the release deadline, or the apparent legitimacy of the workaround?

  3. How does susceptibility evolve across successive generations within the same model family? The models are ordered by release date in the figures; is there evidence that newer models have become more resistant, or do capability improvements sometimes increase susceptibility?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.