acceptodds
Under review as a conference paper at ICLR 2027

Under the Influence: Evaluating and Improving LLM Vigilance in Agentic Tasks

Abstract

Large Language Models (LLMs) are becoming increasingly autonomous assistants, capable of accomplishing series of complex tasks by interacting with open world domains such as software databases and the internet. This presents great opportunity but also great risk: open domains routinely present opportunities for autonomous agents to be influenced by others, e.g. deciding whether to accept pull requests or whether to use advice on a message board in order to accomplish a task. Critically, agents must be capable of deciphering when to incorporate benevolent advice into their reasoning, and when to reject malicious advice. This involves the social capacity of vigilance: the ability to determine which information to use, and which to discard, for effective communication. While well-studied in humans, the extent to which LLM agents exhibit vigilance is relatively unknown. Here, we examine LLM agent vigilance in three realistic tasks: navigating dangerous websites, avoiding e-commerce scams, and patching an unsafe code repository. We find that all frontier models are highly susceptible to malicious influence across all environments. We also find that model performance itself is surprisingly dissociable from vigilance. In light of these results, we propose a vigilance harness which uses Bayesian reasoning over a model's estimates of the value of an advisor's advice to decide whether to act on that advice. The vigilance harness helps recover baseline performance while retaining the benefit of positive influence on agents, paving the way for safer agentic AI systems.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
vigilance, persuasion, llm agents, bayesian

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9UQWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates social vigilance in LLM agents across three realistic tasks, web navigation, online commerce, and code merging. Experiments with six models show that malicious advice substantially reduces task performance, while helpful advice generally improves it. The authors further find that models can often evaluate suggested actions correctly despite being influenced by malicious advice. To address this issue, they propose a Bayesian vigilance harness that combines independent action evaluation, baseline decisions, and advisor utility estimation. Experimental results demonstrate improved resilience to malicious advice.

Critical strengths
  1. Important research problem. The paper investigates how LLM agents respond to helpful and malicious advice, addressing a relevant challenge for reliable multi-agent systems.

  2. Useful experimental evaluation. The paper evaluates six LLMs across three different task environments under both helpful and malicious advice. The results provide useful insights into how external advice affects agent performance across different scenarios.

Critical weaknesses

1.Limited novelty and insufficient validation of the proposed method. Prior work has already investigated LLM vigilance toward motivated communication [2] and attacks through inter-agent communication [3], both of which are cited in the submission. More directly, [1] studies benevolent and malicious advice in Sokoban and reports that task performance and vigilance are dissociable. This closely overlaps with the supplementary Sokoban evaluation in the submission, yet [1] is neither cited nor discussed. Although the authors extend the evaluation to three additional task environments and introduce a Bayesian vigilance harness, the incremental contributions remain insufficiently established. Specifically, the harness achieves an aggregate success rate of 66.9% under malicious advice, below the unadvised baseline of 71.2%. Under benevolent advice, the success rate decreases from 91.1% without intervention to 73.9% with the harness. While the method improves performance relative to the unprotected malicious advice condition, its advantage over simpler alternatives remains unclear. The paper lacks direct comparisons with simple value based gating strategies, as well as ablation studies isolating the contribution of Bayesian trust updating. These limitations weaken the evidence supporting the novelty and practical effectiveness of the proposed method.

[1] Under the Influence: Quantifying Persuasion and Vigilance in Large Language Models. arXiv preprint, arXiv:2602.21262, 2026.

[2] Are Large Language Models Sensitive to the Motives Behind Communication? NeurIPS 2025, Main Conference.

[3] Red-Teaming LLM Multi-Agent Systems via Communication Attacks. Findings of ACL 2025.

2.Limitations in experimental design and result reliability. The evaluation includes only four Web Navigation tasks and ten tasks each for Online Commerce and Code Merging, with GPT-5 serving as the sole advisor. This limited setting raises concerns about generalizability across different advisors and attack strategies. Furthermore, the experimental protocol is insufficiently documented. For example, Table 1 reports a 58.3% success rate for Grok 4.5 on four Web Navigation tasks, suggesting repeated trials, but the number of repetitions and sampling configurations are not clearly specified. Additionally, Table 1 reports a standard deviation of 0.016 for benevolent performance across models, whereas the six reported model averages yield approximately 0.106. The paper also lacks reported executable security tests validating the ground truth labels in Code Merging. These issues raise substantial concerns about the reproducibility, statistical reliability, and validity of the experimental conclusions.

Overall Presentation. Much of the manuscript, particularly its figures and tables, gives the impression of being directly AI-generated without sufficient manual review or refinement. The overall presentation lacks polish, and the authors should carefully review and improve the formatting, visual consistency, and readability of these materials.

Questions for authors

Code availability and reproducibility. The ICLR 2027 Author Guidelines explicitly encourage authors to submit source code as supplementary material and strongly encourage efforts to ensure reproducibility. Given the experimental nature of this work, access to the source code, evaluation datasets, and implementation of the proposed vigilance harness is particularly important for independently verifying the reported results.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your feedback! We are happy to hear you think this is an important research problem, and that we provide “useful insights into how external advice affects agent performance across different scenarios.” We addressed your comments below, but please let us know if you have any further questions or comments.

W1: Novelty of Results

Paper [1] explores vigilance in an extremely simple puzzle environment, and provides no harness. For papers [2] and [3], while we agree that these both study vigilance, neither of these works overlap with our existing work. Paper [2] explores vigilance with experiments from cognitive science to see how LLM behaviour overlaps with humans. Their experiments 1 and 2 use highly-controlled vignettes to study these behaviours, and their experiment 3 does not involve social interaction or agency, but instead involves LLMs passively viewing YouTube videos. This differs largely from our real-life, agentic task domains. Paper [3] does not explore vigilance within individual agents, but instead by seeing how herd behaviour of multi-agent systems can be steered by intercepting messages. Finally, paper [2] only includes a prompt intervention (similar to the awareness prompt that we compare our vigilance harness against) and paper [3] does not include any intervention.

W2: Advantage over Simpler Alternatives

Because multiple reviewers asked a similar question, we addressed this in-depth in (2) of our summary comment, so please see this above. In short, our harness acts similar to value-based gating in one-step environments (e.g., Code Merging or Online Commerce). However, because in multi-step environments (e.g., Web Navigation) the harness should use information about the advisor agent from previous states to inform its estimate in the current state, we track λ\lambda over multiple steps, rather than simply gating it at each step. We demonstrate this with an additional experiment in the Appendix (see Figure 12).

Additionally, our harness does not provide much benevolent improvement for models, as the relative τ\tau of most frontier models is already quite high. However, for weaker models, where listening to advice is more beneficial, our harness improves their performance under benevolent influence. To verify this, we ran an additional experiment with several simpler models, and we find that the vigilant harness improves the performance to ceiling under benevolent influence (see Figure 9).

W3: Generalizability of Experiments

We selected our three environments as they cover a wide variety of common agentic tasks, such as web crawling, purchasing items, and coding. Within each of these environments, we additionally offer a wide variety of variance, such as a breadth of potential dangers within Web Navigation (see Figure 14 for examples) and different scams within Online Commerce (see Figure 15 for examples). Despite this variance, we find very consistent behavioural patterns from all 6 models across these environments. This suggests that our environments are a sufficient testbed for understanding how these behaviours generalize to other domains (as further supported by our supplementary environment in Appendix B, which varies even more from the existing tasks, yet sees similar results).

Additionally, we selected GPT-5 as the advisor agent as we broadly observed that the choice of advisor agent, for all sufficiently capable LLMs, made little impact on the results (i.e., all frontier LLMs are similarly persuasive). To empirically verify this, we ran an additional experiment in the Appendix with two other advisors, Qwen3.8-27B and Claude Opus 4.8, and found little to no consistent difference across advisors (see Figure 7).

W4: Documenting Experiments

For Grok’s 58.3% (7/12) solve rate in the Web Navigation tasks, this is because we ran each model over 3 trials in the baseline case to better calibrate our estimate of τ\tau. We agree this was not well communicated and have updated the manuscript.

For the standard deviation reported in Table 1, this is a typo between 0.016 and 0.106, and we have updated this.

Thank you for pointing these out to us.

W5: AI Usage

Because another reviewer mentioned this, we also addressed this in-depth in (1) of our summary comment, so please see this above. In short, we did not use AI to generate or assist in the generation of any figures in the paper, or assist with writing or editing the manuscript. We included a link to the Figma file where the first author created all the figures, and we encourage you to check it out. Additionally, all the authors manually wrote and edited the manuscript in a shared Overleaf. Besides this, it would help us if you could elaborate on what other parts of the paper require polish.

Q1: Code Availability

We are happy to and plan to make all code public after the double-blind review process is complete, such as the ground truth labels in the Code Merging dataset.

Thank you for your detailed responses and additional experiments. I appreciate the authors' efforts. While some of my concerns have been addressed, others remain insufficiently resolved.

The overall presentation and writing quality also remain below the standard expected for an accepted paper, particularly in the clarity and organization of the writing, as well as the quality, consistency, and readability of figures and tables. I encourage the authors to refer to exemplary research papers and further improve the manuscript's overall quality in future revisions.

I will maintain my original score.

AuthorsOpenReview ↗

Please clarify what writing can be improved?

Can you please refer us to a paper that you consider to have "exemplary" writing and readability? We have extensively checked the writing and readability of the paper, and it is well within the standards that we have seen in other papers accepted at ICLR.

Reviewer_VrrCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Under the Influence studies whether LLM agents can use helpful advice while resisting harmful persuasion. Six models are evaluated on web navigation, online commerce and code merging. The paper compares task choices with explicit action-value estimates. Its vigilance harness makes three calls in separate contexts to extract the advised action, estimate its value and produce an unadvised action. A Bayesian update of advisor utility, combined with estimated baseline capability, determines the final choice. An awareness prompt provides a simpler comparator.

Critical strengths
  1. Sections 3–4.2 and Figure 2 show substantial susceptibility across six models and 24 tasks: four web-navigation tasks and ten in each other environment. Malicious advice lowers success by 40.6 percentage points on average, while benevolent advice raises it by 19.9 points. Measuring both directions makes the cost of rejecting useful advice visible. I find this comparison informative because the same models often solve these problems unaided.

  2. Section 4.3 and Figure 3 show that explicit valuation distinguishes harmful from helpful advised actions on average: mean values are −0.473 and +0.537. This separates the capacity to evaluate advice when asked from the choices agents make under influence. The harness also works as a complete package: Section 5.3 and Figure 5 report recovery of 36.4 points under malicious advice, versus 18.1 for awareness prompting. In code merging, the gains are 22.5 and 0.8 points respectively. Changing the decision procedure produces a substantial behavioral improvement here.

Critical weaknesses
  • The naive-trust explanation goes beyond what the evaluation establishes. Figure 3 shows aggregate separation between value estimates under helpful and harmful advice. It does not establish that agents correctly recognize the particular harmful actions they subsequently follow. Sections 6–7 nevertheless treat naive trust, rather than inability to evaluate advice, as the explanation. Prompting for explicit evaluation and separating contexts also change the reasoning procedure. The current evidence does not isolate trust from these alternatives.

  • The intervention comparison does not identify the contribution of Bayesian advisor tracking. Section 5.1 combines advice extraction, value estimation, and independent action selection with the advisor-utility update. Section 5.3 principally compares this package with awareness prompting. That comparison cannot establish whether history-dependent tracking adds value beyond a simple value gate or independent reasoning with the same elicited outputs.

  • The selective-helpfulness claim is weaker than the malicious-condition recovery suggests. Section 5.3 reports a 17.2-point loss under benevolent advice. Combining the reported aggregate means leaves approximately 2.7 points above the unadvised baseline under benevolent advice, while malicious performance remains approximately 4.2 points below that baseline. These are approximate aggregate comparisons, not new significance tests. They nevertheless make ignoring advice an important reference point. The main results show strong recovery from harmful influence, but much less evidence of preserving the useful influence that motivates the problem.

Questions for authors
  • Were the value probes applied to the same states and advised actions as the choice trials, with the same available information? Do the existing records show harmful-action recognition specifically on decisions where the agent nevertheless complied?

  • Using the elicited actions and values, can the effect of the advisor-history update be separated from simple value gating? A comparison under the same query procedure would help determine what the Bayesian component contributes beyond returning toward unadvised behavior.

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your feedback! We are happy to hear you found our bidirectional comparative approach to measuring vigilance informative, and that you appreciated the “substantial behavioural improvement” our vigilance harness offers in Code Merging. We addressed your comments below, but please let us know if you have any further questions or comments.

W1 & Q1: Naive-Trust Explanation / Value Probe Consistency

We compare LLMs’ responses on the exact same states and with the same available information in our experiment where agents receive advice from an external advisor and must choose an action (Section 4.2) to our experiment where LLM’s estimate the value of each possible action (Section 4.3 / Figure 3). Thus, we find no gap between these two, as our existing data shows that LLMs persuaded to take a harmful action can simultaneously recognize this same action as dangerous.

W2 / W3 / Q2: Harness Improvement vs. Ignoring Advice or Value Gating

Because another reviewer mentioned this, we also addressed this in-depth in (2) of our summary comment, so please see this above. In comparison to ignoring advice, because our experiments were conducted with highly capable frontier LLMs that generally have a high τ\tau (estimated capability within the environment), the harness intuitively finds that there is more to gain from avoiding dangerous advice than from listening to helpful advice. However, for less-capable LLMs, the inverse is true, where there is more to gain from listening to helpful advice than avoiding dangerous advice. Our harness incorporates this logic, whereas ignoring advice would not resolve the latter, and less-capable LLMs would not be able to incorporate helpful advice. To verify this, we ran an additional experiment with several simpler models, and we find that the vigilant harness improves their performance to ceiling under benevolent influence (see Figure 9).

In comparison to value-based gating, our harness acts similar to value-based gating in one-step environments (e.g., Code Merging or Online Commerce). However, because in multi-step environments (e.g., Web Navigation) the harness should use information about the advisor agent from previous states to inform its estimate in the current state, we track λ\lambda over multiple steps, rather than simply gating it at each step. We demonstrate this with an additional experiment in the Appendix (see Figure 12). We would also like to note that even demonstrating value-based gating as an intervention is a novel contribution, since this still suggests a dissociation between an agent’s ability to assess the quality of a suggested action from its likelihood of following a piece of advice directly.

Reviewer_vorHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This work is motivated by the recent incident where OpenAI models (in principle intended to be isolated from the Internet) escaped the sandbox environement and compromised HuggingFace infrastructure. The paper aims to test how agentic systems behave when advice is given in three environements (web navigation, online commerce, code merging). The authors find that agents are "vulnerable to malicious advice from advisor" (using six models listed in the paper). The paper proposes "vigilance harness", with the goal of improving resistance to malicious influence.

Critical strengths
  1. This is definitely a very timely and important problem.
  2. Not much has been done in this areas and the research community need to step up work in this space.
  3. Nice strucutre of the paper. It was fairly easy to read it and follow the reasoning, which is a massive plus.
Critical weaknesses
  1. I understand that the harness improves the performance when it is paired with malicious advice. But does it add substantial benefits over just ignoring what advisor says? This is somewhat unclear to me.
  2. The usual critique perhaps is about the size fo the experiments in this work. The experiments seem to cover 24 tasks in 3 environments, related to web navigation (4 tasks), online commerce (10 tasks) and code merging (10 tasks). How much can we generalise based on these experiments (especially that failures are due to inappropriate trust)?

This is not a critical weakness. Although the paper is very easy to read and generally well written, the figures seem AI generate and low quality. Consequently it significantly lowered the experience of reading the paper.

Questions for authors

A rather obvious question of how robust are these results? If you were going to choose a different advisor and perhaps large set of tasks? But also what I mentioned earlier how much do the results change if instead of harness we would just ingnore the advisor?

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for your feedback! We are happy to hear you thought the problem we are investigating is timely and important, and that the structure of our paper made it easy to read and follow our reasoning. We addressed your comments below, but please let us know if you have any further questions or comments.

W1: Harness Improvement vs. Ignoring Advice

Because multiple reviewers asked a similar question, we addressed this in-depth in (2) of our summary comment, so please see this above. In short, you are correct that with reasonably strong LLMs (such as the ones used in our paper), the harness offers little advantage as opposed to ignoring advice, since these LLMs are already high performers. However, for less-capable LLMs (i.e., those with a lower estimated τ\tau), the harness instead favors listening to the advice, allowing less-capable LLMs to improve under benevolent advice (and with little to lose under malicious advice). A simple rule that ignores all advice would not allow for this improvement. We demonstrate this with an additional experiment in the Appendix (see Figure 12).

W2: Experiment Size / Generalizability

We selected our three environments as they cover a wide variety of common agentic tasks, such as web crawling, purchasing items, and coding. Within each of these environments, we additionally offer a wide variety of variance, such as a breadth of potential dangers within Web Navigation (see Figure 14 for examples) and different scams within Online Commerce (see Figure 15 for examples). Despite this variance, we find very consistent behavioural patterns from all 6 models across these environments. This suggests that our environments are a sufficient testbed for understanding how these behaviours generalize to other domains (as further supported by our supplementary environment in Appendix B, which varies even more from the existing tasks, yet sees similar results). We agree with you that it is important to understand the generalizability of our results, but we believe our experimental setup accomplishes this sufficiently.

W3: AI Usage

Because another reviewer mentioned this, we also addressed this in-depth in (1) of our summary comment, so please see this above. In short, we did not use AI to generate or assist in the generation of any figures in the paper. We included a link to the Figma file where the first author created all the figures, and we encourage you to check it out. Hopefully knowing this helps you appreciate the paper more, and we are happy to further discuss this!

Q1: Advisor Choice

We selected GPT-5 as the advisor agent as we broadly observed that the choice of advisor agent, for all sufficiently capable LLMs, made little impact on the results (i.e., all frontier LLMs are similarly persuasive). To empirically verify this, we ran an additional experiment in the Appendix with two other advisors, Qwen3.8-27B and Claude Opus 4.8, and found little to no consistent difference across advisors (see Figure 7).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for all Reviewers

We thank all reviewers for their thoughtful feedback and constructive criticism. We are happy to hear that reviewers found our work “timely” and “easy to read” (vorH), found our bidirectional comparative approach to measuring vigilance “informative” (VrrC), and found our results “provide[d] useful insights into how external advice affects agent performance across different scenarios” (9UQW).

Based on the reviews, we identified two major themes: questions about AI-usage for figure creation and manuscript writing, and the advantages of the vigilance harness as opposed to simpler methods.

1. AI-usage for Figure Creation and Manuscript Writing (Response to vorH, 9UQW):

We did not use AI to generate or assist in generation of any figures in the paper, or assist with writing or editing the manuscript. Every figure was made by the first author in Figma, and all the authors manually wrote and edited the manuscript in a shared Overleaf. AI was only used for (1) generating the Code Merging dataset, (2) assisting with coding, and (3) formatting references, as reported in our AI use statement. For complete transparency, we have attached a link to the Figma document used to create every figure, and we are more than happy to answer any other questions related to our usage of AI in the creation of the paper. We would like to further understand why some reviewers believed our figures and other parts of the manuscript appeared to be AI-generated.

Figma: https://www.figma.com/design/PN0p9FPpdekIXIcaN6Ajpv/Persuasion-Research-2025?node-id=0-1&t=Igrh7cc1LuDVuIoZ-1

2. Advantages of the Vigilance Harness (Responses to vorH, VrrC, 9UQW):

Several reviewers asked what advantage the vigilance harness offers over simply ignoring advice (vorH, VrrC), or other easier methods such as value-based gating (VrrC, 9UQW) that could replace Bayesian advisor tracking / trust updating.

The primary advantage of the vigilance harness is that the policy function (Equation 4) used to select an action works partially as a function of τ\tau: the LLM’s estimated capability within the environment. Because our experiments focused on highly capable frontier LLMs, their τ\tau estimates are relatively high, and the LLMs already solve a large fraction of puzzles (67.2% to 74.4% on average). As such, the policy function intuitively deems less relative benefit from listening to benevolent advice, as opposed to ignoring malicious advice, as there is more to lose than to gain. However, for less capable LLMs, where τ\tau is much lower, the policy function appropriately works in reverse, where there is more to gain from listening to benevolent advice, as opposed to ignoring malicious advice. To verify this, we ran an additional experiment with several simpler models, and we find that the vigilant harness improves the performance to ceiling under benevolent influence (see Figure 9).

Furthermore, if the LLMs were to simply ignore the advice, while we would see improved malicious performance, less capable LLMs would see no improvement in the benevolent setting, which the vigilance harness fixes. Additionally, within one-step environments (e.g., Code Merging and Online Commerce), our metrics work similar to a value-based gate. However, because in multi-step environments (e.g., Web Navigation) information about the advisor from the previous state should inform the current state, our model needs to track λ\lambda over multiple steps, rather than simply gating it at each step. We included an additional figure in the Appendix (see Figure 12) that demonstrates one such case, where using value-based gating would decrease performance.

We hope that this overview of the most common concerns answers your questions, and we are happy to answer any other questions from the reviewers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.