acceptodds
Under review as a conference paper at ICLR 2027

Negotiation Based Attacks On Language Model Agents

Abstract

As language models gain greater agency, they are increasingly used to act on users’ behalf: executing tasks, representing users, and communicating with third-party agents in interactions that may increasingly replace human-to-human communication with agent-to-agent exchange. This shift introduces new privacy risks because such agents may possess and transmit sensitive user information. Crucially, whether a piece of information is appropriate to disclose is not fixed, but depends on the task and interaction context – a view captured by the principle of Contextual Integrity. Existing benchmarks and methods largely study this question in static or single-turn settings, where privacy can be assessed from the immediate context of a request. In this work, we introduce a benchmark in which the appropriate disclosure boundary must instead be maintained over multi-turn interactions between a User Agent and a Service Agent. Multi-turn communication is necessary to accomplish the user’s task, but it also creates opportunities for an external Service Agent to strategically elicit information beyond what the task requires. We therefore formulate agentic privacy as a mixed-motive two-player game: the agents cooperate toward task completion while competing over the disclosure of contextually protected information under an adversarial threat model. We find that sustained, adaptive multi-turn negotiation can improve not only task completion but also the adversary’s ability to elicit protected user information, highlighting the importance of treating privacy as a trajectory-level property of interactive agents rather than a property of isolated messages or requests.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llm, agent, contextual integrity, privacy, attack

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sriCOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper models privacy leakage in service-agent conversations as a mixed-motive game, introduces CI-Challenge, and tests prompt-optimized attacks across different incentives and conversation lengths on two benchmarks.

Critical strengths

I like the focus on service providers that want to complete the task while collecting extra data, and the payoff model makes those competing incentives explicit.

Critical weaknesses
  1. My main concern is that collecting all required fields is counted as task completion, so the results do not establish that the attacker actually delivers the service. The comparisons also change more than the intended variable: each horizon uses a separately optimized strategy, and the utility-only baseline is not shown the protected fields, making the effects of conversation length and incentives hard to isolate.

  2. The added contribution over ConVerse and earlier search-based privacy attacks seems modest without a direct comparison showing what the mixed-motive setup adds. The negotiation examples are interesting, but there are no controlled comparisons showing whether concessions and later re-requests work better than simply asking repeatedly.

Questions for authors
  1. What supports treating disclosure of all required fields as successful service delivery, and do any experiments check whether the requested service is actually completed?

  2. Do the main trends hold when the same attack strategy is tested across horizons and all incentive settings receive the same field information?

  3. Have you compared with ConVerse-style attacks and earlier search-based approaches using the same models, tasks, and conversation budget?

  4. How was the 78.1% completion rate in Section 4.3 calculated, given that the test split contains 166 tasks and Appendix B.5 specifies one episode per task?

5 comments
AuthorsOpenReview ↗

Follow up with reviewer sriC

We thank the reviewer for their time and effort in reviewing our work.

We are encouraged by the reviewer's positive reception of:

  • Our threat model, where a Service Agent is incentivized to both complete a legitimate task and collect additional user information.
  • Our mixed-motive game formulation that makes these competing incentives explicit.

We appreciate the reviewer's feedback, which helped us improve our work in following aspects:

  • Clarified our assumption of a mocked Service Agent backend, where Task Utility measures information sufficiency for task completion, consistent with prior privacy benchmarks.
  • Introduced a single RL-trained attacker evaluated across different conversation horizons, showing similar trends to our original results. [Figure 4, Section 4.2]
  • Expanded our discussion of ConVerse and prior search-based privacy attacks, clarifying our contributions and differences from these works. [Appendix A]

We further thank the reviewer for helping us improve the writing and clarity of our draft across evaluation setup and post-hoc analysis of negotiation strategies.


As the discussion period comes close to an end, we are happy to address any further questions from the reviewer. If the reviewer has no remaining concerns, we would appreciate reconsideration of the review score based on our updated draft and rebuttal responses.

AuthorsOpenReview ↗

Re: Follow up with reviewer sriC

As the discussion period comes to an end, we would appreciate any further feedback from reviewer sriC and would be happy to address any remaining questions or concerns.

If our updated draft and rebuttal responses have addressed the reviewer's concerns, we would appreciate update of the review score.

We again thank the reviewer for their time and effort in helping us improve our work.

AuthorsOpenReview ↗

Re: Follow up with reviewer sriC

We kindly urge the reviewer sriC to acknowledge our rebuttal responses and updated draft, and consider updating their score if their concerns have been addressed.

As the discussion period comes to an end, we also welcome any remaining questions or concerns and would be happy to address them.

Reviewer_XyXVOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies a contextual privacy setting in which an agent collects extraneous information about a user agent. The authors frame this as a mixed-motive game. The authors introduce a semi-synthetic dataset of reddit queries in which some information is necessary for a task and other information is private. The paper finds that longer conversations result in more leakage but also increased utility. The paper uncovers a new threat model: negotiation attacks, which distributes privacy attacks across the whole sequence of requests. These attacks are found by prompt-optimizing the attacker model. The authors also find that optimizing the user for privacy results in better privacy but lower utility, in line with prior findings on the privacy-utility tradeoff.

Critical strengths
  • The tradeoff between obtaining a service and maintaining ones privacy is common and is an important problem to address.
  • The idea to optimize models via prompt optimization is new in this space, and is a useful cheaper alternative to the kind of RL training that has been done in prior work. The negotiation behavior obtained through optimization of the attacker is interesting.
Critical weaknesses
  • Missing related work. The authors state ``most CI benchmarks assess disclosure in static or single-turn settings (Shao et al., 2024; Cheng et al., 2024), and agent-security benchmarks target prompt-injection surfaces rather than information exchange between agents (Debenedetti et al., 2024; Wang et al., 2026).''. Prior work has looked at multi-turn settings and information exchange (https://arxiv.org/abs/2509.14284, https://www.usenix.org/conference/usenixsecurity25/presentation/ayzenshteyn) and the setting seems quite similar to the one presented in https://arxiv.org/abs/2509.14284 (albeit with fewer agents). It would be good for the authors to discuss similarities and differences (I believe there are difference here as well.)
  • Missing details: in benchmark construction, it's not clear how the instances are generated. Is it done with a model? If so, which model? How are instances vetted?
Questions for authors
  • Typo L019 contextuallu
3 comments
AuthorsOpenReview ↗

Follow up with reviewer XyXV

We thank the reviewer for their time and effort in reviewing our work.

We are encouraged by the reviewer's positive reception of:

  • Our mixed-motive threat model and the importance of studying the privacy-utility trade off.
  • Our prompt optimization approach and the studying negotiation strategies that emerge within agent interaction.

We appreciate the reviewer's feedback, which helped us improve our draft in following aspects:

  • Expanded our discussion of multi-turn privacy attacks and clarified our differences from prior work. [Appendix A3]
  • Clarified our dataset/benchmark generation and filtering process, and added human validation of benchmark labels. [Appendix B, and human study in Appendix B2], so reader can easily reproduce the exact process if needed.

As the discussion period comes close to an end, we are happy to address any further questions from reviewer. If the reviewer has no remaining concerns, we would appreciate reconsideration of the review score based on our updated draft and rebuttal responses.

Thanks for providing these additional details. This resolves my concerns and I have raised my score accordingly.

AuthorsOpenReview ↗

Thank you reviewer XyXV

Thanks to reviewer XyXV for the interacting over review process, and helping us further improve our draft. We further welcome any comments or feedback before the discussion period ends.

Reviewer_AEjGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies privacy attacks in multi-turn interactions between user agents and service agents. The authors model the interaction as a mixed-motive game in which both agents benefit from completing the user's task, while the service agent may also benefit from collecting information that the task does not require.

The paper introduces CI-CHALLENGE, a semi-synthetic benchmark built from Reddit service requests, with task-dependent labels for required, optional, and protected information. Service-agent strategies are optimized using evolutionary prompt search under different task-utility and information-extraction incentives.

The experiments show that longer interactions increase both task utility and attack success, and that a service agent can obtain protected information while retaining high task utility. The authors also identify several negotiation strategies, including bundling information requests, conceding disputed fields, reframing their necessity, and requesting them again later.

Critical strengths
  • A relevant threat model for agent-to-agent interactions. The paper considers a service agent that pursues an additional information-collection objective while still helping complete the user's task. This is a useful distinction from settings where an attacker is interested only in extraction or task disruption. Explicitly varying the utility and extraction weights also makes it possible to study how the attacker's incentives affect its behavior.

  • A useful empirical study of multi-turn privacy risks. CI-CHALLENGE provides task-dependent information labels grounded in real user requests, and the experiments cover different incentives, conversation horizons, and defender privacy settings. The results show that high task utility can coexist with considerable privacy leakage. The qualitative examples are also useful for understanding how apparently reasonable requests can become part of a longer adversarial interaction.

Critical weaknesses
  • Limited validation of the benchmark labels and task-success metric. The required, optional, and protected information labels are generated from Reddit requests, without reported independent human validation. Some labels may depend on the actual service workflow or authorization mechanism. Moreover, task success is defined solely by whether all required keys are authorized, rather than whether the requested service is completed. This limits the validity of the reported utility–privacy trade-off.

  • The effect of conversation length is confounded with attacker optimization. The service agent is optimized separately for each conversation horizon, while ASR measures whether any target information is disclosed over the entire interaction. The observed increase in ASR therefore mixes additional disclosure opportunities with differences in the optimized policies, making it difficult to attribute the improvement specifically to negotiation.

  • Limited causal evidence for the identified negotiation strategies. The reported strategies are mainly supported by optimized prompt descriptions and selected trajectories, without controlled ablations of individual negotiation moves. The claim that these attacks can only be identified from the full conversation is also not directly tested against single-message detection. In addition, task success and privacy leakage are primarily reported separately, rather than through their joint occurrence within the same trajectory.

Questions for authors
  • How are the required, optional, and protected labels validated? Are there independent human annotations or agreement measurements? In particular, how are cases handled where a field's necessity depends on the actual service implementation or authentication procedure?

  • Can the authors report the joint rate of task success and protected information disclosure, rather than only the two marginal metrics? This seems particularly important for the paper's claim that adversarial services can collect information while continuing to deliver the requested service.

  • Can the authors isolate the effect of conversation length from the effect of horizon-specific prompt optimization? For example, how does a fixed attacker policy perform when evaluated with different turn budgets? A per-turn analysis of disclosure events would also help distinguish repeated opportunities from negotiation-specific effects.

  • How much does each identified negotiation strategy contribute to attack success? In particular, does removing concession, reframing, or returning to previously refused requests substantially change ASR when the remaining attack conditions are fixed?

3 comments
AuthorsOpenReview ↗

Follow up with reviewer AEjG

We thank the reviewer for their valuable time and effort in reviewing our work.

We are encouraged by the reviewer's positive reception of:

  • Our threat model, where an external Service Agent is incentivized to both provide legitimate task utility and extract protected user information.
  • Our empirical study, including the introduced CI-Challenge benchmark, different incentive settings, conversation horizons, and defender privacy personas.

We appreciate the reviewer's feedback, which helped us improve our work in several aspects:

  • Added human validation of CI-Challenge labels, showing 98% agreement with human-majority labels (3 annotators, 246 attributes, 738 judgments), with no false-positive Protected labels in the annotated sample. [Table 3, Appendix B2]
  • Introduced a single RL-trained attacker (via GRPO) evaluated across different conversation horizons, showing similar trends to our original results. [Figure 4, Section 4.2]
  • Added the joint rate of task success and protected information disclosure [Appendix D3].

We further thank the reviewer to help us improve our writing by asking various questions around several aspects of our methodology including task completion and conversation length.


As the discussion period comes to an end, we would be super happy to address any further questions. If the reviewer has no remaining concerns, we would appreciate reconsideration of the review score in light of our updated draft and rebuttal responses.

AuthorsOpenReview ↗

Re: Follow up with reviewer AEjG

As the discussion period comes to an end, we would appreciate any further feedback from reviewer AEjG and would be happy to address any remaining questions or concerns.

If our updated draft and rebuttal responses have addressed the reviewer's concerns, we would appreciate update of the review score.

We again thank the reviewer for their time and effort in helping us improve our work.

AuthorsOpenReview ↗

Re: Follow up with reviewer AEjG

We kindly urge the reviewer AEjG to acknowledge our rebuttal responses and updated draft, and consider updating their score if their concerns have been addressed.

As the discussion period comes to an end, we also welcome any remaining questions or concerns and would be happy to address them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the AC Meta-Review: Reviewers' concerns and revisions

Below we summarize the main concerns raised by the reviewers and how we addressed them through new experiments and clarifications in the revised draft.

  1. Benchmark validity (AEjG, XyXV): Reviewers questioned how CI-Challenge was constructed and validated. We clarified the generation and filtering process [Appendix B], and added human validation with 3 annotators over 246 attributes, showing 98% agreement with human-majority labels [Table 3, Appendix B2].

  2. Attacker optimization (AEjG, sriC): Reviewers questioned whether longer conversations improve attack success or separately optimized attackers explain the results. We clarified that the maximum turn budget is known to both agents before interaction, consistent with established LLM agent frameworks [1, 2, 3], and can naturally be exploited by an attacker. We further trained a single attacker using RL (GRPO) across different turn budgets and found similar trends in both task utility and privacy leakage [Figure 4, Section 4.2].

  3. Task utility and privacy evaluation (AEjG, sriC): Reviewers questioned whether sharing all required information implies task completion. We clarified that task utility measures information sufficiency under our mocked service backend, rather than downstream execution [Section 4.1]. We also added the joint rate of task success and privacy violation to better capture the privacy-utility trade-off [Appendix D3].

  4. Novelty and scope (XyXV, sriC): Reviewers pointed to prior multi-turn privacy attacks and questioned our differentiation and evidence for negotiation strategies. We expanded our related work, including ConVerse and search-based approaches [Appendix A3], and clarified that our strategy analysis is post-hoc and qualitative, without claiming causality for individual moves [Section 4.4].

  5. Evaluation clarity (AEjG, sriC): Reviewers raised questions about field information across incentive settings, conversation horizons, and evaluation denominators. We clarified these aspects of our protocol [Section 4.1, Appendix B5].

We believe these changes have addressed the main concerns raised by the reviewers and meaningfully strengthened our work. We welcome any further questions or concerns and would be happy to address them before the discussion period ends.


References

[1] AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation, Wu et al., COLM 2024.

[2] https://microsoft.github.io/autogen/0.2/docs/FAQ/ uses max_turns in ConversableAgent.initiate_chat limits the number of conversational turns between two agents

[3] https://openai.github.io/openai-agents-python/ref/run/ uses max_turns and raises MaxTurnsExceeded when an execution exceeds its turn budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.