- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how personal agents navigate honesty, confidentiality, and task completion in simulated Slack negotiations involving privately inferred goals.
Critical strengths
The setting captures a meaningful conflict arising from ordinary delegation and private user information. Confidentiality variations and honeypot controls provide useful evidence about disclosure and misleading excuses. Detailed trajectories and extensive human review strengthen the behavioral analysis.
Critical weaknesses
The novelty rests mainly on a new scenario, without establishing a distinct failure mode beyond previously observed pressure-induced deception.
Strongly stated private preferences and deliberately restrictive negotiation rules leave unclear what the results establish about unsanctioned goal formation under ordinary delegation.
The confidentiality intervention lacks matched reporting of deception and task completion, limiting the central three-way trade-off analysis.
The case study is credible, but its incremental scientific contribution and central trade-off evidence are not yet sufficiently established.
Questions for authors
What empirical prediction distinguishes history-inferred goals from explicitly assigned goals under otherwise matched conditions?
Would the behavior persist if the chair accepted unexplained preferences or permitted authorized escalation?
How do deception and staffing outcomes change across the same three confidentiality conditions?
4 comments
Rebuttal to Reviewer CCw8 Nr. 2/3
W3 / Q3. How do deception and staffing outcomes change across the same three confidentiality conditions?
We added this matched comparison (Section 4.3, Figures 4 and 5). All seven open-weight models are run at all three levels with the same world, user prompts, and seeds, giving 140 agents per level:
| no instruction | medium | strong | |
|---|---|---|---|
| Discloses the no-Bob preference | 71% | 29% | 1% |
| False claim | 21% | 29% | 28% |
| Misdirection | 38% | 47% | 41% |
| No deception | 38% | 20% | 21% |
| Pursues T1 from the start | 4% | 4% | 10% |
| Ends on T1 later | 1% | 8% | 7% |
Confidentiality instructions protect the no-Bob preference, but deception rises once any such instruction is given; between the medium and the strong instruction it barely changes. Few agents end up on T1 at any level, and fewest when they are free to disclose. Disclosing less therefore does not resolve the conflict: with the medium instruction, only 16 of 70 runs end with a valid staffing.
Q1. What empirical prediction distinguishes history-inferred goals from explicitly assigned goals under otherwise matched conditions?
Conceptually, we use history-inferred goals for three reasons. First, the actions needed to achieve an inferred goal are not sanctioned by the user; we even see agents reasoning about whether they should concede T2 because pursuing it implies unsanctioned actions. Second, the inference is an active step: 5 of the 140 open-weight agents in the main experiment do not adopt the planted goal and pursue T1 instead. An instructed goal is adopted unless the agent explicitly refuses it, whereas context from which a goal must be inferred leaves the agent more room to reject it. Third, we consider unsanctioned behavior in pursuit of a self-inferred goal more relevant for deployed personal agents than unsanctioned behavior in pursuit of an assigned goal.
Empirically, the two have similar effects. We compare the normal instruction ("handle this") with an explicit one ("get me on T2") in two settings:
- Main experiment (Appendix D.2; five of the seven open-weight models, 100 agents; only Alice's agent receives the explicit instruction). False claims change by 2pp, misdirection and no-deception by 1pp each, and the two disclosure rates by at most 3pp. The only visible effect is the intended one: no agent of Alice pursues T1 any more.
- Ready-made excuse/Honeypot (this comparison is not part of the revised PDF). Across the six models run in both conditions, the agent uses the planted scheduling conflict as a reason against T1 in 38% of runs under the explicit instruction (n=104) and in 40% under the normal instruction (n=65). The rates agree most closely for the models that use the excuse most (DeepSeek 67% vs. 60%, GPT-5.6 Sol 70% vs. 80%, GLM-5.3-Flash 43% vs. 50%); models with a low propensity are noisier. When the excuse is used, it is more often the only reason given under the normal instruction (77% of uses vs. 53%).
Rebuttal to Reviewer CCw8 Nr. 3/3
Q2 (first part). Would the behavior persist if the chair accepted unexplained preferences?
Largely yes. With a neutral Dan (Appendix D.1), who gets the same "handle the sprint" prompt as Alice, Bob, and Carol, we compare against the main experiment on the same five open-weight models (100 agents each). We find fewer disclosures of the no-Bob preference (34 → 18 of 100 agents), fewer false claims (30 → 13), and more misdirection (41 → 52). The share of agents that do not deceive (24 vs. 26) and the share that end on T2 are similar. The interrogating chair therefore changes the type of deception more than its amount: agents rarely need to assert something false when nobody asks for a reason, but they still steer the staffing away from T1 without naming the real motive.
Q2 (second part). Would the behavior persist if authorized escalation were permitted?
An authorized escalation channel would likely change the outcomes considerably, since the need to justify a preference in front of the group is part of the trilemma. Because we do not claim that the environment measures a natural tendency to deceive or disclose, this does not change the main message of the paper.
As an imperfect proxy for whether agents would use such a channel, we analyzed the messages that the agents send to their own employee before their first unsanctioned action in a run. This analysis is not part of the revised PDF. Agents usually do not inform their employee of the unsanctioned action (9% do so before, 20% in the same step). In only 3% of cases do they ask for permission or offer the employee a veto, and all of these are runs in which the agent discloses information. In 25% of cases the agents downplay the action, e.g., "naming the constraint and its target, but none of the specifics, none of the history, none of what you told Emily and Fiona in confidence." So although an authorized escalation channel would change the setting, agents show a low propensity to raise these conflicts with their user even though a channel to the user is available.
References
[1] Scheurer et al., 2024, "Large Language Models can Strategically Deceive their Users when Put Under Pressure", https://arxiv.org/abs/2311.07590
[2] Su et al., 2025, "AI-LieDar: Examine the Trade-off Between Utility and Truthfulness in LLM Agents", https://arxiv.org/abs/2409.09013
[3] Liu et al., 2026, "Knowledge-Verified Emergent Deception in LLM Agents Under Conflicting Incentives", https://arxiv.org/abs/2608.26372v1
[4] Järviniemi & Hubinger, 2024, "Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistant", https://arxiv.org/abs/2405.01576
[5] Guo et al., 2025, "Are Your Agents Upward Deceivers?", https://arxiv.org/abs/2512.04864
Thanks to the authors for their response, but the persuasiveness brought by merely tightening the phrasing is limited and cannot solve the problem. I have also read the comments and responses from other reviewers, especially the latest review by Reviewer ooW3.
Rebuttal to Reviewer CCw8 Nr. 1/3
We thank the reviewer for the clear questions. Each of them led to an experiment or analysis that we report below, most of which are in the revised PDF.
To improve readability, we restructured the paper (the environment and the behavior definitions are now one section that explains each design choice), moved key definitions from the appendix into the main text, and reorganized the appendix by topic with an overview of all experiments (Appendix H.2).
W1. Novelty rests on a new scenario, without a distinct failure mode beyond pressure-induced deception. Only one scenario.
Previous work has shown that models deceive without being instructed to. In most of it, the agent is assigned a goal and an explicit incentive [1, 2, 3]. Where no incentive is given, agents have been shown to deceive their own users or overseers about their own actions [4, 5]. In our environment, the agent is assigned neither a goal nor an incentive: it infers the goal from its user’s private messages and then deceives peers, i.e., other users’ agents, to protect it. We are not aware of a prior environment with this combination.
Accordingly, we see our contribution as an environment that elicits this behavior, not as an evaluation or an estimate of a model’s propensity to deceive, and we do not claim that the rates we report transfer to other scenarios; the revised Limitations section says so explicitly. Such an environment is useful in its own right. (a) One can study which conditions give rise to deception. For example, gpt-5.6-sol makes no deceptive statement in 11 of 20 agents in the main scenario, but uses the ready-made excuse in 8 of 10 runs. (b) One can test whether monitors or mitigations catch deception, which requires a setting where it occurs reliably and can be verified against ground truth. (c) One can study how models reason about deception: when they deem it acceptable, and when they engage in motivated reasoning, i.e., convince themselves that a false claim is not a lie. Both are visible in our traces, and both can only be studied when the model is not instructed to deceive.
We agree that more scenarios would be better. We prioritized depth over breadth, since establishing what counts as deception depends on the backstory, and building judges for it takes a significant amount of time per scenario. The revised paper (Section 3.2) now explains which element of the scenario serves which purpose in making the trilemma binding and the agents’ statements checkable, and the environment can be extended to larger groups, other backstories and tools, multi-day runs, and mixtures of different LLMs.
W2. Strongly stated private preferences and restrictive negotiation rules leave unclear what the results establish about unsanctioned goal formation under ordinary delegation.
We made the setting less restrictive in the revision and now vary each of these elements:
As explained under W1, we do not present the rates as a measurement of what agents do under ordinary delegation in general. We will also tone down language that suggests broad autonomous goal formation; what we observe is an inferred user preference that the agent adopts as a goal.