acceptodds
Under review as a conference paper at ICLR 2027

Do Agents Keep Their Word? Promise-Breaking in Multi-Agent LLMs

Abstract

Monitoring what large language model (LLM) agents say they will do is an active field of alignment research for multi-agent systems. However, we know little about whether the agents keep their promise. We evaluate ten frontier models on six social-dilemma games with two protocols. In the exogenous protocol the agent is assigned a public commitment under seven conditions, and we compare its deviation rate with the rate it would reach by ignoring the commitment. In the endogenous protocol the agent privately plans, publicly announces its own commitment, and then acts. We find that promise-breaking depends on the model and the game together, with their interaction accounting for 48% of the variance across model–game cells, twice as much as the model alone. A single instruction to honor public statements reduces deviation from an assigned statement to under 2% on binary games, but self-made promises are still broken eight times as often. Declaring announcements binding has a similar effect, while declaring them cheap talk changes little, which suggests that models treat announcements as non-binding unless told otherwise. In repeated play, promise-breaking falls after the first round and individual models then diverge: over twenty-five rounds, one stops breaking entirely and another breaks more each round. Furthermore, when an agent breaks a promise, its reasoning usually does not mention the announcement. We score this with four LLM judges validated against three human annotators and find no clear relation between a model's awareness and how often it breaks its promises. These results suggest that multi-agent systems should state explicitly that announcements are to be honored, and should still expect some promises to be broken.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
multi-agent systems, LLM, promise-breaking, commitment-breaking, deception, AI Safety

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mgfKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates promise-keeping and commitment defection among LLM agents in multi-agent environments through two complementary evaluation protocols across ten LLMs and six canonical social dilemma games. The authors introduce two complementary evaluation protocols (an Exogenous Protocol and an Endogenous Protocol). Empirical analyses - incorporating repeated interactions, trust scores, heterogeneous ensembles, and human-validated multi-LLM evaluations of reasoning traces - reveal that promise-keeping is neither an intrinsic model trait nor a reliable default behavior. Furthermore, while explicit commitment-binding instructions substantially reduce defection, self-authored endogenous commitments remain less reliable than exogenous ones under identical "Honor" directives, and 63% of violation traces omit any acknowledgment of the broken pledge. These results demonstrate that prompt-level constraints alone cannot guarantee reliable coordination across diverse strategic contexts.

Critical strengths
  1. The paper investigates a problem on the reliability of multi-agent systems: whether LLM agents honor their public commitments during strategic coordination. In collaborative workflows where agents rely on communication and explicit pledges to align their actions, commitment credibility is crucial for system stability and mitigating strategic deception. By translating promise-keeping into a quantifiable evaluation task that compares public declarations against subsequent execution, this work exposes finer-grained behavioral vulnerabilities.

  2. The experimental design is structured around two complementary protocols that capture distinct commitment-formation mechanisms. By pairing an Exogenous Protocol (externally assigned declarations) with an Endogenous Protocol (self-generated pledges), the authors examine how commitment adherence varies across externally imposed versus autonomous contexts. The three-stage endogenous pipeline provides observability into stage-wise behavioral shifts and potential strategic misrepresentation.

  3. The empirical evaluation is extensive and supported by several ablations across a wide matrix of strategic environments. Spanning ten LLMs and six canonical social dilemmas, this work reduces the risk that findings are artifacts of a single model family or game structure. Furthermore, the ablations across prompt framings not only confirm that explicit binding constraints lower violation rates compared to removing payoff-maximization directives, but also preserve granular model- and game-level reporting that highlights heterogeneity in intervention efficacy.

Critical weaknesses
  1. The headline claim that endogenous promises are roughly 8x harder to enforce than exogenous declarations under the "Honor" directive is confounded by inconsistent decision aggregation. The Exogenous Protocol takes a majority vote over five samples per scenario, whereas the Endogenous Protocol evaluates single-sample actions. Because majority voting mathematically suppresses rare events, this discrepancy, alongside unaligned token budgets and interaction histories, invalidates attributing the 8x gap solely to endogenous versus exogenous commitment generation.

  2. The Endogenous Protocol conflates rational strategic updating with promise-breaking due to information asymmetry between Stages 2 and 3. Because agents announce intentions sequentially in Stage 2 before observing the full set of peer announcements in Stage 3, changing actions in anti-coordination games often reflects a rational response to newly revealed peer intentions rather than bad-faith defection. By counting any discrepancy between announcement and final action as a violation, the metric fails to separate legitimate belief updating from genuine promise-breaking.

  3. The multi-round experiments lack the causal controls needed to attribute longitudinal violation trends to reputation mechanisms (Section 7). Because repeated interactions simultaneously expose agents to action histories, realized payoffs, and self-generated trust ratings without an ablated baseline removing trust scores, reputation effects cannot be isolated from strategy learning or context accumulation. Moreover, the 25-round evaluation suffers from selection bias, as it only tracks four models chosen post hoc based on their initial 10-round trends.

Questions for authors

See weaknesses

3 comments

Thank you so much for the thorough review! We address the changes made to the paper below:

W1: We reran the exogenous honor condition with the endogenous protocol's settings and its promise-breaking rate stayed at 1.7% (Appendix C). Second, endogenous agents choose their own announcements, so some combinations of announcements occur far more often than others. We reweighted the exogenous rate by how often each combination actually occurs, which raises it to 3.5% (Appendix F). We still conclude that each of the ten models breaks more of its own promises than of the statements we assign it. We now state the result in the paper this way instead of saying self-made promises are broken eight times as often.

W2: The last speaker hears every announcement before making its own, so it gains no new information between announcing and acting, yet it still breaks 46.3% of its promises, against 48.7% for the first speaker (Section 3.2). New information therefore has a minimal effect on promise-breaking.

W3: In later rounds, agents do not see the full history. They only see the trust ratings and short written assessments of each partner, which Sections 3 and 7 state explicitly. However, we agree with the reviewer that certain effects of the agents’ behaviors cannot be easily isolated, so we no longer claim that agents keep promises to protect their reputation, only that promise-breaking changes over rounds. We also ran the twenty-five-round experiment on the remaining six models, so it now covers all ten (Section 7, Appendix J). Over rounds 11–25, five models break fewer than 5% of their promises and four break more than half.

Furthermore, we reworded the paper so that it is easier to follow without changing any of the results.

Thank you again for the feedback! We hope these revisions addressed your concerns and would appreciate your reconsideration given these revisions!

Dear Reviewer mgfK,

Thank you once again for your time and valuable feedback on our submission.

As the rebuttal period is coming to a close, we wanted to kindly follow up to see if you have had a chance to review our responses. We dedicated significant effort to thoroughly addressing your questions and running additional analyses to strengthen the paper based on your suggestions.

If you have any remaining concerns or additional comments, we would be very happy to discuss them further before the deadline. Otherwise, if our responses have satisfactorily resolved your questions, we would greatly appreciate it if you would consider updating your evaluation accordingly.

Thank you again for your time and consideration.

Best regards, The Authors

Thanks for the response. Some of my concerns are addressed, especially for W3. However, I think limited new insights are presented in this paper, and there is a substantial gap compared to practical agent systems like codex and claude code. I will keep my rating.

Reviewer_UL7COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether LLM agents keep their public statements, testing ten models on six social-dilemma games under two protocols. In the exogenous protocol, the agent is given a public statement and its deviation rate is compared with that of an agent ignoring the statement. In the endogenous protocol, agents privately plan, publicly announce an intended action, then act, over one to twenty-five rounds.

Critical strengths

The central idea is useful for AI safety and oversight. Does the model follow something it put out in tokens as the reasoning progress is interesting.

Critical weaknesses

The exogenous “promise” is supplied to the model. There is existing work that says that models can indeed detect inserted text that it did not make (even steering vectors inserted). The statement is inserted into the prompt as a past fact, and the other agents’ announcements are also invented. The model never commits to anything and this therefore goes too far from what the title promises.

Coming to the Endogenous announcements, these are also statements of intent, not promises. Agents must “clearly state your intended action” the word promise is is not used if I did not miss this detail. So nothing makes the statement binding by default. Especially a deviating from intended plan when deemed optimal cannot be the same as ‘broken promise’. Adding to the previous point, one could argue that the setup invites deviation. Stage 1 asks what to announce separately from what to do, Stage 2 invites persuasion, and Stage 3 stresses that the choice is private and decides “real payoff.” The 29.5% “premeditated deception” looks at least partly produced by these prompts. The result does not look so surprising when we approximate the models as rational agents.

Baseline deviation is also instruction following. The only stated goal is to maximize payoff, and nothing says announcements count. If I read correctly, when told to honor intend, models mostly comply. So the default behavior is rational cheap talk. If you could do a control study on where the breaking stated intent is highest and then see if it correlates with dominant strategy that would further prove this point. Labeling announcements “cheap talk” changes only 5 points, so models already read the intent as cheap talk.

Not sure if this is readily available but an equilibrium study or human benchmark could also explain some results. I couldn't find comparison with game theoretic predictions or with human promise keeping rates from the cited economics literature. That could be interesting.

The concealment claim is a bit strong. A model playing a dominant strategy has no reason to mention a non binding announcement, so not mentioning it shows nothing about hiding it.

Questions for authors

Mostly in the weakness

3 comments

Thank you so much for the thorough review! We address the changes made to the paper below:

Assigned statements are not promises: We agree. The paper calls exogenous deviations commitment-breaking, never lying or promise-breaking, and every one of our claims about promises comes from the endogenous protocol, where the agent writes its own announcement.

Announcements are statements of intent: The prompt never uses the word promise, but 37% of messages contain "I will" or a close variant and 20% a first-person "commit" (Section 3.2). Even when announcements are declared binding commitments, 17.7% are broken (Section 5).

The prompts may invite deviation: We ran a new control with neutral prompts at all three stages (Appendix H). Premeditated deception by llama-4-maverick falls from 85–93% to 2–12%, while claude-opus-5 and gpt-5.6-Terra plan to deviate about as often as before. We updated Section 5 and the Limitations to reflect this, but the interaction and honor results do not depend on premeditation.

Rational cheap talk and dominant strategies: We ran the suggested control. We see that promise-breaking is highest in the two games with an action that pays more whatever the others do (diners 64%, public goods 79%), and most deviations there land on that action (Appendix E). Payoff structure does not explain differences between models on the same game: on diners, gemini-3.7-flash breaks 100% of its promises and gpt-5.6-Terra 2.4%.

Equilibrium and human benchmarks: Against what the other agents announced, 89% of deviations raise the deviator's payoff, so most are best responses to the announcements (Appendix E). Section 2 now cites Vanberg (2008), in which people who promised to cooperate did so 73% of the time, against 52% for those who made no promise.

Concealment Claim: The paper does not claim concealment, and Section 6 now states this directly.

Furthermore, we reworded the paper so that it is easier to follow without changing any of the results.

Thank you again for the feedback! We hope these revisions addressed your concerns and would appreciate your reconsideration given these revisions!

Dear Reviewer UL7C,

Thank you once again for your time and valuable feedback on our submission.

As the rebuttal period is coming to a close, we wanted to kindly follow up to see if you have had a chance to review our responses. We dedicated significant effort to thoroughly addressing your questions and running additional analyses to strengthen the paper based on your suggestions.

If you have any remaining concerns or additional comments, we would be very happy to discuss them further before the deadline. Otherwise, if our responses have satisfactorily resolved your questions, we would greatly appreciate it if you would consider updating your evaluation accordingly.

Thank you again for your time and consideration.

Best regards, The Authors

Thank you for the response. I will keep my rating.

Reviewer_1MvsOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates whether agents in a multi-agent LLM environment actually carry out the actions they have publicly declared. The authors distinguish between "commitment-breaking", violating a declaration imposed from the outside, and "promise-breaking", failing to uphold a commitment generated by the agent itself, and propose exogenous and endogenous evaluation protocols to measure these behaviors. The experiments use 10 different LLMs across 6 social-dilemma games. In the exogenous protocol, agents receive a public declaration, and their actual violation rate is compared against a "structural null"—the expected violation rate if they ignored the declaration. In the endogenous protocol, agents plan their actions, make a public commitment, and then select their final course of action.

Critical strengths

1. A well-motivated research question and evaluation framework

The paper addresses an important question concerning the reliability of LLM agents in multi-agent environments. While much of the existing literature focuses on cooperation or observed actions, this work examines whether agents actually follow through on their publicly stated commitments. I find this distinction meaningful, particularly as LLM agents are increasingly expected to coordinate and negotiate with one another. The separation between exogenous and endogenous protocols is a strength of the evaluation design. I also appreciate the introduction of the structural null, which provides a useful reference for distinguishing commitment-following from behavior that is effectively independent of the assigned announcement. This makes the analysis more informative than simply reporting deviation frequency.

2. Comprehensive empirical evaluation with interesting findings

The empirical evaluation is extensive, covering ten models and six social-dilemma games. One particularly interesting finding is the substantial model–game interaction, which suggests that promise-breaking cannot be adequately characterized by a single model-level score. This is an important observation for designing and interpreting multi-agent reliability benchmarks. The paper also examines several complementary aspects of the problem, including honor instructions, binding versus cheap-talk framing, repeated interactions, heterogeneous agent groups, and awareness of promise-breaking. These analyses provide a more nuanced picture of agent behavior beyond aggregate deviation rates. I also appreciate the effort to validate the LLM-based awareness judgments against human annotations, which adds credibility to that part of the evaluation.

Critical weaknesses
  • Unclear distinction between deception and strategic adaptation: While the paper attempts to distinguish promise-breaking from intentional deception, the private plans in the endogenous protocol are themselves model-generated text and may not reliably reflect the agent's actual intentions.

  • Potential confounding from sequential interactions: Agents observe others' announcements before making their final decisions. Deviations may therefore reflect reasonable strategic adjustments to new information rather than deliberate promise-breaking.

  • Risk of overinterpreting the findings: Although the authors acknowledge these limitations, some conclusions could be interpreted as stronger claims about agent intentions or reliability than the evidence supports. A clearer distinction between strategic replanning and intentional deception would strengthen the analysis.

  • Restricted evaluation settings: The experiments focus on six symmetric social-dilemma games with predefined payoff structures. While these provide controlled environments, they may not capture the complexity of real-world multi-agent collaboration, negotiation, or task coordination.

  • Task-dependent behavior: Given that model–game interactions account for a substantial portion of the observed variance, it remains unclear whether the reported patterns would generalize to other types of multi-agent tasks.

  • Confounding differences between protocols: The exogenous and endogenous protocols differ not only in how commitments are generated but also in their interaction structures, available information, and reasoning configurations. Consequently, the differences in observed behavior cannot be attributed solely to whether commitments are assigned or self-generated.

Questions for authors

Q1. Promise-breaking versus strategic adaptation : In the endogenous protocol, agents can observe other agents' announcements before taking their final actions. How can we distinguish intentional promise-breaking from legitimate changes in strategy based on new information? Could the authors clarify how much of the observed deviation is attributable to strategic adaptation rather than a failure to honor commitments?

Q2. Comparison between exogenous and endogenous protocols : One interesting finding is that honor instructions are less effective for self-generated promises than for assigned statements. However, the two protocols differ in several ways beyond the source of the commitment. Could the authors clarify how much of this difference can be attributed specifically to self-generated commitments? A more controlled comparison would strengthen this conclusion.

Q3. Generalization beyond social-dilemma games : The results show substantial model–game interactions, suggesting that promise-breaking is highly sensitive to the task setting. Would similar patterns hold in collaborative tasks where agents need to coordinate actions toward a shared goal, rather than maximize payoffs in social dilemmas? This seems important for understanding how broadly the findings apply to real multi-agent systems.

Q4. Interpreting awareness from generated reasoning : The paper finds little correlation between awareness scores and promise-breaking rates. However, awareness is measured from generated reasoning, which may not fully reflect the model's decision process. How should we interpret the absence of an explicit reference to a previous promise? Could this reflect differences in reasoning style or verbosity rather than a lack of awareness?

2 comments

Thank you so much for the thorough review! We address the changes made to the paper below:

Q1: The last speaker hears every announcement before making its own, so it gains no new information between announcing and acting, yet it still breaks 46.3% of its promises, against 48.7% for the first speaker (Section 3.2). New information therefore has a minimal effect on promise-breaking.

Q2: We reran the exogenous honor condition with the endogenous protocol's settings and its promise-breaking rate stayed at 1.7% (Appendix C). Second, endogenous agents choose their own announcements, so some combinations of announcements occur far more often than others. We reweighted the exogenous rate by how often each combination actually occurs, which raises it to 3.5% (Appendix F). We still conclude that each of the ten models breaks more of its own promises than of the statements we assign it. We now state the result in the paper this way instead of saying self-made promises are broken eight times as often.

Q3: The closest case to a shared-goal task in our set is weakest link, a pure coordination game, and it has the lowest promise-breaking rate (20%). We have not tested open-ended collaborative tasks, and the Limitations section states that we make no claim about asymmetric or open-ended tasks.

Q4: Section 6 now states that a score of 1 means only that the trace does not refer to the announcement, not that the agent was unaware of it or hid it. For each model, scores correlate with trace length between 0.00 to 0.22 for most models, so most of the overall length correlation reflects which model wrote the trace rather than verbosity.

W1: We agree that Stage 1 text is not verified intent. The paper therefore calls the plan–announcement gap self-reported premeditation and uses "deception" only when the private plan already differs from the announcement.

W2: See Q1.

W3: We agree, and we narrowed the claims so that they describe what agents do, not what they intend. For example, under repeated play, we no longer claim that agents keep promises to protect their reputation, only that promise-breaking changes over rounds.

W4, W5: See Q3. We also state in Section 4 that because of the model–game interaction, a rate measured on one task does not transfer to another.

W6: See Q2.

Furthermore, we reworded the paper so that it is easier to follow without changing any of the results.

Thank you again for the feedback! We hope these revisions addressed your concerns and would appreciate your reconsideration given these revisions!

Dear Reviewer 1Mvs

Thank you once again for your time and valuable feedback on our submission.

As the rebuttal period is coming to a close, we wanted to kindly follow up to see if you have had a chance to review our responses. We dedicated significant effort to thoroughly addressing your questions and running additional analyses to strengthen the paper based on your suggestions.

If you have any remaining concerns or additional comments, we would be very happy to discuss them further before the deadline. Otherwise, if our responses have satisfactorily resolved your questions, we would greatly appreciate it if you would consider updating your evaluation accordingly.

Thank you again for your time and consideration.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.