acceptodds
Under review as a conference paper at ICLR 2027

SharedIntentBench: A Benchmark for Agents That Coordinate for Independent Principals

Abstract

We introduce SharedIntentBench, a benchmark for AI agents that coordinate deals for different people, such as booking a meeting or making a purchase. Each person gives their own agent private requirements and can change them mid-negotiation. The benchmark has 24 two-party task families in four domains. In eight of them, one person changes their requirements while the agents work. An automatic checker sees everyone's private requirements and approvals. The checker scores two opposite failures. First, the agents can close a deal on an offer made before the change, although nothing shows that the person still agrees. We call this a stale completion. Second, the agents can end with no deal although an option acceptable to everyone remains. We call this a walkaway. A deal is correct only if everyone accepts its terms and every approval is current. A no-deal is correct only if no acceptable option remained. Checking only whether a deal was completed misses the first failure. Counting every no-deal as safe misses the second. Across eight models and 3,600 trials, the share of correct endings ranges from 33.5% to 97.5%. Walkaways are about three times as common as stale completions, and they account for most failures of GPT-5.6 Sol and Qwen3-235B. Scoring the same trials with only one of the two checks reorders the models. In controlled tests with open-weight models, an update that changes no requirement still induces agents to close deals on old offers. Blocking such deals cuts stale completions from 40 to 3 in 768 trials, but walkaways remain the most common failure.

Primary area:
datasets and benchmarks
Keywords:
benchmarks, LLM agents, multi-agent systems, coordination for independent principals, negotiation, authorization, agent evaluation, two-sided validity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RUkKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces SharedIntentBench - a benchmark for agents coordinating on behalf of two independent principals.

The information regarding conditions of the acceptance is private to each agent and in some task families one principal's requirements can be updated mid-negotiation, making the previously issued authorizations outdated. The agreement is only valid if the terms satisfy both principals and the authorizations are up to date. The benchmark also evaluates the cases of agents prematurely abandoning the negotiations when an agreement is possible.

Critical strengths

I think this is a strong benchmark with a range of models of different sizes evaluated across multiple task families covering different possible coordination failures and varying levels of difficulty.

I appreciate that the benchmark includes infeasible tasks for which no-deal is the correct outcome. It is also good that the authors verify task solvability and provide scripted valid solutions for all tasks.

The finding that different scoring rules can change model rankings is very interesting. I also appreciate the distinction between different failure modes which provide better insights into agent coordination failures than a single metric of failure/success.

Critical weaknesses
  1. I found some of the writing in the paper quite dry and difficult to follow. Especially the "Discussion and Limitations" section reads like a list of points rather than a discussion. I think the paper would benefit from more careful editing.

  2. The benchmark assumes that any mandate version update invalidates all previously issued authorizations, even if the agreed terms are still acceptable. This requirement seems unnecessarily strict. Could the authors discuss how this design choice affects the generalizability of the benchmark, or investigate different authorization policies where only approvals affected by the update become invalid?

Questions for authors
  1. Section 4 states that unparsable agents' replies consume steps. Among the 'step cap' failure, what percentage do the unparsable replies take? It would be great to be able to distinguish between formatting issues and coordination capability failures.

  2. How sensitive are the success rates and model rankings reported to the step cap restriction? Is it possible that weaker models would perform substantially better if given more steps?

Reviewer_2A1FOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper explores the problem of strategic negotiation between a pair of agents on behalf of their principals. Each agent is given mandates of their principal and then asked to participate in a structured negotiation. The paper examines the two key failure modes in which either an agreement is reached using stale information or an agreement isn't reached when one could have been reached. They introduce an interesting benchmark to test this behavior and evaluate commercial and open weight models. They also evaluate other properties of the failure (for example unsafe completion where the agent reveals private information of the principal.

Critical strengths
  1. The paper addresses an interesting and still understudied problem domain: strategic negotiation with LLMs.
  2. They describe a benchmark with 4 realistic conflict domains and a validator for the benchmark. This could be used by other researchers (though I have not validated the code). They seem to have a fairly detailed family of actions and stages to the negotiation (it isn't just agreeing on a time, I think?). They describe things like offers, quotes, approvals, and signatures though I don't know how they fit into the task.
  3. They do a relatively extensive evaluation across open weight and closed-weight models (with bootstrap CIs on 2000 draws!).
Critical weaknesses
  1. I found the paper very difficult to follow. It relies heavily on Opus 5 style overly active text with short sentences and weak connections between concepts as well as "it" references that hard to parse.
  2. As a consequence of the presentation issues, I am left with only a very limited understanding of how the benchmark actually works? What is the action sequence, how could private information disclosed. Can agents talk to each other or are they limited only to taking actions determined by the game (if so why have actions that leak information)?
  3. I also found a key part of the problem setup confusing. How can a principal change its mandate mid transaction? Is this realistic? If the principal changes its mandate, can't the negotiation just be restarted? There is mention of signatures? Shouldn't the principal refuse to sign negotiated outcome on a stale mandate? I was left with a lot of questions about the process. If the paper was presented more clearly, I think this could all be addressed.
  4. I also came into this work expecting to see more than two agents but the entire analysis is around only pairs of agents while the domain (supply chain, calendaring, project exchange, procurement, all likely involve multiple principals in real life settings).
Questions for authors

I listed a bunch of questions in the weaknesses. I genuinely think this research direction is interesting and important but I found the paper to be very difficult to follow.

Also I am confused a little about the results. The step cap bar seems high for the open-weight models. Is this because they had shorter context limits or because of some repeat error? If I ignore step cap, then Qwen3-14B looks better than qwen3-32B. What is going on here?

Reviewer_upjzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Agents may increasingly be used in the future to conduct negotiations and reach agreements, e.g., for scheduling appointments or for buying/selling. To investigate this more closely, the authors created a new benchmark: SharedIntentBench. The benchmark comprises 24 two-party task families across four domains. They also distinguish between settings in which valid no-deal outcomes exist and those in which the requirements change during the negotiations. Using this benchmark, they tested 8 different models (6 open-weights models and 2 closed-weights models). They were able to show that the share of correct endings ranges between 33.5% and 97.5%.

Critical strengths
  • Agents may increasingly be used in the future to conduct negotiations and reach agreements, e.g., for scheduling appointments or for buying/selling. To investigate this more closely, the authors created a new benchmark: SharedIntentBench.

  • The benchmark comprises 24 two-party task families across four domains. They also distinguish between settings in which valid no-deal outcomes exist and those in which the requirements change during the negotiations. This allows the benchmark to better reflect reality and avoids assuming a "perfect world".

Critical weaknesses
  • The paper is based on the assumption that, when a change occurs, the other agent sees a change but not what was changed. This is already a laboratory situation that I do not regard as reflecting reality.

  • In parts, the paper is very hard to read, because a great many terms are used in relation to the benchmark (task library, domains, conflict classes, families, frozen instances, simulators, interaction mechanism) that attempt to explain the benchmark's structure. Even though most of the terms are explained somewhere in the paper, it takes quite some time to understand how the individual terms relate to one another. A central place in the paper or in the appendix where the terms are explained, along with how they relate to each other, would make the paper considerably easier to understand.

  • The problem formulation conflates the actual problem with the specific implementation used in the paper — particularly in Section 1. A clearer separation between a general problem formulation and the paper's concrete implementation would strengthen the contribution and the clarity of the paper.

  • The paper assumes that the same model is used on both sides of the negotiation. However, this is an assumption that is not very realistic in the real world. What happens when two different models enter into negotiation? In particular, it would be interesting to examine the following combinations: strong model × strong model, weak model × strong model, weak model × weak model. Can the weak model be compensated for by the strong model, so that a suitable outcome is still reached? Which errors persist and which errors disappear?

  • Since the benchmark is new and synthetically created, the analysis of the failed frozen instances in particular should be deepened. This is done in Section 5.1 and in the references to the appendix, but it remains very superficial. A deeper analysis could establish two things:

  1. Do all models fail on the same frozen instances?
  2. From this basis, which strategies could resolve the failures? This would not only improve the quality of the benchmark but also the insights in the paper, since the paper's findings are currently descriptive in nature and do not offer any new solution proposals.

Minor Points:

  • The abbreviation VR is introduced twice.
  • The contributions state: "walkaways outnumber stale completions about three to one". I consider stating such a ratio problematic, because it depends heavily on the selection of models. The ratio of 1/3 only arises through aggregation; but if you look at the ratios of all the individual models, none of them comes close to this ratio.
  • The introduction states: "Both failures occur across eight models". The results in Table 3 do not show this. For 3 out of 8 models there is no error for the error class "Stale compl.".
Questions for authors

In the AI Disclosure Statement, you only wrote something about quality assurance for sources. How was quality assurance carried out for the other uses of AI, in particular for the implementation and for writing the paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.