acceptodds
Under review as a conference paper at ICLR 2027

AτA: A Benchmark for Multi-Agent interaction in Agentic Economies

Abstract

Agents will increasingly take on economically impactful, long-horizon tasks, but most current benchmarks evaluate them in isolated interactions with a single user. While such evaluations are useful for measuring whether agents can accurately accomplish tasks, they fail to capture the broader impact of deploying these agents in real-world marketplaces. In real markets, task completion alone is not enough. Outcomes depend on how well agents satisfy the preferences and constraints of multiple entities with competing objectives. In this work, we introduce Aτ A, a benchmark for multi-agent interaction in agentic economies, in which both users and providers navigate markets through autonomous agents. We evaluate user and provider utility in both symmetric (the same model represents users and providers) and asymmetric (different models represent the two sides) settings, studying how model choice, resource availability, and coordination affect the resulting Pareto trade-offs. Our results show that high booking rates do not necessarily translate into good outcomes for both sides. Even strong models leave substantial room to improve joint outcomes, revealing limitations that evaluations focused on task completion alone do not capture

Primary area:
datasets and benchmarks
Keywords:
Multi-agent Benchmark, LLM Agents, Tool-Calling Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DJ5kOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a synthetic marketplace where consumers and airlines, both represented by LLMs, negotiate with each other to maximize their own reward. They vary several aspects of the system, such as whether the consumers and airlines directly negotiate with each other, and whether they're represented using the same LLMs. Some takeaways include: consumer and airline interests can mutually improve, rather than being a zero-sum game, and no model is uniformly better across roles and environments.

Critical strengths

Studying multi-agent economies is a timely and interesting problem. The paper also grounds the simulation in a economically realistic setting. It is realistic to believe that in the near future, AI agents may represent the interests of consumers by automatically negotiating and booking airline tickets for them.

Critical weaknesses

There don't seem to be any clear, interesting takeaways from this paper. I believe part of the reason is that the reward for consumers and airlines entangles multiple aspects that should ideally be separated: (i) basic instruction following, (ii) negotiation, and (iii) planning and decision-making. For example, the authors claim that the Claude models frequently book the wrong number of passengers, which due to the multiplicative nature of the reward, results in zero reward. This falls under (i) above, and is likely preventable with prompt engineering, emphasizing the importance of booking the correct number of passengers. Evaluating (i) is important, but it doesn't need to be done in a multi-agent setting. Since this is a multi-agent benchmark, the system should be designed so that the experiment's takeaways are largely driven by (ii) and (iii).

Questions for authors

How would you design the benchmark so that the agents' success is largely driven by how well they negotiate, and how well they plan, rather than how well they can follow basic instructions?

3 comments

We thank the reviewer for their valuable time and effort in reviewing the paper. After the initial submission, we studied the traces further to gain more insight into common failure modes across all models. We added two new sections to the revised paper. The first, "Agent Behavior and Market Outcomes" (Section 5), analyzes where current models fail. The second, "Token Economics" (Section 4.4), studies token economics in detail. It addresses a fundamental question: how should providers spend their tokens, and is it worth using a frontier model, and at what cost? We also report standard errors for all mean rewards (Tables 7–12) and provide an anonymous repository with sample traces from GPT-6 Astra (link below). We would be happy to answer any further questions.

Anonymous repository with sample traces: https://anonymous.4open.science/r/ata-traces-56FE/

Q1. There seem to be no clear, interesting takeaways. The reward entangles (i) basic instruction following, (ii) negotiation, and (iii) planning and decision-making. For example, Claude models booking the wrong number of passengers gets zero reward through the multiplicative reward, which is an instruction-following failure likely fixable with prompting. How do you respond?

Thank you for the question. We have updated the paper with two additional sections, mainly covering the failure modes and further insights and analysis on token economics. For rewards, we provide a detailed breakdown of user reward in the appendix (Table 7). The final reward depends on how well the model can follow instructions, which predominantly affects the preference reward, passenger count reward and trip coverage. These reflect the capability of the agent to understand the instruction and execute the booking. For the Astra model, all of these individual metrics are high.

Getting the best deal affects the agent's budget adherence as well as the preference reward, as it depends on the agent's negotiation skills and its interactions.

For the Claude series, we added a special prompt to the Claude models, which improved their scores: Opus 5.5 reaches a user reward of 0.580 and a provider reward of 0.319 under scarce supply, and 0.754 and 0.370 under abundant supply. We will update the preprint with the latest Sonnet 5.5 and Opus 5.5 results.

Q2. How would you design the benchmark so that agents' success is driven largely by how well they negotiate and plan, rather than by basic instruction following?

Thank you for your question. We argue that the current benchmark already measures more than instruction following. The GPT and Gemini models reliably book the correct trip and party size, yet their outcomes remain far from the Pareto frontier. The main reason is that agents do not reach the best deal available. Securing that deal requires negotiation and planning, and the traces show agents doing exactly this where user agents compare prices across airlines, ask for the best discounted fare at first contact, and use competitors' quotes as leverage, while provider agents respond by matching or beating rival offers. We provide representative traces in an anonymous repository (link below), and the revised Section 5 analyses where these interactions fall short, for example providers conceding the full discount even when customers have budget to spare.

Anonymous repository with sample traces: https://anonymous.4open.science/r/ata-traces-56FE/

Dear reviewer

We haven't heard back since we posted our response, where we address both of your concerns and answered your question. We are looking forward your reply, as it is the last day of discussions. Thank you for your time.

Dear Reviewer,

Thank you for taking the time to review our paper and provide valuable feedback. We hope that our responses have adequately addressed your concerns and increased your confidence in our work.

Reviewer_b3tyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The introduced benchmark supports evaluations of interactions between LLM-based user and provider agents in a simulated airline marketplace with shared inventory, heterogeneous preferences, and competing objectives. It tests symmetric and asymmetric model assignments under different supply conditions and reports user satisfaction and provider revenue. Experiments with eight models show Pareto analyisis in different settings.

Critical strengths
  1. The paper extends conventional agent benchmarks by introducing multiple autonomous agents with competing objectives, shared inventory constraints, and state-changing transactions. It enables systematic investigation of multi-agent interactions and their consequences for both sides of a market.
  2. The experiments are comprehensive and demonstrate that model assignment and supply conditions can substantially influence outcomes for both market participants. These findings provide concrete diagnostic value and motivate evaluating agent capabilities in terms of realized outcomes rather than successful transactions alone.
Critical weaknesses
  1. The justification for the metrics and reward is not very clear. It does make some sense to transfer the user satisfaction and provider outcome to the designed reward, but it is difficult to analyze backward. If checking real user satisfaction and provider outcome, the Pareto trade-offs could be different from using reward.
  2. The platform-only and agentic settings differ in their decision-making architectures and objectives, making it unclear how much of the observed improvement can be attributed specifically to direct agent negotiation.
Questions for authors
  1. Do the platform-only and Agentic BASE evaluations share identical settings, such as task instances, inventory initialization, etc.? Are the results from a single run without repeated tests?
  2. How frequently do the simplified seat-preference check and cash-based budget accounting affect evaluated bookings?
  3. Are there existing coordination and BASE runs with matching discount constraints, scoring coverage, and execution protocols?
  4. All the figures could be better visualized to make the difference clearer by changing the range of the axes and adding more annotations/clustering.
5 comments

We thank the reviewer for their valuable time and effort in reviewing the paper. After the initial submission, we studied the traces further to gain more insight into common failure modes across all models. We added two new sections to the revised paper. The first, "Agent Behavior and Market Outcomes" (Section 5), analyzes where current models fail. The second, "Token Economics" (Section 4.4), studies token economics in detail. It addresses a fundamental question: how should providers spend their tokens, and is it worth using a frontier model, and at what cost? We also report standard errors for all mean rewards (Tables 7–12) and provide an anonymous repository with sample traces from GPT-6 Astra (link below). We would be happy to answer any further questions.

Anonymous repository with sample traces: https://anonymous.4open.science/r/ata-traces-56FE/

Q1. What justifies the metrics and reward? If real user satisfaction and provider outcomes were checked, would the Pareto trade-offs differ from those under the designed reward?

We thank the reviewer for raising this point. Our user reward consists of four components: passenger count, trip coverage, budget satisfaction, and preference satisfaction. Passenger count acts as a binary gate (0 or 1), trip coverage provides partial credit for completed journeys, budget satisfaction decreases linearly between the user's target and walk-away budgets, and preference satisfaction uses inverse-rank weighting based on user-specified preferences (Stillwell et al., 1981). We combine these components multiplicatively so that high satisfaction in one dimension cannot fully compensate for poor performance in another. For providers, we measure revenue earned relative to a reference revenue.

To check how much the results depend on our reward design, we evaluated alternative formulations: two user rewards that average components instead of multiplying them, and a provider reward based on the share of the user's walk-away budget the airline captures. The model rankings remain nearly unchanged (Spearman rank correlation 0.90–1.00 across BASE settings; new columns in Tables 7 and 8; see also our response to Reviewer 6tT4, Q5).

Reference: Stillwell, W. G., Seaver, D. A., & Edwards, W. (1981). A Comparison of Weight Approximation Techniques in Multiattribute Utility Decision Making. Organizational Behavior and Human Performance, 28(1), 62–77. https://doi.org/10.1016/0030-5073(81)90015-5

Q2. Platform-only and agentic settings differ in decision-making architecture and objectives. How much of the improvement can be attributed specifically to direct agent negotiation?

Both setups are built on the same tasks, users, airline inventory, booking tools and reward. The only difference is who makes decisions. In the platform-only setup, a single agent handles every user's request, queries airlines and books directly, with the objective of maximizing user reward. In Agentic BASE, each user and each airline is represented by its own agent, and they negotiate prices and services directly before the provider completes the booking. Because everything else is held fixed, the difference in outcomes reflects this change from a single mediator to direct negotiation between separate agents. Under this comparison, Agentic BASE improves both user and provider reward for GPT-6 Astra, GPT-5.4 and both Gemini models under both supply settings (Figure 4).

Q3. Do the platform-only and Agentic BASE evaluations share identical settings, such as task instances and inventory initialization? Are the results from a single run without repeated tests?

Yes. Both setups use the same 100 tasks with five users each, selected with the same evaluation seed, and each task starts from the same fresh inventory, so users face identical options and seat limits in both. Each condition is a single run of 500 user episodes.

Q4. How frequently do the simplified seat-preference check and cash-based budget accounting affect evaluated bookings?

All reported results use cash-based budget accounting, where the user's budget counts only credit-card spending, while certificates and gift cards still count toward provider revenue. The seat-preference check verifies that a seat was selected but does not enforce exact aisle or window matching. We thank the reviewer for pointing this out and will clarify it in the camera-ready version.

Q5. Are there existing coordination and BASE runs with matching discount constraints, scoring coverage, and execution protocols?

Yes. The coordination and BASE runs use the same setting: the same tasks, discount constraints, scoring, and execution protocol. They differ only in whether the coordination channel is enabled.

Q6. Could the figures make differences clearer by changing axis ranges and adding annotations or clustering?

Thank you for the suggestion, and we apologise that the differences were hard to read. We will update the figures in the camera-ready version to use tighter axis ranges, annotate each point with its model, and group points by model family so that differences are easier to see.

Thank you for the detailed response. Most of my concerns are addressed. One question I may need further clarification on is the comparability of the coordination experiments. The response states that Coordination and BASE use identical protocols and discount constraints, but Appendix G.3 and Tables 9–12 still describe differences in protocols and scored coverage, with coordination results coming from historical runs. Could you please explain this inconsistency? If the experiments are actually sufficiently controlled and reliable, I would consider increasing my score to 3.

Reviewer_6tT4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds and studies marketplaces where agents represent customers and providers. The work is an interesting effort to better understand agentic behaviors with sellers and buyers, where there are different preferences, resources, and constraints. The paper demonstrates the value of (and need for) developing benchmarks and evaluation methodologies for realistic market-facing agents: work that is growing in importance given the likely rise of agentic marketplaces.

At the highest level, I see this as a promising benchmark paper addressing an important area of work. A core focus is measuring how well agents serve their principals’ interests when multiple buyers and sellers interact over shared inventory. The authors contribute measures and a set of experiments. They share multiple findings of their experiments, including the result that successful transactions can conceal poor overall service; agents may complete purchases while sacrificing important preferences; stronger customer agents can benefit both sides; and better representation of customer needs can improve both customer satisfaction and provider revenue. They also find direct negotiation helps in some settings but hurts in others. Overall, the work shows interesting examples of methods for building marketplaces and evaluating the quality and consequences of transactions with the synthetic economy.

Critical strengths

Important problem given likely rise of agentic marketplaces.

Valuable direction on developing benchmarks for agentic marketplaces.

Important evaluation problem: Measures whether agents serve customers’ and providers’ interests and preferences, beyond sole focus on completing transactions.

Useful insights on discovery of failures; e.g., high booking rates can conceal such failures as dropping requested passengers.

Critical weaknesses

It is unclear how much of the measured performance reflects market interaction versus capabilities/deficits with preference extraction, search, and reliable booking execution. Extended studies and analysis could help with this. It would be helpful to distinguish potential infrastructure failures from agent failures. It would be helpful to flesh out a taxonomy of failures. It would be useful to report attempted, completed, failed, and scored counts for conditions.

The paper would be improved with the inclusion of representative traces.

The related work would be strengthened if the authors would better distinguish the novelty of their contributions from related work on agentic marketplaces.

Questions for authors

How many evaluations were attempted, completed, failed, and scored in each condition?

How sensitive are results to alternative utility definitions, payment accounting, and restricting evaluation to policy-compliant transactions?

Do the principal findings survive repeated seeds and randomized execution order?

What capabilities or findings does AτA uniquely enable relative to Magentic Marketplace and other related benchmarks? Why do the new research questions require a new environment?

4 comments

We thank the reviewer for their valuable time and effort in reviewing the paper. After the initial submission, we studied the traces further to gain more insight into common failure modes across all models. We added two new sections to the revised paper. The first, "Agent Behavior and Market Outcomes" (Section 5), analyzes where current models fail. The second, "Token Economics" (Section 4.4), studies token economics in detail. It addresses a fundamental question: how should providers spend their tokens, and is it worth using a frontier model, and at what cost? We also report standard errors for all mean rewards (Tables 7–12) and provide an anonymous repository with sample traces from GPT-6 Astra (link below). We would be happy to answer any further questions.

Anonymous repository with sample traces: https://anonymous.4open.science/r/ata-traces-56FE/

Q1. How much of the measured performance reflects market interaction versus capabilities or deficits in preference extraction, search, and reliable booking execution? Can you distinguish infrastructure failures from agent failures?

Thank you for the question. Tables 7 and 8 (Appendix) provide a detailed breakdown of user reward across all experiments, including trip coverage, passenger correctness, budget adherence, and preference satisfaction. Trip coverage and passenger correctness are close to 1.0 for GPT and Gemini models. For GPT-6 Astra, the booking rate under abundant supply is around 98%, suggesting that instruction following and execution are not the main bottlenecks.

For GPT-6 Astra, user reward drops from 0.791 under abundant supply to 0.587 under scarce supply. This decline is mainly driven by a lower booking rate (from 97.8% to 80.6%) and agents securing less preferred or more expensive bookings. The corresponding drops in budget and preference satisfaction reflect the difficulty of negotiating with providers and competing for limited seats.

Infrastructure failures are rare, accounting for around 1% of traces in some experiments and none in others. These traces are excluded from the reported averages rather than assigned zero reward, while customers who complete an episode without booking receive zero reward.

Q2. Can you provide a taxonomy of failures?

We added a new section in the revised paper, "Agent Behavior and Market Outcomes", which discusses two recurring failure modes. The first one is "Individual decisions can undermine collective outcomes", where every agent may complete its purchase while the market as a whole ends up worse off. Under scarce supply, users are affected by this in 85% of tasks: they could not book the seats they wanted because another agent in the same task had already booked them. A well-coordinated plan could have made most of them better off. Despite enabling coordination, agents improve for some models but generally struggle to coordinate effectively. Second, agents are generally good buyers but poor sellers. Provider agents tend to give the maximum 30% discount; for example, for GPT-6 Astra we found the full discount in 100% of bookings. Instead, they could have used their other levers, such as holding the fare, offering a smaller discount, a different cabin or itinerary, or paid services, to find a better deal and capture more value. We have detailed descriptions of the above mentioned phenomenon in the paper with examples.

Q3. How many evaluations were attempted, completed, failed, and scored in each condition?

Each condition runs 100 tasks with 5 users each, so 500 user evaluations are attempted per condition (Section 4.1, Evaluation design). Almost all complete and are scored: environment failures are rare (about 1% in some conditions, none in most) and As stated in Section 2.4, these are excluded rather than counted as zero reward. Among scored evaluations, unbooked customers do count as agent outcomes and receive zero reward. For example, for GPT-6 Astra, 97.8% of customers book under abundant supply and 80.6% under scarce supply. We will add a per-condition table of attempted, completed, failed and scored evaluations to the appendix in the camera-ready version.

Q4. Can you include representative traces?

Thank you for the suggestion. We provide representative traces (GPT-6 Astra, scarce supply) in an anonymous repository:

Anonymous repository with sample traces: https://anonymous.4open.science/r/ata-traces-56FE/

Q5. How sensitive are results to alternative utility definitions, payment accounting, and restricting evaluation to policy-compliant transactions?

Thank you for the suggestion. We evaluated alternative definitions of both the user and the provider reward to check whether our conclusions or the model rankings change. They do not: the overall story remains the same. We added these results as three new columns in Tables 7 and 8 (Appendix).

User reward. The paper defines user reward as U=Ipax⋅Ctrip⋅Sbudget⋅SprefU = I_{\mathrm{pax}} \cdot C_{\mathrm{trip}} \cdot S_{\mathrm{budget}} \cdot S_{\mathrm{pref}}. We evaluated two alternatives that average some factors instead of multiplying them:

U1=Ipax⋅Ctrip⋅Sbudget+Spref2,U2=Ipax⋅Ctrip+Sbudget+Spref3U_1 = I_{\mathrm{pax}} \cdot C_{\mathrm{trip}} \cdot \frac{S_{\mathrm{budget}} + S_{\mathrm{pref}}}{2}, \qquad U_2 = I_{\mathrm{pax}} \cdot \frac{C_{\mathrm{trip}} + S_{\mathrm{budget}} + S_{\mathrm{pref}}}{3}

Model (symmetric, scarce)UU_mainU1U_1U2U_2
GPT-6 Astra0.5870.6820.720
Gemini 3.8 Flash0.5670.6550.690
Gemini 3.7 Flash0.5470.6310.664
GPT-5.40.3410.4400.481

Provider reward. We evaluated how much of the user's budget the provider captures:

Upcap=min⁡(1,revenuewalkaway budget)U_p^{\mathrm{cap}} = \min\left(1, \frac{\text{revenue}}{\text{walkaway budget}}\right)

Model (symmetric, scarce)UpU_p_mainBudget capture UpcapU_p^{\mathrm{cap}}
GPT-6 Astra0.2770.581
Gemini 3.8 Flash0.2560.554
Gemini 3.7 Flash0.2450.524
GPT-5.40.1850.408

Policy compliance. Policy compliance is computed for every booking and reported separately in the Policy column of Tables 7–12; it is not multiplied into the provider reward. When we multiply the provider reward by policy compliance, the BASE ranking of models is unchanged (Spearman 0.89–0.98), and the top pairing stays the same in all four BASE settings.

Ranking agreement. The alternatives change the absolute scores but not the ranking of models. For each BASE setting (symmetric or asymmetric, scarce or abundant), we computed the Spearman rank correlation between the model rankings under the original and the alternative reward; a value of 1 means an identical ranking. The correlation is 0.98–1.00 for U1U_1, 0.97–1.00 for U2U_2, and 0.90–0.98 for budget capture. The lowest value, 0.90, comes from abundant supply, where the top three models are within 0.003 of each other under the original reward.

Q6. Do the principal findings survive repeated seeds and randomized execution order?

Thank you for the question. We evaluated 80 conditions, covering 20 symmetric and asymmetric model pairings under two supply settings, with and without user coordination, each with 500 user episodes (100 tasks with 5 users each). Within a task, users act concurrently, so the order in which they book is not fixed but emerges from the interaction. In the revised paper, Tables 7–12 report the standard error of every mean reward, computed over the 100 tasks. Across all conditions, the standard error is at most 0.024 for user reward and 0.013 for provider reward.

Q7. What capabilities or findings does AτA uniquely enable relative to Magentic Marketplace and other related benchmarks? Why do the new research questions require a new environment?

Thank you for the question. AτA is a substantially harder setting than Magentic Marketplace for several reasons. First, it requires personalization where user agents must extract each customer's preferences, whereas Magentic Marketplace gives consumers their needs explicitly. Second, users always compete with one another for a shared, scarce seat inventory built from real fares, so one agent's booking changes what others can buy. Third, price negotiation is central. Airline agents negotiate discounts within their own policies and limits, whereas Magentic Marketplace does not evaluate negotiation. Fourth, we score both user and provider outcomes rather than consumer welfare alone. Beyond this, AτA lets us study coordination on both sides of the market, different models on each side, and the token economics of provider inference cost (Section 4.4). Finally, AτA is well suited as a reinforcement learning environment: rewards are computed programmatically from the final bookings, without an LLM judge, so every outcome is verifiable, and the task generator can produce far more tasks than our 100-task evaluation set.

Dear reviewer

We haven't heard back since we posted our response, where we address both of your concerns and answered your question. We are looking forward your reply, as it is the last day of discussions. Thank you for your time.

Dear Reviewer,

Thank you for taking the time to review our paper and provide valuable feedback. We hope that our responses have adequately addressed your concerns and increased your confidence in our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.