- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a synthetic marketplace where consumers and airlines, both represented by LLMs, negotiate with each other to maximize their own reward. They vary several aspects of the system, such as whether the consumers and airlines directly negotiate with each other, and whether they're represented using the same LLMs. Some takeaways include: consumer and airline interests can mutually improve, rather than being a zero-sum game, and no model is uniformly better across roles and environments.
Critical strengths
Studying multi-agent economies is a timely and interesting problem. The paper also grounds the simulation in a economically realistic setting. It is realistic to believe that in the near future, AI agents may represent the interests of consumers by automatically negotiating and booking airline tickets for them.
Critical weaknesses
There don't seem to be any clear, interesting takeaways from this paper. I believe part of the reason is that the reward for consumers and airlines entangles multiple aspects that should ideally be separated: (i) basic instruction following, (ii) negotiation, and (iii) planning and decision-making. For example, the authors claim that the Claude models frequently book the wrong number of passengers, which due to the multiplicative nature of the reward, results in zero reward. This falls under (i) above, and is likely preventable with prompt engineering, emphasizing the importance of booking the correct number of passengers. Evaluating (i) is important, but it doesn't need to be done in a multi-agent setting. Since this is a multi-agent benchmark, the system should be designed so that the experiment's takeaways are largely driven by (ii) and (iii).
Questions for authors
How would you design the benchmark so that the agents' success is largely driven by how well they negotiate, and how well they plan, rather than how well they can follow basic instructions?
3 comments
Dear reviewer
We haven't heard back since we posted our response, where we address both of your concerns and answered your question. We are looking forward your reply, as it is the last day of discussions. Thank you for your time.
Dear Reviewer,
Thank you for taking the time to review our paper and provide valuable feedback. We hope that our responses have adequately addressed your concerns and increased your confidence in our work.
We thank the reviewer for their valuable time and effort in reviewing the paper. After the initial submission, we studied the traces further to gain more insight into common failure modes across all models. We added two new sections to the revised paper. The first, "Agent Behavior and Market Outcomes" (Section 5), analyzes where current models fail. The second, "Token Economics" (Section 4.4), studies token economics in detail. It addresses a fundamental question: how should providers spend their tokens, and is it worth using a frontier model, and at what cost? We also report standard errors for all mean rewards (Tables 7–12) and provide an anonymous repository with sample traces from GPT-6 Astra (link below). We would be happy to answer any further questions.
Anonymous repository with sample traces: https://anonymous.4open.science/r/ata-traces-56FE/
Q1. There seem to be no clear, interesting takeaways. The reward entangles (i) basic instruction following, (ii) negotiation, and (iii) planning and decision-making. For example, Claude models booking the wrong number of passengers gets zero reward through the multiplicative reward, which is an instruction-following failure likely fixable with prompting. How do you respond?
Thank you for the question. We have updated the paper with two additional sections, mainly covering the failure modes and further insights and analysis on token economics. For rewards, we provide a detailed breakdown of user reward in the appendix (Table 7). The final reward depends on how well the model can follow instructions, which predominantly affects the preference reward, passenger count reward and trip coverage. These reflect the capability of the agent to understand the instruction and execute the booking. For the Astra model, all of these individual metrics are high.
Getting the best deal affects the agent's budget adherence as well as the preference reward, as it depends on the agent's negotiation skills and its interactions.
For the Claude series, we added a special prompt to the Claude models, which improved their scores: Opus 5.5 reaches a user reward of 0.580 and a provider reward of 0.319 under scarce supply, and 0.754 and 0.370 under abundant supply. We will update the preprint with the latest Sonnet 5.5 and Opus 5.5 results.
Q2. How would you design the benchmark so that agents' success is driven largely by how well they negotiate and plan, rather than by basic instruction following?
Thank you for your question. We argue that the current benchmark already measures more than instruction following. The GPT and Gemini models reliably book the correct trip and party size, yet their outcomes remain far from the Pareto frontier. The main reason is that agents do not reach the best deal available. Securing that deal requires negotiation and planning, and the traces show agents doing exactly this where user agents compare prices across airlines, ask for the best discounted fare at first contact, and use competitors' quotes as leverage, while provider agents respond by matching or beating rival offers. We provide representative traces in an anonymous repository (link below), and the revised Section 5 analyses where these interactions fall short, for example providers conceding the full discount even when customers have budget to spare.
Anonymous repository with sample traces: https://anonymous.4open.science/r/ata-traces-56FE/