acceptodds
Under review as a conference paper at ICLR 2027

TERMS-Bench: Diagnosing LLM Negotiation Agents Beyond Deal Rate

Abstract

Negotiation is a central mechanism of economic exchange, shaping outcomes in markets, procurement, labor agreements, and resource allocation. It also provides a canonical testbed for agentic language models, requiring multi-turn interaction under hidden preferences, strategic communication, and binding constraints. These properties make negotiation difficult to evaluate: unlike math or code, it has no intrinsic verifier. Existing LLM negotiation evaluations rely on LLM versus LLM interaction or aggregate outcome metrics such as deal rate, leaving failures opaque. We introduce **TERMS-Bench** (**T**estbed for **E**conomic **R**easoning in **M**ulti-turn **S**trategy), a Bayesian-game framework that makes the environment itself the verifier by specifying the counterpart's latent type, policy, and payoff structure. We instantiate this framework in bilateral price negotiation, where the counterpart's private state and simulator policy are hidden from the agent but observable to the evaluator. This turns the counterpart from a black-box opponent into a diagnostic instrument, enabling agent-attributable failure analysis and oracle reference policy defined optimality gaps. Evaluating 13 LLM agents spanning frontier systems from major providers, **TERMS-Bench** turns negotiation evaluation from aggregate ranking into actionable diagnosis: *where* agents fail, *why* they fail, and *what* to strengthen. Empirically, frontier models saturate deal rate yet diverge in surplus, cue use ability, belief calibration, and compliance, revealing agent-specific bargaining bottlenecks masked by prior benchmarks. Stress tests of the counterpart preserve the core findings, and a model-versus-model round-robin shows that the capability measured by the simulator largely carries over to head-to-head play between models.

Primary area:
datasets and benchmarks
Keywords:
large language model agents, negotiation, benchmark and evaluation, multi-agent systems, Bayesian games, economic reasoning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_fWW3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TERMS-Bench, a controlled benchmark for diagnosing LLM negotiation abilities beyond deal rate. It evaluates surplus, opponent beliefs, and constraint compliance, using six opponent families and oracle information interventions to probe performance bottlenecks. Experiments with 13 LLMs show that high agreement rates mask substantial differences in bargaining performance. Validation includes robustness checks, ablations, model-versus-model negotiations, and a small human study.

Critical strengths
  1. The paper is well organized and easy to follow, with well-designed figures that clearly explain the framework and findings.

  2. The benchmark reveals differences in surplus, belief accuracy, and constraint compliance that agreement rates alone miss. These diagnostics provide a more informative assessment of negotiation abilities.

  3. Providing posterior information and revealing opponent types help probe why agents underperform. Robustness checks, ablations, and model-versus-model experiments further support the usefulness of these diagnostics and reduce concerns that the findings depend on one simulator configuration.

Critical weaknesses
  1. The benchmark focuses on single-price bargaining with hand-designed opponents. Despite some external validation, it remains unclear whether its findings generalize to realistic negotiations.

  2. Posterior interventions change both available information and its presentation. Failure to benefit may reflect difficulty using the summary rather than accurate prior inference, weakening the claimed separation of inference, uncertainty, and control.

Questions for authors
  1. What evidence supports interpreting the intervention gaps as distinct inference, uncertainty, and control failures, rather than differences in how models use the supplied information? Have you tested sensitivity to the posterior’s presentation?

  2. Beyond the shared reservation-value distribution and small human probe, is there evidence that the benchmark’s rankings or diagnoses generalize to other negotiation settings?

  3. The oracle posterior uses latent cues underlying the messages, while the base agent only sees natural language. How much of the intervention gain comes from this additional information rather than improved inference?

  4. Could the flat or increasing belief error across rounds reflect harder negotiations remaining unresolved? Does this finding persist when comparing the same set of episodes across rounds?

4 comments

We thank reviewer for the thoughtful and constructive review. We appreciate the positive assessment of the paper's clarity, diagnostic contributions, and supporting validation experiments. We conducted additional experiments addressing all four questions, with results summarized below.

W2/Q1: Distinct failure sources versus how models use supplied information

  • Short answer. We tested whether the inference gap reflects difficulty using the supplied posterior. Simpler structured representations do not increase the gain, and additional controls show that the agents respond to the correctness of the information. These results support the diagnostic value of the interventions.

  • Posterior presentation (new). We tested alternative representations for Grok 4.20 on 720 matched episodes with voice off:

Posterior suppliedInference gainDifference vs. paper
Full posterior summary (paper)+0.58 [+0.26, +0.90]—
Point estimate only+0.51 [+0.19, +0.80]−0.06 [−0.37, +0.25]
Cue-free posterior (family known)+0.56 [+0.21, +0.88]−0.02 [−0.33, +0.30]
Plain-language description+0.11 [−0.24, +0.41]−0.47 [−0.83, −0.14]

Neither simpler structured representation improves the inference gain. The plain-language version performs worse, confirming that presentation can affect information use; the paper's structured summary performs best among the tested formats in point estimate.

  • Evidence of prior inference (new). We replayed the exact Bayesian filter on the economic histories recorded for four agents. Their reported reservation-value errors (0.10–0.11 of the price range) are comparable to the filter's (approximately 0.10) and substantially below the prior's (0.19). This suggests that agents already extract meaningful counterpart information from their observations, leaving less additional value for the posterior intervention.

  • Same format, different information (new). We also tested whether type revelation helps merely because an additional structured prompt is provided. Supplying an incorrect type from another episode in the same condition reverses the true-type gains:

AgentTrue-type gainWrong-type effect
Grok 4.20+3.99−3.68 [−4.39, −2.94]
GLM-5.1+1.70−1.31 [−1.79, −0.81]

The agents respond to incorrect types in economically predictable directions: overly optimistic information induces excessive holdout, while pessimistic information induces excessive concessions. The sign reversal demonstrates that the value of type revelation depends on informational correctness, with the presentation held fixed.

Finally, the base-to-revealed-type gain and remaining control-side gap do not depend on posterior presentation. Across four agents, larger stated belief errors are also associated with greater benefits from revealing the true type (partial Spearman ρ = 0.33–0.46; see our response to Reviewer syXk). We agree that these quantities measure intervention effects and need not uniquely identify internal cognitive mechanisms. We will clarify this interpretation in Fig. 1 and §1 and include the new controls in the appendix.

(To be continued in the next comment.)

Q3: How much of the intervention gain comes from privileged latent cues?

Short answer. We directly tested this by replacing the original posterior with one conditioned only on observed economic history. The inference gains and model-specific diagnostic profiles remain nearly identical.

We recomputed the posterior using only the prices and actions the agent observes, marginalizing over all six counterpart families. We evaluated this with voice off, where the agent receives no linguistic cues (720 matched episodes per agent).

Grok 4.20GLM-5.1
Inference gain, original posterior+0.58 [+0.26, +0.90]+0.69 [+0.33, +1.05]
Inference gain, observation-only+0.67 [+0.40, +0.94]+0.62 [+0.33, +0.94]
Observation-only − original+0.09 [−0.17, +0.36]−0.06 [−0.37, +0.27]
Inference / uncertainty / control (%), original11 / 63 / 2616 / 23 / 61
Inference / uncertainty / control (%), observation-only12 / 62 / 2614 / 25 / 61

The central diagnostic distinction persists: Grok 4.20 remains predominantly uncertainty-sensitive (62–63% of its gap), while GLM-5.1 retains a dominant control-side residual (61%). Neither inference-gain difference is statistically detectable. Thus, privileged latent-cue conditioning does not appear to drive these diagnostic profiles in the tested settings. Further details are in our response to Reviewer cfx4.

Q4: Could the belief-error trend reflect harder negotiations remaining unresolved?

Short answer. We directly tested this by tracking the same episodes across rounds. The flat or increasing belief-error pattern persists for 10 of 12 models, showing that changing episode composition alone does not explain the finding.

Fixed-cohort analysis (new). We restricted evaluation to episodes lasting at least six rounds and compared belief errors between rounds 1 and 6 on identical episodes. Five models show statistically significant increases in error, and another five show no statistically detectable improvement, with only two showing decreases. Thus, the central finding persists even when the episode population is held fixed.

(To be continued in the next comment.)

W1/Q2: Do the rankings and diagnoses generalize beyond the original simulator?

  • Short answer. We find consistent surplus rankings and recurring behavioral diagnoses across three complementary evaluations, varying the counterpart, its language responsiveness, and the economic grounding of negotiation scenarios.
EvaluationWhat changesEvidence of transfer
LLM round-robin (App. I.6)Seven LLMs negotiate directly against one anotherSurplus ranking ρ = 0.93 (p = 0.003); negotiation-length ranking ρ = 0.75; GPT-4o-mini retains the lowest surplus and a 35% deal rate (vs. 83–99% for others)
Language-reactive counterpart (new)Agent messages directly affect acceptance and concessionsAggregate ranking unchanged under moderate reactivity (4 agents); type-revelation gains preserve their ordering: Grok +4.26, GLM-5.1 +1.38, Qwen3.5 +0.61 (standard: +3.99, +1.70, +0.57)
Product-grounded suite (App. H.2)Negotiations use real product categories and price statisticsSurplus-ranking correlation ρ = 0.83 (p < 0.01, 9 models); no false agreements
  • Model-specific transfer. Beyond aggregate surplus correlation, the round-robin reproduces identifiable behavioral patterns: negotiation length rank-correlates across settings, GPT-4o-mini continues to struggle to reach agreements, and all six frontier models retain the seller-side surplus advantage. The six-episode human probe provides additional directional evidence, including GPT-4o-mini's early rejection of feasible deals (App. I.7).

  • Language-reactive extension. More importantly for diagnostic validity, the new counterpart allows messages to influence economic decisions while preserving objective utility evaluation. Under moderate reactivity, the relative benefits from revealing counterpart information persist across Grok, GLM-5.1, and Qwen3.5. Stronger language reactivity introduces larger communication effects and can change rankings (full results in our response to Reviewer cfx4).

We agree that bilateral single-price bargaining remains a controlled setting. The results establish transfer across the tested counterpart and product-grounding variations, while realistic multi-issue and human negotiation require further validation. The framework (§2.1) accommodates richer type spaces and negotiation protocols with specified counterpart policies.


We will incorporate the new experiments, clarify the scope of the benchmark, and strengthen the discussion of intervention-based diagnostics in the camera-ready version. We hope these results address the reviewer's soundness concerns and would be grateful if they could be considered in the final assessment.

Dear reviewer, thank you again for your thoughtful review! We wanted to kindly follow up on our rebuttal, including the new experiments on posterior presentation, observation-only inference, and fixed-cohort belief-error analysis, along with additional evidence on diagnostic transfer. As the discussion period closes today, we would greatly appreciate hearing whether these results have addressed your concerns and whether you might consider revisiting your score.

Please don't hesitate to let us know if any questions or concerns remain-we would be very happy to address them. Thank you again for your time and consideration!

Reviewer_syXkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper proposes TERMS-Bench to evaluate LLM negotiation agents. While commonly LLM vs LLM is considered they move beyond blackbox towards a structured opponent with hidden preferences along several dimensions, thus a known policy. It evaluates 13 LLMs across three economic regimes and six opponent families, measuring surplus, agreement, opponent modeling, and protocol compliance. Oracle interventions aim to distinguish inference, uncertainty and control failures.

Critical strengths
  • Opponent grid works to generate structrued opponent policies of desired bahaviour.
  • Deal rate saturates while surplus extraction differs substantially. Extensive ablations and LLM-vs-LLM experiments provide encouraging validation.
Critical weaknesses
  • The paper and figure 1 promis explainations on "WHY" the agent behaved like this. The proposed decomposition measures performance differences under interventions, but does not uniquely identify underlying causes. -the opponent does not react to the agent's natural-language messages. This removes strategic communication towards the opponent and reduces much of the task to price selection under partial observability.
Questions for authors

Are the proposed dimensions of opponent modelling a) mutially exclusive and b) completely exhaustive? LLM present a rich class of opponents, capable of language, opponnent modelling etc. When is the trade off worth it?

There are doubts about the truthfulness of cot. How do we know the belief reports are faithfull?

5 comments

We thank the reviewer for the constructive feedback and for recognizing the value of the structured counterpart, surplus diagnostics, and supporting experiments. We appreciate the positive soundness assessment and will improve the paper's clarity.

Summary of our responses and new evidence:

  • Diagnostic attribution (W1a): We will clarify that the oracle decomposition quantifies recoverable performance gaps under information interventions, rather than uniquely identifying internal failure mechanisms. A new wrong-type placebo shows that correct counterpart information improves performance, while identically formatted incorrect information reverses the gain.
  • Language responsiveness (W1b): We introduce a language-reactive counterpart and find that aggregate rankings and model-specific type-revelation patterns persist under moderate reactivity, while stronger reactivity reveals meaningful communication effects.
  • Belief-report faithfulness (Q2): Beliefs are elicited as structured outputs, not chain-of-thought. Three analyses show that these reports contain counterpart information, relate to subsequent decisions, and predict the benefit of revealing the true type.
  • Presentation: We commit to five concrete revisions, detailed below, to clarify the framework, intervention interpretation, and empirical findings.

W1a: The decomposition does not uniquely identify "why"

Short answer. We agree that the decomposition does not reveal the mechanisms inside a model, and we will clarify this wording. What it does identify is which information improves an agent's performance, by how much, and what gap remains. New experiments further show that these gains depend on the correctness of the information and remain stable when the posterior is restricted to what the agent observes.

What the decomposition measures. We evaluate each agent on the same negotiation episodes under three conditions: (1) using its usual observations to infer the counterpart's hidden type; (2) receiving the exact Bayesian posterior over that type; and (3) knowing the true type. The improvement from (1) to (2) measures the inference gap, from (2) to (3) the uncertainty gap, and the remaining distance to the optimal known-type strategy the control gap.

These distinctions are actionable. Uncertainty accounts for 90% of Claude Opus 4.7's gap (Fig. 2), while GPT-4o-mini retains 98% of its gap even when the true type is revealed. The same contrast appears within the Qwen3.5 family (new): revealing the type recovers 45% of the 27B model's gap but only 4% of the 9B model's. These profiles point to different improvements: better counterpart information for agents that benefit from type revelation, and better decision policies for those that do not.

New intervention controls. To test whether the gains reflect meaningful information use, we conducted two additional experiments on matched episodes (utility differences with 95% CIs):

InterventionGrok 4.20GLM-5.1
True type revealed vs. base+3.99 [+3.53, +4.42]+1.70 [+1.32, +2.08]
Wrong type supplied vs. base−3.68 [−4.39, −2.94]−1.31 [−1.79, −0.81]
Observation-only vs. original posterior (inference gain)+0.09 [−0.17, +0.36]−0.06 [−0.37, +0.27]
  1. Does any type help? We supplied an incorrect type drawn from another episode in the same condition, using the same format as the true-type intervention. Both agents lose utility with the wrong type, reversing their gains from correct information. Their behavior also follows the misinformation: overly optimistic types lead to excessive holdout and missed deals, while pessimistic types lead to excessive concessions. This sign reversal shows that the intervention's benefit depends on the correctness of the supplied information. Grok's larger response is consistent with its greater sensitivity to type revelation.

  2. Does the posterior rely on information unavailable to the agent? We recomputed it using only observed prices and actions, marginalizing over all six counterpart families. The inference gains remain nearly identical, and the diagnostic profiles are preserved: uncertainty still accounts for 62–63% of Grok's gap, while the control-side residual accounts for 61% of GLM-5.1's. The main distinction therefore persists without privileged latent-cue information.

Together, these controls strengthen the diagnostic interpretation: correct information produces economically meaningful gains, and model-specific differences persist under observation-only inference. Further supporting this interpretation, episodes with larger errors in the agents' stated beliefs benefit more from type revelation (Q2, test 2).

(To be continued in the next comment.)

W1b: The counterpart does not react to the agent's messages

  • Short answer. This was a deliberate design trade-off that makes exact oracle evaluation and belief scoring possible. The controlled environment reveals substantial failures in economic decision-making, and new experiments show that the main diagnostic distinctions persist when the counterpart responds to language.

  • Why the trade-off. With a specified counterpart policy, we know how its hidden type and the agent's offers determine acceptance and concessions. This lets us compute reference policies, evaluate optimality gaps, and score beliefs against ground truth. The benefit is substantial: although 12 frontier LLMs reach agreements in 93–99.9% of feasible episodes, they obtain only 41–71% of reference utility (Table 1). These differences are largely hidden by deal rates alone.

  • Language-reactive counterpart (new). We introduced a variant in which a fixed external model scores each agent message for credible reasons and constructive communication, with the score modifying the counterpart's acceptance and concession behavior. The external model governs counterpart behavior; agent performance remains evaluated through objectively computed economic outcomes. We tested four agents on 720 seed-matched episodes per setting (full design in our response to Reviewer cfx4).

TestResult
Moderate language reactivityAggregate model ranking unchanged; deal rates shift by at most 0.7 percentage points
Message-dependent utilityGLM-5.1 gains +0.48 [+0.17, +0.79]; Grok 4.20 loses −0.43 [−0.68, −0.19]
Stronger reactivity (3×)Communication effects increase (GLM-5.1: +0.81; Grok 4.20: −1.04), changing some rankings
Revealed-type diagnosticGrok gains +4.26 (standard: +3.99), GLM-5.1 +1.38 (standard: +1.70), Qwen3.5 +0.61 (standard: +0.57); the relative ordering persists

These results show that TERMS-Bench can incorporate economically consequential communication while preserving the model-specific differences in the value of counterpart information under the tested moderate reactivity. The extension also reveals additional differences in messaging behavior as the counterpart becomes more responsive to language. Additionally, the seven-model round-robin (App. I.6) provides complementary transfer evidence: simulator surplus strongly predicts head-to-head surplus (Spearman ρ = 0.93), and GPT-4o-mini's unusually low agreement rate recurs (35% versus 83–99% for other agents; see our response to Reviewer cfx4).

Q1: Are the counterpart dimensions mutually exclusive and exhaustive? When is the trade-off worthwhile?

  • (a) Separately parameterized, not mutually exclusive. The hidden type contains three dimensions (App. C): reservation value determines acceptable prices, urgency governs time pressure, and stance determines responsiveness to the agent's behavior. Each enters a distinct component of the counterpart policy, and all three can coexist within an opponent. Their separate parameterization allows controlled variation and diagnostic scoring.

  • (b) Exhaustive within the specified model. Together with the counterpart family, these dimensions specify the stochastic counterpart policy, enabling ground-truth beliefs and computable reference policies. We do not claim they exhaust real-world negotiation, where fairness preferences, personality, and multiple objectives may also matter. The general framework (§2.1) permits richer type spaces with specified policies.

  • When is the trade-off worthwhile? The choice depends on the evaluation objective. LLM counterparts offer richer, open-ended interaction but lack known latent preferences and computable reference policies. A specified counterpart provides these ground truths, enabling reproducible diagnosis of economic performance. The approaches are complementary: the round-robin supports transfer of key behavioral patterns, while the language-reactive extension incorporates message-dependent decisions within an explicit economic response mechanism.

(To be continued in the next comment.)

Q2: How do we know the belief reports are faithful?

  • Short answer. The benchmark does not evaluate chain-of-thought. Agents report beliefs in separate structured fields, and three analyses show that these reports contain meaningful counterpart information and are systematically related to the agents' decisions.

  • How beliefs are evaluated. At each round, agents report estimates of the counterpart's reservation value and urgency, along with probabilities over its stance. We score these against the known latent type. Free-text reasoning is never scored.

  • Evidence for belief-report validity:

TestResults
Beliefs predict actions (App. I.5)Across all 9 traced agents, higher reported probability of an aggressive counterpart predicts smaller concessions in the next round
Belief errors predict information value (new)Across 4 frontier agents, larger stated reservation-value errors are associated with greater gains from revealing the true type (partial Spearman ρ = 0.33–0.46, controlling for reservation position, ZOPA width, and urgency)
Reports contain information beyond the prior (new)Across 4 agents, reported reservation-value errors are 0.10–0.11 of the price range, comparable to the exact Bayesian filter (0.10–0.11) and substantially below the prior (0.19)

Interpretation. These tests provide complementary evidence that the belief reports are informative and behaviorally relevant. The reports predict subsequent actions, their errors are associated with the economic value of additional information, and their accuracy approaches that of an exact Bayesian filter on observed histories. Together, these findings support the use of structured belief reports as a meaningful diagnostic of opponent modeling. We note that we do not claim that reported beliefs perfectly reveal the model's internal representations. The paper's primary conclusions rest on objectively computed actions and outcomes, with belief accuracy providing a complementary diagnostic axis.

Presentation

We agree that the paper's main ideas should be easier to follow. In the camera-ready version, we will:

  • Clarify the claims in the abstract, §1, and Fig. 1 to describe recoverable performance gaps under information interventions.
  • Explain the three intervention conditions in plain language before Eq. (3).
  • Add a table in §2 connecting reasoning, communication, and social cognition to the benchmark's diagnostics (see our response to Reviewer bGz7).
  • Streamline §2.3 and Finding 4, using concrete examples to illustrate the diagnostic results.
  • Add a notation table to the appendix.

We hope the new experiments and planned revisions address the reviewer's concerns about diagnostic validity and clarity. We would be grateful if the reviewer would consider this additional evidence in the final assessment. And please don't hesitate to let us know should any concerns persist.

Reviewer_cfx4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TERMS-Bench, a Bayesian-game framework for evaluating LLM negotiation agents beyond aggregate deal rates. The framework replaces black-box LLM counterparts with parameterized policies whose latent states and economic dynamics are accessible to the evaluator.

This enables diagnostic evaluation of surplus extraction, agreement calibration, opponent modeling, and protocol compliance. The authors evaluate 13 LLM agents across six counterpart families and three economic regimes, and introduce oracle-based interventions to characterize performance gaps.

Critical strengths
  1. Diagnostic evaluation beyond aggregate outcomes.

The evaluator-transparent counterpart provides a meaningful alternative to LLM-vs-LLM negotiation benchmarks. By controlling the counterpart's latent type and economic behavior, the framework can examine differences in surplus extraction, belief estimation, and constraint compliance that would otherwise be obscured by aggregate agreement rates.

  1. Empirical evaluation and robustness analysis.

The evaluation covers a diverse set of models, counterpart behaviors, and economic regimes. In particular, the oracle interventions, counterpart perturbations, and model-vs-model round-robin provide evidence beyond simple benchmark rankings. The observation that informative language cues can reduce negotiation performance is also an interesting finding.

Critical weaknesses
  1. The diagnostic conclusions depend on a highly structured counterpart model.

The counterpart's economic decisions are determined by a parameterized kernel, while language realization does not directly affect its committed economic actions. Although this design is appropriate for controlled evaluation, it excludes important aspects of realistic negotiation, where semantic content and persuasive communication can influence subsequent decisions. Consequently, the cue-use and strategic-adaptation failures may be specific to the chosen counterpart dynamics. The limited human-counterpart evaluation does not fully establish their generality.

  1. The oracle interventions do not fully isolate the proposed sources of failure.

The decomposition into inference, uncertainty, and control gaps is based on differences between information interventions. (Appendix E) The oracle posterior uses latent cues underlying the generated messages and is provided to the agent through a low-dimensional summary. This differs from the information directly available to the evaluated agent. Furthermore, additional information may alter the agent's behavior independently of improvements in belief estimation. Although the authors acknowledge that these quantities are intervention effects, interpreting them as distinct capability bottlenecks remains difficult.

Questions for authors
  • Diagnostic validity. How sensitive are the main findings to counterpart policies whose economic decisions respond directly to the semantic content of the agent's messages?

  • Oracle interventions. How would the inference/uncertainty/control decomposition change if the oracle posterior were conditioned only on information recoverable from the agent's actual observations, rather than the latent cues underlying language generation?

  • External validity. Do the identified model-specific diagnostic failures predict corresponding behaviors in the model-vs-model or human-counterpart evaluations, beyond the aggregate performance correlation?

3 comments

We would like to thank the reviewer for the thoughtful feedback. Below are our responses to the reviewer's weaknesses and questions. We ran the experiments described in the first two questions.

W1/Q1: Sensitivity to counterparts whose decisions respond to message content

We agree that the standard kernel does not model persuasion through language. We therefore ran a controlled sensitivity analysis with a language-reactive counterpart.

Design. A fixed external model (Kimi-K2.6, not among the evaluated agents) scores each agent message from 0 to 10 (5 = neutral), rewarding credible reasons, acknowledgment of the other side, and constructive proposals, but not the proposed price itself. The normalized score shifts the counterpart's acceptance logit by beta(q − 0.5), preserving the IR floor, and scales its concession rate by exp(gamma(q − 0.5)). At a neutral score, the original economic response rule is recovered. We note that the language-scoring model is part of the counterpart's behavior, not the evaluator: agent utility and economic outcomes remain objectively computed. The mapping is deliberately stylized. We fixed the main setting (beta = 2.0, gamma = 0.8) before running and then tripled both coefficients to test stronger reactivity. We evaluated four agents.

AgentUtility change: main (95% CI)Utility change: 3× (95% CI)
GLM-5.1+0.48 [+0.17, +0.79]+0.81 [+0.48, +1.15]
Qwen3.5-397B-A17B+0.10 [−0.18, +0.39]+0.63 [+0.34, +0.95]
DeepSeek-V4-Pro−0.32 [−0.74, +0.09]−0.77 [−1.22, −0.36]
Grok 4.20−0.43 [−0.68, −0.19]−1.04 [−1.33, −0.77]

Results. Under moderate language reactivity, the aggregate model ranking remains unchanged across the 720 seed-matched episodes, deal rates shift by at most 0.7 percentage points, and utility changes (at most 0.5) are small relative to the agents' oracle gaps (4.4–6.1). Agents whose messages score above neutral gain utility, while those scoring below neutral lose. At 3× reactivity, these effects grow in the same direction, with utility changes of 0.6–1.0, comparable to the standard-counterpart utility differences among Qwen3.5, Grok 4.20, and DeepSeek-V4-Pro (0.7–1.4). Their ordering consequently changes, while GLM-5.1 remains first and deal rates shift by at most 1.1 percentage points. Thus, aggregate rankings are robust to moderate language reactivity, while stronger reactivity reveals economically meaningful differences in agents' communication behavior that deal rates largely miss.

Model-specific diagnostics. More importantly, we tested whether the diagnostic differences persist by rerunning the revealed-type condition for three agents under moderate reactivity:

AgentRevealed-type gain: standardRevealed-type gain: reactiveDifference
Grok 4.20+3.99 [+3.53, +4.42]+4.26 [+3.81, +4.72]+0.27 [−0.10, +0.64]
GLM-5.1+1.70 [+1.32, +2.08]+1.38 [+1.01, +1.74]−0.32 [−0.69, +0.04]
Qwen3.5-397B-A17B+0.57 [+0.26, +0.87]+0.61 [+0.32, +0.93]+0.04 [−0.34, +0.44]

The ordering of revealed-type gains is preserved: Grok benefits substantially more from type information than GLM-5.1 or Qwen3.5 under both counterpart policies. None of the between-kernel differences is statistically significant. Thus, the model-specific heterogeneity in the value of counterpart information persists under the tested language-reactive extension.

Cue-use and transfer. The new experiment uses voice off to isolate outgoing communication. Evidence for cue-use robustness comes separately from the negative cue penalty across 18/18 kernel perturbations (App. I.3) and under a second vendor's voice model (App. I.4). In the model-vs-model round-robin (App. I.6), where LLM counterparts respond to language, simulator surplus predicts head-to-head surplus (rho = 0.93). We agree that the human-counterpart evaluation is limited: with six episodes, we treat it as a directional probe (App. I.7).

These experiments show that both aggregate performance comparisons and heterogeneity in type-revelation effects persist under moderate language-dependent economic responses. We will include the additional experimental details and results in the appendix.

(To be continued in the next comment.)

W2/Q2: Decomposition using a posterior conditioned only on agent observations

We directly tested whether the decomposition depends on privileged latent-cue information. We recomputed the posterior using only the prices and actions observed by the agent, marginalizing over all six counterpart families. We evaluated this with voice off, isolating economic observations from linguistic cues. The resulting inference effects and model-specific diagnostic profiles are nearly identical to those obtained with the original posterior:

Grok 4.20GLM-5.1
Inference gap, original posterior+0.58 [+0.26, +0.90]+0.69 [+0.33, +1.05]
Inference gap, observation-only+0.67 [+0.40, +0.94]+0.62 [+0.33, +0.94]
Observation-only − original+0.09 [−0.17, +0.36]−0.06 [−0.37, +0.27]
Inference / uncertainty / control (%), original11 / 63 / 2616 / 23 / 61
Inference / uncertainty / control (%), observation-only12 / 62 / 2614 / 25 / 61

The central diagnostic distinction persists: Grok 4.20 remains predominantly sensitive to uncertainty about the counterpart (62–63% of its oracle gap), while GLM-5.1 retains a dominant control-side residual (61%). Grok's profile also closely matches its voice-on decomposition in Fig. 2 (14 / 62 / 24%). These results indicate that the identified differences are not driven primarily by privileged latent-cue conditioning in the tested configurations.

Additional diagnostic evidence. The base-to-revealed-type gain and the residual gap to the reference policy are independent of posterior construction or presentation. Moreover, across four agents, episodes with larger stated reservation-value errors exhibit greater gains from revealing the true type (partial Spearman ρ = 0.33–0.46, controlling for reservation position, ZOPA width, and urgency). This provides complementary evidence linking the information interventions to opponent-modeling limitations.

We agree that information interventions can also change behavior beyond belief correction. Accordingly, we interpret the decomposition as diagnostic intervention effects rather than uniquely isolated cognitive mechanisms (App. E). We will make this distinction explicit in Fig. 1 and §1 and report the new analyses in the appendix.

Q3: Do model-specific diagnoses predict behavior beyond the aggregate correlation?

Yes, for most of the behavioral diagnoses we can test. From the existing round-robin (7 models, App. I.6):

Diagnosis in the simulatorHead-to-head
Surplus rankingρ = 0.93 (p = 0.003)
Negotiation lengthρ = 0.75 (p = 0.05); GPT-4o-mini shortest and GLM-5.1 longest in both settings
GPT-4o-mini refuses winnable deals (deal rate 52%, lowest surplus)deal rate 35% (others 83–99%), lowest surplus; in the human probe it rejects feasible deals in round 1 (App. I.7)
Every frontier model has a seller advantagepresent for all six frontier models

These results show that TERMS-Bench captures model-specific behavioral patterns that recur with unrestricted LLM counterparts. Transfer is weaker for deal-rate rankings (ρ = 0.21), consistent with agreement saturation among frontier models in the simulator (93–100%). We will clarify both the transferable findings and their limitations in the revised discussion.

We will also improve the clarity of the presentation in the camera-ready version. If any point remains unclear, we would be very happy to clarify it further during the discussion period. Please don't hesitate to let us know!

Dear reviewer, thank you again for your thoughtful review! We wanted to kindly follow up on our rebuttal, particularly the new experiments on language-reactive counterparts, observation-only posteriors, and model-specific diagnostic robustness. As the discussion period closes today, we would greatly appreciate hearing whether these additional results have addressed your concerns and whether you might consider revisiting your score.

Please don't hesitate to let us know if any questions or concerns remain - we would be very happy to address them. Thank you again for your time and consideration!

Reviewer_bGz7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes TERMS-Bench, a Bayesian-game-based benchmark for assessing the capabilities of LLM negotiation agents, going beyond outcome-based evaluations such as deal rates. Unlike traditional LLM vs LLM evaluations, it enables failure attribution by utilizing an Environment-Simulated Counterpart that specifies the Latent Type and Counterpart Policy. In a bilateral price negotiation setting, three economic regimes and six counterpart families were constructed, and 13 LLMs and 5 baselines were evaluated. It also analyzes the performance gap among Inference, Uncertainty, and Control using Oracle Intervention. The results showed that even among models exhibiting high deal rates, there were significant differences in Surplus Efficiency, Cue Utilization, Belief Calibration, and Protocol Compliance.

Critical strengths
  • Moving beyond the conventional LLM vs LLM evaluation approach, an interesting method is to enable failure attribution for negotiation agents by utilizing a explicitly defined counterpart policy. I believe this is a meaningful contribution in that it goes beyond simple outcome-based ranking and proposes diagnostic evaluation.

  • The Bayesian-game-based mathematical formulation is well structured, and the analysis distinguishing the performance gap between inference, uncertainty, and control through oracle intervention is impressive. It is also beneficial for utilizing various diagnostic axes such as Terminal Value, Agreement Calibration, Opponent Modeling, and Protocol Compliance.

  • Extensive experiments were conducted with a variety of frontier models and non-LLM baselines. A particular strength is the additional review of the validity and transferability of the proposed benchmarks through robustness analysis, behavioral profile analysis, and model-vs-model round-robin.

Critical weaknesses
  • In the intro, it was explained that negotiation lies at the intersection of Reasoning, Communication, and Social Cognition. However, how these capabilities are connected to the proposed mathematical formulations and diagnostic metrics is not sufficiently revealed. In particular, if the reasons why bilateral price negotiation is appropriate for evaluating these capabilities are explained and how each is measured in the benchmark, the motivation will become clearer.

  • Although it is explained that TERMS-Bench is designed based on Economic Bargaining Theory, there is a lack of motivation regarding how this theory leads to the design of the Economic Regime, Counterpart Families, and specific Counterpart Policies. Although detailed explanations are provided in the appendix, since this is an important foundation of benchmark design, it would be better to explain these connections more specifically in the main text as well.

  • While it is good that various frontier models were evaluated, the use of benchmarks would be more effective if the analysis focused on a more systematic evaluation axis rather than merely comparing individual models. For example, by comparing performance changes based on parameter size or reasoning effort within the same open-weight model family, the relationship between model configuration and negotiation capability can be analyzed more clearly.

Questions for authors

Overall, I think the paper was well written, and I found it an interesting read. Benchmark design and the various diagnostic analyses were particularly impressive. I understand that some content was placed in the appendix due to page limits. However, since proposing a new benchmark, I think it is necessary to explain the design process and theoretical rationale of the benchmark more specifically in the main text so that readers can understand the motivation behind the evaluation environment and the rationale for its design.

While comparing the performance of the latest frontier models through benchmarks is important, it is also worth considering that researchers can use benchmark results to determine model selection or design agent when developing actual negotiation agents in the future. In a landscape where new models emerge rapidly, i believe that analyzing not only between individual models but also focusing on evaluation axes that are clearly distinguishable, such as parameter size and reasoning rffort, will make it more useful for researchers to use TERMS-Bench to understand model capabilities and design follow-up studies.

Q1. Could you explain more clearly how reasoning, communication, and social cognition are connected to the TERMS-Bench's mathematical formulation and diagnostic metrics?

Q2. How has Economic Bargaining Theory been reflected in the design of the Economic Regime and the Counterpart Family, and can you add specific design rationale in the Main Text?

Q3. Can you add capability analysis based on evaluation axes such as parameter size or reasoning effort to better demonstrate the usability of the benchmark?

4 comments
AuthorsOpenReview ↗

Response to Reviewer bGz7

We would like to thank the reviewer for the thoughtful feedback. Below are our responses to the reviewer's weaknesses and questions.

W1/Q1: How reasoning, communication, and social cognition connect to the formulation and metrics

We agree that this connection should be clearer. Negotiation combines reasoning, communication, and social cognition in a semi-verifiable setting: outcomes are observable, but counterpart intentions, information use, and intermediate decisions are difficult to assess. TERMS-Bench makes these dimensions diagnosable by specifying the counterpart's latent type and policy. Bilateral price negotiation preserves hidden information, multi-turn interaction, and language while allowing exact computation of the ZOPA, IR constraints, Bayesian reference beliefs, and full-information policy π⋆\pi^\star, so the core diagnostics require no LLM judge. We will clarify how the three capabilities map to the formulation and diagnostics in §2:

CapabilityFormulationVariationDiagnostics
ReasoningPolicy mapping states to actions under utility, IR, and deadlinesOverlap; Urgency-shift; No-dealSE+, AGR+, FAGR−, CritViol%; control gap
CommunicationDialogue history and agent messagesCue reliability; voice on/off; language-reactive counterpart (new)Cue penalty; voice ablation; language-response effects (new)
Social cognitionBeliefs over counterpart type (reservation, urgency, stance)Latent types; counterpart familiesReservation/urgency belief errors; stance Brier score; inference and uncertainty gaps

Rather than reducing negotiation capability to a single outcome-based ranking, these diagnostics distinguish sources of performance gaps. For example, with voice off, unresolved uncertainty accounts for 63% of Grok 4.20's oracle gap, while the control-side gap accounts for 61% of GLM-5.1's. Such differences point to different improvements for each agent, which a single outcome score would not reveal.

W2/Q2: How bargaining theory shapes the regimes, families, and counterpart policy

We agree that this rationale is an important foundation of the benchmark, and we will move the following from App. A into §2–3 in the camera-ready version.

Economic regimes. Classical bargaining theory defines a problem by its feasible set and disagreement outcome (Nash, 1950; Roth, 1979). Overlap and No-deal determine whether a mutually acceptable agreement exists, making surplus extraction (SE+) and exit discipline (FAGR−) separate objectives. Urgency-shift holds reservation geometry fixed and varies only time pressure, which shapes concessions and disagreement (Stuhlmacher et al., 1998; Karagözoğlu & Kocher, 2019).

Counterpart families. The families span two axes. Economic reactivity captures how much responses depend on the agent's recent behavior rather than the latent type, mirroring the behavior-dependent and time-dependent tactics of Faratin et al. (1998). Cue reliability captures whether language reflects latent stance. The resulting 2×2 gives Candid, Taciturn, Expressive, and Strategic, while Stochastic and Adversarial serve as stress conditions.

Counterpart policy. Reservation values are the canonical private parameter in incomplete-information bargaining (Chatterjee & Samuelson, 1983; Ausubel et al., 2002). Each kernel component instantiates a studied primitive: acceptance conditions with an IR floor (Baarslag et al., 2014), a deadline clock and walk-away hazard for time pressure, and first-offer and concession effects (Petrowsky et al., 2025; Tey et al., 2021).


References

  • Ausubel, L. M., et al. (2002). Bargaining with incomplete information. Handbook of Game Theory with Economic Applications, Vol. 3.
  • Baarslag, T., et al. (2014). Effective acceptance conditions in real-time automated negotiation. Decision Support Systems.
  • Chatterjee, K., & Samuelson, W. (1983). Bargaining under incomplete information. Operations Research.
  • Faratin, P., et al. (1998). Negotiation decision functions for autonomous agents. Robotics and Autonomous Systems.
  • Karagözoğlu, E., & Kocher, M. G. (2019). Bargaining under time pressure from deadlines. Experimental Economics.
  • Nash, J. F. (1950). The bargaining problem. Econometrica.
  • Petrowsky, H. M., et al. (2025). The power and peril of first offers in negotiations. Organizational Behavior and Human Decision Processes.
  • Roth, A. E. (1979). Axiomatic Models of Bargaining. Springer.
  • Stuhlmacher, A. F., et al. (1998). The impact of time pressure in negotiation: A meta-analysis. International Journal of Conflict Management.
  • Tey, K. S., et al. (2021). The impact of concession patterns on negotiations. Organizational Behavior and Human Decision Processes.

(To be continued in the next comment.)

W3/Q3: Capability analysis along parameter size and reasoning effort

We agree that these axes make the benchmark more useful for model selection and agent design. We ran both sweeps on the full suite. UU is mean utility, and the gap is U(π⋆)−UU(\pi^\star) - U, with U(π⋆)=16.39U(\pi^\star) = 16.39.

Reasoning effort (DeepSeek-V4-Pro).

EffortUUGapvs. xhighDeal rate
low11.215.18−0.47 [−0.86, −0.09]0.65
medium11.215.18−0.47 [−0.90, −0.08]0.64
high11.544.85−0.13 [−0.53, +0.27]0.65
xhigh11.684.71—0.64

Higher effort raises utility through better prices: xhigh closes 9% of the gap relative to low, and the benefit appears above medium effort. Deal rate and violation rate (0.3–0.4%) stay flat, so a deal-rate evaluation would report no effect of effort (Finding 1).

Model size (Qwen3.5).

SizeBase UUUU, type revealedGapRecovered by revealing typeControl
9B5.686.1510.714%96%
27B9.0312.317.3645%55%
122B-A10B8.3110.398.0826%74%
397B-A17B10.3210.896.079%91%

The gap falls 43% from 9B to 397B-A17B. The 9B model gains little from the revealed type. The dense 27B model benefits most and outperforms the 122B-A10B mixture-of-experts model at base (+0.72 [+0.41, +1.05]), so total parameter count alone does not order capability. For agent design, these profiles point to different interventions: counterpart information has the most room to help at 27B, while better pricing policy matters more at 9B and 397B-A17B. Because sizes in this family also differ in architecture, we treat this sweep as a within-family case study and leave a general scaling law to future work. We will report both sweeps in §4 and the appendix.

If any point remains unclear, we would be very happy to clarify it further during the discussion period.

Thank you for the detailed responses. My concerns have been adequately addressed, and I will maintain my positive score.

Dear reviewer, thank you again for your encouraging response! We're very glad that our clarifications and new experiments have addressed your concerns. Your suggestions directly motivated the reasoning-effort and model-size analyses, which we believe meaningfully strengthen the benchmark.

As the discussion period closes today, we'd be grateful if you might reconsider whether these additions warrant an upward revision of your score. We sincerely appreciate your thoughtful engagement and support throughout the process!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Rebuttal Summary: Motivation, Methodological Contribution, and New Evidence

We sincerely thank all four reviewers for their constructive feedback. Their suggestions motivated new experiments substantially strengthening TERMS-Bench's diagnostic validity, robustness, and practical usefulness. We close by highlighting the broader motivation, methodological contribution, and key rebuttal findings.


1. Why this problem matters

Negotiation is not merely a stylized AI task: it underpins real economic activity across freight marketplaces, enterprise procurement, sales, and insurance settlements, where human negotiation remains central. As autonomous agents enter these markets, reaching agreements is only the first step. Practitioners must also understand where agents systematically fail, what information they fail to use, and what to improve before deployment. This challenge arose directly from our experience developing negotiation agents for a live industrial spot-freight marketplace.

Negotiation exemplifies a broader methodological gap in semi-verifiable agent evaluation: outcomes are observable, but hidden preferences, intermediate decisions, and counterfactual alternatives are difficult to verify. Unlike math or code, these strategic interactions lack an intrinsic verifier. Outcome-based or LLM-vs-LLM evaluations provide limited guidance for diagnosing systematic failures.

2. What TERMS-Bench contributes

Our central contribution is an environment-as-verifier methodology. By specifying the counterpart's latent type, policy, and payoffs, TERMS-Bench makes reference beliefs, economic constraints, and optimality gaps computable. This enables diagnosis of which information improves an agent's performance, by how much, and what gap remains even with full information. The structured counterpart is a deliberate methodological choice and the source of verifiability. It makes otherwise opaque counterfactual comparisons measurable. Bilateral price negotiation is our first controlled instantiation, retaining hidden preferences, multi-turn interaction, economic constraints, and language.

3. What the new experiments establish

In direct response to the reviewers, we conducted complementary analyses:

  • Practical diagnostic usefulness (Reviewer bGz7). New reasoning-effort and model-size sweeps reveal differences concealed by deal rates. Higher reasoning effort improves DeepSeek-V4-Pro's utility by 0.47 without changing agreement rates. Type revelation recovers 45% of Qwen3.5-27B's oracle gap but only 4% of Qwen3.5-9B's, identifying different improvement opportunities.

  • Robustness to counterpart behavior (Reviewer cfx4). A new language-reactive counterpart allows messages to affect acceptance and concessions. Under moderate reactivity, model-specific type-revelation patterns persist; stronger reactivity reveals meaningful communication effects. These results complement the LLM round-robin and product-grounded evaluations.

  • Validity of information interventions (Reviewers syXk, cfx4, fWW3). A wrong-type placebo reverses gains from correct type revelation (Grok 4.20: +3.99 vs. −3.68; GLM-5.1: +1.70 vs. −1.31), confirming that informational correctness matters. Observation-only posteriors preserve the principal diagnostic distinctions, supported by additional posterior-format tests.

  • Validity of belief diagnostics (Reviewers syXk, fWW3). New analyses link structured beliefs to subsequent decisions and the value of type revelation. A fixed-cohort test confirms that the flat or increasing belief-error pattern persists for 10 of 12 models on identical episodes.

These completed experiments directly address the principal methodological concerns raised during review. We will incorporate them and clarify the interpretation of oracle gaps as recoverable performance differences under specified information interventions.

4. Broader significance

We believe the central contribution of TERMS-Bench is to make systematic failure analysis possible in semi-verifiable agentic interactions, where aggregate outcomes alone provide limited guidance for evaluating and improving agents. Through its environment-as-verifier methodology, TERMS-Bench demonstrates how structured environments can yield computable reference policies, verifiable economic diagnostics, and actionable insights into agent performance. We believe that this opens an important methodological direction for reliable evaluation and improvement of autonomous agents in hard-to-verify domains, particularly as they move into real-world economic workflows where understanding systematic failures is essential before deployment.


We sincerely appreciate the reviewers' thoughtful engagement, including Reviewer bGz7's confirmation that their concerns have been addressed. We hope this methodological contribution, together with the substantial new experimental evidence, will receive consideration in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.