- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces TERMS-Bench, a controlled benchmark for diagnosing LLM negotiation abilities beyond deal rate. It evaluates surplus, opponent beliefs, and constraint compliance, using six opponent families and oracle information interventions to probe performance bottlenecks. Experiments with 13 LLMs show that high agreement rates mask substantial differences in bargaining performance. Validation includes robustness checks, ablations, model-versus-model negotiations, and a small human study.
Critical strengths
-
The paper is well organized and easy to follow, with well-designed figures that clearly explain the framework and findings.
-
The benchmark reveals differences in surplus, belief accuracy, and constraint compliance that agreement rates alone miss. These diagnostics provide a more informative assessment of negotiation abilities.
-
Providing posterior information and revealing opponent types help probe why agents underperform. Robustness checks, ablations, and model-versus-model experiments further support the usefulness of these diagnostics and reduce concerns that the findings depend on one simulator configuration.
Critical weaknesses
-
The benchmark focuses on single-price bargaining with hand-designed opponents. Despite some external validation, it remains unclear whether its findings generalize to realistic negotiations.
-
Posterior interventions change both available information and its presentation. Failure to benefit may reflect difficulty using the summary rather than accurate prior inference, weakening the claimed separation of inference, uncertainty, and control.
Questions for authors
-
What evidence supports interpreting the intervention gaps as distinct inference, uncertainty, and control failures, rather than differences in how models use the supplied information? Have you tested sensitivity to the posterior’s presentation?
-
Beyond the shared reservation-value distribution and small human probe, is there evidence that the benchmark’s rankings or diagnoses generalize to other negotiation settings?
-
The oracle posterior uses latent cues underlying the messages, while the base agent only sees natural language. How much of the intervention gain comes from this additional information rather than improved inference?
-
Could the flat or increasing belief error across rounds reflect harder negotiations remaining unresolved? Does this finding persist when comparing the same set of episodes across rounds?
4 comments
Q3: How much of the intervention gain comes from privileged latent cues?
Short answer. We directly tested this by replacing the original posterior with one conditioned only on observed economic history. The inference gains and model-specific diagnostic profiles remain nearly identical.
We recomputed the posterior using only the prices and actions the agent observes, marginalizing over all six counterpart families. We evaluated this with voice off, where the agent receives no linguistic cues (720 matched episodes per agent).
| Grok 4.20 | GLM-5.1 | |
|---|---|---|
| Inference gain, original posterior | +0.58 [+0.26, +0.90] | +0.69 [+0.33, +1.05] |
| Inference gain, observation-only | +0.67 [+0.40, +0.94] | +0.62 [+0.33, +0.94] |
| Observation-only − original | +0.09 [−0.17, +0.36] | −0.06 [−0.37, +0.27] |
| Inference / uncertainty / control (%), original | 11 / 63 / 26 | 16 / 23 / 61 |
| Inference / uncertainty / control (%), observation-only | 12 / 62 / 26 | 14 / 25 / 61 |
The central diagnostic distinction persists: Grok 4.20 remains predominantly uncertainty-sensitive (62–63% of its gap), while GLM-5.1 retains a dominant control-side residual (61%). Neither inference-gain difference is statistically detectable. Thus, privileged latent-cue conditioning does not appear to drive these diagnostic profiles in the tested settings. Further details are in our response to Reviewer cfx4.
Q4: Could the belief-error trend reflect harder negotiations remaining unresolved?
Short answer. We directly tested this by tracking the same episodes across rounds. The flat or increasing belief-error pattern persists for 10 of 12 models, showing that changing episode composition alone does not explain the finding.
Fixed-cohort analysis (new). We restricted evaluation to episodes lasting at least six rounds and compared belief errors between rounds 1 and 6 on identical episodes. Five models show statistically significant increases in error, and another five show no statistically detectable improvement, with only two showing decreases. Thus, the central finding persists even when the episode population is held fixed.
(To be continued in the next comment.)
W1/Q2: Do the rankings and diagnoses generalize beyond the original simulator?
- Short answer. We find consistent surplus rankings and recurring behavioral diagnoses across three complementary evaluations, varying the counterpart, its language responsiveness, and the economic grounding of negotiation scenarios.
| Evaluation | What changes | Evidence of transfer |
|---|---|---|
| LLM round-robin (App. I.6) | Seven LLMs negotiate directly against one another | Surplus ranking ρ = 0.93 (p = 0.003); negotiation-length ranking ρ = 0.75; GPT-4o-mini retains the lowest surplus and a 35% deal rate (vs. 83–99% for others) |
| Language-reactive counterpart (new) | Agent messages directly affect acceptance and concessions | Aggregate ranking unchanged under moderate reactivity (4 agents); type-revelation gains preserve their ordering: Grok +4.26, GLM-5.1 +1.38, Qwen3.5 +0.61 (standard: +3.99, +1.70, +0.57) |
| Product-grounded suite (App. H.2) | Negotiations use real product categories and price statistics | Surplus-ranking correlation ρ = 0.83 (p < 0.01, 9 models); no false agreements |
-
Model-specific transfer. Beyond aggregate surplus correlation, the round-robin reproduces identifiable behavioral patterns: negotiation length rank-correlates across settings, GPT-4o-mini continues to struggle to reach agreements, and all six frontier models retain the seller-side surplus advantage. The six-episode human probe provides additional directional evidence, including GPT-4o-mini's early rejection of feasible deals (App. I.7).
-
Language-reactive extension. More importantly for diagnostic validity, the new counterpart allows messages to influence economic decisions while preserving objective utility evaluation. Under moderate reactivity, the relative benefits from revealing counterpart information persist across Grok, GLM-5.1, and Qwen3.5. Stronger language reactivity introduces larger communication effects and can change rankings (full results in our response to Reviewer cfx4).
We agree that bilateral single-price bargaining remains a controlled setting. The results establish transfer across the tested counterpart and product-grounding variations, while realistic multi-issue and human negotiation require further validation. The framework (§2.1) accommodates richer type spaces and negotiation protocols with specified counterpart policies.
We will incorporate the new experiments, clarify the scope of the benchmark, and strengthen the discussion of intervention-based diagnostics in the camera-ready version. We hope these results address the reviewer's soundness concerns and would be grateful if they could be considered in the final assessment.
Dear reviewer, thank you again for your thoughtful review! We wanted to kindly follow up on our rebuttal, including the new experiments on posterior presentation, observation-only inference, and fixed-cohort belief-error analysis, along with additional evidence on diagnostic transfer. As the discussion period closes today, we would greatly appreciate hearing whether these results have addressed your concerns and whether you might consider revisiting your score.
Please don't hesitate to let us know if any questions or concerns remain-we would be very happy to address them. Thank you again for your time and consideration!
We thank reviewer for the thoughtful and constructive review. We appreciate the positive assessment of the paper's clarity, diagnostic contributions, and supporting validation experiments. We conducted additional experiments addressing all four questions, with results summarized below.
Short answer. We tested whether the inference gap reflects difficulty using the supplied posterior. Simpler structured representations do not increase the gain, and additional controls show that the agents respond to the correctness of the information. These results support the diagnostic value of the interventions.
Posterior presentation (new). We tested alternative representations for Grok 4.20 on 720 matched episodes with voice off:
Neither simpler structured representation improves the inference gain. The plain-language version performs worse, confirming that presentation can affect information use; the paper's structured summary performs best among the tested formats in point estimate.
Evidence of prior inference (new). We replayed the exact Bayesian filter on the economic histories recorded for four agents. Their reported reservation-value errors (0.10–0.11 of the price range) are comparable to the filter's (approximately 0.10) and substantially below the prior's (0.19). This suggests that agents already extract meaningful counterpart information from their observations, leaving less additional value for the posterior intervention.
Same format, different information (new). We also tested whether type revelation helps merely because an additional structured prompt is provided. Supplying an incorrect type from another episode in the same condition reverses the true-type gains:
The agents respond to incorrect types in economically predictable directions: overly optimistic information induces excessive holdout, while pessimistic information induces excessive concessions. The sign reversal demonstrates that the value of type revelation depends on informational correctness, with the presentation held fixed.
Finally, the base-to-revealed-type gain and remaining control-side gap do not depend on posterior presentation. Across four agents, larger stated belief errors are also associated with greater benefits from revealing the true type (partial Spearman ρ = 0.33–0.46; see our response to Reviewer syXk). We agree that these quantities measure intervention effects and need not uniquely identify internal cognitive mechanisms. We will clarify this interpretation in Fig. 1 and §1 and include the new controls in the appendix.
(To be continued in the next comment.)