acceptodds
Under review as a conference paper at ICLR 2027

Scorable Negotiation Self-Play for Natural-Language Reasoning and Understanding

Abstract

Can learning to negotiate improve language models beyond negotiation itself? We answer this question affirmatively with Language Model Self-Play via Scorable Negotiation Game (LSSG), a post-training framework that transfers skills from multi-turn bargaining to natural-language reasoning and understanding without any downstream reasoning labels. The key idea is to pair open-ended language interaction with continuous outcome rewards computed directly from transaction prices, with no LLM judge. LSSG warm-starts a single buyer–seller model by generalization-aware behavioral cloning on real dialogues, then refines it by stability-aware self-play, which combines payoff-driven updates with supervised retention and semantic diversity and emotional stability regularization from frozen pretrained scorers. Across seven benchmarks spanning commonsense reasoning, textual inference, logical reasoning, sentiment classification, and knowledge-intensive understanding in English and Chinese, LSSG achieves the highest accuracy among all compared methods in every one of the 14 model–benchmark settings on Qwen3-8B and Qwen3-14B, with macro-average gains of 4.61 and 3.14 percentage points over the vanilla models and up to 8.61 points on a single benchmark. Accuracy rises with every self-play iteration, and LSSG further raises win rate and payoff in two interactive negotiation games, also under adversarial social personas, establishing scorable negotiation as a judge-free training environment whose benefits extend well beyond bargaining.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-Play, LLM Negotiation, Natural-Language Reasoning and Understanding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_f95NOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether training a language model to engage in negotiations can simultaneously enhance its capabilities in reasoning and language understanding tasks outside of the negotiation context. The authors propose LSSG, a framework in which a single model assumes the roles of both buyer and seller, with rewards for each side determined by the final agreed-upon price. The training process begins with imitation learning based on human negotiation data, followed by an iterative cycle of self-play—generating dialogues and updating the model to refine its strategy. This process incorporates a supervised learning retention term, as well as regularization terms for semantic diversity and emotional stability. The authors conducted experiments using Qwen3-8B and Qwen3-14B, demonstrating performance gains across seven reasoning and comprehension benchmarks. They also evaluated the model’s performance in novel negotiation scenarios and across various persona styles, while employing ablation studies to examine the contributions of self-play and the two regularization terms.

Critical strengths

In my view, the most valuable aspect of this paper is that the reported performance gains span two model scales and a variety of task types—with improvements extending well beyond the initial imitation learning phase. As shown in Table 1, compared to models that underwent only negotiation-based imitation learning, subsequent self-play yielded further average improvements of approximately 2.43 and 1.90 percentage points for the 8B and 14B models, respectively; this provides a solid rationale for further research into the transferability of negotiation training. Furthermore, the core negotiation reward can be calculated directly from the final agreement, following clear and easily verifiable rules. The authors also incorporated interactive evaluations across different negotiation formats, offering a more comprehensive assessment of the models' practical negotiation capabilities.

Critical weaknesses

My primary reservation concerns the lack of clarity regarding the specific contributions of this paper relative to existing work. SPAG already employs imitation learning for warm-starting, followed by offline self-play, to enhance the reasoning capabilities of language models; meanwhile, SPIRAL has demonstrated that negotiation-based training can yield reasoning gains even in non-negotiation contexts. In the realm of emotion-aware negotiation, EmoDistill [1] focuses on emotional selection and expression; PRISMA [2] utilizes a combination of supervised initialization, DPO, and self-training to develop emotion-aware negotiation strategies and responses; and GENTEEL-NEGOTIATOR [3] incorporates objectives such as politeness, coherence, and the reduction of repetitive responses into the negotiation training process. While these approaches differ from the current paper in their specific goals and implementations, they underscore the need for a more precise articulation of the unique value added by the combination of continuous price-based rewards, semantic diversity, and emotional regularization. Furthermore, the current experimental setup fails to sufficiently validate the individual impact of these proposed design elements. The "G-AG" entry in the tables represents only the imitation learning baseline and does not constitute the full SPAG framework. Although Figure 4 includes an ablation study—removing self-play and regularization terms—it lacks a comparison against alternative methods, such as reward-based selection or supervised learning using filtered trajectories (while maintaining a consistent set of training trajectories and a comparable computational budget). Consequently, the specific value of the continuous price information has not been effectively isolated. Additionally, pushing the output probabilities of an emotion classifier toward 0.5 is not, in itself, sufficient evidence that negotiation decisions have become more stable; nor do the existing experiments on persona-based styles isolate the specific contribution of emotional regularization. In summary, while I acknowledge the performance improvements reported in the paper, I remain unconvinced that these gains are sufficient to establish the independent contribution of the newly introduced design components.

[1] Long, Y., et al. (2026). EmoDistill: Offline Emotion Skill Distillation for Language Model Agents in Adversarial Negotiation. Findings of EMNLP 2026.

[2] Kajare, P. V., et al. (2026). PRISMA: Preference-Reinforced Self-Training Approach for Interpretable Emotionally Intelligent Negotiation Dialogues. ACL 2026.

[3] Priya, P., et al. (2025). GENTEEL-NEGOTIATOR: LLM-Enhanced Mixture-of-Expert-Based Reinforcement Learning Approach for Polite Negotiation Dialogue. AAAI 2025.

Questions for authors

I would like the authors to further clarify the specific training stage(s) in which the primary contributions of this work—relative to SPAG and SPIRAL—are introduced. Additionally, please elaborate on the specific benefits gained from the proposed sentiment regularization and semantic diversity objectives, particularly in comparison to EmoDistill [1], PRISMA [2], and GENTEEL-NEGOTIATOR [3]. Regarding the experiments, could the authors provide a comparison with relevant methods—using the same base model and a comparable training budget—that retain the full self-play phase? Furthermore, by employing constant rewards, shuffled rewards, or direct supervised learning on the same set of successful trajectories, is it possible to disentangle the effects of continuous price information from the impact of trajectory filtering itself? Concerning "emotional stability," are there results from experiments on character-based style—conducted without sentiment regularization—or other independent behavioral metrics that demonstrate the model’s enhanced ability to maintain its negotiation objectives when subjected to emotional interference? Finally, please provide details on the evaluation splits, sample sizes, and the aggregation methods for the results presented in Table 1; specifically, explain how the accuracy for "CB" (Character-Based/Character-Consistent) is calculated, and indicate whether the training process was repeated independently. These clarifications will be instrumental in my assessment of the paper’s novel contributions and the reliability of its experimental results.

Reviewer_56yVOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether training language models through negotiation self-play can improve natural-language reasoning and understanding beyond negotiation. The proposed framework, LSSG, first warm-starts a shared buyer–seller model through behavioral cloning on human negotiation dialogues, then performs iterative self-play updates using role-specific rewards computed from transaction prices. Training additionally incorporates supervised instruction retention, semantic diversity regularization, and sentiment regularization. On Qwen3-8B and Qwen3-14B, the authors report macro-average accuracy improvements of 4.61 and 3.14 percentage points across seven downstream benchmarks, together with improved interaction performance in two additional negotiation environments.

Critical strengths
  1. The research question is valuable, and the outcome feedback is intuitive. Negotiation combines open-ended, multi-turn language interaction with explicit transaction outcomes, enabling continuous, role-specific outcome rewards without an LLM judge. This provides a worthwhile direction for post-training without directly using downstream reasoning answers.

  2. The evaluation covers multiple dimensions of transfer. The authors evaluate two model scales, seven language benchmarks, and two negotiation environments with different rules. The reported improvements extend beyond negotiation to commonsense reasoning, logical reasoning, and Chinese-language understanding. Further gains after behavioral cloning suggest that the subsequent self-play stage merits additional investigation.

Critical weaknesses
  1. The comparisons do not sufficiently isolate the contribution of continuous price feedback or establish an advantage over existing self-play methods.

G-AG, G-NG, and G-SG in Table 1 are behavioral-cloning baselines and do not substitute for comparisons with complete self-play algorithms. The paper discusses closely related approaches such as SPAG and SPIRAL, with SPIRAL also studying negotiation and reasoning transfer. Comparisons under matched backbones, data, and computational budgets are therefore needed to establish the practical advance over prior work.

Moreover, training retains only successful negotiation trajectories. Removing the entire SASP loss in Section 4.4 demonstrates the usefulness of that training component as a whole, but does not distinguish the contribution of continuous price rewards from the effects of selecting successful dialogues and continuing training on them. Controls using constant rewards, shuffled price rewards, or ordinary SFT on the same successful trajectories would help establish the necessity of the central design.

  1. The paper does not sufficiently explain where the downstream capability improvements from negotiation training come from.

The discussion on page 2 concerning offer tracking, role-specific constraints, and multi-turn planning provides an intuitive motivation, but does not demonstrate how these abilities transfer to particular downstream tasks. This is especially important for knowledge-intensive benchmarks such as MedMCQA: improvements in answering medical questions after training on transaction-domain dialogues are surprising and call for more concrete supporting evidence.

The accuracy results alone do not reveal whether the gains arise from better reasoning, improved constraint handling, or more effective use of knowledge already present in the model. Since downstream transfer is the paper’s central contribution, I suggest providing representative comparisons of the model’s answers and reasoning before and after training on the same downstream questions under identical prompts, particularly for medical tasks. Analyzing which reasoning steps improve would make the main contribution more convincing.

Questions for authors
  1. How does LSSG compare with complete self-play methods such as SPAG and SPIRAL under matched model sizes, training data, and computational budgets?

  2. On the same successful negotiation trajectories, how much additional benefit do continuous price rewards provide over constant rewards, shuffled rewards, or ordinary SFT?

  3. Could the authors provide before-and-after comparisons of answers and reasoning on identical downstream questions, particularly MedMCQA, and analyze how the observed improvements relate to negotiation training?

  4. Please clarify the number of runs, result aggregation, and the meaning of “±” in Table 1, particularly how CB accuracies such as 80.04% and 82.62% are obtained under the evaluation protocol.

Reviewer_mcYSOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LSSG, a post-training framework that turns a scorable negotiation game into a judge-free, label-free supervision signal. A shared buyer-seller policy is first warm-started via generalization-aware behavioral cloning, then refined by stability-aware self-play using price-based terminal rewards, supervised retention, and semantic/emotional regularizers. On Qwen3-8B and Qwen3-14B, LSSG reports the best accuracy in all 14 model-benchmark settings across seven language reasoning/understanding benchmarks, and also improves win rate and payoff in two NegotiationArena games, including under adversarial personas.

Critical strengths
  1. The core idea is clean and consequential: negotiation outcomes provide continuous, interpretable, judge-free rewards, and the paper shows that such self-play transfers beyond the training game to seven reasoning/understanding benchmarks. This matters because it supports a label-efficient post-training route without LLM judges or reasoning annotations.

  2. The empirical evidence is broad and consistent: gains appear at two model scales, across all 14 benchmark settings, grow over self-play iterations, and are accompanied by ablations isolating SASP and both regularizers, plus interactive negotiation and persona-robustness evaluations. This substantially strengthens the central claim.

Critical weaknesses
  1. The objective contains several auxiliary components, especially the emotional-stability regularizer toward a 0.5 positive-sentiment target, whose connection to negotiation and downstream reasoning is not well justified beyond ablation gains. This matters because it makes the method look somewhat kit-schy and leaves unclear which mechanism truly drives transfer.

  2. SASP retains only successful agreements and discards failed negotiations. This may bias self-play toward already-successful trajectories and removes potentially useful negative learning signal. The paper does not analyze how this filtering affects stability or downstream transfer, which matters for the claimed outcome-driven refinement.

Questions for authors
  1. Can you provide a clearer mechanistic justification for the emotional-stability regularizer, and analyze whether its gains are specific to SST2-related sentiment or general to reasoning transfer?

  2. How sensitive are the results to retaining only successful agreements? What happens if failed negotiations are included with zero or shaped returns?

  3. Can you report variance or significance for the interactive negotiation results over the 60 runs, and clarify whether the downstream benchmark gains are correlated with negotiation payoff improvements?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.