acceptodds
Withdrawn from ICLR 2027

When “Agreed” Is Not Agreement: Semantic Option Value in LLM-Agent Negotiation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language-model agents can negotiate terms whose wording admits multiple future executions. Existing benchmarks measure agreement or utility but rarely test whether agents select semantic breadth because it preserves a valuable choice after uncertainty resolves. We formalize semantic option value as the expected-utility gain from interpreting a term after ob- serving the state rather than committing ex ante to its best fixed interpretation. We then introduce SOV-Neg, a wording-matched factorial benchmark that crosses an SOV-generating proposer-utility regime with interpretation control and revelation timing. In preregistered self-play, broad-clause agreement increases most when the proposer benefits from flexibility, controls interpretation, and accepts before state revelation. Transcript decomposition locates the response in proposer offers rather than responder offers, and the pattern replicates across benchmark tiers and remains stable under domain-exclusion analyses. The response is nevertheless model-dependent and sub-normative: agents under-realize the available option, obtain no reliable proposer-utility gain, and reduce agreement, responder utility, and social welfare. A utility analysis further shows that the regime changes the full proposer payoff geometry, so the causal estimand is the joint regime-by-protocol interaction rather than a pure effect of scalar option value. Finally, a clarity instruction reduces residual risk and a singleton-only interface removes semantic flexibility. Our interpretation commit-and-reveal protocol pre-serves broad wording but finalizes only matching executable interpretations. The results make contingent semantic breadth measurable while delineating the intervention and protocol guarantees precisely.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, automated negotiation, semantic ambiguity, semantic option value, multi-agent systems, incomplete contracts, agent evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_phB3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether LLM agents favor broad contractual language to preserve valuable future choices. It formalizes semantic option value and introduces payoff-matched experiments varying interpretation control and information timing. Across 4,800 negotiations and four model families, the authors find increased broad-clause agreement in the predicted configuration, with substantial model heterogeneity. These behavioral effects do not yield reliable improvements in proposer utility or social welfare.

Critical strengths
  1. The paper addresses an interesting and consequential question: whether LLM agents favor broad contractual language to preserve strategically valuable choices. This moves negotiation evaluation beyond agreement rates to the execution implications of accepted language.

  2. The controlled experimental design is a substantive strength. Payoff-matched controls, combined with variation in interpretation control and revelation timing, provide a targeted test of incentive-sensitive behavior beyond a generic preference for flexible wording.

Critical weaknesses
  1. The matched controls also alter fixed-option values and dominance structure, limiting attribution of the observed effect specifically to semantic option value.
  2. Predefined clauses and execution sets limit the evidence for reasoning about natural-language ambiguity in realistic negotiations.
  3. The paper offers limited insight beyond responsiveness to valuable flexibility, leaving its broader scientific and practical significance unclear.
Questions for authors
  1. What substantive insight does the study establish beyond agents responding to valuable flexibility, and what practical decision does this insight inform?
  2. What evidence distinguishes sensitivity to semantic option value from responses to the changes in dominance structure and fixed-option values?
  3. What evidence shows that the findings concern natural-language ambiguity beyond choosing among explicitly defined contingent options?
Reviewer_HRuNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is evaluating if LLMs are able to infer if vague, flexible wording in negotiation agreements can be beneficial for them in future events in case of disruptions. The authors create a benchmark to study this. The authors further claim that their results indicate that not all models are able to exploit this.

Critical strengths

[originality] The research idea is nice - it is interesting to explore whether models are able to infer that vague wording can help them with the expected utility in the future. It could be very useful if executed well. It is also not trivial to execute, substantial work needs to be done to showcase this ability.

Critical weaknesses

[clarity] The paper is written really poorly, to the point that it is unreasonably challenging to understand it fully (some examples in questions below). I cannot evaluate the quality of the experiments and contribution because of this. I would suggest heavy rewriting to even be able to judge whether the study is sound.

Questions for authors

I had a lot of trouble going through this paper, despite putting effort into it. Here are some examples of things I did not understand:

  1. SOV first appears in the abstract before being defined, making it completely impossible to understand
  2. First paragraph in the intro gives an example that I did not understand at all
  3. From the introduction, it wasn’t clear to me what either of these are: SOV, payoff-matched benchmark intervention or DDD (this has a three row sentence that to me makes no sense)
  4. In Sec. 3 that describes the SOV, you never described what the states, what are executable outcomes, what is transfer x, or how the policy maps the state to the outcome. It’s also not clearly explained what is fixed or adaptive policy, so I had to guess that these refer to the agreement with clear vs. vague wording?
  5. In Sec 3. utility matrix appears without being defined prior to that as a part of what I think is an example, but this isn’t clear either. Example is, as a consequence, hard for me to understand
  6. In Sec 4, you introduce the term SOV-Neg intervention that is unclear to me.
  7. In Sec 4., what is payoff-matched intervention? What is dispersion here?
  8. Section 5 doesn’t describe any of the games, I didn’t get at all what the benchmark contains.

Given that I didn’t understand the whole framing of these technical sections, I don’t think I can judge results in Section 6, 7 and 8.

The abstract also has a weird formatting issue in line 15 which makes me think that the authors copied it from an existing PDF (as the word observing is broken into ob- and serving in the middle of the line for no reason).

Reviewer_CpidOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether LLM agents strategically exploit semantic ambiguity in negotiations to preserve flexibility under uncertainty. It formalizes Semantic Option Value (SOV) to quantify the benefits of adaptive contractual interpretation and introduces SOV-Neg, a controlled benchmark designed to isolate the effects of semantic flexibility, interpretation rights, and information revelation. Experiments across multiple LLM families provide evidence that agents are more likely to favor ambiguous clauses when they offer strategic advantages, potentially at the expense of their negotiating counterparts. The main contributions are the formalization of SOV, the development of a controlled evaluation framework, empirical insights into strategic ambiguity in LLM-agent negotiations, and an assessment of mitigation mechanisms.

Critical strengths
  1. Well-controlled experimental design A key strength is the careful separation of semantic ambiguity from the strategic benefits of flexible interpretation. By introducing matched zero-SOV controls and varying interpretation rights and information revelation, the authors provide a more convincing test of whether agents deliberately exploit ambiguity. This design makes the central findings more credible and gives the proposed benchmark clear methodological value.
  2. Insightful empirical findings across different LLMs The results reveal substantial differences in how LLMs respond to strategic ambiguity, suggesting that the observed behavior is not universal across models. I also appreciate that the authors distinguish increased use of ambiguous clauses from actual gains in utility or social welfare, rather than overstating their conclusions. This nuanced analysis adds depth to the empirical evidence and strengthens the paper's overall contribution.
Critical weaknesses
  1. Limited interpretability of the main experimental results Although the primary experiment reports a significant interaction effect, the lack of disaggregated results makes it difficult to determine whether the effect is driven by increased preference for ambiguous clauses in the intended experimental condition or by changes in other conditions. Some supporting behavioral analyses are drawn from a complementary experiment with a different design, which weakens their ability to explain the primary findings. In addition, certain claims regarding agreement, social welfare, and cross-play appear stronger than the reported evidence supports. Providing more detailed results and aligning the conclusions with the statistical evidence would improve the paper's credibility.
  2. Insufficient validation of the proposed mitigation mechanisms The evaluation of mitigation strategies does not clearly distinguish actual semantic disagreement from the mere existence of flexible interpretation options. In particular, the definition of unsafe agreements may classify mutually understood arrangements as risky even when no disagreement occurs. Moreover, the effectiveness of ICR partly follows from its design, which eliminates adaptive interpretation before uncertainty is resolved. While this reduces the measured risk, it remains unclear whether ICR offers meaningful practical advantages over simply restricting agreements to unambiguous interpretations. A more detailed analysis of disagreement rates, utility tradeoffs, and practical effectiveness would strengthen this contribution.
Questions for authors
  1. Clarification of experimental findings. Could the authors provide a more detailed breakdown of the matched experiment, particularly the results across different experimental conditions and model families? It would be helpful to clarify whether the observed effects are primarily driven by agents' preference for ambiguous clauses or by changes in other experimental conditions. Additionally, how are state assignments balanced across models and experiments?
  2. Effectiveness of ICR. The evaluation does not clearly distinguish actual semantic disagreement from residual option value. Could the authors clarify this distinction and explain the practical advantages of ICR over simply restricting ambiguous clauses?
  3. Evidence supporting the main conclusions. Some claims regarding reduced agreement rates, social welfare losses, and under-realization of option value appear stronger than the reported statistical evidence. Could the authors clarify which experiments directly support these conclusions and provide the relevant comparisons? A clearer definition of option capture and an explanation of how unsuccessful negotiations are handled would also improve the interpretation of the results.
  4. Lack of an overall framework figure. The absence of an overall framework figure makes the methodology and experimental workflow difficult to follow. A schematic illustrating the negotiation process, experimental design, and evaluation pipeline would improve clarity and readability.
Reviewer_K4QDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies negotiation over clauses that denote sets of executable outcomes. It defines semantic option value (SOV) as the expected-utility advantage of choosing an interpretation after observing the state over choosing the best fixed interpretation beforehand. SOV-Neg crosses proposer payoff alignment, interpretation control, and revelation timing. Its primary experiment uses 100 synthetic games and four models in 4,800 self-play sessions. Two zero-SOV controls preserve statewise proposer payoff distributions while removing cross-state preference reversals.

The primary broad-clause agreement DDD is +10.50 percentage points [3.75, 17.38]. The estimated interaction is positive for GPT and Gemini, inconclusive for DeepSeek, and negative for Claude. The within-target-cell contrast is a separate quantity. The matched experiment reports increased option capture and decreased responder utility, without reliable DDDs for agreement, proposer utility, or total welfare. A complementary benchmark supplies proposal-stage analyses. A mitigation study finds that interpretation commit-and-reveal (ICR) retains broad wording while requiring a shared executable interpretation before state disclosure.

Critical strengths
  1. A useful operationalization with a controlled comparison. Representing a clause as a set of executable outcomes makes the relationship between language and contingent action testable. The two matched-zero controls preserve statewise payoff distributions and reduce dependence on which outcome label dominates. The reported frozen confirmatory plan, game-level inference, and retention of unsuccessful negotiations in the primary endpoint are appropriate design choices.

2.Informative heterogeneity and a scoped protocol intervention. The paper reports the negative Claude contrast and null aggregate welfare findings rather than suppressing them. ICR illustrates how finalization can enforce a shared execution within a defined ontology. Its guarantee is appropriately limited to represented objects; it retains broad wording by committing its execution, rather than preserving the original state-contingent option.

Critical weaknesses
  1. Some conclusions exceed the identified contrast. Matching statewise payoff distributions does not match all payoff geometry. Applying the proposed control to the example in §3 changes the best fixed value from 6 to 9 while preserving adaptive value at 9. Thus, the intervention also changes the attractiveness of precise clauses and, with responder utilities fixed, the alignment between the parties' payoffs. Section 11 recognizes the scope, but statements that payoff geometry is matched or that scalar SOV itself is isolated should consistently describe a payoff-alignment-by-protocol interaction. Moreover, a positive DDD does not establish that the high-versus-zero difference is positive in the target cell; Claude's negative DDD likewise does not demonstrate target-cell ambiguity avoidance. The abstract also asserts reduced agreement and welfare although their matched confidence intervals include zero. The cross-play study should be described as a generalization comparison, since Appendix H explicitly says it does not estimate an SOV DDD.

  2. Missing inputs and decomposed results impede interpretation and replication. Section 5 does specify private utilities, public history, six turns, and a repair attempt. Nevertheless, the PDF supplies neither complete prompts/input examples nor the twelve primary cell means. The artifact is described as containing these materials, but its contents are not established by the manuscript alone. Cellwise agreement and termination counts are particularly useful given 879 turn-limit sessions. Appendix D.3 labels utilities as intention-to-treat, but does not specify nonexecution payoffs or an implementable option-capture formula. The denominator and nonexecution treatment of mitigation welfare are also unclear. These omissions prevent a complete audit of the behavioral and distributional conclusions.

Questions for authors
  1. Please report the twelve matched-condition broad-agreement rates and final-status counts, pooled and by model, together with the target-cell high-versus-zero contrasts and uncertainty. Does the claimed target-cell increase hold, and what do the corresponding Claude contrasts show?
  2. Please provide an accessible artifact location and one complete proposer/responder input example, including the disclosure of utilities, interpretation sets, control, and timing. How are realized states assigned or counterbalanced across treatments within each model, given one run per model-condition? Please also state the exact decoding settings.
  3. Please specify nonexecution utilities, the exact option-capture formula and its treatment of nonagreement, and the mitigation-welfare denominator. Please align the abstract and causal conclusions with the design-specific results and limitations above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.