acceptodds
Under review as a conference paper at ICLR 2027

Strategic Behavior in Large Language Models Depends Critically on Game Structure

Abstract

Large language models (LLMs) are increasingly deployed as strategic agents. However, it remains unclear how their strategic behavior changes with game structure. To address this question, we conduct self-play experiments with 15 LLMs across 10 complete information games, 4 complex incomplete information environments, and an electricity-sector carbon-market bidding game. We find that strategic behavior depends critically on game structure. Equilibrium attainment in sequential games averages 63.2%, about 18 percentage points higher than simultaneous games, and 12 of 15 models perform better in sequential games. Moreover, attaining equilibrium does not imply selecting the Pareto-optimal equilibrium. In complex incomplete information environments, outcome-based metrics alone are insufficient, as models can achieve high scores while exhibiting low-quality interaction processes. To diagnose such failures, we propose Strategic Failure Detection (SFD), which decomposes strategic behavior into planning, coordination, action validity, and outcome. Applying SFD to the carbon-market game improves planning and coordination and raises welfare and terminal payoff achievement rate for DeepSeek-V3 and GPT-4o. These results show that process-level detection can make strategic behavior more observable and easier to improve.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, strategic behavior, game structure, behavioral game theory, complete information games, incomplete information games, carbon-market game

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_W3CwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how LLM behavior varies across game environments, including self-play with 15 models in ten complete-information games, analyses of four complex environments, and a simulated carbon market. It reports higher Nash-equilibrium attainment in its sequential-game group than in its simultaneous-game group, distinguishes equilibrium attainment from efficient equilibrium selection, and argues for evaluating interaction processes alongside outcomes. It also introduces Strategic Failure Detection (SFD), which checks planning, coordination, action validity, and outcomes and selects interventions based on the first detected failure.

Critical strengths
  • Useful breadth and behavioral detail: The study compares multiple model families across varied environments and distinguishes equilibrium attainment, efficiency, and interaction behavior rather than relying on a single outcome score.

  • Substantive paired intervention evidence: The authors report 120 paired records across three small games and two models, plus paired evaluation on 20 held-out carbon-market instances. These support the usefulness of the assistance package in several settings. The appendix also discusses mixed results, including the Wait–Go coordination decline.

Critical weaknesses
  • The structural explanation is not isolated by the comparison: The sequential and simultaneous groups contain different games, so timing and observability vary with payoffs, action spaces, and coordination demands. The 63.2% versus 45.4% equilibrium-attainment gap, consistent across most models, supports a descriptive finding for this suite but does not establish the explanation in Section 5.1. This limits the explanatory advance over prior game studies, including Hua et al., which already examine the same ten named games and structured workflows.

  • The distinctive benefit of SFD’s diagnosis and intervention selection remains unverified: The paired experiments establish package-level effects, but SFD also changes the prompting workflow, elicits opponent predictions, and, in the carbon market, supplies computed best responses and other diagnostic information. Without comparable fixed or generic assistance using the same tools, information, and budget, the gains cannot be attributed specifically to diagnosis-based selection. This leaves the proposed method’s main incremental contribution insufficiently supported.

Questions for authors
  1. Which existing comparison distinguishes timing or observability from the other differences between the game groups? If none does, how would narrowing the structural explanation affect the claimed contribution relative to prior work?

  2. What evidence establishes the benefit of diagnosis-based intervention selection beyond the added workflow and computational assistance? A fixed or generic assistance baseline with comparable resources would directly address this question; I am not assuming such an experiment can be completed during rebuttal.

  3. Where are the confidence intervals and paired tests referenced by Figure 5 as appearing in Table 2? Also, how often is the neutral-prediction fallback used in each carbon-market condition, and how does this affect the comparability of Planning scores?

5 comments
AuthorsOpenReview ↗

Response to Reviewer W3Cw (denoted as R4)

Q4-1 The structural explanation is not isolated by the comparison.

A4-1 We thank the reviewer for questioning whether the structural comparison isolates game format.

(1) Equilibrium invariant test. We converted the five simultaneous games to sequential counterparts while retaining payoff tables and action sets. Across 15 models and 10 runs per game and structure, attainment was:

GameSimultaneous (%)Sequential (%)
Prisoner's Dilemma6792
Stag Hunt7178
Battle of the Sexes3976
Wait-Go2984
Duopolistic competition2146
Overall45.475.2

Bold marks the higher rate in each row. In the Prisoner's Dilemma, both formats have the same unique equilibrium outcome, (D,D)(D,D), while attainment rises from 67% to 92%. The overall 29.8 percentage point difference is descriptive because sequentialization changes strategic opportunities in some games.

(2) Scientific findings. Our contribution extends beyond SFD. We find that strategic behavior varies with structure and scale, equilibrium selection differs across Stag Hunt models, outcome metrics miss process failures in incomplete information games, and observable carbon market failures can be detected and improved.

Q4-2 What establishes the benefit of diagnosis-based intervention selection?

A4-2 We thank the reviewer for asking what SFD adds beyond generic assistance.

(1) Matched control. Generic Feedback and SFD share triggers, evidence types, tools, maximum generation slots, per-call limits, attempt caps, and negotiation. Generic gives fixed feedback, while SFD selects it by diagnosed failure. Packet values may differ as trajectories diverge.

(2) Results. P and C are Planning and Coordination rates. Lower BR gap is better. TPAR is terminal payoff achievement rate. Results are:

ModelMethodPCWelfare (10610^6 CNY)BR gap ↓TPAR
GPT-4oGeneric26.2%18.0%165.91.00998.3%
SFD (ours)100.0%92.0%168.90.005100.0%
Llama3.1-70BGeneric47.7%36.2%146.30.45899.4%
SFD (ours)97.6%82.7%152.30.009100.0%
DeepSeek-V3Generic100.0%100.0%149.60.000100.0%
SFD (ours)100.0%100.0%149.60.000100.0%
GLM-4.5Generic52.0%9.0%109.77.16873.0%
SFD (ours)100.0%95.0%149.60.004100.0%

Bold marks the best value within each model block. Ties are also bold. SFD improves the reported metrics over Generic for three models. DeepSeek-V3 ties at ceiling.

Q4-3 Where are Figure 5’s confidence intervals and paired tests?

A4-3 We thank the reviewer for catching this omission. The revised manuscript reports these tests in Appendix D.7.1.

We compare SFD with Direct on 30 paired instances. The table reports mean paired differences, 95% bootstrap confidence intervals, and two-sided Wilcoxon signed-rank and paired t-test p-values. Welfare is in 10610^6 CNY. pWp_W and ptp_t denote the respective tests.

ModelMetricΔ (SFD vs Direct)95% CIpWp_Wptp_t
DeepSeek-V3P+9.4 pp[+2.7,+16.3]0.0250.013
C+1.7 pp[−1.7,+5.0]0.320.33
Welfare+25.4[+1.0,+60.7]0.0730.12
BR gap ↓−5.2[−13.0,+0.2]0.300.15
TPAR+7.7 pp[−0.3,+15.6]0.0550.073
GPT-4oP+57.5 pp[+52.7,+62.3]<0.001<0.001
C+3.3 pp[+0.8,+6.7]0.0460.043
Welfare+23.3[+14.0,+35.9]<0.001<0.001
BR gap ↓−3.7[−6.1,−1.6]<0.0010.004
TPAR+11.1 pp[+8.0,+14.0]<0.001<0.001

Action validity was 100% in both conditions. Only PP is significant for DeepSeek-V3. For GPT-4o, all five reported metrics are significant under both tests.

Q4-4 How often is neutral-prediction fallback used, and how does it affect Planning comparability?

A4-4 We thank the reviewer for asking us to quantify fallback use.

(1) Fallback. In the revised scorer, neutral prediction fallback is disabled. Its use is therefore zero. Missing or invalid predictions are N/A, not passes.

(2) Planning coverage. The table reports the percentage of scorable Planning decisions in each condition.

ModelDirect LLMCoT PromptingGeneric FeedbackSFD (ours)
GPT-4o95.4%93.9%96.5%95.1%
Llama3.1-70B95.1%94.1%96.1%95.7%
DeepSeek-V3100%100%100%99.8%
GLM-4.599.5%98.3%100.0%98.3%

No Planning score uses a fallback value. We corrected the manuscript's neutral substitution description and specify the rule in revised Appendix D.7.2.

Thank you for the additional experiments. The matched sequentialization study and Generic Feedback baseline substantially address my original concerns. My remaining concern is the incremental contribution over prior game studies and structured workflows. Could you clarify the most important new insight beyond the closest prior work and which revised results establish it? I currently lean toward 2 (Weak Reject) but remain open to raising my rating based on this clarification.

AuthorsOpenReview ↗

Further Discussion on the Incremental Contribution

Dear Reviewer W3Cw,

Thank you for acknowledging that the matched sequentialization study and the Generic Feedback baseline substantially address your original concerns. We understand your remaining concern is whether the contribution is incremental relative to prior game studies and structured workflows. We answer this directly below.

1. Insight. Beyond the closest prior work, the most important new insight is that the gain comes from diagnosis-based selection of repair feedback. In our matched comparisons, the gain is not explained by additional reasoning, generic feedback, extra tools, or larger budgets. The factor that changes is whether repair feedback is selected by the diagnosed failure. Prior behavioral game studies document LLM failures, and prior structured workflows score or guide reasoning. To our knowledge, the closest prior work we compare does not combine a four-dimension diagnosis with a closed repair loop. Our contribution is matched empirical evidence for such a loop.

2. Revised results that establish the insight. The matched Generic Feedback comparison in the carbon-market game is the result that most directly establishes the insight. It covers four models, 30 matched instances, and five runs per instance. Evidence, tools, attempt caps, and budgets are held fixed. Only the feedback selection rule changes. The mean results are:

MethodPCWelfare (10610^6 CNY)BR gap (lower is better)TPAR
Direct LLM19.5%18.8%66.39.67040.5%
CoT Prompting19.7%18.8%68.99.09342.4%
Generic Feedback56.5%40.8%142.92.15992.7%
SFD (ours)99.4%92.4%155.10.005100.0%

In this setting, CoT Prompting yields little improvement over Direct LLM. Generic Feedback helps but remains far below SFD. Relative to Generic Feedback, mean Planning rises from 56.5% to 99.4%, and mean Coordination rises from 40.8% to 92.4%. The complete information results provide generalization evidence. Across eight models, SFD raises equilibrium attainment from 46.3% to 64.8% in simultaneous games and from 59.0% to 99.5% in sequential games. We also mark a boundary openly. In the asymmetric Wait-Go game, GPT-4o falls from 0.50 to 0.30 under SFD.

3. Position relative to prior work. First, behavioral studies such as Akata et al. (2025) characterize failures and modulate behavior with fixed prompts. Within the scope of our comparison, they do not diagnose failures into dimensions or close a repair loop. Second, process benchmarks such as GAMEBoT, DSGBench, and WGSR-Bench score reasoning steps offline. They do not feed evaluation signals back to the agent. Third, PrimeNash derives equilibria analytically. It does not repair the behavior of deployed agents. Relative to these lines, the increment is matched empirical evidence that a detect, decompose, and repair loop with diagnosis-selected feedback improves deployed-agent behavior.

4. Our contribution. To state it plainly, the novelty we claim is closed-loop empirical evidence with a matched control that isolates diagnosis-based intervention selection. The paper also reports discovery-oriented findings. Strategic behavior depends critically on game structure. Equilibrium attainment does not imply Pareto-optimal equilibrium selection. Outcome metrics alone miss process failures. We hope this direct answer clarifies the increment over prior work.

Thank you again for your thoughtful feedback and helpful suggestions.

Best regards,

Authors

Reviewer_TqRGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper evaluates LLMs on 10 complete information games, 4 incomplete ones, and a bidding game for how game structure changes LLMs' strategic behavior. It finds, in sequential games, equilibrium is easier to achieve than in simultaneous games. Then the paper proposes Strategic Failure Detection (SFD), to improve LMs' planning ability in games and see improvements in a carbon-market bidding game.

Critical strengths

The idea Strategic Failure Detection is well-motivated and implemented. It improves performance as the experimental results show.

Critical weaknesses

My biggest concern is about the conclusion that sequential games' equilibrium is easier to attain than simultaneous games: a strong confounder here is the design of the games. The sequential games evaluated seem to have unique equilibriums while the simultaneous games don't.

Questions for authors
  1. In the SFD condition, what information are the models given?
  2. Can you experiment on a game whose equilibrium does not change over time? Or could you provide some insights if any existing game is already a game of such? The purpose is to split a. the strategic difficulty of a game and b. sequential vs. simultaneous format of the game.
3 comments
AuthorsOpenReview ↗

Response to Reviewer TqRG (denoted as R3)

Q3-1 Can you test a game whose equilibrium does not change between simultaneous and sequential play?

A3-1 We thank the reviewer for suggesting this controlled comparison.

(1) Equilibrium invariant case. We converted the five simultaneous games to sequential counterparts while retaining their payoff tables and action sets. In the Prisoner's Dilemma, the unique simultaneous Nash equilibrium and the unique sequential subgame perfect equilibrium path are both (D,D)(D,D). Across 15 models and 10 runs per game and structure, attainment in this game rises from 67% to 92%.

(2) Results. Attainment rates across the five games are:

GameSimultaneous (%)Sequential (%)
Prisoner's Dilemma6792
Stag Hunt7178
Battle of the Sexes3976
Wait-Go2984
Duopolistic competition2146
Overall45.475.2

Bold marks the higher rate in each row. The unchanged Prisoner's Dilemma outcome shows that higher attainment also occurs when the equilibrium outcome is unchanged. The overall 29.8 percentage point difference is descriptive because sequentialization changes strategic opportunities in some games. The full model-by-game results are included in Appendix B.4.

Q3-2 In the SFD condition, what information are the models given?

A3-2 We thank the reviewer for asking us to specify the SFD information set.

(1) Base information. At the initial decision, we provide the game rules and payoffs, the agent's role and state, legal actions, and history observable under the game protocol. SFD does not reveal opponents' private predictions or unsubmitted actions.

(2) Decision and feedback. In simultaneous games, the agent submits an action and a sealed self-reported prediction of current opponent actions. This prediction is not an observation. In sequential games, the agent observes preceding public actions. After a preliminary episode, or before a carbon market final bid, SFD provides evidence relevant to the selected failure, such as payoff comparisons, coordination evidence, or action constraints. After a completed carbon episode, terminal payoff evidence may precede one fresh episode. Carbon feedback never reveals another firm's private proposal or unsubmitted bid.

Reviewer_TqRGOpenReview ↗

acknowledgement

I thank the authors for the clarification and for presenting new results. I would like to keep my positive scores.

AuthorsOpenReview ↗

Thank You to Reviewer TqRG

Dear Reviewer TqRG,

Thank you for your response and for keeping your positive scores. We appreciate your constructive feedback throughout the review process, and we will incorporate it into the final version of the paper.

Best regards,

Authors

Reviewer_87CZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how the behavior of LLMs agents changes across games with different structure and it conducts extensive experiments with ten complete- information game, four complex multi-agent environments, and a carbon-market simulation. The main claim of this paper is that the strategic behavior of LLMs depends critically on game structure, since this paper finds that LLMs can attain higher equilibrium in sequential games compared to simultaneous games. This paper also proposes Strategic Failure Detection (SFD), which separates planning, coordination, action validity, and outcomes, and uses these diagnoses to guide interventions.

Critical strengths

Exploring the behavior of LLMs in different games is a very interesting research direction. Recently, a series of studies have been conducted in this area. These studies have contributed to understanding of the behavior of LLMs in different games and their capabilities under various games.

Critical weaknesses

weakness 1

My biggest concern with this paper is the lack of rigor in its use of mathematical notation, including numerous notations that are introduced without any definition. As an academic research, it requires precise definitions and consistent notation, so the failure to clearly define even the basic notations significantly undermines my confidence in this paper.

For example, in lines 180–181, the stochastic game is defined by the tuple (N,S,Ai,q,ui)(N, S, {A_i}, q, {u_i}), yet no definition of qq is provided in either the main text or the appendix. Besides, although one might infer that NN denotes the set of agents, its meaning should be explicitly stated rather than left to the reader’s interpretation. Similarly, the paper uses sts_t to denote a state, whereas the game definition includes SS. I think that the authors want to use SS to denotes the state space and let st∈Ss_t\in S, but this relationship needs to be made explicit. Although Ui(t)U_i^{(t)} is described as a continuation payoff, its relationship to uiu_i, and the continuation strategy under which it is evaluated, remain unclear. These omissions matter because different continuation assumptions can produce different best responses and diagnostic labels. Likewise, Vi(t+1)V_i^{(t+1)} appears in line 198, but its meaning is never defined anywhere in the paper.

Although these issues may appear minor in isolation, collectively they indicate a troubling lack of precision and care in the presentation of the research, which significantly diminishes my confidence in the subsequent experimental results.

weakness 2

My another concern is that the experiments in the paper may not be sufficient to support its arguments.

For example, in the comparison results with and without SFD, SFD not only tells the model "where the problem is", but also provides additional feedback and even a calculated suggestion for the best response. Therefore, the improved performance after using SFD may be because the model receives more direct action suggestions, more feedback. The existing experiments do not adequately distinguish the role of these factors in "accurately diagnosing the type of failure." To demonstrate the value of the diagnosis itself, a fair control is needed: for example, both groups receive the same number of prompts, the same calculation results, and the same number of attempts, while only one group chooses intervention based on the specific type of failure. This is necessary to determine exactly how much additional improvement the targeted diagnosis contributes.

Besides, it is questionable whether the coordination metric in the paper truly measures coordination ability. In the electricity-sector carbon-market bidding game, assuming four companies participate in four rounds of trading, the paper aims to examine whether these companies made coordinated and reasonable decisions in each round. However, according to the paper's payout formula: when companies buy and sell allowances, the money paid by one company is the money received by another. Adding up the payouts of the four companies, these transaction payments offset each other, and the allowances are merely transferred among the four companies, with no change in the total amount. Therefore, the total payout of the four companies ultimately depends only on the price in the final round. The problem lies in how it scores the earlier rounds. For example, when evaluating the first round, the paper retains the actual operation of the first round and then assumes that the second to fourth rounds are executed according to a pre-calculated reference scheme. This fixes the price in the final round. The result is that as long as the transactions meet the constraints, the total revenue metric remains unchanged regardless of how the operations in the first round are altered. The same problem exists when evaluating the second and third rounds. According to the formula in the paper, the coordination loss in these three rounds will be zero, automatically passing the check. Therefore, the high "coordination pass rate" reported in the paper cannot be used to prove that the model indeed coordinated very well in the early rounds.

Questions for authors
  1. Could the authors provide precise definitions for all notations that are currently left undefined in the paper as pointed in weakness 1?
  2. In light of the concerns raised in weakness 2, could the authors adopt fairer experimental setup for the comparative experiments involving SFD?
  3. The most important question is the coordination metric. According to the issues raised in weakness 2, the coordination metric does not appear to adequately capture how effectively the models coordinate. Could the authors clarify this point? I would appreciate any corrections if I have misunderstood the metric or its interpretation.
3 comments
AuthorsOpenReview ↗

Response to Reviewer 87CZ (denoted as R2)

Q2-1 Several game-theoretic symbols are undefined or inconsistent, including NN, S\mathcal S, qq, and Vi(t+1)V_i^{(t+1)}. Please define the notation precisely.

A2-1 We thank the reviewer for identifying the notation issues. We revised §3.2 and Appendix E.1.

(1) Game notation. We define N={1,…,n}N=\{1,\ldots,n\} as the agent set, S\mathcal S as the state space, and st∈Ss_t\in\mathcal S as the stage tt state. We define Ai(st)\mathcal A_i(s_t) as agent ii’s legal actions and at=(a1,t,…,an,t)a_t=(a_{1,t},\ldots,a_{n,t}) as the joint action. The kernel qt(⋅∣s,a)q_t(\cdot\mid s,a) gives the next state distribution, and q=(q1,…,qT)q=(q_1,\ldots,q_T) collects these kernels.

(2) Payoffs. We define ui,t(st,at)u_{i,t}(s_t,a_t) as the stage payoff and gi(sT+1)g_i(s_{T+1}) as the terminal payoff, so Ui=∑t=1Tui,t(st,at)+gi(sT+1)U_i=\sum_{t=1}^{T}u_{i,t}(s_t,a_t)+g_i(s_{T+1}). We replace the undefined Vi(t+1)V_i^{(t+1)} with Ui(t)U_i^{(t)} in Appendix E.3 and define the diagnostic notation in Appendices E.2 to E.6.

Q2-2 How is continuation payoff Ui(t)U_i^{(t)} related to stage payoff uiu_i, and which continuation strategy is held fixed?

A2-2 We thank the reviewer for asking us to clarify this definition.

(1) Payoff. In revised Appendix E.3, we define Ui(t)=Eκ[∑τ=tTui,τ+gi(sT+1)]U_i^{(t)}=\mathbb E_\kappa[\sum_{\tau=t}^T u_{i,\tau}+g_i(s_{T+1})]. Payoffs before tt are excluded.

(2) Fixed continuation. We hold κ\kappa fixed across candidate actions. We fix current opponent actions in simultaneous games, use backward induction in sequential games, and in the carbon market retain actual past bids and fix future bids to the reference sequence (revised Appendices E.1 to E.3).

Q2-3 Does SFD outperform a fair control with matched information, calculations, prompts, and attempts, differing only in diagnosis-guided intervention selection?

A2-3 We thank the reviewer for requesting a matched control.

(1) Control. Generic Feedback and SFD share diagnostic triggers, calculations, information types, tools, negotiation opportunities, maximum generation slots, per-call limits, and attempt caps. Both receive the same types of payoff and feasibility evidence. Generic gives fixed general feedback, while SFD selects a dimension-specific intervention.

(2) Results. Carbon-market results, with C rescored under the revised criterion, are:

ModelMethodPCAWelfare (10610^6 CNY)BR gap↓TPAR
GPT-4oDirect0.8%0.0%100.0%111.96.34765.3%
CoT1.2%0.0%99.7%111.56.83964.7%
Generic26.2%18.0%100.0%165.91.00998.3%
SFD (ours)100.0%92.0%100.0%168.90.005100.0%
Llama-3.1-70BDirect0.5%0.0%99.7%77.19.14647.0%
CoT2.3%0.0%97.3%87.26.49955.9%
Generic47.7%36.2%99.2%146.30.45899.4%
SFD (ours)97.6%82.7%99.7%152.30.009100.0%
DeepSeek-V3Direct75.0%75.0%100.0%015.7590.0%
CoT75.0%75.0%100.0%015.7590.0%
Generic100.0%100.0%100.0%149.60100.0%
SFD (ours)100.0%100.0%100.0%149.60100.0%
GLM-4.5Direct1.5%0.0%100.0%76.07.42649.7%
CoT0.3%0.0%100.0%76.77.27349.1%
Generic52.0%9.0%100.0%109.77.16873.0%
SFD (ours)100.0%95.0%100.0%149.60.004100.0%

SFD improves over Generic on the reported metrics for GPT-4o, Llama-3.1-70B, and GLM-4.5. DeepSeek-V3 is tied at ceiling.

Q2-4 The former aggregate coordination loss may be unchanged by early bids because inter-firm payments cancel and the fixed future reference bids determine later prices. Does the reported C rate therefore overstate coordination?

A2-4 We thank the reviewer for identifying this limitation.

(1) Revised criterion. We agree the former aggregate-loss criterion could yield zero loss in early periods. We now score C using the maximum enterprise-specific normalized unilateral best response gap under actual opponent bids and the fixed reference continuation. C passes only if all comparisons are scorable and the maximum gap is at most εCcarb=0.10\varepsilon_C^{\mathrm{carb}}=0.10. Otherwise, C is N/A.

(2) Results. C rates:

ModelGeneric FeedbackSFD (ours)
GPT-4o18.0%92.0%
Llama-3.1-70B36.2%82.7%
DeepSeek-V3100.0%100.0%
GLM-4.59.0%95.0%

I appreciate the authors' response. However, given the imprecise notation in the paper, as well as the authors' acknowledgment in their rebuttal (e.g., section A2-4 in their response) that the aggregate-loss criterion may yield zero loss during the early periods, I believe that the coordination metric, which constitutes the paper's core contribution, fails to meaningfully measure coordination ability. Therefore, I believe the paper does not yet meet the acceptance threshold, and I maintain my original score.

AuthorsOpenReview ↗

Further Discussion on the Revised Coordination Metric

Dear Reviewer 87CZ,

Thank you for your continued review. We agree that the former aggregate-loss criterion was not adequate for measuring coordination. Your concern about a core process metric is well justified, and it led us to replace the criterion rather than patch it.

1. Notation and removal of the old criterion. We have addressed Q2-1 and Q2-2 in revised Appendix A.1, where κ\kappa and Ui(t)U_i^{(t)} are defined and each comparison holds κ\kappa fixed while varying only the agent's own current action. The former aggregate coordination loss no longer appears in the manuscript. Every Coordination rate is rescored under the new criterion.

2. Revised definition, aggregation, and edge cases. For enterprise ii at period tt, the unilateral best-response gap under actual opponent bids and the fixed reference continuation κ\kappa is

Ri,tC=max⁡x∈Fi,tUi(t)(x,b−i,t,κ)−Ui(t)(bi,t,b−i,t,κ).R_{i,t}^{C}=\max_{x \in F_{i,t}} U_{i}^{(t)}(x,b_{-i,t},\kappa)-U_{i}^{(t)}(b_{i,t},b_{-i,t},\kappa).

The normalized relative regret is

ρi,tC=Ri,tC(bi,t)Di,t.\rho_{i,t}^{C}=\frac{R_{i,t}^{C}(b_{i,t})}{D_{i,t}}.

The joint Coordination check at period tt is

Ct=1[max⁡i∈Nρi,tC≤εCcarb],εCcarb=0.10.C_{t}=\mathbf{1}[\max_{i \in N}\rho_{i,t}^{C}\leq \varepsilon_{C}^{\mathrm{carb}}],\qquad \varepsilon_{C}^{\mathrm{carb}}=0.10.

Here bi,tb_{i,t} is the actual bid of enterprise ii in period tt, b−i,tb_{-i,t} denotes the actual opponent bids in the same period, and Fi,tF_{i,t} is the feasible bid interval. Ui(t)U_{i}^{(t)} is the remaining payoff from period tt onward, with payoffs received before tt excluded. Di,tD_{i,t} is the attainable payoff range over the feasible interval and serves as the normalizer. Planning conditions on the enterprise's predicted opponent bids, while Coordination uses the actual opponent bids b−i,tb_{-i,t}.

The model-level C rate is a period-level pass rate, namely scorable periods with Ct=1C_t=1 divided by all scorable periods. N/A periods are excluded from the denominator. Appendix A.3 applies this audit rule to every dimension, including Coordination, whose coverage is the ratio of scorable to eligible C periods and is retained with the pass rate. Action Validity stays near 100% in all carbon-market conditions, and an invalid action is the main blocker of a Coordination comparison, so unscorable C periods are rare. When Di,tD_{i,t} is zero, the regret is undefined, the comparison is unscorable, and the joint check is N/A under the Appendix A.2 rule, never a pass. We have made these rules explicit in the revised manuscript.

3. How the revised criterion addresses the early zero-loss concern. Two properties address both mechanisms in Q2-4 concern. First, the joint check takes a maximum of enterprise-specific regrets and never sums them, so inter-firm payments cannot cancel. Second, each enterprise's own current bid is the deviation variable in its own regret test against actual current opponent bids, so an early bid deviation is assessed directly rather than merged across firms. The N/A rule prevents an unscorable early period from counting as a pass.

4. Rescored results. The Coordination rates under the revised criterion are:

ModelDirect LLMCoT PromptingGeneric FeedbackSFD (ours)
GPT-4o0.0%0.0%18.0%92.0%
Llama3.1-70B0.0%0.0%36.2%82.7%
DeepSeek-V375.0%75.0%100.0%100.0%
GLM-4.50.0%0.0%9.0%95.0%
Mean18.8%18.8%40.8%92.4%

Three of the four models score 0.0% under Direct LLM and CoT Prompting. These zero baselines show that the criterion is discriminating in these settings.

The DeepSeek-V3 row also illustrates the scope of C. Its Direct C rate is 75.0%, yet welfare is zero and TPAR is 0.0%. C is a current-period incentive-compatibility test under the fixed reference continuation. Only the current bid is varied in the comparison, which uses the remaining payoff Ui(t)U_i^{(t)} from tt to TT. C does not evaluate full-trajectory efficiency or terminal welfare, which the Outcome dimension captures. Section 5.4 discusses this process-outcome divergence.

Our paper's central claim is that strategic failures are observable, decomposable, and correctable. Strategic behavior depends critically on game structure. Equilibrium attainment does not imply Pareto-optimal equilibrium selection. Outcome metrics alone miss process failures.

We are grateful for this comment, which led us to replace rather than patch the criterion. We hope this clarification is helpful, and we are happy to provide further details if useful.

Best regards,

Authors

Reviewer_J9rZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how the strategic behavior of LLMs changes with game structure. The authors evaluate 15 LLMs, mostly in self-play, across 10 complete-information games (5 simultaneous, 5 sequential), 4 incomplete-information environments (two StarCraft-based combat settings, a cooperative cooking game, and Welfare Diplomacy), and a carbon-market bidding game built from Chinese electricity-sector data.

The key findings are that models reach Nash equilibrium more often in sequential games than in simultaneous ones, and that within the same model family, larger models tend to perform better. Reaching an equilibrium does not mean reaching the optimal one. In incomplete-information environments, outcome metrics such as win rate hide large differences in how models play.

The main claim of the paper is the benefit of the proposed method, Strategic Failure Detection (SFD). It checks each decision for failures in planning, coordination, action validity, and outcome, and gives the model targeted feedback on the first failure found. In the carbon-market game, SFD improves planning and coordination and raises welfare and terminal payoff for the two models it is tested on (DeepSeek-V3 and GPT-4o).

Critical strengths

The paper presents an extensive empirical sweep over 15 LLMs across model families and sizes. Evaluations are carried out across a large task suite, including 4 incomplete-information environments. The breadth of the results makes them a strong reference for the strategic behaviuor of LLM agents. The results show that strategic behaviuor varies substantially across both game types and models. Equilibrium attainment averages 63.2% in sequential games versus 45.4% in simultaneous games. Interestingly, the models with the highest overall equilibrium attainment (GPT-o1, DeepSeek-R1, GPT-OSS-120B) are among the worst at coordinating in Stag Hunt, which merits further investigation.

Critical weaknesses

The paper’s main proposed method, SFD, lacks sufficient evidence to support the claims made. In the main text, it is evaluated on only two of the 15 models and in a single environment. Furthermore, it is only compared to the same setup without SFD; no baseline method is used for comparison (e.g., generic feedback or chain-of-thought prompting).

The authors claim significant improvements in planning and coordination, yet coordination increases only from 80.8% to 82.5% (DeepSeek-V3) and from 75.8% to 79.2% (GPT-4o), on 20 game instances. No statistical significance testing is reported, even though the Figure 5 caption states that confidence intervals and paired tests are given in Table 2, and these changes could be within noise.

In addition, the appendix shows SFD decreasing DeepSeek-V3’s performance on Wait–Go (Table 3: equilibrium rate drops from 0.50 to 0.35), which is not mentioned in the main text. The authors should evaluate SFD across the full task suite, provide significance testing, and discuss where and why it may hurt performance.

Questions for authors

n/a

2 comments
AuthorsOpenReview ↗

Response to Reviewer J9rZ (denoted as R1)

Q1-1 SFD is tested on only two models in one environment. Please evaluate it across the task suite.

A1-1 We thank the reviewer for raising this concern.

(1) Evaluation. We evaluated four methods, Direct LLM (no SFD), CoT Prompting, Generic Feedback, and SFD, in five simultaneous games and five sequential games with complete information using eight models. We also evaluated all four methods in the carbon market game using four models. The carbon market results are reported in A1-2. Results for the simultaneous and sequential games are as follows:

MethodSimultaneous gamesSequential games
Direct LLM46.3%59.0%
CoT Prompting43.8%79.3%
Generic Feedback55.3%94.5%
SFD (ours)64.8%99.5%

SFD has the highest aggregate rate in both structures.

(2) Clarification. In this paper, we evaluate SFD beyond the carbon market. Because of space constraints and the greater complexity of the carbon market, we emphasize its results in the main text. The four-method comparison across all five simultaneous and five sequential games is reported in Table 2 and Appendix B.4.

Q1-2 The paper compares SFD only with the same setup without SFD. Please include baselines such as generic feedback and CoT.

A1-2 We thank the reviewer for this suggestion.

(1) Baselines. We compare four conditions: Direct LLM, CoT Prompting, Generic Feedback, and SFD. Generic Feedback is the key control for targeted diagnosis. It and SFD share diagnostic evidence, computational information, maximum generation slots, per-call token limits, retry caps, and negotiation opportunities. Generic Feedback gives fixed general guidance, while SFD selects an intervention based on the diagnosed failure.

(2) Results. The table summarizes carbon market results across four models. P, C, and A denote Planning, Coordination, and Action Validity pass rates. C uses the revised unilateral deviation criterion. Lower BR gap is better. TPAR denotes terminal payoff achievement rate.

MethodPCWelfare (10610^6 CNY)BR gap↓TPAR
Direct LLM19.5%18.8%66.39.67040.5%
CoT Prompting19.7%18.8%68.99.09342.4%
Generic Feedback56.5%40.8%142.92.15992.7%
SFD (ours)99.4%92.4%155.10.005100.0%

Bold indicates the best value in each column. SFD improves over Generic Feedback on the reported metrics for three models and ties it at ceiling for DeepSeek-V3. Per-model results are reported in Table 3 of the revised manuscript.

Q1-3 The reported planning and coordination gains may be within noise. Please report paired significance tests and confidence intervals.

A1-3 We thank the reviewer for requesting statistical evidence.

We paired 30 instances across Direct and SFD. The table reports mean paired differences, 95% bootstrap confidence intervals, and two-sided p-values from the Wilcoxon signed-rank and paired t-tests. Welfare is in 10610^6 CNY. pWp_W and ptp_t denote the two tests, respectively.

ModelMetricΔ (SFD vs Direct)95% CIpWp_Wptp_t
DeepSeek-V3P+9.4 pp[+2.7,+16.3]0.0250.013
C+1.7 pp[−1.7,+5.0]0.3200.330
Welfare+25.4[+1.0,+60.7]0.0730.120
BR gap ↓−5.2[−13.0,+0.2]0.3000.150
TPAR+7.7 pp[−0.3,+15.6]0.0550.073
GPT-4oP+57.5 pp[+52.7,+62.3]<0.001<0.001
C+3.3 pp[+0.8,+6.7]0.0460.043
Welfare+23.3[+14.0,+35.9]<0.001<0.001
BR gap ↓−3.7[−6.1,−1.6]<0.0010.004
TPAR+11.1 pp[+8.0,+14.0]<0.001<0.001

The table shows that, at the 0.05 level, only the Planning improvement is significant for DeepSeek-V3. For GPT-4o, the reported improvements in Planning, Coordination, Welfare, BR gap, and TPAR are significant under both tests. Action validity was 100% in both conditions. The statistical evidence is included in Appendix D.7.1.

Q1-4 SFD reduced DeepSeek-V3’s equilibrium rate on Wait–Go. Please discuss where and why SFD may hurt performance.

A1-4 We thank the reviewer for asking us to discuss this limitation.

In §6, we explain that Wait–Go’s equilibria, (W,G)(W,G) and (G,W)(G,W), require asymmetric choices. Coordination feedback may instead lead both agents to choose the safer action, producing the non-equilibrium (G,G)(G,G) outcome. For GPT-4o, equilibrium attainment falls from 0.50 to 0.30 under SFD (Appendix B.4). This illustrates that targeted coordination feedback may induce correlated errors in asymmetric games.

AuthorsOpenReview ↗

Clarification index for Reviewer J9rZ

Dear Reviewer J9rZ,

Thank you again for your careful reading and for the four concrete concerns in your review. We hope you are well. We are sharing a short index in case it is helpful for checking our response. There is no need to reply if you are busy.

  1. Single environment and only two models (Q1-1). A1-1 now reports four methods across eight models and ten complete-information games, plus four models in the carbon market. SFD attains the highest aggregate equilibrium rate in both structures.
  2. No comparison baseline (Q1-2). A1-2 adds Direct LLM, CoT Prompting, and a matched Generic Feedback control. SFD improves over Generic Feedback on Planning, Coordination, and BR gap.
  3. Statistical significance (Q1-3). A1-3 and Appendix D.7.1 report paired tests on 30 matched instances with Wilcoxon and paired t-tests and 95% bootstrap confidence intervals. We state the mixed outcome honestly.
  4. The Wait–Go decline (Q1-4). A1-4 and Section 6 explain the mechanism and treat it as a boundary condition.

We hope this makes the new evidence easy to locate. If any point remains unclear, we would be glad to clarify.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the AC Meta-Review: Reviewers' Concerns and Revisions

We thank all four reviewers for their detailed and constructive comments. The paper is a behavioral study and then a method study. From self-play experiments with 15 LLMs, we find that strategic behavior depends critically on game structure, that equilibrium attainment does not imply Pareto-optimal equilibrium selection, and that outcome metrics alone miss process failures. These findings motivate Strategic Failure Detection (SFD), a framework that decomposes decisions into planning, coordination, action validity, and outcome and makes the detected failures observable and correctable.

Below we summarize the main concerns and how we addressed them in the revised draft.

  1. Evaluation scope and baselines (J9rZ, W3Cw): We extended SFD to four methods (Direct LLM, CoT Prompting, Generic Feedback, SFD) across eight models and ten complete information games [Table 2, Appendix B.4] and four models in the carbon market [Table 3]. Generic Feedback is a resource-matched control differing only in diagnosis-based feedback selection. This isolates the main new insight over prior work: the gain comes from diagnosis-based selection rather than additional reasoning or generic feedback. SFD attains the highest equilibrium rate in both structures (64.8% simultaneous, 99.5% sequential).

  2. Statistical significance (J9rZ, W3Cw): We added paired tests on 30 matched instances with Wilcoxon and paired t-tests and 95% bootstrap confidence intervals [Appendix D.7.1]. GPT-4o gains are significant on all five metrics, while for DeepSeek-V3 only the Planning gain reaches the 0.05 level.

  3. Game structure (TqRG, W3Cw): To isolate structure from game design, we converted the five simultaneous games into sequential counterparts with identical payoffs and action sets. Even when the equilibrium outcome itself is unchanged, as the (D, D) outcome in the Prisoner's Dilemma, changing the structure alone raises attainment from 67% to 92%, and the overall rate rises from 45.4% to 75.2% across the five games, showing that game structure is itself a key determinant of strategic behavior [Appendix B.4].

  4. Validity of the coordination metric (87CZ): We replaced the former aggregate-loss criterion with a unilateral best-response regret test under actual opponent bids and a fixed reference continuation, with an explicit N/A rule, and rescored all Coordination rates [Appendices A.1 to A.3, Table 3]. The rescored mean rates are 18.8% for Direct LLM, 18.8% for CoT Prompting, 40.8% for Generic Feedback, and 92.4% for SFD.

  5. Notation and definitions (87CZ): We defined every game-theoretic symbol, including the agent set, state space, transition kernel, stage and terminal payoffs, and continuation payoff, in revised Section 3.2 and Appendices A.1 and E.1.

  6. Limitations (J9rZ): We added the Wait-Go analysis to Section 6, showing that coordination feedback can induce correlated errors in asymmetric equilibria, with GPT-4o equilibrium attainment falling from 0.50 to 0.30 [Appendix B.4].

We believe these revisions have substantially strengthened the paper. The comprehensive empirical evidence and the transparent re-evaluation of our metrics demonstrate that the paper delivers both scientific insights and a validated method.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.