- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces EMPAThE, a benchmark for testing whether emotionally charged natural-language framing changes LLM strategic behavior in game-theoretic environments. The authors keep the nominal payoff matrices fixed and use an automated LLM-based pipeline to generate emotional variants of the prompts. They compare equilibrium adherence across eight games (five simultaneous, three sequential) and five small open-weight LLMs. They evaluate pure-strategy equilibrium adherence (PNE/SPNE) and a mixed-strategy compliance score (MNE) over 100 sampled rounds per condition. The main empirical claim is that emotional framing systematically moves models away from Nash-equilibrium behavior, with particularly large changes in several sequential games and coordination games.
Critical strengths
The paper asks an interesting and potentially important question with a simple experimental design: can emotionally salient language alter strategic decisions when the underlying formal game is unchanged? Separating payoff structure from linguistic framing is conceptually clean (payoff preservation is checked automatically by a regex verifier), and it could grow into a useful benchmark for studying the contextual robustness of LLM agents.
Critical weaknesses
- It seems that the equilibrium metrics, as implemented, do not adequately support the paper’s claims about strategic rationality. For sequential games, the paper scores whether the realized terminal/joint action outcome belongs to a set of pure or SPNE profiles, but subgame-perfect equilibrium is a strategy profile specifying actions at every information/history node, not merely a terminal action sequence. Checking terminal-outcome membership cannot establish that an agent followed a subgame-perfect strategy. There is also a conceptual problem with the MNE metric: the authors average the theoretical mixed strategies of the two player roles into a single probability vector and compare it with pooled action frequencies. This loses the player-specific strategic structure of the equilibrium. A model could match the pooled marginal frequency while using the wrong strategy for each player. Consequently, the reported compliance and degradation scores may not measure equilibrium adherence in the sense the paper claims.
- The emotional manipulation does not isolate emotion from semantic and contextual changes. The central causal claim is that emotion changes strategic reasoning, but the emotional prompts differ from the controls in much more than emotional intensity: length, syntax, narrative structure, renamed action labels, moral valence, and specific content. For example, the control Prisoner’s Dilemma is a generic game description, whereas the emotional version is a detailed scenario about a tenured professor facing scandal, expulsion, betrayal, and tenure, with panic and agony. Observed shifts may therefore stem from role-play, relabeled actions (e.g., “FRAME_RIVAL” instead of “DEFECT”), or distraction rather than emotion. Without neutral controls matched on length, syntax, and action-label structure, the causal attribution to emotion is not supported.
Questions for authors
- The evaluation uses five relatively small and older open-weight models. Do the results replicate with contemporary LLMs?
- Why is there no discussion of closely related work, e.g., GTBench, EAI, B3Emo, and Economics Arena for LLMs?
- Can the authors rerun the experiments with neutral paraphrases that match the emotional prompts in length, syntax, and action-label structure but contain no high-arousal language? Would the NE degradation persist?
- For asymmetric games, can the authors report MNE compliance per player role rather than aggregating across roles? Would the conclusions change?
2 comments
Dear authors, thanks for your efforts. I see a strange addition of embodied agent interface, which is irrelevant to your paper... I believe, it should be clear from context that I meant the paper "EAI: Emotional Decision-Making of LLMs in Strategic Games and Ethical Dilemmas"
We thank the reviewer for recognizing the value and simplicity of our work and recognizing the potential of growing this work into a useful benchmark on contextual robustness of LLMs.
Please refer to the general comment regarding the concerns of keeping the prompt isolated to emotional changes only and on larger models.
Below contains answers to your other questions.