- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SIDE-Bench, a benchmark comprising 150 multi-turn scenarios. It evaluates LLM agents' strategic negotiation in complex social interactions. In addition, this paper proposes the Conversation-Partner Intent Tracking (C-PIT) framework, which tracks the agent's beliefs to infer the partner's intent across conversation turns.
Critical strengths
- The multi-dimensional evaluation protocol of the proposed benchmark is well-designed.
- The proposed benchmark evaluates LLMs in complicated social scenarios involving competing goals, hidden constraints, and relational risk.
Critical weaknesses
- The belief-tracking methodology of the proposed C-PIT framework resembles previous work, but it is not compared with any baseline.
- Lack of deep insights from in-depth case studies (beyond Appendix J) into why existing LLMs fail on the proposed benchmark.
Questions for authors
- Table 1 is outside the page margins. The "Class" column can be removed, as it is less informative.
- Figure 3: It is encouraged to present error bars in the chart to show statistical significance & confidence.
- Missing references: [1] tracks beliefs to enhance Theory of Mind for LLMs; [2] trains LLM agents to infer mental states and then complete the conversation on the Sotopia benchmark.
- [1] Sclar et al. ACL 2023. Minding Language Models’ (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Tracker.
- [2] Hwang et al. ACL 2026. Infusing Theory of Mind into Socially Intelligent LLM Agents.
3 comments
Rebuttal Acknowledgement by Reviewer JpV1
Thank you for the reply. Please include the key points in the revision. Overall, I would raise my rating a bit to reflect the updates. If possible, I would give a 2.5.
Thank you for the feedback. We will incorporate the key points into the revised version of our paper as advised. Regarding the score, the reviewing system does not support a 2.5 rating, and the next available rating is 3 (Weak Accept). We would greatly appreciate it if you could adjust your rating accordingly.
We thank reviewer JpV1 for the feedback. Please see how we address the feedback below:
Comparison with existing work
We thank the reviewer for pointing us to these relevant studies. Sclar et al. use symbolic belief graphs to improve Theory-of-Mind reasoning on narrative reading-comprehension tasks, while Hwang et al. use explicit mental-state generation and training to improve goal-directed dialogue agents in Sotopia. These works establish important components of ToM reasoning and socially intelligent interaction. C-PIT differs by evaluating off-the-shelf agents in high-friction, information-asymmetric, multi-turn interactions with hidden constraints and competing goals, while jointly measuring belief updates and whether subsequent actions are feasible under those beliefs. To the best of our knowledge, this specific evaluation of belief-action consistency in such interactions has not been studied in prior work. We will add and discuss both references in the revised paper.
Baseline clarification
Sclar et al. and Hwang et al. are related methods rather than direct baselines because they target different tasks and objectives. The appropriate baseline for C-PIT is standard SIDE-Bench generation without the explicit turn-level belief-state schema, while keeping the model, scenarios, conversation partner, and outcome evaluation fixed. C-PIT is therefore intended as a diagnostic evaluation layer for measuring belief revision and belief-action consistency in social interaction, rather than as a replacement for existing ToM modeling or agent-training methods. Using the original standard SIDE-Bench runs and the overlapping C-PIT runs, we obtained 348 valid matched scenario pairs across the seven models. The mean Goal Success was 0.810 for standard generation and 0.803 with C-PIT, a small difference of -0.007. The paired bootstrap 95% interval for this difference included zero, approximately [-0.022, 0.008], with mixed changes across models. This indicates that C-PIT is not intended as a performance-improvement intervention, but as a diagnostic evaluation component that exposes belief revision and belief-action mismatches that outcome-only metrics cannot distinguish.
More case studies for insights on model performance
In addition to the qualitative error analysis in Appendix J, we conducted a diagnostic audit of five complete episodes per model drawn from the lower-scoring tail for each of the seven evaluated models. We examined these conversations to identify recurring reasons for poor benchmark outcomes and distinguish behavioral failures from legitimate unresolved interactions. The resulting primary categories are summarized below.
Beyond this summary taxonomy, we documented how these behaviors manifested in individual cases across models, including the specific dialogue turns that supported each classification. We also analyzed the Goal Success score distributions across all 150 scenarios for each model to characterize differences in the lower-performing tails.
Missing references and suggested edits and plots
We thank the reviewer for these helpful suggestions. We will add and discuss the suggested references, fix the table overflow, and update Figure 3 with error bars showing score dispersion across domains. These presentation and related-work changes, together with the additional case-study analysis and score-distribution plot, will be incorporated into the camera-ready version of the paper.
We request the reviewer to consider our feedback and raise the score if they agree. Thank you for your thoughtful comments. They will make our paper better.