- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EmoWeave, a benchmark for evaluating large language models in Social Media Emotional Support Conversation (SM-ESC). The authors formulate emotional support in social media as an interactive setting where models respond to public posts and adapt to subsequent user feedback. They construct a benchmark of 500 text and image cases from FDU-CommentR Weibo posts and propose a three-perspective evaluation protocol involving a supporter model, a profile-conditioned simulated poster, and an independent bystander evaluator. The benchmark evaluates models from multiple dimensions, including emotional trajectory, response quality, and safety. Experiments on nine LLMs show that general model capabilities do not fully align with emotional support performance, and interactive feedback training based on EmoWeave improves Qwen2.5-7B-Instruct on held-out cases.
Critical strengths
1. Meaningful problem formulation and realistic application scenario. The paper identifies an important limitation of existing emotional support benchmarks: many existing settings focus on explicit help-seeking conversations, while social media users often express implicit emotional needs through public posts. The proposed SM-ESC formulation better reflects real-world scenarios where a supporter must infer user needs from limited public evidence and adapt responses as interactions evolve.
2. Comprehensive evaluation framework. The proposed three-perspective evaluation protocol provides a richer evaluation paradigm beyond traditional response-generation metrics. By combining simulated poster feedback, independent quality assessment, and safety evaluation, the benchmark attempts to capture multiple aspects of emotional support, including emotional outcomes, interaction quality, and potential risks.
Critical weaknesses
1. Questionable validity of simulated-user evaluation. The benchmark relies heavily on LLM-based poster simulation to measure emotional improvement and interaction outcomes. Although the authors conduct consistency checks and human evaluations, these studies mainly assess plausibility rather than whether simulated users accurately reflect real human reactions. Therefore, it remains unclear whether EmoWeave measures genuine emotional support ability or merely compatibility with an LLM simulator.
2. Limited empirical validation of benchmark reliability and utility. The benchmark contains only 500 cases, and the paper reports model comparisons mainly based on point estimates without sufficient analysis of statistical reliability, such as ranking stability or variance across subsets. Moreover, while the training experiment demonstrates improvements for one supporter model, it remains unclear whether the learned benefits generalize across different models, evaluator configurations, or real-world user interactions. More comprehensive validation is needed to establish EmoWeave as a reliable framework for both evaluation and model improvement.
Questions for authors
Q1. How well do simulated poster reactions correlate with real user responses? Have the authors conducted human studies comparing simulated feedback with actual user reactions?
Q2. Since multiple LLMs are involved as supporter, poster, and evaluator, how sensitive are the benchmark results to the choice of these models? Does changing the evaluator model significantly alter model rankings?
Q3. How do the authors address potential evaluator preference bias introduced by the simulated poster and bystander models? Could the benchmark unintentionally favor models whose responses align better with the generation style or preference distribution of these evaluators?
Q4. The benchmark measures emotional improvement as an important metric. How do the authors ensure that emotional gain does not simply reflect short-term sentiment changes but corresponds to meaningful emotional support?