acceptodds
Under review as a conference paper at ICLR 2027

EmoWeave: Towards Emotional Support Conversation on Social Media

Abstract

Social media users share emotional experiences that may call for support without an explicit request for help. We formulate Social Media Emotional Support Conversation (SM-ESC), in which a model provides support through public comments and adapts to the poster's feedback. Evaluating new replies requires poster reactions that archived exchanges cannot provide. We introduce EmoWeave, a benchmark of 500 text and image cases grounded in FDU-CommentR Weibo posts. Its three-perspective protocol combines supporter replies based on public evidence, poster reactions simulated from private profiles, and independent bystander judgments of quality and safety. Source evidence grounds these profiles; historical replies and human review assess compatibility and simulation fidelity. Across nine models, automated safety warnings occur in 2.8–6.6% of completed text interactions and can coexist with positive simulated emotion gain. External leaderboard rankings only partly align with support performance. Training with interactive feedback improves Qwen2.5-7B-Instruct's emotional outcomes and response quality while reducing warnings on held-out text cases. The benchmark and code are available at https://anonymous.4open.science/r/EMOWEAVE-6EBF.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Emotional Support Conversation, Social Media Evidence, User Simulation, Interactive Evaluation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Zi7pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EmoWeave, a benchmark for evaluating large language models in Social Media Emotional Support Conversation (SM-ESC). The authors formulate emotional support in social media as an interactive setting where models respond to public posts and adapt to subsequent user feedback. They construct a benchmark of 500 text and image cases from FDU-CommentR Weibo posts and propose a three-perspective evaluation protocol involving a supporter model, a profile-conditioned simulated poster, and an independent bystander evaluator. The benchmark evaluates models from multiple dimensions, including emotional trajectory, response quality, and safety. Experiments on nine LLMs show that general model capabilities do not fully align with emotional support performance, and interactive feedback training based on EmoWeave improves Qwen2.5-7B-Instruct on held-out cases.

Critical strengths

1. Meaningful problem formulation and realistic application scenario. The paper identifies an important limitation of existing emotional support benchmarks: many existing settings focus on explicit help-seeking conversations, while social media users often express implicit emotional needs through public posts. The proposed SM-ESC formulation better reflects real-world scenarios where a supporter must infer user needs from limited public evidence and adapt responses as interactions evolve.

2. Comprehensive evaluation framework. The proposed three-perspective evaluation protocol provides a richer evaluation paradigm beyond traditional response-generation metrics. By combining simulated poster feedback, independent quality assessment, and safety evaluation, the benchmark attempts to capture multiple aspects of emotional support, including emotional outcomes, interaction quality, and potential risks.

Critical weaknesses

1. Questionable validity of simulated-user evaluation. The benchmark relies heavily on LLM-based poster simulation to measure emotional improvement and interaction outcomes. Although the authors conduct consistency checks and human evaluations, these studies mainly assess plausibility rather than whether simulated users accurately reflect real human reactions. Therefore, it remains unclear whether EmoWeave measures genuine emotional support ability or merely compatibility with an LLM simulator.

2. Limited empirical validation of benchmark reliability and utility. The benchmark contains only 500 cases, and the paper reports model comparisons mainly based on point estimates without sufficient analysis of statistical reliability, such as ranking stability or variance across subsets. Moreover, while the training experiment demonstrates improvements for one supporter model, it remains unclear whether the learned benefits generalize across different models, evaluator configurations, or real-world user interactions. More comprehensive validation is needed to establish EmoWeave as a reliable framework for both evaluation and model improvement.

Questions for authors

Q1. How well do simulated poster reactions correlate with real user responses? Have the authors conducted human studies comparing simulated feedback with actual user reactions?

Q2. Since multiple LLMs are involved as supporter, poster, and evaluator, how sensitive are the benchmark results to the choice of these models? Does changing the evaluator model significantly alter model rankings?

Q3. How do the authors address potential evaluator preference bias introduced by the simulated poster and bystander models? Could the benchmark unintentionally favor models whose responses align better with the generation style or preference distribution of these evaluators?

Q4. The benchmark measures emotional improvement as an important metric. How do the authors ensure that emotional gain does not simply reflect short-term sentiment changes but corresponds to meaningful emotional support?

Reviewer_mR1YOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces EmoWeave, a benchmark of 500 cases for emotional support in social-media conversations. A supporter generates replies, a simulated user reacts, and another LLM evaluates response quality and safety. The paper evaluates nine models and trains a smaller model using feedback from this environment.

Critical strengths

1.The task is practically meaningful. Public social-media support involves brief posts, implicit needs, and limited personal context. The task also includes positive sharing and requires replies to adapt to later feedback. These features provide a useful setting for studying support beyond an explicit request for help (Sections 1 and 3.1).

2.The evaluation separates emotional outcomes from response quality and safety. This distinction is useful because positive simulated emotion gain does not necessarily imply an appropriate reply. Section 5.3 reports that 22 of 27 cases with automated safety warnings also show positive emotion gain for GPT-5.6 Terra. Reporting these measures separately provides a more informative view of the simulated interactions.

3.The paper also tests learning from interactive feedback. Training and evaluation use different source posts and authors. Table 4 reports improved quality scores and fewer automated safety warnings after training Qwen2.5-7B-Instruct. This provides an initial demonstration of the environment’s use for training within the proposed evaluation setup (Section 7 and Appendix D).

Critical weaknesses

1.The added scientific value is not sufficiently established. The paper mainly introduces a new setting, a dataset, and an LLM-based evaluation pipeline. Section 2 discusses existing interactive support benchmarks, and the training follows RLVER and GRPO. Controlled comparisons are needed to show what the public setting and three-perspective protocol measure beyond existing approaches, rather than mainly reporting model scores.

2.The evaluation procedure is difficult to follow. Key details are spread across Sections 4–6 and Appendices B–C, including simulated-reply repair, emotion initialization, and the purpose of human review. A clearer end-to-end description should connect these decisions to the final metrics and distinguish checks of simulation behavior from validation of the evaluator.

3.Important categories have limited coverage. The 500 cases include only seven medium-risk cases, three high-risk cases, and seven cases focused on emotional expression (Appendix A.3). These small groups limit conclusions about such settings. Category-level results and uncertainty estimates are needed; the total dataset size alone does not establish adequate coverage.

4.Human alignment of the scores and rankings remains unclear. Two reviewers assess 450 case packages, mainly checking profiles and whether simulated behavior and states appear reasonable. This does not directly validate automatic quality scores, safety labels, or model rankings. Different model groups also use different human-reviewed cases (Appendix C.2). Correlations with LiveBench and EQ-Bench 4 cannot replace direct agreement with human preferences.

5.Internal consistency does not fully establish metric validity. Figure 5 relates LLM-generated quality ratings to simulated emotion changes, while Appendix C.3 derives its BLRI-inspired scores from the bystander’s own outputs. These are not independent human measurements. Moreover, Q5 excludes continuity/adaptation even on later turns (Appendix B.4), leaving a central task requirement outside the main quality score.

Questions for authors
  1. What does EmoWeave measure beyond existing interactive emotional-support benchmarks? Could controlled comparisons show the added value of the public setting and three-perspective protocol?
  2. How well do the automatic quality scores, safety labels, and model rankings agree with independent human judgments on the same cases? What is the agreement between the human reviewers?
  3. Given the small numbers of medium- and high-risk cases, how reliable are the safety findings? Could you provide category-level results and uncertainty estimates?
  4. Why is continuity/adaptation excluded from Q5 on later turns, when it can be assessed? Could you report separate results for adaptation to user feedback?
  5. How stable are the rankings across repeated runs and different simulator or judge models? Do the training gains remain when assessed by humans or an alternative evaluator?
  6. Could you provide one end-to-end description showing how simulation checks, repairs, state updates, and scoring produce the final results?
Reviewer_81pfOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Existing emotional-support evaluations do not fully capture public social-media conversations, where users’ needs may be implicit and change across turns. Historical conversations also cannot provide reactions to newly generated replies. To address this gap, the paper introduces EmoWeave, an interactive benchmark with 500 text and image cases from real Weibo posts. It builds private user profiles from source evidence and checks them through historical replies and human review. The evaluation combines supporter replies, simulated user reactions, and separate judgments of response quality and safety. Experiments on nine models show that general benchmark rankings only partly match support performance, and simulated emotional improvement can coexist with safety warnings. Training with the interactive feedback also improves Qwen2.5-7B’s measured support quality and emotional outcomes while reducing safety warnings.

Critical strengths
  • The task has clear practical value. Public posts often express support needs indirectly. Including both distress and positive sharing makes the setting useful and varied.

  • The evaluation considers several aspects of support. Separating emotional outcomes, reply quality, and safety helps reveal problems that a single score may miss.

  • The benchmark includes several quality checks. The authors distinguish source facts from assumptions, check historical replies, and conduct human review. The training study further shows a possible use of the evaluation feedback.

Critical weaknesses
  • The benchmark measures emotional improvement through states produced by an LLM simulator. The human study checks whether the profiles and conversations appear reasonable. However, it does not show whether the emotion scores reflect human views of helpful support, or whether the resulting model rankings agree with human rankings. Historical replies also guide case selection, which limits their value as independent evidence. This leaves the central measurement claim insufficiently supported.

  • All main results use one fixed user simulator and one fixed judge. The paper does not test whether its conclusions hold when either model is changed. GPT-5.6 Luna also serves as both the judge and an evaluated supporter, which requires checks for possible judging preferences. Without such tests, the reported scores and rankings may depend strongly on the chosen evaluation models. The human review does not resolve this concern, as reviewer agreement and direct human–judge agreement are not reported.

  • The paper discusses existing interactive emotional-support benchmarks, including SAGE, EmoHarbor, and EIBench. However, it provides limited direct evidence showing which important model failures the proposed setting reveals that existing evaluations miss. Comparisons with LiveBench and EQ-Bench show some ranking differences, but they do not isolate the value of the social-media setting, private profiles, or three-perspective protocol. Stronger controlled comparisons are needed to establish the contribution.

  • Only seven cases are labeled medium risk and three high risk, which provides limited coverage of difficult safety situations. Each profile mainly uses one selected interaction trajectory, and nearby model ranks are not tested for statistical significance. These limits weaken confidence in the detailed rankings and safety findings.

Questions for authors
  • Could the authors conduct a small study with real participants to test whether simulated emotion gains predict users’ own ratings of helpfulness and emotional change?

  • How stable are the model rankings when the user simulator or judge is replaced? Could the authors also test a judge that is not among the evaluated supporters?

  • What is the agreement between the two human reviewers? How closely do human quality and safety ratings match the automated judgments?

  • Under the same training setup and budget, how does the proposed feedback compare with RLVER-style emotion feedback alone? Which added feedback component contributes most?

  • Would the main findings hold with more medium- and high-risk cases, repeated interaction runs, and confidence intervals for model differences?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.