- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies how to make emotional support replies from large language models (LLMs) feel more human and less like fixed templates. The authors introduce Octo-Pair, a dataset of 2,653 Reddit post-comment pairs with eight labels. They also propose Octo-CARES, which uses attention from eight classifiers to find similar posts and their paired human replies. The LLM then uses these examples to generate emotional support replies. The results show better retrieval scores on several measures and higher win rates on most response measures under LLM-based evaluation.
Critical strengths
- The paper studies an important problem. A reply can sound kind but still fail to make a user feel understood.
- Using real human replies as examples is a useful idea. It gives the model both a similar situation and an example of how a person responds.
Critical weaknesses
- The main figure is hard to follow. Figure 1 has many arrows. It is hard to see how classifier training, attention weights, retrieval, and response generation connect. The authors should redraw the figure and show the training and test steps in separate parts.
- The training labels come from one LLM, while the test labels come from a vote among three LLMs. The paper does not report human checks or agreement with human labels. The retrieval scores may show agreement with LLM labels rather than a true match with users' support needs.
Questions for authors
-
Could human annotators label a small set of posts? How well do their labels match the labels from the three LLMs?
-
What does attention weighting add beyond normal semantic similarity? Could the authors show a few examples where it finds a better support example?
-
Could the authors compare matched comments with random or wrong comments while keeping the prompt length similar? This would help show whether the gains come from finding the right human reply.
-
Could the authors test the method on the original ESCoT dialogues without turning them into long Reddit-style posts? Could human raters also judge the quality of the replies?