- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces CogFlow, which dynamically combines six cognitive units, including observation, attribution, and motivation, into social reasoning flows. It constructs training data through tree-structured generation and preference-based filtering with deepseek R1, then trains models using SFT and RL. Its main contribution is translating social cognitive theory into a structured reasoning framework, with experiments showing improvements in in-domain test sets.
Critical strengths
-
The direction of studying LLMs for social interactions is interesting.
-
The framework is complete, covering dataset construction, SFT, and RL. However, each stage appears to apply existing methods, making this a solid engineering effort with limited novelty.
Critical weaknesses
-
The title, “Think Socially via Cognitive Reasoning,” seems to overstate what the paper demonstrates. I am not convinced that the method improves social intelligence as a general capability of LLMs. The results mainly show improvements through in-domain training, and outperforming untuned models on an in-domain test set after fine-tuning, as shown in Table 2, is not particularly surprising. What I would find more interesting is whether the method also benefits more advanced LLMs across a broader range of social intelligence benchmarks, such as EmoBench, ETHICS, SocialIQA, and ToMBench, and whether the resulting models can outperform larger LLMs on these benchmarks. For example, Table 3 shows that on SocialIQA, the accuracy of 70.4% is even slightly below the original model’s 70.7%.
-
I find the motivating example in Figure 1 unconvincing. The scenario and question seem unusual, and I do not think the question has a clear or broadly agreed-upon answer. I am also not convinced that the “Cognitive Reasoning” shown in the example leads to a better-supported conclusion. The case seems too ambiguous to clearly illustrate the claimed advantage. So I doubt that the research task of this paper is ill-defined.
Questions for authors
Since this year, it seems to post at most two critical weaknesses of the paper. So I will post other weaknesses I observed in this paper to this section:
-
I find the technical novelty limited. The proposed method appears to combine several established components, including SFT, GRPO, and rejection sampling over rollout trajectories. The paper could explain more clearly what is technically new beyond combining these components.
-
The models used for data generation and training are relatively old, such as DeepSeek-R1 and Qwen2.5. I would encourage the authors to experiment with more recent teacher and backbone models to show whether the proposed method remains useful as model capabilities improve.
-
The expert annotation and pairwise evaluation rely on only six experts. Given the subjective and culturally dependent nature of social judgments, I am concerned about how representative their preferences are. A larger and more diverse pool of annotators would make the human evaluation more convincing.
-
The introduction needs improvement. I do not find the current motivation sufficiently convincing. It does not clearly explain what distinguishes social reasoning from general reasoning, whether social reasoning is a well-defined problem, or why the chosen social cognitive theory provides an appropriate foundation for the proposed framework. These points need to be clarified to better motivate the method.