acceptodds
Under review as a conference paper at ICLR 2027

SD: Skill-Conditioned Self-Distillation for LLM Social Reasoning

21 views

Abstract

While large language models (LLMs) have achieved impressive performance across many domains, their social intelligence, including the ability to reason about others’ beliefs, intentions, and emotions, remains comparatively underexplored. Recent work has applied reinforcement learning (RL) to improve LLMs’ social reasoning capabilities, but sparse verifier outcomes provide little guidance for the intermediate reasoning process. On-policy self-distillation (OPSD) alleviates this by using teacher-side privileged information to provide dense token-level supervision. However, conventional sample-specific privileged signals may fail to capture diverse, reusable reasoning strategies required for social reasoning. To address these limitations, we introduce Skill-Conditioned Self-Distillation for Social Reasoning (SD), a framework that uses structured social reasoning skills as teacher-only privileged information. We extract compact natural-language planning skills from the model’s own correct reasoning trajectories, where each skill describes a successful workflow for solving a class of social reasoning problems. During training, relevant skills are dynamically retrieved and provided only to the teacher, while the student always responds to the plain task prompt and learns to internalize the planning guidance through token-level distillation. We further combine this dense distillation signal with outcome rewards to jointly supervise both the reasoning process and the final answer. Experimental results on multiple social reasoning benchmarks demonstrate that SD substantially outperforms the standard RL baseline, improving both vanilla GRPO and vanilla OPSD.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Social Intelligence, Theory-of-Mind, Self-Distillation, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bVjLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims for enhancing the social intelligence of LLMs. Their method is based on self-conditioned on-policy distillation, which is an existing technique. The key component in the paper is two parts: (1) constructing a skill bank for guiding social reasoning; (2) combining out-come based reward with on-policy distillation reward into one joint objective. Experiments are conduct on just two Qwen models and several Theory-of-mind benchnmarks and EmoBench.

Critical strengths
  • This paper discusses on-policy self-distillation for social intelligence. It is an interesting application of an existing technique to AI's social reasoning.

  • The paper is well written and easy to follow

Critical weaknesses
  • The experimental evidence is limited and does not sufficiently establish the method’s general effectiveness. Although ToMi, SimpleToM, and Hi-ToM are claimed in the paper as out-of-domain benchmarks, they all focus on Theory-of-Mind abilities closely related to those covered by the training data. I am therefore not convinced that improvements on these benchmarks demonstrate broader generalization in social intelligence. There is also no consistent advantage of S³D+GRPO over S³D, leaving the benefit of joint optimization unclear. I would like to see evaluations on a broader range of social intelligence-related tasks, such as social commonsense reasoning on SocialIQA and dedicated benchmarks for moral judgment and value understanding, with a clearer explanation of how they differ from the training distribution. EmoBench in Table 1 provides a somewhat broader evaluation, but the results there do not show positive signals.

  • The technical novelty appears limited. The paper itself cites Skill-SD and Skill-Conditioned Gated Self-Distillation, both of which already use skills as teacher-side privileged information for self-distillation. The main contribution here seems to be adapting this approach to social reasoning and designing the skill organization and retrieval pipeline. Without direct comparisons with these closely related methods, it is difficult to assess the additional technical value of the proposed approach.

Questions for authors
  • Only two qwen models are evaluated. Can the method work on other model families? Can larger size LLMs benefit as well from this method? How is the performance of model in this paper compared with advanced models like GPT-5.6 sol in social intelligence benchmarks?

  • There are many papers so far discussing enhancing social intelligence or theory of mind for LLMs now. I wonder why the authors did not compare with them.

  • The paper does not sufficiently clarify the relationship between social intelligence, social reasoning, and Theory of Mind. Social intelligence is a broader construct, with social reasoning representing one of its component abilities, while ToM focuses more narrowly on understanding and inferring mental states. The authors should clarify these conceptual boundaries and explain how improvements on predominantly ToM-based tasks support their broader claims about social intelligence.

Reviewer_y3F1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

S³D extracts social-reasoning workflows from model trajectories with verified correct answers and retrieves them as teacher-only guidance for on-policy self-distillation, optionally combined with GRPO. Experiments with Qwen3-4B and Qwen3-8B on five benchmarks report improvements, particularly on ToMi and SimpleToM, without inference-time skill retrieval.

Critical strengths
  1. Teacher-only skill conditioning offers a clear way to provide procedural guidance during training while preserving the ordinary inference interface.

  2. Improvements extend beyond comparisons with degraded RL baselines. S³D also raises average accuracy over the starting models at both scales.

  3. Answer- and reference-conditioned OPSD comparisons, alongside analyses of rewards and response lengths, provide useful evidence about alternative supervision and training behavior.

Critical weaknesses
  1. There are no direct ablations of retrieved-skill relevance, query-target routing, or consolidation. Comparisons with answer/reference conditioning alone cannot establish whether specialized reusable skills outperform simpler generic planning guidance.

  2. Section 3.3 averages over all generated tokens, whereas Appendix B defines GRPO using per-response averages. With unequal response lengths, setting β = 0 does not generally recover the stated GRPO objective. The implementation must be clarified to exclude a normalization confound in the hybrid-versus-GRPO comparison.

  3. Table 1 provides no uncertainty estimates. Adding GRPO improves the 8B average by only 0.31 percentage points but reduces the 4B average by 0.52 points. Figure 3(c) also mixes averaging conventions. Its GRPO average uses four benchmarks, while the β = 0.5 average matches the five-benchmark result in Table 1.

Questions for authors
  1. How does S³D compare with length-matched generic plans, randomly selected same-target skills, and retrieval without the structured routing? Do gains persist when skills derived from the same source story are excluded?

  2. Which loss normalization is implemented for GRPO and S³D+GRPO? Do the conclusions persist against validation-tuned GRPO and under common inference-token budgets?

  3. Please specify the teacher-update rule, skill-extraction and embedding models, and the source of OPSD-reference solutions. Does the 4B experiment also use the 8B-derived skill bank?

  4. How were β and checkpoints selected? Please provide repeated-seed results with paired confidence intervals and reconcile the averages in Figure 3(c).

Reviewer_7gk5OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces S3D\text{S}^3\text{D}, a skill-conditioned self-distillation framework for improving LLM social reasoning. The method constructs a bank of reusable natural-language social reasoning skills from verified correct model trajectories. For each training sample, it retrieves a relevant skill and supplies it only to a teacher branch, which provides token-level supervision to a student that receives the original task prompt alone. S3D\text{S}^3\text{D} combine the dense supervision of on-policy self-distillation with outcome-based reward in GRPO to optimize the model. Experiments on Theory-of-Mind demonstrate the effectiveness of the proposed method.

Critical strengths
  • The proposed S3D\text{S}^3\text{D} method is intuitive, which applies reusable reasoning workflows as teacher-only privileged information to perform on-policy self-distillation for social reasoning. This idea is well-motivated.
  • The method is clearly formulated, with explicit procedures for constructing the skill bank, retrieving skills for each instance, and integrating skill-conditioned token-level distillation with outcome-level GRPO optimization.
Critical weaknesses
  • The novelty is limited, as the proposed framework largely extends existing skill-conditioned self-distillation approaches to social reasoning, such as Skill-SD [1] and SGSD [2]. The main difference appears to be the application to social reasoning scenario.
    [1] Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agents.
    [2] Skill-Conditioned Gated Self-Distillation for LLM Reasoning.
  • The performance gains are relatively limited. In Table 1, the improvement of the proposed method over the base model is sometimes marginal, and S3D/S3D+GRPO is not consistently the best-performing method across benchmarks. For example, S3D\text{S}^3\text{D} and S3D\text{S}^3\text{D}+GRPO are both worse than OPSD-reference on ToMBench.
  • The experimental analysis is limited. Beyond the main results and a sensitivity study of the GRPO-OPSD coefficient, the paper provides little analysis of skill quality, retrieval accuracy, skill usage patterns, or failure cases. Such analyses are important for verifying whether the gains truly arise from reusable social-reasoning skills rather than the additional factors.
Questions for authors
  • Could the authors provide further analyses of the proposed method, such as the skill quality, retrieval process, and representative failure cases, to verify the gains of the experiments?
  • Could the authors more clearly clarify the core novelty and findings of the proposed method over existing skill-conditioned self-distillation methods?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.