- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper examines the tension between persona fidelity and cross-context generalization in LLM-based simulation of individual behaviour. The authors first annotate over forty datasets with respect to their behavioural requirements and cluster them; they then appeal to the New Big Five personality theory to collapse the resulting preliminary clusters into a five-dimensional capability taxonomy (FONTS), on top of which they construct a standardized training corpus of roughly 10 million instances. Methodologically, the paper proposes Socio-Foundation, a three-stage recipe. In Stage 1, each task is trained separately with DAPO to yield mutually isolated LoRA-based task experts, so that behaviourally divergent specializations are allowed to mature independently in an environment free of cross-task interference. In Stage 2, exploiting the many-to-many affiliation between tasks and capabilities, the task experts are consolidated into five capability experts via off-policy forward-KL distillation, which effectively re-indexes knowledge from the task axis onto the capability axis. In Stage 3, the five frozen capability teachers supervise the student on its own self-sampled rollouts through multi-teacher on-policy distillation with weighted forward-KL objectives, ultimately delivering a single unified simulator. For evaluation, the authors construct IndiEval.
Critical strengths
-
The pipeline design is grounded in clear mechanistic argumentation, and every design choice is supported by a corresponding ablation.
-
After distillation, the capabilities of the 8B model approach those of general-purpose models. Moreover, the experiments provide interpretable behavioral evidence for the diagnosis that general-purpose models are systematically weak on the N dimension; at the same time, the observation that specialized models collapse to zero on OOD tasks also carries cautionary value.
Critical weaknesses
DeepSeek-V4-Flash simultaneously serves as: (1) the reward model for DAPO; (2) the dialogue counterpart in interactive tasks; (3) the judge for IndiEval; (4) one of the baselines compared in Table 1. The 8B model is optimized with RL against the preferences of DeepSeek-V4-Flash and is then scored by DeepSeek-V4-Flash, since a large number of metrics (CoSER, MirrorBench, HUMANUAL, SOTOPIA, among others) are LLM-judged. In essence, these scores reflect the subjective judgments of DeepSeek-V4-Flash, and this arrangement is therefore unfair to the other general-purpose large models used for comparison.
Questions for authors
- DeepSeek-V4-Flash simultaneously serves as the reward model, the interactive opponent, the judge, and one of the baselines.
- There is no second judge, human validation, or control for judge bias. What are the specific values of \lambda in Eq. 6 (equal weighting, weighting proportional to capability scale, or tuned values)?
- The metrics for N include termination F1, inverse intent overlap, and diversity, which reward stopping when the conversation should stop, whereas the metrics for O (goal, relation, knowledge, benefits, intent coverage, and task score) reward stating everything in full, pushing the interaction forward, and achieving the goal. Why, then, does Table 3 show that the score on the N dimension instead decreases when the O capability expert is removed?
3 comments
I thank the authors for their detailed responses to my questions. The results in Table 12 of the rebuttal show that five of N's seven metrics are coupled with O, and that the capability gap between N and O is concentrated primarily in Termination F1. Once the metrics are integrated with equal weights, the weight assigned to the differing capability shrinks further; the disagreement between the two experts therefore appears more likely to be dissolved by "averaging over the next-token distributions" than to be adjudicated. Therefore, i would like to keep my initial assessment.
Thank you for your suggestions and comments. We also find this phenomenon interesting. In fact, addressing potential conflicts among training samples associated with different capabilities is one of the key motivations behind our efforts to build a unified foundation model. By organizing training samples according to their capability requirements, we have mitigated these conflicts to some extent. Nevertheless, at the capability level, there still exists some degree of mutual influence between capabilities, as you have observed. We believe that finding better ways to integrate and coordinate different capabilities is an important direction for future research. We plan to explore more fine-grained, capability-aware integration mechanisms, including architectural approaches such as Mixture-of-Experts (MoE).
Response for Reviewer dFoN
We sincerely thank the reviewer for their careful reading and valuable feedback. We appreciate the recognition of our pipeline design and experimental analysis.
Socio-Foundation leads evaluated 8B models under all three judges (W1 & Q1 & Q2)
We re-evaluated the models using two additional judges, Kimi-K2.6 and GLM-5.2. We reused the saved trajectories and auxiliary-model outputs, recomputed LLM-judged metrics with the same benchmark prompts and scoring rules, and retained the original values for automatically computed metrics. The average IndiEval scores and ranking correlations are reported in General Response 1. Socio-Foundation remains the highest-ranked 8B model under all three judges. The complete per-benchmark and per-metric results are in Appendix D.6, Tables 13 (Kimi) and 14 (GLM).
Active teachers receive equal weights that sum to one per sample (Q2)
Thank you for asking about the teacher weights in Section 4.4, Eq. (6). Teachers activated by each sample's FONTS labels receive equal weights summing to one, while inactive teachers receive zero weight; no relative teacher weights are tuned. All active teachers supervise the same student-generated prefix. General Response 1 provides the weight definition and details of loss aggregation across positions and continuations.
Removing O improves termination but reduces overall behavioral naturalness (Q3)
We thank the reviewer for highlighting this interesting observation. We would like to clarify that N measures broader behavioral naturalness, rather than only whether the model ends a conversation at the appropriate time. Of the three N metrics mentioned in the review, only Termination F1 directly evaluates whether a conversation ends appropriately. Inverse intent overlap measures how little an utterance mechanically repeats the supplied intent at the lexical level; higher values indicate less surface overlap and can, to some extent, reflect autonomy and naturalness of expression. First-turn diversity measures unigram Jaccard distance between first-turn responses across samples, assessing whether initial responses are varied rather than templated or homogeneous. Neither metric directly rewards early termination.
N also includes MirrorBench GTEval (style and authenticity), CoSER anthropomorphism, AI-detector human likelihood, and role adherence. It therefore covers expression, interaction, initiative, role behavior, and termination decisions, rather than simply rewarding shorter responses.
We decomposed the N results for the full five-expert configuration and w/o O:
Rebuttal Table 12. N metrics for the full configuration and w/o O.
Difference denotes w/o O minus Full.
The local tension identified by the reviewer does exist, but declines in other metrics outweigh the termination gain and reduce the overall N score. Removing O improves Termination F1 by 2.05 points, consistent with reduced pressure to prolong an interaction. However, role adherence is unchanged and every other N metric decreases. These losses outweigh the termination improvement in the aggregate, which falls from 54.41 to 52.46.
A plausible explanation is that O contributes purposeful, context-sensitive interaction, not merely pressure to state everything. Removing it may reduce excessive conversational persistence while making responses more generic, homogeneous, or passive. This interpretation is consistent with the drops in first-turn diversity and human likelihood. CoSER's anthropomorphism rubric explicitly treats "lacks initiative and goals" as unnatural character behavior; initiative and goal-directedness are therefore themselves part of human-like behavior. MirrorBench GTEval likewise assesses similarity to human reference dialogue in style, tone, and authenticity, which can benefit from context-appropriate goal pursuit.
Moreover, not all O metrics reward exhaustive expression. Intent coverage is directly related to expressing needs, but SOTOPIA's goal, relationship, knowledge, and benefits, and AgentSense goal completion, evaluate interaction outcomes rather than response length. Natural human behavior includes both pursuing goals and stopping appropriately. Thus, Rebuttal Table 12 reflects an improvement in one termination-related component after removing O, alongside larger losses in other aspects of naturalness; it does not imply that O and N are inherently opposed.