acceptodds
Under review as a conference paper at ICLR 2027

Learning a Foundation Model of Human Social Reasoning from Heterogeneous Experts

Abstract

Large language models (LLMs) have demonstrated strong capabilities in symbolic reasoning, tool use, and complex task solving, yet they still lack a fundamental understanding of human social reasoning: how people interpret the same situation, weigh relevant information, and arrive at decisions. This limitation largely stems from the scarcity of explicit decision knowledge in pretraining corpora about how individual information leads to behavioral choices. To recover such knowledge from structured behavioral data, we use heterogeneous experts with complementary inductive biases to uncover relationships between individual information and behavioral choices. Building on these relationships, we introduce SocialReasoner to supplement LLMs with the decision knowledge required for human social reasoning through natural language reasoning supervision. Drawing on four major survey programs, we construct nine large datasets for human social reasoning, comprising 281,995 respondents and over 8.7 million observed responses across 365 questions, 13 social domains, and diverse countries and regions. Across multiple model families and scales, SocialReasoner consistently outperforms all evaluated state-of-the-art baselines, with average relative accuracy gains exceeding 40% and substantially better alignment with observed population behavioral distributions. It further generalizes to an unseen survey and extends to decision tasks in finance and healthcare. Code and models will be made publicly available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Human Social Reasoning, Human Decision Making

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_iexROpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SocialReasoner, a framework for improving LLM-based modeling of human decisions from structured survey data. It uses heterogeneous machine-learning experts to identify predictive relationships between respondent attributes, perception information, and behavioral choices, then converts these relationships into natural-language reasoning supervision for LLM fine-tuning. Across nine reconstructed survey datasets covering 281,995 respondents and over 8.7 million responses, the method improves individual response prediction, population-level distribution alignment, and transfer to an unseen survey and additional finance and healthcare tasks.

Critical strengths
  1. The paper is easy to understand.
  2. It proposes a new method and constructs a dataset.
  3. The performance is better than baselines
Critical weaknesses
  1. The definition of “human social reasoning” are overly broad. The paper frames social reasoning as understanding how humans interpret situations, weigh information, and make decisions, but operationalizes it mainly as predicting a held-out survey response from demographics and answers to other survey questions. This task is closer to individual behavioral or preference prediction and does not cover other established aspects of social reasoning such as theory of mind, mental-state inference, pragmatic understanding, or interpersonal reasoning.
  2. The claim that current LLMs fundamentally lack social reasoning is not sufficiently supported. Social reasoning is multidimensional, and established tasks such as Theory of Mind evaluate abilities such as inferring beliefs, intentions, desires, and emotions; recent work such as DialToM[1] shows that modern LLMs can already perform strongly on mental-state inference. Therefore, weak performance on the proposed survey-response benchmark demonstrates a limitation in individualized behavioral prediction, rather than a general deficiency in LLM social reasoning.
  3. The newly constructed benchmark lacks sufficient human validation of its central construct. Although the labels are authentic responses from large-scale human surveys, the paper does not report human evaluation showing that the proposed profile-to-response prediction task is a valid measure of social reasoning. In particular, there is no human baseline, estimate of task predictability, or human verification of whether the generated reasoning traces actually reflect plausible human reasoning processes.
  4. The motivation that current LLMs are fundamentally limited in social reasoning is not fully established. Several key observations in Figure 2 are demonstrated mainly with relatively small or older models, while frontier-model performance is evaluated only on the paper’s own survey-response task. Moreover, even question-specific supervised ML models trained directly on the structured data achieve only around 55–64% accuracy, making it difficult to interpret moderate frontier-model performance without a human upper bound.
  5. The large gains over pretrained models are difficult to interpret because the main evaluation is largely in-domain adaptation. Training and test respondents are disjoint, but the target questions and survey domains are shared, so substantial improvement after fine-tuning on these survey data is expected.
  6. It is unclear how much of the improvement is actually attributable to the proposed social-reasoning supervision. The appendix shows that an answer-only variant using the same expert-selected records and multiplicities already captures a large fraction of the gain, suggesting that expert-guided data selection and reweighting are themselves strong supervision signals. On the unseen GSS2022 survey, LLM-selected evidence already recovers most of the transfer benefit, with SocialReasoner providing only a small additional improvement.

[1] Yadav, Neemesh, et al. "DialToM: A Theory of Mind Benchmark for Forecasting State-Driven Dialogue Trajectories." arXiv preprint arXiv:2604.20443 (2026).

Questions for authors
  1. How do the authors justify using held-out survey-response prediction as a general operationalization of “human social reasoning,” rather than a narrower task of individual behavioral or preference prediction?

  2. What evidence supports the broader claim that current LLMs fundamentally lack social reasoning, given that other established dimensions such as Theory of Mind and mental-state inference have shown substantially stronger performance in modern LLMs?

  3. How was the newly constructed benchmark validated as a measure of social reasoning? In particular, is there any human baseline, estimate of task predictability, or human evaluation verifying that the generated reasoning traces reflect plausible human reasoning processes?

  4. Do the key observations motivating SocialReasoner still hold for frontier models? Figure 2 establishes several of these observations primarily with relatively small or older models, and without a human or oracle upper bound it is difficult to determine whether the moderate accuracy of frontier models reflects a genuine capability gap or intrinsic uncertainty in individual response prediction.

  5. How much of the reported improvement reflects learning transferable social reasoning rather than ordinary in-domain adaptation? The main evaluation holds out respondents but largely retains the same target questions and survey domains, whereas the gains are substantially smaller in the unseen-survey setting.

  6. How much of SocialReasoner’s gain is actually attributable to the proposed reasoning supervision rather than expert-guided data selection and reweighting? The answer-only ablation using the same selected records and multiplicities already captures a large fraction of the improvement.

11 comments
AuthorsOpenReview ↗

Response to Weakness 1 (1/6) and Question 1 (1/6): Scope and Operationalization of Human Social Reasoning

We thank the reviewer for this thoughtful comment. We appreciate the opportunity to clarify a possible misunderstanding regarding the scope of social reasoning in our work.

First, social reasoning in SocialReasoner refers to our methodological approach, rather than a claim about reproducing human cognitive processes or a comprehensive measure of social cognition. Specifically, we investigate how LLMs can reason about human behavioral choices by integrating individual characteristics, perceptions, and decision-relevant knowledge. The term social reasoning emphasizes the social dimension of reasoning, distinguishing it from conventional reasoning tasks centered on mathematics, formal logic, or coding. This terminology is also adopted in prior research investigating LLM reasoning in social contexts, including Scaling LLMs' Social Reasoning [1] and SocialMaze [2]. While these studies explore different social reasoning tasks, they similarly distinguish socially grounded reasoning from general-purpose reasoning. We have already introduced this distinction in our manuscript and will further sharpen its formulation to avoid potential ambiguity.

Second, our evaluation follows an established paradigm in human behavior simulation. Prior studies, including Socrates [3] and OmniSapiens [4], investigate LLM-based human behavior modeling through individual-level behavioral prediction. Following this research direction, we adopt survey-response prediction as an empirically grounded and scalable evaluation task. We recognize that social cognition encompasses broader capabilities, including theory of mind, mental-state inference, and interpersonal understanding. Our work focuses specifically on improving LLM reasoning about human social decisions, rather than claiming to address every dimension of social cognition.

In summary, our contribution lies in developing a knowledge-guided social reasoning approach for human behavior prediction, rather than reconstructing the cognitive processes underlying human decisions.

Manuscript Revisions. We will further clarify the methodological meaning and scope of social reasoning in the Introduction and refine the corresponding claims to avoid similar misunderstandings. We appreciate the reviewer's constructive feedback.

[1] Scaling LLMs' Social Reasoning: Sprinkle Cognitive “Aha Moment” into Fundamental Long-thought Logical Capabilities. Findings of ACL 2025.

[2] SocialMaze: A Benchmark for Evaluating and Enhancing Social Reasoning in Large Language Models in Complex Social Environments. Findings of EMNLP 2026.

[3] Finetuning LLMs for Human Behavior Prediction in Social Science Experiments. EMNLP 2025.

[4] OmniSapiens: A Foundation Model for Social Behavior Processing via Heterogeneity-Aware Relative Policy Optimization. ICML 2026.

AuthorsOpenReview ↗

Response to Weakness 2 (2/6) and Question 2 (2/6): Limitations of Current LLMs in Social Reasoning

We thank the reviewer for this insightful comment. We would like to clarify that social reasoning in SocialReasoner refers to our methodological approach, rather than a claim to comprehensively model human social cognition. Our work focuses on enabling LLMs to reason about human social decisions using individual characteristics, perceptions, and decision-relevant knowledge.

First, our motivation is supported by independent evaluations on well-established and widely adopted survey benchmarks, including ISSP, which has also been used in prior human behavior simulation research such as SocioBench [1]. As illustrated in Figure 1, even frontier LLMs with strong general reasoning capabilities still face substantial challenges in predicting individual human responses.

Second, we would like to highlight an important distinction in the Theory of Mind (ToM) literature. Importantly, DialToM [2], cited by the reviewer, distinguishes between Literal ToM (inferring mental states) and Functional ToM (using mental-state information to predict subsequent social interactions). The study finds that LLMs perform strongly on Literal ToM but remain considerably weaker on Functional ToM, demonstrating that successful mental-state inference does not necessarily translate into accurate social behavior prediction. Similarly, ToMBench [3] and Towards Dynamic Theory of Mind [4] identify persistent limitations in mental-state reasoning and tracking. These findings align with our observation that accurately reasoning about human social decisions remains challenging for current LLMs.

Nevertheless, we appreciate that our original wording may have been overly broad. We will refine our claim to emphasize the limitations of current LLMs in reasoning about individualized human social decisions, rather than suggesting a universal deficiency across all aspects of social cognition.

[1] SocioBench: Modeling Human Behavior in Sociological Surveys with Large Language Models. EMNLP 2025.

[2] DialToM: A Theory of Mind Benchmark for Forecasting State-Driven Dialogue Trajectories. EMNLP 2026.

[3] ToMBench: Benchmarking Theory of Mind in Large Language Models. ACL 2024.

[4] Towards Dynamic Theory of Mind: Evaluating LLM Adaptation to Temporal Evolution of Human States. ACL 2025.

AuthorsOpenReview ↗

Response to Weakness 3 (3/6) and Question 3 (3/6): Human Validation of Behavioral Prediction and Reasoning Traces

We thank the reviewer for this important suggestion. To directly address the concerns regarding human baselines, task predictability, and reasoning trace plausibility, we conducted two complementary human evaluations. Both evaluations involved three expert annotators with extensive knowledge and backgrounds in the social sciences, who performed their assessments independently.

(1) Human Baseline and Task Predictability.

We randomly sampled 100 respondent–question instances from ATP, EVS, and ISSP. Three social science experts independently predicted each respondent's answer using the respondent's profile, survey question, and answer options, without access to the ground-truth response. We evaluated frontier LLMs and SocialReasoner on the same instances.

Table 1. Human Expert and LLM Prediction Accuracy on 100 Survey Instances

PredictorAccuracy (%)
Human Expert 186
Human Expert 281
Human Expert 378
Human Expert Average81.67
GPT-6 Astra64
GPT-5.6 Sol60
DeepSeek-V4-Pro61
SocialReasoner70

The three independent human experts achieve an average accuracy of 81.67%, demonstrating that the task contains meaningful predictive information accessible to informed human judgment. Meanwhile, SocialReasoner achieves 70% accuracy, outperforming the strongest evaluated frontier LLM by 6 percentage points. These results establish a human reference for task predictability while highlighting the remaining difficulty of individual behavioral prediction.

(2) Human Validation of Reasoning Trace Plausibility.

We further randomly sampled 100 constructed reasoning traces and invited three social science experts to independently evaluate each trace, without consulting one another. Each expert provided a binary judgment assessing whether the reasoning was relevant to the survey question, consistent with the respondent's information, and plausible in explaining the behavioral choice.

Table 2. Independent Human Expert Evaluation of Reasoning Traces

MetricResult
Expert 1 / 2 / 3 Approval99% / 99% / 99%
Overall Positive Judgments99% (297/300)
Majority Approval (≥2/3)100%
Unanimous Approval (3/3)97%
Exact Inter-Annotator Agreement97%

Notably, 97% of the reasoning traces received unanimous approval from all three independent social science experts, providing strong evidence for the relevance, consistency, and plausibility of our constructed reasoning supervision.

Together, these evaluations directly address the reviewer's concerns: the human expert baseline establishes the practical predictability of individual survey responses, while independent expert assessments validate the plausibility of the generated reasoning traces. Importantly, our evaluation concerns LLM reasoning about human social decisions, rather than claiming to reconstruct respondents' actual internal cognitive processes.

Manuscript Revisions. We have added both human evaluations, including the expert annotation protocol, evaluation criteria, and inter-annotator agreement results, to the Appendix. We appreciate the reviewer's constructive suggestion.

Reviewer_ZvvkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes SocialReasoner, a framework for improving LLM-based human decision prediction by learning from real survey responses. The authors train heterogeneous ML experts (EBM, LightGBM, MLP) to identify predictive features associated with human choices, filter the extracted evidence using an LLM, and convert it into reasoning traces for supervised fine-tuning (SFT). The method is evaluated across eight survey datasets using three LLM backbones (Qwen3-4B, Qwen2.5-7B, Llama-3.1-8B), achieving average accuracy improvements of approximately 14.9-18.2 percentage points over the corresponding base models and outperforming several frontier LLM baselines. The authors also evaluate population-level response distributions, cross-survey transfer, and applications in finance and healthcare. Ablation studies examine the contributions of expert-derived reasoning, heterogeneous experts, feature selection, and semantic filtering.

Critical strengths
  1. The paper proposes an interesting approach to learning human social reasoning by using heterogeneous ML experts (EBM, LightGBM, MLP) to extract predictive evidence from real survey responses and convert it into reasoning supervision for LLMs. I find this innovative because it grounds the reasoning traces in observed human responses and feature attributions rather than relying entirely on LLM-generated explanations.

  2. The experimental evaluation is extensive, covering multiple survey datasets and LLM backbones (Qwen3-4B, Qwen2.5-7B, and Llama-3.1-8B). SocialReasoner achieves improvements of approximately 14.9 - 8.2 percentage points over the corresponding base models and outperforms several frontier LLM baselines.

  3. The paper includes useful ablation studies examining the contributions of expert-derived reasoning, heterogeneous experts, feature selection, and semantic filtering, which help me understand the benefits of reasoning supervision compared with training-example selection.

Critical weaknesses
  1. Some improvements are very modest. Table 13 shows that answer-only fine-tuning on the same expert-selected records achieves 60.82% accuracy, compared with 63.22% for SocialReasoner (+2.40 percentage points). Table 14 shows that combining heterogeneous experts achieves 63.22%, compared with 62.79% for the strongest repeated single-expert baseline (+0.43 points). Given the additional complexity and cost of training multiple ML experts, extracting feature attributions, and filtering evidence with an LLM, could the authors explain why these components lead to such modest improvements? (A component-wise analysis might help identify what exactly is contributing to the improvements.)

  2. The paper evaluates overall population-level response distributions but does not sufficiently examine whether the model preserves different opinions across demographic subgroups. Prior work such as OpinionQA [1] shows that LLMs can align with dominant opinions while poorly representing the perspectives of specific demographic groups. Since SocialReasoner learns from expert-selected survey responses, it is unclear whether the training process preserves less-represented perspectives or favors commonly observed response patterns. I think the authors should evaluate SocialReasoner on [1] and report demographic-level opinion alignment (e.g., representativeness and steerability across age, gender, education, and other groups) to assess whether the model captures diverse human perspectives rather than only matching overall population distributions.

  3. The authors use a fixed decoding temperature (0.6). A temperature sensitivity analysis is necessary to establish whether the reported accuracy remain consistent across different sampling settings.

  4. It would be useful to evaluate how different inference-time prompting strategies (zero-shot CoT, few-shot CoT, ToT) affect performance. Since the model is trained using expert-derived reasoning traces, maybe using guided reasoning strategies through these cues could lead to further improvements in prediction accuracy.

  5. The reasoning traces are constructed from ML feature attributions and filtered using GLM-5.3-Flash, but their quality is not directly validated through human evaluation. It would be useful to have human annotators assess a representative subset of generated traces for relevance.

[1] Santurkar et al. (2023). Whose Opinions Do Language Models Reflect? ICML 2023

Questions for authors
  1. Could the authors clarify which components contribute most to SocialReasoner's performance gains and whether the improvements justify the additional computational cost?

  2. How does the method ensure that less-represented demographic perspectives are preserved during training, and have the authors considered evaluating demographic-level opinion alignment using OpinionQA?

  3. How sensitive are the reported results to different decoding temperatures?

  4. Have the authors explored whether different inference-time reasoning strategies (zero-shot CoT, few-shot CoT, ToT) can further improve performance?

  5. How do the authors assess the quality and reliability of the constructed reasoning traces, and have they considered human evaluation with inter-annotator agreement?

9 comments
AuthorsOpenReview ↗

Response to Weakness 1 (1/5) and Question 1 (1/5): Component Contributions and Computational Cost (Part 1/3)

We thank the reviewer for these important questions. We appreciate the reviewer's observation regarding the improvements reported in Tables 13 and 14. We believe these gains should be interpreted in the context of the scale and inherent difficulty of human behavioral prediction. ATP152 alone contains 244,976 respondent–question instances, where even small percentage-point improvements correspond to meaningful differences in the number of correctly predicted human responses. Moreover, human behavioral choices are inherently heterogeneous and uncertain, making individual-level prediction particularly challenging. Even a frontier model such as GPT-6 Astra achieves 54.18% average accuracy across our eight survey datasets, illustrating the difficulty of this task.

To provide a more comprehensive assessment, we conduct additional experiments examining the effectiveness and complementarity of heterogeneous experts, the contributions of individual supervision components, and their computational costs.

(1) Effectiveness and Complementarity of Heterogeneous Experts.

We extend the expert-combination experiments to ISSP2009 and ISSP2015, comparing individual experts, repeated single-expert configurations, and heterogeneous combinations.

Table 1. Effectiveness of Heterogeneous Experts Across Social Surveys (Accuracy, % ↑)

Expert ConfigurationISSP2009ISSP2015
Base Model42.9533.38
EBM51.3451.23
LightGBM50.2351.24
MLP51.1950.93
EBM × 351.6251.93
LightGBM × 351.3451.26
MLP × 351.9652.19
EBM + LightGBM + MLP (SocialReasoner)58.1958.18

As shown in Table 1, heterogeneous experts outperform the strongest repeated single-expert configurations by 6.23 percentage points on ISSP2009 and 5.99 points on ISSP2015. These substantial improvements demonstrate that combining different types of experts provides more effective decision knowledge than simply increasing training exposure to a single expert.

Why does heterogeneity help? Relationships between individual characteristics, perceptions, and behavioral choices take different, often nonlinear functional forms, which can be better captured by models with different inductive biases. For example, when predicting an individual's stance on immigration policy, one expert emphasizes the respondent's older age and Republican affiliation as evidence supporting a conservative position, while another focuses on personal experiences with immigrants and attitudes toward cultural diversity.

Accordingly, we adopt three representative modeling paradigms: additive modeling captures nonlinear main effects, tree-based modeling captures nonlinear thresholds and conditional feature interactions, and neural modeling learns complex nonlinear relationships.

Our central assumption is that different modeling paradigms extract complementary decision knowledge from the same behavioral data. We validate this assumption through analyses of evidence diversity and complementary prediction coverage.

(i) Diversity of Extracted Decision Knowledge. We examine whether different modeling paradigms identify distinct predictive evidence. We consider six experts spanning three paradigms: additive models, including Explainable Boosting Machine (EBM) and Logistic Regression (LR); tree-based models, including Light Gradient Boosting Machine (LightGBM) and Random Forest (RF); and neural models, including Multilayer Perceptron (MLP) and Feature Tokenizer Transformer (FT-Transformer).

We compute the pairwise Jaccard similarity between the sets of Top-5 important original input features identified by each expert.

Table 2. Pairwise Jaccard Similarity of Top-5 Decision Evidence

ExpertEBMLRLightGBMRFMLPFT-Transformer
EBM1.0000.6810.4210.3850.6260.551
LR0.6811.0000.4270.3450.6300.533
LightGBM0.4210.4271.0000.3590.4210.432
RF0.3850.3450.3591.0000.3570.368
MLP0.6260.6300.4210.3571.0000.512
FT-Transformer0.5510.5330.4320.3680.5121.000

The results demonstrate that different experts identify distinct predictive evidence. EBM and LR exhibit relatively high evidence overlap (0.681), whereas LightGBM shares substantially less evidence with EBM and MLP (both 0.421). These findings support our heterogeneous expert design by showing that different modeling paradigms extract complementary decision knowledge rather than merely reproducing the same evidence.

AuthorsOpenReview ↗

Response to Weakness 1 (1/5) and Question 1 (1/5): Component Contributions and Computational Cost (Part 2/3)

(ii) Complementary Coverage of Human Decisions. We next examine whether combining experts expands the set of respondent–question instances for which predictive evidence can be extracted. Coverage is defined as the proportion of instances correctly predicted by at least one expert.

Table 3. Coverage of Correctly Predicted Respondent–Question Instances

Expert CombinationCoverage (%)
EBM58.53
LightGBM59.15
MLP58.75
LR58.49
RF58.76
EBM + LightGBM65.13
EBM + MLP62.19
LightGBM + MLP65.19
EBM + LightGBM + MLP66.84
EBM + LightGBM + MLP + LR + RF70.04

Combining EBM, LightGBM, and MLP increases coverage from 59.15% for the strongest individual expert to 66.84%, an improvement of approximately 7.70 percentage points. Further incorporating LR and RF expands coverage to 70.04%, demonstrating that additional experts contribute complementary correctly predicted instances.

Together, evidence diversity and broader prediction coverage explain why heterogeneous experts provide richer decision knowledge for social reasoning supervision, supporting the substantial downstream improvements observed across social surveys.

(2) Component Contributions to Cross-Survey Generalization.

To disentangle the contributions of expert-guided instance selection, evidence selection, and semantic refinement, we conduct controlled ablations using Qwen2.5-7B-Instruct. All expert-guided variants share identical training records, response labels, instance multiplicities, and optimization settings, differing only in supervision content. Standard SFT provides an additional reference using the conventional training distribution.

We train exclusively on ATP152 and evaluate on eight unseen surveys without further adaptation.

Table 4. Controlled Component-wise Ablation on Source Prediction and Cross-Survey Generalization (Accuracy, % ↑)

Training DistributionSupervisionATP152 (Source)ATP142 (OOD)EVS2017 (OOD)ISSP2009 (OOD)ISSP2015 (OOD)ISSP2016 (OOD)ISSP2019 (OOD)ISSP2021 (OOD)GSS2022 (OOD)OOD Avg.
NoneBase Model45.7147.3044.0042.9533.3835.7739.0533.2949.8540.70
Standard SFTAnswer Only58.2342.2341.4536.9433.3436.2738.9433.2948.9238.92
Expert-guidedAnswer Only60.8241.2341.2036.1030.1329.6337.4132.9149.1037.21
Expert-guidedRandom Evidence61.4341.5142.1035.1031.2328.2337.5231.2349.2137.02
Expert-guidedLLM-selected Evidence61.5445.8543.2339.2332.3335.8738.4033.3150.4139.83
Expert-guidedRaw Expert Evidence63.1047.1042.4532.9634.3729.3230.1625.5846.7736.09
Expert-guidedSocialReasoner63.2248.4646.3742.9737.6440.4639.9536.0950.7842.84

All fine-tuned models use Qwen2.5-7B-Instruct and are trained exclusively on ATP152. OOD Avg. is the unweighted average across eight unseen surveys.

The results reveal a clear distinction between in-domain adaptation and transferable decision knowledge.

First, expert-guided data selection improves source-survey prediction but does not independently ensure generalization. Compared with Standard SFT, expert-guided Answer Only increases ATP152 accuracy from 58.23% to 60.82%, confirming the effectiveness of expert-selected training instances. However, its average OOD accuracy decreases to 37.21%, below both Standard SFT (38.92%) and the pretrained model (40.70%).

Second, reasoning supervision provides additional benefits beyond data selection. Under identical expert-guided training distributions, SocialReasoner improves source accuracy from 60.82% to 63.22% (+2.40 points) and average OOD accuracy from 37.21% to 42.84% (+5.63 points) relative to Answer Only.

The evidence controls further clarify the contribution of supervision content. Random Evidence achieves only 37.02% average OOD accuracy, while LLM-selected Evidence improves it to 39.83%. SocialReasoner further increases OOD accuracy to 42.84%, outperforming LLM-selected Evidence by 3.01 percentage points across eight unseen surveys, with positive gains on every dataset.

Third, semantic refinement is particularly important for transferable reasoning supervision. Raw Expert Evidence achieves nearly identical source accuracy to SocialReasoner (63.10% vs. 63.22%), but substantially lower OOD accuracy (36.09% vs. 42.84%). This 6.75-point generalization improvement, despite only a 0.12-point source difference, demonstrates the importance of transforming predictive evidence into semantically coherent reasoning supervision.

AuthorsOpenReview ↗

Response to Weakness 1 (1/5) and Question 1 (1/5): Component Contributions and Computational Cost (Part 3/3)

To illustrate this process, consider the following example from our Case Study.

Illustrative Example. Expert Evidence and Semantic Denoising

ComponentContent
Respondent ProfileAge: 55–64; Gender: Male; Education: Completed compulsory elementary education; Country: France; Perceptions: ...
Target QuestionHow important is religion in your life?
Answer Options1. Very important; 2. Quite important; 3. Not important; 4. Not at all important
Expert Evidence(i) Importance of politics: Not at all important
(ii) Confidence in the church: None at all
(iii) Importance of God in life: Not at all important
(iv) Belief in God: No
Semantic DenoisingFiltered out: (i), irrelevant to the target question. Retained: (ii), (iii), (iv).
Constructed Reasoning Trace<think> The respondent reports no belief in God, considers God not at all important in life, and places no confidence in the church. ... This consistent pattern of secular beliefs suggests that religion is not at all important in his life. Therefore, the answer is 4. </think>
Final Answer4. Not at all important

The expert identifies candidate predictive evidence, semantic denoising removes information irrelevant to the target question, and the retained evidence is transformed into coherent reasoning supervision. This process grounds the reasoning trace in observed predictive relationships instead of relying solely on unconstrained LLM-generated explanations.

Together, the controlled ablations and the illustrative example demonstrate that expert-guided data selection and evidence-grounded reasoning supervision make complementary contributions. The former improves source-survey adaptation, while the latter, particularly semantic refinement, supports stronger transfer across unseen surveys.

(3) Computational Cost and Practical Trade-offs.

We acknowledge that heterogeneous expert training and semantic refinement introduce additional computation. We therefore report the measured offline processing time for constructing decision knowledge.

Table 5. Offline Computational Cost of Decision Knowledge Construction

ComponentOperationTime (Hours)
EBMEncoding + Training + OOF Prediction8.138
LightGBMEncoding + Training + OOF Prediction6.236
MLPEncoding + Training + OOF Prediction4.122
ML Experts Subtotal18.496
LLM Semantic DenoisingAPI-based Evidence Refinement85.010
Total Reported Time103.506

The three ML experts require approximately 18.50 cumulative hours for feature encoding, training, and prediction. This represents a relatively modest offline expert-training cost in the context of large-scale behavioral data processing.

LLM-based semantic refinement requires an additional 85.01 hours of offline API processing. The controlled ablations demonstrate its practical contribution, particularly the 6.75% improvement in average OOD accuracy over Raw Expert Evidence.

Importantly, all expert training and semantic refinement are performed offline during supervision construction. Once SocialReasoner is fine-tuned, inference requires only the student LLM, without invoking the original experts.

Overall, these results provide a more complete assessment of the contributions and computational trade-offs of SocialReasoner. Heterogeneous experts identify diverse and complementary predictive knowledge, while semantic refinement transforms this knowledge into more transferable reasoning supervision. The additional offline processing supports improved prediction and generalization without introducing expert-ensemble overhead at inference time.

Manuscript Revisions. We have extended the expert-combination experiments to ISSP2009 and ISSP2015, added analyses of evidence diversity and complementary prediction coverage, expanded the component-wise evaluation across eight unseen surveys, included illustrative examples of semantic refinement, and reported measured offline computational costs. We sincerely appreciate the reviewer's constructive feedback.

Reviewer_7iJ4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SocialReasoner, a framework for training large language models (LLMs) to predict human decisions from structured behavioral data. It leverages heterogeneous experts to identify decision-relevant relationships in survey responses and distills the resulting evidence into natural-language reasoning supervision. Experiments on nine reconstructed datasets demonstrate consistent improvements across multiple LLM backbones and behavioral modeling and adaptation baselines. The paper addresses an important problem at the area of social computing and LLMs, and the results suggest that structured behavioral data and expert-guided supervision can improve human decision prediction.

Critical strengths

1.A meaningful research problem. The paper addresses the challenge of modeling heterogeneous human decisions with LLMs, an important problem at the intersection of social computing and AI. 2.Broad empirical evaluation. Experiments across nine datasets, three LLM backbones, and multiple evaluation settings demonstrate consistent improvements in human response prediction. The ablation studies further investigate the contributions of key components.

Critical weaknesses

1.The definition and construction of heterogeneous decision knowledge need further clarification. This knowledge is central to SocialReasoner, but the paper could provide a more explicit definition of what constitutes decision knowledge and how it is identified, represented, and validated. 2.The effects of expert-guided supervision and the generality of the learned knowledge require further investigation.
3.Some mathematical formulations require more detailed explanations. For example, the notation (\mathcal{P}_i) in Equation (1) appears to be insufficiently defined. Please clarify this notation and explain the relevant variables including but not limited to Eq 1 and Eq 3. 4.Some abbreviations should be defined at their first occurrence. For example, ATP142, ATP152, and EBM should be introduced more explicitly for readers. 5.The rationale for selecting the three expert models should be better explained. Why are EBM, LightGBM, and MLP chosen to represent additive, tree-based, and neural modeling paradigms? Please clarify the motivation for this choice and how their complementary inductive biases contribute to SocialReasoner. 6.The reference list relies heavily on arXiv preprints. The authors are encouraged to cite peer-reviewed and published works.

Questions for authors

1.How robust and generalizable are the reported improvements? 2.Beyond the existing ablations, can the authors further disentangle the contributions of expert-based instance selection, answer-only supervision, expert-selected evidence, and semantic denoising under controlled training conditions? 3.The provenance and interpretation of the results in Figure 1 should be clarified. Are the results directly adopted from prior work or reproduced and evaluated by the authors?

13 comments
AuthorsOpenReview ↗

Response to Weakness 1 (1/6): Definition and Construction of Heterogeneous Decision Knowledge

We thank the reviewer for highlighting the need to clarify this central concept. We define heterogeneous decision knowledge as the diverse predictive relationships between respondent characteristics, perceptions, and behavioral choices extracted by different machine learning experts. SocialReasoner identifies such knowledge from structured human behavioral data and transforms it into explicit natural-language reasoning supervision.

To illustrate this process, we provide a concrete example of decision knowledge extracted by an MLP expert.

Table 1. Illustration of Expert-Grounded Decision Knowledge Construction

ComponentContent
Respondent ProfileAge: 55–64; Gender: Male; Education: Completed compulsory elementary education; Country: France; Perceptions: ...
Target QuestionHow important is religion in your life?
Answer Options1. Very important; 2. Quite important; 3. Not important; 4. Not at all important
Expert Evidence(i) Importance of politics: Not at all important
(ii) Confidence in the church: None at all
(iii) Importance of God in life: Not at all important
(iv) Belief in God: No
Semantic DenoisingFiltered out: (i), irrelevant to the target question. Retained: (ii), (iii), (iv).
Constructed Reasoning Trace<think> The respondent reports no belief in God, considers God not at all important in life, and places no confidence in the church. ... This consistent pattern of secular beliefs suggests that religion is not at all important in his life. Therefore, the answer is 4. </think>
Final Answer4. Not at all important

This example illustrates how SocialReasoner transforms heterogeneous decision knowledge into explicit reasoning supervision. The expert identifies candidate predictive evidence, semantic denoising removes irrelevant information (i) while retaining decision-relevant evidence (ii–iv), and the resulting reasoning trace connects these relationships to the respondent's observed choice.

Through this process, SocialReasoner transfers diverse decision knowledge extracted by different experts into a unified natural-language representation that LLMs can learn from.

We clarify its identification, representation, and validation as follows.

Identification. We employ three types of experts, additive, tree-based, and neural models, to identify diverse predictive relationships from structured behavioral data. For each expert, we use all correctly predicted training instances and extract the feature-level evidence associated with these predictions. In this way, decision knowledge is grounded in observed human responses and the predictive relationships identified by different types of experts.

Representation. We adopt natural language as a unified representation of heterogeneous decision knowledge, making predictive evidence extracted by different experts accessible to LLMs. To obtain semantically coherent and decision-relevant reasoning supervision, we further employ an LLM to filter irrelevant evidence and organize the retained information into natural language reasoning traces for supervised fine-tuning.

Validation. We evaluate the extracted decision knowledge from four complementary perspectives: (i) in-domain prediction across eight survey datasets, (ii) cross-survey generalization to unseen surveys, (iii) population-level response distribution alignment, and (iv) applicability to other decision-making tasks.

The results demonstrate that SocialReasoner achieves state-of-the-art prediction accuracy across the eight surveys, transfers effectively to unseen surveys, better aligns with observed population response distributions without explicit distribution optimization, and improves performance on additional decision tasks in finance and healthcare. Together, these findings support the effectiveness, transferability, and broader applicability of the extracted decision knowledge.

Revisions in the Manuscript. We have clarified the definition and construction of heterogeneous decision knowledge and added multiple illustrative examples in the Case Study section to demonstrate how expert-derived evidence is transformed into natural-language reasoning supervision. We appreciate the reviewer's constructive suggestion.

AuthorsOpenReview ↗

Response to Weakness 2 (2/6) and Questions 1–2 (1/3, 2/3): Contributions of Expert-Guided Supervision and Generalization (Part 1/3)

We thank the reviewer for these insightful questions. To disentangle the contributions of expert-guided data selection, answer-only supervision, expert-selected evidence, and semantic refinement, while further validating the generalizability of learned decision knowledge, we conduct controlled supervision ablations and four complementary generalization evaluations.

(1) Effects of Expert-Guided Supervision on In-Dataset Prediction.

We first examine how different supervision components contribute to predictive performance across eight survey datasets.

Table 1. Effect of Supervision Strategies on In-Dataset Prediction (Accuracy, % ↑)

MethodATP142ATP152EVS2017ISSP2009ISSP2015ISSP2016ISSP2019ISSP2021Avg.
Base47.3045.7144.0042.9533.3835.7739.0533.2940.18
Standard SFT53.9558.2351.5850.1545.7647.6846.3043.6149.66
Answer only54.9260.8256.7152.5347.8149.6248.2045.5552.02
Random evidence53.2361.4355.4351.4647.2348.3246.5248.6151.53
LLM-selected evidence56.2361.5456.6252.4353.2352.9351.5350.7454.41
SocialReasoner58.2763.2264.1258.1958.1855.8054.3954.6358.35

All experiments use Qwen2.5-7B-Instruct. Avg. is the unweighted mean across eight datasets.

As shown in Table 1, expert-guided instance selection improves average accuracy from 49.66% (Standard SFT) to 52.02% (Answer only), confirming that the expert-selected training distribution itself provides valuable supervision. Incorporating decision-relevant reasoning supervision further improves performance, with SocialReasoner achieving 58.35% average accuracy, outperforming Answer only by 6.33 percentage points and LLM-selected evidence by 3.94 points.

These results demonstrate that expert-guided data selection and reasoning supervision make complementary contributions to in-dataset prediction, with the latter providing substantial additional gains beyond instance selection alone.

(2) Controlled Component Attribution and Cross-Survey Generalization.

To isolate the contribution of reasoning supervision from expert-guided instance selection and reweighting, we construct controlled variants using identical training records, response labels, instance multiplicities, backbone models, and optimization settings. The expert-guided variants differ only in their supervision content, while Standard SFT provides a reference using the conventional training distribution.

We train exclusively on ATP152 and evaluate on eight unseen surveys without further adaptation.

Table 2. Controlled Ablation on Source Prediction and Cross-Survey Generalization (Accuracy, % ↑)

Training DistributionSupervisionATP152 (Source)ATP142 (OOD)EVS2017 (OOD)ISSP2009 (OOD)ISSP2015 (OOD)ISSP2016 (OOD)ISSP2019 (OOD)ISSP2021 (OOD)GSS2022 (OOD)OOD Avg.
NoneBase model45.7147.3044.0042.9533.3835.7739.0533.2949.8540.70
Standard SFTAnswer only58.2342.2341.4536.9433.3436.2738.9433.2948.9238.92
Expert-guidedAnswer only60.8241.2341.2036.1030.1329.6337.4132.9149.1037.21
Expert-guidedRandom evidence61.4341.5142.1035.1031.2328.2337.5231.2349.2137.02
Expert-guidedLLM-selected evidence61.5445.8543.2339.2332.3335.8738.4033.3150.4139.83
Expert-guidedRaw expert evidence63.1047.1042.4532.9634.3729.3230.1625.5846.7736.09
Expert-guidedSocialReasoner63.2248.4646.3742.9737.6440.4639.9536.0950.7842.84

All fine-tuned variants use Qwen2.5-7B-Instruct and are trained exclusively on ATP152. OOD Avg. denotes the unweighted average across eight unseen surveys.

The results reveal a fundamental distinction between learning survey-specific response patterns and acquiring transferable decision knowledge.

First, expert-guided Answer only improves source accuracy to 60.82%, but reduces average OOD accuracy from 40.70% (Base) to 37.21%. Random evidence similarly achieves only 37.02% OOD accuracy. Thus, neither expert-guided data selection nor additional language supervision alone is sufficient to ensure generalization.

AuthorsOpenReview ↗

Response to Weakness 2 (2/6) and Questions 1–2 (1/3, 2/3): Contributions of Expert-Guided Supervision and Generalization (Part 2/3)

Second, the content of reasoning supervision substantially affects transferability. LLM-selected evidence improves OOD accuracy to 39.83%, demonstrating the value of semantically relevant information. However, SocialReasoner further increases OOD accuracy to 42.84%, outperforming LLM-selected evidence by 3.01 percentage points and achieving higher accuracy on all eight unseen surveys.

Most importantly, Raw expert evidence achieves nearly identical source accuracy to SocialReasoner (63.10% vs. 63.22%), yet substantially lower OOD accuracy (36.09% vs. 42.84%). This 6.75-point generalization improvement, despite only a 0.12-point source difference, highlights the importance of semantic refinement in constructing transferable reasoning supervision.

Compared with expert-guided Answer only under the same training distribution, SocialReasoner improves OOD accuracy by 5.63 percentage points. These controlled comparisons demonstrate that reasoning supervision contributes beyond expert-guided data selection and reweighting, particularly in generalization to previously unseen surveys.

(3) Further Validation of Generalizable Decision Knowledge.

We further validate the transferability of learned decision knowledge through (i) progressive cross-survey training, (ii) robustness across training subsets, and (iii) generalization across diverse social decision contexts.

(i) Progressive Cross-Survey Knowledge Transfer.

We progressively expand training across ATP142, ATP152, EVS2017, and ISSP2009, while keeping five evaluation surveys strictly unseen throughout all training stages.

Table 3. Progressive Cross-Survey Knowledge Transfer (Accuracy, % ↑)

Dataset (Status)BaseS1: ATP142S2: +ATP152S3: +EVS2017S4: +ISSP2009
ATP142 (Source from S1)47.3058.2758.3758.4258.63
ATP152 (Source from S2)45.7153.5863.2463.3863.42
EVS2017 (Source from S3)44.0047.7148.1464.1664.17
ISSP2009 (Source from S4)42.9540.7542.8550.0352.05
ISSP2015 (OOD)33.3833.9833.8944.8046.97
ISSP2016 (OOD)35.7739.9840.1945.2346.97
ISSP2019 (OOD)39.0539.6340.1545.3246.56
ISSP2021 (OOD)33.2934.8136.0941.2143.56
GSS2022 (OOD)49.8550.9851.3852.5352.67
Unseen Avg. (OOD)38.2739.8840.3445.8247.35

Source indicates the stage at which a survey enters cumulative training. The five OOD surveys remain excluded from training at every stage. Unseen Avg. is computed exclusively across these five surveys.

As shown in Table 3, average accuracy across the five consistently unseen surveys increases from 38.27% (Base) to 47.35%, an improvement of 9.08 percentage points. Notably, introducing EVS2017 increases OOD accuracy from 40.34% to 45.82%, followed by a further improvement to 47.35% after incorporating ISSP2009.

These findings demonstrate that expanding training across different survey programs improves performance on previously unseen questions and respondents, supporting the accumulation and transfer of learned decision knowledge. As the amount of training data also increases, this experiment evaluates the combined effect of broader survey coverage and additional supervision.

(ii) Robustness Across Training Subsets.

To examine whether cross-survey improvements depend on a particular training subset, we repeat the GSS2022 evaluation using five independently sampled 10% subsets of the eight source surveys. SocialReasoner achieves 50.78%, 51.69%, 52.34%, 52.25%, and 52.93% (52.00% ± 0.81%). All five runs outperform the base model (49.85%), demonstrating consistent transfer performance across different training subsets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.