acceptodds
Under review as a conference paper at ICLR 2027

Examining Social Attribution in LLM Reasoning: A Theory-Guided Probing Methodology

Abstract

Large language models (LLMs) are increasingly deployed in sociotechnical systems to facilitate social interactions, in which *social attribution*, the reasoning process attributing external events to the causes and reasons of agents' social behaviors, plays a critical role. Such reasoning processes involve the judgments of social cause, responsibility, and blame/credit to agents. Although attributional models are well-studied in social psychology and social cognition, represented by Attribution Theory, it is still underexplored in the AI and computing community, and LLM social reasoning in particular. This paper aims to advance the understanding of LLM in a fundamental social aspect by providing the first systematic exploration of LLM social attribution. Our work focuses on responsibility and blame attributions, examining current LLMs' performances in making responsibility and blame judgments, as well as their internal mechanisms underlying such judgments. Specifically, guided by attribution theory, we first construct a social attribution benchmark consisting of a *Vignette* subset based on classic scenarios from attribution theory research and a *Reality* subset constructed based on real-world social narratives, yielding **7,639** responsibility/blame judgment questions. On this basis, we evaluate **32** representative LLMs and **5** basic non-LLM baselines. To further explore the internal mechanisms underlying the LLM judgment process, we develop a probing-based methodology to investigate the latent-space representations of **5** key attribution dimensions and the consistency of their influences on LLM judgments compared to those in human social attribution. Our research findings reveal that current LLMs exhibit measurable but incomplete agreement with human responsibility and blame judgments, and meanwhile, this agreement is positively correlated with model size. At a deep level, some attribution dimensions are systematically decodable from specific positions in LLM hidden states, and their influences on the final judgment are consistent with those indicated by human Attribution Theory.

Primary area:
applications to neuroscience & cognitive science
Keywords:
LLM social attribution, Responsibility and blame judgments, Attribution theory, Latent space dimension probing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Vt4zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds a bilingual benchmark of 7,639 responsibility and blame questions (controlled vignettes plus reconstructed Reddit stories), annotated with five attribution dimensions, and evaluates 32 LLMs. On five small open models, it shows that the dimensions are linearly decodable from hidden states, and that steering along the probe directions shifts judgments in the theory-predicted direction, most consistently for intention and obligation.

Critical strengths
  1. A carefully built, bilingual, double-annotated dataset (κ = 0.826) with dimension labels.
  2. A careful probing protocol: leave-one-scenario-type-out CV, multiple decoding controls, and matched intervention controls.
Critical weaknesses

My main concern is limited novelty and insight. The probe-and-steer methodology closely follows Tak et al. (2025), as the related-work section acknowledges, so the new element is mainly the application domain. The main findings are largely expected: larger models agree more with the reference labels, dimensions written into the narratives are decodable, and pushing them toward "present" increases judgments. The mechanism analysis also does not explain the behavior.

The human reference is narrow. The labels come from two experts, and only items on which both agree are kept. This favors clearer items and reflects two people's judgments rather than a population, so "agreement with humans" should be interpreted cautiously.

Accuracy also largely reflects label calibration. The open-source models' average responsibility accuracy is below the majority-class baseline. Exact match on a subjective four-level scale therefore mostly measures whether a model uses the scale the same way as the two annotators.

Questions for authors

Please refer to the weakness

Reviewer_pay2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a theory-guided probing framework for examining social attribution in large language models (LLMs), focusing on responsibility and blame judgments. The authors construct a bilingual benchmark with 7,639 questions, evaluate 32 LLMs and five non-LLM baselines, and investigate the representation and influence of five attribution dimensions through hidden-state probing and activation interventions.

Critical strengths

The topic is interesting, and the empirical study is extensive.

Critical weaknesses

The strength of its central mechanistic claims is not yet fully supported by the current validation, particularly regarding causal interpretation and statistical dependence in intervention testing. The weakness is shown as follows.

  1. The generalizability of the benchmark and human judgment references requires further validation.
  2. The causal interpretation of the probing and intervention results needs stronger justification.
  3. The statistical testing procedure may overestimate the significance of the intervention results.
  4. The scope of the mechanistic evaluation is relatively limited compared with the breadth of the behavioral benchmark.
Questions for authors

Follow the parts of Critical Weaknesses.

Reviewer_ZN29OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds a bilingual (EN/ZH) benchmark of 7,639 responsibility/blame questions. It has two parts: controlled vignettes, where five attribution dimensions are set by design, and rewritten Reddit AITA stories, where the dimensions are annotated by experts. The dimensions are intention, voluntariness, foreknowledge, controllability, and obligation. 32 LLMs are evaluated; the best reaches about 58% accuracy, and open models improve with size. On five small models, the authors train linear probes for the dimensions and steer along the probe directions. They conclude that intention and obligation are decodable and that steering along them shifts judgments in the direction attribution theory predicts, mostly in middle and later layers.

Critical strengths

The benchmark is carefully built. The dimension definitions are clear, annotation is blind, agreement is high (κ = 0.826), and the evaluation is broad. I think the data is useful.

Critical weaknesses
  1. The intervention results don’t show “consistency with attribution theory.”
  • All five dimensions are positively correlated with the judgment level in this data. The edit is applied at the label-prediction step, one token is generated, and the argmax is restricted to the four labels. Under these conditions, any direction that points toward “higher judgment” would pass.
  • The random-direction and shuffled-label controls do not rule this out.
  • The paper never tests the theory’s differential predictions, for example that voluntariness should matter more for blame than for responsibility. So “consistent with theory” here amounts to “has the same sign as the label correlation.”
  • About 250 cells in Fig. 6 are tested with no multiple-comparison correction.
  1. The probes may be reading agent roles rather than the dimensions. In the vignettes, several dimensions are fixed by role within each template, and the same role structures (superior/subordinate, supervisor/supervisee) recur across scenario types. Holding out one scenario type does not remove this shortcut. The lexical control uses scenario text only, with no information about the target agent, so it is a weak baseline. Even so, the two smallest models do not beat it on foreknowledge and controllability (Figs. 10–11).
Questions for authors
  1. If you residualize each dimension direction against a probe for the judgment level itself, do the intention/obligation effects survive? What happens if you steer along a direction that is decodable but irrelevant to the judgment, such as the randomized gender?
  2. How much of the probe advantage remains against a baseline that is given the agent’s role and description?
  3. MoCa (Nie et al., NeurIPS 2023) already measures LLM–human alignment on causal and moral judgments, and analyzes how models weight the annotated factors compared with people. It is not cited. What does the behavioral part of this paper add beyond MoCa, and what supports the “first systematic study” claim?

Comments on the paper

1 comment
Area_Chair_yuGKOpenReview ↗

Please Review the Author Rebuttal and Participate in the Discussion

Dear Reviewers,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly encourage everyone to actively participate in the discussion and carefully consider the authors' rebuttal.

In particular, if you have not yet responded, please share your thoughts on whether the authors have adequately addressed your concerns. Please also engage with fellow reviewers where appropriate, and provide your final assessment, including any score updates or a brief justification for maintaining your original rating.

Your timely input would be greatly appreciated and will help us reach a fair and well-informed decision.

Thank you again for your time and valuable contributions!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.