- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper builds a bilingual benchmark of 7,639 responsibility and blame questions (controlled vignettes plus reconstructed Reddit stories), annotated with five attribution dimensions, and evaluates 32 LLMs. On five small open models, it shows that the dimensions are linearly decodable from hidden states, and that steering along the probe directions shifts judgments in the theory-predicted direction, most consistently for intention and obligation.
Critical strengths
- A carefully built, bilingual, double-annotated dataset (κ = 0.826) with dimension labels.
- A careful probing protocol: leave-one-scenario-type-out CV, multiple decoding controls, and matched intervention controls.
Critical weaknesses
My main concern is limited novelty and insight. The probe-and-steer methodology closely follows Tak et al. (2025), as the related-work section acknowledges, so the new element is mainly the application domain. The main findings are largely expected: larger models agree more with the reference labels, dimensions written into the narratives are decodable, and pushing them toward "present" increases judgments. The mechanism analysis also does not explain the behavior.
The human reference is narrow. The labels come from two experts, and only items on which both agree are kept. This favors clearer items and reflects two people's judgments rather than a population, so "agreement with humans" should be interpreted cautiously.
Accuracy also largely reflects label calibration. The open-source models' average responsibility accuracy is below the majority-class baseline. Exact match on a subjective four-level scale therefore mostly measures whether a model uses the scale the same way as the two annotators.
Questions for authors
Please refer to the weakness
Please Review the Author Rebuttal and Participate in the Discussion
Dear Reviewers,
Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly encourage everyone to actively participate in the discussion and carefully consider the authors' rebuttal.
In particular, if you have not yet responded, please share your thoughts on whether the authors have adequately addressed your concerns. Please also engage with fellow reviewers where appropriate, and provide your final assessment, including any score updates or a brief justification for maintaining your original rating.
Your timely input would be greatly appreciated and will help us reach a fair and well-informed decision.
Thank you again for your time and valuable contributions!