- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces the Competing Ownership Attribution Task (COAT), comprising 42 scenarios with competing ownership claims, and compares allocations from 108 participants recruited in China with 24 LLM configurations. Respondents distribute 100 points among the claimants, and each model configuration contributes five valid responses per scenario. The analyses examine response similarity, distributional correspondence, allocation evenness, consensus structure, and changes across related contexts. The authors find that model judgments broadly resemble human judgments but are more homogeneous, divide ownership more evenly within individual answers, and sometimes respond differently to contextual changes. The principal contribution is an empirical evaluation task and comparative analysis of ownership judgments.
Critical strengths
- A concrete task with a human comparison. COAT provides a structured set of disputes involving different grounds for ownership and includes individual human responses. This offers potentially useful material for evaluating social judgments in situations where a single reference answer would conceal disagreement.
- A useful distinction between allocation evenness and response diversity. The distinction between dividing ownership among multiple claimants within one answer and preserving different viewpoints across answers is useful. Appendix D.1 also checks model homogeneity without averaging repetitions, addressing a plausible statistical artifact.
Critical weaknesses
- Human reference sample. The final sample of 108 participants lacks a sample-size justification, and participant demographics and relevant backgrounds are insufficiently reported. This leaves the reliability and representativeness of the human reference unclear.
- Validity of the output-based evaluation. The evaluation measures similarity between LLMs’ and human participants’ final ownership allocations, without directly assessing which factors they consider or how they weigh those factors during judgment. Consequently, similar final scores do not establish that humans and LLMs rely on comparable grounds for ownership attribution.
Questions for authors
Does evaluating LLMs through similarity to human participants’ judgments introduce excessive subjectivity? What evidence establishes that the human judgments themselves reflect accurate ownership attribution?