- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This work investigates how LLMs interpret the same utterance under different decision-authority relations. Through behavioral evaluations and causal interventions, the authors examine how relational information is represented in LLMs and influences their interpretations.
Critical strengths
The experimental results are rigorously analyzed and reported, with appropriate statistical tests and confidence intervals.
Critical weaknesses
-
Insufficient data validation. The construction of the synthetic scenarios and their target interpretations requires further validation. Moreover, the agreement between automated evaluation and human judgments is relatively low, raising concerns about evaluation reliability (Section 3.4). Furthermore, the scenarios and interpretation-choice tasks are highly synthetic, raising concerns about how well the findings generalize to real-world social interactions.
-
Inconsistent or unsignificant findings. The observed effects are not always consistent or statistically significant. For example, only two of the four models show significant behavioral effects under both authority-change conditions in Section 4.1. Similar inconsistencies are observed in other experiments, limiting the generalizability of the findings.
-
Limited contribution. The contribution is relatively incremental, as prior research has already shown that LLM behavior is sensitive to social power hierarchies [1-4]. While the probing and causal intervention experiments provide additional insights, the findings are not sufficiently distinguished from existing work.
References
[1] Vijjini et al., Do LLM Agents Mirror Socio-Cognitive Effects in Power-Asymmetric Conversations? ACL 2026
[2] Joswin et al., A Mechanistic View of Authority Hierarchy in LLM Sycophancy, Mechanistic Interpretability Workshop at ICML 2026
[3] Geng et al., Control Illusion: The Failure of Instruction Hierarchies in Large Language Models, AAAI 2026
[4] Atif et al., The Shape of Power: A Multilingual Framework for Social Power Reasoning in Dialogues, Arxiv 2026
Questions for authors
It would be helpful to report target-action accuracy and interpretation flip rates alongside probability changes in the behavioral sensitivity and intervention experiments. Currently, choice accuracy is reported only in certain experiments (e.g., Figure 5). These additional metrics would help assess whether the observed probability shifts translate into meaningful changes in model behavior.