acceptodds
Under review as a conference paper at ICLR 2027

RitualMind: Testing Relational Interpretation in Language Models

Abstract

The same utterance can perform a different social action when decision rights change while social identities remain fixed. Overall pragmatic accuracy leaves open which relationships an interpretation depends on and whether those relationships influence model computation. We introduce RitualMind, a controlled Chinese framework that follows a decision-rights contrast from interpretation preferences to recoverable internal information and its influence under intervention. Sensitivity to decision rights persists across alternative descriptions and recombined content in a subset of the four open-weight models tested. Authority direction also remains recoverable in several models across phrasings and source scenes excluded from decoder training, including at interpretation choice. Controlled replacement tests its influence on interpretation: in GLM-4-9B, opposite-authority states shift relative support beyond an equally large authority-preserving change. An attention component selected before confirmation contributes to this margin shift in new scenarios under new descriptions. Complementary open-response tasks include 504 responses assessed by three trained human experts, providing a human comparison for automated rubric judgments. Together, the experiments link current decision rights to a local computational contribution while distinguishing recoverable social information from its influence under intervention. RitualMind provides an experimental approach to social understanding that connects relationships that change meaning to computations that shape interpretation. Code and data: https://anonymous.4open.science/r/Ritualmind-2C04.

Primary area:
interpretability and explainable AI
Keywords:
large language models, pragmatic interpretation, elational reasoning, mechanistic interpretability, causal interventions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fU7BOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work investigates how LLMs interpret the same utterance under different decision-authority relations. Through behavioral evaluations and causal interventions, the authors examine how relational information is represented in LLMs and influences their interpretations.

Critical strengths

The experimental results are rigorously analyzed and reported, with appropriate statistical tests and confidence intervals.

Critical weaknesses
  1. Insufficient data validation. The construction of the synthetic scenarios and their target interpretations requires further validation. Moreover, the agreement between automated evaluation and human judgments is relatively low, raising concerns about evaluation reliability (Section 3.4). Furthermore, the scenarios and interpretation-choice tasks are highly synthetic, raising concerns about how well the findings generalize to real-world social interactions.

  2. Inconsistent or unsignificant findings. The observed effects are not always consistent or statistically significant. For example, only two of the four models show significant behavioral effects under both authority-change conditions in Section 4.1. Similar inconsistencies are observed in other experiments, limiting the generalizability of the findings.

  3. Limited contribution. The contribution is relatively incremental, as prior research has already shown that LLM behavior is sensitive to social power hierarchies [1-4]. While the probing and causal intervention experiments provide additional insights, the findings are not sufficiently distinguished from existing work.

References

[1] Vijjini et al., Do LLM Agents Mirror Socio-Cognitive Effects in Power-Asymmetric Conversations? ACL 2026

[2] Joswin et al., A Mechanistic View of Authority Hierarchy in LLM Sycophancy, Mechanistic Interpretability Workshop at ICML 2026

[3] Geng et al., Control Illusion: The Failure of Instruction Hierarchies in Large Language Models, AAAI 2026

[4] Atif et al., The Shape of Power: A Multilingual Framework for Social Power Reasoning in Dialogues, Arxiv 2026

Questions for authors

It would be helpful to report target-action accuracy and interpretation flip rates alongside probability changes in the behavioral sensitivity and intervention experiments. Currently, choice accuracy is reported only in certain experiments (e.g., Figure 5). These additional metrics would help assess whether the observed probability shifts translate into meaningful changes in model behavior.

Reviewer_jouBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper introduces RitualMind, a controlled Chinese-language framework for investigating how issue-specific decision authority affects pragmatic interpretation in large language models (LLMs). The authors manipulate decision rights while keeping the target utterance and participant identities fixed, and examine the resulting effects through behavioral evaluation, hidden-state decoding, and activation intervention. Experiments on four open-weight models show that authority-sensitive interpretation persists across alternative descriptions and recombined scenarios in a subset of models. Authority information is recoverable from hidden states across phrasings and held-out source scenes. Activation replacement further provides evidence of a local computational contribution, particularly in GLM-4-9B, with subsequent confirmation of a selected attention component on new scenarios. The paper’s main contribution is a controlled experimental framework connecting social-pragmatic interpretation with mechanistic analysis. While the experimental design is promising, questions remain regarding the validity of the predefined interpretation mapping, generalization beyond synthetic Chinese scenarios, and the strength of the causal conclusions.

Critical strengths
  1. The paper isolates issue-specific decision authority while keeping participant identities and the target utterance fixed. This provides a principled way to investigate relational influences on pragmatic interpretation.

  2. The combination of behavioral sensitivity, cross-form decoding, and activation intervention is valuable. The authors distinguish recoverable information from its behavioral influence and include meaningful controls and subsequent component-level confirmation.

Critical weaknesses
  1. The predefined mapping between authority conditions and candidate interpretations may influence the observed effects. The use of synthetic, historically styled Chinese scenarios also limits conclusions about natural dialogue and cross-linguistic generality.

  2. The intervention controls are useful but do not fully establish that the localized component specifically mediates authority-sensitive interpretation. The lack of prominent necessity or path-level tests, together with description-dependent effects, limits the strength of the mechanistic conclusions.

Questions for authors
  1. How robust are the primary behavioral findings to paraphrasing candidate interpretations, randomizing option order, and changing authority-description wording? Can the authors independently validate the authority-to-interpretation mapping?

  2. Can additional intervention controls, denoising/noising analyses, or path-patching experiments help distinguish specific causal contributions from distribution-shift effects and alternative pathways?

  3. Could the authors report donor effects DD, control effects CC, and control-adjusted effects E=D−CE=D-C together across the principal layer depths, clarifying how the effect magnitudes relate to actual interpretation-choice changes?

  4. How sensitive are the findings to the synthetic, historically styled Chinese setting? If evidence from contemporary dialogue or other languages is unavailable, can the scope of the claims be explicitly limited?

  5. Could the authors clarify the human annotation coverage and aggregation procedures, particularly given the disagreement between expert-based and automated dialogue task-success estimates?

Reviewer_Kc7bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RitualMind, a controlled evaluation framework for investigating how decision authority influences pragmatic interpretation in LLMs. Through behavioral evaluation, representation probing, and causal interventions across four open-weight LLMs, the authors examine whether models recognize, encode, and causally use authority-related information when interpreting utterances in social contexts.

Critical strengths

The paper addresses an interesting and well-motivated problem of how decision authority influences pragmatic interpretation in LLMs. The controlled experimental design effectively isolates changes in decision rights while keeping social identities and utterances fixed. The evaluation combines behavioral analysis, representation probing, and causal interventions across four open-weight LLMs, providing complementary evidence about authority-related information and its influence on interpretation. The paper also includes several robustness checks, such as alternative phrasings, content recombination, and held-out scenarios, to strengthen its findings.

Critical weaknesses
  1. The causal interventions do not sufficiently isolate the contribution of decision authority. The paper uses activation replacement to investigate whether authority-related information causally influences pragmatic interpretation. However, the difference between donor and recipient hidden states may capture lexical and other semantic changes beyond decision authority. Although the authors introduce an equally large same-authority control, matching the magnitude of interventions does not guarantee that their effects are authority-specific. 2.The probing results require stronger lexical controls. In Section 4.2, the authors report that character-level TF–IDF perfectly separates the authority conditions with reversed label direction (raw AUC = 0, orientation-free AUC = 1). This suggests that the experimental conditions contain strong lexical cues that may also contribute to hidden-state decoding performance. Although the authors compare hidden-state probes against word-level TF–IDF, further analysis is needed to establish whether the probes capture authority-related information beyond surface-level lexical differences. Stronger lexical baselines and controlled paraphrases would help address this concern.
  2. Human evaluation does not directly validate the primary pragmatic interpretations. Although the paper includes 504 responses evaluated by three human experts, this evaluation focuses on supplementary open-response tasks rather than validating the interpretation labels used in the main authority-reversal experiments. Furthermore, exact agreement between human and automated evaluations ranges from only 49.8% to 65.3%, and the oracle-minus-active dialogue comparison yields different conclusions. Human validation of the primary interpretation labels and a clearer analysis of human–automated evalua
Questions for authors

In Section 4.2, character-level TF–IDF achieves a raw AUC of 0 but an orientation-free AUC of 1. Could the authors explain this result and clarify whether the hidden-state decoding performance reflects authority-related representations beyond surface-level lexical cues?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.