- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper argues for understanding social authority attribution in LLMs according to two axes: social authority (SA) and triage hierarchy (TH). They propose the "Authority Expectancy Effect", where LLMs implicitly infer authority from latent cues. This builds on prior work that only studies authority in LLMs, and shows that the urgency of an issue (its place in the "triage hierarchy") affects LLM biases in a way that is separable from authority. They study this in the domain of dispute resolution, as an analogy to disputes common in legal settings.
Critical strengths
To my knowledge, the main contribution of this paper is novel, studying how authority interacts with the urgency of an issue.
Critical weaknesses
- The results here are relatively minimal, to my understanding, and this feels more like a workshop paper at best
- Generally poorly written in an LLM-y kind of way.
- E.g. from the abstract: "... evidentiary cues do not exert a fixed directional influence in multi-turn disputes, where identical documentation draws protection to one party or extends it to both depending on which party holds it" - This feels like a very overly complicated and jargon-y way to explain (what I think is) a simple concept. There are many sentences like this throughout the paper, making it very difficult to read.
- Poorly organized and presented
- Section 4.1 feels more like a big list of things the authors tried rather than a principled list of methods. Given how hard it is to parse the opening paragraph, I'm not sure what the goals of this section are
- Similarly, the bolded parts of section 5, and sub-sections under 5.2, read like a long list of various things tried, rather than forming a coherent narrative. I'm also not sure if most of the text under each of these is justified being in the main text, and there is text here that seems like it should instead be in the methods or discussion sections.
The poor presentation of this work makes it very difficult to understand what the authors did, what their main results are, and how to interpret their experiments. In my understanding, which may be affected by this presentation, the authors' core contributions are limited.
Questions for authors
- Can the authors provide more explanation of their basic experimental setup? I struggled to understand what the prompts look like for this kind of dispute resolution.
- For the example (lines 188-190) "Two parties sustain an identical eye injury and each blames the other." This seems like a very uncommon scenario for people to find themselves in. What common real-world situations follow this pattern?
- Are the scenarios the LLMs see similar to this, and if so, are there concerns about ecological validity?
2 comments
Response to Feedback and Manuscript Revisions
Thank you for this feedback. We agreed that the manuscript was overly condensed in several places. In response, we substantially revised the entire manuscript and uploaded a new PDF. The revisions aim to improve readability and clarify the main findings throughout the paper.
Presentation and Structure / Research Questions and Experimental Setup / Key Results / Ecological Validity
Comments on Presentation and Structure
The sentences in the abstract and the main text were overly condensed. Furthermore, sections §4.1 and §5 focused more on listing conditions and numerical results for each model rather than explaining the underlying design intent, which likely made it difficult to grasp the core findings. Below, we provide a brief restatement of the research objectives, experimental design, and key results.
Research Questions and Experimental Setup
Existing research has treated “authority signals”—such as one’s occupation—as factors that add a fixed weight to one side of a judgment. This study examines whether this holds true in situations involving competition or conflict between two parties. In each experiment, we presented the statements of two opposing parties to the models and asked them to decide who should receive a resource (Phase 1), whose fault was greater (V1, V2), and what guidance should be provided to each party (V3). We then varied whether occupation labels and medical reports were present, and which party held them, while keeping the underlying circumstances constant. Each condition was tested 200 times across six different models.
Key Results
First, the effect of a given signal depends on the presence of other signals. For instance, when provided with only a medical report (without information about occupation), Claude refused to make a judgment in 97% of cases; however, when occupational information was also provided, this refusal rate dropped to 22%.
Second, the same piece of evidence can lead models to favor different parties depending on who holds it. In condition V3, when a student held a medical report, several models offered protective guidance exclusively to the student. In contrast, when a professor held the same medical report, GPT and Llama provided protective guidance to both parties.
Third, occupation does not function solely as a marker of authority. In V1, when no occupations were specified, all models consistently assigned equal responsibility to both parties across all 200 trials. However, when the roles of “surgeon” and “painter” were introduced, three models assigned fault to the painter in most runs. The models justified this by pointing to the surgeon’s fine motor control—meaning that the occupation was used not as an indicator of authority, but as a piece of information to infer the circumstances of the incident.
These findings cannot be explained by existing models that treat authority as a fixed weight, nor can they be captured by conventional evaluation methods that only measure average shifts in outcomes.
A Note on Ecological Validity
The “identical eye injury” in V1 is not intended to represent a common real-world occurrence; rather, it serves as a control mechanism to isolate the effect of occupation by keeping the nature of the injury constant. If the injuries differed, it would be impossible to determine whether any disparity in judgment stemmed from the individuals’ occupations or the injuries themselves.
That said, the underlying structure of the scenario is common in real-world contexts. Situations where two parties accuse one another—such as mutual reporting on online platforms, reciprocal grievances in the workplace, counter-allegations in cases of school bullying, or insurance disputes—occur frequently. Large Language Models (LLMs) are already being deployed in areas like report classification, customer service, and dispute resolution support. In such scenarios, models are often presented with unverified claims from both sides, alongside supplementary information such as the individuals’ occupations or relevant documentation.
Multi-party conversation features, in which a single model interacts with several users at once, have begun to be deployed in some models. However, we did not conduct a user study, and therefore have not verified whether users actually engage in disputes of this kind within such group conversations. While the dispute structures described above are already common, this study does not aim to document failures of current deployments; instead, it focuses on disputes that are likely to arise as models increasingly participate in conversations among multiple parties in the near future. In this context, extreme, resource-scarce settings such as a disaster serve as a "stress test," designed to reveal the criteria upon which models base their prioritization. To our knowledge, no prior work has examined how authority and vulnerability signals interact when a model adjudicates between competing parties.
As non-native English speakers, we compressed our findings into dense phrasing, which made several key sentences, including the one quoted from the abstract, ambiguous. We will revise the manuscript throughout for clarity.