- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates how models infer user attributes, such as gender, age, wealth, and personality, from conversations, and whether these inferences influence their responses. The authors compare five methods for reading out user attributes and intervene by modifying either prompts or internal activations, measuring response changes while reusing the same sampling noise. Finally, they apply this framework to occupation-based user profiling audits, examining models’ attribute associations and their effects on generated content.
Overall, I appreciate the distinction between reading out user attributes and measuring their influence on responses. The authors also conduct detailed experiments across multiple models, readout methods, and intervention approaches.
Critical strengths
I appreciate the authors’ systematic comparison of different measurement methods, as well as the clear signposting throughout the paper, which makes it easy to follow.
Critical weaknesses
I am unsure how the paper verifies that its readouts reflect judgments about the user that the model had already formed during the original generation process. The first-stage evaluation primarily checks whether the readouts match the attribute labels associated with the text. This can validate attribute recognition, but I do not understand how it distinguishes between reading out an existing user representation and inferring the attribute anew when the model is asked an additional question.
Current LLMs often exhibit a related problem: they may initially hold one view but shift to agree with a user who expresses a different view, displaying sycophancy. I suspect that the experiments have not ruled out this possibility, which raises concerns about the experimental design itself. If the authors can address this issue, I would be willing to raise my score.
Questions for authors
Can the effects of attribute prompts be attributed to the model’s implicit assumptions about the user when it generated the original response? For example, when recommending a car, explicitly telling the model that “the user is wealthy” may understandably lead it to recommend a more expensive vehicle. However, this result alone does not establish that the original recommendation was already driven by an implicit judgment about the user’s wealth.
6 comments
Rebuttal to Reviewer W3tR (2/3)
W2. Current LLMs often exhibit a related problem: they may initially hold one view but shift to agree with a user who expresses a different view, displaying sycophancy. I suspect that the experiments have not ruled out this possibility, which raises concerns about the experimental design itself. If the authors can address this issue, I would be willing to raise my score.
We thank the reviewer for this important check and for their openness to raising their score if addressed. In summary, if sycophancy occurs, it does so only in the first part of Step 2 (counterfactual generation), but to a limited extent, and its impact on our overall conclusions is limited.
Specifically, (Step 1) all reported belief measurements (including monoculture and stereotypes) are read from unmodified prompts with no stated attribute and a neutral probe, so sycophancy cannot arise there. (Step 2a) while generating counterfactuals might involve stating an attribute value, new experiments comparing our text intervention against text-free Inference-Time Intervention (ITI), which modifies internal activations, show that both shift answers in the same direction 70% of the time (when ITI actually answers the question without drifting), indicating that any pure accommodation effect is modest. (Step 2b) our final attribution evaluates the original, unperturbed answer by measuring which counterfactual it is closest to, which largely insulates the comparison from any residual accommodation shared across the counterfactuals. This is confirmed by new experiments showing that stripping all explicit attribute mentions leaves attribution unchanged (78.8% vs. 78.7%). Below, we provide more detail.
Step 2a. Counterfactual generation. In this step, an attribute value might be stated as a profile field (e.g., "Gender: woman") or via rewriting. To test whether the model merely accommodates the stated label, we compared our text intervention with ITI, which shifts the model's internal state directly in activation space with no change to the text, leaving nothing in the prompt to accommodate.
On the exact same conversations, questions, and random noise (Qwen3-32B and Gemma-4-31B), when ITI stays close to the question (rather than drifting to unrelated content or outputting repeated text), the answer obtained by stating a value is semantically closer to the ITI answer for the same value than to the answer for the opposite value in 70% of cases. This holds specifically on attribute-relevant questions (neutral questions remain at 50%), while a random-steering control stays at 50%.
Explicitly stating a value and silently shifting the internal belief therefore change the answer in a consistent way. While we cannot completely exclude a residual contribution of sycophancy in Step 2a, these results indicate that its effect, if any, is minor, and the final attribution step further reduces its impact on our conclusions.
Step 2b. Attributing the original answer. The original answer is generated with nothing stated, and every counterfactual it is compared with carries a stated value, so any sycophancy applies to all counterfactuals alike, while attribution only asks which one the original is closest to. A new test in which we removed every explicit mention of the attribute leaves attribution unchanged (78.8% vs. 78.7%; please see the response to Reviewer pcfs Q3 for details).
Lastly, sycophancy within a conversation, where the user (not the system) states a view, could be tracked by repeating the readout after every turn. We did not study this in the present work, but it is another potential application of the framework.
Rebuttal to Reviewer W3tR (3/3)
Q1. Can the effects of attribute prompts be attributed to the model’s implicit assumptions about the user when it generated the original response? For example, when recommending a car, explicitly telling the model that “the user is wealthy” may understandably lead it to recommend a more expensive vehicle. However, this result alone does not establish that the original recommendation was already driven by an implicit judgment about the user’s wealth.
We agree: a changed answer after stating "wealthy" does not show that the model had assumed wealth in the original answer. Our evidence rests on a different comparison. We generate the original answer with only the occupation stated (e.g., "I am a janitor"). We then regenerate it with "rich" or "poor" stated explicitly, using the same random noise (Gumbel-max), so the stated value is the only difference. What matters is which counterfactual the original resembles. For the janitor, the original matches the "poor" answer, and the readout places 0.99 on "poor", so the model behaves as if it had already assumed the user is poor.
Because closeness is strong evidence, not proof of an internal judgment, in the revision we will limit causal language to the effect of the stated value and describe the rest as the model behaving as if it had made an assumption. Our main findings, such as the monoculture and the stereotypes, are aggregated over many answers and come from the readout, so they do not depend on closeness.
Rebuttal to Reviewer W3tR (1/3)
Thanks and Summary
We thank the reviewer for the thoughtful question and the clear statement of what would change their assessment. We are glad the reviewer found the comparison of measurement methods systematic and the paper easy to follow. We address the concern about existing judgments versus inference below, and hope this resolves it.
Existing judgment vs. inference when asked (W1). Three results in the paper show that the readout reflects information the model has already formed rather than inferring it on the spot: a probe that reads the model's representation of the conversation alone, with no question asked, recovers the attribute almost perfectly; on prompts with no cue about the user, the readout stays close to uniform, so asking does not create a belief; and the original answers, generated without any question, already track the assumed attribute.
Sycophancy (W2). The readout question expresses no view for the model to agree with, and we average over every option order and ten phrasings, so any tendency to follow the wording cancels out. For the intervention, we verified that the answers match those intervened by ITI, which modifies internal activations and therefore cannot generate sycophancy.
The car example (Q1). The original answer is generated with nothing stated about the user's wealth and is compared with answers generated, using the same random noise, with "rich" or "poor" stated. We will reserve causal language for the effect of the stated value, which does not change our results.
We hope these answers resolve the reviewer's concerns, and would be grateful if the reviewer would reconsider their assessment in light of them.
W1. I am unsure how the paper verifies that its readouts reflect judgments about the user that the model had already formed during the original generation process…
Two results in the paper address this:
The information is present before any question is asked. MEAN-PROBE (and the new LENSES results) reads the model's internal representation of the conversation alone, with nothing appended, and recovers the attribute almost perfectly (κ=0.98−0.99, Table 1). The question-based readouts reach the same accuracy, so they read information the model has already encoded.
Asking does not create beliefs. On real prompts screened to carry no cue about the user (App. H.1), LOGIT stays close to uniform, with little variation across prompts. If the question made the model form a new belief, these prompts would receive confident readings rather than uniform ones. Notably, MEAN-PROBE exhibits substantially stronger deviations from uniformity (mean KL of 0.61 versus 0.25 for LOGIT), reflecting stronger default commitments in the absence of user-specific cues.