acceptodds
Under review as a conference paper at ICLR 2027

Do Language Models Model You? Estimating Implicit Beliefs and Their Generative Effect

13 views

Abstract

Language models often infer user traits, such as gender, age, or wealth, from conversational cues. These implicit assumptions can silently shape responses, leaving users unaware that any profiling occurred. Auditing this behavior requires answering two tightly coupled questions: *what did the model believe?* (**Q1**) and *did that belief shape the output?* (**Q2**). To answer **Q1**, we evaluate five methods for belief readout across activations, logits, and text. We find that as model size increases, probes trained on internal activations perform no better than eliciting the belief directly from the model's output. To address **Q2**, we adopt a causal framework. We edit either the prompt or the model's activations to reach a target belief, and measure how the answer changes. Using the Gumbel-Max trick, the intervened model reuses the sampling noise of the original answer, so the change reflects only the intervention. The **Q1** readout serves as a "thermometer" that verifies whether each intervention reaches the target belief without changing other beliefs. We find that input-level interventions are more selective than activation steering at changing only the target belief. Finally, we use this framework to audit which beliefs 8 models infer from a user's occupation alone (60 jobs), and whether these beliefs change their answers. We find that job titles alone dictate strong monocultural stereotypes about gender and wealth. Crucially, these inferred beliefs can silently alter the substance or the style of the answers.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
beliefs, causality, assumptions, personalization, bias

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_W3tROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how models infer user attributes, such as gender, age, wealth, and personality, from conversations, and whether these inferences influence their responses. The authors compare five methods for reading out user attributes and intervene by modifying either prompts or internal activations, measuring response changes while reusing the same sampling noise. Finally, they apply this framework to occupation-based user profiling audits, examining models’ attribute associations and their effects on generated content.

Overall, I appreciate the distinction between reading out user attributes and measuring their influence on responses. The authors also conduct detailed experiments across multiple models, readout methods, and intervention approaches.

Critical strengths

I appreciate the authors’ systematic comparison of different measurement methods, as well as the clear signposting throughout the paper, which makes it easy to follow.

Critical weaknesses

I am unsure how the paper verifies that its readouts reflect judgments about the user that the model had already formed during the original generation process. The first-stage evaluation primarily checks whether the readouts match the attribute labels associated with the text. This can validate attribute recognition, but I do not understand how it distinguishes between reading out an existing user representation and inferring the attribute anew when the model is asked an additional question.

Current LLMs often exhibit a related problem: they may initially hold one view but shift to agree with a user who expresses a different view, displaying sycophancy. I suspect that the experiments have not ruled out this possibility, which raises concerns about the experimental design itself. If the authors can address this issue, I would be willing to raise my score.

Questions for authors

Can the effects of attribute prompts be attributed to the model’s implicit assumptions about the user when it generated the original response? For example, when recommending a car, explicitly telling the model that “the user is wealthy” may understandably lead it to recommend a more expensive vehicle. However, this result alone does not establish that the original recommendation was already driven by an implicit judgment about the user’s wealth.

6 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer W3tR (1/3)

Thanks and Summary

We thank the reviewer for the thoughtful question and the clear statement of what would change their assessment. We are glad the reviewer found the comparison of measurement methods systematic and the paper easy to follow. We address the concern about existing judgments versus inference below, and hope this resolves it.

  • Existing judgment vs. inference when asked (W1). Three results in the paper show that the readout reflects information the model has already formed rather than inferring it on the spot: a probe that reads the model's representation of the conversation alone, with no question asked, recovers the attribute almost perfectly; on prompts with no cue about the user, the readout stays close to uniform, so asking does not create a belief; and the original answers, generated without any question, already track the assumed attribute.

  • Sycophancy (W2). The readout question expresses no view for the model to agree with, and we average over every option order and ten phrasings, so any tendency to follow the wording cancels out. For the intervention, we verified that the answers match those intervened by ITI, which modifies internal activations and therefore cannot generate sycophancy.

  • The car example (Q1). The original answer is generated with nothing stated about the user's wealth and is compared with answers generated, using the same random noise, with "rich" or "poor" stated. We will reserve causal language for the effect of the stated value, which does not change our results.

We hope these answers resolve the reviewer's concerns, and would be grateful if the reviewer would reconsider their assessment in light of them.


W1. I am unsure how the paper verifies that its readouts reflect judgments about the user that the model had already formed during the original generation process…

Two results in the paper address this:

  1. The information is present before any question is asked. MEAN-PROBE (and the new LENSES results) reads the model's internal representation of the conversation alone, with nothing appended, and recovers the attribute almost perfectly (κ=0.98−0.99\kappa = 0.98 - 0.99, Table 1). The question-based readouts reach the same accuracy, so they read information the model has already encoded.

  2. Asking does not create beliefs. On real prompts screened to carry no cue about the user (App. H.1), LOGIT stays close to uniform, with little variation across prompts. If the question made the model form a new belief, these prompts would receive confident readings rather than uniform ones. Notably, MEAN-PROBE exhibits substantially stronger deviations from uniformity (mean KL of 0.61 versus 0.25 for LOGIT), reflecting stronger default commitments in the absence of user-specific cues.

AuthorsOpenReview ↗

Rebuttal to Reviewer W3tR (2/3)

W2. Current LLMs often exhibit a related problem: they may initially hold one view but shift to agree with a user who expresses a different view, displaying sycophancy. I suspect that the experiments have not ruled out this possibility, which raises concerns about the experimental design itself. If the authors can address this issue, I would be willing to raise my score.

We thank the reviewer for this important check and for their openness to raising their score if addressed. In summary, if sycophancy occurs, it does so only in the first part of Step 2 (counterfactual generation), but to a limited extent, and its impact on our overall conclusions is limited.

Specifically, (Step 1) all reported belief measurements (including monoculture and stereotypes) are read from unmodified prompts with no stated attribute and a neutral probe, so sycophancy cannot arise there. (Step 2a) while generating counterfactuals might involve stating an attribute value, new experiments comparing our text intervention against text-free Inference-Time Intervention (ITI), which modifies internal activations, show that both shift answers in the same direction 70% of the time (when ITI actually answers the question without drifting), indicating that any pure accommodation effect is modest. (Step 2b) our final attribution evaluates the original, unperturbed answer by measuring which counterfactual it is closest to, which largely insulates the comparison from any residual accommodation shared across the counterfactuals. This is confirmed by new experiments showing that stripping all explicit attribute mentions leaves attribution unchanged (78.8% vs. 78.7%). Below, we provide more detail.

Step 2a. Counterfactual generation. In this step, an attribute value might be stated as a profile field (e.g., "Gender: woman") or via rewriting. To test whether the model merely accommodates the stated label, we compared our text intervention with ITI, which shifts the model's internal state directly in activation space with no change to the text, leaving nothing in the prompt to accommodate.

On the exact same conversations, questions, and random noise (Qwen3-32B and Gemma-4-31B), when ITI stays close to the question (rather than drifting to unrelated content or outputting repeated text), the answer obtained by stating a value is semantically closer to the ITI answer for the same value than to the answer for the opposite value in 70% of cases. This holds specifically on attribute-relevant questions (neutral questions remain at 50%), while a random-steering control stays at 50%.

Explicitly stating a value and silently shifting the internal belief therefore change the answer in a consistent way. While we cannot completely exclude a residual contribution of sycophancy in Step 2a, these results indicate that its effect, if any, is minor, and the final attribution step further reduces its impact on our conclusions.

Step 2b. Attributing the original answer. The original answer is generated with nothing stated, and every counterfactual it is compared with carries a stated value, so any sycophancy applies to all counterfactuals alike, while attribution only asks which one the original is closest to. A new test in which we removed every explicit mention of the attribute leaves attribution unchanged (78.8% vs. 78.7%; please see the response to Reviewer pcfs Q3 for details).

Lastly, sycophancy within a conversation, where the user (not the system) states a view, could be tracked by repeating the readout after every turn. We did not study this in the present work, but it is another potential application of the framework.

AuthorsOpenReview ↗

Rebuttal to Reviewer W3tR (3/3)

Q1. Can the effects of attribute prompts be attributed to the model’s implicit assumptions about the user when it generated the original response? For example, when recommending a car, explicitly telling the model that “the user is wealthy” may understandably lead it to recommend a more expensive vehicle. However, this result alone does not establish that the original recommendation was already driven by an implicit judgment about the user’s wealth.

We agree: a changed answer after stating "wealthy" does not show that the model had assumed wealth in the original answer. Our evidence rests on a different comparison. We generate the original answer with only the occupation stated (e.g., "I am a janitor"). We then regenerate it with "rich" or "poor" stated explicitly, using the same random noise (Gumbel-max), so the stated value is the only difference. What matters is which counterfactual the original resembles. For the janitor, the original matches the "poor" answer, and the readout places 0.99 on "poor", so the model behaves as if it had already assumed the user is poor.

Because closeness is strong evidence, not proof of an internal judgment, in the revision we will limit causal language to the effect of the stated value and describe the rest as the model behaving as if it had made an assumption. Our main findings, such as the monoculture and the stereotypes, are aggregated over many answers and come from the readout, so they do not depend on closeness.

Reviewer_gY85OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Language models silently infer user traits (gender, age, wealth, personality) from conversational cues, and those implicit "beliefs" can shape responses without the user's knowledge. The paper builds an end-to-end auditing framework around two coupled questions: Q1, what did the model believe? and Q2, did that belief change the answer? For Q1, the authors evaluate five belief readouts (activation probes vs. output elicitation) against Herrmann & Levinstein's criteria (accuracy, coherence, uniformity) across 8 models (1.7B -- 32B) and 8 concepts, finding that training-free output readouts (LOGIT, SAMPLE) match a fine-tuned LoRA-LOGIT readout at 14B and generalize is better/equal under distribution shift than probes. For Q2, they measure causal influence via interventions (prompt rewriting, profile injection, ITI activation steering), escalated until a readout "thermometer" verifies the target belief flipped without moving others, generating counterfactual answers with Gumbel-Max noise coupling. Prompt interventions prove far more selective than activation steering. A large audit (60 occupations, 8 models, ~59,040 measurements) shows job titles alone trigger strong, shared gender/wealth stereotypes -- models agree with each other more than with labor statistics -- and these beliefs measurably alter answers, including style-only shifts on factual questions. A proof-of-concept transparency interface is sketched.

Critical strengths
  1. Clean conceptual separation of Q1 (what the model believes) from Q2 (whether it changed the answer): aggregate bias benchmarks cannot do instance-level causal attribution; this framework can.
  2. Methodologically careful -- thermometer verification of interventions, a Random-direction control that exposes spurious relevant/neutral separation, and Gumbel-Max coupling so counterfactuals differ only by the intervention.
  3. The Q1 comparison is genuinely useful: training-free output readouts matching supervised probes at 14B (and beating them under shift), ≥\ge extending to API-only models via SAMPLE, is actionable guidance for auditors.
  4. Substantial empirical scale: 8 models, 5 families, ~59,040 audit measurements, per-concept breakdowns.
  5. The monoculture quantification (inter-model agreement exceeding agreement with labor statistics) is a crisp, citable framing, and the style-bias result (facts preserved, delivery shifts) is subtle and important.
Critical weaknesses
  1. No mitigation, and no path to one. The paper introduces and measures the implicit-bias problem but stops there. For a top venue, documenting a known problem without moving toward addressing it limits the contribution.
  2. Headline findings are confirmatory. That models stereotype occupations by gender and wealth is well established; the audit measures it more precisely but does not revise prior understanding.
  3. The framework's payoff is deferred. Its value depends on community adoption and downstream tooling such as the interface -- neither demonstrated.
  4. ΔSem\Delta Sem's dependence on a single embedder (itself an LLM-family model) is a methodological soft spot for the audit's quantitative claims.
  5. One-concept-at-a-time interventions and an explicit-cue Q1 benchmark leave the hardest, most realistic cases (implicit, intersectional) only partially covered.
  6. The user-facing transparency proposal is speculative — no user study, no evidence it helps rather than misleads.
Questions for authors
  1. The framework measures and documents bias but offers no mitigation, nor a path toward one. Even a small demonstration, e.g., whether counterfactual profile injection can neutralize a harmful inferred belief would substantially strengthen the practical contribution. Why was this left entirely out of scope?
  2. The headline audit finding (occupation gender/wealth stereotypes) --> confirms what the bias literature already reports. What is the most surprising result here -- the one that changes what the community believes or does?
  3. Δ\DeltaSem relies on a single external embedder (Qwen3-Embedding-8B). How sensitive are the audit's attribution rates to the embedder choice, given the embedder may share the very stereotypes being measured?
  4. Interventions target one concept at a time, but real profiling is intersectional (e.g., wealth × gender). Do the headline claims survive joint interventions? The appendix's co-movement analysis is only preliminary.
  5. The Q1 benchmark uses explicit cues as a "lower bound," but the audit's value rests on reading beliefs from implicit cues, where no ground truth exists. How many audit readings sit near the lean/consistency reporting threshold -- how much of the monoculture result rests on borderline readings?
  6. Instruction-tuned models sometimes refuse option letters on sensitive concepts (52% parse rate for Phi-4 on gender), patched with anti-refusal templates. Doesn't coaxing the model past its own safeguards complicate the claim that output readouts are the natural, faithful channel?
  7. ITI is compared at the strength needed to reach τ=0.90\tau = 0.90 on a LoRA-LOGIT thermometer. Is the selectivity gap between prompt edits and ITI robust to the choice of thermometer and τ\tau?
  8. The transparency interface is an unevaluated proof of concept. Is there any evidence that surfacing beliefs improves user trust or decisions or could wrong readouts mislead users?
  9. In the manager/janitor example the model assigns 0.99 to wealthy/poor. But is cheaper-car advice to a janitor necessarily a harm rather than reasonable personalization? The paper says the framework "informs but does not resolve" this -- where do the authors themselves draw the line for the audit's headline claims?
  10. Q2 validation covers four concepts on synthetic profiles and screened conversations. Any evidence the selectivity results hold for the personality concepts or noisier real traffic?
7 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer gY85 (1/6)

Thanks and Summary

We thank the reviewer for the detailed and constructive review. We are glad the reviewer found the framework methodologically careful and the monoculture and style findings important. We have addressed every point raised, with new experiments wherever the reviewer asked for evidence, and we hope these resolve the remaining concerns about soundness.

  • Mitigation (W1, Q1). We show how the framework supports mitigation by detecting where a model holds a stereotyped belief and verifying whether a fix removed it, with a demonstration on BBQ across three models.

  • Embedder dependence (W4, Q3). We repeated the attribution analysis with two measures that contain no learned associations; the pattern across question families is unchanged.

  • Intersectional cases (W5, Q4). We ran a new joint gender × wealth audit (4 models, 60 jobs, 11,280 measurements): the headline claims hold, and models also make intersectional assumptions.

  • Borderline readings (Q5). Clarified that this conclusion did not depend on borderline reading but also recomputed the monoculture without readings near the thresholds, and with only readings too weak to report; the results are essentially unchanged.

  • Robustness of the ITI comparison (Q7). We repeated the full validation with a different thermometer: all results hold and the gap widens. A sensitivity analysis over τ=0.7\tau = 0.7, 0.80.8 and 0.90.9 confirmed that even at its best, ITI does not match either input-level edit.

  • The interface (W6, Q8). We are running a user study comparing answers with and without the belief readout and counterfactual.

We also clarify the most surprising findings (Q2), the anti-refusal templates (Q6), where we draw the line between bias and personalisation (Q9), and the coverage of real conversations and personality traits (Q10). We will add all new analyses to the paper, and we hope they address the reviewer's concerns.


W1 & Q1. The framework measures and documents bias but offers no mitigation, nor a path toward one. Even a small demonstration, e.g., whether counterfactual profile injection can neutralize a harmful inferred belief would substantially help.

We agree that mitigation is important. However, our primary goal is transparency, which is both valuable in itself and a precondition for mitigation: hidden assumptions about the user (e.g., gender, age, or wealth) cannot be fixed if they cannot be seen. Furthermore, because the same inferred trait can be an unwanted stereotype for one user and useful personalisation for another, surfacing assumptions lets users and developers decide when to intervene rather than enforcing a fixed policy.

That said, our framework directly supports mitigation in two ways: (1) detection, where our Q1 step (the readout) flags a conversation in which the model holds a confident belief about the user before it answers; and (2) verification, where the same readout (the thermometer of Section 4) and the Q2 step (counterfactuals) check whether a fix actually removed the belief and its influence on the answer, rather than only changing the wording.

We illustrate this with a new experiment on the BBQ dataset (Parrish et al., 2022). In BBQ's ambiguous stories the text does not say who did what, so an answer naming a person reflects a stereotype. Before the model answers, we read its belief about who the answer is. A one-turn prompt asking the model to check its reading against the story fixes 61–83% of stereotyped answers across three models (Qwen3-32B, Gemma-4-31B, Qwen3.8-27B) and breaks at most 0.34% of correct ones. Applying the prompt only where the readout flags a specific person avoids the extra step on most conversations (i.e., reduces latency and token cost): for example, on Qwen3.8-27B, it is needed for only 17% of them, fixes 80% of stereotyped answers and breaks 0.13% of correct ones. After the prompt, the readout names the same option as the final answer on 88–98% of items, so it can also confirm the fix without an external judge.

We will add these results as a short paragraph and an appendix.

AuthorsOpenReview ↗

Rebuttal to gY85 (2/6)

W2 & Q2. The headline audit finding (occupation gender/wealth stereotypes) --> confirms what the bias literature already reports. What is the most surprising result here -- the one that changes what the community believes or does?

We agree that occupational gender and wealth stereotypes are well established. Reproducing these patterns was an intentional validation of our framework, rather than our main contribution. The more interesting findings are both methodological and audit-specific, and are more nuanced than simply confirming known stereotypes.

Methodological findings that go beyond prior work:

  1. We thoroughly evaluate different techniques for estimating the beliefs (different probing, lenses, lora, logits) and for performing the intervention (activation steering, LLM-rewriting, injection) and find the best for each step.
  2. As far as we know, we are the first to go beyond simple belief estimation with our step-2, and showed (see point below) why this is important.
  3. In our audit, 64.1% of answers depend on a stated trait even though the readout shows no reportable belief, and belief strength barely predicts influence (rank correlation 0.036; App. Q.6). Audits that read beliefs alone, the basis of most current work, would miss much of the influence that our coupled counterfactuals reveal.
  4. Checking that an intervention hits its target cannot rely on relevant-versus-neutral differences (Section 4.4). A random activation perturbation separates relevant from neutral questions with AUROC 0.92-0.93 without moving the target belief. Our belief thermometer addresses this issue, allowing us to verify interventions and compare their selectivity. Thanks to this we could establish that input-level edits prove more selective than activation steering.
  5. Output-based readouts generalize better than activation probes under strong distribution shift, and the training-free LOGIT matches supervised methods from 14B, so audits need no labelled data and extend to API-only models.

Audit-specific findings:

  1. Models share a monoculture (Section 5, App. Q.6). Models agree with each other on which jobs are female or poor (median rank correlation 0.903 and 0.913) more than they agree with labour statistics (0.832).
  2. Implicit influence extends to presentation style (Section 5, Appendix Q.4): Inferred traits can change how factual information is presented, even when the underlying facts remain unchanged. For questions inviting explanations, attribution reaches 40–85%.
  3. Interventions reveal links between beliefs that correlations miss (App. Q.7). Across occupations, the gender belief is almost uncorrelated with the age and wealth beliefs (rank correlations of -0.001 and -0.040), yet stating age or wealth moves the gender belief in seven and eight of eight models, respectively.

We will better highlight these contributions in the revised manuscript.


W3. The framework's payoff is deferred. Its value depends on community adoption and downstream tooling such as the interface -- neither demonstrated.

We agree that the societal impact of any technological advancement depends on community adoption and downstream tooling, and that neither can be fully established within a single paper (however, we would also argue that this is true for most algorithmic papers). We therefore undertook a systematic study of state-of-the-art methods for both steps, contributing empirical evidence on their strengths and limitations. By integrating the most suitable approaches into a unified framework, we advance methodological understanding while lowering the barrier to adoption and demonstrating their potential for practical transparency applications.

Adoption. We designed the framework for low friction. The best readout is training-free, needs no labelled data, extends to API-only models through SAMPLE, and covers new attributes on the fly by changing the answer options (see our answer on intersectional attributes). The occupation audit serves as a first application across 8 models. It yields findings the reviewer highlights, such as the cross-model monoculture and the effect on presentation style. We release all 59,040 measurements behind the audit (readouts, counterfactual answers and distances) with code to reproduce our results. We will release the full framework code with the final version, so others can apply it to new models, attributes and settings.

Downstream tooling. The same measurements already support verifying mitigations (W1). They also support surfacing assumptions to users through a UI, which we are now evaluating in a user study (W6, Q8).

AuthorsOpenReview ↗

Rebuttal to Reviewer gY85 (3/6)

W4 & Q3. Sem relies on a single external embedder (Qwen3-Embedding-8B). How sensitive are the audit's attribution rates to the embedder choice, given the embedder may share the very stereotypes being measured?

We agree that relying on a single embedder is a limitation worth testing. ΔSem\Delta_{\text{Sem}} needs some measure of semantic similarity, and any learned measure could carry biases. We chose Qwen3-Embedding-8B because it has been shown to correlate with human judgments of semantic similarity (Zhang et al., 2025). Note that the embedder only sees the generated answers, never the prompt or the occupation, so it cannot read the user's profile directly, and any biases that manifest from the embedder are with respect to the semantic closeness of the generated answers, not the user profiles.

To test whether the embedder's own associations drive the results, we repeated the attribution analysis with two measures that contain no learned associations at all, and therefore cannot share the stereotypes being measured: word-sequence similarity and bag-of-words cosine similarity. Ranking the 51 questions by attribution rate, both agree closely with the original ranking obtained with the semantic embedder (Spearman ρ=0.787\rho = 0.787 and 0.7990.799). The pattern across question families is unchanged:

Question familyQwen3 (original)Word-sequenceBoW
Factual0.3860.4190.421
Career0.9320.9840.987
Job seeking0.9310.9800.982
Purchases0.9510.9840.985
Travel0.9590.9900.991
Attribute-dependent0.9500.9820.983

Factual questions remain far less affected than all other families. The word-level measures give slightly higher rates because they register any change of wording, including changes that do not alter the meaning, which is why we use a semantic embedder in the first place. We will add this analysis to the revised paper.


W5 & Q4. Interventions target one concept at a time, but real profiling is intersectional (e.g., wealth × gender). Do the headline claims survive joint interventions? The appendix's co-movement analysis is only preliminary?

Yes, we performed new experiments with joint interventions and the headline claims survive: the monoculture, the stereotypes and the dependence of answers on inferred traits all hold when gender and wealth are intervened on together.

We agree on the importance of intersectional audits. We used single attributes for simplicity, but the framework is not limited to them. Because the readout is training-free, a new attribute, including an intersectional one, is just a new set of answer options, and the intervention is just a line in the user profile. Neither needs data, labels or probe training. To test this, we defined a joint gender × wealth attribute (poor woman, rich woman, poor man, rich man) and reran the audit in the paper's setting, where the user states only their job: 4 models, 60 jobs, 47 questions, 11,280 measurements, each with four coupled counterfactual answers, one per combined profile, on the same random noise.

  1. The joint readout works. When the profile is stated, it names it in 100% of cases, and its gender and wealth totals agree with the separate readouts (r = 0.81 and 0.87). The model adopts both stated values in every measurement, and stating them barely moves the age belief. This is only a validity check: if it failed, there would be no point in proceeding further. 

  2. Models make intersectional assumptions. The monoculture holds for combined profiles (median inter-model Spearman 0.85–0.89). For 72–83% of jobs, the model links the two attributes: its guess about wealth changes its guess about gender, beyond any such link it shows when no job is given. All four models picture a janitor or plumber as a poor man, a broker or surgeon as a rich man, and a nurse or secretary as a poor woman.

  3. Answers still depend on the profile in 92–96% of advice questions, against 39% of factual ones, and questions where gender or budget should matter are driven by gender and wealth respectively.

We will add this analysis to the paper.

Reviewer_pcfsOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a framework for auditing inferred user attributes and their influence on language-model responses. It compares five belief readout methods operating on activations, logits, and sampled text, evaluating predictive accuracy, paraphrase consistency, and generalization under distribution shift. It then compares prompt-based and activation-based interventions, using a belief readout to verify changes in the target attribute. Shared Gumbel sampling noise enables coupled comparisons between original and intervened responses. Experiments across eight models show that training-free output readouts become competitive with supervised methods at larger model scales, and that input-level interventions induce more selective response changes than the evaluated activation-steering method. An occupation-based audit illustrates applications to demographic inference and response adaptation.

Critical strengths

The comparison of belief readouts is systematic and practically useful. The evaluation extends beyond explicit-label recovery to examine paraphrase robustness, distribution shifts, and cue-free inputs. Template ensembles and answer-option permutations address important sources of readout instability. The finding that training-free output readouts become competitive with supervised methods on larger models could substantially simplify auditing. The per-concept results and calibration analyses further clarify the scope of this finding.

Critical weaknesses

Some causal interpretations go beyond what the intervention design establishes. Section 4 carefully defines influence as the effect of an external edit that changes a measured belief. However, parts of the occupation audit describe the original response as causally attributable to an unstated attribute. Adding an explicit profile field may affect the response through the attribute’s direct mention or through collateral changes in other inferred attributes. Similarly, proximity to one counterfactual response in embedding space does not, by itself, establish that the corresponding attribute caused the baseline response.

Questions for authors

What assumptions support interpreting the nearest-counterfactual attribution rule in Appendix Q.3 as evidence about the cause of the original response? Please report attribution rates separately for interventions with stable versus shifted non-target readouts. Repeating the analysis after removing direct attribute mentions from generated responses would also help distinguish response adaptation from explicit discussion of the newly supplied information.

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer pcfs (1/3)

Thanks and Summary

We thank the reviewer for the careful reading and for the constructive suggestions, which have helped us clarify the paper's claims. We are glad the reviewer found the comparison of readouts systematic and practically useful. We have addressed every point raised and run every analysis the reviewer requested, and we hope these resolve the remaining concerns about soundness.

  • Causal language (W1.1, W1.2, and Q1). We now separate what the intervention establishes---that the answer depends on the attribute---from the weaker claim that the original answer reflects one specific value. We state the assumptions behind each and revise the wording in Appendix Q accordingly.

  • Interventions that shift other beliefs (Q2). As requested, we report attribution rates separately for interventions that do and do not move other beliefs, on the full audit. There is no difference once question and attribute composition are accounted for.

  • Direct mentions of the attribute (Q3). As requested, we removed every explicit mention of the attribute from the generated answers and repeated the analysis. The attribution rate is essentially unchanged (78.8% vs. 78.7%), with controls that behave as expected.

We will add both new analyses to Appendix Q. We hope these clarifications and results address the reviewer's concerns, and would be grateful if the reviewer would consider them in their assessment.


W1.1. Some causal interpretations go beyond what the intervention design establishes. Section 4 carefully defines influence as the effect of an external edit that changes a measured belief. However, parts of the occupation audit describe the original response as causally attributable to an unstated attribute.

We would first like to clarify that the occupation audit uses the same framework as Section 4. It uses the same intervention and profile injection checked with the thermometer, which Section 4 selects as the best among those tested (Fig. 3). It also uses the same quantity: the attribution rule compares the influence of Eq. 1 across the stated values, with all answers decoded under the same random noise.

We agree, however, that one sentence in Appendix Q.3 went beyond Eq. 1. It described the attribution rate as how often the original answer "can be causally traced to a specific assumed trait". Eq. 1 supports the claim that the answer depends on the assumed value. It does not identify the implicit belief as the cause of the original answer. We will revise this sentence and the related headings, and we discuss the distinction in our answer to Q1 below.


W1.2. Adding an explicit profile field may affect the response through the attribute's direct mention or through collateral changes in other inferred attributes.

We explicitly test whether profile injection affects only the stated attribute or also shifts other inferred beliefs. Across 59,040 measurements, the stated value is successfully induced in 98.3% of cases. We find that another attribute's inferred belief shifts by more than 10 percentage points in 20.3% of paired measurements. These effects are highly asymmetric: gender interventions are particularly selective (below 1% spillover), whereas age and wealth can influence each other's inferred beliefs. Importantly, we measure and report these shifts rather than assuming perfect isolation. Accordingly, we interpret the results as evidence that the answer depends on the stated attribute value, not as proof that an internal belief variable causally mediates generation. We have clarified this qualification in Appendix Q.7-Q.8.

AuthorsOpenReview ↗

Rebuttal to Reviewer pcfs (2/3)

W1.3 & Q1. Similarly, proximity to one counterfactual response in embedding space does not, by itself, establish that the corresponding attribute caused the baseline response. | What assumptions support interpreting the nearest-counterfactual attribution rule in Appendix Q.3 as evidence about the cause of the original response?

We agree with the reviewer, and we will separate two claims the paper currently runs together.

The first, that the answer depends on the attribute, is established by the intervention itself: we change only the stated value and regenerate with the same random noise, so any difference in the text is caused by that value. It does not rely on proximity, and it behaves as it should: answers to factual questions change far less than answers to advice questions, and questions where gender or budget should matter are driven by gender and wealth, respectively. This claim relies on three assumptions, each of which we check: the model adopts the stated value (verified with the readout for every measurement, Appendix Q.8); the counterfactuals share the original's random noise, so only the edit differs (verified by a control that re-decodes the original answer verbatim); and a difference in embedding larger than the embedder's own noise reflects a real difference in content (Appendix Q.3).

The second, that the baseline answer reflects one specific value, is a weaker claim. It relies on which counterfactual the baseline is closest to, and it needs a further assumption: that an answer produced under an implicit belief ends up closest to the counterfactual for that belief. This is not guaranteed, because counterfactual answers can diverge for reasons unrelated to the value, and the reviewer is right that closeness alone does not prove cause. We will make this distinction explicit. We will use causal language only for dependence, and replace "attributed to" with "closest to". Claims about which value an answer resembles will be limited to averages over many answers, where they can be checked, for example, against labour statistics.


Q2. Please report attribution rates separately for interventions with stable versus shifted non-target readouts.

We ran the analysis, and our conclusion is that spillover to other beliefs does not drive the attribution rate. Answers are attributed equally often whether or not the intervention also moves other beliefs. Spillover therefore changes what an attribution covers, namely the stated attribute together with the beliefs it moved (Appendix Q.8), but not how often an answer is attributed.

Setup. We ran the analysis on the full audit (58,021 valid measurements, 8 models), using the same definitions as Appendix Q. We call an intervention shifted when stating one value instead of the other moves any other attribute's readout by more than 10 points, and stable otherwise. Note that the 20.3% reported in Appendix Q.8 counts (stated, read) attribute pairs. Counted per measurement, 33.6% of interventions are shifted (age: 61.8%, wealth: 38.3%, gender: 0.8%). Attribute-dependent questions have no readout of the other attributes, so the comparison uses the 36 questions shared by all attributes.

Results. The raw attribution rate is higher for shifted interventions (85.8%) than for stable ones (75.2%). This difference is explained by composition. Shifted interventions are mostly age and wealth, and factual questions, which have the lowest attribution rates, shift other beliefs less often. After standardising over attribute, model, and question, the difference is +0.2 points (95% CI

−1.1,+1.8-1.1, +1.8

). Restricted to measurements where the two counterfactuals differ, it is -0.4 points ([-1.4, +1.0]). Within age and within wealth, the standardised differences are +0.8 ([-1.1, +2.8]) and -1.0 ([-2.8, +1.2]) points. Gender has too few shifted interventions (137) for a separate estimate. Confidence intervals come from a two-way bootstrap over occupations and questions. We will add this analysis to Appendix Q.

AuthorsOpenReview ↗

Rebuttal to Reviewer pcfs (3/3)

Q3. Repeating the analysis after removing direct attribute mentions from generated responses would also help distinguish response adaptation from explicit discussion of the newly supplied information.

We followed the recommendation and ran the experiments. The attribution rates reflect adaptation of the response content, not explicit discussion of the attribute. Removing every direct mention of the attribute from the generated answers leaves the attribution rate essentially unchanged (78.8% vs. 78.7%). The mention affects which counterfactual an answer resembles in some cases, but not whether the answer depends on the stated value.

Setup. We removed every explicit restatement of the stated attribute from all answers of a measurement (original and counterfactuals), re-embedded them with the same embedder, and recomputed attribution with the same four steps. Restatements include a lead-in plus a value ("as a woman plumber", "since you're 67 or older"), the profile field itself ("your age"), and an attribute word directly before the occupation ("an older scientist"). They occur in 35.5% of measurements (18,107 of 51,062 on the shared questions). As a check, re-embedding the unredacted answers reproduces the released attribution outcome in 99.95% of measurements.

Results. The attribution rate is 78.8% before and 78.7% after redaction (-0.1 points, 95% CI [-0.3, +0.1]). Among the 18,107 measurements whose answers contained a restatement, it is 97.9% before and 97.6% after (-0.3 points, [-0.9, +0.2]). No attributed measurement became inert, i.e., no answer became identical to its counterfactual once the mention was removed, so no answer differed from its counterfactual only by the mention. A broader redaction that also removes bare attribute words (pronouns, "rich", "older") gives the same result (-0.1 points, [-0.3, +0.1]). Two controls behave as expected. Deleting an equal number of other words that best distinguish the two counterfactuals changes attribution by -0.1 points ([-0.6, +0.3]), and deleting the occupation, which is identical across answers, changes it by +0.5 points ([-0.2, +1.2]). On factual questions, where we report the style effect, the attribution rate is 38.6% both before and after redaction.

Reviewer_pcfsOpenReview ↗

Response to the author's rebuttal

Thank you for the responses. My concerns are addressed. I appreciate it you add analyses stratified by changes in non-target attributes and a robustness check removing direct attribute mentions. They make sense to me. Therefore i will raise my score to 3.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal Summary (1/2)

Dear Reviewers and AC,

We thank all three reviewers for their careful reading and constructive comments. Their reviews helped us state our claims more precisely, add new experiments, and improve the paper overall.

Agreed across all reviewers

  • Clarity. Two reviewers rated the presentation "3: Clear and easy to follow" (pcfs, gY85). The third wrote that the paper has "clear signposting throughout the paper, which makes it easy to follow" (W3tR).
  • Informativeness. All three reviewers chose "I learned something interesting or useful." In their words: the readout comparison is "systematic and practically useful" and "could substantially simplify auditing" (pcfs); the framework is "methodologically careful", the Q1 comparison "is genuinely useful", the monoculture result "is a crisp, citable framing", and the style result "is subtle and important" (gY85); and "I appreciate the distinction between reading out user attributes and measuring their influence on responses" (W3tR).
  • Soundness. All three reviewers chose "I have minor or fixable concerns about the paper's soundness." We addressed every concern with new analyses or revisions, and we hope this rebuttal resolves them.

Main new results

  1. Mitigation (gY85).

    • Concern: the paper measures bias but offers no mitigation or path to one.
    • New result: firstly, the scope of the paper was, and remains, transparency for the user, which is valuable in itself. However, following the reviewer's suggestion, we show how the framework supports mitigation. On BBQ, the readout flags stereotyped beliefs before the model answers, and one extra turn fixes 61% to 83% of stereotyped answers across three models while breaking at most 0.34% of correct ones. Afterwards, the readout agrees with the final answer on 88% to 98% of stories, so it can confirm the fix without an external judge (new App. R).
  2. User study (gY85).

    • Concern: the transparency interface is an unevaluated proof of concept, with no evidence that it helps users.
    • New result: The interface was simply to show how the framework could be integrated rather than a final UI proposal. That said, we embraced again the user suggestion and ran a user study with 14 participants and 224 paired comparisons, participants preferred our panel over the plain interface 63% vs 17% when it showed both the belief and whether it changed the answer, and 68% vs 17% when the assumption did change it. 71% of participants wanted the combined panel (to be added to App. P).
  3. Sycophancy (W3tR).

    • Concern: stating a value in the prompt might change the answer only because the model goes along with what it is told.
    • Clarification: the original answer and all belief readings come from prompts where nothing about the user is stated, so sycophancy cannot affect them. It could only affect the counterfactuals. Every counterfactual states a value, so it would affect them all alike, and the comparison with the original answer would still hold.
    • New result: we also tested this directly. We changed the internal state with ITI, leaving nothing in the text to agree with, and compared the result with answers for a stated value. The two move the answer in the same direction, 70% when ITI stays on topic, while neutral questions and random steering stay at chance (new App. S).
  4. Intersectional profiles (gY85).

    • Concern: interventions target one attribute at a time, while real profiling is intersectional.
    • Clarification: one advantage of our method is that we can easily test intersectional attributes without any training data.
    • New result: to show the flexibility of our method we ran a joint gender × wealth audit (4 models, 60 jobs, 11,280 measurements) that confirms the headline claims. The monoculture holds, answers depend on the combined profile, and models link the two attributes (new App. Q.10).
  5. Explicit mentions (pcfs, W3tR).

    • Concern: answers might differ only because they repeat the stated attribute, not because their content adapts.
    • New result: removing every mention of the attribute from the answers leaves the attribution rate unchanged, 78.8% vs 78.7%, and the controls behave as expected (new App. Q.9).
  6. Spill-over to other beliefs (pcfs).

    • Concern: stating one attribute may also shift other inferred attributes. The reviewer asked for attribution rates split by whether this happens.
    • New result: the rates do not differ, once we account for the mix of questions and attributes (+0.2 points, 95% CI [-1.1, +1.8]; App. Q.5).
AuthorsOpenReview ↗

Rebuttal Summary (2/2)

  1. Robustness (gY85).
    • Concern: three of our results might depend on our choices of setup. These are the gap between input edits and ITI (thermometer and threshold τ\tau), attribution (a single embedder that could share the stereotypes being measured), and the monoculture result (borderline readings).
    • New results:
      • All Section 4 results hold with a different, training-free thermometer, and the gap widens.
      • Over a sweep of τ\tau even the best ITI setting does not match input-level edits (new App. M.4).
      • The attribution pattern stays the same with two similarity measures that use no embedder.
      • The monoculture result stays the same without borderline readings.

Main clarifications

  • Causal claims (pcfs, W3tR).

    • Concern: parts of the audit used causal language beyond what the intervention shows. Being closest to one counterfactual does not prove the attribute caused the original answer, and information a probe can decode is not necessarily used in generation.
    • Clarification: we agreed and made the claim more precise. Specifically, we separate two claims.
      • The first is that the answer depends on the attribute the context conveys. The intervention establishes this.
      • The second is that a given original answer reflects one specific value.
    • Revisions:
      • We now use causal language only for the first claim, and say "closest to" instead of "attributed to" for the second.
      • Section 4 now links our Q2-step to causal theory explicitly. Coupling the counterfactuals through shared Gumbel-Max noise places them on Pearl's counterfactual rung.
      • We check that each edit moves its target, using the thermometer. We check that it acts through no other path, using neutral questions as negative controls together with mention removal.
  • Readout vs generation (W3tR).

    • Concern: the readout might reflect a judgment the model forms only when asked, not one present during generation.
    • Clarification: the readout question is never part of generation, so asking it cannot change the answer.
  • Novelty (gY85).

    • Concern: the headline audit findings confirm known occupational stereotypes.
    • Clarification: reproducing known stereotypes was performed in order to validate the framework. Beyond that, and in addition to the methodological contributions, we show four things:
      • 64.1% of answers depend on a trait the readout does not report, so audits that only read beliefs would miss them.
      • Relevant-vs-neutral separation alone cannot validate an intervention.
      • Models agree with each other more than with labour statistics.
      • Inferred traits change how facts are presented.

All revisions are marked in colour in the updated PDF. We thank the reviewers for helping us improve the paper.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.