acceptodds
Under review as a conference paper at ICLR 2027

User Model Extraction via Belief Self-Distillation

45 views

Abstract

Large language models (LLMs) implicitly infer attributes of their users and adapt their behavior accordingly, yet these beliefs remain difficult to inspect and causally manipulate. We introduce Belief Self-Distillation (BSD), a unified read-write framework that bridges linear and causal probing by learning a compact user representation that can be both decoded and written back into the model. The frozen LLM acts as its own teacher, distilling beliefs from natural conversations without external annotations. Unlike conventional probing, BSD isolates not only information present in activations, but a state whose causal role can be directly tested. Across multiple model families, BSD faithfully recovers user beliefs and enables substantially stronger interventions than matched hidden-state steering. Crucially, we find that refusal depends not only on the request, but on the model's inferred user intent: changing this belief alters refusal while holding the request fixed. We further uncover a striking cross-model regularity: independently trained LLMs converge on a shared geometry for representing their users. Together, these results reveal implicit user models as readable and causally writable internal states with direct implications for AI safety, shaping how models condition safety decisions on whom they believe they are interacting with.

Primary area:
interpretability and explainable AI
Keywords:
Implicit user model, LLMs, Belief Extraction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aEGrOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduce Belief Self-Distillation (BSD), a unified read-write framework to explain how the model internally represents its judgments about users and whether can change model's behavior based on this. It bridges linear and causal probing by learning a compact user rep-resentation that can be both decoded and written back into the model.

Critical strengths

The research question is meaningful, and the method is relatively clear.

Critical weaknesses
  1. The writing needs improvement. For example, in the Abstract, what's the meaning of "beliefs"? And the motivation is not well described.
  2. The paper does not empirically compare BSD with closely related trained methods such as LatentQA or prior user-model extraction approaches.
    1. Alexander Pan, Lijie Chen, Jacob Steinhardt: LatentQA: Teaching LLMs to Decode Activations Into Natural Language. ICLR 2026
    2. Asma Ghandeharioun, Ann Yuan, Marius Guerard, Emily Reif, Michael A. Lepori, Lucas Dixon: Who's asking? User personas and the mechanics of latent misalignment. NeurIPS 2024
  3. The paper treats flip rate simply as whether a prediction changes after steering, but for multi-class, changing the prediction doesn't mean changing it to the intended target value.
  4. Steering the User Intent direction changes refusal behavior, but this doesn't prove that refusal changes because the model's inferred belief about the user has changed. The intervention may directly affect refusal-related representations.
  5. As shown in Appendix Table 5, BSD is not easier to decode than the raw hidden states, and it can't prove BSD has extract more accurate user beliefs. If using PCA, a supervised linear projection, or train a 128-dimensional bottleneck directly using the teacher’s MCQ attribute labels, would achieve the same or even better results
Questions for authors
  1. Can the authors compare BSD with a linear-probe-guided steering method or a similarly trained low-rank adapter to prove the contribution of belief self-distillation?
  2. How do you build the disagreement set, and what's the relationship between it with the test set?
  3. Does the reported flip rate mean any change in the predicted attribute, or only changes to the intended target value? For multi-class, can the authors report the target-class success rate?
2 comments

We thank the reviewer for the careful reading and specific suggestions. In response, we expanded the comparisons with prior methods and clarified how we define beliefs, measure steering success, and construct the disagreement set.


  1. W1 - Writing and the meaning of "beliefs". We agree that the term needed a clearer definition. A belief is the model’s probability distribution over the predefined values of a user attribute, elicited through multiple-choice questions (Eq. 1). These judgments may differ from the user’s actual attributes. We revised the Abstract and Introduction to clarify why we study them.

  1. W2, W5, Q1 -Comparison with prior methods and baselines. Following the reviewer’s suggestion, we added a LatentQA readout comparison on Llama and a rank-matched PCA readout baseline. We also evaluated steering with probe-weight directions, PCA-128, and LDA (Table 1). LatentQA uses the released Llama-3 decoder, so it is reported for Llama only (Table 1):

    ModelRead F1: BSDPCA-128Raw hRBPLatentQAWrite flip: BSDProbe-weightCAAPCA-128LDA
    Llama-3.1-8B0.670.610.700.430.430.820.380.230.230.09
    Qwen3-8B0.680.650.720.39n/a0.740.660.600.580.41
    OLMo-3-7B0.660.610.700.40n/a0.590.370.320.310.21

    We agree that BSD is not easier to decode than raw h, which we treat as an upper bound; the contribution is that the same compact vector is also causally writable. BSD outperforms PCA-128, LatentQA and RBP on reading, and steers user beliefs better than all baselines, including linear-probe-guided steering, on every model.

    Added baselines for refusal analysis: (Fig. 3c, evaluated on the same StrongREJECT requests):

    BasePersonaLatentQARandomGenderCAABSD
    Refusal (%) ↓98.198.7100.076.989.182.462.6
    SR score ↑0.010.010.000.160.060.120.24

    Neither stating benign intent in the prompt nor LatentQA control lowers refusal; editing the inferred user intent with BSD does.

    On a trained low-rank adapter such as LoReFT: trained on our MCQ objective, it would be optimized directly on the answers we use for evaluation, so it could reach near-perfect flip rates without testing the model's own representation. BSD's flip rate instead tests whether the model causally uses a vector extracted from the conversation.


  1. W3, Q3 - Flip rate. We thank the reviewer for flagging the ambiguity. A trial counts as a success when the post-intervention argmax equals the target class. The reported flip rate is therefore already the target-class success rate, including for multiclass attributes. We clarified this definition in §4.1.

  1. W4 - Does refusal change through the user belief? We added two analyses. First, the UserIntent direction written by BSD is orthogonal to the known refusal and harmfulness directions (Arditi et al., 2024; Zhao et al., 2025; cosine −0.003 and +0.008), while the CAA direction in raw activations overlaps with both (−0.38 and −0.25). Second, the steering also changes the model's own stated belief on the MCQ probes: P(benign) rises from 0.18 to 0.39 as refusal drops.

  1. Q2 - Disagreement set. The disagreement set consists of (test conversation, attribute) pairs where the two models' probes predict different classes; the 25k–27k pairs are all drawn from the ~7k test conversations. The Procrustes map is fit only on class centroids from the training conversations. We clarified this in App. E.

We thank the reviewer again for their constructive feedback. As the discussion period draws to a close, we would welcome a brief follow-up on whether our response addresses their concerns. If any questions remain, we would be glad to clarify

Reviewer_HS23OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(−1)
Read review
Paper summary

This paper introduces a method, belief self-distillation (BSD) that extracts a low-dimensional representation of a user's attributes. This representation can be used to read user attributes as inferred by a model, as well as write beliefs about user attributes into a model. This method can also be used to steer models away from refusing requests. Finally, this method can be used to understand the representational geometry of user attributes.

Critical strengths

This paper provides a clear example of a trained causal adapter (the write matrix, B) improving steering efficacy. Though this has already been demonstrated in a different form in REFT [1], it is useful to see a different training objective (mimicking the impact of attributes inferred from conversation on MCQs) can also provide similar causal benefits.

1: Wu, Z., Arora, A., Wang, Z., Geiger, A., Jurafsky, D., Manning, C. D., & Potts, C. (2024). Reft: Representation finetuning for language models. Advances in Neural Information Processing Systems, 37, 63908-63962.

Critical weaknesses
  1. The work does not deliver on its central claims, nor interpretations of those claims. The work proposes to identify a "low dimensional" subspace encoding user attributes, yet they identify a 128D subspace for only 44 distinct classes spread over 13 attributes. In the worst case, representing each attribute class requires only a 31 dimensional subspace (44 classes - 13 different attribute types). More to the point, training linear probes over a subspace of this dimensionality is equivalent to training a linear probe on the original residual stream, insofar as the down projection, A, does not discard relevant dimensions. Identifying a larger subspace that does not discard these directions is not necessarily a contribution, especially if the point is to eventually decode these values anyway. A more specific contribution might be to form a subspace from the weights of trained linear probes on the original residual stream, and then see whether edits in this more theoretically-motivated subspace can systematically change MCQ answers, or whether conversations move representations in this subspace along some sensible trajectory.
  2. Relatedly, the work compares CAA in residual stream space vs. the discovered belief subspace. However, the core difference between these two is not a more pure representation of user attributes, but the existence of causally-trained write operation (the B matrix) in BSD. To make any claims about the belief subspace, one should compare against an ablation that effectively just maintains causally trained write operation, such as REFT/LoREFT. One low-lift control experiment would be to train linear probes for each attribute on the original residual steam, then optimize the B matrix on a vector of these probe logits. If this works just as well as the BSD approach, then the extra dimensions in the subspace are not doing any work.
  3. The refusal result is already well established in the literature [1]. The additional point that the single refusal direction is related to user persona is also directly implied by the combination of [1] and [2].
  4. The geometry result is potentially interesting, but lacks an important baseline. It is quite possible that the representational geometry is defined by the MCQs, rather than some more mysterious latent user profile. Performing CKA/RSA with a hypothesized matrix that considers the differences between MCQ answers given each conversation would deflate the objection that the representational geometry is determined purely by the training objective, rather than revealing information about the model's encoding of user attributes.

[1] Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., & Nanda, N. (2024). Refusal in language models is mediated by a single direction. Advances in Neural Information Processing Systems, 37, 136037-136083. [2] Ghandeharioun, A., Yuan, A., Guerard, M., Reif, E., Lepori, M. A., & Dixon, L. (2024). Who's asking? User personas and the mechanics of latent misalignment. Advances in Neural Information Processing Systems, 37, 125967-126003.

Questions for authors

No questions. See Critical Weaknesses for concrete suggestions on how to improve the work.

2 comments

We thank the reviewer for the careful reading and concrete questions. In response, we expanded the prior-method and steering comparisons and clarified the meaning of beliefs, the steering metric, and the construction of the disagreement set.


  1. W1 - minimal rank and probe-weight subspace. We agree that the raw hidden state hh provides a reference for how much attribute information is available to linear probes. BSD retains most of the measured readout performance in a 128-dimensional vector, compared with 4,096 dimensions for hh. The same vector supports intervention. We use compact to describe this 32× compression. Our evaluation uses a fixed rank of 128, which limits what we can conclude about the minimum rank needed to preserve both readout and steering performance. Following the reviewer’s suggestion, we added steering directions formed from differences between the weights of linear probes fitted on hh (Table 1b).

    ModelBSDProbe-weightCAAPCA-128LDARandom
    Llama-3.1-8B0.820.380.230.230.090.19
    Qwen3-8B0.740.660.600.580.410.40
    OLMo-3-7B0.590.370.320.310.210.17

    BSD achieved higher mean target-class success on all three models, with paired Wilcoxon p<10−4p < 10^{-4} on Llama and OLMo.

    Takeaway: Probe-weight steering, which directly tests the probe-derived subspace the reviewer suggests, is the strongest baseline -- yet BSD still outperforms it. This suggests that the learned 128-d subspace has functional value beyond decodability, even if the minimal rank needed remains open.


  1. W2 - BSD vs. ReFT/LoReFT. We agree with the reviewer that learning the write map BB matters. Holding the BSD representation fixed, the learned BB steers more effectively than a random write map (Table 2).
    1. Added 128-d PCA: In an additional experiment, we applied CAA to a 128-dimensional PCA projection of the raw hidden state hh, matching BSD’s compression ratio. PCA-128 steering performed similarly to CAA directly on hh (Table 1b). This comparison shows that compression alone did not yield a steering gain in our evaluation.
    2. LoReFT comparison: We clarified the relationship to LoReFT in §2. LoReFT learns low-rank activation edits under a task objective. BSD jointly trains a read map that extracts a vector from a conversation and a write map that injects it into a frozen student to recover the teacher’s conversation-conditioned attribute distributions -- BSD is never trained to steer. Steering directions are computed afterwards from the extracted vectors, so steering success tests whether this extracted representation is causally used. A LoReFT edit trained on our MCQ objective is optimized directly on the answers used to evaluate steering, so LoReFT' success would not tell us whether a representation extracted from the conversation is causally used.

  1. W3 - Relation to Arditi et al. and Ghandeharioun et al. We thank the reviewer for pointing out that prior work established refusal steering and persona-sensitive safety behavior. In response, we reframed refusal as an application of BSD and added two comparisons that connect our results directly to this prior work.
  • Direction overlap (Fig. 3d). We measured the cosine similarity of the BSD UserIntent steering direction with the refusal direction of Arditi et al. (2024) and the harmfulness direction of Zhao et al. (2025). The similarities were close to zero for BSD (−0.003-0.003 and +0.008+0.008), compared with −0.38-0.38 and −0.25-0.25 for CAA on the raw hidden state hh.

  • Persona prompting (Fig. 3c). Inspired by Ghandeharioun et al. (2024), we tested ten benign-intent prefixes on the same StrongREJECT requests. Refusal was 98.7% with the tested prefixes, 98.1% at baseline, and 62.6% under BSD steering.

    These additions show that the BSD intervention changes refusal on fixed requests while having little overlap with the two measured safety directions.


  1. W4 - Geometry vs. MCQ answers. We thank the reviewer for suggesting this comparison. We added the requested MCQ-answer geometry baseline to Table 3. For each model pair, we concatenate the MCQ answer distributions for all 13 attributes and compute cross-model CKA on those distributions. We compare this baseline with CKA on the BSD user vectors:

    PairCKA (user vectors)CKA (MCQ answers)Null
    Qwen -> Llama0.780.680.002
    OLMo -> Llama0.720.600.002
    Qwen -> OLMo0.720.570.002

    The alignment is not inherited from the training objective alone: the models’ own MCQ answers align less across models than their user vectors do on every pair.

Thank you for your thorough response and extra work running these baselines. These results significantly improve my assessment of the novelty and value of the work. I am raising my score from a 1 to a 3 as a reflection of these new results, and hope that the authors prominently include and discuss these baselines in the main text.

Reviewer_qLxhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces Belief Self-Distillation (BSD), which compresses a frozen LLM’s conversation representations into a 128-dimensional vector using the model’s own elicited user-attribute predictions as supervision. A jointly trained write map allows this vector to be injected back into the model. Across three models, BSD preserves most attribute-probing performance and improves steering over raw hidden-state interventions. The paper further argues that inferred user intent influences refusal and that different models share a compatible geometry of user representations.

Critical strengths
  • The overall idea of distilling and learning a compact representation that supports both attribute readout and intervention is a useful contribution. The frozen-model setup requires neither external user labels nor a separately trained language-model decoder.
  • The compression retains most of the measured attribute information. Across three models, macro-F1 remains within four percentage points of the raw hidden-state probes, while steering improves substantially for example, from 21% to 78% on Llama-3.1-8B.
  • The controls address several plausible explanations for the steering gains. Matching intervention magnitudes and replacing the learned write map with a random one help show that the learned direction and write map matter.
Critical weaknesses
  • The novelty is limited and needs more precise positioning. Prior work already extracts user representations and shows that persona steering affects refusal. BSD’s distinctive contribution appears to be the compact, jointly learned read–write interface. The paper should distinguish this methodological contribution from the broader finding that perceived user attributes influence safety behavior.
  • I think it would be great to see the practical benefit of distillation. A comparison with simpler persona-based prompting would help establish the added value of BSD for behavioral control. Which is not clear.
  • The refusal experiment does not establish that the effect operates through beliefs about the user. Section 4.3 shows that steering the UserIntent direction reduces refusal from 98% to 62%, but this direction could also encode request harmfulness or directly affect refusal. The random and gender controls rule out some generic perturbation effects, but not these alternatives. Moreover, four safety-attribute directions have a mean cosine similarity of 0.91, making the attribution specifically to user intent difficult.
  • The cross-model transfer result is narrower than the claim suggests. The Appendix measures whether a target-model probe predicts the source belief from a mapped vector; it does not inject that vector into the target LLM and measure its answers or behavior. The reported 53–54% transfer therefore demonstrates agreement between mapped representations and probes, rather than the target LLM adopting another model’s belief.
  • The explicit-versus-implicit comparison does not control for training diversity. The explicit dataset uses four phrasings per attribute value over 100 Alpaca prompts, whereas the implicit probes train on a much broader conversational corpus. The transfer gap in Table 2 could reflect this distribution difference. It does not, on its own, establish that explicit cues and natural conversations produce fundamentally different user representations.
Questions for authors
  1. What evidence separates user-belief mediation from direct refusal steering? For the same request, does changing user context move the BSD representation and refusal behavior consistently with the proposed intervention?
  2. Does the explicit-versus-implicit transfer gap persist when training size and diversity are matched?
  3. How does BSD compare with persona-based prompting on the same requests and target user attributes?
  4. Does cross-model transfer change the target LLM’s own attribute answers or behavior after injection? Were the conversations used to fit the Procrustes map excluded from evaluation?
2 comments

We thank the reviewer for the thoughtful feedback. The review helped us clarify BSD’s methodological contribution and state more precisely what our experiments establish. In response, we added baselines and analyses and revised the manuscript.


  1. W1 - Novelty. We agree with the reviewer that prior work established user-model extraction and persona-sensitive refusal.

    1. Adjusted writing: To address this concern, we revised parts of the paper to highlight the main novelty of the proposed approach: to our knowledge, BSD is the first method to unify linear readout and causal intervention on inferred user attributes in a single compact representation trained through its write operation. The revised Abstract now leads with BSD's methodological contribution: it "unifies linear and causal probing in a single compact user representation." The revised Figure 1 makes the distinction concrete by showing how the same vector is read with linear probes, and edited and written back for causal tests.

    2. New baselines: To support this contribution, we added training-free and prior-work baselines on both sides of the interface (Table 1): | Model | Read F1: BSD | PCA-128 | Raw h | RBP | LatentQA | Write flip: BSD | Probe-weight | CAA | PCA-128 | LDA | |---|---|---|---|---|---|---|---|---|---|---| | Llama-3.1-8B | 0.67 | 0.61 | 0.70 | 0.43 | 0.43 | 0.82 | 0.38 | 0.23 | 0.23 | 0.09 | | Qwen3-8B | 0.68 | 0.65 | 0.72 | 0.39 | n/a | 0.74 | 0.66 | 0.60 | 0.58 | 0.41 | | OLMo-3-7B | 0.66 | 0.61 | 0.70 | 0.40 | n/a | 0.59 | 0.37 | 0.32 | 0.31 | 0.21 |

BSD retains most of the decodable information at 32× compression and steers user beliefs better than all baselines on average on every model.


  1. W2, Q3 - Persona prompting. Following the reviewer’s suggestion, we tested ten benign-intent prefixes inspired by Ghandeharioun et al. (2024) on the same StrongREJECT requests. We also added LatentQA control and raw-activation CAA (Fig. 3c):
BasePersonaLatentQARandomGenderCAABSD
Refusal (%) ↓98.198.7100.076.989.182.462.6
SR score ↑0.010.010.000.160.060.120.24

BSD produced the largest change in refusal in this comparison. Refusal stayed near baseline with the tested persona prefixes and the evaluated LatentQA control.


  1. W3, Q1 - Does refusal change through the user belief? We thank the reviewer for distinguishing the effect of the intervention from the pathway through which it acts. We added two analyses (Fig. 3d and accompanying text).

    1. Overlap with measured safety directions. The BSD UserIntent direction has cosine similarities of −0.003 and +0.008 with the measured refusal and harmfulness directions, respectively. The corresponding similarities for raw-activation CAA are −0.38 and −0.25. Change in elicited intent. Under the same BSD intervention, the model’s elicited probability of benign intent rises from 0.18 to 0.39 as refusal falls.

    2. We use UserIntent as a representative direction for the four highly aligned safety attributes (mean cosine similarity 0.91). Whether the shift in elicited intent mediates the change in refusal remains open. Varying user context while holding the request fixed would provide a complementary test of whether both the BSD representation and refusal shift.


  1. W4, Q4 - Cross-model transfer. We thank the reviewer for pointing out the distinction between probe agreement and behavioral transfer. We revised the manuscript to clarify that the reported 53–54% measures how often the target model’s probe predicts the source class from the mapped vector. Whether injecting that vector changes the target model’s own answers or behavior remains untested. We fit the Procrustes map on class centroids from training conversations and evaluate probe agreement exclusively on held-out test conversations (Appendix E).

  1. W5, Q2 - Explicit vs. implicit. Following the reviewer’s suggestion, we retrained the natural-conversation probes on subsets matched to the explicit-cue training set by total size or by examples per class (Appendix C, Table 9). The transfer asymmetry persists under both matching schemes. Probes on BSD vectors trained on natural conversations transfer to explicit cues at 0.49–0.70 macro-F1, compared with 0.39–0.43 in the reverse direction. The original difference in training set size therefore cannot by itself explain the gap.

We thank the reviewer again for their thoughtful feedback. With less than two days left in the discussion period, we would appreciate a brief reply on whether our response addresses their concerns. We would be happy to clarify any remaining points

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We thank all three reviewers for their careful reading. We are glad that the reviewers found the research question meaningful (aEGr), saw a compact representation supporting both readout and intervention as a useful contribution that needs neither external user labels nor a trained decoder (qLxh), and recognized the learned write map as a clear example of a causally trained adapter improving steering, with controls that rule out magnitude and random-map explanations (qLxh, HS23).


These points are the paper’s core: BSD is not a claim that a 128-d vector decodes better than the residual stream. The contribution is a single interface in which the same vector is read and written, trained on the model’s own beliefs inferred from natural conversations. The read results establish that little is lost. The write results establish that the vector is causal. We then use the interface to analyze refusal and cross-model geometry.


Revision summary: The revision (changes in green) restructures the paper around this claim and adds the comparisons the reviewers asked for:

  1. New read and write baselines (aEGr, HS23). At equal dimensionality, BSD outperforms PCA-128 on readout by 3-6 points. Flip rate counts only flips to the target class. BSD reaches 82/74/59% on Llama/Qwen/OLMo, above CAA, probe-weight, LDA, and PCA-128 steering.
  2. Added comparison with prior user-modeling methods (aEGr, qLxh). LatentQA and explicit-cue probes (RBP) reach at most 43% on naturally inferred beliefs, against 67% for BSD on Llama. The explicit-versus-natural asymmetry persists when training size is matched.
  3. Refusal changes with the inferred user belief (all reviewers). Editing inferred intent lowers refusal from 98% to 63% and raises the model’s own probability on MCQ that the user is benign from 0.18 to 0.39. The BSD direction is orthogonal to the known refusal and harmfulness directions, whereas CAA on raw h overlaps with both. Stating benign intent in the prompt (99%) or steering through LatentQA (100%) leaves refusal unchanged. The effect is therefore not implied by prior refusal-direction or persona-prompting results.
  4. Geometry (HS23, qLxh). User vectors align more strongly across models (CKA 0.72–0.78) than the models’ MCQ answers do (0.57–0.68), so the alignment is not inherited from the training objective. We have narrowed the transfer claim to what is measured: the target model’s probe decodes the source’s belief.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.