- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduce Belief Self-Distillation (BSD), a unified read-write framework to explain how the model internally represents its judgments about users and whether can change model's behavior based on this. It bridges linear and causal probing by learning a compact user rep-resentation that can be both decoded and written back into the model.
Critical strengths
The research question is meaningful, and the method is relatively clear.
Critical weaknesses
- The writing needs improvement. For example, in the Abstract, what's the meaning of "beliefs"? And the motivation is not well described.
- The paper does not empirically compare BSD with closely related trained methods such as LatentQA or prior user-model extraction approaches.
- Alexander Pan, Lijie Chen, Jacob Steinhardt: LatentQA: Teaching LLMs to Decode Activations Into Natural Language. ICLR 2026
- Asma Ghandeharioun, Ann Yuan, Marius Guerard, Emily Reif, Michael A. Lepori, Lucas Dixon: Who's asking? User personas and the mechanics of latent misalignment. NeurIPS 2024
- The paper treats flip rate simply as whether a prediction changes after steering, but for multi-class, changing the prediction doesn't mean changing it to the intended target value.
- Steering the User Intent direction changes refusal behavior, but this doesn't prove that refusal changes because the model's inferred belief about the user has changed. The intervention may directly affect refusal-related representations.
- As shown in Appendix Table 5, BSD is not easier to decode than the raw hidden states, and it can't prove BSD has extract more accurate user beliefs. If using PCA, a supervised linear projection, or train a 128-dimensional bottleneck directly using the teacher’s MCQ attribute labels, would achieve the same or even better results
Questions for authors
- Can the authors compare BSD with a linear-probe-guided steering method or a similarly trained low-rank adapter to prove the contribution of belief self-distillation?
- How do you build the disagreement set, and what's the relationship between it with the test set?
- Does the reported flip rate mean any change in the predicted attribute, or only changes to the intended target value? For multi-class, can the authors report the target-class success rate?
2 comments
We thank the reviewer again for their constructive feedback. As the discussion period draws to a close, we would welcome a brief follow-up on whether our response addresses their concerns. If any questions remain, we would be glad to clarify
We thank the reviewer for the careful reading and specific suggestions. In response, we expanded the comparisons with prior methods and clarified how we define beliefs, measure steering success, and construct the disagreement set.
W2, W5, Q1 -Comparison with prior methods and baselines. Following the reviewer’s suggestion, we added a LatentQA readout comparison on Llama and a rank-matched PCA readout baseline. We also evaluated steering with probe-weight directions, PCA-128, and LDA (Table 1). LatentQA uses the released Llama-3 decoder, so it is reported for Llama only (Table 1):
We agree that BSD is not easier to decode than raw h, which we treat as an upper bound; the contribution is that the same compact vector is also causally writable. BSD outperforms PCA-128, LatentQA and RBP on reading, and steers user beliefs better than all baselines, including linear-probe-guided steering, on every model.
Added baselines for refusal analysis: (Fig. 3c, evaluated on the same StrongREJECT requests):
Neither stating benign intent in the prompt nor LatentQA control lowers refusal; editing the inferred user intent with BSD does.
On a trained low-rank adapter such as LoReFT: trained on our MCQ objective, it would be optimized directly on the answers we use for evaluation, so it could reach near-perfect flip rates without testing the model's own representation. BSD's flip rate instead tests whether the model causally uses a vector extracted from the conversation.