- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper evaluates 12 linear and nonlinear probe architectures across 12 truth and safety datasets, two models (Gemma-3-4B-IT, Qwen2.5-7B-Instruct), four layer depths, and two prompt framings under the Predictability–Computability–Stability (PCS) framework. The authors introduce hierarchical random-slopes linear probes to model shared versus dataset-specific directions and compare discriminative performance against activation steering efficacy.
Critical strengths
- The experimental grid covers 12 datasets (including two new human-validated datasets,
AnatomyandVDS) with grouped cross-validation that prevents data leakage across paired prompts. - Applying multilevel random-slopes logistic regression () provides a clear mathematical formulation to separate shared concept directions () from dataset-specific adjustments ().
- Applying Ledoit-Wolf shrinkage to the target covariance matrix in the Mahalanobis cosine similarity corrects the circularity between unregularized geometric alignment and prediction correlation in prior work.
Critical weaknesses
- Sec. 3.6 concludes that steering aligns with maximal activation variance () rather than discriminative power. However, within each target dataset, the cosine similarity between and the difference-in-means direction () is . Because the two directions are collinear in these curated datasets, the ablation cannot distinguish whether steering relies on variance () or class separation ().
- Random-slopes probes require known dataset identity to apply the adjustment . In out-of-distribution settings where is unknown (VDS in Figure 5), weighting adjustments by unsupervised activation similarity fails to improve over the shared direction because label-free geometry does not predict alignment across domains.
- Evaluations are limited to two small instruction-tuned models (4B and 7B), and steering is measured only by next-token log-odds shifts () rather than multi-token generation quality or refusal rates.
Questions for authors
- In Appendix B.1, how is the covariance of the random effects estimated in practice—via penalized maximum likelihood with fixed penalty ratios ( or ) or empirical Bayes / REML?
- Why do Mixture-of-Experts (MoE) probes underperform random-slopes probes when dataset identity is linearly decodable from the activation space ?
- Does the cross-concept transfer between truth and safety (Figure A4) persist in base (non-instruction-tuned) models, or is it induced by post-training alignment?
We sincerely thank the reviewer for the detailed methodological comments and constructive suggestions. Our contributions and novelty relative to prior work are detailed in the first comment of our response to Reviewer QAF7.
W1. Diff-in-means versus PC1. We agree that near-collinearity between PC1 and the difference-in-means direction prevents the existing steering experiments from distinguishing maximal variance from class separation as the relevant causal factor, and thank the reviewer for pointing this out. We have since done an analysis using target-absent experiments to help distinguish between the two; the experimental results and details are in the second comment of our response to Reviewer QAF7, but we summarize them below.
When the held-out data corpus determines the direction vectors, the collinearity between PC1 and diff-in-means greatly decreases (cosine similarity between 0.37-0.94). Diff-in-means has significantly higher steering effect than PC1, and PC1 steers as or less effectively than random vectors on Gemma once diff-in-means is projected out, and far less effectively than the diff-in-means on Qwen. This provides more clear evidence other directions such as PC1 appear to steer proportionally to their alignment with the diff-in-means; diff-in-means is causally implicated.
W2. Known dataset identity. Known dataset identity is an explicit assumption of the random-slopes model’s descriptive application, since the model jointly represents shared predictive structure and source-specific variation through partial pooling. Its descriptive contribution can therefore be evaluated independently of the ability to select adjustments for an entirely new domain without labels. For an unknown dataset, the implementation uses the shared component unless an adjustment is supplied separately.
W3. Scale and generation. We agree that the architecture rankings are supported for the two evaluated models and single-turn settings; their persistence at larger scales or multi-turn prompts remains unestablished. Our study uses model sizes and single-turn responses consistent with the majority of prior work we built upon.
We have also re-run our steering experiments to additionally capture the first 50 tokens of the model’s output, and used LLM-as-a-judge (Claude Opus) to judge whether the response coherently reflects the label being steered towards. Full results and experimental details are in Appendix secion C.6, but to summarize: we find that judged free-responses for Qwen show significant response to steering, whereas Gemma shows weaker effects. We find congruent relationships between the effect of L2 regularized probes and diff-in-means to the next-token probability experiment; when L2 regularized directions are projected onto the top PC (almost perfectly reflecting the diff-in-means), the steering effect is far stronger than when projecting onto the top 50 components. We have modified the paper's conclusions about steering to reflect this new result.
Q1. Estimation. We fit the random-slopes probe using Bayesian inference rather than restricted maximum likelihood due to the binary objective and limited number of groups. We have updated the corresponding section in the Appendix with the implementation details.
Q2. MoE probes. Linear decodability of dataset identity does not imply that a MoE router trained to predict truth or safety labels will recover the dataset partition. Our MoE router is trained jointly with the experts using the prediction objective, without supervision on dataset identity or a constraint assigning one expert to each dataset. In assessing how prompts are routed, we found that individual experts handle prompts from multiple sources, showing that the unconstrained routing does not impose the grouping used by the random-slopes model; it does not indicate that source information was unavailable in the activations. In comparing the random-slopes to MoE, we are able to distinguish the benefit of explicitly encoding dataset identity in hierarchical probe performance across known sources.
Q3. Base models. We have run a new experiment to evaluate how cross-concept transfer between truth and safety changes on the base models corresponding to the instruction-tuned Qwen and Gemma models. We show results below for diff-in-means directions trained on the Geometry of Truth dataset (GoT, factual) and XSTest (safety) and applied to one another, using the true/false prompts as input at the 88% depth (base / instruction-tuned AUC using 5-fold CV). We observe that the degree of cross-concept transfer is greater in the instruction-tuned model for Gemma but still quite significant in base models, particularly for Qwen, where the two are comparable.