- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes SVM-STEER, an activation steering framework that trains a linear SVM on per-layer hidden activations and extracts steering vectors from the resulting decision boundary. Three vector variants are considered (difference of class means, the hyperplane normal, and difference of support-vector means), together with a layer-selection score (RMSE divided by SVM accuracy) that picks the top 10 intervention layers without brute-force search, and a projection adjustment applied at inference time. For unlabeled settings, pseudo-labels are derived from the model's own logits. The method is evaluated on bias mitigation (gender, racial dialect) and censorship control (evasion and enhancement, measured on JailbreakBench, SorryBench, and XSTest) across five open-weight models in the 7B to 12B range, with a supervised case study on TruthfulQA and ToxiGen in the appendix. Steered models reduce bias scores by roughly 30 to 40 percent and can move refusal rates substantially in either direction, relative to the unsteered model.
Critical strengths
The evaluation is reasonably broad for a steering paper: five model families, two distinct application domains, and an appendix ablation of the projection component. The layer-selection criterion is a sensible idea, since brute-force layer search is a genuine practical annoyance in this literature.
Critical weaknesses
-
There is no comparison to any existing steering method. The paper states explicitly that "our goal is to evaluate the efficacy of SVM-based steering rather than compare it with other approaches" (Sec. 1 and Sec. 3), and every table uses only the unsteered model as baseline. But the motivation of the paper is that supervised SVM-derived directions should improve on unsupervised mean-difference constructions. Without a head-to-head comparison against CAA-style difference-in-means or the unsupervised method of Cyberey and Evans (2025) under identical data and evaluation, the central claim is untested, and what remains ("steering can change model behavior") is already well established. This alone places the contribution below the acceptance bar for me.
-
The experimental evidence is too fragile to support the conclusions even on its own terms. All numbers are single-run point estimates with one fixed seed (Appendix D), with no variance reported. There is no evaluation of general capability after intervention (no MMLU or similar), so the cost of steering is unknown. And the paper's own Appendix F shows that the projection adjustment, presented as part of the method, is actively harmful for some model-direction pairs (for Llama with C1, removing it changes JailbreakBench refusal from 0.58 to 0.008), meaning a core component does not behave consistently across the very models tested. The steering strength lambda also requires per-model manual tuning (OLMo becomes numerically unstable beyond lambda of 10), with no selection procedure given.
Questions for authors
-
Can you provide a quantitative comparison with at least one unsupervised steering baseline, e.g. difference-in-means/CAA or Cyberey and Evans (2025), using the same data, layers, and benchmarks? Absent this, how should a reader conclude that the SVM construction is responsible for the reported gains?
-
Can you report variance across random seeds for at least one representative setting, and any measurement of general capability (e.g. MMLU) under the lambda values used in Tables 1 and 2?
-
In Sec. 3.1 the text says C1 "leads to a slight increase in disparity (-6.1 percent relative change)" for Qwen1.5 on race, but Table 1 shows disparity decreasing from 0.5154 to 0.4371 relative to the unsteered baseline. Please confirm the -6.1 percent is relative to the N direction and revise the text, which currently reads as an increase over baseline.