- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes K-Shield, an activation-sterring method that partitions benign activations into local regions, constructs a region-specific protected subspace and steering map, and routes each input to the nearest protected subspace. The paper is technically polished and reports strong empirical results. However, the central claim—an improved safety–utility trade-off—is not yet established convincingly
Critical strengths
-
The motivation for local rather than global preservation constraints is intuitive and clearly presented.
-
The paper is well written and generally easy to follow.
Critical weaknesses
-
Null-space projection is already established in representation erasure, continual learning, model editing, refusal steering, and increasingly LLL unlearning. Thus, using a null-space constraint alone is not novel. There are already many papers, such as AlphaSteer [r1], Leace [r2], AlphaEdit [r3], that the paper should compared with.
-
The OT component appears to be a standard entropy-regularized balanced assignment with subspace-reconstruction cost, rather than a novel contribution to OT.
-
Tables 1 and 2 compare one configuration of each method. K-Shield is tuned separately for each model, while the baselines are reproduced using their released settings. Since defense success and utility depend strongly on steering strength and other hyperparameters, these isolated points do not establish that K-Shield has a better trade-off.
-
Near-perfect DSR on static prompts is insufficient evidence of robustness. At least one or two strong attacks should be regenerated with the defense in the loop, with comparable attack budgets across methods.
-
The main tables provide no confidence intervals or significance tests. Several utility improvements are extremely small—for example 0.5–2 percentage points—while MATH and GSM8K contain only 100 questions each. Such differences may be sampling noise.
[r1]. Sheng, Leheng, et al. "Alphasteer: Learning refusal steering with principled null-space constraint." International Conference on Learning Representations. Vol. 2026. 2026.
[r2]. Belrose, Nora, et al. "Leace: Perfect linear concept erasure in closed form." Advances in Neural Information Processing Systems 36 (2023): 66044-66063.
[r3]. Fang, Junfeng, et al. "Alphaedit: Null-space constrained knowledge editing for language models." International Conference on Learning Representations. Vol. 2025. 2025.
Questions for authors
See weaknesses.
5 comments
Q4. Adaptive attack robustness
We have added two adaptive white-box GCG evaluations on Llama-3.1-8B-Instruct, comparing the unsteered model, AlphaSteer, and K-Shield. Adversarial suffixes were optimized separately for each configuration, with the defense active and gradients computed through its steering transformation. Each attack used matched settings across configurations: 200 optimization steps, search width 512, top-k 256, and seed 0.
Starting from the 100 AdvBench behaviors used in Table 1, a preliminary GCG run successfully attacked 53 behaviors on the unsteered model. We evaluated the defenses on these 53 behaviors to avoid crediting them for failures of the attack against the original model. Attack Success Rate (ASR) is therefore conditional on this attackable subset. Using the same judge as Table 1, a behavior counted as successfully attacked if any checkpoint at steps 50, 100, 150, or 200 produced a jailbreak.
Explicitly targeting the steering mechanism. In addition to standard GCG, which optimizes the target-response likelihood through the defended model, we evaluated a defense-aware variant that explicitly penalizes the relative steering magnitude:
Here, is the steering intervention and is the hidden state at the routed token in layer . This objective encourages suffixes that elicit the target response while suppressing the defense’s intervention. A pilot on 10 of the 53 behaviors tested . We selected , the strongest setting against K-Shield and joint-strongest against AlphaSteer, and applied it to both defenses.
| Attack | Unsteered ASR | AlphaSteer ASR | K-Shield ASR | Paired exact McNemar p |
|---|---|---|---|---|
| Standard GCG (β = 0) | 98.1% (52/53) | 32.1% (17/53) | 13.2% (7/53) | ≈ 0.013 |
| Defense-aware GCG (β = 30) | — | 64.2% (34/53) | 11.3% (6/53) | ≈ 7 × 10⁻⁹ |
| Either attack succeeds | — | 73.6% (39/53) | 20.8% (11/53) | ≈ 6 × 10⁻⁸ |
Explicitly targeting steering doubles AlphaSteer’s ASR, whereas K-Shield’s ASR does not increase. Taking the per-behavior worst case over both attacks, K-Shield reduces ASR by 52.8 percentage points relative to AlphaSteer. This aggregate gives each defense the same combined budget of two 200-step attacks.
These results support improved robustness under the evaluated matched-budget adaptive attacks, including an objective explicitly targeting steering.
Q5. Statistical reliability and utility evaluation coverage
We would like to clarify that our main objective is to improve DSR while retaining utility close to the unsteered model, rather than to achieve superior utility. However, we agree that differences of 0.5–2 percentage points should not be interpreted as meaningful utility improvements without statistical evidence, and that 100 questions per benchmark provide limited precision. To address these concerns, we have added a paired statistical analysis of K-Shield against AlphaSteer, our main baseline, and expanded evaluation to 500 questions each on MATH-500 and GSM8K.
Statistical reliability. We compare the methods on the same prompts and report 95% confidence intervals for their score differences. We use Newcombe intervals and exact McNemar tests for individual benchmarks. For average DSR, we use a paired bootstrap within attack families (10,000 resamples) for confidence intervals and a stratified paired permutation test for significance, preserving the paper’s equal weighting of the seven families. Significance tests are Holm-corrected across 36 comparisons.
Differences below are K-Shield − AlphaSteer, in percentage points, based on the original evaluation outputs.
| Model | Average DSR | XSTest compliance | MATH | GSM8K | AlpacaEval |
|---|---|---|---|---|---|
| Llama-3.1-8B | +6.3 [4.8, 7.9] | 0.0 [−2.5, 2.5] | −1.0 [−8.4, 6.5] | +6.0 [0.4, 12.5] | +3.7 [0.8, 6.5] |
| Qwen2.5-7B | +19.5 [16.3, 22.6] | −1.6 [−4.3, 0.8] | +1.0 [−1.3, 4.5] | +3.0 [−1.9, 8.9] | +5.4 [1.6, 9.1] |
| Gemma-2-9B | +22.5 [19.7, 25.1] | +22.4 [16.8, 27.8] | +4.0 [−2.7, 10.6] | +1.0 [−6.4, 8.5] | +10.7 [7.2, 14.1] |
Bold: Holm-adjusted p ≤ 0.05. Brackets show individual, unadjusted 95% confidence intervals; significance is determined by the corrected tests.
K-Shield’s average DSR improvement is statistically significant on all three models. It also significantly improves XSTest compliance on Gemma, indicating reduced over-refusal, and AlpacaEval performance on Qwen and Gemma. The MATH and GSM8K comparisons are not significant after correction. We do not rely on these small differences to claim superior utility or statistical equivalence. Full per-attack results are reported in Appendix G.2, “Significance Tests".
Expanded utility evaluation. We additionally evaluated all three models on the full MATH-500 test set and 500 GSM8K questions:
| Model | Method | MATH-500 | GSM8K (500) |
|---|---|---|---|
| Llama-3.1-8B | Unsteered | 48.0 | 84.6 |
| Llama-3.1-8B | AlphaSteer | 47.2 | 85.8 |
| Llama-3.1-8B | K-Shield | 47.8 | 85.6 |
| Qwen2.5-7B | Unsteered | 73.4 | 91.2 |
| Qwen2.5-7B | AlphaSteer | 71.8 | 86.6 |
| Qwen2.5-7B | K-Shield | 73.6 | 90.4 |
| Gemma-2-9B | Unsteered | 49.0 | 88.0 |
| Gemma-2-9B | AlphaSteer | 42.4 | 74.0 |
| Gemma-2-9B | K-Shield | 48.6 | 86.0 |
K-Shield’s observed accuracy remains within 2 percentage points of the unsteered model across all three models and both benchmarks. It is numerically close to AlphaSteer on Llama and higher on Qwen and Gemma, with the largest gaps on Gemma: +6.2 points on MATH-500 and +12.0 on GSM8K. These expanded results provide additional empirical evidence of utility retention, separate from the statistical comparisons above.
We thank the reviewer for their thoughtful questions and constructive feedback. We provide our responses below, including additional experimental results and clarifications of our contributions and claims. This response is split into three comments.
Q1. Methodological novelty
We agree that null-space projection is established and do not claim the constraint itself as novel. Our contribution concerns where the constraint is applied. Specifically, we identify a limitation of the global null-space formulation used in prior methods and show how locally chosen protected directions can improve refusal-target approximation.
K-Shield's central methodological contribution is therefore the move from global to local null-space constraints, supported by analysis and empirical evidence in refusal steering. This also suggests a promising extension to other applications of null-space-constrained methods, although its benefits beyond refusal steering remain to be evaluated.
Regarding comparisons, AlphaSteer is our closest baseline and is evaluated in Tables 1–2 across all three models and the evaluated attacks. LEACE and AlphaEdit are relevant methodological precedents but address different objectives: LEACE removes linearly recoverable concept information, whereas AlphaEdit edits factual knowledge in model parameters while preserving unrelated knowledge. Neither directly provides a refusal-steering defense. Adapting them would therefore require additional design choices. We will expand the Related Works discussion to clarify these distinctions and position our contribution more precisely within existing uses of null-space projection.
Q2. Optimal transport novelty
We agree with this characterization. We do not propose a new OT algorithm. We use entropy-regularized OT with subspace-reconstruction costs to learn local subspaces through soft, balanced assignments. Our controlled ablations evaluate the practical value of this component. We will clarify that OT is a supporting component of K-Shield, while the central methodological contribution is the move from global to local null-space constraints.
Q3. Baseline comparison fairness and safety–utility trade-off
We agree that Tables 1–2 compare selected configurations and do not establish a generally superior safety–utility trade-off. To examine sensitivity to these operating points, we evaluated K-Shield and AlphaSteer, our closest baseline, across steering strengths of 0.1–0.5 on all three models, holding their remaining hyperparameters fixed. We focus on steering strength because it directly scales the intervention in both methods and provides a shared inference-time control for varying the safety–utility balance without changing the learned transformations.
The main tables report AlphaSteer's reproduced results at its released experimental settings. The sweep shows that increasing its steering strength can improve DSR beyond those values, but may incur substantial utility degradation or collapse. For example, on Gemma-2, increasing λ from 0.14 to 0.5 raises DSR from 76.54% to 86.1%, while average utility falls from 57.8% to 0.7%.
Across all three models, K-Shield achieves higher average DSR at every evaluated shared steering strength, with comparable or higher average utility at multiple operating points. Representative comparisons at the same strength are:
These results show that K-Shield’s empirical safety-utility advantage extends beyond the originally reported operating points. However, we acknowledge that the sweep isolates steering-strength sensitivity with the remaining settings fixed. It therefore does not establish superiority across other hyperparameter choices or fully optimised trade-off frontiers. We will revise the contribution statements and conclusions to limit our claims to the evaluated settings and avoid implying a guaranteed or universally better safety–utility trade-off. We have added the experimental setup and full results to Appendix G.1, “Steering Strength and Safety–Utility Trade-off,” with the sweep summarized in Figure 5 and per-attack and per-benchmark results reported in Tables 12–17.