- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates the source-context efficacy and cross-context transfer of refusal-direction ablation, showing that intervention effectiveness depends on the extraction site and that source-effective directions can exhibit different effects across prompt contexts. Through experiments on Qwen2.5-7B and Llama-3.1-8B, the authors evaluate source and target effects using matched baselines, semantic harmful-compliance scoring, and additional control and human evaluations.
Critical strengths
-
The paper highlights an important evaluation issue by distinguishing source-context efficacy from cross-context transfer. This distinction helps avoid drawing misleading conclusions about transferability from interventions that are ineffective even in their source contexts.
-
The experimental evaluation is relatively thorough, including multiple models, extraction sites, intervention settings, matched baselines, and random-direction controls. The additional semantic scoring, human evaluation, and sensitivity analyses further strengthen the credibility of the findings.
Critical weaknesses
- The novelty and significance of the main contribution are limited. Distinguishing source efficacy from cross-context transfer is a reasonable evaluation practice, but the paper does not clearly establish how this distinction advances our understanding beyond existing intervention-validation methods. Most findings remain descriptive, with limited insight into why different refusal directions behave differently across contexts.
- The main Passive gap-contraction result is difficult to interpret due to the high baseline harmful-compliance rate (65.3%) and potential ceiling effects. Since both candidates already achieve high compliance in Passive, the reduced performance gap may largely reflect limited room for further improvement rather than meaningful differences in cross-context transfer. The current experiments do not adequately disentangle these explanations, weakening one of the paper's central conclusions.
- The writing and presentation quality are poor and need substantial improvement. For example, several table captions are placed too close to the tables without proper spacing, and the figures do not use the PDF format. The text also contains repetitive, formulaic phrasing that reads like insufficiently edited AI-generated content, including excessive parenthetical expressions such as "(N = XXX)" throughout the prose and abbreviations such as "No-Int." that are not clearly explained when introduced. These issues make the paper unnecessarily difficult to read and obscure its main contributions.
Questions for authors
-
The Passive context already has a high baseline harmful-compliance rate (65.3%), and both candidate directions achieve over 80% compliance after intervention. To what extent is the observed gap contraction driven by ceiling effects rather than genuine context-dependent transfer behavior? Could the authors provide a baseline-matched or headroom-adjusted analysis to disentangle these explanations?
-
How robust are the reported source and transfer effects across different extraction seeds and sampling choices? The current bootstrap confidence intervals are conditional on fixed extracted vectors, while the experiments use different vectors across several comparisons. In particular, the newly extracted Site A direction shows a positive source effect under the H/N heuristic, unlike the original Site A result. Can the authors demonstrate that their key conclusions remain consistent across independently extracted directions under the same intervention and semantic evaluation protocol?
-
How reliable are the semantic evaluation results, especially for Passive and Academic contexts? The human–automatic agreement is relatively modest (77.9% overall), and 490 out of 500 Academic responses reach the 256-token limit. Could the authors evaluate longer or complete responses and use additional independent human annotators to verify whether the main conclusions, particularly the Academic gap contraction, remain valid?
-
The prompt wrappers differ substantially in wording, length, framing, and baseline harmful-compliance rates. How can the authors distinguish genuine cross-context transfer effects from changes in the underlying task formulation or evaluation behavior induced by these wrappers? Would more controlled prompt variations, where only one contextual factor changes at a time, lead to the same conclusions?
-
What concrete conclusions about refusal-direction transfer become possible through the proposed evaluation framework that could not be obtained using existing intervention-validation practices? The distinction between source efficacy and target effects is reasonable, but its methodological novelty and broader significance remain unclear. Could the authors clarify which prior assumptions or conclusions are challenged by their findings, beyond demonstrating that different extraction sites and intervention configurations produce different behavioral effects?