- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper evaluates whether large language model refusal behavior remains invariant across four distinct speech act request formulations, namely Validation L0, Delegation L1, Instructional L2, and Exploratory L3. While safe response rates stay consistent across L0 through L2, refusal rates drop sharply by 17 to 21 percentage points under L3 framing. Domain specific medical fine tuning further amplifies this exploratory vulnerability, and mechanistic analyses attribute this failure to delayed commitment and execution routing breakdowns rather than a loss of internal refusal geometry.
Critical strengths
- The L0 through L3 taxonomy grounded in established question classification provides a practical and naturalistic evaluation framework that reveals sharp refusal drops under exploratory framing without relying on adversarial jailbreaks.
- The empirical design is rigorous, incorporating thorough controls for classifier boundaries, steering experiments, and fine-tuning confounds to isolate true model behaviors.
Critical weaknesses
- The mechanistic pipeline filters out queries where single token logit margins disagree with free generation labels and restricts interventions to flipped cases, risking selection bias that could obscure true L3 behavior since per level exclusion rates are omitted.
- Claims about fine-tuning mechanisms lack empirical support, as intervention experiments like patching and steering were conducted exclusively on base models rather than fine-tuned models.
Questions for authors
Please report the exact query exclusion rates per framing level and clarify if commitment depth findings hold without margin filtering.
3 comments
Response to Reviewer 3Xn5 (Part 2/2)
Geometric Profiling and Steering Interventions on Fine-Tuned Models
We thank the reviewer for pointing out the need for validating our mechanistic conclusions on fine-tuned models. We have extended our geometric profiling and steering interventions to both fine-tuned model families (Med42 and Qwen-Med), with the results incorporated into the extended Table 4 and the corresponding RQ3.
| Model Pair | Dataset | Cosine (Base → FT) | Orthogonal-only (Base → FT) | Parallel-only (Base → FT) |
|---|---|---|---|---|
| Llama → Med42 | CARES | +0.32 → +0.21 | 68 → 62% | 0 → 1% |
| Llama → Med42 | MedSafety | +0.39 → +0.28 | 72 → 65% | 0 → 0% |
| Qwen → Qwen-Med | CARES | +0.05 → −0.09 | 97 → 92% | 14 → 12% |
| Qwen → Qwen-Med | MedSafety | −0.10 → −0.15 | 100 → 95% | 56 → 43% |
Geometric changes following fine-tuning. We observe that fine-tuning consistently shifts the L3 framing direction further away from the refusal direction in both model families. For Llama→Med42, the cosine similarities decrease from +0.32/+0.39 to +0.21/+0.28 across CARES and MedSafetyBench, respectively. For Qwen→Qwen-Med, they shift from +0.05/−0.10 to −0.09/−0.15, moving further into the negatively aligned regime. These results provide direct evidence that domain-specific fine-tuning weakens the geometric alignment between exploratory framing representations and the refusal direction.
Steering interventions on fine-tuned models. We additionally perform orthogonal-only and parallel-only steering interventions to examine how refusal recovery is distributed across these components. Both fine-tuned model families exhibit predominantly orthogonal recovery, although with distinct profiles. Med42 exhibits 62–65% orthogonal-only recovery but virtually no parallel-only recovery (0–1%). Qwen-Med exhibits substantially higher orthogonal-only recovery (92–95%), while parallel-only recovery remains lower (12–43%) across the two datasets. These results suggest that medical adaptation does not eliminate the underlying refusal representation or fundamentally change the dominant steering recovery profile. Instead, the L3 framing shift moves further away from the refusal direction, while the effectiveness of steering-based recovery weakens. These findings support that the L3 vulnerability in medically adapted checkpoints is associated with increased geometric decoupling and weakened recovery, consistent with our interpretation of framing-induced execution disruption rather than complete loss of refusal representations.
Thank you for your detailed response and the additional clarifications. I have carefully read the rebuttal and the other reviewers' comments. After consideration, I have decided to maintain my original score. I appreciate the authors' efforts and wish them the best with this work.
Response to Reviewer 3Xn5 (Part 1/2)
Margin-Based Filtering and Commitment-Depth Robustness
We thank the reviewer for highlighting the possibility of selection bias introduced by margin-based filtering. We address this concern by reporting per-level exclusion rates across all three benchmarks and conducting an additional commitment-depth analysis on the full, unfiltered sample.
Per-level exclusion rates. To examine whether margin–generation disagreements disproportionately exclude exploratory queries, we report the percentage of queries excluded at each framing level on MedSafetyBench.
Exclusion rates range from 5.44% to 8.44%, with no systematic increase towards L3. In particular, L3-question has an exclusion rate of 8.44%, only 0.11 percentage points higher than L0 (8.33%), while L3-statement (7.89%) is lower than L0. Thus, the disagreements do not indicate disproportionate exclusion of exploratory queries. CARES and HarmBench exhibit a similar pattern: although their overall exclusion rates are higher, exclusions remain broadly distributed across framing levels rather than being selectively concentrated in L3. We have included the complete per-level breakdowns for all three benchmarks in Appendix D.2.
Commitment depth without filtering. We also recomputed commitment depth (c) on the full, unfiltered sample as suggested, assigning c*=L (the final layer) when the margin never stabilises. The unfiltered results preserve the L0→L3 commitment-depth gap across all three models. Gemma remains nearly flat (approximately 0.55–0.62 across levels) in both filtered and unfiltered analyses. Llama and Qwen show slightly wider L0→L3 gaps in the unfiltered analysis (approximately +0.03), because the few additional L3 boundary cases have marginally deeper commitment trajectories. The intervention analysis is restricted to flipped cases to localise mechanisms associated with an observed behavioural change. The conclusion that L3 delays commitment deeper into the network therefore holds without margin filtering.