- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies when a one-shot activation intervention is most effective at improving the safety of reasoning-model outputs. The authors propose REACT, which learns a safety direction from paired safe and unsafe trajectories and uses a learned gate to apply the intervention at an early CoT position. Experiments across three distilled reasoning models suggest that effective control is concentrated in an early-token window and does not necessarily coincide with positions where risk is most predictable.
Critical strengths
The paper investigates a relatively underexplored question: where in the CoT trajectory a hidden-state intervention can most effectively influence safety. The distinction between risk readability and controllability is conceptually interesting, and the experiments provide evidence that effective intervention positions do not necessarily coincide with those that best predict risk. The substantial safety improvements further support the potential practical value of the approach.
Critical weaknesses
- The paper’s presentation could be substantially improved. The paper lacks a clear and coherent line of argument, and the transitions between paragraphs are often insufficient. As a result, it is difficult to understand the motivation behind individual design choices and how they relate to the central research question. For example, while the first three paragraphs of the Introduction provide a reasonably clear setup, the subsequent paragraphs introduce additional details that make it difficult to identify the paper’s core research question and main findings.
- The rationale for several methodological choices and hyperparameter settings is not sufficiently explained. It is unclear which choices are essential components of the proposed algorithm and which are empirical configurations selected through tuning. Since the paper does not provide corresponding ablation studies to clarify their effects, these choices and the assumptions behind them remain difficult to assess.
- The paper acknowledges a bunch of limitations, but it is unclear why some of them are not addressed through additional experiments. Simply acknowledging these limitations does not fully address their impact on the paper’s central claims. I would encourage the authors to make a greater effort to mitigate at least some of these limitations experimentally, or to clarify why doing so is infeasible.
Questions for authors
- How sensitive are the results to the choice of intervention layer, steering strength, and gating threshold?
- Since REACT and steering without the gate produce identical outputs on StrongREJECT-273, under what conditions does the gate improve safety rather than only preserve utility?
- How do you rule out the possibility that the learned safety direction captures source-specific artifacts or refusal styles rather than generalizable safety-relevant representations?
- Since each position uses a separately learned safety direction, how to disentangle the effect of intervention timing from the effect of the position-specific direction?
- Can the local next-token likelihood analysis explain why interventions at different positions affect free-running generation differently, or is it only a correlational diagnostic?
5 comments
Response to Reviewer 91GP (Part 2/2: Q1–Q5)
Reviewer Q1: "How sensitive are the results to the choice of intervention layer, steering strength, and gating threshold?"
Across all three axes on 8B, safety gains remain stable over wide operating ranges (§5.4, Appendix G & O.5):
| Axis | Range tested | SR-273 Safe | SORRY-450 Safe | Benign-500 |
|---|---|---|---|---|
| Layer (Table 26; 1 seed, t2, α = 1) | L9 / L17 / L20 / L25 / L32 | 76.92%–80.95% (P0: 26.74%; 4.03 pp span) | 72.22%–74.67% (P0: 49.11%; 2.45 pp span) | Refusal 5.0%–5.2%, usefulness 92.83–92.98, triggers 1.00%–4.80% |
| Strength (Table 38; 3 seeds, L20/t2 fixed vector, gate off) | 4× norm range: 0.5× / 1× / 2× (1× ≡ b = 11.21458) | 76.56% / 78.39% / 79.73% (P0: 28.94%) | 74.22% / 74.89% / 74.96% (P0: 48.74%) | Gate triggers read unedited states and are invariant to α |
| Gate threshold (Table 39; 3 seeds, L20/t2) | Dev-FPR caps ≤1%/≤2%, ≤5%, and ≤10% | 81.81% (flat) | 74.96%–75.04% | Triggers 0.73% → 4.67% → 5.93%; refusal 5.40% vs. 10.67% ungated |
(Gate-threshold row: 0.24 pp vs. Table 1's 82.05% reflects 2/819 re-scored identical outputs.)
Reviewer Q2: "Since REACT and steering without the gate produce identical outputs on StrongREJECT-273, under what conditions does the gate improve safety rather than only preserve utility?"
Across all evaluated settings, the gate strictly preserves benign utility rather than adding harmful-set safety (§5.1). Because routing selects between the steered branch and P0, gating could exceed always-steering only if steering degraded safety below P0 on harmful inputs left unedited. Empirically, gated REACT matches ungated safety while restoring Benign-500 refusal:
| Metric | Ungated steering | Gated REACT |
|---|---|---|
| SR-273 Safe (8B) | 82.05% | 82.05% |
| SORRY-450 Safe (8B) | 75.11% | 74.96% |
| Benign-500 refusal (8B; P0: 5.33%) | 10.67% | 5.40% |
| Benign-500 refusal (1.5B / 7B; Table 24) | 14.60% / 15.80% | 7.20% / 9.00% |
Reviewer Q3: "How do you rule out the possibility that the learned safety direction captures source-specific artifacts or refusal styles rather than generalizable safety-relevant representations?"
Rather than claiming a style-invariant semantic subspace, three audits establish cross-source transfer onto native refusal behaviour (§5.1, App. B, H Table 28, O.6 Tables 40–41):
- (1) Paired supervision: Prompts are identical in all 170 training pairs and the first 3 CoT tokens are identical in 123/170 pairs (72.35%), with signed updates training on post-t2 continuations.
- (2) Cross-source transfer: With zero exact-text overlap, the frozen vector raises cross-source SR-273 safety by +52.99 pp (29.06%→82.05%).
- (3) Refusal style: Across 447/419 Unsafe→Safe conversions, opening answer n-gram cosine is closer to native P0 refusals than to training answers in 77.18% (+0.0323, 95% CI [0.0211, 0.0430]) and 72.08% (+0.0260, [0.0144, 0.0392]) of cases (Tables 40–41), while 89.33% of Benign-500 stays unrefused even ungated.
Together with ~98% I → the transmission, the edit tips an early CoT transition onto native refusal trajectories.
Reviewer Q4: "Since each position uses a separately learned safety direction, how to disentangle the effect of intervention timing from the effect of the position-specific direction?"
At matched norm b = 11.21458 (8B L20, gate off, 3 seeds; §5.2, App. O.1, Fig. 3 & Tables 29–30):
| Control | SR-273 ΔSafe (pp) | Isolates |
|---|---|---|
| (1) t2 × t3 cross | δ(t2) at t2/t3: +49.45 ± 2.56 / +45.42 ± 1.94; initial δ(t3): −1.10 ± 0.73 / −2.32 ± 2.38 | Vector quality in Table 2's t3 drop |
| (2) 9-position fixed-δ(t2) sweep | +27.47 to +55.19 across t1–t5 vs. +7.81 at t0 and +8.30 to +12.21 across t6–t8 | Early-context leverage |
Reviewer Q5: "Can the local next-token likelihood analysis explain why interventions at different positions affect free-running generation differently, or is it only a correlational diagnostic?"
It is a local alignment diagnostic rather than a standalone rollout mechanism (§5.2–5.3, App. B): teacher-forced shifts explain why initial δ(t2) vs. δ(t3) differ locally (Table 3: −0.213 / +0.101 vs. +0.201 / +0.311 ΔNLL; fixed-continuation KL contracting 0.0927 → 0.0011 in Table 17), whereas causal rollout propagation is established by our free-generation 4-arm experiment (Fig. 4) and shared-direction sweep (Fig. 3).
Thank you for the rebuttal. The revised version addresses most of my concerns. I've raised my scores accordingly.
Response to Reviewer 91GP (Part 1/2: W1, W2 & W3)
Reviewer W1: "The paper's presentation could be substantially improved. The paper lacks a clear and coherent line of argument, and the transitions between paragraphs are often insufficient..."
Thank you for this constructive critique. We aligned the Abstract, rewrote §1 (Introduction), added explicit design motivations across §3 (Method), and gave §5.1–5.4 conclusion-first topic sentences around one central question: "how does the safety effect of a single activation edit vary across early reasoning states, and what factors govern that variation?" Each step now motivates the next: (1) why runtime control needs causal intervention measurements beyond monitoring (Abstract, §1 ¶1–2); (2) why REACT pairs one edit + free continuation with modular gating (§1 ¶3, §3, Table 51); (3) how equal-norm sweeps separate direction quality from early-context leverage (§5.2, App. O.1); (4) how forced prefixes and the 4-arm rollout isolate next-token propagation (§5.2–5.3, App. O.3); and (5) how probe AUROC/NLL diagnostics complement practical gated control and perturbation efficiency (§5.1, §5.3–5.4).
Reviewer W2: "The rationale for several methodological choices and hyperparameter settings is not sufficiently explained. It is unclear which choices are essential components... and which are empirical configurations..."
§3 and Appendix O.11 (Table 51) now explicitly separate three tiers, backed by ablations (§5.4, App. G, O.4–O.5):
Reviewer W3: "The paper acknowledges a bunch of limitations, but it is unclear why some of them are not addressed through additional experiments..."
We converted the four core limitations directly affecting our claims into controlled experiments (App. B & O.1–O.11), while scoping non-core boundaries transparently:
Scoped boundaries (App. B & D): Zero-init boundary alignment across Tables 1 & 19; Benign-500 scoped to ordinary-instruction refusal/utility rather than math/coding or hard-benign benchmarks; exact-text (rather than semantic near-duplicate) train–test separation (Table 28); and multi-annotator human evaluation left to future work.