acceptodds
Under review as a conference paper at ICLR 2027

Same Conversation, Different Actions: Phase-Conditioned Causal Steering for Multi-Turn LLM Safety

Abstract

In multi-turn LLM dialogue, the same internal correction can improve one answer but impair another, even when both contexts require the same response behavior. We find that these differences are reproducible and that relative dialogue trajectories help select useful corrections. In this paper, we introduce PhaseSteer, which learns a shared correction family and its allocation from complete-response outcomes with the backbone frozen. A paired objective rewards appropriate answers across clarification and refusal boundaries, with separate constraints on quality loss and added harm. On our paired benchmark, PhaseSteer improves assistance-to-clarification and clarification-to-refusal joint success over development-selected comparators by an average of 4.1 and 4.5 percentage points across four backbones. Controlled analyses show improved allocation among the same learned corrections, while interactive tests show better boundary handling under generated histories. These gains are achieved with one action sampled per answer, without scoring candidate responses at inference.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
large language models, multi-turn dialogue, safety alignment

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_8LUXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper looks at multi-turn safety as a problem of changing response requirements, where a model should help, ask for clarification, refuse, or resume helping depending on the turn. The authors build a paired benchmark from four existing sources and propose PairSuccess, which counts a boundary pair (A to D, or D to C) as correct only if both answers satisfy their contracts. The proposed method, PhaseSteer, keeps the backbone frozen and learns a small set of low-rank activation corrections plus an allocator that picks one correction per answer based on how the hidden representation has drifted across turns. The two parts are trained alternately from LLM-judged outcomes, with constraints on quality loss and added harm. Experiments on four 7B to 14B models report gains of about 4 points in PairSuccess over the strongest comparator, along with transfer and interactive results.

Critical strengths
  1. The problem setting is relevant. Treating unnecessary refusal at the clarification stage and failure to resume help after a safe return as errors reflects how assistants are actually used. It complements work that focuses only on refusing escalating attacks.

  2. Requiring both answers in a boundary pair to pass is a reasonable way to avoid rewarding blanket refusal or blanket compliance. Scoring complete answers against per-prefix contracts is also, in principle, better than checking for refusal phrases.

  3. The method has some practical advantages at deployment. The backbone stays frozen, action 0 recovers the reference system, and the system falls back to the reference if calibration checks fail. Inference needs only one sampled action with no candidate scoring, at about 1.1x the reference latency.

Critical weaknesses
  1. Much of the gain over outside baselines may come from more training feedback rather than from the method itself. PhaseSteer is trained by repeatedly scoring answers with Qwen2.5-72B, using the same policy, contracts, and prompts as the final judge (Appendix D.1), at 1,540 GPU hours per selected Qwen-7B run (Table 19). Most baselines are trained on fixed data, and no baseline budgets are reported. There is also no baseline that simply tells the model the policy. The model only gets "You are a helpful assistant", yet adding one fixed instruction already improves over P0 by 3.8 to 6.8 points (Table 2). A same-budget RL baseline and a policy-in-prompt baseline are needed.

  2. All results are scored by LLM judges, and none of the main results is checked by humans. Human labels are only used to measure how often the judge agrees with people (Table 10). That check already shows real errors: the judge misses 6.7 percent of harmful answers overall and 13.0 percent for Qwen2.5-14B. Even so, the Unsafe column has no confidence intervals, although the top methods differ by only 0.2 to 0.4 points. A human check on a sample of each method's outputs, or intervals that include judge error, would help. In addition, the interactive user follows a fixed script, the external benchmarks are single-turn, and several key hyperparameters are not reported.

  3. The 4.1 and 4.5 point gains in the abstract are measured against the authors' own continuous operator (Table 16), which uses the same state and the same feedback budget. So this comparison tests discrete versus continuous corrections, not whether trajectory information helps. In principle the continuous version is at least as flexible, yet it is worse on every model, and the paper does not explain why.

  4. Two benchmark choices need explanation. Direct A to C transitions are dropped from the pairs (Appendix A.2) with no reason given, even though these sudden jumps are where trajectory information should matter most. It is also unclear whether annotators could see the labels of earlier turns (Appendix C.3). If they could, adjacent labels may agree more than they should, which directly affects PairSuccess.

Questions for authors
  1. How many judge calls and GPU hours did each baseline use? Could you add a LoRA baseline trained with online RL on the same judge reward and the same number of judge calls?

  2. How do the models perform if the full policy from Appendix I.1 is put in the system prompt, or if your phase classifier adds a short text instruction at D and C instead of the S and R vectors? What exactly is the instruction used in the History prompt baseline?

  3. Could you have humans check a sample of each method's outputs, and give confidence intervals for Unsafe and Degradation that include judge error? What is the judge's false-alarm rate for harm, and how many harmful cases are in the audit set?

  4. Why does the continuous operator do worse than the four discrete actions? What were its tuning budget and the matching tolerance mentioned in Appendix F.2?

  5. Why are direct A to C transitions excluded? How many are there in the benchmark, and how does PhaseSteer do on them?

  6. When labeling a prefix, could annotators see the labels of earlier turns, and was each conversation labeled by one person?

  7. Do different prefixes really need different actions? Within each phase, how often does the best action differ across prefixes, and does it stay the best on a fresh batch of responses? Do the repeatability results in Figure 3a still hold after controlling for the reference model's success rate, and what are the numbers for M = 1 versus M = 4? Relatedly, what happens in the substitution test if d and v are replaced with random noise of the same size, and why do cross-phase and same-phase donors cause almost the same drop?

11 comments

Thank you for the detailed review. We address feedback fairness, human validation, and the interpretation of the diagnostics below. Rates are percentages, differences are percentage points (pp), and brackets give 95% confidence intervals.

1. Feedback fairness and resources (W1; Q1).

Under matched total training feedback, PhaseSteer outperforms online LoRA-RL on Qwen2.5-7B-Instruct (Q7) by 2.6 [0.5, 4.7] / 3.0 [0.7, 5.3] pp in AD/DC. The baseline optimizes LoRA adapters using sequence-level REINFORCE with a leave-one-out baseline (RLOO) [1]. Both methods share P0P_0, fixed S/R controls, initialization supervision, scorer/contracts, endpoint/pair objective, preservation criteria, and development-only checkpoint selection.

Each receives 24.0M valid training judgments, cumulatively across 24 search configurations and three selected-configuration runs. PhaseSteer / online LoRA-RL consume 25.20M / 25.25M total judge requests and 41,600 / 51,200 A800-80GB GPU-hours before testing. Requests include non-training stages and retries; retries do not increase valid training feedback. The paper's 1,540 GPU-hours is one selected PhaseSteer run included in that total. Matching concerns cumulative training feedback on Q7, without implying equal computation or per-checkpoint exposure.

References

[1] Ahmadian et al., Back to Basics: Revisiting REINFORCE-Style Optimization for Learning from Human Feedback in LLMs, ACL 2024.

2. Policy and text controls (W1; Q2).

Full-policy prompting supplies the complete policy to the raw backbone or P0P_0. Predicted-phase text uses the frozen observer but replaces S/R with text instructions, without learned activation corrections. All prompts are development-selected and fixed, none receives gold phases or per-example contracts.

Q7 controlADDCUnsafe [95% CI]
Raw + full policy72.168.85.8 [4.2, 7.9]
P0P_0 + full policy75.573.74.4 [3.0, 6.2]
Predicted phase to text76.675.04.0 [2.7, 5.7]
PhaseSteer81.881.03.4 [2.3, 4.9]

PhaseSteer's paired advantage over predicted-phase text is 5.2 [1.8, 8.6] / 6.0 [2.4, 9.6] pp. Together, the online-RL and text controls test whether the gains can be explained by feedback quantity or explicit policy guidance alone.

The History baseline instead appends this fixed system-message suffix before the serialized conversation:

Before answering, reassess the current request in light of the full conversation. If authorization, intent, or scope remains unclear, ask a focused clarification or offer bounded help; refuse prohibited assistance, and resume permissible help when the request returns to a safe scope.

3A. Human assessment (W2; Q3).

We audited all 16 methods on Qwen2.5-7B-Instruct (Q7) and Llama-3.1-8B-Instruct (L8), with method identities and automated scores hidden and independent judgments preceding adjudication. Humans and the automated judge scored the same outputs.

The boundary audit covers 300 common lineages: per model, 140 AD and 180 DC pairs share 20 middle endpoints, yielding 620 distinct endpoints and 19,840 method–model–endpoint answers overall. Unsafe uses 500 prefixes per model/method (16,000 answers). Degradation uses 300 independently screened reference-success prefixes per model/method (600 control units per method). These are separate populations. Fresh evaluation answers, including from P0P_0, are independent of screening, so reference degradation need not be zero.

Human scores, equal model weightsADDCUnsafeDegradation
Raw backbone60.851.712.56.7
Frozen reference P0P_068.662.87.92.6
History prompt73.568.25.84.1
CAA62.261.16.48.2
CAST69.167.44.84.0
AlphaSteer70.771.63.12.0
AUSteer73.170.43.83.0
LoRA-DPO73.872.53.44.1
ACT-style76.670.64.63.0
MTSA-style73.575.12.93.7
Continuous operator77.476.33.73.0
Online LoRA-RL78.677.93.82.9
Raw + full policy72.068.75.55.4
P0P_0 + full policy75.473.64.23.6
Predicted phase to text76.875.44.13.1
PhaseSteer81.280.53.52.5

Online LoRA-RL is the strongest human-scored AD/DC baseline. Its direct paired comparison with PhaseSteer is:

Human-evaluation scopeAD gain (pp; 95% CI)DC gain (pp; 95% CI)
Q7: matched-total-feedback comparison2.8 [0.2, 5.4]3.0 [0.4, 5.6]
Q7/L8: equal-model aggregate2.6 [0.4, 4.8]2.6 [0.3, 4.9]

Intervals resample whole lineages within source, preserving method pairing, shared endpoints, and evaluation weights. They quantify sampling uncertainty for audited checkpoints. Both Q7 intervals exclude zero, providing human support for the boundary-handling advantage in the matched-total-feedback comparison. The two-model aggregate broadens human validation. Feedback matching is established only for Q7.

Against the original continuous comparator, human AD/DC gains are 3.8 [1.3, 6.3] / 4.2 [1.5, 6.9] pp, versus automated gains of 3.9 / 4.5 on the same records. Safety comparisons are less conclusive:

Human outcomeContinuousPhaseSteerPaired difference (pp; 95% CI)
Unsafe3.7 [2.4, 5.4]3.5 [2.2, 5.2]-0.2 [-0.7, 0.3]
Degradation3.0 [1.9, 4.4]2.5 [1.5, 3.9]-0.5 [-1.1, 0.1]

Both paired difference intervals include zero: we do not claim a statistically established safety advantage. This comparison uses the original continuous systems, not the additional Q7 continuous fit discussed in our response to Q4.

3B. Judge reliability, evaluation scope, and training (W2; Q3).

The additional harm audit covers all 16 methods on Qwen2.5-7B-Instruct (Q7) and Llama-3.1-8B-Instruct (L8): 500 prefixes per model/method, totaling 16,000 answers.

Judge errors. Training feedback uses Qwen2.5-72B-Instruct; final scoring uses Llama-3.3-70B-Instruct. The shared rubric/wrapper can still induce correlated errors. Taking human-labeled harm as positive:

AuditTPFNFPTNHuman-positive
Original16812141,662180
Additional7524812215,078800

These are raw counts. Inclusion-weighted false-alarm / missed-harm rates are 0.8 [0.4, 1.4] / 6.7 [3.6, 11.5]% originally and 0.8 [0.5, 1.2] / 6.0 [3.9, 8.7]% additionally, with human-negative / human-positive denominators. The Q7/L8 audit does not resolve Qwen-14B's 13.0% missed-harm rate. Human-scored intervals use adjudicated labels and automated intervals remain conditional on automated labels.

Interaction. Actual generated histories enter subsequent turns, but the schedule is fixed: uncertainty at turn 2 or 3, absent authorization next, and permissible return at turn 5. Each model has 400 tasks and four rollouts/task/run. Recovery is conditional on delivered permissible-return opportunities. Figure 8 is an independent qualitative demonstration, not a complete Table 4 rollout. External benchmarks test single-turn behavior, not unrestricted multi-turn robustness.

Training configuration. Across the four main-paper backbones, we learn four nonzero rank-32 corrections and a 387–64–5 GELU allocator, with the backbone, reference controls, observer, and state transformations frozen. Correction and allocator optimization use AdamW, with betas (0.9, 0.999), epsilon 10−810^{-8}, zero weight decay, and global gradient-norm clipping at 1.0. Training seeds are 17, 42, and 2025.

The allocator reuses a fixed outcome table at each correction snapshot. Independent response samples estimate correction gradients while routing probabilities, multipliers, margins, and active constraints remain fixed. Before clipping, this is the unbiased sequence-gradient estimator for the frozen, linearized reward objective in Appendix B.3. Each correction update triggers remeasurement of the changed actions. Outcome and gradient responses use full-softmax sampling at temperature 1, without top-p/top-k filtering, and a 1,024-token cap. Log-probabilities are recomputed under the same parameters and distribution, with dropout disabled.

After freezing the corrections, we refit and check the sparse allocator under its deployment distribution. Q7's quality tolerance is 0.03. Per backbone and training run, the error budgets are 0.04 for the preservation checks and 0.01 for the efficacy check. Acceptance requires all preservation checks to pass and the efficacy lower bound to exceed 2 pp, otherwise the system retains the reference. Configuration and checkpoint selection use development data, not final test outcomes. Appendix E.2 now specifies the stage-wise learning rates, update and sampling schedules, objective weights, constraint tolerances, and calibration checks. Appendix E.4 details the training-feedback accounting.

Reviewer_w2xkOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the problem of safety and usefulness in LLM agents. An agent should be safe and refuse helping with harmful content but it should also not be overly cautious making itself useless when providing knowledge. The authors follow the path of steering internal states of the model in order to change its output behavior. However, they propose a novel approach to this where they train a policy to choose the best steering method based on context. Their training pipeline also trains a set of interventions for the model. The approach is novel and argumentative on how interventions to models should also be context-aware. The paper presents a training and evaluation reliant on two main situations, asking for clarification upon a dubious request, and denying a request whilst also providing a safe course of action. The proposed approach, called PhaseSteer, is not the leanest, it has substantial moving parts, but the authors do a proper job of ablating their method showing how each element affects the final outcome. Their results show that PhaseSteer manages to provide responses that increase clarification and safe course of action whilst maintaining or reducing unsafe responses. Upon a test on a held out set the numbers also increase.

Critical strengths
  1. The problem tackled by the paper is a relevant current issue faced when training and aligning LLMs. The authors' approach for a harmless yet useful response is a form of labeling the conversation beyond the binary helpful and useless or harmful and useful. By labeling the conversation in phases as an attempt to shift them their method takes the broader context into account when trying to steer the conversation.
  2. The method presented in the paper is novel, it combines Reinforcement Learning and mechanistic approaches. The policy that chooses the intervention, and the learning of interventions themselves are adaptable, both context-dependent. This provides us with an indication that internal state adjustment of models should be adaptable to the current context. This approach even has implications outside merely LLM steering but other fields of mechanistic interpretability or neural model control might also benefit from such insights.
Critical weaknesses

Major concerns:

  1. The generalization aspect of the trained models is a central part of the paper to show if the proposed approach can indeed be applied outside its training data. However, the paper leaves a single paragraph dedicated to this aspect whilst pointing to an appendix. I strongly suggest that this be part of the main text. Otherwise the main text is left without one of its most important aspects when proposing a new training or alignment schema.
  2. The values presented in tables throughout the paper do not show confidence intervals. The authors do point out that the numbers are the mean with 95% confidence bootstrapped from 3 seeds. However, the lack of the confidence intervals makes it difficult to understand how much the metho really is better than the alternatives. Table 2 has a reference to an appendix with some intervals but not all. Upon looking in the appendix we can see that the gap between the proposed approach and the baselines is not as wide as the mean seems to show. I suggest the authors introduce the intervals and provide both argument and explanation in text.

Minor concerns:

  1. Clarity of the paper could be improved. The paper is mostly well written, though I would like to draw attention to some paper-specific terms that are not properly introduced. Term such as "phase" and "source" have very specific meanings in the context of the paper but are never formally introduced. I suggest doing a sweep for other terms that seem to follow the same pattern. Those can be troublesome for the reader especially since the method is not straightforward.
Questions for authors
  1. The metrics used to evaluate the PhaseSteer are sound, though they are somewhat specific to what the method proposes and is trained on. How do similar approaches evaluate their efficiency? Have the authors considered evaluating their approach on other, perhaps more established, metrics?
  2. Regarding the previously mentioned confidence intervals, Appendix G.1 apparently has intervals only for PhaseSteer but not for the other methods. Could the authors provide the confidence intervals for Tables 2, 3, and 4?
6 comments

Thank you for recognizing the relevance of the problem and context-dependent intervention selection. We have made the source-transfer evidence visible in the main text and provide the requested complete confidence intervals below. Rates are percentages, differences are percentage points (pp).

1. Generalization in the main text (W1).

Section 5.5 now presents source-transfer results, support counts, and the whole-system holdout protocol. Each target source is excluded from observer fitting, fixed-control construction and calibration, state scaling, initialization, correction and allocator learning, and development selection. Related dialogue derivatives stay in one lineage and split. Thus, holdout covers the complete adaptation pipeline.

Qwen-7B covers all four folds, Llama-8B provides a separate, prespecified X-Teaming confirmation. Gains are relative to the continuous comparator selected on development data from the remaining sources.

Held-out source and backboneSupport: lineages / AD / DCAD gain (pp)DC gain (pp)
WildChat, Qwen-7B612 / 218 / 2684.85.2
MHJ, Qwen-7B537 / 201 / 2904.24.7
Crescendo, Qwen-7B474 / 153 / 2393.15.5
X-Teaming, Qwen-7B504 / 179 / 1994.53.8
X-Teaming confirmation, Llama-8B504 / 179 / 1994.34.6

Source-weighted Qwen-7B AD/DC gains are 4.2 / 4.8 pp. All point gains are positive, although they do not establish significance for each fold. This supports new-source generalization after backbone-specific adaptation, rather than transfer of learned parameters across architectures.

2. Complete confidence intervals and their interpretation (W2; Q2).

The tables at the end of this response provide all 176 method–metric intervals for Tables 2–4: the 24 already reported in Appendix G.1 and 152 additional empirical intervals computed from evaluation outputs using Appendix D.3. All point estimates are unchanged. The revised manuscript summarizes the principal paired comparisons and their interpretation in Appendix G.7, the complete cellwise intervals are provided here.

For the central comparison, the four-model, equally weighted paired differences against the development-selected continuous operator are:

OutcomePhaseSteer-minus-comparator gain [95% CI] (pp)
AD PairSuccess4.1 [1.2, 7.0]
DC PairSuccess4.5 [1.6, 7.4]

These positive intervals support improved aggregate boundary handling. Marginal method intervals can overlap while a paired difference excludes zero because shared-instance dependence is retained. Unsafe/Degradation point differences are -0.2 / -0.6 pp, the marginal intervals do not establish significant reductions. Model-specific preservation bounds test prespecified absolute tolerances, separately from between-method differences.

How the intervals are computed. Bootstrap replicates resample whole lineages within source, retaining endpoints, response repetitions, and action trials. Shared lineage multiplicities preserve method/model dependence. Learned methods use three training runs. A run draw applies to its complete run, with run axes paired only when the training design pairs them. Source weights and evaluated backbones remain fixed. Fixed methods use repeated generations from the same system. We report the 2.5th and 97.5th percentiles, rounded to one decimal place: coverage is pointwise, not simultaneous. Paired differences are computed within each replicate. Interactive recovery retains method-specific realized eligible denominators. These intervals are conditional on automated labels and do not include systematic judge error.

Ablations and interaction. The factorial contrasts separate trajectory information from paired supervision:

Effect and backboneAD intervalDC interval
Trajectory information: Qwen-7B[1.3, 6.7][1.6, 7.0]
Trajectory information: Llama-8B[0.9, 6.6][1.4, 7.2]
Paired supervision: Qwen-7B[-0.2, 4.4][0.2, 5.2]
Paired supervision: Llama-8B[-0.4, 4.5][0.5, 5.7]

Trajectory effects are positive on both boundaries; paired-supervision evidence is clearer on DC, with AD intervals including zero. The highest combined mean alone does not establish positive interaction. In the interactive evaluation, PhaseSteer improves boundary handling by 5.6 [2.0, 9.2] pp over ACT-style on Qwen-7B and 4.4 [0.9, 7.9] pp over MTSA-style on Llama-8B. These gains are measured on generated dialogue histories, whereas static PairSuccess is evaluated on fixed prefixes.

3. Established evaluation metrics (Q1).

The paper evaluates safety, over-refusal, instruction following, and response quality using native benchmark procedures, frozen systems, and no benchmark-specific tuning. Qwen-7B results are:

Benchmark and native metricP0P_0 (%)PhaseSteer (%)
HarmBench: attack success ↓12.510.0
OR-Bench: benign refusal ↓8.07.6
IFEval: strict prompt-level accuracy ↑72.572.1
AlpacaEval: LC win rate ↑21.020.6

The point estimates show lower attack success and over-refusal alongside small utility declines. Computational efficiency is measured separately: Qwen-7B median end-to-end latency is 1.10× its frozen reference, including the observer and fixed controls.

PairSuccess adds a boundary-specific test: both complete answers must satisfy their respective prefix contracts. With binary contract satisfaction qi(Yi)q_i(Y_i), one pair's outcome is

zij=qi(Yi),qj(Yj).z_{ij}=q_i(Y_i)\\,q_j(Y_j).

We average these joint outcomes under the specified repetition/source/model weights. This penalizes blanket refusal or compliance that succeeds at only one endpoint. The separate scorer audit covers 1,920 records, with 1,856 resolved, 93.0% weighted agreement and 0.844 Cohen's kappa. It checks scorer reliability, not comparative method performance or judge-error-adjusted interval coverage.

4. Terminology (minor concern).

Sections 3.1–3.2 define phase as the policy-required response: A, assistance is permitted; D, clarification or bounded help is needed because unresolved authorization, intent, or scope affects permissibility; C, prohibited assistance is declined while permissible help is retained. Phase describes required behavior, not the generated answer. A source is an originating dialogue resource; a lineage groups a dialogue and its derivatives, which remain together during splitting and source holdout.

Complete intervals requested for Tables 2–4. Every cell below is mean [95% CI], in percent. Q7/Q14/L8/G12 denote Qwen-7B/Qwen-14B/Llama-8B/Gemma-12B. AD/DC are backbone-specific, Degradation/Unsafe aggregate four models in Table 2 and two in Table 3.

Table 2 — boundary success, Qwen models

MethodQ7 AD ↑Q7 DC ↑Q14 AD ↑Q14 DC ↑
Raw backbone60.8 [52.5, 68.5]49.2 [42.2, 56.3]67.4 [59.2, 74.6]57.1 [50.0, 64.0]
Frozen reference P068.4 [60.6, 75.3]61.7 [54.8, 68.2]74.2 [66.7, 80.5]68.3 [61.6, 74.4]
History prompt73.8 [65.9, 80.4]66.7 [59.7, 73.0]80.6 [73.2, 86.3]72.8 [66.0, 78.7]
CAA61.9 [53.6, 69.6]60.3 [53.2, 67.0]68.8 [60.7, 75.9]65.9 [58.9, 72.3]
CAST68.9 [60.8, 76.0]66.4 [59.4, 72.8]75.0 [67.2, 81.5]72.3 [65.5, 78.2]
AlphaSteer70.1 [62.0, 77.1]70.6 [63.7, 76.6]76.4 [68.7, 82.7]75.7 [69.1, 81.3]
AUSteer72.8 [64.8, 79.5]69.3 [62.4, 75.4]78.7 [71.1, 84.7]75.0 [68.4, 80.6]
LoRA-DPO73.5 [65.6, 80.1]71.4 [64.6, 77.4]79.5 [72.0, 85.4]76.8 [70.3, 82.3]
ACT-style76.8 [69.1, 83.1]69.8 [62.9, 75.9]82.1 [74.9, 87.6]75.2 [68.6, 80.8]
MTSA-style73.1 [65.2, 79.8]74.6 [67.9, 80.3]79.2 [71.7, 85.1]79.1 [72.7, 84.3]
Continuous operator77.0 [69.7, 83.0]75.5 [69.1, 80.9]82.6 [75.8, 87.8]80.4 [74.3, 85.3]
PhaseSteer81.8 [74.9, 87.1]81.0 [75.0, 85.8]86.0 [79.6, 90.6]84.6 [79.0, 88.9]

Table 2 — boundary success, Llama and Gemma

MethodL8 AD ↑L8 DC ↑G12 AD ↑G12 DC ↑
Raw backbone62.3 [54.0, 69.9]55.6 [48.5, 62.5]66.1 [57.9, 73.5]58.4 [51.3, 65.2]
Frozen reference P069.6 [61.7, 76.3]65.1 [58.3, 71.4]73.1 [65.5, 79.5]69.4 [62.7, 75.4]
History prompt74.5 [66.6, 81.0]71.2 [64.4, 77.2]79.9 [72.4, 85.7]73.2 [66.5, 79.0]
CAA63.7 [55.4, 71.2]63.6 [56.5, 70.1]68.2 [60.0, 75.4]68.0 [61.0, 74.2]
CAST70.5 [62.4, 77.5]69.8 [62.9, 75.9]74.8 [67.0, 81.3]73.1 [66.4, 78.9]
AlphaSteer72.3 [64.3, 79.1]74.1 [67.4, 79.8]75.6 [67.8, 82.0]77.0 [70.5, 82.4]
AUSteer74.6 [66.7, 81.1]72.8 [66.0, 78.7]77.4 [69.7, 83.6]75.9 [69.3, 81.5]
LoRA-DPO75.2 [67.4, 81.7]74.7 [68.0, 80.4]78.6 [71.0, 84.6]77.3 [70.8, 82.7]
ACT-style78.0 [70.4, 84.1]72.6 [65.8, 78.5]80.5 [73.1, 86.2]75.5 [68.9, 81.1]
MTSA-style75.0 [67.2, 81.5]77.5 [71.0, 82.9]78.2 [70.6, 84.3]80.0 [73.7, 85.1]
Continuous operator78.4 [71.2, 84.2]78.2 [72.0, 83.4]81.1 [74.1, 86.5]80.8 [74.8, 85.7]
PhaseSteer82.7 [75.9, 87.9]83.3 [77.5, 87.8]85.0 [78.4, 89.8]84.0 [78.3, 88.4]

Table 2 — Degradation and Unsafe, four-model aggregate

MethodDegradation ↓Unsafe ↓
Raw backbone6.8 [5.2, 8.8]12.4 [10.8, 14.2]
Frozen reference P02.2 [1.4, 3.5]7.4 [6.2, 8.8]
History prompt3.9 [2.8, 5.5]5.6 [4.5, 6.9]
CAA8.1 [6.4, 10.2]6.4 [5.3, 7.8]
CAST3.8 [2.7, 5.4]4.8 [3.8, 6.0]
AlphaSteer1.8 [1.1, 3.0]3.0 [2.2, 4.0]
AUSteer2.8 [1.9, 4.2]3.7 [2.9, 4.8]
LoRA-DPO3.9 [2.8, 5.5]3.4 [2.6, 4.4]
ACT-style2.9 [1.9, 4.3]4.4 [3.5, 5.6]
MTSA-style3.5 [2.4, 5.0]2.8 [2.1, 3.8]
Continuous operator2.7 [1.8, 4.1]2.8 [2.1, 3.8]
PhaseSteer2.1 [1.3, 3.4]2.6 [1.9, 3.5]

Table 3 — fixed-family ablations, boundary success

State / objectiveQ7 AD ↑Q7 DC ↑L8 AD ↑L8 DC ↑
General history / Endpoint75.7 [67.9, 82.1]74.0 [67.3, 79.7]76.9 [69.2, 83.1]75.9 [69.3, 81.5]
Relative trajectory / Endpoint79.8 [72.3, 85.6]78.2 [71.8, 83.5]80.9 [73.5, 86.6]80.0 [73.7, 85.1]
General history / Paired77.9 [70.3, 84.0]76.6 [70.1, 82.1]79.2 [71.7, 85.1]78.8 [72.4, 84.0]
Relative trajectory / Paired81.8 [74.9, 87.1]81.0 [75.0, 85.8]82.7 [75.9, 87.9]83.3 [77.5, 87.8]

Table 3 — Degradation and Unsafe, Qwen-7B/Llama-8B aggregate

State / objectiveDegradation ↓Unsafe ↓
General history / Endpoint3.1 [1.8, 5.3]4.2 [3.0, 5.9]
Relative trajectory / Endpoint2.6 [1.4, 4.7]3.6 [2.5, 5.2]
General history / Paired2.8 [1.6, 5.0]3.7 [2.6, 5.3]
Relative trajectory / Paired2.4 [1.3, 4.5]3.1 [2.1, 4.6]

Table 4 — interactive evaluation. Generated histories follow the scheduled user policy. Steer.* is AUSteer for both backbones; Learn.* is ACT-style for Qwen-7B and MTSA-style for Llama-8B. These metrics are distinct from static PairSuccess, recovery uses each method's realized eligible denominator.

Qwen-7B

OutcomeReferenceSteer.*Learn.*PhaseSteer
Boundary handling62.4 [56.1, 68.3]70.6 [64.5, 76.0]74.2 [68.3, 79.3]79.8 [74.2, 84.4]
Legitimate completion74.8 [69.8, 79.3]73.6 [68.5, 78.1]79.1 [74.3, 83.2]82.6 [78.0, 86.4]
Recovery success67.2 [60.0, 73.7]71.4 [64.4, 77.5]76.8 [70.1, 82.4]83.4 [77.2, 88.2]
Unsafe continuation16.8 [13.4, 20.8]9.8 [7.2, 13.1]7.6 [5.4, 10.6]6.4 [4.4, 9.3]

Llama-8B

OutcomeReferenceSteer.*Learn.*PhaseSteer
Boundary handling65.8 [59.6, 71.5]73.9 [68.0, 79.1]76.3 [70.5, 81.3]80.7 [75.2, 85.2]
Legitimate completion76.9 [72.0, 81.2]75.8 [70.8, 80.2]80.4 [75.7, 84.4]83.1 [78.6, 86.8]
Recovery success63.8 [56.5, 70.5]69.5 [62.4, 75.8]74.6 [67.7, 80.4]82.0 [75.7, 87.0]
Unsafe continuation13.4 [10.4, 17.1]7.2 [5.0, 10.2]5.5 [3.6, 8.2]4.8 [3.1, 7.4]
Reviewer_M999OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes PhaseSteer, a method for selecting internal activation corrections based on how a multi-turn conversation evolves. Rather than applying a fixed safety intervention, the approach jointly learns a family of low-rank corrections and an allocator that selects interventions using current and relative hidden-state representations. The learning objective encourages appropriate responses across assistance, clarification, and refusal boundaries while limiting degradation and unsafe assistance. Experiments across four model backbones report improvements over existing steering and behavioral-learning methods.

Critical strengths
  • Interesting problem formulation. The paper recognizes that the appropriate safety response changes throughout a conversation and, importantly, that the same intervention may have different effects even when the required behavior is identical. Learning context-dependent intervention selection is a meaningful direction.

  • Relatively thorough experimental evaluation. The paper evaluates multiple backbones, compares against several relevant baselines, and includes ablations that isolate the contribution of trajectory information. In particular, fixing the available corrections while varying the allocator provides useful evidence that conversational trajectories improve intervention selection.

Critical weaknesses
  1. The proposed framework is complex, and the conceptual contribution could be clearer.

PhaseSteer combines learned low-rank corrections, a phase estimator, relative trajectory representations, alternating optimization, and preservation constraints. Although the experimental results support the effectiveness of the complete system, it is difficult to identify which aspects constitute the essential methodological advance.

The relative trajectory features are simple differences between hidden representations. The paper demonstrates that these features are useful, but offers limited explanation of why they predict intervention effectiveness or what conversational properties they capture.

  1. Generalizability and practical applicability remain uncertain.

The experiments primarily use relatively small open models (7B–14B), and interactive evaluation is restricted to two model families. It remains unclear how well the approach scales to larger models or transfers across substantially different safety policies and conversational settings.

Moreover, the primary evaluation relies on constructed response contracts and automated judgments. Although the authors provide human validation and additional benchmark evaluations, the extent to which the improvements generalize to open-ended, real-world interactions remains uncertain.

Questions for authors
  • What properties of conversational change are captured by the relative hidden-state differences? Can the authors provide a more interpretable characterization of why these features predict intervention effectiveness?

  • How sensitive are the results to the design of the correction family and the complexity of the allocator? Is there a substantially simpler configuration that retains most of the benefits?

  • How well does PhaseSteer generalize to larger or more recent models and to different safety policies?

5 comments

Thank you for highlighting the contribution, interpretability, and applicability questions. We clarify the central contribution and report additional mechanism, simplification, and generalization experiments. Rates are percentages, differences are percentage points (pp), and brackets give pointwise 95% confidence intervals.

1. Essential contribution (W1a).

PhaseSteer learns how the effectiveness of an internal correction changes with conversational context. Prefixes requiring the same response behavior can benefit from different corrections. We jointly learn a reusable low-rank correction family and its allocation from complete-response outcomes; actions have no predefined behavioral roles. Alternating updates track changing intervention effects, while paired supervision and preservation constraints couple boundary handling with acceptable behavior elsewhere.

Analytical support. For a fixed family and a common feasible set, Appendix B.5 bounds allocation regret by 2(1+λB)ϵ+ζ2(1+\lambda_B)\epsilon+\zeta, where ϵ\epsilon bounds quality-estimation error uniformly, ζ\zeta is the optimization gap, and λB\lambda_B weights the boundary objective. It also bounds action-reuse loss when effect profiles remain close; feasibility transfers when constraint slack covers changes in costs. These bounds motivate evaluating trajectory features through their ability to predict action-effect variation.

With the correction family, feedback, allocator capacity, and optimization budgets fixed, trajectory inputs improve AD/DC PairSuccess by 2.14 / 2.74 pp over the development-selected alternative, averaged over Qwen-7B and Llama-8B. Alternatives include phase posteriors, the current full-history representation, and a general history encoder. This isolates allocation quality. The 4.1 / 4.5 pp headline gains compare the complete method with trajectory-conditioned continuous control across four backbones.

2. What the relative features capture (W1b; Q1).

For pre-intervention user-turn representations, dt=ht−h1d_t=h_t-h_1 measures cumulative displacement and vt=ht−ht−1v_t=h_t-h_{t-1} recent change, including the preceding assistant response. The current representation hth_t supplies context. Both differences are zero initially; endpoints are recomputed within the retained window after truncation.

Semantic changes alter relative action effects. We contrasted scope expansion with narrowing and introduction with partial resolution of authorization uncertainty. Independent annotators verified that both variants remained in phase D and specified contracts before seeing outputs. Topic, length, and turn position were matched; meaning-preserving paraphrases controlled for wording changes.

Define Δa(x)=E[qx(Ya)]−E[qx(Y0)]\Delta_a(x)=\mathbb{E}[q_x(Y_a)]-\mathbb{E}[q_x(Y_0)], where qxq_x indicates contract satisfaction and action 0 recovers the reference. For each factor ff, we fixed the action pair (af,bf)(a_f,b_f) and contrast direction on development data, requiring both actions to be feasible in both variants. We measured

If=E(x−,x+)∣f[(Δaf(x+)−Δbf(x+))−(Δaf(x−)−Δbf(x−))].I_f=\mathbb{E}_{(x^{-},x^{+})\mid f}\Big[\big(\Delta_{a_f}(x^{+})-\Delta_{b_f}(x^{+})\big)-\big(\Delta_{a_f}(x^{-})-\Delta_{b_f}(x^{-})\big)\Big].

The reference term cancels, isolating changes in relative action effectiveness. Each factor uses 120 held-out lineages, equally distributed across four sources; the disjoint factor sets total 240 lineages shared by both models. Each action/variant/batch/run receives 16 responses. Batch 2 uses fresh generations on the same lineages with unchanged actions and corrections. Results average three runs; paired source-stratified lineage/run resampling retains variants and action outcomes together.

Backbone / factorBatch 1 (pp)Batch 2 (pp; 95% CI)
Qwen-7B / scope8.455.64 [2.24, 8.96]
Qwen-7B / authorization uncertainty8.258.47 [4.24, 12.66]
Llama-8B / scope6.356.81 [3.51, 10.12]
Llama-8B / authorization uncertainty6.553.75 [0.16, 7.24]

All four contrasts retain their prespecified direction in both batches; the second-batch pointwise intervals exclude zero.

Trajectory information contributes to selection. On these variants, with correction family, capacity, feedback, and search matched, trajectory allocation outperforms current-state-plus-phase allocation:

BackboneEndpoint-success gain (pp)Gain minus that on paraphrases (pp)
Qwen-7B2.7 [1.0, 4.4]1.8 [0.4, 3.2]
Llama-8B2.4 [0.8, 4.0]1.6 [0.3, 2.9]

To test use of semantically related trajectory components, we fitted low-dimensional semantic readouts on training/development annotations and froze them before testing. We removed the corresponding components only from the allocator's trajectory inputs, keeping current-context features, phase posteriors, corrections, and generation settings fixed. Random controls matched rank and each prefix's perturbation norm. Semantic removal caused 1.4 [0.3, 2.5] / 1.2 [0.2, 2.2] pp more endpoint-success loss than random controls on Qwen-7B/Llama-8B, respectively. These are endpoint diagnostics, separate from AD/DC PairSuccess.

Together, the tests connect conversational meaning, relative intervention effects, and information used for selection. They support a functional interpretation of the tested features, without assigning universal semantic meanings to individual hidden coordinates. Separately, choosing actions in one batch and evaluating them in another yields 2.3 [1.1, 3.5] pp over phase-wise fixed selection, averaged equally over both models. Deployment selects one action without candidate-response scoring.

3. Simpler configurations and sensitivity (W1a; Q2).

Allocator complexity. With the fully learned correction family fixed, a linear allocator achieves 80.7 / 80.0 AD/DC on Qwen-7B, versus 81.8 / 81.0 for the original MLP, at 1.08× / 1.10× reference latency. Degradation/Unsafe differences are +0.1 [-0.2, 0.4] / 0.0 [-0.2, 0.2] pp.

Learning-pipeline complexity. We also tested a combined simplification: after preference initialization, freeze the corrections and fit a linear allocator from one fixed outcome dataset. This removes subsequent correction learning, alternating updates, and repeated outcome refreshes, while retaining the observer, reference, and preservation checks.

BackboneSimplified pipeline: AD/DCFull PhaseSteer: AD/DC
Qwen-7B77.6 / 76.681.8 / 81.0
Llama-8B79.0 / 79.282.7 / 83.3

With equal valid training-feedback counts and development-search trials, the simplified pipeline uses 88.0% of the full method's training GPU-hours on the same hardware, including initialization and calibration. The full method adds 4.1 [2.5, 5.7] pp, averaged equally over both models and boundaries. This provides a simpler training option with a measured performance cost.

Correction family and constraints. Qwen-7B ranks 32, 64, 128 differ from the default mean AD/DC by at most 0.3 pp. Increasing nonzero actions from 1 to 4 with total feedback and search fixed improves AD/DC by 2.2 [1.0, 3.4] / 2.6 [1.3, 3.9] pp, averaged over Qwen-7B/Llama-8B. Removing quality-preservation cost while retaining harmful-assistance control increases Degradation by 2.5 [0.8, 4.2] / 2.3 [0.6, 4.0] pp on the two models. This supports the distinct roles of correction learning, allocation, and preservation. Appendix B.4 explains why its nonnegative costs prevent improvements from offsetting damage in the acceptance check on measured prefixes.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary: Brief Overview and Rebuttal Results at a Glance

Dear Area Chair,

Thank you for overseeing the discussion and considering our work.

Brief Overview of the Work

Two requests can both call for clarification, yet the same internal correction can elicit a useful question in one case and a premature refusal in the other. Learning how the conversation reached the current turn helps predict which correction will work. In matched conversations, changing the scope of the request or what was known about permission shifted the relative benefit of the corrections. The pattern recurred in fresh answers to the same requests. This led us to ask whether the path to the current turn could help the model choose a correction.

  1. Choose a correction using the conversation’s trajectory. From feedback on complete answers, PhaseSteer learns low-rank corrections together with a rule for choosing among them. We compared selectors with the same corrections, feedback and capacity. The selector that tracked changes in hidden state across turns outperformed a general history encoder on both boundaries in Qwen-7B and Llama-8B (Table 3).
  2. Evaluate both sides of a change in behavior. PairSuccess requires appropriate answers on both sides of an assistance-to-clarification or clarification-to-refusal transition. A correct refusal on the second turn cannot erase a premature refusal on the first.
  3. Preserve helpful answers while learning to intervene. With the backbone frozen, training separately constrains quality loss on answers the reference model handles well and increases in prohibited assistance. At inference, PhaseSteer makes one choice for the whole answer, including the option to keep the reference behavior, without scoring candidate replies.

Previous Rebuttal Results at a Glance

8LUX raised their rating from 2 to 3 after the new evidence resolved most of their concerns. 8LUX’s follow-up

w2xk explicitly expressed willingness to raise their rating (which would take it from 3 to 4), confirming that they were satisfied with our response and revisions. w2xk’s follow-up

M999 has not posted a follow-up. The blinded audit discussed with 8LUX and the source holdouts reported to w2xk also address M999’s questions about automated evaluation and transfer to new dialogue sources. In our reply, we examined which corrections helped within the same phase, compared simpler training variants, and tested a 32B backbone and adaptation to two new policies.

Major Concerns and How We Addressed Them

(A) Does tracking the conversation improve the choice of correction?

How we addressed it

  • Among requests that all called for clarification, changing the scope or the uncertainty about authorization shifted the relative benefit of the same corrections. The pattern held on fresh answers.
  • We also compared choosing among several corrections with choosing when to apply a single one. With feedback and search budgets matched, the larger set improved joint success on both boundaries, averaged across Qwen-7B and Llama-8B.

Corresponding questions: M999 Q1–2; 8LUX Q7, follow-up Q8.

(B) Could extra training feedback explain the gains?

How we addressed it

  • On Qwen-7B, PhaseSteer outperformed online LoRA-RL on both boundaries. The methods received the same total primary-training feedback, and PhaseSteer used slightly less in each selected run.
  • On the same backbone, PhaseSteer reached the development targets and passed the preservation checks with fewer judgments than continuous control at the median across seeds.

Corresponding questions: 8LUX Q1 and Q4, follow-up Q1–2 and Q6.

(C) Do human judgments support the main result?

How we addressed it

  • Raters who were blind to the method scored the same Qwen-7B answers as the automated judge. They found higher joint success for PhaseSteer than for online LoRA-RL on both boundaries.

Corresponding questions: 8LUX Q3, follow-up Q4–5.

(D) Does the result extend to new dialogue sources, models and policies?

How we addressed it

  • We excluded each test source and its derived dialogues from every stage of adaptation. On Qwen-7B, PhaseSteer outperformed continuous control on both boundaries for every held-out source.
  • The advantage over continuous control also held on Qwen2.5-32B with matched feedback and search budgets.
  • On Qwen-7B, we kept the learned corrections and refitted and recalibrated the selector for each new policy. This improved on the policy-specific reference under both clarification-first and stricter privacy rules.

Corresponding questions: M999 Q3; w2xk major concern 1.


Complete Problem List (Map)

Reviewer M999

  • Q1: contribution and meaning of trajectory features within a phase.
  • Q2: correction family, allocator complexity and a simpler training pipeline.
  • Q3: larger models, different policies and applicability beyond the constructed benchmark.

Reviewer w2xk

  • Major concern 1: source generalization in the main text.
  • Q1: established evaluation metrics.
  • Q2 and major concern 2: confidence intervals for Tables 2–4.
  • Minor concern: definitions of phase and source.

Reviewer 8LUX

  • Q1: feedback and compute.
  • Q2: full-policy, phase-to-text and history prompts.
  • Q3 and W2: human validation, judge errors, evaluation scope and training settings.
  • Q4: continuous control, tuning and attribution.
  • Q5: direct assistance-to-refusal transitions.
  • Q6: annotation independence and visibility of prior labels.
  • Q7: action stability, donor substitutions and random noise.

Reviewer 8LUX, follow-up

  • Q1: strongest baseline in the abstract.
  • Q2: per-run feedback and other baseline costs.
  • Q3: text controls on Llama-8B.
  • Q4: rater agreement and audit coverage.
  • Q5: shared rubric and missed harmful answers on Qwen-14B.
  • Q6: performance as feedback increases.
  • Q7: direct transitions against the reference and online LoRA-RL.
  • Q8: trajectory information with one or four corrections.

Thank you again to the reviewers and to you as the Area Chair for your time and dedication.

We deeply appreciate your careful consideration of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.