acceptodds
Under review as a conference paper at ICLR 2027

SafelySample: Safety Steering via Contrastive Soft Prompting

Abstract

Autoregressive language models can produce unsafe completions when harmful requests bypass their alignment safeguards. Existing defenses mitigate this risk, but often require extensive model training, auxiliary scorer or expert models, or substantial losses in performance on benign tasks. We introduce SafelySample, a decoding-time method that steers generation toward safer alternatives using a safety signal obtained from the language model itself. We train a small soft prompt on paired safe and unsafe responses while keeping the underlying model frozen and preserving its behavior on benign instructions. During generation, SafelySample compares the model's default next-token distribution with the distribution induced by the safe prompt. It promotes tokens favored by the safe prompt and suppresses tokens associated with the model's default, potentially unsafe continuation, while remaining anchored to the default distribution. SafelySample therefore requires neither an external safety model nor a separately trained unsafe model. We evaluate eight instruction-tuned models from five families on ten benchmarks spanning safety, reasoning, and helpfulness. SafelySample reduces mean attack success rate by 28-45% on six models while keeping average reasoning and helpfulness consistent. We thus demonstrate that SafelySample can steer a foundation model toward safer generation while largely preserving its general capabilities.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Safety, Inference-time Steering, Soft Prompt Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5L8rOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SafelySample is a decoding-time safety method that keeps the base model frozen and trains only a small safe soft prompt via a DPO-style objective with a benign KL regularizer, then at inference compares the safe-prompt next-token distribution against the model's own default distribution and interpolates their logits as z = α·z_s + (1−α)·z_0. Across eight instruction-tuned models and ten benchmarks it cuts average attack success rate by 28–45% on six models while keeping reasoning within five points of default decoding.

Critical strengths

None

Critical weaknesses

The core mechanism — freeze the base model, optimize a small soft prompt, run two forward passes, and combine the two logit vectors — is exactly Safety Alignment of Large Language Models via Contrasting Safe and Harmful Distributions ( Zhang et al., AAAI 2026; arXiv:2406.16743, 2024). The overlap is not merely conceptual: the paper's own ablation arms, S–U Contrast. (z = z₀ + α(z_s − z_u)) and S–U Extrap. (z = αz_s − (1−α)z_u), reproduce ACD's decoding rule essentially verbatim, including a separately trained unsafe soft prompt as the contrastive reference.

This is decisive for my assessment: the paper's claimed insight — "the default model suffices as the contrastive reference" — cannot be evaluated as a departure from anything without the prior work in view. As written, the contribution reduces to running ACD's contrast with the default model instead of the unsafe prompt, a hyperparameter-level change rather than a conceptual one. The derivation in Eq. (7) does not rescue this: it is a standard KL-regularized reward-maximization problem with a textbook exponential-tilting solution, and the closed form z = αz_s + (1−α)z_0 is plain logit interpolation already established in the proxy-tuning / weak-to-strong fusion literature. It rationalizes a rule that was almost certainly chosen first.

Questions for authors

Same as weakness.

Reviewer_QJ5SOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors train a small soft safety prompt on safe/unsafe response pairs without training the underlying LLM. Then, they decode greedily with a weighted mix of logits from the steered and un-steered prompts, aiming to produce safer responses while preserving usefulness. They evaluate this across a range of instruction-tuned LLMs and benchmarks, where they find it lowers compliance with harmful requests.

Critical strengths

The paper addresses an important problem of improving LLM safety against harmful requests.

The evaluation is done on eight models from five families, and reports safety alongside multiple-choice accuracy and instruction following. This allows the safety--usefulness tradeoff to be characterised for a variety of models.

Critical weaknesses

The methodological novelty is insufficiently established, and several close predecessors are neither cited nor evaluated. ACD (Zhang et al., AAAI 2026) already learns safe and adversarial soft prompts and contrasts their logits; DETOXIGEN (Niu et al., 2024) combines soft-prompt tuning with contrastive decoding on a shared frozen backbone; and DRO (Zheng et al., ICML 2024) optimizes safety prompts while preserving general capabilities. Moreover, the main interpolation rule is algebraically a special case of DExperts (Liu et al., 2021). The methodological contribution therefore appears limited to the particular combination of preference-based prompt training, benign KL regularization, and default-logit interpolation. Although the paper includes useful ablations, it does not establish this combination's advantage over the closest predecessors or isolate the benefit of preference-based rather than supervised prompt training.

Questions for authors
  1. What methodological or empirical advance over ACD, DETOXIGEN, and DRO do the proposed combination and experiments establish?
Reviewer_2go1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduce a decoding time safety steering approach for LLMs, called SafelySample. The method freezes the base model and only trains soft prompt with DPO and KL regularization. At inference, it computes logits from both the original model and soft prompt augmented model, then calibrates token distribution toward safer outputs. Extensive evaluations across 8 instruction tuned models and 10 benchmarks are conducted to assess safety, reasoning and helpfulness.

Critical strengths
  1. The paper is well written and proposes a simple and effective decoding time safety steering method. The training cost of soft prompt is relatively low. The method is built based on mature techniques such DPO and logits calibration, making the proposed method technically feasible.
  2. Comprehensive empirical evaluations are performed on 8 models and 10 benchmarks, which demonstrate the effectiveness and generalizability of the proposed method.
Critical weaknesses
  1. The performance relies on the quality of preference datasets for training soft prompt, which requires human annotation and limit generalization toward unseen prompts.
  2. Inference requires two forward passes (one for original input and one with soft prompt), which doubles computational overhead.
  3. The core modules such as soft prompt, DPO and logits calibration are not novel individually. Although it is insightful to combine them and apply to safety steering, the contribution mainly comes from combinatorial innovation of existing components.
Questions for authors

na

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.