acceptodds
Withdrawn from ICLR 2027

ResInject: State-Dependent Residual Adaptation for Joint Safety and Over-Refusal Control

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Safety alignment requires Large Language Models (LLMs) to accurately separate harmful requests that must be declined from benign queries that should be served. Because this safety boundary is learned implicitly rather than specified explicitly, benign prompts sharing lexical similarities with harmful ones often fall on the wrong side, leading to over-refusal. Reducing over-refusal without increasing unsafe generations is notoriously difficult, as both behaviors are governed by the same decision boundary. Prior methods typically rely on external preference datasets or decoding-time representation steering to shift this boundary. In this work, we leverage the inherent stochasticity of the model at the safety boundary. We observe that for borderline prompts, a frozen aligned model naturally generates a mixture of refusals, unsafe responses, and safe compliances. We formalize these self-generated outputs as on-policy preference pairs and propose RESINJECT, a hidden-state intervention technique that trains a low-rank residual stream adapter on these contrastive pairs. By excluding explicit refusal targets, our reference-adjusted preference objective indirectly suppresses over-refusal while preserving the rejection of genuinely harmful requests. Extensive experiments on Llama-3, Qwen-2.5, and OLMoE demonstrate that our method significantly reduces over-refusal on safety boundary prompts and external benchmarks without compromising safety against harmful queries.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Safety Alignment, Over-refusal, Preference Learning, Residual-stream Intervention, Self-generated Supervision

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_k8CwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the joint optimization problem of reducing over-refusal while preserving safety in safety-aligned LLMs. This paper observes that repeated sampling from the same frozen model on a borderline prompt can produce refusals, unsafe responses, and safe non-refusals, and uses the resulting safe–unsafe responses to construct model-local preference pairs. Based on this observation, this paper proposes RESINJECT, a state-dependent low-rank residual adapter that conditions its correction on the current hidden state and is trained with a reference-adjusted preference objective together with a chosen-response likelihood term. Experiments across three model families show that RESINJECT achieves a balanced trade-off between reducing benign over-refusal and harmful unsafe generation.

Critical strengths

1. Clear and meaningful problem formulation. This paper explicitly considers the joint objective of reducing benign over-refusal and harmful unsafe generation, rather than optimizing either aspect in isolation. This provides a practical perspective on the trade-off between safety and model usability.

2. Novel model-local preference construction. RESINJECT constructs safe–unsafe preference pairs from different responses generated by the same frozen model for the same prompt, without relying on external answers or treating refusal as the preferred target. This provides a simple way to exploit the model’s own stochastic behavior as supervision.

3. Simple and well-motivated method design. RESINJECT introduces a state-dependent low-rank residual correction that conditions the intervention on the current hidden state, allowing different inputs to receive different corrections while keeping the number of trainable parameters relatively small.

Critical weaknesses

1. The contribution of state dependence is not sufficiently validated through ablation. The paper attributes the effectiveness of RESINJECT partly to its state-dependent residual adaptation. However, the representation controls in Table 2 do not provide a fully controlled comparison: the fixed-vector and static-residual baselines contain only 4,096 trainable parameters, whereas RESINJECT contains 65,536. Therefore, it remains unclear whether the performance improvement comes from the increased number of trainable parameters or from state dependence itself.

2. Insufficient safety evaluation. The other primary concern is that the paper’s safety evaluation is limited to direct and overtly malicious queries, which may not adequately reflect more realistic adversarial threats. Recent jailbreak attacks, such as AutoDAN [1] and PAIR [2], are specifically designed to bypass the model’s safety alignment. This issue is particularly important for RESINJECT because its correction depends on the current hidden state and is learned from safe–unsafe response pairs sampled from the frozen model itself. Jailbreak prompts may induce hidden state distributions that differ substantially from those observed during training, potentially weakening the learned state-dependent correction.

3. The discussion of related work is incomplete. In particular, the paper does not sufficiently discuss more recent methods such as AlphaSteer [3] or AdaCD [4]. These approaches similarly introduce adaptive intervention or decoding mechanisms to better balance safety and over-refusal, and are closely related to the paper’s motivation of moving beyond fixed global steering directions. A comparison or clearer discussion of how RESINJECT differs from these recent methods would help better establish the novelty of RESINJECT .

[1] Xiaogeng Liu et al., “AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models”, ICLR 2024.

[2] Patrick Chao et al., “Jailbreaking Black Box Large Language Models in Twenty Queries”, SaTML 2025.

[3] Leheng Sheng et al., “AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraint”, ICLR 2026.

[4] Yupeng Qi et al., “Please Refuse to Answer Me! Mitigating Over-Refusal in Large Language Models via Adaptive Contrastive Decoding”, ACL 2026.

Questions for authors
  1. The authors should include a parameter-matched state-independent residual baseline to clarify whether the performance improvement comes from state dependence itself or simply from having more trainable parameters.

  2. The authors should further evaluate RESINJECT under representative jailbreak attacks, such as AutoDAN and PAIR, to verify whether reducing over-refusal can still preserve safety under jailbreak attack prompts.

  3. The authors should consider providing a clearer comparison with recent related methods, such as AlphaSteer or AdaCD, to better clarify the novelty and differences of RESINJECT.

Reviewer_zFgyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the joint problem of reducing over-refusal on benign prompts without increasing unsafe generation on harmful prompts. The authors propose RESINJECT. Specifically, starting from a frozen aligned model, the authors sample borderline prompts until generations contain both safe responses that do not refuse and unsafe responses, creating within-prompt safe-unsafe preference pairs labeled by a fixed offline evaluator. They then train a residual adapter with a DPO-inspired reference-adjusted margin loss. Experiments are conducted on Llama-3.1-8B, Qwen2.5-3B, and OLMoE-1B-7B, where RESINJECT is compared against baselines, such as SFT, parameter-matched LoRA, DPO-LoRA, and ACTOR, to validate the effectiveness of the proposed method.

Critical strengths
  • The method is simple and easy to follow. The motivation is reasonable, and results show that a zero-initialized, state-dependent low-rank residual outperforms the fixed-vector, static-residual, and shuffled-label controls while remaining on the safety frontier across training budgets.

  • The evaluation in this work is well-designed. The authors use matched parameter budgets, per-seed tables, paired prompt-level bootstrap intervals, explicit separation of evidence sets, and report the zero-inclusive Llama external interval. Overall, the evaluation supports the paper’s central claim that reducing over-refusal need not compromise safety.

Critical weaknesses
  • The core design of the proposed method is not new: preference alignment, low-rank parameter-efficient updates, and hidden-state interventions all have prior precedent. The novelty is mainly applying and combining them for this safety setting, especially constructing same-prompt safe-versus-unsafe pairs from the model’s own sampled outputs. The authors should therefore clarify that the main contribution is a training-data construction strategy tailored to this setting.

  • The paper’s organization and presentation could be improved. For example, Figure 1 is rough and difficult to follow, with several mathematical symbols left unexplained in the accompanying text. The two parts of the figure also seem disconnected; adding a clear link between them would help readers understand how they relate. In addition, the discussion of related work is too brief to provide sufficient background.

Questions for authors
  • How does the OLMoE paired bootstrap mean (-16.5 pp, Table 11) reconcile with the -17.9 pp difference of the displayed pooled rates in Table 1?

  • More experimental results are shown in Appendix C.3. However, there is almost no explanation for these tables.

Reviewer_cg75OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ResInject, a state-dependent residual adaptation method for jointly controlling over-refusal and insecure generation. This method aims to reduce unnecessary rejections of benign requests by the model, while avoiding the model generating insecure content on harmful requests.

Critical strengths

1 The author froze the base model and only trained the parameters of the low-rank residual adaptation module, thereby reducing the training cost and also giving this method a certain degree of deployment potential. 2. The author constructed a preference pair between safe and unsafe responses by using the different model outputs under the same prompt, enabling the training process to directly utilize the behavioral differences of the models themselves, rather than relying solely on externally constructed standard answers.

Critical weaknesses
  1. Although the paper has set several common baselines and static intervention controls, it has not fully demonstrated that this method has clear advantages over the existing over-rejection mitigation methods. It is suggested that the authors should compare it with more directly relevant specialized methods.
  2. The experiment only involved three models, and the model size and architecture were limited. The current results are insufficient to show whether this method can stably transfer to different scales, different model families, or other mainstream instruction fine-tuning models.
  3. The authors only retained the prompt words that included both secure non-rejection responses and insecure responses, and used them to construct training preference pairs. The final qualified prompt word pools obtained by the three models were 49, 85, and 108 respectively, which were relatively limited in size. It is necessary to report in more detail the distribution before and after data filtering, the number of samples of different categories, and the potential overlap between the training set and the evaluation set, and further verify the method's sensitivity to data sampling and filtering strategies.
Questions for authors
  1. How does ResInject compare to methods specifically designed to alleviate excessive rejection?
  2. Do these improvement effects apply to models of different sizes, architectures, series, and other models that have been tuned for instructions?
  3. Can the author report the number of categories before and after filtering, check the overlap between training and evaluation, and assess the sensitivity to the selection of sampling and filtering?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.