acceptodds
Withdrawn from ICLR 2027

Safety Alignment with Adaptive Preference Weighting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Safety alignment should discourage harmful assistance while keeping useful answers to benign requests. Preference pairs specify which response to prefer, but averaging their losses can hide differences in how harmful and benign preferences are learned. We introduce Adaptive Preference Weighting (APW), which tracks harmful and benign preference losses separately and adjusts their coefficients within bounded intervals. APW scores responses by their mean log probabilities relative to a frozen reference model and trains one adapter with the resulting weighted loss. When training reuses the model's current errors, APW also caps the share of each harmful category and clips example weights, so that no category or example dominates. With the same base model, 5,333 ordered training pairs, adapter budget, decoder, and evaluation pipeline, APW obtains an Overall error (the mean of the harmful and benign error rates) of 18.20%, lower than all seven preference-optimization baselines. It also has the lowest error on a 600-prompt extension subset, 20.29%. Its harmful error is lower than that of every baseline, and this reduction is larger than its increase in benign refusal. Source breakdowns and prompt-level comparisons show where APW corrects harmful answers. A separate study that trains on the model's current errors tests the replay controls, sampled decoding, and the choice of refusal detector. Tracking harmful and benign losses separately is thus a practical way to set how strongly each kind of preference shapes safety training.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
safety alignment, preference optimization, adaptive weighting, over-refusal, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Wg3XOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Preference Weighting (APW), a preference optimization method designed to balance safety alignment and over-refusal in large language models. APW maintains separate loss coefficients for harmful and benign preference pairs, dynamically adjusting them based on exponentially smoothed training losses and predefined targets. The method uses reference-relative, length-normalized preference scores. The main contributions are (1) an adaptive role-weighting mechanism for harmful and benign preferences, (2) category balancing and weight clipping, and (3) an empirical evaluation of the proposed approach against seven preference optimization baselines. For validation, the authors fine-tune Llama-3.1-8B-Instruct using 5,333 preference pairs and evaluate on 1,600 prompts from multiple benchmarks.

Critical strengths

APW introduces a simple mechanism for dynamically adjusting harmful and benign preference weights using losses already computed during training, without requiring additional reward models or inference-time overhead. This is a practically relevant contribution to the important problem of balancing safety alignment and over-refusal, although the benefits of adaptivity itself remain questionable.

Critical weaknesses
  1. APW differs from the baselines in both its length-normalized preference objective and its adaptive role weighting, making it unclear which component drives the reported improvements. Although the authors provide an ablation comparing adaptive and fixed weighting, it uses a separate 649-pair training configuration and only 100 curated evaluation prompts. The reported improvement from 9.00% to 8.00% is equivalent to just one net error on a uniformly weighted 100-prompt set, providing limited evidence for the effectiveness of adaptive weighting. Moreover, the adaptive coefficients eventually reach their predefined bounds, raising the possibility that the method behaves similarly to static weighting. The lack of controlled comparisons against length-normalized DPO and appropriately tuned fixed weights in the primary setting substantially weakens the evidence for the central methodological contribution.

  2. The experiments are restricted to Llama-3.1-8B-Instruct, without validation on stronger or more recent models. Furthermore, the paper does not report the unmodified model's performance on the primary evaluation benchmarks, making it difficult to assess the absolute improvement achieved through alignment. In the primary comparison, APW outperforms the strongest baseline by only 0.02 percentage points (18.20% vs. 18.22%), with one training seed per method. Although an eight-seed stability study is provided, it uses a different training configuration and does not establish the robustness of APW's advantage over the baselines. Consequently, the evidence is insufficient to demonstrate consistent or practically meaningful improvements over existing preference optimization methods.

Questions for authors
  1. APW differs from the baselines in both its length-normalized preference scores and its adaptive role coefficients. How much of the reported improvement is attributable specifically to adaptive weighting? Could the authors provide a controlled comparison against length-normalized DPO with fixed and adaptive role weights under the primary experimental setup? The existing ablation (Table 7) shows only a 1-percentage-point difference on 100 prompts, which seems insufficient to establish the benefit of adaptive weighting.

  2. Appendix B reports that the harmful and benign coefficients eventually reach their predefined bounds (1.5 and 0.7, respectively). How do these coefficients evolve throughout training, and how much time do they spend at their bounds? In particular, does APW outperform simply using these fixed coefficients throughout training? This would help establish whether the adaptive mechanism provides meaningful benefits beyond static safety-biased weighting.

  3. The improvement over APO-Zero in Overall error is only 0.02 percentage points (18.20% vs. 18.22%). Although Section 6.4 provides an eight-seed stability analysis, it uses a different training configuration and evaluates only APW. Is the improvement over the strongest baselines statistically reliable across multiple training seeds in the primary setting? Additionally, can the authors provide the unmodified Llama-3.1-8B-Instruct model's performance on the primary evaluation to establish the absolute benefit of alignment, and is there evidence of generalization to stronger or more recent models?

Reviewer_vtg4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Standard safety alignment treats all preference pairs equally by averaging their losses. This paper proposes Adaptive Preference Weighting (APW), which computes harmful and benign preference losses separately. APW also caps the share of each harmful category and clips per-example weights so that no single category or example dominates. Results show that APW achieves a lower error than all seven preference-optimization baselines.

Critical strengths
  1. The paper evaluates both harmful compliance and benign refusal, showing the method’s trade-offs.
  2. The method is simple and effective.
Critical weaknesses
  1. The method is evaluated only on Llama-3.1-8B-Instruct, with the primary comparison using 5,333 preference pairs selected under specific harmful/benign role rules. It is unclear whether the findings transfer to other model families, larger models, or differently constructed preference datasets.
  2. The benign evaluation measures refusal rather than response quality or general capability. Without a separate utility evaluation, it is unclear whether the safety improvements preserve useful model performance.
Questions for authors

see weakness

Reviewer_69JDOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on DPO-style method of LLM safety alignment. The authors consider the trade-off that suppresses harmful assistance and preserves helpful responses to benign requests. To address this, the authors introduce Adaptive Preference Weighting (APW), which decouples the training losses of harmful and benign prompts and dynamically adjusts their role weights. Each weight scales up when its role's loss exceeds the target and scales down otherwise within bounded intervals, balancing the two objectives without adding inference overhead. Evaluated on Llama-3.1-8B-Instruct, APW outperforms seven preference-optimization baselines, achieving the lowest overall error and extension error while ensuring harmful error reductions outweigh increases in benign over-refusal.

Critical strengths
  1. The paper introduces a reference-based preference objective that tracks harmful and benign preference losses independently. By adjusting the role coefficients within bounded intervals via smoothed loss deviations from preset targets, the method balances opposing objectives without changing preference labels or adding runtime overhead.
  2. To prevent dominant subsets from skewing the model during error replay, the authors incorporate category capping and per-example weight clipping, thereby constraining the influence of outliers and category imbalances independently from the role coefficients.
  3. The evaluation protocol balances harmful compliance and benign over-refusal through source-level macro-averaging across established benchmarks, accompanied by an investigation into the model's sensitivity to refusal detectors.
Critical weaknesses
  1. APW assumes that every training prompt has a reliable harmful or benign role label. My concern is that ambiguous, dual-use, or mixed requests may not fit a binary role cleanly, and an incorrect role label changes the direction in which the adaptive controller emphasizes the data. The paper does not quantify how label noise or mixed-role prompts affect APW.
  2. The paper does include ablations for replay controls and several training choices, including fixed versus adaptive coefficients, category limits, clipping, replay size, and training length. However, the method still contains many influential choices: the harmful and benign target losses, coefficient bounds, smoothing factor, controller step size, harmful-category cap, clipping quantile and bounds, source multipliers, category multipliers, role factors, and category offsets. The paper does not explain how the target values and bounds were selected, whether the same settings remain effective across datasets and model families.
  3. The comparison uses one model family (Llama-3.1-8B-Instruct). More evaluation on other LLMs is needed to show the effectiveness of their method.
  4. The baseline set contains several methods but does not include the dynamic preference-optimization baseline such as MADPO, AMaPO, AlphaDPO. Since dynamic adjustment is central to the claimed contribution, the authors should justify the baseline selection and discuss whether APW provides an advantage over other adaptive preference methods.
  5. The paper should also clarify its intended regime relative to RLHF or other reward-model-based alignment approaches. If APW is intended for resource-constrained settings or settings that avoid an additional reward/judge model during training, that scope and motivation should be stated explicitly.
Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.