- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Adaptive Preference Weighting (APW), a preference optimization method designed to balance safety alignment and over-refusal in large language models. APW maintains separate loss coefficients for harmful and benign preference pairs, dynamically adjusting them based on exponentially smoothed training losses and predefined targets. The method uses reference-relative, length-normalized preference scores. The main contributions are (1) an adaptive role-weighting mechanism for harmful and benign preferences, (2) category balancing and weight clipping, and (3) an empirical evaluation of the proposed approach against seven preference optimization baselines. For validation, the authors fine-tune Llama-3.1-8B-Instruct using 5,333 preference pairs and evaluate on 1,600 prompts from multiple benchmarks.
Critical strengths
APW introduces a simple mechanism for dynamically adjusting harmful and benign preference weights using losses already computed during training, without requiring additional reward models or inference-time overhead. This is a practically relevant contribution to the important problem of balancing safety alignment and over-refusal, although the benefits of adaptivity itself remain questionable.
Critical weaknesses
-
APW differs from the baselines in both its length-normalized preference objective and its adaptive role weighting, making it unclear which component drives the reported improvements. Although the authors provide an ablation comparing adaptive and fixed weighting, it uses a separate 649-pair training configuration and only 100 curated evaluation prompts. The reported improvement from 9.00% to 8.00% is equivalent to just one net error on a uniformly weighted 100-prompt set, providing limited evidence for the effectiveness of adaptive weighting. Moreover, the adaptive coefficients eventually reach their predefined bounds, raising the possibility that the method behaves similarly to static weighting. The lack of controlled comparisons against length-normalized DPO and appropriately tuned fixed weights in the primary setting substantially weakens the evidence for the central methodological contribution.
-
The experiments are restricted to Llama-3.1-8B-Instruct, without validation on stronger or more recent models. Furthermore, the paper does not report the unmodified model's performance on the primary evaluation benchmarks, making it difficult to assess the absolute improvement achieved through alignment. In the primary comparison, APW outperforms the strongest baseline by only 0.02 percentage points (18.20% vs. 18.22%), with one training seed per method. Although an eight-seed stability study is provided, it uses a different training configuration and does not establish the robustness of APW's advantage over the baselines. Consequently, the evidence is insufficient to demonstrate consistent or practically meaningful improvements over existing preference optimization methods.
Questions for authors
-
APW differs from the baselines in both its length-normalized preference scores and its adaptive role coefficients. How much of the reported improvement is attributable specifically to adaptive weighting? Could the authors provide a controlled comparison against length-normalized DPO with fixed and adaptive role weights under the primary experimental setup? The existing ablation (Table 7) shows only a 1-percentage-point difference on 100 prompts, which seems insufficient to establish the benefit of adaptive weighting.
-
Appendix B reports that the harmful and benign coefficients eventually reach their predefined bounds (1.5 and 0.7, respectively). How do these coefficients evolve throughout training, and how much time do they spend at their bounds? In particular, does APW outperform simply using these fixed coefficients throughout training? This would help establish whether the adaptive mechanism provides meaningful benefits beyond static safety-biased weighting.
-
The improvement over APO-Zero in Overall error is only 0.02 percentage points (18.20% vs. 18.22%). Although Section 6.4 provides an eight-seed stability analysis, it uses a different training configuration and evaluates only APW. Is the improvement over the strongest baselines statistically reliable across multiple training seeds in the primary setting? Additionally, can the authors provide the unmodified Llama-3.1-8B-Instruct model's performance on the primary evaluation to establish the absolute benefit of alignment, and is there evidence of generalization to stronger or more recent models?