- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
DACS DPO addresses noisy preferences by identifying and reversing suspicious preference pairs using reward differences, adapting the correction threshold, and adding a per-example gradient-norm penalty. Experiments cover three datasets and several synthetic flip rates. At 40% flips, reported average win-rate gains are 17.35 percentage points over DPO and 7.83 points over Dr.DPO. These calculations and the main/appendix tables are consistent.
The approach is promising, but implementation discrepancies and incomplete validation weaken the conclusions.
Critical strengths
- Addresses an important alignment problem without requiring an additional clean validation subset.
- Separates selection, correction, and regularization through useful component ablations.
- Evaluates multiple datasets and noise levels, with consistent numerical reporting.
Critical weaknesses
-
Released evaluation code has three confirmed defects. eval_wintie.py loads both response sets from the reference file (lines 29-30), counts reference rather than target wins, and gives ties full credit (line 69). The paper specifies half credit. This does not prove that the reported tables used this script. The authors should identify the actual evaluation version and release original responses and judge records.
-
The implementation differs materially from the paper. The trainer uses a frozen teacher rather than the current policy for selection, a different adaptive coefficient and threshold update, and LoRA rank 16 rather than 32. These changes affect the algorithm. The repository also lacks complete reproduction instructions and a required module. Equations, code, and experimental configurations must be reconciled.
-
Correlation does not establish correction accuracy. Near -1 Spearman correlation between reward difference and estimated noise probability is insufficient evidence of reliable individual label flips. Moreover, the diagnostic subset uses 50% noise, unlike the 10%-40% training rates (Section 3.1). Report detection precision/recall, incorrect flips, and residual noise at the actual thresholds and noise prevalence.
-
Statistical reliability and baseline fairness remain unclear. Sample counts, repeated runs, confidence intervals, and tuning protocols are missing. The Anthropic-HH gain over Dr.DPO is only 1.03 points. In the clean UltraFeedback setting, cDPO/rDPO differ substantially from DPO, although they should reduce to it when their noise parameter is zero (Table 4). This requires clarification under matched settings, not an immediate conclusion of faulty implementation.
-
The stability explanation is too strong. A squared gradient-norm penalty does not generally suppress incorrect updates; a scalar DPO example can instead amplify them. This challenges the general mechanism claim, not the method's empirical effectiveness. Add measured stability, a DPO-plus-regularizer control, and runtime/memory costs. Equation (11) also omits denominator stabilization; the released code already adds epsilon.
-
Novelty and generalization need narrower claims. Since loss_DPO = softplus(-reward_difference), low-margin ranking equals high-loss ranking at the same checkpoint. [PerpCorrect] already detects and corrects noisy preferences. Position the contribution around the adaptive correction procedure and combined regularization. Random flips on Pythia-2.8B do not establish robustness to natural annotation errors or broader model families.
Questions for authors
Please see weaknesses.
5 comments
Comment 3: Additional experiments on runtime and memory costs, the Vanilla DPO + stability penalty control, and measured stability.
Response 3: We evaluate all four methods on UltraFeedback with 40% noisy preferences. The complete comparison is summarized below. Detailed results are reported in Appendix B.2 and Table 5 of the updated manuscript.
| Method | Wall time (min) | Steady time/update (s) | Peak GPU allocated (GiB) | Peak GPU reserved (GiB) | Directional sensitivity ↓ | PA (%) ↑ | WR (%) ↑ |
|---|---|---|---|---|---|---|---|
| Vanilla DPO | 36.17 | 1.258 | 22.53 | 27.83 | 0.000971 | 48.44 | 55.10 |
| DACDPO | 44.05 | 1.650 | 22.54 | 27.84 | 0.001921 | 58.46 | 75.51 |
| Vanilla DPO + stability penalty | 49.50 | 1.916 | 22.56 | 27.97 | 0.000925 | 52.42 | 57.47 |
| DACSDPO | 49.53 | 1.919 | 22.56 | 27.97 | 0.000744 | 65.92 | 78.57 |
a. Runtime and memory costs. Our method (DACSDPO) substantially improves alignment performance while incurring only slight runtime overhead and negligible additional GPU memory consumption. Specifically, compared with Vanilla DPO, DACDPO increases the total training time by 7.88 min and the per-update time by 0.392 s, while increasing peak allocated and reserved GPU memory by only 0.01 GiB each. Adding the stability penalty directly to Vanilla DPO increases the total training time by 13.33 min and the per-update time by 0.658 s, with additional allocated and reserved GPU memory of 0.03 and 0.14 GiB, respectively. When applied to DACDPO, the stability penalty in DACSDPO introduces an additional 5.48 min of total training time and 0.269 s per update, with corresponding increases of only 0.02 GiB in peak allocated memory and 0.13 GiB in peak reserved memory. These results indicate that DACSDPO introduces a relatively small computational overhead without substantially increasing GPU memory requirements.
b. Vanilla DPO + stability penalty control. Adding the stability penalty directly to Vanilla DPO also improves alignment performance compared with the original Vanilla DPO baseline. Specifically, PA increases from 48.44% to 52.42%, representing an improvement of 3.98 %, while WR increases from 55.10% to 57.47%, representing an improvement of 2.37%. Meanwhile, directional sensitivity decreases from 0.000971 to 0.000925, corresponding to a reduction of approximately 4.7%. These results suggest that the stability penalty can provide improvements in both alignment performance and the stability property, even when applied without the correction mechanism.
c. Measured stability. The stability penalty effectively reduces directional sensitivity, indicating improved stability. We measure directional sensitivity as the average squared finite-difference slope of the DPO loss under the same small parameter perturbations across different methods. Compared with Vanilla DPO, adding the stability penalty reduces directional sensitivity from 0.000971 to 0.000925 (approximately 4.7%). More notably, DACSDPO achieves a directional sensitivity of 0.000744, substantially lower than the 0.001921 obtained by DACDPO, corresponding to a reduction of approximately 61.3%. Therefore, DACSDPO exhibits the lowest directional sensitivity among all four methods, demonstrating that the proposed stability penalty effectively improves stability property.
Comment 4: Effectiveness of our method (DACSDPO) across a broader range of model families.
Response 4: Our method remains effective on Qwen2.5-3B, a model family beyond Pythia-2.8B. We compare DPO, DrDPO, and DACSDPO on UltraFeedback with 40% noisy preferences. All methods use the same SFT checkpoint, training and test splits, and evaluation protocol. DACSDPO exceeds DPO by 4.56 percentage points in PA and 12.12 percentage points in WR, and exceeds DrDPO by 3.96 and 9.09 percentage points, respectively. These results support its effectiveness on an additional model family, as detailed in Appendix B.4 and Table 8 of the updated manuscript.
| Method | PA (%) | WR (%) |
|---|---|---|
| DPO | 72.12 | 69.70 |
| DrDPO | 72.72 | 72.73 |
| DACSDPO | 76.68 | 81.82 |
Dear Reviewer,
We appreciate the time you have taken to review our work and consider our response. With the discussion period nearing its end, we would be grateful for any further feedback on whether our revisions resolve your main concerns. We would be glad to clarify any remaining points.
Sincerely,
The Authors
We thank the reviewer for carefully examining the manuscript and the released code, and for identifying issues that require clarification.
Comment 1: Details and explanations of our method (DACSDPO) and its practical implementation.
Response 1: We apologize for the confusion caused by the preliminary example code in our repository. We are preparing a complete and refined version of the code, which will be available in our repository (URL: https://anonymous.4open.science/r/DACSDPO-52BF). We clarify several important issues below.
a. Clarification of the evaluation code. We confirm that the win-rate calculation used in our experiments strictly follows the evaluation protocol described in the paper, assigning half credit to ties.
b. Clarification of the practical implementation. We clarify that our method freezes the current policy for preference selection, adopts the adaptive coefficient and threshold updates, and employs LoRA with a rank of 16. Moreover, Eq. (11) should include the numerical stabilization term ε in the denominator. These implementation details in our repository are now consistent with the descriptions in the updated manuscript.
c. Experimental Variability and Reproducibility. All main experiments in the manuscript are conducted with three random seeds, and we report the mean results. Even under identical experimental configurations, results may vary across independent runs due to training stochasticity. This explains why cDPO and rDPO yield different reported results from DPO in the clean setting, despite reducing to DPO when the noise rate is zero. We have also supplemented the repository with comprehensive reproduction instructions, including dependencies, training and evaluation commands, data splits, and the noise generation procedure.
d. The 50% noise rate setting in the preliminary experiment (Section 3.1). We use a 50% noise rate to purely investigate the relationship between reward differences and noisy preferences while avoiding the effect of unequal overall numbers of noisy and clean pairs. This setting gives the two classes equal chances of representation across reward-difference ranges, so the observed relationship is not affected by class imbalance.
e. Clarification of the stability penalty. We have revised the explanation following Eq. (12) to describe the role of the stability penalty more precisely.
Comment 2: Novelty and reliability of our method (DACSDPO)
Response 2:
a. Our method differs fundamentally from PerpCorrect. PerpCorrect first trains a model on an entirely clean subset to learn the patterns of clean preferences and then uses the trained model to identify noisy preferences by additionally computing perplexity (PPL). Notably, obtaining an entirely clean subset is often challenging in real-world scenarios. This limitation has been discussed in the Related Work section of our paper. In contrast, our method directly leverages reward differences during training to identify and correct noisy preferences, without requiring an additional clean data, or separate perplexity computations.
b. Reliability of noise correction by our method (DACSDPO). We have conducted experiments on UltraFeedback with 40% preference noise to evaluate the reliability of noise correction. DACSDPO achieves an overall accuracy of 84.31%, precision of 80.64%, and recall of 80.01%, leaving only approximately 8% of preference pairs with uncorrected noise. Moreover, the correction performance of DACSDPO remains stable throughout the correction stage. Detailed results are presented in Section 4.3 and Fig. 4 of the updated manuscript.