acceptodds
Desk rejected at ICLR 2027

Reward Differences Matter for Robust LLM Alignment under Noisy Preferences

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Human preference alignment is pivotal for encouraging large language models to generate more helpful and safer responses that align with human values. Direct preference optimization (DPO) is a widely used alignment approach that enables LLMs to learn preferences directly from human preference data. However, human preference data inevitably contain noise arising from annotator inattention and bias, which can substantially degrade DPO performance. To identify noisy preferences, we conduct extensive preliminary experiments and empirically observe that as the reward difference decreases, the noise probability gradually increases during DPO training. Based on this observation, we propose a new method named (DACSDPO), which combines noisy preference correction with stability regularization. Specifically, DACSDPO first uses the reward difference to select and correct potentially noisy preferences during training. To further stabilize optimization in the presence of residual noise, DACSDPO then incorporates a stability penalty based on the squared norm of the sample gradient into the DPO objective, with an adaptive stability-aware coefficient. Experiments on Anthropic-HH, UltraFeedback, and PKU-SafeRLHF under varying noise rates demonstrate that DACSDPO improves robustness over representative preference optimization and robust alignment baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, DPO, Noisy Preferences, Reward Difference, Gradient Regularization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_T6P2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

DACS DPO addresses noisy preferences by identifying and reversing suspicious preference pairs using reward differences, adapting the correction threshold, and adding a per-example gradient-norm penalty. Experiments cover three datasets and several synthetic flip rates. At 40% flips, reported average win-rate gains are 17.35 percentage points over DPO and 7.83 points over Dr.DPO. These calculations and the main/appendix tables are consistent.

The approach is promising, but implementation discrepancies and incomplete validation weaken the conclusions.

Critical strengths
  • Addresses an important alignment problem without requiring an additional clean validation subset.
  • Separates selection, correction, and regularization through useful component ablations.
  • Evaluates multiple datasets and noise levels, with consistent numerical reporting.
Critical weaknesses
  1. Released evaluation code has three confirmed defects. eval_wintie.py loads both response sets from the reference file (lines 29-30), counts reference rather than target wins, and gives ties full credit (line 69). The paper specifies half credit. This does not prove that the reported tables used this script. The authors should identify the actual evaluation version and release original responses and judge records.

  2. The implementation differs materially from the paper. The trainer uses a frozen teacher rather than the current policy for selection, a different adaptive coefficient and threshold update, and LoRA rank 16 rather than 32. These changes affect the algorithm. The repository also lacks complete reproduction instructions and a required module. Equations, code, and experimental configurations must be reconciled.

  3. Correlation does not establish correction accuracy. Near -1 Spearman correlation between reward difference and estimated noise probability is insufficient evidence of reliable individual label flips. Moreover, the diagnostic subset uses 50% noise, unlike the 10%-40% training rates (Section 3.1). Report detection precision/recall, incorrect flips, and residual noise at the actual thresholds and noise prevalence.

  4. Statistical reliability and baseline fairness remain unclear. Sample counts, repeated runs, confidence intervals, and tuning protocols are missing. The Anthropic-HH gain over Dr.DPO is only 1.03 points. In the clean UltraFeedback setting, cDPO/rDPO differ substantially from DPO, although they should reduce to it when their noise parameter is zero (Table 4). This requires clarification under matched settings, not an immediate conclusion of faulty implementation.

  5. The stability explanation is too strong. A squared gradient-norm penalty does not generally suppress incorrect updates; a scalar DPO example can instead amplify them. This challenges the general mechanism claim, not the method's empirical effectiveness. Add measured stability, a DPO-plus-regularizer control, and runtime/memory costs. Equation (11) also omits denominator stabilization; the released code already adds epsilon.

  6. Novelty and generalization need narrower claims. Since loss_DPO = softplus(-reward_difference), low-margin ranking equals high-loss ranking at the same checkpoint. [PerpCorrect] already detects and corrects noisy preferences. Position the contribution around the adaptive correction procedure and combined regularization. Random flips on Pythia-2.8B do not establish robustness to natural annotation errors or broader model families.

Questions for authors

Please see weaknesses.

5 comments

We thank the reviewer for carefully examining the manuscript and the released code, and for identifying issues that require clarification.

Comment 1: Details and explanations of our method (DACSDPO) and its practical implementation.

Response 1: We apologize for the confusion caused by the preliminary example code in our repository. We are preparing a complete and refined version of the code, which will be available in our repository (URL: https://anonymous.4open.science/r/DACSDPO-52BF). We clarify several important issues below.

a. Clarification of the evaluation code. We confirm that the win-rate calculation used in our experiments strictly follows the evaluation protocol described in the paper, assigning half credit to ties.

b. Clarification of the practical implementation. We clarify that our method freezes the current policy for preference selection, adopts the adaptive coefficient and threshold updates, and employs LoRA with a rank of 16. Moreover, Eq. (11) should include the numerical stabilization term ε\varepsilon in the denominator. These implementation details in our repository are now consistent with the descriptions in the updated manuscript.

c. Experimental Variability and Reproducibility. All main experiments in the manuscript are conducted with three random seeds, and we report the mean results. Even under identical experimental configurations, results may vary across independent runs due to training stochasticity. This explains why cDPO and rDPO yield different reported results from DPO in the clean setting, despite reducing to DPO when the noise rate is zero. We have also supplemented the repository with comprehensive reproduction instructions, including dependencies, training and evaluation commands, data splits, and the noise generation procedure.

d. The 50% noise rate setting in the preliminary experiment (Section 3.1). We use a 50% noise rate to purely investigate the relationship between reward differences and noisy preferences while avoiding the effect of unequal overall numbers of noisy and clean pairs. This setting gives the two classes equal chances of representation across reward-difference ranges, so the observed relationship is not affected by class imbalance.

e. Clarification of the stability penalty. We have revised the explanation following Eq. (12) to describe the role of the stability penalty more precisely.

Comment 2: Novelty and reliability of our method (DACSDPO)

Response 2:

a. Our method differs fundamentally from PerpCorrect. PerpCorrect first trains a model on an entirely clean subset to learn the patterns of clean preferences and then uses the trained model to identify noisy preferences by additionally computing perplexity (PPL). Notably, obtaining an entirely clean subset is often challenging in real-world scenarios. This limitation has been discussed in the Related Work section of our paper. In contrast, our method directly leverages reward differences during training to identify and correct noisy preferences, without requiring an additional clean data, or separate perplexity computations.

b. Reliability of noise correction by our method (DACSDPO). We have conducted experiments on UltraFeedback with 40% preference noise to evaluate the reliability of noise correction. DACSDPO achieves an overall accuracy of 84.31%, precision of 80.64%, and recall of 80.01%, leaving only approximately 8% of preference pairs with uncorrected noise. Moreover, the correction performance of DACSDPO remains stable throughout the correction stage. Detailed results are presented in Section 4.3 and Fig. 4 of the updated manuscript.

Comment 3: Additional experiments on runtime and memory costs, the Vanilla DPO + stability penalty control, and measured stability.

Response 3: We evaluate all four methods on UltraFeedback with 40% noisy preferences. The complete comparison is summarized below. Detailed results are reported in Appendix B.2 and Table 5 of the updated manuscript.

MethodWall time (min)Steady time/update (s)Peak GPU allocated (GiB)Peak GPU reserved (GiB)Directional sensitivity ↓PA (%) ↑WR (%) ↑
Vanilla DPO36.171.25822.5327.830.00097148.4455.10
DACDPO44.051.65022.5427.840.00192158.4675.51
Vanilla DPO + stability penalty49.501.91622.5627.970.00092552.4257.47
DACSDPO49.531.91922.5627.970.00074465.9278.57

a. Runtime and memory costs. Our method (DACSDPO) substantially improves alignment performance while incurring only slight runtime overhead and negligible additional GPU memory consumption. Specifically, compared with Vanilla DPO, DACDPO increases the total training time by 7.88 min and the per-update time by 0.392 s, while increasing peak allocated and reserved GPU memory by only 0.01 GiB each. Adding the stability penalty directly to Vanilla DPO increases the total training time by 13.33 min and the per-update time by 0.658 s, with additional allocated and reserved GPU memory of 0.03 and 0.14 GiB, respectively. When applied to DACDPO, the stability penalty in DACSDPO introduces an additional 5.48 min of total training time and 0.269 s per update, with corresponding increases of only 0.02 GiB in peak allocated memory and 0.13 GiB in peak reserved memory. These results indicate that DACSDPO introduces a relatively small computational overhead without substantially increasing GPU memory requirements.

b. Vanilla DPO + stability penalty control. Adding the stability penalty directly to Vanilla DPO also improves alignment performance compared with the original Vanilla DPO baseline. Specifically, PA increases from 48.44% to 52.42%, representing an improvement of 3.98 %, while WR increases from 55.10% to 57.47%, representing an improvement of 2.37%. Meanwhile, directional sensitivity decreases from 0.000971 to 0.000925, corresponding to a reduction of approximately 4.7%. These results suggest that the stability penalty can provide improvements in both alignment performance and the stability property, even when applied without the correction mechanism.

c. Measured stability. The stability penalty effectively reduces directional sensitivity, indicating improved stability. We measure directional sensitivity as the average squared finite-difference slope of the DPO loss under the same small parameter perturbations across different methods. Compared with Vanilla DPO, adding the stability penalty reduces directional sensitivity from 0.000971 to 0.000925 (approximately 4.7%). More notably, DACSDPO achieves a directional sensitivity of 0.000744, substantially lower than the 0.001921 obtained by DACDPO, corresponding to a reduction of approximately 61.3%. Therefore, DACSDPO exhibits the lowest directional sensitivity among all four methods, demonstrating that the proposed stability penalty effectively improves stability property.

Comment 4: Effectiveness of our method (DACSDPO) across a broader range of model families.

Response 4: Our method remains effective on Qwen2.5-3B, a model family beyond Pythia-2.8B. We compare DPO, DrDPO, and DACSDPO on UltraFeedback with 40% noisy preferences. All methods use the same SFT checkpoint, training and test splits, and evaluation protocol. DACSDPO exceeds DPO by 4.56 percentage points in PA and 12.12 percentage points in WR, and exceeds DrDPO by 3.96 and 9.09 percentage points, respectively. These results support its effectiveness on an additional model family, as detailed in Appendix B.4 and Table 8 of the updated manuscript.

MethodPA (%)WR (%)
DPO72.1269.70
DrDPO72.7272.73
DACSDPO76.6881.82

Dear Reviewer,

We appreciate the time you have taken to review our work and consider our response. With the discussion period nearing its end, we would be grateful for any further feedback on whether our revisions resolve your main concerns. We would be glad to clarify any remaining points.

Sincerely,

The Authors

Reviewer_2UfFOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Difference-Aware Noise Correction for Stable DPO (DACS DPO) to improve the robustness of direct preference optimization under noisy human preferences. The authors first empirically observe a negative correlation between reward differences and the probability of noisy preference labels. Based on this observation, DACS DPO identifies and corrects potentially noisy preference pairs using an adaptive reward-difference threshold, together with a warm-up strategy to improve the reliability of noise identification. Additionally, the method incorporates a gradient-norm-based stability penalty with an adaptive coefficient to mitigate the effects of residual noise during optimization. Experiments on three preference datasets under varying synthetic noise rates demonstrate that DACS DPO generally outperforms representative preference optimization and robust alignment baselines in terms of preference accuracy and win rate.

Critical strengths
  1. DACS DPO achieves competitive or superior performance compared with a wide range of preference optimization and robust alignment baselines across three datasets and multiple noise rates. The improvements are particularly notable under high noise rates.
Critical weaknesses
  1. Limited novelty compared with closely related work. The core ideas of using reward differences to identify unreliable preferences and correcting potentially flipped preference labels have been explored in prior work. Specifically:
    • Reward-difference-based noise identification: PLC-DPO [1] uses the same DPO policy-reference reward margin as Eq. (4) to infer whether a preference pair is clean, flipped, or ambiguous.
    • Preference label correction: PerpCorrect [2] already detects potentially noisy preference pairs using a perplexity-based score and corrects them by reversing their preference labels. PLC-DPO also performs online label correction using the DPO reward margin, with soft routing among clean, flipped, and ambiguous pairs.
    • Compared with these methods, the main technical distinctions of DACS DPO appear to lie in its adaptive selection threshold, warm-up stopping criterion, and gradient-norm stability regularization. However, the paper does not sufficiently establish the individual benefits of these components. A clearer discussion of the differences from PLC-DPO and PerpCorrect, together with more targeted ablations, would help substantiate the claimed novelty.
  2. Insufficient ablation and unclear implementation of the proposed components. Table 2 compares DAS DPO, DAC DPO, and DACS DPO, but the configurations of these variants are not clearly specified. In particular, both DAS DPO (Eq. 6) and DAC DPO (Eq. 7) are initially defined with a fixed threshold before the adaptive threshold mechanism is introduced in Eqs. (8)–(9). Although Eq. (10) subsequently incorporates the adaptive threshold into DAC DPO, it remains unclear whether DAS DPO and DAC DPO in Table 2 both employ this mechanism. This ambiguity makes it difficult to isolate the contribution of preference correction. Furthermore, while the comparison between DAC DPO and DACS DPO evaluates the overall stability penalty, it does not isolate the effectiveness of the adaptive stability-aware coefficient. Additional ablations on the adaptive threshold and adaptive stability coefficient would better substantiate the contributions of these components.
  3. Unclear computational overhead of the stability penalty. The proposed stability penalty in Eq. (12) involves the squared norm of per-sample gradients, whose optimization generally requires second-order differentiation. This may introduce substantial computational and memory overhead compared to standard DPO. However, the paper does not clarify how this penalty is implemented or report its computational cost. The authors should provide implementation details and compare the training time and memory consumption of DACS DPO with those of vanilla DPO and DAC DPO.
  4. Unclear selection and sensitivity of the warm-up patience parameter. The proposed warm-up stopping criterion relies on the patience parameter pp (Section 3.2 and Algorithm 1), but the paper does not specify its value or how it is selected. Since pp directly affects the identified warm-up stopping step and potentially the reliability of subsequent noise correction, the authors should clarify its setting and provide a sensitivity analysis to demonstrate the robustness of the proposed stopping criterion.

Reference

[1] PLC-DPO: Posterior Label Correction in Noisy and Ambiguous Preference Optimization, Findings of EMNLP, 2026.

[2] Perplexity-aware Correction for Robust Alignment with Noisy Preferences, NeurIPS, 2024.

Questions for authors

See Weaknesses.

3 comments

We thank the reviewer for the careful assessment and constructive suggestions on the method and experimental validation.

Comment 1: The novelty of our method compared with closely related work.

Response 1: Our method is fundamentally different from PerpCorrect and PLC-DPO, and has obvious contributions and advantages.

a. PerpCorrect. PerpCorrect and our method propose two fundamentally different approaches to identifying potentially noisy preferences. PerpCorrect first trains on an "absolutely clean" subset to obtain the ability to identify noisy preferences by additionally calculating perplexity (PPL). However, obtaining an entirely clean subset can be difficult in real-world scenarios. In contrast, our method uses reward differences produced during training as the primary evidence for identifying and correcting noisy preferences, without requiring additional clean data or separate perplexity computations.

b. PLC-DPO. Although PLC-DPO and our method both refer to the concept of reward differences, the ways they detect noisy preferences are totally different. PLC-DPO estimates the mean reward difference of samples within a mini-batch and detects noisy preferences by computing the deviation between this mean and each sample’s reward difference. Its basic assumption is that the deviation can be considered key evidence for judging whether a preference is noisy. In contrast, our method directly uses reward differences to identify and correct noisy preferences. Our method is based on the observation that, as reward differences decrease, noise probability gradually increases, exhibiting a clear negative correlation.

Comment 2: Insufficient ablation and unclear implementation of the proposed components.

Response 2:

a. Clarification of Table 2 and preference correction. Both DASDPO and DACDPO in Table 2 use the same adaptive threshold τt\tau_t. We have revised Eqs. (6) and (7) to state this explicitly.

b. The ablation of the adaptive threshold. The adaptive threshold outperforms every tested fixed threshold in our method (DACSDPO). We retain preference correction and the adaptive stability coefficient, changing only the threshold mechanism. As shown in Table 6 and Appendix B.3 of the revised manuscript, the adaptive threshold achieves 65.92% PA and 78.57% WR. The best tested fixed threshold, τ=−4\tau=-4, achieves 62.62% PA and 69.70% WR. Thus, adaptation improves PA and WR by 3.30 and 8.87 percentage points, respectively.

Threshold mechanismFixed thresholdPA (%)WR (%)
Adaptive τt\tau_t—65.9278.57
Fixed τ\tau−561.5559.60
Fixed τ\tau−462.6269.70
Fixed τ\tau−361.0566.67
Fixed τ\tau−261.1263.64
Fixed τ\tau−160.3161.62

c. The ablation of the adaptive stability coefficient. The adaptive coefficient achieves the best PA and WR among the fixed coefficient settings. We keep the adaptive threshold, correction rule, and penalty unchanged, replacing only its coefficient αt/λ\alpha_t/\lambda with a fixed multiplier ww. As shown in Table 7 and Appendix B.3, the adaptive coefficient achieves 65.92% PA and 78.57% WR. The best tested fixed coefficient, w=0.1w=0.1, achieves 50.70% PA and 54.55% WR. This comparison supports the effectiveness of the adaptive coefficient.

Stability coefficientPA (%)WR (%)
Adaptive αt/λ\alpha_t/\lambda65.9278.57
Fixed w=0.1w=0.150.7054.55
Fixed w=0.3w=0.336.6846.06
Fixed w=0.5w=0.534.6449.49
Fixed w=1.0w=1.035.2446.46

Comment 3: The computational overhead of the stability penalty is unclear.

Response 3: Our method substantially improves alignment performance while incurring only slight runtime overhead and negligible additional GPU memory consumption.

Specifically, compared with Vanilla DPO, DACDPO increases the total training time by 7.88 min and the per-update time by 0.392 s, while increasing peak allocated and reserved GPU memory by only 0.01 GiB each. Adding the stability penalty directly to Vanilla DPO increases the total training time by 13.33 min and the per-update time by 0.658 s, with additional allocated and reserved GPU memory of 0.03 and 0.14 GiB, respectively. When applied to DACDPO, the stability penalty in DACSDPO introduces an additional 5.48 min of total training time and 0.269 s per update, with corresponding increases of only 0.02 GiB in peak allocated memory and 0.13 GiB in peak reserved memory. These results indicate that the proposed stability penalty introduces a relatively small computational overhead without substantially increasing GPU memory requirements. Detailed results are reported in Section 4.4 and Table 3 of the updated manuscript.

MethodWall time (min)Steady time/update (s)Peak GPU allocated (GiB)Peak GPU reserved (GiB)
Vanilla DPO36.171.25822.5327.83
DACDPO44.051.65022.5427.84
Vanilla DPO + stability penalty49.501.91622.5627.97
DACSDPO49.531.91922.5627.97

Comment 4: Sensitivity of the warm-up patience parameter pp.

Response 4: Our method (DACSDPO) maintains relatively stable performance across various patience pp. We evaluate the sensitivity of DACSDPO to the patience parameter pp on UltraFeedback with 40% noisy preferences. We vary p∈10,30,50,100p \in \\{10, 30, 50, 100\\} in the dynamic warm-up stopping criterion.

As shown in the table below, PA varies by 0.24 percentage points and WR by 1.53 percentage points across these settings. Thus, the final alignment results are relatively insensitive to pp over the evaluated range. We select p=50p=50 because it achieves the highest observed PA and WR on UltraFeedback with 40% noisy preferences. We have added this sensitivity analysis to Appendix B.1 of the updated manuscript.

ppPA (%)WR (%)
1065.8777.55
3065.7178.06
5065.9278.57
10065.6877.04

Dear Reviewer,

Thank you again for your careful review and constructive feedback. As the discussion period is nearing its end, we would be grateful to know whether our responses and revisions have addressed your main concerns. If any questions remain, we would welcome the opportunity to clarify them further.

Sincerely,

The Authors

Reviewer_RamrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines how DPO can learn from preference data containing incorrect labels. The authors observe that pairs with smaller implicit reward differences are more likely to have been mislabeled, and use this signal to identify and reverse potentially incorrect preferences. Their method, DACSDPO, combines an adaptive correction threshold with a gradient-norm penalty. A separate warm-up criterion determines when reward differences are considered reliable enough for correction. The experiments cover Anthropic-HH, UltraFeedback, and PKU-SafeRLHF, with 10%–40% of training preferences randomly flipped. The reported results show gains over standard and robust preference optimization baselines in preference accuracy and response win rate. Ablations compare sample filtering, preference correction, and the addition of the gradient penalty.

Critical strengths

1.The correction mechanism of the current paper utilizes the existing reward differences in DPO training and does not require a separate noise detection model. This feature gives it an advantage in the current approach. Additionally, the paper attempts to recover useful information from the suspected erroneous labeled samples and does not discard them. Therefore, when it is difficult to obtain additional preference annotations, this method has good practical application value. 2.The improvement effect of the report was observed across three datasets and multiple noise levels. With a 40% label flipping rate, DACSDPO achieved the highest preference accuracy on all three datasets compared to the baseline methods, and the response hit rate also improved. These results indicate that this method still performs well even in the presence of significant synthetic label noise. The comparison in Table 2 of the filtering, correction, and complete methods also helps to evaluate the advantages brought by retaining and correcting suspicious pairs.

Critical weaknesses

1.Figures 2 and 3 show that there is a negative correlation between the reward difference and the estimated noise probability. However, this correlation alone cannot determine that pairs below the selected threshold can be reliably regarded as incorrect annotations. The inconsistency between the current model and the annotations may stem from annotation noise or the limitations of the model itself. The authors should clarify how the adaptive threshold distinguishes between these two situations and explain why reversing the selected preference can correct the annotation errors rather than reinforcing the existing errors of the model. 2.Equation (11) uses the previous reward difference as the denominator. However, if the initialization strategy is based on the reference model, the initial reward difference is zero. The author should explain how to handle this situation. Algorithm 1 also uses selection or correction targets during the warm-up phase, although the introduction of the warm-up is to address the issue of unreliable early reward differences. It is necessary to clarify whether these operations are activated during the warm-up period and, if so, how to avoid premature correction. For Equation (12), the paper should explain how the per-sample gradient penalty is implemented, including whether the gradient is calculated separately for each sample and whether the penalty term is optimized through higher-order differentiation or an approximation method.

Questions for authors
  1. Figures 2 and 3 show a negative correlation between the reward difference and the estimated noise probability. However, the inconsistency between the annotation results might stem from label noise or the limitations of the model itself. Could the authors further explain what the basis is for reversing the pairing below the adaptive threshold? And how does this rule distinguish between incorrect annotations and correct annotation examples that the model has not yet learned?
  2. Equation (11) is divided by the previous reward difference, and when the strategy matches the reference model during initialization, this difference is zero. How should this be handled in this case? Algorithm 1 also uses selection or correction operations during the warm-up phase. Are these operations effective before the reward difference becomes reliable? Additionally, please further explain the calculation and derivation of the gradient penalty for each sample in Equation (12), including whether the gradient of individual examples is approximated and whether higher-order differentials are used.
3 comments

We sincerely thank the reviewer for the careful evaluation of our manuscript and the constructive comments.

Comment 1: How does the adaptive threshold distinguish noisy preferences from correct preferences that the model has not yet learned, so that reversing the selected preference does not reinforce model errors?

Response 1:

a. Human preferences that the model has not yet learned are assigned a stricter threshold τt\tau_t to avoid erroneous corrections. Specifically, when human preferences that the model has not yet learned are encountered, the model lacks the knowledge to distinguish between preferred and rejected responses, leading to a decrease in δt\delta_t and, consequently, a stricter threshold according to Eqs. (8) and (9) in the manuscript. Therefore, such correct human preferences are less likely to be incorrectly reversed by our method.

b. Introducing a small amount of noise during training is a common strategy for improving model robustness. We evaluate the precision and recall of preference correction during alignment on UltraFeedback with 40% noisy preferences. Throughout the correction stage, our method achieves an overall accuracy of 84.31%, precision of 80.64%, and recall of 80.01%, leaving only approximately 8% of preference pairs with uncorrected noise. Detailed results are presented in Section 4.3 and Fig. 4 of the updated manuscript. Although some erroneous corrections may still occur, they are insufficient to reinforce the model's existing errors and may even contribute to improving its robustness.

Comment 2: Details of the Warm-Up procedure and stability penalty

Response 2:

a. In practice, we add ε=10−6\varepsilon=10^{-6} to the denominator of Eq. (11) to ensure numerical stability. We have included this detail in the updated manuscript.

b. The model is trained with ℓDPO(θ)\ell_{\rm DPO}(\theta) during warm-up, and selection and correction are activated only after the warm-up stage. We have clarified this procedure in Algorithm 1 of the updated manuscript.

c. We clarify the implementation of the stability penalty. We compute the gradient for each sampled preference pair separately and optimize the stability penalty using a first-order finite-difference approximation following [1]. Specifically, for gi=∇θℓi(θ)g_i=\nabla_\theta\ell_i(\theta) and vi=gi∥gi∥2v_i=\frac{g_i}{\|g_i\|_2}, the gradient of the stability penalty is approximated as follows:

\approx 2w_t\lVert g_i\rVert_2 \frac{\nabla_\theta\ell_i(\theta+\epsilon v_i)-g_i}{\epsilon}.$$ We have included these implementation details for the finite-difference approximation of **Eq. (12) in the section 3.3 of updated manuscript.** [1] Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Bias, ICML 2023.

Dear Reviewer,

Thank you for taking the time to review our work and provide feedback. As the discussion period approaches its conclusion, we would sincerely appreciate any additional comments on whether our revisions have adequately addressed your primary concerns. We would be happy to provide further clarification on any remaining issues.

Sincerely,

The Authors

Thank you for the responses. Some of my concerns have been addressed. I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.