acceptodds
Under review as a conference paper at ICLR 2027

Style-Debiased DPO: Updating LLM Knowledge with Factuality-Aware Synthetic Preference Data

Abstract

Continued pretraining (CPT) with data augmentation such as paraphrasing can store inside a large language model (LLM) the knowledge of a small source corpus, a collection of documents describing new knowledge. The stored knowledge, however, is not always retrieved correctly. We study the eliciting side rather than the storing side: we use preference optimization, which learns from pairs of a preferred (chosen) and a dispreferred (rejected) response, so that the model elicits its stored knowledge more accurately. One proposed approach takes the model's own erroneous response as rejected and the gold answer as chosen, so as to suppress the error. When the target knowledge is partially known, however, most of these rejected responses are factually correct. Using direct preference optimization (DPO) as the preference optimization method then pushes down rejected responses that contain correct knowledge and differ from the chosen answer only in style, such as length and wording. We propose style-debiased DPO (SD-DPO), which scores whether the rejected response of each pair is factually correct, inverts the preference of such pairs, and weights them so that the learning signal due to differences in style cancels out as a whole. We first test whether, on top of EntiGraph, a representative storing-side method that runs CPT on text synthesized from the corpus, our method adds accuracy efficiently. On QuALITY, the reading-comprehension QA benchmark on which EntiGraph was evaluated, SD-DPO exceeds a baseline we CPT on EntiGraph's synthetic data from the same base model and evaluate with the same procedure. The training tokens this requires are a few dozen times fewer than the additional CPT needed for the same gain. For knowledge updating, the main goal of this work, we use AToKE, a knowledge-editing benchmark for facts that change over time. There, SD-DPO reaches an overall accuracy of 0.982 and answers with the new or the old fact according to the queried period.

Primary area:
foundation or frontier models, including LLMs
Keywords:
preference optimization, direct preference optimization, DPO, knowledge updating, knowledge editing, continued pretraining, synthetic preference data, factuality, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_DhQYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper learns the use of model-generated "rejected answers" to construct DPO data for knowledge updating. The authors observe that many rejected answers are factually correct but differ in expression from the chosen gold answers. They propose style-debiased DPO (SD-DPO), which scores whether the rejected response of each pair is factually correct, inverts the preference of such pairs, and weights them so that the learning signal due to differences in style cancels out as a whole.

Critical strengths

The motivation behind the issue is clear and universal.

The method attempts to separate facts from style in a structured manner.

The baseline and mechanism ablation studies are extensive.

Critical weaknesses
  1. The theory covers only idealized initial gradients and does not fully explain the actual SD-DPO loss. The main motivation is about decompose the margin into Δ(F)\Delta^{(F)} and Δ(S)\Delta^{(S)}. But the actual loss have confidence weighting, group rescaling, knowledge-preservation hinge... Those add-on looks like engineering parts and not discussed in theory part. The performance cannot be explained by the motivation and theory.

  2. The baselines are not convincing. In Appendix, the length of Llama output increases from 6 to 272 after using SD-DPO. If accuracy is calculated based on whether the ground truth answer appears in the response, then it is very unfair to the baseline. Also, all baselines were tested using the same learning rate, and no hyperparameter experiments regarding the learning rate were conducted. Authors can not demonstrate that this learning rate was not the result of cherry-picking. Although the authors consider this issue a topic for future work, a fair comparison against baselines is a fundamental requirement.

  3. Although the authors promised to release the code, they did not provide an anonymous link; however, there are currently many common websites and tools available for sharing code. Furthermore, while this paper provides highly detailed parameters and prompts, it fails to present any training loss or gradient descent curves, making it challenging to substantiate the theory of "style-gradient cancellation."

Questions for authors

Please see Weaknesses above. I have no more questions. Thank you.

Reviewer_nrUQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a new method, SD-DPO, that can elicit stored knowledge from continual pre-training by separating knowledge differences from stylistic differences. It trains on preferences that differ on correct knowledge and inverts preferences with correct knowledge but different style, with the aim of reducing stylistic differences. It's evaluated against two datasets (QuALITY & AToKE) and seems to show performance increase with token efficiency.

Critical strengths
  1. Controlling for knowledge and style separately for preference data is a worthwhile direction to explore.
  2. The method (SD-DPO) seems to improve on both datasets, QuALITY & AToKE.
  3. The method seems to be token-efficient.
Critical weaknesses
  1. Improvement is modest against SamDPO on QuALITY, under 2 points for a single metric.
  2. There is no analysis of the surprising effectiveness of the SD-DPO method for AToKE, making the meaningfulness of the outcome questionable. Why does it improve largely on AToKE while being marginal on QuALITY?
  3. The discussion and conclusion are generally lacking, condensed into 1 section. The analysis of results is generally lacking overall.
  4. While learning rate ablations are appreciated, the mechanism ablations are incomplete, such as removing or reducing the inversion effect (Table 2). More thorough methodological ablations are needed, such as cFA & pFA ablations.
Questions for authors

Figure 2 is unclear. The horizontal axis seems to indicate the number of CPT tokens, while there's a counterintuitive number of tokens displayed (10.9M & 21.7M). The figures should be self-contained, and the caption should explain them.

Reviewer_c31tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the use of synthetic preference data for knowledge updating, where rejected responses may already be factually correct. It proposes SD-DPO, which distinguishes factual errors from stylistic differences and reverses the preference direction when both responses are factually correct. The paper provides a theoretical analysis of the proposed method and evaluates its effectiveness on QuALITY and AToKE.

Critical strengths
  1. The paper identifies a relevant problem with synthetic preference data, where rejected responses may already be factually correct. This provides a reasonable motivation for reconsidering how DPO is applied to knowledge updating.
  2. The proposed method is relatively simple, and the experiments demonstrate improvements over standard DPO on both QuALITY and AToKE. Although the method does not consistently outperform SFT, the results suggest that it is a promising approach to knowledge updating.
Critical weaknesses
  1. It is unclear whether reversing preferences between two factually correct responses is well justified. The theoretical guarantee holds only at initialization under strong assumptions, and it remains unclear whether preference inversion may interfere with factual learning during training.
  2. It is unclear whether cFA reliably distinguishes factual correctness from stylistic differences, as it also relies on lexical overlap. The substantial difference in the proportion of responses classified as factually incorrect by pFA and cFA (11% vs. 33%) raises concerns about the reliability of this distinction, particularly without independent validation.
Questions for authors
  1. The paper discusses tie training (Moya et al., ICML 2026) but does not include it as a baseline. Given the close relationship between the two methods, a direct comparison would be important to establish the contribution of SD-DPO.
  2. The paper emphasizes training-token efficiency, but SD-DPO produces substantially longer responses. How does its overall computational cost, including response generation and LLM-based judging, compare with that of the baselines?
1 comment

Dear reviewer, as the deadline is approaching, please check the rebuttal and consider participating in the discussion.

Reviewer_jRPPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies knowledge updating in large language models by distinguishing knowledge storage from knowledge elicitation. The authors observe that, after continued pretraining (CPT), many model-generated rejected responses in self-constructed preference datasets are already factually correct and differ from the chosen responses primarily in style. Standard DPO may therefore introduce undesirable stylistic biases while suppressing previously acquired knowledge. To address this issue, the authors propose Style-Debiased DPO (SD-DPO), which uses an LLM judge to identify factually equivalent preference pairs, reverses their preference direction, and applies a weighting scheme designed to cancel style-related gradients. The method is theoretically motivated by a decomposition of preference margins into factual and stylistic components. Experiments on QuALITY and AToKE demonstrate improvements over standard DPO and several alternative preference optimization methods, with promising results on temporal knowledge updating across multiple model families.

Critical strengths
  1. The paper provides an interesting perspective on knowledge updating by distinguishing knowledge storage from knowledge elicitation. The observation that a large proportion of self-generated rejected responses are factually correct is insightful and highlights an important limitation of applying standard DPO to partially acquired knowledge.

  2. SD-DPO introduces a simple and intuitive approach to mitigating style bias in preference optimization. Instead of discarding factually equivalent pairs, the method leverages them through preference inversion and a theoretically motivated weighting scheme. The analysis provides a useful explanation for how style-related gradients can be canceled while preserving factual learning signals.

  3. The experiments demonstrate promising improvements over CPT and several preference optimization baselines, including vanilla DPO, IPO, KTO, and SamPO. The ablation studies support the effectiveness of preference inversion, while the results on AToKE across Llama, Qwen, and Gemma suggest that the approach can improve both new-fact acquisition and historical-fact recall. The additional comparison with SFT provides useful insights into robustness under different question formats.

Critical weaknesses
  1. The theoretical analysis assumes that style gradients are approximately shared across samples, which may not hold given the diversity of model-generated responses. Moreover, the cancellation is only guaranteed at initialization. It would be helpful to empirically verify whether style gradients are reduced during training.

  2. The method heavily relies on LLM judgments to distinguish factual errors from stylistic differences. Incorrect judgments may lead to reinforcing wrong answers, especially when the judge only sees truncated source documents. A sensitivity analysis with noisy labels or different judges would strengthen the results.

  3. The advantage over SFT is not entirely clear. SFT achieves comparable or better accuracy under the original evaluation format, while SD-DPO's advantage under format changes is not consistent across models. More comprehensive comparisons would help clarify when SD-DPO is preferable.

  4. SD-DPO tends to produce overly long or repetitive responses, with 14% reaching the generation limit at the final checkpoint. This raises concerns about introducing new style biases. The efficiency comparison also overlooks the additional costs of data generation and LLM judging.

Questions for authors

see weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.