acceptodds
Under review as a conference paper at ICLR 2027

Domain-Grounded Counterfactual Augmentation for Mitigating Knowledge Conflicts in LLM Fine-Tuning

Abstract

Domain-specific fine-tuning of pre-trained LLMs exposes a persistent failure mode: existing parametric knowledge persists as a dominant inductive bias, so models produce confident, internally consistent predictions that remain inconsistent with the target domain even after supervised fine-tuning on it. Such failures are not fully captured by label noise or distributional shift in the conventional sense; they reflect a domain-conditioned knowledge-priority problem, in which generalizations valid under the original training distribution are overridden by more specific target-domain facts, which creates a need for targeted supervision and consistency across domain-preserving views. We introduce CIDA (Conflict-Immunized Data Augmentation), a framework that applies counterfactual invariance to this misalignment. CIDA identifies samples prone to conflict via distributional consistency under stochastic decoding, constructs knowledge-conditioned rewrites intended to vary prior-triggering surface cues while preserving the target answer, and optimizes a composite objective coupling conflict-aware sample weighting with an invariance regularizer on predictions over original and counterfactual pairs. Evaluated on an internal domain-specific dataset and public benchmarks covering diverse domains, CIDA outperforms competitive baselines from data filtering and sample reweighting paradigms. Gains concentrate on high-conflict samples, while low-conflict performance remains comparable to standard fine-tuning. Sample- and update-matched comparisons separate additional training exposure from the augmentation pipeline, while a same-data ablation identifies the additional benefit of paired invariance. Together, these comparisons support retrieval-grounded augmentation and prediction consistency as complementary contributors to the gains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Domain-Specific Fine-Tuning, Knowledge Conflicts, Counterfactual Data Augmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jC4HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies knowledge conflict in domain-adapted language models, focusing on cases where a model continues to produce stable predictions that disagree with the correct domain-specific answer after supervised fine-tuning. The proposed method, CIDA, assigns a continuous conflict score based on agreement across multiple generations and disagreement with the target answer. It generates intended answer-preserving rewrites conditioned on the target answer and retrieved domain evidence, filters the rewrites for quality, and trains jointly on the original and rewritten questions. A KL-based invariance loss encourages the model to produce similar output distributions for each pair.

The experiments cover four QA datasets and three 7B to 8B language models, all adapted using LoRA. CIDA consistently outperforms vanilla fine-tuning in the main experiments, with the largest gains on examples classified as high-conflict. The ablations indicate that these gains cannot be attributed solely to additional training examples or optimization steps. They also provide evidence that grounded rewriting and the invariance objective make separate contributions.

CIDA performs worse than vanilla fine-tuning on multi-hop and temporal QA. These results suggest that the method depends on whether the rewrite preserves both the original answer and the reasoning conditions needed to derive it.

Critical strengths

Stable target disagreement after domain fine-tuning

This paper addresses an important failure mode in domain adaptation. A model may produce stable and plausible answers based on pretrained knowledge even when those answers conflict with domain-specific ground truth. Such errors are especially relevant in settings where organization-specific rules, terminology, or conventions differ systematically from general knowledge. Distinguishing stable conflicting predictions from unstable errors provides a useful basis for targeted intervention instead of treating all errors in the same way.

Coherent conflict-targeted intervention

CIDA uses the conflict score for three linked decisions: selecting examples for rewriting, downweighting the original, and upweighting the retained rewrite. Rather than discarding high-conflict samples, it converts them into additional supervision through domain-grounded, answer-preserving rewrites. This method then encourages agreement between predictions for the original and rewritten questions, and combines data augmentation with an explicit constraint on model behavior.

Empirical improvements aligned with the intended mechanism

The empirical results are consistent with the intended purpose of the method. Improvements are concentrated on high-conflict examples, while performance on low-conflict examples changes little. This pattern suggests that CIDA is addressing the targeted failure mode rather than merely improving average task performance. The ablations further indicate that the gains are not explained only by additional data or optimization steps, and that both grounded rewriting and the invariance loss contribute to the result.

Critical weaknesses

The evidence does not support a broad claim of general knowledge conflict resolution

The experiments are limited to LoRA-based adaptation of 7B to 8B decoder-only models and primarily involve single-fact QA tasks for which answer-preserving rewriting is comparatively feasible. By contrast, performance drops on multi-hop and temporal QA. These negative results expose an assumption of CIDA: the rewritten question must preserve the answer and all reasoning-relevant conditions. The results therefore support a narrower claim than the paper may appear to make. CIDA is effective for the evaluated answer-preserving domain QA settings, but its applicability to tasks involving compositional reasoning, temporal conditions, or answer-changing semantic details remains unestablished.

The reliability and interpretation of the conflict-detection and rewriting pipeline

The conflict score is estimated from only three generations and depends on fixed choices of temperatures, sentence-embedding model, similarity function, and thresholds. The score operationalizes stable disagreement with the target; it does not independently establish that the disagreement is caused by pretrained general knowledge rather than another systematic error. The rewrite filters rely largely on embedding-based proxies, which cannot guarantee that the answer, entities, numerical constraints, temporal conditions, or reasoning structure remain unchanged. The independent answer-recovery and literal-overlap audits help address direct answer leakage, but they do not directly verify semantic or reasoning-condition preservation. The results on HotpotQA and TimeQA indicate that violations of the preservation assumption can have a substantial effect.

All experiments use LoRA. Although the CIDA objective is not formally tied to a particular parameterization, the experiments do not establish whether its gains extend to full fine-tuning or other PEFT methods. The observed improvements may reflect a general benefit of conflict-aware training, an interaction with LoRA's low-rank update constraint, or a combination of the two.

Questions for authors

Scope, answer preservation, and the negative results:

Is the intended scope of CIDA limited to tasks for which answer preservation can be reliably established? Please report a human or independent semantic audit of whether the rewrites preserve the answer and all answer-determining conditions, with separate results for negation, numerical values, temporal expressions, named entities, and conditional or multi-hop questions. For HotpotQA and TimeQA, can the authors disentangle rewrite errors, retrieval quality, conflict-sample selection, sample weighting, and the invariance loss, for example by evaluating CIDA only on independently verified answer-preserving pairs?

Conflict-score validity, sensitivity, and dataset-level effects:

How stable are the selected high-conflict examples under changes to the number of generations, decoding temperatures, sentence encoder, similarity threshold, and random seed? As the score measures stable target disagreement rather than directly identifying a pretrained-knowledge cause, can the authors validate that high-scoring cases actually reflect conflicts with parametric knowledge? Please test whether downstream gains are associated with conflict prevalence after accounting for dataset size, answer format, retrieval quality, and rewrite acceptance rate.

Generalization across adaptation and rewrite mechanisms:

Does CIDA provide comparable gains under full fine-tuning or another PEFT method, and when the rewrites are generated by a different language model or written by humans? Please report whether the gain remains under full fine-tuning and with rewrites produced by a different model. As the paper explicitly states that full KnowTuning requires a reference model and an external domain knowledge base unavailable for CreditQA, please clarify the practical resource gap and avoid interpreting the KnowTuning-Lite comparison as evidence of superiority over full KnowTuning.

Reviewer_ZovHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses knowledge conflicts that arise when fine-tuning pretrained LLMs on domain-specific data. Even after supervised fine-tuning, a model may continue to rely on its pretrained knowledge and produce answers that conflict with the target domain. The authors propose CIDA, a framework that identifies potentially conflicting samples, generates counterfactual rewrites grounded in retrieved domain evidence, filters low-quality rewrites, and trains the model with conflict-aware sample weighting and a consistency loss between original and rewritten questions. Results on different datasets show the effectiveness of this method.

Critical strengths
  1. The studied problem is important and clearly motivated.

  2. The framework is reasonable and practical. The use of retrieval-grounded rewriting and quality filtering is sensible, given the risk that unconstrained paraphrasing may introduce semantic drift or misleading cues.

Critical weaknesses
  1. The methodological novelty is somewhat limited. CIDA combines several established techniques, including stochastic decoding for sample selection, retrieval-grounded LLM rewriting, data filtering, sample reweighting, and KL-based consistency regularization. The integration is reasonably motivated, but the individual components are not particularly novel. The theoretical analysis provides a local condition under which the consistency update can reduce task risk, but it does not establish that this condition is generally satisfied during training. As a result, the main contribution appears to be the design and empirical validation of a task-specific pipeline rather than a substantial algorithmic advance.

  2. The score is based on consistency across sampled outputs and similarity to the target answer. However, consistency does not necessarily indicate a strong incorrect prior, and embedding similarity may not reliably capture semantic correctness. The score also depends on decoding settings and similarity thresholds. More evidence on its calibration and sensitivity across models and domains would strengthen the motivation for using it to select samples and determine training weights.

  3. Consistency between an original question and its rewrite is useful only when the rewrite preserves all information relevant to the answer. This becomes difficult for multi-hop reasoning and temporal questions, where changing an intermediate entity, relation, or time qualifier can change the correct answer. Indeed, CIDA performs substantially worse than vanilla fine-tuning on HotpotQA and TimeQA. Although these results are reported as limitations, they raise questions about how broadly the proposed approach can be applied beyond relatively simple, answer-preserving QA tasks.

Questions for authors

See weakness.

1 comment
Reviewer_ZovHOpenReview ↗

Thanks for the response

The clarification is appreciated. After reading reviews from other fellow reviewers, I tend to keep my negative rating.

Reviewer_YktmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address what the authors call "stable disagreement" in domain-adapted large language models (LLMs), i.e. cases where the correct answer to a common question changes because of the domain-specific context, even though the (now incorrect) general answer is already in the parametric knowledge of the LLM, so that the model is confidently and consistently incorrect. Specifically, they propose a method, CIDA (Conflict-Immunized Data Augmentation), to i) identify such cases, ii) rewrite them into augmented copies that make the domain context explicit, and finally iii) re-weigh them and their augmented copies during training to improve overall domain adaptation. More specifically, a question is flagged when three sampled answers agree with each other but differ from the gold label, and the model itself rewrites it using the gold answer and three passages retrieved from the training corpus. During training the originals are down-weighted, the rewritten copies are up-weighted, and a KL term pushes the model to give the same answer distribution for each question and its rewrite. The main contribution is turning these conflicting samples into extra, grounded training examples, where earlier work filters or down-weights them, which they evaluate with LoRA on Qwen3-8B, Llama-3-8B and Mistral-7B over a private lending dataset (CreditQA), MedMCQA, PopQA and SciQ.

Critical strengths
  1. The method combines ideas from several directions, namely detecting conflicts by sampling the model, augmenting data with grounded rewrites, filtering the data for quality, reweighting samples, and consistency training with a KL-divergence training term, though the main gain seems to be using conflicting examples as extra training data instead of outright dropping or down-weighing them, which is a simple but useful idea.

  2. On that point, the experiments and the ablations are mostly careful, with a central ablation table at the core of the paper. Specifically, table 2 carefully ablated the various knobs of the approach to show the effectiveness of each component. The authors are also open about some of the limits of the paper, reporting losses on HotpotQA and TimeQA (Table A7).

Critical weaknesses
  1. Each component of the method adds another layer of ad-hoc selections, none of which are evaluated or optimized, making it difficult to determine which elements of the method actually contribute to performance. The conflict score evaluates the model’s greedy response versus the gold label using the cosine similarity of a mini-sentence encoder (all-MiniLM-L6-v2), which assesses how close in wording two sentences are (not their accuracy), thus, a greedy response can be very dissimilar to the gold label, yet share high word similarity because of a number, negation, etc. Further, the Conflict Score is halved when there is disagreement among the sampled responses and only questions with scores greater than or equal to 0.5 will be rewritten. Therefore, a question that is uncertain by the model can only qualify if its greedy response has a similarity score of 0 to the gold label. All of the fixed thresholds (i.e. 0.5 for conflicts, 0.7 for internal consistency, etc.), the fixed number of samples, the fixed training weights (-0.7 for original responses and +0.5 for rewritten responses) and the KL weight are defined within Table A4 with no sweep, and Appendix G.5 referenced for Threshold Sensitivity (L197) includes no evaluations. Additionally, the Quality Filter employs the same evaluation metric as the Conflict Score and never validates whether a rewritten response maintains the same correct answer. The proposed ablation studies (Table 2) evaluate removing entire components of the architecture (which is good), but do not remove the additional up- or down-weighting applied, and the ablation was only performed on Qwen. In order to validate the conflict detection algorithm, perhaps the authors can evaluate it directly via a measure of correctness (e.g. exact match for MCQs or entailment/judge models for free-text responses) and provide the frequency of questions passing the agreement check, being reduced by half and reaching .5, across each dataset. The authors could also perform sweeps over the primary thresholds and weights and include ablations where only the down-weighing and up-weighing is removed.

  2. While CIDA produces improvements relative to the strongest baselines used for evaluation, the size of those improvements are relatively modest and do not preclude other methods or even larger amounts of training data from explaining those improvements. On Qwen, CIDA outperforms KnowTuning Lite by 1.03 points on CreditQA and 1.53 points on MedMCQA; while on Llama and Mistral CIDA outperforms the best baseline by 1.2 to 2.4 points (with standard deviation values ranging from .5 to .9 and no significance testing provided). Importantly, CIDA requires from 1.19 to 1.59 times as many samples during training as compared to the Table 1 baselines, in addition to requiring more decoding and rewriting costs that are not explicitly stated (e.g. in Appendix H, only plain fine-tune is allocated budget), and KaFT (the most closely related work) is not provided as a baseline comparison for Llama or Mistral. Additionally, the 2.55 point improvement produced due to inclusion of the KL term (Table 2) is also difficult to interpret, as the rewrites are generated based upon knowing the correct answer for each question, and therefore linking each question to its corresponding rewrite may transfer some information regarding that answer to the model. Since the only control is pairing each question with an unrelated rewrite it cannot isolate this effect from more general benefits associated with consistency-based training (e.g. as in UDA [1] and R-Drop [2]). Therefore, the authors could run KaFT and KnowTuning Lite (or FT-Top) at CIDA’s sample and step budget on all three backbones, including a consistency control involving plain paraphrasing or employing R-Drop on the original questions, and providing significance tests or more seeds.

[1] Xie, Q., et al.: Unsupervised data augmentation for consistency training. NeurIPS (2020). https://arxiv.org/abs/1904.12848 [2] Liang, X., et al.: R-Drop: Regularized dropout for neural networks. NeurIPS (2021). https://arxiv.org/abs/2106.14448

Questions for authors
  1. The paper never describes how accuracy is computed.

  2. Appendix C.3 (training settings) is empty - please provide details.

  3. Several averages in Tables 1 and 2 (most CreditQA and SciQ values, and some PopQA values) cannot come from three runs scored right or wrong on the stated 237 or 1,000 evaluation items. Please give the number of correct answers and the number of items for each seed, so the numbers can be checked.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.