acceptodds
Under review as a conference paper at ICLR 2027

When Models Struggle with Negation: Learning from Sound and Unsound Reasoning

Abstract

Direct Preference Optimization (DPO) is only as effective as the preference data it learns from. As human annotation is expensive, synthetic preference pipelines increasingly rely on large language models (LLMs) to generate candidate responses and on LLM judges to identify preferred ones. Yet this pipeline implicitly assumes that the generator can already produce sufficiently reliable responses for the target capability. When models struggle with that capability, preference construction becomes circular: it presupposes the very capability that post-training is intended to improve. We study this problem through negation, a pervasive linguistic phenomenon that remains challenging for LLMs across multiple tasks. We introduce PACE, a framework that constructs preference data without requiring models to first solve the target problem reliably. Given known correct and incorrect answers, PACE elicits corresponding justifications that reflect contrasting sound and plausible but unsound reasoning. PACE further uses pairwise comparisons with Bradley-Terry aggregation to rank candidate justifications within each answer-conditioned set while mitigating positional bias. Across five negation-focused benchmarks spanning question answering, natural language inference, commonsense reasoning, and information retrieval, DPO training on approximately 5K PACE preference pairs yields gains across four Llama and Qwen backbone LLMs, with statistically significant improvements in 17 out of 20 model-benchmark settings. Controlled experiments show that removing justifications conditioned on incorrect answers consistently weakens performance, demonstrating that plausible but incorrect reasoning provides an important learning signal. Our results suggest that reliable preference construction is a critical prerequisite for effective preference optimization when models struggle with the target capability. To our knowledge, we release the first preference dataset for negation understanding, along with our implementation at https://anonymous.4open.science/r/NEG.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
natural language understanding, negation understanding, preference learning, synthetic preference data, direct preference optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_jZstOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PACE constructs preference data by generating justifications conditioned on known correct and incorrect answers. It filters candidates, ranks them within each answer-conditioned set using pairwise judgments and Bradley–Terry aggregation, and uses the selected pairs for DPO. Approximately 5K pairs improve four backbone models across five negation-related benchmarks.

Critical strengths
  1. Consistent and sometimes substantial improvements. PACE improves all 20 model–benchmark combinations, with reported significance in 17. The Llama3-8B improvements on ScoNe and NeQA are particularly substantial (Table 1).
  2. A practical, documented data-construction contribution. The pipeline uses relatively small generators and includes filtering, cross-backbone evaluation, and a human quality check. These make the proposed resource useful beyond a single demonstration.
Critical weaknesses
  1. The necessity of unsound reasoning is not established. Removing incorrect-answer candidates also removes the explicit correct-versus-incorrect answer contrast. Thus, the ablation cannot distinguish learning from flawed reasoning from learning the answer labels. Matched answer-only DPO and SFT on the same correct responses are missing; these are central alternatives given the available gold labels.
  2. The positional-bias attribution is incomplete. More uniform winner-position histograms do not demonstrate invariance to reordering the same candidates. Moreover, order-consistency filtering precedes BT ranking, so the contributions of filtering, bidirectional comparisons, and aggregation remain entangled (Figure 2; Appendices D.2 and F.4).
Questions for authors
  1. Experimental clarification: How does PACE compare with answer-only DPO and SFT on the same correct answers/justifications, using matched instances and training budgets?
  2. Bias clarification: Were candidate identities randomized across positions? Can you report same-candidate permutation consistency and separate the effects of consistency filtering, bidirectional judging, and BT aggregation?
5 comments
AuthorsOpenReview ↗

Response to Q1 (The necessity of unsound reasoning)

Q1. The necessity of unsound reasoning is not established.
Removing incorrect-answer candidates also removes the explicit correct-versus-incorrect answer contrast. Thus, the ablation cannot distinguish learning from flawed reasoning from learning the answer labels. Matched answer-only DPO and SFT on the same correct responses are missing; these are central alternatives given the available gold labels. How does PACE compare with answer-only DPO and SFT on the same correct answers/justifications, using matched instances and training budgets?

Response to Q1: Unsound Reasoning Provides Critical Preference Supervision.

We thank the reviewer for the suggestion. Our existing ablation (Table 1) demonstrates the importance of incorrect-answer-conditioned generation, as removing these candidates consistently weakens performance. We acknowledge, however, that this ablation does not fully isolate the contribution of unsound justifications from the answer-label contrast.

To address this concern, we conducted two additional experiments following the reviewer’s suggestions: (1) Answer-only DPO and (2) SFT on correct justifications. We trained all four backbone models and evaluated them on ScoNe and NevIR. The accuracy results (%) are shown below, alongside those of PACE-trained models.

BenchmarkMethodLlama-3BLlama-8BQwen-3BQwen-7B
ScoNeOff-the-shelf52.0861.1756.7571.33
SFT49.1759.7550.0071.17
Answer-only DPO51.5873.1758.8371.67
PACE60.3383.9269.1773.00
NevIROff-the-shelf12.654.6327.0440.93
SFT2.820.7211.6438.76
Answer-only DPO13.024.9927.2641.21
PACE13.3820.3929.1441.36

PACE consistently outperforms both Answer-only DPO and SFT across all model–benchmark settings. We also observe a clear trend: Answer-only DPO consistently outperforms SFT, while SFT underperforms the off-the-shelf models in all eight settings. These findings further strengthen our central claim that contrasting sound and unsound reasoning provides critical preference supervision beyond answer labels or correct justifications alone.

We thank the reviewer for suggesting these additional experiments, which provide stronger empirical support for our findings.

AuthorsOpenReview ↗

Response to Q2 (The positional-bias attribution)

Q2. The positional-bias attribution is incomplete.
More uniform winner-position histograms do not demonstrate invariance to reordering the same candidates. Moreover, order-consistency filtering precedes BT ranking, so the contributions of filtering, bidirectional comparisons, and aggregation remain entangled (Figure 2; Appendices D.2 and F.4). Were candidate identities randomized across positions? Can you report same-candidate permutation consistency and separate the effects of consistency filtering, bidirectional judging, and BT aggregation?

Response to Q2: PACE Mitigates Positional Bias Under Candidate Reordering.

We thank the reviewer for the comments. We would like to clarify two aspects of our design and provide additional experimental evidence.

1. Permutation Consistency and Positional Bias

In our original direct-judging experiments, candidates followed a fixed order (Qwen/Llama/Ministral) rather than randomized positions. We agree that the observed preference could partly reflect generator quality rather than positional bias. To disentangle these factors, we evaluated direct judging under all six permutations of the same three candidates. The table below reports the top-1 selection rate (%) for each response position (1st/2nd/3rd), rather than generator identity.

Candidate Order1st Selected2nd Selected3rd Selected
Qwen / Llama / Ministral69.96%6.81%23.23%
Llama / Qwen / Ministral42.90%28.92%28.18%
Qwen / Ministral / Llama69.67%23.30%7.03%
Llama / Ministral / Qwen41.51%35.77%22.72%
Ministral / Qwen / Llama72.98%21.10%5.92%
Ministral / Llama / Qwen76.67%8.44%14.89%
Overall62.28%20.72%16.99%

The first position is consistently preferred across all six permutations, regardless of which generator occupies it. This corroborates our findings in Figure 2 and provides direct evidence of positional bias in direct judging, rather than merely reflecting generator quality.


2. Component Attribution

PACE explicitly addresses order sensitivity (Section 3.1; Appendices D.2 and G). For each candidate set (A, B, C), we evaluate all six ordered response pairs (AB, BA, AC, CA, BC, CB), ensuring balanced exposure across both pairwise positions. We then fit separate BT models to forward and reverse judgments and average their scores.

Importantly, our comparisons use the same candidate pool and judge model, with only the preference extraction strategy varying (Section 4.3, Table 2). Our original positional-bias analysis (Figure 2) evaluates the benefit of the overall PACE pipeline rather than attributing bias reduction to any single component. The following experiments further clarify the roles of these components.

We further measured same-candidate permutation consistency, defined as the percentage of candidate sets for which the same response is selected across all six permutations. Forward-only pairwise judging uses one comparison per candidate pair, whereas bidirectional judging uses both orders.

SettingDirect JudgingForward-only Pairwise + BTBidirectional Pairwise + BT
Before filtering19.22%45.37%88.25%
After filtering26.11%66.86%100.00%

On the same filtered candidate pool, bidirectional pairwise judging with BT achieves 100% permutation consistency, compared with 66.86% for forward-only pairwise judging and 26.11% for direct judging. Consistency filtering further retains more order-stable examples, improving consistency across all three strategies.

To examine BT aggregation, we also compared it with simple vote counting. With all six judgments, vote counting produces tied top votes in 30.39% of cases. Whenever a unique winner exists, it agrees with BT.

Together, these results provide direct evidence of PACE's robustness to candidate reordering. They further show that bidirectional judging and consistency filtering are central to mitigating positional bias, while BT provides score-based aggregation of pairwise preferences.

AuthorsOpenReview ↗

Additional Response to Q1 (The necessity of unsound reasoning)

Additional Results on TIND.
As a follow-up, we further evaluated Answer-only DPO and SFT on TIND using the same four backbone models. The accuracy results (%) are compared with PACE below.

BenchmarkMethodLlama-3BLlama-8BQwen-3BQwen-7B
TINDOff-the-shelf51.9954.8751.2561.67
SFT53.0149.9149.5261.24
Answer-only DPO52.0755.6052.1161.69
PACE53.6364.5155.5262.12

Across three distinct downstream tasks (NLI, information retrieval, and commonsense reasoning), PACE consistently outperforms both baselines in all 12 model–benchmark settings. Overall, the previously observed trend (PACE > Answer-only DPO > Off-the-shelf > SFT) largely holds. These consistent results further strengthen our conclusion regarding the importance of contrasting sound and unsound reasoning in preference construction.

We thank the reviewer for this valuable suggestion and hope these additional results address any remaining concerns.

Reviewer_X8srOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PACE, a framework for constructing synthetic preference data without requiring LLMs to reliably solve the target task. PACE generates contrasting justifications conditioned on known correct and incorrect answers, and uses pairwise comparisons with Bradley-Terry aggregation to rank candidate justifications while mitigating positional bias. Experiments across five negation-focused benchmarks and four Llama and Qwen models show that DPO training on approximately 5K preference pairs consistently improves performance.

Critical strengths
  1. Extensive experiments: The paper evaluates multiple LLMs across diverse benchmarks in an interesting negation-focused setting.

  2. Important topic: The paper addresses an important challenge in constructing reliable preference data for LLM alignment.

Critical weaknesses
  1. Limited generalizability: Although PACE is motivated as a general framework for constructing preference data when models struggle with a target capability, all experiments focus on negation understanding. It remains unclear whether the proposed approach generalizes to other challenging capabilities, such as mathematical or complex reasoning.

  2. Limited exploration of online RL: The paper only evaluates PACE using offline DPO, leaving its applicability to online reinforcement learning unexplored. It would be interesting to investigate whether the constructed preference data can effectively train reward models and whether these reward models can provide useful learning signals for online RL methods.

Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Q1 (Limited generalizability)

Q1. Limited generalizability.
Although PACE is motivated as a general framework for constructing preference data when models struggle with a target capability, all experiments focus on negation understanding. It remains unclear whether the proposed approach generalizes to other challenging capabilities, such as mathematical or complex reasoning.

Response to Q1: Additional Experiments Show PACE's Applicability Beyond Negation.

We agree that evaluating PACE on other challenging capabilities, such as mathematical reasoning, is valuable. However, such experiments would extend our explicitly stated empirical scope rather than address a generalization claim made in the paper. We would like to clarify three points first:

  • Our paper explicitly defines its scope as negation understanding. This focus is clearly stated in the Title, Abstract (L17-18), and Contributions (L107). Moreover, we explicitly acknowledge the extension to other capabilities as future work in the Conclusion (L484-485) and Appendix A (L826-830).
  • PACE's core methodology is not inherently specific to negation. As described in Section 3.1, this mechanism does not rely on negation-specific rules and can potentially be applied to other tasks with known correct and incorrect answers.
  • Our experiments already demonstrate cross-task robustness. We evaluate PACE across five benchmarks spanning question answering, natural language inference, commonsense reasoning, and information retrieval. PACE improves performance across all 20 model–benchmark settings, with 17 statistically significant gains (Table 1). These results demonstrate that the benefits are not limited to a single task, benchmark, or model.

Additional Experiments on Mathematical Reasoning. Following the reviewer's suggestion, we applied the same PACE pipeline to construct preference data from three DeepMind Mathematics tasks, followed by DPO training on four LLMs. We then evaluated these models on DeepMind Math (Higher Complexity) and AWPNLI, an independent arithmetic word-problem inference benchmark. The relative accuracy improvements (%) over the off-the-shelf models are shown below.

BenchmarkLlama-3BLlama-8BQwen-3BQwen-7B
DeepMind Math+3.39%+0.96%+2.58%+1.59%
AWPNLI+1.30%+1.00%+2.07%+0.47%

PACE consistently improves all four models on both mathematical reasoning benchmarks. These results provide further empirical evidence that PACE's benefits extend beyond negation understanding to mathematical reasoning.

We thank the reviewer for suggesting these experiments, as we now have more evidence supporting our work.

AuthorsOpenReview ↗

Response to Q2 (Limited exploration of online RL)

Q2. Limited exploration of online RL.
The paper only evaluates PACE using offline DPO, leaving its applicability to online reinforcement learning unexplored. It would be interesting to investigate whether the constructed preference data can effectively train reward models and whether these reward models can provide useful learning signals for online RL methods.

Response to Q2: PACE Supports Reward Modeling and Online RL.

We appreciate the reviewer's suggestion.

  • We would like to clarify an important distinction: PACE focuses on preference data construction, not preference optimization. As explicitly stated in the Introduction (L88-99), Section 2 (L136-138), Section 3.1 (L189-244) and Appendix G (L1477-1478), PACE constructs preference data, while DPO serves only as the downstream optimizer.

  • We deliberately adopt DPO as a computationally lightweight optimizer that directly learns from preference pairs, allowing us to focus on the effectiveness of preference construction without the additional complexity of reward modeling and online RL (L131–138). Our experiments demonstrate improvements across all 20 model–benchmark settings (Table 1).

  • The value of preference construction should be assessed independently of whether every possible downstream optimization method has been explored. We agree that evaluating PACE through reward modeling and online RL would be valuable, but such experiments would extend our study to a different optimization setting rather than address the central preference-construction problem investigated in this work.

Additional Experiments on Online RL. Nevertheless, following the reviewer's suggestion, we trained reward models using the original PACE preference pairs and applied online GRPO to two computationally efficient 3B instruction-tuned backbones under consistent training settings.

We selected reward-model checkpoints based on development-set ranking accuracy, measuring how often the reward model scored the chosen response above the rejected response. During online GRPO, each policy generated four fresh responses per training question, scored by the frozen reward model. GRPO checkpoints were selected using a fixed development set.

We evaluated the GRPO-trained models against their off-the-shelf counterparts on ScoNe and CondaQA. The reward-model ranking accuracies and relative accuracy improvements (%) are shown below.

BackboneRM Ranking Acc.ScoNeCondaQA
Llama-3B82.94%+1.02%+2.20%
Qwen-3B83.33%+0.91%+0.88%

Both reward models achieve around 83% development ranking accuracy, and RM-guided GRPO consistently improves both backbones across both downstream benchmarks. We believe these results directly address the reviewer's two concerns: PACE preference data can effectively train reward models, which in turn provide useful learning signals for online RL.

AuthorsOpenReview ↗

Additional Response to Q1 (Limited generalizability)

Extension to Complex Reasoning.
Following the reviewer's suggestion, we further evaluated PACE on StrategyQA, a challenging benchmark requiring implicit multi-hop reasoning. We compared the same four models against their off-the-shelf counterparts under the same zero-shot evaluation protocol. The relative accuracy improvements (%) are shown below.

BenchmarkLlama-3BLlama-8BQwen-3BQwen-7B
StrategyQA+1.70%+0.24%+3.65%+3.27%

PACE improves all four models on StrategyQA (4/4 settings).
Taken together, these results demonstrate that PACE's benefits extend beyond negation understanding to other challenging capabilities, including mathematical and complex reasoning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.