- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PACE constructs preference data by generating justifications conditioned on known correct and incorrect answers. It filters candidates, ranks them within each answer-conditioned set using pairwise judgments and Bradley–Terry aggregation, and uses the selected pairs for DPO. Approximately 5K pairs improve four backbone models across five negation-related benchmarks.
Critical strengths
- Consistent and sometimes substantial improvements. PACE improves all 20 model–benchmark combinations, with reported significance in 17. The Llama3-8B improvements on ScoNe and NeQA are particularly substantial (Table 1).
- A practical, documented data-construction contribution. The pipeline uses relatively small generators and includes filtering, cross-backbone evaluation, and a human quality check. These make the proposed resource useful beyond a single demonstration.
Critical weaknesses
- The necessity of unsound reasoning is not established. Removing incorrect-answer candidates also removes the explicit correct-versus-incorrect answer contrast. Thus, the ablation cannot distinguish learning from flawed reasoning from learning the answer labels. Matched answer-only DPO and SFT on the same correct responses are missing; these are central alternatives given the available gold labels.
- The positional-bias attribution is incomplete. More uniform winner-position histograms do not demonstrate invariance to reordering the same candidates. Moreover, order-consistency filtering precedes BT ranking, so the contributions of filtering, bidirectional comparisons, and aggregation remain entangled (Figure 2; Appendices D.2 and F.4).
Questions for authors
- Experimental clarification: How does PACE compare with answer-only DPO and SFT on the same correct answers/justifications, using matched instances and training budgets?
- Bias clarification: Were candidate identities randomized across positions? Can you report same-candidate permutation consistency and separate the effects of consistency filtering, bidirectional judging, and BT aggregation?
5 comments
Response to Q2 (The positional-bias attribution)
Q2. The positional-bias attribution is incomplete.
More uniform winner-position histograms do not demonstrate invariance to reordering the same candidates. Moreover, order-consistency filtering precedes BT ranking, so the contributions of filtering, bidirectional comparisons, and aggregation remain entangled (Figure 2; Appendices D.2 and F.4). Were candidate identities randomized across positions? Can you report same-candidate permutation consistency and separate the effects of consistency filtering, bidirectional judging, and BT aggregation?
Response to Q2: PACE Mitigates Positional Bias Under Candidate Reordering.
We thank the reviewer for the comments. We would like to clarify two aspects of our design and provide additional experimental evidence.
1. Permutation Consistency and Positional Bias
In our original direct-judging experiments, candidates followed a fixed order (Qwen/Llama/Ministral) rather than randomized positions. We agree that the observed preference could partly reflect generator quality rather than positional bias. To disentangle these factors, we evaluated direct judging under all six permutations of the same three candidates. The table below reports the top-1 selection rate (%) for each response position (1st/2nd/3rd), rather than generator identity.
| Candidate Order | 1st Selected | 2nd Selected | 3rd Selected |
|---|---|---|---|
| Qwen / Llama / Ministral | 69.96% | 6.81% | 23.23% |
| Llama / Qwen / Ministral | 42.90% | 28.92% | 28.18% |
| Qwen / Ministral / Llama | 69.67% | 23.30% | 7.03% |
| Llama / Ministral / Qwen | 41.51% | 35.77% | 22.72% |
| Ministral / Qwen / Llama | 72.98% | 21.10% | 5.92% |
| Ministral / Llama / Qwen | 76.67% | 8.44% | 14.89% |
| Overall | 62.28% | 20.72% | 16.99% |
The first position is consistently preferred across all six permutations, regardless of which generator occupies it. This corroborates our findings in Figure 2 and provides direct evidence of positional bias in direct judging, rather than merely reflecting generator quality.
2. Component Attribution
PACE explicitly addresses order sensitivity (Section 3.1; Appendices D.2 and G). For each candidate set (A, B, C), we evaluate all six ordered response pairs (AB, BA, AC, CA, BC, CB), ensuring balanced exposure across both pairwise positions. We then fit separate BT models to forward and reverse judgments and average their scores.
Importantly, our comparisons use the same candidate pool and judge model, with only the preference extraction strategy varying (Section 4.3, Table 2). Our original positional-bias analysis (Figure 2) evaluates the benefit of the overall PACE pipeline rather than attributing bias reduction to any single component. The following experiments further clarify the roles of these components.
We further measured same-candidate permutation consistency, defined as the percentage of candidate sets for which the same response is selected across all six permutations. Forward-only pairwise judging uses one comparison per candidate pair, whereas bidirectional judging uses both orders.
| Setting | Direct Judging | Forward-only Pairwise + BT | Bidirectional Pairwise + BT |
|---|---|---|---|
| Before filtering | 19.22% | 45.37% | 88.25% |
| After filtering | 26.11% | 66.86% | 100.00% |
On the same filtered candidate pool, bidirectional pairwise judging with BT achieves 100% permutation consistency, compared with 66.86% for forward-only pairwise judging and 26.11% for direct judging. Consistency filtering further retains more order-stable examples, improving consistency across all three strategies.
To examine BT aggregation, we also compared it with simple vote counting. With all six judgments, vote counting produces tied top votes in 30.39% of cases. Whenever a unique winner exists, it agrees with BT.
Together, these results provide direct evidence of PACE's robustness to candidate reordering. They further show that bidirectional judging and consistency filtering are central to mitigating positional bias, while BT provides score-based aggregation of pairwise preferences.
Additional Response to Q1 (The necessity of unsound reasoning)
Additional Results on TIND.
As a follow-up, we further evaluated Answer-only DPO and SFT on TIND using the same four backbone models. The accuracy results (%) are compared with PACE below.
| Benchmark | Method | Llama-3B | Llama-8B | Qwen-3B | Qwen-7B |
|---|---|---|---|---|---|
| TIND | Off-the-shelf | 51.99 | 54.87 | 51.25 | 61.67 |
| SFT | 53.01 | 49.91 | 49.52 | 61.24 | |
| Answer-only DPO | 52.07 | 55.60 | 52.11 | 61.69 | |
| PACE | 53.63 | 64.51 | 55.52 | 62.12 |
Across three distinct downstream tasks (NLI, information retrieval, and commonsense reasoning), PACE consistently outperforms both baselines in all 12 model–benchmark settings. Overall, the previously observed trend (PACE > Answer-only DPO > Off-the-shelf > SFT) largely holds. These consistent results further strengthen our conclusion regarding the importance of contrasting sound and unsound reasoning in preference construction.
We thank the reviewer for this valuable suggestion and hope these additional results address any remaining concerns.
Response to Q1 (The necessity of unsound reasoning)
Response to Q1: Unsound Reasoning Provides Critical Preference Supervision.
We thank the reviewer for the suggestion. Our existing ablation (Table 1) demonstrates the importance of incorrect-answer-conditioned generation, as removing these candidates consistently weakens performance. We acknowledge, however, that this ablation does not fully isolate the contribution of unsound justifications from the answer-label contrast.
To address this concern, we conducted two additional experiments following the reviewer’s suggestions: (1) Answer-only DPO and (2) SFT on correct justifications. We trained all four backbone models and evaluated them on ScoNe and NevIR. The accuracy results (%) are shown below, alongside those of PACE-trained models.
PACE consistently outperforms both Answer-only DPO and SFT across all model–benchmark settings. We also observe a clear trend: Answer-only DPO consistently outperforms SFT, while SFT underperforms the off-the-shelf models in all eight settings. These findings further strengthen our central claim that contrasting sound and unsound reasoning provides critical preference supervision beyond answer labels or correct justifications alone.
We thank the reviewer for suggesting these additional experiments, which provide stronger empirical support for our findings.