acceptodds
Under review as a conference paper at ICLR 2027

RLearner-LLM: Balancing Logical Grounding and Fluency in Large Language Models via Hybrid Direct Preference Optimization

Abstract

Direct Preference Optimization (DPO) has emerged as an efficient alternative to PPO-based RLHF, yet its application to knowledge-intensive generation tasks reveals a fundamental limitation: standard preference signals—whether from human annotators or LLM judges—exhibit a systematic verbosity bias that rewards linguistic fluency over logical correctness. We show empirically that this reward signal blindspot leaves a large logical alignment gap: SFT-trained models achieve NLI entailment scores of only 0.05–0.22, despite producing fluent, confident-sounding text. We propose RLearner-LLM, a framework that resolves this through Hybrid-DPO: an automated preference construction pipeline that fuses a DeBERTa-v3 NLI entailment signal with a verifier LLM score, eliminating the need for human annotation while overcoming the "alignment tax" that degrades fluency under single-signal optimization. Evaluated across five academic domains (Biology, Medicine, Law) with three base architectures (LLaMA-2-13B, Qwen3-8B, and Gemma 4 E4B-it), RLearner-LLM achieves up to 6x NLI improvement over SFT baselines, with NLI gains in 11 of 15 (architecture, domain) cells and consistent answer-coverage gains. On Gemma 4 E4B-it—a 4.5B-effective-parameter dense model—Hybrid-DPO lifts NLI entailment in four of five domains (from +11.9% on Auckland Law to +2.4x on UK Medicine Year 2), with faster inference across all five, demonstrating that the approach scales down to compact modern base models without losing the alignment-tax mitigation. Our Qwen3-8B RLearner-LLM wins 95% of blind pairwise comparisons against its own SFT baseline, while GPT-4o-mini in turn wins 95% against our concise RLearner-LLM output—a result that, viewed alongside the 69% win rate the same judge awards a verbose SFT baseline over our DPO-aligned model, reproduces the verbosity bias we document on a frontier comparator and reinforces the case for logic-aware automatic metrics (NLI, ACR) over LLM-as-a-judge on knowledge-intensive generation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Model Alignment, Direct Preference Optimization, Preference Learning, Logical Grounding, Verbosity Bias

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7g7hOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper focuses on an issue that a verbosity bias might be introduced from human annotators or LLM judges during DPO training, making rewards fluency over logical correctness. This paper proposes RLearner-LLM and its Hybrid-DPO. The proposed method is an automated perference-construction pipeline, without human annotations. The authors conducted experiments on several academic domains using three base LLMs, and achieved promising results. The results show clear improvements upon SFT baselines. The authors also claim that the method is robust to question-quality filtering and the method could introduce gain of reasoning quality.

Critical strengths
  1. The author points out the issue that the preference signal does not measure logical correctness. This phenomenon is new to me and has not been reviewed by previous works. In addition, the alignment tax is also a novel and meaningful concept.

  2. The experiments are extensive with 5 academic domains and 3 base models. Most results from these experiments show improvements from the proposed method.

Critical weaknesses
  1. The writing of the abstract is really bad. I cannot understand what actually RLearner-LLM and its Hybrid-DPO are by reading the abstract (even Introduction). The abstract spends more than 2/3 words to details the experimental results. This is not a good abstract. Please confirm that the abstract is not a LLM-summarized version.

  2. The related work 2.2 section discussed hallucination problem. But it's not clear how hallucination can be addressed by the proposed method.

  3. Could the authors clarify some concept like, NLI entailment, before using this?

  4. The Hybrid Reward uses 0.5 and 0.5 weights. Why does these two factor weights equals? Are there any ablations on this point?

  5. In the limitation (i), the method is surpassed by ILearner-LLM in some cases. Could the author provide reasons on this?

Questions for authors
  1. What the "R" in "RLearner-LLM" means?
3 comments

Thank you. Your presentation comments shaped the rewrite.

In short. The abstract is rewritten, method first; "R" is reinforcement learning (done with DPO); NLI is defined at first use; §2.2 no longer claims to address hallucination; the 0.5/0.5 weight is ablated; ILearner-LLM's Law lead disappears after an evaluation fix; and the alignment tax is now measured.

Abstract. Rewritten from scratch, and written by us. It now opens with what the method does: "For each multiple-choice question we sample several explanations from a fine-tuned model and score each with a natural-language-inference (NLI) model: the explanation is the premise, the correct option the hypothesis, and the score is the probability of entailment. We then train with DPO on the highest- versus lowest-scoring pair." Results follow in three sentences, one per finding; the abstract is now 253 words, of which the method and the masking test take the first 98.

Readability. Besides the abstract: §3 follows the pipeline in order (scores, pairs, training), Fig. 2 is a five-box left-to-right diagram of it, the reward formulas are in one table, and Fig. 1 shows every result for every model in one place.

What does the "R" mean? Reinforcement learning, in the RLHF sense of learning from preferences; we implement it with DPO, which needs no RL loop. ILearner-LLM (Bao et al., 2025) refined explanations iteratively; RLearner-LLM replaces that with preference optimisation. This is now stated in §1.

Defining NLI. Added at first use (§1): an NLI model gives the probability that a premise entails a hypothesis. Here the premise is the explanation, and the hypothesis is the correct option.

Hallucination (§2.2). You are right that the method does not address hallucination. The section is rewritten around entailment and faithfulness rewards. Hallucination now appears only as the measured rate of fabricated URLs in SFT outputs.

Why 0.5/0.5? Ablation? It was a default, not a tuned value. We now ablate it: H_A with NLI weight 0.25, 0.5 and 0.75, plus the endpoints (verifier-only = 0, NLI-only = 1), all trained on pairs from the same Gemma 4 candidate pool. Masked held-out NLI: verifier-only 0.086 (99 words), w=0.25 0.087 (66 words), w=0.5 0.103, w=0.75 0.100, NLI-only 0.133 (53 words); SFT 0.062 (55 words). The gain rises up to w=0.5, is flat at 0.75, and is largest for NLI-only, while length falls; so 0.5 was a reasonable but not optimal choice. The pairs explain why: the verifier rates most candidates alike (Appendix C), so its weight mostly adds noise and a length bias.

Why does ILearner-LLM win on Law (limitation i)? With the corrected LLaMA-2 evaluation (General response) it no longer clearly does: single-pass RLearner DPO reaches NLI 0.415 on Law (SFT 0.253, 95% CI of the change [+0.06, +0.27]) against 0.400 for ILearner-LLM, and is ahead on the other four courses too (e.g. Med Y1 0.262 vs 0.122). The Law margin is small and the two come from separate runs, so we claim parity there, with one generation pass instead of ILearner-LLM's five refinements. The Law gain also survives masking (0.259→0.368).

The alignment tax. You found this concept useful, so we measured it directly. The tax we predicted (NLI-only training degenerating into short, incoherent text) did not appear on Gemma 4 or LLaMA-2, where NLI-only outputs keep their length (Gemma 4 55→53 words); on Qwen3 they halve (58→28). A different tax does appear: on five courses, three LLM judges rate the NLI-only policy's reasoning below SFT's (39–42% of non-tied comparisons), although its entailment is the highest. The same holds on corrected LLaMA-2 (39–42%) and Qwen3 (31–41%). The NLI policy writes shorter explanations there, which judge length bias penalises, but on items within 20 words the rate is still 40% (LLaMA-2, p=0.008) and 38% (Qwen3); on Gemma 4 the gap is mostly length. On the student test (a small model answering from the masked explanation) it costs nothing, however. Adding a reasoning-judge score to the NLI score removes this cost (judged level with SFT, preferred to NLI-only by 59–60%) while keeping a significant entailment gain, on Gemma 4 and, in a pre-specified replication, on LLaMA-2 (preferred to NLI-only by 52–58% over three seeds). On Qwen3 it removes the cost too (three seeds: NLI-only 24–45% vs SFT; judge+NLI 49–56%). The revision describes the tax in these measured terms.

Novelty of the problem. Thank you for finding the verbosity issue new. The revision adds a concrete tool and finding: a cheap masking test tells whether an entailment gain is answer restatement, and on two of three models it shows the gain is genuine support.

We hope the rewritten abstract and introduction make the method clear. We would be grateful if you would reconsider your presentation and overall scores.

Thanks for your detailed reply. Most of my concerns have been resolved. I improved the rate to weak acceptance. Thanks.

Thank you for reading our response and the revision, and for raising your score. Your comments on the abstract, the figures and the alignment tax made the paper much easier to follow, and we will keep improving the presentation in the final version.

Reviewer_ew1QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that DPO preference signals reward fluency over logical correctness in educational explanation generation. Hybrid-DPO builds preference pairs from an NLI entailment score (explanation as premise, correct option as hypothesis) and an Alpaca-7B verifier score, combined additively (HAH_{A}) or multiplicatively with an ACR gate and length penalty (HMH_{M}). Experiments cover five PeerWise domains and three base models, reporting NLI gains over SFT in 11 of 15 cells (up to 6.6×), agreement of a held-out NLI scorer in 12 of 15 cells, robustness to question-quality filters, a separate SimPO experiment, and GPT-4o-mini comparisons read as verbosity bias.

Critical strengths

S1. Unfavourable results are disclosed. The selection rule is labelled "post-hoc rationalisation" (§3.1, l.241), circular use of the training scorer is acknowledged (§6.4(ii)), and Tables 2-6 list every metric, including those that drop. Why this matters to my assessment: most concerns below can be checked against the paper's own tables.

S2. The Gemma 4 held-out check re-scores the same generations. DeBERTa-v3-large agrees in direction on all five Gemma 4 domains and both Tier-C corpora (Table 12; caption, l.941-942, l.952). Why this matters to my assessment: it is the best evidence that the entailment gain is not scorer-specific.

Critical weaknesses

W1. Lack of novelty, and the paper's own ablations do not support the dual signal as the load-bearing design. The authors write that "entailment-as-reward has precedent in summarization faithfulness (Pasunuru & Bansal, 2018; Roit et al., 2023), which we adapt to DPO preference construction" (§2.1, l.147-148), and that the objective "follows the standard DPO formulation" (§3.3, l.257); the verifier comes from Bao et al. (2025) (§4.3, l.316). What remains is the claim that the combination is necessary (§3.1, l.223, l.245), which is not isolated:

  • Verifier-only DPO v2 (no NLI signal, §5, l.322-323) reaches 88.5% (Cardiff) and 74.0% (Sydney) of RLearner's LLaMA-2 gain over SFT (Tables 2-3).
  • In the only cell with all four variants (Qwen3 Cardiff, §3.1, l.243-245; Table 16), every variant is below SFT 0.1959.
  • The "alignment tax" of NLI-only training (§1, l.67-69) is never measured; the one NLI-only run (Llama-3-8B, §6.2, l.422-426) improves NLI and ACR.
  • §3.3 (l.262-265) says every chosen example wins "on both dual-signal axes ... regardless of which Hybrid variant". HAH_{A} admits pairs "when either signal is sufficiently better" (§3.1, l.231-232), and §6.1 (l.403-405) reports "indistinguishable verifier scores".

Why this matters to my assessment: without a clean single- versus dual-signal comparison, the contribution reduces to DPO with an existing entailment reward.

W2. The evaluation does not support the headline claims.

  • The main metric is the training scorer (§4.3, l.314-316), and the authors note that "an explanation can entail the correct option by restating it" (§6.3, l.430-431). "Logical Grounding" in the title and "structured proof" (§5.5, l.395) go beyond what is measured.
  • Table 1 and Table 12 are different runs for LLaMA-2 and Qwen3 (l.950-952); LLaMA-2 SFT NLI differs between them (Med Y2 0.2319 vs 0.046) without explanation. No seeds or intervals are reported.
  • §5.2 (l.356-357) calls Law and Medicine zero-shot, yet Gemma 4 trains on a "5-domain merged pool" (§4.2, l.307; Table 9).
  • The headroom correlation (§6.2, l.414-416) is not mechanistic evidence: in Table 1, SFT and post-DPO NLI are uncorrelated (r≈0.00); randomly re-pairing them yields mean correlations near −0.73 and −0.80, the reported values.
  • Summaries contradict tables: Table 16's mean HM−HAH_{M}-H_{A} is +0.89 pp, not +1.5 pp; "all three architectures improve NLI" on Medicine (§5.2, l.361) fails for Qwen3 Med Y1 (0.2457→0.2104); "stronger base models" in the abstract's ACR claim is undefined, and Gemma 4 ACR drops in four of five domains.
Questions for authors
  1. Could you please redraw figure 1 and figure 2. I's hard for readers to understand. And the current version is not suitable fore publiacation

  2. Please map every row of Tables 1, 12, 16 and Appendix A to its SFT checkpoint, preference pool, reward variant and test set. Why do the LLaMA-2 SFT values differ between Tables 1 and 12?

  3. Which domains, and how many questions per domain, enter each preference pool? Does the Gemma 4 pool contain Law or Medicine questions?

  4. How often does an explanation contain the correct-option text verbatim, before and after DPO, and how does NLI change if that text is masked?

  5. For Table 17, what are the output lengths, and how do two "averaged" verdicts (Appendix J, l.991-992) yield integer counts with zero ties? Row 3 has GPT-4o-mini judging its own outputs.

References with something wrong:

  1. "Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. RLAIF: Scaling reinforcement learning from human feedback with AI feedback. In International Conference on Machine Learning (ICML), 2024. arXiv:2309.00267." The ICML 2024 version is titled "RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback"; the cited title is that of an earlier arXiv version.
5 comments
AuthorsOpenReview ↗

Response to Reviewer ew1Q (1/2)

Thank you for the most careful review we have received. We checked every point against our files, and you were right on each. The revision is built around your two central points.

In short. (1) Restatement (W2.1/Q4): we ran your masking test. On LLaMA-2 and Gemma 4 the masked gain survives in 10/10 cells with no more copying; on Qwen3 nothing changes reliably. (2) Dual signal (W1): on one shared pool the verifier adds nothing; a reasoning-judge score adds judged reasoning on all three models. The necessity claim is withdrawn. (3) Tables 1 vs 12, zero-shot, headroom, summaries: all corrected (part 2).

W2.1 / Q4 — restating the answer. We ran exactly the test you proposed (revised §4, Table 2, Fig. 1). For each cell, the share of explanations that contain the correct-option text, before → after DPO, and held-out NLI with the answer's key terms masked:

  • LLaMA-2 (corrected evaluation, see General response): copying does not rise (−6 to +2 points), outputs get shorter in 4/5, and masked held-out NLI goes up in 5/5 test sets, each with a CI above zero (e.g. Med Y1 0.136→0.289).
  • Gemma 4: copying does not rise (−15 to +2 points), outputs get shorter (by 1–16 words), and masked held-out NLI goes up in 5/5 (e.g. Med Y1 0.042→0.106).
  • Qwen3 (corrected): NLI rises on Cardiff and Sydney (held-out scorer: Cardiff only); no masked change has a CI excluding zero; outputs get shorter in 4/5 (the earlier "triple in length" was the evaluation bug).

We also turned the test into a reward: masked NLI, gated on answer coverage (ACR), which gives no credit for restating the answer. It works as intended but is not better than plain NLI at the same size: only 93 pairs pass the gate, and masked-NLI DPO (0.084 ± 0.007, 3 seeds) matches NLI-DPO on 93 random pairs (0.083 ± 0.010). Reasoning judges prefer its outputs to full NLI-DPO's (57% / 54% / 57%), so it trades entailment for judged quality. On Gemma 4 plain NLI already does not restate (masked score 0.062→0.133, copies 52%→47%). "Logical Grounding" is gone from the title, "structured proof" from §5.5.

W1 — novelty. We agree that entailment rewards and DPO are not new, and the revision no longer claims they are. What we think is new is what happens when an entailment reward is used to train an explainer: (a) a gain in entailment may be answer restatement, which the usual NLI score cannot see; (b) a masking test, needing only the option text and one extra NLI pass, separates restatement from support (related to rationale-leakage metrics such as LAS and REV, which we now cite, but applied to auditing a training reward); and (c) the test shows the gain is real support on two of three models, yet entailment and judged reasoning come apart on all three, so the second signal has to measure reasoning. You also noted that all four variants are below SFT on Qwen3 Cardiff; we agree, and the revision states that Qwen3 shows no reliable change under any reward.

W1 — is the dual signal load-bearing? Our earlier single- vs dual-signal runs (Table 9) used separately sampled candidates and the buggy LLaMA-2/Qwen3 evaluation, so we no longer draw conclusions from them. Instead we added the clean ablation on one shared Gemma 4 pool (all variants, same candidates; new Table 3). NLI-only is the best entailment reward (held-out 0.307→0.473, masked 0.062→0.133); H_A gains grow with the NLI weight; verifier-only lowers NLI and doubles length. So the verifier is not load-bearing for entailment. What the second signal must supply is reasoning quality, which entailment does not measure: three judges prefer SFT to NLI-DPO on all three base models (Gemma 4 39–42%, LLaMA-2 39–42%, Qwen3 31–41%). Summing NLI with a reasoning-judge score on length-matched pairs fixes this: the masked entailment gain stays significant (0.062→0.096), judges rate it level with SFT and prefer it to NLI-DPO (59–60%, all three judges). A pre-specified replication on LLaMA-2 (three seeds) gives the same result with smaller margins (preferred to NLI-only in all 9 seed×judge cells, 52–58%); on Qwen3 (three seeds), judge+NLI is preferred to NLI-only by 58–78% but, like NLI-DPO there, has no masked gain. Claude Opus 5.5 agrees: judge+NLI beats NLI-only (56%, 58%) and is level with SFT (46–53%, none significant). So a dual signal is load-bearing, but the second signal has to measure reasoning; the verifier does not. Two caveats. The verifier ablation is on Gemma 4 only and uses a reproduction of the lost verifier (Pearson 0.73 with the original's scores); we have no equally clean ablation on LLaMA-2. And judge+NLI is a trade-off, not a free gain: its masked gain is about half of NLI-only's (Gemma 4 +0.034 vs +0.071), and its judged reasoning equals SFT rather than exceeding it. It keeps an entailment gain without the reasoning cost.

AuthorsOpenReview ↗

Response to Reviewer ew1Q (2/2)

W1 (cont.) — an LLM judge as a reward. This may look at odds with our motivation (judge labels favour length). We found the same bias in it: unmatched judge pairs won 60–61% through longer outputs. Restricting pairs to |Δ| ≤ 15 words removes it, so a judge reward is usable when pairs are length-matched. We withdraw the claims that the NLI+verifier combination is necessary and that NLI-only training degenerates, and revise the contribution accordingly.

W2.2 / Q2 — Tables 1 vs 12, seeds, intervals. LLaMA-2 decodes by sampling (T=0.5; Qwen3 and Gemma 4 decode greedily). The same LLaMA-2 SFT checkpoint therefore ranges, e.g., 0.045–0.21 NLI on Cardiff over 11 runs, and Table 1 had paired some DPO rows with SFT rows from other runs. Med Y2's 0.2319 matches none of 9 runs (max 0.123). The revised Table 2 reads SFT and DPO from the same evaluation file for every cell, adds 95% bootstrap CIs, and puts the held-out scorer on those same generations. Revised Appendix A (Table 5) maps every row to its SFT checkpoint, preference pool, reward variant, DPO adapter and evaluation file. With same-run pairing (and the corrected LLaMA-2/Qwen3 evaluation), the CI excludes zero in 8 of 15 cells and the two scorers agree on direction in 15 of 15. We have one DPO seed per cell, stated as a limitation; Cardiff SimPO has three seeds (76.4 ± 0.8%), and the LLaMA-2 judge+NLI and NLI-only SimPO policies three training seeds each (masked gain +0.026 to +0.050 and +0.042 to +0.066; judge+NLI preferred to NLI-only in all 9 seed×judge cells), as do the Qwen3 ones. On the shared Gemma 4 pool we now also run three DPO seeds for H_M (masked held-out 0.086 ± 0.006), masked NLI (0.084 ± 0.007) and the 93-pair NLI control (0.083 ± 0.010); all seed means are above SFT (0.062). Because LLaMA-2 samples, we also re-ran its SFT and H_A DPO evaluation with two more sampling seeds (each pair from one run): the masked held-out gain is positive in all 15 (seed, course) cells, CI above zero in 13, pooled +0.109 / +0.112 / +0.089 (each CI above zero), with outputs 60–68 words shorter each time.

W2.3 / Q3 — zero-shot and pool composition. You are right, and it applies to all three models. Every pool contains Law questions: LLaMA-2 has 77 Law, 24 Sydney and 137 from Cardiff or UK Medicine (the two courses share question text in our files, so we count them together); Gemma 4 has 29 / 20 / 113 (Table 8). No pool question appears in any test set. "Zero-shot" is removed.

W2.4 — headroom correlation. Withdrawn. We reproduced your argument: r(SFT, post) = −0.10 (p = 0.71), and randomly re-pairing SFT and post-DPO values gives a mean r(SFT, gain) of −0.75, close to the submitted −0.81 (which used relative gains).

W2.5 — summaries vs tables. All fixed: the mean is +0.89 pp (and two of the H_M values were one run listed twice); the claim that Qwen3 improves on Med Y1 is removed; the claim of ACR gains on stronger base models is removed.

Q1 — Figures. Fig. 1 is now a data figure: for each model, masked entailment, three LLM judges and the student test side by side; the per-cell held-out vs. masked plot is Fig. 3. Fig. 2 is a five-box left-to-right pipeline, with the reward formulas moved to a table.

Q5 — Table 17. Each item was judged once with a random order, not twice and averaged. Our description was wrong, and the judge never answered "Tie". Lengths: 234 vs 88, 59 vs 183, 183 vs 237 words; the longer output wins every row, and Row 3 is self-judging, as you note. The revision presents the table only as length bias and draws no quality conclusion (its DPO outputs also come from the buggy evaluation, which the table now states).

Reference. RLAIF now cites its ICML 2024 title.

The paper is narrower than the submission but, we believe, sound, and much of that is due to your review. We would be grateful if you would reconsider your score in light of the revision.

Dear Reviewer ew1Q, thank you again for your detailed review. We ran the answer-masking test you proposed (the masked gain holds in 10/10 LLaMA-2 and Gemma 4 cells; Qwen3 shows no reliable change), added a single- vs dual-signal ablation on one shared pool, and corrected every point you raised on Tables 1/12, zero-shot, the headroom correlation and the summaries. The revised PDF is uploaded. If anything is still unclear, we would be glad to answer before the discussion period ends on Nov 18.

Reviewer_enkhOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RLearner-LLM combines NLI and verifier scores to construct automatic preferences for educational explanations. Hybrid-DPO is evaluated across three model architectures and five academic datasets. A separate follow-up uses reasoning preferences and SimPO to study explanation quality under length controls.

Critical strengths

Section 3 directly addresses the trade-off between answer entailment and presentation quality. Table 1 compares SFT and preference training across multiple architectures and academic datasets. Appendix H adds held-out NLI scoring, while Appendices K/N provide multi-judge and human evaluation for the SimPO follow-up.

Critical weaknesses
  1. Section 3.3 claims that every chosen explanation improves both scores, but scalar reward ranking does not ensure componentwise dominance.
  2. For equal-length, ACR-passing candidates, (1,0.5) outranks (0.2,0.75) beyond both 0.05 thresholds while lowering the verifier score, contradicting that guarantee.
  3. Section 6.3 and Appendices K/N evaluate SimPO with a different reward, so its reasoning gains require separation from those of Hybrid-DPO.
Questions for authors
  1. Does the implementation apply an additional componentwise-dominance filter omitted from the paper?
  2. How often do chosen candidates in the existing preference data sacrifice one of the two scores?
  3. Do the existing reasoning evaluations in Section 6.3 include the original Hybrid-DPO policy as a comparator?
2 comments

Thank you. Your counterexample is correct and led us to measure exactly how pairs behave.

In short. (1) No dominance filter was used; the §3.3 claim is removed, and we report how often the chosen candidate loses on one score. (2) Your proposed filter, tested, does not help. (3) Hybrid-DPO is now in the reasoning comparison: judges prefer the reasoning reward to it (70%, 77%), so the §5.3 gain comes from that reward, not from SimPO or Hybrid-DPO.

Q1 — is there a dominance filter? No reported result uses one. The builder keeps the best- vs. worst-scoring candidate whenever the scalar gap is ≥ 0.05. (An experimental mode requiring 2-of-3 metric wins exists in the code but built no pool in the paper.) The §3.3 sentence is removed.

Q2 — how often does the chosen candidate lose on one score? In the four pools behind Table 1 (revised Appendix C, Table 8):

Poolpairschosen has lower NLIchosen has lower verifierverifier tieboth higher
LLaMA-2, H_A2397.5%26.8%15.5%50.2%
Qwen3, H_M (5 candidates per question)1431.4%8.4%76.9%14.0%
Qwen3, H_A6327.3%3.5%63.4%25.2%
Gemma 4, H_M1641.2%34.1%36.0%29.9%

In the Gemma 4 pool chosen and rejected have mean verifier scores of 3.05 and 3.06. Pairs are effectively ranked by NLI, which also explains why the verifier adds little (below).

Your proposed fix, tested. We trained Gemma 4 on pairs where the chosen candidate is at least as good on both NLI and verifier, from the same candidate pool as every other variant. It does not help: the dominance filter leaves 117 of 159 pairs, and masked held-out NLI rises only 0.062→0.080 (CI [−0.002, +0.038]) against 0.086 ± 0.006 for H_M over three seeds. Requiring agreement mostly removes informative pairs, because the verifier barely separates candidates. (The original verifier checkpoint is no longer available; these runs use a reproduction trained on the same data, which matches its scores at Pearson 0.73 and reproduces the 35% disagreement rate; new Appendix.)

Q3 — is Hybrid-DPO a comparator in §6.3? It is now (revised §5.3, Table 4). Same SFT checkpoint, same 191 Cardiff test questions, same judge prompt and order flip, greedy decoding. These judges run locally in bf16, while §5.3's are served by ollama, so SimPO vs SFT differs slightly (74% here, 70% in §5.3 under Qwen2.5):

JudgeSimPO vs SFTHybrid-DPO vs SFTSimPO vs Hybrid-DPO
Qwen2.5-7B-Instruct74% (p=5e-11)51% (p=0.94)70% (p=2e-8)
Llama-3.1-8B-Instruct84% (p=1e-22)58% (p=0.02)77% (p=1e-14)

On the same outputs, Hybrid-DPO has the highest entailment (training-scorer NLI 0.39 vs 0.19 for SFT and 0.17 for SimPO), but it also copies the answer more (71% of outputs vs 53%) and writes longer (78 vs 66 words); its masked held-out NLI still rises (0.086→0.136). So the two rewards do different things, and the §5.3 gain comes from the reasoning reward. We now say so explicitly, and §5.3 no longer serves as evidence for the Hybrid reward.

To separate the objective from the reward: SimPO with the NLI reward (same pairs as NLI-DPO) is rated below SFT (39% / 36% / 31%), but its pairs are not length-matched and its outputs shrink to 29 words. The cleaner test is our replications on LLaMA-2 and Qwen3, where both policies use SimPO on the same pool and only the score differs: judge+NLI beats NLI-only in all 18 seed×judge cells (52–78%). SimPO by itself does not produce the reasoning gain; the reward does.

We hope this answers your concerns about the paper's soundness. If it does, we would be grateful if you would reconsider your score.

Dear Reviewer enkh, thank you again for your review. In short: no dominance filter was used and the §3.3 claim is removed; your proposed filter, tested on a shared candidate pool, does not help; and Hybrid-DPO is now included in the reasoning comparison (judges prefer the reasoning reward to it, 70% and 77%). The revised PDF is uploaded. If any concern remains, we would be glad to answer before the discussion period ends on Nov 18.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.