- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper focuses on an issue that a verbosity bias might be introduced from human annotators or LLM judges during DPO training, making rewards fluency over logical correctness. This paper proposes RLearner-LLM and its Hybrid-DPO. The proposed method is an automated perference-construction pipeline, without human annotations. The authors conducted experiments on several academic domains using three base LLMs, and achieved promising results. The results show clear improvements upon SFT baselines. The authors also claim that the method is robust to question-quality filtering and the method could introduce gain of reasoning quality.
Critical strengths
-
The author points out the issue that the preference signal does not measure logical correctness. This phenomenon is new to me and has not been reviewed by previous works. In addition, the alignment tax is also a novel and meaningful concept.
-
The experiments are extensive with 5 academic domains and 3 base models. Most results from these experiments show improvements from the proposed method.
Critical weaknesses
-
The writing of the abstract is really bad. I cannot understand what actually RLearner-LLM and its Hybrid-DPO are by reading the abstract (even Introduction). The abstract spends more than 2/3 words to details the experimental results. This is not a good abstract. Please confirm that the abstract is not a LLM-summarized version.
-
The related work 2.2 section discussed hallucination problem. But it's not clear how hallucination can be addressed by the proposed method.
-
Could the authors clarify some concept like, NLI entailment, before using this?
-
The Hybrid Reward uses 0.5 and 0.5 weights. Why does these two factor weights equals? Are there any ablations on this point?
-
In the limitation (i), the method is surpassed by ILearner-LLM in some cases. Could the author provide reasons on this?
Questions for authors
- What the "R" in "RLearner-LLM" means?
3 comments
Thanks for your detailed reply. Most of my concerns have been resolved. I improved the rate to weak acceptance. Thanks.
Thank you for reading our response and the revision, and for raising your score. Your comments on the abstract, the figures and the alignment tax made the paper much easier to follow, and we will keep improving the presentation in the final version.
Thank you. Your presentation comments shaped the rewrite.
In short. The abstract is rewritten, method first; "R" is reinforcement learning (done with DPO); NLI is defined at first use; §2.2 no longer claims to address hallucination; the 0.5/0.5 weight is ablated; ILearner-LLM's Law lead disappears after an evaluation fix; and the alignment tax is now measured.
Abstract. Rewritten from scratch, and written by us. It now opens with what the method does: "For each multiple-choice question we sample several explanations from a fine-tuned model and score each with a natural-language-inference (NLI) model: the explanation is the premise, the correct option the hypothesis, and the score is the probability of entailment. We then train with DPO on the highest- versus lowest-scoring pair." Results follow in three sentences, one per finding; the abstract is now 253 words, of which the method and the masking test take the first 98.
Readability. Besides the abstract: §3 follows the pipeline in order (scores, pairs, training), Fig. 2 is a five-box left-to-right diagram of it, the reward formulas are in one table, and Fig. 1 shows every result for every model in one place.
What does the "R" mean? Reinforcement learning, in the RLHF sense of learning from preferences; we implement it with DPO, which needs no RL loop. ILearner-LLM (Bao et al., 2025) refined explanations iteratively; RLearner-LLM replaces that with preference optimisation. This is now stated in §1.
Defining NLI. Added at first use (§1): an NLI model gives the probability that a premise entails a hypothesis. Here the premise is the explanation, and the hypothesis is the correct option.
Hallucination (§2.2). You are right that the method does not address hallucination. The section is rewritten around entailment and faithfulness rewards. Hallucination now appears only as the measured rate of fabricated URLs in SFT outputs.
Why 0.5/0.5? Ablation? It was a default, not a tuned value. We now ablate it: H_A with NLI weight 0.25, 0.5 and 0.75, plus the endpoints (verifier-only = 0, NLI-only = 1), all trained on pairs from the same Gemma 4 candidate pool. Masked held-out NLI: verifier-only 0.086 (99 words), w=0.25 0.087 (66 words), w=0.5 0.103, w=0.75 0.100, NLI-only 0.133 (53 words); SFT 0.062 (55 words). The gain rises up to w=0.5, is flat at 0.75, and is largest for NLI-only, while length falls; so 0.5 was a reasonable but not optimal choice. The pairs explain why: the verifier rates most candidates alike (Appendix C), so its weight mostly adds noise and a length bias.
Why does ILearner-LLM win on Law (limitation i)? With the corrected LLaMA-2 evaluation (General response) it no longer clearly does: single-pass RLearner DPO reaches NLI 0.415 on Law (SFT 0.253, 95% CI of the change [+0.06, +0.27]) against 0.400 for ILearner-LLM, and is ahead on the other four courses too (e.g. Med Y1 0.262 vs 0.122). The Law margin is small and the two come from separate runs, so we claim parity there, with one generation pass instead of ILearner-LLM's five refinements. The Law gain also survives masking (0.259→0.368).
The alignment tax. You found this concept useful, so we measured it directly. The tax we predicted (NLI-only training degenerating into short, incoherent text) did not appear on Gemma 4 or LLaMA-2, where NLI-only outputs keep their length (Gemma 4 55→53 words); on Qwen3 they halve (58→28). A different tax does appear: on five courses, three LLM judges rate the NLI-only policy's reasoning below SFT's (39–42% of non-tied comparisons), although its entailment is the highest. The same holds on corrected LLaMA-2 (39–42%) and Qwen3 (31–41%). The NLI policy writes shorter explanations there, which judge length bias penalises, but on items within 20 words the rate is still 40% (LLaMA-2, p=0.008) and 38% (Qwen3); on Gemma 4 the gap is mostly length. On the student test (a small model answering from the masked explanation) it costs nothing, however. Adding a reasoning-judge score to the NLI score removes this cost (judged level with SFT, preferred to NLI-only by 59–60%) while keeping a significant entailment gain, on Gemma 4 and, in a pre-specified replication, on LLaMA-2 (preferred to NLI-only by 52–58% over three seeds). On Qwen3 it removes the cost too (three seeds: NLI-only 24–45% vs SFT; judge+NLI 49–56%). The revision describes the tax in these measured terms.
Novelty of the problem. Thank you for finding the verbosity issue new. The revision adds a concrete tool and finding: a cheap masking test tells whether an entailment gain is answer restatement, and on two of three models it shows the gain is genuine support.
We hope the rewritten abstract and introduction make the method clear. We would be grateful if you would reconsider your presentation and overall scores.