- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper constructs preference-tuning data from conversation logs from weak and strong models viewing them as rejected and chosen. The authors applies formatting normalization, sentence-level paraphrasing of both responses, and length balancing on both sides to rule out style difference and focus on content difference between the pair. The experiments train Phi-4 and Qwen-2.5-14B with DPO on 77K–11M preference pairs constructed from conversation logs, using strong-model logged answers as chosen and weak-model generations as rejected. Evaluation covers tasks on WildChat, and AlpacaEval/Arena-Hard prompts evaluated by a custom binary LLM-judge rubric. The ablation-like main experiment includes the starting models, SFT, raw DPO, and preprocessing ablations. The full pipeline improves Phi-4’s test set score from 61.9% to 69.7%.
Critical strengths
-
The motivation is significant as leveraging model capacity as label for multi-turn dialog could potentially save the cost of training corresponding reward models, the compute side saving is clear.
-
Detailed ablations: The paper examines message processing normalization, balancing, paraphrasing, rejection strength on their effect towards the alignment performance.
-
Generalization: The authors also conducted experiments on cross family settings, demonstrating the robustness of the method.
Critical weaknesses
-
Missing comparisons to preference pair construction under reward model The main experiment comparisons predominantly vary components of the proposed source-labeling pipeline and seems a bit more of a ablation study. The authors did not compare DPO with data curated under supervision of a real LLM reward model. This omission is central to the motivation. The method assumes that source identity supplies sufficiently reliable preferences; a reward model could offer another imperfect but potentially useful preference signal and allow a wider spectrum of model to supply preference data. More experiments are needed to demonstrate what are the strengths and weaknesses of capacity implicit labelling versus a real reward model like Skywork-Reward-V2-Qwen3-1.7B/4 or RLHFlow’s 8B. For example, the authors may use all the models from the main experiment as a model pool and randomly route a single question to two of the models, do the same pre-processing and then rank them with a reward model and investigate the performance of this type of DPO.
-
The computational motivation is insufficiently established. The computational motivation is insufficiently established. The pipeline generates rejected responses with a 3.8B model and paraphrases both answers with a 14B model. Decision models such as Jev and Laya return scores or preference decisions without autoregressive text generation, providing alternatives worth evaluating for preference labeling or filtering. Without measuring preprocessing GPU-hours and downstream performance for these alternatives, the paper does not establish an efficiency advantage for its judge-free approach.
-
The theory provides limited justification for the claimed mechanism The learnability discussion relies on stronger conditional-noise conditions than the stated marginal better-than-random assumption supplies. The main transformation proposition assumes source-information contraction and bounded quality drift, then derives a preference-gap perturbation bound. These statements provide conditional motivation, but do not establish that the implemented transformation selectively removes stylistic information, preserves difficult reasoning, or improves DPO generalization. Likewise, reduced source-classifier accuracy cannot distinguish removal of undesirable stylistic cues from loss of useful quality information. The authors acknowledge some of these limitations; the theoretical contribution should be assessed accordingly.
-
Semantic and reasoning preservation are insufficiently measured The paraphraser processes sentences independently, without the full reasoning chain or dialogue context. This can plausibly alter logical conditions, cross-sentence references, or technical details. It may also correct rejected answers while leaving their labels unchanged.
My main concern is that the proposed labeling scheme assigns preferences based on differences in model size and overall capability, rather than the quality of individual responses. This may provide supervision that is too coarse for long-horizon interactions and difficult reasoning tasks, where informative negatives may require subtle, context-dependent errors rather than a broad capability gap. The paper does not establish whether this scheme produces sufficiently informative negatives in these settings. Moreover, it does not compare against conventional reward-model labeling or filtering in terms of downstream performance and total computational cost. Consequently, the practical benefit of avoiding explicit preference evaluators remains unclear; the “unsupervised” formulation alone does not establish an advantage over these alternatives.
Questions for authors
As discussed in weakness