acceptodds
Under review as a conference paper at ICLR 2027

Unsupervised Preference Alignment of LLMs from Multi-Turn Dialog Logs

Abstract

Multi-turn preference alignment is hindered by long-range context dependencies and costly human annotation. We present a fully unsupervised framework that constructs Direct Preference Optimization (DPO) data from production dialog logs without human labels or LLM judges during pair construction. Responses from a stronger deployed LLM system are chosen; weaker-model responses to the same context are rejected. This cross-model construction creates a source-model confound: preference-alignment methods, including DPO, can exploit *generative fingerprints*—model-specific stylistic and distributional patterns—instead of semantic quality. We formalize the confound and show that shared, sentence-level paraphrasing of both responses can reduce fingerprint separability while bounding semantic-signal drift; our pipeline combines normalization, length balancing, and both-side paraphrasing. Across 77K–11M pairs from  12M real multi-turn conversations, the regularized pipeline improves with scale, whereas standard DPO overfits at all scales. Phi-4 (14B) gains up to +7.8pp held out, and +6.2pp, +3.1pp, and +5.0pp on WildChat, AlpacaEval, and Arena-Hard prompts under our binary rubric. Using Phi-family preference data, Qwen-2.5-14B gains +6.7pp held out and +2.9pp on WildChat. Two additional judge families and a target-independent paraphraser control confirm the key held-out gain, and public WildChat data replicates the pipeline.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
LLM alignment, unsupervised preference learning, Direct Preference Optimization, multi-turn dialogue, shortcut learning.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5WBVOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper constructs preference-tuning data from conversation logs from weak and strong models viewing them as rejected and chosen. The authors applies formatting normalization, sentence-level paraphrasing of both responses, and length balancing on both sides to rule out style difference and focus on content difference between the pair. The experiments train Phi-4 and Qwen-2.5-14B with DPO on 77K–11M preference pairs constructed from conversation logs, using strong-model logged answers as chosen and weak-model generations as rejected. Evaluation covers tasks on WildChat, and AlpacaEval/Arena-Hard prompts evaluated by a custom binary LLM-judge rubric. The ablation-like main experiment includes the starting models, SFT, raw DPO, and preprocessing ablations. The full pipeline improves Phi-4’s test set score from 61.9% to 69.7%.

Critical strengths
  • The motivation is significant as leveraging model capacity as label for multi-turn dialog could potentially save the cost of training corresponding reward models, the compute side saving is clear.

  • Detailed ablations: The paper examines message processing normalization, balancing, paraphrasing, rejection strength on their effect towards the alignment performance.

  • Generalization: The authors also conducted experiments on cross family settings, demonstrating the robustness of the method.

Critical weaknesses
  1. Missing comparisons to preference pair construction under reward model The main experiment comparisons predominantly vary components of the proposed source-labeling pipeline and seems a bit more of a ablation study. The authors did not compare DPO with data curated under supervision of a real LLM reward model. This omission is central to the motivation. The method assumes that source identity supplies sufficiently reliable preferences; a reward model could offer another imperfect but potentially useful preference signal and allow a wider spectrum of model to supply preference data. More experiments are needed to demonstrate what are the strengths and weaknesses of capacity implicit labelling versus a real reward model like Skywork-Reward-V2-Qwen3-1.7B/4 or RLHFlow’s 8B. For example, the authors may use all the models from the main experiment as a model pool and randomly route a single question to two of the models, do the same pre-processing and then rank them with a reward model and investigate the performance of this type of DPO.

  2. The computational motivation is insufficiently established. The computational motivation is insufficiently established. The pipeline generates rejected responses with a 3.8B model and paraphrases both answers with a 14B model. Decision models such as Jev and Laya return scores or preference decisions without autoregressive text generation, providing alternatives worth evaluating for preference labeling or filtering. Without measuring preprocessing GPU-hours and downstream performance for these alternatives, the paper does not establish an efficiency advantage for its judge-free approach.

  3. The theory provides limited justification for the claimed mechanism The learnability discussion relies on stronger conditional-noise conditions than the stated marginal better-than-random assumption supplies. The main transformation proposition assumes source-information contraction and bounded quality drift, then derives a preference-gap perturbation bound. These statements provide conditional motivation, but do not establish that the implemented transformation selectively removes stylistic information, preserves difficult reasoning, or improves DPO generalization. Likewise, reduced source-classifier accuracy cannot distinguish removal of undesirable stylistic cues from loss of useful quality information. The authors acknowledge some of these limitations; the theoretical contribution should be assessed accordingly.

  4. Semantic and reasoning preservation are insufficiently measured The paraphraser processes sentences independently, without the full reasoning chain or dialogue context. This can plausibly alter logical conditions, cross-sentence references, or technical details. It may also correct rejected answers while leaving their labels unchanged.

My main concern is that the proposed labeling scheme assigns preferences based on differences in model size and overall capability, rather than the quality of individual responses. This may provide supervision that is too coarse for long-horizon interactions and difficult reasoning tasks, where informative negatives may require subtle, context-dependent errors rather than a broad capability gap. The paper does not establish whether this scheme produces sufficiently informative negatives in these settings. Moreover, it does not compare against conventional reward-model labeling or filtering in terms of downstream performance and total computational cost. Consequently, the practical benefit of avoiding explicit preference evaluators remains unclear; the “unsupervised” formulation alone does not establish an advantage over these alternatives.

Questions for authors

As discussed in weakness

Reviewer_vzhSOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

the paper builds dpo preference data from production multi-turn chat logs without human labels or llm judges: the final answer of a stronger deployed model becomes the chosen response, a weaker model's answer to the same context becomes the rejected one. the authors point out that this couples the label to model identity, so dpo can learn source-model writing style instead of answer quality — the "generative fingerprint" shortcut they name. their fix is a data pipeline of style normalization, sentence-level paraphrasing of both sides through the same model, and length balancing, motivated by a formal argument that such transformations contract source information while roughly preserving quality. experiments use production conversations at several scales and two model families, with held-out and public-benchmark gains for both phi-4 and qwen targets. there are also component ablations, cross-model transfer, additional judges, a paraphraser control and a replication on public wildchat data.

Critical strengths

the problem is real and well motivated: when the label is determined by the source model, the shortcut is automatic, and the authors are careful not to claim they were first to notice model fingerprints. the method is simple enough to be adopted without changing the preference-learning objective, which i find appealing. the ablation set is unusually thorough for this kind of paper, and the negative small-model result is reported rather than hidden. the wildchat replication is the result i trust most, because the processed data is compared against the raw versions of exactly the same pairs, which rules out pair selection and data volume as explanations. cross-judge agreement and the paraphraser swap also rule out two obvious alternative stories. overall the empirical result looks solid.

Critical weaknesses

first, the mechanism is not really isolated, the direct source-classification probe moves only slightly, and since correctness alone predicts the source, a quality-matched probe is needed to say how much stylistic information actually remainss ... and the proposition does not help here, because the strict contraction is assumed rather than derived. second, the headline number is not the method's own gain, the full pipeline beats the cheaper normalization-plus-balancing variant by only a small margin, so most of the reported improvement comes from plain preference training plus the easier preprocessing steps.

also worth flagging, the evaluation rests on a binary llm judge with only moderate human agreement, the additional judges confirm the sign but only for the small-scale model, and part of the test set is single-turn, so the multi-turn claim is never tested on its own.

Questions for authors

could you run the cheaper and the full pipeline at the largest scale on the same source pairs, so the paraphrasing gain is reported separately from the rest? and how would a symmetrically normalized large-scale baseline behave relative to the asymmetric one? is it possible to measure source separability on pairs matched for judged quality?

Reviewer_85A5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds preference pairs from chat logs: deployed-model answers are preferred over Phi-3-mini answers to the same prompts. It trains Phi-4 and Qwen-2.5-14B with DPO on 77K–9M pairs. To reduce style bias, it normalizes formatting, paraphrases both answers and balances length, then evaluates mainly with Claude Opus 4.6.

Critical strengths

The controls are helpful. WildChat uses the same pairs for raw and rewritten training. Changing the rewriting model barely changes results. Several judges agree on a small gain, and repeated 1M runs differ by only 0.3 points.

Critical weaknesses

The main comparison is not against ordinary DPO. It uses E0+, which only cleans up the preferred answers and introduces the style bias the paper tries to fix. At 77K pairs, raw DPO gets 65.6 and E3 gets 66.0. E3 is only 1–2 points above E2, within evaluation uncertainty.

The abstract also combines the best numbers from different models: 6.7M pairs for one test, 1M for two others, and 9.3M for AlpacaEval. No single checkpoint achieves all the claimed gains. Table 2 has one run per setting, no paired tests, and different batch sizes from Figures 2–3.

Questions for authors

Which single checkpoint is the headline model, and what are its four test numbers? Are Table 2 rows selected on the development set or at the test-set peak?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.