- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a preference-data selection criterion combining the winner–loser implicit-reward margin with their summed log likelihood ratios. Cross-fitted DPO proxies score the data, and the selected subset is used for preference training. Experiments evaluate alignment performance, selection ratios, proxy settings, computational costs, and transfer across models and datasets.
Critical strengths
- The method is simple and well motivated, combining individual response scores with the preference margin to capture information overlooked by margin-only selection. Its lightweight scoring rule is easy to integrate into existing preference-training pipelines without changing the training objective.
- The method is supported by substantial performance gains over existing data-selection baselines. Targeted ablations, evaluations across multiple model families, and repeated LLaMA runs further strengthen the empirical evidence for its effectiveness.
Critical weaknesses
- The proxy settings behind the headline results are insufficiently specified. Page 7, line 366 states that experiments mainly use three proxy-estimation rounds. However, Table 2's LLaMA 50% LC/WR of 27.7/28.8 matches Table 4's one-round result of 27.69/28.84, while its 10% result of 19.2/24.1 matches the two-round result of 19.23/24.05. The three-round 10% LC is 16.12, below full-data DPO's 18.2. If different configurations are used within the same table, the configuration associated with each result should be clearly specified.
- The interpretation of proxy-based scores could be better supported. High LLR may partly reflect responses that the DPO proxy learns readily, rather than higher intrinsic data quality. Conversely, low LLR may indicate either poor-quality data or valid examples that the proxy has not yet learned. The current analyses do not clearly distinguish these explanations, leaving some uncertainty about what the criterion identifies beyond its demonstrated performance gains.
Questions for authors
- Should the parameter domain in Proposition 4.3 be restricted, for example by requiring both original parameters to lie in [−1, 1]? The stated condition alone does not ensure that the reduction preserves the ranking. Please clarify the applicable range.
- Which proxy-estimation settings were used in Tables 2, 4, 6, 11, and 12? Please clarify the matching results across Tables 2 and 4 and specify which configuration the runtime comparison measures.
2 comments
The additional clarifications and analyses are helpful. I retain my rating.
Thank Reviewers for their sincere efforts! We will answer to the questions one by one as follows. Here, W means Weaknesses and Q means Questions.
We thank the reviewer for carefully cross-checking the results, and we agree that the proxy estimation settings should have been stated explicitly.
LLaMA: We evaluated m∈1,2,3, and Table 2 reports the best-performing m for each selection ratio: m=2 at 10% and m=1 at 50%.
Mistral and Qwen: We did not perform a sweep over m and instead used the default setting of m=3. Thus, their reported results were not tuned with respect to this hyperparameter.
Robustness across proxy settings: Table 4 already reports the results for all three values of m. At the 50% selection ratio, our method outperforms both full-data DPO (18.2/15.5) and the strongest selection baseline, Loss (21.1/18.0), for every m. Even the weakest configuration (m=3, 25.1/24.1) maintains this advantage.
We will clarify the detailed proxy estimation settings in the appendix and upload the revised manuscript as soon as possible.
We agree that a proxy-based score does not by itself establish intrinsic data quality, and we will temper this language in the revised manuscript.
One clarification is important, however. Because of cross-fitting (Algorithm 1), each sample's likelihood ratio is computed by a proxy that never trained on that sample. A high LLR therefore does not simply mean that the proxy fitted this example easily.
To examine what LLR reflects, we compared it with two independent reward models (see our response to Reviewer MDV7, Q1). Proxy LLRs correlate positively with external reward scores, including within reward-margin strata, but the highest-LLR pairs also show higher label disagreement. LLR thus tracks the overall reward level of a pair rather than label reliability, while the margin term in our score preserves the preference signal.
Several existing results further support this distinction:
Prioritizing low-LLR samples does not help. If low-LLR samples were mostly valid but unlearned, prioritizing them (λ<0) should help, as in active learning. Instead, it underperforms (Figs. 4 and 10).
Low-LLR gradients are mutually near-orthogonal. These gradients (Table 1) do not point in a shared direction that the model has yet to learn.
Performance of the selected subsets is robust to the proxy architecture (Fig. 19), which argues against a purely proxy-specific notion of "easiness."
We thank the reviewer for the question. Ranking preservation is ensured by the restriction λ,λ1≥−1 imposed in the proof (Appendix A.3), which keeps the common ratio in Eq. 17 positive. This restriction was stated in the proof but omitted from the main statement, and we will make it explicit.
Under this restriction, the applicable range is λ1≥−1 and λ2−λ1<2. This is the region where 1+λ1 exceeds λ2−1. The boundary λ2−λ1=2 corresponds to selection by the summed likelihood ratio, recovered as λ→∞ (case (4) in Appendix A.3). Outside it, the score would reward a pair more for the likelihood of its losing response than of its winning response, contradicting the preference signal that the criterion is designed to preserve. Such weightings are therefore outside the intended scope of the selection criterion. The domain suggested by the reviewer, [−1,1]2, is contained in this range except for the degenerate corner (−1,1), where the score is identically zero, and is thus a valid sufficient condition.
In the revised manuscript, we will explicitly state the condition λ2−λ1<2 in Proposition 4.3 and add the corresponding one-line check to case (1) of the proof.
Please see our response to W1 for a detailed explanation. For clarity, we summarize the proxy-estimation settings used in each table: