- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper considers improving the efficiency of "tie training", a technique proposed by Moya et al 2026. Tie training seeks to reduce the reliance of preference learning on spurious parts of the features, that are not predictive of preference in test scenarios. Tie training proceeds by adding comparisons which differ only in spurious features (not "core"/"causal" ones) to the dataset, assigning a win/lose value at random. The present paper considers ways to reduce the number of added comparisons while maintaining the effectiveness of tie training. Using standard first-order sensitivity tools ("influence functions"), the candidate comparisons are scored using the loss' gradient and hessian. This score being costly to compute (>= 1 backward passes), an approximation that only involves the comparison margin (1 forward pass) is proposed; it matches the initial score in expectation. The method is evaluated on synthetic and one Llama 3.2 1B on one dataset, and shows that choosing ties at random is less efficient than choosing based on the proposed scores.
As far as I understand, the contribution lies in the proposed way to reduce the number of pairs needed to perform the "tie training" from Moya et al 2026. The mathematical content has a limited depth.
Critical strengths
- I believe that this paper deals with a relevant topic for the preference learning community, that of handling or leveraging ties in data, which are otherwise ignored by the Bradley-Terry model.
Critical weaknesses
-
The mathematical content is limited: the proofs essentially consist of combining the Implicit Functions Theorem, Taylor expansions, and linear algebra.
-
The analysis centers around isolated minimizers with local quadratic growth of the loss. But preference learning, particularly in modern LLM pipelines, is done in one epoch. This leaves the final checkpoint of DPO, from which tie learning is applied ( here) away from critical points, and in particular isolated minimizers with local quadratic growth. Besides, it is far from clear that typical losses do indeed admit such minimizers. Learning losses can show valleys of minimizers, for instance when there are symmetry invariances in the network, and thus break this assumptions.
Questions for authors
Any elements on the reported weaknesses would be helpful in updating my score.
2 comments
Response to Reviewer cbtK (2/2)
W2. Assumption of an Isolated Minimizer
Concern 1. The analysis assumes an isolated minimizer with local quadratic growth, but one epoch of DPO ends away from critical points, and neural losses can have valleys of minimizers.
Response. We agree with both points for neural models. Our analysis assumes an isolated minimizer only in the log-linear model. In this model, the DPO loss is convex. At the solution we analyze, we assume that the gradient vanishes and the Hessian is positive definite. With convexity, these conditions imply a unique minimizer with local quadratic growth and no valley of minimizers. In our experiments on the log-linear model, we check that the gradient norm is at most and that the Hessian is positive definite. Scoring stops if either check fails (App. B.1).
Outside the log-linear model, our analysis does not apply. The last paragraph of Sec. 4 states that both selectors rest on the log-linear assumption, and Sec. 6.2 states that the log-linear theory no longer applies to a neural network (see also App. E). For neural models, we therefore make no theoretical claim. Our language models train for one epoch, as you note, so their checkpoints need not satisfy the assumptions of the analysis. We test whether both selectors outperform uniform selection in these models (W1).
In our main experiments, the checkpoint only scores the ties. We then return to the initialization of the scoring policy and train a new policy on the strict comparisons and the selected ties (Sec. 2.3; App. C.1). We report continued training from the checkpoint separately, as an ablation (App. C.4.1, C.5.4–C.5.5).
Changes in the Manuscript
Your review showed us two places where the paper was unclear: the scope of the analysis and the role of the checkpoint. We revised both.
- Scope (Sec. 1, Sec. 3, Sec. 4). The first contribution now opens with "For log-linear policies," and Sec. 3 and Sec. 4 each now state the scope of their analysis after the opening paragraph.
- Role of the checkpoint (Sec. 2.3, Sec. 5). We now state that the checkpoint only scores the ties and that training restarts from the initialization.
Closing. We hope these clarifications address your concerns. If you have additional questions or concerns, please let us know. If not, we would appreciate an updated assessment of our paper.
Response to Reviewer cbtK (1/2)
Thank you for your review. We appreciate that you found ties relevant to preference learning. Your concerns helped us clarify the contributions and the scope of our analysis. We address them below.
Our response to all reviewers restates our contributions, clarifies the scope of our mathematical analysis, and lists the new evidence in this revision.
W1. Depth of the Mathematical Content
Concern 1. The proofs combine the implicit function theorem, Taylor expansions, and linear algebra.
Response. We agree that these tools are standard. We chose them because they suffice to answer our research question: which ties reduce spurious reliance, and why? Previous work showed that ties reduce spurious reliance in aggregate, but left this question open (Moya et al., 2026). To make this question tractable, we study a policy that is log-linear in core and spurious features. This choice follows theoretical work on spurious correlations, which analyzes models that are linear in core and spurious features (Arjovsky et al., 2019; Nagarajan et al., 2021; Bombari & Mondelli, 2025). We treat the results of this model as predictions to test in neural networks and language models.
In this model, the standard tools explain which ties to select and why:
These results yield two selectors. Influence selection takes the ties with the largest influence, which is first-order optimal because influences add (Thm. 2), and pure margin selection ranks ties by absolute margin (Thm. 3). In the log-linear model, the experiments support the predictions: across ties added one at a time, the predicted influence matches the observed reduction with 4.2% relative ℓ2 error, and the conditional mean of Thm. 3 lies within the 95% confidence interval of the observed mean in every margin group (Fig. 1). Beyond the log-linear model, both selectors outperform uniform selection at matched budgets below the full pool in a neural network, Llama-3.2-1B on two tasks, Qwen3-4B (one seed), and OLMo-3-7B (Sec. 6.2–6.3; App. C.5.6, D.2).
These standard tools therefore explain, in the log-linear model, which ties to select and why, and they yield a selection rule that requires only forward passes. The contribution of our analysis lies in this answer and in the predictions it makes, which our experiments support.
We continue in the next comment (2/2).