acceptodds
Under review as a conference paper at ICLR 2027

Selective Tie Learning: Mitigating Spurious Correlations in Preference Optimization

Abstract

Preference optimization can learn spurious correlations, and augmenting the preference data with ties (equally preferred responses) can reduce spurious reliance. However, ties differ in their corrective effect, making their selection important. For log-linear policies, we derive each tie's influence on spurious features and show that taking the largest predicted reductions is first-order optimal among subsets of equal size. Computing these influences requires knowledge of the spurious features and the curvature of the training objective, which is often unavailable or expensive to estimate. We therefore study the absolute implicit reward margin as a proxy for tie influence, giving a selection rule that requires two forward passes per tie. To test the rules beyond the theoretical setting, we develop practical influence estimates using a counterfactual proxy for reliance on spurious features and a generalized Gauss–Newton approximation to curvature. Across log-linear policies, neural models, and large language models, our selection rules consistently achieve comparable reductions in spurious reliance with fewer ties than uniform selection.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Preference optimization, Spurious correlations, Data selection, Influence functions, Language model alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cbtKOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper considers improving the efficiency of "tie training", a technique proposed by Moya et al 2026. Tie training seeks to reduce the reliance of preference learning on spurious parts of the features, that are not predictive of preference in test scenarios. Tie training proceeds by adding comparisons which differ only in spurious features (not "core"/"causal" ones) to the dataset, assigning a win/lose value at random. The present paper considers ways to reduce the number of added comparisons while maintaining the effectiveness of tie training. Using standard first-order sensitivity tools ("influence functions"), the candidate comparisons are scored using the loss' gradient and hessian. This score being costly to compute (>= 1 backward passes), an approximation that only involves the comparison margin (1 forward pass) is proposed; it matches the initial score in expectation. The method is evaluated on synthetic and one Llama 3.2 1B on one dataset, and shows that choosing ties at random is less efficient than choosing based on the proposed scores.

As far as I understand, the contribution lies in the proposed way to reduce the number of pairs needed to perform the "tie training" from Moya et al 2026. The mathematical content has a limited depth.

Critical strengths
  • I believe that this paper deals with a relevant topic for the preference learning community, that of handling or leveraging ties in data, which are otherwise ignored by the Bradley-Terry model.
Critical weaknesses
  • The mathematical content is limited: the proofs essentially consist of combining the Implicit Functions Theorem, Taylor expansions, and linear algebra.

  • The analysis centers around isolated minimizers with local quadratic growth of the loss. But preference learning, particularly in modern LLM pipelines, is done in one epoch. This leaves the final checkpoint of DPO, from which tie learning is applied (θ0\theta_0 here) away from critical points, and in particular isolated minimizers with local quadratic growth. Besides, it is far from clear that typical losses do indeed admit such minimizers. Learning losses can show valleys of minimizers, for instance when there are symmetry invariances in the network, and thus break this assumptions.

Questions for authors

Any elements on the reported weaknesses would be helpful in updating my score.

2 comments
AuthorsOpenReview ↗

Response to Reviewer cbtK (1/2)

Thank you for your review. We appreciate that you found ties relevant to preference learning. Your concerns helped us clarify the contributions and the scope of our analysis. We address them below.

Our response to all reviewers restates our contributions, clarifies the scope of our mathematical analysis, and lists the new evidence in this revision.

W1. Depth of the Mathematical Content

Concern 1. The proofs combine the implicit function theorem, Taylor expansions, and linear algebra.

Response. We agree that these tools are standard. We chose them because they suffice to answer our research question: which ties reduce spurious reliance, and why? Previous work showed that ties reduce spurious reliance in aggregate, but left this question open (Moya et al., 2026). To make this question tractable, we study a policy that is log-linear in core and spurious features. This choice follows theoretical work on spurious correlations, which analyzes models that are linear in core and spurious features (Arjovsky et al., 2019; Nagarajan et al., 2021; Bombari & Mondelli, 2025). We treat the results of this model as predictions to test in neural networks and language models.

In this model, the standard tools explain which ties to select and why:

  • Two components of influence. A tie has no difference in core features, so its gradient has no core part. When we account for the curvature of the DPO loss, the influence of a tie factors into two terms: a margin factor, which sets how strongly the tie pushes the policy, and an alignment factor, which sets whether the push opposes spurious reliance (Prop. 1).
  • Selection with forward passes. The alignment factor requires the curvature and the spurious features, but the margin requires only a forward pass. The margin and the alignment factor are both linear in the tie's difference in spurious features. Under the Gaussian model of Sec. 4.2, the expected alignment thus grows linearly with the margin, and selecting the ties with the largest absolute margins maximizes the expected first-order reduction in spurious reliance among selectors that see only margins (Thm. 3).
  • Not every tie helps. Although ties reduce spurious reliance in aggregate, Prop. 1 shows that an individual tie raises it, to first order, when its margin factor and its alignment factor have opposite signs. Under the Gaussian model of Sec. 4.2, Thm. 3 shows that the expected influence is positive for every nonzero margin and grows with the absolute margin. The margin thus identifies the ties that help in expectation, although an individual tie with a large margin can still raise spurious reliance. Our estimates of practical influence show both negative influences and a positive association with absolute margin. On hotels with Llama-3.2-1B (seed 0), 93 of the 2,000 candidate ties (4.65%) have negative practical influence, but none of the 100 ties with the largest absolute margins does (App. D.5). A criterion that removes the margin can also perform worse than uniform selection, as our adapted ActiveDPO criterion does at intermediate budgets (App. D.1; Reviewer 1, W1).

These results yield two selectors. Influence selection takes the ties with the largest influence, which is first-order optimal because influences add (Thm. 2), and pure margin selection ranks ties by absolute margin (Thm. 3). In the log-linear model, the experiments support the predictions: across ties added one at a time, the predicted influence matches the observed reduction with 4.2% relative ℓ2\ell_2 error, and the conditional mean of Thm. 3 lies within the 95% confidence interval of the observed mean in every margin group (Fig. 1). Beyond the log-linear model, both selectors outperform uniform selection at matched budgets below the full pool in a neural network, Llama-3.2-1B on two tasks, Qwen3-4B (one seed), and OLMo-3-7B (Sec. 6.2–6.3; App. C.5.6, D.2).

These standard tools therefore explain, in the log-linear model, which ties to select and why, and they yield a selection rule that requires only forward passes. The contribution of our analysis lies in this answer and in the predictions it makes, which our experiments support.

We continue in the next comment (2/2).

AuthorsOpenReview ↗

Response to Reviewer cbtK (2/2)

W2. Assumption of an Isolated Minimizer

Concern 1. The analysis assumes an isolated minimizer with local quadratic growth, but one epoch of DPO ends away from critical points, and neural losses can have valleys of minimizers.

Response. We agree with both points for neural models. Our analysis assumes an isolated minimizer only in the log-linear model. In this model, the DPO loss is convex. At the solution we analyze, we assume that the gradient vanishes and the Hessian is positive definite. With convexity, these conditions imply a unique minimizer with local quadratic growth and no valley of minimizers. In our experiments on the log-linear model, we check that the gradient norm is at most 10−1010^{-10} and that the Hessian is positive definite. Scoring stops if either check fails (App. B.1).

Outside the log-linear model, our analysis does not apply. The last paragraph of Sec. 4 states that both selectors rest on the log-linear assumption, and Sec. 6.2 states that the log-linear theory no longer applies to a neural network (see also App. E). For neural models, we therefore make no theoretical claim. Our language models train for one epoch, as you note, so their checkpoints need not satisfy the assumptions of the analysis. We test whether both selectors outperform uniform selection in these models (W1).

In our main experiments, the checkpoint only scores the ties. We then return to the initialization of the scoring policy and train a new policy on the strict comparisons and the selected ties (Sec. 2.3; App. C.1). We report continued training from the checkpoint separately, as an ablation (App. C.4.1, C.5.4–C.5.5).

Changes in the Manuscript

Your review showed us two places where the paper was unclear: the scope of the analysis and the role of the checkpoint. We revised both.

  • Scope (Sec. 1, Sec. 3, Sec. 4). The first contribution now opens with "For log-linear policies," and Sec. 3 and Sec. 4 each now state the scope of their analysis after the opening paragraph.
  • Role of the checkpoint (Sec. 2.3, Sec. 5). We now state that the checkpoint only scores the ties and that training restarts from the initialization.

Closing. We hope these clarifications address your concerns. If you have additional questions or concerns, please let us know. If not, we would appreciate an updated assessment of our paper.

Reviewer_jdVoOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to select a limited number of ties to mitigate spurious correlations in preference optimization. For log-linear policies, it derives each tie’s first-order influence on spurious reliance and proves that selecting the largest predicted reductions is first-order optimal. Under a Gaussian model of tie contrasts, ranking by absolute reward margin maximizes expected first-order improvement among margin-only selectors. The authors extend influence scoring to neural models using counterfactual energy and approximate curvature. Experiments on log-linear models, an MLP, and controlled language-model tasks show that targeted selection achieves comparable mitigation with fewer ties than uniform selection.

Critical strengths
  • S1. The paper connects a concrete selection problem to an interpretable theoretical decomposition. A tie’s corrective value depends on both its margin and its curvature-adjusted alignment with the direction of spurious reliance. The log-linear experiments quantitatively validate the influence predictions and demonstrate the benefit of selecting informative ties under a fixed budget.

  • S2. Pure margin selection offers a practical simplification. It requires only forward passes and performs comparably to influence selection in several settings, while outperforming practical influence on the reported Llama emoji task. The experiments distinguish adversarial accuracy from counterfactual sensitivity, avoiding the assumption that improved accuracy alone demonstrates reduced shortcut reliance.

Critical weaknesses
  • W1. The practical scope of the margin-ranking guarantee remains insufficiently characterized. Theorem 3 assumes Gaussian contrasts with covariance proportional to the spurious-feature second moment. This is a substantive assumption, and the controlled validation does not establish its applicability to language-model tie pools. Since equally sized margins can have different influences, the paper would benefit from directly checking whether high-margin ties reliably receive positive estimated influence in those pools.

  • W2. The additional practical value of influence scoring is unclear relative to its computational cost. It requires a held-out probe, gradient computations, and approximate curvature, yet PM tracks it on hotels and performs better on the reported Llama emoji task. The experiments support selective tie learning more strongly than they establish a reason to prefer the expensive selector. The continued-training results also show that the local analysis does not by itself explain mitigation over longer training trajectories.

Questions for authors
  • Q1. For one LLM task, please report the Spearman correlation between practical influence and absolute margin, and the fraction of the top-100 margin-selected ties with negative estimated influence. These statistics would help assess the proxy’s empirical reliability, without being treated as verification of the Gaussian assumption.

  • Q2. How stable are the PM–UTL gains and the ordering between PM and practical influence across seeds? Please summarize paired per-seed differences at representative budgets and distinguish the three-seed Llama results from the single-seed Qwen check.

  • Q3. Is there a tested setting where practical influence provides a meaningful advantage over PM? Please report scoring time and memory alongside any benefit, and clarify when the additional computation is justified.

  • Q4. Continued training improves accuracy while mean counterfactual energy remains above the strict baseline. Can signed probe-margin statistics clarify whether this reflects shortcut reversal, stronger sensitivity on a subset of ties, or another effect? How should this limit the interpretation of the local influence scores?

4 comments
AuthorsOpenReview ↗

Response to Reviewer part jdVo (1/2)

Thank you for your review. We appreciate that you found the decomposition of a tie's influence into margin and alignment interpretable, and pure margin selection (PM) a practical simplification. We address your concerns below.

W1. Scope of the Guarantee for Pure Margin Selection

Concern 1. Strength of the Gaussian assumption.

Response. We agree that this Gaussian assumption is strong. Section 4.2 introduces it for tractability without claiming that it holds for pools of ties in language models. Following Moya et al. (2026), the covariance cΣssc\Sigma_{ss} models ties that inherit the spurious geometry of the preference data. Under this Gaussian model, pure margin selection ranks ties by absolute margin using only forward passes and maximizes the expected first-order reduction in spurious energy among selectors that see only margins (Theorem 3). Beyond this Gaussian setting, on hotels we still draw spurious attributes from the preference data without assuming Gaussian contrasts. Across all four evaluated pools, we test pure margin selection directly and find that absolute margin correlates positively with influence (Concern 2).

Concern 2 (Q1). Do ties with large margins receive positive influence?

Response. On hotels with Llama-3.2-1B, yes. None of the 100 ties with the largest margins has negative practical influence. Because practical influence replaces the energy with counterfactual energy and the Hessian with a damped GGN in language models, this comparison measures agreement between two selectors rather than testing the Gaussian assumption. Moreover, Theorem 3 predicts this relationship in expectation rather than tie by tie, because a tie's alignment can take either sign (Section 3.1). Thus, the two rankings need not coincide.

Evidence. At the strict checkpoint (seed 0; App. D.5, Fig. 11), 93 of the 2,000 hotel candidates (4.65%) have negative practical influence, but none of the 100 that pure margin picks first. The Spearman correlation between absolute margin and influence is 0.977 on hotels, 0.928 in the log-linear model, where Oracle influence is exact, 0.950 for the neural network, and 0.668 on emojis. The weaker agreement on emojis may explain why the two selectors differ on that task (W3).

W2. Value of Influence Relative to Its Cost

Concern 1. Influence needs a probe, tie gradients, and curvature, yet PM tracks it on hotels and exceeds it on emojis. Why prefer the expensive selector?

Response. We do not prefer influence by default. Instead, we recommend pure margin selection. Our analysis first derives influence ranking as first-order optimal (Theorem 2). Because computing influence is costly, Theorem 3 conditions influence on the margin to obtain a selector that requires only forward passes. Influence therefore defines the theoretical target, whereas pure margin selection provides our practical method. The margin, however, reflects only spurious feature differences on exact ties (Section 3.1). When a tie also changes core content, pure margin selection cannot separate core from spurious contributions, whereas influence can isolate spurious reliance using a separate probe of exact ties.

Concern 2 (Q3). Is there a tested setting where practical influence gives a meaningful advantage over PM? Report scoring time and memory, and say when the extra computation is justified.

Response. We find no consistent advantage in language models. In the log-linear model, influence with an exact Hessian helps when candidate ties have substantial core changes (Concern 1). With three seeds, practical influence leads pure margin by at most 1.5 percentage points on average (hotels, k=100k = 100; W3), and its larger lead on Qwen3-4B (hotels, k=500k = 500: 66.2% vs. 62.6%) comes from one seed. The extra computation may therefore be justified when candidate ties have substantial core changes and a probe with smaller core changes is available.

Evidence. In the log-linear model, we added to each candidate tie a random core difference as large, on average, as the core difference between the two responses of a strict comparison (30 seeds, k=100k = 100; App. D.6, Fig. 12). Pure margin then reaches 51.2% accuracy under QadvQ_{\text{adv}}, while influence reaches 71.2% with an exact probe and 69.5% with a probe whose core changes are half as large as those of the candidates. With a probe whose core changes are as large as those of the candidates, influence loses this advantage: it reaches 48.2%, and across seeds it does not differ reliably from pure margin. On hotels with perturbed ties, practical influence leads by at most 1.0 point at the largest perturbation (Table 4, App. D.3). Influence also costs more computation: scoring 2,000 hotel ties with Llama-3.2-1B on one RTX 4090 took 68 s with PM and 6.8 min with practical influence, which stored 22.5 GB of curvature in host memory. With OLMo-3-7B, practical influence took 17–18 min and 40 GB.

AuthorsOpenReview ↗

Response to Reviewer part jdVo (2/2)

W3. Stability Across Seeds

Concern 1 (Q2). How stable are the gains of PM over uniform tie learning (UTL) and the ordering between PM and practical influence across seeds?

Response. The gains over UTL are stable across seeds. The ordering against practical influence is not. The Qwen3-4B check uses one seed and one draw of UTL, so it gives no estimate of variation, but it agrees with the Llama results.

Evidence. For Llama-3.2-1B, PM exceeds UTL in all 18 paired comparisons (three seeds, k∈100,500,1500k \in \\{100, 500, 1500\\}, hotels and emojis). At k=500k=500, the mean gains are 17.5 points on hotels and 9.1 on emojis. Against practical influence, the sign changes across seeds on hotels, and the mean difference stays within 1.5 points. On emojis, the two selectors nearly tie at k=100k = 100 and k=1500k = 1500, and PM wins in every seed at k=500k = 500. For Qwen3-4B, both selectors exceed UTL at every budget below the full pool, and neither is better than the other at every budget on hotels (App. C.5.6). Each seed has its own data, pool of ties, strict checkpoint, and scores. Please refer to App. D.5 and C.1 for full details.

W4. Local Scores and Longer Training

Concern 1. The results of continued training show that the local analysis does not by itself explain mitigation over longer training.

Response. We agree. The scores predict the first-order change in spurious energy at the strict checkpoint, and the paper states this limit (Sec. 3.2, "Scope: first-order result"). We use the analysis to derive the selectors, not to predict training trajectories, and we judge the selectors by their effect after fresh training, our main protocol (Sec. 6). Continued training is an ablation beyond this protocol.

Evidence. In the log-linear model, the objective has a single minimizer, so fresh and continued training agree to numerical precision (App. C.3.1). In the neural network, mean accuracies after fresh and continued training differ by less than 0.47 percentage points under QadvQ_{\text{adv}}, and both keep the advantage of selection (App. C.4.1). In language models, they diverge: on hotels, continued PM leads at k=100k = 100 (67.3% vs. 55.9%) but falls behind at k=2000k = 2000 (80.2% vs. 95.3%), and its counterfactual energy stays above the strict baseline at every budget (App. C.5.4). Concern 2 examines this gap.

Concern 2 (Q4). Continued training improves accuracy while counterfactual energy stays above the strict baseline. Is this reversal of the shortcut, stronger sensitivity on a subset of ties, or another effect, and how should it limit the interpretation of the scores?

Response. Our new evidence suggests that continued training strengthens reliance on core features while retaining spurious reliance. Higher accuracy under QcleanQ_{\text{clean}} supports this explanation, but does not rule out shortcut reversal or stronger sensitivity on some ties. Our local analysis does not establish how spurious reliance evolves throughout continued training.

Evidence. Instead of signed probe margins, we measured core energy, the mean squared margin on test pairs that differ only in core fields (App. C.5.4). On 2,000 such hotel pairs (seed 0), continuing the strict checkpoint with 500 PM ties raises core energy from 19.8 to 130.2 and accuracy under QadvQ_{\text{adv}} from 30.8% to 83.2%, while counterfactual energy rises. At the full pool, continued PM raises accuracy under QcleanQ_{\text{clean}} from 73.4% to 86.9% (App. C.5.4).

Questions

Q1 is answered in W1, Concern 2; Q2 in W3; Q3 in W2, Concern 2; and Q4 in W4, Concern 2.

Closing. We hope these clarifications address your concerns. If you have additional questions or concerns, please let us know. If not, we would appreciate an updated assessment of our paper.

Thank you for the detailed response and additional experiments. The margin–influence correlation analysis and the results across seeds address my questions about the reliability of PM. The runtime comparison and perturbed-tie experiments also clarify the trade-offs between PM and influence scoring.

I have three remaining concerns. The applicability of the Gaussian assumption in Theorem 3 to LLM tie pools remains unclear. Influence scoring shows limited additional benefits over PM in the LLM experiments despite its higher computational cost. The core-energy analysis does not fully explain the behavior observed during continued training.

The rebuttal has addressed several of my concerns and increased my confidence in the empirical results. I will maintain my original rating of 3 (Weak Acceptance).

AuthorsOpenReview ↗

Response to Reviewer jdVo

Thank you for your reply and for confirming that the new analyses increased your confidence in the empirical results. We address the three remaining points below.

Gaussian assumption in Theorem 3. We agree. Theorem 3 holds under a Gaussian model of the tie contrasts, and whether it extends to the arbitrary geometry of tie pools in language models remains open. In a language model, the spurious features are not observed, so the distribution of contrasts cannot be checked directly. We therefore test its prediction under relaxed conditions. The prediction is that the influence of a tie grows with its absolute margin. It holds on hotels (Spearman 0.977) and weakens on emojis (0.668), and PM exceeds UTL on both tasks (App. D.5). In Sec. 4.2, we will state that extending Theorem 3 beyond the Gaussian model remains open, and we will point to App. D.5 for the test of its prediction.

Value of influence over PM. We agree that influence costs more than PM and gives no consistent gain in our experiments with language models. In the paper, influence explains where selection arises for log-linear models. It factors into margin and alignment, which shows which ties reduce spurious reliance and why (Prop. 1, Thm. 2). Because influence is costly, it then leads to its proxy, PM, which needs only forward passes (Thm. 3). Influence therefore provides the mathematical basis from which we derive PM, rather than a competing selector, and we recommend PM in practice.

Core energy and continued training. We agree. Our method trains a new model on the strict comparisons and the selected ties, from the same initialization as the strict model (Sec. 6). Continued training from the strict checkpoint is an ablation outside our mathematical analysis. The rise in core energy offers one possible explanation of its behavior, but other explanations may also hold, as your original question suggested. How spurious reliance evolves under continued training therefore remains open, and we will state this point in the limitations.

Thank you again for your careful review.

Reviewer_15zXOpenReview ↗
Rating
2(+1)
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of spurious correlations in preference optimization by proposing a selective Tie Learning approach based on Influence Functions and Implicit Reward Margins, with theoretical guarantees of first-order optimality and expected first-order optimality, respectively. Experimental results demonstrate that, compared with random Tie Selection, the proposed methods can effectively reduce the model's reliance on spurious features using fewer samples. Notably, Margin Selection, which requires only forward passes, achieves performance comparable to the more computationally expensive Influence Selection.

Critical strengths
  1. Practical relevance of the research problem. The paper addresses spurious correlations in preference optimization and investigates how to select more effective Tie samples under a limited data budget to reduce the model's reliance on spurious features. This problem has clear practical significance.

  2. Simple and computationally efficient method. The proposed Pure Margin Selection requires only forward passes, without additional Hessian or gradient computations, while achieving performance comparable to the more computationally expensive Influence-based Selection.

  3. Connection between theory and experiments. The authors analyze the debiasing effects of individual samples using Influence Functions and validate the theoretical predictions in a Log-linear model. Furthermore, experiments on neural networks and LLMs demonstrate that selective Tie Learning is more sample-efficient than random selection.

Critical weaknesses
  1. Insufficient comparisons with existing active learning and data selection methods. The paper primarily compares the proposed approach with Uniform Tie Learning and its own Influence-based Selection, without direct experimental comparisons against existing methods such as active learning, or uncertainty-based and gradient-based data selection strategies. Therefore, it remains unclear whether the observed improvements arise from the specific selection criteria proposed in this paper or whether similar benefits could be achieved through general data selection strategies. This weakens the novelty of the method and the experimental evidence supporting its advantages over prior work.

  2. Limited theoretical contributions. The theoretical analysis mainly builds upon classical Influence Functions and first-order approximations. The optimality result for Top-k Selection is relatively straightforward, while the theoretical guarantees for Margin Selection rely on strong Gaussian distribution assumptions. The paper lacks theoretical analysis for more general distributions and nonlinear models.

  3. Limited scale and overly idealized LLM experiments. The experiments are conducted only on Llama-3.2-1B-Instruct and primarily focus on the Hotel and Emoji tasks with artificially controlled spurious correlations. The paper lacks validation on larger-scale models and real-world human preference datasets.

  4. Dependence on high-quality Tie construction. The proposed method assumes that Tie samples can be constructed by preserving Core Features while modifying only Spurious Features. However, such disentanglement is difficult to guarantee in real-world language tasks. The paper also lacks a systematic analysis of errors in Tie construction and their impact on the effectiveness of the proposed method.

Questions for authors
  1. Lack of active learning and data selection baselines.

The paper primarily compares the proposed method with Uniform Tie Learning. Could the authors further include comparisons with Active Preference Learning, ActiveDPO, and data selection methods based on uncertainty, gradients, or other sample importance metrics? Can these methods achieve similar debiasing effects under the same Tie Budget?

  1. Distinction between the theoretical contributions and existing methods.

The first-order optimality results mainly build upon classical Influence Functions and Top-k ranking. Could the authors further clarify the main theoretical novelty of this work and its substantive differences from existing Influence-based Data Selection methods?

  1. Validation on larger-scale LLMs.

The current experiments only use Llama-3.2-1B-Instruct. Could the authors evaluate the effectiveness of the proposed method on 7B/8B or larger models? In particular, does the advantage of Pure Margin Selection over other selection strategies remain consistent as model scale increases?

  1. Real-world preference data and reliability of Tie construction.

The paper primarily considers tasks with artificially controlled spurious correlations and assumes that Tie samples can preserve Core Features while modifying Spurious Features. Could the authors validate the proposed method on real-world human preference datasets and investigate how inaccuracies in Tie construction affect the effectiveness of the selection strategy?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 15zX part (1/2)

Thank you for your review. We appreciate that you found the problem relevant and our theory linked to the experiments. We address your concerns below.

W1. Comparison with Active Learning & Data Selection

Concern 1. Missing baselines.

Response. In preference optimization, methods for active learning and data selection use gradients or uncertainty to choose strict comparisons (Muldrew et al., 2024; Lin et al., 2026). These methods aim to cut annotation costs, reduce uncertainty, or improve preference prediction. By contrast, we aim to reduce spurious reliance by selecting known ties under a fixed budget. While active selection could guide how to construct ties, that step lies outside our scope. Following your suggestion, we adapted two criteria to select from a fixed pool of ties: the gradient norm from supervised selection (Paul et al., 2021) and ActiveDPO (Lin et al., 2026). We compare both criteria with our selectors in App. D.1.

Concern 2 (Q1). Are the gains specific to the selection, or could existing methods achieve similar debiasing?

Response. Not all of them. Among the criteria we tested, only those that keep the margin match our selectors. This result agrees with Prop. 1, which factors a tie's influence into its margin and its alignment with spurious reliance. We compared these criteria across five seeds in our log-linear model and neural network. Each selector scored a shared pool of ties at the strict checkpoint, and we retrained on the strict comparisons and selected ties at matched budgets (Fig. 8, App. D.1):

  • Gradient norm keeps the margin because in the log-linear model ∣gj∣2=β2tanh⁡(∣mj∣/2),∣dj∣2\\|g_j\\|_2 = \tfrac{\beta}{2}\tanh(|m_j|/2)\\,\\|d_j\\|_2. Under the assumptions of Thm. 3, its expected value increases with ∣mj∣|m_j|, as expected influence does. It closely tracks pure margin selection.
  • ActiveDPO normalizes margin gradients to favor unexplored directions, without using the margin. It yields lower adversarial accuracy and higher energy than pure margin selection, and influence, and it falls below uniform tie learning at intermediate budgets.

All selectors coincide at full pool. Under fixed budget, the choice of selector matters. Criteria that keep the margin perform best. ActiveDPO was designed to select comparisons across annotation rounds with model updates, whereas our adaptation scores ties once at the strict checkpoint. Adapting ActiveDPO's coverage criterion to target spurious reliance therefore deserves its own study. Our tractable models offer a testbed for this direction (App. D.1).

W2. Depth of Theory

Concern 1 (Q2). Novelty beyond influence functions.

Response. We did not start from influence functions. We arrived at them by asking how a single tie changes spurious reliance. While our mathematical tools are classical, we introduce the tie gradient, the symmetric packet, and the resulting tie analysis. Because a tie gradient has no core component, its effect passes through the Schur complement KsK_s, factoring influence into a margin term and an alignment term (Prop. 1). Top-kk optimality (Thm. 2) is simple by design because additive influences reduce selection to ranking, so the contribution lies in the quantity ranked. Whereas classical influence functions score training points by test loss (Koh & Liang, 2017), our analysis targets spurious reliance over candidate ties. This tie structure yields a selector that requires only forward passes (Thm. 3).

Concern 2. Strength of the Gaussian assumption.

Response. We agree the assumption is strong, and the paper states its scope (Sec. 4.2). Under this Gaussian model, ranking by absolute margin maximizes the expected first-order reduction in spurious reliance among selectors that see only margins (Thm. 3). Beyond this Gaussian setting, we test whether the margin still predicts influence. At the strict checkpoint (one seed), the Spearman correlation between absolute margin and influence is 0.928 in the log-linear model, 0.950 in the neural network, and 0.977 and 0.668 on hotels and emojis with Llama-3.2-1B (Fig. 11, App. D.5). On hotels, none of the 100 ties with the largest margins has negative influence (Reviewer 2, Q1).

Concern 3. Analysis beyond log-linear.

Response. We agree that our guarantees hold only for log-linear policies, and we state this scope (Sec. 4.2 and App. E). Linear models with core and spurious features are standard in spurious correlation theory because they isolate how training produces unstable predictors (Nagarajan et al., 2021; Arjovsky et al., 2019; Bombari & Mondelli, 2025). We adopt these assumptions to isolate how ties change spurious reliance and derive our selectors. We then test both selectors beyond these assumptions, in a neural network and in language models, and their gains over uniform tie learning persist qualitatively (Sec. 6.2–6.3; App. C.5.6, D.2).

AuthorsOpenReview ↗

Response to Reviewer 15zX part (2/2)

W3. Scale of the LLM Experiments

Concern 1 (Q3). Model scale beyond Llama 1B.

Response. Beyond Llama-3.2-1B, our paper evaluates Qwen3-4B (a second model family) on hotels and emojis with one seed (App. C.5.6). To address your concern, we added OLMo-3-7B-Instruct on hotels across three seeds (Fig. 9, App. D.2). As noted in our response to all reviewers, these language models test whether selection gains persist qualitatively with hidden representations, autoregressive sequence modeling, and scale, not to certify the theory.

Concern 2 (Q3). Whether the advantage of pure margin persists with scale.

Response. It does in the models we tested. With Llama-3.2-1B and Qwen3-4B, pure margin selection outperforms uniform selection at every sub-pool budget on hotels and emojis, winning all 18 paired Llama-3.2-1B comparisons across tasks, budgets, and seeds (App. D.5). With OLMo-3-7B, the three selectors perform similarly at k=100k = 100, but at k=500k = 500 adversarial accuracy reaches 34.4% for pure margin selection, 32.8% for practical influence, and 14.8% for uniform selection. At k=1,500k = 1{,}500, accuracy rises to 77.7%, 77.3%, and 61.5% (Fig. 9, App. D.2). Across these experiments, pure margin selection remains competitive with practical influence at substantially lower scoring cost (App. D.6).

Concern 3 (Q4). Validation on natural preference data.

Response. The emoji task already uses natural responses from PKU-SafeRLHF with a spurious attribute that we control. To test a spurious attribute that arises without our intervention, we ran a pilot experiment on verbosity bias in OpenAssistant Conversations (OASST1, Köpf et al., 2023, App. D.4). To construct each tie, Qwen3-8B shortens a preferred response while preserving its content and pairs the rewrite with the original response. Trained on 2,203 natural comparisons, the strict Llama-3.2-1B checkpoint prefers the verbose response in 89.5% of 531 test ties, compared with 50% for an unbiased model. Across three seeds, pure margin selection and practical influence reduce both this verbosity preference and counterfactual energy more than uniform selection at every budget up to k=500k = 500 (Fig. 10). Accuracy on 161 validation comparisons matches the strict checkpoint (74.5% to 76.0% vs. 74.7%). Thus, selective tie learning reduces verbosity reliance while preserving preference accuracy.

W4. Quality of the Constructed Ties

Concern 1 (Q4). Difficulty of constructing exact ties.

Response. We assume exact ties to isolate tie selection from tie construction (Sec. 3.1). Because selective tie learning takes a tie pool as input (Sec. 2.3), constructing ties for arbitrary data remains open, as stated in Sec. 8 and App. E. Still, several domains simplify tie construction. In verifiable domains, verified responses with different spurious attributes could provide candidate ties. In natural language, language model rewrites yield imperfect ties, as in our OpenAssistant Conversations experiment (W3, Concern 3).

Concern 2 (Q4). Effect of errors in tie construction.

Response. We agree that this analysis was missing. We now relax exact ties to imperfect ties, whose responses also differ in core features, and define the magnitude of this core difference as core leakage. Because an imperfect tie's margin mixes core and spurious contributions, large margins need not signal strong spurious reliance, potentially weakening pure margin selection. By contrast, when a small probe of exact ties is available, influence measures spurious reliance on the probe rather than on the candidate pool and can limit this degradation. We test both predictions in two controlled settings:

  • Log-linear (500 candidate ties, 100 exact probe ties, Fig. 12, App. D.6). Leakage scales from 0 (exact ties) to 1 (the core difference of a typical strict comparison). At smaller scales and k=100k=100, the selectors differ by at most 0.61 points. At scale 1 and k=100k = 100, influence exceeds pure margin selection by 20.0 points of adversarial accuracy.
  • Hotels (Table 4, App. D.3). The tie operator also perturbs distance and price, with leakage δ∈{0.25,0.5,1}\delta \in \{0.25, 0.5, 1\} in units of utility; strict pairs differ by at least 5 units. Relative to exact ties, both selectors lose 1–2 points of adversarial accuracy at 500 ties and less than one point at 1,500 ties. Neither selector wins consistently.

Questions

Q1 is answered in W1; Q2 in W2, Concern 1; Q3 in W3, Concerns 1–2; and Q4 in W3, Concern 3, and W4.

Closing. We hope these clarifications address your concerns. If you have additional questions or concerns, please let us know. If not, we would appreciate an updated assessment of our paper.

Thank you for the additional baselines and experiments on larger models and more realistic datasets, which strengthen the empirical evidence supporting the paper. However, I still have some reservations regarding the theoretical novelty and practical applicability of the proposed approach. Nevertheless, I acknowledge the improvements and will raise my rating to 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We thank the reviewers for their careful feedback. In this comment, we restate our paper's objective and contributions, clarify our mathematical and experimental scope, and list the new evidence in this revision.

Reminder of contribution. Previous work (Moya et al., 2026) showed that ties reduce spurious reliance in preference optimization. To explain which ties drive this reduction and select them under a fixed budget, we analyze log-linear policies to first order. We make three contributions: (1) We decompose a tie's influence into its margin, which sets how strongly the tie pushes the policy, and its alignment, which sets whether this push opposes spurious reliance. Selecting the ties with the largest influence achieves first-order optimality (Proposition 1, Theorem 2). (2) Computing influence, however, requires curvature and spurious features that remain costly or unavailable in practice. Because margins require only forward passes, our practical pure margin selector ranks ties by absolute margin. Under a Gaussian model of the ties, this ranking maximizes the expected first-order reduction in spurious reliance among selectors that see only margins (Theorem 3). (3) Beyond log-linear policies, we estimate influence in neural networks (Section 5) and benchmark both selectors against uniform selection in neural networks and language models.

Scope of the mathematical analysis. To derive which ties to select, our analysis isolates how a single tie changes spurious reliance under two assumptions. First, following standard theory on spurious correlations (Arjovsky et al., 2019; Nagarajan et al., 2021; Veitch et al., 2021; Bombari & Mondelli, 2025), we assume the policy is log-linear in features that split into core and spurious parts. This log-linear structure makes the DPO loss convex, and our experiments verify that training reaches its unique minimizer (Appendix B.1). Second, we add ties to the training objective with a small weight and keep only their first-order effect. Because these first-order effects add up, choosing the best set of ties reduces to ranking individual ties. These guarantees do not certify neural networks or language models, which have non-convex losses and do not expose a split between core and spurious features. For those models, we test whether both selectors still outperform uniform selection (Appendix E).

Scope of the experiments. The neural network and language models test whether selection gains persist with hidden representations, autoregressive sequence modeling, and scale. To evaluate spurious reliance, our tasks control the spurious attribute. Specifically, we construct ties that change only this attribute and build test sets that reverse its correlation with the preference. For example, the emoji task inserts this controlled attribute into natural PKU-SafeRLHF responses (Ji et al., 2025). By contrast, our new OASST1 experiment (Köpf et al., 2023) examines a verbosity preference without imposing a correlation between verbosity and labels. In this setting, the strict checkpoint still prefers the verbose response in 89.5% of ties whose responses give the same answer (Appendix D.4).

New evidence in this revision.

  • Selection criteria from data selection (Appendix D.1; R1). In our log-linear model and neural network (five seeds), we adapted the gradient norm (Paul et al., 2021) and ActiveDPO (Lin et al., 2026) to ties. The gradient norm retains the margin and closely tracks pure margin selection. By contrast, ActiveDPO removes the margin and falls below uniform selection at intermediate budgets.
  • A 7B model (Appendix D.2; R1). On hotels with OLMo-3-7B (three seeds), pure margin selection reaches 34.4% accuracy under the reversed correlation with 500 ties, compared with 14.8% for uniform selection.
  • Ties that change core content (Appendices D.3 and D.6; R1, R2). On hotels, perturbing distance and price changes both selectors' accuracy by at most two points. In the log-linear model, influence outperforms pure margin selection with strongly perturbed candidates and a less perturbed probe.
  • Natural preference data (Appendix D.4; R1). On OASST1, both selectors reduce verbosity preference more than uniform selection up to 500 ties, while preserving preference accuracy.
  • Stability across seeds (Appendix D.5; R2). With Llama-3.2-1B, pure margin selection exceeds uniform selection in all 18 paired comparisons across seeds, budgets, and tasks.
  • Continued training (Appendix C.5.4; R2). On hotels, continuing from the strict checkpoint strengthens reliance on core features without reducing reliance on spurious features.

We answer the remaining concerns of each reviewer in our individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.