acceptodds
Under review as a conference paper at ICLR 2027

Sinkhorn-DPO: Distributionally Robust Preference Alignment via Entropic Optimal Transport

Abstract

Reinforcement learning from human feedback (RLHF) is a standard approach for aligning large language models (LLMs) with human preferences, while direct preference optimization (DPO) provides a simpler offline alternative that avoids explicit reward modeling. However, DPO remains sensitive to distribution shifts in preference data, motivating robust formulations based on distributionally robust optimization (DRO). Wasserstein-DPO (WDPO) methods provided a principled treatment of preference distribution shift, but their exact robust objectives involved nonsmooth dual structures that complicate gradient-based analysis and finite-sample estimation. We propose Sinkhorn-DPO (SDPO), a distributionally robust preference optimization framework based on entropically regularized optimal transport. Its dual formulation yields a smooth log-sum-exp envelope, enabling explicit derivative characterizations of the robust objective. Under mild regularity and a data-coverage condition imposed on the fixed reference measure, we prove smoothness and strong convexity of the profiled objective. Our finite-sample analysis establishes a parameter estimation rate of near- with sample size , which is a faster sample-size dependence than the bound of WDPO. The robustification is implemented through Sinkhorn-induced sample weights, so the policy is updated using a weighted first-order DPO gradient. We evaluate SDPO against DPO, Kullback-Leibler DPO (KLDPO), PO, and WDPO in linear experiments and in computational profiling with LLMs. The linear experiments examine radius calibration, parameter stability, and behavior under an induced within-corpus shift, while the LLM study reports runtime, peak GPU-memory usage and validation accuracy.

Primary area:
learning theory
Keywords:
Distributionally robust optimization, Direct preference optimization, Sinkhorn distance, Entropic optimal transport, LLM alignment, Distribution shift, Finite-sample analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_c54zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper suggests using a distributionally robust optimization called Sinkhorn-DPO for handling distribution shifts in LLMs.

Critical strengths

This work extends DPO through a distributionally robust optimization called Sinkhorn-DPO, which induces explicit DPO sample weights. This is easy to integrate with existing pipelines.The authors provide detailed analysis, strong convexity, and a faster parameter estimation rate than existing WDPO. The authors have shown their method is scalable as SDPO memory is dominated by static weights, unlike WDPO’s double backprop reliance.

Critical weaknesses

The conducted experiments are small-scale. The topic shift experiment is done on 80 training atoms, and the LLM comparison is also a single run of 150 steps. The experimental results show only marginal improvement, and up to a 5-times higher time cost of SDPO against baselines makes it the worst.

Questions for authors

The manuscript is readable and could benefit from more intuitive explanations (e.g., effective radius). Can you move important experimental settings to the main article for better clarity? Is it possible to assess the proposed method on Open LLM Leaderboard v2 and Emotion Alignment, similar to the prior work by Xu et al. (2025b), for a better comparison?

3 comments

We thank the reviewer for the constructive suggestions. We answer your comments below.

1. Experimental scale and computational trade-off. We acknowledge that our experiments are small scale. The linear distribution-shift study uses 80 training atoms per replication, while the LLaMA-3.1-8B experiment reports a single 150-step optimization trajectory (Figure 3, Appendix B.12). The available computational resources are limited to 32-GB V100 GPUs, which restrict the batch size and training budget; WDPO also encounters an out-of-memory error in our 8B configuration.

We acknowledge the additional computational cost of SDPO. In the linear benchmark, SDPO is approximately four to five times slower than WDPO. This overhead mainly comes from evaluating the entropic log-sum-exp envelope and solving the scalar dual optimization at each iteration, which are particularly visible when the underlying linear model computations are inexpensive (Appendix B.12).

The LLM profiling involves a different computational regime. As detailed in Appendix B.12, the SDPO runtime includes an upfront warm up calibration, involving multiple forward passes to estimate global feature scales without gradient tracking, as well as the scalar dual optimization for each mini-batch. Meanwhile, WDPO requires double backpropagation through its input gradient penalty approximation, adding substantial computational and intermediate memory overhead at larger model scales. Thus, the linear runtime ratio should not be directly extrapolated to LLM training. Although SDPO still incurs additional computation, its weighted first-order update avoids the double backpropagation overhead of WDPO and enables the tested 8B configuration to run within the 32GB V100 memory limit, whereas WDPO encounters an out-of-memory error.

For context, the LLaMA experiments in Xu et al. (2025b) used 8 H100 GPUs with an effective batch size of 128. Under our substantially more restricted hardware budget, conducting comparable multi-seed alignment experiments is not feasible. We therefore position the current experiments as preliminary evaluations of algorithmic feasibility, parameter behavior, and computational trade-offs, rather than evidence of superior downstream alignment quality.

2. Broader alignment benchmarks. We agree that Open LLM Leaderboard v2 and Emotion Alignment would provide valuable additional evaluations. However, such evaluations, especially with repeated seeds and distribution-shift comparisons, require substantially greater GPU resources than are currently available to us. For example, the WDPO study used a 40-GB A100 for its emotion-alignment experiments and 8 H100 GPUs for its LLaMA experiments (Xu et al., 2025b). Our present 32-GB V100 environment does not support comparable evaluation budgets.

The focus of this paper is the theoretical development of SDPO as a smooth, computationally tractable surrogate for WDPO, together with its finite sample guarantees, first-order implementation and practical implementation strategies. The current experiments provide initial evidence of its executability and memory advantages. A comprehensive comparison of downstream alignment quality on broader benchmarks remains an important direction for future work.

3. Effective radius and experimental settings. We have clarified the interpretation of the effective radius in Assumption 3.3 and Remark 3.4. The strict margin imposed in Assumption 3.3 ensures that the dual minimizer remains strictly positive, as required by our derivative based analysis. We have also included the principal experimental settings in the main text, including the preference data size, feature dimension, transport cost, regularization parameter, radius calibration, model scales, GPU configuration, and batch size. Core experimental configurations are summarized in Section 6.1, with full details in Appendix B to respect main text page limits.

We emphasize that the central contribution of this work is the development and theoretical analysis of an entropically smoothed transport robust DPO method. Its smooth dual enables a weighted first-order update without WDPO's double backpropagation, and its small scale experiments demonstrate practical execution and favorable memory scalability. Broader alignment evaluations under larger computational budgets remain future work.

Thank you for the response and the revised manuscript. After evaluating the response, I am keeping my rating, and my reasons are as listed below.

  1. No new experimental results were added. The topic-shift study is still using 80 training atoms, also the 8B comparison is done on a single 150-step run with a batch size of 2, where the gaps between methods are within noise (e.g., 0.645 vs 0.650). Although I understand the hardware limits you mentioned, part of my request seems feasible on a 32GB V100.
  2. I understand the clarification made for 4-5X figure relative to WDPO in the linear benchmark and should not be extrapolated to LLM training. However, Table 7 shows SDPO is 2-3X slower than DPO and is the slowest method in Figure 3. SDPO has a memory advantage over WDPO, but this is shown against one gradient-penalty implementation.

We thank the reviewer for the additional comments and appreciate the suggestion to further assess the method under the available GPU resources. Regarding the proposed alignment benchmarks, our current experiments already include LLM scale profiling on 32GB V100 GPUs, including LLaMA-3.1-8B with batch size 2 and a 150 step optimization trajectory. We also explored reduced budget comparisons under the same hardware constraints. To assess what can be reliably measured under our hardware, we conducted exploratory runs comparing DPO, WDPO, and KLDPO under the reduced batch sizes and shorter training horizons permissible on a 32GB V100. In those exploratory trials, even these established baselines produced closely overlapping metrics where small-batch variance obscured the clear performance ordering reported in Xu et al. (2025b). Although these configurations were executable, the resulting differences were too limited and unstable to provide a reliable ranking of alignment performance. Conducting further SDPO comparisons in a regime where the baselines themselves are indistinguishable would yield noisy, inconclusive figures rather than scientifically informative validation. We therefore focus the reported LLM experiments on optimization feasibility and computational profiling. For context, the WDPO study of Xu et al. (2025b, Appendix F) used a single 40GB A100 GPU for its emotion-alignment experiments and an 8×H1008 \times \text{H100} setup with effective batch size 128 for its LLaMA experiments. Their 8B leaderboard evaluation also reported KLDPO results due to computational constraints. These configurations illustrate the substantial difference between our available 32GB V100 environment and the computational budgets used for comprehensive large-scale alignment evaluations. In particular, a short executable run on a constrained GPU and a reproducible multi-seed benchmark under comparable training budgets represent different evaluation requirements. Regarding runtime, we acknowledge that SDPO incurs additional computational cost. Table 7 reports approximately 4.50–40.34 seconds per linear fit for SDPO, compared with 0.87–9.58 seconds for WDPO and approximately 0.02 seconds for DPO. The additional cost in the linear setting mainly comes from evaluating the entropic log-sum-exp envelope and solving the scalar dual optimization, whose overhead becomes particularly visible when the underlying model computation is inexpensive. At LLM scale, Appendix B.12 further identifies the upfront warm up calibration and the per-mini-batch dual optimization as the main sources of additional runtime. The computational advantage of SDPO is therefore primarily in memory scalability rather than wall clock speed. WDPO's input gradient penalty requires double backpropagation, resulting in substantial intermediate activation memory. Under our matched 32GB V100 setting, SDPO completes the tested 8B configuration, whereas WDPO encounters an out-of-memory error. We acknowledge that this comparison concerns the tractable gradient penalty WDPO implementation evaluated in our experiments, and interpret the result within this specific computational setting. WDPO is the most direct methodological comparator because SDPO develops an entropically smoothed transport-robust alternative to Wasserstein-DPO, while DPO, KLDPO, and χ\chiPO provide additional baselines. Our empirical results consequently characterize the computational trade-off of this first-order approach, rather than claiming a general runtime advantage or a statistically established improvement in downstream alignment. We agree that broader multi-seed evaluations on Open LLM Leaderboard v2 and Emotion Alignment would provide valuable additional evidence. Such evaluations remain an important direction under larger computational budgets. We believe these experimental considerations should be considered alongside the paper's principal contributions: the smooth Sinkhorn-DPO formulation, its finite-sample theoretical analysis, and its executable first-order optimization pathway.

Reviewer_8rWbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper discussing post-training (RLHF vs DPO) variants, and propose Sinkhorn-DPO (SDPO) based on regularized OT. The evaluations compare SDPO against DPO, KLDPO, XPO, WDPO in linear experiments and profiling with LLMs. The linear experiments examine radius calibration, parameter stability, and behavior under an induced within-corpus shift, and the LLM study reports runtime, peak GPU-memory usage and validation accuracy.

Critical strengths
  • Once the scalar dual variable is selected, the method constructs normalized transport weights and updates the policy through a weighted sum of ordinary DPO gradients. This yields a practical implementation of the log-sum-exp objective without differentiating through a gradient penalty. T

  • The log-sum-exp objective with λ>0\lambda>0: the paper derives gradients, Hessians, and mixed derivatives. The covariance and Schur-complement argument also provide a nice way to control the curvature change caused by profiling over λ\lambda. These are substantive analytical ingredients, but the global smoothness and statistical conclusions depend on assumptions that require correction.

  • Profiling covers models from GPT-2 to LLaMA-3.1-8B, with batch sizes and sequence lengths matched across methods at each model scale. SDPO completes the tested 8B configuration on a 32GB V100, and the WDPO implementation using double backprop runs out of memory. The authors also specify the additional runtime of SDPO.

Critical weaknesses
  • Assumption 3.1 places θref\theta_{\mathrm{ref}} inside Θ\Theta. At θ=θref\theta=\theta_{\mathrm{ref}}, however, the DPO score is identically zero and ℓ(z;θref)=log⁡2\ell(z;\theta_{\mathrm{ref}})=\log 2 for every zz, contradicting Assumption 3.2’s requirement that the loss be non-constant for every parameter. Moreover, Aθref=ZA_{\theta_{\mathrm{ref}}}=\mathcal Z implies ρˉcrit(θref)=0\bar\rho_{\mathrm{crit}}(\theta_{\mathrm{ref}})=0, making the strictly positive radius required by Assumption 3.3 impossible. Therefore, Lemma 3.5 and Theorems 3.8 and 4.1 have unsatisfiable joint hypotheses in the stated model.

  • Section 2 defines the entropy penalty relative to the product of the two transport marginals, KL(π∥P⊗Q)\mathrm{KL}(\pi\|P\otimes Q), including the unknown adversarial distribution. Proposition 2.2 invokes the log-sum-exp dual associated with a fixed reference measure ν\nu. These formulations are not interchangeable: with nominal marginal P^\widehat P and adversarial marginal PP, the fixed-reference penalty additionally includes KL(P∥ν)\mathrm{KL}(P\|\nu), which varies with PP. The cited Sinkhorn-DRO result explicitly uses the fixed-reference formulation. Therefore, the proposed dual and subsequent weighting algorithm do not correspond to the primal ambiguity set as currently written.

  • Theorem 4.1 fixes ν\nu and studies estimation of the outer nominal distribution. In contrast, Appendix B.3 re-estimates ν\nu from each bootstrap sample, while the LLM implementation uses the current mini-batch as both the nominal and reference support. The appendix acknowledges that this introduces bias through the nonlinear log-sum-exp operation, but the analysis supplies no reference-estimation or minibatch-bias term. Therefore, even after correcting the preceding mathematical issues, the stated O(log⁡n/n)O(\sqrt{\log n/n}) guarantee would not yet apply to the implemented estimator.

  • The linear shift results are mixed: SDPO loses to DPO on all three nominal clusters, and its advantage over χ\chiPO on the largest held-out cluster is only 0.5157 versus 0.5115. The LLM experiments primarily establish computational feasibility, with no explicit distribution-shift evaluation. After 150 optimization steps, SDPO reaches validation preference accuracy 0.650 versus 0.645 for DPO and χ\chiPO, without reported uncertainty for this comparison. These results are insufficient to establish improved robust alignment quality independently of the theory claims.

Questions for authors
  • How can Assumptions 3.2 and 3.3 hold at θ=θref\theta=\theta_{\mathrm{ref}}, where the loss is constant and the critical effective radius is zero? Can the authors provide a revised theorem that handles the reference policy and boundary dual solutions with λ⋆=0\lambda^\star=0, or clearly specify a consistent restricted regime? This directly determines if the main smoothness and estimation guarantees remain valid.

  • Can the authors establish an estimation guarantee for the sample-dependent reference measure actually used in the bootstrap experiments, and quantify the mini-batch approximation used at LLM scale?

  • Do the reported predictive improvements persist across training seeds and under an explicit LLM preference-distribution shift? Please clarify the validation-set size and uncertainty of the 0.650 versus 0.645 comparison. In the linear experiments, can parameter norms, prediction margins, or controls with comparable regularization distinguish improved robustness from shrinkage toward the reference policy, particularly given the cross-validation losses near log⁡2\log 2?

3 comments

We thank Referee 2 for the thorough theoretical review and constructive questions. We have revised the theoretical assumptions, clarified the reference measure conventions in the algorithm and experiments, and added numerical diagnostics. We respond to the main concerns below.

1. Consistency of the theoretical assumptions. We agree that the original assumptions were inconsistent at θref\theta_{\mathrm{ref}}, where the DPO loss is constant. In the revised manuscript, Assumption 3.1 now defines Θ⊆θ∈Rd:∥θ∥2≤B\Theta \subseteq \\{\theta \in \mathbb{R}^d : \lVert \theta \rVert_2 \le B\\} as a compact convex set satisfying dist⁡(Θ,θref)≥δθ>0\operatorname{dist}(\Theta, \theta_{\mathrm{ref}}) \ge \delta_\theta \gt 0 (Remark 3.8). Assumptions 3.2 and 3.3 further impose uniform loss non-degeneracy and a strictly positive margin from the critical effective radius. Under these consistent conditions, Lemma 3.5 ensures an interior, strictly positive dual minimizer λ⋆(θ)∈[λ‾,λ‾]⊂(0,∞)\lambda^\star(\theta) \in [\underline{\lambda}, \overline{\lambda}] \subset (0,\infty) (excluding λ⋆=0\lambda^\star=0), and Theorems 3.9 and 4.1 hold rigorously. In addition, Theorem 4.3 extends the finite-sample estimation result to a finite union of convex domains U=⋃i=1NΘi\mathcal{U} = \bigcup_{i=1}^N \Theta_i covering an annular region around the reference parameter θref\theta_{\mathrm{ref}}. Practically, Algorithm 1 optimizes over BB\mathbb{B}_B for convenience, and Table 3 confirms the iterates stay strictly separated from θref\theta_{\mathrm{ref}}.

2. Fixed reference Sinkhorn distance in Section 2. We revised Section 2 to define the primal transport cost Wϵν(Po,P)W_\epsilon^\nu(P_o, P) with relative entropy H(π∥Po⊗ν)H(\pi \parallel P_o \otimes \nu) against the fixed reference measure ν∈P(Z)\nu \in \mathcal{P}(\mathcal{Z}), aligning the ambiguity set Bρ,ϵν(Po)\mathcal{B}_{\rho,\epsilon}^\nu(P_o) exactly with Proposition 2.2 from Wang et al. (2026).

3. Fixed ν\nu in theory vs. empirical reference protocols. We have clarified the relationship between the fixed-reference theoretical formulation, Algorithm 1, and the reference-measure choices in different experiments:

  • Theory and core algorithm: Theorems 4.1 and 4.3 assume a fixed reference measure ν\nu. Consistently, Algorithm 1 specifies the discrete reference measure ν\nu on R\mathcal{R} before the optimization loop; only the outer nominal support Bt\mathcal{B}_t is sampled during iterations. Thus, Bt\mathcal{B}_t and R\mathcal{R} play distinct roles.
  • CV and linear experiments: In each linear training fit, R\mathcal{R} is constructed from that fit's training support and remains fixed throughout optimization. The same convention is used in cross-validation: the reference support is the training support, including during held-out evaluation, where only the outer support is changed to the test set. In the bootstrap stability study, for each sample size nn and replication, the bootstrap support is used for both the outer and reference empirical measures. This keeps the two supports matched to the available bootstrap sample when examining parameter stability across sample sizes. Accordingly, ν\nu may vary across bootstrap samples and sample sizes, but remains fixed within each individual fit. We acknowledge that the resulting bootstrap protocol involves a sample-dependent reference measure, which is beyond the fixed-ν\nu theory of Theorem 4.1.
  • LLM mini-batching: For LLM training, the full reference support is computationally expensive under our 32GB V100 memory constraint. We therefore use each mini-batch drawn from the fixed training corpus as both the outer and reference support at that iteration. This provides a practical stochastic approximation to the full empirical Sinkhorn objective. As clarified in Appendix B.3, the resulting approximation is generally biased because the empirical average appears inside the nonlinear log-sum-exp operation; related biased mini-batch estimators were also discussed in Sinkhorn-DRO (Wang et al., 2026), whose theoretical quantification remains open.

4. Empirical performance, parameter behavior, and computational resources.

  • Separating robustness from shrinkage toward θref\theta_{\mathrm{ref}}: Section 6.2 and Table 3 (Appendix B.5) report ∥θ^k−θref∥2\lVert \widehat{\theta}_k - \theta_{\mathrm{ref}} \rVert_2 and mean held-out ∣margin∣\lvert \mathrm{margin} \rvert. Across non-degenerate CV radii (ρ≥0.10\rho \ge 0.10), ∥θ^k−θref∥2\lVert \widehat{\theta}_k - \theta_{\mathrm{ref}} \rVert_2 grows monotonically from 0.0327 (ρ=0.10\rho=0.10) and 0.0439 (ρcv=0.15\rho_{\mathrm{cv}}=0.15) to 0.3141 (ρ=0.70\rho=0.70), and ∣margin∣\lvert \mathrm{margin} \rvert grows from 0.0022 to 0.0145. Moreover, the full-pool parameter norms of SDPO and WDPO are comparable across the reported optimization horizons. Together with the similar bootstrap deviations (0.0468 for SDPO and 0.0466 for WDPO at n=2,000n=2{,}000), these results provide preliminary evidence of related parameter stability behavior in the linear setting.

    On full-pool iterations T∈60,120,240T \in \\{60, 120, 240\\} (ρcv=0.15\rho_{\mathrm{cv}}=0.15), SDPO maintains norms of 0.0397, 0.0334, 0.0395 (comparable to WDPO's 0.0533, 0.0531, 0.0340). This shows active parameter separation from θref=0\theta_{\mathrm{ref}}=0 rather than infinite shrinkage, while the near log⁡2\log 2 losses and 0.49–0.64 accuracies across all five baselines reflect the limited capacity of 24-dim TF-IDF/SVD features (Section 6.3).

    We agree that these diagnostics do not establish better policy quality. The linear benchmark uses a rank-24 TF-IDF/SVD representation and produces modest preference accuracies, making it unsuitable for reliably distinguishing the alignment quality of the different DPO variants. The topic-shift results are also mixed. We therefore interpret these experiments as evaluations of numerical feasibility, parameter behavior, and sensitivity to the tested distribution changes.

  • LLM validation scope: Regarding LLM evaluation, the 8B results are based on a single run of 150 steps with batch size 2 (Table 1). Our aim is not to claim that the small accuracy differences establish statistically significant improvements. Our available computational resources are limited to 32-GB V100 GPUs. By comparison, the LLaMA experiments in Xu et al. (2025b) used 8 H100 GPUs with an effective batch size of 128. Consequently, a reliable multi-seed comparison of alignment quality under comparable large-scale training configurations is beyond our current resources. Running all baselines under substantially reduced configurations would not reproduce the training conditions or performance levels reported in those literature.

    Instead, our contribution focuses on establishing SDPO as a theoretically sound, smoothed surrogate for WDPO (with matching finite sample parameter stability in Figure 2(a)) and demonstrating an executable first-order pathway that scales to 8B where WDPO fails with OOM (Sections 6.5, 7).

The central contribution of this work is the development and theoretical analysis of an entropically smoothed transport-robust DPO method. Its smooth dual enables a weighted first-order update without WDPO's double backpropagation, and its small-scale experiments demonstrate practical execution and favorable memory scalability. Broader alignment evaluations under larger computational budgets remain future work.

I thank the authors for the rebuttal, and apologize for my late response. This has been taken into consideration.

The 2 mathematical revisions is the right move, and there is now a more defensible theoretical result. However the theory covering the bootstrap and LLM using biased approximation has been noted yet not resolved. In fact the nonzero parameter norms and small preference margins reinforce my concerns. There remain several of my concerns that are acknowledged in the rebuttal but not resolved.

The new margin details are the least convincing part of the rebuttal, and actually strengthens my shrinkage point (especially Table 3). The stability comparison is compounded with parameter scale. The description of the exp protocol is now more clear, but it does not repair the theorem's validity. The LLM approximation is still an unresolved concern.

Given these considerations, I shall maintain my score, but encourage the authors to revise the manuscript without over-claiming and with better clarity. 5 assumptions, 3 remarks, 2 theorems, 1 prerequisite, within 9 pages is a lot. Many of these are written back to back and essentially two-liners. The combination of the unsteady theory, unclear results, and difficult reading is a hard gap to overcome.

Reviewer_4iXEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Sinkhorn-DPO (SDPO), a distributionally robust optimization (DRO) algorithm for preference optimization, designed to handle potential distribution shift in preference data. Building on the Sinkhorn-DRO framework of Wang et al. (2026), the authors derive a smooth dual formulation that leads to a weighted first-order DPO algorithm. The main advantage over WDPO is that the proposed method avoids double backpropagation, reducing GPU memory requirements. The authors also establish a parameter estimation rate of O(log⁡n/n)O(\sqrt{\log n/n}), improving upon the O(n−1/4)O(n^{-1/4}) bound previously established for WDPO. The method is evaluated through experiments on parameter stability, distribution shift, computational efficiency, and LLM preference accuracy.

Critical strengths

The proposed method offers an appealing way to incorporate DRO into DPO through adversarial sample reweighting, while retaining first-order policy updates. In particular, avoiding the double-backpropagation overhead of WDPO provides a meaningful computational advantage. The experiments demonstrate that SDPO can be applied to an 8B language model under memory constraints where WDPO fails. I appreciate this practical improvement, especially given the computational challenges of applying DRO to large-scale preference optimization.

Critical weaknesses

While SDPO seems to offer statistical and computational advantages over WDPO, I have several concerns regarding the mathematical formulation and the actual benefits for preference alignment.

  1. (Mathematical validity) I found two potentially serious issues with the theoretical formulation and analysis.

    • Proposition 2.2: The Sinkhorn distance defined in Section 2 uses entropic regularization with respect to P⊗QP\otimes Q. However, the dual formulation borrowed from Wang et al. (2026) appears to require a fixed reference measure ν\nu for the adversarial marginal; see Remark 2 of Wang et al. (2026). These formulations are generally not equivalent, as they differ by an additional KL divergence term. Consequently, I do not see how Proposition 2.2 follows from the stated definition. Since the subsequent theoretical development and algorithm rely on this proposition, I consider this a fundamental concern.

    • Assumptions 3.1--3.3: These assumptions also appear mutually inconsistent. At θ=θref∈Θ\theta=\theta_{\mathrm{ref}}\in\Theta (by Assumption 3.1), the DPO loss is identically log⁡2\log 2, which contradicts the non-degeneracy condition in Assumption 3.2. Furthermore, the critical effective radius in Assumption 3.3 becomes zero at this parameter, contradicting the strictly active condition. Hence, I do not see how the assumptions required for Theorems 3.8 and 4.1 can simultaneously hold.

    These issues are particularly concerning because the theoretical guarantees are presented as one of the main contributions of the paper.

  2. (Unclear benefits for preference alignment) While the paper clearly motivates its computational advantages over WDPO, I am less convinced by its statistical and alignment benefits.

    • Statistical guarantees: Theorem 4.1 establishes a parameter estimation rate relative to the population optimizer of the proposed robust objective. However, I am not sure whether this is the most informative guarantee for preference alignment. What ultimately matters is the quality of the learned policy, rather than parameter estimation alone. The authors do not sufficiently explain how their improved parameter rate translates into better alignment. Moreover, the comparison with WDPO concerns different population objectives, so the faster rate does not necessarily imply better alignment performance.

    • Empirical results: The experiments do not convincingly demonstrate improved alignment. Figure 2(a) measures parameter stability relative to each method's own full-data solution, which does not necessarily indicate better preference learning. Figure 2(b) shows mixed results under distribution shift, with no consistent advantage over other robust baselines. In Table 1, SDPO achieves a slightly higher peak validation accuracy, but the differences from DPO and χ\chiPO are marginal, especially at the final checkpoint. Without uncertainty estimates or repeated runs, it is difficult to assess whether these improvements are meaningful.

  3. (Presentation and novelty) Finally, the method itself could be explained much more clearly. My understanding is that it essentially applies the existing Sinkhorn-DRO framework to the DPO loss, resulting in an adversarially weighted DPO gradient. However, this interpretation is surprisingly difficult to extract from the presentation. I would appreciate a clearer distinction between the contributions inherited from Sinkhorn-DRO and those specific to preference optimization.


Overall, while the computational advantage over WDPO is useful, I do not find sufficient evidence that the proposed method leads to meaningfully better preference alignment. If there is any misunderstanding in my review, I would be happy to correct my assessment.

Questions for authors
  1. (Proposition 2.2) Could the authors clarify the discrepancy between the Sinkhorn distance defined in Section 2 and the dual formulation in Proposition 2.2? In particular, does the intended formulation use a fixed reference measure ν\nu, as in the Sinkhorn-DRO duality result of Wang et al. (2026)? If not, how is Proposition 2.2 derived?

  2. (Theoretical assumptions) How can Assumptions 3.2 and 3.3 hold at θ=θref\theta=\theta_{\mathrm{ref}}, where the DPO loss is constant? Can the authors establish the main theoretical results under a consistent set of assumptions?

  3. (Statistical guarantees) Theorem 4.1 establishes a parameter estimation rate relative to the population optimizer of SDPO. However, since SDPO and WDPO optimize different population objectives, it is unclear whether the improved rate implies better preference alignment. Could the authors clarify the implications of this guarantee for the learned policy, particularly its preference risk under distribution shift?

  4. (Empirical improvements) Could the authors provide stronger evidence that SDPO improves preference alignment over DPO and χ\chiPO, particularly under distribution shift? Repeated experiments with uncertainty estimates would help determine whether the relatively small improvements in Table 1 are meaningful.

4 comments

We thank the reviewer for the detailed and constructive comments. We have revised the manuscript to resolve the mathematical concerns, clarified the methodological contribution, and added numerical diagnostics. Our responses are as follows.

1. Mathematical validity of Proposition 2.2 and Sinkhorn ambiguity set. We agree with the reviewer that the original formulation was inconsistent. In Section 2, we revised the primal transport cost to the fixed reference formulation

W_\epsilon^\nu(P_o, P) := \inf_{\pi \in \Pi(P_o, P)} \left\\{ \int_{\mathcal{Z}\times\mathcal{Z}} c(x,z)\\,d\pi(x,z) + \epsilon H(\pi \parallel P_o \otimes \nu) \right\\}

and the ambiguity set to Bρ,ϵν(Po):=P∈P(Z):Wϵν(Po,P)≤ρ\mathcal{B}_{\rho,\epsilon}^\nu(P_o) := \\{P \in \mathcal{P}(\mathcal{Z}) : W_\epsilon^\nu(P_o, P) \le \rho\\}. This is precisely the fixed reference measure formulation underlying the Sinkhorn-DRO duality of Wang et al. (2026). The revised dual formulation therefore follows directly from Wang et al. (2026).

2. Consistency of Assumptions 3.1–3.3. We agree that the original assumptions were incompatible at θref\theta_{\mathrm{ref}}. In Section 3, we resolved this via two revisions:

  • Assumption 3.1 now restricts Θ⊆θ∈Rd:∥θ∥2≤B\Theta \subseteq \\{\theta \in \mathbb{R}^d : \lVert \theta \rVert_2 \le B\\} to a compact convex set separated from θref\theta_{\mathrm{ref}} (dist⁡(Θ,θref)≥δθ>0\operatorname{dist}(\Theta, \theta_{\mathrm{ref}}) \ge \delta_\theta \gt 0; see Remark 3.8), while Assumptions 3.2–3.3 impose uniform variance (σℓ2>0\sigma_\ell^2 \gt 0) and active-radius (δρ>0\delta_\rho \gt 0) margins on Θ\Theta. Under these consistent hypotheses, Theorems 3.9 and 4.1 establish local mFm_F-strong convexity and the O(log⁡n/n)\mathcal{O}(\sqrt{\log n/n}) rate.
  • Theorem 4.3 and Definition A.7/Remark A.8 further extend the O(log⁡n/n)\mathcal{O}(\sqrt{\log n/n}) parameter guarantee to finite union of convex domains U=⋃i=1NΘi\mathcal{U} = \bigcup_{i=1}^N \Theta_i covering an annular region around θref\theta_{\mathrm{ref}}. Empirically, our new diagnostics (Section 6.2, Table 3) show that practical iterates over BB\mathbb{B}_B remain strictly bounded away from θref=0\theta_{\mathrm{ref}}=0.

3. Statistical guarantees and their implications for alignment. We agree that SDPO and WDPO optimize different population objectives (Sinkhorn vs. Wasserstein balls). Rather, comparing estimation rates across distinct robust objectives is standard in DRO-DPO research (Xu et al., 2025b; Huang et al., 2025; Chowdhury et al., 2024). Here, our aim is to show that SDPO has similar behavior to other DPO-variant methods in the current toy small size linear models to show its preliminary feasible. In addition, we notice that SDPO is naturally regarded as an entropically smoothed surrogate for WDPO (Wang et al., 2026; Cuturi, 2013). Replacing WDPO's nonsmooth Moreau–Yosida infimum with a smooth log-sum-exp envelope removes the O(n−1/4)\mathcal{O}(n^{-1/4}) square root bottleneck and yields the O(log⁡n/n)\mathcal{O}(\sqrt{\log n/n}) rate toward the Sinkhorn robust target. Consistently, our finite-sample linear study (Figure 2(a), Table 4) shows that SDPO and WDPO achieve nearly identical parameter deviations (0.0468 vs. 0.0466 at n=2,000n=2{,}000, vs. >0.91\gt 0.91 for DPO/KLDPO/χ\chiPO), supporting SDPO's role as an analytically tractable surrogate for WDPO.

4. Empirical evidence, computational limitations and core contribution. We emphasize that the primary contribution of this work is theoretical analysis and an executable first-order surrogate for WDPO, rather than comparison about large-scale alignment tuning performance numerical results:

  • Linear experiments: On the 24-dimensional TF-IDF/SVD features, all five DPO variants stay in a modest 0.49–0.64 accuracy range (Section 6.3), indicating that low-dimensional linear representations cannot meaningfully differentiate downstream policy quality. Instead, the linear benchmarks verify radius calibration, parameter stability (Figure 2(a)), and non-collapse away from θref\theta_{\mathrm{ref}} (Table 3).
  • LLM compute constraint: Prior large scale LLM alignment comparisons across DPO variants rely on multi-GPU clusters (e.g., 8×H1008 \times \text{H100} in Xu et al. (2025b)). Under our single 32GB V100 GPU environment, reproducing full scale multi-seed alignment benchmarks from prior work is computationally infeasible and cannot match results obtained with cluster scale compute. Consequently, our 8B experiments (Table 1, Figure 3) are designed to verify first-order optimization feasibility and memory scalability (completing 8B training where WDPO fails with OOM), while large scale OOD alignment evaluation remains compute gated future work (Sections 6.5, 7). We hope the paper is evaluated on its core theoretical guarantees and memory feasible algorithmic pathway rather than cluster scale empirical benchmarks unattainable under our hardware constraints.

5. Specific contributions beyond generic Sinkhorn-DRO. We have clarified the distinction between the inherited Sinkhorn-DRO framework and our specific contributions. The entropic transport duality representation from follows from Wang et al. (2026), while our contributions are its specialization to DPO problems: (i) deriving uniform pointwise loss, gradient, and Hessian bounds for log-linear DPO (Lemma A.3); (ii) proving strong convexity of the objective while relaxing data-coverage assumptions to a single fixed reference measure ν\nu (Theorem 3.9, Remark 3.7); (iii) establishing O(log⁡n/n)\mathcal{O}(\sqrt{\log n/n}) guarantees on separated convex and annular domains (Theorems 4.1, 4.3); and (iv) translating the dual into explicit first-order sample weights (Algorithm 1) that eliminate WDPO's double backpropagation.

I appreciate the authors' detailed response and their acknowledgment of the mathematical inconsistencies in the original formulation and assumptions.

However, I remain concerned about the implications of the revised theoretical assumptions. In particular, the new analysis restricts the parameter domain to be separated from θref\theta_{\mathrm{ref}}, whereas the original algorithm initializes at θref\theta_{\mathrm{ref}} and optimizes over the full parameter ball. Although the authors provide empirical diagnostics in Table 3, these only report the distances of fitted parameters at selected checkpoints, and do not establish that the revised assumptions hold for the original optimization problem. In particular, it remains unclear whether the new statistical guarantees apply to the estimator produced by the proposed algorithm.

Given that the main contribution of this submission is theoretical, as the authors themselves emphasize, I believe these substantial revisions to the mathematical formulation and assumptions require more careful development and validation. My concerns regarding the limited empirical evidence of improved preference alignment also remain.

I therefore maintain my original assessment.

We thank the reviewer for the follow up comments. We would like to further clarify the relationship between our theoretical guarantees and the practical training algorithm.

First, Theorems 4.1 and 4.3 establish statistical guarantees for constrained empirical estimators over separated convex domains and their finite union, respectively. Algorithm 1, in contrast, provides a practical first-order training procedure over the bounded parameter ball BB\mathbb{B}_B. Its purpose is to translate the Sinkhorn dual formulation into explicit sample weights and executable DPO gradient updates. Furthermore, Definition A.7, Remark A.8, and Theorem 4.3 extend this guarantee to a finite union of convex slices Uα=⋃i=1NΘi\mathcal{U}_\alpha = \bigcup_{i=1}^N \Theta_i, where Uα\mathcal{U}_\alpha can cover the parameter ball outside an arbitrarily small neighborhood of θref\theta_{\mathrm{ref}}. On each convex slice Θi\Theta_i, strong convexity ensures a unique minimizer θi⋆\theta_i^\star, and Theorem 4.3 (together with Remark A.9) controls both the distance dist⁡(θ^n,SU⋆)\operatorname{dist}(\widehat{\theta}_n, \mathcal{S}_{\mathcal{U}}^\star) and the excess robust preference risk. Therefore, the finite sample rate is an estimator-level theoretical guarantee under the stated assumptions, while Algorithm 1 provides its practical first-order implementation.

This distinction also appears in closely related DRO-DPO literature. In particular, Xu et al. (2025b, Sections 5.1–5.2 and 6) established sample complexity guarantees for the empirical WDPO and KLDPO robust objectives, while their practical algorithms employed a gradient penalty approximation for WDPO and an approximate worst-case kernel for KLDPO. Thus, the theoretical analysis of an idealized robust estimator and the development of a tractable large scale training algorithm are complementary components of distributionally robust preference optimization. We believe our theoretical and algorithmic contributions should be assessed within this established methodological context.

For computational convenience, Algorithm 1 performs first-order updates directly over BB\mathbb{B}_B rather than solving NN separate directional subproblems over Θ1,…,ΘN\Theta_1, \dots, \Theta_N.

Regarding Table 3, its purpose is specifically to document parameter non-collapse in the tested linear experiments. It reports the distances of the fitted CV parameters from θref\theta_{\mathrm{ref}}, together with the terminal parameter norms at T∈60,120,240T\in\\{60,120,240\\} in the full pool experiments. Figure 2(a) complements this analysis by reporting bootstrap parameter deviations from the corresponding full pool solutions across sample sizes. Together, these diagnostics provide evidence of non-collapse at the reported fitted solutions and finite sample parameter stability.

We therefore emphasize the distinction between the scope of the statistical theorems and the practical implementation. The stated theoretical results remain valid under their consistent hypotheses, and establish the smoothness, strong convexity, and near-parametric estimation rate of the proposed Sinkhorn robust objective on the specified domains. Meanwhile, the practical algorithm demonstrates how its smooth dual can be converted into an executable weighted first-order DPO update, with substantial memory advantages over the tested WDPO implementation at the 8B scale.

We also clarify that the parameter estimation rate concerns the population minimizer of the corresponding robust objective, rather than a direct ranking of the alignment performance across different robust objectives. The principal contribution is the combination of a tractable entropic robust formulation, its derivative-based statistical analysis, and a memory efficient first-order implementation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.