- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper suggests using a distributionally robust optimization called Sinkhorn-DPO for handling distribution shifts in LLMs.
Critical strengths
This work extends DPO through a distributionally robust optimization called Sinkhorn-DPO, which induces explicit DPO sample weights. This is easy to integrate with existing pipelines.The authors provide detailed analysis, strong convexity, and a faster parameter estimation rate than existing WDPO. The authors have shown their method is scalable as SDPO memory is dominated by static weights, unlike WDPO’s double backprop reliance.
Critical weaknesses
The conducted experiments are small-scale. The topic shift experiment is done on 80 training atoms, and the LLM comparison is also a single run of 150 steps. The experimental results show only marginal improvement, and up to a 5-times higher time cost of SDPO against baselines makes it the worst.
Questions for authors
The manuscript is readable and could benefit from more intuitive explanations (e.g., effective radius). Can you move important experimental settings to the main article for better clarity? Is it possible to assess the proposed method on Open LLM Leaderboard v2 and Emotion Alignment, similar to the prior work by Xu et al. (2025b), for a better comparison?
3 comments
Thank you for the response and the revised manuscript. After evaluating the response, I am keeping my rating, and my reasons are as listed below.
- No new experimental results were added. The topic-shift study is still using 80 training atoms, also the 8B comparison is done on a single 150-step run with a batch size of 2, where the gaps between methods are within noise (e.g., 0.645 vs 0.650). Although I understand the hardware limits you mentioned, part of my request seems feasible on a 32GB V100.
- I understand the clarification made for 4-5X figure relative to WDPO in the linear benchmark and should not be extrapolated to LLM training. However, Table 7 shows SDPO is 2-3X slower than DPO and is the slowest method in Figure 3. SDPO has a memory advantage over WDPO, but this is shown against one gradient-penalty implementation.
We thank the reviewer for the additional comments and appreciate the suggestion to further assess the method under the available GPU resources. Regarding the proposed alignment benchmarks, our current experiments already include LLM scale profiling on 32GB V100 GPUs, including LLaMA-3.1-8B with batch size 2 and a 150 step optimization trajectory. We also explored reduced budget comparisons under the same hardware constraints. To assess what can be reliably measured under our hardware, we conducted exploratory runs comparing DPO, WDPO, and KLDPO under the reduced batch sizes and shorter training horizons permissible on a 32GB V100. In those exploratory trials, even these established baselines produced closely overlapping metrics where small-batch variance obscured the clear performance ordering reported in Xu et al. (2025b). Although these configurations were executable, the resulting differences were too limited and unstable to provide a reliable ranking of alignment performance. Conducting further SDPO comparisons in a regime where the baselines themselves are indistinguishable would yield noisy, inconclusive figures rather than scientifically informative validation. We therefore focus the reported LLM experiments on optimization feasibility and computational profiling. For context, the WDPO study of Xu et al. (2025b, Appendix F) used a single 40GB A100 GPU for its emotion-alignment experiments and an setup with effective batch size 128 for its LLaMA experiments. Their 8B leaderboard evaluation also reported KLDPO results due to computational constraints. These configurations illustrate the substantial difference between our available 32GB V100 environment and the computational budgets used for comprehensive large-scale alignment evaluations. In particular, a short executable run on a constrained GPU and a reproducible multi-seed benchmark under comparable training budgets represent different evaluation requirements. Regarding runtime, we acknowledge that SDPO incurs additional computational cost. Table 7 reports approximately 4.50–40.34 seconds per linear fit for SDPO, compared with 0.87–9.58 seconds for WDPO and approximately 0.02 seconds for DPO. The additional cost in the linear setting mainly comes from evaluating the entropic log-sum-exp envelope and solving the scalar dual optimization, whose overhead becomes particularly visible when the underlying model computation is inexpensive. At LLM scale, Appendix B.12 further identifies the upfront warm up calibration and the per-mini-batch dual optimization as the main sources of additional runtime. The computational advantage of SDPO is therefore primarily in memory scalability rather than wall clock speed. WDPO's input gradient penalty requires double backpropagation, resulting in substantial intermediate activation memory. Under our matched 32GB V100 setting, SDPO completes the tested 8B configuration, whereas WDPO encounters an out-of-memory error. We acknowledge that this comparison concerns the tractable gradient penalty WDPO implementation evaluated in our experiments, and interpret the result within this specific computational setting. WDPO is the most direct methodological comparator because SDPO develops an entropically smoothed transport-robust alternative to Wasserstein-DPO, while DPO, KLDPO, and PO provide additional baselines. Our empirical results consequently characterize the computational trade-off of this first-order approach, rather than claiming a general runtime advantage or a statistically established improvement in downstream alignment. We agree that broader multi-seed evaluations on Open LLM Leaderboard v2 and Emotion Alignment would provide valuable additional evidence. Such evaluations remain an important direction under larger computational budgets. We believe these experimental considerations should be considered alongside the paper's principal contributions: the smooth Sinkhorn-DPO formulation, its finite-sample theoretical analysis, and its executable first-order optimization pathway.
We thank the reviewer for the constructive suggestions. We answer your comments below.
1. Experimental scale and computational trade-off. We acknowledge that our experiments are small scale. The linear distribution-shift study uses 80 training atoms per replication, while the LLaMA-3.1-8B experiment reports a single 150-step optimization trajectory (Figure 3, Appendix B.12). The available computational resources are limited to 32-GB V100 GPUs, which restrict the batch size and training budget; WDPO also encounters an out-of-memory error in our 8B configuration.
We acknowledge the additional computational cost of SDPO. In the linear benchmark, SDPO is approximately four to five times slower than WDPO. This overhead mainly comes from evaluating the entropic log-sum-exp envelope and solving the scalar dual optimization at each iteration, which are particularly visible when the underlying linear model computations are inexpensive (Appendix B.12).
The LLM profiling involves a different computational regime. As detailed in Appendix B.12, the SDPO runtime includes an upfront warm up calibration, involving multiple forward passes to estimate global feature scales without gradient tracking, as well as the scalar dual optimization for each mini-batch. Meanwhile, WDPO requires double backpropagation through its input gradient penalty approximation, adding substantial computational and intermediate memory overhead at larger model scales. Thus, the linear runtime ratio should not be directly extrapolated to LLM training. Although SDPO still incurs additional computation, its weighted first-order update avoids the double backpropagation overhead of WDPO and enables the tested 8B configuration to run within the 32GB V100 memory limit, whereas WDPO encounters an out-of-memory error.
For context, the LLaMA experiments in Xu et al. (2025b) used 8 H100 GPUs with an effective batch size of 128. Under our substantially more restricted hardware budget, conducting comparable multi-seed alignment experiments is not feasible. We therefore position the current experiments as preliminary evaluations of algorithmic feasibility, parameter behavior, and computational trade-offs, rather than evidence of superior downstream alignment quality.
2. Broader alignment benchmarks. We agree that Open LLM Leaderboard v2 and Emotion Alignment would provide valuable additional evaluations. However, such evaluations, especially with repeated seeds and distribution-shift comparisons, require substantially greater GPU resources than are currently available to us. For example, the WDPO study used a 40-GB A100 for its emotion-alignment experiments and 8 H100 GPUs for its LLaMA experiments (Xu et al., 2025b). Our present 32-GB V100 environment does not support comparable evaluation budgets.
The focus of this paper is the theoretical development of SDPO as a smooth, computationally tractable surrogate for WDPO, together with its finite sample guarantees, first-order implementation and practical implementation strategies. The current experiments provide initial evidence of its executability and memory advantages. A comprehensive comparison of downstream alignment quality on broader benchmarks remains an important direction for future work.
3. Effective radius and experimental settings. We have clarified the interpretation of the effective radius in Assumption 3.3 and Remark 3.4. The strict margin imposed in Assumption 3.3 ensures that the dual minimizer remains strictly positive, as required by our derivative based analysis. We have also included the principal experimental settings in the main text, including the preference data size, feature dimension, transport cost, regularization parameter, radius calibration, model scales, GPU configuration, and batch size. Core experimental configurations are summarized in Section 6.1, with full details in Appendix B to respect main text page limits.
We emphasize that the central contribution of this work is the development and theoretical analysis of an entropically smoothed transport robust DPO method. Its smooth dual enables a weighted first-order update without WDPO's double backpropagation, and its small scale experiments demonstrate practical execution and favorable memory scalability. Broader alignment evaluations under larger computational budgets remain future work.