- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates non-strict preferences in multi-image multimodal large language models (MLLMs). The authors identify four categories of preference ties and propose TieDPO, which preserves standard DPO for strictly preferred response pairs while introducing a margin-based regularization loss for tied pairs. Unlike reference-anchored preference objectives, TieDPO directly constrains the policy's own log-likelihood difference to encourage indifference between equally preferred responses. The paper also introduces MultiTie-22k, a ternary preference training dataset, and TieBench-MI, a benchmark for evaluating tie recognition and preference biases. Experiments on two MLLM backbones demonstrate improvements in multi-image understanding, tie discrimination, and preference calibration.
Critical strengths
-
Meaningful problem setting and structured evaluation. The paper highlights an important limitation of binary preference optimization in multi-image reasoning. The proposed taxonomy of four tie categories, particularly evidence-path ties, provides a useful perspective on non-strict preferences. The construction of MultiTie-22k and TieBench-MI also contributes valuable resources for studying this problem.
-
Comprehensive experimental evaluation. The experiments cover multiple benchmarks, two MLLM backbones, several preference optimization baselines, and extensive ablation studies. The results demonstrate consistent improvements in tie discrimination and reduced preference biases while maintaining competitive performance on standard benchmarks.
Critical weaknesses
-
Limited novelty and insufficient differentiation from prior work. My primary concern is that the core idea of explicitly modeling tied preferences in DPO has already been explored in several studies, including TODO (ICLR 2025) and On Extending Direct Preference Optimization to Accommodate Ties (NeurIPS 2025). Furthermore, Spurious Correlation Learning in Preference Optimization (ICML 2026) has investigated how tie training can mitigate spurious preference biases. These works are highly relevant but are not adequately discussed in the related work. The proposed TieDPO objective mainly replaces reference-anchored tie regularization with a margin-based penalty on the policy's own log-likelihood difference. Although this modification is empirically effective, its methodological novelty appears relatively limited. The paper should more clearly distinguish its contributions from these existing tie-aware preference optimization approaches.
-
Insufficient theoretical justification for policy-level indifference. The paper argues that tied responses should have similar log-likelihoods under the policy rather than merely preserve the reference model's relative preferences. However, equal response quality does not necessarily imply equal sequence probabilities, particularly when responses differ in length or expression. Under the KL-regularized formulation underlying DPO, equally rewarded responses may legitimately retain different probabilities inherited from the reference policy. Therefore, the motivation for directly constraining the policy's log-likelihood difference requires stronger theoretical justification. Although the ablation comparing policy-level and reference-anchored penalties is useful, further comparisons with simpler reference-free tie objectives would help establish whether the proposed margin-based formulation is necessary.
Questions for authors
-
How does TieDPO fundamentally differ from existing tie-aware preference optimization methods, particularly TODO (ICLR 2025) and the DPO variants proposed by Chen et al. (NeurIPS 2025)? Could the authors clarify the methodological novelty beyond the choice of the tie regularization objective?
-
What is the theoretical justification for encouraging equal policy log-likelihoods for equally preferred responses? Since equal preference rewards do not necessarily imply equal generation probabilities, especially for responses of different lengths, how does the proposed objective maintain consistency with the preference modeling assumptions underlying DPO?
-
Could the authors compare TieDPO with a simpler reference-free tie objective, such as a symmetric logistic loss applied directly to the policy's preference score? This would help determine whether the improvements arise primarily from removing reference-model bias or from the specific margin-based regularization design.
5 comments
Response to Reviewer yGTq (2)
Q3. Comparison with a simpler reference-free objective.
We ran the control you suggested: a symmetric logistic loss applied directly to sθ. Together with existing rows, this gives a 2×2 design (all cells share the same Lsft anchor):
| Anchor | Loss on ties | LLaVA MI-Avg | AUC | Over-conf. | Qwen MI-Avg | AUC | Over-conf. |
|---|---|---|---|---|---|---|---|
| Δθ | symmetric logistic (= DPO-soft) | 56.52 | 0.688 | 0.048 | 71.18 | 0.709 | 0.044 |
| Δθ | margin (Table 17) | 57.02 | 0.719 | 0.048 | 71.31 | 0.728 | 0.043 |
| sθ | symmetric logistic (new) | 57.75 | 0.758 | 0.024 | 71.65 | 0.768 | 0.021 |
| sθ | margin (TieDPO) | 58.14 | 0.774 | 0.019 | 71.88 | 0.781 | 0.017 |
The anchor matters most. Moving from Δθ to sθ adds +1.23 / +1.12 MI-Avg on LLaVA-OV and +0.47 / +0.57 on Qwen3-VL (logistic / margin loss), and roughly halves over-confidence (about 0.048 to about 0.02). The margin form is a smaller refinement. Moving from logistic to margin adds +0.50 / +0.39 on LLaVA-OV and +0.13 / +0.23 on Qwen3-VL (Δθ / sθ anchor), with little change in over-confidence. This agrees with Table 17, where dropping the quadratic shape costs 0.28.
So the simple reference-free logistic loss already recovers most of the benefit, which supports our claim that the key ingredient is removing the reference model's bias. We have added this table to the main text.
Dear Reviewer yGTq,
Thank you again for your review. The discussion period ends on 11.18, and we would like to know whether our response addresses your questions. In brief:
- Q1: We added significance tests for the main comparisons. The improvements are significant except on LLaVA-OV, where the result is borderline (p = 0.06).
- Q2: We clarified when the method applies and when it does not, depending on whether the reference model's preferences can be trusted.
If anything is still unclear, we can provide more details or additional results before the deadline. If the response resolves your concerns, we would appreciate it if you could take it into account in your final assessment.
Best regards, The Authors
Response to Reviewer yGTq
Thank you for the related-work pointers and the theoretical question.
Q1. Novelty relative to TODO, Chen et al., and Moya et al.
We now discuss all three works and include them as baselines. Our "Davidson" baseline is the DPO-D of Chen et al. (NeurIPS 2025), and we now attribute it accordingly. We also add DPO-RK from Chen et al. and TODO (Guo et al., ICLR 2025).
Table R1. Comparison with existing tie-aware DPO variants (3 seeds).
All rows, here and throughout the paper, use the same SFT anchor as TieDPO (λsft = 1.0, averaged over both responses on tie pairs),
Against TODO, the strongest of these, the benchmark gap is +1.21 / +0.78 (paired bootstrap p = 0.06 / 0.04), which is borderline on LLaVA-OV. The tie-side gaps are clear on both backbones (all p < 0.01): AUC is +0.039 / +0.043, over-confidence is roughly halved, and LC-Flip drops from 0.262 to 0.219.
All three prior works study text-only LLMs (TODO: Mistral-7B and Llama-3-8B; Chen et al.: machine translation and summarization; Moya et al.: log-linear models and LLMs), and all treat ties as a single unstructured class. None of them considers multimodal or multi-image alignment, distinguishes tie categories, or measures category-specific failure modes.
On the objective itself, the difference lies in what the tie term is anchored to. DPO-D, DPO-RK and TODO extend Bradley–Terry with ties in different ways, but all apply the tie likelihood to the reference-anchored Δθ. Their fixed point on a tied pair is therefore sθ = b, the reference model's own preference on that pair. TieDPO applies the tie term to the policy's score sθ, whose fixed point is sθ = 0. The 2×2 comparison under Q3 shows that this choice accounts for most of the gain. The bias results agree: all Δθ-anchored methods stay at the base model's level of length bias (LC-Flip 0.262–0.268 vs. 0.271), and only TieDPO goes below it.
Moya et al. (ICML 2026) show that training on ties can reduce spurious correlation learning in preference optimization. Our results add a condition: if the reference model itself carries a spurious preference (length, position), a Δθ-anchored tie loss pulls the policy toward that preference instead of removing it, while anchoring on sθ removes it. We have rewritten Section 2.3 accordingly. Our main finding is that reference-anchored tie objectives, which include all existing tie-aware DPO variants, preserve the reference model's biases on tied pairs, and that anchoring the tie term on the policy's own score fixes this. The taxonomy and the evaluation framework are what make this failure visible and measurable.
Q2. Theoretical justification for policy-level indifference.
You are right that under the KL-regularized optimum π* ∝ π_ref·exp(r/β), equal rewards give Δθ = 0, so the policy inherits the reference preference, sθ = b. TieDPO deliberately departs from this, and we now explain why in the paper.
By construction, a tie pair differs only along an axis that annotators judged irrelevant to quality: length, order, format, or evidence subset. On such a pair, the reference preference b is therefore a preference along an irrelevant axis. We can check this directly, because for the base model π_θ = π_ref, so b equals the base model's sθ on each pair. On tie pairs, b correlates with the length difference (Spearman ρ = 0.31 on detail ties) and with the position of the first-discussed image (ρ = −0.24 on order-invariance ties). After training, the policy's sθ on tie pairs still tracks b under DPO-D (ρ = 0.62) but largely stops tracking it under TieDPO (ρ = 0.18).
There is also a KL-consistent way to read this. sθ = 0 is the KL-optimal policy under a corrected reward whose difference on the tied pair is −b, so TieDPO stays inside the KL-regularized framework. It treats a tie label as evidence that the reference prior is biased along that axis, not as a reason to keep the reference ratio. On strict pairs, where no such evidence exists, the DPO objective is unchanged.
On length and expression: the indifference zone m tolerates residual differences, and 51% of tie pairs already have |b| < m and receive no tie gradient. A per-token length-normalized variant is slightly worse on both backbones (Table 17). The policy does not flatten its likelihoods: perplexity goes from 8.94 to 9.07, and generation length from 214 to 203 tokens (Table 16).
When the reference model's preferences can be trusted to be unbiased, Δθ-anchoring is the right choice. We now state this condition in Section 4.2.