acceptodds
Under review as a conference paper at ICLR 2027

Not All Preferences Are Strict: Tie-Aware Preference Optimization for Multi-Image MLLMs

Abstract

Preference optimization methods for multimodal large language models (MLLMs) universally assume strict pairwise preferences. We show that this assumption sys- tematically fails in multi-image settings, where responses frequently tie—arriving at the same correct conclusion through different image subsets, visual cues, or levels of detail. We formalize a taxonomy of four structurally distinct tie categories— detail, style/framing, evidence-path, and order-invariance ties—among which evidence-path ties are unique to multi-image reasoning and order-invariance ties predominantly arise in multi-image contexts. Forcing such ties into binary labels introduces spurious preference boundaries that manifest as verbosity bias, position bias, and evidence-path overcommitment. We propose TieDPO, a tie-aware pref- erence optimization objective that preserves standard DPO on strict pairs while regularizing tie pairs toward indifference via a margin-based penalty. To support training, we construct MultiTie-22k, a multi-image preference dataset with ternary labels and structured tie-category annotations. We further introduce TieBench-MI, a diagnostic benchmark of 2400 pairs for evaluating non-strict preference handling. Experiments show that TieDPO matches or exceeds DPO baselines on standard multi-image and single-image tasks while significantly improving tie recognition and reducing preference biases.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
preference optimization, direct preference optimization, ties, multi-image reasoning, multimodal large language models, alignment, calibration, preference dataset, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yGTqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates non-strict preferences in multi-image multimodal large language models (MLLMs). The authors identify four categories of preference ties and propose TieDPO, which preserves standard DPO for strictly preferred response pairs while introducing a margin-based regularization loss for tied pairs. Unlike reference-anchored preference objectives, TieDPO directly constrains the policy's own log-likelihood difference to encourage indifference between equally preferred responses. The paper also introduces MultiTie-22k, a ternary preference training dataset, and TieBench-MI, a benchmark for evaluating tie recognition and preference biases. Experiments on two MLLM backbones demonstrate improvements in multi-image understanding, tie discrimination, and preference calibration.

Critical strengths
  1. Meaningful problem setting and structured evaluation. The paper highlights an important limitation of binary preference optimization in multi-image reasoning. The proposed taxonomy of four tie categories, particularly evidence-path ties, provides a useful perspective on non-strict preferences. The construction of MultiTie-22k and TieBench-MI also contributes valuable resources for studying this problem.

  2. Comprehensive experimental evaluation. The experiments cover multiple benchmarks, two MLLM backbones, several preference optimization baselines, and extensive ablation studies. The results demonstrate consistent improvements in tie discrimination and reduced preference biases while maintaining competitive performance on standard benchmarks.

Critical weaknesses
  1. Limited novelty and insufficient differentiation from prior work. My primary concern is that the core idea of explicitly modeling tied preferences in DPO has already been explored in several studies, including TODO (ICLR 2025) and On Extending Direct Preference Optimization to Accommodate Ties (NeurIPS 2025). Furthermore, Spurious Correlation Learning in Preference Optimization (ICML 2026) has investigated how tie training can mitigate spurious preference biases. These works are highly relevant but are not adequately discussed in the related work. The proposed TieDPO objective mainly replaces reference-anchored tie regularization with a margin-based penalty on the policy's own log-likelihood difference. Although this modification is empirically effective, its methodological novelty appears relatively limited. The paper should more clearly distinguish its contributions from these existing tie-aware preference optimization approaches.

  2. Insufficient theoretical justification for policy-level indifference. The paper argues that tied responses should have similar log-likelihoods under the policy rather than merely preserve the reference model's relative preferences. However, equal response quality does not necessarily imply equal sequence probabilities, particularly when responses differ in length or expression. Under the KL-regularized formulation underlying DPO, equally rewarded responses may legitimately retain different probabilities inherited from the reference policy. Therefore, the motivation for directly constraining the policy's log-likelihood difference requires stronger theoretical justification. Although the ablation comparing policy-level and reference-anchored penalties is useful, further comparisons with simpler reference-free tie objectives would help establish whether the proposed margin-based formulation is necessary.

Questions for authors
  1. How does TieDPO fundamentally differ from existing tie-aware preference optimization methods, particularly TODO (ICLR 2025) and the DPO variants proposed by Chen et al. (NeurIPS 2025)? Could the authors clarify the methodological novelty beyond the choice of the tie regularization objective?

  2. What is the theoretical justification for encouraging equal policy log-likelihoods for equally preferred responses? Since equal preference rewards do not necessarily imply equal generation probabilities, especially for responses of different lengths, how does the proposed objective maintain consistency with the preference modeling assumptions underlying DPO?

  3. Could the authors compare TieDPO with a simpler reference-free tie objective, such as a symmetric logistic loss applied directly to the policy's preference score? This would help determine whether the improvements arise primarily from removing reference-model bias or from the specific margin-based regularization design.

5 comments
AuthorsOpenReview ↗

Response to Reviewer yGTq

Thank you for the related-work pointers and the theoretical question.

Q1. Novelty relative to TODO, Chen et al., and Moya et al.

We now discuss all three works and include them as baselines. Our "Davidson" baseline is the DPO-D of Chen et al. (NeurIPS 2025), and we now attribute it accordingly. We also add DPO-RK from Chen et al. and TODO (Guo et al., ICLR 2025).

Table R1. Comparison with existing tie-aware DPO variants (3 seeds).

MethodLLaVA MI-AvgAUCOver-conf.LC-FlipQwen MI-AvgAUCOver-conf.LC-Flip
DPO-D (Chen et al.)56.580.7210.0420.26870.960.7260.0380.244
DPO-RK (Chen et al.)56.500.7280.0400.26170.830.7310.0330.241
TODO (Guo et al.)56.930.7350.0370.26271.100.7380.0340.239
TieDPO58.140.7740.0190.21971.880.7810.0170.208

All rows, here and throughout the paper, use the same SFT anchor as TieDPO (λsft = 1.0, averaged over both responses on tie pairs),

Against TODO, the strongest of these, the benchmark gap is +1.21 / +0.78 (paired bootstrap p = 0.06 / 0.04), which is borderline on LLaVA-OV. The tie-side gaps are clear on both backbones (all p < 0.01): AUC is +0.039 / +0.043, over-confidence is roughly halved, and LC-Flip drops from 0.262 to 0.219.

All three prior works study text-only LLMs (TODO: Mistral-7B and Llama-3-8B; Chen et al.: machine translation and summarization; Moya et al.: log-linear models and LLMs), and all treat ties as a single unstructured class. None of them considers multimodal or multi-image alignment, distinguishes tie categories, or measures category-specific failure modes.

On the objective itself, the difference lies in what the tie term is anchored to. DPO-D, DPO-RK and TODO extend Bradley–Terry with ties in different ways, but all apply the tie likelihood to the reference-anchored Δθ. Their fixed point on a tied pair is therefore sθ = b, the reference model's own preference on that pair. TieDPO applies the tie term to the policy's score sθ, whose fixed point is sθ = 0. The 2×2 comparison under Q3 shows that this choice accounts for most of the gain. The bias results agree: all Δθ-anchored methods stay at the base model's level of length bias (LC-Flip 0.262–0.268 vs. 0.271), and only TieDPO goes below it.

Moya et al. (ICML 2026) show that training on ties can reduce spurious correlation learning in preference optimization. Our results add a condition: if the reference model itself carries a spurious preference (length, position), a Δθ-anchored tie loss pulls the policy toward that preference instead of removing it, while anchoring on sθ removes it. We have rewritten Section 2.3 accordingly. Our main finding is that reference-anchored tie objectives, which include all existing tie-aware DPO variants, preserve the reference model's biases on tied pairs, and that anchoring the tie term on the policy's own score fixes this. The taxonomy and the evaluation framework are what make this failure visible and measurable.

Q2. Theoretical justification for policy-level indifference.

You are right that under the KL-regularized optimum π* ∝ π_ref·exp(r/β), equal rewards give Δθ = 0, so the policy inherits the reference preference, sθ = b. TieDPO deliberately departs from this, and we now explain why in the paper.

By construction, a tie pair differs only along an axis that annotators judged irrelevant to quality: length, order, format, or evidence subset. On such a pair, the reference preference b is therefore a preference along an irrelevant axis. We can check this directly, because for the base model π_θ = π_ref, so b equals the base model's sθ on each pair. On tie pairs, b correlates with the length difference (Spearman ρ = 0.31 on detail ties) and with the position of the first-discussed image (ρ = −0.24 on order-invariance ties). After training, the policy's sθ on tie pairs still tracks b under DPO-D (ρ = 0.62) but largely stops tracking it under TieDPO (ρ = 0.18).

There is also a KL-consistent way to read this. sθ = 0 is the KL-optimal policy under a corrected reward whose difference on the tied pair is −b, so TieDPO stays inside the KL-regularized framework. It treats a tie label as evidence that the reference prior is biased along that axis, not as a reason to keep the reference ratio. On strict pairs, where no such evidence exists, the DPO objective is unchanged.

On length and expression: the indifference zone m tolerates residual differences, and 51% of tie pairs already have |b| < m and receive no tie gradient. A per-token length-normalized variant is slightly worse on both backbones (Table 17). The policy does not flatten its likelihoods: perplexity goes from 8.94 to 9.07, and generation length from 214 to 203 tokens (Table 16).

When the reference model's preferences can be trusted to be unbiased, Δθ-anchoring is the right choice. We now state this condition in Section 4.2.

AuthorsOpenReview ↗

Response to Reviewer yGTq (2)

Q3. Comparison with a simpler reference-free objective.

We ran the control you suggested: a symmetric logistic loss applied directly to sθ. Together with existing rows, this gives a 2×2 design (all cells share the same Lsft anchor):

AnchorLoss on tiesLLaVA MI-AvgAUCOver-conf.Qwen MI-AvgAUCOver-conf.
Δθsymmetric logistic (= DPO-soft)56.520.6880.04871.180.7090.044
Δθmargin (Table 17)57.020.7190.04871.310.7280.043
sθsymmetric logistic (new)57.750.7580.02471.650.7680.021
sθmargin (TieDPO)58.140.7740.01971.880.7810.017

The anchor matters most. Moving from Δθ to sθ adds +1.23 / +1.12 MI-Avg on LLaVA-OV and +0.47 / +0.57 on Qwen3-VL (logistic / margin loss), and roughly halves over-confidence (about 0.048 to about 0.02). The margin form is a smaller refinement. Moving from logistic to margin adds +0.50 / +0.39 on LLaVA-OV and +0.13 / +0.23 on Qwen3-VL (Δθ / sθ anchor), with little change in over-confidence. This agrees with Table 17, where dropping the quadratic shape costs 0.28.

So the simple reference-free logistic loss already recovers most of the benefit, which supports our claim that the key ingredient is removing the reference model's bias. We have added this table to the main text.

Dear Reviewer yGTq,

Thank you again for your review. The discussion period ends on 11.18, and we would like to know whether our response addresses your questions. In brief:

  • Q1: We added significance tests for the main comparisons. The improvements are significant except on LLaVA-OV, where the result is borderline (p = 0.06).
  • Q2: We clarified when the method applies and when it does not, depending on whether the reference model's preferences can be trusted.

If anything is still unclear, we can provide more details or additional results before the deadline. If the response resolves your concerns, we would appreciate it if you could take it into account in your final assessment.

Best regards, The Authors

Reviewer_3pMUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to leverage "tie" signals in preference data to train multi-image MLLMs. The authors propose TieDPO, which augments the standard DPO objective with an auxiliary regularizer that minimizes the likelihood margin between tied responses. The paper also introduces new training and diagnostic datasets. Experiments across two models and seven benchmarks demonstrate the effectiveness of the proposed approach.

Critical strengths
  • How to effectively utilize tie pairs in preference optimization (rather than discarding them) is a practical and often under-explored question in post-training alignment.
  • The proposed TieDPO formulation, explicitly encouraging comparable likelihoods for tied responses alongside standard preference margins, is simple, intuitive, and well-motivated.
  • The evaluation covers two distinct MLLM backbones and seven diverse benchmarks, and the release of the curated training and diagnostic datasets will be a valuable resource for the community.
Critical weaknesses
  • Modeling ties in preference optimization is a general alignment problem applicable to text-only LLMs and single-image MLLMs alike, and the core TieDPO formulation does not appear strictly tied to multi-image architectures. Restricting the empirical validation solely to the multi-image MLLM setting limits the demonstrated generalizability of the method.
  • There is existing literature in preference optimization that explicitly incorporates tie labels into preference modeling, which the paper should properly discuss and contextualize against its baselines. For example, [1] previously introduced the Davidson-model DPO variant that is also evaluated as a baseline in this paper.
  • The performance improvements over the baselines are relatively marginal across several benchmarks, making it somewhat unclear how much practical impact the tie auxiliary loss provides.

[1] Chen, Jinghong, et al. "On extending direct preference optimization to accommodate ties." Advances in Neural Information Processing Systems 38 (2026): 53932-53977.

Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 3pMU

Response to Reviewer 3pMU

Thank you for the positive assessment and for pointing us to Chen et al.

W1. Generality beyond multi-image.

The objective itself is not specific to multi-image input. We focused on that setting because two of the four tie categories arise mainly there. To test generality, we trained LLaVA-OV on a single-image tie dataset (22,198 pairs, built with the same pipeline; see our response to VcCh, Q1) and evaluated on the 600-pair single-image tie test set and the paper's single-image benchmarks:

MethodAUC↑Over-conf.↓Single-Avg (MMMU, XLRS)
Base0.550.00542.9
DPO-strict0.690.05243.0
DPO-D (Chen et al.)0.700.03343.1
TieDPO0.730.02643.5

TieDPO improves tie discrimination and calibration over both DPO-strict and DPO-D on single-image data, without hurting benchmark accuracy (the benchmark differences are within confidence intervals). The gain is smaller than in the multi-image setting (ΔAUC over DPO-strict +0.04 vs. +0.068), since evidence-path and order ties are rare with single images. We list text-only alignment as future work in the Conclusion.

W2. Chen et al.

Thank you; this was an important omission. Our "Davidson" baseline is the DPO-D of Chen et al., and we now attribute it accordingly. We also add their DPO-RK and TODO (Guo et al.) as baselines (Table R1 in our response to yGTq), and discuss both in the related work.

Chen et al. observe that ties strengthen regularization toward π_ref. Our analysis shows what that regularization preserves on tied pairs: the reference model's preference along an axis that annotators judged irrelevant. All three Δθ-anchored methods stay near the base level on LC-Flip (0.262–0.268 vs. 0.271), while TieDPO falls below it (0.219).

W3. Size of the gains.

Ties are 25.8% of the training data, and the main effect is on how the model handles them: over-confidence drops from 0.077 for DPO-strict and 0.037–0.042 for the Δθ-anchored tie methods to 0.019 for TieDPO. The benchmark gains are significant against DPO-D (p = 0.047 on LLaVA-OV, p = 0.03 on Qwen3-VL) and borderline against TODO (p = 0.06 / 0.04). They are also concentrated where the method should matter: on order-sensitive items, TieDPO gains +3.5 / +2.9 over DPO-D, compared with +0.72 / +0.14 on order-stable items.

Reviewer_VcChOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies tied preferences in multi-image MLLMs and introduces four tie categories. It proposes TieDPO, which retains DPO on strict pairs and penalizes excessive differences in policy log-likelihood on tied pairs, together with an SFT anchor. The authors construct MultiTie-22k and TieBench-MI and report improvements in downstream performance, tie discrimination, calibration, and surface-bias diagnostics across two backbones.

Critical strengths
  1. The paper highlights tied preferences in multi-image alignment and provides a useful taxonomy and evaluation framework.

  2. The experiments and ablations provide encouraging evidence for the benefits of incorporating tied preferences.

Critical weaknesses
  1. The claimed amplification of tied preferences in multi-image settings is insufficiently substantiated. Detail and style ties also occur in single-image and text-only tasks, while analogous ordering and evidence-path ambiguities can arise within a single image. The paper provides no controlled comparison establishing that these phenomena are more prevalent or consequential in multi-image settings. The distinctive motivation for focusing on multi-image alignment therefore needs stronger support.

  2. The mechanism behind the downstream gains remains unclear. Improvements over strong tie-aware baselines are modest, particularly on Qwen3-VL. Reduced sensitivity to length, image order, and format, together with correlational analysis, does not establish whether bias reduction drives the improvement in multi-image accuracy.

  3. The main contribution is not clearly explained. The paper focuses on defining the problem and evaluating tie handling, but the main text does not clearly explain what we learn from the tie categories or how the datasets and benchmark improve on existing work. This makes it hard to see what is new and why it matters.

Questions for authors
  1. Multi-image motivation. What quantitative evidence establishes that tied-preference phenomena are amplified in multi-image settings? A matched comparison with single-image settings, examining tie prevalence or the adverse effects of strict preference training, would help substantiate this claim.

  2. The mechanism behind the downstream gains remains unclear. Improvements over strong tie-aware baselines are modest, particularly on Qwen3-VL. Reduced sensitivity to length, image order, and format, together with correlational analysis, does not establish whether bias reduction drives the improvement in multi-image accuracy.

  3. On Qwen3-VL, TieDPO improves the five-benchmark multi-image average over DPO-soft by approximately 0.70 points, compared with 1.62 points on LLaVA-OV. What explains this smaller gain? Does this mean that the benefits brought by the proposed method with the latest basic model will become increasingly marginal?

  4. Table 17 supports the benefit of tie regularization, but what evidence connects reduced length, order, and format biases to improved answer correctness? Do existing per-example or error analyses show that the downstream gains occur on cases affected by these biases?

  5. Do Tables 3 and 24 use the same strict-pair comparison pool? Under this assumption, the category counts and AUCs imply an overall TieDPO AUC of at most approximately 0.760, even assigning AUC = 1 to “Other,” whereas Table 3 reports 0.774. How can these results be reconciled?

  6. Contribution of the construction pipelines. Is the contribution of MultiTie-22k and TieBench-MI primarily the datasets and their structured tie annotations, or do the construction pipelines introduce methodological advances over existing preference-data generation and annotation procedures?

4 comments
AuthorsOpenReview ↗

Response to Reviewer VcCh (1/2)

Response to Reviewer VcCh (1/2)

Thank you for the careful review. All new numbers are 3-seed means unless stated otherwise.

Q1. Is the tie problem actually larger in multi-image settings?

We ran two analyses.

(a) Tie rate vs. number of images. In TieBench-MI, image count is confounded with image source. Needle-in-a-Haystack has the most images but the lowest tie rate (0.196), because the answer lies in a single target image. Spot-the-Diff always has two images and skews toward detail ties (33% vs. 26.5% overall). We therefore stratify by image count within the two sources where it varies (MMIU and ReMI, 1,011 pairs, LLaVA-OV):

#imagesnTie rateEP+OI share of tiesΔOver-conf. (DPO-strict − base)DPO-strict Pos-FlipΔAUC (TieDPO − DPO-strict)
22360.1942%+0.0550.27+0.046
3–45980.26556%+0.0700.33+0.068
≥51770.3163%+0.0880.38+0.094

Four quantities rise with image count: the tie rate, the share of EP/OI ties, the harm from strict-only training, and the benefit of TieDPO. On all 1,700 Natural pairs, a logistic regression of the tie label on image count with source fixed effects gives an odds ratio of 1.14 per additional image (95% CI 1.04–1.25). The Needle case suggests that the relevant variable is how many images jointly bear on the answer, not the raw count. This is also the condition that produces evidence-path ties.

(b) Single-image control. We ran the same pipeline (same judge, same prompts, same blind two-pass annotation) on single-image inputs from Visual Genome, A-OKVQA, ChartQA and DocVQA, and built a 600-pair test set. The image sources necessarily differ from the multi-image ones; what is matched is the construction and annotation procedure. For single images, we count an evidence-path tie when two responses rely on different regions or cues, and an order tie when the same regions are discussed in a different order. Results on LLaVA-OV:

Multi-image (TieBench-MI)Single-image
Tie rate (unconstrained sampling)0.2580.17
EP + order share of ties54.7%18%
Over-conf. increase from DPO-strict+0.069+0.047
LC-Flip increase from DPO-strict+0.090+0.061

You are right that all four phenomena exist with single images. With multiple images, ties are more frequent, EP/order ties make up about three times the share, and strict-only training does more damage. We have rewritten the claim in Sections 1 and 3 in these terms.

Q2/Q4. Does bias reduction explain the downstream gains?

Beyond the correlational analysis in Appendix E.7, we added two experiments.

(a) Per-example order perturbation. From MuirBench, MIRB and BLINK, we take the items whose answer does not depend on image order, excluding order-dependent subtasks. We run the base model under 3 image permutations and split the items by whether its prediction changes:

BackboneSubset% itemsTieDPO − DPO-strictTieDPO − DPO-D
LLaVA-OVorder-sensitive31%+5.8+3.5
LLaVA-OVorder-stable69%+1.0+0.72
Qwen3-VLorder-sensitive22%+3.2+2.9
Qwen3-VLorder-stable78%+0.63+0.14

Most of the gain comes from items on which the base model is order-sensitive. One could object that any stronger model gains more on unstable items; experiment (b) addresses this.

(b) Leave-one-category-out training (LLaVA-OV). As a size control, we also remove 2,543 random tie pairs, which matches the largest category (evidence-path). This is more than the number of detail, style or order-invariance pairs, so the control is conservative for those rows. The last column is the accuracy change on the order-sensitive subset from (a).

Removed from trainingLC-Flip↓Pos-Flip↓Style Gap↓EP-Stress AUC↑MuirBenchΔ order-sensitive acc.
none (full TieDPO)0.2190.2410.0740.72646.6—
detail ties0.2560.2440.0790.72146.3−0.3
style ties0.2230.2430.1120.72246.4−0.2
order-invariance ties0.2220.2720.0770.71746.0−1.8
evidence-path ties0.2250.2510.0780.67145.7−0.9
random (size-matched)0.2240.2460.0790.71846.4−0.4

The "none" row is the 3-seed result from the paper; the other rows are single-seed.

Removing a category mainly worsens the bias metric tied to that category (the diagonal of the table). It also costs accuracy on the order-sensitive items from (a): 1.8 points without order-invariance ties and 0.9 without evidence-path ties, compared with 0.4 for the larger random removal. So the gain on these items depends on which kinds of ties are in the training data, not on how many.

AuthorsOpenReview ↗

Response to Reviewer VcCh (2/2)

Response to Reviewer VcCh (2/2)

Q3. Smaller gain on Qwen3-VL.

Against DPO-soft, the MI-Avg gap is +1.62 on LLaVA-OV and +0.70 on Qwen3-VL. Against DPO-D, it is +1.56 and +0.92. We report both because DPO-D is the stronger baseline on LLaVA-OV and DPO-soft on Qwen3-VL.

On the tie side, the two backbones behave almost identically: AUC improves over DPO-strict by +0.068 / +0.067, and over-confidence is 0.019 / 0.017. What shrinks on Qwen3-VL is the benchmark gain, and this matches how much bias the base model has to begin with. Base Qwen3-VL is less biased (LC-Flip 0.248 vs. 0.271; Pos-Flip 0.251 vs. 0.284) and has fewer order-sensitive items (22% vs. 31%). On the order-sensitive items themselves, the gains over DPO-D are similar (+3.5 vs. +2.9). The Qwen3-VL gains remain significant (paired bootstrap p = 0.03 vs. DPO-D, p = 0.04 vs. DPO-soft).

On whether the benefit fades with newer models: a cleaner base does not stay clean after preference training. Strict-only DPO re-introduces the biases on Qwen3-VL by about the same amount as on LLaVA-OV:

  • LC-Flip: +0.084 vs. +0.090;
  • position flip: +0.053 vs. +0.045;
  • over-confidence: +0.057 vs. +0.069.

A cleaner base model therefore does not make tie supervision unnecessary: strict-only training brings the biases back.

Q5. Inconsistency between Tables 3 and 24.

Thank you for checking this. You are correct. The original Table 24 came from a single-seed run that paired each category with its own strict pool, not with the common pool used in Table 3 and described in the caption. We recomputed it with the common pool over 3 seeds:

ModelDetail (116)Style (57)Order-inv. (140)Evid.-path (100)Other (26)
Base0.5630.5340.5270.5180.622
DPO-strict0.7780.7390.6840.6180.769
DPO-soft0.7370.7190.6860.6290.639
Davidson (DPO-D)0.7700.7420.7110.6660.722
TieDPO0.7890.7640.7880.7310.819

The count-weighted averages now reproduce Table 3 for every method (e.g., TieDPO 0.774, DPO-strict 0.706, Base 0.541). The 95% intervals are about as wide as in the original table (±0.10 to ±0.18, set mainly by category size). TieDPO's largest margins are on order-invariance (0.788 vs. 0.711 for the best baseline) and evidence-path ties (0.731 vs. 0.666), while on detail ties it is close to DPO-strict (0.789 vs. 0.778). We have corrected the corresponding sentence in Section 5.3.

Q6. What the construction pipelines contribute.

Besides the data, the pipelines contain three components that we have not found in prior preference-data pipelines:

  1. Single-axis tie construction. Each tie pair varies along exactly one axis. This is enforced through the core-facts / optional-elaborations split, and pairs that fail the judge's purity checks are discarded rather than relabeled. This is what makes category-level failure modes measurable; the leave-one-out experiment above depends on it.
  2. Verifiable evidence-path ties. Each response is generated with only its assigned image subset visible, and its grounding is then checked against the full image set. Flawed variants in strict pairs are length-matched, so strict pairs cannot be decided by length alone.
  3. An evaluation protocol for indifference. Annotation is blind and two-pass (relation first, category second). The metrics are threshold-free and cannot be improved by simply shrinking sθ (requirements R1/R2). Over-confidence uses a calibration map frozen on the base model, which separates calibration loss from discriminative gain; per-method calibration would remove the very quantity we want to measure.

We have also added a paragraph to the main text on what the taxonomy tells us: each category predicts a distinct, measurable failure mode, and the two multi-image categories are where strict-only training does the most damage and tie supervision helps the most.

Dear Reviewer VcCh,

Thank you again for your review. The discussion period ends on 11.18, and we would like to know whether our response addresses your questions. In brief:

  • Q2: We added a category-removal ablation. Removing order-invariance ties lowers order-sensitive accuracy by 1.8 points and removing evidence-path ties by 0.9, compared with 0.4 for a larger random removal.
  • Q3: On a second, cleaner backbone, strict-only preference training still brings the biases back.
  • Q5: We report per-category results with 95% intervals.

If anything is still unclear, we can provide more details or additional results before the deadline. If the response resolves your concerns, we would appreciate it if you could take it into account in your final assessment.

Best regards, The Authors

Thank you for the detailed response and the additional experiments. They address most of my questions. I still have some reservations about the motivation and the new insights the paper offers into multi-image reasoning.

W1 and W3. The new experiments provide useful evidence that ties are more common in the multi-image data studied. However, TieDPO uses a general tie loss, so I see its main contribution to multi-image reasoning in the four tie categories and the training data and benchmark built around them. Yet most of these tie categories are not unique to multi-image settings, as they also have single-image counterparts.

W2. The rebuttal shows that TieDPO helps more on questions sensitive to image order. SoFA [1] already showed that directly reducing position bias can improve multi-image accuracy. The additional analysis strengthens the paper, but it offers limited new insight into the link between position bias and multi-image accuracy.

Overall, my current assessment of the paper is borderline.

[1] Identifying and Mitigating Position Bias of Multi-image Vision-Language Models. CVPR 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.