acceptodds
Under review as a conference paper at ICLR 2027

Self-Supervised Preference Learning for Multimodal Foundation Models

Abstract

Preference optimization for multimodal foundation models typically relies on external judgment, such as human annotations, model-based scoring, or task-specific supervision. We introduce structural-neighbor preference learning, a self-supervised method based on a simple principle: structurally similar inputs should receive similar descriptions. For each image in a data set of interest, we identify others with similar underlying patterns–for example, charts with similar trends and fluctuations–and collect the model’s descriptions of the image and its neighbors. We rank these candidate descriptions by how much they agree with descriptions of similar images and disagree with those of dissimilar images, selecting the highest- and lowest-ranked candidates as preferred and rejected responses. We apply this method to DPO on Qwen2-VL-7B and Gemma-3-4B, testing across financial price paths, electrocardiograms, and seismic traces. We find the preference signals our approach learns transfer zero-shot to held-out tasks: relative to the supervised fine-tuned baseline, the detection of large price moves increases by 31%, crash-risk prediction by 14%, electrocardiogram abnormality detection by 10%, and seismic event detection by 3.3%. Our method also outperforms image-conditioned, self-rewarding, and external-judge preference baselines, and remains effective when similarity between inputs is determined from image-derived representations rather than the underlying signal. We also observe improvements on other types of images, including histograms, scatter plots, and natural images.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-supervised, preference-learning, vision language models, post-training, foundation models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2HfdOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper constructs DPO preference pairs without any external judgment by exploiting relations among unlabeled inputs. The authors build a neighborhood for each image from a representation that preserves the input structure, then prompt the current policy to emit a structured sequence of (state, texture) tuples. The anchor's own description and those of its neighbors form a candidate pool, scored by a similarity-weighted normalized edit distance to the neighbors minus a contrastive term over non-neighbors, with the argmax and argmin taken as the chosen and rejected responses. The procedure runs for three rounds of iterative DPO on Qwen2-VL-7B and Gemma-3-4B across financial price charts, ECG waveforms and seismic traces, and is evaluated zero-shot on downstream tasks that never appear in training.

Critical strengths

The motivation for learning preferences from structural relationships among unlabeled inputs is clear. The paper is well organized, and the proposed method is easy to follow.

Critical weaknesses
  1. The compared baselines are relatively dated and mainly cover methods from 2024. More recent work has explored automatic construction of DPO preference datasets for VLMs [1][2], but these methods are not included in the comparisons. Comparisons with more recent methods would help assess the competitiveness of the proposed approach.
  2. The evaluation mainly focuses on time-series charts. These results do not yet establish whether the gains extend to general visual question answering and multimodal understanding. Evaluating the model before and after preference training on representative benchmarks, such as MMBench or LLaVA-Bench for general understanding and POPE or AMBER for hallucination, would provide more direct evidence for the broader claims about visual grounding.

[1] Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization. [2] Self-Consistency as a Free Lunch: Reducing Hallucinations in Vision-Language Models via Self-Reflection.

Questions for authors
  1. Could you evaluate the model before and after preference training on MMBench or LLaVA-Bench for general understanding and POPE or AMBER for hallucination?
  2. The related work mainly covers earlier methods and omits several recent approaches to automatic preference data construction for VLMs. Could you update this discussion to cover relevant work from 2025 and 2026 and clarify how the proposed method differs from these approaches?
  3. Could you add comparisons with recent baselines from 2025 and 2026 for automatic construction of DPO preference datasets, including relevant methods such as [1][2]?
  4. For each round, what fraction of training pairs has the anchor's own description as the chosen response, as opposed to a neighbor's description? Please also report within-neighborhood description diversity per round.
Reviewer_GQhSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes structural-neighbor preference learning for vision-language models. It ranks model-generated descriptions by their agreement with descriptions of similar inputs and separation from those of dissimilar inputs, then uses the resulting preference pairs for iterative DPO. Experiments on Qwen2-VL-7B and Gemma-3-4B report zero-shot downstream improvements across financial charts, ECG waveforms, and seismic traces, with additional results on histograms, scatterplots, and bird images.

Critical strengths
  1. The paper is clearly structured and easy to follow.
  2. The figures and tables present the method and results clearly.
  3. The method is intuitive and straightforward to understand.
Critical weaknesses
  1. Unclear novelty. Neighborhood consistency and consensus-based caption selection are established ideas. The paper needs to distinguish its contribution more clearly from Devlin et al. (2015) and Re-Align, ideally through direct comparisons.
  2. Incomplete evaluation details. Key information about task labels, prediction thresholds, data splits, and downstream scoring is missing. This makes the main results difficult to reproduce and leaves potential overlap between training and evaluation data unresolved.
  3. Consistency does not establish correctness. The scoring function favors neighborhood agreement, which may suppress important differences between similar inputs. Better clustering and cross-modal correlation alone do not establish more accurate image descriptions.
  4. The scope of self-supervision needs clarification. Preference construction is self-supervised, but initialization relies on descriptions from a stronger teacher. The teacher model, SFT dataset size, and generation procedure are insufficiently specified, making the pipeline’s dependence on external supervision difficult to assess.
  5. Limited evidence of generalization beyond charts. The main experiments focus on chart-based tasks. Although gains are reported on bird-attribute verification, the experimental setup, absolute performance, and baseline comparisons are insufficiently documented. A full evaluation of this setting in the main experiments would make the claim of broader applicability more convincing.
  6. Limited model coverage. Experiments use only Qwen2-VL-7B and Gemma-3-4B. Testing a newer, stronger backbone of comparable size, such as Qwen3-VL-8B, would help establish whether the gains extend beyond correcting weaknesses in older models.
Questions for authors
  1. Could you clarify what is new relative to Devlin et al. (2015) and Re-Align, and what advantages the proposed method offers over these approaches?
  2. Could you provide more detail on the evaluation setup, particularly label construction, prediction thresholds, data splits, and how model outputs are scored?
  3. Could you include a full evaluation on bird-attribute verification, with absolute scores, baseline comparisons, and enough detail to reproduce the results?
  4. Could you test the method on newer backbones, such as Qwen3.5-4B or a comparably sized Gemma 4 variant, to see whether the gains hold on more recent models?
Reviewer_tprvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper starts with pointing out a hypothesis that structurally similar inputs should have similar descriptions. The authors use mostly time series data - price, ECG waveforms and seismic - to showcase the method. The method is basically finding structurally similar images / samples and finding common descriptions and following DPO on these samples-common description pairs rather than initial sample-description pairs. Authors use Qwen2 and Gemma3 models to test the proposed method. The proposed method outperforms all the baselines significantly.

Critical strengths

The problem mentioned by the authors and the solution being self-supervised method is the strongest part of the paper. The method makes the unlabelled samples a part of DPO training in a simple and an interpretable way.

Critical weaknesses

The problem mentioned by the authors is not validated. It's mentioned in the introduction and authors immediately move on to the solution in the method section.

The result section is inadequate despite the small space it requires to mention all the results. Table 1 only mentions for financial case along with baselines and for the rest of cases (ECG and seismic) only the relative improvement compared to SFT is mentioned and nothing about baseline results. Given that the space that would have taken in the main paper to show all these results is very small, I cannot think of an excuse.

Also the authors do not provide any ablation whether the improvement comes more information (or better information) or due to the neighborhood grouping (the exact proposed method). The neighboring graphs are calculated using raw time series values Line 118 which provides much cleaner than provided images.

Questions for authors

Try to answer W1 and W3 to the best of your ability

Extend Table 1 for other two cases with baselines as well.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.