acceptodds
Under review as a conference paper at ICLR 2027

Modality Disagreement Guided Fine-Tuning for MLLM-Based Sequential Recommendations

Abstract

Multimodal sequential recommendation combines interaction histories with item images and text to model users' preferences. Recent approaches fine-tune multimodal large language models (MLLMs) to rank candidate items from these inputs. However, candidates that are similar in one modality may differ substantially in the other. We ask whether such cross-modal disagreement can serve as a content-based criterion for deciding which comparisons to emphasize during recommendation training. We propose MoDiFy-Rec, a modality-disagreement-guided fine-tuning method that uses these differences to weight candidate comparisons in MLLM-based sequential recommendation. Specifically, MoDiFy-Rec measures disagreement using frozen visual and textual representations. It first maps pairwise similarities to percentiles in their respective training-set distributions, making their relative positions comparable across modalities. An exponential kernel and symmetric normalization then convert the absolute differences between these normalized similarities into fixed positive comparison weights. Following cross-entropy initialization, MoDiFy-Rec fine-tunes the MLLM with full-candidate cross-entropy and a weighted pairwise logistic loss comparing the observed next item against every other candidate. For fixed contexts with positive conditional label probabilities, we show that the combined objective preserves cross-entropy’s population minimizers over unrestricted scores for any nonnegative auxiliary coefficient. On three recommendation datasets, MoDiFy-Rec improves ranking quality over cross-entropy fine-tuning and pairwise baselines.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
MLLM, Sequential Recommendations

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MerGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

MoDiFy-Rec uses disagreement between frozen visual and textual candidate similarities to weight pairwise comparisons during multimodal recommendation fine-tuning. Similarities are transformed into training-set percentiles, and an exponential kernel with symmetric normalization produces fixed positive weights. Following shared cross-entropy initialization, the method optimizes full-candidate cross-entropy plus weighted pairwise logistic loss. The paper proves that these objectives share conditional population minimizers over unrestricted scores under the stated assumptions. Experiments on three datasets evaluate covered-event reranking, kernel ablations, disagreement slices, training-seed variation, and modality interventions.

Critical strengths
  1. The objective is principled and the theoretical claims are appropriately delimited. Computing fixed, label-independent symmetric weights in a common scoring context supports a clean population argument. The paper explicitly distinguishes unrestricted score-space minimizers and curvature from the behavior of a restricted LoRA model. It also acknowledges that the result applies to any symmetric positive kernel rather than proving that disagreement is universally preferable.
  2. The controlled experiments meaningfully isolate the proposed signal. Shared initializations, candidates, and identifier permutations reduce important confounders. Comparisons with uniform and mass-matched weights, raw similarity differences, single-modality kernels, and permuted weights directly examine whether candidate-specific cross-modal disagreement matters. On MicroLens, covered NDCG@5 improves from 0.4337 to 0.4475, with larger gains on the high-disagreement slice. The paper also reports failed or inconclusive conditions instead of uniformly favorable conclusions.
Critical weaknesses
  1. The demonstrated benefit has a narrow retrieval-conditioned scope. Evaluation retains only events whose target is among 16 candidates retrieved by the same frozen encoders used to define disagreement. From Table 4, the covered test fractions are approximately 23.1%, 31.2%, and 25.8% on MicroLens, MovieLens-1M, and Netflix, respectively, calculated as test hits divided by users. This is a valid controlled reranking study, but it does not establish broad end-to-end recommendation improvements or robustness to a different learned retriever. The random-candidate supplement and retrieval-source diagnostics are useful, but do not remove this limitation.
  2. The advantage over the closest alternatives is less conclusive than the advantage over Full-CE. The primary-metric gains over HaNoRec are only 0.0013 on MovieLens-1M and 0.0018 on Netflix, and Table 6 shows that paired differences can reverse across seeds. The user-level bootstrap intervals condition on seed-averaged predictions and therefore do not incorporate training-run uncertainty. The paper acknowledges these facts, which is a strength of its reporting, but they limit the breadth of the superiority claim. I view the main contribution as evidence for a useful objective under the studied protocol, rather than a decisive general improvement over existing preference objectives.
Questions for authors
  1. Reporting clarification: Can you report all-event metrics alongside covered-event metrics, assigning zero credit to retrieval misses? This can be computed from the existing outputs and would make the practical effect size explicit. Please also clarify whether any existing experiments use a retriever independent of the disagreement encoders; a new retriever study would be an extension rather than a prerequisite for validating the current conditional result.

  2. Statistical clarification: Can you summarize seed-level paired uncertainty against HaNoRec and mass-matched Global-Sym, particularly on MovieLens-1M and Netflix, and distinguish it from the reported test-user intervals? Would the cross-dataset superiority conclusion remain after incorporating training-seed variation?

Reviewer_9HYfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MoDiFy-Rec, a fine-tuning method for MLLM-based sequential recommendation. It uses cross-modal disagreement between frozen visual and textual item representations to weight pairwise ranking comparisons. For each candidate pair, the method maps visual and textual cosine similarities to training-set percentiles. The absolute difference between these percentiles defines disagreement, which an exponential kernel and symmetric normalization convert into fixed, positive comparison weights. Starting from a full-candidate cross-entropy checkpoint, MoDiFy-Rec jointly optimizes full-candidate CE and a weighted pairwise logistic loss over LoRA parameters. The theoretical analysis shows that, for any symmetric positive weights, the combined population objective preserves CE’s minimizers and characterizes the local score-space Hessian. The authors evaluate the method on MicroLens, MovieLens-1M, and Netflix using label-agnostic 16-candidate retrieval, shared candidates and initialization, and 3 seeds. They also report slice analyses, kernel and normalization ablations, and modality interventions. The reported gains over full-candidate CE concentrate on high-disagreement candidate sets. On low-disagreement sets, performance is non-inferior on two datasets; results on MovieLens-1M are inconclusive.

Critical strengths

The population-risk analysis is a substantive contribution. It shows that any symmetric positive auxiliary pairwise weights preserve cross-entropy’s population minimizers and explains why row-normalized kernels require an offset. This provides a principled constraint for designing the auxiliary pairwise loss.

The empirical protocol uses label-agnostic candidates, shared initialization and candidate sets, prespecified slice tests, and multiple ablations. The permuted-weight ablation is particularly informative: it preserves the weight values and row sums while breaking candidate alignment, and the gain disappears. This supports the claim that the benefit depends on aligning weights with candidate content.

Critical weaknesses

The gains appear modest and are not fully robust across the reported analyses. Proposition 1 applies to any symmetric positive weights, so it does not establish a specific theoretical role for disagreement. On MovieLens-1M, low-slice non-inferiority is inconclusive. For the high-disagreement slice, lower bounds fall below the pre-specified threshold on two datasets. MoDiFy-Rec ranks first in 7/9 per-seed runs. These results leave open whether the gains reflect generic hard-negative weighting; isolating the contribution of cross-modal disagreement would strengthen the claim.

The candidate protocol evaluates only M=16 retrieved candidates and covered events. It does not include full-catalog retrieval or evaluation at a larger M, which limits assessment of the practical benefit over CE fine-tuning.

Questions for authors
  1. Please clarify the Q1 column in Table 2. Why does MoDiFy-Rec show “X/X” when the text says that all three conditions hold on MicroLens? What does each mark represent?

  2. Can you report results with a larger candidate set or a trained retriever that uses encoders independent of the disagreement computation? How much of the gain remains?

  3. Proposition 1 holds for any symmetric positive weights. What evidence distinguishes the effect of cross-modal disagreement from generic hard-negative or difficulty-based weighting? Comparisons with margin-based or content-similarity-based weights would help isolate this effect.

  4. Does the high-disagreement gain persist when cross-channel candidate pairs are excluded or downweighted? This would help determine whether the effect reflects modality disagreement or retrieval-channel composition.

  5. Please report paired absolute gains and confidence intervals for MovieLens-1M and Netflix more prominently. How sensitive are the conclusions to the 5% high-slice threshold and the non-inferiority margin?

1 comment
Reviewer_9HYfOpenReview ↗

Official Comment

Since there are no responses to my questions, I will not change my score. I will reconsider if the authors provide a response.

Reviewer_TJuhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MoDiFy-Rec, a fine-tuning method for multimodal LLM-based sequential recommendation. It uses disagreement between normalized visual and textual item similarities to construct symmetric comparison weights, combining a weighted pairwise ranking loss with cross-entropy training. Experiments on MicroLens, MovieLens-1M, and Netflix report improved reranking performance over cross-entropy fine-tuning, with larger gains on high-disagreement candidate sets.

Critical strengths
  1. The method offers a simple way to use visual–textual disagreement to weight ranking comparisons during training, without computing disagreement weights at inference.

  2. Experiments on three datasets with multiple random seeds show improvements over cross-entropy fine-tuning, supported by targeted ablation studies.

Critical weaknesses
  1. The relative NDCG@5 gains over HaNoRec are about 0.3% on MovieLens-1M and 0.4% on Netflix, with some seeds performing worse. These results do not show a consistent advantage over HaNoRec.

  2. The main experiments use one backbone and a fixed retriever. Retrieval and disagreement scoring also use the same encoders. It remains unclear whether the gains hold with other backbones or retrievers.

Questions for authors
  1. Could you clarify how HaNoRec was adapted to your ranking setup, including which original components were retained and how its hyperparameters were selected?

  2. The average gains over HaNoRec are small on MovieLens-1M and Netflix. Using the existing runs, could you provide paired confidence intervals for the overall NDCG@5 differences to complement the per-seed results in Table 6?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.