- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
MoDiFy-Rec uses disagreement between frozen visual and textual candidate similarities to weight pairwise comparisons during multimodal recommendation fine-tuning. Similarities are transformed into training-set percentiles, and an exponential kernel with symmetric normalization produces fixed positive weights. Following shared cross-entropy initialization, the method optimizes full-candidate cross-entropy plus weighted pairwise logistic loss. The paper proves that these objectives share conditional population minimizers over unrestricted scores under the stated assumptions. Experiments on three datasets evaluate covered-event reranking, kernel ablations, disagreement slices, training-seed variation, and modality interventions.
Critical strengths
- The objective is principled and the theoretical claims are appropriately delimited. Computing fixed, label-independent symmetric weights in a common scoring context supports a clean population argument. The paper explicitly distinguishes unrestricted score-space minimizers and curvature from the behavior of a restricted LoRA model. It also acknowledges that the result applies to any symmetric positive kernel rather than proving that disagreement is universally preferable.
- The controlled experiments meaningfully isolate the proposed signal. Shared initializations, candidates, and identifier permutations reduce important confounders. Comparisons with uniform and mass-matched weights, raw similarity differences, single-modality kernels, and permuted weights directly examine whether candidate-specific cross-modal disagreement matters. On MicroLens, covered NDCG@5 improves from 0.4337 to 0.4475, with larger gains on the high-disagreement slice. The paper also reports failed or inconclusive conditions instead of uniformly favorable conclusions.
Critical weaknesses
- The demonstrated benefit has a narrow retrieval-conditioned scope. Evaluation retains only events whose target is among 16 candidates retrieved by the same frozen encoders used to define disagreement. From Table 4, the covered test fractions are approximately 23.1%, 31.2%, and 25.8% on MicroLens, MovieLens-1M, and Netflix, respectively, calculated as test hits divided by users. This is a valid controlled reranking study, but it does not establish broad end-to-end recommendation improvements or robustness to a different learned retriever. The random-candidate supplement and retrieval-source diagnostics are useful, but do not remove this limitation.
- The advantage over the closest alternatives is less conclusive than the advantage over Full-CE. The primary-metric gains over HaNoRec are only 0.0013 on MovieLens-1M and 0.0018 on Netflix, and Table 6 shows that paired differences can reverse across seeds. The user-level bootstrap intervals condition on seed-averaged predictions and therefore do not incorporate training-run uncertainty. The paper acknowledges these facts, which is a strength of its reporting, but they limit the breadth of the superiority claim. I view the main contribution as evidence for a useful objective under the studied protocol, rather than a decisive general improvement over existing preference objectives.
Questions for authors
-
Reporting clarification: Can you report all-event metrics alongside covered-event metrics, assigning zero credit to retrieval misses? This can be computed from the existing outputs and would make the practical effect size explicit. Please also clarify whether any existing experiments use a retriever independent of the disagreement encoders; a new retriever study would be an extension rather than a prerequisite for validating the current conditional result.
-
Statistical clarification: Can you summarize seed-level paired uncertainty against HaNoRec and mass-matched Global-Sym, particularly on MovieLens-1M and Netflix, and distinguish it from the reported test-user intervals? Would the cross-dataset superiority conclusion remain after incorporating training-seed variation?
Official Comment
Since there are no responses to my questions, I will not change my score. I will reconsider if the authors provide a response.