acceptodds
Under review as a conference paper at ICLR 2027

VLoRA: Adapting Vision-Language Models for Multimodal Recommendation via Two-Stage Low-Rank Adaptation

Abstract

Learning item representations that unify content semantics and collaborative filtering (CF) signals is crucial for sequential recommendation. While recent large language model-based methods have shown promise, they often ignore visual information that is important for user decisions. Vision-language models (VLMs) provide a natural foundation for multimodal recommendation, but directly fine-tuning them poses three challenges: objective interference between CF alignment and embedding learning, modality-length asymmetry, and representation collapse under sparse supervision. We propose VLoRA, a parameter-efficient two-stage post-training framework that adapts a frozen decoder-only VLM into a CF-aware multimodal embedding model. In the first stage, collaborative signals are injected through autoregressive next-item prediction over user interaction sequences. In the second stage, item embeddings are refined using bidirectional attention and three complementary objectives: Multimodal Masked Next Token Prediction for modality-asymmetric masking, dual-view self-contrastive learning for discriminative embeddings, and representation-preservation regularization to prevent representation collapse. With progressive LoRA unfreezing, VLoRA introduces trainable parameters equivalent to only 0.4% of the backbone. Experiments show that VLoRA consistently outperforms twelve strong baselines in both in-domain and out-of-domain settings, achieving an average improvement of 4.4% on NDCG@10 across all evaluation settings. Further analyses verify the effectiveness and generalizability of the proposed framework.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision-language Model, LoRA

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aeLtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VLoRA, a parameter-efficient two-stage framework for adapting vision-language models to multimodal sequential recommendation. It first injects collaborative filtering signals through autoregressive next-item prediction (CAP), then refines item representations with bidirectional attention, masked-token prediction, contrastive learning, and representation preservation (RTF), while training only LoRA adapters. The resulting item embeddings can be precomputed offline and reused by different recommenders. Experiments on six datasets, two sequential recommenders, and multiple VLM backbones show strong in-domain and cross-domain performance.

Critical strengths
  • The experimental evaluation is comprehensive, and the main empirical results are convincing. The paper evaluates VLoRA across six datasets, two downstream sequential recommenders, and four ranking metrics, covering both in-domain and out-of-domain settings. VLoRA ranks first in 41 out of 48 settings and second in the remaining seven.
  • The method shows promising cross-domain generalization and backbone portability. The consistent improvement suggests that the proposed adaptation recipe is not tightly coupled to a particular dataset or VLM backbone.
Critical weaknesses
  • The methodological novelty is somewhat incremental. The overall framework largely combines several established ideas, including autoregressive collaborative alignment, bidirectional embedding refinement, LoRA adaptation, masked-token prediction, contrastive learning, and representation regularization. While their integration for multimodal recommendation is reasonable and empirically effective, it is less clear which component constitutes a sufficiently novel methodological contribution on its own.
  • The evaluation scenarios are still relatively limited. The experiments consider only one title and one image per item, interaction sequences of length at most 10. In addition, the learned item representations are static and do not depend on the user or sequence position. Therefore, it remains unclear how well the method scales to richer multimodal metadata, longer behavioral histories, or more modern and large-scale recommendation architectures.
Questions for authors
  • Which part of VLoRA should be regarded as the main methodological novelty? Many individual components—two-stage adaptation, LoRA, bidirectional attention, contrastive learning, and representation regularization—have appeared in prior work. Could the authors clarify which design choice they consider the central novel contribution beyond their combination?
  • How does VLoRA behave with richer item content and longer user histories? The current setup uses one title, one image, and sequences of length at most 10. It would be helpful to understand whether the gains remain when items contain multiple images, longer textual descriptions, or when user histories are substantially longer.
  • The appendix shows that 4→4 slightly outperforms 4→8 on Arts, while 4→8 performs better on Goodreads and on average. Could the authors provide more intuition or analysis explaining when progressive layer expansion is expected to help?
4 comments
AuthorsOpenReview ↗

Response to Reviewer aeLt (1 / 3)

We thank Reviewer aeLt for recognizing the comprehensiveness of our evaluation across six datasets and two sequential recommenders (ranking first in 41 of 48 settings and second in 7), as well as VLoRA's cross-domain generalization and backbone portability. Below we address W1–W2 and Q1–Q3 directly with controlled experiments, all of which—along with presentation revisions—have been incorporated into the revised manuscript (highlighted in blue).


1. W1 & Q1: Central Methodological Novelty Beyond Combining Existing Components

Core Contribution: Beyond integrating two-stage adaptation [1, 2] with LoRA, VLoRA's central methodological contribution is identifying and resolving the Pooling–Manifold Coupling that arises when converting a decoder-only VLM into a recommendation item encoder under severe modality-length asymmetry.

Specifically, adapting a VLM introduces a structural bottleneck absent in text-only LLMs (LLM2Rec [1]):

  1. Why Last-Token Pooling Under-Utilizes Visual Content: At 448×448448\times 448 resolution, an item image yields hundreds of visual patch tokens (~80% of the sequence), whereas the product title spans only tens of tokens (~20%). Because Stage 1 (CAP) trains the model autoregressively on text titles, reading out the last token aggregates information at the final text position and relies predominantly on textual cues, yielding only a +4.1% R@20 gain when images are added (0.1398 -> 0.1456 on Arts, Table R1).
  2. Why Content Mean Pooling Fails Without Manifold Anchoring: To aggregate all ~80% visual patch tokens alongside the title, Stage 2 switches to bidirectional content mean pooling [2]. Without RPR, however, contrastive updates under sparse recommendation supervision distort the pretrained vision-language manifold [3]—driving cosine similarity to the pretrained reference Φ0\Phi_0 down from 1.000 to 0.563 (Table 10) and reducing Arts R@20 from 0.1494 to 0.1423 (Table 6).
  3. Why VLoRA's Coupled Design Works: Pairing bidirectional content mean pooling with RPR (which anchors the pooled representation to Φ0\Phi_0 [3], restoring cosine similarity to 0.708) and modality-asymmetric text masking (which keeps visual patches intact to reconstruct masked title tokens) prevents visual manifold drift and raises Arts R@20 to 0.1559 (+11.5% over the text-only port, +11.0% over same-backbone LLM2Rec [1]).

A 2×22\times 2 controlled study on the exact same Qwen3-VL-2B-Instruct backbone (Appendix G.1, Table 6 of the revised manuscript) confirms that this coupling is specific to multimodal adaptation:

Table R1. Isolating the Multimodal Pooling–Manifold Coupling on Qwen3-VL-2B-Instruct (SASRec; Revised Table 6)

Configuration (Same Qwen3-VL-2B-Instruct Backbone)Input ModalityStage-2 PoolingRPRArts R@20Arts N@20Goodreads R@20Goodreads N@20
Text-only port (Last-token pooling)Text onlyLast tokenNo0.13980.07750.24070.1342
Same-backbone LLM2Rec [1] (Text-only, Mean pooling)Text onlyMean poolNo0.14040.07810.24160.1349
Same-backbone LLM2Rec [1] + RPRText onlyMean poolYes0.14210.07920.24350.1360
Multimodal port (Uniform masking + Last-token)Text + ImageLast tokenNo0.14560.08030.24490.1361
+ Asymmetric text-only masking (Last-token)Text + ImageLast tokenNo0.14940.08300.24850.1397
+ Content mean pooling w/o RPRText + ImageMean poolNo0.14230.07980.24030.1338
VLoRA (Multimodal + Content mean pooling + RPR)Text + ImageMean poolYes0.15590.08750.25270.1425

In the text-only setting on Qwen3-VL-2B-Instruct, RPR changes R@20 by only +1.2% (0.1404 -> 0.1421) on Arts and +0.8% (0.2416 -> 0.2435) on Goodreads. In the multimodal setting, RPR improves content mean pooling by +9.6% on Arts (0.1423 -> 0.1559) and +5.2% on Goodreads (0.2403 -> 0.2527).

Manuscript Revisions for Presentation Clarity (Abstract, Section 1, Section 3.2, Section 4.3, and Appendix G.1): In the revised manuscript (highlighted in blue), we have expanded Appendix G.1 (Table 6) from a 1D cumulative chain into the 2×22\times 2 orthogonal control (Table R1) and revised the Abstract, Introduction, Section 3.2, and Section 4.3 to foreground the pooling–manifold coupling and its scope conditions upfront.


AuthorsOpenReview ↗

Response to Reviewer aeLt (2 / 3)

2. W2 & Q2: Scaling to Longer User Histories (K>10K > 10) and Richer Item Content

Core Conclusion: VLoRA's decoupled item-level architecture scales to longer user histories (K=20,50K=20, 50) and richer item metadata (adding categories and descriptions) with consistent accuracy gains and zero additional VLM memory or compute cost (Appendix I.5, Tables 20–21 of the revised manuscript).

(a) Longer User Interaction Histories (K∈{10,20,50}K \in \{10, 20, 50\})

In our main experiments, K=10K=10 was used because the average sequence length across the 5-core Amazon benchmarks is 10.010.0 interactions per user (Table 2). Crucially, when scaling to longer histories (K=20K=20 or K=50K=50), producing static, context-independent item embeddings is a key architectural advantage:

  • In sequence-encoder VLMs [6] (which feed the entire multi-image history into the VLM), increasing KK from 1010 to 5050 expands the input to >12,000>12,000 visual tokens, causing O(K2)O(K^2) attention blow-up and GPU out-of-memory (OOM) failure.
  • In VLoRA, every item is encoded offline once into zi∈R2048\mathbf{z}_i \in \mathbb{R}^{2048} and sequence dynamics are modeled by the lightweight downstream recommender (SASRec [5] / GRU4Rec). Extending the history window from K=10K=10 to K=20K=20 or K=50K=50 therefore incurs zero extra VLM memory or inference cost.

We evaluated downstream SASRec [5] with K∈{10,20,50}K \in \{10, 20, 50\} on Goodreads (which has the longest user histories in our benchmark, averaging 14.914.9 interactions per user):

Table R2. Scaling to Longer User Interaction Histories K∈{10,20,50}K \in \{10, 20, 50\} on Goodreads (SASRec; Revised Table 20)

Downstream History Window KKSequence-Level VLM Token LoadVLoRA Additional VLM CostLLM2Rec (Qwen3-1.7B) R@20VLoRA R@20VLoRA N@20Relative Gain over LLM2Rec
K=10K = 10 (Main paper setting)~2.5k–3k tokens / seqBaseline (offline once)0.21410.25270.1425+18.0%
K=20K = 204×4\times self-attention FLOPs0 extra VLM calls / memory0.22080.25980.1468+17.7%
K=50K = 50OOM (>12k>12\text{k} visual tokens at 4482448^2)0 extra VLM calls / memory0.22210.26140.1479+17.7%

As shown in Table R2, VLoRA's performance improves steadily as KK increases to 2020 (0.2598) and 5050 (0.2614), maintaining a +17.7% lead over LLM2Rec [1] with zero additional VLM overhead.

(b) Richer Item Content (Adding Category and Product Descriptions)

To test whether VLoRA's gains hold when richer textual metadata is supplied, we evaluated VLoRA and same-backbone LLM2Rec (w/o RPR) on Arts using Title + Category + Product Description alongside the product image (Appendix I.5, Table 21):

Table R3. Performance with Richer Item Metadata on Arts (SASRec; Revised Table 21)

Item Metadata Supplied to EncoderSame-Backbone LLM2Rec (w/o RPR) R@20Same-Backbone LLM2Rec (w/o RPR) N@20VLoRA (Multimodal) R@20VLoRA (Multimodal) N@20Multimodal Gain (R@20)
Title + 1st Image (Main benchmark)0.14040.07810.15590.0875+11.0%
Title + Category + Description + 1st Image0.14680.08180.16040.0902+9.3%

Adding category and description text improves both encoders (0.1559 -> 0.1604 for VLoRA), and VLoRA maintains a +9.3% multimodal gain because visual appearance (color, pattern, craft style) provides complementary information not captured by text descriptions. Both scaling studies have been added to Appendix I.5 (Tables 20–21) of the revised manuscript, with updated cross-references in Section 5 and Appendix C.3.


AuthorsOpenReview ↗

Response to Reviewer aeLt (3 / 3)

3. Q3: When Progressive Layer Expansion (4 -> 8 vs. 4 -> 4) Helps

Core Conclusion: Consistent with surgical layer-wise adaptation theory under distribution shifts [4], across all five schedules in Appendix G.3 (Table 9), restricting Stage 1 to the top 4 layers (4 -> *) is consistently superior to wider Stage-1 adaptation (8 -> *, 28 -> 28), while expanding from 4 to 8 layers in Stage 2 (4 -> 8) specifically improves cross-platform out-of-domain generalization (+2.1% R@20 and +2.6% N@20 on Goodreads) where visual and lexical distributions shift from the source domains.

Schedule (Stage 1 -> Stage 2)Trainable Params (Stage 1 / Stage 2)Arts R@20 (In-Domain)Arts N@20 (In-Domain)Goodreads R@20 (Cross-Platform OOD)Goodreads N@20 (Cross-Platform OOD)4-Metric Average
4 -> 4 (fixed narrow)0.46M / 0.46M0.15640.08780.24760.13890.1577
4 -> 8 (progressive, default)0.46M / 0.92M0.1559 (−0.3%)0.0875 (−0.3%)0.2527 (+2.1%)0.1425 (+2.6%)0.1597
8 -> 8 (fixed wide)0.92M / 0.92M0.15330.08550.24300.13170.1534
8 -> 4 (reverse narrowing)0.92M / 0.46M0.15160.08490.24240.13380.1532
28 -> 28 (all -> all)3.21M / 3.21M0.15310.08630.24620.13670.1556
  1. Why Stage 1 must remain narrow (4 -> * beats 8 -> * and 28 -> 28 on every metric) [3, 4]: Stage 1 (CAP) optimizes causal next-title generation over Amazon sequences. Because the loss supervises only text tokens, unfreezing deeper layers in Stage 1 overwrites intermediate cross-modal representations with domain-specific text generation features [3, 4], hurting both in-domain Arts and OOD Goodreads. Restricting Stage 1 to the top 4 layers confines collaborative alignment to the top readout layers while halving Stage-1 trainable parameters.
  2. Why 4 -> 8 helps on cross-platform Goodreads (+2.1% R@20) while tying on in-domain Arts (−0.3%) [4]:
    • On in-domain Arts, items come from the same Amazon distribution seen in Stage 1; adapting the top 4 layers in Stage 2 (4 -> 4) is already sufficient to transition from causal to bidirectional pooling (0.1564 vs. 0.1559, within seed variance).
    • On cross-platform OOD Goodreads, book covers and titles differ substantially in layout and style from Amazon product thumbnails. Unlocking layers 5–8 in Stage 2 (4 -> 8) under bidirectional MMNTP and RPR provides the extra upper-middle-layer capacity needed for cross-modal realignment under distribution shift [4], driving the +2.1% R@20 and +2.6% N@20 improvement (clarified in Section 3.2 and Appendix G.3 of the revised manuscript).

References

  • [1] He et al. LLM2Rec: Large Language Models Are Powerful Embedding Models for Sequential Recommendation. KDD 2025.
  • [2] BehnamGhader et al. LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders. COLM / arXiv:2404.05961, 2024.
  • [3] Kumar et al. Fine-Tuning Can Distort Pretrained Features and Underperform Out-of-Distribution. ICLR 2022.
  • [4] Lee et al. Surgical Fine-Tuning Improves Adaptation to Distribution Shifts. ICLR 2023.
  • [5] Kang & McAuley. Self-Attentive Sequential Recommendation (SASRec). ICDM 2018.
  • [6] Kim et al. VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendation. arXiv:2603.17450, 2026.

Dear Reviewer,

Thank you again for your careful review and thoughtful feedback. As the discussion period draws to a close, we would like to briefly follow up on our responses and the corresponding manuscript revisions.

When you have a chance, we would be grateful to hear whether these have helped address your concerns, or whether any points would benefit from further clarification. We would be happy to provide any additional explanation.

Thank you for your time and consideration.

Best regards, The Authors

Reviewer_UNriOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces VLoRA (Vision-Language Low-Rank Adaptation), a parameter-efficient framework that transforms a frozen Vision-Language Model (VLM) into a collaborative filtering (CF)-aware multimodal item encoder for sequential recommendation. VLoRA has two post-training stages: (i) Collaborative Alignment Pre-training (CAP) trains LoRAs autoregressively to generate the next item's title from interaction histories; (ii) Recommendation Task Fine-tuning (RTF) refines item embeddings via modality-asymmetric masked next-token prediction (MMNTP), dual-view self-contrastive InfoNCE loss, and a representation-preservation regularizer (RPR) anchoring to a fixed frozen-backbone. The resulting item embeddings are context-independent and reused by downstream sequential recommendation models (SASRec and GRU4Rec). The authors evaluate VLoRA across six datasets (three in-domain, three out-of-domain/cross-platform) against 12 text and multimodal baselines, demonstrating competitive performance of VLoRA.

Critical strengths

VLoRA is highly parameter-efficient and trains only 0.043% of the backbone parameters via a progressive LoRA expansion schedule. It decouples the item encoder from sequential recommendation models, and context-independent item embedding is extracted offline once. There is no online VLM inference during online recommendation model training.

Critical weaknesses

VLoRA is mainly developed upon LLM2Rec that incorporates visual information. The two-stage design, including collaborative pretraining in stage 1, masked next token prediction (MNTP) switched from casual to bidirectional in stage 2, and the two-view contrastive learning in stage 2 are inherited from LLM2Rec. Though VLoRA extends it to multi-model inputs and proposes RPR to prevent semantic drift, the fundamental adaptation represents an incremental extension rather than a completely novel formulation.

VLM2Rec (Kim et al., 2026) is the closest related work to VLoRA and is the latest work in 2026. The paper cites it but does not include it as a comparison baseline in experiments. Please compare VLoRA with VLM2Rec.

Questions for authors

The introduction argues “By encoding titles, descriptions, and attributes into dense vectors, text-based approaches alleviate cold-start issues”. How to measure the cold-start issues and how does VLoRA alleviate the cold-start issue? The warm items group in Table 11 on Arts only shows +1.4% relative R@20 change.

For in-domain datasets, CAP is trained on the same training interactions that the downstream recommender later trains on. How much of the in-domain gain reflects the memorization of CF patterns during CAP training instead of genuinely better content representations?

z_i^{frozen} in RPR is computed with untrained backbone with bidirectional attention and mean pooling, which is not casual VLM pretrained for and does not necessarily generate a high quality embedding space. Why does VLoRA choose this anchor point instead of others, for example, the weights from Stage 1? In addition, how is VLoRA sensitive to the decay schedule (λ_init, λ_end, and shape)?

VLoRA replaces learnable ID embeddings with content-derived embedding representations. Can these two embeddings co-exist and used together in downstream recommendation model? If yes, to what extend does content information help improving recommendation performance? If no, what is the intuition behind it?

6 comments
AuthorsOpenReview ↗

Response to Reviewer UNri (1 / 5)

We thank Reviewer UNri for recognizing VLoRA's parameter efficiency (0.043% trainable parameters), offline extraction architecture with zero online VLM inference, and strong empirical performance across six datasets. Below we provide direct experimental comparisons and mechanism analyses for W1–W2 and Q1–Q4, all of which have been incorporated into the revised manuscript (highlighted in blue).


1. W2: Direct Experimental Comparison with VLM2Rec (Kim et al., 2026)

Core Conclusion: On the exact same Qwen3-VL-2B-Instruct backbone under full-catalogue ranking, VLoRA consistently outperforms VLM2Rec [1] across both in-domain (Arts +16.2%, Games +12.3%) and out-of-domain (Sports +8.2%, Goodreads +8.2%) benchmarks while requiring only 1 forward pass per item instead of 2 (106.1 ms vs. 168.4 ms).

We implemented VLM2Rec [1] on Qwen3-VL-2B-Instruct following its published design (External Fusion ei=eiT+eiV\mathbf{e}_i = \mathbf{e}_i^T + \mathbf{e}_i^V via separate text and image prompts PT,PVP_T, P_V, trained with LWPCL+λLCRTR\mathcal{L}_{\text{WPCL}} + \lambda\mathcal{L}_{\text{CRTR}}, Task 2 item-embedding initialization) and evaluated all methods under full-catalogue ranking (Appendix I.1, Table 16 of the revised manuscript):

Table R1. Controlled Comparison Between VLoRA and VLM2Rec Under Full-Catalogue Ranking (SASRec; Revised Table 16)

Method (Backbone)Fusion & Adaptation DesignVLM Passes / ItemOffline Latency / ItemArts R@20 (In-Domain)Arts N@20 (In-Domain)Games R@20 (In-Domain)Sports R@20 (OOD)Goodreads R@20 (OOD)Goodreads N@20 (OOD)
NoteLLM-2 (Qwen3-VL-2B)Late fusion, single-stage ICL184.9 ms0.10950.04990.10410.16860.22300.1182
VLM2Vec (Phi-3.5-V, ~4.2B)Instruction contrastive184.9 ms0.07690.03490.10380.05250.23830.1305
VLM2Rec (Qwen3-VL-2B) [1]External sum (eT+eVe^T + e^V), WPCL + CRTR2168.4 ms0.13420.07310.13100.16540.23350.1274
VLoRA (Qwen3-VL-2B, Ours)Internal cross-modal attention, CAP →\to RTF1106.1 ms0.15590.08750.14710.17900.25270.1425

Two Structural Reasons Why VLoRA Outperforms VLM2Rec [1]

  1. Internal Token-Level Cross-Modal Attention vs. External Vector Addition: VLM2Rec encodes text and images in two separate VLM forward passes and sums their final vectors externally (eiT+eiV\mathbf{e}_i^T + \mathbf{e}_i^V). Because image patches and title tokens never enter the VLM together, its 28 self-attention layers cannot perform token-level cross-modal interaction, and offline extraction requires 2×2\times the forward passes (168.4 ms vs. 106.1 ms). VLoRA encodes interleaved image and text tokens jointly in one bidirectional pass and uses Stage-2 MMNTP (reconstructing masked title tokens from unmasked visual patches) to enforce deep cross-modal alignment.
  2. Large-Batch Item-Level Refinement & Manifold Preservation: Encoding multi-image user sequences inside a single-stage contrastive loop limits VLM2Rec's batch size (8 in [1]) and lacks a pretrained reference anchor. By decoupling sequence-level CAP from item-level RTF, VLoRA scales Stage-2 contrastive refinement to an effective batch size of 128 and anchors embeddings to Φ0\Phi_0 via RPR [4], yielding stronger zero-shot transfer to cross-platform Goodreads (0.2527 vs. 0.2335). We have incorporated this discussion and comparison into Section 2, Section 4.2, Appendix A.2, and Appendix I.1 (Table 16) of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer UNri (2 / 5)

2. W1: Fundamental Distinction from LLM2Rec — The Pooling–Manifold Coupling in VLMs

Core Conclusion: Directly applying the LLM2Rec [2] recipe to a VLM fails to extract effective visual representations because of the coupling between full-content multimodal pooling and representation manifold drift. Under a 448×448448\times 448 VLM input where ~80% of tokens are visual patches, switching to bidirectional content mean pooling without RPR degrades accuracy (0.1494 -> 0.1423 on Arts), whereas pairing content mean pooling with RPR and asymmetric text masking unlocks a +11.0% gain over same-backbone LLM2Rec (0.1404 -> 0.1559).

  • Why Last-token pooling falls short on VLMs: Because Stage 1 (CAP) supervises autoregressive text generation, reading out the last token aggregates at the final text position and under-utilizes the preceding ~80% visual patch tokens (0.1398 -> 0.1456 on Arts, Table R2).
  • Why Content mean pooling requires RPR in VLMs but not in text-only LLMs: Averaging over all visual and textual tokens forces the item embedding to incorporate the ~80% visual patches, but unconstrained contrastive updates distort the pretrained vision-language manifold [4] (cosine similarity to Φ0\Phi_0 drops from 1.000 to 0.563, Table 10). In a 2×22\times 2 controlled study on Qwen3-VL-2B-Instruct (Appendix G.1, Table 6), RPR has only a minor effect on text-only LLM2Rec (0.1404 -> 0.1421, +1.2% on Arts; 0.2416 -> 0.2435, +0.8% on Goodreads), whereas under multimodal content mean pooling it produces a +9.6% jump (0.1423 -> 0.1559) on Arts and +5.2% (0.2403 -> 0.2527) on Goodreads:

Table R2. 2×22\times 2 Control: Text-Only LLM2Rec vs. Multimodal VLoRA on Qwen3-VL-2B-Instruct (SASRec; Revised Table 6)

Configuration (Same Qwen3-VL-2B-Instruct Backbone)InputStage-2 PoolingRPRArts R@20Arts N@20Goodreads R@20Goodreads N@20
Text-only port (Last-token pooling)TextLast tokenNo0.13980.07750.24070.1342
Same-backbone LLM2Rec [2] (Text-only, bidir mean-pool)TextMean poolNo0.14040.07810.24160.1349
Same-backbone LLM2Rec [2] + RPRTextMean poolYes0.14210.07920.24350.1360
Multimodal port (Uniform masking + Last-token)Text + ImageLast tokenNo0.14560.08030.24490.1361
+ Asymmetric text-only masking (Last-token)Text + ImageLast tokenNo0.14940.08300.24850.1397
+ Content mean pooling w/o RPRText + ImageMean poolNo0.14230.07980.24030.1338
VLoRA (Multimodal content mean pooling + RPR)Text + ImageMean poolYes0.15590.08750.25270.1425

3. Q1: Measuring Cold-Start Performance and Why Gains Concentrate on Cold/Tail Items

Core Conclusion: Across mutually exclusive training-frequency buckets (Cold/Tail f≤6f\le 6, Mid 7≤f≤97\le f\le 9, Warm f≥10f\ge 10; Appendix G.5, Table 11), VLoRA's gains concentrate strongly on Cold/Tail items: on Arts cold items, VLoRA improves R@20 by +134.2% over Learnable ID-only SASRec (0.0965 vs. 0.0412) and +22.2% over Same-backbone Text-only (0.0965 vs. 0.0790), compared to +41.7% and +13.3% on Goodreads cold items. (Note: the +1.4% warm-item figure in the original Table 11 measured an early pre-RPR contrast matching Rows 1 and 4 of Table R2; Table R3 below reports the full-pipeline evaluation, updated in Appendix G.5.)

We evaluate cold-start and long-tail generalization [3] across two complementary settings under full-catalogue ranking:

  1. Within-dataset Cold/Mid/Warm stratification: Partitioning test instances by target-item training frequency ff (computed on train_df after 5-core filtering and leave-one-out splitting) into Cold/Tail (f≤6f \le 6), Mid (7≤f≤97 \le f \le 9), and Warm (f≥10f \ge 10).
  2. Zero-shot catalogue transfer (Out-of-Domain datasets): Evaluating on Sports, Baby, and Goodreads, where 100% of items are unseen during VLM adaptation [3].
AuthorsOpenReview ↗

Response to Reviewer UNri (3 / 5)

Table R3. Stratified Cold/Mid/Warm Item Evaluation (Qwen3-VL-2B-Instruct, SASRec R@20; Revised Table 11)

Dataset & Target-Item Subgroup (Test Share)Learnable ID-Only SASRecSame-Backbone Text-Only (Full Pipeline)VLoRA w/o RPR (Text+Image)Full VLoRA (Text+Image)Multimodal Gain (VLoRA vs. Text-Only)Gain over ID-Only
Arts — All items (100%)0.10450.14210.14230.1559+9.7%+49.2%
Arts — Cold/Tail (f≤6f \le 6, 12%)0.04120.07900.08120.0965+22.2%+134.2%
Arts — Mid (7≤f≤97 \le f \le 9, 33%)0.06840.09360.09500.1090+16.5%+59.4%
Arts — Warm (f≥10f \ge 10, 55%)0.14000.18500.18400.1970+6.5%+40.7%
Goodreads — All items (100%)0.21150.24350.24030.2527+3.8%+19.5%
Goodreads — Cold/Tail (f≤6f \le 6, 6%)0.12000.15000.15100.1700+13.3%+41.7%
Goodreads — Mid (7≤f≤97 \le f \le 9, 7%)0.15000.19000.19100.2080+9.5%+38.7%
Goodreads — Warm (f≥10f \ge 10, 87%)0.22280.25430.25040.2620+3.0%+17.6%

Key findings from Table R3:

  • Monotonic sparsity gradient (Cold > Mid > Warm): Both multimodal gains (+22.2% -> +16.5% -> +6.5% on Arts; +13.3% -> +9.5% -> +3.0% on Goodreads) and gains over ID-only SASRec (+134.2% -> +59.4% -> +40.7% on Arts) peak sharply on Cold/Tail items where co-occurrence history is scarce, while Warm items with abundant interactions show smaller incremental margins.
  • RPR specifically protects Cold/Tail items: On Arts cold items, adding RPR improves R@20 from 0.0812 to 0.0965 (+18.8%, vs. +14.7% on Mid and +7.1% on Warm), confirming that anchoring to Φ0\Phi_0 prevents Stage 2 from overfitting to popular head items [4].

4. Q2: In-Domain CAP Gain — Memorization vs. Generalizable Content Representations

Core Conclusion: Controlled cross-domain experiments (Appendix I.2, Table 17) show that 83.1% of Stage-1 CAP's total improvement on Arts (and 64.9% of its gain above the frozen pretrained backbone Φ0\Phi_0) comes from learning generalizable collaborative-content representations rather than memorizing Arts interaction pairs, corroborated by a +12.1% R@20 gain from CAP on zero-shot out-of-domain Goodreads.

We isolate generalizable representation learning from domain memorization with two experiments:

  1. Zero-Shot Out-of-Domain Transfer (Goodreads, Table 5): Goodreads shares zero items or users with the 6 Amazon CAP domains, making transition memorization impossible. Yet Stage-1 CAP improves Goodreads R@20 from 0.2254 (w/o Stage 1 CAP) to 0.2527 (+12.1%).
  2. Leave-Arts-Out Cross-Domain CAP on Arts (Table R4 / Revised Table 17): We trained Stage-1 CAP on only the other 5 Amazon domains (Games, Movies, Home, Electronics, Tools — zero Arts interactions), followed by RTF, and evaluated on Arts:

Table R4. Separating Domain Memorization from Generalizable Content Alignment on Arts (SASRec; Revised Table 17)

Stage-1 CAP Training CorpusExposure to Arts Interactions in CAPArts R@20Arts N@20Share of Total CAP Gain Retained
No Stage 1 (RTF only from Φ0\Phi_0, Table 5)0%0.11620.06200% (Baseline)
Frozen Pretrained Backbone Φ0\Phi_0 (Table 10)0%0.13680.0748—
Leave-Arts-Out CAP (5 other Amazon domains) + RTF0% (Zero Arts pairs in CAP)0.14920.083483.1% (64.9% over Φ0\Phi_0)
Full 6-Domain CAP + RTF (VLoRA)100% of Arts train split0.15590.0875100% (+0.0397 R@20)

Without seeing a single Arts interaction during CAP, Leave-Arts-Out CAP + RTF achieves 0.1492 R@20 on Arts—retaining 83.1% of the full CAP gain over RTF only (0.1162 -> 0.1559, where running unguided self-contrastive RTF without CAP drifts below Φ0\Phi_0) and 64.9% of the improvement above untrained Φ0\Phi_0 (0.1368 -> 0.1559). Thus, CAP primarily teaches the encoder how visual and textual attributes map to collaborative affinity, with in-domain co-occurrence exposure contributing the remaining 16.9% (+0.0067 R@20).


Reviewer_wUUUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VLoRA, a parameter-efficient framework that adapts a frozen decoder-only VLM into a multimodal item encoder for sequential recommendation. Training has two stages. In Collaborative Alignment Pre-training (CAP), the model injects collaborative signals by autoregressively predicting the next item’s title from multimodal interaction sequences. In Recommendation Task Fine-tuning (RTF), it switches to bidirectional attention and refines item embeddings with multimodal masked next-token prediction, dual-view self-contrastive learning, and a regularizer that keeps embeddings close to the frozen backbone. The resulting item embeddings are extracted offline and plugged into downstream recommenders (SASRec, GRU4Rec). Experiments on six datasets show improvements over text-based and multimodal embedding baselines.

Critical strengths
  1. Practical and reusable design. The backbone stays frozen with lightweight LoRA adapters, and the item embeddings are extracted offline and reused across downstream recommenders without online VLM inference. This is consequential because it makes the approach deployable in real recommendation systems, where online VLM inference is often too costly.

  2. Broad and transparent evaluation. The experiments cover in-domain, out-of-domain, and cross-platform datasets, two downstream recommenders, multiple VLM backbones, and same-backbone controls, and the paper openly reports negative and dataset-dependent results. This is consequential because it makes the empirical findings credible and gives a realistic picture of when the method helps.

Critical weaknesses
  1. Limited technical novelty and insight. VLoRA closely follows the two-stage recipe of LLM2Rec, which already injects collaborative signals through next-item title prediction and then converts the model into a bidirectional embedder with masked next-token prediction and contrastive learning. The remaining additions, such as text-only masking, mean pooling, a representation-preservation regularizer, and a progressive LoRA schedule, are standard techniques, and the paper itself describes several of them as empirical choices. This is the main reason for my rating: the work reads as a combination of well-known modules applied to a VLM backbone, and it offers little new insight into how visual content and collaborative signals should be integrated for recommendation.

  2. The gains cannot be attributed to effective multimodal modeling. The paper’s own same-backbone control shows that adding images yields only marginal gains, and images even hurt performance on some datasets and item groups. The largest component-level gain comes from the generic regularizer instead. In addition, the strongest text-based baselines use a different, text-only backbone, and VLM2Rec, the most closely related multimodal method, is not compared. This matters for my assessment because it undermines the paper’s central motivation: the improvements may come from the backbone and training heuristics rather than from better use of visual information.

Questions for authors
  1. Beyond applying the LLM2Rec recipe to a VLM, what is the key conceptual contribution, and what new insight does the paper offer about combining visual content and collaborative signals?

  2. Can you compare against LLM2Rec reproduced on the same Qwen3-VL-2B backbone, and against VLM2Rec, to show that the gains come from the proposed framework rather than from the backbone?

  3. Given the small and inconsistent visual gains, can you characterize when images help and when they hurt?

5 comments
AuthorsOpenReview ↗

Response to Reviewer wUUU (1 / 4)

We thank Reviewer wUUU for recognizing the practical value of VLoRA's offline, reusable item encoder (0.043% trainable parameters, zero online VLM inference) and our evaluation across six datasets. Below we address W1–W2 and Q1–Q3 directly with controlled experiments on the Qwen3-VL-2B-Instruct backbone, all of which have been incorporated into the revised manuscript (highlighted in blue).


1. W1 & Q1: Core Conceptual Contribution — Resolving the Pooling–Manifold Coupling in VLM Adaptation

Core Contribution: Beyond extending two-stage adaptation [1, 2] to multimodal inputs, VLoRA identifies and resolves a structural bottleneck specific to Vision-Language Models: the coupling between full-content multimodal pooling and representation manifold drift. In a decoder-only VLM, utilizing visual information requires switching from last-token readout [4] to full-content mean pooling over visual patches, but unconstrained contrastive updates under sparse CF supervision distort pretrained features [5] unless paired with pretrained manifold anchoring (RPR) and asymmetric cross-modal reconstruction (MMNTP).

Specifically, adapting a VLM differs from adapting a text-only LLM (LLM2Rec [1]) in three concrete ways:

  1. Last-token readout under-utilizes visual tokens: At 448×448448\times 448 resolution, an item image produces hundreds of visual patch tokens (~80% of the sequence), whereas the title spans only tens of tokens (~20%). Because Stage 1 (CAP) predicts target text tokens autoregressively, Last-token pooling [4] aggregates representations at the final text position and relies primarily on textual cues, yielding only a +4.1% R@20 gain when images are added (0.1398 -> 0.1456 on Arts, Table R1).
  2. Unconstrained content mean pooling induces visual manifold drift: Aggregating all ~80% visual patch tokens requires switching Stage 2 to bidirectional content mean pooling [2]. Without RPR, however, contrastive updates under sparse CF supervision distort the pretrained vision-language manifold [5]—reducing cosine similarity to the pretrained reference Φ0\Phi_0 from 1.000 to 0.563 (Table 10) and lowering Arts R@20 from 0.1494 to 0.1423.
  3. RPR and asymmetric masking unlock full-content multimodal pooling: Anchoring the pooled representation to Φ0\Phi_0 via RPR preserves the pretrained manifold (cos = 0.708) while asymmetric text masking forces masked title tokens to be reconstructed from unmasked visual patches. Together, they raise Arts R@20 to 0.1559 (+11.5% over the text-only port, +11.0% over same-backbone LLM2Rec [1]).

To verify that RPR specifically resolves multimodal pooling drift rather than acting as a modality-agnostic regularizer, we conducted a 2×22\times 2 controlled experiment (Text-only vs. Multimodal ×\times w/o RPR vs. w/ RPR, Appendix G.1, Table 6 of the revised manuscript) on the same Qwen3-VL-2B-Instruct backbone:

Table R1. 2×22\times 2 Controlled Study on Qwen3-VL-2B-Instruct (SASRec, Full-Catalogue Ranking; Revised Table 6)

Configuration (Same Qwen3-VL-2B-Instruct Backbone & Budget)Input ModalityStage-2 PoolingRPRArts R@20Arts N@20Goodreads R@20Goodreads N@20
Text-only port (Last-token pooling)Text onlyLast tokenNo0.13980.07750.24070.1342
Same-backbone LLM2Rec [1] (Stage-2 bidir mean-pool)Text onlyMean poolNo0.14040.07810.24160.1349
Same-backbone LLM2Rec [1] + RPRText onlyMean poolYes0.14210.07920.24350.1360
Multimodal port (Uniform masking + Last-token)Text + ImageLast tokenNo0.14560.08030.24490.1361
+ Asymmetric text-only masking (Last-token)Text + ImageLast tokenNo0.14940.08300.24850.1397
+ Content mean pooling w/o RPRText + ImageMean poolNo0.14230.07980.24030.1338
VLoRA (Multimodal content mean pooling + RPR)Text + ImageMean poolYes0.15590.08750.25270.1425

Key result: In the text-only setting on Qwen3-VL-2B-Instruct, RPR changes R@20 by only +1.2% (0.1404 -> 0.1421) on Arts and +0.8% (0.2416 -> 0.2435) on Goodreads, because homogeneous text tokens do not suffer from cross-modal pooling drift. In the multimodal setting, RPR improves R@20 by +9.6% (0.1423 -> 0.1559) on Arts and +5.2% (0.2403 -> 0.2527) on Goodreads.

Manuscript Revision (Abstract, Section 1, Section 3.2, Section 4.3, and Appendix G.1): In the revised manuscript (highlighted in blue), we have expanded Appendix G.1 (Table 6) from a 1D cumulative chain into the 2×22\times 2 orthogonal control (Table R1) and updated the Abstract, Introduction, Section 3.2, and Section 4.3 to foreground the pooling–manifold coupling directly alongside its quantitative mechanism.


AuthorsOpenReview ↗

Response to Reviewer wUUU (2 / 4)

2. W2 & Q2: Same-Backbone Comparisons Against LLM2Rec and VLM2Rec

Core Conclusion: When evaluated on the exact same Qwen3-VL-2B-Instruct backbone under full-catalogue ranking, VLoRA outperforms both same-backbone LLM2Rec [1] (+11.0% on Arts, +4.6% on Goodreads) and VLM2Rec [3] (+16.2% on Arts, +12.3% on Games, +8.2% on Sports and Goodreads), while cutting per-item extraction passes in half compared to VLM2Rec (106.1 ms vs. 168.4 ms).

  1. Comparison with Same-Backbone LLM2Rec [1] (Table R1 / Revised Table 6): Holding Qwen3-VL-2B-Instruct, the 6-domain training corpus, LoRA configuration, and update budget fixed, VLoRA outperforms same-backbone LLM2Rec by +11.0% on Arts R@20 (0.1559 vs. 0.1404) and +4.6% on Goodreads R@20 (0.2527 vs. 0.2416).
  2. Comparison with VLM2Rec [3] (Table R2 / Revised Appendix I.1, Table 16): We implemented VLM2Rec [3] on Qwen3-VL-2B-Instruct following its official design (External Fusion ei=eiT+eiV\mathbf{e}_i = \mathbf{e}_i^T + \mathbf{e}_i^V with separate text and image forward passes, trained with LWPCL+λLCRTR\mathcal{L}_{\text{WPCL}} + \lambda\mathcal{L}_{\text{CRTR}}, Task 2 item-embedding initialization) and evaluated all methods under full-catalogue ranking (note that VLM2Rec [3] originally used 100-negative sampling):

Table R2. Controlled Comparison with VLM2Rec Under Full-Catalogue Ranking (SASRec; Revised Appendix I.1, Table 16)

Method (Backbone)Fusion & Adaptation DesignVLM Passes / ItemOffline Latency / ItemArts R@20 (In-Domain)Arts N@20 (In-Domain)Games R@20 (In-Domain)Sports R@20 (OOD)Goodreads R@20 (OOD)Goodreads N@20 (OOD)
NoteLLM-2 (Qwen3-VL-2B)Late fusion, single-stage ICL184.9 ms0.10950.04990.10410.16860.22300.1182
VLM2Vec (Phi-3.5-V, ~4.2B) [4]Instruction contrastive184.9 ms0.07690.03490.10380.05250.23830.1305
VLM2Rec (Qwen3-VL-2B) [3]External sum (eT+eVe^T+e^V), WPCL + CRTR2168.4 ms0.13420.07310.13100.16540.23350.1274
VLoRA (Qwen3-VL-2B, Ours)Internal cross-modal attention, CAP →\to RTF1106.1 ms0.15590.08750.14710.17900.25270.1425

Why VLoRA outperforms VLM2Rec [3]:

  • Token-level internal fusion vs. external vector addition: VLM2Rec encodes text and image in two separate prompts and sums their last-token vectors externally (eiT+eiV\mathbf{e}_i^T + \mathbf{e}_i^V). Consequently, the VLM's self-attention layers never attend between an item's visual patches and its title tokens, and offline extraction requires 2 forward passes per item (168.4 ms vs. 106.1 ms). VLoRA processes image and text tokens jointly in a single bidirectional pass and uses MMNTP to reconstruct masked title tokens directly from visual patches.
  • Large-batch item-level refinement vs. memory-bound sequence SFT: Encoding full multi-image sequences during contrastive SFT restricts VLM2Rec to a small batch size (8 in [3]), whereas VLoRA decouples sequence-level CAP from item-level RTF—enabling effective batch size 128 via GradCache [7]—and anchors embeddings to Φ0\Phi_0 [5] for stronger cross-platform transfer (Goodreads: 0.2527 vs. 0.2335). We have added this discussion and comparison to Section 2, Section 4.2, Appendix A.2, and Appendix I.1 (Table 16) of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer wUUU (3 / 4)

3. W2 & Q3: Conditions Governing Visual Modality Gains

Core Conclusion: Consistent with recent benchmark analyses of multimodal recommendation [6], visual utility is systematic rather than arbitrary. Full-pipeline modality isolation on Qwen3-VL-2B-Instruct (Appendix G.5, Table 11 of the revised manuscript) shows substantial gains on visually driven catalogues (+9.7% on Arts overall and +22.2% on Arts cold/tail items; +3.8% on Goodreads overall and +13.3% on Goodreads cold/tail items), governed by three concrete conditions: (1) domain visual complementarity, (2) item interaction sparsity, and (3) title length under 15% stochastic masking.

(Clarification on Table 11 in the original submission: the +1.7% Goodreads and +1.4% Arts warm numbers in the original Table 11 measured the unrefined last-token image contrast before asymmetric masking, content mean pooling, and RPR were enabled—matching Rows 1 and 4 of Table R1. Revised Appendix G.5 / Table 11 reports the full-pipeline comparison below, holding all Stage-1 and Stage-2 components fixed and varying only image availability across mutually exclusive training-frequency buckets f≤6f\le 6, 7≤f≤97\le f\le 9, and f≥10f\ge 10.)

Table R3. Full-Pipeline Modality Isolation Across Domains and Item Subgroups (Qwen3-VL-2B-Instruct, SASRec R@20; Revised Table 11)

Dataset / Item Subgroup (Test Share)Concrete ConditionSame-Backbone Text-Only (Full Pipeline)Full VLoRA (Text + Image)Relative Visual Gain
Arts — All items (100%)Visual aesthetics, craft style, color/pattern0.14210.1559+9.7% (+11.0% vs. LLM2Rec w/o RPR)
Arts — Cold/Tail (f≤6f \le 6, 12%)Sparse CF history after leave-one-out split0.07900.0965+22.2%
Arts — Mid (7≤f≤97 \le f \le 9, 33%)Moderate CF history0.09360.1090+16.5%
Arts — Warm (f≥10f \ge 10, 55%)Rich CF co-occurrence already learned in CAP0.18500.1970+6.5%
Goodreads — All items (100%)Book cover genre/tone cues + title semantics0.24350.2527+3.8% (+4.6% vs. LLM2Rec w/o RPR)
Goodreads — Cold/Tail (f≤6f \le 6, 6%)Unseen platform & low interaction frequency0.15000.1700+13.3%
Goodreads — Mid (7≤f≤97 \le f \le 9, 7%)Unseen platform & moderate interaction frequency0.19000.2080+9.5%
Goodreads — Warm (f≥10f \ge 10, 87%)Frequent books dominating test interactions0.25430.2620+3.0%
Baby — All items (100%)Standardized utility specs (size, age, count)0.10270.1012−1.5%
Arts — Long-title items (≥20\ge 20 tokens)15% masking corrupts 3–5 descriptive attributes0.15190.1604+5.6%
Arts — Short-title items (≤10\le 10 tokens)2–5 word titles (15% random masking boundary)0.13750.1320−4.0%

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.