acceptodds
Withdrawn from ICLR 2027

Deep Interest Mining for Intent-Enriched Semantic IDs in Multimodal Generative Recommendation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

14 views

Abstract

Semantic IDs (SIDs) provide the discrete item vocabulary used by generative recommendation, but their quality depends on what item evidence is preserved before quantization. In product recommendation, surface metadata often misses latent usage intent, visual evidence may be only weakly reflected in text, and downstream policy learning provides sparse feedback about whether a generated SID corresponds to a semantically useful item. We introduce DeepInterestGR, an intent-enriched SID framework for generative recommendation. Before SID quantization, CMSA enriches item representations through two complementary evidence paths: recommendation-oriented VLM captions and projected image embeddings. DCIM then uses an LLM to mine item-side intent descriptors, which capture latent usage motivations implied by product content rather than personalized user states. During policy training over the constructed SIDs, QARM adds a target-gated descriptor-quality bonus on top of standard SID rewards, applying the bonus only when the generated SID decodes to the target item. Thus, descriptor quality cannot reward a fluent but irrelevant item prediction. Experiments on three primary Amazon Product Review categories show that DeepInterestGR improves over competitive generative and RL-based baselines, with relative gains of up to 15.1% in NDCG@5 and 13.9% in NDCG@10 over the strongest per-metric baseline; additional category-level checks on two more categories provide a compact robustness check without expanding the claim to full-catalog generalization. Component ablations, CMSA branch analyses, reward variants, and SID-level case studies support a bounded claim: enriching pre-quantization item evidence with visual cues and item-side intent descriptors, together with target-gated descriptor-quality weighting, improves SID-based generative recommendation under the evaluated settings.

Primary area:
generative models
Keywords:
Generative Recommendation, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_12qZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DeepInterestGR, a framework for enriching item representations before SID generation. DeepInterestGR is composed of three modules: CMSA, DCIM, and QARM. CMSA exploits visual evidence from images, covering both caption-level descriptions and image features. DCIM extracts item-side information from textual evidence so that distinct usage contexts can be properly utilized. QARM evaluates the quality of generated descriptors when the prediction is correct, improving the precision quality by policy learning. DeepInterestGR overcomes two failure modes mentioned in the paper, surface ambiguity and evidence omission, and outperforms three families of baselines across all datasets and evaluation metrics.

Critical strengths
  • The paper identifies a limitation in generative recommendation systems that important information can be discarded before SID quantization. Therefore, the paper aims at enriching the context before SID quantization, which can help improve the accuracy of downstream generators.
  • The paper exploits item information from both images and textual descriptions. Omitted and ambiguous item context can be discovered and utilized for downstream tasks.
Critical weaknesses
  • Despite the contributions by CMSA, DCIM, and QARM, the entire framework, as well as its quality, is largely dependent on the pretrained models such as Qwen2.5-VL-7B-Instruct for caption generation and Qwen2.5-7B-Instruct for DCIM. In this sense, the novelty of the paper remain limited.
  • Potential error propagation. It is stated in the paper that DCIM extracts usage contexts also from CMSA captions. If generated CMSA captions contain false information, this can also affect the effectiveness of DCIM.
Questions for authors
  • Could the authors provide some qualitative examples where the model makes mistakes on surface ambiguity and evidence omission? Also, could the authors provide examples where the model overcomes above two failure modes but still gives incorrect predictions? The latter is important for detecting deficiencies in the proposed framework.
  • Could the authors explain and provide evidence or examples of how the framework performs compared to other baselines when two failure modes, surface ambiguity and evidence omission, are absent from product contexts?
  • Could the authors provide results on broader categories or datasets?
Reviewer_y3JLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The main idea of the paper is centered around augmenting the item representation that is quantized into Semantic IDs, on the premise that evidence missing before quantization cannot be exposed by the resulting codes. DeepInterestGR keeps a MiniOneRec-style pipeline (three-level RQ-VAE, Qwen2.5-7B generator trained with SFT and then GRPO) and adds three components. CMSA captions each item's main image with Qwen2.5-VL-7B and fuses a projected, frozen CLIP ViT-L/14 embedding with the text embedding. DCIM prompts Qwen2.5-7B for item-side usage or purchase contexts, which are appended to the title, description and caption before Qwen3-Embedding-4B encoding. QARM has the same LLM label each descriptor as specific and grounded or not, aggregates the labels into a confidence-weighted item score qiq_i, and adds 0.5 qi0.5\,q_i to the GRPO reward only when the generated SID decodes to the target item. On Amazon Beauty, Sports and Instruments (5-core, leave-last-out), Table 1 reports gains of 9.2% to 15.1% over the strongest of 14 re-run baselines, plus a two-baseline check on Industrial and Toys. Mechanism evidence comes from Beauty ablations, a Surface-only rewrite control and SID-neighborhood diagnostics on all three datasets, rewriting controls on a hard slice, human validation of the quality labels, three-seed shuffled-qq controls and an analysis of when the bonus survives GRPO normalization.

Critical strengths

Backbone-matched headline comparison for a cheap, offline-only change. The strongest baseline is a MiniOneRec re-run with "the same Qwen2.5-7B backbone and evaluation protocol" (C.1, l.804-806), and the tokenizer is MiniOneRec's RQ-VAE with "only the pre-quantization input representation" changed (B.4, l.793-795). Taken at face value, all 12 Table 1 deltas recompute (+9.2% to +15.1%, MiniOneRec strongest in every column), they correspond to unpaired two-proportion z of about 2.8-3.8 for HR@5 at the stated 22K-36K test users, and no Table 1 row violates the HR/NDCG consistency bounds. The added stages cost roughly 4-47 A100-hours offline against 15-120 per GRPO run (Table 13), online inference calls only the SID generator (l.1023-1024), and the released code writes the exact-target gate as described (src/deepinterestgr/qarm.py:196-207) with DCIM and QARM inputs limited to title, description and caption (src/deepinterestgr/schemas.py:92-123), so I see no interaction-leakage path at those interfaces.

Controls aimed at the obvious alternative explanations, with candid scoping. Against "it is just LLM rewriting," a Surface-only prompt holds the LLM, evidence, output budget, SID pipeline and reward fixed (D.1, l.853-858), and summarization and paraphrasing controls run on a title-TF-IDF hard slice (Table 8); against "it is just reward magnitude," a fixed-permutation shuffled-qq control runs over three seeds (Tables 5 and 11). The quality labels are checked against two annotators and a cross-family judge, and Table 10 recomputes exactly from the E.1 confusion counts (κ\kappa = 0.684 for QARM vs. adjudicated labels, 0.762 between annotators); App. E.3 also reports openly that group normalization removes a target-only bonus in two-level groups (l.1011-1015), so QARM changes advantages in only 14.0% of groups. With the explicit limits at l.099-100, l.388-391, l.475-476 and l.809, the design asks the right questions.

Critical weaknesses

The reported numbers, Table 1 included, show regularities that independently trained and evaluated runs should not produce. (i) In Table 1 (p.6), HR@10/HR@5 differs between Sports and Instruments by at most .0083 for all 15 methods (SASRec 1.617/1.615, MiniOneRec 1.575/1.579, DeepInterestGR 1.555/1.560). With one test item per user, each such ratio carries a multinomial sampling error of about .02-.04 even at 36K test users, so even if every method's true ratio were identical on the two datasets, the largest difference is 0.22 standard errors and χ2≈0.25\chi^2 \approx 0.25 on 15 df (lower-tail P≈10−11P \approx 10^{-11} if the methods' errors were independent; methods share test users, so their errors are correlated, but PP stays at or below about 5×10−55\times10^{-5} for any common correlation up to 0.9). For the seven classical baselines the ratio also agrees across all three datasets more closely than sampling allows (e.g., SASRec 1.621/1.617/1.615), and the Instruments/Beauty HR@5 ratio is .901-.920 for all 15 methods (CV 0.51%; 0.67% for N@5, 0.58% for N@10), below the 0.9-1.5% that Table 11's own seed SDs imply for a ratio of two single runs. (ii) In Table 7 (p.17), every Sports Surface-only entry is the exact midpoint of the w/o-DCIM and DCIM entries (steps of 17/17, 23/23, 11/11 and 13/13 in units of 1e-4), as are three of the four Instruments entries (23/23, 33/33, 15/15; the fourth, 19/20, is as close as rounding allows), although the variants have independently trained codebooks (l.413, l.910-911) and separate Exact+Valid+LCP training (D.1, l.856-857). Placing the true Surface-only value exactly at the midpoint and adding seed noise of Table 11's size, my simulations put the chance of all eight cells landing there on the order of 1e-5 at a cross-metric correlation of 0.95, 4e-4 at 0.99 and 3e-3 even with perfectly correlated metrics. (iii) In Table 9 (p.18), the six context-consistency increments (Surface-only minus w/o DCIM and DCIM minus Surface-only, three datasets) are .085, .089, .087, .088, .084 and .089 (SD .0021; the Prefix-AUC increments are as uniform), whereas per-anchor consistency moves in steps of 0.2 over 100 anchors, which puts each increment's standard error at about .01 or more even with shared anchors (probability about 1e-3 at that floor). All nine intervals (±.032 to ±.037) are also 10-17% narrower than the 95% binomial width for 500 binary judgments (±.038 to ±.044), although anchor-level resampling (l.917) should widen them unless judgments within an anchor are negatively correlated. (iv) In Table 11 (p.19), Seed 1 (seed 42, artifacts/seeds_and_preprocessing.json:7) equals the rounded three-seed mean in all eight Strict and Shuffled rows, though not in the True QARM rows (where it is the median), and the Seed 1 runs are the ones printed in the Beauty rows of Tables 1, 3-5 and 7; on its own this is only suggestive. (v) Table 6's Toys S-DPO row (p.16, l.819) has N@10 - N@5 = .0066, below the single-target minimum (HR@10−HR@5)/log⁡211=.0241/3.46=.0070(\mathrm{HR@10}-\mathrm{HR@5})/\log_2 11 = .0241/3.46 = .0070 under any rounding (the 1/log⁡2(r+2)1/\log_2(r+2) convention is excluded because the row's N@5/HR@5 of .709 exceeds its ceiling of .631); it is the only such violation among the paper's HR/NDCG rows, and on its own I would read it as a transcription slip. These tables carry the headline comparison and three of the four evidence lines the abstract names ("controlled surface-only comparisons, direct SID-semantic diagnostics, multi-seed reward controls", l.023-025), plus the claims at l.406-415, l.423-426 and l.460-463, and the single-seed caveat (l.475-476) speaks to statistical power, not to why separately evaluated runs agree this closely. Each pattern could have a clerical cause. Per-run logs and per-user rank files (or checkpoints) for Tables 1, 6, 7, 9 and 11, a statement of how each Table 1 entry was obtained, the exact procedure behind the Table 9 intervals and a corrected Table 6 could resolve this; any entry that was not independently measured should be rerun or withdrawn.

Even at face value, the evidence does not cleanly separate the three components, and the gain is not calibrated against the closest prior work. The "most critical" claim for DCIM (l.397-401) rests on Table 3's -11.8% HR@5, but qiq_i is computed from DCIM descriptors (Eqs. 3-4; l.430-431), the row's reward is never stated despite the caption's "each row removes one component" (l.399-400), and its values sit below Table 7's matched no-QARM w/o-DCIM row on all four metrics (.0598 vs .0601 HR@5), which is hard to reconcile with that row keeping a QARM bonus; the -11.8% therefore most likely bundles DCIM with the loss of QARM. In the matched comparison, removing DCIM costs 6.4% HR@5 on Beauty (7.7% on Sports and Instruments): still the largest measured effect, but within run-to-run noise of CMSA (5.5%) and QARM (5.3%) on Beauty, the only dataset where all three are measured. QARM is the best-isolated component (three seeds and a shuffled control), whereas CMSA's benefit is shown only with DCIM present: Table 7's w/o-DCIM rows keep the caption text and the shared SID construction (l.398-399; D.1, l.853-857) yet sit within about 2% of the controlled MiniOneRec on HR@5 (-1.8% Beauty, +2.0% Sports, -0.2% Instruments), which, if MiniOneRec used the same reward (not stated), suggests that CMSA works mainly through the caption it feeds to DCIM and the judge (l.718). Table 4's separation of the two paths is also weak: Text-Only keeps DCIM "unchanged" (l.845) and may retain caption-mined descriptors, the single-path gains (+.0011 and +.0007 HR@5, no seeds reported) are 1.0-1.6 SDs of a single-run difference (Table 11 seed SD) before any RQ-VAE retraining variance, and their combination (+.0037) is twice their sum without comment. Externally, nine of the 14 Table 1 baselines (GRU4Rec, Caser, HGN, SASRec, BERT4Rec, FDSA, BIGRec, D3, S-DPO) have no reference entry, none has a stated configuration or tuning budget beyond MiniOneRec's backbone (C.1, l.804-809), and as published all of them use only IDs, attributes or text, while DeepInterestGR adds images (as captions and CLIP features) and LLM descriptors. The closest academic work (multimodal SID tokenization such as MQL4GRec, ICLR 2025; LLM knowledge augmentation such as KAR, RecSys 2024; LLM-mined e-commerce intentions in FolkScope, Findings of ACL 2023) is neither compared nor cited (Sec. 2 covers these lines only through OneRec-V2, GPR and Align3GR, l.131-133, and through HLLM, Yan et al. and RecGPT, l.136-138). The Surface-only and rewriting controls restate item text rather than add world knowledge, so "images plus LLM text" is answered only by variants of the authors' own pipeline, and Table 6's unexplained +23.9% to +33.5% gains, about 2.3 times Table 1's, add to the calibration concern. These points bear on contribution bullets 2 and 4 (l.111-113, l.117-119) and would keep me below acceptance on their own, but they are fixable: a CMSA × DCIM factorial under one stated recipe (QARM added within the DCIM cells) with three or more seeds including RQ-VAE seeds and paired per-user tests, a baseline-configuration table with citations, one multimodal-SID baseline given the same images, and a KAR-style factual-knowledge prompt at DCIM's budget.

Questions for authors
  1. Please share per-run logs and per-user rank files (Tables 1, 6, 7, 9, 11), explain Table 7's midpoints and Table 11's Seed 1 = rounded mean, and correct Table 6's Toys S-DPO row.
  2. How were Table 9's intervals computed (replicates, method, 95% or 90%), is the judge binary, and why is Table 1's Instruments/Beauty HR@5 ratio 0.90-0.92 for all 15 methods (CV 0.5%; Sports/Beauty: 2.4%)?
  3. Does descriptor or item text reach the generator (Fig. 2 vs. l.229-230)? Which rewards train the controlled MiniOneRec and Table 3's w/o-DCIM row (with which qq), and does Table 4's Text-Only drop caption-mined descriptors?
  4. How were the 13 other baselines configured and tuned, were α=0.5\alpha=0.5 and checkpoints chosen on validation data, and why are Table 6's gains about 2.3 times Table 1's?
  5. Could QARM's gain reflect popularity or metadata richness (qiq_i vs. target frequency, description length)? With 220 of 500 E.1 descriptors negative and no stated filtering (Eq. 14), how many are unsupported or contradictory?
  6. Reference fixes (cited -> correct): Li et al. (2025, tri-decoupled survey; no venue) -> Preprints.org preprint, doi:10.20944/preprints202512.0203.v1; Wang et al. (2025d, NEZHA) -> WWW 2026, pp. 8073-8082; Yan et al. (2025) -> WSDM 2026, pp. 798-807; Ye et al. (2025, "Align3gr") -> Align³GR, AAAI 2026, 40(19):16154-16162; Zeng et al. (2026) -> ICMR 2026, pp. 1260-1269; Zhang et al. (2025, GPR) -> KDD 2026, pp. 7477-7488, author list beginning with Jie Jiang; Zheng et al. (2024, LC-Rec) -> add Ming Chen between Wayne Xin Zhao and Ji-Rong Wen (ICDE 2024, pp. 1435-1448).
Reviewer_NGKGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents DeepInterestGR, a framework for enriching semantic IDs (SIDs) in multimodal generative recommendation. There are two main components in this paper's contribution. First, it enhances item features before quantization using multi-modal information which is done by combining image captions, projected visual embeddings, and LLM-extracted descriptions of plausible usage contexts. Second, it introduces QARM, a policy-learning reward that grants an additional descriptor-quality bonus only when the generated SID identifies the ground-truth target item. The proposed framework is evaluated via experiments on three primary Amazon categories, with additional category checks and controlled ablations. The extracted intent descriptors represent item-side usage contexts rather than personalized user intent.

Critical strengths

This work presents a useful approach for feature engineering for SID construction in generative recommendation. Enriching item representations before quantization is well motivated: information absent from the input representation cannot be explicitly preserved in the resulting SID semantics. Combining visual evidence with item usage contexts offers a practical way to improve these representations. The proposed method involves quality-dependent reward design. QARM incorporates descriptor reliability into policy optimization while preventing an incorrect recommendation from receiving the quality bonus. The strict no-QARM and shuffled-quality controls provide useful initial evidence about this design.

Critical weaknesses

Statistical support is incomplete for the central feature-engineering contribution. Although Table 11 reports three-seed QARM results on Beauty, the controlled Surface-only/DCIM comparison in Table 7 is explicitly a single-seed experiment. This comparison is particularly important because it tests whether extracting usage contexts provides benefits beyond matched surface-level rewriting. The HR@10 improvements over Surface-only are approximately 2.8%–3.7%, making uncertainty estimates important for assessing their robustness.

Since the variants also involve independently trained SID codebooks, the reported differences may reflect variability in both quantization and downstream learning. The anchor-level confidence intervals in the SID diagnostics do not capture this training variability. Repeating the central representation comparison across seeds would strengthen the evidence.

The multimodal fusion component is not specified sufficiently for reproduction. Appendix B.3 introduces a learnable visual projection and a fusion matrix before RQ-VAE quantization, but their training procedure remains unclear. In particular, the paper should specify the objectives used to train these transformations, whether they are optimized jointly with the RQ-VAE, which components are frozen, and how the modalities are normalized.

Referring to the inherited RQ-VAE configuration does not fully resolve these questions because the proposed method introduces additional transformations upstream of quantization. These details are necessary to understand how visual information is retained and how the fused representation is learned.

The connection between descriptor reliability and the proposed recommendation reward is insufficiently established. This is my main concern. A reliable item descriptor does not necessarily imply stronger user preference or that correctly recommending that item should receive a larger reward. An alternative interpretation is that descriptor reliability identifies training examples whose semantic representations are more useful for learning. That interpretation is plausible, but it requires clearer justification and supporting controls.

Under exact-target gating, the reward can be written as r(y)=r_{\mathrm{format}}(y) +\left(1+\alpha q_{i^}\right)\mathbf{1}[\hat i(y)=i^].

Thus, QARM directly adjusts the correct-answer reward according to a fixed target-item quality score. The advantage audit establishes that this adjustment survives group normalization in some groups, but does not establish why descriptor quality is the appropriate adjustment signal.

The shuffled-quality control supports the usefulness of the association between items and scores. However, it does not rule out explanations involving metadata completeness, popularity, or prediction difficulty. A full training comparison against a constant quality bonus and a tuned exact-match reward coefficient would help determine whether the gains require descriptor-quality information, rather than a more general adjustment of the reward balance.

Questions for authors

See critical weaknesses. Here are few additional questions:

  1. Can the authors provide multi-seed results for the matched Surface-only/DCIM comparison, including variability from SID construction?

  2. What objectives and gradient paths train the visual projection and fusion matrices in Equations (15)–(17)? Are these transformations trained jointly with the RQ-VAE?

  3. Is QARM intended to express recommendation utility or to allocate learning emphasis toward examples with more reliable semantic representations? What evidence supports this choice?

  4. How does QARM compare with a constant quality bonus, qi=qˉq_i=\bar q, or a tuned exact-match reward coefficient? Table 12 includes a Uniform advantage audit, but does not report recommendation performance after training with that control.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.