acceptodds
Under review as a conference paper at ICLR 2027

Progressive Learning from Content to Relevance to Personalization for CTR Prediction

Abstract

Multimodal click-through rate (CTR) prediction is central to search systems, where ranking depends on item content, query relevance, and user preference. Existing approaches incorporate content through dense embeddings or coarse discrete IDs, yet effectively integrating fine-grained heterogeneous content with ID-based predictors remains challenging. We propose FineSID, a progressive learning framework that unifies content tokenization, query–item semantic pretraining, and personalized post-training via a shared fine-grained discrete representation. Each stage optimizes an explicit objective while preserving token-level access to content. First, a pretrained multi-scale visual tokenizer and text tokenizers convert heterogeneous content into fine-grained Semantic IDs (SIDs), explicitly aligning them with the ID features. Second, full-parameter semantic pretraining under query–item relevance supervision establishes a robust matching prior, jointly optimizing the interactions of multimodal SIDs and conventional fields within a shared predictor. Third, this initialized predictor is directly fine-tuned with user click supervision to capture implicit personalized preferences. Extensive offline and online evaluations demonstrate substantial gains in both standard CTR prediction and core business metrics. Furthermore, FineSID improves within-product creative discrimination, demonstrating its semantic generalization beyond conventional ID memorization.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Learning, Semantic IDs, Click-Through Rate Prediction, Search, Fine-Grained Representation Learning, Content Pre-training

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8RLUOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates multimodal click-through rate (CTR) prediction and proposes a unified progressive learning framework that effectively integrates fine-grained heterogeneous content with ID-based predictors.

Critical strengths

This paper focuses on research questions that have practical significance.

Critical weaknesses

The FineSID proposed in the article is more like a combination of engineering methods, lacking innovation. Why adopt the current three-stage strategy?

The experiment was not sufficient. The article was only validated on the Industrial Taobao dataset and a publicly available MMCTR dataset. The ablation experiments in the article are not sufficient, lacking ablation on the selection of various stage strategies and hyperparameters.

This paper is difficult to understand.

Questions for authors

Suggest adding some visual explanation of the core issues, core methods, key results, etc., of the paper.

Reviewer_ss83OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FineSID, a three-stage framework for multimodal CTR prediction that progressively learns from content to relevance to personalization. Stage I discretizes multimodal content into fine-grained Semantic IDs (SIDs), preserving spatial structure (multi-scale image codes) and field-reserved text tokens to align with ID-based predictors. Stage II pretrains the shared predictor on query–item relevance using LLM-distilled soft labels, and Stage III fine-tunes the same predictor end-to-end on CTR logs. On large-scale Taobao data and a public MMCTR benchmark, FineSID shows consistent gains in AUC/GAUC (especially for cold items) and improves within-item creative discrimination (NSCTR), with positive online A/B impacts on clicks and GMV.

Critical strengths
  • Leveraging multi-scale, spatially indexed visual SIDs (from a reconstruction-trained tokenizer) to preserve fine-grained content, rather than compressing to a single global code, is a thoughtful and practically motivated design.
  • The progressive learning pipeline (content tokenization → relevance pretraining → CTR finetuning) cleanly bridges semantic alignment and personalization without complex multitask coupling at serving time.
Critical weaknesses
  • The writing and architecture diagram give a strong impression of AI-generated content.
  • Relevance pretraining relies on an LLM teacher (Qwen3.5) with “million-scale expert annotations,” but the paper gives limited detail about the teacher dataset composition, quality control, and domain coverage relative to the CTR distribution.
  • Ordered fusion and latent-slot attention choices (HiFormer x2 + shared MLP) are reused from prior work; the novelty primarily lies in the discrete, fine-grained interface and staged training rather than a new interaction architecture.
  • More detail on the exact relevance-head architecture and its mapping to the shared representation would improve reproducibility.
Questions for authors

NA

Reviewer_Soe1OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This manuscript proposes FineSID, a three-stage progressive framework for multimodal CTR prediction on Taobao search, structured as an explicit Item → Item–Query → Item–Query–User progression across 3 stages:

Stage 1\ discretizes multimodal content into fine-grained Semantic IDs using image, text, and native categorical features embedded via independent embedding methods and consolidated using a HiFormer and an MLP.

Stage 2\ performs full-parameter semantic pretraining of the entire shared predictor under soft relevance supervision from a fine-tuned Qwen3.5.

Stage 3\ discards the relevance head and fine-tunes the entire model end-to-end on click labels.

The central claim is that fine-grained token-level SIDs (over global RQ-VAE-based SIDs) allow the predictor to learn explicit cross-modal alignment instead of memorizing global embeddings.

What paper also adds is the missing generalization piece with producing SIDs: fine-grained content tokens plus relevance pretraining, so the model can reason about what an item actually contains and whether it matches the query, even when it has never seen that item before.

The evaluation is comprehensive: offline in-house Taobao results and the public MMCTR benchmark show significant gains, which are further strengthened by an online A/B test. The ablation study also demonstrates the benefits of the proposed approach.

Critical strengths

1\ Comprehensive experimentation and validation chain from offline to online.

2\ The within-product creative discrimination (NSCTR) evaluation is very insightful. By fixing the item ID authors test the memorization-vs-generalization claim in a way AUC cannot. The 0.75% gain is the paper's most convincing evidence that fine-grained tokenization matters.

Critical weaknesses

1\ The novelty is somewhat limited, as existing works such as PLUM or COINS have already successfully employed fine-grained SID tokens in production. He, R., et. al, PLUM: Adapting Pre-trained Language Models for Industrial-scale Generative Recommendations. Zhao, Q., et. al, COINS: SemantiC Ids Enhanced COLd Item RepresentatioN for Click-through Rate Prediction in E-commerce Search.

2\ The system is not fully end-to-end across all stages, which leaves open questions regarding how much potential performance is left on the table.

3\ The authors do not provide a comparison with a traditional multi-stage system to clearly demonstrate the benefits of moving toward an end-to-end architecture, which is positioned as the main contribution of the paper.

Questions for authors

1\ Please better position the innovation compared to prior works like PLUM and COINS.

2\ Additional experiments with multi-stage system and a complete end-to-end system would be greatly appreciated to understand the trade-offs

3\ How much of Stage II's value comes from the Qwen3.5 teacher (fine-tuned on million-scale expert annotations) versus the progressive framework itself? What happens with hard labels, a smaller teacher, or fewer expert annotations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.