acceptodds
Under review as a conference paper at ICLR 2027

CONFIDENCE-GATED DISTILLATION OF LANGUAGE MODEL SEMANTICS FOR COLD-START RECOMMENDATION

Abstract

Collaborative filtering fails on items with no interaction history, and a large language model is an inexpensive source of the side information that could fill the gap. The open question is not whether such side information helps but which part of it does, and existing evaluations answer neither: the protocols labelled cold-start are usually leave-one-out splits in which the held-out item is still trained, and the controls that would isolate the language model’s contribution are absent. We present DualDistill, which distils an LLM-derived item geometry into a graph collaborative backbone through four auxiliary objectives and combines the two branches with a dual confidence gate that is made identifiable by a fused ranking loss. We evaluate it under a strict item cold-start protocol in which held-out items have exactly zero training interactions, alongside standard leave-one-out, on three catalogs spanning two orders of magnitude in density. DualDistill reaches 0.1203 nDCG@20 on ML-1M against 0.0540 for the strongest baseline (123%), and under strict cold start it reaches 0.0796 against 0.0619 (29%), with consistent gains on Amazon-Beauty and Book-Crossing. A sixteen-way ablation over 5 seeds separates the components: replacing the semantic neighbourhood with a random graph is the single most damaging change (-0.0144 nDCG@20), and freezing the gates costs -0.0094. Inspecting the trained gate shows it does not degenerate — its output spans 0.25 to 1.00 across the catalog, so it is selecting rather than rescaling. We also report where the method does not win: on Book-Crossing a tuned item-based nearest-neighbour baseline is within 8% of the full model, and substituting raw catalog metadata for the LLM text costs only -0.0035 on ML-1M, which bounds how much of the gain is attributable to world knowledge rather than to text of any kind.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Cold-start Recommendation, Large Language Models, Knowledge Distillation, Graph Collaborative Filtering, Semantic Representation, Confidence Gating, Recommender Systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_ZzvcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes DualDistill for item cold-start recommendation. It transfers text-derived semantic information into a collaborative filtering backbone through multiple distillation objectives and combines two scoring branches using confidence gates. The experiments evaluate the method under leave-one-out and strict item cold-start protocols, supported by baseline comparisons and ablation studies.

Critical strengths
  1. The strict item cold-start protocol removes held-out interactions from the training graph, providing a clearer evaluation of cold-start performance.
  2. The comprehensive ablation studies and metadata controls help assess the contributions of individual components and semantic information sources.
Critical weaknesses
  1. Held-out items may still receive embedding updates through semantic distillation during training. I am concerned that this limits the evidence for generalization to entirely unseen items.
  2. The fused ranking loss makes the confidence gates trainable, but this alone does not establish parameter identifiability. I find the additional benefit of the dual-gate mechanism over simpler fusion strategies insufficiently demonstrated.
Questions for authors
  1. Are the embeddings of held-out items updated through auxiliary objectives during training? I would like to understand how the method performs when these embeddings are excluded from training.
  2. How is the identifiability of the gate parameters established beyond the presence of gradients? I would appreciate a comparison with simpler learnable fusion strategies to clarify the necessity of the proposed design.
Reviewer_HJzEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes DualDistill, a framework for transferring LLM-derived item semantics into graph collaborative filtering for cold-start recommendation. The method aims to address the lack of interaction-derived signals for cold-start items and examine the sources of improvement from semantic information. DualDistill uses four auxiliary objectives to transfer semantic geometry into LightGCN and combines collaborative and semantic scores through a dual confidence gate. Experiments on three datasets under leave-one-out and strict item cold-start protocols show improvements over existing baselines, while raw metadata recovers most of the benefits of LLM-generated descriptions.

Critical strengths
  1. The paper evaluates both leave-one-out and strict item cold-start protocols, which helps distinguish sparse user history from zero-interaction item cold start.
  2. The paper includes raw-metadata controls and component ablations to analyze the contributions of semantic information and confidence gates.
Critical weaknesses
  1. Unclear research objective and disconnected contributions. The paper lacks a coherent research objective. The introduction mainly discusses evaluation issues, including strict cold-start protocols, metadata controls, and gate behavior. However, these issues do not naturally motivate the proposed DualDistill framework. If the paper aims to provide an evaluation study, its findings are mostly limited to the proposed model and do not offer sufficient general insights into existing recommendation methods. If it aims to propose a new method, it fails to identify a clear limitation of existing approaches or explain why the proposed distillation framework is necessary. The evaluation motivation and methodological contribution appear disconnected, making the main contribution of the paper unclear.
  2. Limited methodological novelty and insufficient cold-start-specific motivation. The proposed method mainly combines existing techniques, including InfoNCE, relational distillation, KL distillation, BPR, and confidence-based score fusion. The paper does not clearly explain why these objectives are needed together or what limitations of existing semantic alignment methods they address. Moreover, the collaborative confidence gate depends only on the user's interaction count, rather than the interaction history of the target item. For an active user, the model may still assign a high weight to the collaborative score even when the target item has no training interactions. This design does not appear to directly address the main challenge of item cold-start recommendation.
Questions for authors
  1. What specific limitations of existing semantic alignment and cold-start recommendation methods motivate the four distillation objectives and the dual confidence gate?
  2. If the main contribution is improved evaluation, what general findings does the paper provide beyond the performance analysis of DualDistill?
Reviewer_ytvtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DualDistill, a framework for investigating the contributions of LLM-derived semantic information to cold-start recommendation. The method uses an LLM to generate item descriptions and transfers their semantic geometry into LightGCN through four auxiliary objectives. It also combines collaborative and semantic scores through a dual confidence gate. Experiments on three datasets under leave-one-out and strict item cold-start protocols show improvements over existing baselines, while a raw-metadata control shows that most of the semantic benefits can also be obtained from existing catalog text.

Critical strengths
  1. The paper evaluates strict item cold start with zero training interactions for held-out items and includes cold-start-specific baselines, providing a more direct evaluation of side-information transfer.
  2. The raw-metadata control provides a useful finding that existing catalog text can recover most of the benefits of LLM-generated descriptions.
Critical weaknesses
  1. Overemphasized motivation in the abstract and introduction. The abstract and introduction strongly emphasize that existing evaluations cannot clearly identify the sources of improvement from LLM-based semantic information. However, the experiments mainly focus on comparing DualDistill with existing baselines and analyzing its own components. Although the raw-metadata control provides some useful findings, the analysis is limited to the proposed framework and does not sufficiently investigate this issue in existing LLM-based recommendation methods. Therefore, the paper does not fully address the problem emphasized in the abstract and introduction, making its main contribution unclear.
  2. Insufficient details about the LLM and unclear contribution of LLM-generated information. The paper does not specify which LLM is used to generate item descriptions or provide sufficient details about the generation process. This makes the experiments difficult to reproduce. Moreover, the raw-metadata control achieves performance close to the LLM-generated descriptions, raising questions about the actual benefit of using an LLM. Another concern is potential overlap between the LLM's pretraining data and the evaluation datasets, especially for public datasets such as MovieLens-1M. Although the authors discuss the limitations of leave-one-out evaluation, they do not consider whether the LLM may have seen relevant information during pretraining. Without details about the LLM and its training data, it is difficult to assess the source and reliability of the semantic information used in the experiments.
Questions for authors
  1. How do the proposed method and experiments answer the main research question about the sources of improvement from LLM-based semantic information?
  2. Which LLM and prompts are used to generate item descriptions, and how do the authors assess potential overlap between its pretraining data and the evaluation datasets?
Reviewer_QViLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

this paper looks at how language-model-based item information can help recommend new items with no interaction history. DualDistill uses several distillation losses to transfer semantic information into a LightGCN recommender. It then combines the semantic and collaborative scores using gates based on the user’s history and the item’s semantic coherence

the authors test the method on three datasets using both standard leave-one-out evaluation and a stricter item cold-start setting. they compare it with several collaborative and cold-start baselines and include ablations and a raw-metadata control. DualDistill gets the best average results, although the gains over strong cold-start baselines are fairly small, and most of the semantic benefit can already be obtained from raw item metadata

Critical strengths

the evaluation is carefully designed. the paper clearly separates standard leave-one-out evaluation from true zero-interaction item cold start. This makes the results more relevant to the problem the paper is trying to solve

the paper includes useful controls and honest analysis. the raw-metadata comparison, gate analysis, and detailed ablations help show where the gains come from

Critical weaknesses

the cold-start setup is not fully clean. it is unclear whether held-out cold items are still used in the auxiliary losses or sampled as training negatives. the semantic user profile also includes the current positive item during training, but not at test time. these issues could create extra training signals or a train-test mismatch, directly affecting the main cold-start results

the confidence gate is not strongly validated. semantic neighborhood density is treated as confidence, but the paper does not show that it predicts semantic reliability. Moreover, missing items share a mean-vector fallback and may therefore receive artificially high confidence. the item gate is also close to constant for most items and has only a small ablation effect, weakening the paper’s main methodological contribution

Questions for authors

are strict-cold items excluded from all auxiliary losses and negative-sampling pools? if not, should this setting be described as transductive cold start?

when computing the semantic user profile during training, is the current positive item excluded?

do mean-vector fallback items participate in the semantic KNN graph? If so, how do you prevent identical fallback vectors from receiving artificially high confidence?

do Heater, GAR, and PAD-CLRec use exactly the same semantic features and tuning budget as DualDistill?

Reviewer_deUxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DualDistill for item cold-start recommendation. An LLM generates a structured description for each item, which is encoded by a frozen sentence encoder to build a semantic K-NN graph and a per-item coherence score. This semantic geometry is distilled into a LightGCN backbone through four auxiliary losses (pointwise, relational, distributional, reconstruction). The collaborative and semantic scores are then fused by a dual confidence gate, with an extra fused ranking loss so that the gate parameters actually get trained. The method is evaluated under leave-one-out and a strict item cold-start protocol on three datasets, with a raw-metadata control and a sixteen-variant component analysis.

Critical strengths
  1. The raw-metadata control is a useful design. It shows that most of the benefit comes from text-derived item geometry rather than from LLM-generated descriptions, which is a practical finding for work on LLM-enhanced recommendation.
  2. The dual gate is simple and interpretable. It decides how much to trust the semantic score based on user history and item coherence, and the trained gate does assign lower weight to low-coherence items instead of collapsing to a constant.
Critical weaknesses
  1. The large leave-one-out gains do not seem to come mainly from the semantic signal. On ML-1M, the random semantic graph variant still reaches 0.1059, about 70% above XSimGCL and roughly twice LightGCN. It is also unclear how the baselines were tuned, since Table 7 only lists DualDistill’s settings. This makes the headline numbers hard to interpret.
  2. Under strict cold start, which is the setting the method targets, the gain over PAD-CLRec is only 4–6%, about one or two standard deviations. In addition, none of the LLM-based methods discussed in Section 2 are compared. Given that the components are mostly standard distillation losses plus a simple sigmoid gate, the overall contribution feels limited.
Questions for authors
  1. Which LLM and prompt are used to generate the item descriptions? And what content features do the cold-start baselines receive?
  2. In the strict cold-start setting, what is the candidate set for ranking? LightGCN and ItemKNN have no information about the held-out items, yet they get 0.0349 and 0.0294 on ML-1M. What would a random ranker get?
  3. The ML-1M few-shot threshold is 20, but every ML-1M user has at least 20 ratings. How many users are actually in this slice?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.