acceptodds
Under review as a conference paper at ICLR 2027

Learn Before You Fuse: Preserving Complementarity in Generative Recommendation

Abstract

Semantic IDs (SIDs) compress continuous item representations into discrete codes for generative recommendation. A single SID, however, preserves only one coarse organization of the item space, while quantization removes fine-grained item information. Different representation-learning objectives can induce different geometries over the same item space, suggesting that independently constructed SID views can capture complementary relational structures, while dense representations retain item-level details obscured by quantization. Yet simply combining such representations does not automatically yield complementary predictors. We identify a recurring failure mode: models exploit readily accessible signals before alternative predictive structures are sufficiently established, leaving complementary pathways under-trained. To address this, we propose Learn-Before-Fuse (LBF), which first establishes a complementary predictor before introducing readily exploitable signals, and instantiates it in CVGR2, a Complementary-View Generative Recommendation and Re-Ranking framework. CVGR2 models complementary SID views through interleaved cross-view interaction while preserving view-specific prediction spaces. Following LBF, it establishes the SID predictor before adding dense refinement; its reranker likewise learns history-conditioned item-level relevance before incorporating multi-view SID-based matching evidence. Across diverse item dense representations, LBF consistently improves over joint-learning in both retriever and reranker, with clear gain isolated in our extensive ablations. Across ten public datasets, CVGR2 improves mean Recall@10 by 23.06% over the strongest baseline. Code is available at https://anonymous.4open.science/r/MSGR2-40EF.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Generative Recommendation, Semantic IDs, Item Tokenization, Sequential Recommendation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_KvNGOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LBF, a training-order principle for combining complementary information sources in SID–based generative recommendation. The authors observe that readily exploitable signals can dominate early optimization and prevent complementary but harder-to-learn predictors from developing their own structure. They characterize this phenomenon as Training-time Optimization Accessibility (TOA). LBF proposes: establish the lower-TOA complementary predictor first, then introduce higher-TOA signals for refinement, and instantiates this as CVGR$^{2}$.

Critical strengths

S1: The central idea that optimization can prevent complementary pathways from fully emerging if one pathway is too easy early, is plausible.

S2: CVGR$^2$ implements staged learning on both the generation side and the ranking side: on the generation side, it performs dense refinement after the SID predictor; on the ranking side, it fuses history-conditioned relevance before SID-matched evidence.

Critical weaknesses

W1: The idea that training simpler or harder-to-learn tasks before easier ones is well-established in curriculum learning, multi-task learning. Scheduling auxiliary losses or features sequentially is not new. While the application to SID-based recommendation is domain-specific, the fundamental contribution is incremental.

W2: The paper does not provide a specific definition or measurement standard for TOA. At line 1101, it also acknowledges that Eq. (26) is “not as a direct numerical estimator of TOA.” In addition, it does not show that TOA mismatch can explain the gains under different settings. Therefore, it is difficult to determine whether the performance improvement truly comes from the TOA mechanism, rather than from the general benefits of staged training or regularization.

W3: The paper uses GAOQ from ReSID as the quantizer, FAMAE from ReSID to learn item representations, and Node2Vec to learn graph-based representation. However, it does not provide reasons for why these methods are chosen. Also, under other quantizers and representation-learning methods, how does the CVGR$^{2}$ perform?

Questions for authors

see the weaknesses.

2 comments

We appreciate the reviewer's thoughtful comments. Below, we provide detailed responses to the concerns, supported by additional experiments.

W1: How Does LBF Differ from Existing Staged Training Methods?

We recognize that staged training and the sequential introduction of auxiliary signals have been explored in prior work. Traditional curriculum learning often orders training examples by difficulty, while staged multi-task learning commonly schedules different tasks. However, our focus is on a specific asymmetry between input pathways within the same prediction task: one pathway may already contain readily exploitable predictive information, while a complementary predictor must still be learned from scratch.

LBF therefore establishes the complementary predictor first, then introduces the readily exploitable signal for refinement, differing from the common easy-to-hard curriculum. In this generative recommendation setting, the representative balancing strategies do not exploit the complementary pathways as effectively as LBF. We compare LBF with several multimodal balancing strategies adapted to our architecture on BP. For reranking, all methods use the same candidate sets generated by the same model.

Training strategyGenerator R@10 (%)Reranker R@10 (%)
Early Fusion4.17154.7961
Dense-path Dropout4.44455.0598
OGM-GE-adapted [1]4.27814.7701
Stochastic Path Training [2]4.20025.0078
UMT-adapted [3]4.15555.0451
LBF (ours)4.71495.1350

We implemented the strategies as follows:

  • Dense-path Dropout: Randomly disable the supplementary pathway during training (dense in generation; evidence in reranking), while using both pathways at inference.
  • OGM-GE-adapted [1]: Estimate pathway contributions from correct-target probabilities with each pathway used alone, suppress gradients of dominant-path-specific parameters, and add Gaussian noise; shared prediction parameters are not directly modulated.
  • Stochastic Path Training [2]: Inspired by MLA's alternating unimodal adaptation, randomly choose base-only, supplementary-only, or joint-input training with equal probability under the same task loss, and use both inputs at inference.
  • UMT-adapted [3]: Train separate SID-only/dense-only teachers for generation and history-only/evidence-only teachers for reranking. Freeze the teachers, then train a joint-input student from scratch using the task loss together with feature-level MSE losses against the corresponding teachers.

LBF achieves the best performance in both stages among the tested strategies. Our contribution is not the general idea of staged training itself, but addressing how to better leverage complementary information in generative recommendation. We propose the CVGR² framework to integrate multiple predictive signals and a simple yet effective LBF strategy to address the resulting fusion training challenges, achieving strong overall performance. We will clarify this contribution and add the related work and experiments to the paper.

[1] Balanced Multimodal Learning via On-the-Fly Gradient Modulation.

[2] Multimodal Representation Learning by Alternating Unimodal Adaptation.

[3] Improving Multi-Modal Learning with Uni-Modal Teachers.

W2: How Is TOA Defined, and Does It Explain LBF's Gains?

As defined in Section 2.2, Training-time Optimization Accessibility (TOA) describes how readily a model can exploit a given input pathway to reduce training loss under its current optimization state. We treat TOA as a qualitative comparison between pathways rather than a calibrated numerical measure. To examine this property empirically, we independently train SID-only and Graph Dense-only predictors from scratch under the same training configuration to predict the same target SIDs. The table reports the sum of mean token-level negative log-likelihoods (NLLs) on BP at selected epochs.

Historical inputEpoch 10Epoch 50Epoch 100
SID-only8.74248.40878.2766
Dense-only8.68078.25958.0182

Dense-only yields lower NLL at all three checkpoints, indicating that graph-based dense inputs are more readily used to fit the prediction target under this setup.

To distinguish the benefits of dense refinement from those of continued training, we conducted additional controls on BP. Each strategy continues from its validation-best Stage-1 checkpoint using the same Stage-2 optimization settings as LBF. Early retains dense inputs, SID-only remains unchanged, and LBF introduces graph-based dense refinement.

Training strategyStage 1 R@10 (%)Stage 2 R@10 (%)
Early → Early4.17154.0650
SID-only → SID-only4.64964.6656
SID-only → Graph LBF4.64964.7149

Continuation alone provides little or no benefit, whereas LBF achieves a clear improvement.

For reranking, we also separated delayed evidence introduction from freezing, using two-stage training under the corresponding configurations:

Evidence introductionStage-2 base rankerTest R@10 (%)
EarlyUnfrozen4.7961
EarlyFrozen4.8121
DelayedUnfrozen5.0831
DelayedFrozen5.1350

Early continues from a history-plus-evidence model, whereas Delayed starts from a history-only model and then introduces decoder evidence. Delayed introduction yields the larger improvement in either freezing condition, suggesting that fusion timing is the main source of the observed difference.

We will clarify TOA's definition, present it as an empirically supported explanation of the training behavior, and incorporate these experiments into the revision.

W3: Does CVGR² Generalize to Other Representations and Quantizers?

Our goal is not to propose a new representation learner or quantizer, but to investigate how different item relationships and their predictive signals can be effectively combined. In the original configuration, FAMAE, Sentence-T5 text embeddings, and Node2Vec capture behavioral/field information, semantic content, and graph-based collaborative relations, respectively. GAOQ provides a unified, reproducible procedure for constructing multi-view SIDs.

To test dependence on these components, we replaced the representation sources and quantizer in additional BP experiments:

SID representations / quantizerSID-only G1LBF G2History-only R1Evidence-added R2
FAMAE+Text+Graph / GAOQ4.64964.71494.41265.1350
FAMAE+Text+BERT4Rec / GAOQ4.72684.79084.35205.1337
BERT4Rec+Text+SASRec / GAOQ4.65294.67024.33475.0777
BERT4Rec+Graph+SASRec / GAOQ4.49914.52584.36594.9972
FAMAE+Text+Graph / RQKMeans4.19284.22014.15224.7029

For BERT4Rec and SASRec, we obtain embeddings through training and concatenate item-ID and field embeddings to form item representations. RQKMeans uses three residual KMeans levels with 256 centroids per level, plus a deduplication token. We retrain the generator for each SID configuration.

Across all tested configurations, G2 outperforms G1 and R2 outperforms R1. These results show that CVGR² accommodates different SID representation sources and quantization schemes while retaining the predictive benefits of adding complementary information. We will include the component-selection rationale, implementation details, and replacement results in the paper.

Reviewer_aDtGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to combine complementary information in generative recommendation. CVGR uses separate behavioral, textual, and graph-based SIDs, exchanges information through an interleaved temporal-view encoder, and adds dense representations and a history-aware reranker. The authors also propose Learn-Before-Fuse (LBF), which trains one prediction pathway before introducing signals that are easier to exploit. Across ten Amazon Reviews 2023 categories, the full system has superior performance.

Critical strengths
  • The distinction between having complementary information and actually learning to use it is useful.
  • The multi-view generator and history-aware reranker have reasonable, complementary roles.
  • The overall results are promising, with consistent generator improvements over ReSID across ten datasets.
Critical weaknesses
  • The motivation overlaps strongly with existing work on learning imbalance, but the relevant baselines are missing. Easier pathways dominating training is already studied in multimodal learning. It is unclear what makes this setting different and why LBF is preferable to existing balancing methods. Comparing mainly against Early and Joint Fusion does not answer this. This is my main concern about the contribution.
  • The evidence shows dense reliance, but does not clearly establish insufficient SID learning. Dense-Off performance drops can also reflect co-adaptation and the input change caused by removing a pathway. The learning curves and representation norms do not resolve this ambiguity.
  • The value of dense refinement itself is less clear. In Table 3, LBF substantially improves over Early fusion, but the gains over SID-only are small for text and graph representations, while FAMAE remains worse. This supports avoiding harmful fusion more clearly than it supports the claimed benefit of adding complementary dense information.
Questions for authors
  • My main concern is the gap to existing work. Prior work [1–3] already addresses closely related problems, including faster-learning modalities dominating training and leaving other modalities under-optimized. What is different here that calls for LBF? EAGER and LIGER are relevant recommendation baselines, but they do not answer this question. Without a comparison to a reasonably adapted balancing method, or a concrete explanation of why it does not apply, I am not convinced of LBF’s added value over this line of work.
  • I am not sure Dense-Off shows what you claim it shows. Removing dense inputs clearly hurts Early models, but this does not necessarily mean the SID pathway failed to learn. The shared network may simply have adapted to using both inputs. Also, dense inputs still feed the same SID generation task. Is the claim about weak prediction from discrete inputs alone, or an under-learned SID decoder? These are different. The current diagnostics do not clearly distinguish them, and they do not establish gradient conflict either.
  • What useful information does the dense branch add once the SID predictor already works? In Table 3, SID-only gets 4.6496% Recall@10, compared with 4.6895% for text and 4.7149% for graph under LBF. FAMAE stays below SID-only. You acknowledge this result, but it leaves me unsure how much the dense branch contributes to the motivation. LBF clearly helps relative to Early fusion. The less clear part is why adding this branch is worthwhile in the first place.

[1] What Makes Training Multi-Modal Classification Networks Hard? [2] Characterizing and Overcoming the Greedy Nature of Learning in Multi-modal Deep Neural Networks [3] Balanced Multimodal Learning via On-the-Fly Gradient Modulation

2 comments

We thank the reviewer for the constructive feedback. We address the concerns and present additional experimental results below.

W1/Q1: How Does LBF Differ from Existing Balancing Methods?

We acknowledge that LBF shares some underlying motivations with prior work on multimodal learning imbalance, and appreciate the suggestion to compare it with these methods. Our focus, however, is on a specific asymmetry: one input pathway may already encode information learned from the training data that is closely aligned with the prediction target, while a complementary predictive capability must still be learned from scratch.

In generation, graph-based dense representations already capture collaborative information, whereas SID prediction from discrete histories must still be learned. In reranking, SID-based matching evidence is readily available from the trained generator, whereas history-conditioned item-level relevance must be learned. LBF addresses this asymmetry by establishing the complementary predictor before introducing the readily exploitable signal for refinement. However, existing representative balancing strategies do not adequately address this pronounced asymmetry.

To evaluate this, we compare LBF with several balancing strategies adapted to our architecture on BP. For reranking, all methods rerank the same candidate sets generated by the same model. The results are as follows:

Training strategyGenerator R@10 (%)Reranker R@10 (%)
Early Fusion (ours)4.17154.7961
Dense-path Dropout4.44455.0598
OGM-GE-adapted [3]4.27814.7701
Stochastic Path Training [4]4.20025.0078
UMT-adapted [5]4.15555.0451
LBF (ours)4.71495.1350

LBF primarily controls when an input pathway is introduced and can be flexibly applied across architectures. In contrast, existing balancing methods often rely on specific architectures or training procedures and thus require adaptation. Since the original training procedures of [1,2] are difficult to directly apply to our architecture, we adapt OGM-GE [3] and implement additional strategies inspired by [4,5], as detailed below:

  • Dense-path Dropout: Randomly disable the supplementary pathway (dense representations in generation; decoder evidence in reranking) during training, and use both pathways at inference.
  • OGM-GE-adapted [3]: Estimate pathway contributions from the correct-target probabilities under each single-path input, suppress gradients of parameters specific to the dominant pathway, and inject Gaussian noise. Shared prediction parameters are not directly modulated.
  • Stochastic Path Training [4]: Inspired by MLA's alternating unimodal adaptation, sample base-only, supplementary-only, and joint-input training with equal probability under the same task loss; use both inputs at inference.
  • UMT-adapted [5]: Train SID-only and dense-only teachers for generation, and history-only and evidence-only teachers for reranking. Freeze the teachers, then train a joint-input student from scratch with the task loss and feature-level MSE distillation losses against the respective teachers.

LBF achieves the highest R@10 in both stages among the tested strategies. These results support establishing the complementary predictor before incorporating readily available signals in our setting. We will clarify the connection to prior work and add the adaptation details and comparisons to the paper.

References [1]–[3] are the three papers cited by the reviewer.

[4] Multimodal Representation Learning by Alternating Unimodal Adaptation.

[5] Improving Multi-Modal Learning with Uni-Modal Teachers.

W2/Q2: Does Dense Reliance Indicate Insufficient SID Learning?

We thank the reviewer for the helpful comment. We acknowledge that Dense-Off alone cannot distinguish insufficient learning from co-adaptation. We therefore conducted additional input removal and shuffling experiments to further investigate pathway reliance.

For removal, we zero out either the dense residuals or the historical SID embeddings at the model input. For shuffling, we replace one pathway's entire input history with that of another user of the same history length, while keeping the other pathway unchanged. If shuffling and removal yield similar results, it suggests that misleading information from other users has little additional impact on the predictions, which indicate limited reliance on that input pathway.

StrategyFullDense removedSID removedDense shuffledSID shuffled
Early4.17151.84113.66811.54343.6009
LBF4.71494.60033.04424.27341.4196

Early fusion suffers a large drop when dense inputs are removed, while removing or shuffling historical SIDs has a smaller and similar effect. This suggests substantially greater reliance on dense inputs and weaker use of the SID history. In contrast, LBF retains strong performance without dense inputs, while removing SIDs still leaves some predictive capability from the dense pathway. Shuffling either pathway also degrades LBF, particularly when historical SIDs are shuffled, indicating that both sources contribute to its predictions. We will include these experiments in the paper.

W3/Q3: Does Dense Refinement Add Value Beyond SID-Only?

Dense refinement is intended to recover fine-grained item information not fully preserved by quantization. For example, textual dense representations provide richer item semantics, while graph-based dense representations retain continuous collaborative relationships.

To further assess their incremental value, we conducted four-seed paired experiments on all ten datasets. The table reports mean R@10 (%) ± sample standard deviation across four runs.

DatasetSID-onlyLBF
BP4.6423 ± 0.01214.7169 ± 0.0157
BPC3.8289 ± 0.00243.9231 ± 0.0035
ACS5.2941 ± 0.01145.3746 ± 0.0187
VG9.8620 ± 0.01129.9547 ± 0.0216
MI6.8415 ± 0.03876.9507 ± 0.0396
IS5.4043 ± 0.02475.4308 ± 0.0458
SO3.4954 ± 0.00663.5840 ± 0.0097
TG4.2877 ± 0.01544.5474 ± 0.0119
HH3.3093 ± 0.04233.3862 ± 0.0588
BK8.3574 ± 0.04378.6551 ± 0.0590

LBF has a higher mean R@10 than SID-only on all ten datasets, supporting a modest but consistent positive average contribution from an appropriate dense representation. We will add these multi-seed, cross-dataset results and the corresponding statistical details.

Comments on the paper

2 comments

We thank the reviewer for the detailed suggestions. We provide further clarifications and new experimental evidence to address the key concerns.

W1/Q1: Are Generator Training Budgets and Optimization Settings Fair?

We agree that the original comparison did not fully isolate the effect of fusion timing from other training factors. In our original experiments, Early Fusion and SID-only used the same Stage-1 training settings as LBF, and both triggered validation-based early stopping well before the maximum epoch limit. In Generator Stage 2 (G2), the main network remains trainable, while the pretrained dense item representations are frozen. To ensure a fairer comparison, we further conducted two-stage training comparisons on BP, using the same stage-wise training settings across all strategies.

Training strategyStage 1 R@10 (%)Stage 2 R@10 (%)
Early → Early4.17154.0650
SID-only → SID-only4.64964.6656
SID-only → Graph LBF4.64964.7149

Neither continuing Early Fusion nor further training SID-only substantially improves performance. Adding dense refinement after SID-only training yields the best result under these controls.

W1/Q2: Delayed Fusion or Freezing: Which Drives the Reranking Gains?

To distinguish the contributions of fusion timing and freezing, we conducted a 2 × 2 ablation on BP, crossing evidence introduction (Early vs. Delayed) with the Stage-2 base-ranker update policy (Unfrozen vs. Frozen).

Evidence introductionStage-2 base rankerTest R@10 (%)
EarlyUnfrozen4.7961
EarlyFrozen4.8121
DelayedUnfrozen5.0831
DelayedFrozen5.1350

In Stage 2, Early continues from the validation-best history-plus-evidence model, while Delayed starts from the validation-best history-only model and introduces the evidence pathway. Frozen keeps the history backbone and base scorer fixed, updating only the evidence branch and adapter; Unfrozen updates all trainable parameters.

Both factors improve R@10 in this experiment, but delayed introduction contributes the larger gain under either freezing policy. We will add this ablation to the paper.

W1: Does Dense-Off Indicate Insufficient SID Learning?

To examine how training order affects the use of historical SIDs, we added bidirectional input-removal and shuffling experiments. Removal zeroes out the dense residual or historical SID embedding outputs. Shuffling replaces one pathway's entire history with that of a different user with the same history length, while keeping the other pathway unchanged.

StrategyFullDense removedSID removedDense shuffledSID shuffled
Early4.17151.84113.66811.54343.6009
LBF4.71494.60033.04424.27341.4196

Early Fusion suffers a substantial performance drop when dense inputs are removed, whereas removing or shuffling historical SIDs has a relatively smaller effect. Moreover, the similar results under SID removal and shuffling suggest that even misleading SID histories from other users have little additional impact on its predictions, indicating limited reliance on SID information.

In contrast, LBF retains strong predictive capability when either pathway is removed, while shuffling either pathway degrades performance. This suggests that LBF effectively utilizes both SID and dense information, with dense refinement providing complementary gains. We will include these results in the revised paper.

W2/Q3: Are the Improvements Stable Across Seeds and Datasets?

To assess the stability of the relatively small SID-only versus LBF improvement, we conducted four-seed paired experiments on ten datasets. For each seed, SID-only was independently trained, and LBF was initialized from its validation-best checkpoint. The results below are mean R@10 (%) ± sample standard deviation over four runs.

DatasetSID-onlyLBF
BP4.6423 ± 0.01214.7169 ± 0.0157
BPC3.8289 ± 0.00243.9231 ± 0.0035
ACS5.2941 ± 0.01145.3746 ± 0.0187
VG9.8620 ± 0.01129.9547 ± 0.0216
MI6.8415 ± 0.03876.9507 ± 0.0396
IS5.4043 ± 0.02475.4308 ± 0.0458
SO3.4954 ± 0.00663.5840 ± 0.0097
TG4.2877 ± 0.01544.5474 ± 0.0119
HH3.3093 ± 0.04233.3862 ± 0.0588
BK8.3574 ± 0.04378.6551 ± 0.0590

LBF improves the mean over SID-only on every dataset, providing cross-dataset, multi-seed evidence of positive incremental value from dense refinement. We will include these results and their statistical details in the revision.

Q4: Why Does the BP Evaluation Count Differ from the User Count?

Table 7 reports the total number of users, not the number of valid test instances. The BP test set contains 150,185 instances, and all compared methods are evaluated on exactly the same test set. We initially reserve each user's last two interactions for validation and testing. Users with interaction sequences of length at most three do not contribute evaluation instances. In addition, when a held-out target item is absent from the training set, the corresponding interaction is returned to training to avoid cold-start evaluation. Consequently, some users do not contribute validation or test instances. We will clarify this filtering procedure and the evaluation denominator in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.