acceptodds
Under review as a conference paper at ICLR 2027

FUMO: Frequency Unified Embedding Optimization for Sequential Recommendation

Abstract

Sequential recommendation systems suffer from highly imbalanced item frequency distribution in real-world datasets. This imbalance can contribute to popularity bias and significantly influences recommendation accuracy. Training trajectories show that frequent item accuracy plateaus early while rare items require continued training for accuracy improvement, revealing unequal learning progress under the shared training schedule. To solve this problem, we propose Frequency Unified eMbedding Optimization (FUMO), an optimizer that uses item frequency to adjust recommendation updates for item embeddings. FUMO applies larger updates for recommendation loss on rare items and combines frequency-dependent weight decay with adaptive shrinkage to control the overfitting, without modifying the model architecture or introducing additional loss functions. Across six sequential recommendation models and four datasets, FUMOshows compelling performance compared with other baseline optimizers and improves HR@20 up to 18.5% and NDCG@20 up to 25.0% over Adam. Training trajectories additionally show earlier improvements in rare item validation. These results support item frequency as a useful signal for adjusting embedding optimization across diverse sequential recommendation systems, and show that FUMO is a competitive optimizer for sequential recommendation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Recommendation Systems, Popularity bias, Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Pj9bOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Rather than modifying model architectures or loss functions, FUMO operates purely on the embedding update rule with zero inference overhead. Two design choices are particularly well-thought-out: (a) scaling only the recommendation gradient in the first moment while keeping the auxiliary gradient unscaled and computing the second moment from the unscaled summed gradient, so rare embeddings are strongly regularized when idle (receiving only negative-sampling push-away gradients) but minimally damped when actively updated.

The authors evaluate FUMO across 6 representative sequential recommenders spanning attention (SASRec), RNN (GRU4Rec), MLP (FMLPRec), frequency-domain, and contrastive paradigms on 4 datasets against 5 optimizer baselines (Adam, AdamW, AdamD, FAL, Helen). FUMO outperforms Adam in all 48/48 model–dataset–metric combinations and ranks first in 39/48 settings (Table 1). Crucially, to address the concern that FUMO introduces two extra hyperparameters, the authors evaluate a restricted 4-configuration budget and show FUMO still beats Adam in 41/48 cases and the fully tuned best baseline in 31/48 cases.

Critical strengths

Rather than modifying model architectures or loss functions, FUMO operates purely on the embedding update rule with zero inference overhead. Two design choices are particularly well-thought-out: scaling only the recommendation gradient in the first moment while keeping the auxiliary gradient unscaled and computing the second moment from the unscaled summed gradient. Rare embeddings are strongly regularized when idle but minimally damped when actively updated.

The authors evaluate FUMO across 6 representative sequential recommenders spanning attention (SASRec), RNN (GRU4Rec), MLP (FMLPRec), frequency-domain, and contrastive paradigms on 4 datasets against 5 optimizer baselines (Adam, AdamW, AdamD, FAL, Helen). FUMO outperforms Adam in all 48/48 model–dataset–metric combinations and ranks first in 39/48 settings (Table 1). Crucially, to address the concern that FUMO introduces two extra hyperparameters, the authors evaluate a restricted 4-configuration budget and show FUMO still beats Adam in 41/48 cases and the fully tuned best baseline in 31/48 cases.

Critical weaknesses

FUMO adjusts both the first-moment recommendation gradient and the inverse-second-moment weight decay solely as a heuristic function of raw logged item count. Treating raw count as an intrinsic optimization hyperparameter suffers from three foundational confounds:

Observed interaction counts are not ground-truth item popularity or intrinsic learning difficulty; they are the product of the previous production RecSys engine's serving policy, including retrieval candidate filtering, ranking biases, position bias, and cold-start exploration rules. Hardcoding optimizer step sizes and weight decay to raw couples gradient dynamics directly to the historical serving policy's feedback loop—penalizing or shrinking items that were merely under-exposed by the production engine rather than intrinsically rare or noisy.

Upstream data pipelines routinely reshape item count$ via filtering (e.g., 5-core removal), user-activity capping, head-item/negative downsampling, temporal window truncation, and sliding-window sequence expansion vs. single-target sequences. While Section 4.1 claims scale invariance, this identity holds only when every item's count is multiplied by a uniform constant, and breaks under any non-uniform pipeline subsampling or sequence-slicing policy that alters the relative frequency curve or the ratio of context occurrences to target occurrences.

FUMO changes the stationary points of training; rare/under-exposed items have smaller second moments, shrinking their embedding norms more aggressively on idle steps and shifting inner-product logits. The paper never validates that FUMO improves intrinsic tail representations rather than acting as a popularity-norm confounder.

Questions for authors

While FUMO is a clean, drop-in heuristic that achieves consistent benchmark gains under a fixed 5-core leave-one-out preprocessing script (Table 1, Table 7), tying an optimizer's gradient and weight-decay dynamics directly to raw offline item counts raises a fundamental question of scientific validity: item frequency 𝑛_i is controlled by upstream data-pipeline sampling and exposure logging, not intrinsic optimization geometry. Without isolating the impact of data-pipeline sampling rather than acting as an uncalibrated popularity-norm confounder (especially given the tail-exposure drop on LastFM in Table 6 and missing final rare-item test metrics), the approach lacks the rigor needed for clear acceptance. See the weakness above too.

1 comment

Thank you for highlighting exposure, preprocessing, and norm-related confounds. We added experiments addressing each concern.

  • W1 — Historical exposure: FUMO uses counts as a proxy for available learning signals, without assuming they measure intrinsic popularity. Appendix C.1 adds the KuaiRand dataset with random-exposure feedback, retaining even single-interaction items. FUMO beats Adam in 12/12 comparisons and ranks first in 10/12, extending the evidence beyond upstream recommendation policy.
  • W2 — Preprocessing dependence: Appendix C.2 tests three non-uniform pipelines involving frequent-item removal, random replacement, and changed filtering thresholds. With recomputed counts and no hyperparameter retuning, FUMO improves both metrics in 19/24 settings. This establishes robustness of FUMO, and its gain does not solely depend on specific preprocessing pipeline.
  • W3 — Embedding-norm confounding: Appendix B.7.2 finds negative frequency–norm correlations: rarer items tend to have larger norms. More directly, after normalizing candidate item embeddings, FUMO beats other optimizers on SASRec and GRU4Rec in 12/16 comparisons and beats Adam in 15/16.
  • Q1 — We agree that logged counts depend on exposure and preprocessing. FUMO uses them as a proxy for available target and context learning signals, without assuming they measure intrinsic popularity. We added the checks addressing your concern:
  1. Exposure and preprocessing: FUMO beats Adam in 12/12 comparisons on KuaiRand, which includes random-exposure feedback (Appendix C.1). Across three non-uniform preprocessing pipelines, it improves both metrics in 19/24 settings without retuning (Appendix C.2).
  2. Norm confounding: After normalizing candidate item embeddings to unit length, FUMO still has the best result in 12/16 comparisons. Its gains therefore extend beyond candidate-norm effects (Appendix B.7.2). These results distinguish recommendation accuracy from exposure share. We present FUMO as an accuracy optimizer supported by these additional controls, while final rare-item gains and popularity-bias improvements are not universal.
Reviewer_iLX6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Motivated by unequal learning progress between frequent and rare items, this paper proposes FUMO, a frequency-aware optimizer for sequential recommendation. It combines a frequency-dependent step multiplier for recommendation updates with frequency-dependent adaptive weight decay to control overfitting. Experiments across six models and four datasets show that FUMO achieves relatively competitive performance against five baseline optimizers.

Critical strengths
  • The paper presents a clear motivation, supported by an empirical analysis of training dynamics for frequent and rare items.
  • The proposed method modifies only the embedding optimizer. It can be applied to different backbones without introducing additional loss or modifying the model architecture.
Critical weaknesses

W1: The role of weight decay in preventing rare-item overfitting is not directly evaluated. The only ablation without decay (A1) also removes frequency scaling, while the others retain either frequency-dependent or uniform decay. No variant combines frequency-dependent recommendation updates with no decay. Moreover, the paper does not directly compare rare-item overfitting with and without the proposed decay, for example, by analyzing the train–validation gap. Therefore, it remains unclear whether the decay is necessary and whether its benefits arise from mitigating rare-item overfitting as claimed.

W2: The improvements are marginal on several datasets. On Beauty and Sports, differences between FUMO and the strongest baseline are often within ±0.0005\pm 0.0005. FUMO falls behind the strongest baseline in 7 of 12 comparisons on Beauty. The largest relative gains are reported on LastFM, which contains only 1090 test users, making these gains less reliable.

Questions for authors

Q1: Figure 2 suggests that FUMO may reduce accuracy on frequent items. For example, on Yelp, FUMO’s frequent-item NDCG@20 appears slightly lower than that of some baselines in later epochs. Could the authors report test HR@20 and NDCG@20 separately for frequent and rare targets at the selected checkpoints? This would examine whether the overall gains preserve frequent-item accuracy or reflect a trade-off between frequent- and rare-item performance.

Q2: The motivation of FUMO is that a single stopping point cannot accommodate both frequency groups. However, rare-item accuracy is reported only at early epochs (Table 2, Figure 2). Could the authors report the actual stopping epochs, together with the frequency-group results requested in Q1? This would help clarify how the early rare-item improvements translate into the final overall gains.

Q3: Weight decay is applied only to the embedding layer in FUMO but to all parameters in the baselines (Appendix B.2). Since decaying encoder weights may itself affect performance, the main comparisons in Table 1 mix the frequency-dependent design with this difference in decay scope. Do ablations A2 and A3 (Table 7, Appendix B.5) retain embedding-only decay? Neither variant removes both frequency-dependent mechanisms. Could the authors report FUMO with γ=1\gamma=1 and ϕ≡1\phi\equiv1 (i.e., AdamD with embedding-only decay), keeping all other settings unchanged, to isolate the overall contribution of frequency awareness?

1 comment

Thank you for identifying the missing controls. The revision adds direct ablations and final checkpoint results.

  • W1 — Necessity of decay: We add a new ablation A3 that retains frequency-aware updates while removing decay in Appendix B.5. Accuracy decreases in all 16 comparisons. Appendix B.7.1 also shows less stable rare-item learning without decay: GRU4Rec’s late normalized validation progress averages 80% without decay versus 91% with FUMO. This supports stabilization, although these curves do not directly measure a train–validation gap.
  • W2 — Strength of improvements: We acknowledge that some baselines remain competitive. Nevertheless, FUMO still ranks first in 39/48 comparisons and second in the remaining 9 comparisons. FUMO achieves an average improvement of 3.58% compared with the best baseline optimizers and 7.43% compared with standard Adam.
  • Q1 — Final frequent/rare accuracy: We report test results of frequent and rare groups in Appendix D. FUMO improves both frequent-item metrics in all 24 settings. Mean rare-item HR@20 and NDCG@20 improve by 8.3% and 8.1% over Adam, respectively, although the improvement is not universal. FUMO accelerates rare item learning progress but does not guarantee a higher final test result.
  • Q2 — Training epochs: Appendix D also shows that FUMO reaches its optimal epoch later than Adam in 14/24 settings, averaging 42.3 epochs versus Adam’s 34.8. Earlier rare item progress can coexist with a later, higher overall validation peak.
  • Q3 — Isolating frequency awareness: Yes, these two ablations (now A2 and A4) retain embedding-only decay. We add a new ablation A5 in Appendix B.5 that isolates all frequency-aware components and sets γ=1\gamma=1 and ϕr≡1\phi_r \equiv 1 (equivalent to AdamD-style embedding-only decay). Performance decreases in all 16 comparisons, supporting FUMO’s frequency-aware design beyond decay scope.
Reviewer_wahwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the FUMO to address the issue of uneven learning progress caused by extreme item frequency imbalance in sequence recommendation. Through analysis of training trajectories, the authors observed that when using Adam, the accuracy for popular items saturates very early, while rare items require longer training to improve; a single training stopping point struggles to accommodate both types of items. To address this, FUMO sets a decreasing step multiplier for the recommendation loss gradient based on item frequency, enabling rare items to receive larger relative updates. This multiplier applies only to the recommendation gradient in the first-order moment, while the auxiliary loss gradient and second-order moment remain unchanged. Additionally, FUMO combines adaptive weight decay—correlated with frequency and the second-order moment—to control overfitting for rare items. The entire method requires no modifications to the model architecture and introduces no additional loss. The authors conducted experiments on four datasets and six sequence recommendation models. The results show that FUMO achieved the best performance in all comparisons.

Critical strengths
  1. The author uses training trajectories grouped by frequency to clearly demonstrate that popular items reach saturation early, while rare items take longer to learn, and explains this based on the gradient structure.
  2. FUMO is plug-and-play; it only modifies the optimization method at the embedded layer without altering the model architecture or adding a loss function, and it can be directly applied to a variety of existing models.
  3. The evaluation of the experiment covered four datasets and various models, and FUMO achieved the best results in all cases.
Critical weaknesses
  1. FUMO's grid search includes 80 sets of configurations, while AdamW and AdamD have only 2 to 3 sets, so the comparison in the main table may favor FUMO.
  2. FUMO applies decay only to the embedded layer, whereas other optimizers apply decay to all parameters, which affects the fairness of the comparison.
  3. The experimental datasets are all commonly used, small-scale public datasets; they have not been validated on large-scale industrial-grade item repositories, where frequency imbalance is most pronounced.
  4. Long-tail exposure and average recommendation popularity improved in only about two-thirds of the combinations, and showed limited improvement on LastFM.
Questions for authors

See Weaknesses

1 comment

Thank you for the constructive feedback. We address each concern below.

  • W1 — Tuning budget: Table 4 restricts FUMO to four configurations, a 95% reduction. It still beats Adam in 41/48 comparisons and the fully tuned strongest baseline in 31/48, with one additional tie against each.
  • W2 — Decay scope: We add a new ablation A9 in Appendix B.5 that combines FUMO embeddings with AdamW on non-embedding parameters and beats AdamW in 14/16 comparisons. Standard FUMO also beats our zero-decay Adam baseline in 48/48 comparisons, so its gains cannot be explained solely by avoiding encoder decay.
  • W3 — Industrial data: We add KuaiRand, an industry dataset with 567,198 positive interactions and random-exposure feedback in Appendix C.1. Its Gini coefficient is 0.695, versus 0.457/0.437/0.499/0.490 for Yelp/LastFM/Beauty/Sports, indicating substantially stronger imbalance. FUMO ranks first in 10/12 comparisons and beats Adam in all comparisons. This verifies FUMO’s performance on highly skewed industrial data.
  • W4 — Popularity bias: FUMO is an accurate optimizer with favorable average popularity-bias metrics. It improves tail exposure in 17/24 settings and lowers recommendation popularity in 16/24. LastFM, with only 1,090 users and 3,646 items, remains an explicit exception.

Comments on the paper

1 comment

Dear Area Chairs, Thank you for overseeing the review process. We have expanded the manuscript with additional experiments and controls directly addressing the reviewers’ concerns. Below, we summarize the major changes during the revision period.

  • One additional dataset and three preprocessing pipelines — Appendix C. In response to wahw W3 and Pj9b W1, we added KuaiRand, an industrial dataset containing random-exposure feedback and substantially stronger frequency imbalance. For Pj9b W2, we tested three non-uniform preprocessing pipelines without re-tuning.
  • Three additional ablations — Appendix B.5. A3, requested by iLX6 W1, removes decay while retaining frequency-aware updates. A5, requested by iLX6 Q3, removes the frequency-aware components while retaining embedding-only decay. A9 addresses wahw W2 by combining FUMO embeddings with AdamW on non-embedding parameters, demonstrating compatibility with encoder weight decay.
  • A detailed study of stronger rare-item decay — Appendix B.7. Addressing iLX6 W1 and Pj9b W3, we added learning trajectories with and without decay, frequency–norm correlations, and evaluation with unit-normalized candidate item embeddings.
  • Final frequent/rare test results and epochs — Appendix D. Requested by iLX6 Q1–Q2, and also addressing Pj9b Q1, these results cover all 24 model–dataset settings. Mean rare-item HR@20 and NDCG@20 improve by 8.3% and 8.1%. We explicitly report exceptions and distinguish early learning progress from final accuracy.
  • Clearer presentation and corrected notation. We added a roadmap in Section 1 and direct references from the main text to the supplementary analyses. We corrected the notation mistakes in Section 4.3 and Equation 15. These are additional editorial improvements.

We believe these additions are helpful in resolving the reviewers’ central concerns about tuning fairness, weight decay, data dependence, preprocessing robustness, frequent–rare trade-offs, and embedding-norm confounding. Our detailed responses explain the corresponding results and acknowledge cases where improvements are not universal. We respectfully ask the Chairs to consider the extent of these revisions and the new empirical evidence when making the final decision.

FUMO Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.