- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Rather than modifying model architectures or loss functions, FUMO operates purely on the embedding update rule with zero inference overhead. Two design choices are particularly well-thought-out: (a) scaling only the recommendation gradient in the first moment while keeping the auxiliary gradient unscaled and computing the second moment from the unscaled summed gradient, so rare embeddings are strongly regularized when idle (receiving only negative-sampling push-away gradients) but minimally damped when actively updated.
The authors evaluate FUMO across 6 representative sequential recommenders spanning attention (SASRec), RNN (GRU4Rec), MLP (FMLPRec), frequency-domain, and contrastive paradigms on 4 datasets against 5 optimizer baselines (Adam, AdamW, AdamD, FAL, Helen). FUMO outperforms Adam in all 48/48 model–dataset–metric combinations and ranks first in 39/48 settings (Table 1). Crucially, to address the concern that FUMO introduces two extra hyperparameters, the authors evaluate a restricted 4-configuration budget and show FUMO still beats Adam in 41/48 cases and the fully tuned best baseline in 31/48 cases.
Critical strengths
Rather than modifying model architectures or loss functions, FUMO operates purely on the embedding update rule with zero inference overhead. Two design choices are particularly well-thought-out: scaling only the recommendation gradient in the first moment while keeping the auxiliary gradient unscaled and computing the second moment from the unscaled summed gradient. Rare embeddings are strongly regularized when idle but minimally damped when actively updated.
The authors evaluate FUMO across 6 representative sequential recommenders spanning attention (SASRec), RNN (GRU4Rec), MLP (FMLPRec), frequency-domain, and contrastive paradigms on 4 datasets against 5 optimizer baselines (Adam, AdamW, AdamD, FAL, Helen). FUMO outperforms Adam in all 48/48 model–dataset–metric combinations and ranks first in 39/48 settings (Table 1). Crucially, to address the concern that FUMO introduces two extra hyperparameters, the authors evaluate a restricted 4-configuration budget and show FUMO still beats Adam in 41/48 cases and the fully tuned best baseline in 31/48 cases.
Critical weaknesses
FUMO adjusts both the first-moment recommendation gradient and the inverse-second-moment weight decay solely as a heuristic function of raw logged item count. Treating raw count as an intrinsic optimization hyperparameter suffers from three foundational confounds:
Observed interaction counts are not ground-truth item popularity or intrinsic learning difficulty; they are the product of the previous production RecSys engine's serving policy, including retrieval candidate filtering, ranking biases, position bias, and cold-start exploration rules. Hardcoding optimizer step sizes and weight decay to raw couples gradient dynamics directly to the historical serving policy's feedback loop—penalizing or shrinking items that were merely under-exposed by the production engine rather than intrinsically rare or noisy.
Upstream data pipelines routinely reshape item count$ via filtering (e.g., 5-core removal), user-activity capping, head-item/negative downsampling, temporal window truncation, and sliding-window sequence expansion vs. single-target sequences. While Section 4.1 claims scale invariance, this identity holds only when every item's count is multiplied by a uniform constant, and breaks under any non-uniform pipeline subsampling or sequence-slicing policy that alters the relative frequency curve or the ratio of context occurrences to target occurrences.
FUMO changes the stationary points of training; rare/under-exposed items have smaller second moments, shrinking their embedding norms more aggressively on idle steps and shifting inner-product logits. The paper never validates that FUMO improves intrinsic tail representations rather than acting as a popularity-norm confounder.
Questions for authors
While FUMO is a clean, drop-in heuristic that achieves consistent benchmark gains under a fixed 5-core leave-one-out preprocessing script (Table 1, Table 7), tying an optimizer's gradient and weight-decay dynamics directly to raw offline item counts raises a fundamental question of scientific validity: item frequency 𝑛_i is controlled by upstream data-pipeline sampling and exposure logging, not intrinsic optimization geometry. Without isolating the impact of data-pipeline sampling rather than acting as an uncalibrated popularity-norm confounder (especially given the tail-exposure drop on LastFM in Table 6 and missing final rare-item test metrics), the approach lacks the rigor needed for clear acceptance. See the weakness above too.
Thank you for highlighting exposure, preprocessing, and norm-related confounds. We added experiments addressing each concern.