acceptodds
Withdrawn from ICLR 2027

Frequency-Graded Directional Collapse in Full-Softmax Sequential Recommenders

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Full-softmax sequential recommenders degrade on rare items, but the failure's locus and the mechanism of content-based repair remain unclear. We identify frequency-graded directional concentration after cross-entropy training in the output, rather than input, embedding table: rare-item directions form a low-rank cone anti-aligned with typical user states. Untied probes, capacity controls, and a frequency-capping ladder with volume-matched controls support an imbalance-driven optimization explanation. History dropout strengthens tail retrieval, whereas a frozen semantic residual favors popular items, consistent with an anisotropy-associated popularity prior. Whitening the semantic matrix before training reduces this anisotropy; tested post-hoc centering, normalization and gating do not reproduce the training-time gains. Across three Amazon corpora, the repaired retriever raises Recall@10 by 20–29% over the matched d = 192 pure-ID control. A 17.4K-parameter residual then reranks only the retrieved top-10 using training-set co-occurrence, preserving Recall@10 exactly. Under matched full-catalog evaluation, the final system achieves state-of-the-art performance against ten reproduced baselines, improving all four metrics across three corpora by approximately 17–48% over the strongest five-seed baseline per metric. Repeated tests reject frequency gating; excluding consumed items alone raises NDCG@10 by 28.7%. These results connect output-side geometry to ranking behavior under a frozen evaluation protocol.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
sequential recommendation, long-tail, embedding geometry, class imbalance, weight tying, evaluation protocol, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8xXGOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why full-softmax sequential recommenders underperform on rare items. Using untied probes, the authors argue the failure lives in the output embedding table: rare-item output vectors form a low-rank cone that is anti-aligned with typical user states, while rare-item input vectors stay more dispersed. They support an imbalance story with a frequency-capping ladder (plus volume-matched deletion) and a layer-peeled analysis. The proposed fix combines history dropout, a frozen flan-t5-xxl semantic residual that is PCA-whitened before training, and a small (~17.4K) residual head that only reranks the retrieved top-10 using score, rank, frequency, and training-set co-occurrence. Whitening improves Recall@10 by roughly 20–29% over a matched d=192 pure-ID control. Under one full-catalog protocol on Beauty / Toys / Sports, the full system beats ten reproduced baselines on all reported metrics.

Critical strengths
  1. Separating the input table from the output table is the right diagnostic move. Weight tying makes these roles look like one object when they are not. On Beauty, the rare-item output side collapses (roughly 1.93× a random-direction reference, stable rank ~4.8/64, mean cosine to the average user state around −0.36), while the input side stays nearer the random baseline until the hottest frequency group. That finding is concrete and useful.
  2. The whitening comparison is fairly clean. Raw vs whitened semantics keep the same encoder, dimension, and trainable parameter count. Whitening wins on all four metrics across the three datasets. Post-hoc centering / row-norm / gating do not reproduce the training-time gain. Frequency gating hurts tail Recall@10 and is correctly dropped from the final recipe. I appreciate that the negative results are reported.
  3. Retrieval and reranking are separated by design. The residual only permutes the top-10, so Recall@10 is unchanged by construction. The protocol audit is also honest: excluding consumed items on one Beauty checkpoint moves NDCG@10 from 0.0432 to 0.0556 — same order of magnitude as several claimed model gains.
  4. The appendix is unusually transparent about awkward cases (constant-output ActionPiece seed, short-prefix BERT4Rec recipe that collapses performance, development-time test inspection). It takes some work to keep diagnostic checkpoints and the five-seed system straight, but the information is there.
Critical weaknesses
  1. The headline 17–48% gains lean on a baseline panel that the paper itself shows is uneven. Reproduced SASRec on Beauty gets R@10 0.0643, while the authors' own d=64 pure-ID trunk on the same splits reaches 0.0719 under a different optimizer / batch / LR. BERT4Rec is trained on very short prefixes (median length around 3) and lands near 0.008. ActionPiece's Sports mean includes a seed that emits the same top-10 for all 35,598 users; without that seed the 4-seed R@10 is 0.0480, and the relative gain over ActionPiece drops from ~+37.7% to ~+16%. LLM-ESR also comes in below SASRec. I trust the whitening-vs-raw comparison. I do not trust the "over the strongest baseline" percentages until the panel is retuned more carefully.
  2. A large chunk of the NDCG lift is not a cone repair. Score/rank calibration moves validation NDCG@5 by at most ~0.4%; adding frequency adds at most ~0.8%; the full feature set with co-occurrence counts adds 10.7–20.4%. History dropout is a generic augmentation. Only whitening is aimed at the diagnosed geometry. The abstract still makes the whole stack sound like a consequence of that diagnosis.
  3. The causal reading of the geometry is stronger than the evidence. As Appendix E notes, adding the same vector to every output embedding can change cosine / stable-rank stats without changing softmax rankings. Whitening also rescales features and changes optimization. The frequency-cap ladder changes which users/contexts survive. Whitening as an intervention is solid. "We fixed directional collapse and that is why accuracy rose" is not established.
  4. The rare-item narrative does not fully survive the paper's own group metrics. Relative to pure ID, whitening lowers catalog coverage on every corpus. On Sports, matched tail Recall@10 is 0.01430 vs 0.01667 for pure ID. Aggregate accuracy improves; recovery of rare targets does not improve uniformly.
  5. Width 192 was selected on Beauty and then reused. With the same reranker, width 128 is better on all four test metrics for Toys and Sports. So the reported recipe is not the best configuration the authors tried on 2/3 datasets.
Questions for authors
  1. Evidence: please retrain the baseline panel with the trunk optimizer recipe (and the trunk with the panel recipe). How much do the "over strongest baseline" numbers shrink? I need this to separate method from tuning.
  2. Clarification: which Sports number should readers use — the 5-seed mean that includes the constant-output run, or the 4-seed mean? The appendix has the sensitivity analysis; the abstract’s upper end of the 17–48% range still reflects the inflated five-seed Sports figure.
  3. Evidence: do the frequency-graded trends remain under a translation-invariant margin / centered output table? If the cone is mostly a shared offset, it does not explain rare-item loss.
  4. Clarification: how much of the final NDCG gain remains without the co-occurrence head, and how much does that head give when attached to the pure-ID trunk? Please put the geometry result and the count-feature result in separate rows.
Reviewer_gbDrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This manuscript seems completely AI-generated and the presentation is extremely poor. I cannot understand what it is saying, so I cannot write a paper summary.

A paper with this quality should be viewed as an AI slop and should not be accepted by ICLR.

Critical strengths

None

Critical weaknesses
  1. This paper is AI-generated and has extremely poor presentation.

  2. The problems, methodologies, results and contributions are unclear.

  3. Many terminologies are AI hallucination.

Questions for authors
  1. What is a full-softmax sequential recommender? How is it defined? Is it a collaborative filtering recommender, a content-based recommender or a knowledge-based recommender? Is there any existing study on it?

  2. What is directional collapse? How is it defined? Why is it a serious problem in recommender system?

  3. What is your approach in fixing the issue of directional collapse? Why is your approach novel?

  4. In Line 159, can authors explain the mechanism of PCA whitening?

  5. In Line 247, what does "input-output geometry" mean? Where is your geometry?

  6. In Tables 1 and 2, why are the values "Ours, retrieval" for the "R@10" column missing?

  7. In Line 210, what does the "Amazon history window" mean?

  8. Line 211, "ML-1M is used only for a window-length protocol control". MovieLens-1M is a dataset. What do you mean by using it for protocol control?

  9. In Line 151, what does the number 40964096 mean?

  10. In Eq. (1), what does the number 10−810^{-8} mean?

  11. What are the relationships between Eqs. (1)(2)(3)?

  12. What is the mathematical meaning behind Eq. (3)?

  13. In Line 232, what does the "retriever seeds" mean? Why should we care about it?

  14. In Line 15, what does the "untied probes" mean?

  15. In Line 272, "Ours, final uses one shared fixed reranker". What does ``one shared fixed reranker'' mean?

  16. In Line 251, what is an "approximate isotropic reference"?

  17. In Line 249, what does the "stable rank" mean? Why do you measure it?

  18. In Lines 432-440, why do you list the two cases here? Do they explain anything?

  19. What does "frequency decile" mean in your Figure 2?

  20. In Line 268, what is a "frequency-capping ladder"?

Reviewer_FAEzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work focuses on why sequential recommenders perform poorly on rare items. The authors find that the models’ output embeddings end up clustered in similar directions, and they attribute this to imbalanced training. The problem appears from the output embedding table rather than the input one, where rare items form a low-rank cone that may point away from typical user states. To overcome this problem, they propose a method that has three parts: history dropout, whitened text embeddings, and a small reranker that reorders the top 10 recommendations, which leaves Recall@10 unchanged. On three Amazon datasets, the retriever raises Recall@10 by 20–29% over matched ID-only models, and the full system beats the baselines they tested. Against ten reproduced baselines, it improves all four metrics by roughly 17–48% over the strongest baseline for each one.

Critical strengths
  1. The diagnosis of why sequential recommenders perform poorly on rare items is novel and interesting. Separating input and output embeddings shows that history representations and poor candidate-scoring geometry can appear at the same time, indicating there might be certain intuition about the mechanism here. This also gives researchers a concrete place to investigate rare-item failures, beyond simply increasing model size.
  2. The experiment design is clear and helps strengthen the understanding of component contributions. The raw-versus-whitened comparisons hold the text encoder, representation dimension, and parameter count fixed. Separating retrieval from reranking also distinguishes improvements in finding relevant items from improvements in ordering them.
Critical weaknesses
  1. Better overall accuracy does not consistently benefit rare items. On the Sports dataset, tail Recall@10 decreases from 0.01667 with ID-only embeddings to 0.01430 with whitened semantics. Tail coverage, which should be the fraction of rare items recommended at least once, also decreases. That being said, the proposed method improves aggregate accuracy while potentially recommending a narrower range of rare items.
  2. Most of the experiments are done using Amazon datasets. This adds a potential domain concern that it is unknown whether the observations and experimental results can be generalized to a wider range of corpora.
Questions for authors

Based on the first point in the weaknesses section, could the authors further discuss or explain, with additional results or examples, whether the method causes the model to recommend a narrower range of rare items, even if aggregate accuracy improves? Additionally, does a similar pattern exist in other datasets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.