- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem that increasing the depth of CTR prediction models can lead to performance saturation or even degradation. It suggests that redundant representations across layers and insufficient use of information learned at different depths may limit the benefits of greater depth. To address these issues, the authors propose DeCORA, which combines cross-depth decorrelation regularization with depth-prefix aggregation to encourage complementary representations across layers and facilitate their reuse. Experiments on multiple recommendation backbones and two datasets demonstrate improved predictive performance, with mean AUC consistently increasing as model depth grows within the evaluated range.
Critical strengths
- The motivation of this research is clear and interesting. It is meaningful to discuss the relationship between increasing the depth of CTR prediction models and the prediction accuracy.
- The authors gave plenty of experiments for their research.
- The authors gave theoretical proofs for their methods.
Critical weaknesses
- The evaluation is limited to two datasets, which raises questions about the generalizability of the findings across different CTR prediction scenarios. Could the authors clarify the rationale for restricting the evaluation to these two datasets? Additional experiments on CTR datasets with different characteristics would strengthen the empirical evidence and make the conclusions more convincing. If the author can give convincing reasons, it is also acceptable.
- The analysis and theoretical results appear largely domain-agnostic, and the paper also discusses possible future extensions. Could the authors clarify the motivation for focusing on recommender systems and explain which aspects of DeCORA specifically address challenges in CTR prediction? Have any experiments been conducted in other domains to assess the method’s broader applicability? Such evidence would help clarify its scope and generalizability.
- The connection of Theorem 5 and Corollary 3 to the proposed method and empirical results appears limited. Could the authors clarify their specific roles in supporting the paper’s main claims?
Questions for authors
See Weaknesses
4 comments
Official Comment by Authors (2/2)
W2: Why recommender systems, and what is CTR-specific
Thank you for the question. Our focus is motivated by two observations on deep recommenders: (1) depth mechanisms from LLMs and vision (DenseFormer, mHC, ORU, AttnRes) give uneven and sometimes negative gains on recommenders (Table 2; Tables 9–12); (2) in vanilla recommenders, deeper layers correlate weakly with the heterogeneous field embeddings (input tokens) but strongly with each other (Fig. 3).
DeCORA addresses both in a recommendation-specific way: it keeps the input tokens accessible at every depth, reduces cross-depth redundancy per token without mixing heterogeneous tokens, and wraps any interaction operator (FM-based, mixer, hierarchical, or attention-based).
Hence, while our analysis is stated generally, DeCORA is tailored to embedding-based recommendation. Within this scope it extends beyond multi-field CTR, to sequential CTR (W1), and to newly added GNN-based collaborative filtering (Appendix A.6), where it improves 8-layer LightGCN [5] by 49% in Recall@20. Extending DeCORA beyond recommendation is a promising future direction.
W3: Roles of Theorem 5 and Corollary 3
Thank you for the careful reading. Our theory is intended to explain why the paired design is needed, rather than to provide training guarantees. Corollary 3 supports the main argument, and Theorem 5 is a supplementary analysis of our loss.
- Corollary 3 explains why depth saturates, and what must change. Theorem 4 shows that a new layer improves prediction only through its new, target-aligned information. Corollary 3 adds that, along a fixed sequence of representations, these gains share a finite budget and must vanish, so stacking more layers helps only if each new layer keeps contributing such information. This matches what we observe: vanilla peaks early (Fig. 2) and its late layers add almost no new representational geometry (IKR ≈ 0 at the last source for RankMixer; Fig. 3), while DeCORA keeps IKR high (≈ 0.59) and is the only variant whose AUC still improves from 8× to 16× in all 8 backbone–dataset pairs (Table 4).
- Theorem 5 is a supplementary interpretation of our loss, not a requirement of the method. Our loss penalizes correlations between pairs of depths, whereas redundancy is a property of the whole depth stack. Theorem 5 relates the two: the coding-rate redundancy of the whole stack (in the spirit of MCR² [6]) is zero exactly when all cross-depth correlations vanish, and, locally, its leading term is proportional to the uniform-weight version of our pairwise penalty, including after the shared adapter. Hence the pairwise loss can be viewed as targeting whole-stack redundancy at pairwise cost, in line with the lower DCO of DeCORA (Fig. 3).
We have made these roles explicit in Sec. 3.4 (one sentence each), noting that Theorem 5 is a supplementary view (Appendix D).
We hope our responses address your concerns. Thank you again for the constructive comments, which have helped us improve the paper.
References
[5] LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. In SIGIR 2020.
[6] Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction. In NeurIPS 2020.
Thanks for your responses. The authors solved most of my concern.
Thank you for the confirmation
Dear Reviewer,
Thank you very much for taking the time to review our responses and for your continued positive assessment. We are very glad to hear that our updates have resolved most of your concerns.
Best regards,
Official Comment by Authors (1/2)
We sincerely thank the reviewer for the positive assessment and for highlighting the strength of our motivation, empirical validation, and theoretical analysis. Below we address each concern; all revisions are highlighted in blue in the updated manuscript.
W1: Choice of datasets
Thank you for the question. Avazu (40.3M samples) and Criteo (45.9M samples) are large-scale benchmarks widely used for convincing offline evaluation of deep and scaled-up recommenders [1, 2], and they are large enough for depth effects to emerge.
To cover different data characteristics, here we add two sequential CTR datasets with user behavior sequences, TaobaoAd [3] and KuaiVideo [4], comparing vanilla, mHC, and DeCORA. As shown below, while vanilla degrades with depth (AUC −0.0086 to −0.0199 from 4× to 16×) and mHC provides only modest gains, our DeCORA consistently excels across all settings and widens its margin as depth grows, outperforming vanilla by 0.0136–0.0255 AUC at 16×. The results have been added to Appendix A.6 of the updated submission.
References
[1] Wukong: Towards a Scaling Law for Large-Scale Recommendation. In ICML 2024.
[2] Bagpipe: Accelerating Deep Recommendation Model Training. In SOSP 2023.
[3] Deep Match to Rank Model for Personalized Click-Through Rate Prediction. In AAAI 2020.
[4] Routing Micro-videos via A Temporal Graph-guided Recommendation System. In MM 2019.