acceptodds
Under review as a conference paper at ICLR 2027

Making Depth Matter: Complementary Representation Learning for Deep Recommender Systems

Abstract

Making a recommender deeper does not necessarily make it better. We study this problem through the joint design of representation learning and reuse, introducing DeCORA (Depthwise Correlation-Regularized Aggregation), a wrapper that couples token-conditional cross-depth correlation control with depth-prefix aggregation. A training-only objective regularizes source correlations with learned, normalized pair weights, while depth-specific maps shared across sources aggregate the embedding and all available hidden representations, retaining the backbone's interaction operators. Our conditional analysis connects correlation control to coordinate-function overlap and its transport through affine maps, clarifying the roles of target alignment and access after aggregation in making distinct responses useful for prediction. Across four structurally different backbones on Avazu and Criteo, DeCORA improves AUC and Logloss over vanilla in all 40 matched Base– configurations, with a mean absolute AUC gain of at depth. Among the component and parameter-matched variants, only the full method improves mean AUC in all eight backbone–dataset pairs from to ; representation diagnostics further show that the full method consistently exhibits the geometric patterns motivated by our analysis. The code is available in an anonymous repository.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Recommendation, Deep Recommenders, Complementary Representation, CTR Prediction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fR1LOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem that increasing the depth of CTR prediction models can lead to performance saturation or even degradation. It suggests that redundant representations across layers and insufficient use of information learned at different depths may limit the benefits of greater depth. To address these issues, the authors propose DeCORA, which combines cross-depth decorrelation regularization with depth-prefix aggregation to encourage complementary representations across layers and facilitate their reuse. Experiments on multiple recommendation backbones and two datasets demonstrate improved predictive performance, with mean AUC consistently increasing as model depth grows within the evaluated range.

Critical strengths
  1. The motivation of this research is clear and interesting. It is meaningful to discuss the relationship between increasing the depth of CTR prediction models and the prediction accuracy.
  2. The authors gave plenty of experiments for their research.
  3. The authors gave theoretical proofs for their methods.
Critical weaknesses
  1. The evaluation is limited to two datasets, which raises questions about the generalizability of the findings across different CTR prediction scenarios. Could the authors clarify the rationale for restricting the evaluation to these two datasets? Additional experiments on CTR datasets with different characteristics would strengthen the empirical evidence and make the conclusions more convincing. If the author can give convincing reasons, it is also acceptable.
  2. The analysis and theoretical results appear largely domain-agnostic, and the paper also discusses possible future extensions. Could the authors clarify the motivation for focusing on recommender systems and explain which aspects of DeCORA specifically address challenges in CTR prediction? Have any experiments been conducted in other domains to assess the method’s broader applicability? Such evidence would help clarify its scope and generalizability.
  3. The connection of Theorem 5 and Corollary 3 to the proposed method and empirical results appears limited. Could the authors clarify their specific roles in supporting the paper’s main claims?
Questions for authors

See Weaknesses

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

We sincerely thank the reviewer for the positive assessment and for highlighting the strength of our motivation, empirical validation, and theoretical analysis. Below we address each concern; all revisions are highlighted in blue in the updated manuscript.

W1: Choice of datasets

Thank you for the question. Avazu (40.3M samples) and Criteo (45.9M samples) are large-scale benchmarks widely used for convincing offline evaluation of deep and scaled-up recommenders [1, 2], and they are large enough for depth effects to emerge.

To cover different data characteristics, here we add two sequential CTR datasets with user behavior sequences, TaobaoAd [3] and KuaiVideo [4], comparing vanilla, mHC, and DeCORA. As shown below, while vanilla degrades with depth (AUC −0.0086 to −0.0199 from 4× to 16×) and mHC provides only modest gains, our DeCORA consistently excels across all settings and widens its margin as depth grows, outperforming vanilla by 0.0136–0.0255 AUC at 16×. The results have been added to Appendix A.6 of the updated submission.

ModelgAUC (Taobao)AUC (Taobao)Logloss (Taobao)gAUC (KuaiVideo)AUC (KuaiVideo)Logloss (KuaiVideo)
RankMixer (4× / 16×)0.5771 / 0.56440.6533 / 0.63570.1926 / 0.19620.6749 / 0.66280.7527 / 0.74410.4271 / 0.4384
w/ mHC0.5783 / 0.57900.6544 / 0.65530.1919 / 0.19200.6762 / 0.67640.7539 / 0.75380.4242 / 0.4240
w/ DeCORA0.5795 / 0.58160.6560 / 0.65830.1911 / 0.19000.6779 / 0.68010.7546 / 0.75770.4234 / 0.4209
WuKong (4× / 16×)0.5749 / 0.56290.6512 / 0.63130.1933 / 0.19900.6698 / 0.65770.7502 / 0.74100.4303 / 0.4469
w/ mHC0.5760 / 0.57590.6530 / 0.65310.1925 / 0.19290.6719 / 0.67240.7525 / 0.75280.4275 / 0.4263
w/ DeCORA0.5772 / 0.57970.6539 / 0.65680.1920 / 0.19110.6745 / 0.67870.7532 / 0.75530.4260 / 0.4230

References

[1] Wukong: Towards a Scaling Law for Large-Scale Recommendation. In ICML 2024.

[2] Bagpipe: Accelerating Deep Recommendation Model Training. In SOSP 2023.

[3] Deep Match to Rank Model for Personalized Click-Through Rate Prediction. In AAAI 2020.

[4] Routing Micro-videos via A Temporal Graph-guided Recommendation System. In MM 2019.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

W2: Why recommender systems, and what is CTR-specific

Thank you for the question. Our focus is motivated by two observations on deep recommenders: (1) depth mechanisms from LLMs and vision (DenseFormer, mHC, ORU, AttnRes) give uneven and sometimes negative gains on recommenders (Table 2; Tables 9–12); (2) in vanilla recommenders, deeper layers correlate weakly with the heterogeneous field embeddings (input tokens) but strongly with each other (Fig. 3).

DeCORA addresses both in a recommendation-specific way: it keeps the input tokens accessible at every depth, reduces cross-depth redundancy per token without mixing heterogeneous tokens, and wraps any interaction operator (FM-based, mixer, hierarchical, or attention-based).

Hence, while our analysis is stated generally, DeCORA is tailored to embedding-based recommendation. Within this scope it extends beyond multi-field CTR, to sequential CTR (W1), and to newly added GNN-based collaborative filtering (Appendix A.6), where it improves 8-layer LightGCN [5] by 49% in Recall@20. Extending DeCORA beyond recommendation is a promising future direction.

W3: Roles of Theorem 5 and Corollary 3

Thank you for the careful reading. Our theory is intended to explain why the paired design is needed, rather than to provide training guarantees. Corollary 3 supports the main argument, and Theorem 5 is a supplementary analysis of our loss.

  • Corollary 3 explains why depth saturates, and what must change. Theorem 4 shows that a new layer improves prediction only through its new, target-aligned information. Corollary 3 adds that, along a fixed sequence of representations, these gains share a finite budget and must vanish, so stacking more layers helps only if each new layer keeps contributing such information. This matches what we observe: vanilla peaks early (Fig. 2) and its late layers add almost no new representational geometry (IKR ≈ 0 at the last source for RankMixer; Fig. 3), while DeCORA keeps IKR high (≈ 0.59) and is the only variant whose AUC still improves from 8× to 16× in all 8 backbone–dataset pairs (Table 4).
  • Theorem 5 is a supplementary interpretation of our loss, not a requirement of the method. Our loss penalizes correlations between pairs of depths, whereas redundancy is a property of the whole depth stack. Theorem 5 relates the two: the coding-rate redundancy of the whole stack (in the spirit of MCR² [6]) is zero exactly when all cross-depth correlations vanish, and, locally, its leading term is proportional to the uniform-weight version of our pairwise penalty, including after the shared adapter. Hence the pairwise loss can be viewed as targeting whole-stack redundancy at pairwise cost, in line with the lower DCO of DeCORA (Fig. 3).

We have made these roles explicit in Sec. 3.4 (one sentence each), noting that Theorem 5 is a supplementary view (Appendix D).

We hope our responses address your concerns. Thank you again for the constructive comments, which have helped us improve the paper.


References

[5] LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. In SIGIR 2020.

[6] Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reduction. In NeurIPS 2020.

Thanks for your responses. The authors solved most of my concern.

AuthorsOpenReview ↗

Thank you for the confirmation

Dear Reviewer,

Thank you very much for taking the time to review our responses and for your continued positive assessment. We are very glad to hear that our updates have resolved most of your concerns.

Best regards,

Reviewer_rTXWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

To scale the depth of recommender systems, the paper proposed DenseNet like aggregation of previous layers and added regularization in the form of weighted cross-layer feature correlations. The paper showed its effectiveness on SOTA models like DHEN and Wukong. The math formulation is over complicated and can be simplified. There are missing symbol definition and misleading usage of some, making the method and experiments hard to understand.

Critical strengths
  • results on SOTA models like DHEN and Wukong are promising
Critical weaknesses
  • The math formulation is over complicated and can be simplified. There are missing symbol definition and misleading usage of some, making the method and experiments hard to understand.
Questions for authors
  • how does the method compare with simple baselines like residual and densenet?
  • the token definition is unclear in recommendation models
3 comments

We thank the reviewer for the valuable feedback and recognition of our empirical results. Below we address each concern; all revisions are highlighted in blue in the updated manuscript.

W1: Math formulation and symbol definitions

Thank you for noting where the definitions and notation could be more explicit. We have addressed these expository details with targeted clarifications in the revised manuscript. The core method has two components: 1) each layer receives an adapter-transformed average of preceding representations; 2) a training-only loss discourages cross-depth correlation. Importantly, the mathematical depth is devoted to analysis and diagnostics rather than algorithmic complexity: Eqs. (1)–(6) define DeCORA, whereas Eqs. (7)–(10) interpret its behavior and Eqs. (11)–(13) define diagnostics; formal results and proofs are in Appendices B–D.

Specifically, we clarified:

  • Defined at first use: backbone layers, prediction head, depth LL, number of tokens TT, and width dd (Sec. 3.1); per-token batch statistics in Eq. (4) and the L2L_2 inner product (Sec. 3.2); and coherence μs\mu_s (Sec. 3.3).
  • Disambiguated notation: token-indexed sample quantities marked tt are distinguished from their fixed-token population counterparts (Sec. 3.2).

These notation edits are minimal and do not alter the method or theorem statements.

Q1: Comparison with simple residual and DenseNet

Thank you for the suggestion. We add both baselines on RankMixer and WuKong (details in Appendix A.6) below. We observe that DeCORA is the best in all 8 settings on both metrics. Simple residual brings minimal benefits (lower AUC in 6 of 8 settings), and DenseNet helps only modestly: at 16×, DeCORA's AUC gain over vanilla is 1.9–3.5× that of DenseNet, at a lower cost. Consistent with our Aggregation only ablation (Table 4), access to earlier layers alone is not enough; pairing it with correlation control is the key.

ModelAUC (Avazu)Logloss (Avazu)AUC (Criteo)Logloss (Criteo)
RankMixer (4× / 16×)0.79496 / 0.794430.37086 / 0.371180.81374 / 0.813360.43901 / 0.43917
w/ Simple Residual0.79491 / 0.794200.37080 / 0.371250.81368 / 0.813170.43910 / 0.43935
w/ DenseNet [1]0.79508 / 0.795180.37071 / 0.370600.81370 / 0.813830.43871 / 0.43848
w/ DeCORA0.79579 / 0.796710.37023 / 0.369570.81425 / 0.815010.43792 / 0.43723
WuKong (4× / 16×)0.79429 / 0.793280.37234 / 0.372220.81380 / 0.813440.43901 / 0.43922
w/ Simple Residual0.79436 / 0.793030.37233 / 0.372450.81387 / 0.813280.43889 / 0.43940
w/ DenseNet [1]0.79444 / 0.794690.37217 / 0.371990.81374 / 0.813860.43877 / 0.43838
w/ DeCORA0.79500 / 0.795900.37088 / 0.370190.81400 / 0.814630.43806 / 0.43739

Q2: Definition of "token"

We apologize for not defining this term. Following recent recommendation research (e.g., RankMixer [2], FAT [3]), heterogeneous inputs (multi-field sparse features, user behaviors, etc.) are embedded into a unified set of vectors called tokens, so that diverse inputs share one interface to the stacked interaction layers; tokens might not correspond one-to-one to raw fields. Concretely, a token is one row of the input matrix of shape [number of tokens TT, embedding dimension dd]. We use this term throughout and have now added the definition to Sec. 1 and Sec. 3.1.

We hope these clarifications and new results address your concerns, and we would be grateful if you would take them into account in your final assessment. Thank you again for your time and feedback.


References

[1] Densely Connected Convolutional Networks. In CVPR 2017.

[2] RankMixer: Scaling Up Ranking Models in Industrial Recommenders. In CIKM 2025.

[3] From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction. In KDD 2026.

Reviewer_rTXWOpenReview ↗

concerns resolved with updated version and added experiments

as titled

AuthorsOpenReview ↗

Thank you for your feedback and upgraded score

Dear Reviewer,

Thank you very much for re-evaluating our work and for raising your score based on our revised draft and added experiments. We truly appreciate your time and feedback throughout the process, and we are very glad that our updates have resolved your concerns.

Best regards,

Reviewer_nmspOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the problem of implementing deep recommendation models, where existing approaches show a degradation in performance when stacking numerous layers. To handle this issue, current solutions propose to enhance connections among layers of the recommendation models. However, none of them consider the interplay between two key aspects, namely: 1) the correlation among different depths, and 2) the aggregation from previous layers (indicated as prefix). To this end, the authors propose to jointly address these two points by designing a novel recommendation approach named DeCORA. DeCORA works by controlling the correlation among different depths, meaning that each layer should be driven to produce outputs that are not redundant; at the same time, they ensure each layer receives as input also the aggregation from previous layers. The authors maintain the two aspects should be considered in a jointly manner, because only one contribution would not be enough: only leveraging the aggregated output from previous layers would not ensure each depth is learning novel (not redundant) information, and only de-correlating layers would not ensure the differences among depths would be accessible at later layers. Additionally, the authors provide an analytical view of representation and access across depths, by connecting correlation control with coordinate-function overlap. This further motivates complementary diagnostics in terms of source overlap, kernel representational novelty, and aggregate-prefix alignment. Experiments for two datasets and four backbones demonstrate how the proposed DeCORA can outperform a Vanilla version on both AUC and LogLoss for depths up to 16x, by still retaining limited memory consumption and low running time.

Critical strengths

I think the paper has the merits to address a foundational problem in recommender systems, namely, how to properly exploit deep architectures to improve recommendation performance, where existing methodologies show performance degradations after certain depths. The proposed solution is well-placed within the existing literature, and the authors clearly outline how the approach is conceptually solving the current issues by combining depth de-correlation and previous depth aggregation, where none of them can exist alone without the other. Finally, the conducted experiments are quite extensive and provide enough empirical support to the goodness of the proposed methodology.

Critical weaknesses

I think there are two main concerns regarding the paper, where the most critical is presentation-wise, while the other (minor) aspect is about some of the presented results.

In the former case, I admit it was not completely easy to assess the whole methodology in Section 4. There are some terminologies and notations that, to my understanding, were used without proper introduction. Even considering the page limitations, I would have appreciated an introductory section with some useful background to introduce the whole problem and the existing issues in the related literature.

In the latter case, a minor aspect I would like to further understand is about results from Table 2. In particular, from what I see, memory and time consumption are not quite limited for the proposed DeCORA methodology, which always (or quite always) settles within the worst methodologies in the benchmark. I understand there might be some performance-memory/time consumption trade-off to consider, but still I would ask the authors to further expand on this aspect, understanding if (maybe) exploiting deeper layers of recommendation architectures would come at the expense of higher memory and time footprint.

Questions for authors

I have two main questions, especially related to the second weakness I raised.

  1. Can authors further elaborate on why DeCORA is always (or quite always) placing within the last ranks in terms of memory and time consumption? Is there any performance-efficiency trade-off to be met?

  2. This is more of a comment, but I would like to ask the authors if they are familiar with other similar situations in recommender systems where depth might be a limiting factor. For instance, the literature shows how propagation depth in graph neural networks for recommendation (and not only for recommendation) is a limiting factor, where models do not propagate messages for more than 3 layers. This is generally known as the oversmoothing problem, and does not only affect GNNs for recommendation. Are the authors familiar with this problem and, in case, do they know if their proposed approach might be applied to this same problem and scenario?

3 comments

We sincerely thank the reviewer for the positive assessment and recognition of our contributions. Below we address each concern; all revisions are highlighted in blue in the updated manuscript.

W1: Background and notation

Thank you for noting where the definitions and notation could be more explicit. We have addressed these presentation points with targeted clarifications; these notation edits are minimal and do not alter the method or theorem statements:

  • Task background (Sec. 1, ¶1). The Introduction now opens with the setting: in modern recommendation backbones, especially for CTR prediction, heterogeneous inputs are embedded into a unified set of tokens, following recent backbones, and processed by stacked feature-interaction layers, so increasing depth produces a sequence of depth-indexed representations of the same tokens.
  • Open issue in the literature (Sec. 1, ¶2). Existing depth-connection methods (DenseFormer, mHC, ORU, AttnRes) were developed mainly for language and vision models; on recommendation backbones their gains are uneven, ranging from slight losses to moderate improvements (Table 2; Tables 9–12).
  • Notation (Sec. 3). Terms and symbols (e.g., backbone layers, prediction head, tokens, per-token batch statistics) are now defined at first use.

We hope this makes the problem and the motivation of DeCORA much easier to follow.

W2 & Q1: Memory and time

Yes, DeCORA incurs a training overhead over vanilla (+8.7~18.5% time/epoch; +0.6~1.2 GB GPU memory across 16 settings), but its cost is comparable to AttnRes's while achieving better AUC and Logloss in all 16 settings (Sec. 4.3). At inference, the correlation loss is removed; only lightweight prefix aggregation and adapters remain. The newly added table below reports inference GPU memory (GB) / total inference time (min) for RankMixer and WuKong at 16× on Avazu and Criteo, measured under the same hardware and evaluation setup.

MethodRankMixer (Avazu)RankMixer (Criteo)WuKong (Avazu)WuKong (Criteo)
Vanilla1.9 / 6.62.4 / 9.31.7 / 2.22.0 / 4.8
DenseFormer2.3 / 6.82.6 / 9.82.0 / 2.32.1 / 5.0
mHC2.4 / 6.92.6 / 9.92.0 / 2.32.2 / 5.0
AttnRes2.3 / 7.02.5 / 10.02.1 / 2.32.2 / 5.1
DeCORA2.3 / 6.92.5 / 9.92.0 / 2.32.2 / 5.0

DeCORA adds 0.1–0.4 GB and 0.1–0.6 min over vanilla. Crucially, at inference it is within 0.1 GB and 0.1 min of the training-lighter DenseFormer in all four settings, and uses no more memory or time than mHC or AttnRes in any of them, while achieving better AUC and Logloss (Table 2; Appendix A.3). Thus, its measured accuracy gains come at a real but competitive resource cost. As a further check, the newly added DenseNet baseline [1] (not DenseFormer) uses more training memory and time yet yields lower AUC than DeCORA in all eight matched settings (Appendix A.6, Table 18).

Q2: Depth limitation in GNN-based recommenders (over-smoothing)

Thank you for the insightful comment. Yes, we are aware of over-smoothing, which typically limits GNN-based recommenders to a few propagation layers. As an additional experiment, we apply DeCORA to LightGCN [2] on Amazon-Book with 3 and 8 layers:

LightGCNRecall@20NDCG@20HitRate@20
3 layers0.04070.03180.2353
w/ DenseFormer0.04350.03360.2622
w/ mHC0.04670.03690.2973
w/ DeCORA0.05010.04090.3135
8 layers0.03640.02640.2119
w/ DenseFormer0.04510.03600.2750
w/ mHC0.04890.03950.3098
w/ DeCORA0.05440.04520.3326

Vanilla LightGCN degrades with depth, whereas DeCORA benefits from it, improving Recall@20 over vanilla by 49% at 8 layers and outperforming DenseFormer and mHC. That said, we do not claim that DeCORA resolves over-smoothing itself: over-smoothing concerns node representations within a layer, while DeCORA acts across layers, so the two are distinct. This is also consistent with prior findings that deep GCNs can still degrade despite dense skip connections [4], and that their degradation also involves feature over-correlation [3]. The results are added to Appendix A.6.


References

[1] Densely Connected Convolutional Networks. In CVPR 2017.

[2] LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. In SIGIR 2020.

[3] Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective. In KDD 2022.

[4] Simple and Deep Graph Convolutional Networks. In ICML 2020.

Dear Authors,

Thank you for your rebuttal and your modifications to the paper, which I carefully assessed.

  • W1. The modifications you made to the paper regarding background and notations are sufficient in my opinion; I think now it is easier to follow the narrative of the methodology throughout the paper.

  • W2 + Q1. Yes, I acknowledge the proposed methodology is on par with AttnRes (the main competitor) on time efficiency and memory consumption; the additional results you reported about inference time provide further support for this.

  • Q2. Thanks for these additional experiments. As I said, this was more of a comment really, but it was interesting to see how your approach can improve over LightGCN. However, I agree with you these results cannot alone ensure the over-smoothing problem is solved.

All in all, I am satisfied with the rebuttal and the modifications to the paper, and I do not have further comments or doubts to discuss.

AuthorsOpenReview ↗

Thank you for the positive feedback

Dear Reviewer,

Thank you very much for your time, positive feedback, and thoughtful evaluation throughout the process. We are glad that our clarifications and additional results successfully addressed your concerns.

Best regards,

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.