acceptodds
Under review as a conference paper at ICLR 2027

Beyond Interaction Capacity: Estimator Scaling with Recursive Models for CTR Prediction

Abstract

Click-Through Rate (CTR) prediction, a core task in recommendation and advertising systems, relies on modeling interactions among sparse categorical features. Explicit cross networks are a central paradigm for CTR prediction, and recent progress has largely come from increasing the interaction capacity of a single predictor through deeper cross networks and more expressive cross operators. We revisit whether continually increasing interaction capacity remains the most effective way to improve predictive performance, and find that its benefits quickly exhibit diminishing returns even as capacity continues to grow. This motivates a complementary scaling direction that we call estimator scaling, where additional resources are used to incorporate multiple related estimators rather than only enlarging a single predictor. Through theoretical analysis and diagnostic experiments, we show that the gains from estimator scaling are governed by the amount of non-shared predictive variation available across estimators. However, exploiting this variation naively can be expensive: independently trained models provide substantial estimator diversity but require deployment cost to grow with ensemble size. This motivates a parameter-efficient realization of estimator scaling that can incorporate diversity from multiple estimator sources without maintaining multiple full models. Building on this view, we introduce RECursive Averaged Predictor (RECAP), a parameter-efficient recursive CTR model that operationalizes estimator scaling at three levels: distillation across independently trained models, exponential moving averaging over training trajectories, and aggregation over inference-time routes within a weight-shared recursive backbone. This design improves predictive performance without requiring parameter growth proportional to the ensemble size. Experiments across multiple CTR benchmarks show that RECAP approaches the accuracy of a five-model ensemble of a strong cross-network baseline at one fifth of its deployed parameters, while doubling the baseline's interaction capacity recovers only a third of that gain. Overall, these results establish new state-of-the-art predictive performance on standard benchmarks, while placing the proposed approach on a favorable performance–parameter Pareto frontier.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Click-through rate prediction, Recursive models, Knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_L6kEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RECAP, a parameter-efficient framework for CTR prediction based on estimator scaling. The paper argues that increasing feature interaction capacity yields diminishing returns, while averaging multiple estimators can further improve predictive performance. RECAP proposes three mechanisms: ensemble distillation across independently trained models, exponential moving averaging over training trajectories, and aggregation over inference-time routes within a weight-shared recursive backbone. The theoretical analysis demonstrates averaging gains to non-shared predictive variance. Experiments on five CTR benchmarks demonstrate competitive performance.

Critical strengths
  1. The paper investigates estimator scaling as an alternative to continuously increasing feature interaction capacity. This perspective offers a potentially useful direction for improving mature CTR architectures.
  2. The paper evaluates RECAP on five CTR benchmarks and compares it against a wide range of existing methods. RECAP achieves the highest reported AUC among the compared baselines across all five datasets.
Critical weaknesses
  1. The motivation and design of recursive LoRA aggregation are not sufficiently justified. The paper does not clearly explain why recursive LoRA aggregation is necessary. In particular, it remains unclear how the model handles potential inconsistencies between their intermediate representations, and what encourages different adapters to learn complementary information rather than similar prediction functions. These issues raise concerns about the effectiveness and necessity of the proposed architecture.
  2. The theoretical analysis provides limited support for the proposed method. The theoretical analysis mainly explains the general benefits of averaging estimators with non-shared predictive variation. However, these results are not specific to recursive models or LoRA-based route aggregation. The same analysis could apply to conventional ensembles or combinations of different architectures. Therefore, the theoretical results do not adequately justify why the proposed recursive architecture is preferable to alternative ensemble designs.
Questions for authors
  1. How does the proposed architecture ensure diversity among different LoRA routes?
  2. What is the specific advantage of recursive LoRA aggregation over alternative estimator constructions?
5 comments
AuthorsOpenReview ↗

Author Response - Part 1

Thank you for your constructive review. We are glad you found our investigation into estimator scaling to be a useful and promising direction for improving mature CTR architectures. We also appreciate your recognition of the strong empirical evidence across the five CTR benchmarks.

Below, we address your questions regarding the design, theoretical motivation, and practical advantages of recursive LoRA aggregation compared to alternative ensembling methods. We hope the inclusion of the new theoretical framework and explicit empirical comparisons thoroughly resolves your concerns regarding the necessity of our architecture, and that these revisions encourage you to reconsider your score.

[W1] The motivation and design of recursive LoRA aggregation are not sufficiently justified. The paper does not clearly explain why recursive LoRA aggregation is necessary. In particular, it remains unclear how the model handles potential inconsistencies between their intermediate representations, and what encourages different adapters to learn complementary information rather than similar prediction functions. These issues raise concerns about the effectiveness and necessity of the proposed architecture.

We thank the reviewer for highlighting these architectural questions. The core motivation for recursive LoRA aggregation is to obtain the predictive benefits of an ensemble without the parameter cost of maintaining multiple independent models. Recursion and LoRA play complementary roles to achieve this: recursion shares the core interaction weights across computation steps, while lightweight adapters introduce variations in those transformations.

Intermediate representation consistency: Regarding the question about how we align intermediate representations across routes. The key architectural detail here is that we do not need to. Aggregation occurs strictly at the final prediction-logit level, not by averaging or exchanging intermediate hidden states. Within a recursive step, selected adapters modify the shared transformation applied to the current state. Each inference route then maintains its own independent hidden-state trajectory—starting from the same reused input embedding and executing the complete recursion under its prescribed route override. Because intermediate states are never mixed across routes, cross-route hidden-state alignment is not required.

Encouraging complementary predictions: To prevent different adapters from collapsing to similar prediction functions, we rely on state-dependent sparse routing and leave-one-out expert dropout during training. By randomly exposing the backbone to alternative adapter configurations during training, we actively discourage the network from relying exclusively on a single expert path. At inference, predefined route overrides (such as excluding a specific expert or forcing the selection of a designated expert) force the model down these alternatively trained computation paths. Together, these mechanisms successfully encourage predictive diversity.

Empirical evidence of diversity: The ultimate test of whether routes learn complementary information is whether aggregating them improves the final prediction. Our route-aggregation diagnostics explicitly measure this. Route aggregation alone improves AUC by 0.042, 0.007, 0.037, 0.060, and 0.212 percentage points on Avazu, Criteo, ML-1M, KDD12, and KKBox, respectively. These consistent positive gains prove that the evaluated routes provide useful, non-shared predictive differences, rather than collapsing into identical functions.

Ultimately, we do not claim that recursion is the only way to ensemble; rather, it is the highly parameter-efficient construction RECAP uses to extract these measured route-aggregation benefits from a shared parameterization.

AuthorsOpenReview ↗

Author Response - Part 2

[W2] The theoretical analysis provides limited support for the proposed method. The theoretical analysis mainly explains the general benefits of averaging estimators with non-shared predictive variation. However, these results are not specific to recursive models or LoRA-based route aggregation. The same analysis could apply to conventional ensembles or combinations of different architectures. Therefore, the theoretical results do not adequately justify why the proposed recursive architecture is preferable to alternative ensemble designs.

We thank the reviewer for highlighting the connection between the theoretical analysis and the architectural design. We have strengthened this connection by adding a new subsection 4.2, “Why Recursive LoRA Routing?”, containing a proposition on the predictive variation of recursive LoRA routes, with the full proof and parameter-budget analysis in Appendix B.4.

Role of the original analysis. Section 3 deliberately provides an architecture-independent framework for studying estimator scaling across independent models, training trajectories, and inference routes. Eqs. 3–6 relate local expected BCE gains to non-shared predictive variation, while Eq. 7 characterizes the residual variation available when combining estimator sources. The new analysis complements this framework by examining how RECAP’s recursive adapter choices produce the predictive variation that aggregation exploits.

New architecture-specific analysis. For a fixed input and finite recursion depth, under smooth core updates and sufficiently small effective adapter perturbations, the new proposition establishes

zk(x)=z0(x)+∑q,t⟨Gq,t(x),Uq,t(k)(x)⟩F+OT(ε2)z_k(x) = z_0(x) + \sum_{q,t} \left\langle G_{q,t}(x), U^{(k)}_{q,t}(x) \right\rangle_F + \mathcal{O}_T(\varepsilon^2)

where Uq,t(k)U^{(k)}_{q,t} is route kk’s effective adapter perturbation at tower qq and step tt, and Gq,tG_{q,t} measures its influence on the final prediction through the subsequent recursive updates and readout. This identifies when different adapter selections generate different predictions: their differences must act along directions to which the recursive predictor is sensitive. Reusing an adapter at different steps can therefore have different predictive effects because the intermediate states and subsequent transformations differ.

The proposition further connects the resulting route-logit variance to the local BCE improvement of mean-logit aggregation relative to the average loss of the constituent routes. Importantly, it accommodates predefined routes with state-dependent adapter mixtures, retaining dependencies across steps and towers rather than assuming independent route sampling.

Parameter efficiency and comparative evidence. The accompanying parameter accounting makes the architectural trade-off explicit. For the simplified two-tower formulation with one d×dd\times d core matrix per tower and MM rank-rr adapters per tower, the core matrices and adapter banks contain 2d2+4Mdr2d^2+4Mdr parameters, independent of recursion depth and the number of evaluated routes. This component-level count excludes embeddings, routers, gates, and readouts. Recursive sharing avoids separate core matrices at every depth, while additional route evaluations reuse the same embedding table, backbone, and adapter bank. They increase computation, but do not require additional copies of these stored parameters.

Together, the new analysis and the empirical results provide complementary justification: the analysis characterizes how recursive LoRA routing produces predictive diversity at a controlled storage cost, while the route-aggregation gains across all five datasets and the accuracy–resource comparisons establish its practical value. We do not claim that the theory uniquely selects recursive LoRA over every alternative; it now directly characterizes the mechanism underlying RECAP rather than relying solely on a general ensemble-variance argument.

AuthorsOpenReview ↗

Author Response - Part 3

[Q1] How does the proposed architecture ensure diversity among different LoRA routes?

RECAP encourages diversity among LoRA routes through three mechanisms: state-dependent sparse routing, stochastic expert dropout during training, and constrained route selection at inference.

  • State-dependent Routing selects the top two experts from a bank of eight at each recursive step, conditioned on the current hidden state, step position, and recursion-depth budget. Different inputs and intermediate states can therefore activate different expert combinations, exposing the adapters to different training examples and computation paths.
  • Leave-one-out route dropout randomly disables one expert per tower during training, encouraging the model to learn alternative computation paths rather than relying exclusively on a particular expert.
  • Constrained inference routing evaluates multiple predefined expert restrictions, including leave-one-out and single-expert routes. Each route starts from the same initial state and executes (T=4) recursive steps, producing distinct candidate predictions that are subsequently aggregated.

Empirical Evidence of Diversity: To verify that these mechanisms successfully induce diversity, we explicitly measure the non-shared predictive variation (variance) and the resulting AUC gain from aggregation across all five datasets:

DatasetEstimatorVarianceAUC Gain
AvazuSeeds0.0980.336
Trajectory0.0080.052
Routes0.0090.042
CriteoSeeds0.0330.145
Trajectory0.0120.081
Routes0.0020.007
ML-1MSeeds0.1830.300
Trajectory0.1690.392
Routes0.0260.037
KDD12Seeds0.1270.625
Trajectory0.0190.152
Routes0.0180.060
KKBoxSeeds0.2300.481
Trajectory0.1500.448
Routes0.0860.212

The results confirm that LoRA routes exhibit measurable non-shared predictive variation and provide positive aggregation gains on all five datasets. Route averaging improves AUC by 0.042, 0.007, 0.037, 0.060, and 0.212 percentage points on Avazu, Criteo, ML-1M, KDD12, and KKBox, respectively. While independent seeds generally expose greater predictive variation, LoRA routes provide an additional estimator source within a shared-parameter architecture. These findings are consistent with our theoretical analysis (Eqs. 3–6), which relates the benefits of estimator averaging to the non-shared predictive variation available across estimators.

[Q2] What is the specific advantage of recursive LoRA aggregation over alternative estimator constructions?

The specific advantage of recursive LoRA aggregation is that it constructs alternative prediction paths at a fixed recursion depth while reusing the same embedding table, core interaction weights, and adapter banks. Recursion shares the interaction parameters across steps, while LoRA routing varies the effective transformations within the computation. Additional routes therefore increase inference computation without requiring additional copies of the stored backbone.

We compare the complete RECAP configuration against multiple-branch FCN, TabM, and multi-depth ensembling. RECAP achieves the highest reported AUC across all five datasets, improving over the strongest listed alternative by 0.22, 0.09, 0.14, 0.56, and 0.34 AUC percentage points on Avazu, Criteo, ML-1M, KDD12, and KKBox, respectively.

MethodAvazuCriteoML-1MKDD12KKBox
FCN79.6681.6290.7480.5285.01
FCN + Multiple Branches79.7081.6790.7480.8385.69
TabM (Batch Ensemble)79.6081.5690.6680.8585.58
RRT + Multi-Depth Ensemble79.4381.5790.4580.8485.21
RECAP79.9281.7690.8881.4186.06

These results support the predictive effectiveness of the complete RECAP design. The route-aggregation diagnostics separately show that its route predictions contain useful differences that improve performance when combined. The architectural advantage is therefore the ability to expose an additional estimator source within a shared parameterization, rather than a guarantee of maximal diversity or universal superiority over other ensemble constructions.

Reviewer_VrAtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper starts from a simple observation: making CTR interaction networks deeper or wider eventually stops helping much

Instead, the authors propose scaling the number of predictors being combined. RECAP brings together information from different training seeds, checkpoints, and inference paths using ensemble distillation, EMA, and LoRA-based route averaging

Experiments on five CTR datasets show that RECAP performs strongly and nearly matches a five-model ensemble while keeping roughly a single model’s parameter footprint

Critical strengths

The paper clearly shows that making interaction networks deeper or wider can quickly saturate, and frames estimator diversity as a complementary scaling direction. This is consequential because it offers a practical alternative to designing increasingly complex feature-interaction operators

RECAP nearly matches a five-model FCN ensemble while using roughly 1/5 of its deployed parameters

Critical weaknesses

The efficiency comparison only measures deployed parameters\textbf{deployed parameters}. It excludes the cost of training multiple teachers and the additional inference compute from evaluating multiple routes. Without FLOPs, latency, or throughput measurements, the claimed serving advantage is not fully supported

Potential staleness under distribution shift\textbf{staleness under distribution shift} is not evaluated. RECAP relies on a high-decay EMA and distillation from independently trained teachers, which may slow adaptation when user populations or item distributions change rapidly. The paper only evaluates static offline benchmarks; chronological or streaming experiments are needed to establish its suitability for production CTR serving

Questions for authors

Does the efficiency comparison include training the teacher ensemble, generating soft labels, and training the student?

Were the data splits chronological? How does the high-decay EMA and teacher distillation behave under rapid distribution shift?

Is the 81.76 vs. 81.74 AUC difference between RECAP and FCN+EMA+KD statistically significant under the same training and parameter budget?

5 comments
AuthorsOpenReview ↗

Author Response - Part 1

Thank you for the detailed and constructive review. We are ve rygrateful for your assessment that framing estimator diversity as a complementary scaling direction is a consequential and practical alternative to designing increasingly complex feature-interaction operators. We are also glad you found our parameter-efficiency results compelling.

To address your concerns, we have added comprehensive online inference FLOPs measurements, as well as rigorous chronological and streaming evaluations to test RECAP under actual distribution shift.

We hope these new streaming benchmarks and efficiency metrics fully address your concerns and enable a re-evaluation of our work.

[W1] The efficiency comparison only measures deployed parameters. It excludes the cost of training multiple teachers and the additional inference compute from evaluating multiple routes. Without FLOPs, latency, or throughput measurements, the claimed serving advantage is not fully supported

We thank the reviewer for highlighting this important distinction. We agree that deployed parameter counts alone do not capture the full computational profile, and that both inference compute and teacher training costs are critical factors in CTR systems.

Training vs. Serving Compute: We acknowledge that maintaining a teacher ensemble and generating soft labels incurs additional training compute. However, in production CTR pipelines, where both teachers and students are typically updated continuously to track shifting distributions, often with heavier teachers updated at a lower cadence—training and serving FLOPs are fundamentally asymmetric. Training updates are throughput-oriented and asynchronous, whereas online serving operates at orders-of-magnitude higher query volume under strict tail-latency and memory constraints. Because each model checkpoint serves a massive number of inference requests before the next update, training overhead is heavily amortized, making per-query serving footprint and compute the primary deployment bottlenecks.

Online Inference Compute (FLOPs): To directly evaluate online serving cost, we have measured per-sample inference FLOPs. A key architectural advantage of RECAP is its shared embedding stage: while each member of an independent FCN ensemble must store its own embedding table and execute a separate embedding lookup and interaction pass, RECAP computes the input embedding once and reuses it across both recursive towers and all 12 inference routes. Adding routes therefore avoids duplicating embedding storage and lookup, confining additional arithmetic strictly to the compact recursive predictor.

We have added the comparison below and included both the table and the corresponding accuracy-compute Pareto analysis in Appendix G.

ModelTest AUCMFLOPs
FCN81.6219
FCN 2-seed ensemble81.7238
FCN 4-seed ensemble81.7776
FCN 5-seed ensemble81.7895
TabM81.5668
LoopCTR81.3652
RECAP (Ours)81.7639

On Criteo, the full RECAP configuration achieves 81.76% AUC at 39 MFLOPs per sample, accounting for its recursive and multi-route computation. This is approximately twice the arithmetic cost of a single FCN, but close to that of a two-seed FCN ensemble (38 MFLOPs), with higher reported AUC (81.76% versus 81.72%). Compared with the four- and five-seed FCN ensembles, RECAP requires approximately 49% and 59% fewer FLOPs, respectively, with AUC differences of only 0.01 and 0.02 percentage points. RECAP also achieves higher reported AUC with fewer FLOPs than the evaluated TabM and LoopCTR configurations.

AuthorsOpenReview ↗

Author Response - Part 2

[W2] Potential staleness under distribution shift is not evaluated. RECAP relies on a high-decay EMA and distillation from independently trained teachers, which may slow adaptation when user populations or item distributions change rapidly. The paper only evaluates static offline benchmarks; chronological or streaming experiments are needed to establish its suitability for production CTR serving

[Q2] Were the data splits chronological? How does the high-decay EMA and teacher distillation behave under rapid distribution shift?

The data splits for the experiments in the paper are all made randomly following the FuxiCTR and BARS frameworks. The original experiments used random 8:1:1 train/validation/test splits. To explicitly evaluate temporal generalization and adaptation under distribution shift, we have conducted additional chronological and streaming experiments on Avazu, including an EMA-decay sensitivity analysis.

Chronological evaluation: For the chronological experiment, we train on October 21–26 (23.87M examples), use October 27 for validation and early stopping (3.23M examples), and evaluate on October 28–30 (13.34M examples). On this future-period holdout, RECAP improves AUC over FCN by 0.92 percentage points and reduces logloss from 0.3893 to 0.3838.

Train: Oct 21–26, 23.87M rows (59%). Validation: Oct 27, 3.23M rows (8%), used for early stopping. Test: Oct 28–30, 13.34M rows (33%).

MethodAUCLogloss
FCN73.810.3893
RECAP (ours)74.730.3838

Streaming evaluation: For the streaming experiment, we pretrain the model on October 21–22 (9.46M examples), use October 23 for validation and early stopping (3.87M examples), and process October 24–30 as a chronological stream containing 27.10M examples across 168 hourly windows. For each hour, we first record predictions for its examples and then perform one incremental training pass on that hour’s labeled data. Teachers do not access an hour’s labels before its predictions are recorded. RECAP without EMA achieves 76.77% AUC, compared with 76.61% for plain FCN, while reducing logloss from 0.3877 to 0.3867. Across the tested EMA decays from 0.1 to 0.8, RECAP retains similar performance, with AUC between 76.72% and 76.77%.

Pretrain: Oct 21–22, 9.46M rows. Validation: Oct 23, 3.87M rows, used for early stopping Stream: Oct 24–30, 27.10M rows across 168 hours in time order. Each hour is predicted first, then used for one incremental training pass. Teachers never see an hour’s labels before that hour is predicted.

MethodAUCLogloss
FCN76.610.3877
RECAP (No EMA)76.770.3867
RECAP (EMA 0,1)76.770.3867
RECAP (EMA 0.2)76.770.3867
RECAP (EMA 0.4)76.760.3867
RECAP (EMA 0.5)76.760.3867
RECAP (EMA 0.6)76.750.3868
RECAP (EMA 0.7)76.740.3868
RECAP (EMA 0.8)76.720.3869
RECAP (EMA 0.99)76.180.3901
RECAP (EMA 0.999)75.370.3964
RECAP (EMA 0.9995)74.980.3999
RECAP (EMA 0.9999)74.240.4061

EMA horizon and adaptation. The sweep confirms that large EMA decays can introduce substantial prediction lag in this streaming setting. When EMA is updated after each optimizer step, its half-life is h1/2=ln⁡2−ln⁡βh_{1/2}=\frac{\ln 2}{-\ln \beta}, measured in optimizer updates. At approximately 16–17 updates per hour of stream data, the offline decay of (β=0.9995)(\beta=0.9995) has a half-life of approximately 1,386 updates, equivalent to 3.4–3.6 days of data. For (β=0.9999)(\beta=0.9999), the half-life increases to approximately 17–18 days, exceeding the entire seven-day stream. These long averaging horizons are consistent with the observed degradation. In comparison, (β=0.8)(\beta=0.8) has a half-life of approximately three updates and retains performance close to serving the current model weights.

These experiments extend the evaluation beyond static random splits and show that RECAP can generalize to future observations and learn incrementally from a chronological stream. They also identify a practical configuration requirement: the averaging horizon must be matched to the update cadence. In the evaluated stream, serving current weights or using a shorter EMA horizon preserves adaptation, whereas transferring the high-decay offline setting unchanged substantially reduces performance. We have added the results for the chronological and streasming experiments in Appendix H in the paper.

AuthorsOpenReview ↗

Author Response - Part 3

[Q1] Does the efficiency comparison include training the teacher ensemble, generating soft labels, and training the student?

The reviewer is correct that the reported efficiency comparisons measure deployment efficiency rather than total end-to-end compute. Specifically, we evaluate deployed parameters and inference FLOPs, explicitly excluding the offline costs of training the teacher ensemble and the student. We made this distinction because RECAP relies on a deliberate system trade-off: investing additional offline training computation to maximize predictive accuracy while preserving a lightweight online serving footprint. This trade-off is most attractive when serving costs dominate the overall computational budget. In such settings, the additional cost of each training cycle can be amortized over a large number of subsequent predictions. This motivation is supported by industrial CTR studies. For example, Ling et al. (WWW 2017) estimated that a 0.1% improvement in prediction accuracy in their Bing Ads production setting could yield hundreds of millions of dollars in additional earnings. This is a deployment-specific estimate, rather than a direct conversion from our offline AUC gains to revenue. Nevertheless, it illustrates why modest predictive improvements can justify additional computational investment at scale. Our results characterize the accuracy–deployment-cost trade-off; they do not establish a reduction in total training-and-serving cost.

Reference:

  • Ling et al.: Model Ensemble for Click Prediction in Bing Search Ads (Ling et al., WWW 2017)

[Q3] Is the 81.76 vs. 81.74 AUC difference between RECAP and FCN+EMA+KD statistically significant under the same training and parameter budget?

Yes, the difference is statistically significant. We provide a direct comparison of RECAP and FCN enhanced with EMA and ensemble distillation, including variability across repeated runs and a statistical assessment of their difference.

On Criteo, the enhanced FCN baseline achieves (81.74 \pm 0.001) AUC (%), while RECAP achieves (81.76 \pm 0.007). Using the reported means and sample standard deviations, and assuming five independent runs per method, a two-sided Welch’s t-test yields an approximate (p=0.0028). The estimated improvement is 0.02 AUC percentage points, with a 95% confidence interval of [0.011, 0.029]. This provides direct statistical support for the incremental gain beyond the improvements already obtained through EMA and ensemble distillation.

Furthermore, this statistical significance is not limited to Criteo. We have added a unified comparison across all five datasets to Appendix F, reporting mean AUC, standard deviations, and the corresponding two-sided Welch pp-values. As shown below, the incremental gains are statistically significant (p<0.01p < 0.01) on four out of the five benchmarks, making the magnitude and reliability of RECAP’s architectural contribution explicit.

Method / statisticAvazuCriteoML-1MKDD12KKBox
FCN79.68 ±\pm 0.05781.62 ±\pm 0.00690.72 ±\pm 0.03180.82 ±\pm 0.05685.72 ±\pm 0.044
FCN + EMA + Distillation79.83 ±\pm 0.00581.74 ±\pm 0.00190.84 ±\pm 0.01081.37 ±\pm 0.00185.98 ±\pm 0.012
RECAP79.93 ±\pm 0.03481.76 ±\pm 0.00790.87 ±\pm 0.02681.44 ±\pm 0.03086.07 ±\pm 0.008
Gain over FCN + EMA + Distillation+0.10+0.02+0.03+0.07+0.09
Two-sided Welch pp-value0.00250.00280.0590.00642.4×10−62.4\times10^{-6}
Reviewer_gipgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how to improve CTR prediction when increasing feature-interaction capacity yields diminishing returns. It proposes estimator scaling, motivated by an analysis linking the benefits of prediction averaging to non-shared predictive variation. The proposed RECAP model combines weight-shared recursive LCN and ECN branches with ensemble distillation, exponential moving weight averaging, and inference-time route aggregation using lightweight LoRA adapters. The authors evaluate RECAP on five public CTR benchmarks through baseline comparisons, diagnostic experiments, component ablations, and performance–parameter analyses. The reported results show improved AUC across the evaluated datasets and performance approaching a five-model FCN ensemble with approximately one-fifth of its deployed parameters.

Critical strengths

This paper has clear motivation supported by diagnostic experiments. The paper compares increasing interaction depth and the number of interaction towers with model ensembling. These experiments motivate the question of how additional resources should be allocated once further increases in interaction capacity yield diminishing returns.

Critical weaknesses
  1. This paper misses comparisons with closely related methods. The experimental evaluation omits several particularly relevant baselines, including the CTR ensemble-distillation approach of Zhu et al. (CIKM 2020), TabM, and LoopCTR. These methods address closely related questions concerning ensemble distillation, parameter-efficient ensembling, or recursive computation. Comparisons with them would help establish RECAP’s advantages over existing approaches.

  2. The incremental improvement over the strengthened FCN baseline remains uncertain. FCN with EMA and ensemble distillation already achieves an AUC of 81.74%, compared with 81.76% for the full RECAP model, a difference of only 0.02 percentage points. The paper does not report variability estimates or a statistical significance analysis for this comparison, making it difficult to assess the reliability of this incremental gain.

Questions for authors
  1. Could you add experimental comparisons with the CTR ensemble-distillation approach of Zhu et al. (CIKM 2020), TabM, and LoopCTR, along with variability estimates and significance tests for the ablation results? These experiments would be helpful, particularly for assessing the incremental gain over FCN with EMA and distillation.

  2. Could you add experiments measuring practical inference costs, including throughput and latency? These results would help assess whether the additional execution time from recursive computation and multi-route inference limits the feasibility of online deployment.

  3. Could you carefully revise the writing and presentation? Some figures give the impression of AI-generated graphics, with dense text inside rounded rectangular boxes. For example, in Figure 2, the “weights shared across steps” labels in the LCN and ECN modules overlap the arrows. Citation formatting also disrupts readability: Table 1 and the model list on page 8, lines 395–402, use constructions such as “Model Author et al. (Year).” A more natural presentation would be “Model (Author et al., Year).”

  4. Could you release the source code and experimental configurations? Without these resources, researchers in the community will face substantial difficulty reproducing the results and building on this work.

If the concerns raised above are adequately addressed, I would consider raising my score.

5 comments
AuthorsOpenReview ↗

Author Response - Part 1

Thank you for the detailed and constructive review. We are glad you found our core motivation clear and appreciated the diagnostic experiments comparing interaction-capacity scaling with model ensembling.

To address the concerns, we have conducted extensive additional experiments to directly answer your requests: we have added comparisons against the specific baselines you requested (Zhu et al., TabM, and LoopCTR), provided variance and statistical significance tests across all five datasets, evaluated practical inference costs via FLOPs analysis, corrected the figure and citation formatting, and provided an anonymous link to our codebase.

We hope these comprehensive updates fully resolve your concerns and enable a re-evaluation of our work. Below, we respond to your points in detail.

[W1] This paper misses comparisons with closely related methods. The experimental evaluation omits several particularly relevant baselines, including the CTR ensemble-distillation approach of Zhu et al. (CIKM 2020), TabM, and LoopCTR. These methods address closely related questions concerning ensemble distillation, parameter-efficient ensembling, or recursive computation. Comparisons with them would help establish RECAP’s advantages over existing approaches.

We thank the reviewer for suggesting these closely related baselines. We have added experimental comparisons with Zhu et al. (CIKM 2020), TabM, and LoopCTR. Their results are reported below and included in the revised Table 1.

MethodAvazuCriteoML-1MKDD12KKBox
FCN79.6681.6290.7480.8685.72
Zhu et al.79.6581.6590.8581.2185.06
LoopCTR78.8581.3689.9780.4683.60
TabM79.6081.5690.6680.8585.58
RECAP79.9281.7690.8881.4186.06

RECAP achieves the highest reported AUC among the compared methods on all five datasets. The added baselines address three closely related directions; their relationship to RECAP is summarized below.

LoopCTR (Tang et al., 2026): LoopCTR proposes the idea of applying a recursive Transformer block, using hyper-connected residuals and a mixture-of-experts (MoE) architecture to strengthen its representation capacity. Supervision at multiple loop depths supports both multi-loop inference and a zero-loop mode that bypasses the recurrent block. RECAP shares the principle of recursive parameter reuse and multi-depth supervision. However, in RECAP, we use a two tower approach where each tower shares its core interaction weights across recursive steps, while selections from a shared adapter bank introduce route-specific, low-rank variations in the effective transformations. This constructs multiple related predictors within a common recursive backbone, whose outputs are aggregated at inference. Combined with ensemble distillation and EMA, this design enables RECAP to study estimator scaling across independent training runs, optimization trajectories, and recursive routes.

TabM (Gorishniy et al., ICLR 2025): TabM proposes an ensemble method by using multiple MLP predictors within one model, sharing most weight matrices and using lightweight, member-specific multiplicative adapters. These predictors are trained jointly, and their predictions are combined at inference. This closely relates to RECAP’s goal of obtaining multiple estimators with limited parameter growth. RECAP constructs its estimators through different LoRA-conditioned routes within a shared recursive cross-network backbone. It further combines these route estimators with information from independently trained teachers and the optimization trajectory, studying the incremental benefit of each source.

Zhu et al. (CIKM 2020): This work transfers knowledge from an ensemble of CTR predictors into a single student model. It introduces sample-adaptive teacher gating and early stopping based on the distillation loss, enabling the student to benefit from multiple teachers while serving only the student at inference. RECAP builds on ensemble distillation as one source of estimator information and combines it with two additional mechanisms: EMA across the training trajectory and aggregation across LoRA-conditioned recursive routes. Our contribution is both analytical and empirical. Our central question is when and why increasing the number of estimators helps relative to increasing interaction capacity. For mean-logit ensembles, Eqs. 3–6 relate the local expected BCE improvement to curvature-weighted, non-shared predictive variance; Fig. 1$(c)$ additionally demonstrates an empirical correspondence with AUC gains. This provides an analytical perspective on estimator-scaling saturation that is qualitatively consistent with the diminishing gains from additional teachers reported by Zhu et al. The ablations and baseline comparisons provide direct evidence of the benefit beyond distillation alone.

AuthorsOpenReview ↗

Author Response - Part 2

[W2] The incremental improvement over the strengthened FCN baseline remains uncertain. FCN with EMA and ensemble distillation already achieves an AUC of 81.74%, compared with 81.76% for the full RECAP model, a difference of only 0.02 percentage points. The paper does not report variability estimates or a statistical significance analysis for this comparison, making it difficult to assess the reliability of this incremental gain.

To assess the benefit of RECAP over the strengthened FCN baseline (FCN+EMA+Distillation), we provide the following direct comparison across all five datasets. Results are mean AUC (%) (±)(\pm) sample standard deviation. To rigorously assess the reliability of these incremental gains, we also report the pp-values from two-sided Welch’s tt-tests.

Method / statisticAvazuCriteoML-1MKDD12KKBox
FCN79.68 ±\pm 0.05781.62 ±\pm 0.00690.72 ±\pm 0.03180.82 ±\pm 0.05685.72 ±\pm 0.044
FCN + EMA + Distillation79.83 ±\pm 0.00581.74 ±\pm 0.00190.84 ±\pm 0.01081.37 ±\pm 0.00185.98 ±\pm 0.012
RECAP79.93 ±\pm 0.03481.76 ±\pm 0.00790.87 ±\pm 0.02681.44 ±\pm 0.03086.07 ±\pm 0.008
Gain over FCN + EMA + Distillation+0.10+0.02+0.03+0.07+0.09
Two-sided Welch pp-value0.00250.00280.0590.00642.4×10−62.4\times10^{-6}

On Criteo, RECAP’s additional gain of 0.02 AUC percentage points has a 95% confidence interval of [0.011, 0.029], with p=0.0028p=0.0028. This directly supports the reliability of the improvement over the enhanced FCN baseline under the stated analysis.

RECAP also achieves higher mean AUC on the other four datasets, with statistically significant improvements on Avazu, KDD12, and KKBox (p<0.01p<0.01). Thus, the additional predictive benefit is supported statistically on four benchmarks rather than being limited to the original Criteo point comparison.

We have included the repeated-run comparison and corresponding uncertainty estimates in Appendix F, “Comparison with FCN Enhanced by EMA and Ensemble Distillation.”

[Q1] Could you add experimental comparisons with the CTR ensemble-distillation approach of Zhu et al. (CIKM 2020), TabM, and LoopCTR, along with variability estimates and significance tests for the ablation results? These experiments would be helpful, particularly for assessing the incremental gain over FCN with EMA and distillation.

We have addressed these requests through the additional experiments described in W1 and W2. The revised Table 1 includes comparisons with Zhu et al., TabM, and LoopCTR, while Appendix F reports the repeated-run comparison against FCN enhanced with EMA and ensemble distillation, including standard deviations and two-sided Welch tests. In particular, the Criteo comparison gives p=0.0028p=0.0028, with a 95% confidence interval of [0.011, 0.029] AUC percentage points for RECAP’s additional gain.

[Q2] Could you add experiments measuring practical inference costs, including throughput and latency? These results would help assess whether the additional execution time from recursive computation and multi-route inference limits the feasibility of online deployment.

We thank the reviewer for raising this concern and would like to highlight a key architectural advantage of RECAP, its shared embedding stage. Embedding tables account for most of the model parameters in the CTR setting. Each independently trained FCN ensemble member retains its own learned embedding table and executes its own embedding lookup and interaction network. RECAP computes the input embedding once and reuses it across both recursive towers and all inference routes. Adding routes therefore avoids duplicating the largest parameter component and repeating the input-embedding stage, while the additional computation takes place in the recursive predictor with LoRA adapters. This provides a parameter-efficiency advantage alongside the overall FLOP savings.

We have added the comparison below and included both the table and the corresponding accuracy–compute Pareto analysis in Appendix G.

ModelTest AUCMFLOPs
FCN81.6219
FCN 2-seed ensemble81.7238
FCN 4-seed ensemble81.7776
FCN 5-seed ensemble81.7895
TabM81.5668
LoopCTR81.3652
RECAP (Ours)81.7639

On Criteo, RECAP achieves 81.76% AUC at 39 MFLOPs per sample, compared with 81.62% at 19 MFLOPs for a single FCN. Its arithmetic cost is close to that of the two-seed FCN ensemble (39 versus 38 MFLOPs), while providing higher reported AUC (81.76% versus 81.72%). Relative to the four- and five-seed ensembles, RECAP requires approximately 49% and 59% fewer FLOPs, respectively, with AUC differences of only 0.01 and 0.02 percentage points. RECAP also achieves higher reported AUC with fewer FLOPs than the evaluated TabM and LoopCTR configurations.

AuthorsOpenReview ↗

Author Response - Part 3

[Q3] Could you carefully revise the writing and presentation? Some figures give the impression of AI-generated graphics, with dense text inside rounded rectangular boxes. For example, in Figure 2, the “weights shared across steps” labels in the LCN and ECN modules overlap the arrows. Citation formatting also disrupts readability: Table 1 and the model list on page 8, lines 395–402, use constructions such as “Model Author et al. (Year).” A more natural presentation would be “Model (Author et al., Year).”

We have addressed the identified presentation issues in the revised manuscript. The figure layouts have been simplified to reduce text density and visual clutter, with improved spacing, typography, and label placement. In Figure 2, we have repositioned the “weights shared across steps” annotations in both the LCN and ECN modules to eliminate their overlap with the arrows.

We have also standardized model citations to the format “Model (Author et al., Year)”, including those in Table 1 and the model list originally appearing on page 8, lines 395–402. We have reviewed the remaining figures, tables, and text for consistent formatting and readability.

[Q4] Could you release the source code and experimental configurations? Without these resources, researchers in the community will face substantial difficulty reproducing the results and building on this work.

We strongly agree with the importance of reproducibility. The implementation is available through an anonymous link at the end of the abstract, and the configurations used for the main experiments are documented in the Appendix E. After the review period, we will publicly release the source code and accompanying experimental configurations to support reproduction of our results and enable the community to build on this work.

Reviewer_nqbDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies diminishing returns from increasing feature-interaction capacity in CTR models and explores incorporating multiple related estimators as a complementary source of improvement. The authors use a local BCE analysis and predictive-variance diagnostics to explain aggregation gains from different estimator sources. They propose RECAP, which combines ensemble distillation, EMA, and aggregation across LoRA-defined routes within a weight-shared recursive two-tower backbone. The experiments cover five public datasets, component ablations, scaling behavior, and hyperparameter sensitivity, demonstrating a favorable trade-off between predictive accuracy and deployed parameter count.

Critical strengths
  1. The analysis and cross-dataset diagnostics support a useful scaling direction. The authors examine interaction-capacity scaling, different estimator sources, and the residual variance after combining them. Comparisons that apply estimator scaling to the standard FCN backbone also support gains beyond the particular recursive architecture.

  2. The method demonstrates an attractive accuracy–parameter trade-off. Multi-level aggregation on a shared backbone approaches the performance of a larger FCN ensemble at approximately a single-model parameter count, providing a useful option for CTR systems with constrained deployment parameter budgets.

Critical weaknesses
  1. The additional value of the recursive and route designs over a simpler enhanced model is not sufficiently established. In Table 3, FCN with both EMA and ensemble distillation reaches 81.74 on Criteo, while the corresponding value for full RECAP in Table 1 is 81.76. This is a point comparison across two tables, and neither the uncertainty of that difference across repeated runs nor the corresponding results on the other datasets are reported. The evidence supports estimator scaling overall, but leaves the magnitude of the extra benefit from the more complex implementation unclear.

  2. Key computational settings for the main experiments are insufficiently specified. Section 4 defines the number of inference routes, recursion depth, and training across multiple depths, but the main text and Appendix D do not fully provide the configurations used for the main results, the depth weights, or the route sampling rules. These settings affect performance and the computation required by the configuration shown in Figure 4.

Questions for authors
  1. What is the uncertainty across repeated runs in the difference between full RECAP (Table 1) and FCN with EMA and ensemble distillation (Table 3)? Table 1 already reports a two-tailed t-test over five runs; has this cross-table difference been tested in the same way? Is this additional gain also observed on the other datasets?

  2. Please specify the inference route count K, recursion depth T, training depth set and its weights, and the route selection rules used during training and inference for the main results.

2 comments
AuthorsOpenReview ↗

Author Response - Part 1

Thank you for the thoughtful and constructive feedback. We are glad you found our analysis and cross-dataset diagnostics to strongly support estimator scaling as a useful and complementary direction for CTR models. We are also encouraged by your positive assessment of RECAP's accuracy-parameter trade-off, and agree that it provides a highly practical option for CTR systems operating under constrained deployment budgets.

Below, we respond to your specific questions regarding the incremental value of the recursive architecture and the exact computational settings used in our experiments. We hope these additional results and clarifications address your concerns and enable a re-evaluation of our work. All feedback has been incorporated into the revised manuscript and appendix.

[W1] The additional value of the recursive and route designs over a simpler enhanced model is not sufficiently established. In Table 3, FCN with both EMA and ensemble distillation reaches 81.74 on Criteo, while the corresponding value for full RECAP in Table 1 is 81.76. This is a point comparison across two tables, and neither the uncertainty of that difference across repeated runs nor the corresponding results on the other datasets are reported. The evidence supports estimator scaling overall, but leaves the magnitude of the extra benefit from the more complex implementation unclear.

[Q1] What is the uncertainty across repeated runs in the difference between full RECAP (Table 1) and FCN with EMA and ensemble distillation (Table 3)? Table 1 already reports a two-tailed t-test over five runs; has this cross-table difference been tested in the same way? Is this additional gain also observed on the other datasets?

We thank the reviewer for pointing this out and agree that the original cross-table comparison did not sufficiently quantify the benefit of RECAP over FCN equipped with both EMA and ensemble distillation. To resolve this, we now provide a unified comparison across all five datasets, reporting mean AUC (%) and standard deviation over five independently seeded runs per method. To rigorously assess the reliability of these incremental gains, we also report the pp-values from two-sided Welch’s tt-tests.

Method / statisticAvazuCriteoML-1MKDD12KKBox
FCN79.68 ±\pm 0.05781.62 ±\pm 0.00690.72 ±\pm 0.03180.82 ±\pm 0.05685.72 ±\pm 0.044
FCN + EMA + Distillation79.83 ±\pm 0.00581.74 ±\pm 0.00190.84 ±\pm 0.01081.37 ±\pm 0.00185.98 ±\pm 0.012
RECAP79.93 ±\pm 0.03481.76 ±\pm 0.00790.87 ±\pm 0.02681.44 ±\pm 0.03086.07 ±\pm 0.008
Gain over FCN + EMA + Distillation+0.10+0.02+0.03+0.07+0.09
Two-sided Welch pp-value0.00250.00280.0590.00642.4×10−62.4\times10^{-6}

RECAP achieves higher mean AUC than the strengthened FCN baseline on all five datasets, with improvements ranging from 0.02 to 0.10 percentage points. In particular, the additional gains on Avazu, KDD12, and KKBox are 0.10, 0.07, and 0.09 points, respectively, extending the evidence beyond the original Criteo comparison.

These results support the additional predictive value of the complete RECAP design beyond the substantial gains already obtained by applying EMA and ensemble distillation to FCN. We have added this unified comparison to Appendix F, reporting mean AUC and standard deviations across runs to make both the magnitude of the improvement and the variability of each method explicit.

AuthorsOpenReview ↗

Author Response - Part 2

[W2] Key computational settings for the main experiments are insufficiently specified. Section 4 defines the number of inference routes, recursion depth, and training across multiple depths, but the main text and Appendix D do not fully provide the configurations used for the main results, the depth weights, or the route sampling rules. These settings affect performance and the computation required by the configuration shown in Figure 4.

[Q2] Please specify the inference route count K, recursion depth T, training depth set and its weights, and the route selection rules used during training and inference for the main results.

We thank the reviewer for requesting these execution details. We have added a dedicated Appendix E, “Main Experimental Configuration,” documenting the hyperparameters and execution settings used for the main results. It specifies the inference route count and recursion depth, the training depths and their loss weights, and the route-selection procedures during training and inference.

Recursion budgets and depth weights. Inference uses K=12K=12 routes at recursion depth T=4T=4. During training, all three depths in Dtrain=4,8,12\mathcal{D}_{\mathrm{train}}={4,8,12} are evaluated in every minibatch, with equal weights α4=α8=α12=1/3\alpha_4=\alpha_8=\alpha{12}=1/3. Thus, training includes all three depths rather than sampling a single depth per update.

Training-time routing. Each of the two towers has a bank of eight rank-96 LoRA adapters, shared across recursive steps. At each step, a learned MLP router with hidden dimension 64 selects and mixes the top two eligible adapters using softmax temperature 1.0. The router is conditioned on the current state, step, phase, update norm, and depth budget. With probability 0.5 per tower and rollout, one uniformly selected adapter is disabled for that rollout; routing then operates over the remaining adapters.

Inference-time routing. The evaluated routes use predefined overrides that specify the eligible adapters and the number selected per step. An override may exclude a designated adapter while retaining top-2 selection, or restrict the rollout to a single designated adapter. The sample's embedding is computed once and reused across route evaluations. For each tower and prescribed override, recursion restarts from the same initial state and executes four steps. The final-step tower logits are retained and combined after the route evaluations.

To support reproducibility, we have also added an anonymous link to the codebase at the end of the abstract. The key computational settings for the RECAP backbone are summarized below:

HyperparameterValue
Dataset splitRandom 8:1:1 (train / validation / test)
Seeds2021, 2022, 2023, 2024, 2025
Epochs100
Early stoppingPatience 4 on validation AUC
OptimizerAdam
Initial learning rate10−310^{-3}
LR scheduleCosine over 20 epochs to 10−510^{-5}
Number of towers2
Adapter bank size8 per tower; shared across recursive steps
LoRA rank96
RouterLearned MLP with hidden dimension 64
Training recursion depths{4,8,12}\{4,8,12\}; all three unrolled every batch
Depth loss weightsα4=α8=α12=1/3\alpha_4=\alpha_8=\alpha_{12}=1/3
Training route dropout0.5 per tower and rollout
Inference recursion depth4
Distillation teacher5-seed FCN ensemble, mean logit
EMA decay0.9995
KD weight1.0
Inference route count12

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Manuscript Revisions

We thank all reviewers for their constructive comments and suggestions for new experiments, which we believe have substantially improved our manuscript.

We have highlighted all revisions in the updated manuscript in blue for ease of review. Below, we summarize the key modifications mapped to each reviewer's feedback.

Reviewer nqbD

  • W1 & Q1: Added a multi-seed comparison and statistical significance analysis (Appendix F) demonstrating RECAP's consistent, significant gains over the FCN + EMA + Distillation baseline across multiple datasets.
  • W2 & Q2: Added a dedicated "Main Experimental Configuration" section (Appendix E) detailing exact hyperparameter values, depth weights, routing rules, and inference execution settings to ensure full reproducibility.

Reviewer gipg

  • W1 & Q1: Expanded Table 1 to include experimental comparisons against the three requested baselines: Zhu et al. (CIKM 2020), TabM, and LoopCTR.
  • W2 & Q1: Added Appendix F to provide variance estimates and two-sided Welch’s tt-tests, confirming the statistical reliability of RECAP's incremental gains over the enhanced FCN baseline.
  • Q2: Added an online inference compute analysis (Appendix G), explicitly measuring MFLOPs to demonstrate RECAP's serving efficiency and accuracy–compute Pareto advantages.
  • Q3: Refined Figure 2 to eliminate text overlap and visual clutter, and standardized all model citations to the requested Model (Author et al., Year) format.

Reviewer VrAt

  • W1 & Q1: Added an inference FLOP analysis (Appendix G) to explicitly evaluate online serving compute, highlighting the efficiency gained from RECAP's shared embedding stage.
  • W2 & Q2: Conducted new chronological and streaming evaluations (Appendix H), including an EMA-decay sensitivity sweep, to validate RECAP's adaptation capabilities under temporal distribution shift.
  • Q3: Included rigorous multi-seed statistical testing (Appendix F) confirming the significance of RECAP's gains over FCN + EMA + KD.

Reviewer L6kE

  • W1, W2 & Q2: Introduced a new theoretical proposition (Section 4.2) and full proof/parameter-budget analysis (Appendix B.4) that mathematically justifies how recursive LoRA routing produces the necessary predictive diversity for estimator scaling.
  • Q1: Expanded the architectural description in the main text to explicitly detail how state-dependent sparse routing and route dropout ensure adapter diversity during training and inference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.