- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RECAP, a parameter-efficient framework for CTR prediction based on estimator scaling. The paper argues that increasing feature interaction capacity yields diminishing returns, while averaging multiple estimators can further improve predictive performance. RECAP proposes three mechanisms: ensemble distillation across independently trained models, exponential moving averaging over training trajectories, and aggregation over inference-time routes within a weight-shared recursive backbone. The theoretical analysis demonstrates averaging gains to non-shared predictive variance. Experiments on five CTR benchmarks demonstrate competitive performance.
Critical strengths
- The paper investigates estimator scaling as an alternative to continuously increasing feature interaction capacity. This perspective offers a potentially useful direction for improving mature CTR architectures.
- The paper evaluates RECAP on five CTR benchmarks and compares it against a wide range of existing methods. RECAP achieves the highest reported AUC among the compared baselines across all five datasets.
Critical weaknesses
- The motivation and design of recursive LoRA aggregation are not sufficiently justified. The paper does not clearly explain why recursive LoRA aggregation is necessary. In particular, it remains unclear how the model handles potential inconsistencies between their intermediate representations, and what encourages different adapters to learn complementary information rather than similar prediction functions. These issues raise concerns about the effectiveness and necessity of the proposed architecture.
- The theoretical analysis provides limited support for the proposed method. The theoretical analysis mainly explains the general benefits of averaging estimators with non-shared predictive variation. However, these results are not specific to recursive models or LoRA-based route aggregation. The same analysis could apply to conventional ensembles or combinations of different architectures. Therefore, the theoretical results do not adequately justify why the proposed recursive architecture is preferable to alternative ensemble designs.
Questions for authors
- How does the proposed architecture ensure diversity among different LoRA routes?
- What is the specific advantage of recursive LoRA aggregation over alternative estimator constructions?
5 comments
Author Response - Part 2
[W2] The theoretical analysis provides limited support for the proposed method. The theoretical analysis mainly explains the general benefits of averaging estimators with non-shared predictive variation. However, these results are not specific to recursive models or LoRA-based route aggregation. The same analysis could apply to conventional ensembles or combinations of different architectures. Therefore, the theoretical results do not adequately justify why the proposed recursive architecture is preferable to alternative ensemble designs.
We thank the reviewer for highlighting the connection between the theoretical analysis and the architectural design. We have strengthened this connection by adding a new subsection 4.2, “Why Recursive LoRA Routing?”, containing a proposition on the predictive variation of recursive LoRA routes, with the full proof and parameter-budget analysis in Appendix B.4.
Role of the original analysis. Section 3 deliberately provides an architecture-independent framework for studying estimator scaling across independent models, training trajectories, and inference routes. Eqs. 3–6 relate local expected BCE gains to non-shared predictive variation, while Eq. 7 characterizes the residual variation available when combining estimator sources. The new analysis complements this framework by examining how RECAP’s recursive adapter choices produce the predictive variation that aggregation exploits.
New architecture-specific analysis. For a fixed input and finite recursion depth, under smooth core updates and sufficiently small effective adapter perturbations, the new proposition establishes
where is route ’s effective adapter perturbation at tower and step , and measures its influence on the final prediction through the subsequent recursive updates and readout. This identifies when different adapter selections generate different predictions: their differences must act along directions to which the recursive predictor is sensitive. Reusing an adapter at different steps can therefore have different predictive effects because the intermediate states and subsequent transformations differ.
The proposition further connects the resulting route-logit variance to the local BCE improvement of mean-logit aggregation relative to the average loss of the constituent routes. Importantly, it accommodates predefined routes with state-dependent adapter mixtures, retaining dependencies across steps and towers rather than assuming independent route sampling.
Parameter efficiency and comparative evidence. The accompanying parameter accounting makes the architectural trade-off explicit. For the simplified two-tower formulation with one core matrix per tower and rank- adapters per tower, the core matrices and adapter banks contain parameters, independent of recursion depth and the number of evaluated routes. This component-level count excludes embeddings, routers, gates, and readouts. Recursive sharing avoids separate core matrices at every depth, while additional route evaluations reuse the same embedding table, backbone, and adapter bank. They increase computation, but do not require additional copies of these stored parameters.
Together, the new analysis and the empirical results provide complementary justification: the analysis characterizes how recursive LoRA routing produces predictive diversity at a controlled storage cost, while the route-aggregation gains across all five datasets and the accuracy–resource comparisons establish its practical value. We do not claim that the theory uniquely selects recursive LoRA over every alternative; it now directly characterizes the mechanism underlying RECAP rather than relying solely on a general ensemble-variance argument.
Author Response - Part 3
[Q1] How does the proposed architecture ensure diversity among different LoRA routes?
RECAP encourages diversity among LoRA routes through three mechanisms: state-dependent sparse routing, stochastic expert dropout during training, and constrained route selection at inference.
- State-dependent Routing selects the top two experts from a bank of eight at each recursive step, conditioned on the current hidden state, step position, and recursion-depth budget. Different inputs and intermediate states can therefore activate different expert combinations, exposing the adapters to different training examples and computation paths.
- Leave-one-out route dropout randomly disables one expert per tower during training, encouraging the model to learn alternative computation paths rather than relying exclusively on a particular expert.
- Constrained inference routing evaluates multiple predefined expert restrictions, including leave-one-out and single-expert routes. Each route starts from the same initial state and executes (T=4) recursive steps, producing distinct candidate predictions that are subsequently aggregated.
Empirical Evidence of Diversity: To verify that these mechanisms successfully induce diversity, we explicitly measure the non-shared predictive variation (variance) and the resulting AUC gain from aggregation across all five datasets:
| Dataset | Estimator | Variance | AUC Gain |
|---|---|---|---|
| Avazu | Seeds | 0.098 | 0.336 |
| Trajectory | 0.008 | 0.052 | |
| Routes | 0.009 | 0.042 | |
| Criteo | Seeds | 0.033 | 0.145 |
| Trajectory | 0.012 | 0.081 | |
| Routes | 0.002 | 0.007 | |
| ML-1M | Seeds | 0.183 | 0.300 |
| Trajectory | 0.169 | 0.392 | |
| Routes | 0.026 | 0.037 | |
| KDD12 | Seeds | 0.127 | 0.625 |
| Trajectory | 0.019 | 0.152 | |
| Routes | 0.018 | 0.060 | |
| KKBox | Seeds | 0.230 | 0.481 |
| Trajectory | 0.150 | 0.448 | |
| Routes | 0.086 | 0.212 |
The results confirm that LoRA routes exhibit measurable non-shared predictive variation and provide positive aggregation gains on all five datasets. Route averaging improves AUC by 0.042, 0.007, 0.037, 0.060, and 0.212 percentage points on Avazu, Criteo, ML-1M, KDD12, and KKBox, respectively. While independent seeds generally expose greater predictive variation, LoRA routes provide an additional estimator source within a shared-parameter architecture. These findings are consistent with our theoretical analysis (Eqs. 3–6), which relates the benefits of estimator averaging to the non-shared predictive variation available across estimators.
[Q2] What is the specific advantage of recursive LoRA aggregation over alternative estimator constructions?
The specific advantage of recursive LoRA aggregation is that it constructs alternative prediction paths at a fixed recursion depth while reusing the same embedding table, core interaction weights, and adapter banks. Recursion shares the interaction parameters across steps, while LoRA routing varies the effective transformations within the computation. Additional routes therefore increase inference computation without requiring additional copies of the stored backbone.
We compare the complete RECAP configuration against multiple-branch FCN, TabM, and multi-depth ensembling. RECAP achieves the highest reported AUC across all five datasets, improving over the strongest listed alternative by 0.22, 0.09, 0.14, 0.56, and 0.34 AUC percentage points on Avazu, Criteo, ML-1M, KDD12, and KKBox, respectively.
| Method | Avazu | Criteo | ML-1M | KDD12 | KKBox |
|---|---|---|---|---|---|
| FCN | 79.66 | 81.62 | 90.74 | 80.52 | 85.01 |
| FCN + Multiple Branches | 79.70 | 81.67 | 90.74 | 80.83 | 85.69 |
| TabM (Batch Ensemble) | 79.60 | 81.56 | 90.66 | 80.85 | 85.58 |
| RRT + Multi-Depth Ensemble | 79.43 | 81.57 | 90.45 | 80.84 | 85.21 |
| RECAP | 79.92 | 81.76 | 90.88 | 81.41 | 86.06 |
These results support the predictive effectiveness of the complete RECAP design. The route-aggregation diagnostics separately show that its route predictions contain useful differences that improve performance when combined. The architectural advantage is therefore the ability to expose an additional estimator source within a shared parameterization, rather than a guarantee of maximal diversity or universal superiority over other ensemble constructions.
Author Response - Part 1
Thank you for your constructive review. We are glad you found our investigation into estimator scaling to be a useful and promising direction for improving mature CTR architectures. We also appreciate your recognition of the strong empirical evidence across the five CTR benchmarks.
Below, we address your questions regarding the design, theoretical motivation, and practical advantages of recursive LoRA aggregation compared to alternative ensembling methods. We hope the inclusion of the new theoretical framework and explicit empirical comparisons thoroughly resolves your concerns regarding the necessity of our architecture, and that these revisions encourage you to reconsider your score.
We thank the reviewer for highlighting these architectural questions. The core motivation for recursive LoRA aggregation is to obtain the predictive benefits of an ensemble without the parameter cost of maintaining multiple independent models. Recursion and LoRA play complementary roles to achieve this: recursion shares the core interaction weights across computation steps, while lightweight adapters introduce variations in those transformations.
Intermediate representation consistency: Regarding the question about how we align intermediate representations across routes. The key architectural detail here is that we do not need to. Aggregation occurs strictly at the final prediction-logit level, not by averaging or exchanging intermediate hidden states. Within a recursive step, selected adapters modify the shared transformation applied to the current state. Each inference route then maintains its own independent hidden-state trajectory—starting from the same reused input embedding and executing the complete recursion under its prescribed route override. Because intermediate states are never mixed across routes, cross-route hidden-state alignment is not required.
Encouraging complementary predictions: To prevent different adapters from collapsing to similar prediction functions, we rely on state-dependent sparse routing and leave-one-out expert dropout during training. By randomly exposing the backbone to alternative adapter configurations during training, we actively discourage the network from relying exclusively on a single expert path. At inference, predefined route overrides (such as excluding a specific expert or forcing the selection of a designated expert) force the model down these alternatively trained computation paths. Together, these mechanisms successfully encourage predictive diversity.
Empirical evidence of diversity: The ultimate test of whether routes learn complementary information is whether aggregating them improves the final prediction. Our route-aggregation diagnostics explicitly measure this. Route aggregation alone improves AUC by 0.042, 0.007, 0.037, 0.060, and 0.212 percentage points on Avazu, Criteo, ML-1M, KDD12, and KKBox, respectively. These consistent positive gains prove that the evaluated routes provide useful, non-shared predictive differences, rather than collapsing into identical functions.
Ultimately, we do not claim that recursion is the only way to ensemble; rather, it is the highly parameter-efficient construction RECAP uses to extract these measured route-aggregation benefits from a shared parameterization.