acceptodds
Under review as a conference paper at ICLR 2027

ELiT: An Expressive Lite Transformer for CTR Prediction with Error-Bounded Attention Approximation

Abstract

Recommender systems (RS) have long been dominated by the Embedding & MLP paradigm, which is now approaching its intrinsic performance ceiling. Motivated by the scaling laws of LLMs, recent studies have begun to adapt Transformer backbones to RS. However, direct transplantation incurs prohibitive parameter overhead, as heterogeneous input features rule out globally shared QKV projections and necessitate token-level parameterization, thereby causing parameter explosion, whereas lightweight variants typically sacrifice global interaction modeling. To resolve this trade-off, we propose the Expressive Lite Transformer (ELiT), a Transformer variant tailored to heterogeneous features that reconciles strong representational power. Our design is motivated by a key empirical insight: in RS, inter-token affinities are governed predominantly by feature fields rather than by specific feature values, rendering the inter-token affinity relationships largely invariant across different inputs and thus permitting aggressive pruning of redundant self-attention (SA) parameters. Concretely, ELiT introduces: (i) a progressive two-stage token mixing mechanism, fortified by a sparse MoE module, that substitutes for SA in modeling inter-token relations, thereby achieving sample-adaptive global interaction with negligible overhead; (ii) theoretical guarantees that the approximation error of the attention matrix remains tightly bounded, rigorously justifying its lightweight yet highly expressive architecture; (iii) a masked SwiGLU module that leverages the parameters freed by the simplified SA in (i) to upgrade the standard FFN for higher representational capacity. To avoid the substantial inference overhead of vanilla SwiGLU’s dual weight matrices, we replace them with a shared FP16 matrix and binary masks, alleviating the memory-bandwidth bottleneck at negligible computational cost. Extensive experiments on large-scale industrial and public datasets and online A/B tests consistently show that ELiT outperforms SOTA baselines, achieving an exceptional efficiency–expressiveness trade-off in real-world RS.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Scaling Laws, Ranking Model, Recommender System

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_otekOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ELiT proposes a lightweight CTR Transformer for heterogeneous features, motivated by the authors' hypothesis that feature-field structure largely determines token interactions. It combines local and global token mixing, sample-dependent expert routing, and shared-weight masked SwiGLU. The paper derives an equivalent mixing computation and proposes an attention-approximation guarantee under explicit assumptions. Public and industrial experiments report stronger ranking performance at comparable dense parameter counts; repeated public runs and online A/B tests broaden the evidence. Component ablations support the combined architecture, while design sweeps examine ranking quality alongside parameter size and analytical storage savings.

Critical strengths
  1. The architecture diagram and comparison with self-attention make the roles of local mixing, global mixing and feed-forward processing clear.

  2. The mixer combines separate local transforms with input-dependent global interactions. Its shared-transform construction has an exact algebraic rearrangement that avoids forming the large recombination matrix.

  3. The public and industrial comparisons relate ranking quality to model size, and repeated public runs report variability. Online tests add evidence on business metrics.

  4. The component comparisons test simpler mixing and feed-forward alternatives, giving direct evidence for retaining these choices in the tested model.

Critical weaknesses

In the part of Theorem 1 in Sec. 3.3 where the residual dimension exceeds the number of active experts, the proof treats the routed output as the best approximation within the expert span without justification.

The oracle calculation therefore does not establish the claimed tight bound for the actual routed outputs in Eq. (8). Please distinguish the oracle result from a guarantee for the proposed router.

The case where the residual dimension does not exceed the expert count remains an existence result with unrestricted mixing coefficients.

Questions for authors
  1. Which model, layer and sample population generated Fig. 2, and what does the reference matrix in Theorem 1 represent relative to the mixer in Eq. (6)? This would clarify the intended attention approximation.

  2. Sec. 3.4 claims lower inference latency, while Figure 4b presents storage accounting. Please report matched masked or vanilla-SwiGLU serving timings or restrict the claim to storage savings.

Reviewer_Jyc5OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

ELiT replaces heterogeneous self-attention with token-specific local mixing followed by an MoE-personalized global mixing matrix. It reallocates saved parameters to a masked SwiGLU that shares an FP16 matrix through binary masks. The paper reports public, industrial, scaling, ablation, and online A/B-test results and presents a theorem intended to bound attention-matrix approximation error.

Critical strengths
  1. ELiT targets a consequential efficiency/expressiveness trade-off and reports consistent offline improvements on multiple datasets at comparable parameter and FLOP budgets.
  2. The evaluation includes public data, industrial data, component ablations, scaling sweeps, and online deployment results.
Critical weaknesses
  1. The central theorem does not match the implemented MoE. Equation (8) uses nonnegative softmax weights summing to one, so outputs lie in a convex hull; the proof assumes arbitrary real coefficients and reasons about a linear span. Thus r≤k does not yield exact representation. In the r>k case, the proof bounds only the residual but does not ensure that the mean matrix belongs to the expert span. The assumed low-dimensional/low-variance property is not measured.
  2. The efficiency claims lack direct evidence. With five masks, masked SwiGLU performs substantially more matrix computation than standard SwiGLU, yet no latency, throughput, or peak-memory measurements validate the claim that this overhead is negligible. Baseline tuning, industrial variability, and online-test significance are insufficiently reported.
Questions for authors
  1. Can you give a corrected theorem under convex-combination routing and empirically estimate residual variance and effective dimension?
  2. What are end-to-end latency, throughput, and memory results for masked SwiGLU and the full model?
  3. Were baselines tuned independently, and are gains significant against the strongest baseline rather than only GDCN?
1 comment

Thank you for the response. My concern on efficiency (Q2) and the significance part of Q3 are resolved; the theorem concern (Q1) is only partly resolved, since the revised bound holds only under optimal experts and routing, and the baseline tuning details, industrial variability, and online A/B significance remain unaddressed. Based on the response, I am raising my score from 1 to 2.

Reviewer_ZFSiOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes ELiT, a lightweight Transformer variant for CTR prediction in recommender systems. The core observation is that, unlike in NLP, inter-token affinities in recommender systems are governed mainly by feature fields rather than specific feature values, so attention patterns are largely invariant across different inputs. Based on this, ELiT replaces self-attention with a Learnable & Personalized (LP) Token Mixing layer: the full recombination matrix is compressed via Kronecker decomposition into a two-stage mixing process, and the global mixing matrix G is personalized at the user level with a sparse MoE. On two public datasets (Criteo, RecFlow), two billion-scale industrial datasets, and in online A/B tests, ELiT consistently outperforms all baselines at comparable parameter budgets, with steeper scaling curves.

Critical strengths
  1. The design is validated on public datasets,billion-scale industrial datasets ,steeper scaling curves ,real online A/B gains.
  2. The paper correctly identifies that online RS serving is memory-bandwidth-bound rather than compute-bound, and trades a 1-bit mask per weight for halved weight-read volume. An elegant, deployment-aware design.
Critical weaknesses
  1. The assumptions of Theorem 1 (Ḡ and R(x) lying in an r-dimensional subspace with bounded variance) presuppose exactly the low-variance structure the theorem is supposed to support, yet the paper never measures them — there is no estimate of r or σ, and no curve of approximation error versus the number of experts k.

  2. Neither the masked SwiGLU nor the MoE routing overhead is backed by any measured latency/throughput/memory-bandwidth results, which is a significant gap for a paper motivated by online-serving efficiency.

Questions for authors
  1. Can you report measured inference latency/throughput and memory-bandwidth usage of ELiT vs. RankMixer/FAT/Transformer at matched parameter counts, with a separate breakdown for the masked SwiGLU and the MoE routing overhead?

  2. In the "w/o MSwiGLU" ablation in Table 4, what exactly replaces the module — a standard FFN or the vanilla SwiGLU?

  3. What are the total number of experts m, the number of activated experts k, and the router input? Is any load-balancing/auxiliary loss used? How sensitive are the results to k?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.