- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ELiT proposes a lightweight CTR Transformer for heterogeneous features, motivated by the authors' hypothesis that feature-field structure largely determines token interactions. It combines local and global token mixing, sample-dependent expert routing, and shared-weight masked SwiGLU. The paper derives an equivalent mixing computation and proposes an attention-approximation guarantee under explicit assumptions. Public and industrial experiments report stronger ranking performance at comparable dense parameter counts; repeated public runs and online A/B tests broaden the evidence. Component ablations support the combined architecture, while design sweeps examine ranking quality alongside parameter size and analytical storage savings.
Critical strengths
-
The architecture diagram and comparison with self-attention make the roles of local mixing, global mixing and feed-forward processing clear.
-
The mixer combines separate local transforms with input-dependent global interactions. Its shared-transform construction has an exact algebraic rearrangement that avoids forming the large recombination matrix.
-
The public and industrial comparisons relate ranking quality to model size, and repeated public runs report variability. Online tests add evidence on business metrics.
-
The component comparisons test simpler mixing and feed-forward alternatives, giving direct evidence for retaining these choices in the tested model.
Critical weaknesses
In the part of Theorem 1 in Sec. 3.3 where the residual dimension exceeds the number of active experts, the proof treats the routed output as the best approximation within the expert span without justification.
The oracle calculation therefore does not establish the claimed tight bound for the actual routed outputs in Eq. (8). Please distinguish the oracle result from a guarantee for the proposed router.
The case where the residual dimension does not exceed the expert count remains an existence result with unrestricted mixing coefficients.
Questions for authors
-
Which model, layer and sample population generated Fig. 2, and what does the reference matrix in Theorem 1 represent relative to the mixer in Eq. (6)? This would clarify the intended attention approximation.
-
Sec. 3.4 claims lower inference latency, while Figure 4b presents storage accounting. Please report matched masked or vanilla-SwiGLU serving timings or restrict the claim to storage savings.
Thank you for the response. My concern on efficiency (Q2) and the significance part of Q3 are resolved; the theorem concern (Q1) is only partly resolved, since the revised bound holds only under optimal experts and routing, and the baseline tuning details, industrial variability, and online A/B significance remain unaddressed. Based on the response, I am raising my score from 1 to 2.