acceptodds
Under review as a conference paper at ICLR 2027

Maglev: Improving the Recommendation Scaling Laws via Unified Scaling

Abstract

Industrial recommenders face diminishing returns from dense scaling, while sparse and depth scaling make global feature interaction and distributed training pro- hibitively expensive. We introduce Maglev, a model–system co-design that decom- poses a recommender into hardware-local pipeline stages. It combines (1) Maglev Indexers, which learn capacity-bounded stage inputs, as soft projections within one scale-up domain or hard feature partitions across them; (2) Maglev Induct, which carries a low-rank residual across stages; and (3) Maglev Rail, a pipeline schedule that stays bubble-free with synchronous gradient clipping. We conduct an industrial-scale study that jointly scales sparse capacity and dense compute (28B samples, 700 features, four tasks). Applied to Wukong and DHEN, the strongest of 10 screened backbones, Maglev improves the scaling law along every axis: every Maglev configuration outperforms the best baseline configuration at equal or lower compute and sparse capacity, and the lead widens as sparse capacity and compute grow. The full system trains up to 1.3× faster than hybrid parallelism. On three public benchmarks, Maglev improves five further backbones in all 15 matched- compute settings. Maglev has been launched in three flagship recommenders at a major social media company, improving RLL by up to 0.4% in A/B tests.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Recommendation models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cgx3OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Maglev restructures recommendation models into hardware-local sequential stages. Soft Indexers learn bounded feature projections; Hard Indexers convert learned affinities into feature assignments that respect storage budgets and coverage. Induct carries a gated residual digest between stages, and Rail schedules stage-local embedding lookups to fill pipeline warm-up slots while retaining exact synchronous gradient clipping. The evaluation combines a 28B-sample proprietary study of Wukong and DHEN, five additional backbones on three public datasets, systems measurements, and reported production A/B tests. Table 2 reports 116 of 116 wins against comparable same-backbone frontiers and a median 0.15 percentage-point RLL margin. Public Soft-Indexer experiments use 12 paired seeds per setting, with positive mean gains in all 15 settings. The main contribution is a practical model and systems design that improves the observed quality and compute trade-off.

Critical strengths
  1. The evaluation supports a useful architectural contribution beyond one backbone or benchmark. Sections 4.2-4.3 compare quality under explicit compute and sparse-capacity budgets. Table 1 provides paired confidence intervals across five backbones and three datasets; Appendix A.12.2 additionally tests Hard Indexers, including an unfavorable FiGNN result on Criteo. Appendix A.11 checks non-interpolated frontiers, additional converged runs, and nearby budgets. These checks materially strengthen the argument that the gains are not simply an artifact of one selected comparison.
  2. The model and systems components address a coherent deployment constraint. Learned feature placement makes lookups local, Induct restores useful cross-stage conditioning, and Rail uses the resulting lookup independence to improve scheduling. The analysis explicitly limits zero bubbles to a specified latency model and excludes the common optimizer tail, while measured throughput includes that tail. Appendix A.4 reports 6.3%-11.7% schedule gains across 12 configurations, and A.12.5 separates the schedule contribution from sharding and communication overlap. This separation makes the practical contribution more credible than a single aggregate speedup.
Critical weaknesses
  1. The strongest scaling conclusions remain specific to a partially reproducible industrial campaign. Each grid configuration has one seed. The repeatability check in A.10.1 compares four-layer Wukong with eight-stage Maglev-DHEN at 38.6% different FLOPs on later data; the authors correctly state that it does not estimate uncertainty for the frontier. Thus, it cannot resolve the smallest ablation gains or uncertainty in fitted slope ratios. The public experiments offer useful complementary evidence but do not reproduce the industrial scaling regime. The appendix also omits consequential settings: industrial train/validation/test construction and temporal leakage controls, numerical optimizer and learning-rate settings, completion criteria, and some co-tuned widths. Public configurations list depth and Indexer width without the full model and training configuration. These omissions limit independent assessment of the scaling claims. A representative repeated, near-matched industrial contrast and complete public configurations would materially strengthen the evidence; repeating the entire industrial grid is not necessary.
  2. The representation analysis supports a coefficient approximation bound, but some interpretation goes beyond what it establishes. In Section 3.2, U = B-dagger W is called the optimal linear readout. Appendix A.9 shows that it minimizes ||W-BU||_F, whereas the stated output is ZBU; it does not generally minimize ||Z(W-BU)||_F. For example, with Z = [[2,1],[1,2]], B = [1,0]^T and W = [0,1]^T, the proposed U is 0, but prediction least squares gives U = 0.8. This does not invalidate the displayed upper bound for the chosen U, but the optimality claim should be qualified. Similarly, the spectral tail is a lower bound on coefficient residual, not automatically a lower bound on prediction error. A fixed width O also leaves the orientation of the learned subspace relevant. The Lipschitz statement in A.9 appropriately disclaims a full prediction guarantee, but an explicit mapping from the deployed nonlinear digest to the surrogate would be needed to justify stronger claims of recovered interactions.
Questions for authors
  1. Clarification of reproducibility: Please provide the complete public experiment configurations, including embedding dimensions, all backbone widths, Induct width, optimizer, learning rates, regularization, stopping rule and tuning search. For the industrial study, what are the temporal splits, history cutoffs and completion criteria? How are leakage from future events and overlap between training and evaluation controlled?
  2. Evidence that could affect the scaling assessment: Can you provide paired repetitions for one or two same-backbone, approximately matched-FLOP industrial comparisons, including a larger sparse-capacity setting? Please distinguish uncertainty of those fixed contrasts from the descriptive slopes fitted to the selected grid. This would help assess the smaller effects without requiring another full campaign.
  3. Clarification of the theory: Is U = B-dagger W intended to optimize the coefficient residual only? What conditions would make it optimal for the Z-weighted prediction objective, and how is B constructed from the actual gated stages? Please also separate the coefficient spectral-tail statement from a prediction-error floor.
  4. Clarification of implementation: A.7 distributes ordered sequence chunks across participating stages, while Section 3.1 states that each industrial feature field is assigned exactly once. What unit is partitioned for sequence fields, how do chunks contribute to the storage budgets, and how are stage order and unequal sequence lookup costs chosen? Does Listing 1's update of the digest before the current backbone block match the exact implementation described in Section 3.2?
Reviewer_v2k9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Maglev, a model-system jointly-designed framework for improving scaling efficiency and ability of industrial LLM-scale recommendation models. The authors identify the bottlenecks in the existing recommendation system architectures and why they face intrinsic challenges in scaling up sparse capacity and dense computation modules. The authors introduced decomposition of global all-vs-all feature interaction modeling with a few coordinated hardware-local stages in order to solve these problems. Their framework contains three main components: 1) Maglev Indexers learn feature affinities and transform the original feature set into stage-specific feature representations, either through soft indexers or hard indexers; 2) Maglev Induct introduces a low-rank gated residual mechanism for information propagation across different stages, acting as a makeup for cross-stage feature interactions lost in partitioning; 3) Maglev Rail improves training efficiency through pipelining and scheduling optimizations. The authors did extensive experiments on large-scale industrial recommendation scenarios, as well as a handful of public datasets and different architectures. Results show improvements in scaling behavior across sparse, dense and depth dimensions, and up to 1.3x improvement in training throughput. The authors also showed positive results in three real-world production A/B tests.

Critical strengths

1. Extensive empirical validation and practical significance

I appreciate that the authors have done extensive experimental evaluation of the proposed framework, and this is the strongest aspect I reckon as of this paper. They have conducted experiments across multiple recommendation system datasets, architectures, as well as an industrial-scale study, with reasonably careful control of computational budgets and experimental setups. I also appreciate the comprehensive ablation studies which addressed many of the concerns I initially had when reading the main text, especially regarding the effectiveness of feature partitioning, Maglev Induct and different system optimization decisions. Together with the real-world A/B tests, the experiments provide strong evidence that the proposed framework is practically effective. This is a major supporting evidence for the framework's strength.

2. A well-rounded model-system joint design

The proposed framework provides an interesting and practically meaningful approach to addressing the scaling bottlenecks of industrial recommendation systems. Instead of simply increasing the model size or introducing a more efficient interaction module, the authors jointly consider the model architecture, feature storage, communication overhead, and distributed training. Although some of the individual components are related to existing techniques, I like how they are combined into a framework that gained improvements in both predictive performance and training efficiency. I think this is a meaningful contribution to practical training of industry-scale recommender systems.

Critical weaknesses

1. Limited justification and interpretability of the feature assignment strategy

My main concern is that the affinity-based feature assignment strategy is not sufficiently justified from the perspective of preserving important feature interactions. The Hard Indexer uses the average absolute projection coefficients learned by the Soft Indexer to determine feature assignments. However, it remains unclear whether these marginal affinity scores can adequately reflect the importance of pairwise or higher-order interactions between features.

Specifically, the greedy coverage correction removes duplicate feature assignments to ensure that every feature is assigned to at least one stage. One particular failure case I can think of is that it may remove potentially important stage-specific assignments for features that interact strongly with multiple groups. This could make certain feature interactions more difficult to capture within individual stages or through cross-stage information propagation.

I acknowledge that the authors have conducted experiments comparing learned partitioning against random and importance-based strategies, as well as experiments on feature reuse in Appendix A.12.2. These results demonstrate the effectiveness of the proposed approach. However, I would still appreciate a more direct analysis of how the learned feature assignments relate to the underlying interaction structure. For example, analyzing whether features with stronger interactions across multiple groups benefit more from feature reuse could provide additional insights into the proposed design.

I consider this a limitation bevause feature partitioning is a central feature of Maglev, yet the current evidence set does not fully explain whether important feature interactions are successfully preserved, approximated, or even lost.

2. Limited theoretical understanding of the cross-stage interaction mechanism

The second concern I had, is that the theoretical analysis of Maglev Induct relies on a simplified linear low-rank approximation, while the actual architecture involves nonlinear interaction modules and gated residual updates. Although I understand that theoretically characterizing such architectures remains challenging, it is unclear to what extent the derived approximation bounds explain the information preservation properties of the actual Maglev Induct mechanism. Nevertheless, I acknowledge that the authors have showed extensive experiments and ablations that helped compensate the information loss introduced by feature partitioning.

Another issue I wanted to bring up is that there remains a gap between the theoretical analysis and the actual architecture. In particular, the paper does not directly characterize how much important cross-feature interaction information is lost through partitioning and subsequently recovered through Induct. At least some preliminary interpretability or theoretical analysis comparing the original global interaction architecture with the partitioned architecture would be appreciated.

This is more of an limitation in understanding and interpretation of the framework. I believe a better understanding of this mechanism could also provide valuable insights for future model development based on the sequential divide-and-conquer philosophy behind this work.

3. Minor suggestions include figure remaking.

The current figures have fonts that are too small for display. Especially figure 1, text is near unreadable there.

Questions for authors

I don’t have any additional outstanding questions beyond the concerns I have already discussed in the Critical Weaknesses section. I would appreciate the authors’ thoughts on these concerns. If the authors adequately address my concerns, I'd be willing to raise my score to clear acceptance.

2 comments

We thank the reviewer again for recognizing our work. We hope the rebuttal submitted earlier was to your satisfaction and you would consider raising our score as you kindly offered. We're happy to provide any further clarifications if needed.

Thank you for your careful response! These substantially addressed my concerns about the feature assignment strategy and interaction preservation. Some limitations remain in connecting the theory to the full nonlinear architecture and isolating the contribution of Induct. Nevertheless, given the strong empirical evidence and practical significance, I recommend acceptance and will maintain my current weak acceptance score.

Reviewer_mU5dOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that industrial ranking models hit three scaling "walls" (Sec. 2, Fig. 1b–d): dense returns plateau beyond ~40 GFLOPs/sample; under iso-FLOPs, sparse capacity plateaus beyond ~300B parameters because feature interaction is super-linear in feature count; and at fixed FLOPs, quality is U-shaped in depth and degrades past 16 layers. All-to-all embedding exchange also stops scaling beyond one hardware scale-up domain (Fig. 2). Maglev decomposes the model into stages, each pairing a feature subset with one unmodified backbone block:

  • Indexers. Soft Indexers learn per-stage linear combinations of all features; Hard Indexers turn the learned affinities into a feature partition (Sec. 3.1).
  • Induct. A gated low-rank residual digest carried across stages, with a linear-surrogate error bound (Sec. 3.2, Eq. 1).
  • Rail. A zero-bubble pipeline schedule compatible with synchronous gradient clipping (Sec. 3.3). Validation: on a 28B-sample, 700-feature (600 sparse + 100 sequence), four-task proprietary dataset, Maglev-Wukong and Maglev-DHEN lie above their baselines' quality–compute frontiers in all 116 comparisons (median +0.15 pp RLL, Table 2), with isolated sparse/dense/depth sweeps (Fig. 1); on three public datasets, five backbones improve in all 15 matched-FLOPs settings with 12 seeds (Table 1); three production launches report +0.1/0.3/0.4% RLL (Sec. 4.4); training throughput is up to 1.3× that of hybrid parallelism.
Critical strengths

A simple, backbone-agnostic staging idea, evaluated with careful statistical hygiene on the settings it does cover.

  • Soft Indexers (one learned projection per stage) plus a gated cross-stage digest are easy to describe and drop into an existing block, and the same structure maps naturally onto hardware locality (Hard Indexers + Rail).
  • The public-benchmark protocol is above the usual standard in this area: 12 paired seeds, FLOPs matched within 1.09%, parameters within 4.8%, equal tuning budgets, and Holm-corrected paired intervals (Table 1).
  • The industrial comparison uses same-backbone frontiers rather than single points, and is robust to the interpolation choice (App. A.11: 116/116 without interpolation, 152/152 over all converged runs), with a five-seed two-hardware repeatability check (seed SD 0.011–0.013 pp, Table 7).
  • Why it matters: the engineering and the measurement discipline are real, and this is what keeps the paper worth discussing despite the coverage problems below.
Critical weaknesses
  1. The validated regime does not match the claim. The backbones are not today's unified sequence / non-sequence models, and the dense scale is one to two orders of magnitude below industrial practice.
  • (a) Missing unified sequence + feature-interaction backbones. The transformed backbones are Wukong and DHEN industrially (Sec. 4.1) and RankMixer, FinalMLP, AutoInt, DESTINE, FiGNN publicly (Table 1). All of these are feature-interaction architectures. None of the current unified backbones that jointly model behaviour sequences and feature interaction is evaluated — MixFormer, HyFormer, OneTrans, LONGER, HSTU, TokenMixer-Large. This matters specifically for Maglev's own mechanism: App. A.7 handles sequence features by splitting each sequence's temporal axis into ordered chunks across stages, so each stage applies the backbone's sequence interaction only to its local chunk, with Maglev Induct carrying context forward. On a unified backbone, where cross-position attention over the whole history is the main source of quality, that chunking is exactly the step most likely to cost accuracy. The paper never tests it. Since the claim is that Maglev "improves the scaling law along every axis" for industrial recommenders, and the field has moved to unified backbones, the central claim is unverified where it matters most.
  • (b) The dense scale is too small to support an industrial-scale claim. The only reported parameter counts are for the public study: 0.93M–41.33M total trainable parameters at 0.18–5.00 MFLOPs per sample (Table 1), i.e. tens of M at most. Industrial ranking models are now at several hundred M to ~1B dense parameters. For the industrial campaign the paper reports FLOPs per sample (3.47–110 GFLOPs) and sparse capacity (104–514B embedding parameters), but no dense parameter counts (Sec. 4, App. A.6), so the reader cannot verify that the dense regime where "diminishing returns" are claimed corresponds to the dense regime deployed today. The dense wall at ~40 GFLOPs is also demonstrated on a single baseline family (Wukong), narrowing MLPs to hold FLOPs fixed; this may say as much about that family's input tokenization under an iso-FLOPs constraint as about dense scaling in general.
  • Why it matters: the paper's contribution is defined by the regime it claims to serve. At tens of M dense parameters and without a unified backbone, the results do not establish that Maglev helps large-scale industrial recommender systems.
  1. The infra baselines are insufficient to establish that the proposed system design is what delivers the gains.
  • (a) Only one partitioning alternative. The sole infra-side model comparison is DMT (Table 3, 104B capacity). Other established alternatives are discussed but not compared: Meta Lattice and Kunlun are set aside as "composite systems" (Sec. 5), and TorchRec-style sharding planners, locality-aware reduction schemes and token-level MoE routing are only cited. With a single comparator, the claim that learned affinity partitioning plus a cross-stage digest beats the alternatives is not established.
  • (b) Only one schedule and one parallelism baseline. Maglev Rail is compared against ZB2P+FSDP end-to-end (1.3×) and against ZB2P in isolation (6.3–11.7%, App. A.4). 1F1B and interleaved schedules appear only in the analysis, and the zero-bubble property is proved under the standard equal-latency model rather than measured against them.

combined with 1, what the paper demonstrates is that one staged variant of two classic backbones beats those backbones at small-to-mid dense scale. It does not isolate which part of the system is responsible, nor show superiority over the infra designs it aims to replace.

Questions for authors
  1. Unified backbones. Please apply Maglev to at least one unified sequence + feature-interaction backbone (e.g. MixFormer, HyFormer, OneTrans, LONGER or HSTU) at matched compute, and report the effect of the A.7 temporal chunking on quality. Does the frontier gain survive when cross-position sequence interaction is split across stages?
  2. Dense scale. Please report the dense (non-embedding) parameter counts for every industrial configuration, and extend the dense sweep into the few-hundred-M to ~1B dense-parameter regime. Does the frontier margin persist, grow or vanish there?
  3. Infra baselines. Please add infra comparators beyond DMT: at minimum a TorchRec/Lattice-style sharding or partitioning planner, and pipeline schedules 1F1B and interleaved alongside ZB-H1/ZB2P, all at matched compute and memory.
  4. Single-stage control and multi-seed ablation. At matched FLOPs and width, how does a backbone with one learned input compression (a K = 1 Soft Indexer) compare with multi-stage Maglev? Can the Fig. 4 ablation be repeated with multiple seeds, even on a reduced-scale proxy, so that 0.01–0.04 pp differences can be distinguished from noise?
  5. Online metrics. What business metrics or confidence intervals underlie the +0.1/0.3/0.4% RLL launches?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.