acceptodds
Under review as a conference paper at ICLR 2027

nUniMixer: Breaking the Barrier to a Unified and Scalable Backbone for Recommender Systems

Abstract

In recent years, the scaling laws of recommendation models have attracted increasing attention. Three mainstream architectures have emerged for scaling ranking models: attention-based, TokenMixer-based, and factorization-machine-based architectures. These architectures differ fundamentally in both their design philosophies and structural characteristics. In this paper, we propose UniMixer, a unified theoretical framework for model scaling that establishes connections among these three mainstream blocks. By parameterizing the rule-based TokenMixer, we develop a Dual-Axis Heterogeneous Mixer (DAHM) that learns token-mixing patterns during training and achieves global element-wise feature mixing. To relax the constraints on the parameter matrices and further improve the return on investment (ROI) of model scaling, we further develop a normalized UniMixer block, termed nUniMixer. Extensive offline and online experiments across diverse tasks and application scenarios demonstrate the superior scalability of nUniMixer.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Ranking Models, Recommender Systems, Feature Interaction, Scaling Law

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BgLBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a scalable recsys model backbone for learnable token and feature mixing The backbone model incorporates nGPT-style normalized hidden states, BallNorm weight constraints, and an interpolated residual structure. To better explain the effectiveness, it proposes a unified perspective, that explain attention, TokenMixers, and Factorization Machine modules as matrix-based mixing operations.

Critical strengths

S1: effective combination of existing modules. Although all modules are available but combo them and verified them in large-scale recsys data is valuable in industry

S2: strong empirical results. Compared to TokenMixer-Large with ~103 million params, proposed model achieves 0.754199 AUC vs. 0.748410 by baseline. The computational cost only increases by ~19%.

Critical weaknesses

W1: This should be a KDD or RecSys paper instead of a ICLR paper from technical novelty perspective. the major credit is engineering practice (combination) and industry data eval and testing, not too much theoretical or technical innovation. The conceptual novelty of the proposed approach is over-sold.

  • pre-normalized linear operator in DAHM adopts the well-known "two-factor Monarch" form
  • categorizing the proposed backbone as an "attention mechanism over feature maps" would require not only the inclusion of projection weights but also the removal of the Softmax function and the incorporation of the 1/d1/\sqrt{d} scaling factor It is more a shared template than a novel mode

W2: Although this architectural combination is noteworthy, the current experimental results fail to clarify which specific component drives the performance gains (existing one just demonstrate normalzaion has a value), nor do they justify the necessity of introducing DAHM. It also lacks a control experiment—such as a "dense layer/MLP mixture"—that employs the same normalization scheme while maintaining a comparable computational cost.

W3: More details needed for experiments reproduciblity

  • Checkpoint selection is unclear.
  • Equally tuned recipes and one checkpoint should be used to rule out the differences that prevents us from attributing the 15/17 wins to the backbone.
  • Proposed task towers a learning rate is different from the other baselines compared?
  • Include 100-update validation also for other modles compared?
Questions for authors

plz provide details to justify the challenge raised in weakness parts

5 comments

We are grateful for your insightful review and constructive feedback on our manuscript. We have carefully considered each point you raised, and our detailed responses are outlined below. For clarity, we have denoted Weaknesses with W.

W1: Regarding technical novelty of nUniMixer Thank you for your comments. As mentioned in this paper, we summarize the main technical contributions as follows:

  • Contribution in parameterizing and training the TokenMixer. First, we provide a clear mathematical proof of the equivalent parameterization of the TokenMixer (Theorem 3.1), theoretically demonstrating the existence of a permutation matrix that enables this equivalent parameterization. Second, directly applying Theorem 3.1 to parameterize and train the TokenMixer leads to an explosion in both parameter count and computational cost, making parameterized TokenMixer training particularly difficult. We theoretically derive a Kronecker-product-based compression scheme (Theorem 3.2 Compressibility) to reduce the parameter count. However, this scheme still requires constructing prohibitively large intermediate matrices during computation. We therefore further derive an equivalent computational-flow optimization scheme (Appendix E) that avoids these large intermediate matrices. The following concrete example is provided to demonstrate the substantial performance improvement achieved through this process. Given a hidden state with T=16 and D=1024, using FP32 precision, the parameter counts and FLOPs before and after optimization are as follows:
StageModuleFLOPsTotal GPU memory usagePrecisionBytes per parameter
Before optimizationWpermvec⁡(X)W^{\mathrm{perm}}\operatorname{vec}(X)536.87 M1 GiBFP324 bytes
After optimizationWGlobal[x1WLocal1⋮xLBWLocalLB]W_{\mathrm{Global}}\left[\begin{array}{c}x_1W_{\mathrm{Local}}^1\cr\vdots\cr x_{\frac{L}{B}}W_{\mathrm{Local}}^{\frac{L}{B}}\end{array}\right]10.49 M4.25 MiBFP324 bytes
Reduction—98.05% ↓99.58% ↓——

, where B=D/TB=D/T and L=TDL=TD. In addition, the Sinkhorn-Knopp and the temperature annealing operations are used to ensure the properties of mixer parameter matrices given in Theorem 3.2.

  • Contribution in the Unified Perspective These closely connected theoretical results (Theorems 3.1 and 3.2, computational-flow optimization scheme) lead to the parameterized formulations of TokenMixer and UniMixer (Eq. 3~5) in our paper. More importantly, these theoretical results and heterogeneous transformation reveal the relationships among TokenMixer, attention, and factorization machines (FMs). These insights are crucial for designing the scalable model in recommender systems. The unified perspective (Eq. 6 and Appendix F) no longer views these architectural paradigms in isolation. Actually, G(X,WGlobal)G(X,W_\text{Global}) in Eq. (6) supports combinations of different feature interaction mechanisms. Specifically, G(X,WGlobal)G(X,W_\text{Global}) can be divided into blocks, allowing an appropriate interaction mechanism to be selected for each block. This unified perspective enables multiple interaction mechanisms to be implemented within a single block. We think that this compatible, unified perspective can effectively integrate the strengths of different paradigms, thereby providing a solid theoretical foundation for the further development of scalable backbone architectures.
  • Contribution in designing the scalable backbone. The nUniMixer backbone is developed to achieve the global element-wise feature mixing (Appendix G) and address the complicated training strategy caused by the constraints on the global and local mixing matrices, where the nDAHM and npSwiGLU modules are designed to achieve improvements in both scaling efficiency and performance.

pre-normalized linear operator in DAHM adopts the well-known ''two-factor Monarch'' form

Thanks for your comments. We extend UniMixing to DAHM to achieve global element-wise mixing. Formally, DAHM can be expressed in a two-factor Monarch form. We also find that DAHM has a formulation similar to that of the two-dimensional discrete Fast Fourier transform (2D FFT). Despite their similar formulations, they operate in different domains and are derived from fundamentally different motivations. DAHM is designed to achieve global element-wise mixing, and its current formulation is further derived from the theoretical framework established by UniMixer. This connection reveals the relationships among scalable backbone design for recommender systems, Monarch matrices, and the 2D FFT. In this sense, UniMixer provides broader perspectives for designing scalable recommendation backbones. Furthermore, DAHM, together with input/weight normalization and the corresponding per-token normalized SwiGLU, constitutes nUniMixer. This design removes the requirements of double stochasticity, sparsity, and symmetry imposed on the global and local mixing matrices, thereby simplifying the otherwise complex training strategy. Therefore, directly adopting the two-factor Monarch form or a 2D FFT-based formulation to design scalable backbones for recommender systems leads to performance degradation. As shown in Table 3, removing the BallNorm operation from nDAHM results in a 0.1191% decrease in AUC.

categorizing the proposed backbone as an "attention mechanism over feature maps" would require not only the inclusion of projection weights but also the removal of the Softmax function and the incorporation of the scaling factor. It is more a shared template than a novel mode

Thanks for your comments. From the perspective of global element-wise mixing, attention weights play a role equivalent to that of the global mixing matrix. The key difference is that both attention weights and FMs (Table 5) are input-dependent. We introduce the unified perspective of heterogeneous feature interaction to move beyond viewing each design paradigm in isolation. Moreover, as mentioned above, G(X,WGlobal)G(X,W_\text{Global}) in Eq. (6) can be partitioned into blocks, and the interaction mechanism for each block can be selected adaptively. This allows appropriate feature interaction mechanisms to be allocated according to the computational budget, without being restricted to any single interaction paradigm. Establishing a unified perspective across diverse feature interaction paradigms is crucial for designing scalable backbones for recommender systems. This unified theoretical framework (Eq. 6), or shared template, enables us to further develop the DAHM module, reveal its connections to the 2D FFT and Monarch matrices, and thereby broaden the design space for scalable backbones.

W2: Regarding performance gains of specific components Thanks for your comments. According to the ablation results given in Table 3, the nUniMixer variant in the "w/o DAHM" row is obtained by replacing the DAHM module in nUniMixer with the UniMixing module (Eq. 5). As shown by the results, removing DAHM leads to the 0.1238% decrease in AUC relative to the complete nUniMixer model. Compared with the other components, DAHM is the second most influential module in terms of performance. Meanwhile, we also replace the DAHM module in nUniMixer with a token-mixing module (given in RankMixer), and the corresponding experimental results are shown in the following table, which further demonstrate the importance of the DAHM.

SettingAUC ↑ΔAUC ↑UAUC ↑ΔUAUC ↑ParamsFLOPs/Batch
nUniMixer0.750446–0.739548–6.8M0.098T
w/o DAHM (DAHM→UniMixing\text{DAHM}\to\text{UniMixing})0.749208−0.1238%0.738946−0.0602%7.8M0.098T
DAHM→TokenMixing\text{DAHM}\to\text{TokenMixing}0.747131-0.3315%0.736789-0.2759%6.5M0.082T

The corresponding modification has been added in Table 3 in this version.

W3: Regarding Experiments Reproducibility

  • Sorry for the unclear description. As shown in Table 1, the results of the nUniMixer and other baseline models follow the same checkpoint selection rule. For each training run, each model is trained for one epoch, and the checkpoint saved at the end of that epoch is used to calculate the test AUC for all tasks on the test dataset. The description in Lines 1005–1007, “For nUniMixer, full validation ... number of training epochs,” is intended to describe how nUniMixer is monitored during training. Checkpoint selection is fixed at the end of the epoch, and the logic and criteria remain consistent across all models and datasets. The original statement could easily cause confusion. We will revise this statement as the following clearer and more accurate description about our checkpoint selection process: "We apply the same checkpoint selection strategy to all models and datasets. For each run, we train for one epoch and use the single checkpoint saved at the end of this epoch to compute test AUC for all tasks."

  • For a given dataset, all models use the same tokenization scheme, and the same number and partitioning of input tokens. The identical tower input dimensions and architectures are used to all model. The numbers of the dense parameter across different models are given in the following table.

ModelQK-VideoKuaiRandTAAC-25MerRec
UniMixer12.694412.801312.526412.8105
nUniMixer12.509513.015712.733112.8048
FAT12.610312.213012.386212.9192
Wukong12.752512.594012.598012.4596
RankMixer13.090112.589312.639512.6095
TokenMixer-Large12.686212.892412.849312.8368
HiFormer12.692412.551012.630812.7181
Reviewer_nGWwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes UniMixer and its normalized variant, nUniMixer, as scalable backbones for large-scale recommendation. It first expresses the TokenMixer operation as a permutation-matrix transformation and uses a structured decomposition to motivate learnable local and global mixing. It then introduces DAHM (Dual-Axis Heterogeneous Mixer) to enable global element-level exchange without explicitly constructing a large permutation projection, and develops nUniMixer with BallNorm-based parameter normalization, normalized hidden states, per-token normalized SwiGLU, and learnable residual updates.

The paper frames attention-based, TokenMixer-based, and FM-based recommendation models within a common local/global mixing perspective. Evaluation includes 17 prediction tasks across four public datasets, industrial offline experiments on a dataset with over 0.7 billion user samples, comparisons of parameter counts and FLOPs, component ablations, depth-scaling experiments, and an online A/B-test report. nUniMixer achieves the highest AUC on 15 of the 17 public-dataset tasks and shows favorable performance/compute trade-offs in the reported industrial offline results. The paper also reports more than 15% average improvement in cumulative active days over a 30-day window in multiple advertising scenarios.

Critical strengths
  1. A meaningful architecture contribution with a practical efficiency/performance trade-off. The paper goes beyond a simple relabeling of existing blocks: it develops learnable local/global mixing, introduces DAHM, and then proposes nUniMixer to improve the behavior of the mixing architecture. On the industrial dataset, nUniMixer-2-Blocks reports AUC 0.752018 with 51.8M parameters and 0.76T FLOPs per batch, compared with 0.744577, 132.7M parameters, and 1.68T FLOPs for the heterogeneous-attention reference. This is a consequential result if reproduced under the reported setup.

  2. Broad empirical coverage and useful component/depth studies. The paper evaluates 17 tasks across four public datasets and reports that nUniMixer ranks first on 15 tasks. It also includes component ablations and 2/4/8-block comparisons, which provide evidence that the proposed design is not supported by a single isolated benchmark. The results suggest that the architecture is competitive across several recommendation settings and that its components contribute to the reported performance.

Critical weaknesses
  1. The theoretical-unification and scaling-law claims exceed what is established. The permutation-matrix representation and the local/global mixing formulation offer a useful common perspective, but the paper does not establish general equivalence among attention, TokenMixer, and FM architectures, nor does it provide broad theoretical guarantees about expressivity or scaling. Similarly, the reported scaling curves cover a limited set of model configurations and do not report uncertainty or fit diagnostics. The evidence supports favorable scaling trends within the tested range, not a generally predictive scaling law. This matters because theoretical unification and scaling are central to the paper’s framing.

  2. Some key comparisons and the online claim are insufficiently documented. The public-dataset setup uses one training epoch and different learning rates for nUniMixer versus other models. Different learning rates can be justified, but the paper does not make sufficiently clear that all baselines received comparable tuning and optimization opportunities. In addition, the online A/B-test section reports an average CAD increase of more than 15% without describing the control configuration, traffic allocation, sample sizes, aggregation procedure, confidence intervals, or significance tests. The reported result is promising, but the paper does not provide enough information to judge its robustness or attribute the gain confidently to the proposed model.

Questions for authors
  1. Scaling-law evidence: How many model-size/compute configurations and random seeds underlie Figure 3, and can the authors report fit uncertainty or goodness-of-fit measures? How sensitive are the fitted exponents and coefficients to the chosen range? This would materially affect how strongly the scaling-law claims can be interpreted.

  2. Depth versus width: The paper concludes that scaling along depth is more efficient than scaling along width. Were depth and width compared at matched parameter counts, FLOPs, and training budgets? If not, can the authors narrow this conclusion to the tested depth configurations?

  3. Baseline tuning and training budget: Why are the public-dataset experiments trained for one epoch, and how were the different learning rates selected? Were baseline hyperparameters tuned with comparable effort, and do the conclusions persist under a longer or convergence-aware training budget?

  4. Online A/B testing: What were the control model, traffic allocation, sample sizes, and aggregation method behind the reported average CAD improvement of more than 15%? Were confidence intervals or statistical significance tests computed, and were other product or delivery changes controlled during the experiment?

6 comments

We are grateful for your insightful review and constructive feedback on our manuscript. We have carefully considered each point you raised, and our detailed responses are outlined below. For clarity, we have denoted Weaknesses with W and Questions with Q.

W1: The theoretical-unification and scaling-law claims exceed what is established Thank you for your comments. As mentioned in this paper, we summarize the main technical contributions as follows:

  • Contribution in parameterizing and training the TokenMixer. First, we provide a clear mathematical proof of the equivalent parameterization of the TokenMixer (Theorem 3.1), theoretically demonstrating the existence of a permutation matrix that enables this equivalent parameterization. Second, directly applying Theorem 3.1 to parameterize and train the TokenMixer leads to an explosion in both parameter count and computational cost, making parameterized TokenMixer training particularly difficult. We theoretically derive a Kronecker-product-based compression scheme (Theorem 3.2 Compressibility) to reduce the parameter count. However, this scheme still requires constructing prohibitively large intermediate matrices during computation. We therefore further derive an equivalent computational-flow optimization scheme (Appendix E) that avoids these large intermediate matrices. The following concrete example is provided to demonstrate the substantial performance improvement achieved through this process. Given a hidden state with T=16 and D=1024, using FP32 precision, the parameter counts and FLOPs before and after optimization are as follows:
StageModuleFLOPsTotal GPU memory usagePrecisionBytes per parameter
Before optimizationWpermvec⁡(X)W^{\mathrm{perm}}\operatorname{vec}(X)536.87 M1 GiBFP324 bytes
After optimizationWGlobal[x1WLocal1⋮xLBWLocalLB]W_{\mathrm{Global}}\left[\begin{array}{c}x_1W_{\mathrm{Local}}^1\cr\vdots\cr x_{\frac{L}{B}}W_{\mathrm{Local}}^{\frac{L}{B}}\end{array}\right]10.49 M4.25 MiBFP324 bytes
Reduction—98.05% ↓99.58% ↓——

, where B=D/TB=D/T and L=TDL=TD. In addition, the Sinkhorn-Knopp and the temperature annealing operations are used to ensure the properties of mixer parameter matrices given in Theorem 3.2.

  • Contribution in the Unified Perspective These closely connected theoretical results (Theorems 3.1 and 3.2, computational-flow optimization scheme) lead to the parameterized formulations of TokenMixer and UniMixer (Eq. 3~5) in our paper. More importantly, these theoretical results and heterogeneous transformation reveal the relationships among TokenMixer, attention, and factorization machines (FMs). These insights are crucial for designing the scalable model in recommender systems. The unified perspective (Eq. 6 and Appendix F) no longer views these architectural paradigms in isolation. Actually, G(X,WGlobal)G(X,W_\text{Global}) in Eq. (6) supports combinations of different feature interaction mechanisms. Specifically, G(X,WGlobal)G(X,W_\text{Global}) can be divided into blocks, allowing an appropriate interaction mechanism to be selected for each block. This unified perspective enables multiple interaction mechanisms to be implemented within a single block. We think that this compatible, unified perspective can effectively integrate the strengths of different paradigms, thereby providing a solid theoretical foundation for the further development of scalable backbone architectures.
  • Contribution in designing the scalable backbone. The nUniMixer backbone is developed to achieve the global element-wise feature mixing (Appendix G) and address the complicated training strategy caused by the constraints on the global and local mixing matrices, where the nDAHM and npSwiGLU modules are designed to achieve improvements in both scaling efficiency and performance.

Thank you for your valuable suggestions. As shown in Fig. 3, the five points with five model-size/compute configurations on the scaling curve of the RankMixer (2 Blocks), UniMixer (2 Blocks), and nUniMixer (2 Blocks) correspond one-to-one with each other. For each pair of corresponding points, they have the identical parameter configuration. Each configuration is currently evaluated with one random seed. A single random seed does not allow us to estimate training variance across random initializations. Therefore, the reuslts given in Fig. 3 are empirical power-law-like scaling trends over the evaluated range. For each model, we fit ΔAUC=aXb\Delta\mathrm{AUC}=aX^b using ordinary least squares in log-log space, where XX denotes either the number of dense parameters or FLOPs. We report the fitted coefficient aa, exponent bb, their 95% regression confidence intervals, and the log-space R2R^2. These confidence intervals quantify regression uncertainty across the five tested configurations.

Dense Parameters

ModelPointsaa (95% CI)bb (95% CI)Rlog⁡2R^2_{\log}
RankMixer (2 Blocks)50.002718 [0.002594, 0.002848]0.1160 [0.1027, 0.1294]0.9961
UniMixer (2 Blocks)50.003767 [0.003402, 0.004170]0.1419 [0.1097, 0.1742]0.9849
nUniMixer (2 Blocks)50.004608 [0.004262, 0.004982]0.1198 [0.0969, 0.1426]0.9893

FLOPs

ModelPointsaa (95% CI)bb (95% CI)Rlog⁡2R^2_{\log}
RankMixer (2 Blocks)50.002022 [0.001858, 0.002199]0.1166 [0.1024, 0.1309]0.9956
UniMixer (2 Blocks)50.002338 [0.001982, 0.002758]0.1353 [0.1108, 0.1598]0.9904
nUniMixer (2 Blocks)50.003308 [0.002944, 0.003716]0.1218 [0.1024, 0.1411]0.9926

To evaluate sensitivity to the selected fitting range, we additionally conducted a leave-one-configuration-out analysis, in which each of the five configurations was removed in turn and the power-law curve was refitted. Across both dense parameters and FLOPs, all three models retain strictly positive exponents and high goodness of fit.

In the dense-parameter analysis, the log-space R2R^2 values range from 0.9849 to 0.9961. The leave-one-out exponent ranges are [0.1100,0.1209][0.1100,0.1209] for RankMixer (2 Blocks), [0.1289,0.1558][0.1289,0.1558] for UniMixer (2 Blocks), and [0.1081,0.1280][0.1081,0.1280] for nUniMixer (2 Blocks). In the FLOPs analysis, the corresponding R2R^2 values range from 0.9904 to 0.9956, and the leave-one-out exponent ranges remain similarly narrow. These results indicate that the fitted trends are not driven by a single model-size or compute configuration. Therefore, RankMixer (2 Blocks), UniMixer (2 Blocks), and nUniMixer (2 Blocks) all exhibit stable positive empirical power-law-like trends with respect to both dense parameter count and FLOPs. All log-space fits achieve R2>0.984R^2>0.984.

W2: Some key comparisons and the online claim are insufficiently documented

Sorry for the insufficiently rigorous experiments. Actually, we tuned the task-tower learning rate over {1×10−4,2×10−3}\{1\times10^{-4}, 2\times10^{-3}\}. This coarse-grained tuning was insufficiently rigorous. Therefore, we conducted a new, more rigorous set of experiments on the public datasets. The task-tower learning rates are tuned in {1×10−4,4×10−4,8×10−4,1×10−3,2×10−3}\{1\times10^{-4}, 4\times10^{-4}, 8\times10^{-4}, 1\times10^{-3}, 2\times10^{-3}\}. For each model, we selected the learning rate that achieved the best AUC for that model on the largest proportion of tasks, and then compared all SOTA models using their respective selected learning rates. The results are reported in the table below. nUniMixer achieves the highest AUC on 15 of the 17 tasks, including all tasks on KuaiRand, TAAC-25, and MerRec, as well as the Click and Like tasks on QK-Video.

DatasetHeadUniMixernUniMixerFATWukongRankMixerTokenMixer-LargeHiFormer
QK-VideoClick0.9303980.9345310.9313680.9313450.9297980.9323960.931283
QK-VideoFollow0.9076730.9167520.9176570.9143880.9153370.9179490.920816
QK-VideoLike0.9434710.9471360.9432500.9408550.9437070.9456540.946948
QK-VideoShare0.9286430.9316840.9323050.9298920.9306970.9320930.935005
KuaiRandClick0.7924520.7970170.7754470.7762800.7829930.7795430.771839
KuaiRandFollow0.8635410.8881980.8743920.8723050.8760480.8811610.878605
KuaiRandLike0.9208930.9290610.9205270.9203640.9234080.9211530.907368
KuaiRandComment0.8834150.8993760.8832470.8852760.8785930.8911480.882770
KuaiRandForward0.8423870.8727500.8476420.8408190.8514100.8582220.853178
KuaiRandLong View0.8101510.8140210.7899630.7943290.8015500.7970510.789431
TAAC-25Click0.8222660.8345890.8247870.8224360.8185110.8317360.828790
TAAC-25Conversion0.8735460.8850600.8792260.8779730.8742840.8803300.879132
MerRecLike0.7602420.7684910.7330900.7341850.7329820.7398780.729209
MerRecCart0.8099800.8209100.7942460.7987610.7960620.8087890.798337
MerRecOffer0.7528900.7820640.7461170.7459740.7262450.7738080.751192
MerRecCheckout0.8313450.8557370.8272740.8204320.8333920.8518370.823843
MerRecPurchase0.8282090.8557200.8184310.8189800.8377260.8537850.819639

To verify the stability of the model performance, we conducted experiments using three random seeds. All models used the same parameter configurations, including the same validation and checkpoint selection policy and so on. nUniMixer still achieves the highest mean test AUC on 15 of the 17 tasks across three random seeds, including all tasks on KuaiRand, MerRec, and TAAC-25, and Click and Like on QK-Video. The table reports mean ± sample standard deviation. nUniMixer outperforms UniMixer on all evaluated tasks, demonstrating robust improvements across different recommendation scenarios.

DatasetHeadTokenMixer-LargenUniMixerUniMixerFATWukongRankMixerHiFormer
QK-VideoClick0.931951 ± 0.0002000.934351 ± 0.0003610.930503 ± 0.0003850.930021 ± 0.0002290.931068 ± 0.0002790.930137 ± 0.0004060.931017 ± 0.000240
QK-VideoFollow0.923897 ± 0.0002960.919770 ± 0.0005450.904967 ± 0.0028920.919229 ± 0.0004970.916917 ± 0.0004350.914449 ± 0.0007820.920376 ± 0.000382
QK-VideoLike0.947620 ± 0.0002240.947977 ± 0.0003480.943358 ± 0.0003680.944714 ± 0.0002750.944749 ± 0.0001750.942764 ± 0.0004770.946442 ± 0.000448
QK-VideoShare0.936285 ± 0.0006260.933520 ± 0.0007700.927291 ± 0.0022420.932832 ± 0.0006790.932119 ± 0.0005560.931119 ± 0.0006200.934630 ± 0.000545
KuaiRandClick0.773425 ± 0.0034940.796366 ± 0.0005900.792188 ± 0.0000580.770936 ± 0.0015930.780661 ± 0.0019910.780922 ± 0.0027210.766685 ± 0.004766
KuaiRandFollow0.881334 ± 0.0012110.883346 ± 0.0010520.861146 ± 0.0029690.875739 ± 0.0027970.878426 ± 0.0034280.870297 ± 0.0021190.876493 ± 0.002902
KuaiRandLike0.923013 ± 0.0012530.927323 ± 0.0006330.920439 ± 0.0001650.915507 ± 0.0023410.918555 ± 0.0022550.919841 ± 0.0029090.910132 ± 0.003849
KuaiRandComment0.894616 ± 0.0013270.895357 ± 0.0016430.880449 ± 0.0022860.887412 ± 0.0001750.885348 ± 0.0028280.879705 ± 0.0037620.886119 ± 0.002949
KuaiRandForward0.855546 ± 0.0038540.863816 ± 0.0022810.841560 ± 0.0085660.854116 ± 0.0014630.853448 ± 0.0050250.849156 ± 0.0029870.853519 ± 0.003792
KuaiRandLong View0.791017 ± 0.0036640.813215 ± 0.0005730.809648 ± 0.0005610.784823 ± 0.0015390.797771 ± 0.0022920.800217 ± 0.0020590.782027 ± 0.006477
MerRecLike0.743153 ± 0.0049850.766452 ± 0.0025600.760793 ± 0.0040520.730130 ± 0.0071220.736700 ± 0.0015660.730924 ± 0.0087510.731070 ± 0.003560
MerRecCart0.806397 ± 0.0030610.825237 ± 0.0025050.814781 ± 0.0011690.795219 ± 0.0051910.790600 ± 0.0015830.795246 ± 0.0010510.796846 ± 0.001817
MerRecOffer0.766029 ± 0.0026000.780213 ± 0.0019180.738441 ± 0.0151120.751905 ± 0.0068070.729940 ± 0.0045280.729390 ± 0.0140360.748952 ± 0.003849
MerRecCheckout0.842633 ± 0.0031060.851583 ± 0.0038580.825981 ± 0.0020210.820882 ± 0.0037030.809114 ± 0.0086230.816510 ± 0.0115590.819766 ± 0.004357
MerRecPurchase0.845131 ± 0.0028690.851708 ± 0.0037330.825417 ± 0.0001670.818131 ± 0.0085610.807437 ± 0.0031810.825322 ± 0.0092700.822367 ± 0.005682
TAAC-25Click0.822825 ± 0.0009010.833327 ± 0.0015350.821773 ± 0.0006160.824695 ± 0.0013990.823294 ± 0.0042070.817667 ± 0.0015110.827389 ± 0.001938
TAAC-25Conversion0.875919 ± 0.0009000.882903 ± 0.0008670.875108 ± 0.0013280.877256 ± 0.0004310.877013 ± 0.0026940.873606 ± 0.0004970.876311 ± 0.004465

In the online A/B test, the control group is served by RankMixer. 5.48% of traffic assigned to UniMixer with more than 5500 Queries Per Second. We measure user engagement using the Cumulative Active Days (CAD) over a 30-day observation window, excluding the installation day (day 0). During the A/B testing period (7 days), for nUniMixer, the 95% Confidence Interval of CAD is [9.21%, 21.55%]. There were no other product or delivery changes. Since nUniMixer outperforms UniMixer, nUniMixer has now been fully deployed and are serving across multiple advertising delivery scenarios.

Reviewer_NfcCOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper seeks a unified, scalable backbone for ranking models.

  • Parameterizing TokenMixer. It shows that RankMixer's rule-based, non-parametric TokenMixer is exactly a fixed permutation matrix acting on vec(X) (Thm 3.1), and that this matrix is Kronecker-compressible (W_perm = G ⊗ I), doubly stochastic, sparse, and symmetric when T = H (Thm 3.2). Compressibility reduces the parameter cost of making mixing learnable from O(T²D²) to O(T⁴ + (D/T)²).
  • UniMixer. Learnable global and per-block local mixing matrices give UniMixing (Eqs. 2–5), with a pipeline optimization reducing cost from O(L²) to O(L²/B + LB) and avoiding materializing W_perm. Sinkhorn–Knopp, a temperature and symmetrization enforce the properties of Thm 3.2. DAHM adds a transposed axis for global element-wise mixing; MoBE-style bases generate the mixing matrices; SiameseNorm stabilizes depth.
  • Unification. Under Eq. 6 / Table 5, self-attention, heterogeneous attention, TokenMixer and Wukong's FM block are expressed as different choices of the global mixing pattern G(X, W_Global) over the same local mixing form.
  • nUniMixer. To avoid the complicated training induced by those constraints, the normalized variant relaxes them and instead applies nGPT-style hypersphere normalization, column-wise BallNorm on all mixing and SwiGLU weights, per-token normalized SwiGLU and learnable residual interpolation (Sec. 4).
  • Validation. 17 tasks across four public datasets under the UniRank protocol, with dense parameters matched at ~12.2–13.1M (Table 1, App. H Table 7); a ~100M-parameter industrial user-retention dataset of 0.7B samples against six SOTA backbones (Table 2); component ablations (Table 3); depth scaling against RankMixer (Table 4); fitted parameter/FLOPs scaling curves (Fig. 3); and an online A/B test reporting >15% average CAD improvement, now fully rolled out (Sec. 5.4).
Critical strengths
  1. Turning RankMixer's non-parametric token mixing into a learnable operator, with the theory needed to make it practical. This is the paper's core contribution and it is a valuable one for ranking models.
  • Thm 3.1 identifies the rule-based TokenMixer as a fixed permutation on vec(X). This reframing is what makes "learn the mixing pattern" a well-posed design, instead of an ad hoc architectural tweak.
  • Thm 3.2 is what makes it feasible: Kronecker compressibility turns an O(T²D²) parameterization into O(T⁴ + (D/T)²), and the pipeline optimization (Eqs. 3–5, App. E) avoids constructing the TD × TD intermediate, cutting cost from O(L²) to O(L²/B + LB). Removing the T = H constraint also frees the interaction granularity, which the rule-based operator fixes by construction.
  • The learnable formulation then yields the unified view of attention / TokenMixer / FM as choices of the global mixing pattern (Eq. 6, Table 5), which is a useful lens for designing scaling blocks rather than treating these families as unrelated.
  • Why it matters: rule-based mixing is the main design limitation of the TokenMixer family, and this paper gives both the formulation and the complexity reduction needed to replace it. That is a genuine step for scalable recommendation backbones.
  1. Fairly solid and broad experimental validation.
  • 17 tasks across four public datasets under a shared external benchmark protocol (UniRank), with dense parameter counts matched across all models (~12.2–13.1M, App. H Table 7); nUniMixer is best on 15 of 17 tasks and improves on UniMixer on every task (Table 1).
  • An industrial study at ~100M parameters on 0.7B user-retention samples against six recent backbones, including RankMixer, TokenMixer-Large, Wukong, FAT and HiFormer, with reported AUC/UAUC and parameters/FLOPs (Table 2); the strongest baseline is then carried into a scaling comparison (Fig. 3).
  • Depth scaling is examined at 2/4/8 blocks for both variants (Table 4), with a component ablation (Table 3), plus an online A/B test and a full rollout (Sec. 5.4).
  • Why it matters: the breadth across public benchmarks, an industrial dataset and a deployment is what lifts the contribution from a plausible idea to a usable one.
Critical weaknesses
  1. Closely related published work on parameterizing RankMixer's mixing is not cited or compared, so the delta is hard to size. Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendation (RankElastor; KDD 2026, arXiv:2605.23191, public since May 2026) formulates RankMixer's mixing as (P ⊗ I)·vec(X), replaces the fixed P with a learnable full-mixing matrix, argues that block/Kronecker-structured mixing limits the attainable interactions relative to element-wise full mixing, and pairs this with GLU-style per-token FFNs — motivated by a layer-wise effective-rank analysis. These are close to UniMixing/DAHM plus per-token SwiGLU. A discussion and an experimental comparison are needed to establish what the Kronecker-structured W_Global ⊗ {W_Local^i} formulation and DAHM add beyond learnable full mixing (presumably an efficiency / expressivity trade-off, which would be worth stating explicitly).
  • Why it matters: this does not change my view that the contribution is real, but without the comparison the incremental novelty and the claimed scaling advantage cannot be sized against the closest existing design.
  1. Parts of the empirical comparison are not controlled, which weakens the quantitative scaling claims.
  • (a) Unequal optimization. Per App. H (p.19), nUniMixer's task towers use learning rate 2×10⁻³ while UniMixer and all external baselines use 10⁻⁴, and nUniMixer additionally runs full validation every 100 updates. With one-epoch training, this makes large UniMixer → nUniMixer jumps hard to attribute to architecture (e.g. MerRec-Offer 0.7357 → 0.7821; KuaiRand-Follow 0.8575 → 0.8882, Table 1). No seeds or standard deviations are reported anywhere, although many gaps are 0.1–0.3% AUC.
  • (b) A baseline that looks under-tuned. RankMixer degrades from 2 to 4 blocks (Table 4, −0.1066% AUC), which conflicts with the depth scaling reported for RankMixer and suggests its hyperparameters were not re-tuned at depth. Since the depth-scaling claim is made relative to this curve, it is weakened.
  • (c) An efficiency claim contradicted by the paper's own numbers. Sec. 5.2 claims superiority "under smaller parameter budgets and computational costs," yet in Table 2 UniMixer-2/4 use 2.17T/4.24T FLOPs per batch versus 1.68T for RankMixer. The gap between UniMixer-2 (42.4M, 2.17T) and nUniMixer-2 (51.8M, 0.76T) is also unexplained.
  • (d) The ablation does not isolate the core idea. Table 3 is a single 6.8M-parameter run and removes DAHM and the two BallNorm placements. It does not compare learnable mixing against a fixed TokenMixer under the identical normalization recipe, so the gain attributable to learnability itself — the paper's main claim — is not separated from the imported normalization machinery (nGPT, BallNorm, SiameseNorm, MoBE bases). No internal diagnostic of the learned mixing is reported.
  • Why it matters: these are fixable within a rebuttal, but until then the magnitude of the reported gains, and their attribution to learnable mixing, remain uncertain.
Questions for authors

1.Attribute the gain to learnability. At matched parameters, FLOPs and the same normalization recipe, please compare: (i) fixed TokenMixer, (ii) learnable UniMixing, (iii) learnable UniMixing + DAHM. This isolates the paper's central claim from the nGPT/BallNorm/SiameseNorm/MoBE components. 2. Matched tuning with variance. Please re-run Table 1 with the same learning rate and the same validation/checkpoint-selection policy for all models (or an equal per-model tuning budget), reporting mean ± std over ≥3 seeds. Does nUniMixer still lead on 15 of 17 tasks? 3. Comparison with RankElastor. Please discuss and compare against parameterized full mixing with GLU per-token FFNs (RankElastor, KDD 2026) at matched parameters and FLOPs, and state what the Kronecker-structured global/local factorization and DAHM provide beyond it — e.g. the efficiency/expressivity trade-off. 4. Which properties the final model keeps. nUniMixer relaxes double stochasticity, sparsity and symmetry. Is there a measurement showing that the learned mixing behaves as Thm 3.2 and the DAHM analysis predict (e.g. effective rank or mixing-pattern statistics across layers), and does keeping any of those constraints help at all? 5. RankMixer depth and FLOPs accounting. Were RankMixer's hyperparameters re-tuned at 4 blocks (Table 4)? Please also reconcile the FLOPs figures in Table 2 with the efficiency claim on p.7, and explain the UniMixer-2 versus nUniMixer-2 FLOPs gap.

6 comments

We are grateful for your insightful review and constructive feedback on our manuscript. We have carefully considered each point you raised, and our detailed responses are outlined below. For clarity, we have denoted Weaknesses with W and Questions with Q.

W1: Comparison with the closest existing design Thanks for your valuable suggestions. As mentioned by the reviewer, the paper [1] proposed the following parameterized full mixing formulation LN((P⊗I)vec(X)+vec(X))=LN((Wperm+I)vec(X))\text{LN}((P\otimes{I})\text{vec}(X)+\text{vec}(X))=\text{LN}((W^\text{perm}+I)\text{vec}(X)). This core observation is strictly proven by our work (Theorem 3.1), theoretically demonstrating the existence of a permutation matrix that enables this equivalent parameterization. Besides, directly applying the parameterized full mixing formulation proposed by [1] to parameterize and train the TokenMixer leads to an explosion in both parameter count and computational cost, making parameterized TokenMixer training particularly difficult. We theoretically derive a Kronecker-product-based compression scheme (Theorem 3.2 Compressibility) to reduce the parameter count. However, this scheme still requires constructing prohibitively large intermediate matrices during computation. We therefore further derive an equivalent computational-flow optimization scheme (Appendix E) that avoids these large intermediate matrices. The following concrete example is provided to demonstrate the substantial performance improvement achieved through this process. Given a hidden state with T=16 and D=1024, using FP32 precision, the parameter counts and FLOPs before and after optimization are as follows:

StageModuleFLOPsTotal GPU memory usagePrecisionBytes per parameter
Before optimizationWpermvec⁡(X)W^{\mathrm{perm}}\operatorname{vec}(X)536.87 M1 GiBFP324 bytes
After optimizationWGlobal[x1WLocal1⋮xLBWLocalLB]W_{\mathrm{Global}}\left[\begin{array}{c}x_1W_{\mathrm{Local}}^1\cr\vdots\cr x_{\frac{L}{B}}W_{\mathrm{Local}}^{\frac{L}{B}}\end{array}\right]10.49 M4.25 MiBFP324 bytes
Reduction—98.05% ↓99.58% ↓——

, where B=D/TB=D/T and L=TDL=TD. In addition, the Sinkhorn-Knopp and the temperature annealing operations are used to ensure the properties of mixer parameter matrices given in Theorem 3.2.
On the other hand, RankElastor in [1] requires the matrix multiplication Wpermvec⁡(X)W^{\mathrm{perm}}\operatorname{vec}(X) during modeling, which is difficult to be satisifed in large-scale experiments. As shown in the table above, learnable full mixing leads to prohibitive memory consumption and FLOPs, which is precisely the issue that the DAHM is designed to address. In our experiments, even the smallest model uses T=32T=32 and D=128D=128. Directly modeling the feature interactions with learnable full mixing would require 134 MiB of memory for a single WpermW^\text{perm} matrix under FP32 precision and incur 67.11 M FLOPs. Compared with RankElastor, DAHM provides element-wise mixing capability and superior parameter efficiency and computational efficiency. The corresponding discussion has been added to Lines 163–168 of the revised manuscript, which is highlited with the blue font.

[1] Guoming Li, Shangyu Zhang, Junwei Pan, Wentao Ning, Jin Chen, Gengsheng Xue, Chao Zhou, Shudong Huang, Haijie Gu, and Menglin Yang. Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendation. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2569–2579, 2026.

W2: Empirical Comparison Thanks for your valuable comments. We provide a point-by-point response as follows:

  • (a) Sorry for the ambiguous description and the insufficiently rigorous experiments. Actually, we tuned the task-tower learning rate over {1×10−4,2×10−3}\{1\times10^{-4}, 2\times10^{-3}\}. This coarse-grained tuning was insufficiently rigorous. Therefore, we conducted a new, more rigorous set of experiments on the public datasets. The task-tower learning rates are tuned in {1×10−4,4×10−4,8×10−4,1×10−3,2×10−3}\{1\times10^{-4}, 4\times10^{-4}, 8\times10^{-4}, 1\times10^{-3}, 2\times10^{-3}\}. For each model, we selected the learning rate that achieved the best AUC for that model on the largest proportion of tasks, and then compared all SOTA models using their respective selected learning rates. The results are reported in the table below. nUniMixer achieves the highest AUC on 15 of the 17 tasks, including all tasks on KuaiRand, TAAC-25, and MerRec, as well as the Click and Like tasks on QK-Video.
DatasetHeadUniMixernUniMixerFATWukongRankMixerTokenMixer-LargeHiFormer
QK-VideoClick0.9303980.9345310.9313680.9313450.9297980.9323960.931283
QK-VideoFollow0.9076730.9167520.9176570.9143880.9153370.9179490.920816
QK-VideoLike0.9434710.9471360.9432500.9408550.9437070.9456540.946948
QK-VideoShare0.9286430.9316840.9323050.9298920.9306970.9320930.935005
KuaiRandClick0.7924520.7970170.7754470.7762800.7829930.7795430.771839
KuaiRandFollow0.8635410.8881980.8743920.8723050.8760480.8811610.878605
KuaiRandLike0.9208930.9290610.9205270.9203640.9234080.9211530.907368
KuaiRandComment0.8834150.8993760.8832470.8852760.8785930.8911480.882770
KuaiRandForward0.8423870.8727500.8476420.8408190.8514100.8582220.853178
KuaiRandLong View0.8101510.8140210.7899630.7943290.8015500.7970510.789431
TAAC-25Click0.8222660.8345890.8247870.8224360.8185110.8317360.828790
TAAC-25Conversion0.8735460.8850600.8792260.8779730.8742840.8803300.879132
MerRecLike0.7602420.7684910.7330900.7341850.7329820.7398780.729209
MerRecCart0.8099800.8209100.7942460.7987610.7960620.8087890.798337
MerRecOffer0.7528900.7820640.7461170.7459740.7262450.7738080.751192
MerRecCheckout0.8313450.8557370.8272740.8204320.8333920.8518370.823843
MerRecPurchase0.8282090.8557200.8184310.8189800.8377260.8537850.819639

On the other hand, as shown in Table 1, the results of the nUniMixer and other baseline models follow the same checkpoint selection rule. For each training run, each model is trained for one epoch, and the checkpoint saved at the end of that epoch is used to calculate the test AUC for all tasks on the test dataset. The description in Lines 1005–1007, “For nUniMixer, full validation ... number of training epochs,” is intended to describe how nUniMixer is monitored during training. Checkpoint selection is fixed at the end of the epoch, and the logic and criteria remain consistent across all models and datasets. The original statement could easily cause confusion. We will revise this statement as the following clearer and more accurate description about our checkpoint selection process: "We apply the same checkpoint selection strategy to all models and datasets. For each run, we train for one epoch and use the single checkpoint saved at the end of this epoch to compute test AUC for all tasks."

To verify the stability of the model performance, we conducted experiments using three random seeds. All models used the same parameter configurations, including the same validation and checkpoint selection policy and so on. nUniMixer still achieves the highest mean test AUC on 15 of the 17 tasks across three random seeds, including all tasks on KuaiRand, MerRec, and TAAC-25, and Click and Like on QK-Video. The table reports mean ± sample standard deviation. nUniMixer outperforms UniMixer on all evaluated tasks, demonstrating robust improvements across different recommendation scenarios.

DatasetHeadTokenMixer-LargenUniMixerUniMixerFATWukongRankMixerHiFormer
QK-VideoClick0.931951 ± 0.0002000.934351 ± 0.0003610.930503 ± 0.0003850.930021 ± 0.0002290.931068 ± 0.0002790.930137 ± 0.0004060.931017 ± 0.000240
QK-VideoFollow0.923897 ± 0.0002960.919770 ± 0.0005450.904967 ± 0.0028920.919229 ± 0.0004970.916917 ± 0.0004350.914449 ± 0.0007820.920376 ± 0.000382
QK-VideoLike0.947620 ± 0.0002240.947977 ± 0.0003480.943358 ± 0.0003680.944714 ± 0.0002750.944749 ± 0.0001750.942764 ± 0.0004770.946442 ± 0.000448
QK-VideoShare0.936285 ± 0.0006260.933520 ± 0.0007700.927291 ± 0.0022420.932832 ± 0.0006790.932119 ± 0.0005560.931119 ± 0.0006200.934630 ± 0.000545
KuaiRandClick0.773425 ± 0.0034940.796366 ± 0.0005900.792188 ± 0.0000580.770936 ± 0.0015930.780661 ± 0.0019910.780922 ± 0.0027210.766685 ± 0.004766
KuaiRandFollow0.881334 ± 0.0012110.883346 ± 0.0010520.861146 ± 0.0029690.875739 ± 0.0027970.878426 ± 0.0034280.870297 ± 0.0021190.876493 ± 0.002902
KuaiRandLike0.923013 ± 0.0012530.927323 ± 0.0006330.920439 ± 0.0001650.915507 ± 0.0023410.918555 ± 0.0022550.919841 ± 0.0029090.910132 ± 0.003849
DatasetHeadTokenMixer-LargenUniMixerUniMixerFATWukongRankMixerHiFormer
KuaiRandComment0.894616 ± 0.0013270.895357 ± 0.0016430.880449 ± 0.0022860.887412 ± 0.0001750.885348 ± 0.0028280.879705 ± 0.0037620.886119 ± 0.002949
KuaiRandForward0.855546 ± 0.0038540.863816 ± 0.0022810.841560 ± 0.0085660.854116 ± 0.0014630.853448 ± 0.0050250.849156 ± 0.0029870.853519 ± 0.003792
KuaiRandLong View0.791017 ± 0.0036640.813215 ± 0.0005730.809648 ± 0.0005610.784823 ± 0.0015390.797771 ± 0.0022920.800217 ± 0.0020590.782027 ± 0.006477
MerRecLike0.743153 ± 0.0049850.766452 ± 0.0025600.760793 ± 0.0040520.730130 ± 0.0071220.736700 ± 0.0015660.730924 ± 0.0087510.731070 ± 0.003560
MerRecCart0.806397 ± 0.0030610.825237 ± 0.0025050.814781 ± 0.0011690.795219 ± 0.0051910.790600 ± 0.0015830.795246 ± 0.0010510.796846 ± 0.001817
MerRecOffer0.766029 ± 0.0026000.780213 ± 0.0019180.738441 ± 0.0151120.751905 ± 0.0068070.729940 ± 0.0045280.729390 ± 0.0140360.748952 ± 0.003849
MerRecCheckout0.842633 ± 0.0031060.851583 ± 0.0038580.825981 ± 0.0020210.820882 ± 0.0037030.809114 ± 0.0086230.816510 ± 0.0115590.819766 ± 0.004357
MerRecPurchase0.845131 ± 0.0028690.851708 ± 0.0037330.825417 ± 0.0001670.818131 ± 0.0085610.807437 ± 0.0031810.825322 ± 0.0092700.822367 ± 0.005682
TAAC-25Click0.822825 ± 0.0009010.833327 ± 0.0015350.821773 ± 0.0006160.824695 ± 0.0013990.823294 ± 0.0042070.817667 ± 0.0015110.827389 ± 0.001938
TAAC-25Conversion0.875919 ± 0.0009000.882903 ± 0.0008670.875108 ± 0.0013280.877256 ± 0.0004310.877013 ± 0.0026940.873606 ± 0.0004970.876311 ± 0.004465

In this revised version, the original incomplete experiments have been replaced with a more precise description and rigorous experimental evidence, which are highlighted with the blue font.

  • (b) Some studies and experiments have reported that RankMixer is typically configured with two layers. For example, Paper [2] states: “TokenMixer is typically configured with a shallow depth (e.g., two layers in RankMixer) in industrial settings. As model depth increases, achieving training stability and performance gains becomes challenging. The current TokenMixer lacks designs tailored for deep architectures.” In TokenMixer-Large [2], the authors further introduce a lightweight auxiliary loss to address insufficient training of lower-layer parameters as network depth increases, thereby enabling more thorough parameter learning and more stable performance in deep models. In the experiments about effects of the block number (given in Table 4), the only experimental variable is the number of model layers and all other settings are kept identical.
  • (c) Sorry for the inaccurate description. Accroding to the Table 2, the smaller nUniMixer-2-Blocks model, with 51.8M parameters and 0.76T FLOPs/batch, outperform other SOTA models. On the other hand, accroding to Fig. 3, it can be observed that, under smaller parameter budgets and computational costs, both UniMixer and nUniMixer architectures outperform the strongest SOTA model (RankMixer). Therefore, this conclusion should be drawn by considering both Table 2 and Fig. 3 together, rather than from Table 2 alone. In this revised paper, we have modified the current description to " The main performance results of our models and the SOTA models are provided in Table 2. It can be observed that, under smaller parameter budgets and computational costs, the nUniMixer architecture significantly outperform other SOTA models across multiple metrics."
  • (d) Thanks for your comments. According to the ablation results given in Table 3, the nUniMixer variant in the "w/o DAHM" row is obtained by replacing the DAHM module in nUniMixer with the UniMixing module (Eq. 5). As shown by the results, removing DAHM leads to the 0.1238% decrease in AUC relative to the complete nUniMixer model. Compared with the other components, DAHM is the second most influential module in terms of performance. Meanwhile, we also replace the DAHM module in nUniMixer with a token-mixing module (given in RankMixer), and the corresponding experimental results are shown in the following table, which further demonstrate the importance of the DAHM. | Setting | AUC ↑ | ΔAUC ↑ | UAUC ↑ | ΔUAUC ↑ | Params | FLOPs/Batch | |:--|--:|--:|--:|--:|--:|--:| | nUniMixer | 0.750446 | – | 0.739548 | – | 6.8M | 0.098T | | w/o DAHM (DAHM→UniMixing\text{DAHM}\to\text{UniMixing}) | 0.749208 | −0.1238% | 0.738946 | −0.0602% | 7.8M | 0.098T | | DAHM→TokenMixing\text{DAHM}\to\text{TokenMixing} | 0.747131 |-0.3315% | 0.736789 | -0.2759% |6.5M | 0.082T|

Comments on the paper

1 comment

Final Response

Dear AC, SAC, and PC,

We sincerely appreciate the reviewers' time and the constructive feedback provided on our manuscript. We are encouraged by the positive reception of our work, specifically:

A meaningful architectural contribution with strong empirical results: All reviewers (NfcC, nGWw, and BgLB) recognized the effectiveness of the proposed architecture. Specifically, the breadth across public benchmarks, an industrial dataset and a deployment is what lifts the contribution from a plausible idea to a usable one.

A valuable contribution to ranking models: Reviewer NfcC commended that the equivalent parameterization of the TokenMixer, the Kronecker-product-based compression scheme, and the computational-flow optimization scheme constitute valuable contributions to ranking models.

We have addressed the concerns raised during the review process, including the more rigorous experiments, more comprehensive ablation studies, discussion of the closest existing design, and model stability under multiple random seeds. Based on the valuable suggestions from Reviewers, we have implemented the following revisions and additional experiments (Please note that revised content has been highlighted with the blue font in the updated paper to facilitate the review process) :

More rigorous experiments: We have tuned each state-of-the-art model over a broader range of learning rates and rigorously compare its performance with that of our proposed model. The corresponding results have been added in this version (Table 1, Lines 1027--1034 in Appendix H).

Model Stability: To verify the stability of the model performance, we have conducted experiments using three random seeds and the results have been added in this version (Lines 373--377, Appendix I, Table 8).

More Comprehensive Ablation Studies: We compare (i) fixed TokenMixer, (ii) learnable UniMixing, (iii) learnable UniMixing + DAHM with each other to isolate the contribution of DAHM from the nGPT/BallNorm/SiameseNorm/MoBE components (Table 3).

Clarifications: We have clarified the potentially confusing descriptions and added more online A/B testing information in this version (Lines 480--485, 1027--1034).

Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.