- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a scalable recsys model backbone for learnable token and feature mixing The backbone model incorporates nGPT-style normalized hidden states, BallNorm weight constraints, and an interpolated residual structure. To better explain the effectiveness, it proposes a unified perspective, that explain attention, TokenMixers, and Factorization Machine modules as matrix-based mixing operations.
Critical strengths
S1: effective combination of existing modules. Although all modules are available but combo them and verified them in large-scale recsys data is valuable in industry
S2: strong empirical results. Compared to TokenMixer-Large with ~103 million params, proposed model achieves 0.754199 AUC vs. 0.748410 by baseline. The computational cost only increases by ~19%.
Critical weaknesses
W1: This should be a KDD or RecSys paper instead of a ICLR paper from technical novelty perspective. the major credit is engineering practice (combination) and industry data eval and testing, not too much theoretical or technical innovation. The conceptual novelty of the proposed approach is over-sold.
- pre-normalized linear operator in DAHM adopts the well-known "two-factor Monarch" form
- categorizing the proposed backbone as an "attention mechanism over feature maps" would require not only the inclusion of projection weights but also the removal of the Softmax function and the incorporation of the scaling factor It is more a shared template than a novel mode
W2: Although this architectural combination is noteworthy, the current experimental results fail to clarify which specific component drives the performance gains (existing one just demonstrate normalzaion has a value), nor do they justify the necessity of introducing DAHM. It also lacks a control experiment—such as a "dense layer/MLP mixture"—that employs the same normalization scheme while maintaining a comparable computational cost.
W3: More details needed for experiments reproduciblity
- Checkpoint selection is unclear.
- Equally tuned recipes and one checkpoint should be used to rule out the differences that prevents us from attributing the 15/17 wins to the backbone.
- Proposed task towers a learning rate is different from the other baselines compared?
- Include 100-update validation also for other modles compared?
Questions for authors
plz provide details to justify the challenge raised in weakness parts
5 comments
pre-normalized linear operator in DAHM adopts the well-known ''two-factor Monarch'' form
Thanks for your comments. We extend UniMixing to DAHM to achieve global element-wise mixing. Formally, DAHM can be expressed in a two-factor Monarch form. We also find that DAHM has a formulation similar to that of the two-dimensional discrete Fast Fourier transform (2D FFT). Despite their similar formulations, they operate in different domains and are derived from fundamentally different motivations. DAHM is designed to achieve global element-wise mixing, and its current formulation is further derived from the theoretical framework established by UniMixer. This connection reveals the relationships among scalable backbone design for recommender systems, Monarch matrices, and the 2D FFT. In this sense, UniMixer provides broader perspectives for designing scalable recommendation backbones. Furthermore, DAHM, together with input/weight normalization and the corresponding per-token normalized SwiGLU, constitutes nUniMixer. This design removes the requirements of double stochasticity, sparsity, and symmetry imposed on the global and local mixing matrices, thereby simplifying the otherwise complex training strategy. Therefore, directly adopting the two-factor Monarch form or a 2D FFT-based formulation to design scalable backbones for recommender systems leads to performance degradation. As shown in Table 3, removing the BallNorm operation from nDAHM results in a 0.1191% decrease in AUC.
categorizing the proposed backbone as an "attention mechanism over feature maps" would require not only the inclusion of projection weights but also the removal of the Softmax function and the incorporation of the scaling factor. It is more a shared template than a novel mode
Thanks for your comments. From the perspective of global element-wise mixing, attention weights play a role equivalent to that of the global mixing matrix. The key difference is that both attention weights and FMs (Table 5) are input-dependent. We introduce the unified perspective of heterogeneous feature interaction to move beyond viewing each design paradigm in isolation. Moreover, as mentioned above, in Eq. (6) can be partitioned into blocks, and the interaction mechanism for each block can be selected adaptively. This allows appropriate feature interaction mechanisms to be allocated according to the computational budget, without being restricted to any single interaction paradigm. Establishing a unified perspective across diverse feature interaction paradigms is crucial for designing scalable backbones for recommender systems. This unified theoretical framework (Eq. 6), or shared template, enables us to further develop the DAHM module, reveal its connections to the 2D FFT and Monarch matrices, and thereby broaden the design space for scalable backbones.
W2: Regarding performance gains of specific components Thanks for your comments. According to the ablation results given in Table 3, the nUniMixer variant in the "w/o DAHM" row is obtained by replacing the DAHM module in nUniMixer with the UniMixing module (Eq. 5). As shown by the results, removing DAHM leads to the 0.1238% decrease in AUC relative to the complete nUniMixer model. Compared with the other components, DAHM is the second most influential module in terms of performance. Meanwhile, we also replace the DAHM module in nUniMixer with a token-mixing module (given in RankMixer), and the corresponding experimental results are shown in the following table, which further demonstrate the importance of the DAHM.
| Setting | AUC ↑ | ΔAUC ↑ | UAUC ↑ | ΔUAUC ↑ | Params | FLOPs/Batch |
|---|---|---|---|---|---|---|
| nUniMixer | 0.750446 | – | 0.739548 | – | 6.8M | 0.098T |
| w/o DAHM () | 0.749208 | −0.1238% | 0.738946 | −0.0602% | 7.8M | 0.098T |
| 0.747131 | -0.3315% | 0.736789 | -0.2759% | 6.5M | 0.082T |
The corresponding modification has been added in Table 3 in this version.
W3: Regarding Experiments Reproducibility
-
Sorry for the unclear description. As shown in Table 1, the results of the nUniMixer and other baseline models follow the same checkpoint selection rule. For each training run, each model is trained for one epoch, and the checkpoint saved at the end of that epoch is used to calculate the test AUC for all tasks on the test dataset. The description in Lines 1005–1007, “For nUniMixer, full validation ... number of training epochs,” is intended to describe how nUniMixer is monitored during training. Checkpoint selection is fixed at the end of the epoch, and the logic and criteria remain consistent across all models and datasets. The original statement could easily cause confusion. We will revise this statement as the following clearer and more accurate description about our checkpoint selection process: "We apply the same checkpoint selection strategy to all models and datasets. For each run, we train for one epoch and use the single checkpoint saved at the end of this epoch to compute test AUC for all tasks."
-
For a given dataset, all models use the same tokenization scheme, and the same number and partitioning of input tokens. The identical tower input dimensions and architectures are used to all model. The numbers of the dense parameter across different models are given in the following table.
| Model | QK-Video | KuaiRand | TAAC-25 | MerRec |
|---|---|---|---|---|
| UniMixer | 12.6944 | 12.8013 | 12.5264 | 12.8105 |
| nUniMixer | 12.5095 | 13.0157 | 12.7331 | 12.8048 |
| FAT | 12.6103 | 12.2130 | 12.3862 | 12.9192 |
| Wukong | 12.7525 | 12.5940 | 12.5980 | 12.4596 |
| RankMixer | 13.0901 | 12.5893 | 12.6395 | 12.6095 |
| TokenMixer-Large | 12.6862 | 12.8924 | 12.8493 | 12.8368 |
| HiFormer | 12.6924 | 12.5510 | 12.6308 | 12.7181 |
We are grateful for your insightful review and constructive feedback on our manuscript. We have carefully considered each point you raised, and our detailed responses are outlined below. For clarity, we have denoted Weaknesses with W.
W1: Regarding technical novelty of nUniMixer Thank you for your comments. As mentioned in this paper, we summarize the main technical contributions as follows:
, where B=D/T and L=TD. In addition, the Sinkhorn-Knopp and the temperature annealing operations are used to ensure the properties of mixer parameter matrices given in Theorem 3.2.