- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RISE, a plug-in module for unified ranking models. It combines Attention Modulation and Position Adaptation to add a low-dimensional cumulative correction to native attention scores. RISE is integrated into five ranking backbones and evaluated on four datasets. The paper reports label-macro ROC-AUC, theoretical properties, component ablations, width studies, and order interventions.
Critical strengths
-
RISE is a clearly defined and broadly applicable plug-in. It preserves each backbone's native weighting and value aggregation while adding a low-dimensional score correction. The implementation covers five substantially different ranking backbones, which supports the engineering generality of the interface.
-
The reported aggregate gains are directionally consistent. RISE improves label-macro ROC-AUC in all 20 model-dataset cells. The equally weighted macro AUC rises from
0.859529to0.865410. The HyFormer--MerRec ablation also shows that full AM+PA outperforms content-only, AM-only, and PA-only variants.
Critical weaknesses
-
The main statistical evidence is incomplete and asymmetric. RISE results are reported as means over three seeds, but the paper does not clearly state whether baselines use the same matched seeds. Table 1 reports no standard deviations, confidence intervals, or paired tests. Several QK-Video gains are only about
0.0004--0.0011, so it is unclear whether they exceed training variation. -
Some insertion sites accumulate state across artificial packing boundaries without direct controls. For example, HyFormer query decoding allows later global queries to inherit increments from earlier packed queries. Other sites mix sequential and non-sequential layouts. In these cases, the state may encode serialization choices rather than only behavioral context. The current reverse and shuffle probes do not fully isolate this effect, and Table 1 does not report per-label results that could reveal important regressions hidden by macro averaging.
Questions for authors
- Clarification and potentially score-changing evidence: Were the baseline models also evaluated with the same three matched seeds? Please report per-seed paired gains and uncertainty for all 20 cells.
- Potentially score-changing analysis: Do any of the 17 individual labels regress in ROC-AUC or PR-AUC? Please report per-label performance, positive rates, and seed variance.
- Potentially score-changing control: What happens when state is reset or token order is permuted at each affected insertion site, including HyFormer global-query decoding and packed sequential/non-sequential layouts?
2 comments
W2 & Q3. How sensitive is RISE to the packing order of global queries and nonsequential features?
Feature order is a longstanding issue in CTR prediction, and ranking backbones such as OneTrans use it to determine which feature tokens can attend to one another [1, 2]. We kept the backbone layout fixed and tested both the original and reversed RISE scan orders on three backbone–dataset pairs. RISE improved macro ROC-AUC over the baseline on average in both directions.
Prior work. Earlier CTR studies asked what happens when a ranker assigns positions to fields that have no natural order. Chan et al. trained the same CNN architecture on 50 random field orders. Changing the order changed which fields fell within a convolutional window. Across those orders, AUC ranged from 0.7410 to 0.7461 on Avazu and from 0.6861 to 0.7488 on Tencent [1, Fig. 2]. The larger spread on Tencent shows that sensitivity to field order can vary across datasets.
In unified ranking, OneTrans illustrates where this dependency enters the backbone. Every nonsequential token can attend to the full behavioral history. If feature A comes before feature B, B can also attend to A. Swapping them reverses which feature can read the other [2, §3.3.1]. A related experiment in OneTrans-V2 changed the order of three decisions in its retrieval decoder. Purchase HR@128 was 0.4590 when the purchase decision came first and 0.4520 when it came last [3, Table 7].
For RISE, the question is whether adding the module can improve ranking under more than one order even when the order affects performance. Liu et al. made this comparison in CTR prediction. For each of ten field orders, they evaluated FGCNN with and without a layer that combines features across convolutional windows. On both Criteo and Huawei App Store, AUC varied with field order, yet the layer improved AUC under every order shown [4, Fig. 6].
Our experiments. We reversed only the RISE scan over HyFormer’s global queries or OneTrans’s nonsequential tokens, keeping the behavioral history and native attention mask fixed. The table reports single-seed macro ROC-AUC, with each gain over baseline in parentheses. Arrows indicate training order → evaluation order.
| Backbone–dataset | Baseline | Original → original | Reverse → reverse |
|---|---|---|---|
| HyFormer–MerRec | 0.7856338 | 0.8008724 (+0.0152386) | 0.8014624 (+0.0158286) |
| HyFormer–KuaiRand | 0.8566752 | 0.8619408 (+0.0052656) | 0.8624053 (+0.0057301) |
| OneTrans–KuaiRand | 0.8611243 | 0.8618298 (+0.0007055) | 0.8640480 (+0.0029237) |
| Mean gain over baseline | — | +0.00707 | +0.00816 |
The last row averages the three gains in each column. For OneTrans–KuaiRand, the original-order entry comes from the main experiment. The separate order-control run scored 0.8608158, slightly below baseline. Using that run gives a mean original-order gain of +0.00673 across the same three pairs.
HyFormer’s global queries read the same behavioral history [5, Eq. 9]. On MerRec, we isolated the queries by letting each use only its own increment alongside that history. The comparison with full RISE also reflects a change in how the two branches are computed.
| HyFormer–MerRec configuration | Macro ROC-AUC (single-seed) |
|---|---|
| Full RISE (original order) | 0.8008724 |
| RISE with query isolation | 0.7982020 |
Both scan directions preserve the behavioral history available to each query. Reversing the scan changes which other queries or features can influence a score, so the size of the gain can vary with the order. On HyFormer–MerRec, RISE remained above the baseline even when each query used only its own increment, suggesting that part of the gain can be obtained without cross-query accumulation.
References
[1] Patrick P. K. Chan et al. Convolutional Neural Networks based Click-Through Rate Prediction with Multiple Feature Sequences.
[2] Zhaoqi Zhang et al. OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender.
[3] Hannan Cao et al. OneTrans-V2: Unifying Retrieval, Pre-rank, and Fine-rank with One Transformer in Industrial Recommender.
[4] Bin Liu et al. Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction.
[5] Yunwen Huang et al. HyFormer: Revisiting the Roles of Sequence Modeling and Feature Interaction in CTR Prediction.
Thank you for your careful review and helpful feedback.
W1 & Q1. Were the baselines evaluated with multiple seeds, and how stable are the gains?
We now report results for each baseline across three random seeds. RISE improved mean macro ROC-AUC in all 20 backbone–dataset settings, raising the equally weighted average from 0.859420 to 0.865410 (+0.005990). Each cell in the three result columns gives the mean ± sample standard deviation across seeds. The last column uses the gain for each seed (RISE minus baseline), so its standard deviation shows how much the gain varied across runs.
W2 & Q2. Do any individual labels regress in ROC-AUC or PR performance?
Yes. Of 85 backbone–label combinations, 80 improved and five declined in multi-seed mean ROC-AUC. For single-seed PR-AUC, 81 improved and four declined. The table gives the counts by dataset and highlights one label from each dataset with gains in both metrics. Each Δ is RISE minus baseline.
For HiFormer, ROC-AUC declined on MerRec Offer and Checkout and on KuaiRand like and comment. The fifth decline was OneTrans on TAAC-25 conversion. PR-AUC declined on MerRec Checkout with HiFormer, MerRec Purchase with HyFormer and OneTrans, and TAAC-25 click with OneTrans.
Due to space constraints, we have uploaded the full per-label results for all 85 backbone–label combinations as Supplementary Material.