acceptodds
Under review as a conference paper at ICLR 2027

More Than a Match: State-Informed Attention for Unified Ranking

Abstract

Unified ranking models use heterogeneous features and behavioral histories as evidence for predicting user feedback. Standard attention selects this evidence through contextualized query–key compatibility, expressing content matching and predictive usefulness through the same score. We introduce RISE (Ranking with Input-dependent State Enhancement), a plug-in attention module that provides an explicit score-level channel for contextual importance. RISE combines Amplitude Modulation (AM), which accumulates input-dependent decay controls to regulate auxiliary interaction strength, and Phase Alignment (PA), which accumulates input-dependent phase controls to adjust the relative orientation of low-dimensional auxiliary projections. The transformed projections produce a signed, query-specific correction to the original attention score, while preserving the backbone's native score-to-weight rule and value aggregation. Experiments across multiple backbones and datasets show that RISE improves average AUC over the corresponding baselines. These results support explicit contextual-importance modeling alongside the representations learned by ranking backbones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Attention Mechanisms, Recommender Systems, Unified Ranking

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CgfJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RISE, a plug-in module for unified ranking models. It combines Attention Modulation and Position Adaptation to add a low-dimensional cumulative correction to native attention scores. RISE is integrated into five ranking backbones and evaluated on four datasets. The paper reports label-macro ROC-AUC, theoretical properties, component ablations, width studies, and order interventions.

Critical strengths
  1. RISE is a clearly defined and broadly applicable plug-in. It preserves each backbone's native weighting and value aggregation while adding a low-dimensional score correction. The implementation covers five substantially different ranking backbones, which supports the engineering generality of the interface.

  2. The reported aggregate gains are directionally consistent. RISE improves label-macro ROC-AUC in all 20 model-dataset cells. The equally weighted macro AUC rises from 0.859529 to 0.865410. The HyFormer--MerRec ablation also shows that full AM+PA outperforms content-only, AM-only, and PA-only variants.

Critical weaknesses
  1. The main statistical evidence is incomplete and asymmetric. RISE results are reported as means over three seeds, but the paper does not clearly state whether baselines use the same matched seeds. Table 1 reports no standard deviations, confidence intervals, or paired tests. Several QK-Video gains are only about 0.0004--0.0011, so it is unclear whether they exceed training variation.

  2. Some insertion sites accumulate state across artificial packing boundaries without direct controls. For example, HyFormer query decoding allows later global queries to inherit increments from earlier packed queries. Other sites mix sequential and non-sequential layouts. In these cases, the state may encode serialization choices rather than only behavioral context. The current reverse and shuffle probes do not fully isolate this effect, and Table 1 does not report per-label results that could reveal important regressions hidden by macro averaging.

Questions for authors
  1. Clarification and potentially score-changing evidence: Were the baseline models also evaluated with the same three matched seeds? Please report per-seed paired gains and uncertainty for all 20 cells.
  2. Potentially score-changing analysis: Do any of the 17 individual labels regress in ROC-AUC or PR-AUC? Please report per-label performance, positive rates, and seed variance.
  3. Potentially score-changing control: What happens when state is reset or token order is permuted at each affected insertion site, including HyFormer global-query decoding and packed sequential/non-sequential layouts?
2 comments

Thank you for your careful review and helpful feedback.

W1 & Q1. Were the baselines evaluated with multiple seeds, and how stable are the gains?

We now report results for each baseline across three random seeds. RISE improved mean macro ROC-AUC in all 20 backbone–dataset settings, raising the equally weighted average from 0.859420 to 0.865410 (+0.005990). Each cell in the three result columns gives the mean ± sample standard deviation across seeds. The last column uses the gain for each seed (RISE minus baseline), so its standard deviation shows how much the gain varied across runs.

BackboneDatasetBaseline ROC-AUCRISE ROC-AUCGain (RISE − baseline)
HiFormerQK-Video0.932707 ± 0.0010380.934270 ± 0.000286+0.001563 ± 0.000759
HiFormerKuaiRand0.859532 ± 0.0010150.864227 ± 0.001879+0.004696 ± 0.002170
HiFormerTAAC-250.852001 ± 0.0045150.858847 ± 0.006541+0.006846 ± 0.002060
HiFormerMerRec0.802346 ± 0.0050080.806951 ± 0.006506+0.004605 ± 0.004751
HyFormerQK-Video0.930242 ± 0.0003450.931238 ± 0.000074+0.000996 ± 0.000271
HyFormerKuaiRand0.854982 ± 0.0033890.862748 ± 0.000993+0.007766 ± 0.003195
HyFormerTAAC-250.856040 ± 0.0043410.868615 ± 0.005336+0.012576 ± 0.002266
HyFormerMerRec0.790963 ± 0.0046170.804529 ± 0.003174+0.013565 ± 0.001487
RankMixerQK-Video0.928509 ± 0.0004510.929470 ± 0.000580+0.000961 ± 0.000389
RankMixerKuaiRand0.854013 ± 0.0014820.862167 ± 0.001631+0.008154 ± 0.000914
RankMixerTAAC-250.839276 ± 0.0049120.851183 ± 0.003679+0.011907 ± 0.001518
RankMixerMerRec0.790716 ± 0.0036700.798453 ± 0.003310+0.007738 ± 0.000598
ZenithQK-Video0.934570 ± 0.0003850.935301 ± 0.000321+0.000732 ± 0.000143
ZenithKuaiRand0.859037 ± 0.0007070.863164 ± 0.002825+0.004128 ± 0.002406
ZenithTAAC-250.843662 ± 0.0038020.856290 ± 0.006811+0.012629 ± 0.003652
ZenithMerRec0.802084 ± 0.0050650.808353 ± 0.001786+0.006269 ± 0.003404
OneTransQK-Video0.933384 ± 0.0002140.934123 ± 0.000587+0.000740 ± 0.000578
OneTransKuaiRand0.859221 ± 0.0018910.863997 ± 0.001877+0.004776 ± 0.003633
OneTransTAAC-250.857725 ± 0.0047390.858627 ± 0.002821+0.000902 ± 0.007473
OneTransMerRec0.807397 ± 0.0030330.815654 ± 0.002550+0.008257 ± 0.001368

 

 

W2 & Q2. Do any individual labels regress in ROC-AUC or PR performance?

Yes. Of 85 backbone–label combinations, 80 improved and five declined in multi-seed mean ROC-AUC. For single-seed PR-AUC, 81 improved and four declined. The table gives the counts by dataset and highlights one label from each dataset with gains in both metrics. Each Δ is RISE minus baseline.

DatasetHigher ROC-AUC (multi-seed)Higher PR-AUC (single-seed)example (backbone / label)ΔROC-AUCΔPR-AUC
MerRec23/2522/25HyFormer / Like+0.0219560+0.05132989
QK-Video20/2020/20HiFormer / like+0.0013143+0.01128754
KuaiRand28/3030/30HyFormer / is_follow+0.0066803+0.02959346
TAAC-25 (TencentGR)9/109/10Zenith / is_click+0.0144323+0.01194862

For HiFormer, ROC-AUC declined on MerRec Offer and Checkout and on KuaiRand like and comment. The fifth decline was OneTrans on TAAC-25 conversion. PR-AUC declined on MerRec Checkout with HiFormer, MerRec Purchase with HyFormer and OneTrans, and TAAC-25 click with OneTrans.

Due to space constraints, we have uploaded the full per-label results for all 85 backbone–label combinations as Supplementary Material.

W2 & Q3. How sensitive is RISE to the packing order of global queries and nonsequential features?

Feature order is a longstanding issue in CTR prediction, and ranking backbones such as OneTrans use it to determine which feature tokens can attend to one another [1, 2]. We kept the backbone layout fixed and tested both the original and reversed RISE scan orders on three backbone–dataset pairs. RISE improved macro ROC-AUC over the baseline on average in both directions.

Prior work. Earlier CTR studies asked what happens when a ranker assigns positions to fields that have no natural order. Chan et al. trained the same CNN architecture on 50 random field orders. Changing the order changed which fields fell within a convolutional window. Across those orders, AUC ranged from 0.7410 to 0.7461 on Avazu and from 0.6861 to 0.7488 on Tencent [1, Fig. 2]. The larger spread on Tencent shows that sensitivity to field order can vary across datasets.

In unified ranking, OneTrans illustrates where this dependency enters the backbone. Every nonsequential token can attend to the full behavioral history. If feature A comes before feature B, B can also attend to A. Swapping them reverses which feature can read the other [2, §3.3.1]. A related experiment in OneTrans-V2 changed the order of three decisions in its retrieval decoder. Purchase HR@128 was 0.4590 when the purchase decision came first and 0.4520 when it came last [3, Table 7].

For RISE, the question is whether adding the module can improve ranking under more than one order even when the order affects performance. Liu et al. made this comparison in CTR prediction. For each of ten field orders, they evaluated FGCNN with and without a layer that combines features across convolutional windows. On both Criteo and Huawei App Store, AUC varied with field order, yet the layer improved AUC under every order shown [4, Fig. 6].

Our experiments. We reversed only the RISE scan over HyFormer’s global queries or OneTrans’s nonsequential tokens, keeping the behavioral history and native attention mask fixed. The table reports single-seed macro ROC-AUC, with each gain over baseline in parentheses. Arrows indicate training order → evaluation order.

Backbone–datasetBaselineOriginal → originalReverse → reverse
HyFormer–MerRec0.78563380.8008724 (+0.0152386)0.8014624 (+0.0158286)
HyFormer–KuaiRand0.85667520.8619408 (+0.0052656)0.8624053 (+0.0057301)
OneTrans–KuaiRand0.86112430.8618298 (+0.0007055)0.8640480 (+0.0029237)
Mean gain over baseline—+0.00707+0.00816

The last row averages the three gains in each column. For OneTrans–KuaiRand, the original-order entry comes from the main experiment. The separate order-control run scored 0.8608158, slightly below baseline. Using that run gives a mean original-order gain of +0.00673 across the same three pairs.

HyFormer’s global queries read the same behavioral history [5, Eq. 9]. On MerRec, we isolated the queries by letting each use only its own increment alongside that history. The comparison with full RISE also reflects a change in how the two branches are computed.

HyFormer–MerRec configurationMacro ROC-AUC (single-seed)
Full RISE (original order)0.8008724
RISE with query isolation0.7982020

Both scan directions preserve the behavioral history available to each query. Reversing the scan changes which other queries or features can influence a score, so the size of the gain can vary with the order. On HyFormer–MerRec, RISE remained above the baseline even when each query used only its own increment, suggesting that part of the gain can be obtained without cross-query accumulation.

References

[1] Patrick P. K. Chan et al. Convolutional Neural Networks based Click-Through Rate Prediction with Multiple Feature Sequences.

[2] Zhaoqi Zhang et al. OneTrans: Unified Feature Interaction and Sequence Modeling with One Transformer in Industrial Recommender.

[3] Hannan Cao et al. OneTrans-V2: Unifying Retrieval, Pre-rank, and Fine-rank with One Transformer in Industrial Recommender.

[4] Bin Liu et al. Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction.

[5] Yunwen Huang et al. HyFormer: Revisiting the Roles of Sequence Modeling and Feature Interaction in CTR Prediction.

Reviewer_XfkEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes two mechanisms (Amplitude Modulation (AM) and Phase Alignment (PA)) that augment default attention scores to help unified recommendation rankers focus on salient user interactions.

Critical strengths
  • The approach is a relatively straightforward modification to any attention-based unified ranker -- basically plug-and-play.
  • A well-designed empirical study shows consistent improvements across 5 models and 4 datasets. Ablations provide evidence that the AM and PA signals are highly complementary.
Critical weaknesses
  • Even after several readings, I do not really understand the intuition behind the proposed AM and PA mechanisms. Especially: Why are they needed on top of (very powerful and general-purpose) attention? If these kinds of signals are valuable, I would expect attention to learn them itself.
  • The magnitude of the empirical improvements is small, averaging less than 0.001 AUC. To me, the quality of the empirical results is especially important here, since I do not feel that the motivation/intuition behind the AM and PA mechanisms was well explained (see above).
Questions for authors
  • Why are the AM and PA mechanisms needed on top of (very powerful and general-purpose) attention?
1 comment

We appreciate your insightful comments and suggestions.

W1 & Q1. Why add AM and PA to an attention-based ranker?

What does standard attention leave implicit? A standard attention head computes the unnormalized score for a historical item from the candidate’s query and that item’s key, so intervening events affect this score only through what those two vectors encode. RISE gives those events a direct role in scoring the same pair. It accumulates a learned decay and rotation from each event to adjust the strength (AM) and relative alignment (PA) of an auxiliary match, then adds that match to the native score.

Why does this matter for ranking? The same past interaction can provide different evidence for the same candidate depending on what the user did next. In Figure 1, the user previously viewed a tent, and the ranker scores the same tent candidate in both histories. Subsequent views of a sleeping bag and a camping stove make the earlier tent view evidence of continued camping interest. Views of a suitcase and a shirt provide less support for that interpretation. Zhou et al. [1, Table 2] found that DIEN, which models how interests evolve for a given target, outperformed DIN on Amazon Books and Electronics. On Taobao, Zhang et al. [2, Fig. 10] found that historical category sequences such as AAB, ABA, and BAA had similar associations with the click probabilities predicted by DIN and DIEN.

Why accumulate two controls? An intervening event can affect both how much of a match carries forward and how the auxiliary query and key align. In the query-referenced, unclipped case of Eq. (5), subtracting the endpoint prefix states isolates the updates over that interval. AM determines how much of the auxiliary match carries forward. Lin et al. [3, Eq. (11)] used a related idea in language modeling, accumulating input-dependent forget gates over each query–key interval to attenuate attention. PA adjusts alignment. Su et al. [4] showed how rotating the query and key makes their dot product depend on their relative angle. In RISE, the events along the interval determine that angle, so they can even change the sign of the auxiliary score. RISE adds the resulting correction before the backbone aggregates values.

Can explicit sequence structure help models that already use attention? A multilayer Transformer can already encode sentence context. Yang et al. [5, Table 1] nevertheless improved English–German translation by feeding a sentence summary directly into the encoder’s queries and keys. Huang et al. [6, Fig. 4] tested how readily self-attention learns sequence structure with limited data. Adding recurrent structure to Transformer-XL improved sample efficiency on subsets of the Enwik8 training set.

What do AM and PA add to rankers that already use attention? HiFormer, HyFormer, and OneTrans all use multi-head attention. Adding RISE improved macro ROC-AUC for each of these three backbones on each of the four datasets (12/12 settings, Table 1). The single-seed HyFormer–MerRec ablation tests which parts of RISE contribute to this ranking gain. The auxiliary content score alone reached 0.7831 macro ROC-AUC. Adding AM alone raised it to 0.7959, adding PA alone raised it to 0.7952, and combining them gave the highest score, 0.8009. At the same insertion sites, a standard auxiliary attention head reached 0.7808 with nearly the same number of added parameters (125,958 for the head and 125,976 for RISE).

 

 

W2. How large and consistent are the AUC gains?

We evaluate five strong baselines for large-scale recommendation, and further AUC gains on these models can be difficult to achieve. Each backbone already models both behavioral histories and feature interactions (§4.1). For context, Feng et al. [7, Table 1] reported absolute AUC gains of 0.0032 on advertising and 0.0042 on recommendation when comparing DSIN with DIEN. In our three-seed evaluation on TAAC-25, adding RISE improves macro ROC-AUC by more than 0.010 for three different backbones (HyFormer, RankMixer, and Zenith). For HyFormer, the gain also exceeds 0.010 on MerRec.

RISE improves all five backbones on all four datasets in the submitted Table 1 (20/20 settings). The gains vary in magnitude, but every backbone improves across video recommendation, advertising, and e-commerce.

 

References

[1] Guorui Zhou et al. Deep Interest Evolution Network for Click-Through Rate Prediction.

[2] Hengyu Zhang et al. Deep Pattern Network for Click-Through Rate Prediction.

[3] Zhixuan Lin et al. Forgetting Transformer: Softmax Attention with a Forget Gate.

[4] Jianlin Su et al. RoFormer: Enhanced Transformer with Rotary Position Embedding.

[5] Baosong Yang et al. Context-Aware Self-Attention Networks.

[6] Feiqing Huang et al. Encoding Recurrence into Transformers.

[7] Yufei Feng et al. Deep Session Interest Network for Click-Through Rate Prediction.

Reviewer_zcDnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RISE, a score-level module for recommendation models. It adds a low-dimensional correction to native attention scores: Amplitude Modulation (AM) uses cumulative decay controls to change interaction strength, while Phase Alignment (PA) uses cumulative phase controls to rotate auxiliary query and key projections. The authors evaluate RISE on five backbones and four datasets, reporting higher label-averaged AUC in all 20 model–dataset settings.

Critical strengths
  1. The paper asks a relevant modeling question. Historical events can have different predictive value depending on surrounding behavior, and RISE provides an explicit mechanism for injecting order-dependent context into attention scores.
  2. The method is designed to fit existing ranking architectures. It adds a correction to native scores while retaining the backbone’s weighting rule and value aggregation, making the proposal relatively easy to compare across different model families.
Critical weaknesses
  1. The paper should clarify its main contribution. RISE computes an additional score correction from learned projections of the same token representations, so it is unclear how “contextual importance” differs from another compatibility function. The paper does not establish whether RISE represents functions that standard multi-head attention cannot; if it does not, its contribution is better characterized as a parameterization or inductive bias than as a new modeling capability.
  2. The component ablation lacks a parameter-matched auxiliary attention baseline, leaving it unclear whether the gains come from the AM/PA cumulative-state mechanism or simply increased model capacity.
Questions for authors
  1. Can standard multi-head attention represent RISE’s score correction? If so, what is RISE’s distinct contribution?
  2. Could the authors compare RISE with a parameter-matched auxiliary attention head to determine whether the gains come from the AM/PA cumulative-state mechanism or simply increased model capacity?
1 comment

Thank you for your thoughtful comments and questions.

W1 & Q1. Can standard multi-head attention represent RISE’s score correction? What is RISE’s distinct contribution?

Standard dot-product attention computes each pairwise score from a query and a key, so changes in the intervening sequence affect that score only through the information encoded in these two endpoint representations. When such changes are not fully captured, the score may respond weakly to behavioral context that changes a historical item’s predictive value. RISE therefore uses cumulative AM/PA controls to let intervening inputs explicitly modulate the auxiliary match, complementing the backbone’s native multi-head attention.

Previous work. Yang et al. [1] show that context summarized by earlier Transformer layers can remain underused in matching. Explicitly incorporating global and deep context into Q/K improves translation even though the representations are already contextualized. Huang et al. [2] combine recurrence encodings with ordinary self-attention through a learned gate. Their improved sample efficiency over the Transformer baseline shows that explicitly parameterizing sequence dynamics can help even when MHA is already present.

In recommendation, DIEN [3] identifies a related limitation in DIN’s target-based attention: it can select relevant historical behaviors while underusing dependencies between successive behaviors. In unified ranking, these dependencies matter because intervening events can change the predictive value of an otherwise relevant historical item. In Figure 1, the same historical tent interaction and tent candidate appear in both histories: sleeping-bag and camping-stove interactions support ongoing camping interest, whereas suitcase and shirt interactions provide less corroboration.

At the score level, Lin et al. [4] use input-dependent forget gates accumulated along the query–key interval to construct an additive logit bias, giving intervening inputs a direct role in modulating the pair’s attention score.

RISE makes this sequence-dependent modulation separately learnable alongside native compatibility. Its input-dependent controls accumulate along the supplied order: AM adjusts the auxiliary interaction’s strength, while PA adjusts its alignment. The resulting correction is added to the native score before the backbone’s original weighting and value aggregation.

Our experiments. HiFormer, HyFormer, and OneTrans already use multi-head attention. Adding RISE improves the multi-seed mean macro ROC-AUC in all 12 backbone–dataset settings for these three models across four datasets. These gains show that RISE remains useful alongside the MHA already present in the backbones. Across all five evaluated backbones, the gains hold in all 20 backbone–dataset settings.

Figure 6 examines whether the trained models use the sequence information supplied through RISE. Reversing or shuffling only RISE’s history computation, while holding native Q/K/V fixed and mapping corrections back to the original records, increases click BCE at both evaluated pooling sites. The trained predictors therefore use the ordering supplied to this additional pathway.

The Content-only variant in the HyFormer–MerRec ablation already adds a learned auxiliary compatibility score. AM-only and PA-only both improve this variant, and combining them gives the highest AUC and lowest logloss. Full RISE also outperforms the near-parameter-matched ordinary attention branch (Q2). Together, these comparisons support the effectiveness of the combined score parameterization in this setting.

References

[1] Baosong Yang et al. Context-Aware Self-Attention Networks.

[2] Feiqing Huang et al. Encoding Recurrence into Transformers.

[3] Guorui Zhou et al. Deep Interest Evolution Network for Click-Through Rate Prediction.

[4] Zhixuan Lin et al. Forgetting Transformer: Softmax Attention with a Forget Gate.

 

 

W2 & Q2. Could the authors compare RISE with a parameter-matched auxiliary attention head?

We extended the HyFormer–MerRec component ablation with a standard auxiliary Q/K/V/O attention branch whose additional parameter budget nearly matches full RISE. The control uses the same insertion sites and training protocol. The table reports results for all five configurations.

ConfigurationMacro ROC-AUC ↑
Content-only0.7831212
AM-only0.7958800
PA-only0.7952302
Matched Aux-Attn0.7808444
Full RISE0.8008724

AM-only and PA-only both improve on content-only, and combining them in full RISE gives the highest AUC. Compared with matched Aux-Attn, full RISE improves macro ROC-AUC by 0.0200280, supporting its combined score-level design as a more effective use of the additional capacity in this setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.