acceptodds
Under review as a conference paper at ICLR 2027

PRECISION-WEIGHTED ATTENTION: RESTORING UNCERTAINTY TO THE TRANSFORMER FOR SEQUENTIAL RECOMMENDATION

Abstract

The Transformer treats every token with uniform confidence: attention weights a position by how relevant it is to the query, never by how much evidence supports its representation. Sequential recommendation violates this assumption in every attention row, where an item observed a handful of times sits beside one observed thousands of times. We read the pre-norm Transformer layer as one cycle of a Bayesian filter: attention observes, the residual connection updates, and the feedforward sublayer predicts. Under this reading exactly one quantity is missing from the standard layer, the precision of its state. Restoring it gives precision-weighted attention (PWA): each token carries a positive precision alongside its hidden state, and one observe–update–predict cycle per layer uses it as a bias on the attention logits toward better-supported positions, as the gain of the residual update, and as a quantity transported through the feed-forward sublayer by its Jacobian. The observation precision is closed-form, computed from the dispersion of the pooled values and their effective sample size, and PWA adds 132 parameters, four of which train, with no learned gate. On SASRec, BERT4Rec, and HSTU across six datasets, under one recipe fixed for every backbone and dataset, with 20 matched seeds per cell (720 runs), PWA improves mean NDCG@10 in 17 of 18 backbone– dataset pairs, 15 surviving Holm correction, with the largest gains on rare items and short histories. A matched-capacity control and component ablations locate the gain in the observe and update steps, and SASRec+PWA outperforms STOSA, the closest uncertainty-aware recommender, on every dataset. The filter reading is a structured approximation, not a calibrated posterior; we state each simplification and every negative cell. Precision as a first-class state modifies the Transformer itself, and its extension beyond recommendation is future work

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
attention mechanisms, uncertainty estimation, Kalman filtering, sequential recommendation, Bayesian deep learning, transformers, recommender systems

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_tf7SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes precision-weighted attention (PWA) for sequential recommendation to account for unequal evidence supporting item representations. PWA maintains a per-token precision state that influences attention weights, residual updates, and propagation across layers. The filtering-inspired approach is evaluated on three recommendation backbones across six datasets.

Critical strengths

S1. Relevant and well-motivated problem. Item exposure varies substantially in the evaluated datasets, motivating a distinction between an interaction’s relevance and the reliability of its representation. Explicitly modeling this distinction is a meaningful direction for sparse recommendation.

S2. Interpretable design with minimal additional parameters. The combination of a layerwise precision state, a closed-form observation estimator, and precision-dependent residual gains is conceptually appealing. The derivation clearly states its assumptions, and only four added parameters receive gradients in the evaluated SASRec configuration.

Critical weaknesses

W1. The contribution of precision modeling is not sufficiently isolated. PWA also changes the residual pathway, positional bias, and dropout normalization, so the main comparisons evaluate a composite architectural modification. The matched control is limited to SASRec on two datasets with five seeds. Broader matched controls and simpler residual-gating baselines are needed to establish the specific advantage of the proposed precision mechanism.

W2. The reliability interpretation lacks direct validation and has a concrete implementation problem. The estimator’s assumptions do not hold for learned, correlated attention values, and ranking gains alone do not establish that precision measures reliability. More seriously, an attention row whose entries are all dropped receives maximum observation precision despite having no observation. This behavior requires correction and an experiment measuring its impact.

W3. The exposure prior complicates evaluation fairness. PWA’s exposure counts include validation-target identities, allowing validation-label information to influence model computation before checkpoint selection. Although this is not test-target leakage, it introduces an additional information channel. Disabling the entire prior does not isolate this effect; a training-only exposure-count comparison is needed.

Questions for authors

Please refer to the weaknesses.

Reviewer_J6nmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Precision-Weighted Attention (PWA) for sequential recommendation. The authors interpret Transformer layers as a filtering cycle and introduce a precision state to modify attention weights and residual updates. Observation precision is computed from value dispersion and effective sample size, and an approximate feed-forward Jacobian propagates the accumulated precision between layers. Experiments on three backbones and six datasets report ranking improvements over the corresponding baselines.

Critical strengths
  • The experiments report ranking improvements across several backbones and datasets.
Critical weaknesses
  • The central motivation is not empirically verified.
  • The proposed precision estimator and residual gain lack sufficient justification and comparisons with relevant alternatives.
Questions for authors

I have the following concerns and questions for the authors:

  • Motivation. The central motivation of the paper is not yet empirically verified. The mere absence of an explicit precision state does not prove that standard attention fails to account for reliability. Furthermore, Table 2 groups held-out targets by frequency rather than examining unreliable history items at comparable relevance levels. The authors should directly demonstrate this claimed failure mode in standard attention and show how PWA corrects it.

  • Observation Precision. Eq. (5) is motivated by a pseudo-likelihood assuming fixed weights and independent values sharing a common mean and variance. However, these assumptions do not hold for learned attention, and the paper does not establish whether the estimated precision actually tracks true reliability. Note that the frequency control in Table 4 alters the input prior and thus does not address this concern. Please validate the precision estimate under controlled perturbations and compare it against a simpler proxy, such as min⁡{C,a/(S+b)}\min\{C,a/(S+b)\} with positive a,ba,b tuned on the same validation data.

  • Residual Gain. Eq. (6) applies a precision-fusion gain directly to the residual increment ee, whereas standard fusion of independent state estimates would yield h+K⊙(e−h)h+K\odot(e-h). The paper currently lacks a principled residual or innovation model to justify both this specific gain formulation and the precision accumulation. Moreover, the K≡1K\equiv1 ablation does not rule out the effect of simple residual damping. The authors should compare the proposed mechanism against tuned constant gains as well as a lightweight learned sigmoid gate [R3], while keeping the carrier, attention, position bias, and dropout fixed.

  • Experiments. Relevant prior works such as AC-TSR [R1] and Rec-Denoiser [R2] are cited but omitted from the empirical comparisons. Because both methods directly modify attention in sequential recommendation, comparing only against standard backbones and STOSA is insufficient to establish PWA's unique contribution beyond existing attention-correction techniques. Please evaluate these baselines on a shared SASRec backbone under matched training objectives, data splits, checkpoint selection criteria, and candidate masking.

References:

  • [R1] Attention Calibration for Transformer-based Sequential Recommendation. CIKM 2023.
  • [R2] Denoising Self-Attentive Sequential Recommendation. RecSys 2022.
  • [R3] Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free. NeurIPS 2025.
Reviewer_9gAWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article proposes an attention mechanism called PWA for sequential recommendation, which is based on precision-weighting. PWA maintains a precision state for each hidden representation and uses it to adjust attention weights, thereby controlling residual updates. The experiments cover multiple SR backbones (SASRec, BERT4Rec, HSTU) and show that PWA can improve ranking performance in most settings.

Critical strengths
  1. The question is well motivated. It's reasonable to distinguish the 'relevance' and 'reliability' of historical behavior, especially for SR where there are significant differences in interaction frequency and history length.

  2. The precision state connects attention weighting, residual updating, and cross-layer propagation, which provides a coherent structure for incorporating reliability into the model.

Critical weaknesses
  1. The relationship with existing work needs further clarification. [1] already introduced a prior and frequency correction in user-attention modeling but have not been cited. Can you provide a comparison with [1]?

  2. The interpretation of λobs\lambda^{obs} is not enough validated. The current ablation experiments support that the mechanism is effective, but don't establish whether its benefits come from reliability estimation or scaling/regularization. Additional controls including: a constant residual gain, a conventional learnable sigmoid gate, and shuffled precision signals that preserve their marginal distribution would help the explanation.

  3. For dataset sources and scale: Although the paper used six datasets, five of them were sourced from Amazon and one from MovieLens, which only covers two sources. In addition, the maximum size of the dataset is about 1M interactions, and the maximum item corpus is about 18K. This may not reflect the impact of large-scale recommendations.

[1] Liu, Hu, et al. "Kalman filtering attention for user behavior modeling in ctr prediction." Advances in Neural Information Processing Systems 33 (2020): 9228-9238.

Questions for authors

Please refer to Weakness

Reviewer_gL9WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper applies uncertainty difference to transformer’s attention layers in sequential recommendation and proposes precision-weighted attention (PWA). Instead of distributing equal weights to cold-start items and long interaction histories items, the author gets inspiration from Bayesian filter where the precision reshapes the belief of latent states. This method evaluates and updates the precision of items for attention distinguishing and residual gains scaling. The author adopts PWA to SASRec, BERT4Rec and HSTU and reports the comparisons with STOSA.

Critical strengths

S1. This paper presents a clear research motivation and a cross-domain connection in sequential recommendation, Bayesian filter, and transformer attention. The method links local weighted discreteness, cold-start items, and residual magnitude, with only small parameter increments.

Critical weaknesses

W1. The ablation experiment is not sufficiently proved that the combination of precision control method in PWA is more effective than single residual scaling and normal gate methods. For example in Table 4, when Jacobian transport component is removed, Beauty and ML-1M get +0.62% and +0.08% in NDCG@10, which implies that the Jacobian transport is not necessary in PWA. This paper claims that the whole observe–update–predict cycle contributes to the improvement, but the effectiveness of predict phase is not clearly stated.

W2. In equation (2), the qi is the rank of item i by pre-test exposure count. However, the value of qi only represent the relative ranking among the item list, which lost the evidence of absolute difference gap between the items. For an item list, if every item gains a same large amount of exposure, the value of each qi will not change. However, it is believed that the cold-start items will gain more confidence than long-histories items which we have already convinced in such situation.

Questions for authors

Q1. What is the contribution of Jacobian transport in PWA? How does PWA compare with a residual scale and a simple learned sigmoid gate method?

Q2. How does the relative rank qi measure the uncertainty of the item list? In different datasets the same exposure number may rank differently. Please add the comparison and explain it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.