acceptodds
Under review as a conference paper at ICLR 2027

Does Positional Encoding Matter for Semantic ID-based Generative Recommendation?

Abstract

Semantic IDs have emerged as an effective representation for generative recommendation, where each item is represented by multiple discrete codes. These codes are typically serialized along the user interaction sequence into a flattened token sequence for generative modeling. However, such a formulation couples two structurally different sources of positional variation: the sequential order of user interactions and the hierarchical order induced by residual quantization. We argue that these two structures should be modeled on their respective positional axes. Based on this insight, we propose SimPos, a simple structure-aware positional modeling method that replaces the Transformer backbone's positional encoding with separate interaction-level and residual-level attention biases, requiring no additional learnable parameters. Experiments on four real-world recommendation datasets with different backbones and residual tokenizers show that SimPos consistently outperforms existing positional encoding methods, with relative gains of up to +25.25% in Recall@5. These results indicate that positional encoding matters in semantic ID-based generative recommendation, and that directly inheriting it from the Transformer backbone is suboptimal. Our code and data are available at https://anonymous.4open.science/r/SimPos-8CFC/.

Primary area:
generative models
Keywords:
Generative Recommendation, Semantic IDs, Positional Encoding, Residual Quantization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bZK4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies positional encoding in Semantic ID based generative recommendation. The authors argue that flattening multi-code items into a single sequence mixes interaction positions with residual code positions. They propose SimPos, which models these two positional relations using separate linear attention biases without introducing additional learnable parameters.

Critical strengths

The proposed method is simple and easy to integrate into existing generative recommendation models without introducing additional learnable parameters.

Critical weaknesses

W1: The idea of modeling item structure and separating positional information is not entirely new. DiscRec has already introduced within-item positional embeddings for Semantic ID recommendation, while BARGE also studies the structural issues caused by flattening item tokens. The paper does not sufficiently distinguish its contribution from these related methods.

W2: The motivation and experimental evidence are not fully aligned. Flattening fixed-length Semantic IDs does not inherently lose positional information, but changes how positional relations are represented. Moreover, the proposed interaction bias introduces a strong recency preference. It remains unclear how much of the improvement comes from separating the two positional axes rather than simply imposing a better recency bias. The assumption that nearby residual levels should receive stronger attention also deserves further justification.

Questions for authors

See weakness.

7 comments
AuthorsOpenReview ↗

Response to Reviewer bZK4 (1/4)

We sincerely thank the reviewer for recognizing the simplicity and ease of integration of SimPos, as well as for the constructive feedback. We carefully address these concerns below. (All new results use five training seeds per configuration.)

W1. Distinction from DiscRec and BARGE.

While DiscRec [1] and BARGE [2] also consider item-level structure in generative recommendation, they leverage this structure for different purposes.

DiscRec introduces learnable item-level position embeddings to distinguish code positions within each SID, together with localized intra-item attention in a separate collaborative branch to disentangle semantic and collaborative signals. BARGE addresses the item-boundary gap by using Item Context-Aware Attention (ICA) to incorporate item-level context into token representations, enabling the encoder to better preserve item grouping in flattened SID sequences.

SimPos, in contrast, directly models the two distinct positional relationships between tokens. It replaces the backbone's original positional encoding with two parameter-free relative biases that separately capture distances between interactions and between residual levels. Unlike DiscRec's position embeddings or BARGE's item-context aggregation, SimPos explicitly models both positional relationships without additional learnable modules.

We have expanded §2.1 to discuss DiscRec in the revised PDF and clarify its distinction from SimPos. We also note that BARGE was first posted to arXiv in July 2026, shortly before our submission.

[1] Liu, Chang, et al. "DiscRec: Disentangled Semantic–Collaborative Modeling for Generative Recommendation." arXiv:2506.15576, 2025.

[2] Zeng, Junchao, et al. "Bridging the Structural Gap: Adapting Autoregressive Generation for Recommendation." arXiv:2607.21028, 2026.

AuthorsOpenReview ↗

Response to Reviewer bZK4 (2/4)

W2a. Structural inductive bias rather than information loss.

The advantage of SimPos lies in its structural inductive bias, rather than in recovering information lost by flattening. For fixed-length SIDs with KK codes, a flattened absolute position pp determines both coordinates, t=⌈p/K⌉t = \lceil p / K \rceil (interaction position) and k=((p−1) mod K)+1k = ((p-1) \bmod K) + 1 (residual level), so LPE retains full positional information in principle. However, SimPos outperforms LPE across all 32 setting–metric combinations (Table 3), indicating that preserving positional information alone is insufficient to achieve the same performance.

Since flattening changes how positional relationships are represented rather than what information is available, we further examine whether explicitly encoding both positional coordinates is sufficient. We introduce a factorized absolute PE baseline that encodes (t,k)(t,k) through two learnable absolute embeddings, Eitem(t)+Elevel(k)E_{\mathrm{item}}(t) + E_{\mathrm{level}}(k), added to the input rather than through relative attention biases. This baseline performs similarly to LPE, improving in only 4 of the 32 setting–metric combinations (median relative change: −1.16%). As shown in Table R1, SimPos significantly outperforms factorized absolute PE in all 32 comparisons, with relative gains of 8.19%–38.4% (median: 16.54%). These results suggest that explicitly representing both positional coordinates alone is insufficient, highlighting the benefit of directly modeling their relative relationships through attention biases.

We have clarified this distinction in §3.2 of the revised PDF and revised the corresponding statements in the Introduction and Conclusion to avoid implying that flattening inherently loses positional information.

Table R1. Comparison of SimPos with LPE, factorized absolute PE, and ALiBi, averaged across four datasets for each backbone–tokenizer setting.

Backbone / TokenizerLPEFactorized abs. PEALiBiSimPos
T5 / RQ-VAE0.0217 / 0.01360.0214 / 0.01340.0224 / 0.01390.0257 / 0.0164
T5 / RQ-KMeans0.0227 / 0.01420.0224 / 0.01400.0237 / 0.01510.0273 / 0.0180
GPT-2 / RQ-VAE0.0217 / 0.01360.0214 / 0.01340.0205 / 0.01310.0242 / 0.0157
GPT-2 / RQ-KMeans0.0230 / 0.01490.0230 / 0.01480.0221 / 0.01460.0261 / 0.0169

Each cell reports R@5 / N@5, averaged over five seeds and four datasets (Beauty, Toys, Sports, and CDs). SimPos significantly outperforms all three baselines on both metrics across all datasets and settings (one-sided paired t-tests over five seeds, all Holm-adjusted p < 0.005 across 96 comparisons). The full per-dataset results are reported in Appendix C of the revised PDF.

AuthorsOpenReview ↗

Response to Reviewer bZK4 (3/4)

W2b. Contribution Beyond Recency Bias.

The interaction-level bias in SimPos introduces a recency preference, while its residual-level bias captures a separate structural relationship independent of interaction distance. To examine whether recency alone explains the performance gains, we conduct two complementary comparisons.

  • Recency without the residual axis. We consider an interaction-only variant (λres=0\lambda_{\mathrm{res}}=0), which retains the same interaction-level recency bias as SimPos while removing the residual-level component.

  • Tuned recency without axis separation. We introduce a flattened linear relative bias (FlatBias), −λ ∣p−p′∣-\lambda\ |p - p'|, which keeps SimPos's linear relative-bias form and a recency prior but uses a single flattened distance instead of separate positional axes. We sweep its strength λ over seven values (Table R2).

As shown in Figures 3 and 4 (§4.3.1), SimPos consistently outperforms the interaction-only variant, indicating that the residual-level bias provides additional benefits beyond interaction-level recency alone. Furthermore, Table R2 shows that SimPos maintains its advantage across the tested bias strengths. The per-dataset results in Appendix C of the revised PDF confirm that SimPos outperforms FlatBias across all 32 setting–metric comparisons, with relative gains of 2.70%–31.22% in R@5 and 6.97%–29.41% in N@5.

Together, these results suggest that SimPos's improvements cannot be explained solely by recency bias, but also arise from explicitly modeling interaction-level and residual-level positional relationships separately. In the revised manuscript, we have added this comparison as §4.3.4 and Appendix C, and revised the recency discussion in §4.3.2 accordingly.

Table R2. Sweep over the strength λ of the flattened linear relative bias, averaged over five seeds and all 16 settings (4 datasets × 2 backbones × 2 tokenizers).

λ0.20.40.60.81.01.52.0SimPos
R@50.02180.02220.02220.02220.02250.02210.02100.0258
N@50.01400.01420.01430.01420.01440.01410.01340.0167

SimPos significantly outperforms the best configuration of this baseline in all 32 setting–metric comparisons (one-sided paired t-tests over five seeds, all Holm-adjusted p<0.01p < 0.01). The sweeps per backbone–tokenizer setting and the per-dataset results are reported in Appendix C of the revised PDF.

Reviewer_ALLzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper considers generative recommendation via semantic IDs, a widely used technique in industrial recommendation systems. The authors focus on one question: does positional encoding matter in this area of research? To understand this, they first revisit the widely used methods including RQ-VAE and RQ-KMeans, and then propose a novel method called SimPos, which is a parameter-free positional encoding for semantic-ID based and generative recommendation systems. In particular, as compared to classical methods that use relative position information, the authors use the reasonable intuition that the positional encoding should be decoupled as interaction-level and residual-level positions.

Experiments show that SimPos improves all the performance metrics across different tokenizers (RQ-VAE and RQ-KMeans) and backbones (GPT-2 and T5).

Critical strengths
  1. The authors present a novel idea on how one should design positional embeddings in generative recommenders -- instead of considering each token position, one should design interaction-level embeddings and residual-level embeddings.

  2. Experiments on both RQ-VAE and RQ-KMeans tokenizers with GPT-2 or T5 as backbones are presented. Results show improvements on all metrics. Code is available to reproduce the results.

Critical weaknesses
  1. There are similar ideas in prior works such as [1] and [2]. This might be a novelty concern for this paper.

  2. Current results are only on some public datasets and it is still unclear the advantages of this method in industrial-scale datasets.

References

[1] CoFiRec: Coarse-to-Fine Tokenization for Generative Recommendation.

[2] Train short, test long: Attention with linear biases enables input length extrapolation.

Questions for authors
  1. Some prior work, for example [1], also discuss the usage of positional encoding in generative recommendation systems. Could the authors have a comparison or discussion for these prior works?

  2. Any industrial-scale adoption of this type of positional encodings? The current setup is on public datasets and not proven on industrial-scale datasets yet.

  3. NoPE seems worse than all methods as claimed by the authors. Any insights on why it is not useful in recommendation systems but it is useful in some LLMs, for example [2], from frontier labs?

References

[1] CoFiRec: Coarse-to-Fine Tokenization for Generative Recommendation.

[2] KIMI K3: OPEN FRONTIER INTELLIGENCE.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ALLz (1/3)

We sincerely thank the reviewer for recognizing the motivation of SimPos and its consistent improvements across different backbones and tokenizers. We address the concerns and questions below. (All new results use five training seeds per configuration.)

W1 / Q1. Novelty and comparison with prior work.

While CoFiRec and ALiBi are related to our work, they address positional information differently.

  • CoFiRec incorporates positional information through learnable level and item-position embeddings added to the input representations: et(k)=Eid(sit(k))+Elevel(k)+Epos(t)e_t^{(k)} = E_{\mathrm{id}}(s_{i_t}^{(k)}) + E_{\mathrm{level}}(k) + E_{\mathrm{pos}}(t) (Eq. 12), where kk denotes the semantic level within its SID and tt denotes the item position in the interaction history. In contrast, SimPos directly replaces the backbone's original positional encoding with parameter-free relative attention biases that separately model interaction-level and residual-level distances. To further examine this distinction, we introduce a factorized absolute PE baseline that explicitly encodes both positional dimensions through separate learnable embeddings, similar to CoFiRec's positional formulation. Across the 32 setting–metric combinations (4 datasets × 2 backbones × 2 tokenizers × 2 metrics), this baseline improves over LPE in only 4 cases, with a median relative change of −1.16%. SimPos significantly outperforms the factorized absolute PE baseline in all 32 comparisons, with relative gains of 8.19%–38.40% (median: 16.54%). Table R1 summarizes these results, indicating that modeling the two positional dimensions through relative attention biases is more effective than using separate learnable absolute embeddings.
  • ALiBi has already been discussed in §2.2 and included as a baseline in Tables 3–4. It introduces linear attention biases based on distances along a single flattened token sequence, with fixed head-specific slopes. SimPos instead models interaction-level and residual-level distances with separate bias coefficients. Additional experiments with tuned ALiBi and a tuned single-axis linear bias (Table R2 in our response to Reviewer 76mt; §4.3.4 and Appendix C of the revised PDF) further show that SimPos consistently outperforms both variants, suggesting that its improvements cannot be explained solely by better slope selection.

We have also expanded §2.1 to discuss CoFiRec and DiscRec in the revised PDF and clarify how SimPos differs from existing approaches to modeling SID structure.

Table R1. Comparison of SimPos with LPE, factorized absolute PE, and ALiBi, averaged across four datasets for each backbone–tokenizer setting.

Backbone / TokenizerLPEFactorized abs. PEALiBiSimPos
T5 / RQ-VAE0.0217 / 0.01360.0214 / 0.01340.0224 / 0.01390.0257 / 0.0164
T5 / RQ-KMeans0.0227 / 0.01420.0224 / 0.01400.0237 / 0.01510.0273 / 0.0180
GPT-2 / RQ-VAE0.0217 / 0.01360.0214 / 0.01340.0205 / 0.01310.0242 / 0.0157
GPT-2 / RQ-KMeans0.0230 / 0.01490.0230 / 0.01480.0221 / 0.01460.0261 / 0.0169

Each cell reports R@5 / N@5, averaged over five seeds and four datasets (Beauty, Toys, Sports, and CDs). SimPos significantly outperforms all three baselines on both metrics across all datasets and settings (one-sided paired t-tests over five seeds, all Holm-adjusted p < 0.005 across 96 comparisons). The full per-dataset results are reported in Appendix C of the revised PDF.

AuthorsOpenReview ↗

Response to Reviewer ALLz (2/3)

W2 & Q2. Evaluation on industrial-scale datasets.

As already acknowledged in §5 (Limitations and Future Work), evaluation on industrial-scale datasets remains an important direction for future work. To further strengthen our empirical evaluation, we conducted additional experiments on Amazon Books, a larger public recommendation benchmark containing 139,089 users, 113,175 items, and 3.34 million interactions after 10-core filtering (about 4.6× as many interactions as Toys, the largest dataset in our main experiments). We follow the experimental setup in §4.1 and set λint=0.6\lambda_{\mathrm{int}}=0.6 and λres=0.1\lambda_{\mathrm{res}}=0.1 for SimPos across all four settings. As shown in Table R3 (see Appendix A of the revised PDF for details), across both T5 and GPT-2 backbones and two SID tokenizers (RQ-VAE and RQ-KMeans), SimPos consistently outperforms all compared positional encoding baselines, with relative gains over the strongest baseline of +8.41% to +16.32% in R@5 and +6.47% to +17.36% in N@5.

Although we have not evaluated SimPos in an industrial setting, its parameter-free design introduces only additive attention biases, without modifying the SID tokenizer or training objective. We also note that representative SID-based generative recommendation studies, including TIGER [1], LETTER [2], and RPG [3], primarily evaluate on publicly available benchmarks. Our additional results strengthen the evidence for SimPos's effectiveness on larger-scale public data, while industrial-scale validation remains beyond the scope of our current evaluation.

[1] Rajput, Shashank, et al. "Recommender systems with generative retrieval." NeurIPS, 2023.

[2] Wang, Wenjie, et al. "Learnable item tokenization for generative recommendation." CIKM, 2024.

[3] Hou, Yupeng, et al. "Generating long semantic ids in parallel for recommendation." KDD, 2025.

Table R3. Performance of different positional encodings on Amazon Books with two backbones and two tokenizers.

Backbone / TokenizerLPERelPERoPEALiBiSimPosImprov.
T5 / RQ-VAE0.0308 / 0.01920.0381 / 0.02530.0342 / 0.02310.0397 / 0.02470.0441 / 0.0289+11.08% / +14.23%
T5 / RQ-KMeans0.0312 / 0.01980.0393 / 0.02420.0339 / 0.02180.0335 / 0.02120.0431 / 0.0284+9.67% / +17.36%
GPT-2 / RQ-VAE0.0184 / 0.01140.0190 / 0.01180.0175 / 0.01070.0186 / 0.01220.0221 / 0.0138+16.32% / +13.11%
GPT-2 / RQ-KMeans0.0210 / 0.01320.0199 / 0.01280.0213 / 0.01360.0214 / 0.01390.0232 / 0.0148+8.41% / +6.47%

Amazon Books, R@5 / N@5, averaged over five seeds (std ≤ 0.0007 for all entries). Improv. is relative to the strongest baseline per metric. SimPos significantly outperforms all four baselines on both metrics in every setting (one-sided paired t-tests over five seeds, all Holm-adjusted p < 0.001 across 32 comparisons).

AuthorsOpenReview ↗

Response to Reviewer ALLz (3/3)

Q3. Why does NoPE behave differently in SID-based generative recommendation and LLMs?

The effectiveness of NoPE depends on the model architecture and task setting. We provide three possible explanations:

  1. Architecture-dependent positional information. Without positional encoding, a bidirectional self-attention encoder is permutation-equivariant and cannot distinguish different interaction orders, effectively treating the interaction history as an unordered set. In contrast, GPT-2 uses causal attention, which can implicitly capture positional information even without explicit positional encoding [1]. This helps explain why the performance gap between NoPE and SimPos is substantially larger on T5 than on GPT-2, as observed in §4.3.1.
  2. Alternative positional mechanisms in Kimi K3. Kimi K3 [2] uses NoPE in its Gated MLA layers, which are interleaved with Kimi Delta Attention (KDA) layers at a 3:1 ratio. As explained in §2.1.2 and §3.4 of [2], the KDA layers provide position-sensitive and recency-aware sequence modeling through recurrent gating and decay mechanisms. Kimi K3 thus captures positional information through other architectural components, whereas our T5 and GPT-2 backbones lack such recurrent mechanisms.
  3. Structural requirements of SID sequences. Unlike ordinary text sequences, SID-based generative recommendation involves two distinct positional dimensions: interaction order and residual level. Even when positional information can be implicitly inferred, the model may not adequately capture the relationships between codes across and within items. SimPos explicitly models these relationships through parameter-free relative attention biases, providing a structural inductive bias tailored to SID sequences.

These differences help explain why NoPE can be effective in frontier LLMs such as Kimi K3 but underperforms in our SID-based generative recommendation setting.

[1] Haviv, Adi, et al. "Transformer language models without positional encodings still learn positional information." Findings of EMNLP, 2022.

[2] Kimi Team. "Kimi K3: Open Frontier Intelligence." arXiv:2607.24653, 2026.

Thank you again for your thoughtful review and supportive rating. We are happy to answer any follow-up questions. We hope our clarifications and additional results address your remaining concerns and further strengthen your confidence in the paper.

AuthorsOpenReview ↗

Follow-up and Kind Reminder

Dear Reviewer ALLz,

Thank you again for your thoughtful review and for recognizing the motivation of SimPos and its consistent improvements across backbones and tokenizers. Your questions motivated the factorized absolute PE comparison in relation to CoFiRec, the expanded discussion of related work in §2.1, and the new results on Amazon Books.

As the discussion period draws to a close, we wanted to check whether our responses have resolved your remaining questions. If our clarifications and additional results have addressed your concerns, we would be grateful if you could consider reflecting this in your assessment. We sincerely appreciate your time and support.

Reviewer_76mtOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SIMPOS replaces standard positional encoding with two attention biases: one for distance between interactions and one for distance between residual levels. It keeps the semantic IDs unchanged and adds no learnable parameters. The paper compares positional encodings on four Amazon datasets using T5 and GPT-2 with two residual tokenizers, and includes component ablations and a second item embedding model.

Critical strengths
  1. Easy to integrate: two attention biases, no new learned parameters, and no changes to the tokenizer or training objective.
  2. Keeping the IDs and training setup fixed makes the PE comparison useful. The gains hold across both backbones and tokenizers, with further support from the second item embedding model.
Critical weaknesses
  1. I don't think the analysis yet explains why the two-axis bias is better. Section 3.2 shows collisions in relative offsets, but absolute flattened positions still identify both coordinates for fixed-length SIDs. That isn't an information-loss argument against all the baseline PEs. The component ablations help, but SIMPOS gets a coefficient search while comparable tuning of the baseline biases isn't described. It's hard to tell how much comes from separating the axes rather than choosing a better locality scale.
  2. Table 3 claims significance under a paired t-test but doesn't state the number of runs or what is paired. There's also an inconsistency in Figure 4(a,b): according to the caption, both contain (lambda_int, lambda_res) = (0.8, 0.1), yet the plotted Recall@5 differs. This could be a caption or run-aggregation issue, not necessarily an incorrect result, but it makes the component analysis harder to assess without variability estimates.
Questions for authors
  1. How much of the gain remains if a flattened linear distance bias gets a comparable validation-search budget?
  2. What observations are paired in Table 3's t-test, and how many independent training seeds are used?
  3. In Figure 4(a,b), is the fixed coefficient in the caption correct, or were the shared points run seperately?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 76mt (1/3)

We sincerely thank the reviewer for acknowledging SimPos’s simplicity and our controlled evaluation. We address each concern below. (All new results use five training seeds per configuration.)

W1a. Structural inductive bias rather than information loss.

The advantage of SimPos lies in its structural inductive bias, rather than in recovering information lost by flattening. For fixed-length SIDs with KK codes, a flattened absolute position pp determines both coordinates, t=⌈p/K⌉t = \lceil p / K \rceil (interaction position) and k=((p−1) mod K)+1k = ((p-1) \bmod K) + 1 (residual level), so LPE retains full positional information in principle. However, retaining these coordinates does not itself provide an explicit model of their distinct structural relationships. SimPos directly incorporates interaction-level and residual-level distances through separate relative attention biases. In the revised manuscript, we have clarified this distinction in §3.2 and revised the corresponding statements in the Introduction and Conclusion to avoid implying that flattened positional encodings inherently lose structural information.

To further examine the role of this structural inductive bias, we introduce two complementary baselines:

  • Factorized absolute PE. This baseline explicitly provides the same positional coordinates as SimPos through two learnable absolute embeddings, Eitem(t)+Elevel(k)E_{\mathrm{item}}(t) + E_{\mathrm{level}}(k), added to the input rather than through relative attention biases. Across the 32 setting–metric combinations (4 datasets × 2 backbones × 2 tokenizers × 2 metrics), it improves over LPE in only 4 cases (median relative change: −1.16%). SimPos significantly outperforms it in all 32 comparisons, with relative gains of 8.19%–38.40% (median: 16.54%); see Table R1. Thus, explicitly providing the two coordinates through separate absolute embeddings is insufficient to obtain SimPos's full gains.
  • Flattened linear relative bias (FlatBias). This baseline retains SimPos's linear relative-bias form but applies it to flattened token distances, −λ∣p−p′∣-\lambda|p-p'|. Unlike ALiBi, it uses a shared slope across heads. SimPos significantly outperforms this baseline in all 32 setting–metric comparisons. This indicates that applying a shared-slope linear bias on a single flattened axis does not reproduce SimPos's gains. The effect of tuning its strength is further examined under W1b/Q1 (Table R2).

Together, these results support the benefit of explicitly modeling interaction-level and residual-level relationships through separate relative attention biases.

Table R1. Comparison of SimPos with LPE, factorized absolute PE, and ALiBi, averaged across four datasets for each backbone–tokenizer setting.

Backbone / TokenizerLPEFactorized abs. PEALiBiSimPos
T5 / RQ-VAE0.0217 / 0.01360.0214 / 0.01340.0224 / 0.01390.0257 / 0.0164
T5 / RQ-KMeans0.0227 / 0.01420.0224 / 0.01400.0237 / 0.01510.0273 / 0.0180
GPT-2 / RQ-VAE0.0217 / 0.01360.0214 / 0.01340.0205 / 0.01310.0242 / 0.0157
GPT-2 / RQ-KMeans0.0230 / 0.01490.0230 / 0.01480.0221 / 0.01460.0261 / 0.0169

Each cell reports R@5 / N@5, averaged over five seeds and four datasets (Beauty, Toys, Sports, and CDs). SimPos significantly outperforms all three baselines on both metrics across all datasets and settings (one-sided paired t-tests over five seeds, all Holm-adjusted p < 0.005 across 96 comparisons). The full per-dataset results are reported in Appendix C of the revised PDF.

AuthorsOpenReview ↗

Response to Reviewer 76mt (2/3)

W1b / Q1. Effect of bias-strength tuning in single-axis baselines.

To assess whether SimPos’s gains stem solely from choosing a better locality scale, we conduct bias-strength sweeps for two single-axis baselines:

  • Tuned ALiBi. We keep ALiBi’s original head-specific slopes and multiply them by a shared scaling factor α∈0.2,0.4,0.6,0.8,1,2,4\alpha \in \\{0.2,0.4,0.6,0.8,1,2,4\\}, where α=1\alpha = 1 is the default ALiBi in Table 3.
  • Flattened linear relative bias. For the shared-slope baseline introduced under W1a, we sweep λ∈0.2,0.4,0.6,0.8,1.0,1.5,2.0\lambda \in \\{0.2, 0.4, 0.6, 0.8, 1.0, 1.5, 2.0\\}.

Table R2 yields two observations:

(1) Both sweeps peak inside the grid, with limited gains from tuning. The highest aggregate scores occur at α=0.8\alpha = 0.8 and λ=1.0\lambda = 1.0, respectively. The flattened linear relative bias yields similar aggregate scores for λ\lambda between 0.4 and 1.5. For ALiBi, tuning improves the scores averaged across all 16 settings by only 1.35% in R@5 and 1.41% in N@5 over the default configuration.

(2) A performance gap remains even at the best bias strengths. Compared with the strongest aggregate results of the two baselines, SimPos improves R@5 and N@5 by 14.67% and 15.97%, respectively. The detailed results in Appendix C show the same pattern across all backbone–tokenizer settings. Together, these results indicate that SimPos's gains cannot be explained solely by choosing a better single-axis locality scale. In the revised PDF, we have added these baselines in a new §4.3.4 and reported the full sweeps and per-dataset results in Appendix C.

Table R2. Sweeps over the bias strength of tuned ALiBi (α) and the flattened linear relative bias (λ), averaged over five seeds and all 16 settings (4 datasets × 2 backbones × 2 tokenizers).

α0.20.40.60.81 (default)24SimPos
R@50.02200.02200.02200.02250.02220.02070.01990.0258
N@50.01410.01410.01410.01440.01420.01330.01280.0167
λ0.20.40.60.81.01.52.0SimPos
R@50.02180.02220.02220.02220.02250.02210.02100.0258
N@50.01400.01420.01430.01420.01440.01410.01340.0167

SimPos significantly outperforms the best configuration of each tuned baseline in all 32 setting–metric comparisons per baseline (one-sided paired t-tests over five seeds, all Holm-adjusted p<0.01p < 0.01). The sweeps per backbone–tokenizer setting and the per-dataset results are reported in Appendix C of the revised PDF.

AuthorsOpenReview ↗

Response to Reviewer 76mt (3/3)

W2a / Q2. Statistical significance and variability.

Our original significance tests were paired across three training seeds, while Table 3 displayed results from a single seed. To better assess variability across training runs and make the statistical reporting explicit, we rerun the experiments using five independent training seeds per configuration. We recompute significance using one-sided paired t-tests, pairing SimPos and the baseline by training seed.

Under this protocol, SimPos significantly outperforms the strongest baseline in all 32 setting–metric combinations after Holm correction (maximum unadjusted p=2.3×10−3p = 2.3 \times 10^{-3}; maximum adjusted p=4.2×10−3p = 4.2 \times 10^{-3}). Across all entries, the std is at most 0.0008. SimPos's relative improvements over the strongest baseline are 5.00%–24.88% in R@5 and 5.91%–24.64% in N@5.

In the revised PDF, Table 3 replaces the single-seed scores with five-seed means, and its caption reports the standard deviation bound and corrected significance results. Section 4.1 specifies the statistical test protocol. The same five-seed protocol is used for all new experiments reported in the rebuttal.

W2b / Q3. Inconsistency in Figure 4.

We thank the reviewer for catching this. We rechecked the reported values and identified typographical errors in Table 2 and inconsistencies in the captions and plotted values in Figures 3 and 4. We have corrected these errors. In particular, the shared configuration in both figures is (λint,λres)=(0.8,0.2)(\lambda_{\mathrm{int}}, \lambda_{\mathrm{res}}) = (0.8, 0.2). The updated Figures 3 and 4 show mean ±\pm std over five training seeds, and each shared configuration is run once and used in both panels. We apologize for the confusion and appreciate the reviewer's careful reading.

Thanks again for the careful review. We are happy to answer any follow-up questions. If our responses and new results have addressed your primary concerns, we would be grateful if you could consider raising your score.

AuthorsOpenReview ↗

Follow-up and Kind Reminder

Dear Reviewer 76mt,

Thank you again for your careful and constructive review, and for highlighting SimPos's simplicity and the thoroughness of our controlled evaluation. In response to your comments, we added a factorized absolute PE baseline, bias-strength sweeps for tuned ALiBi and the flattened linear relative bias, and a five-seed rerun of the main results with paired significance tests. We have also corrected Figures 3–4 and Table 2 accordingly.

With about one day left in the discussion period, we wanted to kindly ask whether our responses and new results have addressed your main concerns. Please let us know if any point remains unresolved or would benefit from further clarification. We would be very grateful for the opportunity to address it in the remaining time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank all reviewers for their careful reviews and constructive feedback. A shared question is: given SimPos’s simplicity and consistent effectiveness, do its gains arise from its structural design or merely from access to positional coordinates and a stronger recency bias? Our central clarification is that SimPos provides a structural inductive bias rather than recovering information lost through flattening. For fixed-length SIDs, flattened absolute positions retain both interaction and residual coordinates in principle. SimPos explicitly models their distinct relationships through separate relative attention biases.

To address the reviewers’ concerns, we provide additional experiments and clarifications, grouped below by the questions they address. Detailed tables and analyses appear in the individual responses. All new experiments use five training seeds per configuration.

Structural inductive bias or access to positional coordinates?

  • Table R1 (responses to 76mt, ALLz, and bZK4): Factorized absolute PE explicitly provides both coordinates through separate learnable interaction-position and residual-level embeddings. It improves over LPE in only 4 of 32 setting–metric comparisons, whereas SimPos significantly outperforms it in all 32 comparisons, with relative gains of 8.19%–38.40%.

Modeling relative positional relationships is more effective than simply providing positional coordinates.

Separate positional axes or a better-tuned locality bias?

  • Table R2 (responses to 76mt and bZK4): We sweep the bias strengths of ALiBi and a flattened linear relative bias (FlatBias). FlatBias retains SimPos’s linear bias form and shared slope across heads but operates on a single flattened positional axis. SimPos significantly outperforms the best configuration of each baseline in all 32 setting–metric comparisons per baseline.
  • Figures 3–4: SimPos also consistently outperforms its interaction-only variant, showing that the residual-level component provides benefits beyond interaction-level recency alone.

Separating the two positional axes is more effective than tuning a single-axis locality bias.

Does the residual-level bias formulation matter?

  • Table R4 (response to bZK4): We compare the linear residual-level bias with three alternatives: a same-level indicator, a learnable level-pair bias, and a reversed distance-based bias. All variants retain the same interaction-level bias. SimPos significantly outperforms each alternative across all 16 evaluated setting–metric comparisons.

A distance-based decay over residual levels is more effective than distance-agnostic or learned alternatives.

Does SimPos remain effective on a larger dataset?

  • Table R3 (response to ALLz): We add Amazon Books, containing 139,089 users, 113,175 items, and 3.34 million interactions. Across both backbones and both tokenizers, SimPos outperforms the strongest positional encoding baseline by 8.41%–16.32% in R@5 and 6.47%–17.36% in N@5.

The gains persist on a larger public benchmark.

Are the improvements robust across training seeds?

  • Updated Table 3: We report means over five independent training seeds, with standard deviations at most 0.0008. SimPos significantly outperforms the strongest baseline in all 32 setting–metric comparisons under one-sided paired t-tests with Holm correction (all adjusted p < 0.005).

SimPos’s improvements remain statistically significant across five training seeds.

Revisions to the manuscript

We have incorporated the following changes into the revised manuscript (marked in blue):

  1. Clarified the motivation behind SimPos in the Introduction, §3.2, and Conclusion, emphasizing structural inductive bias rather than information loss from flattening.

  2. Expanded §2.1 to discuss CoFiRec, DiscRec, and BARGE and clarify their distinctions from SimPos.

  3. Added §4.3.4 and Appendix C for factorized absolute PE, tuned ALiBi, and FlatBias, including bias-strength sweeps and per-dataset results, and updated the recency discussion in §4.3.2 accordingly.

  4. Added Amazon Books experiments in Appendix A and residual-level bias ablations in Appendix B.

  5. Updated Table 3 with five-seed means and statistical reporting, specified the testing protocol in §4.1, and corrected errors in Table 2 and Figures 3–4. The updated figures report mean ± std over five seeds. Related numbers are updated accordingly (not color-marked).

Thank you again for helping us improve the clarity and quality of this work. We welcome any follow-up questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.