acceptodds
Under review as a conference paper at ICLR 2027

ICR: Scaling In-Context Learning for Sequential Recommendation

Abstract

Sequential recommendation in new domains typically requires extensive per-dataset training data and compute. We present ICR, a zero-shot sequential recommender in the Prior-Fitted Network framework that adapts to new domains at inference time through in-context learning. Given a set of support sequences from the target, ICR produces next-item predictions in a single forward pass without requiring dataset-specific fine-tuning. Our design is motivated by three observations: (1)sequential recommendation dynamics are well-characterized by a hierarchy of aggregation functions, enabling a synthetic prior with provable full support over finite item sequences; (2)dependence on absolute embedding coordinates is not justified for a foundational ICL model—we prove this formally—motivating an ID-based architecture with a pointer mechanism that selects items directly from context; and (3)identity-restricted cross-attention scales sublinearly with context size, enabling performance to improve with more context rather than degrade. On the majority of 11 benchmark datasets, ICR outperforms every zero-shot baseline and, more remarkably, matches or exceeds standard supervised baselines—to the best of our knowledge the first zero-shot recommender to do so, allowing for lower end-to-end cost than supervised training for realistic user populations.

Primary area:
learning on time series and dynamical systems
Keywords:
recommender systems, zero shot, prior-fitted networks, synthetic data, sequential prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_PyShOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a prior-fitted model called ICR for sequential recommendation without target-domain training. The model learns from synthetic sequences and uses random item embeddings, identity-restricted cross-attention, and a pointer head for prediction. The authors also provide some theoretical analysis to support the design. Experiments are conducted on 11 datasets, covering comparisons with zero-shot, supervised, and retrieval-based methods.

Critical strengths
  1. Identity-restricted cross-attention and the pointer head provide a reasonable way to handle larger support contexts without target-domain training.
  2. The paper covers 11 datasets, includes recent 2026 baselines, and provides multiple runs and context-scaling analyses.
Critical weaknesses
  1. I notice that ICR uses 128 support sequences by default, but RecPFN uses only 8. Although context-scaling experiments are provided, this difference may affect the reliability of the main comparison.
  2. There seems to be some inconsistency. Theorem 3 allows repeated items, but Appendix F.3 enables repeat suppression during synthetic sequence generation. This difference deserves further clarification.
  3. The benefit of IRCA seems to be concentrated on Dianping. Removing it causes a large performance drop on this dataset but has little effect on the other five datasets. Its effectiveness across different data distributions may need further investigation.
Questions for authors
  1. Could the authors explain why RecPFN is evaluated with only 8 support sequences, compared to 128 for ICR?
  2. Could the authors clarify how the full-support guarantee in Theorem 3 relates to the actual synthetic data generation process, where repeat suppression is enabled?
  3. Could the authors further explain why IRCA provides a substantial improvement on Dianping but only marginal gains on the other datasets?
3 comments

We would like to thank the reviewer for the thoughtful review and for recognizing the context-scaling design, broad benchmark coverage, and multiple-run evaluation. We will address the reviewer concerns below:

Reviewer concern: Why RecPFN is evaluated with only 8 support sequences, compared to 128 for ICR

We used 8 support sequences for RecPFN because this is its default context size in the original paper, and provides a strong operating point rather than disadvantaging it with an oversized context. Our context-scaling experiments show that RecPFN peaks at small contexts and then degrades as more support is added, consistent with the context sensitivity reported in the RecPFN paper. Thus, increasing its headline budget to 128 would weaken it as a baseline.

Importantly, the context-scaling section also compares performance at matched context sizes, sweeping from 2 to 1024 support sequences per query. ICR outperforms RecPFN and SRPFN across all tested context sizes in the aggregate comparison. As context grows, ICR improves with diminishing returns, RecPFN degrades after its early peak, and SRPFN remains relatively constant.

Reviewer concern: How the full-support guarantee in Theorem 3 relates to the actual synthetic data generation process

Thank you for identifying this discrepancy. Repeat suppression was not enabled in either training stage. The training configurations set repeated_items=True, meaning repeated items are allowed; the exclusion branch runs only when this flag is false. This can be verified by referring to the linked code repository in the paper. The "repeat suppression enabled" entries in Table 8 are therefore a documentation error, which we will correct, together with the conditional presentation of the optional step in Algorithm 1. We apologize for the error and thank the reviewer for identifying it.

Reviewer concern: Why IRCA provides a substantial improvement on Dianping but only marginal gains on the other datasets

Dianping has a substantially weaker global popularity head than the other ablation datasets: its most popular 1% of items account for only 7.2% of training interactions, compared with 16.3-35.1% elsewhere. We believe this suggests that query-specific evidence is particularly important on Dianping, making the benefit of explicitly identity-aligned attention plausible.

I appreciate the authors' efforts to address my concerns. The matched-context scaling analysis is helpful and largely alleviates my concern about the different support sizes used for ICR and RecPFN. The clarification regarding repeat suppression is also reasonable.

However, the revised manuscript (which I have downloaded at 2026-11-16 11:03 am) still lists repeat suppression as enabled for both training stages in Table 8, which remains inconsistent with the authors' response. I encourage the authors to correct this discrepancy. Finally, the additional popularity statistics provide a plausible explanation for the dataset-specific gains of IRCA on Dianping, but the underlying relationship is not yet sufficiently established.

Overall, I acknowledge the improvements and find the responses helpful. Nevertheless, some concerns remain, particularly regarding the consistency between the theoretical guarantee and the actual implementation. I therefore prefer to maintain my original rating.

Thank you for acknowledging the improvements and for your continued feedback. We have just uploaded the revised manuscript; the version you downloaded preceded this update. The incorrect repeat-suppression entries in Table 8 and the unused step in Algorithm 1 have now been removed to reflect the actual training configuration. If any concerns remain after this update, we would appreciate clarification and the opportunity to address them.

Reviewer_k9JkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an in-context learning approach trained on synthetic environments to address the zero-shot performance in sequential recommendation. The approach leverages an identity-restricted attention mechanism with a pointer head to allow it to scale to larger support sets and reports strong performance against other zero-shot, non-parametric, and supervised baselines across 11 datasets.

Critical strengths
  • The paper is well-structured and clearly motivated. Zero-shot performance is a relevant topic for certain industrial scenarios.
  • Good reported zero-shot performance, competitive (even outperforming) supervised baselines on most benchmarks.
  • Provided ablations and experiments are useful to identify context-scaling effects and understand where its contributions come from to some extent.
Critical weaknesses
  • Baseline tuning is skipped despite different dataset selection and split criteria than the ones used for the recommended hyperparameters in the respective papers/repositories. This weakens the paper claims against the supervised baselines.
  • In my opinion, non-parametric comparisons should appear in the main paper (count-based) and stronger PCTM or SeqRules should be added to the comparison set, using both, full training data and identical retrieved support. This would help establish whether the proposed ICR approach is actually performing higher-order modelling or mainly estimating recency-weighted pairwise transitions (concern recently raised by Petrov et al. (2026)).
  • Considering W is explored in {1, 2, 3}, it's unclear the ability of the model to learn longer-range dependencies and the F_{full} synthetic prior, unlike the sequential recommenders it tries to represent, is limited by this n-gram/windowed formulation.
  • The unconventional primary cutoff of 3 should be justified and complemented by more standard HR/MRR at 5, 10, or 20.
  • Certain clarifications or additional analyses are required to fully understand the paper:
    • [Important] The paper states that dataset splits are carried so that they contain disjoint user sets. However, this split does not necessarily protect against temporal leakage, which considering the fact that the model relies solely on support sequences from the target domain, it becomes an evaluation constraint that must be applied. Therefore, if evaluation is done with leave-one-out on hold-out user sequences, these user sequences must be split based on timestamp of the last user interaction. Clarification on this critical implementation detail should be added to the paper.
    • [Important] The choice for random embeddings per model call is justified but further ablation experiments should be provided regarding this part: 1) measure ranking consistency across repeated embedding assignments for query-support pairs and different context sizes to check how robust the decoder modeling is to the same inputs with different random initializations; 2) Test the design choice where the model is trained with semantic-embeddings as inputs. Right now, semantic embeddings are used as output embedding expansion, but checking how item geometry based on semantic information affects the modelling capabilities of ICR is a more useful insight in my opinion.
    • [Medium] It is not clear whether the pointer aggregation introduces some popularity/support-frequency effects due to the scatter-summing operation and maybe a frequency-normalized ablation could provide further clarifications on this issue.
    • [Medium] The paper only explores ICR pretrained on synthetic environments but experiments training on real-domain sequences with held-out domain evaluation would also strengthen the paper.
    • [Low] Table 1 bottom row indicates on how many "cells" ICR is able to outperform its baselines. "cells" and the overall sentence is unclear without actually looking into the Appendix tables. Please, think of a more clear way to present this information.
Questions for authors

Clarify the split that was used to evaluate the model and confirm that no temporal leakage happens within the support sequences. Experiments with semantic embeddings as input to train the decoder model under item space pretrained geometry. Comparison against the SeqRules or any strong non-parametric baseline on the full target training data.

9 comments

We would like to thank the reviewer for the thoughtful review and for recognizing the paper's motivation, strong zero-shot performance, and informative ablations. We will address the reviewer's concerns below:

Reviewer concern: Baseline tuning is skipped despite different dataset selection and split criteria

Exhaustive per-dataset hyperparameter searches for every supervised baseline were computationally prohibitive, so we used recommended generic defaults. Across 11 datasets, the closest supervised competitors, VQ-Rec and BERT4Rec, trail ICR by mean relative HR@3/MRR@3 differences of 22%/23% and 26%/27%, respectively. These margins suggest that tuning may not erase the advantage in most cases, but do not establish superiority over optimally tuned baselines. The same synthetic-pretrained ICR requires neither target-domain parameter training nor dataset-specific hyperparameter search; additional baseline tuning incurs per-dataset adaptation cost.

Reviewer concern: Non-parametric comparisons should appear in the main paper; stronger PCTM or SeqRules should be added

Thank you for suggesting these comparisons. We evaluated SeqRules and PCTM from the official spotify-research/sequential-capacity-probes repository accompanying Petrov et al., Do Sequential Recommendation Benchmarks Really Require Higher-Order Sequence Modelling?, Proceedings of the 20th ACM Conference on Recommender Systems (2026), adapted to our evaluation settings. Both use dataset-level fitting on the full training-user partition.

Values below are HR@3 / MRR@3. ICR uses the paper's four-training-run average, without rerunning it.

DatasetICRSeqRulesPCTM
Appliances0.2589 / 0.21250.2199 / 0.17850.2199 / 0.1702
Arts0.2029 / 0.18120.0993 / 0.08960.0980 / 0.0885
Games0.0737 / 0.05950.0658 / 0.05100.0639 / 0.0493
Grocery0.1078 / 0.09200.0835 / 0.07180.0774 / 0.0661
Office0.1204 / 0.10350.0885 / 0.07920.0776 / 0.0685
Scientific0.0971 / 0.07750.0733 / 0.06630.0712 / 0.0636
Software0.1840 / 0.15570.1544 / 0.11920.1665 / 0.1238
Dianping0.0607 / 0.04900.0290 / 0.02020.0266 / 0.0183
ML-20M0.0805 / 0.05420.0899 / 0.06490.1143 / 0.0818
Washington0.0253 / 0.02000.0263 / 0.01890.0247 / 0.0177
Yelp0.0214 / 0.01550.0199 / 0.01300.0201 / 0.0134

ICR leads these methods on 9/11 datasets for HR@3 and 10/11 for MRR@3. PCTM is strongest on ML-20M; SeqRules slightly exceeds ICR's HR@3 on Washington. These non-parametric methods also outperform many neural supervised baselines, consistent with Petrov et al.'s observations.

Reviewer concern: Ability of the model to learn longer-range dependencies

We chose short synthetic windows to emphasize recent dependencies and avoid making training excessively difficult. Our broader-prior 3× ablation includes windows up to W=7 and achieves mean changes of +0.9% HR@3 and +0.2% MRR@3 across six ablation datasets. Thus, longer-window training preserves strong performance. Because several prior parameters change together, this supports robustness to broader prior specifications, including window size, rather than isolating its effect.

Reviewer concern: The unconventional primary cutoff of 3

HR@3 and MRR@3 emphasize the first few recommendations, as in RecGPT and InstructRec. The k=3, 5, and 10 summaries below show that ICR remains ahead of all baselines on most datasets at larger cutoffs. We will include these summaries in the revised evaluation.

All tables cover 11 datasets. Wins counts datasets where the listed baseline strictly beats ICR. Mean Δ (%) averages 100 × (baseline − ICR) / ICR across datasets; negative values favor ICR. Supervised baselines are trained for 50 epochs. The superscript marker is written as ^ for the LLM-embedding variants, matching the paper's notation.

HR@3 and MRR@3 — zero-shot baselines

BaselineHR wins / 11MRR wins / 11Mean ΔHR (%)Mean ΔMRR (%)
RecPFN01-42-42
SRPFN00-43-52
RecFormer00-62-62
RecGPT01-60-52
UniSRec00-75-76
VQ-Rec01-56-50
EmbKNN00-65-70
EmbKNN-R11-41-41

HR@3 and MRR@3 — supervised baselines

BaselineHR wins / 11MRR wins / 11Mean ΔHR (%)Mean ΔMRR (%)
BERT31-26-27
BERT^00-83-85
FDSA00-67-71
GRU00-70-74
GRU^00-68-68
SAS00-54-59
SAS^00-68-69
UniSRec00-32-36
VQ-Rec21-22-23

At k=3, ICR strictly beats all 8 zero-shot baselines on 20/22 dataset–metric comparisons, and all 9 supervised baselines on 17/22.

HR@5 and MRR@5 — zero-shot baselines

BaselineHR wins / 11MRR wins / 11Mean ΔHR (%)Mean ΔMRR (%)
RecPFN00-41-42
SRPFN00-37-50
RecFormer00-60-61
RecGPT00-64-54
UniSRec00-73-76
VQ-Rec00-60-52
EmbKNN00-61-69
EmbKNN-R01-41-42

HR@5 and MRR@5 — supervised baselines

BaselineHR wins / 11MRR wins / 11Mean ΔHR (%)Mean ΔMRR (%)
BERT31-21-26
BERT^00-80-84
FDSA00-60-69
GRU00-64-72
GRU^00-67-68
SAS00-48-57
SAS^00-68-69
UniSRec00-28-34
VQ-Rec21-17-22

At k=5, ICR strictly beats all 8 zero-shot baselines on 21/22 dataset–metric comparisons, and all 9 supervised baselines on 17/22.

HR@10 and MRR@10 — zero-shot baselines

BaselineHR wins / 11MRR wins / 11Mean ΔHR (%)Mean ΔMRR (%)
RecPFN00-38-41
SRPFN10-27-47
RecFormer00-56-60
RecGPT00-69-56
UniSRec00-72-75
VQ-Rec00-64-54
EmbKNN00-54-68
EmbKNN-R01-38-41

HR@10 and MRR@10 — supervised baselines

BaselineHR wins / 11MRR wins / 11Mean ΔHR (%)Mean ΔMRR (%)
BERT33-13-24
BERT^00-76-83
FDSA00-52-67
GRU10-55-69
GRU^00-67-67
SAS10-39-55
SAS^00-68-69
UniSRec20-21-32
VQ-Rec32-13-21

At k=10, ICR strictly beats all 8 zero-shot baselines on 19/22 dataset–metric comparisons, and all 9 supervised baselines on 14/22.

Reviewer concern: Disjoint user sets do not necessarily protect against temporal leakage

We use strong generalization: training, validation, and test users are disjoint, unlike weak generalization where users overlap. This is stricter for transfer to unseen users, but does not establish temporal validity. This protocol appears commonly in literature including Mult-VAE (Liang et al., WWW 2018), EASE (Steck, SIGIR 2019), RecVAE (Shenbin et al., WSDM 2020) and [RecPFN (Tan et al., SIGIR 2026) (https://dl.acm.org/doi/pdf/10.1145/3805712.3809696)]. This differs from the weaker conventional per-user leave-one-out protocols of SASRec, BERT4Rec, UniSRec, VQ-Rec, RecFormer, and SRPFN where evaluation users are withheld from shared training.

While we concede that the current protocol does not ensure that the train and test split events are strictly temporally isolated, it is computationally prohibitive to rerun all baselines under a different protocol and we maintain that our evaluation protocol is consistent with existing literature and believe that the resulting conclusions drawn are accurate.

Reviewer concern: Ranking consistency across repeated embedding assignments

ICR's reported results average four independent training runs with different trained weights and randomly initialized item-embedding stores. We show the variability across runs below.

DatasetHR@3HR relative SDMRR@3MRR relative SD
Appliances0.2589 ± 0.00411.58%0.2125 ± 0.00502.38%
Arts0.2029 ± 0.00060.29%0.1812 ± 0.00221.22%
Games0.0737 ± 0.00121.57%0.0595 ± 0.00060.95%
Grocery0.1078 ± 0.00050.42%0.0920 ± 0.00020.23%
Office0.1204 ± 0.00110.95%0.1035 ± 0.00181.72%
Scientific0.0971 ± 0.00030.31%0.0775 ± 0.00091.20%
Software0.1840 ± 0.00703.80%0.1557 ± 0.00221.39%
Dianping0.0607 ± 0.00071.24%0.0490 ± 0.00071.38%
ML-20M0.0805 ± 0.00040.50%0.0542 ± 0.00050.95%
Washington0.0253 ± 0.00010.59%0.0200 ± 0.00020.89%
Yelp0.0214 ± 0.00083.85%0.0155 ± 0.00052.93%

Across most cells where ICR leads all baselines, ICR's improvements are statistically significant. Together with the small variability, this supports reproducibility across models and embedding realizations. These statistics vary both trained model weights and the random embedding realization, which we believe provides a broader reproducibility check than varying input embedding assignments alone.

Reviewer concern: Test the design choice where the model is trained with semantic-embeddings as inputs

We trained a fresh two-stage ICR with BGE-M3 embeddings sampled from BEIR as backbone inputs. Evaluation uses BGE-M3 item-metadata embeddings for queries and supports. A shared, fixed orthonormal 1024-to-256 projection followed by L2 normalization preserves input dimension and learned parameter count.

Input representationMean HR@3ChangeMean MRR@3Change
Random embeddings0.11458Reference0.09503Reference
BGE-M3, fixed projection0.10838-5.4%0.08302-12.6%

Across 11 evaluation datasets, semantic inputs reduce mean HR@3/MRR@3 by 5.4%/12.6%. While we emphasize that this is in no way direct empirical proof of Theorem 4, it is aligned with the theorem's predictions.

Reviewer concern: Popularity/support-frequency effects due to the scatter-summing operation

While scatter-summing does give repeatedly occurring support items more opportunities to accumulate attention, we regard this as an intentional aggregation of corroborating evidence and would like to emphasize that frequency within query-retrieved support is not the same as global catalogue popularity, although the two can be correlated. The attention weights also depend on the query and contextual representations, so the pointer is not simply ranking items by raw frequency.

To isolate the contribution of multiplicity, we compared the original summed pointer scores with a frequency-normalized variant that divides each item's score by its number of valid occurrences in that query's support, then renormalizes over candidates. Padding is excluded; within-sequence repeats are counted. Both conditions use identical original IRCA-trained weights, retrieved support, candidate IDs and target coverage.

Pointer aggregationMean HR@3ChangeMean MRR@3Change
Sum over occurrences0.11458Reference0.09503Reference
Mean per occurrence0.02633-77.0%0.01802-81.0%

Across all 11 datasets, frequency normalization lowers mean HR@3 by 77.0% and MRR@3 by 81.0%; HR@3 decreases on every dataset. The result shows that multiplicity is important to the existing trained predictor, consistent with aggregating support evidence rather than discarding repeated observations.

Reviewer concern: Training on real-domain sequences with held-out domain evaluation

We train ICR from scratch on real interaction sequences from Amazon Fashion, Grocery, Music, Sports and Steam, with equal batch budgets for each dataset and cycling through the datasets. Supports are sampled from that domain's training users, excluding the query users; checkpoint selection uses disjoint validation users, and test users are not used for training or validation. We evaluate on the ten headline domains not used for pretraining, excluding Grocery.

Pretraining sequencesMean HR@3ChangeMean MRR@3Change
Synthetic0.11506Reference0.09509Reference
Real, five domains0.10013-13.0%0.07566-20.4%

As seen from the results, there is significant degradation - this supports our claim that recommendations is highly context dependent and that pre-training on a real world datasets leads to the model learning and memorizing specific behaviour that does not generalize well to other datasets.

Reviewer_YX6zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ICR, a zero-shot sequential recommender in the Prior-Fitted Network framework. ICR is trained only on sequences generated by a synthetic structural causal model and adapts to a target domain at inference time by conditioning on retrieved support sequences, without dataset-specific gradient updates. Its main components are an ID-based representation, identity-restricted cross-attention (IRCA) for scaling to longer contexts, and a pointer head that predicts items appearing in the retrieved context. The paper also develops a hierarchy of projected-kernel models and proves full support for the general synthetic prior. Experiments on 11 datasets report strong HR@3 and MRR@3 against zero-shot and supervised baselines, together with ablations, context-scaling results, efficiency measurements, and context-coverage analysis.

Critical strengths
  1. Strong zero-shot results across diverse domains. ICR is evaluated on 11 datasets with catalogs ranging from roughly 22K to 340K items. Without target-domain training, it achieves the best zero-shot HR@3 and MRR@3 on 10 of 11 datasets and is competitive with or better than standard supervised recommenders. This is a practically meaningful result for domains where repeated per-dataset training is costly.

  2. A scalable, task-driven PFN design. IRCA exploits repeated item identities to increase the usable support context while avoiding the attention dilution and linear context dependence observed in RecPFN. The paper supports this design with context-scaling, latency, ablation, and coverage analyses, making the empirical study broader than a main-results comparison alone.

Critical weaknesses

The final submission should retain only the one or two weaknesses most consequential to the recommendation.

  1. The practical benefit of full support is not empirically established. Compared with the concurrent methods RecPFN and SRPFN, the paper does not demonstrate that ICR's advantage comes from full support. Figure 2 attributes the clearest gains to IRCA and context scaling, while Figure 3 mainly validates the IRCA and pointer-head mechanisms; neither establishes a performance benefit from full support. Thus, the full-support results remain theoretical, weakening the practical significance of the first contribution. Additional empirical analysis would help dispel the concern that the theory merely serves as an unrelated mathematical wrapper around the method.

  2. The full-support theorem does not apply to the actual training prior. Repeat suppression is enabled in both training stages (Table 8), and Algorithm 1 assigns zero probability to previously seen items. Thus, sequences containing repeated items have zero probability under the actual training generator. Appendix G.3 explicitly excludes repeat suppression from Theorem 3, meaning that the full-support guarantee applies only to an abstract SCM rather than the prior used to train ICR. This further weakens the practical relevance of the first contribution.

  3. IRCA lacks an identity-agnostic sparse-attention control. IRCA is a task-driven architectural optimization that exploits repeated item identities across query and support sequences to mitigate attention dilution. However, sparse attention, masking, and gather-attend-scatter are established techniques. The ablation compares IRCA only with dense cross-attention, without an identity-agnostic sparse-attention baseline matched for sparsity and computational cost. Such a control is needed to isolate whether the gains come from ID-based routing or from sparsification alone. If identity-agnostic sparse masking performs similarly, the claimed novelty and significance of the second contribution would be substantially weakened.

  4. Candidate coverage is not controlled in the comparisons. Although the evaluation is described as full-catalog ranking, ICR's pointer head can predict only items present in the retrieved context; Figure 4 reports a context coverage of only 25--67% at the default setting. In contrast, RecPFN retrieves over the full catalog, while SRPFN and the supervised baselines explicitly score all catalog items. This asymmetry confounds the architectural comparison: restricting the output space imposes a recall ceiling, but also removes most distractors and avoids full-catalog scoring. A matched-candidate or matched-coverage evaluation is needed to determine whether ICR's accuracy and efficiency gains come from the model or from candidate restriction.

Questions for authors
  1. Table 8 enables repeat suppression, but Appendix G.3 excludes it from Theorem 3. Does the full-support guarantee apply to the actual training prior?

  2. Can the authors isolate the effect of full support? The linear-versus-extended SCM ablation changes more than this property.

  3. How does IRCA compare with a compute-matched sparse-attention baseline without ID-based masking?

  4. Can the authors compare the methods under the same candidate coverage? ICR predicts only context items, while the other methods search the full catalog.

3 comments

First and foremost, we would like to thank the reviewer for the thoughtful review and for highlighting our strong zero-shot results across diverse domains and scalable, task-driven PFN design. We will answer the reviewer's concerns below:

Reviewer concern: The practical benefit of full support is not empirically established

Our first contribution is a model-informed synthetic prior and its formal characterization, not full support as an independent source of empirical gains. The projected-kernel formulation connects the generator to sequential recommenders: lag-dependent aggregation motivates the linear component, content-dependent weighting motivates routing, and content-dependent contributions motivate pattern modulation. These connections guide structured subfamilies and parameter ranges emphasizing plausible recommendation dynamics.

Theorem 3 establishes a complementary property: every finite item sequence has positive probability under the general SCM prior. This is the "insurance" interpretation in Section 3 and Appendix J.3: task-informed structure without categorically excluding finite sequences. Full support establishes coverage, not an isolated empirical performance benefit.

Reviewer concern: The full-support theorem does not apply to the actual training prior

Thank you for identifying this discrepancy. Repeat suppression was not enabled in either training stage: both set repeated_items=True. The optional exclusion branch runs only when this flag is false and attempts to remove all previously visited items, not just consecutive repeats. We will correct the erroneous "repeat suppression enabled" entries in Table 8 and make the optional step conditional in Algorithm 1. We apologize for the error and thank the reviewer for identifying it.

Reviewer concern: IRCA lacks an identity-agnostic sparse-attention control

Original IRCA's average attention occupation (percentage of eligible attention edges retained) is:

DatasetOriginal IRCA average attention occupation
Grocery6.57%
Games4.66%
Office7.60%
Software4.66%
Dianping2.36%
Washington3.02%
Six-dataset mean4.81%

We test random sparse and deterministic recency-based attention on the same IRCA-trained checkpoint. Uncapped matched-random routing retains exactly IRCA's per-query key count, D. We also compare both under the same 1% cap, min(D, max(1, floor(0.01L))), where L is the valid context-token count. IRCA uses fewer keys when fewer identity matches exist. These controls match attention-edge budgets, not necessarily wall-clock cost.

We additionally test random and recency routing retaining 15% or 40% of valid context tokens, without limiting their budgets by D. Recency selects the most recent valid events across support sequences. These higher-occupation controls are compared with unchanged, uncapped IRCA.

Backbone routingMean HR@3Mean MRR@3
IRCA, uncapped0.097580.08260
Random, matched uncapped IRCA budget0.066700.05397
IRCA, 1% density cap0.097090.08250
Random, matched 1%-capped IRCA budget0.056670.04171
Random, 15% retained0.074410.06004
Recency, 15% retained0.069790.05558
Random, 40% retained0.079420.06288
Recency, 40% retained0.078730.06277

These are six-dataset means. The uncapped matched-random row averages three routing seeds; other rows use one. Uncapped IRCA performs best on aggregate, and the 1%-capped version loses little accuracy. Increasing random/recency occupation improves their aggregate performance but does not recover IRCA's results. Thus, identity-aligned key selection matters beyond sparsity alone for this trained checkpoint. These inference interventions do not establish that separately trained alternatives cannot perform comparably.

Reviewer concern: Candidate coverage is not controlled in the comparisons

We evaluate RecPFN with 8 supports under full-catalog ranking and ranking restricted to ICR's 128-support pointer candidates. Both rankings reuse the same forward-pass scores; only output candidates change. Restricted RecPFN and ICR therefore have identical candidates and target coverage. All test users count, including misses when targets are absent from the candidates.

Evaluation conditionMean HR@3HR@3 Δ vs ICRMean MRR@3MRR@3 Δ vs ICR
RecPFN, full catalogue0.06583−42.5%0.05201−45.3%
RecPFN, ICR-matched candidates0.08179−28.6%0.06122−35.6%
ICR, same matched candidates0.114540.0%0.095000.0%

These are unweighted means across all 11 benchmarks. Candidate restriction improves RecPFN but bridges less than a third of either accuracy gap, so it contributes to, but does not explain, ICR's advantage.

Thank you again for your thoughtful review. As the discussion period concludes tomorrow, we would appreciate knowing whether any concerns remain following our rebuttal, particularly regarding the sparse-routing controls, matched-candidate comparison, and clarification of the theoretical contribution. The revised manuscript also corrects the repeat-suppression documentation. We would appreciate the opportunity to address any remaining concerns. If our responses resolve your principal concerns, we would be grateful if you would consider updating your assessment.

Area_Chair_7AXWOpenReview ↗

Please Respond to the Authors’ Rebuttal

Dear Reviewer YX6z,

The discussion period ends in a few hours. Could you kindly check the authors’ responses and let us know whether your concerns have been addressed?

Thank you!

Best regards,

The AC

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Manuscript Revisions in Response to Reviewer Feedback

We thank all reviewers for their constructive feedback. We have revised the manuscript as follows:

  • Synthetic-data-generation correction (YX6z and PySh): Removed the unused repeat-suppression setting and algorithm step. Repeated items were allowed in both training stages; this corrects the documentation, not the training procedure.
  • Non-parametric baselines (k9Jk): Added SeqRules and PCTM with citations and implementation details, and included non-parametric comparisons in Table 1 and Appendix A.2.
  • Larger evaluation cutoffs (k9Jk): Added HR@5/MRR@5 and HR@10/MRR@10 summaries for all three baseline classes in Appendix A.3.

Additions and revised text are highlighted in red for ease of review.

AuthorsOpenReview ↗

Summary of Responses to Reviewer Concerns

We thank all reviewers for their thoughtful feedback, which has helped us strengthen the paper and clarify its contributions. As the discussion period concludes tomorrow, we summarize how our responses address the concerns raised:

  • Theoretical guarantee and training consistency (YX6z, PySh): We verified that repeat suppression was disabled in both training stages: repeated items were allowed in the actual training generator. We corrected the erroneous description in Table 8 and Algorithm 1 in the revised manuscript. We also clarified the role of full support as a finite-sequence coverage guarantee, rather than claiming that this property independently explains the empirical gains.

  • Identity-based routing versus sparsification (YX6z): We provided identity-agnostic random sparse-attention controls matched to IRCA’s per-query attention budget, alongside broader random and recency-based routing controls. IRCA performs better in these comparisons, supporting the importance of identity-aligned routing beyond sparsification alone. These are inference-time controls using the same trained checkpoint.

  • Candidate restriction and support-size fairness (YX6z, PySh): We evaluated RecPFN over the exact candidate set available to ICR, retaining uncovered targets as misses. Candidate restriction improves RecPFN but does not close the performance gap. We also explained RecPFN’s original eight-support configuration and provided matched-context scaling comparisons, showing that ICR’s aggregate advantage persists when support sizes are matched.

  • Stronger baselines and conventional cutoffs (k9Jk): We added SeqRules and PCTM fitted on the target-domain training partition, incorporated non-parametric comparisons into the main results, and added HR/MRR summaries at k=5 and k=10 for all three baseline classes. We also addressed baseline-tuning limitations and discussed the broader-prior ablation relevant to the concern about short dependency windows.

  • Evaluation protocol and embedding robustness (k9Jk): We clarified the user-disjoint evaluation protocol and training-only support retrieval, while distinguishing held-out-user generalization from global temporal separation. We reported per-dataset variability across four independently trained models and embedding realizations, and tested semantic embeddings as backbone inputs through fresh two-stage training with a fixed projection and unchanged learned parameter count.

  • Support frequency, pretraining, and dataset-specific gains (k9Jk, PySh): We compared summed pointer scores with frequency-normalized scores under identical weights, supports, and candidates; evaluated real-domain pretraining on held-out domains; and supplied popularity statistics offering a plausible, though not conclusive, explanation for IRCA’s larger gains on Dianping.

The revised manuscript highlights additions and revised text in red. We hope these clarifications, controls, and revisions address the reviewers’ concerns. We would appreciate knowing whether any issues remain and would welcome the opportunity to address them before discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.