acceptodds
Under review as a conference paper at ICLR 2027

Markovian Pre-Trained Transformer for Next-Item Recommendation

Abstract

We introduce the Markovian Pre-trained Transformer (MPT) for next-item recommendation, a transferable model fully pre-trained on synthetic Markov chains, yet capable of achieving state-of-the-art performance by fine-tuning a lightweight adaptor. This counterintuitive success can be attributed to the observed `Markovian' characteristics: advanced sequential recommenders tend to rely predominantly on the latest interaction for prediction, while historical interactions primarily provide contextual signals for inferring the user's general, non-sequential identity. These characteristics necessitate the capabilities of a universal recommendation model to (1) effectively summarize the user sequence, with (2) particular emphasis on the latest interaction. MPT acquires these inherent capabilities from carefully constructed synthetic data. On the one hand, when trained to predict the next state of an arbitrary Markov chain, it must learn to (1) estimate transition probabilities from contextual information (one adaptive manner for sequence summarization), and (3) attend to the most recent state to ensure an accurate restart point. On the other hand, unlike the heterogeneous interaction data, an unlimited number of controllable Markov chains is available to boost the model capacity. Extensive experiments on five public datasets from three distinct platforms validate the superiority of Markovian pre-training over traditional recommendation pre-training (+39% on average) and recent language pre-training (+24% on average) paradigms.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
sequential recommendation, synthetic data pre-training, transfer learning, Markov chains

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jjrkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work empirically identifies a “Markovian characteristic” common in modern sequential recommenders: predictions are dominated by the latest interaction, while earlier items mostly serve as non-sequential context for user identity. Based on this insight, it proposes MPT, pre-trained entirely on synthetic Markov chains, which achieves competitive next-item recommendation by fine-tuning only a lightweight input adaptor. It also analyzes scaling behavior and derives a theoretical Bayes-optimal performance limit.

Critical strengths

(1) Unlike prior works that manually built Markovian structures into architectures, this paper empirically shows that even expressive sequential models spontaneously operate in a Markovian mode. It then proposes pre-training on pure synthetic Markov chains, wichi is a counterintuitive idea opening a interesting direction for universal recommendation pre-training. (2) It starts with systematic shuffle experiments to demonstrate the Markovian property, then designs the pre-training task to match those required capabilities, then validates extensively with benchmarks, scaling analysis, attention maps, and theoretical limits. The logic builds cleanly from insight to method to evidence. (3) Matches standard SASRec inference speed while outperforming both dedicated pre-training and LLM-based methods.

Critical weaknesses

(1) The Markovian property breaks down on platform-driven datasets (e.g. MovieLens), and all main tests are on e-commerce. The method’s real scope is not clearly bounded. (2) The paper shows empirically that synthetic-to-real transfer works, but provides little theoretical explanation for why ideal Markov chains transfer to real user behavior. (3) The Transformer backbone is completely standard, and all contribution comes from the data paradigm, leading to limited technical depth. (4) Unbalanced LLM baseline comparison. Qwen2.5-7B is orders of magnitude larger than MPT; zero-shot and cross-domain LLM strengths are not evaluated. (5) There’s no control test with a randomly initialized backbone plus the same adaptor fine-tuning. You can’t be sure the gains come from Markov pre-training vs. just the adaptor learning the task from scratch.

Questions for authors

(1) The core “Markovian characteristic” finding is tested almost entirely on short-session e-commerce datasets. How do we know this is a property of the models themselves, rather than just a property of these datasets (which are naturally close to first-order Markov dynamics)? Have you tested this pattern on longer-sequence, strongly sequential domains like content recommendation? (2) There’s no control experiment with a randomly initialized backbone plus the same lightweight adaptor fine-tuning. How can we be sure the performance gains come from capabilities transferred via Markovian pre-training, rather than the adaptor simply re-learning the task from scratch on downstream data? (3) Have you compared MPT to similarly sized small language models, and tested it in zero-shot / cross-domain settings?

12 comments
AuthorsOpenReview ↗

[1/4] Scope of the "Markovian Characteristic" Finding [W1, Q1]

Responses

  1. Is the "Markovian characteristic" primarily observed in e-commerce scenarios? NO. We evaluate two additional widely used datasets from non-e-commerce domains, Steam and LastFM. The SASRec NDCG@10 results below show that the "Markovian characteristic" can also be observed in these settings:

    SASRecSteamLastFM
    ① Chronologically ordered0.07350.0685
    ② Partially shuffled0.07310.0701
    ③ Completely shuffled0.07260.0180
  2. How do we know this is a property of the models themselves, rather than just a property of these datasets? As discussed in the manuscript, establishing that the "Markovian characteristic" holds across all scenarios is neither technically feasible nor necessary for our main argument. Our central point is that the ability of advanced sequential recommenders to make predictions in a "Markovian" fashion may offer useful insights for developing universal recommenders.

Misunderstandings

  1. Should MovieLens datasets be used as sequential recommendation benchmarks? NO. We include MovieLens because it is widely used in traditional recommendation research. However, [1] and [2] have emphasized that MovieLens should not be used as a sequential recommendation benchmark because its recorded movie sequences reflect a rating elicitation process governed by a fixed algorithm rather than users' actual viewing histories.

[1] Harper F. M., et al. The movielens datasets: History and context. TIIS, 2015.

[2] Woolridge D., et al. Sequence or Pseudo-Sequence? An Analysis of Sequential Recommendation Datasets. RecSys, 2021.

AuthorsOpenReview ↗

[2/4] LLM baseline comparison [W4, Q3]

Responses

  1. Why was Qwen2.5-7B selected? We selected Qwen2.5-7B because it is widely used, and models with 7B parameters are a common choice in prior work on LLM-based recommendation.

  2. NDCG@10 Results for Qwen3-0.6B. We acknowledge the reviewer's concern regarding a fair comparison and provide additional results for the smaller Qwen3-0.6B model.

    BeautyToysSportsYelpRetail
    Qwen2.5-7B0.04640.04330.02560.02040.0731
    Qwen3-0.6B0.05650.05790.03160.02190.0735
    MPT0.06730.07050.03730.02240.0741

    Qwen3-0.6B achieves better results than Qwen2.5-7B but still underperforms MPT across all five datasets. These results further motivate the question raised by the attention maps in Figure 4c: does language pre-training provide mechanisms that adapt effectively to recommendation tasks?

  3. Zero-shot / Cross-domain settings. MPT requires recommendation-specific fine-tuning because its pre-training and downstream task spaces differ. Figure 6 demonstrates its ability to adapt with limited downstream data. Moreover, all results in Table 1 use the same pre-trained backbone, with a separate adaptor fitted to each downstream domain. This demonstrates its reuse across multiple recommendation domains.

AuthorsOpenReview ↗

[3/4] Control Experiment with a Randomly Initialized Backbone [W5, Q2]

Misunderstandings

  1. We would like to clarify that Table 9 already reports a control experiment using a randomly initialized backbone with the same lightweight adaptor fine-tuning procedure. The results are reproduced below:

    BeautyToysSportsYelpRetail
    Randomly initialized0.06230.06640.03360.02210.0679
    Markovian pre-trained0.06730.07050.03730.02240.0741

Changes

  1. We have replaced "Training from scratch" with "Randomly initialized" to avoid confusion.
Reviewer_dE6VOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes the Markovian Pre-trained Transformer (MPT), a compact Transformer whose backbone is pre-trained on synthetic first-order Markov trajectories and then transferred to next-item recommendation through a lightweight input adaptor. The motivation is an empirical perturbation study showing that, for several sequential recommenders and datasets, shuffling interactions before the latest item changes performance little, whereas shuffling the entire sequence is harmful. The authors interpret this as evidence for two transferable capabilities: aggregating historical context and emphasizing the latest interaction. MPT is trained to predict the next state of Dirichlet-sampled Markov chains using independently resampled orthogonal state representations, which are intended to discourage token memorization. Downstream experiments on five datasets compare MPT with scratch-trained, recommendation-pretrained, and language-pretrained baselines. MPT is competitive or best on most reported metrics while keeping the backbone frozen; the paper also reports scaling studies, ablations, a same-architecture scratch control, efficiency measurements, and a Dirichlet--multinomial Bayes estimator for the synthetic pre-training task.

Critical strengths
  1. The capability-centric synthetic pre-training idea is original and potentially important. Instead of transferring a fixed recommendation corpus or item vocabulary, MPT uses procedurally generated trajectories and resamples orthogonal state representations for every trajectory. This is a thoughtful attempt to induce a reusable sequence-processing computation rather than memorize identities. The same-architecture comparison in Table 9 further shows that the Markov-pretrained initialization improves over training MPT from scratch, so the contribution is not merely a new downstream architecture.

  2. The empirical and practical package is unusually broad. The paper combines ordering interventions, five downstream datasets, frozen-backbone adaptation, scaling and sensitivity studies, data-efficiency analysis, generative and discriminative baselines, and wall-clock measurements. Adapter-only MPT is strong on most reported metrics while retaining inference cost close to compact sequential recommenders. Even if the proposed mechanism is ultimately narrowed, this result makes synthetic procedural pre-training a worthwhile direction for recommendation research.

Critical weaknesses
  1. The central mechanism is not identified by the current experiments. Robustness to partially shuffling earlier items shows approximate invariance to their order, yet this does not uniquely demonstrate general user identity summarization. Short histories, co-occurrence pooling, exposure effects, weak temporal signals, or architectural recency bias are all consistent with this observation. Similarly, Table 9 confirms the benefit of pre-training, but the work lacks synthetic control experiments matched on tokens, compute, architecture and representations, including IID or exchangeable sequences, shuffled transition pairs, higher-order chains, and hidden Markov models. As such, the evidence more strongly supports transfer from large-scale procedural sequence pre-training, rather than the paper’s core claim that Markov-chain transition inference underpins the two proposed capabilities.

  2. The headline superiority and universality claims are not yet supported by a sufficiently controlled or statistically complete comparison. MPT's main configuration uses sentence-T5-xl semantic features, whereas important scratch baselines use learned item IDs; objectives, trainable-parameter budgets, tuning protocols, pre-training resources, and even the number of runs also differ. Table 1 reports point estimates without dispersion or paired uncertainty despite three-run versus five-run protocols, which is consequential for the small Online Retail differences. Moreover, three of the five downstream datasets are Amazon categories, and MovieLens is an explicit counterexample to the motivating shuffle pattern. Matched-input comparisons, seed-level uncertainty, and either broader order-sensitive evaluation or narrower ``universal'' claims are needed before the reported gains can be attributed to the Markov-pretrained backbone and generalized beyond the tested regime.

Questions for authors
  1. Can the authors provide token-, compute-, architecture-, and representation-matched pre-training controls using IID/exchangeable sequences, shuffled Markov trajectories, higher-order chains, and hidden Markov models? On held-out synthetic tasks, do MPT predictions track the prefix-conditioned Dirichlet posterior predictive distribution and respond correctly to interventions on transition counts while the last state is fixed?

  2. Can the authors report a representation- and objective-matched comparison in which the strongest sequential baselines receive the same sentence-T5-xl inputs, full-corpus loss, trainable-parameter budget, and tuning budget as MPT? Please also provide all seed-level results and paired uncertainty intervals or tests for the principal Table 1 contrasts.

  3. Does the Bayes-estimator line in Figure 4 use only transitions observed in the causal prefix at each prediction position? Appendix A defines counts over the entire trajectory, which would expose future transitions if used directly. Please derive the position-dependent posterior predictive loss and state explicitly which quantity is bounded; this bound should be separated from downstream NDCG.

  4. What measurable property predicts when the proposed pre-training is appropriate? In particular, can the MovieLens counterexample be converted into a prospective applicability test, or can the authors add a downstream dataset with reliable, strongly order-sensitive sessions? The answer would determine whether I view the current ``universal'' framing as supported or whether it should be substantially narrowed.

6 comments
AuthorsOpenReview ↗

[1/4] Controlled Comparisons and Statistical Reporting [W2, Q2]

Responses

  1. Why do baselines trained from scratch use learned item ID embeddings? We follow the configurations recommended in their original papers.

  2. Is the evaluation in Table 1 fair? For each baseline, we tune the learning rate over {1e-4, 5e-4, 1e-3, 5e-3}, weight decay within [0, 0.1], and dropout rate within [0, 0.5] on Beauty. We fix the embedding dimension at 256, as this is a key factor affecting recommendation capacity. All results are obtained from the checkpoint with the highest validation NDCG@10. These choices provide a consistent tuning and checkpoint-selection protocol, although trainable parameter counts and the number of runs differ across methods.

  3. Representation- and Objective-Matched Comparison. We report below the results for SASRec+ (namely SASRec with a full-corpus loss) using the same sentence-T5-xl input representations:

    DepthBeautyToysSportsYelpRetail
    SASRec+ (1 layer)0.06190.06120.03100.01840.0639
    SASRec+ (2 layers)0.06070.06010.03120.01940.0678
    SASRec+ (4 layers)0.05900.05760.03080.02010.0711
    SASRec+ (8 layers)0.05880.05750.03020.01960.0735
    MPT (4 layers)0.06730.07050.03730.02240.0741
  4. Seed-level results and statistical comparisons. We report standard errors in most figures and tables, omitting them from Table 1 due to space limitations. Below, we provide the seed-level NDCG@10 results and the corresponding paired t-test pp-values. "Failed" denotes a training failure.

    SeedBeautyToysSportsYelpRetail
    SASRec+00.05790.06330.03350.02000.0725
    SASRec+10.05940.06540.03310.02120.0714
    SASRec+20.06030.06450.03410.02140.0731
    SASRec+30.05890.06430.03360.02110.0712
    SASRec+40.05900.06490.03410.02110.0715
    E4SRec00.04670.04980.0217Failed0.0743
    E4SRec10.04720.04750.02160.01400.0738
    E4SRec20.04690.05030.02210.01380.0767
    MPT00.06740.06980.03720.02250.0746
    MPT10.06720.07070.03750.02260.0737
    MPT20.06610.06990.03650.02240.0750
    MPT30.06860.07160.03760.02250.0731
    MPT40.06710.07040.03770.02200.0739
    pp-valueBeautyToysSportsYelpRetail
    MPT vs. SASRec+2.91E-049.54E-054.70E-046.15E-034.18E-05
    MPT vs. E4SRec4.76E-042.99E-038.97E-041.50E-034.92E-01

Changes

  1. Appendix C.6: Include comparisons with SASRec+ in Appendix C.6.
  2. Appendix C.9: Provide additional statistical tests in Appendix C.9.
AuthorsOpenReview ↗

[2/4] Bayes-Estimator Line in Figure 4 [Q3, Q1]

Responses

  1. Does the Bayes-estimator line in Figure 4 use only transitions observed in the causal prefix at each prediction position? YES. Both the Bayes estimator and LNSP\mathcal{L}_{\text{NSP}} are evaluated using only transitions observed before the prediction target. The derivation in Appendix A provides a standalone introduction to estimation from an arbitrary observed sequence.

  2. Position-dependent posterior predictive loss. Given a state sequence [s1,…,sk=i,…,sT][s_1,\ldots,s_k=i,\ldots,s_T], the transition probabilities for predicting the next state from sk=is_k=i are estimated as follows:

    \text{Bayes estimator:}\quad \hat{p}_{ij}^{\text{bayes},(k)} &= \frac{c_{ij}^{(k)}+\alpha_j} {\sum_{j'\in\mathcal{S}}\left(c_{ij'}^{(k)}+\alpha_{j'}\right)}, \quad c_{ij}^{(k)}=\sum_{t=1}^{k-1}\mathbb{I}[s_t=i,s_{t+1}=j],\\\\ \text{MPT:}\quad \hat{p}_{ij}^{\text{mpt},(k)} &= \frac{\exp(\mathbf{h}_k^\top\mathbf{e}_j)} {\sum_{j'\in\mathcal{S}}\exp(\mathbf{h}_k^\top\mathbf{e}_{j'})}, \quad \mathbf{h}_k=\text{MPT}([s_1,\ldots,s_k];\Theta). \end{aligned}$$ Here, $\\{\mathbf{e}_j\\}_{j\in\mathcal{S}}$ denotes the set of orthogonal embeddings. For each estimator, the pointwise loss in Figure 4a is calculated as $-\log\hat{p}_{i,s_{k+1}}^{(k)}$.
  3. Connection between the bound and downstream performance. We would like to clarify that we do not interpret the gap between the two estimators as an indicator of downstream performance. Figure 4a illustrates an interesting scaling phenomenon that may inform future synthetic data design, such as using higher-order chains to increase task complexity.

  4. Do MPT predictions track the prefix-conditioned Dirichlet posterior predictive distribution? We report the pointwise predictive losses at different prefix lengths below:

    Prefix Length−log⁡p^i,targetmpt-\log \hat{p}_{i, \text{target}}^{\text{mpt}}−log⁡p^i,targetbayes-\log \hat{p}_{i, \text{target}}^{\text{bayes}}
    13.453.40
    103.063.04
    502.342.31
    1002.041.98
    5001.531.42
    10241.451.32

    MPT predictions closely track the prefix-conditioned Dirichlet posterior predictive distributions, although the gap widens at longer prefixes.

Changes

  1. Line 794-801: Provide a detailed derivation, as above, to clarify how the curves in Figure 4a are calculated.
  2. Line 414-416: Revise the discussion of Figure 4a to clarify its intended interpretation.
AuthorsOpenReview ↗

[3/4] Pre-training Controls [W1, Q1]

Responses

  1. Can the authors provide token-, compute-, architecture-, and representation-matched pre-training controls? We report the results of additional pre-training controls below. Implementation details are provided in Appendix C.7 of the revised manuscript:

    BeautyToysSports
    Shuffled Markov0.06420.06610.0352
    Hidden Markov models0.06480.06810.0361
    Second-order Markov0.06640.07170.0368
    MPT (First-order Markov)0.06730.07050.0373

    These results indicate that Markovian pre-training is not the only way to achieve transferability. However, first- and second-order Markov pre-training yield the strongest results among the evaluated settings, suggesting that a well-defined next-state prediction structure should be prioritized when designing synthetic pre-training data.

Changes

  1. Appendix C.7: Include the pre-training controls in the appendix and clarify that Markovian pre-training is not the only approach that enables transfer.
Reviewer_MgyYOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that state-of-the-art sequential recommenders behave in a largely “Markovian” way: when all historical interactions except the most recent one are shuffled, performance is nearly unchanged, whereas shuffling the entire sequence causes a clear drop. The authors interpret this as evidence for two transferable capabilities: summarizing non-sequential user context and attending to the latest item. They propose the Markovian Pre-trained Transformer (MPT), which is pre-trained entirely on synthetic Markov chains sampled from Dirichlet transition matrices with random orthogonal state representations, and then fine-tuned on downstream recommendation data using only a lightweight input adaptor over semantic item embeddings. Experiments on five public datasets report strong gains over domain-specific and pre-trained baselines, with ablations on data scale, model scale, embedding source, and attention maps. The paper also claims that Markovian pre-training has a Bayes-optimal limit due to Dirichlet-multinomial conjugacy.

Critical strengths
  • The inference-time shuffling experiment is a simple and informative diagnostic. It raises a legitimate question about how much sequential order advanced recommenders actually exploit, and it could motivate more careful evaluation protocols in the recommendation community.
  • The synthetic pre-training pipeline is conceptually clean and computationally efficient. If the empirical gains hold under stronger controls, pre-training on controllable synthetic sequences could be a practical alternative to recommendation-specific or language pre-training.
Critical weaknesses
  • The central “Markovian” framing is inconsistent with the paper’s own evidence. The paper shows that using only the latest item is insufficient, and that the second-to-last or third-to-last item cannot substitute for the full non-sequential context. What the experiments support is a “last-item-anchored set summarization” behavior, not a Markov property. Naming the model Markovian and building the pre-training story around Markov chains therefore overstates the conceptual connection. This matters because the paper’s main contribution is the claim that Markovian pre-training stimulates the required recommendation capabilities.
  • The causal attribution to Markovian synthetic data is not established. The paper varies the state-space size and Dirichlet concentration, but it does not compare against other synthetic pre-training distributions, such as non-Markovian sequences, random sequences, n-gram processes, or other in-context estimation tasks. Without such controls, the gains could come from generic in-context sequence modeling, the input adaptor, semantic embeddings, or the larger effective pre-training data, rather than from Markovian structure. This is a critical gap for the paper’s central thesis.
Questions for authors
  1. How do you reconcile the term “Markovian” with the finding that the latest item alone is insufficient and that non-sequential context is essential? Please give a precise definition of the claimed property and explain why it should be called Markovian rather than last-item-anchored set summarization.
  2. What control synthetic pre-training tasks were run to show that Markov chains specifically, rather than generic in-context sequence modeling, are responsible for the downstream gains? If none were run, can you provide evidence that a non-Markovian synthetic generator would not achieve similar results?
  3. The Bayes estimator from Dirichlet-multinomial conjugacy bounds the synthetic pre-training loss. What is the analogous bound for downstream recommendation risk? How can Figure 4a be used to support a downstream performance limit rather than a limit on the synthetic task?
  4. The paper reports non-Markovian behavior on MovieLens and attributes it to pseudo-sequential data collection. What pre-specified criteria determine which datasets are suitable for the Markovian analysis? Would the conclusions hold on large-scale industrial recommendation data with different collection mechanisms?
6 comments
AuthorsOpenReview ↗

[1/4] The Meaning of "Markovian" and the Role of Non-Sequential Context [W1, Q1]

Responses

  1. Does the importance of historical context contradict our "Markovian" framing? NO. First, as discussed in Remark 1, we deliberately use quotation marks to indicate the informal meaning of "Markovian". Second, a first-order Markov chain with transition matrix P\mathbf{P} satisfies

    = \mathbb{P}(s_{t+1}\mid s_t;\mathbf{P}).$$ We view non-sequential historical context as playing an analogous role in characterizing user-specific transition behavior. In other words, with sufficient historical context, the model summarizes user preferences into a representation that plays a role analogous to the transition matrix $\mathbf{P}$: $$\mathbb{P}(v_{t+1}\mid v_t,v_{t-1},\ldots,v_1; \Theta) \approx \mathbb{P}\left(v_{t+1}\mid v_t; \underbrace{\\{v_1,\ldots,v_{t-1}\\}, \Theta}_{\text{role analogous to }\mathbf{P}}\right).$$ The SASRec NDCG@10 results across different maximum sequence lengths, reported below, suggest that the contextual prior provided by $\{v_1,\ldots,v_{t-1}\}$ stabilizes as the (inference) history length increases: |Max Len.| Beauty| Yelp| Retail| |-|:-:|:-:|:-:| |1| 0.0339| 0.0085| 0.0399| |3| 0.0374| 0.0108| 0.0468| |5| 0.0374| 0.0121| 0.0516| |10| 0.0369| 0.0138| 0.0550| |20| 0.0365| 0.0143| 0.0595| |50| 0.0362| 0.0142| 0.0601|
  2. Is "last-item-anchored set summarization" a better description? YES. We acknowledge that this phrase more formally describes the phenomenon illustrated in Figure 1. Nevertheless, we prefer to retain the informal term "Markovian" because: (1) it emphasizes the importance of the most recent item; and (2) "set summarization" does not convey the role of historical context as analogous to P\mathbf{P}.

Changes

  1. Line 195-204: Include P\mathbf{P} in the Markov-property equation and explain its analogy with summarized user context.
AuthorsOpenReview ↗

[2/4] Why Markov Chains Specifically? [W2, Q2]

Responses

  1. Is the causal attribution to Markovian synthetic data not established? (1) The top panel of Figure 4a shows that downstream recommendation performance improves as more Markovian synthetic data is incorporated. (2) Table 9 in the appendix compares pre-trained MPT with randomly initialized MPT, with the latter exhibiting substantially worse performance.

    BeautyToysSportsYelpRetail
    Randomly initialized0.06230.06640.03360.02210.0679
    Markovian pre-trained0.06730.07050.03730.02240.0741
  2. What control synthetic pre-training tasks were run? Thank you for this constructive suggestion. We have since conducted additional experiments using various types of synthetic trajectories (also suggested by Reviewer dE6V).

    BeautyToysSports
    Shuffled Markov0.06420.06610.0352
    Hidden Markov models0.06480.06810.0361
    Second-order Markov0.06640.07170.0368
    MPT (first-order Markov)0.06730.07050.0373

    These results indicate that Markovian pre-training is not the only way to achieve transferability. However, first- and second-order Markov pre-training yield the strongest results among the evaluated settings, suggesting that a well-defined next-state prediction structure should be prioritized when designing synthetic pre-training data.

Changes

  1. Line 451-456 & Appendix C.7: Include the additional control experiments and discuss their implications for the choice of Markov chains.
AuthorsOpenReview ↗

[3/4] Bayes Estimator and Downstream Performance [Q3]

Responses

  1. How can Figure 4a be used to support a downstream performance limit rather than a limit on the synthetic task? We would like to clarify the interpretation of Figure 4a. The bottom panel shows that LNSP\mathcal{L}_{\text{NSP}} plateaus as the number of training tokens increases, suggesting that further learning is constrained by the complexity of the synthetic task. In other words, incorporating more Markov chains yields diminishing gains for MPT. This saturation may help explain why downstream recommendation performance does not continue to improve as conventional scaling laws might suggest. Although this observation does not establish a downstream performance limit, it suggests an interesting connection between synthetic-task complexity and downstream scaling behavior that warrants further investigation.

Changes

  1. Line 414-416: We have clarified the interpretation of Figure 4a.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to the Reviewers

We thank the reviewers for their careful reading and constructive feedback. Our replies are organized by concern, with headings indicating the corresponding weaknesses (W) and questions (Q). Within each section, Responses provides our answers and supporting evidence, Misunderstandings clarifies points of potential confusion, and Changes identifies the corresponding manuscript revisions. Changes in the revised manuscript are marked in red.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.