Markovian Pre-Trained Transformer for Next-Item Recommendation

We introduce the Markovian Pre-trained Transformer (MPT) for next-item recommendation, a transferable model fully pre-trained on synthetic Markov chains, yet capable of achieving state-of-the-art performance by fine-tuning a lightweight adaptor. This counterintuitive success can be attributed to the observed `Markovian' characteristics: advanced sequential recommenders tend to rely predominantly on the latest interaction for prediction, while historical interactions primarily provide contextual signals for inferring the user's general, non-sequential identity. These characteristics necessitate the capabilities of a universal recommendation model to (1) effectively summarize the user sequence, with (2) particular emphasis on the latest interaction. MPT acquires these inherent capabilities from carefully constructed synthetic data. On the one hand, when trained to predict the next state of an arbitrary Markov chain, it must learn to (1) estimate transition probabilities from contextual information (one adaptive manner for sequence summarization), and (3) attend to the most recent state to ensure an accurate restart point. On the other hand, unlike the heterogeneous interaction data, an unlimited number of controllable Markov chains is available to boost the model capacity. Extensive experiments on five public datasets from three distinct platforms validate the superiority of Markovian pre-training over traditional recommendation pre-training (+39% on average) and recent language pre-training (+24% on average) paradigms.