S${}^2$LMR: Supervised Sequential Long-Context Multi-Step Retriever

Multi-step retrieval over long contexts is a key capability for retrieval-augmented generation, but strong existing approaches are costly and complex to train. Supervised retrievers are far simpler and cheaper, yet they fall well behind in this setting. We show that this gap stems not from supervised training itself, but from three misalignments between training and inference: easy training negatives, noisy retrieval prefixes, and position-agnostic scoring. Guided by this analysis, we introduce S${}^2$LMR, a simple yet effective supervised sequential retriever that models multi-step retrieval as the evolution of an explicit textual state encoded by a single shared Transformer. On long-context reasoning, multi-hop QA, and needle-in-a-haystack benchmarks with contexts from 4K to 10M tokens, S${}^2$LMR matches or outperforms state-of-the-art methods and far surpasses prior supervised retrievers, while training over 20$\times$ faster and remaining stable across a wide range of hyperparameters.