Lost in Evaluation? Rethinking Sequential Recommendation Benchmarking via a Multi-dimensional and GEO Risk Assessment

The rapid development of LLM has drawn significant attention in Sequential Recommendation (SR). However, a comprehensive evaluation of SR models remains lacking due to the limitations of existing benchmarks: 1) an overemphasis on accuracy metrics neglects real-world demands such as quality, and overlooks emerging adversarial threats like Generative Engine Optimization (GEO), which can stealthily manipulate ranking results. 2) existing evaluation methods fail to fully exploit the semantic capabilities of LLM, leading to unfair comparisons between Neural Network-based SR (NN-SR) and LLM-based SR (LLM-SR) models; and 3) a reliable parsing mechanism is absent to extract task-specific results from unstructured LLM outputs. To address this gap, we propose SRBench, a fair and comprehensive benchmark. SRBench incorporates 1) a novel four-dimensional evaluation framework aligned with practical demands to holistically evaluate recommendation accuracy, quality, stability, and efficiency, which also serves as a latent GEO risk screening signal; 2) a unified prompt-enhanced input paradigm enables fair comparisons across recommendation paradigms; 3) a prompt-extractor-coupled extraction mechanism captures answers from LLM outputs through prompt-guided output formatting and a numeric-oriented extractor. Evaluating 14 mainstream models across several datasets with SRBench reveals several valuable insights, such that LLM-SR models overfocus on item popularity while lacking a deep understanding of item quality, and LLM-SR models with stronger semantic capabilities show behavioral divergence from NN-SR models, indicative of latent GEO risk. Thus, SRBench promotes holistic cross-paradigm evaluation and risk screening signals, providing a solid basis for reliable and trustworthy LLM-SR research and applications.