- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SRBench, a sequential recommendation (SR) benchmark for evaluating both neural‑network‑based (NN‑SR) and large‑language‑model‑based (LLM‑SR) models. It makes three main contributions: (i) a multi‑dimensional evaluation framework measuring accuracy, quality (ARP/ARQ), stability (ARQV/ARR), and efficiency (ART). Aligned with practical deployment needs, it leverages the divergence‑based GEO Risk Score (GRS) as a heuristic indicator for latent Generative Engine Optimization (GEO) risks; (ii) a unified prompt‑enhanced input paradigm that transforms structured interaction histories and item metadata into standardized prompts to ensure fair cross‑paradigm evaluation; and (iii) a prompt‑extractor‑coupled mechanism to constrain semi‑structured outputs, extract valid item IDs from LLM generations, and eliminate hallucinated results. Evaluations across 14 models on ML‑100K, Beauty and Yelp datasets show that LLM‑SR models commonly suffer from popularity‑driven exposure bias and degraded stability versus NN‑SR baselines, and that few‑shot user histories are adequate for maintaining LLM‑SR recommendation accuracy.
Critical strengths
- Introduces an evaluation design that goes beyond accuracy by operationalizing quality (ARP, ARQ), stability (ARQV, ARR), and efficiency (ART) with output-only metrics applicable to both NN-SR and LLM-SR.
- Broad model coverage (six NN-SR and multiple LLM-SR models, including few-shot variants) on three real datasets; ablations on prompt types; extractor robustness study with low mismatch rate.
Critical weaknesses
- The GEO risk assessment (GRS) is a heuristic divergence from NN-SR outputs, not validated against controlled GEO attacks; it risks conflating paradigm differences, dataset artifacts, or prompt choices with adversarial susceptibility.
- Efficiency comparisons conflate model inference with network/API latency for hosted LLMs, making cross-paradigm results hard to interpret.
- LLM‑SR models are evaluated in zero‑shot mode with no recommendation‑specific fine‑tuning, while NN‑SR baselines are fully trained on target datasets. This mismatch in training setups hinders direct performance comparison.
Questions for authors
- The inputs of LLM‑SR models contain no information about candidate items. How does the model know what to output? Does this depend on the LLM memorizing this dataset?
- How is the latency of NN‑SR and LLM‑SR measured? Which models are deployed locally offline, and which ones call APIs? Is it unfair to compare locally‑deployed services against API calls?