acceptodds
Under review as a conference paper at ICLR 2027

Lost in Evaluation? Rethinking Sequential Recommendation Benchmarking via a Multi-dimensional and GEO Risk Assessment

Abstract

The rapid development of LLM has drawn significant attention in Sequential Recommendation (SR). However, a comprehensive evaluation of SR models remains lacking due to the limitations of existing benchmarks: 1) an overemphasis on accuracy metrics neglects real-world demands such as quality, and overlooks emerging adversarial threats like Generative Engine Optimization (GEO), which can stealthily manipulate ranking results. 2) existing evaluation methods fail to fully exploit the semantic capabilities of LLM, leading to unfair comparisons between Neural Network-based SR (NN-SR) and LLM-based SR (LLM-SR) models; and 3) a reliable parsing mechanism is absent to extract task-specific results from unstructured LLM outputs. To address this gap, we propose SRBench, a fair and comprehensive benchmark. SRBench incorporates 1) a novel four-dimensional evaluation framework aligned with practical demands to holistically evaluate recommendation accuracy, quality, stability, and efficiency, which also serves as a latent GEO risk screening signal; 2) a unified prompt-enhanced input paradigm enables fair comparisons across recommendation paradigms; 3) a prompt-extractor-coupled extraction mechanism captures answers from LLM outputs through prompt-guided output formatting and a numeric-oriented extractor. Evaluating 14 mainstream models across several datasets with SRBench reveals several valuable insights, such that LLM-SR models overfocus on item popularity while lacking a deep understanding of item quality, and LLM-SR models with stronger semantic capabilities show behavioral divergence from NN-SR models, indicative of latent GEO risk. Thus, SRBench promotes holistic cross-paradigm evaluation and risk screening signals, providing a solid basis for reliable and trustworthy LLM-SR research and applications.

Primary area:
datasets and benchmarks
Keywords:
Large Language Model, Sequential Recommendation Benchmark, Generative Engine Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_sfpVOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SRBench, a sequential recommendation (SR) benchmark for evaluating both neural‑network‑based (NN‑SR) and large‑language‑model‑based (LLM‑SR) models. It makes three main contributions: (i) a multi‑dimensional evaluation framework measuring accuracy, quality (ARP/ARQ), stability (ARQV/ARR), and efficiency (ART). Aligned with practical deployment needs, it leverages the divergence‑based GEO Risk Score (GRS) as a heuristic indicator for latent Generative Engine Optimization (GEO) risks; (ii) a unified prompt‑enhanced input paradigm that transforms structured interaction histories and item metadata into standardized prompts to ensure fair cross‑paradigm evaluation; and (iii) a prompt‑extractor‑coupled mechanism to constrain semi‑structured outputs, extract valid item IDs from LLM generations, and eliminate hallucinated results. Evaluations across 14 models on ML‑100K, Beauty and Yelp datasets show that LLM‑SR models commonly suffer from popularity‑driven exposure bias and degraded stability versus NN‑SR baselines, and that few‑shot user histories are adequate for maintaining LLM‑SR recommendation accuracy.

Critical strengths
  1. Introduces an evaluation design that goes beyond accuracy by operationalizing quality (ARP, ARQ), stability (ARQV, ARR), and efficiency (ART) with output-only metrics applicable to both NN-SR and LLM-SR.
  2. Broad model coverage (six NN-SR and multiple LLM-SR models, including few-shot variants) on three real datasets; ablations on prompt types; extractor robustness study with low mismatch rate.
Critical weaknesses
  1. The GEO risk assessment (GRS) is a heuristic divergence from NN-SR outputs, not validated against controlled GEO attacks; it risks conflating paradigm differences, dataset artifacts, or prompt choices with adversarial susceptibility.
  2. Efficiency comparisons conflate model inference with network/API latency for hosted LLMs, making cross-paradigm results hard to interpret.
  3. LLM‑SR models are evaluated in zero‑shot mode with no recommendation‑specific fine‑tuning, while NN‑SR baselines are fully trained on target datasets. This mismatch in training setups hinders direct performance comparison.
Questions for authors
  1. The inputs of LLM‑SR models contain no information about candidate items. How does the model know what to output? Does this depend on the LLM memorizing this dataset?
  2. How is the latency of NN‑SR and LLM‑SR measured? Which models are deployed locally offline, and which ones call APIs? Is it unfair to compare locally‑deployed services against API calls?
Reviewer_bsuiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

SRBench compares trained neural recommenders against off-the-shelf LLMs on Yelp, ML-100K, and Beauty. Beyond Recall/NDCG, it reports popularity (ARP), average rating (ARQ), within-list rating variance (ARQV), cross-run overlap (ARR), and latency (ART). Histories are converted into prompts, and a parser extracts item IDs from free-form LLM outputs. The distinctive claim is a GEO Risk Score (GRS): a weighted distance between an LLM's four-dimensional profile and the average neural model, presented as a screening signal for Generative Engine Optimization risk.

Critical strengths
  1. Looking past accuracy for LLM recommenders is a reasonable goal. Invalid IDs, run-to-run variability, and latency are real deployment issues.
  2. The empirical sweep is large: 6 neural models, 8 LLMs, 3 datasets, repeated runs, a short-history setting, and hallucination rates. Prompt templates and the ID filter are concrete enough to reuse. Code is linked.
  3. Appendix D does eventually state that GRS is a behavioral indicator on clean inputs, not an attack detector. That paragraph is useful. The abstract and title do not follow it.
Critical weaknesses
  1. GRS does not measure GEO risk. In the main tables it is a weighted absolute gap between an LLM and the neural average on accuracy / quality / stability / efficiency, with no rewritten item text and no target being promoted. Appendix D says this explicitly, and the limitations leave validation against attack outcomes as future work. The only injection experiment uses two Llama models on one dataset, and GRS is never correlated with the resulting rank lift. Despite that, the title, abstract, and "insights" talk about latent GEO risk. Distance from the neural average is not vulnerability.
  2. The score treats every difference as risk, and latency can dominate. A model that recommends fewer popular items, or is simply slower, moves away from the neural average and looks "riskier." On Yelp, Grok-3's GRS of 0.7114 sits on an efficiency term near 1 because a call takes ~20s. There is no sensitivity check with equal weights or with efficiency removed. The component weights themselves (0.7/0.3 popularity vs rating; 0.6/0.4 variance vs overlap) are chosen by argument.
  3. Some reported numbers look suspicious. On Yelp, Llama-2 and Llama-3.2 (full history and 5-item history) all have ARR = 0.4500 exactly. The written ARR formula does not define the overlap clearly enough to reproduce, and four identical values across different settings need an explanation before that column can be trusted. Separately, Table 5 says NN-SR beats LLM-SR on ML-100K quality, while the Figure 21 discussion says LLM-SR models have better quality scores on that dataset. Those cannot both be right.
  4. The NN-vs-LLM comparison does not mean what the paper claims. Neural models are trained; LLMs are zero-shot, with no fine-tuned / LoRA recommender in the pool. NCF, NGCF, and LightGCN are not sequential models, yet they enter the average that defines GRS. Latency compares local inference to API time. Overlap compares a deterministic checkpoint to a sampled API. Calling LLMs "less stable" is close to restating the decoding setup.
  5. Several metrics do not match their names. Within-list rating variance can be high for a fixed mixed-rating list and zero for a randomly changing equal-rated list — that is not run-to-run stability. Average star rating is not personalized quality. Deleting hallucinated IDs shortens the prediction list, so high-hallucination models (e.g., Claude-Sonnet-4 on Yelp) lose Recall for a mechanical reason.
  6. It is unclear what catalog information the LLM actually sees for IDs outside the history. The prompt includes history text and a max item ID, but not an explicit mapping for arbitrary unseen IDs. If that mapping is missing, low accuracy may just mean the generation task is under-specified. The paper needs to show a complete model input.
  7. The research questions do not really stress-test the benchmark. Running 14 models and getting a table is treated as evidence the evaluation is reliable. A higher rate of parseable outputs is treated as evidence the augmented prompt is a better recommender interface. There are no confidence intervals, even when conclusions are drawn from Recall gaps on the order of 0.0001.
Questions for authors
  1. Evidence: under one fixed rewrite protocol, what is the correlation across the 8 LLMs between GRS and the attacked item's rank lift? If you do not have this, GRS should be removed from the title and abstract. I would still reject, but then I would no longer be objecting to a false claim.
  2. Clarification: what catalog text / ID list / ID-to-title map is visible to the LLM for items outside the history? Please quote one complete prompt.
  3. Clarification: how exactly is ARR computed, and why is it 0.4500 for four different Llama settings on Yelp? For neural stability numbers, were models retrained each repetition or only re-inferred?
Reviewer_spQJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the limitations of existing sequential recommendation benchmarks and proposes SRBench, a unified evaluation framework for neural network-based and LLM-based recommendation models. The framework evaluates recommendation performance across four dimensions: accuracy, quality, stability, and efficiency. It also introduces a GEO Risk Score (GRS) to assess potential ranking manipulation risks based on behavioral differences between model paradigms. In addition, SRBench incorporates a prompt-enhanced input paradigm and an output extraction mechanism to standardize LLM-generated recommendations. Experiments on 14 models across Yelp, MovieLens-100K, and Beauty demonstrate differences in recommendation accuracy, popularity preference, stability, and computational efficiency.

Critical strengths

The main strength of this paper is its multi-dimensional evaluation framework, which extends traditional accuracy-based benchmarking to include recommendation quality, stability, and efficiency. The unified input paradigm and output extraction mechanism provide a practical approach to comparing different recommendation models. Furthermore, the evaluation across 14 models and three datasets offers useful insights into the performance differences between neural network-based and LLM-based sequential recommendation methods.

Critical weaknesses

The main limitation is that the proposed GEO Risk Score primarily measures behavioral differences between LLM-based and neural network-based recommendation models, without sufficient validation of its relationship to actual GEO attack vulnerability. For example, in Table 12, Grok-3 receives a high GRS of 0.7114, partly driven by its large inference-time deviation, which does not necessarily indicate greater susceptibility to ranking manipulation. In addition, the fairness of cross-paradigm comparisons requires further justification, since the neural network models are trained specifically for recommendation tasks while the LLM-based models operate without recommendation-specific fine-tuning and receive additional semantic information. Some evaluation metrics also require stronger justification. In particular, ARQV measures rating variance within recommendation lists, which may not directly reflect model stability across repeated executions.

Questions for authors
  1. The proposed GRS measures behavioral divergence from neural network-based models rather than directly measuring GEO attack vulnerability. Could the authors further clarify this relationship using the existing GEO manipulation experiments in Appendix F.3? What are the main limitations of GRS as a risk screening metric?
  2. Since the neural network-based models are trained specifically for recommendation tasks while the LLM-based models operate without task-specific fine-tuning, could the authors further justify the fairness of the comparison? How might these differences in training and input information affect the conclusions?
  3. ARQV measures rating variance within individual recommendation lists rather than consistency across repeated executions. Could the authors clarify why ARQV is included in the stability dimension and how it differs conceptually from the ARR metric?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.