acceptodds
Under review as a conference paper at ICLR 2027

Fixed Rankings, Moving Leaders: A Relevance-Label and Source-Identity Audit of Scientific Retrieval Evaluation

Abstract

We show that a scientific retrieval benchmark can change its conclusions while every retrieval system stays fixed. In CER-Bench, a historical collection of 304 synthetic biomedical retrieval tasks over 4,936 PubMed records, replacing seed relevance labels with pooled automated judgments reverses nine of 45 pairwise system orderings on identical saved rankings (Kendall's τ_b = 0.584) and moves the mean-recall leader from a three-round search agent to a dense retriever. Label incompleteness is a known problem in retrieval evaluation, but it is rarely audited together with source identity: whether an identifier in the corpus refers to the article text attached to it. We audit both. An authoritative re-fetch of all 4,936 records traces a parser error that let cited-paper identifiers replace an article's own identifiers, changing 247 PMC identifiers, 226 of which occur in reference lists. A controlled Monte Carlo experiment with 1,000 paired, nested disclosures of the existing judgment pool locates the change of leader between six and nine revealed judgments per query, with paired empirical intervals overlapping zero at both points. We release a source-identity-checked retrieval dataset (4,936 documents, 10,313 chunks, 525 documents with conservatively restored own-article text), a hash-bound BM25 substrate, all saved rankings, judgments, disclosure arrays, and audit code. We deliberately do not release a corrected leaderboard: without human relevance validation or new model runs, the defensible output is an explicit boundary between reproducible arithmetic, verified source identity, and still-unvalidated relevance.

Primary area:
datasets and benchmarks
Keywords:
scientific information retrieval, benchmark evaluation, relevance judgments, data provenance, biomedical retrieval, benchmark auditing, retrieval evaluation, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_dgjLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper reviews the scientific literature retrieval benchmark CER-Bench, identifying errors regarding the source identities of certain documents within its corpus and analyzing how the construction of relevance labels affects the evaluation results of retrieval systems. By expanding the original benchmark labels with relevance judgments generated by an LLM, the study observed changes in retrieval system rankings and subsequently reconstructed a retrieval corpus with verified source identities.

Critical strengths

The paper has a relatively complete structure and content, and provides implementation details to ensure reproducibility.

Critical weaknesses
  1. The paper does not explicitly state the source and publication history of the CER-Bench being reviewed, so that’s unclear whether it is an existing public benchmark or a dataset previously constructed by the authors.

  2. This paper attempts to expand the original benchmark labels with relevance judgments generated by LLM and observes changes in the evaluation results, thus demonstrating that the benchmark is sensitive to the label construction method. However, the relevance judgments generated by LLM have not been independently verified and may not be closer to the true relevance. Therefore, ranking reversal can only show that the evaluation results depend on the label definition, but cannot lead to the conclusions of the original benchmark are invalid.

Questions for authors
  1. What is the provenance of CER-Bench? The paper does not explicitly state its source or release history, nor does it clarify whether it is an existing public benchmark or a dataset constructed by the authors.

  2. How can extended relevance labels generated by an LLM be validated to ensure they are more accurate or better represent true relevance compared to the original benchmark labels?

Reviewer_K5B5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits the reliability of scientific retrieval evaluation by revisiting CER-BENCH under fixed system rankings. It identifies two distinct issues: retrieval conclusions are sensitive to relevance-label construction, and the historical corpus contains source-identity errors caused by incorrect identifier extraction. Experiments further show that system rankings can change substantially as additional existing judgments are revealed, even when retrieval outputs remain unchanged.

Critical strengths
  1. The fixed-ranking analysis reveals a useful evaluation sensitivity. I like the insightful observation that system ordering can change as relevance judgments are expanded even when retrieval outputs remain unchanged. This provides a warning against drawing strong conclusions from retrieval benchmarks.

  2. The source-identity audit is reproducible. The paper traces identifier inconsistencies to a specific parsing issue and provides identity-checked artifacts and verification procedures. This makes the audit practically useful even though its broader scientific contribution is limited.

Critical weaknesses
  1. The contribution is limited beyond the benchmark-specific audit. The paper shows that relevance-label choices can change system rankings, but it provides limited explanation of why particular retrieval methods are affected or what broader retrieval insight follows from this observation.

  2. The two main findings remain empirically disconnected. The paper establishes source-identity errors and separately demonstrates ranking sensitivity to relevance labels, but it does not show how correcting the source-identity problem changes retrieval evaluation. This leaves a key gap in the paper’s ley chain.

  3. The experiments are not sufficient to establish scientific validity. I don't find out the human relevance validation or a validated evaluation on the corrected corpus, so it can show that the original conclusions are sensitive but cannot establish which evaluation better reflects actual relevance.

Questions for authors
  1. Can the authors clarify what generalizable insight follows from the observed ranking sensitivity beyond this specific benchmark? In particular, is there evidence that explains why certain retrieval methods are more sensitive to changes in relevance judgments than others?

  2. Can the authors clarify the empirical connection between the source-identity errors and the ranking sensitivity? Does correcting the source-identity issue change the retrieval evaluation, or should these be understood as two independent audit findings?

  3. Without human relevance validation or a validated evaluation on the corrected corpus, what evidence supports conclusions about scientific evaluation validity?

  4. Moreover, can the current analysis determine which evaluation better reflects actual relevance, rather than only showing that the evaluations disagree?

Reviewer_mPutOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper audits the biomedical retrieval benchmark CER-BENCH, revealing critical vulnerabilities under fixed system rankings. This paper examines CER-BENCH at three aspects: whether scores are calculated correctly, whether article identifiers match the attached text, and whether relevance labels are scientifically sound. Keeping the queries and search results fixed, the authors expanded the original labels with automated judgments. This changed the system rankings, including which system came first. Further tests showed that rankings depended on how many judgments were available, but did not establish a clear threshold. Separately, the authors found that the data-processing code could confuse identifiers in reference lists with an article’s own identifier. They then rebuilt the article text collection using verified sources and documented a basic search implementation with software checks. Without human relevance judgments or new model comparisons, the paper offers a more traceable basis for future evaluation, rather than a corrected biomedical ranking.

Critical strengths
  1. This paper provides a comprehensive analysis of the dataset, revealing several important issues for the datasets and corresponding ranking performance.
  2. Discovers an important question in the retrieval benchmark
Critical weaknesses
  1. The overall innovation is limited, it is more like an analytic paper.
  2. This paper spends a lot of time discussing the experimental results but does not provide a comprehensive introduction to the background or motivation for the study.
  3. It is already well established in information retrieval that incomplete relevance labels can change system comparisons, as the paper’s related work also acknowledges. The main contribution here is to bring label sensitivity and source-identity auditing into the same study, while thoroughly repairing one specific collection. However, the generalizability to other datasets is unknown
  4. There is no human validation, relevant papers outside the judgment pool remain unknown, and the data lack fields identifying which systems contributed the candidates.The authors openly acknowledge these limitations, but they also leave the paper at the diagnostic stage, without independent evidence to establish what the findings mean for retrieval in practice.
Questions for authors

See the above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.