- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper reviews the scientific literature retrieval benchmark CER-Bench, identifying errors regarding the source identities of certain documents within its corpus and analyzing how the construction of relevance labels affects the evaluation results of retrieval systems. By expanding the original benchmark labels with relevance judgments generated by an LLM, the study observed changes in retrieval system rankings and subsequently reconstructed a retrieval corpus with verified source identities.
Critical strengths
The paper has a relatively complete structure and content, and provides implementation details to ensure reproducibility.
Critical weaknesses
-
The paper does not explicitly state the source and publication history of the CER-Bench being reviewed, so that’s unclear whether it is an existing public benchmark or a dataset previously constructed by the authors.
-
This paper attempts to expand the original benchmark labels with relevance judgments generated by LLM and observes changes in the evaluation results, thus demonstrating that the benchmark is sensitive to the label construction method. However, the relevance judgments generated by LLM have not been independently verified and may not be closer to the true relevance. Therefore, ranking reversal can only show that the evaluation results depend on the label definition, but cannot lead to the conclusions of the original benchmark are invalid.
Questions for authors
-
What is the provenance of CER-Bench? The paper does not explicitly state its source or release history, nor does it clarify whether it is an existing public benchmark or a dataset constructed by the authors.
-
How can extended relevance labels generated by an LLM be validated to ensure they are more accurate or better represent true relevance compared to the original benchmark labels?