- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes SCOPE, a new retrieval system that targets financial filings and papers by proposing a series of components based on properties of such long structured documents. Experiments report that the method has a moderate gain in retrieval performance but no improvement in answer quality. The paper is being honest in reporting a major source of the gain, which is not from the proposed components but from a prompt change discovered from the test set.
Critical strengths
None
Critical weaknesses
-
Methodology-wise, the paper is trivial. The proposed components are not validated by experimental gains, and the significant gains come from "A one-line rule for query modification, to keep every year and fiscal period." which carries no scientific value at all.
-
Experiment-wise, the core claim is weird. While the paper has a diagnostic paper-style title, it reads to me like a methodology paper at the beginning, and it indeed describes a new method. However, when we arrive at the experiments, the conclusion of the experiments was that the gains are from an instruction that was discovered "post hoc" in the test set, and that the gains do not improve the answers in RAG. While negative methods could be potentially helpful, I don't see any value of a paper to propose a random method and use experiments to prove that this method does not work at all. I thank the real author of this paper for being honest.
Questions for authors
What is the main claim of this paper?