- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents SPECTRA, a multi-scenario benchmark for granular evaluation of Retrieval-Augmented Generation (RAG) in chemistry. Its defining contribution is a two-dimensional taxonomy: 6 scenarios×3 modifications. However, this paper has several issues in its evaluation stage: the main results are concentrated along a single direction and are insufficient to support the current conclusions. Meanwhile, the findings presented are largely consistent with existing understanding of RAG.
Critical strengths
- The scenario × modification design is the paper's strongest idea. Rather than collapsing RAG quality into one accuracy number, it decomposes performance along what kind of reasoning is required.
- The benchmark design is methodologically rigorous, employing an open-access controlled corpus augmented with explicit distractors and validated by a sufficient panel of human experts.
Critical weaknesses
- Table 2 is run on a single domain (Chelate Metal Complexes) with only two seeds and a fixed embedder and backbone. The cross-domain results that would establish generality are relegated to the appendix and use 50-question, single-seed samples, which the authors themselves describe as "sensitivity studies, not high-precision rankings." This gap is at odds with the contributions the paper sets out to make.
- Even across the eight domains, the topics cluster around the antimicrobial / nano-pharmaceutical area, which I do not believe is representative of the broader CHEMISTRY direction.
- The controlled corpus is a genuine strength for reproducibility, but at a scale of 333 documents it is far from a production-grade chemistry RAG setting. The DOI retrieval results further suggest that the correct paper typically resides within a very small candidate pool, so the benchmark likely underestimates the difficulty of source discovery in a realistic corpus spanning thousands of documents.
- The findings presented here largely coincide with existing intuition—namely, which classes of RAG/embedding approaches tend to be more effective for the chemistry domain.
Questions for authors
Beyond the issues raised in the Weaknesses, I have two more specific questions:
- Can you provide the results across all eight domains under the full configuration, together with a per-domain breakdown that reports associated uncertainties?
- Can you use statistical analysis to characterize how your retrieval conclusions would change at a realistic scale?
4 comments
- To address this concern, we extended the main configuration study to Nanozymes and Oxazolidinone Antibiotics and replaced Table 2 with the expanded results. All seven configurations are now evaluated on the complete question sets of three domains, using two seeds (mean ± SEM). Hybrid retrieval consistently outperforms sparse retrieval, while the leading configuration varies by domain. Seed-to-seed variability is small (Accuracy SEM ≤ 0.02). This substantially strengthens the evidence beyond the original single-domain evaluation and small-sample sensitivity studies.
- Exhaustive coverage of chemistry was not our objective. We prioritized depth over breadth, selecting specialized subdomains where expert knowledge enabled rigorous question construction and validation. This produces challenging, evidence-grounded tasks that assess retrieval and reasoning over scientific publications rather than primarily relying on LLM parametric knowledge. The domains differ in scientific focus and document characteristics (Table 1), while the six scenarios and three modifications target general RAG capabilities. Thus, SPECTRA provides a reusable evaluation methodology for scientific literature beyond the selected chemistry domains.
- We challenge the assumption that realistic RAG applications necessarily require thousands of documents. In our experience with enterprise deployments, knowledge bases containing hundreds of domain-specific documents are common. SPECTRA's 333 full-length publications, including 109 topically related distractors, therefore represent a practically relevant, task-bounded retrieval setting. The controlled design enables reproducible evaluation of source discrimination and downstream reasoning. We do not claim that DOI Recall would remain unchanged at substantially larger scales.
- Agreement with existing intuition does not diminish the value of systematically validating configuration choices. More importantly, SPECTRA's contribution extends beyond aggregate retrieval trends: it reveals scenario- and modification-specific differences and diagnostic failure patterns that inform targeted optimization (Tables 3–4; Appendix F). Its taxonomy and evaluation metrics can also be applied to other scientific domains. We will clarify this methodological contribution in the paper.
Q1. Figure 3 (Appendix F) already reports diagnostic results across all eight domains for Jina-Hybrid, showing both recurring failure patterns and differences in domain difficulty. These differences support selecting benchmark subsets representative of a target application. The updated Table 2 separately provides seven-configuration comparisons across three complete domains, with two-seed uncertainty estimates. We acknowledge that the full seven-configuration, eight-domain matrix with per-domain uncertainties remains outside the current evaluation.
Q2. We consider the current 333-document corpus representative of a realistic class of domain-specific RAG deployments, rather than merely a small-scale approximation. Our retrieval conclusions apply to this setting. Extrapolating them to collections of thousands of documents would require additional experiments, not statistical inference from the present corpus. We will clarify this distinction and the intended scope of our conclusions.
I thank the authors for their response, which addresses my concerns to some extent. However, my primary concern remains that, as this paper proposes an evaluation benchmark, I assess it along several dimensions: whether the proposed taxonomy and evaluation metrics are supported by sufficient empirical evidence, whether they generalize, and whether they yield novel insights. Given the scoring scale of 1–4, I do not find the authors’ response sufficient to justify a higher score. I therefore maintain my original score.
Thank you for your follow-up. We understand that your main concerns are 1) whether SPECTRA provides sufficient empirical support, 2) generalizes beyond the evaluated settings, and 3) offers novel insights. We believe the revised manuscript addresses these points more directly:
- Empirical support. Table 2 compares seven RAG configurations across three complete chemistry domains; Tables 3–4 examine performance across scenarios and modifications; Appendix F reports detailed results across all eight domains; and Appendix D validates the evaluation judges against independent experts, including category-level agreement. Could you clarify what additional empirical evidence you would consider sufficient, given the experiments and validation already included?
- Generalizability. SPECTRA's methodology is not chemistry-specific: its scenario-based taxonomy and diagnostic evaluation of retrieval, evidence localization, and answer quality can be applied to other scientific domains with grounded reference evidence. Related work has already demonstrated the broader applicability of scenario-based evaluation and granular RAG metrics, while our experiments across eight chemistry subdomains provide further empirical support for the framework.
- Novel insights. SPECTRA goes beyond aggregate evaluation: even multiple metrics may not reveal how well a system meets specific workload requirements or where its limitations arise. While individual elements overlap with prior benchmarks, SPECTRA brings together scientific question requirements and separate evaluations of document retrieval, evidence localization, and answer quality, enabling more informative diagnostics. For example, in Multi-turn, Answer Accuracy declines despite relatively stable DOI Recall, showing that degradation cannot be explained solely by losing the relevant documents and helping narrow down potential causes more quickly. The Aggregation results demonstrate SPECTRA's practical value in identifying a challenging workload where standard RAG configurations struggle and providing a testbed for evaluating alternative architectures such as RAGU.
We also noticed that your original Soundness rating was 2, indicating minor or fixable concerns. We have addressed the specific issues raised in our response and revised the manuscript. Could you clarify whether any soundness concerns remain unresolved?
Thank you again for your time and feedback.
We thank the reviewer for the feedback and for recognizing the strengths of SPECTRA's benchmark design and expert validation.
Your concern regarding cross-domain evaluation motivated us to substantially strengthen the experimental evidence through additional experiments and an expanded main results table. We believe the remaining concerns largely reflect differences in expectations regarding the intended scope and contributions of SPECTRA. In our responses below, we clarify these points and emphasize that our primary contribution is a rigorously validated framework for systematic, fine-grained RAG evaluation, rather than exhaustive chemistry coverage or universal conclusions about RAG performance.
We hope our detailed responses and additional results address your concerns and provide grounds for reconsidering your initial rating. We would be grateful if you could reassess your evaluation in light of these clarifications.