acceptodds
Under review as a conference paper at ICLR 2027

SPECTRA: A Multi-Scenario Benchmark for Granular RAG Evaluation in Chemistry

Abstract

Despite rapid progress in large language models (LLMs) and agentic systems, retrieval-augmented generation (RAG) remains indispensable in any setting where authoritative answers must come from a controlled, often proprietary corpus such as scientific literature, internal company documentation, or regulatory archives. We introduce (cenario-artitioned xpert hemistry estbed for AG ssessment), a benchmark for fine-grained evaluation of RAG systems on expert-level questions grounded in open-access scientific publications. The benchmark organizes 2007 questions across 8 chemistry domains into a two-dimensional taxonomy: 6 question scenarios (single-hop extraction, multi-hop reasoning, conditional filtering, comparison, aggregation, and multi-turn dialogue) that isolate distinct system capabilities and 3 orthogonal modification types (multimodal, constrained, and negative queries). We also propose an evaluation framework to assess LLMs and RAG configurations across several metrics and apply it to 7 RAG configurations and 3 standalone LLM baselines, with the primary Answer Accuracy judge validated against two experts on 200 usable examples. The benchmark evaluation yields two key findings. First, scenario- and modification-level evaluation on one domain reveals performance trade-offs invisible to aggregate scoring: configurations that appear equivalent by overall accuracy differ substantially across question types. This allows practitioners to optimize RAG configurations according to their primary use case. Second, the dedicated multi-turn scenario illustrates how per-turn analysis supports fault localization: document-level DOI Recall remains stable across dialogue turns for hybrid configurations, while evidence-level Source Support declines modestly and Answer Accuracy declines more strongly. This shows that degradation is not explained by complete failure to retrieve the source article. While the corpus is chemistry-focused, the scenario taxonomy and evaluation protocol are not intrinsically chemistry-specific and can be adapted to other scientific corpora. Beyond evaluation, the per-scenario performance vectors produced by our framework can support data-driven RAG configuration design, and the semi-automated question generation pipeline can suggest a route toward periodic question refresh to reduce contamination. The dataset and evaluation code are publicly available through anonymized repositories: https://huggingface.co/collections/anonymousauthor2026team/scientific-rag-benchmark-collection and https://anonymous.4open.science/r/1c7ffbd3c15e.

Primary area:
datasets and benchmarks
Keywords:
retrieval-augmented generation, RAG evaluation, benchmarks, scientific literature, chemistry, information retrieval, LLM evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YNaWOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents SPECTRA, a multi-scenario benchmark for granular evaluation of Retrieval-Augmented Generation (RAG) in chemistry. Its defining contribution is a two-dimensional taxonomy: 6 scenarios×3 modifications. However, this paper has several issues in its evaluation stage: the main results are concentrated along a single direction and are insufficient to support the current conclusions. Meanwhile, the findings presented are largely consistent with existing understanding of RAG.

Critical strengths
  1. The scenario × modification design is the paper's strongest idea. Rather than collapsing RAG quality into one accuracy number, it decomposes performance along what kind of reasoning is required.
  2. The benchmark design is methodologically rigorous, employing an open-access controlled corpus augmented with explicit distractors and validated by a sufficient panel of human experts.
Critical weaknesses
  1. Table 2 is run on a single domain (Chelate Metal Complexes) with only two seeds and a fixed embedder and backbone. The cross-domain results that would establish generality are relegated to the appendix and use 50-question, single-seed samples, which the authors themselves describe as "sensitivity studies, not high-precision rankings." This gap is at odds with the contributions the paper sets out to make.
  2. Even across the eight domains, the topics cluster around the antimicrobial / nano-pharmaceutical area, which I do not believe is representative of the broader CHEMISTRY direction.
  3. The controlled corpus is a genuine strength for reproducibility, but at a scale of 333 documents it is far from a production-grade chemistry RAG setting. The DOI retrieval results further suggest that the correct paper typically resides within a very small candidate pool, so the benchmark likely underestimates the difficulty of source discovery in a realistic corpus spanning thousands of documents.
  4. The findings presented here largely coincide with existing intuition—namely, which classes of RAG/embedding approaches tend to be more effective for the chemistry domain.
Questions for authors

Beyond the issues raised in the Weaknesses, I have two more specific questions:

  1. Can you provide the results across all eight domains under the full configuration, together with a per-domain breakdown that reports associated uncertainties?
  2. Can you use statistical analysis to characterize how your retrieval conclusions would change at a realistic scale?
4 comments

We thank the reviewer for the feedback and for recognizing the strengths of SPECTRA's benchmark design and expert validation.

Your concern regarding cross-domain evaluation motivated us to substantially strengthen the experimental evidence through additional experiments and an expanded main results table. We believe the remaining concerns largely reflect differences in expectations regarding the intended scope and contributions of SPECTRA. In our responses below, we clarify these points and emphasize that our primary contribution is a rigorously validated framework for systematic, fine-grained RAG evaluation, rather than exhaustive chemistry coverage or universal conclusions about RAG performance.

We hope our detailed responses and additional results address your concerns and provide grounds for reconsidering your initial rating. We would be grateful if you could reassess your evaluation in light of these clarifications.

  1. To address this concern, we extended the main configuration study to Nanozymes and Oxazolidinone Antibiotics and replaced Table 2 with the expanded results. All seven configurations are now evaluated on the complete question sets of three domains, using two seeds (mean ± SEM). Hybrid retrieval consistently outperforms sparse retrieval, while the leading configuration varies by domain. Seed-to-seed variability is small (Accuracy SEM ≤ 0.02). This substantially strengthens the evidence beyond the original single-domain evaluation and small-sample sensitivity studies.
  2. Exhaustive coverage of chemistry was not our objective. We prioritized depth over breadth, selecting specialized subdomains where expert knowledge enabled rigorous question construction and validation. This produces challenging, evidence-grounded tasks that assess retrieval and reasoning over scientific publications rather than primarily relying on LLM parametric knowledge. The domains differ in scientific focus and document characteristics (Table 1), while the six scenarios and three modifications target general RAG capabilities. Thus, SPECTRA provides a reusable evaluation methodology for scientific literature beyond the selected chemistry domains.
  3. We challenge the assumption that realistic RAG applications necessarily require thousands of documents. In our experience with enterprise deployments, knowledge bases containing hundreds of domain-specific documents are common. SPECTRA's 333 full-length publications, including 109 topically related distractors, therefore represent a practically relevant, task-bounded retrieval setting. The controlled design enables reproducible evaluation of source discrimination and downstream reasoning. We do not claim that DOI Recall would remain unchanged at substantially larger scales.
  4. Agreement with existing intuition does not diminish the value of systematically validating configuration choices. More importantly, SPECTRA's contribution extends beyond aggregate retrieval trends: it reveals scenario- and modification-specific differences and diagnostic failure patterns that inform targeted optimization (Tables 3–4; Appendix F). Its taxonomy and evaluation metrics can also be applied to other scientific domains. We will clarify this methodological contribution in the paper.

Q1. Figure 3 (Appendix F) already reports diagnostic results across all eight domains for Jina-Hybrid, showing both recurring failure patterns and differences in domain difficulty. These differences support selecting benchmark subsets representative of a target application. The updated Table 2 separately provides seven-configuration comparisons across three complete domains, with two-seed uncertainty estimates. We acknowledge that the full seven-configuration, eight-domain matrix with per-domain uncertainties remains outside the current evaluation.

Q2. We consider the current 333-document corpus representative of a realistic class of domain-specific RAG deployments, rather than merely a small-scale approximation. Our retrieval conclusions apply to this setting. Extrapolating them to collections of thousands of documents would require additional experiments, not statistical inference from the present corpus. We will clarify this distinction and the intended scope of our conclusions.

I thank the authors for their response, which addresses my concerns to some extent. However, my primary concern remains that, as this paper proposes an evaluation benchmark, I assess it along several dimensions: whether the proposed taxonomy and evaluation metrics are supported by sufficient empirical evidence, whether they generalize, and whether they yield novel insights. Given the scoring scale of 1–4, I do not find the authors’ response sufficient to justify a higher score. I therefore maintain my original score.

Thank you for your follow-up. We understand that your main concerns are 1) whether SPECTRA provides sufficient empirical support, 2) generalizes beyond the evaluated settings, and 3) offers novel insights. We believe the revised manuscript addresses these points more directly:

  1. Empirical support. Table 2 compares seven RAG configurations across three complete chemistry domains; Tables 3–4 examine performance across scenarios and modifications; Appendix F reports detailed results across all eight domains; and Appendix D validates the evaluation judges against independent experts, including category-level agreement. Could you clarify what additional empirical evidence you would consider sufficient, given the experiments and validation already included?
  2. Generalizability. SPECTRA's methodology is not chemistry-specific: its scenario-based taxonomy and diagnostic evaluation of retrieval, evidence localization, and answer quality can be applied to other scientific domains with grounded reference evidence. Related work has already demonstrated the broader applicability of scenario-based evaluation and granular RAG metrics, while our experiments across eight chemistry subdomains provide further empirical support for the framework.
  3. Novel insights. SPECTRA goes beyond aggregate evaluation: even multiple metrics may not reveal how well a system meets specific workload requirements or where its limitations arise. While individual elements overlap with prior benchmarks, SPECTRA brings together scientific question requirements and separate evaluations of document retrieval, evidence localization, and answer quality, enabling more informative diagnostics. For example, in Multi-turn, Answer Accuracy declines despite relatively stable DOI Recall, showing that degradation cannot be explained solely by losing the relevant documents and helping narrow down potential causes more quickly. The Aggregation results demonstrate SPECTRA's practical value in identifying a challenging workload where standard RAG configurations struggle and providing a testbed for evaluating alternative architectures such as RAGU.

We also noticed that your original Soundness rating was 2, indicating minor or fixable concerns. We have addressed the specific issues raised in our response and revised the manuscript. Could you clarify whether any soundness concerns remain unresolved?

Thank you again for your time and feedback.

Reviewer_YoigOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SPECTRA, a benchmark for evaluating retrieval-augmented generation (RAG) systems on scientific chemistry literature. It contains 2,007 questions across eight chemistry domains, organized into six question scenarios and three modification types. The authors evaluate seven RAG configurations and three standalone LLM baselines using answer accuracy, retrieval metrics, grounding metrics, and diagnostic signals. A key motivation is that aggregate accuracy can obscure meaningful differences in system capabilities, and that scenario-specific evaluation can better inform RAG configuration selection.

The paper also examines multi-turn degradation and reports that document-level DOI recall can remain relatively stable while evidence support and answer accuracy decline. Overall, SPECTRA aims to provide a more granular and actionable evaluation framework for scientific RAG systems.

Critical strengths

Practical and relevant evaluation problem.

Aggregate answer accuracy is insufficient for understanding the failure modes of RAG systems. Separating single-hop retrieval, multi-hop reasoning, conditional filtering, comparison, aggregation, and multi-turn dialogue is a useful direction, particularly for scientific applications where answers must be grounded in specific sources.

Multi-dimensional benchmark design.

The combination of scenario categories and query modifications provides a structured way to examine capabilities that are often evaluated separately. Including constrained, multimodal, and negative queries broadens the scope beyond conventional fact-based question answering.

Useful diagnostic perspective.

The distinction between retrieving the correct source document, locating relevant evidence, and producing a correct answer is valuable. The multi-turn analysis illustrates how these signals can diverge and motivates more detailed evaluation than a single end-to-end score.

Effort toward validation and reproducibility.

The benchmark is grounded in open-access publications, uses expert validation of generated questions, and includes an evaluation of the primary automatic judge against human experts. The reported configuration comparisons and supplementary analyses also provide useful empirical evidence about the behavior of the evaluated systems.

Critical weaknesses
  1. The incremental contribution over existing benchmarks needs to be established more convincingly.

The central contribution is a combination of a scenario taxonomy, query modifications, and a diagnostic evaluation framework. While this combination is potentially useful, the paper does not yet make the incremental value of the combined design sufficiently clear.

In particular, the claim that scenario-level evaluation enables better RAG configuration selection would be stronger with a direct comparison against simpler alternatives, such as selecting configurations using aggregate accuracy or a smaller set of conventional retrieval and answer-quality metrics. It is not enough to show that configurations have different per-scenario scores; the authors should demonstrate that the proposed evaluation leads to materially different or better-supported configuration decisions.

A clearer comparison with existing scientific RAG benchmarks and evaluation frameworks, using explicit differences in task coverage, evidence annotation, diagnostic validity, and practical utility, would strengthen the novelty claim.

  1. The diagnostic metrics do not fully establish the causes of system failures.

Separating document retrieval, evidence support, and answer accuracy is a useful design choice. However, differences between these metrics do not necessarily identify the underlying cause of an error.

For example, stable DOI Recall alongside declining answer accuracy in the multi-turn scenario suggests that complete failure to retrieve the source document is not the only explanation for performance degradation. Nevertheless, it does not establish whether the remaining errors primarily arise from evidence localization, query reformulation, context integration, or answer generation.

The paper would benefit from controlled interventions that isolate these stages. For example, evaluating the same questions with oracle documents or oracle evidence passages, while holding the generator fixed, could help quantify how much of the performance gap is attributable to retrieval versus downstream reasoning and generation. The current results are useful diagnostic signals, but some of the interpretations should remain hypotheses rather than definitive failure attributions.

  1. The robustness and generalizability of configuration comparisons remain uncertain.

The main configuration comparisons rely substantially on the Chelate Metal Complexes domain, while several additional domain analyses use relatively small samples. The primary plots report averages over two random seeds. These experiments provide an initial comparison, but they offer limited evidence that fine-grained rankings are stable across domains, question samples, and execution variability.

This matters because the paper's practical motivation is to help users select an appropriate RAG configuration for a target workload. If the apparent differences between configurations are sensitive to the sampled questions or domain, the resulting recommendations may not generalize.

The authors should clarify which conclusions are supported across multiple domains and which are specific to the main evaluation domain. For small performance differences, uncertainty estimates and paired comparisons should be emphasized, and stronger cross-domain validation would improve confidence in the proposed selection framework.

  1. Automatic evaluation is promising but requires more evidence for difficult question types.

The authors validate Gemini-2.5-Flash against two human experts on 200 usable examples and additionally evaluate an alternative judge. This is a positive aspect of the work. However, the reliability of answer evaluation may vary substantially across direct extraction, multi-hop reasoning, aggregation, multimodal questions, and negative queries.

In particular, a correct-looking answer may omit required items, fail to satisfy a constraint, or provide a plausible response to an unanswerable query. These cases require different evaluation criteria, and aggregate judge agreement may conceal errors in specific categories.

The paper would be stronger if it reported agreement and error patterns separately for the most challenging scenarios and modifications. It would also be useful to clarify how partial answers, incomplete aggregation, unsupported claims, and appropriate abstention are handled, and whether judge errors could change the relative ranking of RAG configurations.

  1. The practical conclusions should be bounded by the benchmark's scope.

SPECTRA is constructed from a controlled corpus of open-access chemistry publications. This makes the evaluation more reproducible, but real scientific search environments can contain larger and more heterogeneous collections, duplicate or conflicting sources, evolving literature, and different distributions of user questions.

The paper also reports that standalone LLMs outperform the evaluated RAG configurations on aggregation questions. This is an interesting observation, but it should not be interpreted as a general limitation of RAG: performance can depend heavily on retrieval coverage, context construction, and the generation model. The appendix itself provides evidence that a different RAG pipeline can substantially change the result.

The authors appropriately acknowledge that the results are pipeline-dependent. The discussion should consistently distinguish conclusions about the evaluated configurations from broader conclusions about RAG systems in general.

Questions for authors

I just wrote all the questions in the critical weaknesses part.

4 comments

We sincerely thank the reviewer for the thoughtful and constructive feedback. We especially appreciate your recognition of the practical relevance of SPECTRA, its multi-dimensional benchmark design, diagnostic perspective, and efforts toward validation and reproducibility.

Your comments helped us strengthen the paper, particularly through additional cross-domain experiments, a more detailed analysis of judge reliability across question types, and clearer statements of the limitations. We also hope to clarify an important aspect of our contribution: SPECTRA is not intended to identify universally optimal RAG configurations or establish exact causes of failures, but to provide a structured benchmark and evaluation methodology that supports both configuration selection and targeted improvement.

Below, we address each of your concerns in detail. We hope our clarifications and additional results will resolve your concerns and demonstrate the value of our contribution. We would be grateful if you could reconsider your initial rating in light of our responses.

  1. We agree that different per-scenario scores alone are not enough. SPECTRA combines question types with diagnostic metrics to support both configuration selection and targeted improvement. Figure 3 (Appendix F) shows distinct failure patterns: Aggregation has much higher D. Incomplete than Single-hop (0.89 vs. 0.38), while multimodal questions show sharp declines in Numeric and Scientific Fact Recall despite high DOI Recall. These results highlight different optimization priorities beyond aggregate Accuracy. Tables 3–4 also show how the preferred configuration changes across workloads. Regarding novelty, Table 5 (Appendix A) compares existing benchmarks in task coverage, evidence annotations, evaluation validity, and practical utility. SPECTRA brings these aspects together in a unified framework with six scenarios, three modifications, gold evidence annotations, and a human-validated answer judge. Its practical value is further demonstrated by a separate study, recently accepted at NeurIPS, which uses SPECTRA to achieve top-performing RAG configurations for the new, unseen use cases. We do not disclose details due to the double-blind reviewing policy, but will include the citation in the camera-ready version.
  2. We agree that diagnostic metrics indicate potential failure points rather than establish exact causes. Our goal is to provide a benchmark and evaluation framework that enables such investigations, not to perform a full causal analysis of individual RAG pipelines. We will clarify this distinction. Oracle-based experiments and deeper analysis of scenario-specific metrics are valuable future directions. We included Multi-turn because dialogue is essential in practical RAG applications. While MTRAG and RAD-Bench focus specifically on dialogue, SPECTRA evaluates it alongside five other scenarios under the same protocol. This allows researchers to study dialogue-related degradation in the context of broader RAG capabilities, while leaving detailed causal analysis to subsequent studies.
  3. To strengthen cross-domain validation, we extended the comparison of all seven RAG configurations to two additional domains, Nanozymes and Oxazolidinone Antibiotics. All three domains were evaluated on their full question sets with two seeds (Table 2). The results show a consistent advantage of hybrid over sparse retrieval, while the leading configuration varies by domain. Seed-to-seed variability is small (Answer Accuracy SEM ≤ 0.02), suggesting limited execution variability. However, we acknowledge that two seeds do not capture question-sampling uncertainty or establish the significance of small differences. SPECTRA does not aim to recommend universally optimal configurations. Instead, it provides a methodology for evaluating systems on representative questions from a target application. We will distinguish cross-domain findings from Chelate-specific results and avoid overstating small ranking differences.
  4. We appreciate the reviewer's recognition of our validation efforts and agree that reliability should be examined by question type. We therefore conducted an additional analysis of the same 200 expert-rated examples, separating agreement by scenario and modification (Table 9, Appendix D). Gemini achieves agreement comparable to human experts on most categories, including challenging Conditional, Multi-hop, Multi-turn, and Constrained questions. Agreement is weaker on Comparison and especially Aggregation. However, on Aggregation, the dedicated D. Incomplete metric achieves expert agreement comparable to inter-expert agreement (F1 = 0.91/0.84 vs. 0.87), demonstrating the value of separate failure diagnostics. Our five-level Answer Accuracy rubric already distinguishes partial from complete answers and penalizes missing or incorrect facts (Appendix L.3.1). Evidence grounding is evaluated separately, while Negative questions are judged against expert-validated references. We acknowledge that the validation subset does not cover unanswerable questions, limiting conclusions about abstention reliability. Finally, independent Jev re-scoring largely preserves aggregate configuration rankings across three domains (Table 2), although fine-grained rankings may be less stable.
  5. We agree that our conclusions should be limited to the evaluated corpus and configurations. In particular, the advantage of standalone LLMs on Aggregation is pipeline-dependent, not a general limitation of RAG. Our RAGU comparison (Appendix I, Table 19) already demonstrates that another RAG architecture can substantially improve Aggregation performance. We will make this distinction clearer throughout the paper and explicitly acknowledge that results may differ with other architectures, larger corpora, and different question distributions.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the expanded cross-domain evaluation and scenario-specific judge validation. The new results substantially strengthen the empirical evaluation, and I consider concerns 3 and 4 weakness points largely addressed.

However, my main concern regarding the incremental practical utility of SPECTRA, the first weakness point, remains unresolved.

The reported scenario-specific failure patterns demonstrate the value of fine-grained diagnostics, but they do not directly establish whether SPECTRA-informed configuration selection improves upon simpler alternatives, such as aggregate-accuracy-based selection. The reference to a separate NeurIPS-accepted study is encouraging, but without accessible experimental details, it is difficult to assess whether this concern has been empirically addressed.

I therefore encourage the authors to distinguish more explicitly between demonstrating diagnostic richness and demonstrating improved configuration-selection decisions.

Overall, I recognize that the rebuttal has meaningfully strengthened the paper, although the evidence for its central claim of actionable configuration selection remains limited.

Thank you for your follow-up and for acknowledging that concerns 3 and 4 are largely addressed.

We agree that diagnostic richness and improved configuration selection are distinct claims, and we will clarify this in the manuscript.

SPECTRA's main contribution is actionable diagnostic evaluation. For example, Hybrid without reranking achieves 0.66 overall Accuracy on Chelate (Table 2), but only 0.49 on Aggregation (Table 3). SPECTRA exposes this hidden weakness, while Appendix F helps characterize it and Appendix I demonstrates how the benchmark can evaluate alternative architectures such as RAGU. Its practical utility is further illustrated by the ML-driven RAG optimization study in Appendix M, which uses SPECTRA data to identify and validate high-performing configurations.

We will clarify that SPECTRA supports informed evaluation and system improvement, without claiming a demonstrated advantage over aggregate-based configuration selection.

Given your acknowledgment of the substantial improvements and our clarification of the remaining concern, we respectfully ask you to reconsider your rating in light of the revised manuscript. We remain available to address any further questions.

Reviewer_GTVSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SPECTRA, a chemistry benchmark with 2,007 questions for evaluating Retrieval-Augmented Generation (RAG) systems. The authors argue that aggregate accuracy hides specific capability differences, making scenario-level evaluation necessary. While the motivation is solid and the dataset took obvious effort to build, I have major concerns about how the benchmark was constructed, its statistical validity, and the narrow experiments. In its current form, the paper does not meet ICLR standards.

Critical strengths
  1. Important Problem: Breaking down RAG evaluation by specific scenarios (retrieval, multimodal, multi-turn) is a great approach.
  2. Good Provenance & Review: The dataset includes clear source DOIs and evidence. The two-stage expert review for Scenario 1 shows real effort (37.3% of questions replaced, 11.3% rephrased).
  3. Transparent Implementation: The paper clearly documents its pipeline (chunking, retrieval, prompts) and openly discusses limitations like the small corpus and single-seed tests.
Critical weaknesses
  1. Unjustified Size & Distribution: The 2,007 questions are unevenly distributed across scenarios without a clear reason or stopping criterion.
  2. Missing Corpus Details: The paper does not provide a reproducible way for how the 224 source articles and 109 distractors were selected.
  3. No Deduplication or Contamination Check: There is no analysis for question duplicates or test-set leakage. For a dataset drafted by an LLM, this is a major omission.
  4. Limited Quality Audits: Detailed expert-correction stats are only provided for Scenario 1. More complex scenarios lack these audits, which is concerning since the authors admit a labeling error slipped past their experts.
  5. Confounded Taxonomy: The benchmark claims to isolate specific RAG capabilities, but fails to do so. Modifications are unevenly distributed (e.g., 53.8% of multi-hop questions are constrained vs. 25.1% of single-hop). You cannot tell if a performance drop is due to the scenario or the modification.
  6. Narrow Experiments: The seven tested RAG configurations are too similar (same embedder, backbone, and prompt). Standalone LLMs (like GPT-5.4) actually beat all RAG models on Aggregation (S5) queries, exposing the limits of testing just one pipeline. Supplementary tests in the appendix are too noisy (only 50 questions, one seed), and the "Parametric Answerability" test fails to prove whether a document was in the LLM's pretraining data.
  7. Weak Statistics: Ranking models based on just two random seeds is unreliable. The authors' own bootstrap analysis shows overlapping confidence intervals for the top systems, yet they still declare clear winners.
  8. Weak "Expert" Validation: The "independent experts" were actually junior (first- and second-year) PhD students. There is no documented calibration or inter-annotator agreement. Furthermore, these two reviewers disagreed heavily on the "Incomplete" metric (flagging 54% vs 41.2% of responses), making it an unreliable signal.
  9. Metric Circularity: The Source Support Max Similarity metric is calculated using JinaEmbV3 - the exact same embedding model used by the dense retriever being evaluated. This heavily biases the results in favor of the dense retriever.
  10. Limited Novelty: The benchmark mostly combines existing ideas (multi-hop, multimodal, diagnostic metrics) into a chemistry context without introducing fundamentally new methodologies.
Questions for authors

Please refer to weaknesses

4 comments

We strongly disagree with the overall assessment, as several criticisms appear to reflect a misunderstanding of the paper's primary contribution.

First, SPECTRA is a benchmark for systematic, fine-grained RAG evaluation, rather than a study proposing or ranking specific RAG architectures. Appendix A (Table 5) explicitly compares SPECTRA with existing benchmarks. Its contribution is the integration of six reasoning scenarios, three orthogonal modifications, expert-validated questions with gold DOI and evidence annotations, and a modular evaluation framework combining retrieval, generation, grounding, and diagnostic metrics. Together, these elements support workload-specific configuration selection and failure analysis beyond aggregate accuracy.

Second, the experiments are intended to demonstrate the capabilities and research opportunities enabled by SPECTRA, not to establish universally optimal RAG configurations. Beyond the presented comparisons, Section 5.1 identifies three research directions supported by the benchmark: data-driven RAG configuration design, session-aware evaluation metrics, and dynamic benchmark generation. These possibilities constitute an important part of the paper's contribution.

Third, SPECTRA has already demonstrated its practical research value. A separate study, accepted at NeurIPS 2026, uses to identify high-performing RAG configurations for previously unseen use cases. To preserve double-blind anonymity, we do not provide the corresponding citation during review, but can include it in the camera-ready version. This provides concrete evidence that SPECTRA enables research beyond the experiments presented in this paper.

Below, we address each criticism individually, including points where the manuscript may benefit from clarification. However, we maintain that these concerns do not undermine the central contributions described above. We respectfully but strongly disagree with the rejection recommendation and ask the reviewer to reconsider the evaluation in light of the paper's actual objectives, demonstrated contributions, and our detailed responses. We remain open for further clarifications.

  1. The distribution was intentional: per domain, we targeted 80 Single-hop questions, 20 per other single-turn scenario, and 20 Multi-turn dialogues of 4–5 turns. Single-hop questions supported the construction of more complex ones; Multi-turn counts represent individual turns. Expert-driven additions and removals explain deviations (Appendix B.2, Table 6).
  2. The 224 open-access source papers were expert-curated within the researchers' specialties to enable rigorous validation of challenging, evidence-grounded questions rather than exhaustive chemistry coverage. The 109 topically related distractors test source discrimination. Our focus is on retrieving and using scientific evidence, not merely recalling parametric knowledge.
  3. As described in Section 3.4, LLMs were used only to draft candidate questions, which subsequently underwent two-stage expert validation. Evaluated RAG systems were not trained on SPECTRA, so benchmark-specific training–test leakage is not applicable. We audited duplicates using normalized exact matching and character 5-gram Jaccard similarity (>0.8), with manual verification (Appendix C). Five repeated opening turns belong to different Multi-turn trajectories and were retained. Five semantically equivalent question pairs were also identified; confirmed redundancies will be removed, with affected tables updated in the camera-ready version.
  4. Draft-to-final correction statistics apply only to Scenario 1, where questions were primarily LLM-drafted. More complex scenarios were predominantly expert-developed with limited LLM assistance. All questions underwent two-stage validation for correctness, evidence sufficiency, answerability, and scenario compliance (Section 3.4).
  5. SPECTRA is designed to compare RAG configurations under different question conditions, rather than establish causal effects of scenarios or modifications themselves. All configurations are evaluated on the same questions, allowing direct comparisons within each scenario or modification. The benchmark also supports more granular analysis by fixing a scenario and examining performance across modifications. While differences in modification distributions can affect comparisons between scenario averages, they do not invalidate configuration-level comparisons.
  6. The primary contribution of this work is a benchmark for systematic RAG evaluation, not an exhaustive comparison of RAG architectures. The experiments are intended to demonstrate its capabilities. Nevertheless, we substantially expanded the evaluation: the updated Table 2 compares all seven configurations on the complete question sets of three domains, using two seeds. Appendix I additionally examines retrieval parameters, generation backbones, and an alternative RAG architecture (RAGU), illustrating the pipeline dependence of Aggregation results. Finally, our parametric answerability analysis is not intended to establish whether specific documents appeared in LLM pretraining data, but to assess which questions can be answered without retrieval.
  7. As noted in Point 6, the experiments demonstrate SPECTRA's evaluation capabilities rather than establish statistically definitive winners among RAG configurations.
  8. Our experts actively research the relevant subdomains and have published directly related work in Q1 journals or A*-ranked venues, including Nature Portfolio and Nature Index journals. PhD seniority is therefore an inadequate proxy for their expertise. Disagreement on D. Incomplete reflects the inherent subjectivity of completeness judgments, not demonstrated scientific incompetence. We explicitly report inter-expert agreement to quantify metric reliability and acknowledge that diagnostic metrics differ in reliability.
  9. Using JinaEmbV3 for both retrieval and Source Support Max Similarity may introduce evaluator–retriever bias. This affects an auxiliary metric, while Answer Accuracy and DOI Recall provide independent evidence. SPECTRA's modular framework also supports alternative evaluators and metrics. This limitation is now acknowledged in Appendix E.4.
  10. SPECTRA's novelty lies in integrating six scenarios, three modifications, expert-validated scientific questions, gold DOI/evidence annotations, and granular diagnostics into one benchmark. Evaluating configurations under different question conditions within the same domain reduces domain-related confounding and supports evidence-grounded assessment beyond parametric recall. Appendix A (Table 5) details the distinctions from existing benchmarks.

Thank you for the detailed rebuttal and clarifications. I appreciate the authors' efforts to address my concerns, particularly regarding the benchmark's objectives, question distribution, deduplication process, expert validation, and expanded experimental evaluation.

I agree that SPECTRA should primarily be evaluated as a benchmark rather than as a proposal for a new RAG architecture. The authors' explanations have helped clarify its contributions and addressed several concerns raised in my original review.

However, important concerns remain regarding the representativeness of the source corpus, reliability of some evaluation metrics, limited statistical evidence, and potential bias introduced by using the same embedding model for retrieval and evaluation.

Overall, I recognize the potential value of the benchmark and appreciate the additional clarifications. I will increase my rating from Clear Reject to Weak Reject, as the rebuttal has improved my assessment of the work. However, the remaining methodological concerns prevent me from recommending acceptance at this stage.

Thank you for reconsidering your rating and acknowledging the improvements to the manuscript. We would like to address the four remaining concerns.

  1. Corpus representativeness. SPECTRA is intentionally expert-curated rather than a random sample of chemistry literature. The 224 source articles span eight specialized domains and enable scientifically meaningful, evidence-grounded questions, while 109 related distractors test retrieval discrimination. Importantly, domain-scoped production RAG systems usually operate over collections of hundreds of documents, making our corpus size a realistic evaluation setting. We believe representativeness should be assessed relative to the intended scientific QA tasks, rather than the entire scientific literature. Could you clarify which specific aspect of corpus representativeness remains concerning?
  2. Metric reliability. We have expanded Appendix D with category-level human validation (Tables 8–9). Overall, Gemini's agreement with two experts (QWK = 0.79/0.76) approaches inter-expert agreement (0.81). For Aggregation, the dedicated D. Incomplete metric achieves F1 = 0.91/0.84 against experts, compared with 0.87 inter-expert agreement. We also explicitly report categories where reliability is weaker rather than assuming all metrics are equally reliable.
  3. Statistical evidence. Table 2 now evaluates all seven configurations across three complete domains using two seeds. We acknowledge that small differences and overlapping confidence intervals do not establish statistically significant winners. However, SPECTRA is an evaluation benchmark, and its validity rests primarily on expert-verified questions and evidence, reliable evaluation, a reproducible protocol, and clearly stated limitations. The configuration experiments demonstrate its evaluation capabilities rather than establish definitive rankings. More extensive statistical validation would strengthen the work, but need not be exhaustive in the initial release of the benchmark.
  4. Evaluator-retriever bias. We agree that sharing JinaEmbV3 between retrieval and Source Support Max Similarity may introduce bias. We now explicitly acknowledge this limitation in Appendix E.4. Source Support Max Similarity is an auxiliary diagnostic metric; Answer Accuracy and gold DOI Recall are evaluated independently of that embedding model. Validation with an alternative embedding model would be a valuable further robustness check.

We appreciate that you have already reconsidered your rating. Given the substantial revisions, additional validation, and clarification of the benchmark's intended scope, we believe the remaining limitations are appropriately acknowledged and do not undermine SPECTRA's central contribution. We therefore respectfully ask you to consider raising your rating further in light of the revised manuscript and our responses. We remain committed to address any outstanding questions and implement your suggestions in the next revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.