acceptodds
Withdrawn from ICLR 2027

Retrieval Augmented Generation Compositionality

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Retrieval Augmented Generation (RAG) grounds large language models in external knowledge. Modern pipelines combine embedding, chunking, retrieval, reranking, query augmentation, context compression, and agentic verification, yet these techniques are typically evaluated one at a time. It remains unclear whether their isolated gains survive composition, whether retrieval quality and answer faithfulness favor the same design choices, and whether conclusions from small corpora hold at large scale. We pre-declare three hypotheses and introduce Lasagna, an open-source framework for controlled RAG evaluation. We test configurations across eight benchmarks and corpora ranging from K to K documents. Three findings emerge. First, individually strong techniques from different phases of a RAG pipeline do not necessarily compose: three retrieval techniques predict a +47% combined improvement but deliver only +17% when used together. Second, component gains stack differently for retrieval and faithfulness, and optimizing one can hurt the other. Third, the best composition changes as the corpus grows: the combination that wins on a small corpus may not remain best at large scale. These findings show that gains measured in isolation are not reliable predictors of realistic performance, and that new RAG techniques should be tested with the full surrounding stack. We call on researchers to do so and supply Lasagna to simplify these experiments.

Primary area:
foundation or frontier models, including LLMs
Keywords:
retrieval-augmented generation, RAG evaluation, compositionality, information retrieval, benchmarking, agentic RAG, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Hr3QOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks whether RAG techniques that help when tested alone still help when combined in a full pipeline. It introduces LASAGNA, an open-source framework that splits the pipeline into separate stages. Each stage sits behind a fixed interface, and a conformance test uses hashes to check that changing one stage does not affect the others. The authors pre-declare three hypotheses: components do not combine additively, one composition can serve several deployment targets, and the ranking of compositions changes with corpus size. They test these with a D-optimal factorial over eight pipeline stages: 242 runs were planned and 167 executed. They add 138 targeted ablation configurations and a scaling sweep from 5K to 200K documents. All benchmarks are merged into one multi-domain corpus. Retrieval is scored with NDCG@10 on 93 queries, and generation with RAGAS faithfulness judged by GPT-5.1 on 56 queries. The main claims are as follows. Retrieval gains combine sub-additively: three single-component gains predict +47% NDCG@10, but the combined pipeline realizes +17%. Faithfulness gains combine roughly additively. Retrieval quality and faithfulness depend on different stages and are uncorrelated across configurations. Four released profiles form a Pareto trade-off. STANDARD leads at 5K–15K documents, while BALANCED leads from 25K and is 13.7% ahead at 200K.

Critical strengths
  1. The fixed per-stage interfaces, decorator-based registration, YAML sweeps and hash-based isolation test address a real attribution problem in existing toolkits such as FlashRAG and BERGEN. LASAGNA could lower the cost of the evaluations the paper recommends.

  2. Most RAG techniques are tested one stage at a time against weak baselines. Asking whether those gains survive composition is a useful methodological challenge. The call to test new components inside full pipelines, on both retrieval and faithfulness, and at the intended scale is a great recommendation.

  3. Embedding, enrichment and reranking drive NDCG@10, while query augmentation and agentic gating drive faithfulness. Together with the near-zero correlation between the two metrics across configurations, this is a concrete warning against choosing pipelines by retrieval metrics alone.

Critical weaknesses
  1. That gains do not stack, that retrieval metrics poorly predict downstream quality, and that rankings depend on scale have each been noted in prior RAG benchmarking work. The contribution is mainly the framework and the systematic sweep. The biggest concern for this paper is that it has 101 pages, but the main text depends heavily on appendix pointers. It is hard to read. Numbers are not always consistent: BALANCED scores 0.483 in Fig. 9, 0.504 in Table 48 and 0.840 in Table 2. “Eight benchmarks” names only seven datasets, with HotpotQA counted on both tracks. These inconsistencies make the claims hard to check.

  2. Under the pre-registration’s own decision rule, H1 is rejected: the interaction η² is 0.033–0.040, below the 5% threshold at which “additivity holds”. H2 was not run as pre-registered. Only H3 is reported as supported. The abstract and introduction nevertheless lead with “components do not compose”. The main-text H2 has also been reworded from the pre-registered claim. Calling the hypotheses “pre-declared” while reframing them after the results are known weakens the main methodological conclusion.

  3. The predicted gain simply adds absolute NDCG gains on a bounded metric, so sub-additivity is expected from diminishing returns and redundant mechanisms alone. One of the “techniques” is disabling MMR, a harmful default in the baseline, which inflates the sum of parts. The realized value is the BALANCED profile, which also changes chunking, compression and merging. It is therefore not the baseline plus exactly the summed components; the authors acknowledge this kind of mismatch for faithfulness but not for retrieval. The comparison also mixes three predicted techniques with a five-technique realized pipeline. A matched comparison would build the exact stack and test it against an additive or multiplicative null.

  4. The pre-registered budget was 2,300 queries per configuration. The executed budget is 93 retrieval queries and 56 generation queries, or 43 per seed for the profiles. Effects are estimated from an unbalanced design with 69% of runs executed and one enrichment level dropped after the fact. Most per-component conclusions, such as the MiniLM 2019 reranker beating newer ones or MMR hurting, could easily be dataset-specific on this sample.

Questions for authors
  1. Can you report a matched composition test for retrieval? That means the exact stack (baseline + reranker + doc-awareness + MMR off) evaluated directly and compared with both an additive and a multiplicative/ceiling-adjusted null, with query-level bootstrap CIs.

  2. How much of the faithfulness gap in Table 2 remains when all profiles use the same answer prompt? Please also report EM/F1 or answer correctness for each profile.

Reviewer_vQLEOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper constructs a comprehensive RAG evaluation framework, composed of configurable stages with many concrete implementations. Extensive experiments over a combination of eight benchmarks reveal findings related to RAG compositionality.

Critical strengths

S1. A lot of effort has been done to create a comprehensive multi-stage RAG evaluation framework.

S2. An exhaustive combination of different settings of these stages leads to a very large-scale evaluation.

Critical weaknesses

W1. It is unclear how to position this work.

The comparison in Table 1 lacks explanation and the compared benchmarks are a bit out of date. Multi-stage or modular evaluation of RAG systems is not new. There have been many such papers published at SIGIR or ACL this year. The empirical findings provided in this paper are mostly trivial. For example, it is well-known that retrieval gains may not deliver to answer correctness; it is also common that a stronger generator helps the lightest pipeline more than the heaviest. After reading the paper, I did not learn much new.

W2. Paper is not self-contained and not self-consistent.

I am shocked that many critical information is only included in the appendix. For example, without knowing the column defintions, how can we understand the meaning of Table 1? Without knowing the eight experimental dimensions, how can we understand your experimental design? Lacking such essential information makes the main text hardly readable. It would be unfair to other submissions if such incompleteness is allowed.

The paper also contains may inconsistencies. For example, the overview of your framework contains "seven" functional blocks, but you use it across "eight" configurable stages of RAG, and then your workflow spans "nine" processing stages. I am wondering whether the authors have carefully checked their paper after AI assistance.

Questions for authors

Q1. What is the contribution of your framework compared with recent evaluation efforts published in 2026 (but not those earlier included in Table 1)?

Q2. Which of your findings are new, not reported in the literature so far?

Q3. You unified eight benchmarks. However, a query in one benchmark may have a relevant document in another benchmark. Have you added such labels to the gold standard for evaluation? Otherwise, how do you assess the quality/sufficiency of the gold labels in your combined benchmark?

Reviewer_vzEPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces LASAGNA, a framework for controlled RAG evaluation. The goal is to evaluate weather the isolated gains from different components of RAG pipelines compose when used together. Experiments are done with several configurations over various benchmarks and corpora.

Critical strengths

The paper asks a relevant question: do the gains reported for individual RAG components still hold when they are combined in a full pipeline? Most RAG papers evaluate one component at a time, so this is useful to know. Looking at both retrieval quality and answer faithfulness, and at how conclusions change with corpus size, is the right way to frame it.

Critical weaknesses
  1. This work evaluates existing components (HyDE, CRAG, Self-RAG, RAPTOR, off-the-shelf rerankers and embedding models) with a fixed generator, and wraps them in a framework. The main findings are also largely known. Prior configuration-search work ([1,2,3,4]) already assumes that components interact. [5] showed that retrieval quality and downstream RAG quality can diverge, and [6] studied how retrieval degrades as the index grows. Under the authors’ own pre-registered rule, the measured interaction effect (η² ≈ 0.04) is small enough to count as “approximately additive”. This could be a useful resource or demo paper.
  • [1] Wang et al., 2024, Searching for Best Practices in Retrieval-Augmented Generation.
  • [2] Li et al., 2025, Enhancing Retrieval-Augmented Generation: A Study of Best Practices.
  • [3] Kim et al., 2024, AutoRAG: Automated Framework for Optimization of Retrieval Augmented Generation Pipeline.
  • [4] Fu et al., 2024, AutoRAG-HP: Automatic Online Hyper-parameter Tuning for Retrieval-Augmented Generation.
  • [5] Reimers & Gurevych, The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes, ACL 2021
  • [6] Salemi & Zamani, Evaluating Retrieval Quality in Retrieval-Augmented Generation, SIGIR 2024
  1. The evaluation is too small for the claims made. It uses 93 retrieval queries and 56 generation queries, against a pre-registered plan of 2,300, and the paper still draws conclusions about production-scale behaviour. The 167 executed runs cannot identify all 28 pairwise interactions, yet the interactions are reported anyway. The “composition penalty” compares the summed gains against a pipeline that uses a different reranker and adds several extra components, so the gap does not measure interaction. Faithfulness comes from a single LLM judge that is also the generator, and agreement with other judges is low. The profiles are scored under different answer prompts, and under a common prompt the agentic profiles’ faithfulness advantage disappears (Table 57). Finally, the seed replicates are effectively deterministic (T = 0), and the 5K and 15K points behind the H3 rank flip are single runs.
  2. The paper is 101 pages, and the main text cannot be read on its own. It points to the appendix more than 40 times, and basic definitions live only there: the profiles, the baseline, the hypothesis tests, and how the headline numbers are computed. I had to go back and forth constantly to follow the argument. The appendix itself reads more like an experiment log than supplementary material. It contains revision notes, local file paths and code line numbers, UI screenshots and code listings, and some numbers appear in several places with different values.
  3. The appendices repeatedly cite local script names, source-code line numbers and LaTeX source lines (e.g., experiment.py:54–66, preregistration_body.tex L357–358) as evidence. These are not accessible and do not substitute for describing the method in the paper. Given the code is not released with the submission, the “every table and figure is reproducible from the released manifest” claim in the Reproducibility statement cannot be assessed. Several artifacts are also stated to be unavailable (App. B.3.4, B.3.6, D.3).
  4. The paper states that no OSF pre-registration was filed (Table 4, Reproducibility statement), yet App. D.3 cites “OSF §189” as the location of the pre-registered H1 test. The pre-registration in App. A.2 has no section 189.
  5. The references contain several errors that suggest it was not checked. The authors themselves declare to use AI model to format the references. Some issues are:
  • Chen et al. is listed as appearing at the “Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026”; it appeared at NeurIPS 2025.
  • The Gao et al. RAG survey lists “Haofen Wang” twice among its authors.
  • Box et al. is given the venue “AMC, 10, 01 2005”.
  • Fedorov (1972), Theory of Optimal Experiments, is a book but is cited as a Biometrika article.
  • The FlashRAG entry (Jin et al., 2025) gives its page range as “737740”.
Questions for authors

NA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.