- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper asks whether RAG techniques that help when tested alone still help when combined in a full pipeline. It introduces LASAGNA, an open-source framework that splits the pipeline into separate stages. Each stage sits behind a fixed interface, and a conformance test uses hashes to check that changing one stage does not affect the others. The authors pre-declare three hypotheses: components do not combine additively, one composition can serve several deployment targets, and the ranking of compositions changes with corpus size. They test these with a D-optimal factorial over eight pipeline stages: 242 runs were planned and 167 executed. They add 138 targeted ablation configurations and a scaling sweep from 5K to 200K documents. All benchmarks are merged into one multi-domain corpus. Retrieval is scored with NDCG@10 on 93 queries, and generation with RAGAS faithfulness judged by GPT-5.1 on 56 queries. The main claims are as follows. Retrieval gains combine sub-additively: three single-component gains predict +47% NDCG@10, but the combined pipeline realizes +17%. Faithfulness gains combine roughly additively. Retrieval quality and faithfulness depend on different stages and are uncorrelated across configurations. Four released profiles form a Pareto trade-off. STANDARD leads at 5K–15K documents, while BALANCED leads from 25K and is 13.7% ahead at 200K.
Critical strengths
-
The fixed per-stage interfaces, decorator-based registration, YAML sweeps and hash-based isolation test address a real attribution problem in existing toolkits such as FlashRAG and BERGEN. LASAGNA could lower the cost of the evaluations the paper recommends.
-
Most RAG techniques are tested one stage at a time against weak baselines. Asking whether those gains survive composition is a useful methodological challenge. The call to test new components inside full pipelines, on both retrieval and faithfulness, and at the intended scale is a great recommendation.
-
Embedding, enrichment and reranking drive NDCG@10, while query augmentation and agentic gating drive faithfulness. Together with the near-zero correlation between the two metrics across configurations, this is a concrete warning against choosing pipelines by retrieval metrics alone.
Critical weaknesses
-
That gains do not stack, that retrieval metrics poorly predict downstream quality, and that rankings depend on scale have each been noted in prior RAG benchmarking work. The contribution is mainly the framework and the systematic sweep. The biggest concern for this paper is that it has 101 pages, but the main text depends heavily on appendix pointers. It is hard to read. Numbers are not always consistent: BALANCED scores 0.483 in Fig. 9, 0.504 in Table 48 and 0.840 in Table 2. “Eight benchmarks” names only seven datasets, with HotpotQA counted on both tracks. These inconsistencies make the claims hard to check.
-
Under the pre-registration’s own decision rule, H1 is rejected: the interaction η² is 0.033–0.040, below the 5% threshold at which “additivity holds”. H2 was not run as pre-registered. Only H3 is reported as supported. The abstract and introduction nevertheless lead with “components do not compose”. The main-text H2 has also been reworded from the pre-registered claim. Calling the hypotheses “pre-declared” while reframing them after the results are known weakens the main methodological conclusion.
-
The predicted gain simply adds absolute NDCG gains on a bounded metric, so sub-additivity is expected from diminishing returns and redundant mechanisms alone. One of the “techniques” is disabling MMR, a harmful default in the baseline, which inflates the sum of parts. The realized value is the BALANCED profile, which also changes chunking, compression and merging. It is therefore not the baseline plus exactly the summed components; the authors acknowledge this kind of mismatch for faithfulness but not for retrieval. The comparison also mixes three predicted techniques with a five-technique realized pipeline. A matched comparison would build the exact stack and test it against an additive or multiplicative null.
-
The pre-registered budget was 2,300 queries per configuration. The executed budget is 93 retrieval queries and 56 generation queries, or 43 per seed for the profiles. Effects are estimated from an unbalanced design with 69% of runs executed and one enrichment level dropped after the fact. Most per-component conclusions, such as the MiniLM 2019 reranker beating newer ones or MMR hurting, could easily be dataset-specific on this sample.
Questions for authors
-
Can you report a matched composition test for retrieval? That means the exact stack (baseline + reranker + doc-awareness + MMR off) evaluated directly and compared with both an additive and a multiplicative/ceiling-adjusted null, with query-level bootstrap CIs.
-
How much of the faithfulness gap in Table 2 remains when all profiles use the same answer prompt? Please also report EM/F1 or answer correctness for each profile.