acceptodds
Under review as a conference paper at ICLR 2027

STELLAR: Stage-Separated Testbed for Error Localization and Attribution in Retrieval

Abstract

Enterprise RAG pipelines are typically two-stage – an embedding retriever followed by one or more rerankers – yet almost universally evaluated with one end-to-end score that cannot say whether a deficiency originates in retrieval or reranking. We introduce STELLAR, a static benchmark built from 25,178 ServiceNow knowledge-base articles and 8,870 LLM-generated queries with mined, verified hard negatives, which scores retrieval and reranking independently – whole-corpus and fixed-pool modes, plus end-to-end – so a deficient score attributes to a specific stage. Every query also carries diagnostic metadata (lexical overlap, distractor hardness, length, product area, content type, LLM-assigned intent; test-retest ), disaggregating failure by cause, not just count. Across 18 model configurations, task formulation predicts quality more reliably than scale: a 1B cross-encoder significantly beats a 32B listwise LLM by 7.5 MRR points (), a 0.6B pointwise reranker matches or exceeds that same 32B model used listwise (directionally consistent but not itself significant at ), two same-family scaling curves are non-monotonic past moderate size, and cross-encoders dominate the quality-latency Pareto frontier outright. A diagnostic ablation surfaces an actionable failure mode: every reranker loses 10-14 MRR points on long source documents.

Primary area:
datasets and benchmarks
Keywords:
retrieval-augmented generation, reranking, information retrieval evaluation, benchmark, embedding retrieval, cross-encoder, listwise reranking, enterprise search, error attribution, RAG evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_uubxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

STELLAR is an evaluation testbed for multi-stage retrieval pipelines with a retriever (L1), a reranker (L2), and a final ranking stage (L3). The main idea is to evaluate downstream ranking on a fixed per-query pool containing one gold document and up to 20 mined hard negatives, independently of retrieval, and then run the connected cascade to track where the designated gold document is lost. The authors instantiate the testbed on 25,178 internal ServiceNow articles with 8,870 query–gold records constructed using GPT-OSS-120B, evaluate 7 embedding models and 12 reranker configurations, and attach per-query diagnostics covering lexical overlap, distractor hardness, length, product area, content type, and intent. Main findings include a 1B cross-encoder outperforming a 32B listwise LLM by 7.5 MRR points, consistent degradation on long gold documents among the seven diagnostically analyzed rerankers, model-dependent sensitivity to low lexical overlap, and a reversal of the BGE/Qwen ordering on public TechQA. The contribution is the protocol, construction pipeline, and empirical study; there is no new retrieval or reranking method.

Critical strengths
  1. The three-mode design (whole-corpus, fixed pool, end-to-end) gives a practical way to distinguish retrieval failures from downstream ranking failures. Section 4.3 shows this concretely: of 8,870 records, 372 designated golds are lost at L1 (Recall@100 = 0.958), and a further 373 are lost when the reranker truncates to its top-21. Table 3 also holds the query subset and candidate width fixed while changing pool composition. This accounting is what a practitioner needs, and it is the paper's main strength.

  2. The reporting gives enough detail to audit the measurements. Appendix H.3 describes source-document-clustered bootstrapping and Holm correction, Appendix G.1 reproduces the included construction prompts verbatim, Table 7 states the artifact release boundary, and the note on A8 corrects an earlier duplicate-query count. The TechQA replication is also reported despite reversing the internal model ordering.

Critical weaknesses

1. The main ServiceNow instance cannot be independently rerun or used to evaluate new models without corpus access, and its validity remains insufficiently established.

The corpus is withheld (Appendix D, Table 7). The released artifacts (query and document identifiers, fixed pools, per-query ranks, code, prompts) support recomputation of reported aggregates, but they do not permit independent model scoring on the ServiceNow instance. Appendix D acknowledges this distinction.

The full query-generation and validation pipeline is demonstrated only on the private corpus. The TechQA build (Section 3.2, "Public replication") uses real questions and embedding-mined negatives, with one embedder and two rerankers. It validates the evaluation protocol under a different pool construction, rather than independently demonstrating the full synthetic construction pipeline.

On the internal build, the same model generates queries and participates in positive and negative validation (Sections 3.1 and 6). The only independent human check is one author auditing 200 of 8,870 query-gold pairs (2.3%) and an unspecified subset of negatives (Appendix C). The 98% gold correctness is a good sign, but this remains a limited single-annotator audit.

Queries have median lexical overlap of 0.80 with their gold documents (Section 3.4); 67.6% have overlap above 0.7, while only 120 records (1.4%) fall below 0.3 (Table 15). So the low-overlap comparison of 0.791 versus 0.594 MRR rests on 120 records and has no accompanying confidence interval or significance test. The generator uses 25 real queries as style exemplars (Appendix G), but the paper does not compare model ordering on real versus synthetic queries from the same knowledge base.

The design also builds on established practices; reranking evaluation is already represented in MTEB, synthetic query generation in InPars, and automated construction of retrieval evaluation data in AIR-Bench. AIR-Bench is not discussed. The paper's value then rests on the usability and validation of its construction pipeline and empirical findings. For a datasets and benchmarks submission, this is my decisive concern.

2. The attribution evidence is weaker than the paper's framing suggests.

(a) L3 is not independently isolated from L2. The authors say this on page 3, lines 109-110. The attribution paragraph in Section 3.3 distinguishes retrieval failures from failures in the downstream L2 and L3 pathway. The "direct on 21" rows in Table 4 evaluate individual rankers on fixed pools; they do not characterize L3 conditional on a controlled L2 output. The connected cascade does provide transition-level gold-survival counts, but this is narrower than independently evaluating all three components.

(b) The L3 gains lack a matched control for informative input ordering. Table 4 reports an improvement from 0.837 MRR for BGE's top-10 to 0.886 after Qwen3-32B, whereas direct Qwen on the fixed pools scores 0.789. This comparison changes both candidate selection and input ordering.

Figure 6 offers a relevant additional observation: at K = 21, the cascade reaches approximately 0.89 MRR on a 1,500-query subset, compared with 0.789 for direct Qwen on all queries in Table 4. At this width, the cascade should retain the full fixed pool, making input order a plausible explanation. The query populations differ, but bge alone scores 0.847 on that subset against 0.839 on all queries (Figure 6, Table 4), so the subset difference is unlikely to explain a 10-point gap. Appendix H.5 reports robustness to reshuffling fixed pools, but does not report a matched comparison between BGE-ranked and randomly ordered inputs to L3. Without that control, the contribution of informative ordering cannot be separated from Qwen's reranking contribution.

(c) Multi-positive augmentation creates a potentially consequential relevance-scoring issue. Appendix G states that augmentation writes additional records with identical query text and different gold documents. Section 3.3 defines relevance using only the record's designated gold; Appendix E acknowledges incomplete relevance judgments. A8 reports that 4,896 records share their query text.

Under deterministic retrieval for identical query text, and assuming each such query's records designate distinct gold documents, whole-corpus Recall@1 is bounded above by 6,013 / 8,870 ≈ 0.678. A known relevant sibling document can then outrank the designated gold without receiving relevance credit. Fixed-pool filtering is intended to remove answering negatives, but this does not resolve the issue when scoring against the full corpus or L1-derived candidates.

Some of the Table 3 drop from 0.854 to 0.638, and some of the 373 designated-gold losses at L2, could reflect sibling positives rather than incorrect answers outranking correct ones. A8 evaluates duplicate-query slices on fixed pools and does not quantify this effect in whole-corpus or end-to-end evaluation. The magnitude remains unverified; Q1 asks for the decisive check.

Smaller points that should be fixed

  • Section 4.6, the conclusion, and Table 14 describe a universal 10–14 MRR-point long-document drop. Table 18 instead gives short-to-long drops ranging from 9.7 to 19.1 points. The paper's own BGE-v2-m3 example, 0.888 → 0.791, is a 9.7-point drop.
  • Appendix H.3 specifies 12 planned comparisons, but Table 10 lists nine. The correction status of the three A2 p-values reported afterwards (0.044, 0.009, 0.094) is unclear. The confidence intervals reported alongside them imply the same values, which suggests they are unadjusted. If so, Holm correction within those three tests gives 0.088, 0.027, and 0.094, respectively, leaving one significant result rather than two. Please specify the comparison family and adjustment status.
  • Table 6 defines K1 > K2 > K3, whereas Section 3.3 uses K1 ≥ K2 ≥ K3 and the main cascade uses 100 → 21 → 21.
  • Table 2 bolds Recall@10 = 0.980 for Qwen3-32B pointwise, although 0.983 is the column maximum.
  • Appendix H.6 refers to a "4–7×" latency figure "quoted in the main text," but that figure does not appear in the main text.
  • The Ramanath et al. (2021) bibliography entry appears mismatched. Its author list matches "Towards Deep and Representation Learning for Talent Search at LinkedIn", published at CIKM 2018, rather than the listed title and KDD 2021 venue. Please verify the intended reference.
  • The 7.5-point headline compares Nemotron with the no-CoT listwise Qwen3-32B configuration. The gap is 6.4 points against listwise Qwen with CoT and 4.7 points against the same Qwen checkpoint used pointwise (Table 2). This configuration dependence should accompany the headline.
  • The abstract on OpenReview and the abstract in the PDF disagree (18 versus 19 model configurations; two non-monotonic scaling curves versus one).
Questions for authors

Clarification questions whose answers would change my assessment:

  1. How many records were added by multi-positive augmentation? In whole-corpus and end-to-end evaluation, are sibling golds with identical query text scored as non-relevant? Please recompute Table 9, Table 3, and the Section 4.3 loss counts with all known golds for each exact query text counted as relevant, aggregating results over distinct queries.
  2. On exactly the same query subset and candidate sets, how does Qwen3-32B listwise perform when the full fixed pool is supplied in BGE order versus random order? Please also provide this ordering control for BGE's top-10 candidates. These comparisons would separate candidate-selection effects from input-order effects.
  3. Are all evaluated checkpoints public releases with no fine-tuning on this corpus or data produced by the construction pipeline? Several prompts in Appendix G.1 describe a "reranker training dataset"; please state which uses of the pipeline output are downstream training and which are evaluation.
  4. For listwise Qwen3-32B and Qwen3-8B, how are pools of up to 21 documents represented within one prompt when documents exceed the available context? Are documents truncated, snippeted, or processed using a windowed listwise procedure? Section 3.3's maximum-chunk-score rule does not fully specify listwise handling. This affects interpretation of both the headline comparison and the long-document finding.
  5. Can the full GPT-OSS construction pipeline (query generation, mining, validation gates) be run on TechQA or another public corpus, with the resulting pools released?
  6. Do you have real queries with gold articles for this ServiceNow knowledge base, and does model ordering on those queries agree with Table 2?
  7. Which 12 comparisons were planned in Appendix H.3, and are the A2 p-values Holm-adjusted?

Suggestion, not required for my assessment: BM25 and hybrid baselines in Table 9 would help interpret lexical-overlap effects in whole-corpus retrieval, given the use of BM25 during negative mining.

Reviewer_ejuYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces STELLAR, a stage-separated testbed for evaluating enterprise RAG systems. It separately evaluates L1 whole-corpus retrieval and the L2/L3 ranking pathway using fixed candidate pools, while tracking gold-document survival across the end-to-end cascade to support error localization and attribution.

I acknowledge the practical value of this work. However, I have significant concerns regarding its methodological novelty and whether the proposed framework adequately supports its central claims. The current evidence does not convincingly demonstrate capabilities beyond established evaluation practices.

Critical strengths

The paper addresses a practically relevant problem and provides useful insights for model selection and optimization in enterprise RAG systems. The authors systematically integrate whole-corpus retrieval, fixed-pool reranking, end-to-end evaluation, and diagnostic analyses, supported by extensive experimental results.

Critical weaknesses

W1. Limited methodological novelty compared with existing evaluation practices.

Independent component evaluation, controlled candidate-pool comparisons, stage-transition logging, and diagnostic slicing or ablation studies are already common practices in IR and RAG research and engineering. While integrating these techniques into a unified testbed has practical value, the paper does not sufficiently establish its methodological novelty or demonstrate what additional diagnostic capabilities STELLAR provides beyond conventional evaluation and logging approaches.

W2. Insufficient connection between stage-separated evaluation and end-to-end error attribution.

The evaluation protocol primarily separates L1 from the combined L2/L3 ranking pathway, using fixed candidate pools constructed independently of L1 retrieval. Although this design isolates downstream ranking from retrieval failures, the candidate distribution differs from actual L1 outputs. Table 3 also demonstrates that candidate-pool composition substantially affects ranking performance.

While the authors conduct end-to-end evaluation, the results primarily demonstrate where gold documents are lost, rather than establishing how isolated evaluation supports deeper attribution of failure causes. Furthermore, the framework does not fully isolate L3 from L2, leaving a gap between the evaluation protocol and its claimed three-stage error attribution capability.

W3. Experimental results do not sufficiently demonstrate the distinctive value of the proposed testbed.

Many experiments primarily compare the performance of different embedding models, rerankers, and LLMs, or analyze their sensitivity to document length, lexical overlap, and other factors. Although these findings are informative, they do not clearly establish the additional diagnostic value of STELLAR over standard component-level evaluation, stage-transition logging, or ablation studies. A framework centered on error attribution should provide more direct evidence of its distinctive diagnostic capabilities.

W4. Limitations in dataset reliability and reproducibility.

GPT-OSS-120B is involved in multiple stages of dataset construction, including query generation, quality validation, and hard-negative mining. The lack of independence between generation and validation may introduce systematic biases. The limited human audit does not fully establish the representativeness of synthetic queries or the reliability of relevance judgments.

Moreover, the proprietary ServiceNow corpus is not publicly available, preventing independent reproduction of the main enterprise-domain experiments. Although the authors provide construction code, evaluation artifacts, and a public TechQA replication, the evidence remains limited for a paper whose central contribution is a benchmark and evaluation framework.

Questions for authors

I would particularly appreciate clarification on the following two questions:

Q1. What is the substantive novelty of STELLAR compared with existing IR/RAG evaluation practices?

Compared with standard component-level evaluation, fixed-pool reranking, stage-transition logging, and ablation studies, what new evaluation methodology or error attribution capability does STELLAR introduce? Is there concrete evidence that it can provide diagnostic insights that conventional approaches cannot?

Q2. How does stage-separated evaluation support error attribution in the actual end-to-end RAG cascade?

(1) Given the substantial differences between fixed candidate pools and actual retrieval outputs, how can isolated evaluation results reliably explain failures in the end-to-end cascade? Does STELLAR's error attribution capability extend beyond identifying the stage where a gold document is lost to explaining the underlying causes of failure?

(2) Since the current evaluation does not fully isolate L3 from L2, how do the authors define and validate the claimed three-stage error attribution capability?

Reviewer_nxsAOpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents STELLAR, a stage-separated testbed for error localization and attribution in retrieval. The central idea is to break one end-to-end overall score into stage evaluations. In general, stage-separated evaluation is not a new idea.

Critical strengths

A systematic work on dividing a pipeline into 3 phases and propose evaluation criteria.

Critical weaknesses

There is no systematic discussion on the validity of the proposed benchmark.

There are many types of retrieval. The paper does not clearly specify the tasks and the assumptions.

The paper does not discuss the implications, limitations, and potential applications of the proposed benchmark.

Questions for authors

Add a clear problem statement

Discuss validity of your benchmarks concretely if you can.

1 comment

I wanted to thank the authors for their efforts. The revised version partially addresses my concerns. As I have low confidence on the topic, I would like to follow the other reviewer and the senior PC's judgements.

Comments on the paper

1 comment
Area_Chair_1oPuOpenReview ↗

Reminder

Dear reviewers,

Please review the authors’ responses carefully and discuss with the authors if you have more concerns or questions. If their responses have changed your assessment, please consider revising your scores accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.