- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
STELLAR is an evaluation testbed for multi-stage retrieval pipelines with a retriever (L1), a reranker (L2), and a final ranking stage (L3). The main idea is to evaluate downstream ranking on a fixed per-query pool containing one gold document and up to 20 mined hard negatives, independently of retrieval, and then run the connected cascade to track where the designated gold document is lost. The authors instantiate the testbed on 25,178 internal ServiceNow articles with 8,870 query–gold records constructed using GPT-OSS-120B, evaluate 7 embedding models and 12 reranker configurations, and attach per-query diagnostics covering lexical overlap, distractor hardness, length, product area, content type, and intent. Main findings include a 1B cross-encoder outperforming a 32B listwise LLM by 7.5 MRR points, consistent degradation on long gold documents among the seven diagnostically analyzed rerankers, model-dependent sensitivity to low lexical overlap, and a reversal of the BGE/Qwen ordering on public TechQA. The contribution is the protocol, construction pipeline, and empirical study; there is no new retrieval or reranking method.
Critical strengths
-
The three-mode design (whole-corpus, fixed pool, end-to-end) gives a practical way to distinguish retrieval failures from downstream ranking failures. Section 4.3 shows this concretely: of 8,870 records, 372 designated golds are lost at L1 (Recall@100 = 0.958), and a further 373 are lost when the reranker truncates to its top-21. Table 3 also holds the query subset and candidate width fixed while changing pool composition. This accounting is what a practitioner needs, and it is the paper's main strength.
-
The reporting gives enough detail to audit the measurements. Appendix H.3 describes source-document-clustered bootstrapping and Holm correction, Appendix G.1 reproduces the included construction prompts verbatim, Table 7 states the artifact release boundary, and the note on A8 corrects an earlier duplicate-query count. The TechQA replication is also reported despite reversing the internal model ordering.
Critical weaknesses
1. The main ServiceNow instance cannot be independently rerun or used to evaluate new models without corpus access, and its validity remains insufficiently established.
The corpus is withheld (Appendix D, Table 7). The released artifacts (query and document identifiers, fixed pools, per-query ranks, code, prompts) support recomputation of reported aggregates, but they do not permit independent model scoring on the ServiceNow instance. Appendix D acknowledges this distinction.
The full query-generation and validation pipeline is demonstrated only on the private corpus. The TechQA build (Section 3.2, "Public replication") uses real questions and embedding-mined negatives, with one embedder and two rerankers. It validates the evaluation protocol under a different pool construction, rather than independently demonstrating the full synthetic construction pipeline.
On the internal build, the same model generates queries and participates in positive and negative validation (Sections 3.1 and 6). The only independent human check is one author auditing 200 of 8,870 query-gold pairs (2.3%) and an unspecified subset of negatives (Appendix C). The 98% gold correctness is a good sign, but this remains a limited single-annotator audit.
Queries have median lexical overlap of 0.80 with their gold documents (Section 3.4); 67.6% have overlap above 0.7, while only 120 records (1.4%) fall below 0.3 (Table 15). So the low-overlap comparison of 0.791 versus 0.594 MRR rests on 120 records and has no accompanying confidence interval or significance test. The generator uses 25 real queries as style exemplars (Appendix G), but the paper does not compare model ordering on real versus synthetic queries from the same knowledge base.
The design also builds on established practices; reranking evaluation is already represented in MTEB, synthetic query generation in InPars, and automated construction of retrieval evaluation data in AIR-Bench. AIR-Bench is not discussed. The paper's value then rests on the usability and validation of its construction pipeline and empirical findings. For a datasets and benchmarks submission, this is my decisive concern.
2. The attribution evidence is weaker than the paper's framing suggests.
(a) L3 is not independently isolated from L2. The authors say this on page 3, lines 109-110. The attribution paragraph in Section 3.3 distinguishes retrieval failures from failures in the downstream L2 and L3 pathway. The "direct on 21" rows in Table 4 evaluate individual rankers on fixed pools; they do not characterize L3 conditional on a controlled L2 output. The connected cascade does provide transition-level gold-survival counts, but this is narrower than independently evaluating all three components.
(b) The L3 gains lack a matched control for informative input ordering. Table 4 reports an improvement from 0.837 MRR for BGE's top-10 to 0.886 after Qwen3-32B, whereas direct Qwen on the fixed pools scores 0.789. This comparison changes both candidate selection and input ordering.
Figure 6 offers a relevant additional observation: at K = 21, the cascade reaches approximately 0.89 MRR on a 1,500-query subset, compared with 0.789 for direct Qwen on all queries in Table 4. At this width, the cascade should retain the full fixed pool, making input order a plausible explanation. The query populations differ, but bge alone scores 0.847 on that subset against 0.839 on all queries (Figure 6, Table 4), so the subset difference is unlikely to explain a 10-point gap. Appendix H.5 reports robustness to reshuffling fixed pools, but does not report a matched comparison between BGE-ranked and randomly ordered inputs to L3. Without that control, the contribution of informative ordering cannot be separated from Qwen's reranking contribution.
(c) Multi-positive augmentation creates a potentially consequential relevance-scoring issue. Appendix G states that augmentation writes additional records with identical query text and different gold documents. Section 3.3 defines relevance using only the record's designated gold; Appendix E acknowledges incomplete relevance judgments. A8 reports that 4,896 records share their query text.
Under deterministic retrieval for identical query text, and assuming each such query's records designate distinct gold documents, whole-corpus Recall@1 is bounded above by 6,013 / 8,870 ≈ 0.678. A known relevant sibling document can then outrank the designated gold without receiving relevance credit. Fixed-pool filtering is intended to remove answering negatives, but this does not resolve the issue when scoring against the full corpus or L1-derived candidates.
Some of the Table 3 drop from 0.854 to 0.638, and some of the 373 designated-gold losses at L2, could reflect sibling positives rather than incorrect answers outranking correct ones. A8 evaluates duplicate-query slices on fixed pools and does not quantify this effect in whole-corpus or end-to-end evaluation. The magnitude remains unverified; Q1 asks for the decisive check.
Smaller points that should be fixed
- Section 4.6, the conclusion, and Table 14 describe a universal 10–14 MRR-point long-document drop. Table 18 instead gives short-to-long drops ranging from 9.7 to 19.1 points. The paper's own BGE-v2-m3 example, 0.888 → 0.791, is a 9.7-point drop.
- Appendix H.3 specifies 12 planned comparisons, but Table 10 lists nine. The correction status of the three A2 p-values reported afterwards (0.044, 0.009, 0.094) is unclear. The confidence intervals reported alongside them imply the same values, which suggests they are unadjusted. If so, Holm correction within those three tests gives 0.088, 0.027, and 0.094, respectively, leaving one significant result rather than two. Please specify the comparison family and adjustment status.
- Table 6 defines K1 > K2 > K3, whereas Section 3.3 uses K1 ≥ K2 ≥ K3 and the main cascade uses 100 → 21 → 21.
- Table 2 bolds Recall@10 = 0.980 for Qwen3-32B pointwise, although 0.983 is the column maximum.
- Appendix H.6 refers to a "4–7×" latency figure "quoted in the main text," but that figure does not appear in the main text.
- The Ramanath et al. (2021) bibliography entry appears mismatched. Its author list matches "Towards Deep and Representation Learning for Talent Search at LinkedIn", published at CIKM 2018, rather than the listed title and KDD 2021 venue. Please verify the intended reference.
- The 7.5-point headline compares Nemotron with the no-CoT listwise Qwen3-32B configuration. The gap is 6.4 points against listwise Qwen with CoT and 4.7 points against the same Qwen checkpoint used pointwise (Table 2). This configuration dependence should accompany the headline.
- The abstract on OpenReview and the abstract in the PDF disagree (18 versus 19 model configurations; two non-monotonic scaling curves versus one).
Questions for authors
Clarification questions whose answers would change my assessment:
- How many records were added by multi-positive augmentation? In whole-corpus and end-to-end evaluation, are sibling golds with identical query text scored as non-relevant? Please recompute Table 9, Table 3, and the Section 4.3 loss counts with all known golds for each exact query text counted as relevant, aggregating results over distinct queries.
- On exactly the same query subset and candidate sets, how does Qwen3-32B listwise perform when the full fixed pool is supplied in BGE order versus random order? Please also provide this ordering control for BGE's top-10 candidates. These comparisons would separate candidate-selection effects from input-order effects.
- Are all evaluated checkpoints public releases with no fine-tuning on this corpus or data produced by the construction pipeline? Several prompts in Appendix G.1 describe a "reranker training dataset"; please state which uses of the pipeline output are downstream training and which are evaluation.
- For listwise Qwen3-32B and Qwen3-8B, how are pools of up to 21 documents represented within one prompt when documents exceed the available context? Are documents truncated, snippeted, or processed using a windowed listwise procedure? Section 3.3's maximum-chunk-score rule does not fully specify listwise handling. This affects interpretation of both the headline comparison and the long-document finding.
- Can the full GPT-OSS construction pipeline (query generation, mining, validation gates) be run on TechQA or another public corpus, with the resulting pools released?
- Do you have real queries with gold articles for this ServiceNow knowledge base, and does model ordering on those queries agree with Table 2?
- Which 12 comparisons were planned in Appendix H.3, and are the A2 p-values Holm-adjusted?
Suggestion, not required for my assessment: BM25 and hybrid baselines in Table 9 would help interpret lexical-overlap effects in whole-corpus retrieval, given the use of BM25 during negative mining.
I wanted to thank the authors for their efforts. The revised version partially addresses my concerns. As I have low confidence on the topic, I would like to follow the other reviewer and the senior PC's judgements.