- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes CoverSynth, a framework to generate a retrieval evaluation dataset based on an unlabelled corpus that explicitly targets coverage, evidence validity, and system comparison in designing the generation and evaluation process. Experiments are conducted on 5 enterprise corpora, and the system is compared with the published reference evaluation data. Experiments argue that CoverSynth generates a better-quality evaluation set based on the defined metrics and distinguishes weaker and stronger models better than existing methods.
Critical strengths
None
Critical weaknesses
-
The paper has no human verification at all. For a paper that mainly targets the automated generation of an evaluation benchmark, human expert verification or comparison with real-world queries is definitely needed. It is possible that the generated data are only technically correct but neither realistic to end users nor truly reflect the requirements of the system to be deployed.
-
The assessment metrics like AD and persona entropy are essentially what the method directly optimizes for. However, these metrics are not directly linked to the end-to-end quality of the generated dataset. Besides, the paper optimizes for harder evaluation data. However, harder does not mean better.
Questions for authors
Besides the critical weaknesses above, can you also clarify the scale of the experiment conducted, especially the ratio passing each gate, the number of documents included, and the final number of queries?
3 comments
Thank you for the thorough response and the additional analyses, which I read with interest. The human study is a welcome addition and partly addresses my concern about validation, although with a small sample of generalist annotators and only half of the ambiguous and underspecified negatives confirmed unanswerable, I do not think the realism and reliability of the generated sets are established yet. On the second point, I appreciate the candor of the re-scoring with independent topic models, but since it shows no significant coverage advantage over the human reference sets, my concern that the gains in Table 1 reflect the method's own objectives remains. The per-gate statistics are helpful and would be worth including in the paper. I keep my rating unchanged.
Reply to Rebuttal Comment by Authors
We thank the reviewer for considering our updated results and hope that the additional commentary below can remedy some of the remaining concerns.
As explained in our comment to Reviewer cUxZ, the human annotation results for the ambiguous and underspecified questions capture the ambiguity needed to identify the correct grounding material — a core capability for trustworthy search agents — rather than simply the right answer. As mentioned in that response, we will continue to expand the scale of our human annotations, including those from expert annotators, and not just generalists.
Regarding the updated topic modeling results, we would like to clarify the findings. Our earlier statement that “no reference set is significantly better” meant that the reference sets never win; it did not mean that the sets tie. Across 18 topic models and 6 coverage metrics, CoverSynth is significantly better than the human sets in 161 of 324 tests (p < 0.05) and significantly worse in none. Against MiRAGE, the sets tie (CoverSynth is better in 4 of 216 tests). MiRAGE's gold consists of 1–11 whole chunks per question (median 2.5K characters), so each of its questions already reaches several topics. On label-free measures of the questions themselves, however, CoverSynth is more diverse than MiRAGE: 91–97% of MiRAGE questions share a 4-gram with another question in the set, compared to 48–58% for CoverSynth, and CoverSynth leads on embedding diversity as measured via a higher Vendi score (40.4 vs. 33.9 on Harvey and 44.7 vs. 34.3 on eCFR).
Rebuttal by Authors
We thank the reviewer for their valuable feedback and hope our responses can address their concerns about the validity of our results.
[W1] We would first like to restate the use of human-generated evaluation datasets as baseline references for EnterpriseRAG, TechQA, and RepLiQA throughout the paper. The use of these human-generated baselines not only enabled comparison of coverage (Table 1) and retrieval performance (Table 9 vs. Table 10) between human and CoverSynth datasets, but also the comparison of gold passages between both strategies (Table 2). However, this analysis of span quality was exclusively LLM-adjudicated, and we acknowledge that LLM-generated content is prone to a myriad of pitfalls such as hallucinations, improper reasoning, and inherent bias.
In order to validate the LLM-adjudicated gates, we ran a calibration study aimed at measuring the alignment between LLM-generated labels and human-annotated labels. For this study, we sampled 96 items balanced across all five corpora, stratified by each judge's acceptance or rejection. Five generalist human annotators, blind to the pipeline's verdicts, labeled the sampled questions and spans for realism, sufficiency, and necessity. 90.5% of queries were rated as realistic (62%) or partially realistic (28.5%), and coherence was confirmed on 95% of the queries (i.e. shipped questions did not arbitrarily fuse unrelated topics to meet the generation input parameters).
Spans fully support the answer in 82% of judgments. Annotators mark 88–90% of quotes as needed, against 85–86% for our corrected probe on the same items (see our reply to Reviewer W8q7). The oracle reader, a different model from the generator, agrees with the humans on 82% of sufficiency and 85% of necessity calls. 78% of negatives of the corpus-unsupported and false-premise types are confirmed unanswerable. Ambiguous and underspecified negatives are confirmed only 50% of the time, signaling an area for potential improvement in both human validation and LLM validation of these questions.
As part of future work, we would like to investigate how SME feedback on the CoverSynth outputs and findings from agent traces in production deployments can be used as an additional steering parameter to automatically generate updated CoverSynth outputs.
[W2] We agree that AD and persona entropy largely confirm that the generator hit its own targets (personas are assigned by hash, depth by quota), and we will label them as such in Table 1. Please refer to our [W1/Q1] comment to Reviewer cUxZ on alternative evaluation metrics we used to confirm CoverSynth’s ability to achieve better coverage over the curated set of query dimensions. Regarding difficulty as a proxy for quality, we agree that harder questions do not imply better downstream utility, and hence we did not set difficulty distribution targets in our generation methodology. Instead, as seen in Tables 1 and 8 and Figures 3 and 4, our main interest in measuring difficulty was two-fold: 1) to determine CoverSynth’s ability to reliably discriminate the performance of different retrieval systems, and 2) to provide an assessment of the reliable headroom for retrieval systems. For example, on the TechQA benchmark, the human-curated reference evaluation set actually skewed further towards hard questions than the CoverSynth output (Table 8, Figure 4). However, upon closer inspection in Figure 3, we find that it is unable to reliably replicate the Haiku < Sonnet < Opus hierarchy in the ReAct and Claude Code agent harnesses. This highlights that aggregate difficulty alone is not able to drive downstream utility in retrieval agent evaluation, but rather it is the interaction of various query characteristics (e.g. question intent, number of hops, realism to downstream users) that drives the utility of these output evaluation sets.
[Q1] The five evaluated benchmarks contained anywhere from 60-500 documents, each generating evaluation datasets with up to 150 questions (100 positives + up to 50 negatives). Gate columns give the share of positive candidates reaching each gate that pass, in pipeline order; failed API calls are excluded. The chain gate runs before drafting and drops bridge candidates whose link is a shared name rather than a real dependency; it removes most candidates on large, loosely linked corpora. Negatives pass a separate verifier and adversarial solver. We will add per-gate counts to the appendix.