acceptodds
Under review as a conference paper at ICLR 2027

CoverSynth: Coverage-Guided Generation of Retrieval Evaluation Sets

Abstract

Enterprise agents need retrieval evaluation sets that expose domain-specific failures and guide developers towards efficient deployment tradeoffs. Neither expert-human-authored sets nor current synthetic generation methods provide a methodology for achieving the necessary coverage, reliability, efficiency, and discriminability required for such decision-making. Simultaneously, the field does not have an assessment framework for determining when an evaluation set has satisfied these properties. We introduce an evaluation set assessment framework for these criteria and CoverSynth - a generator specifically designed to satisfy these criteria without incurring the high cost of human expert labeling. Starting from an unannotated corpus, CoverSynth targets coverage across retrieval demands and system capabilities. Deterministic checks locate evidence quotes in their sources, while model-based checks assess semantic validity to guarantee reliable ground truth. Across five public enterprise-style corpora, our method provides informative variation in difficulty, revealing unseen failure modes such as an inability of agents to handle underspecified or ambiguous requests. On the RepLiQA corpus, 91% of human-generated queries can be answered by all members on a panel of 12 retrieval systems, compared to only 12% of CoverSynth-generated queries. Finally, in a repeated TechQA/RepLiQA study, 12 of 12 within-harness model contrasts have positive nominal 95% intervals on generated sets, compared with 4 of 12 on human sets. CoverSynth makes coverage and evidence quality explicit construction objectives, producing corpus-grounded evaluations that reveal retrieval headroom and differences between agent configurations.

Primary area:
datasets and benchmarks
Keywords:
Agent evaluation, retrieval systems, synthetic dataset generation, information retrieval evaluation, corpus grounding, unanswerability & abstention, Item Response Theory

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HiqZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CoverSynth, a framework to generate a retrieval evaluation dataset based on an unlabelled corpus that explicitly targets coverage, evidence validity, and system comparison in designing the generation and evaluation process. Experiments are conducted on 5 enterprise corpora, and the system is compared with the published reference evaluation data. Experiments argue that CoverSynth generates a better-quality evaluation set based on the defined metrics and distinguishes weaker and stronger models better than existing methods.

Critical strengths

None

Critical weaknesses
  1. The paper has no human verification at all. For a paper that mainly targets the automated generation of an evaluation benchmark, human expert verification or comparison with real-world queries is definitely needed. It is possible that the generated data are only technically correct but neither realistic to end users nor truly reflect the requirements of the system to be deployed.

  2. The assessment metrics like AD and persona entropy are essentially what the method directly optimizes for. However, these metrics are not directly linked to the end-to-end quality of the generated dataset. Besides, the paper optimizes for harder evaluation data. However, harder does not mean better.

Questions for authors

Besides the critical weaknesses above, can you also clarify the scale of the experiment conducted, especially the ratio passing each gate, the number of documents included, and the final number of queries?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for their valuable feedback and hope our responses can address their concerns about the validity of our results.

[W1] We would first like to restate the use of human-generated evaluation datasets as baseline references for EnterpriseRAG, TechQA, and RepLiQA throughout the paper. The use of these human-generated baselines not only enabled comparison of coverage (Table 1) and retrieval performance (Table 9 vs. Table 10) between human and CoverSynth datasets, but also the comparison of gold passages between both strategies (Table 2). However, this analysis of span quality was exclusively LLM-adjudicated, and we acknowledge that LLM-generated content is prone to a myriad of pitfalls such as hallucinations, improper reasoning, and inherent bias.

In order to validate the LLM-adjudicated gates, we ran a calibration study aimed at measuring the alignment between LLM-generated labels and human-annotated labels. For this study, we sampled 96 items balanced across all five corpora, stratified by each judge's acceptance or rejection. Five generalist human annotators, blind to the pipeline's verdicts, labeled the sampled questions and spans for realism, sufficiency, and necessity. 90.5% of queries were rated as realistic (62%) or partially realistic (28.5%), and coherence was confirmed on 95% of the queries (i.e. shipped questions did not arbitrarily fuse unrelated topics to meet the generation input parameters).

Spans fully support the answer in 82% of judgments. Annotators mark 88–90% of quotes as needed, against 85–86% for our corrected probe on the same items (see our reply to Reviewer W8q7). The oracle reader, a different model from the generator, agrees with the humans on 82% of sufficiency and 85% of necessity calls. 78% of negatives of the corpus-unsupported and false-premise types are confirmed unanswerable. Ambiguous and underspecified negatives are confirmed only 50% of the time, signaling an area for potential improvement in both human validation and LLM validation of these questions.

As part of future work, we would like to investigate how SME feedback on the CoverSynth outputs and findings from agent traces in production deployments can be used as an additional steering parameter to automatically generate updated CoverSynth outputs.

[W2] We agree that AD and persona entropy largely confirm that the generator hit its own targets (personas are assigned by hash, depth by quota), and we will label them as such in Table 1. Please refer to our [W1/Q1] comment to Reviewer cUxZ on alternative evaluation metrics we used to confirm CoverSynth’s ability to achieve better coverage over the curated set of query dimensions. Regarding difficulty as a proxy for quality, we agree that harder questions do not imply better downstream utility, and hence we did not set difficulty distribution targets in our generation methodology. Instead, as seen in Tables 1 and 8 and Figures 3 and 4, our main interest in measuring difficulty was two-fold: 1) to determine CoverSynth’s ability to reliably discriminate the performance of different retrieval systems, and 2) to provide an assessment of the reliable headroom for retrieval systems. For example, on the TechQA benchmark, the human-curated reference evaluation set actually skewed further towards hard questions than the CoverSynth output (Table 8, Figure 4). However, upon closer inspection in Figure 3, we find that it is unable to reliably replicate the Haiku < Sonnet < Opus hierarchy in the ReAct and Claude Code agent harnesses. This highlights that aggregate difficulty alone is not able to drive downstream utility in retrieval agent evaluation, but rather it is the interaction of various query characteristics (e.g. question intent, number of hops, realism to downstream users) that drives the utility of these output evaluation sets.

[Q1] The five evaluated benchmarks contained anywhere from 60-500 documents, each generating evaluation datasets with up to 150 questions (100 positives + up to 50 negatives). Gate columns give the share of positive candidates reaching each gate that pass, in pipeline order; failed API calls are excluded. The chain gate runs before drafting and drops bridge candidates whose link is a shared name rather than a real dependency; it removes most candidates on large, loosely linked corpora. Negatives pass a separate verifier and adversarial solver. We will add per-gate counts to the appendix.

CorpusDocsQueries (pos+neg)ChainRealismLocateVerifyNegatives kept
ERAG95150 (100+50)20%87%100%77%50/91
Harvey137150 (100+50)3%83%96%73%50/96
TechQA500128 (100+28)17%86%99%66%28/145
eCFR60148 (100+48)79%71%93%79%48/179
RepLiQA275147 (100+47)45%85%97%68%47/71

Thank you for the thorough response and the additional analyses, which I read with interest. The human study is a welcome addition and partly addresses my concern about validation, although with a small sample of generalist annotators and only half of the ambiguous and underspecified negatives confirmed unanswerable, I do not think the realism and reliability of the generated sets are established yet. On the second point, I appreciate the candor of the re-scoring with independent topic models, but since it shows no significant coverage advantage over the human reference sets, my concern that the gains in Table 1 reflect the method's own objectives remains. The per-gate statistics are helpful and would be worth including in the paper. I keep my rating unchanged.

AuthorsOpenReview ↗

Reply to Rebuttal Comment by Authors

We thank the reviewer for considering our updated results and hope that the additional commentary below can remedy some of the remaining concerns.

As explained in our comment to Reviewer cUxZ, the human annotation results for the ambiguous and underspecified questions capture the ambiguity needed to identify the correct grounding material — a core capability for trustworthy search agents — rather than simply the right answer. As mentioned in that response, we will continue to expand the scale of our human annotations, including those from expert annotators, and not just generalists.

Regarding the updated topic modeling results, we would like to clarify the findings. Our earlier statement that “no reference set is significantly better” meant that the reference sets never win; it did not mean that the sets tie. Across 18 topic models and 6 coverage metrics, CoverSynth is significantly better than the human sets in 161 of 324 tests (p < 0.05) and significantly worse in none. Against MiRAGE, the sets tie (CoverSynth is better in 4 of 216 tests). MiRAGE's gold consists of 1–11 whole chunks per question (median 2.5K characters), so each of its questions already reaches several topics. On label-free measures of the questions themselves, however, CoverSynth is more diverse than MiRAGE: 91–97% of MiRAGE questions share a 4-gram with another question in the set, compared to 48–58% for CoverSynth, and CoverSynth leads on embedding diversity as measured via a higher Vendi score (40.4 vs. 33.9 on Harvey and 44.7 vs. 34.3 on eCFR).

Reviewer_cUxZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CoverSynth, a framework for automatically generating retrieval evaluation sets from unannotated corpora. The method combines concept-graph construction, coverage-guided question generation, evidence verification, and controlled negative-query generation to improve coverage across question types, user personas, and corpus regions. Experiments on five enterprise-style corpora show that CoverSynth achieves greater question diversity and, in certain settings, more effectively discriminates between Retrieval Agents with different capabilities.

Critical strengths
  1. The paper addresses the limited coverage of existing retrieval evaluation sets and their inability to expose system-specific failure modes. This is an important problem for evaluating enterprise RAG systems and Retrieval Agents.
  2. The proposed method integrates coverage control, multi-hop question generation, evidence localization, and negative-query generation into a unified pipeline. Exact quote localization also improves the traceability of supporting evidence.
  3. Beyond measuring coverage and gold-evidence validity, the paper conducts repeated experiments across multiple retrieval systems to assess the discriminability of the generated evaluation sets.
Critical weaknesses
  1. Strong coupling between evaluation metrics and method design. CoverSynth explicitly optimizes coverage across question intents, personas, and corpus regions, while its advantages are primarily demonstrated using metrics that directly reflect these objectives. In particular, personas are inferred by CoverSynth itself and then used to evaluate other benchmarks, potentially introducing bias into the comparison. Moreover, a more balanced distribution does not necessarily imply better alignment with real-world user needs.
  2. Lack of independent validation of ground-truth quality. Exact quote localization only ensures that the supporting evidence exists in the source documents; it does not guarantee semantic correctness. Assessments of answerability, evidence sufficiency, and multi-hop dependencies still rely heavily on LLM-based judgments, with limited independent human validation.
  3. System discriminability improvements may depend on the evaluation panel. CoverSynth demonstrates higher evaluation reliability when considering the six base agents. However, this relative advantage disappears when traditional indexed retrievers are included. This suggests that the reported improvements in system discriminability may depend on the particular set of systems being evaluated, limiting the generalizability of the conclusions to retrieval systems more broadly.
  4. Insufficient analysis of generation efficiency and cost. Although the paper emphasizes avoiding expensive expert-human annotation, the complete generation pipeline involves multiple rounds of LLM calls. The authors do not adequately report the generation cost per accepted question, candidate rejection rates, or quality differences compared with simpler generation methods under matched budgets.
  5. Presentation clarity and missing citations. The overall presentation could be improved. Figure 1 is visually cluttered and does not clearly highlight the key mechanisms of CoverSynth or the relationships between its components. Some technical descriptions are rather abstract and lack intuitive examples or sufficient explanation. In addition, the descriptions of datasets and baselines lack systematic, direct references to the corresponding literature. Several important experimental details and limitations are discussed only in the appendix, which affects the clarity, traceability, and reproducibility of the work.
Questions for authors
  1. How well do the proposed coverage metrics reflect real-world benchmark quality? Could the authors use real user queries or independently defined task taxonomies from domain experts to demonstrate that improvements in coverage metrics translate into meaningful improvements in benchmark quality?
  2. How is the validity of generated questions and ground truth independently verified? Could the authors provide an independent human evaluation of the validity and correctness of generated questions, gold evidence, and hard negatives? Additionally, the descriptions of the MiRAGE oracle-reader comparisons in Table 2 and Appendix A.12 appear somewhat inconsistent. Could the authors clarify the evaluation protocol and comparability of results across different evidence granularities?
  3. Why should controlled edits produce higher-quality hard negatives? Could the authors conduct controlled comparisons against direct LLM-based generation, random edits, and existing negative-query generation methods to demonstrate the effectiveness of the proposed approach?
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for their detailed feedback and questions.

[W1/Q1] We acknowledge that the coverage results in Table 1 use CoverSynth-generated outputs (e.g. user personas, entity clusters) to compare to baselines. This was partially intentional, as the goal was to validate that generation adhered to input generation parameters (e.g. persona, multi-hop, scenario intent) when subjected to various semantic and quality gates. Because this can appear self-preferential, we re-score every output without the use of CoverSynth’s intermediate outputs, such as personas, labels, or clusters:

  • Independent topic model: On 18 topic models fit to each corpus independently (LDA with 5 seeds and NMF, each at 20, 50 and 100 topics), no reference set is significantly better on any of the 6 coverage metrics. CoverSynth leaves fewer topics untested than the human sets (TechQA 0.12 vs 0.33) and ties MiRAGE.
  • Label-free text diversity: Against MiRAGE, CoverSynth is more diverse on every metric (4-gram repetition 0.58 vs 0.91 on Harvey, 0.48 vs 0.97 on eCFR). Against human sets, it is not more semantically diverse.
  • Persona-driven query phrasing: On a fixed rubric (assumed expertise × background given) rated blind, CoverSynth beats MiRAGE on both corpora but on average is about on par with human sets. Senior personas write terser, more expert questions, but not harder ones (IRT β slope −0.04 [−0.15, 0.07]); difficulty follows how many files the answer spans.

Regarding downstream utility for text retrieval evaluation, we agree that coverage alone does not make a set more representative of user needs. Its ability to drive improved benchmark quality lies in covering dimensions where retrieval systems fail differently (e.g. intent, hop depth, negative family). For example, in Figure 2, the negative taxonomy, which was inspired by RefusalBench's defined categories, surfaced that Opus-5 gains mainly on underspecified and ambiguous negatives. In addition, structural depth drives difficulty: IRT difficulty rises with the number of files an answer spans (β 0.65, 1.23, 1.50 for 1, 2, 3 files).

[W2/Q2] We agree that span matching is only a preliminary requirement for ground truth validity. In our blind human study (reply to Reviewer HiqZ), gold spans are sufficient in 82% of judgments, and 78% of corpus-unsupported and false-premise negatives are confirmed unanswerable, but only 50% of ambiguous and underspecified ones.

We thank the reviewer for flagging the discrepancy between Appendix A.12 and Table 2. MiRAGE's gold is whole chunks (1–11 per item, median 2.5K characters) rather than exact spans, so we reran Table 2 with the same reader, judge, and protocol for every column and checked whether this granularity drives the results. It does not penalize sufficiency, since a long chunk passes as long as it supports the answer. Necessity is lower for MiRAGE (0.41 vs 0.78 on Harvey, 0.57 vs 0.77 on eCFR), and the gap holds among questions with four or more gold units (0.32 vs 0.74; 0.41 vs 0.71) and when CoverSynth is scored by chunk instead of span (0.80 or above). We will correct A.12 and label MiRAGE necessity as chunk-level.

[W3] Adding BM25, dense, and hybrid retrieval pushes G(100) to 0.976–0.997 for every set, because the retriever–agent gap dominates system variance. Per Figure 3 and Table 13, even human datasets separate retrievers from agents, so ΔG is uninformative there. The informative test is among comparable agents: G(100) is 0.94 vs 0.74 on TechQA (ΔG +0.20 [+0.05, +0.66]) and 0.93 vs 0.86 on RepLiQA (+0.07 [−0.06, +0.34]). Eight of 15 agent pairs meet the detection criterion, compared with 3 and 4 for the human sets, and only CoverSynth recovers the Haiku < Sonnet < Opus order.

[W4] We leave cost and latency optimization as future work. However, in its present state, the average end-to-end cost to generate 150 questions using CoverSynth on corpora ranging from 60-500 documents is ~$500. The key bottleneck for cost and latency is the entity extraction and dependency graph creation, which is proportional to the corpus size. Please reference our response to Q1 for Reviewer HiqZ for information on rejection rates. Overall, the cost can be optimized for larger production-grade corpora, but even the current cost remains substantially lower than that of human annotation.

[W5] We will redesign Figure 1 around one worked example to demonstrate the utility of each pathway and check. We will also update our bibliography to reflect the contributions of prior works in designing the evaluated benchmarks.

[Q3] The decision to use perturbations to generate negative queries was informed by prior work done by RefusalBench (Muhamed et al., 2025), while defect placement by hop was informed by HopRefusalBench (Xie et al., 2026). In particular, this strategy provided traceability and enabled analysis of the impact of semantics and question content on performance for each question pair.

I thank the authors for their response and additional experiments. I appreciate their efforts to clarify MiRAGE evaluation protocols, System Discriminability, cost and offer additional human evaluation. However, several key concerns remain unresolved:

  1. The lack of comparisons with alternatives such as direct LLM generation and random perturbations makes it difficult to establish the effectiveness of Hard Negative Generation.

  2. The authors have not sufficiently demonstrated that improved coverage translates into better benchmark quality in real-world settings.

  3. Only 50% of the Ambiguous and Underspecified negatives were confirmed to be unanswerable, which raises concerns about benchmark reliability and the validity of the evaluation results.

  4. Although the rebuttal period provided an opportunity to update the manuscript, the promised revisions to Figure 1 and the addition of relevant references have not been incorporated into the current version.

Overall, while the authors’ response addresses some of my concerns, these unresolved issues still undermine my confidence in the proposed method and the reliability of the generated benchmark.

AuthorsOpenReview ↗

Reply to Rebuttal Comment by Authors

We thank the reviewer for taking the time to assess our updated results, and hope that our additional edits and clarifications can alleviate some of the remaining concerns.

We have resubmitted the paper with the promised changes to Figure 1 and benchmark citations.

Regarding the previously reported 50% human acceptance rate for ambiguous and underspecified questions, the annotator’s notes pointed to one prominent failure mode: CoverSynth treated search ambiguity differently from ambiguity in the expected response. For example, one question asked about legal requirements when working with “the Council”. Given the existence of multiple councils with their own respective authoritative documents, CoverSynth considered this question as ambiguous and required surfacing spans from multiple authoritative documents to present for human escalation. However, because the legal requirements for each council was the same, human annotators did not feel clarification was necessary as the final response would have been the same regardless of which document was cited. This type of variation was the source of rejections for 20% of the rejections in the human annotation study, and appears in nearly 48% of all generated negative queries across the 5 evaluated benchmarks. We acknowledge that this type of discrepancy could have been handled differently if we used expert annotators instead of generalists, and will work to gather expert annotations as next steps. However, we would like to clarify that CoverSynth is designed to evaluate and discriminate agents’ search capabilities, and, ultimately, full awareness of the available content in a corpus is central to evaluating retrieval performance, while question-answering and human escalations are a separate LLM capabilities that CoverSynth does not control. Therefore, the ambiguous/underspecified labeling for these types of questions reflects the potential search failure modes of retrieval agents.

Reviewer_W8q7OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to automatically generate evaluation sets for retrieval systems and RAG agents from unlabeled enterprise documents.

The authors propose CoverSynth, a framework that first extracts concepts from documents and constructs a knowledge graph. It then uses Coverage Control to balance the coverage of documents, question types, and retrieval depths, generating both answerable and unanswerable questions. String matching and LLM-based verification are used to assess the reliability of generated questions and their supporting evidence. In addition, the authors introduce an evaluation framework to assess the quality of generated benchmarks in terms of coverage, evidence validity, and system discriminability.

Critical strengths
  1. A comprehensive framework. The paper proposes a framework for automatically generating retrieval evaluation sets. It can generate different types of questions with varying retrieval depths, including unanswerable questions. It also uses evidence verification to improve data quality and help reduce the cost of manually constructing evaluation sets.
  2. Extensive experiments with reasonably convincing results. The authors conduct experiments on five datasets. The results show that the generated evaluation sets achieve better coverage and question diversity, and can more effectively distinguish the performance of different retrieval systems.
Critical weaknesses
  1. Limited methodological novelty. The framework mainly combines existing techniques, including knowledge graph construction, heuristic coverage control, and LLM-based generation and verification. The core Coverage Control relies on standard strategies such as Round-robin, Quota, and Greedy Scheduling, without substantial algorithmic novelty or theoretical guarantees. Moreover, Coverage, Not Averages (ICML 2026) has explored a similar coverage-guided generation approach, but the paper lacks direct comparisons to demonstrate its additional contributions.
  2. Lack of key ablation studies. The paper does not evaluate the individual contributions of key components, particularly Coverage Control. Comparisons with simpler strategies, such as Random Sampling or Uniform Sampling, are also missing. Therefore, it remains unclear whether the proposed mechanisms are necessary to achieve the reported improvements.
  3. Insufficient validation of evidence reliability. Exact string matching only ensures that cited evidence exists in the source documents, but does not guarantee semantic correctness. Moreover, the Gold Evidence Necessity scores in Table 2 are only 0.49–0.67. Although this may be related to intermediate evidence in multi-hop retrieval, further analysis is needed to establish the reliability of the generated evaluation data.
Questions for authors
  1. Theoretical guarantees of Coverage Control. The proposed Coverage Control mainly relies on heuristic strategies. Can the authors provide any theoretical guarantees on the achieved coverage or deviations from target distributions? In particular, can the error bounds or stability be analyzed under limited generation budgets or candidate verification failures?
  2. Differences from prior work. Klearman et al. (ICML 2026) have already proposed coverage-guided question generation based on semantic clustering. Could the authors clarify the main algorithmic contributions of CoverSynth and provide a direct comparison under the same datasets and generation budgets?
  3. Effectiveness of key components. Could the authors provide ablation studies on Coverage Control and compare it with simpler strategies such as Random Sampling and Uniform Sampling? What are the individual contributions of Knowledge Graph Construction and Verification?
  4. Reliability of Gold Evidence. The Gold Evidence Necessity scores in Table 2 are only 0.49–0.67. Could the authors explain these results and distinguish intermediate evidence required for multi-hop retrieval from redundant evidence? Could human evaluation further validate the semantic reliability of the evidence?
1 comment
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for recognizing the breadth of our framework and appreciate their thoughtful questions.

[W1/Q2] We acknowledge that our work leverages existing strategies, such as semantic-cluster coverage measurement from Coverage, Not Averages (CNA; Klearman et al., 2026), perturbation-based negatives from RefusalBench (Muhamed et al., 2025), and Hamilton apportionment, and do not claim these individual components as algorithmic novelty. Instead, as shown in the comparison matrix in Appendix A.1, CoverSynth’s novel contribution comes in the unique application and cumulative utility of these strategies:

  1. Compared to CNA or MiRAGE, CoverSynth natively identifies exact spans for each query, rather than whole documents or large chunks, respectively. This difference is crucial to discriminating performance of agentic retrieval systems whose tool design, tool use, and reasoning quality vary with agent harness and model backbones. For example, the granularity of content exposed via a grep or glob call is different than the top-k sampling of a dense retriever.
  2. CNA measures coverage over a single semantic dimension — entities; CoverSynth optimizes for coverage across multiple dimensions — user persona, negative classification, number of hops, question family, etc. These curated dimensions reflect the failure modes, retrieval capabilities, and constraints that enterprises consider and evaluate before deploying AI agents into production, which other strategies fail to capture exhaustively. Therefore, implementing and evaluating the feasibility of this multi-objective synthetic generation is a core component of CoverSynth’s methodological novelty.

To address Q2, we ran the Coverage, Not Averages query generation methodology for the same datasets. The results show that even for the entity/semantic coverage dimensions that Coverage, Not Averages directly optimizes for — ZQC and SCC — CoverSynth performs better across all datasets. We will add these results as a relevant baseline to Table 1. CoverSynth / CNA; † = Holm-adjusted p < .05

CorpusAD ↑Persona H_norm ↑ZQC ↓SCC ↑
ERAG-Bench0.88 / 0.54†0.98 / 0.72†0.46 / 0.61†0.97 / 0.94
Harvey FK0.89 / 0.23†0.99 / 0.73†0.62 / 0.77†0.86 / 0.85
TechQA0.89 / 0.55†0.99 / 0.73†0.53 / 0.590.97 / 0.93†
eCFR0.82 / 0.27†0.99 / 0.81†0.48 / 0.69†0.58 / 0.15†
RepLiQA0.75 / 0.44†0.99 / 0.65†0.59 / 0.75†0.66 / 0.44

[W2/Q1/Q3] We logged the first gate each rejected candidate failed (see response to Reviewer HiqZ). RepliQA generated 406 positive candidates, of which 68% failed to demonstrate strong chained reasoning dependencies. Ultimately, only 11% of the accepted questions are multi-hop because each document of fictional prose was largely unique, demonstrating that coverage across these dimensions is bounded by the corpus structure. Regarding theoretical guarantees, we will add a proof to Section 3.6 that Coverage Control never overshoots hard targets, even when verification fails, because a failed candidate releases its slot; any shortfall therefore equals the unfilled budget, and total deviation from target stays below ∣J∣+(n−N^)|J| + (n - \hat{N}). Ultimately, we limit the focus of this paper to investigating the feasibility of multi-objective coverage generation and leave investigation regarding optimizing individual sub-modules, such as testing various sampling strategies, as future work.

[W3/Q4] We agree that span matching is only a preliminary requirement for ground truth validity. Therefore, Table 2 measures three LLM-adjudicated metrics — sufficiency, uniqueness, and necessity — whose definitions are provided in Appendix A.12. In our reply to Reviewer HiqZ, human annotators find gold spans sufficient in 82% of judgments, and our oracle agrees with them on 82% of sufficiency and 85% of necessity calls.

Regarding the low necessity scores, we traced this back to the leniency of the judge, which judged a passage unnecessary if the answer without it kept the core details rather than all details of the reference response. With a strict judge and references written from each item’s final gold, CoverSynth necessity is now 0.77–0.96.

To separate multi-hop evidence from redundant evidence, we broke down necessity by span role. To separate multi-hop evidence from redundant evidence, we broke down necessity by span role. Only 3–5% of spans of any type are redundant. In bridge chains, the passage holding the answer has necessity 0.89, but earlier hops only 0.51, as they have no bearing on the final response, only the search process.

Gold span typeNecessityRedundant span
Single-passage evidence0.970.03
Aggregation evidence (one fact per passage)0.960.03
Bridge chain: passage holding the answer0.890.05
Bridge chain: earlier hop (intermediate evidence)0.510.03

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.