acceptodds
Under review as a conference paper at ICLR 2027

Evaluating Multiple LLM Generations with Validated Task Coverage

Abstract

Many LLM applications are most useful when they provide several candidate outputs for comparison, validation, or combination. Predominant evaluation settings, however, still focus on individual outputs or reduce multiple samples to a single success or selected answer. This can miss whether the outputs include several genuinely different useful results. We introduce VTC-Bench, a five-domain benchmark for this setting, together with Validated Task Coverage (VTC) as its core evaluation quantity. The benchmark is built from carefully selected real-data tasks where both output quality and task-relevant distinctness can be checked automatically and reproducibly, without model-based judges. VTC measures how many distinct useful results are obtained within attempts. Across multiple models and inference settings, the benchmark leads to different conclusions from conventional evaluation: configurations that look strongest from single-draw quality are not necessarily those with the best coverage, and simple measures of output variation do not reliably recover task-relevant coverage. These results show that finite candidate sets can be evaluated directly as objects of interest, revealing differences in model behavior that are not apparent from conventional per-output evaluation.

Primary area:
datasets and benchmarks
Keywords:
large language models, evaluation, benchmarking, diversity, coverage, multi-sample generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7MsJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces VTC-Bench, a benchmark for evaluating LLMs based on the number of distinct outcomes generated within a fixed number of attempts. The authors propose Validated Task Coverage (VTC), which measures task-relevant coverage using automatic validation. Through experiments across multiple LLMs and inference configurations, the authors demonstrate that single-generation quality and surface-level diversity do not necessarily reflect validated task coverage.

Critical strengths

S1. Clear presentation and motivation

The paper is clearly written and well organized. The motivation and main claims are easy to follow, and the proposed evaluation framework is presented in a straightforward manner.

Critical weaknesses

W1. Limited Novelty and Unclear Practical Relevance

The novelty of VTC-Bench relative to existing coverage and diversity-based benchmarks, such as HypoSpace and NoveltyBench (in related work), is not sufficiently clear. While the paper extends coverage-based evaluation to multiple domains with deterministic validation, the underlying evaluation principle appears closely related to prior work. It remains unclear what fundamentally new evaluation capabilities or insights VTC-Bench provides beyond these existing approaches.

Furthermore, the practical motivation for independent repeated generation is not fully convincing. Several selected tasks, such as bug finding, molecule design, and evidence search, naturally involve iterative exploration guided by previous outcomes and validation feedback. Although the paper evaluates chained generation, this baseline conditions on previous outputs without incorporating validation feedback. Consequently, it remains unclear how well the proposed evaluation setting reflects realistic multi-candidate problem-solving workflows.

W2. Limited Empirical Insights into Coverage Behavior

While the paper demonstrates that model rankings based on single-generation quality can differ from those based on finite-budget coverage, the empirical analysis provides limited insight into the underlying factors driving these differences.

In particular, low coverage can arise from two distinct failure modes: generating few valid outputs or repeatedly generating the same valid outcomes. Although the paper compares single-draw quality and VTC, it does not sufficiently disentangle the contributions of validity and redundancy to the observed coverage differences.

Questions for authors
  1. What are the key contributions of VTC-Bench beyond existing benchmarks such as HypoSpace and NoveltyBench?
  2. Why should we approach the tasks in the VTC-Bench with 'flat-parallel generation framework'?
  3. Could the authors provide additional empirical insights other than the ones provided in the paper?
  4. Could additional experiments using advanced methods (e.g., self-refinement or evolutionary search) for chained generation be conducted to further demonstrate the necessity of “flat-parallel generation”?
Reviewer_qGMAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a metric, Validated Task Coverage (VTC), and a corresponding benchmark for evaluating the coverage of multiple LLM generations. Unlike traditional metrics such as pass@k, which focus on finding a single correct answer, VTC addresses tasks with multiple valid solutions. It measures coverage by mapping validated outputs to distinct useful outcomes and taking their union. The authors also develop a benchmark spanning five tasks, each with task-specific validation and coverage criteria. Experiments across three model families yield several findings: single-generation quality does not necessarily reflect coverage; higher temperatures generally improve coverage; and thinking offers less consistent benefits for coverage than for one-draw quality.

Critical strengths
  1. The paper tackles a valuable evaluation problem distinct from traditional metrics such as pass@k: measuring coverage across multiple valid solutions. The proposed VTC metric is simple and intuitive.
  2. The authors develop a diverse benchmark spanning five well-defined tasks, each with clear evaluation criteria. Extensive experiments across multiple model families offer useful insights into the shortcomings of existing evaluation methods.
Critical weaknesses
  1. The proposed VTC metric is relatively straightforward, and several empirical findings are unsurprising. For example, prior work such as Codex [1] has discussed the trade-off among sampling temperature, diversity, and single-generation accuracy. NoveltyBench has also explored the distinction between surface diversity and useful diversity. Furthermore, the evaluation covers only four models across three model families and omits stronger frontier models, making it difficult for readers to assess the benchmark's significance and understand how current models perform on it.
  2. The benchmark has different limitations depending on the task. For tasks with predefined reference answers, such as differential diagnosis, VTC is essentially a straightforward extension of pass@k, measuring coverage across multiple correct answers. For open-ended tasks such as repository repair, however, defining distinct, useful outcomes is much more challenging. The authors group patches according to the functions they modify, but fundamentally different repair strategies that modify the same function are treated as identical. This raises concerns about whether the proposed metric reliably captures meaningful diversity. Additional human evaluation or LLM-based verification could help validate these outcome definitions.

[1] Evaluating Large Language Models Trained on Code. https://arxiv.org/pdf/2107.03374

Questions for authors
  1. Could the authors provide stronger justification for their outcome mappings or validate them with human or LLM evaluations, particularly for open-ended tasks? For example, in repository repair, fundamentally different strategies that modify the same function are grouped together, while irrelevant changes to other functions could artificially inflate VTC. In bug finding, could models gain additional coverage credit by exploiting out-of-spec inputs or computational limits?
  2. For tasks with predefined reference answer sets, such as differential diagnosis, what fundamentally distinguishes VTC from a straightforward extension of pass@k to multiple valid answers?
  3. Could the authors include more recent frontier models to provide a clearer understanding of how state-of-the-art models perform?
  4. What motivates the different budget upper bound H across the five tasks?
Reviewer_5hUqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Validated Task Coverage (VTC), a metric for evaluating sets of multiple LLM generations based on how many distinct, useful outcomes are covered within k attempts, rather than reducing them to a single best answer. The authors build VTC-Bench, a five-domain benchmark (molecule design, repository repair, bug finding, differential diagnosis, evidence search) where output validity and distinctness can be checked automatically without model-based judges. Across 24 model-inference configurations, they show that one-draw quality and surface-level output diversity often fail to identify the configurations with the best VTC, and that inference choices like temperature and chained diversity prompting affect coverage inconsistently across tasks.

Critical strengths
  • The core idea of evaluating finite candidate sets as objects of interest rather than reducing to a single output addresses a real gap in current LLM evaluation (pass@k, best-of-N, self-consistency).
  • The benchmark spans five genuinely different domains with fully automatic, reproducible scoring, avoiding reliance on model-based judges for the useful-outcome mapping.
  • The empirical findings are non-trivial and consistent across tasks: quality-based and diversity-based selection both regularly fail to pick the highest-VTC configuration, which has practical implications for how multi-sample systems should be evaluated and tuned.
Critical weaknesses
  • Only four models (Qwen3.6-27B, Qwen3.6-35B-A3B, GLM-4.7, DeepSeek-V4-Flash) are tested, all fairly similar in scale and recency. Testing across a wider range of model families and sizes would strengthen claims about how general the quality-vs-coverage mismatch is.
  • As the authors note, attempt counts aren't directly comparable across tasks in terms of compute or token cost, so it's unclear whether the "best" configuration under VTC remains best once resource cost is factored in. This should be reported more prominently as a caveat on practical takeaways, since inference cost is often the deciding factor in choosing a sampling strategy.
Questions for authors

Please refer to Weaknesses Section

Comments on the paper

1 comment
AuthorsOpenReview ↗

General answer to reviews

We summarize below the main concerns raised by the reviewers and how we have addressed them in the revised paper.

  • Novelty and positioning. We clarified the gap between established quality-oriented benchmarks and existing diversity benchmarks, and how VTC-Bench addresses it:

    • Mainstream quality benchmarks (e.g., HumanEval, MATH, Terminal-Bench) provide deterministic evaluation of individual-output correctness, but do not assess the accumulation of distinct useful outcomes.
    • Existing diversity benchmarks face two important limitations: (1) reliance on learned judgments to assess useful diversity (NoveltyBench), or (2) reliance on simplified, fully enumerable solution spaces to enable deterministic evaluation (HypoSpace).
    • Our contribution is to bridge this gap by combining realistic, heterogeneous tasks with deterministic evaluation of both quality and task-relevant distinctness. This combination is the central contribution of VTC-Bench. We have made this positioning explicit in the revised Introduction and Related Work.
  • Semantic validity of outcome mappings. We acknowledge that our design constraints entail trade-offs in defining distinct useful outcomes. However, we believe our choices provide interpretable and practically meaningful proxies, with their rationale extensively documented in Appendix C. To further assess their validity, we conducted an additional semantic audit of our repository-repair proxy using GPT-6.1 Sol, finding that different function sets are substantially more likely to represent meaningful repair alternatives. We report this new analysis with further discussions in Appendix D.

  • Practical relevance. We clarified that VTC-Bench is agnostic to the inference strategy, with independent sampling a general reference protocol rather than a prescribed approach. We additionally implemented a validation-feedback-guided sequential generation baseline, demonstrating that VTC can evaluate adaptive exploration and capture its benefits (Section 6.3 and Appendix B).

  • Broader model coverage. We expanded the evaluation with Qwen3.8-27B and Nemotron 3 Ultra, including an additional model family. (Experimental Results).

  • Understanding coverage behavior. Following reviewer suggestion, we introduced an explicit validity–nonredundancy decomposition. We then show further experimental insights on VTC scores: many configurations achieving higher finite-budget coverage have lower validity but substantially higher nonredundancy (Results and Appendix D).

We thank the reviewers for their suggestions and hope it has helped improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.