- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces VTC-Bench, a benchmark for evaluating LLMs based on the number of distinct outcomes generated within a fixed number of attempts. The authors propose Validated Task Coverage (VTC), which measures task-relevant coverage using automatic validation. Through experiments across multiple LLMs and inference configurations, the authors demonstrate that single-generation quality and surface-level diversity do not necessarily reflect validated task coverage.
Critical strengths
S1. Clear presentation and motivation
The paper is clearly written and well organized. The motivation and main claims are easy to follow, and the proposed evaluation framework is presented in a straightforward manner.
Critical weaknesses
W1. Limited Novelty and Unclear Practical Relevance
The novelty of VTC-Bench relative to existing coverage and diversity-based benchmarks, such as HypoSpace and NoveltyBench (in related work), is not sufficiently clear. While the paper extends coverage-based evaluation to multiple domains with deterministic validation, the underlying evaluation principle appears closely related to prior work. It remains unclear what fundamentally new evaluation capabilities or insights VTC-Bench provides beyond these existing approaches.
Furthermore, the practical motivation for independent repeated generation is not fully convincing. Several selected tasks, such as bug finding, molecule design, and evidence search, naturally involve iterative exploration guided by previous outcomes and validation feedback. Although the paper evaluates chained generation, this baseline conditions on previous outputs without incorporating validation feedback. Consequently, it remains unclear how well the proposed evaluation setting reflects realistic multi-candidate problem-solving workflows.
W2. Limited Empirical Insights into Coverage Behavior
While the paper demonstrates that model rankings based on single-generation quality can differ from those based on finite-budget coverage, the empirical analysis provides limited insight into the underlying factors driving these differences.
In particular, low coverage can arise from two distinct failure modes: generating few valid outputs or repeatedly generating the same valid outcomes. Although the paper compares single-draw quality and VTC, it does not sufficiently disentangle the contributions of validity and redundancy to the observed coverage differences.
Questions for authors
- What are the key contributions of VTC-Bench beyond existing benchmarks such as HypoSpace and NoveltyBench?
- Why should we approach the tasks in the VTC-Bench with 'flat-parallel generation framework'?
- Could the authors provide additional empirical insights other than the ones provided in the paper?
- Could additional experiments using advanced methods (e.g., self-refinement or evolutionary search) for chained generation be conducted to further demonstrate the necessity of “flat-parallel generation”?
General answer to reviews
We summarize below the main concerns raised by the reviewers and how we have addressed them in the revised paper.
Novelty and positioning. We clarified the gap between established quality-oriented benchmarks and existing diversity benchmarks, and how VTC-Bench addresses it:
Semantic validity of outcome mappings. We acknowledge that our design constraints entail trade-offs in defining distinct useful outcomes. However, we believe our choices provide interpretable and practically meaningful proxies, with their rationale extensively documented in Appendix C. To further assess their validity, we conducted an additional semantic audit of our repository-repair proxy using GPT-6.1 Sol, finding that different function sets are substantially more likely to represent meaningful repair alternatives. We report this new analysis with further discussions in Appendix D.
Practical relevance. We clarified that VTC-Bench is agnostic to the inference strategy, with independent sampling a general reference protocol rather than a prescribed approach. We additionally implemented a validation-feedback-guided sequential generation baseline, demonstrating that VTC can evaluate adaptive exploration and capture its benefits (Section 6.3 and Appendix B).
Broader model coverage. We expanded the evaluation with Qwen3.8-27B and Nemotron 3 Ultra, including an additional model family. (Experimental Results).
Understanding coverage behavior. Following reviewer suggestion, we introduced an explicit validity–nonredundancy decomposition. We then show further experimental insights on VTC scores: many configurations achieving higher finite-budget coverage have lower validity but substantially higher nonredundancy (Results and Appendix D).
We thank the reviewers for their suggestions and hope it has helped improve the paper.