acceptodds
Under review as a conference paper at ICLR 2027

VeriScale: Adversarial Test-Suite Scaling for Formal Specification Evaluation

Abstract

Formal verification can establish that generated code satisfies a formal specification, but this guarantee is meaningful only when the specification faithfully captures the intended behavior. Existing benchmarks evaluate LLM-generated specifications using concrete test cases, whose sparse coverage can allow flawed specifications to pass and thus systematically overestimate SpecGen capabilities. We present VeriScale, a framework for scaling test suites for formal specification evaluation. VeriScale expands candidate inputs through LLM-based seed generation and type-aware mutation, and synthesizes adversarial implementations that exploit weaknesses in generated specifications to produce challenging incorrect outputs. It further constructs compact test suites through boundary-preserving reduction and mutant-guided selection, reusing the adversarial implementations as mutants. Applied to Verina, VeriScale produces VerinaPlus, an approximately expanded benchmark, and VerinaLite, a compact variant. Across eight state-of-the-art LLMs, VerinaPlus reduces the average Spec Score from 61.44 to 55.09, exposing approximately 12 additional task-level failures per model among 189 tasks. VerinaLite obtains an average score of 56.02, retaining 85.45% of the score reduction with only about 16% of the tests in VerinaPlus. Code and data are provided in the supplementary material.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Lean, adversarial test-suite scaling, autoformalization, formal verification, benchmarking

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xNvfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces VeriScale, a framework for strengthening evaluation of LLM-generated preconditions and postconditions. It combines language-model seed generation, type-aware input mutations, reference-based input classification, and adversarial implementations that exploit weaknesses in candidate specifications to produce challenging outputs. Applied to VERINA, the approach produces Plus and Lite variants with approximately 83 and 13 times the original number of tests. Evaluation across eight models and 189 tasks lowers the average reported Spec Score from 61.44 to 55.09 with Plus. Lite retains 85.45% of this score reduction using approximately 16% of the Plus tests.

Critical strengths
  1. Effective benchmark augmentation. All models evaluated on Plus and Lite show a clear performance drop compared to the original VERINA, indicating that the benchmark does become relatively harder after being augmented.

  2. Novel synthesis of unexpected outputs targeting the weaknesses of the spec. Prior work tends to generate outputs via random sampling, which is relatively inefficient; the proposed method can perform targeted hacks against the specification.

Critical weaknesses
  1. Potential bias toward simple inputs. The Precondition-Guided Input Classification part of Section 3.1 states that "Candidates that still lack a definitive verdict due to timeouts or search exhaustion are permanently excluded from the final test suite." This may leave mostly simple inputs in the final suite, since for complex inputs it is hard for Lean's evaluation alone to decide whether the precondition is satisfied—thus biasing the test set. The authors could mitigate this by using a model to generate proofs that the inputs satisfy the precondition.

  2. The construction of unexpected outputs is too strict. An output that differs from the ground-truth code's output is not necessarily incorrect. For example, in a task that allows returning the index of any maximum value, multiple indices may all satisfy the requirement when several maxima exist; likewise, the list representation of an unordered collection may admit multiple valid orderings. Such overly strict tests may be the reason why models' performance drops on the augmented benchmark, and the authors need to clarify this—since Plus and Lite contain far more test cases than the original benchmark, they may inadvertently introduce a large number of overly strict tests.

Questions for authors

See weaknesses.

1 comment

We thank you for these two constructive comments. They helped us improve input logging, introduce a fallback using Lean-checked proofs for unresolved precondition checks, and correct the benchmark data. We report these changes and the updated evaluation results below.

W1: Selection bias from unresolved precondition checks.

We agree that discarding inputs whose precondition checks remain unresolved may introduce selection bias. Our logs report 97,990 classified candidates, of which 75,499 satisfy and 22,491 violate the ground-truth precondition, together with 2,922 unsuccessful validation checks. These failures include timeouts and other checking errors. Our previous implementation did not preserve their candidate values or individual failure causes, so we cannot replay these historical cases or determine how many failed due to timeouts.

Our classifier substitutes each concrete input into the ground-truth precondition and invokes Lean on the resulting closed proposition P(input). Checking time depends on the computation required by this proposition and the available execution resources, so a timeout cannot be interpreted solely in terms of input size. Manual inspection confirms that the retained set includes relatively large and structurally constrained inputs:

TaskRetained inputsRelevant precondition checks
verina_advanced_74Lists of length 100Length bounds and element ranges
verina_advanced_13Inputs containing nine chordsChord shapes, endpoint bounds, and global uniqueness of endpoints
verina_advanced_10Inputs containing nine distinct primes and very large natural numbersElementwise primality and duplicate freedom

These examples demonstrate that such inputs are retained, but do not rule out selection bias.

We have updated the logging to preserve candidate values, including those associated with failed checks. We have also introduced proof generation for P(input) or ¬P(input) as a fallback for unresolved precondition checks, with generated proofs checked by the Lean kernel before assigning a label. For future runs, we will record failure categories separately and retry well-typed candidates whose precondition checks remain unresolved using a longer timeout and reduced concurrency. Candidates that remain unresolved will be preserved and reported separately as unknown.

W2: Validation of unexpected outputs.

We agree that a reference-output mismatch can incorrectly label a valid alternative as an unexpected output. We have revised the pipeline to retain an output as unexpected only when the ground-truth postcondition rejects it. Our audit of VerinaLite and VerinaPlus identified affected unexpected outputs in three tasks:

TaskAffected unexpected outputs in VerinaLiteAffected unexpected outputs in VerinaPlus
verina_advanced_27029
verina_advanced_7661163
verina_basic_9022
Total63194

We corrected these entries and re-ran the evaluation. Mean Spec Scores changed from 56.02 to 56.08 for VerinaLite and from 55.09 to 55.22 for VerinaPlus. The main findings remain unchanged. VerinaPlus reduces the mean Spec Score from 61.44 to 55.22, a decrease of 6.22 percentage points. VerinaLite retains approximately 86.19% of this decrease. The corrected results are reported below.

ModelVerinaVerinaLiteVerinaPlus
Claude-Sonnet-4.660.5857.1456.08
Claude-Opus-4.771.6965.0864.29
GPT-5.3-Codex67.7262.9661.90
GPT-5.569.5862.4361.64
Gemini-3.1-Flash-Lite-Preview43.3939.1539.15
Gemini-3.1-Pro-Preview69.8462.4361.90
Qwen3.6-Max-Preview61.1155.8255.03
DeepSeek-V4-Pro47.6243.6541.80
Average61.4456.08 (↓5.36)55.22 (↓6.22)

Spec Scores are reported as percentages and represent the midpoints of the lower and upper bounds. Values in parentheses indicate decreases relative to Verina in percentage points.

We welcome any further questions or concerns and would be happy to provide additional clarification.

Reviewer_WQuGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem that sparse test suites in formal specification benchmarks can allow flawed LLM-generated specifications to pass and overestimate model performance. The authors propose VeriScale, a test-suite scaling framework that expands inputs via LLM seed generation and type-aware mutation, synthesizes adversarial implementations to produce challenging unexpected outputs, and then reduces the test suite using boundary-preserving and set-cover selection. Applying VeriScale to the Lean-based VERINA benchmark yields two expanded variants: VERINAPLUS (an 83x expansion) and VERINALITE (a compact 13x version). Experiments across eight frontier LLMs show that VERINAPLUS lowers the average specification score from 61.44 to 55.09 by exposing additional postcondition and precondition flaws, while VERINALITE retains about 85% of this score drop using only 16% of the expanded tests.

Critical strengths

First, the paper addresses an important problem in verifiable code generation and exposes a real weakness in current benchmarks. In formal verification, the specification is the foundation of the entire trust chain: a formal proof only guarantees that code matches the specification, so if the specification itself is flawed or too loose, the proof means very little. The experiments clearly show that the original VERINA test suites are too sparse and overestimate model capabilities, as all eight evaluated LLMs see a noticeable score drop (from 61.44 to 55.09 on average), especially on postconditions. This serves as a valuable reminder to the community that existing specification evaluation results may be overly optimistic.

Second, the resulting compact benchmark, VERINALITE, is a practical and useful resource for the community. By keeping only about 16% of the tests from VERINAPLUS while retaining roughly 85% of the score drop, it reduces the heavy Lean compilation overhead of the full expansion. Future researchers can directly use VERINALITE as a drop-in benchmark that is much stricter than the original VERINA while keeping evaluation costs manageable, provided that the new tests are confirmed to be valid.

Critical weaknesses

My main concern is that the methodological novelty of VeriScale is relatively limited, and the paper's core adversarial synthesis component brings a very small empirical gain over simple direct LLM generation. The scaling pipeline is similar to the paradigm of EvalPlus [1], especially the seed generation and mutation, mainly adapting to Lean specification evaluation. While the authors claim that synthesizing adversarial implementations to exploit weak specifications is their core contribution, this idea has been explored in prior work such as AlphaVerus [2] and the paper's own ablation study shows that this core adversarial synthesis component yields a negligible improvement in practice. On VERINALITE, when replacing adversarially synthesized unexpected outputs with outputs directly generated by an LLM (VERINALITE-UNEX), the average Spec Score across the eight evaluated models only changes from 56.02 to 56.51 (just 0.50 points). Such a small difference does not match the central role the paper claims for this component.

[1] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. NeurIPS 2023.

[2] Pranjal Aggarwal, Bryan Parno, and Sean Welleck. AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement. ICML 2025.

Questions for authors
  1. Could the authors clarify the core methodological novelty of VeriScale and its key differences compared to prior work such as EvalPlus and AlphaVerus, especially given that the adversarial synthesis component only changes the average Spec Score by 0.50 points over direct LLM generation in the ablation study?
  2. How do the authors ensure the validity of the newly generated tests? For example, does the ground-truth specification pass all new tests in VERINAPLUS and VERINALITE, and how many of the adversarially synthesized unexpected outputs are verified to be truly incorrect?
4 comments

Thank you for your constructive feedback. Your comments helped us clarify VeriScale’s contribution relative to prior work, extend the ablation experiments across test budgets, and strengthen output validation through explicit ground-truth postcondition checks. We detail these clarifications, additional experiments, and dataset corrections below.

W1 & Q1, Part I: Relation to EvalPlus and AlphaVerus.

As acknowledged in Section 3.1, VeriScale’s seed generation and type-aware mutation build on EvalPlus. We also recognize that mutant-based test reduction has established precedents. Our contribution is to couple adversarial output synthesis with reusable test-suite construction and reduction for formal specification evaluation. EvalPlus filters invalid inputs when evaluating implementations. VeriScale labels and retains both valid and invalid inputs to evaluate generated preconditions, and constructs explicit unexpected outputs to evaluate generated postconditions.

AlphaVerus generates adversarial implementations and proofs to expose and filter translations with exploitable specifications. In VeriScale, we execute our synthesized adversarial implementations to obtain concrete negative output tests, then reuse the same implementations to guide test selection. Their outputs become reusable evaluation data, while their failure patterns guide reduction. This produces the fixed VerinaPlus and VerinaLite benchmarks for evaluating generated specifications across models. We will expand Related Work to explicitly discuss AlphaVerus and clarify how our use of adversarial implementations for test construction and reduction relates to its specification-filtering mechanism.

W1 & Q1, Part II: Magnitude of the ablation gains.

We agree that the incremental gains are modest. The unexpected-output ablation retains the same expected input-output pairs and unexpected inputs, changing only how unexpected outputs are generated. We also compare against random reduction to evaluate our combined reduction strategy, which uses boundary-preserving selection for unexpected inputs and reuses synthesized adversarial implementations to select expected input-output pairs.

To examine the effect of test-suite size, we evaluated all three variants at half, standard, and double the VerinaLite budget. The table reports mean midpoint Spec Scores (%) across the eight models.

Budget relative to VerinaLiteVeriScaleDirect LLM unexpected outputsRandom reduction
0.50×57.7458.04 (+0.30)58.42 (+0.68)
1.00×56.0856.51 (+0.43)56.92 (+0.84)
2.00×55.8256.35 (+0.53)56.86 (+1.04)

1. Parentheses indicate increases in percentage points relative to VeriScale at the same budget. 2. Following your feedback and that of the other two reviewers, we checked unexpected outputs against the ground-truth postcondition, removed mislabeled entries, and re-ran the evaluation. The VerinaLite results reported here reflect this correction, as detailed in our response to Q2.

Across these three settings, the observed gaps increase from 0.30 to 0.53 percentage points for adversarial output generation and from 0.68 to 1.04 for the combined reduction strategy. Both ablations retain substantial shared components. In particular, random reduction still uses the expanded test pool and adversarial outputs. Moreover, Spec Score is a task-level conjunctive metric: once a specification fails, additional counterexamples do not lower its score further. Improved test coverage affects the score only when it changes the overall Pass/Unknown/Fail status. These properties help interpret the modest gains, which remain positive in the reported averages at all three tested budgets.

Q2: Validity of generated tests.

In our original pipeline, inputs were validated against the ground-truth precondition: every retained expected input was confirmed to satisfy it, and every retained unexpected input to violate it. Candidates without a definitive verdict were excluded from the labeled benchmark.

Expected outputs were obtained from the ground-truth implementation, while new unexpected outputs were produced by adversarial implementations and required to differ from the reference outputs. However, these output labels were not explicitly checked against the ground-truth postcondition, and reference disagreement does not establish incorrectness when multiple outputs are valid.

We have therefore added explicit ground-truth postcondition checks for both output types. All audited expected input-output pairs passed the check. Across all unexpected-output entries in the two benchmarks, 36,175 entries in VerinaLite and 208,519 in VerinaPlus were verified to violate the ground-truth postcondition, corresponding to 99.83% and 99.91% of their respective totals before correction.

BenchmarkUnexpected outputs before correctionAccepted by GT postcondition and removedRejected by GT postcondition and retainedVerified-invalid fraction
VerinaLite36,2386336,17599.83%
VerinaPlus208,713194208,51999.91%

We removed the invalid unexpected outputs. After human audit, the removed entries occurred only in verina_advanced_27, verina_advanced_76, and verina_basic_90. We corrected the datasets and re-ran the evaluation. The main findings remain unchanged: VerinaPlus reduces the mean Spec Score from 61.44 to 55.22, a decrease of 6.22 percentage points, while VerinaLite achieves 56.08.

ModelVerinaVerinaLiteVerinaPlus
Claude-Sonnet-4.660.5857.1456.08
Claude-Opus-4.771.6965.0864.29
GPT-5.3-Codex67.7262.9661.90
GPT-5.569.5862.4361.64
Gemini-3.1-Flash-Lite-Preview43.3939.1539.15
Gemini-3.1-Pro-Preview69.8462.4361.90
Qwen3.6-Max-Preview61.1155.8255.03
DeepSeek-V4-Pro47.6243.6541.80
Average61.4456.08 (↓5.36)55.22 (↓6.22)

Spec Scores are reported as percentages and represent the midpoints of the lower and upper bounds. Values in parentheses indicate decreases relative to Verina in percentage points.

We hope these clarifications, additional experiments, and validation corrections address your concerns. We welcome any further questions and would be happy to provide additional details.

Thank you for the new experiments. The new ground-truth postcondition check addresses my concern about the validity of the generated tests. However, I cannot raise my score at this point, because my core concern about the method remains. The rebuttal agrees that the pipeline builds on EvalPlus, and the use of adversarial implementations is still close to AlphaVerus. More importantly, the new ablation shows that, under all three test budgets, the adversarial synthesis component still improves only about 0.3-0.5 points over directly generating outputs with an LLM. This suggests that the main value of this work lies in the benchmark rather than in the method.

Dear Reviewer WQuG,

We are pleased to hear that our response has helped address some of your concerns. Thank you again for taking the time to review our manuscript and for sharing your feedback.

Reviewer_kN9aOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of insufficient test cases in existing formal specification evaluation, which makes it difficult to fully identify specification defects. It proposes the VeriScale test suite expansion framework and constructs VERINAPLUS and VERINALITE based on VERINA. The effectiveness of the expanded test suite is validated on eight large language models.

Critical strengths
  1. The authors investigate the correctness of formal specifications themselves, focusing on their ability to distinguish between valid and invalid behaviors, which is a crucial and timely research direction.

  2. The authors construct negative test cases in a targeted manner by generating adversarial implementations that satisfy flawed postconditions while producing incorrect outputs. The paper also provides concrete examples and algorithmic details, presenting a clear overall pipeline.

  3. VERINAPLUS uncovered issues across all eight models that the original test suites failed to expose, while VERINALITE successfully retains most of the detection capability using significantly fewer test cases.

Critical weaknesses
  1. The input generation and mutation techniques in this paper largely follow existing test augmentation methods. Although Section 4.4 provides relevant ablation studies, the overall advantages of adversarial generation and random reduction shown in Table 4 are only about 0.5 and 0.8 percentage points, respectively. In particular, random reduction yields lower Post S scores across seven out of the eight models, which exhibits a discrepancy with the authors' emphasis on postcondition defect detection. Furthermore, the random reduction ablation replaces two reduction strategies simultaneously, making it impossible to isolate and assess the independent contribution of adversary-killing. While the current results support the effectiveness of the complete framework, evidence regarding the necessity of the core design choices and their actual marginal contributions over existing methods remains limited.

  2. While the authors classify abnormal inputs and erroneous outputs across ten tasks in Tables 2 and 3, these statistics primarily reflect the error types of the generated samples rather than establishing their semantic correctness. Specifically, erroneous outputs are constructed based on discrepancies between the adversarial implementations and the reference implementations; for tasks that admit multiple valid outputs, such discrepancies do not necessarily indicate a violation of the original task requirements. Moreover, a decrease in model scores cannot be directly equated with uncovering genuine specification defects.

  3. Using the same set of adversarial implementations for both erroneous output generation and test case selection might bias VERINALITE toward retaining test cases effective only against those specific implementations, potentially missing other types of specification defects. Although the authors evaluate multiple model families, this primarily demonstrates that the current test suite can detect issues in these specific models; it does not sufficiently establish comparable coverage over independently generated adversarial implementations or diverse failure modes. In particular, the fact that Lite preserves most of Plus's score drops does not imply that both uncover the same spectrum of genuine defects. Validating the reduced test suite on independent adversarial implementations would considerably strengthen the claims regarding generalizability.

  4. A primary value proposition of VERINALITE is reducing evaluation costs; however, this benefit is currently demonstrated mainly through test case counts and score variations. Given that the paper relies on Lean for test execution and verification, and employs an isolated, per-test compilation mechanism in Section 4.2, the relationship between the number of test cases and actual computational overhead is not strictly linear. The paper lacks an empirical runtime comparison among different test suites under identical evaluation settings, leaving Lite's practical efficiency gains in real-world evaluations unclear.

Questions for authors

see weakness

5 comments

Thank you for your detailed and constructive feedback. Your comments helped us clarify VeriScale’s contribution, add ablations of each reduction strategy, strengthen output validation, evaluate VerinaLite on independently generated implementations, and quantify its runtime savings over VerinaPlus. We address each concern below.

W1, Part I: Contributions and separate ablations.

As acknowledged in Section 3.1, our seed generation and type-aware mutation build on EvalPlus, and mutant-based reduction has established precedents. Our contribution is to use the same synthesized adversarial implementations both to generate reusable unexpected-output tests and to guide test selection for formal specification evaluation.

We agree that the original joint ablation did not distinguish the contributions of the two reduction strategies. We have added two variants:

  • Random-UI (random1) replaces boundary-preserving selection of unexpected inputs with random selection, while retaining adversary-killing selection of expected pairs.
  • Random-Pairs (random2) replaces adversary-killing selection of expected pairs with random selection, while retaining boundary-preserving selection of unexpected inputs.

Random-Both replaces both strategies. To examine the effect of test-suite size, we evaluated the variants at half, standard, and double the VerinaLite budget. The table reports mean midpoint Spec Scores (%) across the eight models.

Budget relative to VerinaLiteVeriScaleDirect LLM unexpected outputsRandom-BothRandom-UIRandom-Pairs
0.50×57.7458.04 (+0.30)58.42 (+0.68)58.06 (+0.32)58.15 (+0.41)
1.00×56.0856.51 (+0.43)56.92 (+0.84)56.46 (+0.38)56.64 (+0.56)
2.00×55.8256.35 (+0.53)56.86 (+1.04)56.61 (+0.79)56.08 (+0.26)

Parentheses indicate increases in percentage points relative to VeriScale at the same budget.

The full strategy achieves the lowest mean Spec Score at each tested budget. Removing either selection component raises the mean score, supporting a modest average contribution from each component under the tested settings.

We acknowledge that these gains are modest. The direct-output ablation retains the same selected expected pairs and unexpected inputs, while all random-reduction variants retain adversarial output generation. All ablations retain substantial shared components. In particular, random reduction still uses the expanded test pool and adversarial outputs. Moreover, the task-level conjunctive Spec Score does not count additional counterexamples once a specification already fails. These shared components and the metric’s aggregation help interpret the limited magnitude of the gains.

W1, Part II: Post S under random reduction.

Your observation is correct: in the original Table 4, random reduction yields lower Post S scores for seven of the eight models. Adversarial output synthesis directly targets Post S by generating incorrect outputs that flawed postconditions accept. However, the reduction stage selects inputs according to coverage of synthesized adversarial implementations. This selection objective does not guarantee the strongest Post S evaluation on the tested specifications.

The overall Spec Score is a conjunction of four criteria, so its ordering need not match that of Post S alone. The observed average overall-score advantage therefore does not establish consistent improvements in Post S. We acknowledge this limitation; directly incorporating unexpected-output coverage into reduction is a direction for further improvement.

W2, Part I: Validation of unexpected outputs.

We agree that reference-output disagreement does not establish incorrectness when multiple outputs satisfy the specification. Our original pipeline validated retained expected and unexpected inputs against the ground-truth precondition, but did not explicitly validate output labels against the ground-truth postcondition. Expected outputs came from the reference implementation; newly generated unexpected outputs came from adversarial implementations and were required to differ from the reference outputs.

We have added explicit ground-truth postcondition checks for both output types. All audited expected input-output pairs passed the check. The audit of unexpected-output entries yielded the following results:

BenchmarkUnexpected outputs before correctionAccepted by GT postcondition and removedRejected by GT postcondition and retainedVerified-invalid fraction
VerinaLite36,2386336,17599.83%
VerinaPlus208,713194208,51999.91%

The mislabeled entries were confined to verina_advanced_27, verina_advanced_76, and verina_basic_90. We removed these entries and re-ran the evaluation. The main score trends remain unchanged: VerinaPlus reduces the mean Spec Score from 61.44 to 55.22, a decrease of 6.22 percentage points, while VerinaLite achieves 56.08.

ModelVerinaVerinaLiteVerinaPlus
Claude-Sonnet-4.660.5857.1456.08
Claude-Opus-4.771.6965.0864.29
GPT-5.3-Codex67.7262.9661.90
GPT-5.569.5862.4361.64
Gemini-3.1-Flash-Lite-Preview43.3939.1539.15
Gemini-3.1-Pro-Preview69.8462.4361.90
Qwen3.6-Max-Preview61.1155.8255.03
DeepSeek-V4-Pro47.6243.6541.80
Average61.4456.08 (↓5.36)55.22 (↓6.22)

Spec Scores are reported as percentages and represent the midpoints of the lower and upper bounds. Values in parentheses indicate decreases relative to Verina, in percentage points.

W2, Part II: Score decreases and confirmed specification defects.

We agree that a decrease in midpoint Spec Score alone does not establish additional specification defects. For example, a Pass→Unknown transition lowers the midpoint without confirming a violation. Our reported lower and upper bounds make this uncertainty explicit.

As a confirmed example, Claude-Opus-4.7’s postcondition for the longest-common-prefix task incorrectly requires the input strings to differ at every corresponding position after the prefix. For inputs "xyz" and "ayz", the correct output is the empty prefix. We verified in Lean v4.24.0 that the ground-truth postcondition accepts this input-output pair, whereas the generated postcondition rejects it because the second characters are both y. This is a concrete Post C violation, in which the generated specification rejects a correct output.

This example confirms an actual specification defect, but does not quantify how much of the aggregate score decrease is attributable to confirmed defects. We will make this distinction explicit in the paper. Strengthening automated reasoning to reduce unresolved checks and extending validation against the original task requirements remain important directions for future work.

W3: Generalization to independently generated adversarial implementations.

Following your suggestion, we evaluated VerinaLite’s frozen inputs on independently generated implementations. Using GPT-6.1-Sol as a separate generator, we requested three candidate implementations for each specification produced by Claude-Opus-4.7, GPT-5.5, and Gemini-3.1-Pro-Preview. The generator was not given VerinaLite inputs or the adversarial implementations used during benchmark construction.

Of the 1,701 requested candidates, 1,495 implementations were confirmed to produce outputs violating the ground-truth specification. VerinaLite’s frozen inputs exposed such violations in 1,462 of these implementations, yielding a detection rate of 97.79%. This provides evidence that the selected inputs remain effective beyond the implementations used to construct and reduce the benchmark.

We also agree that preserving aggregate Spec Score reductions does not establish equivalent defect coverage: a single counterexample can make a task fail while additional defects remain undetected. This experiment supports generalization to newly generated implementations, while equivalence of defect coverage between VerinaLite and VerinaPlus remains unestablished. We will clarify this distinction in the paper and discuss a comparison of defect-category coverage based on manually audited counterexamples as future work.

W4: Runtime comparison of VerinaLite and VerinaPlus.

We agree that test counts alone do not establish runtime savings. We have therefore measured evaluation runtime across all eight models, with all three benchmarks evaluated under identical hardware and execution settings. VerinaLite reduces mean runtime from 2252.67s for VerinaPlus to 438.03s, corresponding to an 80.56% reduction and a 5.14× speedup. These measurements directly demonstrate the runtime benefit of test-suite reduction and have been added to the revised manuscript.

ModelVerina (s)VerinaLite (s)VerinaPlus (s)
Claude-Sonnet-4.6108.69330.552284.79
Claude-Opus-4.796.44623.233022.22
GPT-5.3-Codex48.58319.072261.33
GPT-5.5252.97510.292576.77
Gemini-3.1-Flash-Lite-Preview122.15480.201783.48
Gemini-3.1-Pro-Preview271.17537.572984.58
Qwen3.6-Max-Preview272.92425.542040.44
DeepSeek-V4-Pro19.66277.751067.78
Average149.07438.03 (2.94×)2252.67 (15.11×)

Multipliers in the Average row are ratios of mean runtimes relative to Verina.

We hope these clarifications, corrections, and additional experiments address your concerns. We welcome any further questions and would be happy to provide additional details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.