- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces VeriScale, a framework for strengthening evaluation of LLM-generated preconditions and postconditions. It combines language-model seed generation, type-aware input mutations, reference-based input classification, and adversarial implementations that exploit weaknesses in candidate specifications to produce challenging outputs. Applied to VERINA, the approach produces Plus and Lite variants with approximately 83 and 13 times the original number of tests. Evaluation across eight models and 189 tasks lowers the average reported Spec Score from 61.44 to 55.09 with Plus. Lite retains 85.45% of this score reduction using approximately 16% of the Plus tests.
Critical strengths
-
Effective benchmark augmentation. All models evaluated on Plus and Lite show a clear performance drop compared to the original VERINA, indicating that the benchmark does become relatively harder after being augmented.
-
Novel synthesis of unexpected outputs targeting the weaknesses of the spec. Prior work tends to generate outputs via random sampling, which is relatively inefficient; the proposed method can perform targeted hacks against the specification.
Critical weaknesses
-
Potential bias toward simple inputs. The Precondition-Guided Input Classification part of Section 3.1 states that "Candidates that still lack a definitive verdict due to timeouts or search exhaustion are permanently excluded from the final test suite." This may leave mostly simple inputs in the final suite, since for complex inputs it is hard for Lean's evaluation alone to decide whether the precondition is satisfied—thus biasing the test set. The authors could mitigate this by using a model to generate proofs that the inputs satisfy the precondition.
-
The construction of unexpected outputs is too strict. An output that differs from the ground-truth code's output is not necessarily incorrect. For example, in a task that allows returning the index of any maximum value, multiple indices may all satisfy the requirement when several maxima exist; likewise, the list representation of an unordered collection may admit multiple valid orderings. Such overly strict tests may be the reason why models' performance drops on the augmented benchmark, and the authors need to clarify this—since Plus and Lite contain far more test cases than the original benchmark, they may inadvertently introduce a large number of overly strict tests.
Questions for authors
See weaknesses.
W1&W2
We thank you for these two constructive comments. They helped us improve input logging, introduce a fallback using Lean-checked proofs for unresolved precondition checks, and correct the benchmark data. We report these changes and the updated evaluation results below.
We agree that discarding inputs whose precondition checks remain unresolved may introduce selection bias. Our logs report 97,990 classified candidates, of which 75,499 satisfy and 22,491 violate the ground-truth precondition, together with 2,922 unsuccessful validation checks. These failures include timeouts and other checking errors. Our previous implementation did not preserve their candidate values or individual failure causes, so we cannot replay these historical cases or determine how many failed due to timeouts.
Our classifier substitutes each concrete input into the ground-truth precondition and invokes Lean on the resulting closed proposition
P(input). Checking time depends on the computation required by this proposition and the available execution resources, so a timeout cannot be interpreted solely in terms of input size. Manual inspection confirms that the retained set includes relatively large and structurally constrained inputs:verina_advanced_74verina_advanced_13verina_advanced_10These examples demonstrate that such inputs are retained, but do not rule out selection bias.
We have updated the logging to preserve candidate values, including those associated with failed checks. We have also introduced proof generation for
P(input)or¬P(input)as a fallback for unresolved precondition checks, with generated proofs checked by the Lean kernel before assigning a label. For future runs, we will record failure categories separately and retry well-typed candidates whose precondition checks remain unresolved using a longer timeout and reduced concurrency. Candidates that remain unresolved will be preserved and reported separately as unknown.We agree that a reference-output mismatch can incorrectly label a valid alternative as an unexpected output. We have revised the pipeline to retain an output as unexpected only when the ground-truth postcondition rejects it. Our audit of VerinaLite and VerinaPlus identified affected unexpected outputs in three tasks:
verina_advanced_27verina_advanced_76verina_basic_90We corrected these entries and re-ran the evaluation. Mean Spec Scores changed from 56.02 to 56.08 for VerinaLite and from 55.09 to 55.22 for VerinaPlus. The main findings remain unchanged. VerinaPlus reduces the mean Spec Score from 61.44 to 55.22, a decrease of 6.22 percentage points. VerinaLite retains approximately 86.19% of this decrease. The corrected results are reported below.
Spec Scores are reported as percentages and represent the midpoints of the lower and upper bounds. Values in parentheses indicate decreases relative to Verina in percentage points.
We welcome any further questions or concerns and would be happy to provide additional clarification.