- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces SPECAT, a framework for generating and evaluating formal specifications from natural-language descriptions and behavioral examples. The main motivation is that formal verification can establish whether code satisfies a specification, but not whether the specification itself captures the user's intent.
SPECAT uses multiple agents to evaluate the relationships between descriptions, specifications, examples, and formal tests. It has two refinement loops: one that uses existing behavioral examples and verifier feedback, and another that generates counterexamples to identify specification errors missed by the initial examples.
The authors evaluate the framework on 704 benchmark-backend units across four verification tools. The results show that 391 specifications are endorsed and 77 are repaired. On Codeforces, counterexample-guided refinement improves the number of clean specifications from 46 to 54 out of 88, with hidden-test violations decreasing from 5.5% to 1.1%.
Critical strengths
-
Important problem and useful methodology. The paper addresses an important gap in formal verification: ensuring that generated specifications actually represent natural-language requirements. I like the idea of treating specification evaluation as a separate problem rather than assuming that a verified specification is correct. The distinction between specification errors, test-translation errors, and verifier limitations is particularly useful.
-
Good experimental coverage and evidence of improvement. The evaluation spans multiple benchmarks and verification backends. The counterexample-guided refinement results are promising, particularly the improvement on independent Codeforces hidden tests. The authors also manually audit specifications and explicitly report unresolved cases rather than treating every verifier failure or LLM judgment as conclusive.
Critical weaknesses
-
Reliability of the Spec Critic and intent validation. My main concern is whether the Spec Critic can reliably identify actual mismatches with natural-language intent. Formal verification can confirm that a counterexample violates the generated specification, but it does not independently validate the LLM's interpretation of the original description. The manual audit identifies false-positive defects caused by incorrect interpretations, and some endorsed specifications remain incorrect. It is still unclear how often the critic misses important errors and how much assurance an endorsement provides.
-
Limited isolation of the main improvement and evaluation generalization. The counterexample-guided refinement results are encouraging, but it is difficult to separate the benefits of the methodology from using a stronger critic and additional refinement opportunities. The baseline comparisons are not fully equivalent, since the baselines do not receive counterexample-guided repair. Also, independent held-out evaluation is limited to Codeforces, while the manual endorsement audit covers only a subset of the benchmarks. This makes it harder to assess how well the reported improvements generalize.
Questions for authors
-
How reliable is the Spec Critic at identifying genuine specification errors? In particular, how often does it miss errors or propose counterexamples based on an incorrect interpretation of the description? Can you estimate recall using independently validated defective specifications?
-
How do you distinguish a formally confirmed specification mismatch from an incorrect interpretation of natural-language intent? Since both the Spec Critic and Advisor are LLM-based, could they share the same misunderstanding?
-
How much of the improvement comes from counterexample-guided refinement versus the stronger Spec Critic model? Could you compare against simpler iterative refinement using the same model, budget, and counterexample feedback?
-
The manual audit reports high adequacy for endorsed specifications. How were those cases selected, and how representative are they across task difficulties and verification backends? Could you provide a more systematic evaluation, particularly for Frama-C and Why3?
-
How sensitive are the results to the number and quality of behavioral examples and the refinement budget? It would be helpful to see whether fewer examples or additional iterations substantially change the final outcomes.
We thank the reviewer for the detailed questions and will update the manuscript accordingly.
W1 / Q1. Spec Critic reliability. On Codeforces, the counterexample phase flags 23 of the 28 Loop 1 specifications that fail the hidden suite (82% recall, App. E). Of the 12 endorsed or repaired specifications that still fail, 7 stem from checker conventions absent from the problem text, 3 from repairs missing rare edge cases, 1 from a benchmark–validator disagreement, and only 1 is a genuine detection miss (App. K). Endorsement precision is 48/52 (92%) on the hidden suite and 97/101 (96%) in the manual audit (Q4).
Q2. Confirmed mismatch vs. misreading. Specat has a dedicated check for each common failure mode. Too-weak and over-strict specifications are caught by backend-confirmed counterexamples from the Spec Critic. Convention mismatches (e.g., a null pointer vs. an empty array) are pinned down by seed examples, and evidence unrelated to the specification (an unfaithful test, a timeout) is filtered by the Translation and Semantic Critics. A counterexample the backend cannot decide is labeled UNADJUDICATED, never DEFECT.
Every counterexample makes two claims: what the specification does on a behavior ("it accepts this output") and what the description says about it ("the description forbids this output"). Specat checks the first claim with the backend, never trusting an LLM for it. The second is a reading of natural language that no machine can check, so the Spec Critic and Advisor could in principle share a wrong reading. This is inherent to the problem: since the description is informal, no method can formally prove alignment with intent, and even very recent industrial verification efforts from a frontier AI lab keep human experts reviewing specification [1]. We limit this by design: the Spec Critic uses a different model, is adversarially prompted, and doesn't see the refinement history. Disagreements escalated to human review. The audit shows that shared misreadings are rare (App. I.5) and do not reach a shipped specification, because any repair built on a wrong counterexample must still pass every seed example.
W2 / Q3. Loop vs. stronger critic. App. F holds the writer, Advisor, and budget fixed (the same 88 specifications before and after Loop 2), so the gain from 46 to 54 clean is attributable to the loop. To isolate the critic model, we reran the identical Loop 2 with GPT-5.4 (the writer/Advisor model) as the Spec Critic:
With any critic, 83–85% of passed specifications are clean, compared with 67% for Loop 1 alone, so this gain belongs to the method. The critic's quality determines how much passes: GPT-5.4's low precision (27%, Table 7) yields 26 human-review referrals and only 4 repairs, whereas GPT-5.6-Sol turns similar detection into 26 repairs. We will add this comparison to the final version.
Q4. Audit selection and coverage. The 60 endorsed cases were sampled randomly, 20 each from VT-Dafny, LC-Dafny, and VT-Verus. We have now extended the audit to all 21 endorsed Frama-C specifications and a random sample of 20 endorsed Why3 specifications: 20/21 and 19/20 are adequate, for a total of 97/101 (96%). We will add these results to the final version.
Q5. Sensitivity to examples and budget. Seed examples also anchor the interface design and the interpretation of requirements where the prose is open (boundary cases, conventions, input domain), so their effect is hard to reduce to a single score. In a new experiment on 15 VerifyThis/Dafny units exposing only a few examples, all units still passed Loop 1, at lower cost, but some specifications were weaker, missing constraints the visible examples never exercised. The counterexample loop recovered much of this from the description. However, sparse examples make repair harder than detection, since several repairs may be consistent with both the description and the few visible examples. We will add this experiment to the final version.
Budget. The cap ($2.50 per loop) rarely binds. In Loop 1, the median cost is 0.42 and 90% of units finish under 1.40; caps of 1.50 and 2.00 would retain 92% and 96% of the specifications that pass it. In Loop 2, 0.10 covers 70% of units, 0.50 covers 92%, and 1.50 covers 99%. If needed, we can run a larger budget and assess where it rescues FAILURE units. The final version will include the above stats and a detailed analysis of cost by outcome bucket.
References [1] P. Kohli and S. Chaudhuri. Introducing AlphaProof Code: a breakthrough in verifying complex software at depth. Google DeepMind Blog, November 10, 2026. https://deepmind.google/blog/introducing-alphaproof-code/