- Rating
- 1
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces ULPBench, a benchmark for evaluating autonomous agents on end-to-end formal verification of production numerical C code. Each task requires an agent to construct an executable functional model, prove that the C implementation refines that model under formal C semantics, and prove a specified property, such as a floating-point error bound. The benchmark contains 560 tasks, each available in Lean and Rocq. The authors also provide a Lean port of CompCert's Clight semantics and IEEE-754 floating-point semantics, together with supporting proof automation. Experiments evaluate three agents on a 100-task subset with a five-hour budget per task, reporting a highest end-to-end success rate of 34% in Lean and 32% in Rocq. The paper analyzes stage-level performance and computational costs, and presents case studies in which verification attempts led to identifying numerical defects in CPython and GeographicLib.
Critical strengths
The benchmark focuses on verification tasks drawn from real production C programs. Establishing the correctness of numerical software used in practice is an important problem, and a benchmark grounded in such programs is a meaningful direction for evaluating verification agents.
Critical weaknesses
-
Limited contribution beyond the benchmark. The Lean tooling appears primarily to port and integrate established infrastructure, with limited explanation of its technical novelty. The evaluation and case studies are insufficiently supported by the comparisons and methodological details provided below.
-
Insufficient justification for the proof decomposition. The required executable model and refinement proof may add obligations beyond those needed to establish the target property directly. For the integer square-root example in Figure 1, the intended floor-square-root behavior can be specified by , where and the returned value are nonnegative and the arithmetic is over mathematical integers. This gives a direct contract without real-valued square roots or rounding conversions. The paper does not explain why the model-based route illustrated with Real.sqrt is preferable, or whether this direct formulation would reduce verification effort.
-
Insufficient specification detail. Section 4.1 identifies source comments, documentation, and algorithm-derived claims as specification sources, but does not adequately explain how they were formalized and validated. This also makes it difficult to assess whether the case-study findings establish violations of the intended library behavior. At minimum, the relevant specifications and input assumptions should be presented to allow readers to assess these claims.
-
The evaluation does not separate task difficulty from workflow overhead. Why not let the same agent use VeriFast or Frama-C/WP to express the required behavior as a concise source-level contract and supply the annotations needed to verify it? Such a workflow could avoid an explicit executable model and refinement proof while preserving the target property. Without this comparison on supported tasks under matched budgets and semantic assumptions, failures at refinement may reflect the imposed workflow rather than an inability to verify the requested property.
-
Incomplete experimental reporting. The number of attempts per task and the treatment of failed runs in cost comparisons are unclear. Appendix F (Full Results) and Appendix G (Successes) contain only headings. Per-task outcomes and a worked verification example would make the results easier to assess.
Questions for authors
-
Why require agents to construct an executable functional model and prove refinement, rather than let them verify the same target property directly through source-level contracts using VeriFast、Frama-C/WP or even VST? What benefit does this decomposition provide over the agent-plus-verifier alternative?
-
On tasks supported by both approaches, can the same agent verify properties directly that it fails to establish through the required refinement workflow? A comparison under matched budgets, input domains, target properties, and semantic assumptions would help determine whether the reported refinement bottleneck is intrinsic to the tasks or introduced by the evaluation design.
-
How were the benchmark specifications constructed, and what evidence supports their correctness as formalizations of the intended behavior? Please explain how the claims and input assumptions were derived and independently checked, with a concrete example connecting the original requirement to its formal specification.
2 comments
Do the case-study findings establish violations of intended behaviour?
We agree that the paper did not divulge the full information on these claims, so we state each one exactly in the appendix.
For cmath.atanh, we prove in Lean that our model of the function returns infinity at 1 + 2⁻⁵¹¹i and that the correctly rounded result there is finite. The only assumption is a finite input away from the poles. The proof concerns a Lean model validated against CPython, since complex arithmetic falls outside the C fragment our benchmark covers; we now say this in the text. For GeographicLib, we prove under the Clight semantics, assuming only that the math-library calls return the values glibc returns, that the library's solver exits without meeting its own convergence threshold on a specific input, and returns a specific distance. That this distance differs from the true one by 227 micrometres, against a documented bound of 15 nanometers, is a separate high-precision computation and not part of the proof. Since the initial submission, the author of GeographicLib has confirmed the issue and planned a fix.
Incomplete experimental reporting
Each task was run once per agent and prover. Runs that did not solve the task are excluded from the time and token figures, which plot each solved task at the cost of its successful run.
Limited contribution beyond the benchmark; novelty of the Lean tooling
The benchmark and its evaluation are the primary contribution. The tooling exists because we knew of no infrastructure for verifying C against a formal semantics in Lean. It is a port only at the level of the semantics, which follows CompCert's deliberately, so that the two can be compared definition by definition. The proof layer above it is not ported from Rocq: an executable interpreter for Clight, proved sound against the semantics, and tactics that perform symbolic execution by evaluating it.
Thank you for the detailed review. The revision supplies the missing material you identified, though not the controlled comparison: a worked example, and the specifications and how they were validated. On the decomposition we concede part of your point and defend the rest, below.
Why require a model and a refinement proof at all?
We state a few reasons. The immediate reason is that the extracted Clight AST is often monstrously large and cumbersome to directly reason with for nontrivial functions. This wasn't possible to see in the initial submission as we didn't include many examples nor the dataset, but indeed the median AST (through the typical clightgen generation path) is over 700 lines in our dataset. A model allows for a simplified lean-native object to be used. Also, this model can serve to prove further properties of the same function, or later be composed with other functional models.
Our data do not measure the cost of the no-refinement route. Two observations are relevant. Runs that proved the property of their model but not the refinement are the clearest candidates for a direct proof, and there are two to four per agent and prover. And in the two solved submissions where an agent used the specification's own reference function as its model, the nearest thing in our data to a direct proof, the property became trivial and the refinement proof grew several-fold. We state in Limitations that we ran no direct baseline.
Task difficulty versus workflow overhead: why not Frama-C/WP or VeriFast?
We have not run this comparison, so we cannot answer the second question empirically, and the revision says so in Limitations. A contract-based baseline without a model would be informative, especially for the integer and byte-level half of our tasks, where automated solvers are strong, and we regard it as a valuable next experiment.
How were the specifications built and validated?
We agree this was missing and have added it. Each specification records its source: a statement in the library's own comments or documentation, quoted with its location, or a standard or mathematical definition. Specifications were drafted by AI agents following a written protocol, which the revised AI use statement now says, and validated mechanically. An executable transcription is tested against the compiled C, exhaustively for input spaces up to 2³² and on large samples otherwise. The Lean and Rocq statements are checked for agreement on shared inputs. Every precondition needs an input on which the statement fails without it. A second agent then tries to show the statement false or trivial. We did not review every statement by hand.
As an example, openlibm documents
flooras "Return x rounded toward -inf to integral value". Our property statement is that for every finitexthe result is finite and equals ⌊x⌋; finiteness is its only assumption. The appendix shows this statement and its proof. Fifteen tasks deliberately state a weaker bound than their source claims, and four are false by construction; the appendix identifies both groups.