- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces LACERTA, a Lean framework for reasoning about WASM binaries, and LACERTA-BENCH, 400 formalized CSES tasks. Models submit a binary with a correctness proof, and accepted submissions are compared by deterministic gas. Six models are tested on 100 tasks with 30-minute budgets and ten tasks with repeated eight-hour attempts. The results distinguish sample passing, proof completion and execution efficiency.
Critical strengths
- Requiring a proof about the submitted WASM module and its byte interface is a useful angle. Keeping the specification and codec fixed allows different implementations to be judged against the same requirement.
- The task formulation is clear. Section 2.1 separates the semantic specification, byte codec and binary behavior, and the Missing Number example helps explain what the model must establish.
Critical weaknesses
- The long-budget evaluation provides limited support for the model comparisons. The 100-task study provides breadth, but the detailed efficiency, cost and repeated-attempt comparisons cover ten tasks. Five runs per task help characterize repeatability, but do not establish how the comparisons extend across the benchmark. Figure 6 provides useful timing information, yet the paper does not summarize each model’s average completion time alongside its completion rate and final gas quality. This makes it difficult to assess the eight-hour allowance and understand the trade-off between completing quickly and producing a better solution. These limitations reduce my confidence in the main empirical comparisons.
- The proposed leaderboard score is not evaluated in the empirical study. Section 3.2 and Eq. (8) define logarithmic per-test scores that are averaged over tests, while the experiments use total and relative gas. Appendix A.6 acknowledges this distinction, but does not show whether it changes the comparisons. These aggregations can produce different orderings. Since the scoring protocol is part of the contribution, leaving its relationship to the reported results unresolved is a substantive gap in the benchmark evaluation.
Questions for authors
- Completion time and quality: What is each model’s mean time to a complete, scoreable solution among successful attempts, with its completion rate reported alongside that average? On shared tasks, how does completion time relate to final gas quality, and do these observations support the eight-hour allowance?
- Task coverage: Using the existing results, what task-level uncertainty can be attached to the ten-task comparisons? How sensitive are the main differences between models to the tasks included?
- Does applying Eq. (8) to the evaluated submissions preserve the main comparisons? Please clarify why total and relative gas were used instead of the proposed leaderboard score.
- Failure interpretation: Do the existing evaluation records show how unproved binaries perform on the full functional tests? Separating incorrect outputs, execution-limit failures and binaries that pass those tests but lack proofs would help explain the sample/proof gap.
4 comments
Question 4.
Failure interpretation: Do the existing evaluation records show how unproved binaries perform on the full functional tests? Separating incorrect outputs, execution-limit failures and binaries that pass those tests but lack proofs would help explain the sample/proof gap.
We appreciate this suggestion, which we found very useful. We ran the final binary of every attempt without an accepted proof through the hidden tests under the same gas cap, and Table 8 separates the outcomes into the three categories you describe. Most unproved binaries pass every hidden test, the remainder mostly exceed the gas cap, and incorrect outputs are rare. Counting proved and unproved binaries together, 84% of eight-hour attempts end with a binary that is correct on every hidden test within the cap, while 63% have an accepted proof. The sample/proof gap therefore consists mostly of correct binaries that lack a proof. We now report this in Section 4.3.
We are thankful again for these precise questions, which have helped us clarify the scope of the experiments and the role of each measure. We hope the revised paper addresses your concerns, and we would welcome your view on whether the distinction between gas and the leaderboard score is now clear. We are happy to discuss any point further.
Review of rebuttal
Thank you for the additional experiments. The hidden-test results and the new timing and uncertainty analyses address several of my earlier concerns.
I still have two questions after reading the revision.
1. The interpretation of the new results.
Table 8 shows that most binaries without accepted proofs still pass the full hidden tests. This seems to be an important finding: the main difficulty is often completing the proof, not producing a working program.
However, the abstract and conclusion remain largely unchanged. I wonder whether the new results should play a larger role in the paper's main argument. The original motivation focuses on the limitations of test-based evaluation, while the new experiments seem to reveal more about the difficulty of proof construction. How do the authors see these findings affecting the paper's main contribution and conclusions?
2. The evaluation choices.
I appreciate the clarification that the ten long-budget tasks were selected rather than randomly sampled. But what were the specific selection criteria? Covering eight categories and a range of solver counts does not fully explain why these ten tasks were chosen, particularly given how much of the analysis depends on them.
I also remain curious whether using the proposed leaderboard score in Eq. (8), instead of total gas, would change the main model comparisons.
I appreciate the revisions and would be interested in the authors' views on these points.
Thank you for the follow-up and for reading the revision closely. We address each point below.
1. The interpretation of the new results.
We agree, and we have revised the paper so that this finding is part of its main argument. We see it as evidence for the original motivation rather than a departure from it. The motivation is that passing tests does not establish correctness. The new results show that current models already write programs that pass the hidden tests in most attempts, but often cannot prove them correct. A test-based benchmark would count most of these programs as successes and, as the new Table 1 shows, would also rank the models differently. The benchmark measures precisely the step from a working program to a proved one, which test-based evaluation cannot observe. The abstract, the contributions, Section 4.3 and the conclusion now state this, and Section 4.2 shows that the same pattern holds across the 30-minute evaluation.
2. The selection of the ten tasks.
The ten problems were selected by hand, before any evaluation was run, with two aims: problems from different CSES sections whose solutions rely on different core techniques, such as dynamic programming, range queries, string matching, maximum flow, geometry and trees, and a range of recorded solver counts. Section 4.1 and Appendix B.2 now state this, and Table 10 now reports each problem's rank in the corpus by recorded solvers, so that the difficulty of the selection can be judged directly. Because the problems were fixed before any results were available, the choice could not favour any system. The 90 problems drawn at random complement this selection, and the main finding of the eight-hour evaluation, that most binaries without a proof still pass every hidden test, holds on them as well (Section 4.2).
3. The leaderboard score of Eq. (8).
We applied Eq. (8) to all 165 scoreable eight-hour artifacts, taking the evaluated cohort as the leaderboard snapshot. Averaged over the 50 attempts of each system, with failures scored zero, it ranks the systems in the same order as the number of scoreable artifacts, and within each problem it orders 94% of pairs of artifacts in the same way as total gas. The main comparisons are therefore unchanged under the leaderboard score.
We thank you again for these questions, which have helped us make the main finding of the paper explicit. We are happy to discuss any point further.
Thank you for the careful review and for your kind words on the task formulation and the worked example. We found your questions precise and constructive, and we have revised the paper accordingly. We answer each point below.
Weakness 1 and Question 1.
We appreciate this point. We agree that the eight-hour comparisons characterize ten tasks, and the revised paper states this more plainly. The 100-problem evaluation provides breadth, while the ten-problem evaluation provides depth through repeated attempts. As requested, Appendix B.6 now reports for each system the number of scoreable artifacts, their mean and median completion time, and their gas quality (Table 10). Completion time and relative gas are essentially unrelated (Spearman ρ=0.05), so longer attempts do not yield more efficient solutions. The eight-hour allowance therefore matters for completion rather than quality, since about a fifth of the scoreable artifacts were completed after four hours, mostly by the systems with lower completion rates. In revising Section 4.1, we also corrected an error in the submitted description. The ten problems were not sampled at random but selected to span eight CSES sections and a range of difficulty, and the ten-task comparisons should be read in that light. No reported number changes.
Question 2.
We appreciate this question. Appendix B.6 now reports paired task-bootstrap intervals for the main comparisons, resampling the ten tasks with replacement while keeping all attempts and systems for each sampled task (Table 11), together with leave-one-task-out checks. The differences among the three leading systems lie within the task-level uncertainty, and removing a single task can change their order. Their separation from DeepSeek and GLM, by contrast, is robust to the choice of tasks. The main comparative claims of the paper are of this second kind, and Section 5.2 now states this limitation explicitly.
Weakness 2 and Question 3.
We appreciate this point and see that the distinction was not stated clearly enough. The quantity the benchmark measures is gas. Eq. (8) is the rule by which the public leaderboard converts gas into a score, so that submissions made at different times can be ranked against an evolving best. The experiments instead compare a fixed cohort of systems, and for this purpose we report gas directly, as total gas and as gas relative to the lowest observed total. Both are derived from the same measurements and differ only in how they are aggregated. Relative gas keeps the comparison in directly interpretable units and shows the full profile across allowances (Figure 5a), which a single averaged score would conceal.