acceptodds
Under review as a conference paper at ICLR 2027

AuXSpec: Learning to Autoformalize Program Specifications via Executable Postconditions

Abstract

Formal specifications define intended program behavior and underpin verification, but writing them in theorem provers such as Lean is slow and expertise-intensive, making specification a major bottleneck to scaling formally correct code. Autoformalization from natural-language problems is promising, yet training specification models requires scalable feedback on whether generated specifications capture intended behavior. Prior approaches obtain this feedback through explicit proofs or separately implemented code with formal correspondence proofs, both expensive. We introduce AuXSpec, a Lean framework that pairs declarative postconditions with executable counterparts through typeclass resolution. Test cases can then evaluate declarative specifications by execution, letting us use agreement as a reinforcement-learning reward. Using AuXSpec, we synthesize 3,037 LeetCode problems with proved equivalence theorems and expand training to 19.5K samples through expert iteration on Codeforces. Experiments show that training Qwen3.8-27B and Qwen3.5-9B with the execution-based reward raises SpecGen pass@4 on AuXSpec-Test 0.7% 60.3% and 0.7% 40.3%. On the out-of-domain Verina benchmark, both models improve 21.9% 84.7% and 7.0% 69.8%, and the 27B model surpasses VeriSpecGen-30B-A3B, the prior state of the art among open-source specification models, on both pass@1 (47.6% 60.2%) and pass@4 (63.9% at pass@10 84.7%). We release the 3,037 problems with test cases and equivalence proofs, the weights of the trained 9B and 27B models, and the 300-problem AuXSpec-Test benchmark with expert review and correctness proofs for future evaluation.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Specification generation, Autoformalization, Verifiable code generation, Lean, Reinforcement learning, Data synthesis

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_mWZxOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes AuXSpec, a framework that enables test-based feedback for training LLMs to generate formal program specifications in Lean 4. The key idea is to pair declarative postconditions with executable counterparts, allowing test cases to provide automated and reliable reward signals for specification generation. The authors construct a large-scale dataset and train models using supervised fine-tuning and reinforcement learning, achieving substantial improvements on both in-domain and out-of-domain benchmarks.

Critical strengths
  1. Important problem. Automated ways to check the correctness of LLM-generated specifications are crucial to LLM-based automated program verification.

  2. Bringing established PL/FV techniques to the AI community. Executable specifications are a well-established technique in the Programming Languages (PL) and Formal Verification (FV) communities. This is the right way to evaluate LLM-generated specifications and improve LLM-based specification generation. I am fully convinced because execution of specifications is standard in the PL/FV community. I appreciate the paper's merit in bringing this approach to the AI community and integrating it with LLM-based specification generation and training.

  3. Clear and careful presentation. The paper is clearly and carefully written. I particularly appreciate its attention to potential pitfalls and detailed methodological explanations. It makes me feel I'm reading a rigorously written PL article and I enjoy it.

Critical weaknesses
  1. In the end, the paper integrates established techniques from the PL/FV and AI communities, rather than introducing a fundamental breakthrough. This is not to say that the work lacks novelty. Given the substantial gap between these two research communities, bringing well-established PL/FV techniques into the AI community is itself a valuable contribution.
  2. The paper should mention in its Related Work section the long-standing research history of executable specifications in the PL/FV communities.
  3. Figure 1 is too small.
Questions for authors

I have no questions.

3 comments

Thank you for the positive assessment and for pointing us to the history of executable specifications in the PL and formal verification communities. In the revised PDF, the following changes are marked in blue:

  1. the new paragraph “Executable specifications” in the related work (Section 6, page 10);
  2. the sentence of the introduction that points to this history (page 2);
  3. the caption of the redrawn Figure 1 (Section 3, page 4), now points to the detailed example in Figure 12 of Appendix G.

W1: Bringing established PL/FV techniques to LLM specification generation.

We agree with this view of the contribution. AuXSpec applies executable specifications to training models that generate Lean specifications from natural-language problems. The models generate both the declarative and the executable postcondition. Running the executable postcondition on test cases filters the training data and forms part of the reinforcement-learning reward.

W2: Research history of executable specifications.

We have added a paragraph on this history to the related work. A short summary follows, and the full paragraph with all references is in Section 6.

The executability of specifications has long been debated in formal methods. Executable specifications can be validated against intended behavior by execution (Kemmerer, 1985; Fuchs, 1992), whereas declarative specifications are more expressive and independent of any algorithm (Hayes & Jones, 1989).

  • Runtime assertion checking and property-based testing run contracts and properties as tests (Meyer, 1992; Cheon & Leavens, 2002; Claessen & Hughes, 2000), and proof assistants test conjectures before proving them (Berghofer & Nipkow, 2004; Plausible).
  • Boolean reflection pairs a proposition with a Boolean function and a proof that the two agree (Gonthier & Mahboubi, 2010; Paraskevopoulou et al., 2015).
  • Executable counterparts can also be derived automatically from inductive relations in proof assistants, but the derivation covers a restricted class of relations, and users still write executable variants by hand and prove that the two agree (Berghofer et al., 2009; Paraskevopoulou et al., 2022).

AuXSpec combines both sides of this trade-off. Models generate the executable variants, a step that previously required manual curation. In our curated corpus, 35.5% of the executable postconditions come from decidability instances in Lean and agree with the declarative postcondition by construction. For the other 64.5% models generate a separate executable postcondition. The postcondition equivalence theorem then certifies the agreement, as the agreement proof does in Boolean reflection. This theorem is provable as generated for 99.0% of the 3,037 synthesized problems. The revised introduction now points to this connection where the two postconditions are first described.

W3: Figure 1 is too small.

We redrew Figure 1 as a compact diagram of the stages, the frozen inputs and the accepted components. The submitted figure pairs each stage with the input of a running example and the Lean code that the stage produces. It now appears at the start of Appendix G as Figure 12 at the full text width, next to the detailed description of each stage. At this size the figure remains legible (page view).

Thank you again for the thoughtful review. We learned a lot from reading the relevant literature, which made the connection to PL/FV much clearer. Hoping these refinement would address your concerns, and we are glad to discuss them further!

Dear Reviewer mWZx,

Thank you again for the encouraging review and for pointing us to the history of executable specifications, which we learnt a lot through reading. The discussion period ends on November 18 (AoE), and we would like to know whether the revision addresses your comments. The changes are marked in blue in the revised PDF.

W2: Research history of executable specifications

  1. We rewrite the related work paragraph on executable specifications to place AuXSpec within this history (Section 6, page 10). You can read the full paragraph with all references in Section 6.
  2. The introduction now points to this connection where the two postconditions are first described (page 2).

W3: Size of Figure 1

  1. Following your suggestion, we redraw Figure 1 as a compact diagram of the pipeline (Section 3, page 4).
  2. The submitted figure moves to Appendix G as Figure 12 at full text width, next to the description of each stage. A page view shows that the figure stays legible at this size.

W1 and further experiments: the value of executable feedback

Since your review, we have also run more detailed ablations of the RL recipe, which show that test execution provides a necessary training signal (Appendix F.2).

  1. Without test execution, both generation tasks improve more slowly, and CodeGen falls furthest behind, since no other stage of the reward runs the implementation.

A matched run that skips test execution and keeps all other parts of the recipe falls behind AuXSpec-9B-RL by 5.0 and 10.6 points in SpecGen and by 19.5 and 30.6 points in CodeGen. Its reward also accepts implementations that the test cases would reject (49.7% vs. 19.7% of CodeGen rollouts accepted in steps 41 to 50).

  1. With the execution-based reward, SFT on a small dataset followed by RL outperforms SFT alone on far larger data.

We validate this by training the same recipe on Qwen3-4B, the backbone of VeriSpecGen-4B, with 19.5K SFT samples. After SFT our model scores 6.9 SpecGen pass@1 on Verina, and RL improves this score to 54.2. This 4B model outperforms the 30B-A3B models of VeriSpecGen (47.6 pass@1 at best), which train with SFT only on 122K to 344K distilled samples. This comparison supports the effectiveness of executable postconditions for specification generation (Appendix F.1).

We would be happy to discuss anything further before the deadline. If the revision addresses your concerns, we would be grateful if you could consider updating your rating.

Best regards,

The authors

I am satisfied with the authors’ response and will increase my score. In fact, my original inclination was a 4, but I wanted to first consider the opinions of the other reviewers. Given the rebuttal and the overall discussion, I no longer find a reason to maintain a weak accept.

Reviewer_FkY8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

AuXSpec pairs Lean postconditions with executable checks to obtain training feedback for specification generation. The paper builds a 3,037-problem corpus and evaluates trained models on a reviewed 300-problem test set and Verina.

Critical strengths

The corpus construction includes equivalence proofs, plus manual review and implementation-correctness proofs for the test set. The gains over SFT also transfer to Verina without test cases in the prompt. That is stronger evidence than gains on the synthesized test set alone. There is substantial work in making the reference specifications checkable rather than accepting them on compilation alone. The external transfer result matters because it suggests the training benefit is not limited to the particular synthesized test cases.

Critical weaknesses

The corpus has stronger guarantees than newly generated specifications do. Candidates can be accepted without proving their equivalence theorem, so tests and LLM checks must catch the remaining errors. Appendix G.2 finds five false positives in 60 accepted outputs, mostly involving preconditions or declarative requirements. This leaves uncertainty about how often reported successes are actually correct. The audit is useful, and errors found in the reference test set were revised. My concern is specifically the acceptance rate for new outputs.

Several reward stages use LLM judgments. The results show that the combined reward works, but do not establish how much the execution signal itself contributes. The reward-variance analysis supports a training effect, but does not replace a trained control without execution feedback.

Questions for authors

For a sample of accepted outputs, how often can the equivalence theorem be proved? Is there a matched training run that removes the execution stage but keeps the judge stages? For the proof check, it would help to separate failed proof search from an actual counterexample to equivalence.

3 comments
AuthorsOpenReview ↗

Response, (part 1 of 2)

We appreciate the careful reading and the point about the gap between the guarantees of the corpus and those of newly generated specifications. The revision addresses this gap with proofs and an expert review of accepted outputs, and it adds the matched ablation without test execution that you asked for. Since submission we have also explored a stronger RL recipe and enlarged the RL data pool, so the reported models are new. The revised PDF is uploaded, which you can open from the PDF link in this submission, and the changes relevant to your review are marked in blue (Section 2, Section 5.2 and Appendices F.2, H.4 and I.2).

A summary of our latest results is on the anonymous page: https://anonymous.4open.science/r/AuXSpec-Anon-4D79

The reported models

The new AuXSpec-27B-RL reaches 65.7 SpecGen pass@1 on AuXSpec-Test, compared with 35.6 in the submission and 57.7 for the strongest frontier model (Table 4). Without test cases in the prompt, it reaches 84.5 SpecGen pass@1 on Verina, compared with 60.2 in the submission. The proofs and the expert review below cover these models.

W1: Proofs and acceptance precision of accepted outputs

We now validate accepted SpecGen outputs of all 13 models in Table 4 by proving their postcondition equivalence theorems, 1,330 proofs in total (Appendix H.4). For each model and each problem with an accepted output, we draw one accepted output at random. Fixed tactics from the corpus proofs close 557 of the 2,012 drawn outputs. The proving loop of the 3,037 corpus problems then settles every remaining output of the 2 RL models and up to 40 remaining outputs of each other model.

Table R1. Postcondition equivalence theorems of accepted SpecGen outputs on AuXSpec-Test, one output per problem. Accepted indicates the number of problems with an accepted output. Proved, Counterexample and Float indicate the outcomes of the proof attempts, where Float indicates a theorem whose executable postcondition evaluates a Float operation, which Lean treats as opaque, so the theorem can be neither proved nor refuted. Mismatch indicates the estimated percentage of accepted outputs whose two postconditions disagree. For the SFT and frontier models, at most 40 outputs left open by fixed tactics enter the proving loop, so their outcomes do not add up to Accepted.

ModelAcceptedProvedCounterexampleFloatMismatch (%)
Ours
AuXSpec-27B-RL253251200.8
AuXSpec-9B-RL248246110.4
AuXSpec-27B-SFT12273101.8
AuXSpec-9B-SFT7362101.7
Frontier
Claude-Opus-5244133000.0
Claude-Sonnet-511887000.0
DeepSeek-V4-Flash23079000.0
DeepSeek-V4-Pro16390000.0
GPT-5.6-Luna8655102.0
GPT-5.6-Sol17997000.0
Kimi-K3251112000.0
Backbone
Qwen3.5-9B22000.0
Qwen3.8-27B4343000.0
  • Every attempt ends in a proof, a counterexample on a valid input or a Float case, so no output stays undecided for lack of proof effort. All 773 proofs from the proving loop pass SafeVerify, which replays each proof against the original theorem statement.
  • The theorem is provable for 99.2% of the accepted outputs of both RL models (497/501), and the estimated mismatch stays at or below 2.0% for every model.
  • The 6 counterexamples place the error in 3 declarative postconditions, 2 executable postconditions and 1 validity predicate.

A proof establishes that the two postconditions agree but not that they state the problem. For this second question we add an expert review of 100 accepted outputs of the two RL models, 50 per model (Appendix I.2). The reviewers find no error in any postcondition. All 10 rejected outputs miss an input constraint in the validity predicate, so 90.0% of the reviewed successes are correct (90/100). We plan to extend the review to more outputs, and given the short discussion period we report these 100 outputs first.

AuthorsOpenReview ↗

Response, (part 2 of 2)

W2: Matched run without test execution

Is there a matched training run that removes the execution stage but keeps the judge stages?

Yes, we train a variant of the 9B RL recipe from AuXSpec-9B-SFT that removes test execution and keeps everything else. Each rollout is compiled without executing its test cases, and all LLM judge stages stay unchanged. None of the judges reads execution results, so each judge receives the same input as in the full recipe and the comparison isolates the execution signal. The variant reaches a given step after a different number of RL training prompts, so we compare it with AuXSpec-9B-RL at approximately the same number.

Table R2. pass@1 (%) on AuXSpec-Test with and without test execution. SpecGen-Repair indicates the task of revising a rejected specification from compiler or judge feedback. RL training prompts indicates the number of prompts used in RL up to each checkpoint. Bold indicates the better value of each group.

ModelRL training promptsSpecGenCodeGenSpecGen-Repair
About 2.4K RL training prompts
AuXSpec-9B-RL2,14921.843.554.3
w/o test execution2,60416.824.063.0
About 4.1K RL training prompts
AuXSpec-9B-RL4,16234.965.049.6
w/o test execution4,11124.334.466.8

Test execution provides a necessary training signal, and both generation tasks improve more slowly without it. After approximately the same number of RL training prompts, the variant falls behind AuXSpec-9B-RL by 5.0 and 10.6 points in SpecGen and by 19.5 and 30.6 points in CodeGen. Its SpecGen-Repair pass@1 is higher at both points (63.0 vs. 54.3 and 66.8 vs. 49.6), so the gap comes from generating a correct output from scratch.

A closer look at each checkpoint of the variant shows another difference in CodeGen.

Table R3. pass@1 (%) on AuXSpec-Test of the variant without test execution at every checkpoint. Step 0 indicates AuXSpec-9B-SFT.

StepRL training promptsSpecGenCodeGen
0 (SFT)010.113.1
2596112.221.8
501,83313.726.0
752,60416.824.0
1003,36218.128.9
1254,11124.334.4
1504,79631.437.6

Moreover, test execution is essential for CodeGen, since the LLM judges do not run the implementation and are insufficient for this task. CodeGen pass@1 of the variant fluctuates over training. It improves from 21.8 at step 25 to 26.0 at step 50, decreases to 24.0 at step 75 and reaches 37.6 at step 150. Without test execution the reward accepts implementations that the test cases would reject, and about half of the CodeGen rollouts of the variant pass the remaining stages in steps 41 to 50 (49.7% vs. 19.7% with test execution; panel (b2) of Figure 11). CodeGen pass@1 of AuXSpec-9B-RL improves at every checkpoint up to step 210 (43.5, 65.0 and 74.9 at steps 70, 140 and 210; panel (a2)).

Besides test execution, we also run experiments to isolate the LLM judges and self-repair. The conclusions are:

  • Without the LLM judges, the model exploits the reward by restating the given test cases as the specification. We note that SpecGen pass@1 improves quickly in early training, reaches its highest value around step 100 and then decreases. The training dynamics confirm this: in panel (c1) of Figure 11, the fraction of SpecGen rollouts that pass all stages keeps increasing while SpecGen pass@1 (dashed line) decreases.
  • Self-repair provides additional learning signal from each training problem. AuXSpec-9B-RL outperforms the variant without self-repair in generation at all 3 comparison points, even with fewer RL training prompts. Beyond this gain, the repair tasks are a key part of our design, in which a model revises its output from feedback over several rounds. On 75 sophisticated problems of ICPC-Eval, refinement for up to 16 rounds outperforms 16 independent samples (82.7% refine@16 vs. 61.3% pass@16; Appendix D.5).

For the tables and details, please refer to our response to Reviewer aGL7 (part 2 of 2) or Appendix F.2.

We hope these results answer your questions, and we are glad to discuss them further.

Dear Reviewer FkY8,

Thank you again for your thoughtful review, and in particular for pointing out the gap between the guarantees of the corpus and those of newly generated specifications. The discussion period ends on November 18 (AoE), and we would like to know whether our responses address your concerns. A short recap:

W1: How often accepted outputs are correct

  1. The test cases and LLM judges rarely accept a specification whose two postconditions disagree. The equivalence theorem is provable for 99.2% of the accepted outputs of both RL models, and the estimated mismatch stays at or below 2.0% for every model.

    Beyond the test cases and LLM judges, we now prove the postcondition equivalence theorems of newly generated specifications. For each of the 13 models in Table 4, we take one accepted output per problem, 1,330 proofs in total (Appendix H.4). To separate failed proof search from actual disagreement, we continue each attempt until the theorem is either proved or refuted by a counterexample on a valid input. The only exceptions are theorems whose executable postcondition evaluates a Float operation, which are opaque to Lean.

  2. The accepted postconditions also state the intended problems, and the remaining errors lie in the input constraints.

    Two experts review 100 accepted outputs of the two RL models and find no error in any postcondition (Appendix I.2). The 10 outputs they reject all miss an input constraint in the validity predicate.

W2: What each part of the RL recipe contributes

  1. Test execution. Without test execution, both generation tasks improve more slowly, and CodeGen falls furthest behind, since no other stage of the reward runs the implementation.

    We isolate the execution signal with a matched run that keeps the judge stages but skips test execution. This run falls behind AuXSpec-9B-RL by 5.0 and 10.6 points in SpecGen and by 19.5 and 30.6 points in CodeGen. Its reward also accepts implementations that the test cases would reject (49.7% vs. 19.7% of CodeGen rollouts accepted in steps 41 to 50).

  2. LLM judges. Without the judges, the reward can be exploited by restating the given test cases as the specification.

    A second variant drops the LLM judges and keeps the rest of the recipe unchanged. Its training reward keeps increasing while SpecGen pass@1 drops after step 100 (35.8 at step 100 vs. 20.0 at step 150).

  3. Self-repair. The repair tasks keep refinement from feedback in the training objective without costing generation performance.

    A third variant without the repair tasks falls behind AuXSpec-9B-RL in generation at all 3 comparison points, even though AuXSpec-9B-RL uses fewer RL training prompts.

You can find the tables in Appendix F.2 and the training dynamics of all three variants in Figure 11, which shows where the training reward and the evaluation scores of each variant diverge.

Comparison on a shared backbone

With the execution-based reward, SFT on a small dataset followed by RL outperforms SFT alone on far larger data.

We validate this by training the same recipe on Qwen3-4B, the backbone of VeriSpecGen-4B, with 19.5K SFT samples. After SFT our model scores 6.9 SpecGen pass@1 on Verina, and RL improves this score to 54.2. This 4B model outperforms the 30B-A3B models of VeriSpecGen (47.6 pass@1 at best), which train with SFT only on 122K to 344K distilled samples (Appendix F.1).

We would be happy to answer any remaining questions before the deadline. If our responses address your concerns, we would be grateful if you could consider updating your rating.

Best regards,

The authors

Reviewer_aGL7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AuXSpec studies autoformalization of program specifications in Lean from natural-language descriptions, constraints, and tests. It pairs declarative postconditions with executable Boolean checkers, uses execution and LLM judgments to synthesize training examples, and trains models through supervised fine-tuning and a progressive reinforcement-learning reward. The paper reports 3,037 formalized LeetCode problems with equivalence proofs and a 300-problem reviewed benchmark with implementation correctness proofs. The reported results improve on both AuXSpec-Test and Verina, although differences in backbones and training setups leave the method's advantage over prior work unclear.

Critical strengths
  • Separating a declarative postcondition from an executable checker is a useful design for obtaining scalable feedback while retaining a specification suitable for verification. The staged synthesis, frozen tests, local repair, and reference equivalence proofs make the proposed pipeline concrete and technically substantial.
  • The dataset and validation effort are substantial. Human review and correctness proving strengthen the benchmark references, and the paper openly documents errors missed by automatic checks. The backbone-to-SFT and SFT-to-RL gains at both model sizes, including transfer to Verina without prompt tests, provide meaningful evidence that the overall training pipeline is effective (Tables 4–5).
Critical weaknesses
  • The comparison with the closest baseline, VeriSpecGen, does not control the backbone or training resources. AuXSpec uses Qwen3.5-9B and Qwen3.8-27B, whereas VeriSpecGen uses Qwen3-4B and Qwen3-Coder-30B-A3B (Section 5.1, Table 5). These results compare trained systems but do not isolate the advantage of the proposed method.
  • The specific contribution of executable feedback is insufficiently isolated. The RL recipe combines execution, alignment, LLM judgments, self-repair, and optimization changes. Section 5.4 and Appendix D.3.1 show that progressive scoring increases reward variation, but do not provide a matched downstream training ablation separating executable feedback from the other ingredients.
  • Reference proof coverage does not extend automatically to newly generated specifications, whose acceptance relies on finite tests and LLM judges. The human audit confirms 55 of 60 accepted outputs, which is useful evidence but does not yield human-corrected benchmark scores. Moreover, the audit uses the 9B RL checkpoint at step 120, while the reported model is selected at step 80 (Appendices D.3 and G.2). This limits validation of semantic correctness for the reported checkpoint.
Questions for authors
  • Could the authors compare AuXSpec with VeriSpecGen on a shared backbone, under comparable source-problem and training budgets and a common evaluation protocol?
  • Could the authors provide matched RL ablations isolating executable feedback from LLM judgments and self-repair, and report downstream specification-generation performance?
  • Could the authors audit the reported 9B RL checkpoint at step 80 and report its acceptance precision?
3 comments
AuthorsOpenReview ↗

Response, (part 1 of 2)

Thanks for the careful and constructive review. We acknowledge that the contribution of the components in the RL recipe was not sufficiently distinguished, and we have run new experiments to isolate it.

Since submission we have explored a stronger RL recipe, enlarged the RL data pool to 8,579 problems and retrained both models, which we summarize first. We answer W1/Q1 and W3/Q3 below and W2/Q2 in the next comment. The revised PDF is uploaded, which you can open from the PDF link in this submission, and the changes relevant to your review are marked in blue.

A summary of our latest results is in the anonymous page: https://anonymous.4open.science/r/AuXSpec-Anon-4D79

Update since submission

AuXSpec-27B-RL now outperforms all 7 frontier models on AuXSpec-Test in SpecGen and CodeGen. Without test cases in the prompt, both variants also outperform VeriSpecGen-30B-A3B on Verina (84.5 and 73.7 vs. 47.6 SpecGen pass@1). We fix issues in the evaluation pipeline like answer extraction and re-evaluate all models. The full results can be viewed in Table 4.

Table R1. Latest results of both RL models in pass@1/pass@4 (%). ID indicates AuXSpec-Test and OOD indicates Verina with test cases in the prompt. Best indicates the strongest frontier model in each column. Bold indicates the best value overall.

ModelID SpecGenID CodeGenOOD SpecGenOOD CodeGen
Submission
27B-RL35.6/60.338.1/58.375.9/92.677.0/94.7
9B-RL20.2/40.330.2/50.753.8/81.075.8/93.1
Revision
27B-RL65.7/84.394.8/99.791.9/96.891.8/93.7
9B-RL58.6/82.777.1/96.087.0/96.892.6/98.4
Frontier
Best57.7/83.793.0/98.386.2/97.956.6/79.4

W1/Q1: Shared backbone with VeriSpecGen

We fix the backbone to Qwen3-4B-Instruct-2507 (the backbone of VeriSpecGen-4B), train it with our SFT data and RL recipe, then evaluate it on Verina with the default setting reported in the paper.

Table R2. pass@1/pass@4 (%) on Verina without test cases in the prompt. Best VeriSpecGen indicates the strongest of the 4 reported variants of each size, as their pass@1/pass@10. Bold indicates our values above every VeriSpecGen variant and italic those above every VeriSpecGen-4B variant only, with our pass@4 compared with their pass@10.

ModelSpecGenCodeGen
AuXSpec on Qwen3-4B-Instruct-2507
Before training0.0/0.00.4/0.5
SFT6.9/24.921.6/54.5
RL step 2023.0/54.550.3/79.4
RL step 4032.9/59.857.9/85.2
RL step 6039.3/68.864.8/86.8
RL step 8046.6/75.767.3/91.5
RL step 10050.7/78.874.3/92.6
RL step 12054.2/84.177.2/95.2
Best VeriSpecGen
4B30.3/45.243.5/72.7
30B-A3B47.6/63.961.2/84.0

From step 40 our model outperforms every VeriSpecGen-4B variant in all 4 tasks. From step 100 it outperforms every VeriSpecGen variant including 30B-A3B, and both scores still improve at step 120.

On the shared backbone, SpecGen pass@1 is 0.0 before training (VeriSpecGen reports 14.7 with its own harness) and 6.9 after SFT. VeriSpecGen trains with SFT only, since its declarative specifications cannot be checked on test cases during RL. We check our specifications through the executable postcondition, and RL improves SpecGen pass@1 from 6.9 to 54.2. Our SFT data contain 19.5k samples, compared with 121k for the No-Test models of VeriSpecGen.

Appendix F.1 compares all 8 VeriSpecGen variants on Verina, and reports the in-domain results.

W3/Q3: Audit of the reported checkpoints

The revision reports the final checkpoints after 1 epoch of RL, step 208 for AuXSpec-27B-RL and step 434 for AuXSpec-9B-RL, in place of the submitted step-80 checkpoint, so no checkpoint selection is need. Two additional experts review 100 accepted SpecGen outputs of these checkpoints, and the review protocol is in Appendix I.2. They confirm 90% of the outputs (43/50 for AuXSpec-27B-RL and 47/50 for AuXSpec-9B-RL), and agree on all overlapping problems.

Every declarative and executable postcondition is correct. All 10 errors are input constraints missing from the validity predicate, such as the guarantee that the answer fits in 32 bits. We plan to extend the review to more outputs, and given the short discussion period we report these 100 first. Section 5.2 now reports this acceptance precision, and Appendix J discusses validity predicates at constraint boundaries.

We also validate that the postcondition equivalence theorem is provable for 99.2% of the accepted outputs of both models (497/501; Appendix H.4).

AuthorsOpenReview ↗

Response, (part 2 of 2)

W2/Q2: Isolating the execution signal from the other components

We train all variants from AuXSpec-9B-SFT to ablate components of the RL recipe, each with the same RL training data, order as AuXSpec-9B-RL. The variants reach a given step after different numbers of RL training prompts, so we compare them at approximately the same number (~).

(1) Test execution. Each rollout is compiled w/o executing its test cases, and all other component, including LLM judges stay unchanged.

Table R3. pass@1 (%) on AuXSpec-Test with and without test execution. RL training prompts indicates the number of prompts used in RL up to each checkpoint. Bold indicates the better run of each group.

ModelRL training promptsSpecGenCodeGen
~2.4K RL training prompts
AuXSpec-9B-RL2,14921.843.5
w/o test execution2,60416.824.0
~4.1K RL training prompts
AuXSpec-9B-RL4,16234.965.0
w/o test execution4,11124.334.4

Test execution provides a necessary training signal, and both tasks improve more slowly without it. After approximately the same number of RL training prompts, the variant falls behind AuXSpec-9B-RL by 5.0 and 10.6 points in SpecGen and by 19.5 and 30.6 points in CodeGen.

A closer look at each checkpoint of the variant shows another difference in CodeGen.

Table R4. pass@1 (%) on AuXSpec-Test of the variant at every checkpoint. Step 0 indicates AuXSpec-9B-SFT.

StepRL training promptsSpecGenCodeGen
0 (SFT)010.113.1
2596112.221.8
501,83313.726.0
752,60416.824.0
1003,36218.128.9
1254,11124.334.4
1504,79631.437.6

Note CodeGen pass@1 of the variant fluctuates. Therefore we believe test execution is essential for CodeGen, since the LLM judges do not run the implementation which are insufficient for this task. It improves from 21.8 at step 25 to 26.0 at step 50, decreases to 24.0 at step 75 and reaches 37.6 at step 150. The reward then accepts implementations that the test cases would reject: about half of the CodeGen rollouts pass the remaining stages in steps 41 to 50 (49.7% vs. 19.7% with test execution; panel (b2) of Figure 11). AuXSpec-9B-RL improves at every checkpoint (43.5, 65.0 and 74.9 at steps 70, 140 and 210; panel (a2)).

We also ablate the LLM judges and self-repair, with the training dynamics of all three variants in Figure 11 of Appendix F.2.

(2) LLM judges. Without them, the model exploits the reward by restating the given test cases as the specification. We note that the training reward keeps increasing, while SpecGen pass@1 reaches its highest value around step 100 and then decreases (35.8 at step 100 vs. 20.0 at step 150; panel (c1) of Figure 11). In steps 141-150, 84.7% of SpecGen rollouts are accepted.

Table R5. pass@1 (%) on AuXSpec-Test without LLM judges, with AuXSpec-9B-RL after a similar number of RL training prompts.

ModelRL training promptsSpecGenCodeGen
w/o LLM judges1,62615.140.2
w/o LLM judges3,03035.867.0
w/o LLM judges4,55620.054.8
AuXSpec-9B-RL4,16234.965.0

(3) Self-repair. The repair tasks are designed for settings in which a model revises its output from feedback over rounds, such as long-horizon verification and collaboration between agents, where we expect this mode to be useful. During training, the data are built from failed rollouts, so each problem has chance to yields repair tasks. Refinement from feedback thus stays in the training objective, so that improving generation wouldn't harm the side repair task. Without self-repair, we fill the batch slot with new RL training prompt, so the process consumes data faster. AuXSpec-9B-RL still outperforms it in generation at all 3 comparison points, even with fewer RL training prompts.

Table R6. pass@1 (%) on AuXSpec-Test with and without self-repair, with RL training prompts as in Table R3. Bold indicates the better run of each group.

ModelRL training promptsSpecGenCodeGen
~2.3K RL training prompts
AuXSpec-9B-RL2,14921.843.5
w/o self-repair2,39614.726.2
~4.2K RL training prompts
AuXSpec-9B-RL4,16234.965.0
w/o self-repair4,34623.053.7
~6.2K RL training prompts
AuXSpec-9B-RL6,18646.774.9
w/o self-repair6,33443.170.2

To assess the value of the repair capability, we evaluate AuXSpec-27B-RL on 75 sophisticated problems of ICPC-Eval. The results show that refinement from compiler and judge feedback for up to 16 rounds outperforms 16 independent samples (82.7% refine@16 vs. 61.3% pass@16; Appendix D.5). We provide 10 refinement traces on ICPC problems.

We are glad to discuss any point further.

I appreciate the authors for running comprehensive additional experiments and providing detailed analysis. All my concerns are addressed and I am glad to have raised my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their thoughtful and constructive feedback. We are encouraged by the positive comments, including:

  • Design. Pairing a declarative postcondition with an executable checker gives scalable feedback while keeping a specification suitable for verification (aGL7, mWZx).
  • Data and validation. The dataset and its validation, with equivalence proofs, expert review and correctness proofs, are a substantial effort (aGL7, FkY8).
  • Transfer. The gains on Verina without test cases in the prompt show that the training benefit is not limited to the synthesized test cases (aGL7, FkY8).
  • Positioning and presentation. Executing specifications is the right way to evaluate and improve LLM-generated specifications, and the presentation is careful (mWZx).

Response to the Key Concerns

  1. Updated models since submission (all reviewers). With a stronger RL recipe and a larger RL data pool, the new AuXSpec-27B-RL outperforms all frontier models on AuXSpec-Test in SpecGen and CodeGen, and both RL models outperform VeriSpecGen-30B-A3B on Verina.
  2. Comparison on a shared backbone (aGL7). We trained our recipe on the backbone of VeriSpecGen-4B. RL with the execution-based reward accounts for most of the gain, and the resulting 4B model outperforms every VeriSpecGen variant, including the 30B-A3B models, with far less data point.
  3. Contribution of the execution signal (aGL7, FkY8). We ran matched ablations of 1) test execution, 2) LLM judges and 3) self-repair. Without test execution, CodeGen performance becomes unstable during training, since the reward no longer runs the implementation and accepts impls that the test cases would likely reject. Without the LLM judges, the reward can be exploited by restating the given test cases as the specification, and without self-repair, generation also falls behind.
  4. Correctness of accepted outputs (aGL7, FkY8). We proved the postcondition equivalence theorems of accepted outputs from all evaluated models, and every attempt ends in a proof or a counterexample except for theorems over Float operations. The two postconditions agree for almost all accepted outputs of the RL models. An expert review of accepted outputs finds no error in any postcondition, and the remaining errors are input constraints missing from the validity predicate.
  5. Reported checkpoints (aGL7). The revision reports the final checkpoints after 1 epoch of RL, and the proofs and expert review above cover these checkpoints.
  6. Positioning and presentation (mWZx). We added a new related-work paragraph on the history of executable specifications in the PL/FV communities and a pointer to it in the introduction. Also, Figure 1 is redrawn, and the previous figure was enlarged at full text width in Appendix G.

All our revisions are marked in blue in the revised PDF.

Again, we would like to thank all reviewers for their valuable time and effort in reviewing our manuscript, your suggestions were really instrumental in helping us improve our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.