- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the generation of loop invariants without providing the target assertion to the model. CRAFT decomposes model responses into clauses, aggregates them and filters them through Houdini, then combines the surviving clauses to complete verification. Training is designed around this pipeline as well: SFT uses labels obtained by composing multiple responses, while RL assigns rewards based on the ability of surviving clauses to rule out candidate negative traces. Experiments show that this training improves the verification success rate after composition, even when the independent success rate of the model's full responses declines.
Critical strengths
- The paper identifies a meaningful problem: if the system ultimately uses only the valid fragments of a response, then assigning rewards based on whether the entire response is correct may discard useful training signal. The authors carry this idea through inference, SFT, and RL, making the overall approach fairly coherent.
- The main training gains are quite clear. For Qwen3-8B, compose@1 rises from 37.86% to 63.94% after SFT and further to 69.23% after RL. Multiple models show similar trends, so the results do not hinge on a single model.
- The reward ablation provides valuable observations. Removing the constraint that a reward is given only when all clauses pass leads to a marked improvement in compositional verification. At the same time, the authors report different trends for pass and compose, rather than cherry-picking only the metrics that favor their method.
Critical weaknesses
- The SFT comparison alone is not yet sufficient to isolate the effect of compositional labels. The paper only states explicitly that the two training sets have approximately the same number of samples, but the compositional labels result from multiple rounds of sampling and coverage filtering. It is currently unclear whether the single-response labels were produced with the same procedure, the same generation budget, and the same filtering criteria. As a result, the gap between 63.94% and 23.20% may reflect the combined influence of search budget and label quality.
- The comparison with Loopy is affected by parser issues. The appendix reports that 98.8% of its initial responses contain invariant annotations, yet only 1.75% are passed by the extractor to Houdini. Under these circumstances, a lower verification rate can hardly be taken as direct evidence that its reasoning method is inferior to CRAFT. At a minimum, a result obtained after fixing the parsing adapter would be needed to judge the actual gap between the two.
- The additional benefit of the Shapley term remains uncertain. After SFT, it raises compose@1 from 67.91% to 69.23%, a gain clearly smaller than that brought by the clause reward itself. In the absence of repeated training runs, I would be cautious about the conclusion that it reliably improves exploration. Moreover, the trained 8B model still reaches only 12% on the nonlinear subset, indicating that the overall improvement does not yet extend to such difficult tasks.
Questions for authors
- Could you compare, on the same batch of training programs and with the same candidate generation budget, "selecting the single response with the highest coverage as the label" against "composing multiple responses as the label"? This would more directly show how much of the SFT improvement comes from composition itself.
- Why do most of Loopy's responses fail to reach Houdini? How much would the results change after fixing the format adapter or using a shared extractor? I do not think the baseline needs to be redesigned, but the pervasiveness of parsing failures undermines the persuasiveness of the current comparison.
- There are two reporting points I find somewhat confusing and would like clarified. First, does the SFT (single) in K.2 also use the same coverage threshold as the compositional labels? Does "approximately the same number of training samples" mean that both groups use the same batch of programs? Second, in Table 23, GPT-5-nano's pass@1 is 3.80% on Linear and 3.86% on Loopy, whereas in Table 25 it is 2.53% and 4.72%; both tables report an overall value of 3.61%, and the text states that the same sample pool is used. Is this a typo in one of the tables, or do the two evaluations differ in how they are processed?
6 comments
Response to Reviewer 3 (2/3): Loopy and reporting clarifications
Q2 — Why do most of Loopy's responses fail to reach Houdini? How much would the results change after fixing the format adapter or using a shared extractor? I do not think the baseline needs to be redesigned, but the pervasiveness of parsing failures undermines the persuasiveness of the current comparison.
Thank you for highlighting the extraction issue. Loopy's extractor dropped ACSL annotations without paired Markdown fences before Houdini. Correcting this raises its verification rate from 18.75% to 68.03%.
To align the prompting setup with Loopy, we use Loopy's prompt in the supplementary GPT-5-nano/minimal evaluation on all 832 programs.
| Method | Rate | Tokens/program |
|---|---|---|
| CRAFT @1 | 37.86% | 928 |
| CRAFT @4 | 54.69% | 1,812 |
| CRAFT @8 | 63.10% | 3,610 |
| CRAFT @16 | 68.75% | 7,233 |
| CRAFT trained @1 (Qwen3-8B + SFT + RL) | 69.23% | 1,348 |
| Loopy, corrected initial composition (15 responses) | 66.47% | 6,904 |
| Loopy, corrected initial + repairs (15 initial + up to 7 repair responses) | 68.03% | 10,016 |
At nearly equal recorded token usage, CRAFT@16 reaches 68.75% versus 66.47% for Loopy's initial composition. It also exceeds Loopy including repairs by 0.72 percentage points with 27.8% fewer recorded tokens. In this target-hidden evaluation, the existing repair responses add 13 programs (1.56 points) beyond initial composition; their incremental benefit is much smaller than that of composition.
Our core contribution is composition-aligned training to improve search efficiency. With composition-aligned training, Qwen3-8B reaches 63.94% compose@1 after SFT and 69.23% after RL, both exceeding its compose@32 of 57.81% before our SFT/RL training, using just one response.
Q3 — There are two reporting points I find somewhat confusing and would like clarified.
Q3.1 — Second, in Table 23, GPT-5-nano's pass@1 is 3.80% on Linear and 3.86% on Loopy, whereas in Table 25 it is 2.53% and 4.72%; both tables report an overall value of 3.61%, and the text states that the same sample pool is used. Is this a typo in one of the tables, or do the two evaluations differ in how they are processed?
Thank you for identifying this reporting error. The original GPT-5-nano pass@1 results in Tables 23 and 25 came from independent runs, contrary to the shared-pool description. We apologize for this inconsistency and the confusion it caused.
While reconciling the experimental records, we also found that the historical GPT-5-nano batch used an earlier prompt variant that lacked some ACSL syntax instructions and examples included in the paper’s reported prompt. Our checks confirmed that evaluations of the other API models and the trained models used the prompt reported in the paper. We have therefore reevaluated GPT-5-nano on the full 832-program benchmark using the reported prompt and updated the affected tables, figures, and accompanying text. Both tables now report results from the same corrected response pool. These corrections preserve the paper’s main conclusions.
Q3.2 — First, does the SFT (single) in K.2 also use the same coverage threshold as the compositional labels? Does "approximately the same number of training samples" mean that both groups use the same batch of programs?
Thank you for requesting this clarification. SFT (single) does not use the same coverage threshold. Both datasets start from exactly the same input-loop pool and use the same SFT settings. Composed labels use multi-round composition, filtering, and coverage-based acceptance; SFT (single) generates one response per loop, removes incorrect clauses, and skips examples with none remaining.
"Approximately the same number of training samples" refers to the resulting dataset sizes. The procedures retain slightly different inputs: over 93% of the distinct prompts in the composed-label dataset also appear in the single-response dataset. We will clarify this distinction between the original program pool and the retained training sets in K.2.
Response to Reviewer 3 (3/3): Nonlinear tasks and Shapley credit
Q4 — Moreover, the trained 8B model still reaches only 12% on the nonlinear subset, indicating that the overall improvement does not yet extend to such difficult tasks.
Thank you for highlighting the limited performance on the nonlinear subset. Qwen3-8B improves from 10% at compose@32 before our training to 12% at compose@1 afterward. On the same subset, CRAFT with GPT-5-nano/minimal reaches 18% at compose@1 and 44% at compose@16.
Our training relies on model-generated candidates, so missing nonlinear relations limit both composition and useful training supervision. The results support this explanation and show that CRAFT can solve more nonlinear tasks with stronger-model candidates, motivating models with better nonlinear reasoning.
Q5 — The additional benefit of the Shapley term remains uncertain. After SFT, it raises compose@1 from 67.91% to 69.23%, a gain clearly smaller than that brought by the clause reward itself. In the absence of repeated training runs, I would be cautious about the conclusion that it reliably improves exploration.
Thank you for asking about stability. Three independent training runs with different seeds and the same training and evaluation settings show small positive Shapley gains. Each compares Full with the corresponding clause-decomposed reward without Shapley:
| Repetition | Shapley gain in compose@1 (pp) | Shapley gain in compose@32 (pp) |
|---|---|---|
| 1 | +0.6 | +0.1 |
| 2 | +1.8 | +1.7 |
| Reported in Table 27 | +1.3 | +1.7 |
The modest gains after SFT are consistent with our expectations and mechanism. Our SFT data construction collects multiple responses, jointly filters their clauses, and selects composed labels using negative-trace coverage. It substantially improves compositional verification with few responses: compose@1 reaches 63.94% after SFT, exceeding compose@32 before SFT (57.81%). Thus, SFT makes a fixed sampling budget more effective.
Our method for constructing discriminative negative traces also provides an informative signal for clause-level rewards. Effective SFT and this reward signal make a smaller marginal benefit from additional credit assignment plausible. At each fixed sampling budget, Table 27 shows that Shapley gains decrease after SFT: from 3.00 to 1.32 percentage points at compose@1 and from 4.32 to 1.68 points at compose@32, consistent with this explanation.
For context, Appendix K.10 (Figure 12) analyzes distribution sharpening under whole-response RL: pass@1 rises from 6.37% to 15.50%, while pass@32 falls from 25.96% to 18.39%. Our clause-decomposed training and compositional inference mitigate this coverage loss.
Response to Reviewer 3 (1/3): SFT label construction
Q1 — Could you compare, on the same batch of training programs and with the same candidate generation budget, "selecting the single response with the highest coverage as the label" against "composing multiple responses as the label"? This would more directly show how much of the SFT improvement comes from composition itself.
Thank you for suggesting this comparison. We first clarify the original SFT data construction, then present a supplementary controlled comparison.
Both SFT datasets use the same input-program pool, Qwen3-8B generator, and training settings, with approximately equal sizes. Over 93% of the distinct prompts in the composed-label dataset also occur in the single-response dataset.
In the original SFT ablation, construction budgets differ: SFT (single) generates one response per input loop, removes incorrect clauses, and uses the remaining valid clauses as the training label; inputs with no valid clauses remaining are skipped. Composed-label SFT jointly filters 16 responses per round for at most five rounds, accepting labels at negative coverage >= 0.5; SFT (single) has no such threshold. The 63.94% versus 23.20% compose@1 result compares these complete construction procedures.
In the supplementary experiment, we compare highest-coverage selection and composition on all 466 Loopy programs using the same first N GPT-5-nano/minimal responses. This matches model-call counts and generation token costs.
All three comparisons use the same N responses. Single-response selection filters each response separately and selects the one with the highest coverage on the same fixed negative traces; composition merges all clauses and filters them together. Filtering costs therefore differ with N and proof difficulty. Targets are used only for evaluation, not selection. “Any single succeeds” counts a program as solved if at least one independently filtered response proves its target.
At N=16, composition verifies 50 more programs than highest-coverage selection (+10.73 percentage points), including 16 programs that no individual filtered response verified under the tested configuration (+3.43 points over "Any single succeeds"). At N=4/8/16, composition also retains every success achieved by an individual response.
Recorded cumulative filtering times are:
At N=16, joint filtering reduces recorded cumulative time by 57.3% while producing higher-quality candidates. Appendix K.6 (Table 21) provides further evidence of lower filtering cost.
Paired SFT retraining was not feasible within the rebuttal period due to time and compute constraints. Under identical generation budgets, composition outperforms best-of-N single-response selection using the same coverage signal. The original SFT ablation further shows that training on labels produced by the complete composition-based procedure yields substantial downstream gains over the single-response filtering baseline.