- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
In recent years, LLMs have become increasingly useful for code generation, and many developers now rely on the use of coding agents to solve coding problems faster and create new products. However, trusting and verifying the generated code of an LLM agent remains challenging. Testing only covers a part of the code execution. Especially in safety critical domains, it is crucial to verify that the generated code matches the intended specifications.
Modern verification engines rely on loop invariants, which are either provided by the user or automatically generated. However, generating invariants is still a substantial challenge. For both developers and tools, it is hard to find invariants that are correct and helpful for verify the code. Since LLMs became more and more powerful in many fields, researchers started to develop invariant generation methods based on LLMs. Currently, there are two main challenges that remain for many LLM-based methods:
- the generated invariants are often not correct, and
- the methods are evaluated on datasets that are public available on the internet and thus have likely been used in the training of the newest LLMs.
The authors address the two challenges by the following contributions:
- They introduce a new pipeline for LLM-based invariant synthesis. Instead of simply prompting an LLM, the key idea to use a pre-processing step to enrich the prompt with additional information. On the one hand, the pipeline uses static and dynamic code analysis to generate further insights into the program. On the other hand, the pipeline generates a number of related programs and corresponding invariants as additional examples to be included into a prompt. Once the preprocessing ends, an LLM is prompted with this additional information to output an invariant for the orginal program (in several iterations if neccessary).
- They generate a novel, synthetic dataset using an LLM. In particular, this dataset contains problems with specific properties (e. g., dead code).
In the evaluation, the authors implemented their pipeline with three diffrerent LLMs (claude-sonnet-5, Gemini-3.5-flash-light, GPT-5.4-mini). An extensive evaluation shows that their method outperforms other LLM-based and non LLM-based methods in most cases. In addition, the authors analyse the token usage and the effectiveness of each part of their pipeline.
Critical strengths
-
The pipeline is a good step in the right direction to generate better invariants by an LLM. It is insightful to see that it helps to additionally provide static and dynamic code analysis results to the LLM. Also, the inclusion of potential unsound information as hints is helpful.
-
The paper is well structured and the evaluation process is sound and easy follow. In particular:
- The paper is well written and easy to understand.
- The authors discuss challenges and limitations in detail and provide mitigation strategies that they used to reduce their effect.
- The code and experimental results are provided in an anonymous repo. Since LLM API keys are needed to test and compare the results, we did not run the code. However, the code seems to be a reasonably good shape, and the instructions to run the code are provided by the authors.
Critical weaknesses
While the topic and the new ideas of the paper are good overall, the following three main concerns remain:
-
Although the paper is generally easy to follow, some paragraphs need editing to me better assessible (e. g., the second part of Section 3.2 or Lines 260-267). In both cases, an example would greatly help the reader to understand the mechanism better. Furthermore, the appendix seems divorced from the main body. There is no reference by the main paper to the appendix, and Table 2 is never referenced or discussed.
(Concern (1) had only a moderate effect on the final rating.)
-
The paper lacks critical information, and some methodological decisions need further explanations. In particular:
- Related work in the area of LLM-based invariant synthesis is missing, including work that enrich the prompt with failure information. Examples include Ranking LLM-Generated Loop Invariants for Program Verification (Chakraborty et al., EMNLP 2023), Combining Reasoning Optimized LLMs and SMT Solvers for Automated Loop Invariant Synthesis (Bharti et al., AIware 2025), Quokka: Accelerating Program Verification with LLMs via Invariant Synthesis (Wei et al., ICLR 2026).
- When comparing the inference cost, no input tokens are reported for the best performing model (Sonnet), while this information is provided for all other models. Without this information, however, aby comparison is incomplete. To mitigate this issue, I recommend logging, including and analysing the input token usage for all models in the paper.
- It remains unclkear why the authors chose to experiment with relatively small models. Current frontier models are know to perform much better on coding and code analysis tasks. Therefore, it is questionable to which extent the results do generalize. Since the authors have access to Opus 5 (the dataset is generarted using it), it would be valuable for the paper to include at least Opus in the experiemntal evaluation.
- It remains unclear to what extent the novel dataset is tailored to the proposed pipeline. For instance, the novel programs are generated to contain dead code, while the first step in the proposed pipeline is to remove it immediately. As a consequence, a comparison to existing methods on this particular dataset seems unfair.
(Concern (2) had the most significant effect on the final rating.)
-
One of the paper's most significant contributions is the newly created dataset, which the authors motivate through data poisioning: many verification tasks are likely contained in the training data of the LLM models, resulting in high performance on these tasks. However, the way the novel dataset is produced is also problematic: as it is completely AI generated, the novel dataset is likely to inherit this problem. The fact that Sonnet performs the best, while Opus (same model family) generated the dataset, is a strong indicator for this potential issue.
(Concern (3) had a significant effect on the final rating.)
Questions for authors
- Did you also evaluate SOTA frontier models or can you include them?
- In the repo: Is the "showcase" dataset the "looptrans" dataset?
- What is the reason for not reporting on the consumed input tokens of Sonnet?
2 comments
-
What is the reason for not reporting on the consumed input tokens of Sonnet?
We thank the reviewer for pointing out this shortcoming in our reporting; the number of input tokens for our Sonnet experiments was not recorded because of a bug in our experiment-running script (
comparison/llm_utils.pyfunction:_query_claude_code); the experiments therefore only captured the input usage of the last inference queries per benchmark, rather than for each one individually. We have fixed this bug and will rerun the Sonnet experiments in order to fully report usage for all models before the camera-ready deadline. With the fixed script, we have also recorded token usage for the new agentic variant: it uses 3.1–4.5× (excluding Sonnet) the tokens of the STRIVE harness (see the updated Figure 4 and the graphs in Appendix A). The results for the other models show the pattern that the unverified transformations and full/agentic variants are the most costly. -
In the repo: Is the “showcase” dataset the “looptrans” dataset?
Yes, the showcase dataset in the artifact is simply a renaming of the looptrans dataset in the paper. It originally started as a way to showcase the value of dead code elimination; the examples were then switched to be sourced from real-world loop benchmarks with dead code injected into only a third of the files, as described above. looptrans is the correct name.
Related work in the area of LLM-based invariant synthesis is missing…
We thank the reviewer for directing us to this additional related work, and have added the following to the updated draft paper.
Chakraborty et al. (EMNLP 2023) rank a set of invariants to decide which are most useful. This is a different problem than STRIVE, which focuses on generating the candidates, although ranking the generated candidates is also important. Their ranking is complementary to STRIVE: it could be used to rank and filter the candidate invariants recycled from the unverified transformations before they reach the verifier.
Bharti et al. (AIware 2025) translate programs into their own intermediate verification language and encode it to SMT, rather than using an existing verifier. They evaluate only on code2inv, where they solve fewer problems than STRIVE.
Quokka (Wei et al., ICLR 2026) identifies that a simple harness produces strong results over the previous heavily engineered options. This result is also found in our results, where a simple harness backing STRIVE outperforms other tools. They classify individual invariants as true, false, or undecided, which is done by Frama-C's (our verifier backend) per-goal results. Their model training is orthogonal and could be combined with our analyses.
Did you also evaluate SOTA frontier models or can you include them?
In response to this review, we performed an evaluation with the frontier model gpt-6.1-terra to investigate generalization to newer models. We chose a non-Anthropic model to simultaneously avoid concerns over the benchmark assembled by Opus being easier to solve for Anthropic models. gpt-6.1-terra is the strongest model we evaluated; it saturates code2inv, fib_46 and looptrans in every configuration, which leaves little to no room for the analyses and transformations to add solves. So, on these benchmarks we cannot tell how much the largest models still gain from them. The analyses and transformations seem to still help on SV-COMP, with unverified transformations raising Terra from 47 to 56 solves. Measuring the gain for frontier models will require harder benchmarks.
These results also address the data contamination concern. If looptrans favored the model family that generated it, a non-Anthropic model should do worse on it than Sonnet. Instead, Terra solves all 52 looptrans instances with the harness alone, more than any Sonnet configuration.
It remains unclear to what extent the novel dataset is tailored to the proposed pipeline.
Our work includes the novel application of dead code elimination to the task of invariant synthesis. We argue that the looptrans dataset is a useful comparison in our work because (1) realistic input programs contain dead code, (2) no existing invariant synthesis benchmarks represent this feature of realistic codebases, and (3) we have designed our benchmark to include real programs that are not overfit to our approach.
Regarding (1), a recent study (Romano, Vendome, Scanniello, Poshyvanyk, IEEE TSE 2020) identified that dead code is both widespread and harmful. All popular production compilers (LLVM, GCC, etc.) remove it. Regarding (2), the existing benchmark sets used to evaluate prior work:
code2inv(Si et al., 2020),fib_46(Dillig et al., 2013),dig_hola(Nguyen et al., 2014), and selections fromSV-COMPandFrama-C(Wen et al., 2024) are toy-sized and do not capture the full range of features in real code, so existing tools do not consider these features either. In particular, dead code is entirely absent from the existing benchmarks.Given that no existing benchmark set captures this realistic feature of production code, we set about constructing a benchmark set that fairly measures invariant synthesis approaches' success on programs that contain dead code. The looptrans benchmark (described in Section 3.6) consists of 52 programs sourced from real-world loop transformation benchmarks (HecBench, PolyBench, and tests from GCC/LLVM/Pluto). We then used the Opus 5 model to inject dead code regions into the existing loop-containing functions of a third of the files; this process is a well-established practice (bug planting/injecting) in work on bug detection (Dolan-Gavitt et al., 2016; Hazimeh, Herrera, Payer, 2020;). The injected dead code varies in detection difficulty, from trivially syntactic uncalled functions to complex cases that require reasoning about possible values. The model used to plant the dead code had no access to STRIVE's implementation or any documentation of our work, and therefore operated independently and had no way of tailoring programs to STRIVE's implementation. As described in our answer on frontier models above, our additional experiments with gpt-6.1-terra also rule out the possibility that the benchmark favors Anthropic models because an Anthropic model generated it.