acceptodds
Under review as a conference paper at ICLR 2027

Precise Invariant Generation via Interleaved Agentic Reasoning and Program Analysis

Abstract

Programmers increasingly rely on Large Language Models (LLMs) to generate code, yet nothing guarantees the code they generate is correct. Recent work has attempted to mitigate this issue by leveraging formal verification tools to certify that LLM-written code is correct. However, existing verifiers cannot scale to real-world software unless they are provided with invariants to guide the verification process. Generating correct and useful invariants is a long-standing unsolved problem: existing formal methods approaches cannot handle larger programs, while LLM-based approaches struggle to reason about the low-level semantics necessary to generate precise invariants. We present a novel approach to invariant generation that achieves the best of both worlds, interleaving LLM-based invariant generation with structured formal analyses. Our strategy first uses program transformations to reduce the input program and tailor its structure for LLM-based reasoning. It then runs a medley of existing and novel analyses on the input program to construct a formal knowledge base of program analysis information. The combination of the reduced program with the formal knowledge base enables an invariant generation agent to produce invariants that are correct and precise enough for program verification. We implement this approach in a tool (STRIVE). STRIVE solves up to as many problems as existing LLM-based approaches and up to as many as solver-based approaches.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Invariant Generation, Program Verification, Compiler Transformations, Static Analysis, Dynamic Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FtjKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In recent years, LLMs have become increasingly useful for code generation, and many developers now rely on the use of coding agents to solve coding problems faster and create new products. However, trusting and verifying the generated code of an LLM agent remains challenging. Testing only covers a part of the code execution. Especially in safety critical domains, it is crucial to verify that the generated code matches the intended specifications.

Modern verification engines rely on loop invariants, which are either provided by the user or automatically generated. However, generating invariants is still a substantial challenge. For both developers and tools, it is hard to find invariants that are correct and helpful for verify the code. Since LLMs became more and more powerful in many fields, researchers started to develop invariant generation methods based on LLMs. Currently, there are two main challenges that remain for many LLM-based methods:

  1. the generated invariants are often not correct, and
  2. the methods are evaluated on datasets that are public available on the internet and thus have likely been used in the training of the newest LLMs.

The authors address the two challenges by the following contributions:

  1. They introduce a new pipeline for LLM-based invariant synthesis. Instead of simply prompting an LLM, the key idea to use a pre-processing step to enrich the prompt with additional information. On the one hand, the pipeline uses static and dynamic code analysis to generate further insights into the program. On the other hand, the pipeline generates a number of related programs and corresponding invariants as additional examples to be included into a prompt. Once the preprocessing ends, an LLM is prompted with this additional information to output an invariant for the orginal program (in several iterations if neccessary).
  2. They generate a novel, synthetic dataset using an LLM. In particular, this dataset contains problems with specific properties (e. g., dead code).

In the evaluation, the authors implemented their pipeline with three diffrerent LLMs (claude-sonnet-5, Gemini-3.5-flash-light, GPT-5.4-mini). An extensive evaluation shows that their method outperforms other LLM-based and non LLM-based methods in most cases. In addition, the authors analyse the token usage and the effectiveness of each part of their pipeline.

Critical strengths
  1. The pipeline is a good step in the right direction to generate better invariants by an LLM. It is insightful to see that it helps to additionally provide static and dynamic code analysis results to the LLM. Also, the inclusion of potential unsound information as hints is helpful.

  2. The paper is well structured and the evaluation process is sound and easy follow. In particular:

    • The paper is well written and easy to understand.
    • The authors discuss challenges and limitations in detail and provide mitigation strategies that they used to reduce their effect.
    • The code and experimental results are provided in an anonymous repo. Since LLM API keys are needed to test and compare the results, we did not run the code. However, the code seems to be a reasonably good shape, and the instructions to run the code are provided by the authors.
Critical weaknesses

While the topic and the new ideas of the paper are good overall, the following three main concerns remain:

  1. Although the paper is generally easy to follow, some paragraphs need editing to me better assessible (e. g., the second part of Section 3.2 or Lines 260-267). In both cases, an example would greatly help the reader to understand the mechanism better. Furthermore, the appendix seems divorced from the main body. There is no reference by the main paper to the appendix, and Table 2 is never referenced or discussed.

    (Concern (1) had only a moderate effect on the final rating.)

  2. The paper lacks critical information, and some methodological decisions need further explanations. In particular:

    • Related work in the area of LLM-based invariant synthesis is missing, including work that enrich the prompt with failure information. Examples include Ranking LLM-Generated Loop Invariants for Program Verification (Chakraborty et al., EMNLP 2023), Combining Reasoning Optimized LLMs and SMT Solvers for Automated Loop Invariant Synthesis (Bharti et al., AIware 2025), Quokka: Accelerating Program Verification with LLMs via Invariant Synthesis (Wei et al., ICLR 2026).
    • When comparing the inference cost, no input tokens are reported for the best performing model (Sonnet), while this information is provided for all other models. Without this information, however, aby comparison is incomplete. To mitigate this issue, I recommend logging, including and analysing the input token usage for all models in the paper.
    • It remains unclkear why the authors chose to experiment with relatively small models. Current frontier models are know to perform much better on coding and code analysis tasks. Therefore, it is questionable to which extent the results do generalize. Since the authors have access to Opus 5 (the dataset is generarted using it), it would be valuable for the paper to include at least Opus in the experiemntal evaluation.
    • It remains unclear to what extent the novel dataset is tailored to the proposed pipeline. For instance, the novel programs are generated to contain dead code, while the first step in the proposed pipeline is to remove it immediately. As a consequence, a comparison to existing methods on this particular dataset seems unfair.

    (Concern (2) had the most significant effect on the final rating.)

  3. One of the paper's most significant contributions is the newly created dataset, which the authors motivate through data poisioning: many verification tasks are likely contained in the training data of the LLM models, resulting in high performance on these tasks. However, the way the novel dataset is produced is also problematic: as it is completely AI generated, the novel dataset is likely to inherit this problem. The fact that Sonnet performs the best, while Opus (same model family) generated the dataset, is a strong indicator for this potential issue.

    (Concern (3) had a significant effect on the final rating.)

Questions for authors
  • Did you also evaluate SOTA frontier models or can you include them?
  • In the repo: Is the "showcase" dataset the "looptrans" dataset?
  • What is the reason for not reporting on the consumed input tokens of Sonnet?
2 comments
  • Related work in the area of LLM-based invariant synthesis is missing…

    We thank the reviewer for directing us to this additional related work, and have added the following to the updated draft paper.

    Chakraborty et al. (EMNLP 2023) rank a set of invariants to decide which are most useful. This is a different problem than STRIVE, which focuses on generating the candidates, although ranking the generated candidates is also important. Their ranking is complementary to STRIVE: it could be used to rank and filter the candidate invariants recycled from the unverified transformations before they reach the verifier.

    Bharti et al. (AIware 2025) translate programs into their own intermediate verification language and encode it to SMT, rather than using an existing verifier. They evaluate only on code2inv, where they solve fewer problems than STRIVE.

    Quokka (Wei et al., ICLR 2026) identifies that a simple harness produces strong results over the previous heavily engineered options. This result is also found in our results, where a simple harness backing STRIVE outperforms other tools. They classify individual invariants as true, false, or undecided, which is done by Frama-C's (our verifier backend) per-goal results. Their model training is orthogonal and could be combined with our analyses.

  • Did you also evaluate SOTA frontier models or can you include them?

    In response to this review, we performed an evaluation with the frontier model gpt-6.1-terra to investigate generalization to newer models. We chose a non-Anthropic model to simultaneously avoid concerns over the benchmark assembled by Opus being easier to solve for Anthropic models. gpt-6.1-terra is the strongest model we evaluated; it saturates code2inv, fib_46 and looptrans in every configuration, which leaves little to no room for the analyses and transformations to add solves. So, on these benchmarks we cannot tell how much the largest models still gain from them. The analyses and transformations seem to still help on SV-COMP, with unverified transformations raising Terra from 47 to 56 solves. Measuring the gain for frontier models will require harder benchmarks.

    These results also address the data contamination concern. If looptrans favored the model family that generated it, a non-Anthropic model should do worse on it than Sonnet. Instead, Terra solves all 52 looptrans instances with the harness alone, more than any Sonnet configuration.

  • It remains unclear to what extent the novel dataset is tailored to the proposed pipeline.

    Our work includes the novel application of dead code elimination to the task of invariant synthesis. We argue that the looptrans dataset is a useful comparison in our work because (1) realistic input programs contain dead code, (2) no existing invariant synthesis benchmarks represent this feature of realistic codebases, and (3) we have designed our benchmark to include real programs that are not overfit to our approach.

    Regarding (1), a recent study (Romano, Vendome, Scanniello, Poshyvanyk, IEEE TSE 2020) identified that dead code is both widespread and harmful. All popular production compilers (LLVM, GCC, etc.) remove it. Regarding (2), the existing benchmark sets used to evaluate prior work: code2inv (Si et al., 2020), fib_46 (Dillig et al., 2013), dig_hola (Nguyen et al., 2014), and selections from SV-COMP and Frama-C (Wen et al., 2024) are toy-sized and do not capture the full range of features in real code, so existing tools do not consider these features either. In particular, dead code is entirely absent from the existing benchmarks.

    Given that no existing benchmark set captures this realistic feature of production code, we set about constructing a benchmark set that fairly measures invariant synthesis approaches' success on programs that contain dead code. The looptrans benchmark (described in Section 3.6) consists of 52 programs sourced from real-world loop transformation benchmarks (HecBench, PolyBench, and tests from GCC/LLVM/Pluto). We then used the Opus 5 model to inject dead code regions into the existing loop-containing functions of a third of the files; this process is a well-established practice (bug planting/injecting) in work on bug detection (Dolan-Gavitt et al., 2016; Hazimeh, Herrera, Payer, 2020;). The injected dead code varies in detection difficulty, from trivially syntactic uncalled functions to complex cases that require reasoning about possible values. The model used to plant the dead code had no access to STRIVE's implementation or any documentation of our work, and therefore operated independently and had no way of tailoring programs to STRIVE's implementation. As described in our answer on frontier models above, our additional experiments with gpt-6.1-terra also rule out the possibility that the benchmark favors Anthropic models because an Anthropic model generated it.

  • What is the reason for not reporting on the consumed input tokens of Sonnet?

    We thank the reviewer for pointing out this shortcoming in our reporting; the number of input tokens for our Sonnet experiments was not recorded because of a bug in our experiment-running script (comparison/llm_utils.py function: _query_claude_code); the experiments therefore only captured the input usage of the last inference queries per benchmark, rather than for each one individually. We have fixed this bug and will rerun the Sonnet experiments in order to fully report usage for all models before the camera-ready deadline. With the fixed script, we have also recorded token usage for the new agentic variant: it uses 3.1–4.5× (excluding Sonnet) the tokens of the STRIVE harness (see the updated Figure 4 and the graphs in Appendix A). The results for the other models show the pattern that the unverified transformations and full/agentic variants are the most costly.

  • In the repo: Is the “showcase” dataset the “looptrans” dataset?

    Yes, the showcase dataset in the artifact is simply a renaming of the looptrans dataset in the paper. It originally started as a way to showcase the value of dead code elimination; the examples were then switched to be sourced from real-world loop benchmarks with dead code injected into only a third of the files, as described above. looptrans is the correct name.

Reviewer_1tZtOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles automatically generating invariants for program verification using language models, but augment the invariant generation with three program enhancements. First is an optimization (dead code elimination), which allows the model to reason about less code. The second is to generate simplified programs using unsound transformations as a way to give the model some guidance. The third is to annotate/provide the language model with information from program analysis. Their approach outperforms all the tested baselines.

Critical strengths

I think the approach is clever, specifically generating easier but related programs as guidance. The results are fairly convincing, as they seem to outperform the baselines by a fairly large margin.

Critical weaknesses

My main critique is that the ablations show that their actual approach isn't very necessary (if I understood this correctly). Specifically, table 2 in the appendix shows that using ONLY the harness solves 340 of the problems, and using all 3 together only solves 328? And the next best match (using only VR) solves 347, which feels well within noise to me. And the harness, by the authors words on line 290, is "an inference pipeline that is fairly standard and similar to all existing LLM-based invariant generation tools". So it's very confusing to me why it would outperform the baselines so much, and also confusing to me what the actual contribution of the paper is.

Questions for authors
  • Do the three sections of the methodology (dead code elimination, unverified transformations, and program analysis) actually contribute significantly to the performance? Table 2 in the appendix suggests that they do not.
  • Along the same lines, why does harness-only perform so well? If the bulk of the benefit comes from this simple harness, what is the intellectual contribution of the paper?
2 comments
  • Do the three sections of the methodology (dead code elimination, unverified transformations, and program analysis) actually contribute significantly to the performance? Table 2 in the appendix suggests that they do not.

    Our results show that each transformation and analysis contributes significantly to improved performance, but each one also has an innate trade-off, which is why the aggregate numbers are not as impressive. When an analysis or transformation is relevant to a problem, it can solve problems the harness alone cannot (Venn diagram, Figure 3). When it is not relevant, it adds context and reasoning cost on easy problems, increasing the likelihood of a timeout, and unsound results can mislead the model into missing the necessary invariants. The conclusion of our work is therefore not as simple as “always run every analysis for better overall results”, but that for many instances the information from the analyses is what enabled the solution. The new agentic results highlight this fact, where for Sonnet, Gemini, and GPT-mini, the ability of the models to selectively call the analysis and transformation tools results in a large overall performance boost compared to both the harness and the STRIVE tool. To demonstrate how each analysis and transformation helps, we provide several examples below, each an instance that, for the smaller and medium-size models, only STRIVE with the relevant analysis or transformation solved:

    Dead code elimination (looptrans s31). Existing benchmarks are small, toy-sized programs with no dead code. In the program below, aux1 is never called. Without DCE, every LLM-based tool generates invariants for the loops in aux1 and the verifier must then prove them; when the dead code is complex enough, this causes failure. DCE removes it at negligible cost.

    // never called; DCE removed
    static int aux1(int n) { 
        int a[12]; int i, lo, hi;
        for (i = 0; i < 12; i++) a[i] = (i * 29 + n) % 64;
        lo = a[0]; hi = a[0];
        for (i = 1; i < 12; i++) { 
            if (a[i] < lo) lo = a[i]; 
            if (a[i] > hi) hi = a[i];
        }
        return hi - lo;
    }
    long long kernel(int n) {
        long long s = 0; int i;
        for (i = 0; i < n; i++) s = s + 5;
        return s;
    }
    

    Unverified transformations (SV-COMP nla-digbench_prodbin). The models failed to generate the necessary invariants for the original loop:

    x = a; y = b; z = 0;            // assume(b >= 1)
    while (1) {
      assert(z + x * y == (long long)a * b);
      if (!(y != 0)) break;
      if (y % 2 == 1) { z = z + x; y = y - 1; }
      x = 2 * x; y = y / 2;
    }
    

    After the LLVM loop canonicalization passes turned the infinite loop with a break into a guarded condition-based loop, the model produced invariants that verified and were recycled into the original program, solving the problem:

    assert(z + x * y == (long long)a * b);
    if (y != 0) {
      /*@ loop invariant z + x * y == (long long)a * b;
          loop invariant y >= 0;
          loop assigns x, y, z; */
      do {
        if (y % 2 == 1) { z = z + x; y = y - 1; }
        x = 2 * x; y = y / 2;
        assert(z + x * y == (long long)a * b);
      } while (y != 0);
    }
    

    Program analysis (code2inv test 83). For the program below, the OEIS analysis suggested that x=y2−y−100002x = \frac{y^2 - y - 10000}{2}. No run without the OEIS analysis ever found or suggested this formula, and it appears verbatim in the invariants that solved the problem.

    int x = -5000; int y = 0;
    while (x < 0) { x = x + y; y = y + 1; }
    assert(y > 0);
    
    /*@ loop invariant x == (y * y - y - 10000) / 2;
        loop invariant y >= 0;
        loop assigns x, y; */
    
  • Along the same lines, why does harness-only perform so well? If the bulk of the benefit comes from this simple harness, what is the intellectual contribution of the paper?

    We agree that the harness alone outperforms existing LLM-based invariant generation tools. We attribute this mainly to engineering choices in those tools (not only related to tool calling), not to anything novel in our harness. The paper does not claim that the improvements made by the harness are a component of our conceptual contribution in this work; rather, our contribution has the effect of solving the additional cases detailed in Figure 3. As described in our response to the first question, the additional analysis results also have the cost of increasing the number of timeout cases.

    Specifically, LEMUR (with CBMC or ESBMC) and ESBMC-ibmc rely on bounded model checkers. These are built to unroll loops and solve the resulting straight-line code, which is not possible for benchmarks with symbolically bounded loops. Although symbolic constraints are possible to encode in SMT formulas used by the underlying solvers, the bounded model checkers are not natively designed for this, and the tools partially inherit this limitation. STRIVE and AutoSpec instead use the Frama-C WP plugin, which is designed for general-purpose invariants and specifications.

    UAutomizer is interpolation-based and performs well even on symbolically bounded loops, but the only LLM-based tool that uses this is LEMUR. LEMUR generates rounds of one invariant, then two, then three, and so on. This aims for a minimal set of invariants rather than solving the problem by any means, and it consumes much of the inference budget. Current commercial models are good at generating the number and type of invariants a problem needs without this hand holding, so this scaffolding now costs more than it helps.

    AutoSpec uses the same verifier as STRIVE, but it aggressively rewrites the model's invariants after generation, in some cases from a lookup table of known invariants. This can replace or remove invariants the proof needs. It also generates invariants scope by scope, inside out (innermost loop to outer loops, function body to call site). That helps with context size, but giving the model the whole program at once, as STRIVE does, generates all invariants in a single inference and attacks unproven goals in several places at once. This takes much less inference time and yields more solves. There is a scalability concern, although for the size of these benchmark files and the context capabilities of modern commercial models that does not come into play.

    Many of these choices predate current models and were reasonable when they were made. The harness is not the paper's main contribution; it is the baseline against which we measure the impact of the analyses and transformations. We compare against each of these baselines to demonstrate the relationship of our work to prior approaches, and our novel analysis and transformation steps lead to solving dozens of instances for which both the existing approaches and our harness fail.

Reviewer_ej9kOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents STRIVE, a loop-invariant generation method that interleaves program analysis (static simplification, compiler optimization, mathematical sequence matching) with LLM reasoning, combining the precision of formal methods with LLM flexibility. Across six benchmarks, STRIVE significantly outperforms pure solver tools (CBMC, ESBMC, UAutomizer) and LLM-only baselines, and introduces a new dead-code benchmark, looptrans. The idea is novel and well-motivated, but the evaluation rigor, mechanism explanation, and ablation analysis have several key gaps.

Critical strengths

STRIVE delivers a clear performance edge: it solves 328 instances in total, 51% more than the next-best tool, UAutomizer (216). On the authors' own looptrans benchmark, it solves 42 instances—far more than any other tool—showing a distinct strength on programs with dead code and complex structure.

Critical weaknesses
  1. Conflates "method design gains" with "native LLM capability limits" The paper treats "LLM generates candidate invariants + external verifier checks them" as a fixed pipeline, but sets no baseline to answer: if a base LLM had the same tool access (running analyses, querying OEIS, calling solvers, self-checking), how far could it get on its own? STRIVE hard-codes these capabilities into dedicated components upfront, so it's unclear whether the gains come from architectural innovation or just from replacing tool calls the LLM could have made itself with hard-coded scheduling.

  2. No "pure agent baseline" comparison Other LLM baselines (AutoSpec, LEMUR, etc.) are restricted to "text generation only, no active tool calls," which makes the comparison unfair from the start—STRIVE's core advantage is multi-tool orchestration, yet it's never fairly compared against comparable LLM methods with equal tool access.

  3. Tool capability and model capability are conflated The claim that "LLMs struggle with low-level semantics" may be overstated; a more reasonable explanation is that baseline systems simply lack a tool-calling channel. Once an LLM is allowed to run scalar evolution, value-range analysis, OEIS retrieval, and loop simplification on demand, it could obtain these cues itself—STRIVE doesn't yet show that these must be pre-executed by the system and packaged into context.

Questions for authors

Why not run a baseline where the LLM can directly call analyses, OEIS, and the verifier itself? Could those gains come from tool access rather than your design?

1 comment
  • Why not run a baseline where the LLM can directly call analyses, OEIS, and the verifier itself? Could those gains come from tool access rather than your design?

    We thank the reviewer for this insightful suggestion, and have now conducted this experiment; the full results are reproduced in the table in our overall response. We found that the "freeform" agent with access to each analysis/transformation as a tool outperforms the fixed-order STRIVE result by a considerable margin for three of the four models. We argue, however, that this result reinforces the contribution of our work rather than detracting from it. Our primary conceptual contribution is in interleaving novel analyses and transformations with LLM-based reasoning, especially the design of the novel OEIS-based analysis (Section 3.3) and the invariant lifting conducted during unverified transformations (Section 3.2). While our original implementation of STRIVE used a fixed order of the three tools and called each one on every input program, the phase ordering is largely an implementation choice, not the main contribution of STRIVE. The unique-solve Venn diagram (Figure 3) already showed problems that were solved only with the addition of the transformations and analyses, and the new results show the same: the smaller models gain substantially through the agentic pipeline using the analyses and transformations, and for gpt-6.1-terra the best configuration is the unverified-transformation pipeline. In both cases the additional solves come from the analyses and transformations, not from how they are orchestrated. The results for a freeform agent, described below, show that the design of each tool is helpful in both modes and give some evidence that a freeform design outperforms a fixed ordering for some models.

    In particular, our results show that, for claude-sonnet-5, gemini-3.5-flash-lite, and gpt-5.4-mini, the agentic system is best, outperforming full STRIVE which forces all 3 components (verified reductions, unverified transformations, and program analyses) to be run without choice. The models were fairly aggressive in tool use, particularly with the program analyses, which are low-cost tools that give useful additional information (e.g., Gemini solved 82 more benchmarks than the full STRIVE and agentically requested the program analysis results for 53.9% of the problems). gpt-6.1-terra (itself a new experimental result, see above) is the exception. It saturates code2inv, fib_46 and looptrans in every configuration, and the agentic style performs slightly worse than the full STRIVE. One possible explanation is that the strongest model can separate the useful parts of the analysis and transformation output from the noise, so receiving every result costs it little, while the agentic style sometimes skips a tool that would have helped. Conversely, weaker models are more easily misled by irrelevant output, so letting them choose pays off. Because saturation limits how much we can learn from these benchmarks, we would like to evaluate this on more complex benchmarks.

  • The claim that ‘LLMs struggle with low-level semantics’ may be overstated; a more reasonable explanation is that baseline systems simply lack a tool-calling channel.

    We agree with the reviewer that this sentence was phrased poorly. We meant that the models on their own, without calling out to tools, struggle with these semantics; the tool calls, whether chosen agentically or hardcoded in the architecture, are what let the LLM reason about them. We have reworded the claim in the updated paper draft to make this clear.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall Comment

We thank the reviewers for their time and insightful comments. Here we provide an overall response and answer each reviewer's questions in a specific comment below.

All three reviewers asked about the source of STRIVE's performance gains and whether they result from our contributions. Our core contribution is the design of the analysis and transformation tools, including the novel OEIS lookup and the validation process for unverified transformations, not the particular order in which they are called. We conducted additional experiments to determine which parts of this contribution are responsible for the gains, and incorporated the results into an updated draft (changes marked in teal). The new results (Table 1 and embedded below) show that providing the transformation and analysis tools yields additional solved instances whether the tools are run in a fixed order or offered as optional tool calls to an agent. In particular, we compare STRIVE with a fixed sequence of tool calls to an agent that may call each tool at will. This “freeform” agent outperforms the fixed orchestration for three of the four models (Sonnet 357 vs. 328, Gemini 275 vs. 193, GPT-mini 191 vs. 164) and is slightly behind it for (newly added) gpt-6.1-terra at 359 vs. 365. Each type of transformation or analysis contributes additional solved instances in the fixed-order mode, as detailed below. The freeform agents used tool calls on 57% of problems and saw large performance improvements.

We also agree with Reviewers ej9k and 1tZt that engineering improvements in our harness, relative to the baseline tools, contributed some of the performance gains. This can be seen in the updated Table 1, where the harness-only and all-tool configurations are comparable for each tested model. The key distinction is that each analysis or transformation contributes unique solves that are not covered by any other existing method, including our harness-only approach. Two phenomena combine to explain the results. On the one hand, providing additional transformations and analysis information takes more time and requires more reasoning tokens; this overhead causes some instances that are easy for the baseline to time out with STRIVE. More importantly, many more difficult instances are solved because the agent is provided with additional information and transformations; we detail the counts of these instances in Figure 3 and provide a breakdown in the individual rebuttals below. Overall, with Sonnet, 18 additional instances (unsolved by the harness) are solved based on verified reduction, 20 based on unverified transformations, 12 based on program analyses, and 11 by the combination of all three, for which every individual component fails. In total, 34 instances unsolved by the harness are solved by a STRIVE variant with the added analysis information or transformations.

In addition to the new experiments with a freeform agent, we also ran STRIVE with a larger, non-Anthropic model, gpt-6.1-terra. While Terra saturates some of the benchmarks (code2inv, fib_46 and looptrans), the transformations still help on SV-COMP, where unverified transformations raise it from 47 to 56 solves. Below, we include the updated version of Table 1:

AuthorsOpenReview ↗

Updated Results Table

GroupMethodModelcode2invSVCOMPdig_holaframa_cfib_46looptransTotal
# Instances1339146534652421
SolverCBMC--1021432058
ESBMC--1433464272
UAutomizer--99739312713216
LLM-basedAutoSpecSonnet11713018230171
Gemini1061102220141
GPT-mini11912020220173
GPT-terra12318016220179
ESBMC ibmcSonnet1213171160158
Gemini98314000115
GPT-mini1133170150148
GPT-terra1144150130146
LEMURSonnet3111151711085
Gemini2188180055
GPT-mini19109189065
GPT-terra3110171910087
STRIVE variants (ours)harnessSonnet1264332484645340
Gemini10723424340192
GPT-mini10218520290174
GPT-terra1294738484652360
VRSonnet1254536464451347
Gemini10520324290181
GPT-mini10019419270169
GPT-terra1295038484652363
UTSonnet1265032484444344
Gemini11012420271174
GPT-mini10118218250164
GPT-terra1295637484652368
PASonnet1253635494548338
Gemini11215419320182
GPT-mini9816218310165
GPT-terra1294837484652360
VR + UT + PASonnet1264326464542328
Gemini11521422301193
GPT-mini10114218290164
GPT-terra1295337484652365
AgenticSonnet1295133484650357
Gemini1242913413929275
GPT-mini11217626300191
GPT-terra1294539484652359

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.