acceptodds
Under review as a conference paper at ICLR 2027

Evidence Before Assertion: Calibrated LLM-Assisted Verification of Code Changes

Abstract

Large language models can explain code changes fluently, but fluency is a poor substitute for evidence when a reviewer asks whether a route is affected, whether a test actually proves the changed behavior, or whether an unexecuted check is safe to ignore. We present an LLM-assisted verifier for pull requests built around an explicit epistemic contract: a verification claim may be reported as established only when it is grounded in inspectable structural or execution evidence; LLM outputs may propose missing facts, but cannot silently promote them to proof. The system represents behavioral claims as verification obligations, attaches evidence at the strongest scope that can be demonstrated, and degrades attribution rather than confidence when exact ownership cannot be established. We study this design through controlled pull requests on eight real open-source repositories spanning Python, TypeScript, Java, Go, and Rust. The study surfaced seven reproducible failure classes at the boundaries between structural analysis, LLM recovery, test mapping, execution, and reporting. Two cases are particularly informative. First, an apparently intermittent call-resolution failure was shown, after isolating the deterministic subsystem, to be a fully reproducible structural defect masked by occasional LLM recovery; after three generic fixes, the same harness resolved twice and the full pipeline resolved the path in runs. Second, whole-suite execution produced 13 passing and 4 unrelated environment failures for a narrowly scoped change; after wiring mapped-test execution and improving assertion recognition, the system executed exactly one relevant test and attributed a pass to the changed behavior without running the unrelated suite. These results are not a benchmark-scale accuracy claim. They are an empirical systems study of a recurring problem in hybrid LLM tools: correctness failures concentrate at evidence boundaries, and reliable behavior requires explicit provenance, scope-aware evidence ownership, and a calibrated handoff when the environment cannot complete verification.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llm, code, verification, validation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_eXykOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

The paper presents an LLM-assisted verifier for pull requests built on an evidence contract, where LLM outputs can only propose facts and claims become established only with structural or execution evidence, attached at the most specific scope that can be shown. Tests that cannot run are left unresolved and handed off to the developer. The evaluation is a defect-driven case study on controlled changes in several open-source repositories, cataloging failure classes at component boundaries and presenting two before-and-after cases, one where LLM recovery hid a deterministic bug and one where targeted test execution replaced running the whole suite.

Critical strengths

The epistemic framing is clear and useful. Separating LLM proposals from admissible evidence, weakening attribution instead of inflating confidence, and allowing an explicit unresolved state are sensible principles for tools whose output reviewers may act on.

The failure analysis is honest and contains at least one insight that should transfer to other hybrid systems: an LLM fallback can make a deterministic defect look intermittent, so components need to be evaluated in isolation and not only through end-to-end runs.

Critical weaknesses

The evaluation is somewhat limited in supporting the claims about the approach. It consists of a small number of controlled changes, several of them constructed and adjudicated by the authors, and with no comparison to LLM-only or static analysis baselines. We do not learn whether the verifier is more accurate or more useful than simpler alternatives, and the quantitative results are really checks that specific bugs were fixed.

Questions for authors

How were the evaluated changes selected, and how many came from real upstream changes versus being constructed?

Could authors report a modest comparison against an LLM-only reviewer and a static analysis-only pipeline on the same changes?

Reviewer_9mvUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper presents an approach for generating evidence that supports pull request changes. The idea is that each change in the PR corresponds to a property and is documented with a corresponding piece of evidence, e.g., a test that demonstrates the PR does what it should.

I really like the idea. The authors should discuss related work on root cause analysis of software bugs etc. There is a lot of work in this area that pre-dates LLMs.

I have two main concerns about the paper though: 1) it is impossible to tell how the technique presented actually works and 2) it appears to assume that properties that are supported by a single piece of evidence are therefore true. i.e., they are existentially quantified properties.

Critical strengths

This is a domain where there is potential for a lot of impact, and the evaluation is performed on real code respositories.

Critical weaknesses

Lack of comparison to related work

The authors should discuss the body of work on root causes of software bugs

Handling of universally quantified properties

  • It's not clear how you deal with universally quantified properties and I believe the way you deal with them may be unsound. It's possible that my claim is something like "this function only returns even numbers", i.e., it reasons over all possible outputs of a function. Then a piece of evidence could be an execution trace that shows the function returns "2", and that supports my claim, but it's not enough to guarantee that my claim is actually true.

Clarity of methodology

Section 2.2 starts off formalizing the problem, but there are some issues wiht this formaliation:

  • A returns a boolean sometimes and an integer sometimes.
  • c hat is a proposal, but it's not clear if this is a proposed claim (a claim is c, so it probably is a proposed claim? But if it is a claim, why does it need to be passed to A? Because A takes in evidence, not proposals, and we don't need claims to be admissable as proof, we just need them to be supported by evidence
  • In general, tne details of how A works are quite important to this paper, and not really covered in the write up.

The partial order idea presented in section 2.2 is nice, but again, I don't understand how this actually works, and it;s never referred to again

How does the verified derive obligations from changed conditional behaviour? Also this seems slightly cyclical? I thought the verifiers role was to verify that obligations (otherwise known as claims) were supported by evidence?

Evaluation details

  • the actual properties seem quite simple
  • I couldn't figure out how many pull requests the evaluation considered, whether it was 1 per repository (so 8) or more.
  • There doesn't appear to be a comparison to any other approach? Not even simply directly prompting an LLM and asking for evidence?
  • I couldn't find the actual results on accuracy. On the 8 repositories, how many pull requests did yiou look at, how many times was the evidence correct and complete, how many times was it correct but incomplete, how many times was it incorrect? How accurate is the verified? Does it produce false positives or false negatives or both?
Questions for authors

Please could you clarify how the verifier works and how you handle universally quantified properties

Reviewer_ZA7mOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents a tool that checks pull requests by combining an LLM with static analysis and test execution. The tool uses static analysis to work out which behaviors a change may affect, and it reports each one as confirmed only if static analysis or a test run supports it. The LLM can suggest call paths or tests that the analysis missed, but these suggestions must pass further checks. Tests that cannot be run are handed to the developer instead of being reported as failed. The evaluation uses one small, controlled change in each of eight open-source repositories written in five programming languages. While running it, the authors found and fixed seven kinds of bugs in the tool, and they present two of these fixes as case studies.

Critical strengths

The paper tackles a real problem with LLM-based code review: the model can claim that a change affects an endpoint, or that a test verifies it, without any proof. The proposed remedy is sensible: report such a claim as confirmed only when static analysis or an actual test run backs it up.

Critical weaknesses
  1. The novelty is limited. Most of the core ideas have been studied in earlier work, and apart from two papers on regression test selection, including Rothermel & Harrold [1], the paper does not cite any of it.

    • LLMDFA [2], IRIS [3] and RepoAudit [4] also let an LLM propose facts about code and then confirm them with program analysis.
    • Checked coverage [5] and the work on pseudo-tested methods [6] study the difference between a test that merely runs the changed code and one that actually checks its result.
    • Running only the tests affected by a change is the long-studied problem of regression test selection; see, e.g., Ekstazi [7].
    • ChangeGuard [8] and Testora [9] check whether a pull request changes program behavior by running code that specifically exercises the changed parts of the program.

    Combining these ideas can be a contribution, but only if the paper shows that the combined system works better than each idea used alone. The paper does not test this.

  2. The experiments are not enough to show that the design works.

    • The same eight changes (one per repository) were used to find the tool's bugs, to fix them, and to run the final evaluation. So even good results on them would only show that the tool now works on these changes; they say nothing about new pull requests.
    • Apart from the case study on one TypeScript change (Sec. 4.5) and an example report for another repository (App. B), the paper does not report what the final version of the tool produced for each change, or whether that result was correct. There is no accuracy measure (for example, how many claims marked as confirmed were actually wrong), no comparison with a simpler baseline such as an LLM-only reviewer, and no ablation showing what each component contributes.
    • Both case studies use the same TypeScript change (L351–352), so all the before/after numbers in Table 3 come from a single pull request.
    • The paper describes the call-resolution component as deterministic, which means it gives the same result every time. The 20 repeated runs rule out the nondeterminism the authors first suspected (L328–334), but beyond that they add little: going from 0 of 20 successful runs to 20 of 20 only shows that the bug was fixed for this one change.
    • The central rule (Sec. 2.2) says a claim is confirmed when admissible evidence supports it. The paper describes these conditions only informally (identity and ownership checks at L114–115, test tiers in Sec. 3.3) and never defines how the tool decides admissibility (the paper's A) or support (its "supports" relation). Sec. 3.5 says LLM suggestions must pass structural checks, but its only concrete example is that the file and function names match the changed code; it does not say how the tool checks that a suggested multi-step call path actually exists.
    • According to Sec. 2.4, a passing admissible test can mark the claimed behavior as verified, and nothing requires that the test would have failed on the code before the change. A test that passes both before and after a change says nothing about whether the change is correct; SWE-bench [10], for example, requires tests that fail on the original code and pass on the fixed code.
  3. The writing is also a serious problem. The AI Use Statement says generative AI was used extensively, including to draft and edit the paper, and the text reads as if it received little human revision.

    • The paper states its main rule for handling uncertain evidence three times (L019, L057, L132–133), and the three versions do not agree. The rule also refers to "confidence", but no such quantity appears in the formal model.
    • The notation introduced in Sec. 2 is never used again. The paper also never explains how the results for individual claims are combined into the final verdict for a pull request.
    • The same numbers and disclaimers are repeated many times. The call-resolution result (0/20 → 20/20) alone is restated in the abstract, the introduction, Sec. 4, the conclusion, the reproducibility statement and two appendices.
    • Sec. 4 reads like a log of bug fixes. It is not organized around the three research questions (L265–269), and none of them is explicitly answered.
    • The only figure (Fig. 1, L162–166) is two lines of text listing the pipeline stages. A diagram of the system architecture, a figure showing how the different kinds of evidence lead to the final verdict, and plots of the results would make the paper much easier to follow.

References

[1] G. Rothermel and M. J. Harrold. A Safe, Efficient Regression Test Selection Technique. ACM TOSEM, 6(2):173–210, 1997. [2] C. Wang et al. LLMDFA: Analyzing Dataflow in Code with Large Language Models. NeurIPS 2024. [3] Z. Li, S. Dutta, and M. Naik. IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities. ICLR 2025. [4] J. Guo, C. Wang, X. Xu, Z. Su, and X. Zhang. RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing. ICML 2025. [5] D. Schuler and A. Zeller. Assessing Oracle Quality with Checked Coverage. ICST 2011. [6] O. L. Vera-Pérez, B. Danglot, M. Monperrus, and B. Baudry. A Comprehensive Study of Pseudo-tested Methods. Empirical Software Engineering, 24(3):1195–1225, 2019. [7] M. Gligoric, L. Eloussi, and D. Marinov. Practical Regression Test Selection with Dynamic File Dependencies. ISSTA 2015. [8] L. Gröninger, B. Souza, and M. Pradel. ChangeGuard: Validating Code Changes via Pairwise Learning-Guided Execution. FSE 2025. [9] M. Pradel. Testora: Using Natural Language Intent to Detect Behavioral Regressions. ICSE 2026. [10] C. E. Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024.

Questions for authors
  1. What does the system add beyond ChangeGuard [8] and Testora [9], and beyond LLMDFA [2], IRIS [3] and RepoAudit [4]?
  2. How exactly does the tool decide that evidence is admissible (A in Sec. 2.2), that it supports a claim (the "supports" relation), and that an LLM suggestion can be accepted? Does marking a behavior as verified require a test that fails on the code before the change?
  3. For each of the eight changes, could you report what the tool produced and whether it was correct, including how many claims were left unresolved? Could you also report results on pull requests that were not used during development?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.