acceptodds
Under review as a conference paper at ICLR 2027

SpecGuard: Proving a Task Is Broken Before the Agent Cheats

Abstract

As autonomous coding agents get increasingly deployed, the risk that accidental or adversarially injected misspecifications in tasks lead to dangerous agent behavior is critical to address. Prior work has shown that agents given such tasks rarely flag the conflict and instead cheat, editing tests or hard-coding expected outputs, and the actions taken to cheat can cause real damage, such as deleting a security defense to make a corrupted test pass. It remains unclear whether such conflicts can be established with independently verifiable evidence before the agent acts. We present SpecGuard, which detects and formally certifies these conflicts between task intent and tests. Given only the task description and codebase, SpecGuard autoformalizes the intended behaviour into a Lean 4 specification. The tests are formalized independently, and the Lean kernel checks whether any implementation could satisfy both formalizations, producing a machine-checked certificate when none can. On conflicted SWE-bench tasks, SpecGuard detects up to 72.8% of conflicts and formally certifies up to 51.1%, with a nearly fivefold lower conflict miss rate than model-based judgment. SpecGuard provides a pre-execution safety check that identifies reward-hacking opportunities through formal certification of task-level conflicts, before any agent behavior is observed.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
reward hacking, specification gaming, formal verification, Lean, coding agents, autoformalization, AI safety, verifiers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_635LOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

SpecGuard investigates whether conflicts between a coding task’s intended behavior and its tests can be identified before an autonomous agent begins execution. It treats the task description as authoritative and allows the tests to contain accidental or adversarial errors.

The system uses three agents. A SpecAgent reads the description and repository, with tests withheld, and produces an executable Lean specification. A TestAgent independently formalizes test requirements. A Certifier connects the representations, enabling Lean to check for disagreement. Outputs include certified conflicts, execution-supported conflicts, no-conflict verdicts, and inconclusive results. Certificates establish inconsistency between the generated representations; their correspondence to the original task requires separate validation.

On 349 mutated SWE-bench tasks across four models, SpecGuard reports conflict detection of up to 72.8% and certification of up to 51.1%. Providing both original and corrupted tests raises Sol’s certification rate from 51.1% to 67.0%. A manual audit classifies 52 of 60 sampled certificates as faithful.

Additional evaluations cover 22 naturally occurring GitHub conflicts, yielding nine detections, eight certified, and 95 LiveCodeBench tasks, yielding 80 execution-supported detections. On VERINA, 180 of 187 generated specifications are proved equivalent to human-written references. The failure analysis identifies alignment between specifications and test representations as a major bottleneck.

Critical strengths

None

Critical weaknesses
  • Unclear added value beyond structured inference-time checking: Most of SpecGuard’s work lies in generating and aligning representations, while Lean certifies the resulting computation. The paired-test Python baseline achieves 74.8% detection versus SpecGuard’s 73.9% at similar cost, and the headline miss-rate reduction substantially reflects abstention. Comparisons with matched information, compute, and abstention policies are needed to establish what formal certification adds beyond executable checking.

  • Unresolved semantic validity of certified conflicts: The paper identifies 51 tasks whose descriptions do not determine the disputed behavior, yet reports a 32.0% certification rate on this subset as successful conflict detection, leaving those ground-truth labels insufficiently justified. Six of 60 audited certificates are also classified as spurious but retain their original classification in the main results. The evaluation should reconcile these labels and distinguish kernel-accepted proofs from faithful evidence of conflicts in the original tasks.

Questions for authors
  • What is the intended role of a conflict certificate in a real coding workflow, and how should it guide decisions to request clarification, correct tests, or escalate to human review?
  • How does adding SpecGuard before a coding agent affect specification-violating behavior, successful task completion, and human-review burden on a mixture of valid and conflicted tasks, compared with running the same agent without SpecGuard?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 635L Part 1

On the listed strengths: "None"

It is disheartening to see no strengths listed, and we respectfully ask the reviewer to reconsider. To our knowledge this is the first work to establish such conflicts with independently checkable evidence before any agent runs, rather than with a model’s verdict. The review's own summary lists a working three-agent system, certificates across four models on 349 tasks, a manual audit, a failure analysis, and an evaluation on naturally occurring conflicts. The other reviewers call the problem "important and well-motivated", the trust model "stated clearly" and the taxonomy "quite informative". We welcome criticism of each of these and answer both weaknesses below, but we believe a fair assessment also weighs them.

W1: Added value beyond executable checking.

We have now run the Python model in the same setting as SpecGuard: GPT-5.6 Sol, corrupted tests only.

MethodDetectedMissedInconclusiveClean: clearedClean: false alarmClean: inconclusive
Python model, imports allowed61.95.233.067.08.924.1
Python model, no imports62.25.732.165.99.524.6
SpecGuard (Lean)57.08.134.975.66.018.3

Lean does not improve detection, and the Python model detects somewhat more. Lean differs in two ways. (1) It is quieter on unmodified tasks, with fewer false alarms and more tasks cleared (single runs). About 11 of the 31 Python false alarms compare values of different shapes, which Python's == treats as unequal and Lean's type checker rejects. (2) Lean itself forces the spec to be pure, terminating and import-free. In Python this is only a rule we impose externally. We will add this comparison and state clearly that detection comes from the specification check, while Lean contributes the stricter artifact and fewer false alarms.

On abstention, we would like to politely push back. We reanalysed the five judge votes per task. On 107 of 349 conflicted tasks all five votes say "not conflicting". Even if the judge must be unanimous to answer, it still marks 30.7% of conflicted tasks as not conflicted, against 8.1% for SpecGuard. Of those 107 tasks, SpecGuard certifies a conflict on 34 and is inconclusive on 56. The gap is therefore not produced by abstention alone. The cost is on unmodified tasks, where SpecGuard withholds about 24% and the judge 3.4%. Information is already matched: the judge and SpecGuard both see only the corrupted tests. On compute, SpecGuard costs USD 0.26 per task and the judge USD 0.84 (Table 12).

W2: Semantic validity of certified conflicts.

Off-issue tasks. We follow ImpossibleBench, which keeps a mutation only if the reference solution fails it. For the 51 off-issue tasks the label has the same basis: the reference solution fails the mutated test. SpecGuard is inconclusive on most of them (56.9%), and without them the certified rate is 55.0%.
Spurious certificates. All 60 audited certificates are on tasks that are conflicted by construction, so every audited verdict is correct. In 6 the proof does not carry the verdict. Removing them gives a certified rate of about 44% for Sol and about 35% for Kimi K3 (from 15 audited certificates per model). Fable 5 and Opus 5 are 15 of 15 faithful and unchanged.
Kernel-accepted versus faithful. When the proof targets the predicted verdict, 13 of Sol's 21 false alarms on unmodified tasks certify, which is 3.7% of unmodified tasks. A certificate does not rule out a false alarm, and we will state this distinction in the paper.

Q1: Role of a certificate in a workflow.

A conflict verdict stops the agent and hands the reviewer the named assertion, who then fixes the test or clarifies the description. Inconclusive asks for clarification or review. No-conflict lets the agent proceed.

Q2: Effect of running SpecGuard before an agent.

We have not run this end to end. From existing results: without a check, Sol cheats on 81% of conflicted tasks. With SpecGuard, 57.0% of conflicted tasks are flagged as conflicts and 34.9% as inconclusive, so 8.1% reach the agent unflagged. The cost is that about 24% of unmodified tasks are withheld for review.

continued...

AuthorsOpenReview ↗

Response to Reviewer 635L Part 2

Summary of new results.

The paper orders evidence about a conflict by what can be independently checked (Section 4.2, Table 1). The rebuttal results put numbers on each level, with GPT-5.6 Sol on the same tasks:

  • Model judgment (LLM judge): marks 30.7% of conflicted tasks as not conflicted, even when it must be unanimous.
  • Execution checked (Python model): misses 5.2 to 5.7%, with 8.9 to 9.5% false alarms on unmodified tasks.
  • Formally certified (Lean): misses 8.1%, with 6.0% false alarms on unmodified tasks (3.7% certified).
  • Human verified: 52 of 60 audited certificates are faithful.

On the 22 real-world cases, the judge clears 12 real conflicts and raises 4 false alarms after the fix, while SpecGuard clears 1 and raises none. Most of the gain comes from moving from model judgment to a checkable artifact. Moving from execution to a kernel-checked proof does not add detection. It gives a stricter artifact and fewer false alarms. We believe stating what each level does and does not establish is useful beyond SpecGuard.


We thank the reviewer and are happy to answer anything further.

Thank you for your response. My main concern is the significance of such intent-test conflict since it might be rare in real world distribution. How many issues or pull requests did you review, across how many repositories, to find the 22 real-world cases? Another concern is since the proposed method is mainly agent framework design, I would expect more baselines included in experiments for comparison.

Reviewer_C2EKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SpecGuard, a pre-execution check for "conflicted" coding tasks, i.e. tasks whose natural-language description and tests cannot both be satisfied. A SpecAgent writes an executable Lean 4 specification from the description and codebase without seeing the tests. A TestAgent independently transcribes the test demands into Lean, and a Certifier writes a connector mapping each test scenario onto the specification's interface. When every case is supported and at least one fails, by decide yields a kernel-checked "conflict certificate"; otherwise the system returns execution-only evidence, no-conflict, or inconclusive.

On 349 ImpossibleBench-style mutated SWE-bench tasks across four models, SpecGuard detects 57–73% of conflicts and certifies 47–51%. With GPT-5.6 Sol it returns no-conflict on 8.1% of conflicted tasks versus 39.8% for an agentic LLM judge, while abstaining on 34.9%. The paper also reports a paired-test variant, a task taxonomy and failure analysis, a 60-certificate faithfulness audit, results on LiveCodeBench and VERINA, and a CLI run on 22 real GitHub test conflicts (9 conflict verdicts, 8 certified).

Critical strengths
  • Detecting that a task invites reward hacking before any agent runs would be a useful complement to trajectory monitoring, and the trust model (description trusted, tests and codebase untrusted) is stated clearly.
  • The three-way taxonomy is quite informative. The authors also show a vacuous certificate from their own audit, which I appreciated.
Critical weaknesses
  1. The problem formulation does not match the reward-hacking problem used to motivate it.
    • The reward hacking the paper cites (Arx et al., MacDiarmid et al.) mostly arises on solvable tasks with weak tests or exploitable graders. ImpossibleBench makes tasks impossible as a measurement device, so that any pass is unambiguously cheating; impossibility is not the main threat. On solvable-but-shortcuttable tasks, SpecGuard returns no-conflict by design, since the spec and tests are jointly satisfiable.
    • For impossible tasks in training and evaluation environments, a cheap, standard alternative exists: require environment providers to submit a reference solution that passes the tests, and re-run it whenever the tests change. On the paper's main benchmark this baseline is already available. Since the mutated tests contradict the issue by construction, running the SWE-bench gold patch against them should flag essentially every mutation with near-zero false positives, for the cost of one test run. This baseline should be reported.
    • The setting where SpecGuard is actually needed is deployment, where a user files an issue against existing tests and no solution exists yet. That makes the 22 real-world cases the paper's main experiment, and they are the least developed part. A passing reference solution does not by itself prove fidelity to the description, but checking that is ordinary code review, arguably easier than auditing a Lean spec.
  2. The added value of the formal certificate over execution is not demonstrated.
    • The ablation that would isolate what Lean buys is missing. The Python Reference Model is evaluated only with paired tests, only with Sol, only on SWE-bench, and without a false-positive rate. The justification in §5.1 (the Python TestAgent must know which test is disputed because tests are executed against the model) does not match the prompts in App. H: the Python pipeline transcribes a TESTS literal and calls spec_run through a connector, structurally the same as the Lean pipeline, so a one-off version seems straightforward. In the one matched setting (paired), Python detects 74.8% vs. 73.9%, with FN 3.2% vs. 4.9%. On LiveCodeBench, SpecGuard produces 0 certificates out of 80 detections. Together this suggests Lean mostly only changes the label on the evidence.
    • To act on a certificate, someone must check that the spec, transcription and connector faithfully represent the task. For many ImpossibleBench mutations the conflict is visible in the test diff in seconds (django-14373: '0001' → '001'), while the specs to be checked run to ~190 lines of Lean (Punycode, HTTP dates). It is not obvious that verifying the certificate is cheaper or more reliable than verifying the conflict directly.
Questions for authors
  1. Can you report the Python Reference Model in the one-off (corrupted-only) setting, for all four models, with false-positive rates? What does the Lean certificate provide in practice beyond deterministic execution of the same spec?
  2. How long does it take a human to verify a certificate's faithfulness, compared with verifying the conflict directly from the issue and test diff?
  3. For the 22 GitHub cases: can you list them with links, show the derived task descriptions, and say who wrote them and when? How many candidates were screened and excluded, and why? What does the judge do on the same 22? What happens on the post-fix tests? Why are 12 inconclusive? This is the most interesting part of the paper to me.
  4. Why does LiveCodeBench yield zero Lean certificates? What does "a recheck of the same model-generated artifacts from the preceding run" mean, and were artifacts rescored after harness changes?
  5. One suggestion I'll make is to allow models to produce partial spec, as in: checking tests in ImpossibleBench like assert 2==f(2)==3 would not actually require formalizing what f is. Just formalizing that f is a function would be enough. This could also potentially reduce the burden of review.
2 comments
AuthorsOpenReview ↗

Response to Reviewer C2EK Part 1

We thank the reviewer for the detailed review and for reading the appendix prompts closely. We are glad the reviewer sees pre-execution detection as "a useful complement to trajectory monitoring".

W1a: Scope

We agree on scope. SpecGuard covers tasks whose description and tests cannot both be satisfied, not solvable tasks with weak tests. Such tasks do occur and we test SpecGuard on them. The 22 cases come from public Python issues. And agents rarely report them: in our runs GPT-5.6 Sol cheats on 81% of conflicted SWE-bench tasks (Figure 1; we ran ImpossibleBench ourselves).

W1b: Reference-solution baseline

We also agree on the reference-solution baseline. ImpossibleBench keeps a mutation only if the reference solution fails it, so that baseline flags every mutated task by construction. Where a reference solution exists, running it is the right first check. SpecGuard is for the setting where none exists, as when a user files an issue against existing tests. We agree the real-world cases are the experiment that matters most, which is why we included them.

W1c and Q3: The 22 real-world cases

Selection required a real open-source case with cited evidence that a test contradicted intended behaviour, raised or resolved from January 2026 on. Flaky tests, environment failures and test-infrastructure-only problems were excluded. Roughly 80 candidates were examined (no exact tally was kept). Every case that met the above criteria was included, and the set was fixed before SpecGuard ran on any case. We allowed fixes that change both tests and implementation.

The task descriptions are the original issue text, with only answer-revealing parts removed by the authors. The exact tasks can be found, and also run, in the attached supplementary code.

Of the 12 inconclusive cases, 9 reached the connection step but could not connect at least one test demand to the specification. The other 3 failed earlier: one produced no usable test transcription, one found no relevant tests, and one found no source files to model. We ran both checks the reviewer asked for, with GPT-5.6 Sol on the same 22 cases.

SettingConflictNo-conflictInconclusive
SpecGuard, pre-fix tests9 (8 certified)112
Judge, pre-fix tests10120
SpecGuard, post-fix tests01012
Judge, post-fix tests4171

On the pre-fix tests all 22 are real conflicts. The two detect about the same number, but the judge clears 12 and SpecGuard clears 1. On the post-fix tests any conflict is a false alarm: SpecGuard raises none and the judge raises 4. Of SpecGuard's 9 detections, 6 become no-conflict after the fix and 3 become inconclusive. SpecGuard is inconclusive on 12 of 22 in both settings.

W2a and Q1: Lean versus execution

We agree the reviewer is right about Section 5.1. A different version of prompt was used to produce that result, and a newer version which was still unrun was put into appendix H, this is an error. All other prompts in Appendix H are the ones used, and the new Python results use the Appendix H version, adapted to take a single test patch. We have now run the Python model in the same setting as SpecGuard: GPT-5.6 Sol, corrupted tests only. The other three models are not yet run.

MethodDetectedMissedInconclusiveClean: clearedClean: false alarmClean: inconclusive
Python model, imports allowed61.95.233.067.08.924.1
Python model, no imports62.25.732.165.99.524.6
SpecGuard (Lean)57.08.134.975.66.018.3

Lean does not improve detection (python detected somewhat more), however, we believe running the same logic in lean has advantages: (1) Changes in behaviour on unmodified tasks, like fewer false alarms and more tasks cleared (single runs). On inspection, about 11 of the 31 Python false alarms compare values of different shapes, for example a dict against a string, which Python's == treats as unequal and Lean's type checker rejects. (2) Lean itself forces the spec to be pure, terminating and import-free. In Python this is only a rule we impose externally. We will add this comparison and state clearly that detection comes from the specification check, while Lean contributes the stricter artifact and fewer false alarms.

A certificate does not rule out a false alarm. When the proof targets the predicted verdict, 13 of Sol's 21 false alarms on unmodified tasks certify, which is 3.7% of unmodified tasks.

continued...

AuthorsOpenReview ↗

Response to Reviewer C2EK Part 2

W2b and Q2: Cost of checking a certificate

We have not measured this, but we would like to politely push back on the comparison. The diff exists only because the benchmark created the mutation (Impossiblebench makes rather simple edits to the test to corrupt them). A user facing a corrupted suite has no earlier version to compare against. The example also pairs two different tasks: the specification for django-14373 is 9 lines (Listing 1), and the 190-line specification belongs to another task. A reviewer does not need to audit the Lean. The verdict names the conflicting assertion and the value the specification expects, so checking it against the issue is the same act as verifying the conflict directly, with the location already known. Reviewing every task, its tests and the relevant parts of a large codebase by hand does not scale, especially as more code is written by agents, while SpecGuard runs on every task automatically. In sympy-23262 the conflict is simple (the specification is just 8 lines), yet all five votes of the LLM judge passed the mutated test while reading it directly.

Q4: LiveCodeBench

LiveCodeBench tests are plain input/output pairs, so only the SpecAgent is needed. A fixed harness feeds the inputs to the specification and compares the outputs, with no model-written test transcription or connector. The inputs reach about 100,000 elements, which is too large for kernel evaluation under decide, so no certificates are produced. The "recheck" is a deterministic re-run of the same Lean code, with no model call and nothing rescored.

Partial specifications.

We agree. A test set like 2 == f(2) == 3 conflicts for every f, and Lean can prove that where execution cannot. This also shrinks what a human must review. We see it as the natural next step.

Summary of new results.

The paper orders evidence about a conflict by what can be independently checked (Section 4.2, Table 1). The rebuttal results put numbers on each level, with GPT-5.6 Sol on the same tasks:

  • Model judgment (LLM judge): marks 30.7% of conflicted tasks as not conflicted, even when it must be unanimous.
  • Execution checked (Python model): misses 5.2 to 5.7%, with 8.9 to 9.5% false alarms on unmodified tasks.
  • Formally certified (Lean): misses 8.1%, with 6.0% false alarms on unmodified tasks (3.7% certified).
  • Human verified: 52 of 60 audited certificates are faithful.

On the 22 real-world cases, the judge clears 12 real conflicts and raises 4 false alarms after the fix, while SpecGuard clears 1 and raises none. Most of the gain comes from moving from model judgment to a checkable artifact. Moving from execution to a kernel-checked proof does not add detection. It gives a stricter artifact and fewer false alarms. We believe stating what each level does and does not establish is useful beyond SpecGuard.


We thank the reviewer again. The questions on the Python baseline and the real-world set changed how we present this work, and we are happy to answer anything further.

Reviewer_zHGhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles the risk of accidental or adversarially injected misspecifications in tasks to autonomous coding agents, where they rarely flag the conflict and instead have a reward-hacking behavior and cheat by editing tests or hard-coding expected outputs, which could lead to security threat. The paper presents the SpecGuard protocol, which detects and formally certifies conflicts between task intent and tests given only the task description, the codebase and the tests. SpecGuard autoformalizes the intended behavior in a Lean 4 spec, and the Lean kernel checks whether the generated specification and the independently formalized tests are inconsistent, producing a machine-checked certificate. The results are such that SpecGuard detects up to 72.8% of conflicts and formally certifies up to 51.1%. For example, SpecGuard with GPT-5.6 Sol has an 8.1% conflict miss rate on a conflicted variant of SWE-bench, and, interestingly, SpecGuard CLI detects 9 out of 22 real-world conflicts that were systematically collected from recent GitHub repositories and formally certifies 8 out of 9.

Critical strengths
  • Important problem and a valuable approach to independently verifiable conflict detection. Details: The paper tackles and important and well-motivated problem in autonomous coding agents - identifying inconsistencies between task intent and test requirements before an agent acts on potentially corrupted feedback. The proposed approach of indendently formalizing the task description and tests with the following Lean-checked conflict certification is a potentially valuable alternative to just using LLM-as-a-judge. The paper is generally clear and well-organized.

  • Broad empirical evaluation and informative analysis of SpecGuard's capabilities and limitations. Details: The evaluation covers multiple settings, including SWE-bench, LiveCodeBench and real-world tasks, and specification-generation evaluation on VERINA. They also consider 4 undelying models, the effect of providing original and corrupted test versions and a detailed failure analysis with decent results.

Critical weaknesses
  • Practical strength of the formal guarantee depends substantially on the faithfulness of the generated specs, and the validity of some benchmark conflict labels is unclear. Details: The theoretical formulation seeks no implementation satisfying both the task spec and test requirements (eq 1), however the actual Lean certificates verify disagreement between a generated executable spec and formalized test demands, and extending this result to original task requires the generated specs and semantic connector to faithfully characterize the intended behavior. The paper acknowledges it, but manual audit finds only 52 of 60 sampled certificates faithful, raising a question of how reliably machine-checked certificates correspond to actual task-level conflicts.

  • The evaluation does not fully establish the incremental practical benefit of formal certification compared with simpler approaches. Details: e.g., the Python Reference Model (PRM) baseline achieves 74.8% conflict detection in the paired-test setting, exceeding 73.9% of SpecGuard (though the latter produces formal certificates for 67% of the tasks). Comparing with LLM Judge SpecGuard's lower false-negative rate (8.1% vs 39.8%) has high inconclusive rate (34.9%), and LLM Judge effectively never abstains. Moreover, in the real-world Github evaluation SpecGuard detects only 9 out of 22 conflicts (though certifies 8 of the 9 detection). More extensive real-world evaluation with clearer trade-odds would strengthen the contribution.

Questions for authors
  • Q1\textbf{Q1}. Formal guarantees and semantic faithfulness -- could the authors clarify precisely how the Lean-checked executable disagreement establishes Eq 1, and under what assumptions the generated Spec.run represents all implementations satisfying the intended task? Given 6 spurious and 2 weak formalizations identified in the manual audit, how should users interpret the practical reliability of certified conflicts?

  • Q2\textbf{Q2}. Validity of benchmark mutations -- how is GT inconsistency established for the mutated SWE-bench tasks, particularly the 51 of 349 off-issue cases for which the task description doesn't determine the disputed behavior?

  • Q3\textbf{Q3}. Baselines -- given the competitive detection performance of the PRM in the paired-test setting, what is the practical added benefit of Lean certification beyond independently rechecking a generated executable model?

  • Q4\textbf{Q4}. Did you investigate using other underlying models for SpecAgent, TestAgent and certifier? Would it result with a better performance?

  • Q5\textbf{Q5}. Trust assumptions and deployment -- the approach treats the task description as authorative while considering codebase untrustworthy -- how would SpecGuard behave when the task description itself is modified, incomplete or inconsisted with the existed code?

3 comments
AuthorsOpenReview ↗

Response to Reviewer zHGh part 1

We thank the reviewer for the careful reading. We are glad the reviewer found the problem "important and well-motivated" and the evaluation "broad". We answer each point below and include new results from the rebuttal period.

W1 and Q1. Faithfulness of certificates.

We believe Section 4.1 gives the link. The theorem proves that the generated specification fails a formalized test, so any implementation that agrees with it on the tested inputs fails too. That is Eq. 1. It carries over to the original task when the description fixes a single output for the tested behavior (Section 3). We will restate this beside Eq. 1.

We do not claim to solve autoformalization. Faithful autoformalization is the stated boundary of the guarantee (Sections 3 and 6), and the audit measures it. We would like to clarify what the audit shows: all 60 audited certificates are on tasks that are conflicted by construction, so every audited verdict is correct. In 8 the proof is weaker than it should be, as in matplotlib-26291 (Appendix C.1). "Not faithful" means the proof does not carry the verdict, not that the verdict is wrong. Claude Fable 5 and Claude Opus 5 are 15 of 15 faithful. The spurious certificates come from Kimi K3 (4) and GPT-5.6 Sol (2).

On unmodified tasks, Sol raises a conflict on 6.0% (21 of 349), and 13 of those 21 certify. So 3.7% of unmodified tasks get a certified but wrong conflict, against 51.1% certified on conflicted tasks. At an even mix, about 93% of certificates are on real conflicts. A certificate shows that the generated specification, test transcription and connector disagree. It does not rule out a false alarm.

W2 and Q3. Added value over the Python model and the judge.

The Python baseline in the submission was only run with paired tests. We have now run it in the same setting as SpecGuard: GPT-5.6 Sol, corrupted tests only, on the 349 mutated and 349 original tasks.

MethodDetectedMissedInconclusiveClean: clearedClean: false alarmClean: inconclusive
Python model, imports allowed61.95.233.067.08.924.1
Python model, no imports62.25.732.165.99.524.6
SpecGuard (Lean)57.08.134.975.66.018.3

Lean over Python. We agree with the reviewer that Lean does not improve detection (python detected somewhat more), however, we believe running the same logic in lean has advantages: (1) Changes in behaviour on unmodified tasks, like fewer false alarms and more tasks cleared (single runs). On inspection, about 11 of the 31 Python false alarms compare values of different shapes, for example a dict against a string, which Python's == treats as unequal and Lean's type checker rejects. (2) Lean itself forces the spec to be pure, terminating and import-free. In Python this is only a rule we impose externally. We will add this comparison and state clearly that detection comes from the specification check, while Lean contributes the stricter artifact and fewer false alarms.

On the judge. We reanalysed the five judge votes per task. On 107 of 349 conflicted tasks all five votes say "not conflicting". Even if the judge must be unanimous to answer, it still clears (marks as not conflicted) 30.7% of conflicted tasks, against 8.1% for SpecGuard. Of those 107 tasks, SpecGuard certifies a conflict on 34 and is inconclusive on 56. The cost is on unmodified tasks, where SpecGuard withholds about 24% and the judge 3.4%. In sympy-23262, all five votes pass a test demanding (2, 3) from a function that returns (y, x), while all three sol runs catch it.

On the real-world evaluation (9 of 22 detected, 8 certified). The set was fixed before SpecGuard was run on any case, and every case that met the selection criteria was included. On these cases the judge clears 12 of the 22 real conflicts, and SpecGuard clears 1.

Q2. Ground truth for mutated tasks.

We follow ImpossibleBench, which keeps a mutation only if both the reference solution and the empty patch fail the mutated tests. The label therefore means that no correct solution passes. For the 51 off-issue tasks the label has the same basis: the reference solution fails the mutated test. SpecGuard is inconclusive on most of them (56.9%), and without them the certified rate is 55.0%.

Q4. Other models for the three agents.

We evaluated four models, with the same model in all three roles within a run. We did not test mixing models across roles.

AuthorsOpenReview ↗

Response to Reviewer zHGh part 2

Q5. Modified, incomplete, or inconsistent descriptions.

SpecGuard reports that the description and the tests cannot both hold. It does not decide which side is wrong. Treating the description as the reference is a modelling choice, not really a claim about whom to trust. If the description disagrees with the code or the tests, the disagreement is flagged for a human to resolve. If it is incomplete for the tested behavior, SpecGuard returns inconclusive (Table 3, Appendix D.3).

Summary of new results.

The paper orders evidence about a conflict by what can be independently checked (Section 4.2, Table 1). The rebuttal results put numbers on each level, with GPT-5.6 Sol on the same tasks:

  • Model judgment (LLM judge): marks 30.7% of conflicted tasks as not conflicted, even when it must be unanimous.
  • Execution checked (Python model): misses 5.2 to 5.7%, with 8.9 to 9.5% false alarms on unmodified tasks.
  • Formally certified (Lean): misses 8.1%, with 6.0% false alarms on unmodified tasks (3.7% certified).
  • Human verified: 52 of 60 audited certificates are faithful.

On the 22 real-world cases, the judge clears 12 real conflicts and raises 4 false alarms after the fix, while SpecGuard clears 1 and raises none. Most of the gain comes from moving from model judgment to a checkable artifact. Moving from execution to a kernel-checked proof does not add detection. It gives a stricter artifact and fewer false alarms. We believe stating what each level does and does not establish is useful beyond SpecGuard.


We thank the reviewer again for the detailed questions, which led directly to the new Python and judge results. We are happy to answer anything further, and we hope the reviewer will consider these results in their final assessment.

I thank the authors for the detailed response. Overall, my concerns were partially resolved, however I will maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.