- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SpecGuard investigates whether conflicts between a coding task’s intended behavior and its tests can be identified before an autonomous agent begins execution. It treats the task description as authoritative and allows the tests to contain accidental or adversarial errors.
The system uses three agents. A SpecAgent reads the description and repository, with tests withheld, and produces an executable Lean specification. A TestAgent independently formalizes test requirements. A Certifier connects the representations, enabling Lean to check for disagreement. Outputs include certified conflicts, execution-supported conflicts, no-conflict verdicts, and inconclusive results. Certificates establish inconsistency between the generated representations; their correspondence to the original task requires separate validation.
On 349 mutated SWE-bench tasks across four models, SpecGuard reports conflict detection of up to 72.8% and certification of up to 51.1%. Providing both original and corrupted tests raises Sol’s certification rate from 51.1% to 67.0%. A manual audit classifies 52 of 60 sampled certificates as faithful.
Additional evaluations cover 22 naturally occurring GitHub conflicts, yielding nine detections, eight certified, and 95 LiveCodeBench tasks, yielding 80 execution-supported detections. On VERINA, 180 of 187 generated specifications are proved equivalent to human-written references. The failure analysis identifies alignment between specifications and test representations as a major bottleneck.
Critical strengths
None
Critical weaknesses
-
Unclear added value beyond structured inference-time checking: Most of SpecGuard’s work lies in generating and aligning representations, while Lean certifies the resulting computation. The paired-test Python baseline achieves 74.8% detection versus SpecGuard’s 73.9% at similar cost, and the headline miss-rate reduction substantially reflects abstention. Comparisons with matched information, compute, and abstention policies are needed to establish what formal certification adds beyond executable checking.
-
Unresolved semantic validity of certified conflicts: The paper identifies 51 tasks whose descriptions do not determine the disputed behavior, yet reports a 32.0% certification rate on this subset as successful conflict detection, leaving those ground-truth labels insufficiently justified. Six of 60 audited certificates are also classified as spurious but retain their original classification in the main results. The evaluation should reconcile these labels and distinguish kernel-accepted proofs from faithful evidence of conflicts in the original tasks.
Questions for authors
- What is the intended role of a conflict certificate in a real coding workflow, and how should it guide decisions to request clarification, correct tests, or escalate to human review?
- How does adding SpecGuard before a coding agent affect specification-violating behavior, successful task completion, and human-review burden on a mixture of valid and conflicted tasks, compared with running the same agent without SpecGuard?
5 comments
Response to Reviewer 635L Part 2
Summary of new results.
The paper orders evidence about a conflict by what can be independently checked (Section 4.2, Table 1). The rebuttal results put numbers on each level, with GPT-5.6 Sol on the same tasks:
- Model judgment (LLM judge): marks 30.7% of conflicted tasks as not conflicted, even when it must be unanimous.
- Execution checked (Python model): misses 5.2 to 5.7%, with 8.9 to 9.5% false alarms on unmodified tasks.
- Formally certified (Lean): misses 8.1%, with 6.0% false alarms on unmodified tasks (3.7% certified).
- Human verified: 52 of 60 audited certificates are faithful.
On the 22 real-world cases, the judge clears 12 real conflicts and raises 4 false alarms after the fix, while SpecGuard clears 1 and raises none. Most of the gain comes from moving from model judgment to a checkable artifact. Moving from execution to a kernel-checked proof does not add detection. It gives a stricter artifact and fewer false alarms. We believe stating what each level does and does not establish is useful beyond SpecGuard.
We thank the reviewer and are happy to answer anything further.
Thank you for your response. My main concern is the significance of such intent-test conflict since it might be rare in real world distribution. How many issues or pull requests did you review, across how many repositories, to find the 22 real-world cases? Another concern is since the proposed method is mainly agent framework design, I would expect more baselines included in experiments for comparison.
Response to Reviewer 635L Part 1
On the listed strengths: "None"
It is disheartening to see no strengths listed, and we respectfully ask the reviewer to reconsider. To our knowledge this is the first work to establish such conflicts with independently checkable evidence before any agent runs, rather than with a model’s verdict. The review's own summary lists a working three-agent system, certificates across four models on 349 tasks, a manual audit, a failure analysis, and an evaluation on naturally occurring conflicts. The other reviewers call the problem "important and well-motivated", the trust model "stated clearly" and the taxonomy "quite informative". We welcome criticism of each of these and answer both weaknesses below, but we believe a fair assessment also weighs them.
W1: Added value beyond executable checking.
We have now run the Python model in the same setting as SpecGuard: GPT-5.6 Sol, corrupted tests only.
Lean does not improve detection, and the Python model detects somewhat more. Lean differs in two ways. (1) It is quieter on unmodified tasks, with fewer false alarms and more tasks cleared (single runs). About 11 of the 31 Python false alarms compare values of different shapes, which Python's == treats as unequal and Lean's type checker rejects. (2) Lean itself forces the spec to be pure, terminating and import-free. In Python this is only a rule we impose externally. We will add this comparison and state clearly that detection comes from the specification check, while Lean contributes the stricter artifact and fewer false alarms.
On abstention, we would like to politely push back. We reanalysed the five judge votes per task. On 107 of 349 conflicted tasks all five votes say "not conflicting". Even if the judge must be unanimous to answer, it still marks 30.7% of conflicted tasks as not conflicted, against 8.1% for SpecGuard. Of those 107 tasks, SpecGuard certifies a conflict on 34 and is inconclusive on 56. The gap is therefore not produced by abstention alone. The cost is on unmodified tasks, where SpecGuard withholds about 24% and the judge 3.4%. Information is already matched: the judge and SpecGuard both see only the corrupted tests. On compute, SpecGuard costs USD 0.26 per task and the judge USD 0.84 (Table 12).
W2: Semantic validity of certified conflicts.
Off-issue tasks. We follow ImpossibleBench, which keeps a mutation only if the reference solution fails it. For the 51 off-issue tasks the label has the same basis: the reference solution fails the mutated test. SpecGuard is inconclusive on most of them (56.9%), and without them the certified rate is 55.0%.
Spurious certificates. All 60 audited certificates are on tasks that are conflicted by construction, so every audited verdict is correct. In 6 the proof does not carry the verdict. Removing them gives a certified rate of about 44% for Sol and about 35% for Kimi K3 (from 15 audited certificates per model). Fable 5 and Opus 5 are 15 of 15 faithful and unchanged.
Kernel-accepted versus faithful. When the proof targets the predicted verdict, 13 of Sol's 21 false alarms on unmodified tasks certify, which is 3.7% of unmodified tasks. A certificate does not rule out a false alarm, and we will state this distinction in the paper.
Q1: Role of a certificate in a workflow.
A conflict verdict stops the agent and hands the reviewer the named assertion, who then fixes the test or clarifies the description. Inconclusive asks for clarification or review. No-conflict lets the agent proceed.
Q2: Effect of running SpecGuard before an agent.
We have not run this end to end. From existing results: without a check, Sol cheats on 81% of conflicted tasks. With SpecGuard, 57.0% of conflicted tasks are flagged as conflicts and 34.9% as inconclusive, so 8.1% reach the agent unflagged. The cost is that about 24% of unmodified tasks are withheld for review.
continued...