acceptodds
Under review as a conference paper at ICLR 2027

SpecGuard: What Should LLMs See to Generate Assertions That Catch RTL Bugs?

Abstract

Hardware bugs that escape the rigorous verification and validation process are costly to fix, often requiring recalls, software patches that mask the error rather than fix it, and similar remedial actions. This motivates rigorous verification of each implementation against its intended behavior. Assertion-based verification captures this intent using SystemVerilog Assertions (SVA), which can be used in both formal and simulation-based verification. However, composing SVAs remains largely manual and time-consuming, causing a bottleneck in time-to-market. LLMs show promise in automating this by translating a design’s natural-language specification into SVA without manual formalization. In practice, SVA generation through LLM is impeded by restating shown implementation behavior, placing claims in the wrong clock cycle, and naming signals outside the checked module. Recent studies show SVAs generated by frontier models detect only 5.1–11.7% of injected bugs. In this paper, we analyze assertion suites generated by existing LLM-based SVA generation methods. We evaluate them against injected bugs and identify four failure modes, along with two attributes that place the resulting assertions beyond repair after generation, rendering them ineffective for actual bug detection. These failure modes and attributes are not captured by the compile and proof rates typically used to evaluate generators. While a stronger model improves these rates, its generated assertions do not necessarily detect more bugs. We subsequently develop SpecGuard, an SVA generation framework that addresses each failure by separating what an assertion must draw from a model versus what can be fixed with design’s structure. SpecGuard was evaluated on the 83-design AssertLLM2 benchmark, using a local 14.8-B model with int4 quantization. Its assertions detected up to 66% of the AssertLLM2 bugs in a single run, compared with 11.7% for the best-performing of six frontier models. Furthermore, on more complex hierarchical designs from NVDLA and Google Coral NPU, it achieves up to a 51% bug-detection rate, compared with 14% for existing state-of-the-art assertion-generation techniques.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Assertion-based verification, LLM-based hardware verification, Design specification translation, Hardware bug detection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bTAROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper investigates why LLM-generated hardware assertions often fail to detect RTL bugs. The authors identify four failure modes in existing assertion generation methods and propose SpecGuard, which provides LLMs with structural information extracted from RTL while withholding implementation behavior. The framework also decomposes specifications into fine-grained questions and uses deterministic rules to repair assertion timing and module binding. Experiments on AssertLLM2 and hierarchical hardware designs report substantial improvements in bug detection over existing methods.

Critical strengths
  1. The authors identify an important problem: LLM-generated assertions may reproduce implementation defects rather than detect them. The proposed separation between specification and implementation behavior is an interesting approach to improving the reliability of assertion generation.
  2. The FifoX case study shows that decomposing a general property into specific cases can substantially improve bug detection. This provides useful insights into assertion generation.
  3. SpecGuard combines structural information extraction, question decomposition, and deterministic assertion repair. The reported improvement in bug detection is substantial.
Critical weaknesses
  1. My major concern is that there are several inconsistencies in the experimental results and settings, which led me to give a low soundness score:
    • The paper claims evaluation on all 83 designs of AssertLLM2. However, Table 14 reports only 10 designs, and the aggregate detection rate (57.9%) also matches Table 13. It is unclear whether the results cover the full benchmark.
    • The paper claims that Table 15 reports the numbers behind Figure 4, but Figure 4 reports a detection rate of 51% for FifoX, whereas Table 15 reports 94.4%.
    • Appendix D.3 states that "6 bugs become newly detected", but Figure 6 reports much larger numbers of newly detected bugs.
    • Section 5.1 states that the baselines use GPT-4o, whereas Appendix A states that all methods use Qwen3-14B.
  2. A much simpler prompt engineering approach might already address the main problem. For example, we could provide both the specification and RTL to the LLM and explicitly prompt it to follow the specification and treat the RTL as potentially buggy. However, the paper does not compare against this simple baseline, so it is unclear whether the proposed much more complicated approach is really necessary.
  3. It is possible that the problem studied in this paper is mainly due to the limited capabilities of the tested LLMs, since the main experiments use Qwen3-14B and GPT-4o, which are already relatively old models. More recent and capable LLMs may be better at following the specification without being misled by incorrect RTL implementations. Therefore, it is unclear whether SpecGuard's advantage would persist as LLM capabilities improve.
  4. The theoretical formulation is unnecessarily complicated. Lemma 1 and Propositions 1 and 2 are fairly intuitive observations that follow directly from the definitions (Their one- or two-sentence proofs are almost trivial), and I do not think they require such extensive formalization. The optimization formulation in Equation (2) also seems to mainly restate the proposed design principle without providing additional theoretical insights. Meanwhile, Lemma 2 makes a more substantive claim, but its brief proof does not adequately justify it. Overall, I think the theoretical presentation could be substantially simplified without losing the main ideas.
  5. For presentation, the paper could benefit from more concrete examples to explain its main ideas. For instance, the Introduction uses an off-by-one error in a loop bound to illustrate how LLM-generated assertions may reproduce bugs in the implementation, but the explanation is entirely verbal, without any code examples. A simple example showing the specification, buggy implementation, and resulting incorrect assertion would make the problem much easier to understand.
Questions for authors
  1. Could the authors explain the inconsistencies in the experimental results mentioned in Weakness 1?
  2. How does SpecGuard compare with a simple prompting approach that provides both the specification and RTL but explicitly prompts the LLM to follow the specification and treat the RTL as potentially buggy?
  3. Have the authors evaluated SpecGuard using more recent and capable LLMs? Does its advantage over existing methods persist with stronger models?
4 comments

We thank the reviewer for the detailed review. We have reconciled the reported results, added the suggested prompting baseline, simplified the theoretical presentation, and added a concrete example in the revised manuscript.

W1 / Q1. Inconsistencies. We thank the reviewer for identifying these oversights in the submitted text, which the revision corrects.

  • (i) 83 vs. 10 designs. We evaluated the benchmark in batches of 10 designs, and the submission reported the first batch in the main paper without clarifying the batching. The revision states the scope explicitly. Fifteen designs do not build in our toolchain (11 fail to parse, 4 fail in Verilator), and 51 mutants do not compile, so all methods are scored on the same 68 designs and 1,309 mutants. SpecGuard detects 57.7% against 23.6% for the strongest baseline (Tabs. 5, 16, 17), and Tab. 6 details the first batch, the 10 development designs (57.9%).
  • (ii) FifoX, 51% vs. 94.4%. These are two configurations. SpecGuard was developed on FifoX, and its development configuration sets the lane and pointer structure by hand, so its 94.4% is optimistic. The general pipeline runs without that structure, and its 51.4% is the number we report (Fig. 6, Tab. 7). Tab. 18 lists both, and the ablation of Tab. 20 is labeled as the development configuration (Sec. 3.1, App. D.4).
  • (iii) Repair counts. For clarity, we replaced the submitted Fig. 6 with a per-rule account. Disabling one rule at a time and rescoring the same 494 assertions, retiming adds 4 detected bugs and the reset-guard and polarity rules add 1 each, for 6 in total with none lost (App. D.4, Tab. 25).
  • (iv) GPT-4o vs. Qwen3-14B. Every method in Sec. 5 runs Qwen3-14B (Sec. 5.1, App. A). GPT-4o appears only in the Tab. 2 case study.

W2 / Q2. A "follow the spec, the RTL may be buggy" prompt. We added this baseline in its strongest form. For each mutant the model receives the specification and that mutant's buggy RTL, with AssertLLM2's own bug-hunting prompt, and writes a separate suite (App. A, Tab. 8). It detects 18.3% against 57.7% for SpecGuard on the same 68 designs (Tab. 5). On FifoX, adding the RTL source to SpecGuard's own prompt lowers detection from 94.4% to 20.8% (Tab. 20, development configuration). Told to distrust the RTL, the model still restates it, and equivalence invariance means no later rewrite can undo that (Secs. 4.1, 5.3).

W3 / Q3. Stronger models. We have not run SpecGuard with a frontier model, as stated in Sec. 5.6. Three findings suggest that capability alone does not close the gap. AssertLLM2's six frontier models, including GPT-5.2, Claude-Sonnet-4.5, and Gemini-2.5-Pro, detect at most 11.7% (Tab. 1). Moving prior generators from Qwen3-14B to GPT-4o improves every surface measure, yet five of six suites still detect none of the 48 FifoX bugs (Tab. 2). With the model fixed, detection rises from 5 to 17 to 41 of 48 as only the questions get finer (Tab. 3). Equivalence invariance and self-filtering follow from the definition of detection, so they hold for any model shown the buggy RTL.

W4. Theory. We agree and have simplified Sec. 4. The optimization program and the Lemma and Proposition apparatus are removed. Sec. 4.1 states the constraint in prose and presents equivalence invariance and self-filtering as direct consequences of the definition of detection. Structural determinacy, formerly Lemma 2, is now argued from register and wire semantics, and retiming supports it empirically, raising detection from 10 to 14 of 20 bugs on versatile_counter with no model call (App. D.4).

W5. Concrete example. We agree and have added Fig. 1. It shows a specification line, the buggy RTL line of a FifoX bug, the assertion copied from that line, and the assertion written from the specification, with each assertion's verdict on correct and buggy RTL.

Thank you for the detailed response and the revised manuscript. Now most of the inconsistencies have been resolved, so I have increased my soundness assessment. However, I still notice that some inconsistencies remain in the revised manuscript. For example, Figure 5 is identical to Figure 3 in the original submission, so it appears to show the old results for the 10 designs rather than the updated results for 68 designs. Could the authors check whether this figure needs to be updated?

Also, since the revised manuscript contains substantial changes, it would be helpful to highlight the revisions in a different color so that reviewers can easily identify what has changed.

We sincerely thank the reviewer for their comments and apologize for the oversight. We have revised the figure accordingly, and all changes made in response to the review are highlighted in blue in the revised manuscript.

Dear Reviewer,

As the discussion period ends today, we would appreciate your consideration of our rebuttal and additional results. We hope they address your concerns and would welcome any remaining questions. Thank you for your time and consideration.

Reviewer_2ai6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why LLM-generated hardware verification oracles (SystemVerilog Assertions) catch so few bugs — frontier models detect only 5.1–11.7% of injected mutants on AssertLLM2 — and argues the limit lies in what the model is shown, not which model is used. Four failure modes are diagnosed: implementation capture, missing structure, wrong scope, and incomplete enumeration. Two formal properties explain why these failures are unrepairable after generation: equivalence invariance and self-filtering. The remedy is information-constrained generation: the model's context may depend on the design under test only through its structure (declarations, widths, module scope, drive kind, update timing), never its behavior.

Critical strengths

1 This work presents a clean conceptual contribution. Casting oracle generation as an information-leakage problem and formalizing information-constrained generation gives the field a portable vocabulary; equivalence invariance and self-filtering precisely explain two widely observed but poorly understood phenomena, post-hoc refinement barely helps and compile/proof rates overstate quality. 2. The diagnosis is causally demonstrated. Fixing model, specification, and bug set while varying question granularity on FifoX (5 → 17 → 41 of 48 bugs detected) cleanly shows "what is asked" dominates "which model answers."

Critical weaknesses
  1. The ceiling is set by specification completeness, and enumeration currently needs per-design configuration. Specs that never name a bug bound any spec-reading generator; the FifoX lane/pointer structure was hand-configured; question templates are validated on few design styles; scaling of question/token cost to industrial designs (thousands of registers) is uncharacterized.
  2. Detection is an any-assertion-fails metric, and suite size makes spurious kills possible. With ~384 assertions per design on average, how much of the 57.9% comes from effective assertions versus incidental kills, and the false-alarm burden, are unquantified.
  3. No experiments outside hardware despite motivating software oracles; robustness of the guard and elaboration-dependent structure (generate blocks, parameter-defined widths) is unclear.
Questions for authors
  1. How enumerations are produced and per-design effort;
  2. whether the 19% of mutants violating Assumption 1 detect differently;
  3. please analyze the share of detection attributable to effective assertions plus false-alarm burden;
  4. it is better to provide misclassification rates of the deterministic repairs on non-standard clock/reset styles.
3 comments

We thank the reviewer for the constructive assessment. We have clarified how enumeration is produced and what it costs, the share of detection from effective assertions, the false-alarm burden, and the scope of Assumption 1 in the revised manuscript.

W1 / Q1. Specification ceiling, enumeration, cost. We agree that no specification-reading generator can detect a bug the specification never states. The same holds for human-written suites. An oracle needs an independent source of correct behavior (Weyuker, 1982), and where no specification suffices, that source is the tester's knowledge of informal specifications, norms, and domain information (Barr et al., TSE 2015). In hardware, the verification plan is derived from the functional specification, since verification "can only detect errors that it is looking for" (Bergeron et al., VMM for SystemVerilog, 2006). We measure this ceiling in Sec. 5.5 (App. D.3, Tab. 19) and state it in Sec. 5.6.

Enumeration is computed by code, not by the model. From the parsed inventory, code derives one question per member of each signal family, such as each register that may hold and each one-bit flag, repeated for every submodule. For a rule over a family, code computes the reachable cases. On FifoX the rank of lane j cannot exceed j, so four lanes give 10 lane and pointer pairs rather than 16. Unanswered questions are recorded as silence (Sec. 4.3, App. D.7, Tab. 23). AssertLLM2 needs no per-design configuration. FifoX's lane structure was set by hand only in the development configuration (94.4%), and the general pipeline detects 51.4% without it.

Generation averages 105 model calls and 227k prompt tokens per design. Cost grows with the number of selector questions, from 68k tokens for 5 questions on srdy-drdy-library to 575k for 68 on reed_solomon_decoder (Sec. 5.3, App. D.11). In terms of effort, this cost is compute only, as no engineer writes questions or per-design configuration, and one local GPU runs each design. Since the questions are independent and can be batched, we expect token usage to grow linearly with the number of questions. Wall-clock time per design ranges from 15 minutes for the smallest design to 4 hours for NVDLA CMAC.

W2 / Q3. Effective assertions and false alarms. The 383.7 assertions are the total over the 10 development designs, so SpecGuard ships 38.4 assertions per design there and 39.5 over all 68 designs (Tabs. 6, 16). Tab. 6 adds two columns that separate effective from incidental kills. On average 5.7 assertions achieve all of a design's kills, and 51.8% of kills are made by one assertion only. Spurious kills by false assertions are excluded, since only assertions proved on the correct RTL earn credit. The false-alarm burden is the broken share, the assertions that fail on correct RTL. It is 13.9% for SpecGuard against 54.3–62.2% for the prior generators, and 39.8% of SpecGuard's assertions are effective against 4.0–8.1% (Tab. 5).

Q2. Mutants violating Assumption 1. We now measure the assumption directly by comparing each mutant's parse with its pre-mutation file. Structure is unchanged for 381 of 398 scored mutants (95.7%, Sec. 4.1). All 17 exceptions are NVDLA mutants that delete a net's only driver, changing its drive kind. Every AssertLLM2 mutant therefore satisfies the assumption, and the exceptions cannot affect the AssertLLM2 results. The submission's 19% included mutants that do not compile or synthesize, now excluded.

Q4. Repair misclassification on non-standard clock/reset styles. A deterministic repair can misclassify only when the structural fact it reads is wrong (Tab. 24). For reset styles, the inferred reset polarity matches the specification on all 6 designs that state it, a misclassification rate of 0%. For temporal form, a blind diagnosis attributes 24 of 248 failing rows (9.7%) to a misclassified timing (App. B.4). We have not measured misclassification rates on non-standard clock styles, such as multiple or gated clocks, or on mixed synchronous and asynchronous resets.

W3. Beyond hardware; guard; generate blocks and parameters. Structure is extracted after elaboration, so generate blocks and parameter-defined widths are resolved before the model sees them (Sec. 5.6). The guard halts a run whenever a prompt contains RTL beyond a one-line quotation of the specification (Sec. 4.2). The method itself carries over to software, since it reads only the specification and the structure of a parse tree. For a unit-test oracle, function signatures, types, and module scope from the abstract syntax tree play the role of the inventory, the documentation plays the role of the specification, and the guard would withhold function bodies (Sec. 4.1). A software-specific study is future work for us (Secs. 5.6, 6).

Area_Chair_uhzuOpenReview ↗

Follow-up on Author Rebuttal

Dear Reviewer,

A quick reminder as the discussion deadline is approaching: please review the authors' rebuttal and let us know whether your concerns have been addressed or if any major issues remain.

Your feedback would be appreciated before the discussion period closes.

Thank you!

Dear Reviewer,

As the discussion period ends today, we would appreciate your consideration of our rebuttal and additional results. We hope they address your concerns and would welcome any remaining questions. Thank you for your time and consideration.

Reviewer_Qw3NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The author proposes SpecGuard, which addresses the risk that LLM-generated hardware assertions reproduce defects in the RTL they are intended to verify. It restricts the model’s implementation context to structural information, derives behavioral relations from specifications, decomposes generation into focused questions, and applies deterministic transformations for timing, reset handling, and module binding. The paper also analyzes four failure modes and formalizes the limitations of equivalent rewrites and filtering assertions against a potentially faulty design.

Critical strengths
  1. An assertion that agrees with an implementation does not necessarily express the intended behavior, particularly when that implementation contains a defect. The paper makes this distinction explicit and explains why retaining only assertions that pass on the design under test can discard the properties needed to expose its bugs.

  2. SpecGuard combines specification-based reasoning with explicit case enumeration, structural grounding, and deterministic assertion construction.

Critical weaknesses
  1. Passing simulation does not establish the correctness of assertions credited with detecting bugs. The paper defines detection using assertions that hold on the correct RTL and fail on a mutant, but some reported assertions are validated only through simulation. An incorrect assertion can pass a limited reference simulation and fail on a mutant, producing an apparent detection that does not establish a valid specification violation.

  2. Portability and the strength of the information boundary remain insufficiently demonstrated. The supplementary implementation contains extensive repair, grounding, clock-handling, and harness rules. Such engineering is not inherently a weakness, but the evaluation does not clearly establish how much additional configuration or rule development is required for an unfamiliar design.

Questions for authors
  1. Which reported detection rates use simulation-only validation, bounded formal validation, or the supplementary code’s formal-corrected scores? How is Table 23’s UART assertion 12 consistent with normal transmission completion, and how do the results change if it is excluded?

  2. Which model and configuration were actually used for each baseline in Figures 3–4? How should readers reconcile Section 5.1 with Appendix A, and what effect does disabling ChIRAAG’s iterative refinement have on the comparison?

  3. Can you provide the per-design and per-run results supporting Table 13 and reconcile its aggregation with Table 14, the FifoX discrepancy between Figure 4 and Table 15, and the repair counts in Figure 6 versus Table 22?

3 comments

We thank the reviewer for the constructive questions. We have clarified the detection metric, the baseline models and settings, the per-design configuration, and the per-design and per-run results in the revised manuscript.

W1 / Q1. Simulation-only validation; UART assertion 12. We agree and apologize for this oversight. Every reported detection rate now uses the proved rule. An assertion earns credit only if k-induction (yosys-smtbmc with Z3, k ≤ 300, one hour per assertion) proves it on the correct RTL and it fails on the mutant (Sec. 5.2, App. A). For transparency, Tab. 6 also reports the simulation-only and BMC rates per design. On the 10 development designs they are 71.1%, 66.7% and 57.9%, and Tab. 17 gives all three for every design.

UART assertion 12 (now Tab. 26) is false. tx_busy falls when a frame completes, which takes at least 160 cycles, beyond both the original simulation and depth-30 BMC. Under k-induction to depth 300 the base case violates it, so it earns no credit, and the reference simulation now completes full frames. UART detection moves from 58.3% to 55.0%. The rule is conservative, since a true assertion the solver cannot prove also earns nothing (31 of 412 time out, Sec. 5.6).

W2. Portability and the information boundary. We thank the reviewer for recognizing that this engineering is not inherently a weakness, as our contribution is the information constraint, which the rules only carry out. We agree that their cost for an unfamiliar design should be clear, and summarize it below.

  • Configuration. AssertLLM2 needs no per-design configuration, as the specification is the only per-design input. Only FifoX's development configuration set its lane structure by hand, and the general pipeline detects 51.4% without it (Tab. 7).
  • Rules. Each rule reads at most one structural fact, such as drive kind, reset polarity, or module scope, and none reads behavior, a design name, or a simulation result. Ambiguous cases are refused rather than guessed (Tab. 24).
  • Boundary. The context is identical for a correct design and a mutant whenever the mutant preserves structure, which holds for 95.7% of scored mutants (Sec. 4.1).

Q2. Baseline models and settings. Every method in Sec. 5 runs the same local Qwen3-14B with the settings of App. A (Tab. 8), and each baseline keeps its published flow with only the backend swapped. GPT-4o appears only in the Sec. 3.1 case study (Tab. 2), and the 5.1–11.7% frontier figures are quoted from AssertLLM2 (Tab. 1). ChIRAAG detects 21.2% with refinement off and 24.1% with it on, against 57.7% for SpecGuard. The impact of the refinement is low as it aims at functional correctness, which mostly aims for making the assertions compile and pass rather than detect bugs. Refining until assertions pass on the design under test also keeps only those that agree with it, which is self-filtering (Sec. 4.1).

Q3. Per-design and per-run results. Tab. 17 in App. D.2 lists every AssertLLM2 design with three repetitions, giving assertions, soundness, minimal detecting set, single-kill share, and the simulation, BMC, and proved rates. Tab. 16, also in App. D.2, pools these for all six methods, and Tab. 6 in Sec. 5.3 details the 10 development designs. The FifoX and repair-count discrepancies are answered under Reviewer 3, W1 (ii) and (iii).

Area_Chair_uhzuOpenReview ↗

Follow-up on Author Rebuttal

Dear Reviewer,

A quick reminder as the discussion deadline is approaching: please review the authors' rebuttal and let us know whether your concerns have been addressed or if any major issues remain.

Your feedback would be appreciated before the discussion period closes.

Thank you!

Dear Reviewer,

As the discussion period ends today, we would appreciate your consideration of our rebuttal and additional results. We hope they address your concerns and would welcome any remaining questions. Thank you for your time and consideration.

Comments on the paper

1 comment

We thank the reviewers for taking the time and considering our submission to this cycle of ICLR. Taking into account the comments posted by the reviewers, we have made the following modifications in our manuscript and supplementary materials:

Reviewer Qw3N: We now credit detection only to assertions proved by k-induction on the correct RTL (Section 5.2), and Table 6 reports the simulation, bounded model checking, and proved rates for each design. We have revised Section 5.1 and Appendix A (Table 8) so that every method runs the same Qwen3-14B, and we report ChIRAAG both with and without its iterative refinement. Per-design and per-run results for all AssertLLM2 designs are added in Appendix D.2 (Tables 16 and 17), and Section 5.6 discusses the configuration each design requires.

Reviewer 2ai6: We have added the Min. set and Single-kill columns to Table 6 to separate effective from incidental kills, and Table 5 reports the effective and broken shares of every method. Section 4.1 now measures Assumption 1 directly, and Appendix D.11 reports the question and token cost per design. We have further added discussions to Section 5.6 on the specification ceiling, per-design configuration, elaborated structure, and repair accuracy.

Reviewer bTAR: We have run every method on all 83 AssertLLM2 designs and reconciled Tables 5, 6, 16, and 17 (Tables 13 and 14 in the initial submission). We have separated FifoX's development configuration (94.4%) from the general pipeline (51.4%) in Section 3.1 and Tables 7, 18, and 20, and replaced the incorrect repair figure (Figure 6 in the initial submission) with Table 25. We have also added the bug-hunt baseline (Table 5), simplified Section 4, and added a motivating example (Figure 1). We have accordingly updated the abstract, contributions, and conclusions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.