- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper investigates why LLM-generated hardware assertions often fail to detect RTL bugs. The authors identify four failure modes in existing assertion generation methods and propose SpecGuard, which provides LLMs with structural information extracted from RTL while withholding implementation behavior. The framework also decomposes specifications into fine-grained questions and uses deterministic rules to repair assertion timing and module binding. Experiments on AssertLLM2 and hierarchical hardware designs report substantial improvements in bug detection over existing methods.
Critical strengths
- The authors identify an important problem: LLM-generated assertions may reproduce implementation defects rather than detect them. The proposed separation between specification and implementation behavior is an interesting approach to improving the reliability of assertion generation.
- The FifoX case study shows that decomposing a general property into specific cases can substantially improve bug detection. This provides useful insights into assertion generation.
- SpecGuard combines structural information extraction, question decomposition, and deterministic assertion repair. The reported improvement in bug detection is substantial.
Critical weaknesses
- My major concern is that there are several inconsistencies in the experimental results and settings, which led me to give a low soundness score:
- The paper claims evaluation on all 83 designs of AssertLLM2. However, Table 14 reports only 10 designs, and the aggregate detection rate (57.9%) also matches Table 13. It is unclear whether the results cover the full benchmark.
- The paper claims that Table 15 reports the numbers behind Figure 4, but Figure 4 reports a detection rate of 51% for FifoX, whereas Table 15 reports 94.4%.
- Appendix D.3 states that "6 bugs become newly detected", but Figure 6 reports much larger numbers of newly detected bugs.
- Section 5.1 states that the baselines use GPT-4o, whereas Appendix A states that all methods use Qwen3-14B.
- A much simpler prompt engineering approach might already address the main problem. For example, we could provide both the specification and RTL to the LLM and explicitly prompt it to follow the specification and treat the RTL as potentially buggy. However, the paper does not compare against this simple baseline, so it is unclear whether the proposed much more complicated approach is really necessary.
- It is possible that the problem studied in this paper is mainly due to the limited capabilities of the tested LLMs, since the main experiments use Qwen3-14B and GPT-4o, which are already relatively old models. More recent and capable LLMs may be better at following the specification without being misled by incorrect RTL implementations. Therefore, it is unclear whether SpecGuard's advantage would persist as LLM capabilities improve.
- The theoretical formulation is unnecessarily complicated. Lemma 1 and Propositions 1 and 2 are fairly intuitive observations that follow directly from the definitions (Their one- or two-sentence proofs are almost trivial), and I do not think they require such extensive formalization. The optimization formulation in Equation (2) also seems to mainly restate the proposed design principle without providing additional theoretical insights. Meanwhile, Lemma 2 makes a more substantive claim, but its brief proof does not adequately justify it. Overall, I think the theoretical presentation could be substantially simplified without losing the main ideas.
- For presentation, the paper could benefit from more concrete examples to explain its main ideas. For instance, the Introduction uses an off-by-one error in a loop bound to illustrate how LLM-generated assertions may reproduce bugs in the implementation, but the explanation is entirely verbal, without any code examples. A simple example showing the specification, buggy implementation, and resulting incorrect assertion would make the problem much easier to understand.
Questions for authors
- Could the authors explain the inconsistencies in the experimental results mentioned in Weakness 1?
- How does SpecGuard compare with a simple prompting approach that provides both the specification and RTL but explicitly prompts the LLM to follow the specification and treat the RTL as potentially buggy?
- Have the authors evaluated SpecGuard using more recent and capable LLMs? Does its advantage over existing methods persist with stronger models?
4 comments
Thank you for the detailed response and the revised manuscript. Now most of the inconsistencies have been resolved, so I have increased my soundness assessment. However, I still notice that some inconsistencies remain in the revised manuscript. For example, Figure 5 is identical to Figure 3 in the original submission, so it appears to show the old results for the 10 designs rather than the updated results for 68 designs. Could the authors check whether this figure needs to be updated?
Also, since the revised manuscript contains substantial changes, it would be helpful to highlight the revisions in a different color so that reviewers can easily identify what has changed.
We sincerely thank the reviewer for their comments and apologize for the oversight. We have revised the figure accordingly, and all changes made in response to the review are highlighted in blue in the revised manuscript.
Dear Reviewer,
As the discussion period ends today, we would appreciate your consideration of our rebuttal and additional results. We hope they address your concerns and would welcome any remaining questions. Thank you for your time and consideration.
We thank the reviewer for the detailed review. We have reconciled the reported results, added the suggested prompting baseline, simplified the theoretical presentation, and added a concrete example in the revised manuscript.
W1 / Q1. Inconsistencies. We thank the reviewer for identifying these oversights in the submitted text, which the revision corrects.
W2 / Q2. A "follow the spec, the RTL may be buggy" prompt. We added this baseline in its strongest form. For each mutant the model receives the specification and that mutant's buggy RTL, with AssertLLM2's own bug-hunting prompt, and writes a separate suite (App. A, Tab. 8). It detects 18.3% against 57.7% for SpecGuard on the same 68 designs (Tab. 5). On FifoX, adding the RTL source to SpecGuard's own prompt lowers detection from 94.4% to 20.8% (Tab. 20, development configuration). Told to distrust the RTL, the model still restates it, and equivalence invariance means no later rewrite can undo that (Secs. 4.1, 5.3).
W3 / Q3. Stronger models. We have not run SpecGuard with a frontier model, as stated in Sec. 5.6. Three findings suggest that capability alone does not close the gap. AssertLLM2's six frontier models, including GPT-5.2, Claude-Sonnet-4.5, and Gemini-2.5-Pro, detect at most 11.7% (Tab. 1). Moving prior generators from Qwen3-14B to GPT-4o improves every surface measure, yet five of six suites still detect none of the 48 FifoX bugs (Tab. 2). With the model fixed, detection rises from 5 to 17 to 41 of 48 as only the questions get finer (Tab. 3). Equivalence invariance and self-filtering follow from the definition of detection, so they hold for any model shown the buggy RTL.
W4. Theory. We agree and have simplified Sec. 4. The optimization program and the Lemma and Proposition apparatus are removed. Sec. 4.1 states the constraint in prose and presents equivalence invariance and self-filtering as direct consequences of the definition of detection. Structural determinacy, formerly Lemma 2, is now argued from register and wire semantics, and retiming supports it empirically, raising detection from 10 to 14 of 20 bugs on versatile_counter with no model call (App. D.4).
W5. Concrete example. We agree and have added Fig. 1. It shows a specification line, the buggy RTL line of a FifoX bug, the assertion copied from that line, and the assertion written from the specification, with each assertion's verdict on correct and buggy RTL.