- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents ISAudit, an LLM-based framework that checks RISC-V cores against the prose ISA specification. For each normative clause, an RTL-grounded worker and a specification-grounded worker each write a SystemVerilog assertion, and an LLM adjudicator reconciles the two drafts into one property (Refutation-Driven Formalization). The property is checked by bounded model checking, and an LLM reviewer attributes each counterexample to the RTL, the property, or the verification environment (Evidence-Backed Checking). On five open-source cores, ISAudit reports 43 true positives, 33 of them previously unknown, at 95.6% precision, compared with 1 to 20 true positives for Cascade, riscv-formal, Claude Code, and Codex. On unprivileged integer instructions, dual-path formalization raises property precision from 75.5% to 97.1% over a single-path variant.
Critical strengths
-
S1. A well-posed problem and safeguards against properties that pass for the wrong reason. The paper separates two failure points of letting an LLM judge conformance directly: misreading a clause (interpretation fidelity) and issuing a verdict without checkable evidence (verdict trust). The checking stage guards against the false PASS that the paper identifies as the more dangerous outcome: repairs may not remove assertions, narrow applicability, or add assumptions; a property whose cover is unreachable is recorded as vacuous instead of passing; and structural checks flag disabled assertions and constant-true conditions.
-
S2. Traceable findings, some with real impact. Each reported bug is tied to a specification clause, an RTL location, and a counterexample trace, and Appendix B is detailed enough to check every finding individually. Some findings affect software behavior: in Kronos, every csrr read of a counter disturbs it (KRO-RTL-001) and a WFI issued with interrupts globally disabled never wakes (KRO-RTL-002); in VexRiscv, sstatus exposes MPRV (VEX-RTL-018), and because the MMU plugin maps the bit read-write, S-mode can also set it.
Critical weaknesses
W1. The claimed methodological contribution, dual-path formalization, is not new and is not shown to work.
- Known components. Its ingredients have clear precedents: AssertLLM (Yan et al., 2025) generates SVA from design specifications with multiple LLMs, AssertionForge (Bai et al., 2025) combines specifications and RTL for the same task, and semantic triangulation (Dai et al., 2025) cross-checks LLM outputs. The checking stage is standard formal-verification practice carried out by an LLM.
- The clause list is not controlled for. Before the dual path runs, a single LLM extractor turns the selected chapters into property records, each fixing a clause's RTL locations, observation event, and applicability conditions, and no bug-finding experiment separates the dual path from these records. The comparison with Claude Code and Codex, the paper's main evidence for bug finding, gives the agents only the RTL and the specification, without the clause list, so ISAudit's margin may come from systematic enumeration alone. Once a clause is known, most findings can be confirmed by reading the source, since a missing CSR is an absent decode entry and a hardwired field is a constant in a write mask; even without a clause list, Claude Code found 8 true positives on Kronos, against 11 for ISAudit. RQ3 (27 vs. 11 true positives with gpt-5.6-sol) fixes the model but not the inputs.
- The ablation does not measure bug finding. Direct Formalization does share the records, but it is evaluated only on property precision and recall for unprivileged integer instructions, which contain at most one of the 33 new bugs, and its bug count over the full clause set is not reported.
W2. The headline evidence, bug counts relative to the baselines, does not measure the claimed contribution.
- Many counted bugs are of little or doubtful value. The abstract and contributions rest on "33 previously unknown bugs, 3–33x as many as the baselines". I checked each finding in Appendix B against the specification, the cores' source code, and public issue trackers:
- Five have clear functional or security impact (KRO-RTL-001, -002, -004, -007, VEX-RTL-018), four of them in Kronos, a 1.6k-line core with no commits since December 2022.
- Nineteen are genuine but low-impact deviations, such as unmasked mepc[0], missing reset values, FENCE writing its reserved rd field (PIC-RTL-002), and TW/TSR/TVM hardwired to zero; VEX-RTL-010 is the constant misaExtensionsInit = 66 in VexRiscv's example presets.
- Five are requirements introduced in privileged specification v1.12 (VEX-RTL-005, -015, -016) or v1.13 (VEX-RTL-019, CVA6-RTL-001), and the paper does not state which version each core targets.
- Three are doubtful. For CVA6-RTL-002, the evaluated commit (6348e9e) decodes all pmpcfg and pmpaddr CSRs without trapping and holds entries beyond NrPMPEntries at zero, which the WARL rule permits, as CVA6 contributors also concluded in openhwfoundation/cva6#3050. PIC-RTL-006 requires disabling the documented CATCH_MISALIGN option, and KRO-RTL-010 matches SonalPinto/kronos#17 from August 2024.
- The remaining finding, PIC-RTL-005, has unclear trigger conditions.
- Only counts are compared. Appendix A summarizes Claude Code in one sentence and says nothing about Codex, so readers cannot tell which bugs the agents found, how many overlap with ISAudit's, or how severe each tool's unique findings are. If the agents' true positives (12 for Claude Code, 20 for Codex) include the few impactful bugs above, ISAudit's surplus consists mostly of low-impact or doubtful entries; on Rocket, Codex found two bugs that ISAudit missed. Since the introduction faults agents for lacking reproducible evidence, the relevant comparison is the quality of evidence, which the paper does not make. The 3,200% margins over Cascade and riscv-formal mostly reflect what each tool checks; the paper itself notes that CSR-existence checks lie outside Cascade's reach.
W3. Inaccurate and incomplete reporting.
- Appendix A states that all roles run at temperature 0, but Claude Opus 5.5 and Claude Sonnet 5 do not accept sampling parameters in the Anthropic API.
- The effort level of each ISAudit role is not reported, while the Claude Code baseline used high effort.
- The "selected chapters" that define the clause list are not identified.
- Table 2 reports only percentages, without the numbers of reference and generated properties or the rule used to match them.
Questions for authors
- Baseline inputs and overlap. Did Claude Code and Codex receive the list of normative tags or any other ISAudit artifact, or only the RTL and the specification? Which true positives did each agent find, and how many overlap with ISAudit's 43?
- Specification versions. Which privileged-specification version does each core target, and how many of the 33 findings remain violations under that version?
- Disputed finding. The WARL rule allows PMP entries to be read-only zero. Which behavior in the CVA6-RTL-002 counterexample violates the specification?
- Direct Formalization. Was Direct Formalization run over the full clause set? If so, how many bugs did it find, and how many overlap with ISAudit's?