acceptodds
Withdrawn from ICLR 2027

ISAudit: Finding Bugs in RISC-V Cores by Checking Against the ISA Specification

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

14 views

Abstract

Hardware flaws are permanent: a CPU bug cannot be patched once fabricated, so it can only be fixed before tape-out. Whether a core behaves as specified by its instruction set architecture (ISA) is thus settled by pre-silicon verification. Yet existing techniques check proxies for the ISA specification rather than the specification itself, either a reference implementation that can itself be wrong or hand-written properties that cover only a subset of the specification, so a core can pass every check and still violate the specification. While Large Language Models (LLMs) offer automation potential, letting one interpret the specification and judge conformance directly faces two challenges: interpretation fidelity, where a prose clause may be misinterpreted, and verdict trust, where even a correct interpretation yields a verdict without mechanically checkable evidence. To bridge this gap, we introduce ISAudit, a specification-grounded auditing framework that makes the LLM’s interpretation of the specification executable and checkable. It first turns each normative statement into a formal property traceable to its source clause, binds it to the RTL, and derives two heterogeneous representations, each iteratively refined using the other to improve fidelity. Once the two representations reach a fixed point, it checks the settled property with a model checker, backing each violation with a concrete counterexample for triage. Evaluation on five opensource RISC-V cores shows that ISAudit uncovers 33 previously unknown bugs, outperforming baselines by 200–3,200%.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
LLM agents, RISC-V, ISA specifications, hardware verification, processor bugs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fHt7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents ISAudit, an LLM-based framework that checks RISC-V cores against the prose ISA specification. For each normative clause, an RTL-grounded worker and a specification-grounded worker each write a SystemVerilog assertion, and an LLM adjudicator reconciles the two drafts into one property (Refutation-Driven Formalization). The property is checked by bounded model checking, and an LLM reviewer attributes each counterexample to the RTL, the property, or the verification environment (Evidence-Backed Checking). On five open-source cores, ISAudit reports 43 true positives, 33 of them previously unknown, at 95.6% precision, compared with 1 to 20 true positives for Cascade, riscv-formal, Claude Code, and Codex. On unprivileged integer instructions, dual-path formalization raises property precision from 75.5% to 97.1% over a single-path variant.

Critical strengths
  • S1. A well-posed problem and safeguards against properties that pass for the wrong reason. The paper separates two failure points of letting an LLM judge conformance directly: misreading a clause (interpretation fidelity) and issuing a verdict without checkable evidence (verdict trust). The checking stage guards against the false PASS that the paper identifies as the more dangerous outcome: repairs may not remove assertions, narrow applicability, or add assumptions; a property whose cover is unreachable is recorded as vacuous instead of passing; and structural checks flag disabled assertions and constant-true conditions.

  • S2. Traceable findings, some with real impact. Each reported bug is tied to a specification clause, an RTL location, and a counterexample trace, and Appendix B is detailed enough to check every finding individually. Some findings affect software behavior: in Kronos, every csrr read of a counter disturbs it (KRO-RTL-001) and a WFI issued with interrupts globally disabled never wakes (KRO-RTL-002); in VexRiscv, sstatus exposes MPRV (VEX-RTL-018), and because the MMU plugin maps the bit read-write, S-mode can also set it.

Critical weaknesses

W1. The claimed methodological contribution, dual-path formalization, is not new and is not shown to work.

  • Known components. Its ingredients have clear precedents: AssertLLM (Yan et al., 2025) generates SVA from design specifications with multiple LLMs, AssertionForge (Bai et al., 2025) combines specifications and RTL for the same task, and semantic triangulation (Dai et al., 2025) cross-checks LLM outputs. The checking stage is standard formal-verification practice carried out by an LLM.
  • The clause list is not controlled for. Before the dual path runs, a single LLM extractor turns the selected chapters into property records, each fixing a clause's RTL locations, observation event, and applicability conditions, and no bug-finding experiment separates the dual path from these records. The comparison with Claude Code and Codex, the paper's main evidence for bug finding, gives the agents only the RTL and the specification, without the clause list, so ISAudit's margin may come from systematic enumeration alone. Once a clause is known, most findings can be confirmed by reading the source, since a missing CSR is an absent decode entry and a hardwired field is a constant in a write mask; even without a clause list, Claude Code found 8 true positives on Kronos, against 11 for ISAudit. RQ3 (27 vs. 11 true positives with gpt-5.6-sol) fixes the model but not the inputs.
  • The ablation does not measure bug finding. Direct Formalization does share the records, but it is evaluated only on property precision and recall for unprivileged integer instructions, which contain at most one of the 33 new bugs, and its bug count over the full clause set is not reported.

W2. The headline evidence, bug counts relative to the baselines, does not measure the claimed contribution.

  • Many counted bugs are of little or doubtful value. The abstract and contributions rest on "33 previously unknown bugs, 3–33x as many as the baselines". I checked each finding in Appendix B against the specification, the cores' source code, and public issue trackers:
    • Five have clear functional or security impact (KRO-RTL-001, -002, -004, -007, VEX-RTL-018), four of them in Kronos, a 1.6k-line core with no commits since December 2022.
    • Nineteen are genuine but low-impact deviations, such as unmasked mepc[0], missing reset values, FENCE writing its reserved rd field (PIC-RTL-002), and TW/TSR/TVM hardwired to zero; VEX-RTL-010 is the constant misaExtensionsInit = 66 in VexRiscv's example presets.
    • Five are requirements introduced in privileged specification v1.12 (VEX-RTL-005, -015, -016) or v1.13 (VEX-RTL-019, CVA6-RTL-001), and the paper does not state which version each core targets.
    • Three are doubtful. For CVA6-RTL-002, the evaluated commit (6348e9e) decodes all pmpcfg and pmpaddr CSRs without trapping and holds entries beyond NrPMPEntries at zero, which the WARL rule permits, as CVA6 contributors also concluded in openhwfoundation/cva6#3050. PIC-RTL-006 requires disabling the documented CATCH_MISALIGN option, and KRO-RTL-010 matches SonalPinto/kronos#17 from August 2024.
    • The remaining finding, PIC-RTL-005, has unclear trigger conditions.
  • Only counts are compared. Appendix A summarizes Claude Code in one sentence and says nothing about Codex, so readers cannot tell which bugs the agents found, how many overlap with ISAudit's, or how severe each tool's unique findings are. If the agents' true positives (12 for Claude Code, 20 for Codex) include the few impactful bugs above, ISAudit's surplus consists mostly of low-impact or doubtful entries; on Rocket, Codex found two bugs that ISAudit missed. Since the introduction faults agents for lacking reproducible evidence, the relevant comparison is the quality of evidence, which the paper does not make. The 3,200% margins over Cascade and riscv-formal mostly reflect what each tool checks; the paper itself notes that CSR-existence checks lie outside Cascade's reach.

W3. Inaccurate and incomplete reporting.

  • Appendix A states that all roles run at temperature 0, but Claude Opus 5.5 and Claude Sonnet 5 do not accept sampling parameters in the Anthropic API.
  • The effort level of each ISAudit role is not reported, while the Claude Code baseline used high effort.
  • The "selected chapters" that define the clause list are not identified.
  • Table 2 reports only percentages, without the numbers of reference and generated properties or the rule used to match them.
Questions for authors
  1. Baseline inputs and overlap. Did Claude Code and Codex receive the list of normative tags or any other ISAudit artifact, or only the RTL and the specification? Which true positives did each agent find, and how many overlap with ISAudit's 43?
  2. Specification versions. Which privileged-specification version does each core target, and how many of the 33 findings remain violations under that version?
  3. Disputed finding. The WARL rule allows PMP entries to be read-only zero. Which behavior in the CVA6-RTL-002 counterexample violates the specification?
  4. Direct Formalization. Was Direct Formalization run over the full clause set? If so, how many bugs did it find, and how many overlap with ISAudit's?
Reviewer_BEeqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ISAUDIT turns RISC-V spec clauses into SystemVerilog assertions. Two LLM workers write each assertion, one from the RTL and one from the spec, and an adjudicator reconciles them. A model checker then tests each assertion, and an LLM triages any failures. It claims 43 bugs (33 new) across five cores at 95.6% precision.

Critical strengths
  1. The manuscript finds actual microarchitectural bugs, proving the efficacy of the proposed method.
  2. The authors have included sufficient safeguards which collect trace data.
Critical weaknesses
  1. Choice of baselines. The comparison against Cascade, riscv-formal, and general-purpose coding agents is informative. However, since ISAUDIT is itself an LLM-based method, the evaluation would be stronger with comparisons to LLM-based assertion-generation approaches, such as AssertLLM, AssertionForge, or SpecAlign, which the related work section already discusses. These are the closest competitors, and including them would help readers see more clearly where ISAUDIT's gains come from.
  2. Different models for the two workers. According to Appendix A, the RTL-grounded worker uses Claude-Opus-5.5 while the spec-grounded worker uses Claude-Sonnet-5. Since the two models differ in capability, part of the observed benefit may come from model diversity rather than from the separation of evidence that the paper emphasizes. Clarifying this would strengthen the paper's central claim.
  3. Possible influence of the RTL on property generation. Several components have access to the RTL, including the extractor that writes the shared record q, which both workers receive. It would be valuable to understand whether information from the RTL could inadvertently shape the formalized requirement, leading properties to reflect what the implementation does rather than what the specification requires. Addressing this would further increase confidence in the reported findings.
Questions for authors
  1. Could the authors include a comparison with existing LLM-based assertion-generation methods on the same cores?
  2. Could the authors add an ablation that runs both workers on the same model, to separate the effect of model diversity from that of separating the evidence?
  3. Could the authors provide an analysis of potential RTL-induced bias? One option is to inject known specification violations into the RTL and check whether ISAUDIT still flags them or instead formalizes properties that accept the faulty behavior.
Reviewer_uW6COpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multi-agent workflow for detecting bugs in RISC-V cores. Two agents independently generate formal properties from the specification and RTL, which are then checked using open-source formal verification tools. A reviewer agent compares the verification feedback from these two heterogeneous paths to identify bugs. Evaluation results show that the proposed multi-agent flow outperform state-of-the-art coding agent in identifying new bugs in RISC-V cores.

Critical strengths

(1) This work evaluates state-of-the-art coding agents on bug hunting across several sufficiently complex open-source designs. (2) It proposes a multi-agent workflow featuring tool feedback and isolated context memory specifically tailored for bug identification. (3) It demonstrates significant performance improvements over baseline implementations such as Claude Code and Codex.

Critical weaknesses

(1) The core contribution is a multi-agent workflow augmented with tool feedback. However, proposing invariants directly from RTL or specifications and relying on another agent to review tool feedback feels somewhat naive. While evaluations using recent frontier models like Claude Code and Codex are appreciated, the paper lacks a detailed analysis of their specific failure modes. (2) Furthermore, the overall framework is not end-to-end verifiable. Although the identified bugs were manually inspected, they have not been confirmed by the original authors of the respective designs, which diminishes the soundness of the empirical results. Despite integrating formal tools into the loop, there are no formal guarantees that a mismatch between the specification and RTL stems from an agent failure versus an actual design bug. Relying on state-of-the-art closed models to make these judgments without offering further insights or directions for improvement limits the paper's overall impact.

Questions for authors

(1) Can you explain how the flow recovers when the agent misjudges whether a CEX is due to a wrongly proposed property or an actual design bug? (2) Can you explain why Claude Code and Codex perform much worse than the proposed flow? Is it because they are not running for long enough, or is some tool feedback disabled for them? While the paper presents component ablations, more insights would be appreciated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.