acceptodds
Under review as a conference paper at ICLR 2027

RISCForge-Bench: Benchmarking the Gap Between Local RTL and Processor-Extension Delivery

Abstract

Module-level RTL benchmarks ask whether an AI system can generate functionally correct hardware. A deployable processor extension must instead preserve one software-visible operation across software, an ISA contract, executable semantics, temporal interfaces, a processor core, workloads, and implementation constraints. We introduce RISCForge-Bench, a family-based executable benchmark spanning 12 independently admitted RISC-V extension families across four workload domains. Each family is one statistical breadth unit; together, they define 48 registered variation axes and 8,403 enumerated sealed-test entries. Across matched model–agent configurations, systems pass 95/108 isolated compute-RTL cells. When systems also author processor-facing RTL, 6/108 cells pass integrated RTL functionality, but none passes protocol stress. End-to-end delivery under full cross-artifact authorship remains 0/108. Same-artifact replay finds 177 locally correct RTL modules and no corresponding delivery. Supplying the admitted implementation restores 32/32 provider-completed deliveries, showing that the evaluator can accept correct artifacts. Responsibility controls place the earliest tested break at processor-facing RTL without claiming to isolate protocol reasoning. Targeted repair recovers nonzero delivery from a near-complete implementation. These results expose a transfer gap between local artifact competence and cross-artifact executable consistency, and establish a versioned target for evaluating long-horizon hardware agents under bounded engineering budgets and auditable scoring.

Primary area:
datasets and benchmarks
Keywords:
RISC-V, processor extensions, hardware agents, RTL generation, hardware–software co-design, executable benchmarks, long-horizon agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QVkeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

RISCForge-Bench evaluates whether model-agent systems can carry a software-visible processor extension from design intent to a deliverable implementation on CV32E40X / CV-X-IF. It contains 12 frozen task families (AI, DSP, cryptography, stream/memory) with native workloads, independent oracles, mutation-tested judges, and 12 ordered strict-delivery gates. The main finding: frontier systems solve isolated compute RTL but rarely survive processor-facing authorship—few A0 cells pass integrated RTL functionality, none pass protocol stress, B0 delivery is zero, and local RTL passes do not transfer to delivery.

Critical strengths

A reasonable matched-scope design. The A-1 / A0 / B0 contrast (Table 2) fixes family intent, sealed distribution, core, and evaluator while varying only the agent-authored artifacts, giving a sensible construct for separating local compute RTL competence from cross-artifact delivery.

Critical weaknesses
  1. The motivation is not clear, and the gap the benchmark closes is never established. §2 asserts that RISCForge-Bench "asks the complementary upstream question," but never shows what systematic misjudgment existing benchmarks (VerilogEval, CVDP, HSCO-Bench, FluxBench) would produce without it, nor why the scope in Table 1 must be covered jointly. The choice of 12 families and the exclusion of compiler backends, OS support, and coherence (§3.4, §7) are likewise asserted rather than justified, so the reader cannot tell what gap this benchmark uniquely fills.
  2. The local-RTL-to-delivery gap is localized but not explained. §5.2–5.4 show where the break occurs and rule out time and workflow support, but the paper does not isolate the causal mechanism behind the gap or argue why the observed breakpoint should be attributed to the capability the benchmark claims to measure. The headline conclusion therefore reports where failure occurs rather than why, and the absence of a justified account of the gate ordering (§3.2) leaves it unclear whether the breakpoint reflects capability or benchmark construction.
Questions for authors

N.A

Reviewer_1ZZoOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RISCForge-Bench, demonstrating that AI system can generate individually correct RTL modules but struggle to deliver a complete, working RISC-V processor extension. The benchmark covers 12 task families across four domains: AI, DSP, cryptography, and stream/memory operations. It evaluates a chain of artifacts, including ISA specifications, software mappings, executable semantics, compute RTL, processor-facing RTL, processor execution, and synthesis feasibility.

Critical strengths
  1. Compared to prior benchmarks, such as HSCO-Bench, the proposed RISCFORGE-BENCH provides a comprehensive evaluation framework that extends beyond isolated RTL generation to cover ISA specifications, executable semantics, software integration, processor-facing RTL, core execution, and implementation constraints. This broad scope enables a more realistic assessment of AI agents' ability to maintain cross-artifact consistency throughout processor-extension development.

  2. The paper has extensive benchmark validity checks and well-designed responsibility controls including local compute RTL, processor-facing RTL and cross-artifact delivery.

Critical weaknesses
  1. The benchmark's primary metric is FullDeliveryRate, but all tested model–agent configurations achieve zero end-to-end delivery, which means the primary metric cannot distinguish the capabilities of the evaluated agents. A useful benchmark should ideally distinguish systems at different capability levels and remain sensitive to incremental improvements.
Questions for authors

The paper identifies a substantial gap between local RTL correctness and end-to-end delivery. However, many failures occur at submission validity or processor-facing RTL. Can the authors further distinguish failures caused by cross-artifact semantic inconsistency from those caused by incomplete workflows, interface implementation errors, or insufficient tool use?

Reviewer_xQywOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The proposed RISCForge-Bench evaluates whether language-model agents can deliver a processor extension whose ISA contract, software mapping, executable semantics, RTL, processor interface, and implementation constraints remain consistent. The benchmark comprises 12 task families across four application domains on the CV32E40X/CV-X-IF platform. Its main contribution is an executable delivery criterion supported by experiments that vary the artifacts agents must author.

Critical strengths
  1. The benchmark addresses a meaningful gap in hardware-agent evaluation. It requires the same software-visible operation to remain consistent across instruction encoding, software, computation, processor interaction, and implementation constraints. This is an important engineering capability that module-level correctness alone cannot establish.
Critical weaknesses
  1. The supplied artifact does not support the claimed level of reproducibility. Section 3.4, the Reproducibility Statement, and Appendix H state that third parties can evaluate all delivery gates locally without curator scoring, with evaluation inputs and reference implementations available to the reproducer. However, the supplementary package’s top-level README explicitly excludes independent oracles and hidden inputs. It also states that public validation only checks task structure, required files, editable paths, and contract schemas; it does not compile or simulate candidate RTL. Official functional, protocol, core, workload, and synthesis scoring remains curator-run.

  2. The experiments establish a bounded delivery gap more clearly than they explain its causes or demonstrate its generality. The paper appropriately acknowledges that G6 and G7 identify rejection boundaries rather than specific reasoning failures. Nevertheless, the available evidence leaves several materially different explanations unresolved. G6 failures can involve compilation, wrapper construction, or functional behavior, while the public records do not expose the concrete violations behind the six G7 failures. Representative counterexamples and targeted repairs would provide a more useful diagnosis than gate labels alone.

Questions for authors
  1. Can the authors reconcile Section 3.4 and Appendix H with the supplementary README? Is there an accessible package containing the executable functional, protocol, and core evaluators, together with the required reference implementations and evaluation inputs? Which gates can a third party currently run on a new submission without curator assistance?
  2. What specific violations caused the six A0 candidates that passed G6 to fail G7? Could the authors provide representative counterexamples or traces, together with a breakdown of the main G6 failure types?
  3. Can the authors provide a controlled comparison in which agents complete substantially more model/tool interactions and validation cycles, rather than merely receiving a larger wall-clock allowance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.