- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
RISCForge-Bench evaluates whether model-agent systems can carry a software-visible processor extension from design intent to a deliverable implementation on CV32E40X / CV-X-IF. It contains 12 frozen task families (AI, DSP, cryptography, stream/memory) with native workloads, independent oracles, mutation-tested judges, and 12 ordered strict-delivery gates. The main finding: frontier systems solve isolated compute RTL but rarely survive processor-facing authorship—few A0 cells pass integrated RTL functionality, none pass protocol stress, B0 delivery is zero, and local RTL passes do not transfer to delivery.
Critical strengths
A reasonable matched-scope design. The A-1 / A0 / B0 contrast (Table 2) fixes family intent, sealed distribution, core, and evaluator while varying only the agent-authored artifacts, giving a sensible construct for separating local compute RTL competence from cross-artifact delivery.
Critical weaknesses
- The motivation is not clear, and the gap the benchmark closes is never established. §2 asserts that RISCForge-Bench "asks the complementary upstream question," but never shows what systematic misjudgment existing benchmarks (VerilogEval, CVDP, HSCO-Bench, FluxBench) would produce without it, nor why the scope in Table 1 must be covered jointly. The choice of 12 families and the exclusion of compiler backends, OS support, and coherence (§3.4, §7) are likewise asserted rather than justified, so the reader cannot tell what gap this benchmark uniquely fills.
- The local-RTL-to-delivery gap is localized but not explained. §5.2–5.4 show where the break occurs and rule out time and workflow support, but the paper does not isolate the causal mechanism behind the gap or argue why the observed breakpoint should be attributed to the capability the benchmark claims to measure. The headline conclusion therefore reports where failure occurs rather than why, and the absence of a justified account of the gate ordering (§3.2) leaves it unclear whether the breakpoint reflects capability or benchmark construction.
Questions for authors
N.A