acceptodds
Under review as a conference paper at ICLR 2027

HARVEST: Hierarchically Adaptive RTL Verification via Structured Episode Trajectories

Abstract

Automatic testbench generation with large language models can reduce the cost of hardware verification, but a testbench that compiles and passes its own checks may still miss critical bugs. This creates a fundamental challenge for automated verification: the signal used to decide whether a generated testbench is adequate cannot simply be the agent's own report of success. We introduce HARVEST, a framework for adaptive testbench generation that routes designs among heterogeneous verification agents and uses an external adequacy signal}, mutation kill rate, to identify insufficient testbenches and trigger complementary verification. A contextual bandit selects among three agents targeting structural, semantic, and adversarial coverage. When the selected testbench has low external adequacy, HARVEST escalates to a second agent andcomposes the resulting testbenches into a single suite, preserving complementary tests rather than discarding the first attempt. On 100 open-source Verilog designs, we find that 13% of testbenches that pass their own checks nevertheless kill fewer than one-third of injected bugs, revealing a substantial gap between agent-reported success and actual verification adequacy. Routing to a single agent achieves 91.7% of the oracle quality, while adequacy-gated escalation further improves quality at modest additional cost. Upon acceptance we will release 300 structured trajectories spanning the 100 designs as a benchmark for adaptive verification under external adequacy signals. Together, these results show that external adequacy can provide a reliable control signal for detecting false-confidence testbenches and coordinating complementary verification agents.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
RTL Verification, Verilog, VHDL, Testbench Generation, RTL, Bandit, Mutation Kill Rate

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_o4QvOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

HARVEST uses three LLM agents to generate RTL testbenches, with structural, semantic, and boundary-testing prompts. A router first selects an agent from static RTL features. If the resulting testbench has a mutation kill rate (MKR) below 0.5, the system runs a second agent. On 100 designs, the paper reports that this escalation reduces mean regret under its composite reward from 0.122 for one routed run to 0.070.

The underlying problem matters: a testbench can pass its own assertions while missing faults. The paper makes a reasonable case for inspecting the first run before deciding whether to spend more compute. It does not make a comparable case for routing before that run.

Critical strengths
  1. The paper identifies a real weakness of self-passing testbenches: passing assertions alone do not establish fault-detection ability.
  2. Evaluating all three strategies on the same designs reveals substantial differences between strategies on individual designs.
  3. The paper reports several unfavorable comparisons, including the lack of a significant gain over the best fixed strategy and the similar performance of coverage and MKR gating.
Critical weaknesses
  1. The motivation explains why one might inspect a testbench and then run another agent. It does not explain why the first agent must be chosen by a router. Different strategies winning on different designs establishes only an opportunity for selection in hindsight. Lines 162–164 say that the signal about which strategy wins lies in execution outcomes rather than source structure, which further weakens the case for routing from static features. Routing does not significantly outperform always choosing the semantic agent (regret 0.122 versus 0.139, Table 2). The fixed strategy also selects the best arm on 46 designs, compared with 43 for the router (Table 11). Routing is central to the proposed method, but its necessity is not established.

  2. Section 3.5 is difficult to follow. It introduces LinUCB updates and selection, then says that outcomes for all strategies were collected in advance, the exploration parameter has no effect, the method reduces to ridge regression, and no online regret guarantee is claimed (lines 203–209). The section explains what the “bandit” does not do in this experiment, without explaining why LinUCB was chosen or what the “Bandit Router” contributes.

  3. The main gain appears to come from making a second attempt, rather than from MKR or testbench composition specifically. Coverage gating has slightly lower regret and reported cost than MKR gating (0.068 at about 1,115 seconds versus 0.070 at about 1,160 seconds, Table 2). A more direct comparison would start with a fixed agent, then let an LLM use the first run's reward, coverage gaps, and surviving mutants to decide how to improve the testbench. The paper's LLM-designated baseline sees only static features before the first run; the complementarity predictor in Appendix F does not use this first-run feedback. The headline regret of 0.070 comes from keeping the better of two runs. The reported composition result of about 0.069 is mainly estimated from the per-metric maxima of the two runs, rather than from executing a merged suite. Only 6 of 35 replayed trigger designs yielded valid per-mutant unions, and none showed kills beyond the stronger arm.

  4. Several central numbers do not agree. The abstract says that routing to one agent reaches 91.7% of oracle quality. Table 2 gives 0.719 for one routed run and 0.841 for the oracle, or 85.5%; 91.7% corresponds to the escalated reward of 0.771. Table 1 reports 237 passing episodes among 294 scored episodes but gives a pass rate of 82.7%; the quotient is 80.6%. Section 4 describes the primary designs as having 29–484 RTL lines, while Table 9 places 35 designs above 500 lines. The size-bin means in Table 9 yield a weighted mean reward of about 0.821, rather than the reported “All” value of 0.858.

  5. Line 161 abruptly says that the final features replaced “degenerate v1 features.” Tables 3, 11, and 13 and Appendix G continue to use internal labels such as “enriched v2” and “dead feat.” These unexplained version references read like unedited drafting artifacts, possibly from AI-assisted writing.

  6. The flowchart is rough, and its content does not match the principal evaluation. Figure 1 and the abstract describe merging TB1 and TB2 into the final suite. Table 2's main result of 0.070 instead keeps the higher-scoring single run. Appendix F mainly estimates the “merged” result from the maximum of each metric across the two runs; it does not report execution of the merged suite on the full corpus. The system shown in the figure is therefore not the system behind the headline result.

Questions for authors

in weaknesses

Reviewer_oLxWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HARVEST, a hierarchically adaptive framework for automated hardware verification. The key ideas include (1) deploying a portfolio of heterogeneous verification agents specialized in structural, semantic, and adversarial testing, (2) routing designs to the most promising agent strategy using an execution-grounded contextual bandit, and (3) leveraging mutation kill rate as an external adequacy signal to detect insufficient testbenches, trigger secondary escalation, and compose complementary tests into a unified suite. Evaluation on an open-source Verilog design corpus across simulated mutation fault injection and multi-agent execution trajectories shows promising bug-detection adequacy, high structural code coverage, and cost-effective test-time compute efficiency.

Critical strengths

LLM-generated testbeches routinely compile and pass their own self-authored assertions while silently missing critical hardware bugs. HARVEST replaces untrustwrothy agent self-reports with Mutation Kill Rate as an active runtime control signal, establishing an objective, execution-aware criterion to detect inadequate verification and dynamically steer test-time compute.

Unlike traditional cascading frameworks that discard rejected attempts, HARVEST recognizes that verification coverage is inherently cumulative. By decomposing verification into orthogonal agent personas (structural, semantic, and adversarial) and merging their outputs upon escalation, the system combines complemetary stimulus patterns.

Critical weaknesses

However, I also have a few concerns.

First, the core gating mechanism relies on mutation kill rate as an online runtime signal to trigger agent escalation. In practical hardware verification, mutation testing are computationally intensive, requiring the compilation and cycle-accurate execution of numerous mutated hardware variants for every design under test. The paper treats mutation analysis as a lightweight oracle, but in production electronic design automation environments, the wall-clock and server simulation costs of running a mutant farm at test time routinly dwarf the inference cost of the language models themselves. If extracting the adequacy signal consumes more simulation time than simply executing all verification strategies concurrently, the practical justification for adaptive compute allocation breaks down.

Second, the evaluation is limited to relatively small, standalone open-source hardware modules such as basic arithmetic units and isolated encryption blocks. However, to allow the proposed framework usable in practical design process, it needs to support heirarchical systems-on-chip featuring asynchronous clock domains, multi-tiered interconnects, and industrial universal verification methodology environments.

Third, the verification schemes represent static prompt variations rather than specialized, fine-tuned agent policies. The only learned component in the architecture is a shallow linear contextual bandit operating over hand-crafted sturctural design features. As such, the system lacks closed-loop policy learning across verification episodes. When a generated testbench exhibits poor bug-hunting efficacy, the surviving mutants and failed execution traces are not used to update the generator or explain why verification gaps occurred. Treating the generators as frozen black boxes reduces the framework to an external prompt-selection heuristic rather than a genuinely adaptive agentic system.

Finally, the escalation decision depend on a globally fixed mutation kill threshold to separate adequate from inadequate testbenches. In digital circuit design, achievable mutation kill rates fluctwaite substantially depending on circuit topology, pipeline depth, and the natural prevalence of functionally equivalent mutants. In lots of nested control logic, a modest kill rate may represent the theoretical coverage ceiling, triggering wasteful and redundant escalations. As such, in straightforward datapaths, a potentially high kill rate can still leave critical corner cases unexercised while falsely signaling adequate verification. Without an adaptive or statistically grounded calibration mechanism, static thresholding remains brittle across heterogeneous hardware domains.

Questions for authors
  1. What is the total wall-clock simulation overhead of online mutation testing compaed to LLM inference time?
  2. Is gating actually cheaper than running all three agents in parallel?
  3. How sensitive is the escalation decision to the fixed mutation threshold across different circuit families?
  4. Were equivalent mutants filtered to prevent false escalations?
Reviewer_QjdKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses test-time compute allocation for LLM-based RTL testbench generation using three verification agents (structural, semantic, and adversarial). HARVEST uses an offline-trained LinUCB contextual bandit to select an initial strategy; if the generated testbench achieves a mutation kill rate (MKR) below 0.5, a meta-controller triggers a second strategy and merges the two testbenches. Evaluated on 100 open-source Verilog designs and four RealBench IPs, the results show that while single-run routing performs similarly to the best fixed strategy, running a second agent on low-MKR designs improves the composite reward at roughly 1.33× the compute cost.

Critical strengths
  1. The paper tackles a practical issue in automated hardware verification: a generated testbench that compiles and passes its own assertions can still be functionally weak. Using mutation testing as an external check to allocate extra compute is intuitive, and separating first-pass routing from second-pass escalation is a clean problem formulation.
  2. The evaluation includes useful baselines (fixed-best strategy, line-coverage and pass/fail gates, random escalation, and always-top-2) and checks a coverage-only reward to rule out circularity. I also appreciate the authors' candor in reporting negative results in the appendices, such as feature degeneracy and the lack of union MKR gains in the per-mutant re-evaluation.
Critical weaknesses
  1. The results do not show why MKR gating is needed over simple line-coverage gating. In Table 2, coverage gating (lc < 70%) achieves slightly lower regret than MKR gating (0.068 vs. 0.070) at slightly lower cost (about 1115s vs. 1160s). Section 5.3 admits the two gates are statistically indistinguishable, yet still calls MKR's value "distinct, irreplaceable." Meanwhile, the 13% "false-confidence" subset cited throughout the paper (passing simulation with MKR < 0.3) has a mean line coverage of only 56.5%, so a coverage gate would catch most of it anyway. Since single-run bandit routing also fails to significantly beat the fixed-best semantic strategy (p = 0.57), Table 2 needs baselines that escalate from a fixed first strategy or simply re-run the same strategy twice (best-of-2) to isolate what the heterogeneous hierarchy actually adds.
  2. The claim regarding testbench composition relies on a scalar proxy rather than actual merged execution. Table 2's main result (0.070 regret) simply keeps the better of the two runs (keep-max), while the composition gain in Appendix F (+0.0008 reward) is just the per-dimension maximum of scalar scores. When the authors actually recover per-mutant kill sets in Appendix F, 29 of 35 triggered designs are excluded due to pipeline errors or failing unmutated baselines. On the 6 measurable designs, the kill sets are completely nested (zero union gain, 95% CI includes 0), which does not support presenting suite composition as an established practical benefit.
  3. Several numerical inconsistencies across the paper need explanation. The abstract claims single-agent routing reaches 91.7% of oracle quality, whereas Table 2 gives 85.5% (0.719 / 0.841) for single-run routing and 91.7% (0.771 / 0.841) for MKR-gated escalation. Furthermore, Section 4 states the corpus spans 29 to 484 RTL lines, yet Table 9 includes 35 designs over 500 lines and reports a mean bandit reward of 0.858 across 97 designs versus 0.719 across 100 designs in Table 2. Since rewards are non-negative, dropping 3 designs cannot explain that jump; moreover, the sample-weighted mean of Table 9's four size buckets is roughly 0.821, not 0.858.
Questions for authors
  1. Under a matched budget, how does HARVEST compare against (a) starting with the fixed-best strategy (semantic) and applying the same escalation rule, and (b) running the first strategy a second time? Also, how often does a passing testbench satisfy the coverage gate (line coverage >= 70%) but fail the MKR gate (MKR < 0.5), and what regret reduction comes from those specific cases?
  2. Could you clarify the numerical discrepancies across the abstract, Table 2, Table 9, and Appendix F, specifically why Table 9 has 35 designs above 500 lines and a mean reward of 0.858 (while its bucket-weighted average is 0.821), and how failing unmutated baselines are handled in the main tables?
Reviewer_MQUCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies LLM agents that write testbenches for 100 open-source Verilog RTL designs. HARVEST uses a LinUCB contextual bandit (the Bandit Router) on six static RTL features to pick one of three agents: structural, semantic, or adversarial. A Meta Controller runs the second-ranked agent if the first testbench has a mutation kill rate (MKR) below 0.5, and it merges both testbenches. With one Claude Sonnet 4.6 run per design and agent, the gate cuts regret against a per-design oracle from 0.122 to 0.070.

Critical strengths
  • Table 2 compares the MKR gate with coverage-gated, pass/fail-gated, and random escalation at similar cost. The random baseline (regret 0.116 against 0.070) shows that the choice of escalated designs matters. The paper also reports openly that learned routing only ties the fixed-best agent.
  • A concrete measure of false confidence. Of 237 testbenches that pass their own checks, 30 have MKR < 0.3 and only 56.5% mean line coverage. Escalation on runs with MKR < 0.5 costs about 1.33× single-run compute and cuts regret significantly. If the numbers hold, the recipe is cheap and practical.
Critical weaknesses
  • The MKR gate (regret 0.070) does not beat a cheaper coverage gate; Sec. 5.3 calls them statistically indistinguishable. The merge score is a per-dimension maximum, so its gain over the better run cannot be negative. With the true union kill rate, its CI includes 0 . With one run per design–agent pair and no same-agent re-run baseline, I cannot separate agent diversity from LLM sampling noise.
  • Unclear metric source and inconsistent numbers. The prompt tells each agent to print its own pass, coverage, and mutation_kill values; App. C also uses an "agent-reported" MKR. If the corpus uses these printed values, MKR is not external to the agent, which contradicts the premise in the abstract. Several numbers also conflict: the abstract gives 91.7% of oracle quality to single-agent routing, but this is the MKR-gated value. Table 9 reports 35 designs above 500 lines and a bandit mean reward of 0.858, but Table 2 gives 0.719.
Questions for authors
  • Do the corpus pass, coverage, and MKR values come from the agent's HARVEST_RESULT line or an independent harness? Do all three agents use the same mutant set?
  • Which corpus version (n = 100 or n = 104, App. F) produced Table 9 and Figure 2a? Figure 2a's tier bars imply at least 68% mean line coverage over 294 episodes, but Table 1 gives 63.9%.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.