- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
HARVEST uses three LLM agents to generate RTL testbenches, with structural, semantic, and boundary-testing prompts. A router first selects an agent from static RTL features. If the resulting testbench has a mutation kill rate (MKR) below 0.5, the system runs a second agent. On 100 designs, the paper reports that this escalation reduces mean regret under its composite reward from 0.122 for one routed run to 0.070.
The underlying problem matters: a testbench can pass its own assertions while missing faults. The paper makes a reasonable case for inspecting the first run before deciding whether to spend more compute. It does not make a comparable case for routing before that run.
Critical strengths
- The paper identifies a real weakness of self-passing testbenches: passing assertions alone do not establish fault-detection ability.
- Evaluating all three strategies on the same designs reveals substantial differences between strategies on individual designs.
- The paper reports several unfavorable comparisons, including the lack of a significant gain over the best fixed strategy and the similar performance of coverage and MKR gating.
Critical weaknesses
-
The motivation explains why one might inspect a testbench and then run another agent. It does not explain why the first agent must be chosen by a router. Different strategies winning on different designs establishes only an opportunity for selection in hindsight. Lines 162–164 say that the signal about which strategy wins lies in execution outcomes rather than source structure, which further weakens the case for routing from static features. Routing does not significantly outperform always choosing the semantic agent (regret 0.122 versus 0.139, Table 2). The fixed strategy also selects the best arm on 46 designs, compared with 43 for the router (Table 11). Routing is central to the proposed method, but its necessity is not established.
-
Section 3.5 is difficult to follow. It introduces LinUCB updates and selection, then says that outcomes for all strategies were collected in advance, the exploration parameter has no effect, the method reduces to ridge regression, and no online regret guarantee is claimed (lines 203–209). The section explains what the “bandit” does not do in this experiment, without explaining why LinUCB was chosen or what the “Bandit Router” contributes.
-
The main gain appears to come from making a second attempt, rather than from MKR or testbench composition specifically. Coverage gating has slightly lower regret and reported cost than MKR gating (0.068 at about 1,115 seconds versus 0.070 at about 1,160 seconds, Table 2). A more direct comparison would start with a fixed agent, then let an LLM use the first run's reward, coverage gaps, and surviving mutants to decide how to improve the testbench. The paper's LLM-designated baseline sees only static features before the first run; the complementarity predictor in Appendix F does not use this first-run feedback. The headline regret of 0.070 comes from keeping the better of two runs. The reported composition result of about 0.069 is mainly estimated from the per-metric maxima of the two runs, rather than from executing a merged suite. Only 6 of 35 replayed trigger designs yielded valid per-mutant unions, and none showed kills beyond the stronger arm.
-
Several central numbers do not agree. The abstract says that routing to one agent reaches 91.7% of oracle quality. Table 2 gives 0.719 for one routed run and 0.841 for the oracle, or 85.5%; 91.7% corresponds to the escalated reward of 0.771. Table 1 reports 237 passing episodes among 294 scored episodes but gives a pass rate of 82.7%; the quotient is 80.6%. Section 4 describes the primary designs as having 29–484 RTL lines, while Table 9 places 35 designs above 500 lines. The size-bin means in Table 9 yield a weighted mean reward of about 0.821, rather than the reported “All” value of 0.858.
-
Line 161 abruptly says that the final features replaced “degenerate v1 features.” Tables 3, 11, and 13 and Appendix G continue to use internal labels such as “enriched v2” and “dead feat.” These unexplained version references read like unedited drafting artifacts, possibly from AI-assisted writing.
-
The flowchart is rough, and its content does not match the principal evaluation. Figure 1 and the abstract describe merging TB1 and TB2 into the final suite. Table 2's main result of 0.070 instead keeps the higher-scoring single run. Appendix F mainly estimates the “merged” result from the maximum of each metric across the two runs; it does not report execution of the merged suite on the full corpus. The system shown in the figure is therefore not the system behind the headline result.
Questions for authors
in weaknesses