acceptodds
Withdrawn from ICLR 2027

VeriEvolve: Harness Evolution for Hardware Verification Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-improving agents need mechanisms for turning task experience into reusable improvements in reasoning and tool use. Hardware verification makes this challenge concrete: success depends on coordinating temporal reasoning, fault activation, discriminating checks, and preservation of previously supported behavior. We present VeriEvolve, a framework for EDA-guided harness evolution in multi-agent verification systems. The central idea is to make the organization of verification experiments an explicit, executable search space. An inner multi-agent system coordinates specification analysis, artifact construction, and experimental challenge, while an outer loop uses development experience to revise resource allocation, diagnostic priorities, and repair or recheck decisions. Verification-specific failure distinctions guide policy proposals, and a fixed execution kernel requires reproducible failure evidence and preservation checks before accepting repairs. This separates adaptation of a task artifact from improvement of a policy reused across designs. We instantiate the framework on stimulus generation, checker construction, assertion generation, and RTL repair in CVDP, and define matched comparisons of instruction-level and executable-policy search with selected policies frozen for transfer evaluation. The formulation treats domain structure as an inductive bias over agent policies and makes its value for self-improvement a testable question across design families and model backbones.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Self-improving Agents, Multi-agent Systems, Hardware Verification

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5BLUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

VeriEvolve frames improving a hardware-verification agent as a search over harness policies (stage budgets, diagnostic priorities, reaction rules, and role instructions) while model weights stay fixed. Each policy runs as a multi-agent executor with an evidence-validation kernel. On 73 CVDP agentic tasks (cid012–016), the Sonnet multi-agent system reaches 26.0% versus 23.3% for a single agent (p=0.774), and the authors acknowledge there is no gain from evolution.

Critical strengths

The paper is unusually candid about its negative results, and the grader-hidden, final-only protocol is a useful step toward evaluation that resists reward hacking.

Critical weaknesses
  1. Limited novelty. Harness and agent-design search is established prior work (ADAS, DGM, GEPA, AFlow, SBCO), so the contribution rests on the EDA-specific policy space and proposal prior. Neither is ever shown to help. What evidence shows that the verification-specific policy space improves over generic harness search, beyond the framing?

  2. There is no evolution. The EDA-guided proposal fails schema validation, the other proposals tie the initial policy, and selection keeps the ancestor in every condition. No evolved policy is ever evaluated. Can you show even one evolved policy that beats the initial policy on held-out designs?

  3. The evidence kernel almost never acts. Across 73 episodes, it admits 1 diagnosis and executes 0 repairs, so the preservation guarantee in Equation 3 is essentially never exercised. Is the kernel so restrictive that it blocks useful repairs, and how many valid diagnoses did it reject?

  4. HORIZON (Yu et al., 2026) reaches 100% on cid012–016 with a multi-iteration agent loop. VeriEvolve reaches 26.0%, and only 41.1% even in its own grader-stop setting, yet HORIZON is mentioned only in an appendix. How does HORIZON, or a plain multi-iteration loop, perform under your final-only protocol, and what does VeriEvolve add beyond it?

  5. It is below a simple baseline. VeriEvolve with Sonnet 5 (26.0%) scores below the plain single-agent CVDP Sonnet 5 wrapper (31.5%). Why does the added multi-agent and policy machinery hurt performance?

  6. The 41.1% Gemma figure is the better of two historical runs (the other scored 32.9%), and it uses grader-stop feedback and a skill store curated with grader success. Can you report all results under one matched, final-only protocol with multiple seeds?

  7. The sample sizes are tiny. Budget, architecture, and policy controls use 8 designs with one rollout each, transfer uses 16 designs, and several cells hold a single Hard design. Can you add repeated rollouts and larger development and transfer sets so the conclusions are statistically meaningful?

  8. The presentation is hard to follow. Historical, pilot, rebuilt-kernel, and matched cohorts are interleaved, which makes the main result hard to reconstruct. Can you restructure the paper around one pre-registered matched comparison?

Questions for authors
  1. Limited novelty. Harness and agent-design search is established prior work (ADAS, DGM, GEPA, AFlow, SBCO), so the contribution rests on the EDA-specific policy space and proposal prior. Neither is ever shown to help. What evidence shows that the verification-specific policy space improves over generic harness search, beyond the framing?

  2. There is no evolution. The EDA-guided proposal fails schema validation, the other proposals tie the initial policy, and selection keeps the ancestor in every condition. No evolved policy is ever evaluated. Can you show even one evolved policy that beats the initial policy on held-out designs?

  3. The evidence kernel almost never acts. Across 73 episodes, it admits 1 diagnosis and executes 0 repairs, so the preservation guarantee in Equation 3 is essentially never exercised. Is the kernel so restrictive that it blocks useful repairs, and how many valid diagnoses did it reject?

  4. HORIZON (Yu et al., 2026) reaches 100% on cid012–016 with a multi-iteration agent loop. VeriEvolve reaches 26.0%, and only 41.1% even in its own grader-stop setting, yet HORIZON is mentioned only in an appendix. How does HORIZON, or a plain multi-iteration loop, perform under your final-only protocol, and what does VeriEvolve add beyond it?

  5. It is below a simple baseline. VeriEvolve with Sonnet 5 (26.0%) scores below the plain single-agent CVDP Sonnet 5 wrapper (31.5%). Why does the added multi-agent and policy machinery hurt performance?

  6. The 41.1% Gemma figure is the better of two historical runs (the other scored 32.9%), and it uses grader-stop feedback and a skill store curated with grader success. Can you report all results under one matched, final-only protocol with multiple seeds?

  7. The sample sizes are tiny. Budget, architecture, and policy controls use 8 designs with one rollout each, transfer uses 16 designs, and several cells hold a single Hard design. Can you add repeated rollouts and larger development and transfer sets so the conclusions are statistically meaningful?

  8. The presentation is hard to follow. Historical, pilot, rebuilt-kernel, and matched cohorts are interleaved, which makes the main result hard to reconstruct. Can you restructure the paper around one pre-registered matched comparison?

Reviewer_VRzLOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies whether the verification harness can be treated as an explicit adaptation target in the context of EDA verification tasks. The paper provides a formalization of the problem, and presents a method consists of three parts: a fixed execution kernel, an evidence-conditioned repair mechanism, and a development-time search over policies that are then frozen for transfer evaluation. Experimental evaluations are conducted on four categories to show the effectiveness of the proposed method.

Critical strengths
  • The paper studies an important problem. Verification harnesses in EDA are a real bottleneck, and the harness is a natural adaptation target when model weights are fixed.

  • The evaluation protocol is carefully designed. The paper separates final-only vs grader-stop evaluation, matched vs unmatched comparisons, etc.

Critical weaknesses
  • The main empirical claim is not supported. The search never improves the policy (Table 4), the MAS is not significantly better than single-agent (Table 3), the transfer gain is not significant (Table 15), and the MAS is worse on Hard tasks (Table 16).

  • The paper does not quantify the cost of the harness search relative to the benefit. The evolution loop is costly. If the search does not improve over the ancestor (Table 4), the cost is pure overhead.

  • The presentation can be significantly improved.

Questions for authors
  1. What is the actual improvement of the proposed method over the state-of-the-art, both theoretically and empirically?

  2. Why is the repair path never exercised in most cases?

Reviewer_P9hmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies electronic design verification using a fixed LLM paired with a modifiable (“malleable”) harness. Agents take written specification documents (and, in the main setting, the RTL source) and produce verification artifacts (stimuli, checkers, and assertions) intended to expose design flaws. A fourth task type asks the agents to repair the RTL directly. Evaluation uses 73 tasks from the agentic subset of CVDP (Comprehensive Verilog Design Problems). The paper asks two questions: (1) whether a multi-agent harness outperforms a single agent, and (2) whether searching over harness configurations outperforms a fixed harness. A side experiment withholds the RTL from the agents.

Critical strengths

The problem is well justified based on prior work. They document the cost of verification time, and the authors tie this to real engineering work where verification environments are often built before the RTL is finished.

The authors are honest about the limitations. They state in the abstract that no evolution gain is established, provide the p-value in the main text, and willingly provide other limitations.

The experiment design is reasonable. Retraining models is often impractical, and continual learning is difficult, so focusing on the harness is a reasonable approach.

Critical weaknesses

In the main study, the search mechanism is very small, with one proposal per category; each is scored once on eight tasks, making it difficult to draw any conclusions about the harness evolution.

The paper often uses notation and terms before defining them (if at all); for example, T_K, μ, and H0 are never defined, and "modifier," "relay," and "skill store" are used without explanation, among others.

Some parts are overly complicated or opaque. For example, Equation 2 amounts to "the LLM proposes an action, the kernel allows or downgrades it, and the state updates." The extra mathematical formulation is unnecessary and confuses readers about what the authors are doing.

The best description of the authors' approach is Figure 2. However, panel A adds little information, and the figure relies on a symbol defined later in the paper. A concrete example of a policy change would be more helpful.

The authors define their objective in Equation 1 as an expectation over run-to-run randomness but never estimate it. All the numbers appear to come from a single rollout per task, which weakens their claims.

Some experiments use no clean holdout or test set. In these cases, they appear to have some overlap with development. For example, the 73-task evaluation "include[s] previously studied tasks" (Appendix N).

Questions for authors

The key differntiator is not very clear from fixed vs evolved harness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.