- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
VeriEvolve frames improving a hardware-verification agent as a search over harness policies (stage budgets, diagnostic priorities, reaction rules, and role instructions) while model weights stay fixed. Each policy runs as a multi-agent executor with an evidence-validation kernel. On 73 CVDP agentic tasks (cid012–016), the Sonnet multi-agent system reaches 26.0% versus 23.3% for a single agent (p=0.774), and the authors acknowledge there is no gain from evolution.
Critical strengths
The paper is unusually candid about its negative results, and the grader-hidden, final-only protocol is a useful step toward evaluation that resists reward hacking.
Critical weaknesses
-
Limited novelty. Harness and agent-design search is established prior work (ADAS, DGM, GEPA, AFlow, SBCO), so the contribution rests on the EDA-specific policy space and proposal prior. Neither is ever shown to help. What evidence shows that the verification-specific policy space improves over generic harness search, beyond the framing?
-
There is no evolution. The EDA-guided proposal fails schema validation, the other proposals tie the initial policy, and selection keeps the ancestor in every condition. No evolved policy is ever evaluated. Can you show even one evolved policy that beats the initial policy on held-out designs?
-
The evidence kernel almost never acts. Across 73 episodes, it admits 1 diagnosis and executes 0 repairs, so the preservation guarantee in Equation 3 is essentially never exercised. Is the kernel so restrictive that it blocks useful repairs, and how many valid diagnoses did it reject?
-
HORIZON (Yu et al., 2026) reaches 100% on cid012–016 with a multi-iteration agent loop. VeriEvolve reaches 26.0%, and only 41.1% even in its own grader-stop setting, yet HORIZON is mentioned only in an appendix. How does HORIZON, or a plain multi-iteration loop, perform under your final-only protocol, and what does VeriEvolve add beyond it?
-
It is below a simple baseline. VeriEvolve with Sonnet 5 (26.0%) scores below the plain single-agent CVDP Sonnet 5 wrapper (31.5%). Why does the added multi-agent and policy machinery hurt performance?
-
The 41.1% Gemma figure is the better of two historical runs (the other scored 32.9%), and it uses grader-stop feedback and a skill store curated with grader success. Can you report all results under one matched, final-only protocol with multiple seeds?
-
The sample sizes are tiny. Budget, architecture, and policy controls use 8 designs with one rollout each, transfer uses 16 designs, and several cells hold a single Hard design. Can you add repeated rollouts and larger development and transfer sets so the conclusions are statistically meaningful?
-
The presentation is hard to follow. Historical, pilot, rebuilt-kernel, and matched cohorts are interleaved, which makes the main result hard to reconstruct. Can you restructure the paper around one pre-registered matched comparison?
Questions for authors
-
Limited novelty. Harness and agent-design search is established prior work (ADAS, DGM, GEPA, AFlow, SBCO), so the contribution rests on the EDA-specific policy space and proposal prior. Neither is ever shown to help. What evidence shows that the verification-specific policy space improves over generic harness search, beyond the framing?
-
There is no evolution. The EDA-guided proposal fails schema validation, the other proposals tie the initial policy, and selection keeps the ancestor in every condition. No evolved policy is ever evaluated. Can you show even one evolved policy that beats the initial policy on held-out designs?
-
The evidence kernel almost never acts. Across 73 episodes, it admits 1 diagnosis and executes 0 repairs, so the preservation guarantee in Equation 3 is essentially never exercised. Is the kernel so restrictive that it blocks useful repairs, and how many valid diagnoses did it reject?
-
HORIZON (Yu et al., 2026) reaches 100% on cid012–016 with a multi-iteration agent loop. VeriEvolve reaches 26.0%, and only 41.1% even in its own grader-stop setting, yet HORIZON is mentioned only in an appendix. How does HORIZON, or a plain multi-iteration loop, perform under your final-only protocol, and what does VeriEvolve add beyond it?
-
It is below a simple baseline. VeriEvolve with Sonnet 5 (26.0%) scores below the plain single-agent CVDP Sonnet 5 wrapper (31.5%). Why does the added multi-agent and policy machinery hurt performance?
-
The 41.1% Gemma figure is the better of two historical runs (the other scored 32.9%), and it uses grader-stop feedback and a skill store curated with grader success. Can you report all results under one matched, final-only protocol with multiple seeds?
-
The sample sizes are tiny. Budget, architecture, and policy controls use 8 designs with one rollout each, transfer uses 16 designs, and several cells hold a single Hard design. Can you add repeated rollouts and larger development and transfer sets so the conclusions are statistically meaningful?
-
The presentation is hard to follow. Historical, pilot, rebuilt-kernel, and matched cohorts are interleaved, which makes the main result hard to reconstruct. Can you restructure the paper around one pre-registered matched comparison?