- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This manuscript studies the test-time scaling for Verilog code generation processes that utilize multiple candidates and finally choose one without simulation. The authors show that while existing works generate correct designs, the cross-verifier has a lower correctness rate. The manuscript’s contribution lies in the study of the accuracy loss, development of a selector, and cost calculation of price-matched one-shots.
Critical strengths
• The paper has a detailed ablation study on the experiments and results. • The manuscript’s coverage ≠accuracy is correct, and the baseline fairness by keeping the cost of generation is a good way to maintain fairness for different generation methods. • The observations and insights are informative and honest, even when the observations go against them.
Critical weaknesses
- Value relative to one-shot generation. The proposed selector clearly reduces the selection loss of existing candidate-selection methods, and the frozen-pool design makes this gain easy to attribute. However, by the authors' own evaluation, frontier one-shot generation achieves comparable or better accuracy at lower cost (e.g., 91.2% vs. 90.8% on VerilogEval-v2 at ≥5× the cost). This raises a broader question that the paper could address more directly: under what conditions is improving candidate-based generation preferable to one-shot generation? A clearer discussion of the settings where the approach is expected to pay off would considerably strengthen the paper's motivation.
-
Clarity of the novel contribution. Much of the reported improvement appears to come from placing a more capable model (Claude-Opus-4.8) in the selector slot. The second frontier judge (gpt-5) did not reproduce the gain, which suggests the improvement may depend on the specific model rather than on the framework itself. It would help if the authors more clearly identified which components of VERDICT are novel and which drive the gains independently of judge choice. Spec-waveform replay, for instance, seems like a promising idea that deserves more emphasis and a clearer positioning relative to prior work. -
Scope of baselines. The main evaluation compares primarily against one family of approaches: one-shot generation and pool-internal selectors. Other established design-generation paradigms are directly relevant, such as iterative tool-feedback methods (e.g., AutoChip by Thakur et al.) and multi-agent workflows (e.g., VerilogCoder, which is cited in the paper but not evaluated against). Including at least one such baseline under the same protocol would make the conclusions considerably more convincing. -
Readability. The paper is rigorous and commendably transparent, but it is quite dense and can be difficult to follow. It introduces many specialized terms (e.g., floor pick, capture, problem-seeds, behavior, disputed problems). Some notation, such as “pp,” is defined only briefly at first use and then used throughout. A short glossary or notation table, a more streamlined main text, and a clearer organization of the appendices (the corrections log, for example, might fit better in supplementary material) would substantially improve accessibility.
Questions for authors
- How does the proposed method compare with iterative (tool-feedback) and multi-agent design-generation approaches such as AutoChip or VerilogCoder? These seem to be natural alternatives to candidate-based generation, and a comparison under your paper-fair protocol would be very informative.
-
Given that frontier one-shot generation performs comparably to the improved harness, could the authors elaborate on the practical scenarios in which they would recommend VERDICT? For example, are there settings involving cost constraints, model availability, or particular specification types where the approach offers a clear advantage?
Thank you for the careful review and the two direct questions. For this response we made a cheap model's own samples the main setting, replicated it on pre-registered fresh seeds, isolated each component and the judge, and ran AutoChip and MAGE with the testbench hidden. Changes for your review are marked in violet in the revised PDF. A problem-seed is one problem under one generation seed; "a / b" counts fixes / breaks, exact McNemar test.
1. Value relative to one-shot generation, and when to recommend VERDICT (W1, Q2)
VERDICT pays when a cheap model writes 5 samples and VERDICT picks one; Section 5 now evaluates exactly this.
Cost. VERDICT@5 costs 37% of an Opus 5.5 one-shot; Opus 5.5 is 0.85 points higher, not significant. A pre-registered bootstrap puts VERDICT +1.61 points (95% CI [+0.28, +2.92]) above the best price-matched mix of one-shots, at list prices (Appendix B.5).
When it pays. The gain follows the headroom between the first sample and the best one: Haiku has 20 problem-seeds of headroom and gains 16; Opus 5.5 has 3 and gains little. This is descriptive, not a tested rule.
Recommendations (Section 7). Tight budget: VERDICT over 3-5 cheap samples. No judge budget: the largest behaviour group gives about half the gain. Specifications that print a waveform: turn replay on. Frontier generator: use one-shot.
Scope: RTLLM-v2 repeats the gain on 5 of 6 seeds but is not significant (8 / 2, p = 0.11). Appendices A.3 and B.1-B.2 give the timeline and replication.
2. Which components are novel and drive the gain independently of the judge (W2)
Three mechanisms are new, and none depends on one judge model.
What this shows. The gate keeps the judge's accuracy at about a sixth of its calls. A second-vendor judge works: gpt-5-mini gives 14 / 3 (p = 0.013) over tools + largest group, against Haiku's 14 / 1. gpt-5 on the submitted pool gave a smaller, non-significant gain (17 / 7, p = 0.064). Appendices B.3, C.1 and C.2 report these.
3. Tool-feedback and multi-agent baselines (W3, Q1)
We ran AutoChip and MAGE on Haiku 5.5 in a sandbox holding only the specification; a system-call trace shows 0 testbench reads.
Reading. VERDICT is not significantly different from MAGE. Each baseline ran once, without the high-effort setting VERDICT uses, so the paper reports them as context. VerilogCoder takes the official testbench as a required input and cannot run under this protocol (Appendix G.3). Appendix G.1 gives the MAGE scoring rule.
4. Readability (W4)
Thank you again. We welcome any further questions.