acceptodds
Under review as a conference paper at ICLR 2027

VERDICT: Verifier-Decoupled Inference-Time Cross-Verification for Verilog Generation

Abstract

Test-time scaling for Verilog samples several designs and must return one without a testbench. With the pool's own selector, the return, not the sampling, is where correct designs are lost: on VerilogEval-v2, nine samples from two cheap model families contain a correct design for of problems, more than Claude-Opus-4.8 solves in one shot, yet the pool's cross-verifier returns one for only , and twelve pool-internal selectors all stay below one-shot. We present VERDICT, a verifier-decoupled, testbench-free harness that separates selection from generation. VERDICT freezes a cheap pool so that every selector is scored on the same candidates; makes selection a swappable slot whose default extends ChipMATE's Verilog-Python cross-verification; and, before the slot, runs execution tools that drop candidates which fail to compile or contradict a waveform the specification itself prints, plus a behavior gate that calls the selector only where survivors differ. Filling the slot with Claude-Opus-4.8, which judges but never writes Verilog, recovers percentage points on VerilogEval-v2 and on RTLLM-v2, positive on every seed, bringing the harness to , statistically indistinguishable from a matched frontier one-shot () that writes the design itself. On VerilogEval-v2 the tools add points to the unchanged cross-verifier without losing a problem; waveform replay covers the of specifications that print one and was calibrated on them. Run over the frontier model's own samples, the same harness reaches against one-shot's (not significant). VERDICT's accounting also explains when a harness pays off: a pool gains over one-shot exactly when its margin exceeds its selection loss. VERDICT thus turns "sample more" into three quantities to report: coverage, selection loss, and cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI4Chip, RTL, Verilog generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_uvisOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This manuscript studies the test-time scaling for Verilog code generation processes that utilize multiple candidates and finally choose one without simulation. The authors show that while existing works generate correct designs, the cross-verifier has a lower correctness rate. The manuscript’s contribution lies in the study of the accuracy loss, development of a selector, and cost calculation of price-matched one-shots.

Critical strengths

• The paper has a detailed ablation study on the experiments and results. • The manuscript’s coverage ≠accuracy is correct, and the baseline fairness by keeping the cost of generation is a good way to maintain fairness for different generation methods. • The observations and insights are informative and honest, even when the observations go against them.

Critical weaknesses
  1. Value relative to one-shot generation. The proposed selector clearly reduces the selection loss of existing candidate-selection methods, and the frozen-pool design makes this gain easy to attribute. However, by the authors' own evaluation, frontier one-shot generation achieves comparable or better accuracy at lower cost (e.g., 91.2% vs. 90.8% on VerilogEval-v2 at ≥5× the cost). This raises a broader question that the paper could address more directly: under what conditions is improving candidate-based generation preferable to one-shot generation? A clearer discussion of the settings where the approach is expected to pay off would considerably strengthen the paper's motivation.
  2.   Clarity of the novel contribution. Much of the reported improvement appears to come from placing a more capable model (Claude-Opus-4.8) in the selector slot. The second frontier judge (gpt-5) did not reproduce the gain, which suggests the improvement may depend on the specific model rather than on the framework itself. It would help if the authors more clearly identified which components of VERDICT are novel and which drive the gains independently of judge choice. Spec-waveform replay, for instance, seems like a promising idea that deserves more emphasis and a clearer positioning relative to prior work.
    
  3.   Scope of baselines. The main evaluation compares primarily against one family of approaches: one-shot generation and pool-internal selectors. Other established design-generation paradigms are directly relevant, such as iterative tool-feedback methods (e.g., AutoChip by Thakur et al.) and multi-agent workflows (e.g., VerilogCoder, which is cited in the paper but not evaluated against). Including at least one such baseline under the same protocol would make the conclusions considerably more convincing.
    
  4.   Readability. The paper is rigorous and commendably transparent, but it is quite dense and can be difficult to follow. It introduces many specialized terms (e.g., floor pick, capture, problem-seeds, behavior, disputed problems). Some notation, such as “pp,” is defined only briefly at first use and then used throughout. A short glossary or notation table, a more streamlined main text, and a clearer organization of the appendices (the corrections log, for example, might fit better in supplementary material) would substantially improve accessibility.
    
Questions for authors
  1. How does the proposed method compare with iterative (tool-feedback) and multi-agent design-generation approaches such as AutoChip or VerilogCoder? These seem to be natural alternatives to candidate-based generation, and a comparison under your paper-fair protocol would be very informative.
  2.   Given that frontier one-shot generation performs comparably to the improved harness, could the authors elaborate on the practical scenarios in which they would recommend VERDICT? For example, are there settings involving cost constraints, model availability, or particular specification types where the approach offers a clear advantage?
    
1 comment

Thank you for the careful review and the two direct questions. For this response we made a cheap model's own samples the main setting, replicated it on pre-registered fresh seeds, isolated each component and the judge, and ran AutoChip and MAGE with the testbench hidden. Changes for your review are marked in violet in the revised PDF. A problem-seed is one problem under one generation seed; "a / b" counts fixes / breaks, exact McNemar test.

1. Value relative to one-shot generation, and when to recommend VERDICT (W1, Q2)

VERDICT pays when a cheap model writes 5 samples and VERDICT picks one; Section 5 now evaluates exactly this.

VerilogEval-v2Acc. (%)$ / problem-seedVERDICT@5 vs it
VERDICT@5, Haiku 5.595.10.0030
Haiku first sample, seeds 1-391.716 / 0, p = 3.1e-5
Same, fresh seeds 4-6 (pre-registered)92.114 / 1, p = 9.8e-4
Sonnet 5.5 one-shot93.40.004811 / 3, p = 0.057
Opus 5.5 one-shot95.90.00804 / 8, p = 0.39

Cost. VERDICT@5 costs 37% of an Opus 5.5 one-shot; Opus 5.5 is 0.85 points higher, not significant. A pre-registered bootstrap puts VERDICT +1.61 points (95% CI [+0.28, +2.92]) above the best price-matched mix of one-shots, at list prices (Appendix B.5).

When it pays. The gain follows the headroom between the first sample and the best one: Haiku has 20 problem-seeds of headroom and gains 16; Opus 5.5 has 3 and gains little. This is descriptive, not a tested rule.

Recommendations (Section 7). Tight budget: VERDICT over 3-5 cheap samples. No judge budget: the largest behaviour group gives about half the gain. Specifications that print a waveform: turn replay on. Frontier generator: use one-shot.

Scope: RTLLM-v2 repeats the gain on 5 of 6 seeds but is not significant (8 / 2, p = 0.11). Appendices A.3 and B.1-B.2 give the timeline and replication.

2. Which components are novel and drive the gain independently of the judge (W2)

Three mechanisms are new, and none depends on one judge model.

  • The one-cycle-shift tie. Execution agreement cannot separate a correct design from its one-cycle-late copy. A pre-registered rule that sees only the agreement probe recovers 0 of 12 such losses; each Claude 5.5 judge that reads the specification recovers 12 of 12 (Appendix F).
  • Spec-waveform replay. It turns the timing table a specification prints into an executable check. On held-out specifications it adds 14 / 0 (p = 0.0001) and wrongly drops 2 of 2,875 correct designs (Appendix E.1). AlphaCode filters with input/output examples; replay aligns clock and reset and compares every cycle.
  • The disagreement gate. Only problems where simulated behaviour groups disagree go to a judge, which picks among existing samples. Grouping itself is prior work (MBR-exec).
Haiku 5.5 own samples, 936 problem-seedsAcc. (%)Judge callsVERDICT@5 vs it
MBR-exec (largest group)93.3017 / 1, p = 1.5e-4
Judge alone, reads all 593.973810 / 0, p = 0.002
Tools + judge, no gate94.67345 / 1, p = 0.22
VERDICT@5, Haiku judge95.0129
VERDICT@5, gpt-5-mini judge94.81295 / 3, p = 0.73

What this shows. The gate keeps the judge's accuracy at about a sixth of its calls. A second-vendor judge works: gpt-5-mini gives 14 / 3 (p = 0.013) over tools + largest group, against Haiku's 14 / 1. gpt-5 on the submitted pool gave a smaller, non-significant gain (17 / 7, p = 0.064). Appendices B.3, C.1 and C.2 report these.

3. Tool-feedback and multi-agent baselines (W3, Q1)

We ran AutoChip and MAGE on Haiku 5.5 in a sandbox holding only the specification; a system-call trace shows 0 testbench reads.

Testbench hidden, Haiku 5.5Acc. (%)VERDICT@5 vs it
MAGE (one run)92.314 / 7, p = 0.19
AutoChip, compiler feedback only89.118 / 1, p = 7.6e-5

Reading. VERDICT is not significantly different from MAGE. Each baseline ran once, without the high-effort setting VERDICT uses, so the paper reports them as context. VerilogCoder takes the official testbench as a required input and cannot run under this protocol (Appendix G.3). Appendix G.1 gives the MAGE scoring rule.

4. Readability (W4)

  • One order: diagnosis (Section 3), method (4), main result (5), frozen-pool judge study (6).
  • Glossary: Table 1 defines six terms; floor pick, capture and disputed are gone; "pp" is spelled out as percentage points.
  • Appendices: cut to 8 (A to H); the corrections log moved to the supplement; the main text fits in 9 pages.

Thank you again. We welcome any further questions.

Reviewer_SD2TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In this work, the authors analyze the tradeoffs in harness design for verilog generation. The core takeaway is that, if multiple implementations are generated, selecting the best one is difficult. Overall, authors show that if a testbench is used as an oracle, the success rate of multi-model harnesses with cheaper models can outperform a frontier model on some benchmark. Authors also propose some mechanisms to improve solution selection without the oracle, and showcase a range of experiments that examine tradeoffs in design choices. The general conclusion seems to be that investing more in better approaches for solution selection is useful, and that generation is not such a bottleneck.

Critical strengths
  • Motivation is clear from the fact that the correct solution does not get selected even if it is contained in the generated implementations.
  • Experiments try to isolate individual choices in harness design, and the insights/tricks in the paper are helpful for practitioners. Especially the setup to run larger cheap pools and use frontier for selection seems fairly nice.
  • Finding issues in earlier evaluations is useful (appendix C).
  • The appendices contain a lot of interesting results around other datasets and specific costs.
Critical weaknesses
  • The analysis seems to be mainly a comparison of design decisions in a fairly specific harness setup. While I appreciate that this is valuable to the automated design community, I do not really see the overall relevance to an ML audience at ICLR.
  • Frontier model performance only sticks to opus 4.8 (and a gpt-5 eval in appendix). This model essentially matches the suggested improvements at one-shot, and if its given 3-5 shots just beats the suggestions (Table 5, excluding Oracle solution selection). Overall, frontier one-shot seems ideal from the cost/benefit. Notably, for cost one-shot from frontier and cheap models are both on the pareto front, whereas none (except exploratory) of the VERDICT setups are (Table 7).
  • A lot of the limitations appendix (appendix S on page 39-41) seems like it would be relevant in the main paper (or at least before the 27 pages of other appendices). Overall, the paper with its appendices reads very confusing. A lot of claims seem more nuanced than they appear initially.
  • Appendix B (line 690) states that the 3 models used in the harness are likely 2 models (deepseek-coder/deepseek-chat api endpoints were the same model). This should much more clearly be stated in the main text (and preferably rerun with another model included as is done in a later appendix).
Questions for authors
  • If frontier model one-shots looks best overall, and frontier judges also seem good with a cheap generation pool, what is the point of not using the best models at one shot?
  • How would current frontier models (astra/fable), or current cheap models (GLM/KIMI) do on these evals?
1 comment

Thank you for the careful review and the two direct questions. For this response we ran a pre-registered fresh-seed replication, a cost-frontier bootstrap, six RTLLM-v2 seeds, a component ablation, a gpt-5-mini judge inside VERDICT, and own samples of two non-Anthropic models. Changes for your review are marked in amber in the revised PDF. A problem-seed is one VerilogEval-v2 problem under one generation seed (156 x 3 = 468); "15 / 1" means our method is right and the comparator wrong on 15, the reverse on 1 (exact McNemar).

1. Why not just use the best model one-shot? (Q1, W2)

If only accuracy matters, use the strongest one-shot; if cost matters, VERDICT over a cheap model's own samples is the most accurate measured point at its price. Seeds 1-3, list prices:

SetupAcc. (%)$ / problem-seedOutcome
Haiku 5.5 one-shot92.10.00045VERDICT@5 beats it 15 / 1, p = 5.2e-4
VERDICT@5, Haiku samples + judge95.10.00295on the cost frontier
Best mix of two one-shots, same price93.40.00295below VERDICT
Sonnet 5.5 one-shot93.40.0048off the frontier
Opus 5.5 one-shot95.90.0080higher, n.s. (VERDICT 4 / 8, p = 0.39)

Frontier samples. Over Opus 4.8's own 5 samples, VERDICT lifts Opus 4.8 from 90.4% for its first sample to 92.9% (12 / 0), so extra frontier shots also need a picker.

Fresh seeds. On pre-registered seeds 4-6, VERDICT@5 scores 94.9% against 92.1% for the first sample (14 / 1, p = 9.8e-4) and is again on the frontier. Over seeds 1-6 it stays on the frontier in 99.3% of 10,000 bootstrap replicates at list prices. On RTLLM-v2 (six seeds) the gain is the same direction but not significant (8 / 2, p = 0.11). The edge is price (about half a cent per problem-seed), so Section 7 calls it exploratory. Appendices B.2 and B.5 report this.

2. Current frontier and cheap models (Q2, W2)

The result does not hinge on Opus 4.8 or on one vendor. We added the Claude 5.5 family (Haiku, Sonnet, Opus) as one-shots, judges and samplers. A gpt-5-mini judge inside VERDICT gains like the Haiku judge (14 / 3 over tools + largest group, p = 0.013; the two judges differ on only 8 problem-seeds). Over DeepSeek's own samples, tools plus grouping lift 71.8% to 77.8% (32 / 4); over gpt-5-mini's own samples the gain is n.s. (19 / 10). GLM, Kimi, gpt-6-astra and Fable 5.2 had no API route or verified price within our cap. Appendices B.4, C.2 and H report this.

3. Relevance to an ML audience (W1)

Picking one of several samples without a checker is a general test-time-scaling problem, and our mechanisms address why agreement-based selection fails:

  • The one-cycle-shift tie. Execution agreement cannot separate a correct design from a copy one clock cycle late. A pre-registered LLM-free tie-break recovers 0 of 12 such losses; each Claude 5.5 judge reading the spec recovers 12 of 12 (Appendix F).
  • Spec-waveform replay. Turning a spec's printed timing table into clocked stimulus gives 14 / 0 (p = 0.0001) on held-out specs (Appendix E.1).
  • Disagreement-gated judging. Calling a judge only where behaviour groups disagree beats grouping alone (MBR-exec) 17 / 1 and the judge alone 10 / 0, with judges from two vendors, statistically as accurate as judging every survivor with about a sixth of the calls (Appendix B.3).
  • Judge findings. The common "exactly one candidate is correct" prompt makes cheap judges spend output hunting for absent bugs (accuracy did not change in a pre-registered check), and count sensitivity depends on the judge (Opus 4.8 drops from 2 to 9 candidates, Haiku 5.5 does not; Appendix D).

The scope is one domain; HumanEval and LiveCodeBench runs in the supplement cover coverage and selection only.

4. DeepSeek aliases (W4)

Section 6 of the main text now says the pool is three API names, likely two models (25.4% of cross-name sample pairs are byte-identical, against 3.7% across families; Appendix A.1). The main evaluation no longer uses this pool; it runs on each model's own samples.

5. Structure and limitations (W3)

  • One story. Selection loses designs (Section 3), VERDICT (Section 4), own samples as the main evaluation (Section 5), the frozen pool as a controlled study (Section 6).
  • Limitations up front. Section 9 states the key limits in the main text; Appendix H is the single full list.
  • Length. 8 appendices (A-H), down from 19; older material moved to a supplement, and Table 1 is a glossary.
  • Nuance stated once. The abstract now separates the measured loss and the mechanisms from the exploratory cost payoff.

Thank you again. We welcome any further questions.

Reviewer_3c62OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an empirical study of testbench-free selection for LLM-generated Verilog. VERDICT freezes candidate pools and compares different selectors, separating pool coverage from the accuracy of the returned design. The study finds that agreement-based selection can leave substantial coverage unused, and that replacing the default judge with Claude recovers part of this loss on VerilogEval-v2 and RTLLM-v2. It also examines execution filters, behavior-based gating, candidate-pool scaling, and inference cost. The study shows that increased candidate coverage need not improve final selected accuracy, and that the evaluated cheap-pool configurations do not demonstrate an advantage over frontier one-shot generation in accuracy or cost.

Critical strengths
  1. Freezing candidate pools provides a useful controlled setting for studying selection separately from generation. Replacing the default judge with Claude improves accuracy by 5.6 percentage points on VerilogEval-v2 and 4.2 points on RTLLM-v2, with positive gains on every generation seed. The analyses of disputed problems and sequential designs also help pinpoint where the default cross-verifier loses correct solutions that are already available in the pool.
  2. The study reports a broad range of unsuccessful interventions and carefully revisits its baselines and cost accounting. In particular, the matched-prompt one-shot comparison and price-matched mixtures prevent improvements over the default selector from being mistaken for practical advantages over simpler alternatives. The explicit reporting of negative results and experimental limitations is valuable.
Critical weaknesses
  1. The additional insight beyond prior work remains unclear. CodeT already studies selection among generated programs using tests and execution agreement, and Large Language Monkeys explicitly reports that selection methods can plateau as sampling coverage increases. The paper acknowledges these works, but its central observations largely reproduce some known bottleneck in RTL. I am not yet convinced that the study offers a sufficiently new understanding of the bottleneck or a distinctive RTL-specific finding.
  2. The experiments do not sufficiently establish the conditions under which these observations hold. In the pool comparisons in Appendix K, candidate count, composition, and quality change together, so the increasing selection loss does not isolate an effect of pool size. The Claude gain is not reproduced by the second frontier judge, and the CVDP comparison also changes the judge prompt, which makes the scope of the finding hard to interpret. Meanwhile, the waveform-filter benefit is evaluated on the same 12 specifications used for calibration. These limitations are acknowledged, but they leave the paper with several configuration-specific observations and limited evidence about which properties of the task, pool, or selector determine the outcome.
Questions for authors
  1. Which finding do the authors consider the main conceptual advance over prior work on execution-based code selection and the coverage–selection gap? Could the paper identify a specific conclusion that changes our understanding of selection in RTL, rather than primarily confirming that imperfect selectors fail to recover all available correct candidates?
  2. Could the authors add a controlled analysis that varies candidate count while holding the number of correct candidates and the candidate-source mixture fixed, with randomized candidate order and a fixed judge protocol and budget? Repeating this comparison across the two frontier judges would help determine whether the observed increase in loss is a consistent selection phenomenon or depends on the particular pool and judge.
1 comment

Thank you for the careful review and two concrete questions. For this response we ran your controlled candidate-count study with three judges, a pre-registered shift-aware tie-break, a held-out replay test, a gpt-5-mini judge inside VERDICT, and fresh seeds. Changes for your review are marked in teal in the revised PDF. A problem-seed is one benchmark problem under one generation seed (VerilogEval-v2: 156 x 3 = 468); "a / b" counts fixes (VERDICT right, the other wrong) and breaks, exact McNemar test.

1. The new understanding beyond CodeT and Large Language Monkeys (W1, Q1)

Those works show that agreement-based selection plateaus; we show why it fails on hardware and what recovers the loss.

FindingWhat is newOutcome
One-cycle-shift tieExecution agreement cannot tell a correct design from its one-cycle-late copyReference votes tie in 16 of 35 such pools; an LLM-free shift-aware tie-break recovers 0 of 12 shift losses, each Claude 5.5 judge 12 of 12
Spec-waveform replayThe spec's printed timing table becomes clocked, reset-aligned stimulus14 / 0 on held-out specs (exact McNemar p = 0.0001)
Disagreement-gated judgeA judge is called only where simulated behaviour groups disagree17 / 1 over MBR-exec, 10 / 0 over the judge alone; as accurate as judging every survivor (5 / 1, n.s.) with about 1/6 of the calls

Why this is RTL-specific. The 0 / 12 control shows a rule that sees only random-probe outputs lacks the timing information. A judge reading the specification recovers 12 / 12, and replay supplies it where a waveform is printed. So the fix is a timing signal, not more samples. Loss types are our reading of each specification. Appendix F reports the loss breakdown and the tie-break null (F.2).

2. Varying only the candidate count (W2, Q2)

The earlier pool comparisons changed count, mix and quality together, so we ran your design with three judges: one correct design plus n-1 wrong ones, nested sets, three random orders each (35 problems; gpt-5 20). The source mix of wrong designs was not held fixed.

Judgen = 2359Trend p
Opus 4.887.685.777.165.78.5e-8
gpt-583.380.078.368.30.0046
Haiku 5.594.392.493.394.30.86

What this shows. Count sensitivity depends on the judge, so it is not a fixed law of best-of-n selection. Every judge loses less than independent errors would predict. About half of Opus 4.8's drop is replies cut off at its output limit, and output limits differ by judge, so judge and budget are not separated. Appendix D reports the study, the third judge (D.2) and the null model (D.3).

3. The second judge, the CVDP prompt, and replay on its tuning set (W2)

A judge from a second vendor now reproduces the gain inside VERDICT.

Two vendors. Over Haiku 5.5's own samples (936 problem-seeds), the Haiku judge beats tools + largest group 14 / 1 and gpt-5-mini 14 / 3 (p = 0.013); the two judges differ on only 8 problem-seeds. On the frozen cheap pool, each Claude 5.5 judge gains 28-29 / 0 over the cross-verifier with tools; gpt-5's gain is smaller and not significant (17 / 7, p = 0.064). Appendices C.1 and C.2 report this.

Held-out replay. We froze replay's code and added a waveform table, simulated from the reference, to the 144 specs that print none. It gives 14 / 0 and wrongly drops 2 of 2,875 correct designs. These tables are synthetic, so this is a mechanism test (Appendix E.1).

CVDP. Its judge prompt differs, so it is no longer used to argue when the finding holds and stays as context only.

4. Which properties decide the outcome (W2)

The property we measured is headroom: the best of a model's samples must beat its first one. Gains follow it: Haiku 5.5 16 / 0, confirmed on fresh seeds 4-6 (14 / 1, p = 9.8e-4); Opus 4.8 12 / 0; little where headroom is small. gpt-5-mini has headroom but no significant gain, so headroom is needed but not enough. Section 7 calls it a descriptive pattern, not a rule. Scope: RTLLM-v2 is directional (8 / 2 over six seeds, n.s.), and an Opus 5.5 one-shot scores higher than VERDICT on Haiku, not significantly, at about 2.7x the cost. Appendix B reports all own-sample runs.

Thank you again. We welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.