acceptodds
Under review as a conference paper at ICLR 2027

When Do RTL Verifiers Lie? Measuring Proxy-Reward Overoptimization in LLM Hardware Design

Abstract

Reinforcement learning with verifiable rewards (RLVR) and inference-time search are increasingly used to improve generated register-transfer-level (RTL) designs. While these techniques require access to rigorous test scripts, high-quality RTL data, and verification infrastructure, these resources are scarce and often proprietary. Recent pipelines therefore rely on capable teacher models to generate synthetic RTL and automated testbenches. However, optimizing against these scalable verification proxies risks Goodhart effects: visible reward may improve without a corresponding gain in true functional correctness. This risk is especially acute because RTL correctness spans input sequences and clock cycles, whereas teacher-generated testbenches, randomized differential simulation, and bounded model checking (BMC) provide scalable but finite checks, with the fidelity of simulation and bounded checking depending on verification effort. We compare four testbench conditions generated by Grok 4.6 and DeepSeek V4 Flash from either the specification or golden RTL, alongside differential simulation and BMC. A hidden formal-equivalence Oracle evaluates candidate RTL against its golden reference. The study covers 100 tasks, eight Qwen3 configurations, 50,948 candidates, Best-of- search, multi-turn repair, and two-epoch RL. Qwen3-32B reasoning reaches 82% Oracle Pass@64, but teacher-generated testbenches select proved designs on only 61–66% of tasks despite visible success rates of up to 93%; tuned BMC and differential simulation reach 78% and 76%, respectively. Increasing verification effort narrows this gap, while weak direct-mode repair feedback can continue improving visible success after hidden progress stalls. In the Qwen3-4B RL run, the proxy–Oracle gap widens and rewarded formal counterexamples become more frequent without a clear improvement in Oracle proof rate; the 8B run shows a persistent proxy–Oracle discrepancy over the same training budget.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reward Overoptimization, Goodhart's Law, Reinforcement Learning with Verifiable Rewards (RLVR), Hardware Design, RTL Synthesis, Verilog, Formal Verification, Best-of-N Search, LLM Reasoning, AI for EDA

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_1nc7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether passing an RTL verifier really means that an LLM has produced a correct design. It compares generated testbenches, differential simulation, and bounded model checking against a separate formal-equivalence oracle. The experiments cover candidate selection, iterative repair, and RL training. The main finding is that a higher test pass rate does not always translate into more correct designs. The paper also studies how much additional verification helps and when the gains start to level off.

Critical strengths
  1. The paper shows why a better RTL generator still needs a reliable checker. I liked the comparison on the same candidate pool. Table 1 (p. 6) shows that a proved solution exists for 82% of tasks, but the generated testbenches select one for only 61–66%. BMC and differential simulation do considerably better. This makes the problem concrete: the model may already produce a correct answer, yet the system fails to identify it. I find this useful for understanding where improvements in generation can get lost.

  2. The verification-budget experiments offer practical guidance. Section 5.3 (p. 6) shows both the benefit of stronger checking and where that benefit starts to flatten. This helps readers think about how much verification effort is worthwhile, rather than simply choosing the strongest available checker. I also appreciated that Section 3.1 (p. 3) distinguishes an actual counterexample from an unfinished proof, since these outcomes support different conclusions.

Critical weaknesses
  1. The RL findings are promising, but their consistency is still unclear. Section 5.5 (p. 8) shows that testbench reward improves while the oracle proof rate changes little. However, the early and late windows were chosen after looking at the trajectory. Section 7 (p. 9) also notes that there is only one run per model configuration, with different rollout counts across configurations. I would be more convinced by another seed or a matched run using a stronger reward verifier. The authors acknowledge these limits, but the RL evidence remains less convincing than the candidate-selection results.

  2. Missing experimental details make the results harder to reproduce. Figure 3’s caption (p. 8) says that the run tag does not establish the exact model checkpoint, while Appendix L (p. 26) says that historical tool versions were not preserved. Appendix P (p. 30) also notes that some cost measurements came from different machines or load conditions. Since the paper’s contribution depends heavily on these measurements, I would feel more comfortable with clearly identified checkpoints, a fixed tool environment, and cost comparisons collected under the same conditions.

Questions for authors

My background is in RL training, and I am relatively new to RTL/EDA. I understand the evaluation setup in Section 4.1 (p. 4), but I would appreciate a concrete example showing how it fits into an actual design workflow. In particular, it would help to explain where the golden reference comes from, why a new implementation is needed when that reference already exists, and what the user hopes to gain. Connecting this example to the results would also help me understand what becomes easier or more reliable once the verification problem is addressed.

I would be interested in your perspective on how this study fits alongside recent methods such as CodeV-R1, ChipSeek, and STG [1–3]. A short discussion of the main bottlenecks in these workflows would help me place the contribution. For someone building a system today, I would especially like to understand whether these findings should change how they construct training rewards, select candidate designs, or provide tool feedback. Since Section 4.4 (pp. 4–5) evaluates repair and RL separately, it would also be useful to discuss which lessons might carry over to multi-turn tool-using agents, while keeping those expectations separate from the findings directly supported here.

The disclosures in Figure 3 and Appendix L leave me unsure how closely someone could reproduce the reported runs. It would help to clarify which exact model checkpoints and tool versions can still be recovered. If some information is no longer available, a confirmation run in a pinned environment would make the main findings easier to trust and build on.

[1] Yaoyu Zhu et al. QiMeng-CodeV-R1: Reasoning-Enhanced Verilog Generation. NeurIPS 2025.

[2] Zhirong Chen et al. ChipSeek: Optimizing Verilog Generation via EDA-Integrated Reinforcement Learning. arXiv:2507.04736, 2025; revised 2026.

[3] En-Ming Huang et al. Structured Testbench Generation for LLM-Driven HDL Design and Verification-Oriented Data Curation. arXiv:2606.12983, 2026.

3 comments

We thank the reviewer for requesting stronger RL controls, clearer evaluation settings, and a concrete account of the intended workflow. The revised manuscript makes the six-run Qwen3-32B study the main RL evidence.

Are the RL results consistent across seeds and stronger rewards?

The revised manuscript now centers the requested repeated seeds and stronger-reward controls. Section 5.5 includes all six Qwen3-32B runs, a three-reward validation figure, and this late held-out comparison. Rates average checkpoints 185, 190, 195, and 200, each with 500 nonpadding outputs:

Training rewardProof success, seed 0 / seed 1Rewarded counterexamples / all outputs, seed 0 / seed 1
DeepSeek testbench17.75% / 16.95%16.95% / 18.10%
GoldDiff L520.65% / 20.80%1.15% / 1.35%
Oracle25.65% / 21.95%0% / 0% — by construction

Oracle rewards support held-out learning in both seeds: proof success rises from 13.80% to 25.65% and from 12.80% to 21.95%, with gains of +11.85 pp [4.35, 19.45] and +9.15 pp [1.90, 16.50]. This empirical learning result is distinct from zero reward disagreement: Oracle-control reward and evaluation use the same online judgment, so zero is an identity.

L5 more closely tracks formal outcomes on its learned policies' outputs. Its late rewarded-counterexample rates are much lower than DeepSeek's, and its signed gaps are −2.10/−3.60 pp versus +16.80/+17.05 pp. Negative gaps reflect rejected proofs as well as residual accepted counterexamples; L5 is not perfect. This is consistent with the earlier selection and effort results. L5's late proof advantage over DeepSeek has task intervals including zero in each seed, so we do not infer a universal final proof-rate ranking.

All six runs improve proof success from their own baseline. DeepSeek training disagreement grows in both seeds; held-out gap growth is clear only in seed 1. Proxy-policy proof labels use a common offline formal protocol, while Oracle-control curves use their online judgments. The runs share an update recipe, not equal total verification compute.

The main figures show both DeepSeek training seeds and all three rewards in validation. Full three-reward training curves and paired-effect tables are in the appendix. The earlier 4B/8B experiments are now supplementary historical evidence.

How are the model, evaluation settings, and cost claims clarified?

We agree that reproducibility requires clearly identified model checkpoints and software environments. The statements in the original Figure 3 caption and Appendix L were misleading and have been corrected. The run-tag caveat meant that a label alone does not distinguish repeated runs using the same model and verifier under different configurations; it did not indicate uncertainty about the model checkpoint. Similarly, the statement about historical tool versions should not have implied that the software environment was undocumented. All experimental data, results, model checkpoints, and associated configuration and environment records are available, and we will release them publicly.

The main RL evidence now comes from six Qwen3-32B non-thinking runs: three rewards, two seeds per reward, and 200 updates under a shared training recipe. This directly addresses the request for repeated seeds and stronger-reward controls. The RL configuration appendix specifies sampling parameters, optimizer settings, rollout counts, validation frequency, and reward definitions. The earlier 4B/8B runs remain separately identified supporting experiments with their own configurations.

All DeepSeek/L5 training and validation outputs receive the same offline formal evaluation, with a 600-second limit per check and common memory and concurrency settings specified in the appendix. The shared protocol fixes the checker code, configuration, and tool identities, in addition to the resource limits. Oracle-control curves use their online judgments, as stated explicitly. This makes the evaluation conditions and the interpretation of each comparison clear.

The cost concern in the original Appendix P is separate: the revised paper distinguishes checker construction, candidate checking, and formal evaluation. It reports CPU-based verification costs while recognizing that measurements across different hosts or concurrency settings do not establish a controlled end-to-end efficiency comparison. The new reward controls support conclusions about learning and reward reliability; they do not establish an equal-cost advantage.

Where does the golden reference come from, and why generate a new implementation?

Consider training on an existing trusted RAM and its specification. The model receives only the specification and proposes an implementation; the reference provides verification feedback. Repeating this over many known blocks teaches RTL generation. The purpose is to learn a generator, rather than to reconstruct one known block for its own sake. The new Oracle-reward runs provide a concrete example of learning under this reference-available setup.

The same reference can also define the behavior of alternative implementations, for example when exploring area or timing trade-offs. We measure functional reliability, not PPA improvements. The RAM witness illustrates the checking problem: a generated design can pass clock-edge tests yet violate required behavior between edges. Selection or training can then reward the wrong behavior despite having a trusted target available.

The practical lesson is to distinguish whether a proved solution exists, whether the checker accepts it, and whether selection recovers it. A test pass rate alone does not answer these questions.

How should these findings inform related methods and tool-using agents?

Our audit complements the methods highlighted by the reviewer. CodeV-R1 motivates validating reference-based verification used for data construction and RL. ChipSeek motivates auditing functional acceptance before interpreting downstream hardware-efficiency rewards. STG motivates extending evaluation to structured test generation rather than treating generic benches as representative of all verifiers.

For system builders, the supported recommendations are to measure selection losses against available proved candidates, audit rewarded training outputs independently, distinguish counterexamples from unresolved proofs, and report checking effort and coverage. The new reward controls strengthen this message: learning under a reward does not by itself establish the reliability of the outputs it accepts.

Independent outcome evaluation is also relevant to tool-using agents. Our repair experiments, however, use assigned feedback and restricted history; unrestricted tool choice, self-revised tests, and autonomous budget allocation are extensions, not measured results here.

Area_Chair_mQ6dOpenReview ↗

Follow-up on Author Rebuttal

Dear Reviewer,

A quick reminder as the discussion deadline is approaching: please review the authors' rebuttal and let us know whether your concerns have been addressed or if any major issues remain.

Your feedback would be appreciated before the discussion period closes.

Thank you!

Reviewer_xCbxOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper examines whether scalable RTL verification proxies stay reliable when LLM-generated hardware designs are optimized against them. Using 100 held-out CodeV-R1 tasks and 50,948 candidates from eight Qwen3 configurations (4B–32B, direct and reasoning modes), the authors compare syntax checking, four LLM-generated testbench conditions (Grok 4.6 or DeepSeek V4 Flash, each prompted from either the specification or the golden RTL), randomized differential simulation against the golden design (GoldDiff), and bounded model checking (BMC). Each proxy is scored against a hidden SymbiYosys formal-equivalence Oracle under three forms of optimization pressure: Best-of-N selection, ten-turn iterative repair, and two-epoch GRPO training with a DeepSeek testbench reward. Scale and reasoning enlarge the pool of correct designs, with Qwen3-32B reasoning reaching 82% Oracle Pass@64. Teacher-generated testbenches select a proved design on only 61–66% of tasks despite visible pass rates of up to 93%, while tuned BMC and GoldDiff recover 78% and 76%. Verifier effort (BMC depth and simulation length) strongly controls false positives, with diminishing returns beyond moderate effort. Under weak repair feedback, visible success keeps rising after hidden correctness stalls. In a Qwen3-4B RL run, the proxy–Oracle gap widens by 2.78 percentage points and testbench-passing outputs with formal counterexamples grow by 2.18 points, while the Oracle proof rate stays essentially flat. The paper concludes that while model scale and reasoning produce better pools of RTL candidates, the verifier decides how much of that capability is actually recovered. LLM-generated testbenches can report high visible success while selecting formally incorrect designs, whereas tuned BMC and differential simulation recover most of the Oracle ceiling, depending on verification effort. The same proxy–Oracle gap appears under iterative repair and testbench-reward RL, though without a clear drop in true correctness. The authors therefore recommend that hardware optimization work report visible reward, hidden correctness, selection regret, verifier effort, and Oracle coverage together.

Critical strengths
  1. A rigorous measurement setup that the field lacks. The paper keeps the optimized reward separate from a strictly stronger hidden formal Oracle. It is careful with metrics: fixed denominators, unresolved Oracle outcomes kept apart from real counterexamples, and paired bootstrap intervals. Most LLM-for-RTL papers report testbench pass rates as if they were correctness. This study shows quantitatively how misleading that can be, with a 61–66% hidden vs. up to 93% visible gap, which makes it a useful reference point for the area.
  2. Actionable results on verifier effort. The BMC-depth and simulation-length sweeps, reported alongside false-positive rates and CPU cost, show where reward reliability saturates. That gives practitioners concrete guidance for choosing verification budgets, instead of only the general warning that proxies can fail.
Critical weaknesses

The comparison isn’t quite fair, and it skips the most realistic case. GoldDiff and BMC check candidates against the same golden design the Oracle uses, so it’s not surprising they agree with it more than LLM testbenches do. And if you already have the golden design, you’d likely just use BMC or formal checking as the reward anyway. The case where LLM testbenches are actually needed, when no golden design exists, isn’t tested. It’s also unclear how many of the testbench “mistakes” are real bugs versus reasonable readings of a vague spec.

Questions for authors
  1. Among the designs that pass the LLM testbench but fail the Oracle, how many are actual bugs, and how many are reasonable readings of an ambiguous spec that just differ from the golden design’s timing? Could you manually check a sample? The RAM example could arguably be either.
  2. Since the Oracle proof rate doesn’t drop during RL, why frame the results as overoptimization rather than as an imperfect, increasingly permissive reward? What result would you consider evidence of true overoptimization?
  3. Do you have more than one training run (different seeds) for the 4B model? Without that, how confident are you that the widening gap isn’t noise, especially since the analysis windows were chosen after looking at the curves?
  4. GoldDiff, BMC and the Oracle all use the golden design. In settings where golden RTL is available, why would anyone use an LLM testbench as the reward instead of BMC? And what does your study tell us about the realistic case where no golden design exists?
4 comments

We thank the reviewer for highlighting specification ambiguity and the interpretation and robustness of the RL evidence. We have revised the main RL study and clarified what the observed disagreement establishes.

Are counterexamples actual bugs or reasonable interpretations of ambiguous specifications?

The RAM example violates an explicit requirement; we cannot yet give a representative ambiguity rate for the full dataset. A formal counterexample establishes disagreement with the reference under the equivalence model, which does not by itself resolve an ambiguous natural-language specification.

The example's specification defines an 8-bit × 256-word RAM, with inputs clk, wren, 8-bit address and data, and 8-bit output q. Writes occur on the rising clock edge when wren is high. Its read requirement states:

The read operation should be performed by assigning the value at the registered address to the output q.

The candidate instead reads the current address combinationally. Our witness initializes addresses 0x00 and 0xFF to 0xAB and 0xCD, disables writes, and clocks address 0x00. Both outputs are then 0xAB. Changing the address to 0xFF before the next clock edge leaves the reference output at 0xAB but changes the candidate to 0xCD. This contradicts the registered-address requirement without relying on simultaneous read/write semantics.

Section 6.1 now states this specification requirement alongside the example, and the RAM appendix gives the witness. This is one inspected coverage failure. A representative manual classification of other counterexamples remains necessary before estimating the contribution of specification ambiguity.

Why call this overoptimization if formal correctness does not decline?

We agree that increasing reward disagreement is a narrower conclusion than correctness degradation. The checker is frozen during training; its rules do not become more permissive. What changes is the policy's output distribution. The revised abstract, Section 5.5, and conclusion explicitly distinguish improved proof success from increased rewarded errors.

The new DeepSeek-reward runs show:

Updates 1–20 → 181–200Seed 0Seed 1
Reward-checker acceptance27.68% → 56.99%30.29% → 48.73%
Formal proof success20.92% → 40.90%22.50% → 33.93%
Increase in acceptance minus proof, pp [95% task-bootstrap interval]+9.34 [4.36, 14.20]+7.01 [2.16, 11.95]
Increase in rewarded counterexamples / all outputs, pp [95% task-bootstrap interval]+7.23 [3.17, 11.32]+6.25 [2.11, 10.41]

Each training window contains 5,120 outputs. Changes use unrounded rates. Learning and increasing rewarded errors coexist in both observed training seeds. On held-out checkpoints 5, 10, 15, 20 versus 185, 190, 195, 200, gap growth is +1.45 pp [−2.00, 4.80] for seed 0 and +4.35 pp [1.10, 7.90] for seed 1. The revised paper reports the inconclusive seed-0 result explicitly.

Stronger evidence of correctness degradation would require repeated reward improvement accompanied by declining independently measured correctness, after accounting for unresolved outcomes. Establishing a learned exploit would additionally require program and witness analysis. Our claim is reward disagreement under optimization, not an overall correctness collapse.

Do additional seeds and window checks support the RL finding?

We added a new repeated study and made it the main RL evidence. It uses Qwen3-32B non-thinking, three rewards, two seeds per reward, 200 updates, and 16 prompts × 16 rollouts per update. Validation uses the same 100 tasks with five outputs each, at checkpoint 0 and every five updates. The earlier 4B/8B runs remain in the appendix; they are not treated as seed replications.

The revised figures show the full training and validation trajectories. Early/late windows remain exploratory. As a check on changing task difficulty, we also compare epochs 1 and 4 using the 785 tasks present in both within each run, with equal task weights. DeepSeek gap growth remains +5.39 pp [3.77, 7.05] and +4.91 pp [3.29, 6.59] for seeds 0/1.

The intervals resample tasks 10,000 times, keeping each task's outputs together. They measure variation across benchmark tasks for these trained policies; two seeds do not establish how much the effect would vary across all future training runs. The revised statistics appendix states this distinction.

Is the shared golden reference an unfair advantage, and why not use formal checking directly?

Reference access is part of the comparison and must be explicit. GoldDiff, BMC, and the Oracle check against the same behavioral target. We compare complete pipelines' recovery of proved candidates, not architecture alone under identical information and compute.

Golden-prompted teacher controls partly address the information concern: they receive the reference during authoring, yet recover only 64–66% proved best-of-64 selection against the 82% available ceiling. Thus, access to the reference in the prompt does not eliminate the selection gap. It still differs from live reference access during checking and does not equalize construction budgets.

When a trusted reference and adequate resources are available, our results favor stronger checking. The new Oracle-reward training runs demonstrate that this can support held-out learning, while L5 offers a finite-simulation reward with smaller observed discrepancies than DeepSeek. These results are useful for training on known specification/reference pairs and generating equivalent implementations. They do not establish numerical rankings for greenfield tasks without a reference.

Area_Chair_mQ6dOpenReview ↗

Follow-up on Author Rebuttal

Dear Reviewer,

A quick reminder as the discussion deadline is approaching: please review the authors' rebuttal and let us know whether your concerns have been addressed or if any major issues remain.

Your feedback would be appreciated before the discussion period closes.

Thank you!

Reviewer_xCbxOpenReview ↗

Post-rebuttal assessment

Thank you for the additional experiments and careful clarifications. The repeated 32B RL study, the distinction between increasing rewarded errors and declining formal correctness, and the explicit reference-available scope address most of my concerns. The RAM witness also resolves my concern about that example.

A representative manual audit of specification ambiguity is still missing, so please keep clear that a formal counterexample establishes disagreement with the reference and does not necessarily establish a violation of an unambiguous natural-language specification. I am maintaining my rating of 3 (weak acceptance).

Reviewer_3xzHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a controlled auditing framework for hardware verification rewards in order to address proxy-reward overoptimization in using LLM for hardware design, where optimizing candidate circuits against synthetic testbenches, simulation, or bounded checks creates an illusion of progress while failing to guarantee true functional correctness. The key ideas include (1) separating scalable proxy verifiers from a hidden unbounded formal-equivalence oracle to systematically expose gaps between visible reward improvements and mathematical circuit equivalence, (2) stress-testing proxy-reward alignment under inference-time candidate selection, iterative multi-turn design repair, and policy-gradient reinforcement learning, and (3) evaluating how scaling checker effort through deeper bounded model checking and extended differential simulation directly suppresses false-positive selections and mitigates reward gaming.

Critical strengths

This a a timely study that exposes how synthetic proxy verifiers induce Goodhart effects in hardware synthesis, as that causes visible reward gains to diverge from true functional correctness.

The paper whows how scaling checker effort through deeper bounded checking and longer simulation suppresses false passes across candidate selection, repair, and training. This provides actionable, grounded principles for balancing compute allocation between circuit generation and formal verification in automated physical design.

Critical weaknesses

However, I also have a few concerns.

First, the study seems to mainly function as an observasinal audit of proxy failure. The key operational point is to increase checker effort through deeper bounded checking and extended simulation, which represents brute-force compute scaling rather than an algorithmic breakthrough in reward formulation. Without incorporating a resilient verifier architecture or adaptive reward mechanism that resists gaming under fixed compute budgets, the proposed framework can diagnose a known vulnerability without providing a path forward.

Second, the experimental methodolagy seems to depend on access to pre-existing golden reference designs to construct differential simulation harnesses and formal equivalence miter circuits. Inpractice, AI models are useful because no golden implementation exists, leaving designers with only incomplete specifications and high-level descriptions. By relying on complete golden circuits to detect proxy divergence, the diagnostic framework evaluated would not function in authentic greenfield electronic design automation workflows.

Third, the proposed reliance on scaling bounded model checking can impose computational limits on complex modern hardware architectures. In circuits with deep sequential state machines, wide arithmetic datapaths, or large internal memory arrays, increasing unrolling depth rapidly triggers exponetial state-space explosion and solver timeouts. Therefore, adopting deeper bounded checking as a universal antidote to proxy divergence can fail to account for the scalability barriers inherent to formal automated reasoning on realistic processor and accelerator blocks.

Fourth, the study only evaluates generic foundation models generating static script testbenches, while domain-specialized verification agents that synthesize robust algorithmic reference models were not considered. Without benchmarking against reasoning-trained verification agents like Pro-V-R1 and RefEvo, it is hard to determine whether proxy gaming is an intrinsic flaw of automated checking or merely an artifact of naive prompting.

Finally, the study only focuses on monolithic, prompt-generated Verilog testbenches, whereas industry-standard verification methodologies such as universal verification environments or formal temporal assertions were not considered. Modern hardware teams typically relies on constrained-random stimulus generation, functional coverage groups, and layered checking structures specifically engineered to prevent silent simulation escapes. By lmiting its audit to primitive script-level testbenches, the proposed framework may lead to vulnerabilities in basic prompting baselines while leaving open whether sophisticated, coverage-driven verification frameworks suffer from equivalent reward gaming.

Questions for authors
  1. How can the proposed audit methodology guide proxy selection in authentic greenfield design settings where no golden implementation exists to build differential or formal miter circuits?
  2. Beyond increasing brute-force checker compute, did the authors identify specific verifier structures or prompt strategies that inherently resist reward gaming under fixed budgets?
  3. How frequently did bounded model checking encounter solver timeouts or state-space explosion on complex sequential circuits when unrolled to the deeper depths required for fidelity?
  4. Do the observed reward gaming dynamics persist when evaluating structured universal verification environments and formal assertions, rather than monolithic script testbenches?
3 comments

We thank the reviewer for the questions about the contribution, reference access, scalability, and specialized verification agents. We address each point below, including an additional PRO-V-R1-8B experiment.

What is the contribution beyond increasing checker compute?

The contribution is a controlled measurement of how much available generation capability a verifier recovers, and how that changes under optimization. At best-of-64 on the same 100 tasks, a proved candidate exists for 82% of tasks. Teacher testbenches select one on only 61–66%, compared with 78% for BMC and 76% for GoldDiff. This identifies a measurable verification bottleneck even when the generator already produces a correct answer.

The effort sweeps show where additional checking improves recovery and where it saturates. The revised RL section extends this result to training: among 2,000 late held-out outputs per run, DeepSeek rewards 339 and 362 outputs with formal counterexamples in seeds 0/1 (16.95% and 18.10%). GoldDiff L5 rewards 23 and 27 (1.15% and 1.35%). Each denominator is 100 tasks × five outputs × four checkpoints (185, 190, 195, 200). These rates describe different learned policies, not equal-compute comparisons on identical candidates.

Section 5.5 now discusses these stronger-reward controls alongside the earlier selection and effort findings. The contribution is an empirical framework and measured guidance for verifier choice; we have not established an adaptive architecture or prompt strategy that inherently resists gaming at a fixed budget.

Does the method apply when no golden implementation exists?

Our quantitative evaluation concerns reference-available workflows, including training. The revised abstract and introduction state this explicitly. Three concrete uses are:

  • RL training: start with specification/reference pairs for existing blocks, such as RAMs or FIFOs. Give the model the specification and use the reference to evaluate its generated implementations. The model learns from many such tasks; it is not given the reference as its answer.
  • Synthetic-data filtering: sample several implementations of a known specification, verify them against the reference, and retain suitable examples for training. Our selection results measure how often a checker misses a proved candidate that is already in that pool.
  • Implementation transformation: generate another implementation of an existing block while preserving its behavior. Area or timing improvements can motivate this workflow, although our experiments measure functional reliability rather than PPA gains.

The paper also explains the information supplied to testbench authors. “Spec-prompted” means the teacher sees the specification and ports when writing the bench. We then run that bench on the golden implementation to check that it accepts the reference. For 15 Grok-Spec tasks, expected values or timing were corrected using the reference. Thus, specification-only prompting does not mean the complete checker-construction process is reference-free.

When no trusted reference or independent correctness criterion exists, this equivalence audit cannot be applied directly. Independent reward validation remains a useful principle, but the numerical rankings here do not establish greenfield error rates.

How does deeper BMC behave, and what are its scalability limits?

Deeper checking did not cause sequential timeouts in this benchmark, but its selection benefit saturated. Across tested sequential depths 1, 2, 4, 8, 16, and 32, the sequential subset has 0 timeouts among 3,392 checks at each depth. The complete sweep has 8/6,397 timeouts (0.125%) at each setting; those eight checks are combinational and remain at depth one. Unsupported outcomes are counted separately and are not successful checks.

For larger designs, formal equivalence can be decomposed into smaller partitions with separate proof strategies, as implemented in YosysHQ EQY. This provides a practical direction for applying the audit at block or partition level. Decomposition does not remove the need to establish that the partitions collectively verify the intended design. Our small-circuit experiments do not quantify industrial-scale performance or evaluate such a partitioned flow.

Do the findings extend beyond generic testbench prompting?

To address this question, we evaluated PRO-V-R1-8B's stimulus/reference components on the same 100 tasks and 6,397 candidates. The model generates Python stimuli and algorithmic reference models. We execute them to produce frozen tests, then score candidates against those tests and the paper's existing formal-equivalence labels. This additional experiment is reported here in the rebuttal.

We use two conditions. Spec-only references receive the specification/interface and are selected for execution and interface validity, without requiring golden agreement. Golden-assisted references additionally receive golden RTL and must pass a finite golden self-check. Both conditions use the same selected stimuli per task. Construction uses uniform interface/clock-reset corrections, at most four stimulus attempts and four reference proposals per condition, and at most 128 vectors or sequential cycles per task (32 cycles per scenario). Candidate-scoring outcomes do not guide construction or repairs.

Result at best-of-64Spec-onlyGolden-assisted
Tasks with an available checker86/10064/100
Tasks with a checker-accepted selection47/10063/100
Tasks with a proved selection26/10037/100
Task-bootstrap 95% interval for proved selection18–35%28–47%
Tasks with an accepted and proved selection14/10037/100
Tasks with an accepted selection and a formal counterexample29/10023/100

Missing checkers remain in the 100-task denominator. The paper's selector returns the earliest usable zero-score candidate if no candidate passes; 12 of Spec-only's 26 proofs are such rejected fallbacks, leaving 14 accepted-and-proved selections.

Coverage materially affects the comparison. On the same 64 tasks where both conditions have checkers, proved selection is 37/64 (57.8%) for golden-assisted PRO-V, 39/64 (60.9%) for Grok-Spec, and 43/64 (67.2%) for DeepSeek-Spec; BMC achieves 53/64 (82.8%). Thus, the assisted 8B verifier approaches Grok-Spec's point estimate on this shared subset. However, this is not evidence of parity: it uses golden assistance, omits tasks where its construction failed, and does not match authoring budgets. The PRO-V spec-only condition reaches 21/64 (32.8%) on that subset; full-set results remain 26%/37%, versus 61–66% for the teacher benches. The 8B result is informative, but this experiment does not isolate the effect of model size or specialized training.

The key reliability finding is that all 64 assisted checkers pass their golden self-check, yet 23 tasks yield accepted selections with formal counterexamples. Algorithmic references therefore also require independent evaluation of incorrect acceptance in this setup.

This experiment integrates the two released generation components with bounded construction and frozen candidate tests; it does not reproduce the full published Judge/majority-voting system. RefEvo was not evaluated. The result extends the audit beyond generic testbench generation without establishing that every specialized verification system has an intrinsic flaw.

What about UVM, coverage-driven verification, and temporal assertions?

Those comparisons remain open. The paper includes differential simulation and formal equivalence alongside generated testbenches, but does not evaluate UVM/coverage-driven environments or specification-derived temporal-property suites. The additional PRO-V experiment tests algorithmic-reference construction, not those separate approaches. Our conclusions concern the evaluated pipelines; equivalent failure rates for untested structured systems cannot be inferred from them.

Area_Chair_mQ6dOpenReview ↗

Follow-up on Author Rebuttal

Dear Reviewer,

A quick reminder as the discussion deadline is approaching: please review the authors' rebuttal and let us know whether your concerns have been addressed or if any major issues remain.

Your feedback would be appreciated before the discussion period closes.

Thank you!

Comments on the paper

1 comment

We thank the reviewers for identifying where stronger experiments and clearer scope were needed. The revised manuscript now makes six Qwen3-32B RL runs (three rewards, two seeds per reward, 200 updates per run) the main training evidence. Section 5.5 includes the repeated DeepSeek training result, a three-reward held-out comparison, and an explicit discussion of Oracle-reward learning and GoldDiff L5's closer agreement with formal outcomes. The earlier 4B/8B experiments are retained separately in the appendix.

The abstract, methods, results, and conclusion now make the central distinction explicit: formal correctness can improve while rewarded errors also increase. DeepSeek training shows this in both seeds; held-out disagreement growth is clear only in seed 1. The paper also states its reference-available scope. To address Reviewer 3xzH's specialized-agent question, we additionally report a PRO-V-R1-8B comparison in our response to Reviewer 3xzH.

“Proved” means formally equivalent to the reference under the checking model. A rewarded counterexample is an output accepted by the reward checker but rejected with a formal counterexample; unresolved proofs are separate. Changes in rates are reported in percentage points (pp).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.