acceptodds
Withdrawn from ICLR 2027

LASSO: Learning Agentic Self-Synthesized Oracles for Golden-Free RTL Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Agentic RTL generation succeeds when a trusted testbench supplies feedback, but a specification-to-silicon flow rarely starts with one: the agent must construct its own verifier, and a lenient verifier silently approves incorrect hardware. We argue that learning in this setting should follow the structure of hardware verification. We first introduce LASSO, a golden-free agentic flow whose stages are inspired by formal-verification practice: specification-linked conditions, reconciliation of independently generated reference models through minimized distinguishing traces, vacuity checks, mutation-based adequacy qualification, and counterexample triage that separates design faults from verifier faults. With the same DeepSeek V4 backbone, LASSO improves pass@1 over direct generation and over the golden-free ChipMATE flow on all four benchmarks we evaluate; with Qwen3.8-27B on VerilogEval-v2, it closes most of the gap to golden-testbench feedback. A final-outcome reward does not directly evaluate verifier quality: when the final design is faulty, a verifier that falsely accepts it receives the same credit as one that correctly rejects it. On held-out CVDP bug-fix tasks, outcome-reward training does not improve Qwen3.8-27B (62.9% vs. 65.7% pass@1). We therefore propose VCPO, in which each artifact of the flow is learned against its own verification contract—soundness and adequacy for the verifier, correct, evidence-backed verdicts for diagnosis, and counterexample elimination for the design—and compared only with alternatives for the same artifact on the same specification. On the same tasks, VCPO raises pass@1 to 71.4%, closing 40% of the gap to golden-testbench feedback, and cuts the false self-acceptance rate from 14.3% to 5.7%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LASSO lets RTL agents build their own verifiers without golden testbenches, and VCPO trains each artifact against its own verification contract rather than final-outcome reward.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VaBZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Introduce LASSO to address the register-transfer-level hardware from NL specs while no trusted reference implementation is available during inference.

LASSO constructs and checks its own verification feedback before using any of it to refine a design

The test bench is a result coming from the combination of trace replay and agent-authored cocotb tests.

New thing: VCPO change credit assignment rather than interfere with inference workflow, author report that LASSO outperforms direct generation and ChipMATE across four benchmarks with the same DeepSeek V4 backbone.

The paper presents these methods as simulation-based techniques inspired by formal verification, not formal correctness guarantees. Its stated limitations include finite stimulus and mutation coverage, shared specification misunderstandings across reference models, and preliminary VCPO evidence limited to one 35-task (?) evaluation without ablations of individual contracts or contract-conditioned grouping.

Critical strengths

The math is internally sound theoretically. The strongest contribution of the paper would be a structured source of self-generated verification feedback and reports that this improves the resulting hardware design.

Critical weaknesses

The definition and math sometimes use words that make it feel like an overstatement. At the same time the reward (final label grading from Eq. 6 and Eq 7) seems to also allow credit for the wrong behavior. Both equations grade verdicts against the final design’s label. In the trajectory where a false rejection drives a repair that breaks a correct design (a failure mode Section 4.4 documents), both contracts credit the harmful verdict as correct. Since rollout-level labels cannot separate this from a true rejection, the reported gains cannot be attributed to improved verification. Same problem with Equation 4 where I do not understand why it drops E_x from the denominator.

Questions for authors

From the critical weakness, here is a couple of question (more like a series of question coming from the same point) i would like to ask:

  1. Why drop E_x from the denominator? On face value, this made it much worse for the trace test since T_trace compares every output at every step X against R* and V matches R* there. Thus, T_trace will kill basically any distinguishable-on-X mutant and basically leaves the scored survivors empty which mean that m_all = 1 almost always and m_agent = |K_agent| / |K| instead. For example (with some make up number), with 50 mutants, X exercised only on 2 of them then m_agent = 2/2 = 1 and the gate passed even though 48 mutants, possibly real faulty ones, survived the suite and were indistinguishable under X per definition.

  2. The first question leads to the second question: If dropping E_x was deliberate, then wouldn’t the explanation for table 5 result (falsely accept 20.5% of cid003 tasks) can also be interpreted as “the stimulus never reached the faulty behavior, and the gate passed anyway because unexercised mutants were excluded.” instead of the paper explanation that it is just a reference sharing the same misleading specs?

  3. Is the intended interpretation therefore authored-test effectiveness among detected mutants, rather than fault-detection adequacy over the generated mutant pool? What mechanism prevents weak stimulus coverage from producing a perfect qualification score? Reporting the observed sizes of K, L, E_x would help establish whether this concern arises in practice.

Reviewer_NvuJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LASSO, a golden-free agentic framework for RTL generation that constructs and validates its own testbenches using reference reconciliation, mutation testing, and counterexample analysis. It further introduces Verification-Contract Policy Optimization (VCPO), which assigns stage-specific rewards to improve verifier reliability and RTL correctness. Experiments demonstrate improvements over existing golden-free approaches, with VCPO increasing held-out pass@1 from 65.7% to 71.4% and reducing false acceptance.

Critical strengths

The paper addresses the important challenge of reliable RTL generation without trusted testbenches. Integrating verification-inspired techniques into an agentic workflow is practically meaningful and produces consistent improvements across four benchmarks.

Critical weaknesses

VCPO is evaluated on only 35 held-out tasks, where the improvement from 65.7% to 71.4% corresponds to just two additional successful tasks. Furthermore, the individual reward contracts and contract-conditioned grouping are not ablated, making it difficult to establish which components drive the improvement.

The framework relies on simulation-based checks and mutants derived from potentially incorrect reference models. Shared specification misunderstandings may remain undetected, as reflected by the 20.5% false-acceptance rate on CVDP cid003 even after qualification. This limits the reliability claims of the proposed approach.

Questions for authors
  1. Can the authors provide ablation studies isolating the effects of individual verification contracts and contract-conditioned group-relative optimisation? How do these compare against simpler stage-specific reward formulations?

  2. Can the authors evaluate VCPO across additional held-out benchmarks and multiple training seeds to establish whether the observed improvements are statistically reliable?

  3. How does LASSO address systematic specification misunderstandings shared by independently generated reference models? Can the authors quantify how frequently these errors survive reference reconciliation and mutation qualification?

1 comment

Given there is no resp. so far I shall keep the original score...

Reviewer_BWcvOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces an agentic workflow for RTL generation without access to a trusted verifier which they call LASSO. The method is inspired by formal verification practice. Lasso is demonstrated both with a fixed underlying model, and with a fine-tuned model, which is trained with a reward signal derived from a golden evaluator using a process they call VCPO. Empirical performance is strong, with LASSO outperforming the equivalent LLM, and a competing golden-free method across four benchmarks. With VCPO, empirical performance is improved further, and VCPO outperforms a naive outcome-only training reward.

Critical strengths
  1. Strong empirical performance for LASSO compared to other methods across four benchmarks, and compared to the golden test bench reference.
  2. Although limited in scope, empirical performance of VCPO is also encouraging.
  3. The ablations are useful and well discussed.
Critical weaknesses
  1. With VCPO, the complexity of the reward formulas (and the associated high number of hyperparameters) doesn't seem adequately justified, either in the text or in the experimental results. For example, there are no ablations of the individual contract rewards. The authors acknowledge this limitation in section 5, but given the weight placed on VCPO throughout the rest of the paper I think it requires further empirical analysis. And related to question 1, how would a much simpler reward impact the performance of VCPO?
  2. Overall I found the paper quite difficult to follow, particularly around VCPO. Figure 1 has a lot of potential to provide a clear 'roadmap' through the paper, but isn't referred back to sufficiently, or with consistent language in section 3. It would be really helpful to include 'pointers' to the component(s) that VCPO interacts with in Figure 1. (e.g. the connections between the LASSO workflow, the verification contract rewards, and the associated policy updates).
  3. It is unclear to me which experimental results use VCPO. In particular do sections 4.4, 4.5, and 4.6 use VCPO? I think it's only in section 4.3, but it isn't entirely clear to me.
Questions for authors
  1. I don't follow the claim that 'when the design fails, a verifier that falsely accepts it and one that correctly rejected it receive the same zero'. Isn't that just a design choice, not a fundamental truth? E.g. you could use only the equation 6 part of VCPO to better assign credit to the verifier?
  2. Which results actually use VCPO? Other than section 4.3/table 2, is VCPO used anywhere else?
  3. Why is Deepseek V4 used for the results in Section 4.2, but Qwen3.8-27B used elsewhere?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.