- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Introduce LASSO to address the register-transfer-level hardware from NL specs while no trusted reference implementation is available during inference.
LASSO constructs and checks its own verification feedback before using any of it to refine a design
The test bench is a result coming from the combination of trace replay and agent-authored cocotb tests.
New thing: VCPO change credit assignment rather than interfere with inference workflow, author report that LASSO outperforms direct generation and ChipMATE across four benchmarks with the same DeepSeek V4 backbone.
The paper presents these methods as simulation-based techniques inspired by formal verification, not formal correctness guarantees. Its stated limitations include finite stimulus and mutation coverage, shared specification misunderstandings across reference models, and preliminary VCPO evidence limited to one 35-task (?) evaluation without ablations of individual contracts or contract-conditioned grouping.
Critical strengths
The math is internally sound theoretically. The strongest contribution of the paper would be a structured source of self-generated verification feedback and reports that this improves the resulting hardware design.
Critical weaknesses
The definition and math sometimes use words that make it feel like an overstatement. At the same time the reward (final label grading from Eq. 6 and Eq 7) seems to also allow credit for the wrong behavior. Both equations grade verdicts against the final design’s label. In the trajectory where a false rejection drives a repair that breaks a correct design (a failure mode Section 4.4 documents), both contracts credit the harmful verdict as correct. Since rollout-level labels cannot separate this from a true rejection, the reported gains cannot be attributed to improved verification. Same problem with Equation 4 where I do not understand why it drops E_x from the denominator.
Questions for authors
From the critical weakness, here is a couple of question (more like a series of question coming from the same point) i would like to ask:
-
Why drop E_x from the denominator? On face value, this made it much worse for the trace test since T_trace compares every output at every step X against R* and V matches R* there. Thus, T_trace will kill basically any distinguishable-on-X mutant and basically leaves the scored survivors empty which mean that m_all = 1 almost always and m_agent = |K_agent| / |K| instead. For example (with some make up number), with 50 mutants, X exercised only on 2 of them then m_agent = 2/2 = 1 and the gate passed even though 48 mutants, possibly real faulty ones, survived the suite and were indistinguishable under X per definition.
-
The first question leads to the second question: If dropping E_x was deliberate, then wouldn’t the explanation for table 5 result (falsely accept 20.5% of cid003 tasks) can also be interpreted as “the stimulus never reached the faulty behavior, and the gate passed anyway because unexercised mutants were excluded.” instead of the paper explanation that it is just a reference sharing the same misleading specs?
-
Is the intended interpretation therefore authored-test effectiveness among detected mutants, rather than fault-detection adequacy over the generated mutant pool? What mechanism prevents weak stimulus coverage from producing a perfect qualification score? Reporting the observed sizes of K, L, E_x would help establish whether this concern arises in practice.
Given there is no resp. so far I shall keep the original score...