- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Since cycle-by-cycle matching against a reference implementation rejects valid designs with task-permitted latency differences, this paper proposes BEHAVE, an agentic framework that lets an LLM agent write both RTL and behavior model B to help validate the correctness of RTL. Both artifacts are compared with a timing-free golden behavior model B⋆ to test the effectiveness of the agent. The current policy can act as an Analyst to identify capability gaps, which guide the construction of new training pairs for RL.
Critical strengths
The paper is clearly written and experiments are extensive. In the experiments, Figure 5(c) partially addresses one of my concerns, namely whether having the LLM agent write a behavior model B actually helps: it shows that doing so improves the pass rate to some extent(from 63.3% to 68.3%).
Critical weaknesses
I think this paper has several main weaknesses:
-
In Sec 3.4, the authors write "Builder constructs S and B1 from the sources and objectives... Implementer produces B2 from S", I think here builder and Implementer should be different models because S is natural-language text, which can be vague. Maybe the same model has a specific understanding of a vague text S.
-
The framework lets an LLM agent write both RTL and behavior model B, but the model B does not seem to help much. It would be enough to compare RTL with the timing-free golden behavior model B⋆ . According to Figure 5(c) and (d), writing RTL alone already performs reasonably well: its pass rate is slightly lower (63.3% vs. 68.3%, a difference of just 3 out of 60 tasks), while its cost is much lower.
-
I feel that this framework is primarily engineering-oriented rather than research-oriented, and its novelty is somewhat limited.
Questions for authors
In Sec 3.4, the paper states that the Analyst “then selects relevant high-level implementations and sets objectives for new tasks.” I would like to know the sources of these new materials and the criteria used to select from them.
3 comments
Response to Reviewer TPg5 - Part 2
W3: Research contribution
Our work asks: How can we scale up training for hardware design and verification agents? This requires reliable correctness feedback and access to more training tasks.
Correctness feedback. The first challenge is obtaining correctness feedback without forcing agents to reproduce one reference design's timing. Behavior IR describes the intended computation, while BEHAVE-Sim checks correctness under the specification's requirements, leaving agents free to explore valid latency and PPA choices. Our cycle-by-cycle comparison found 138 valid implementations that strict reference-cycle matching would reject, representing 41.8% of the 330 implementations reported as passing. (Section 4.2.)
Expanding the training data. The second challenge is the scarcity of RTL training data. Our approach provides training rewards from specification-Behavior pairs without requiring reference RTL. This makes widely available high-level implementations usable as sources of training tasks. We show that the current policy can search these sources, construct and check new training pairs, and continue training on them.
Resources for future work. We also release BEHAVE-Train and BEHAVE-Eval, containing 600 human-reviewed specification-Behavior pairs, as reusable resources for training and evaluating hardware agents on realistic workloads. (Abstract and Section 1.)
Q1: Material sources and selection criteria
Source library. The library contains 374 files from 67 repositories. We started from the source materials used to build BEHAVE-Train, excluded BEHAVE-Eval source files and materials associated with the 60 initial tasks and their variants, and removed duplicate content. We also audited the library and the 100 tasks acquired in the original SI experiment against BEHAVE-Eval using annotated functional operators, input/output semantics, numerical rules, and state behavior. The semantic comparison identified 273 pairs for detailed review, in which we found no equivalent tasks. Repository sharing is permitted because different components of a large library need not implement the same computation. (Appendix C.6.)
Selection criteria. Analyst uses training feedback to select materials that address capability gaps, add new learning objectives, and provide enough detail to define a hardware task. For example, repeated non-submissions on top-k attention motivated smaller selection and fixed-point division tasks. The resulting tasks undergo reconstruction and source-fidelity checks before admission. (Section 4.5 and Appendix C.6.)
Thank you again for your feedback. We would be happy to answer any further questions.
Follow-up on Our Rebuttal
Dear Reviewer TPg5,
Thank you again for your time. When you have a chance, we would appreciate your feedback on our rebuttal and additional results. We would be happy to clarify any remaining concerns.
Submission 8155 Authors
Response to Reviewer TPg5 - Part 1
Dear Reviewer TPg5,
Thank you for your careful review and constructive comments. We address your concerns about task construction, joint development, and our research contribution below. Revisions are highlighted in blue in the manuscript, with relevant locations noted in parentheses below.
W1: Same-model task construction
Self-improvement setting. Using the same model is deliberate: we study whether the current policy can construct useful training data without a second teacher model. Builder produces the specification and B1 from source material, while Implementer reconstructs B2 in a separate context, seeing only the specification and public interface documentation. Disagreements trigger repairs, and Reviewer also checks fidelity to the source. These checks help expose ambiguity, although separate contexts do not eliminate shared interpretation errors.
Training versus benchmarking. For SI, we use automated checks to control the quality of specification-Behavior pairs without requiring human review of every pair. We assess the usefulness of this supervision through held-out performance. For building evaluation datasets, we agree that cross-checking with different models is valuable. Combined with source checks and human review, it can help identify shared mistakes and reduce the manual review burden. The framework permits this choice outside our same-model SI experiment.
Downstream results. We compare Full-data RL sampling from the full 540-task BEHAVE-Train pool, Initial-task RL on the same 60 initial tasks without acquisition, and SI starting from those 60 tasks. Each setting uses three training seeds. Each SI run independently acquires tasks using its own training feedback. Extending SI from 18 to 24 updates includes two additional acquisition rounds, adding 40 tasks per run. The table reports RTL / Behavior accuracy (%), averaging three evaluation runs per trained model and then reporting mean ± sample standard deviation across training runs.
SI reaches 70.7% and 71.7% RTL accuracy on BEHAVE-Eval after 18 and 24 updates, compared with 65.0% and 63.3% for Initial-task RL. These results support the usefulness of the acquired training supervision without introducing another teacher model. (Section 4.5 and Figure 7(e).)
W2: Role and cost of the agent-generated Behavior model
RTL-only does not mean verification-free. This setting requires only an RTL submission, but retains the development tools for compilation, simulation, and agent-written tests. GPT-5.6 Luna ran self-written RTL simulations in 42/60 RTL-only attempts on BEHAVE-Eval. Thus, the baseline already includes self-testing in most attempts. Joint development additionally requires an explicit, reusable functional model that exposes the agent's assumptions and can be checked independently of the RTL with BEHAVE-Sim. (Section 4.3.)
A reference for developers. We agree that joint development adds cost. In Figure 5, adding BEHAVE-Sim reduces this overhead to about 18% over RTL-only, while further improving RTL accuracy. The generated Behavior provides an explicit functional reference for development-time checks, when the golden model is hidden. Final RTL evaluation can compare directly against the golden model without requiring the agent's own Behavior. In practical development, however, a trusted golden model may not yet exist. Producing both RTL and Behavior makes the agent's functional interpretation explicit: a developer can inspect, test, and revise the Behavior model, then approve it as a reference for subsequent RTL development. This provides a human-reviewable interface independent of the RTL's microarchitecture. (Section 4.6.)