- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MCTS-SFT, a reasoning-trace curation framework for Verilog code generation. Motivated by the limited control over intermediate design decisions and their supervisory representation in full-response CoT distillation, the framework uses functional verification results from generated code to guide search over intermediate states. Its Plan-MCTS variant organizes interface, timing, state, and behavioral decisions into structured hardware plans and examines the effect of retaining these plans in supervised fine-tuning targets. With search trajectories and final Verilog implementations held fixed, explicit-plan supervision increases pass@10 from 52.56% to 57.70%. The 6k-trace study uses rollout-only targets and reports accuracy improvements across several VerilogEval tasks while approximately halving the average response length on VerilogEval-v2. On 172 multi-module CVDP tasks, Plan-MCTS and the CoT baseline achieve success rates of 12.21% and 9.30%, respectively, providing preliminary evidence of cross-benchmark transfer, although absolute success rates remain low.
Critical strengths
-
The paper provides a task-specific approach to reasoning-data curation for Verilog generation by incorporating hardware design decisions into both the search state and the supervision target. Structured plans explicitly represent interfaces, timing, state, and update rules, allowing intermediate decisions to be progressively refined and evaluated through functional verification of the generated code. Using executable feedback for trajectory selection without training an additional process reward model offers a practical approach to constructing supervision for hardware design reasoning.
-
The experiments examine how the representation of structured hardware plans affects supervised fine-tuning of Qwen2.5-Coder-7B-Instruct for Verilog generation. With search trajectories, final implementations, and verifier rewards held fixed, explicit structured plans outperform both plan omission and natural-language plan rendering, indicating that the organization of hardware design information affects its value as supervision. Comparisons with one-shot planning, random tree exploration, and reasoning compression further examine the main design choices. Reporting functional accuracy alongside response length also characterizes generation quality and output efficiency.
Critical weaknesses
-
The baseline comparisons do not adequately establish the method’s advantages over existing training approaches. The experiments primarily compare CoT, planning, and search variants constructed within this study, without directly evaluating existing Verilog generation models. This leaves the method’s relative performance unclear. Moreover, the answer-only baseline is derived from conventional CoT data rather than the same Plan-MCTS samples. The absence of a matched code-only comparison leaves the contributions of the search-generated Verilog implementations and intermediate reasoning supervision insufficiently separated.
-
The performance–cost trade-off of MCTS relative to one-shot planning remains unclear. Plan-CoT achieves 57.05% pass@10, close to the 57.70% obtained by Plan-MCTS with explicit-plan supervision, and attains a higher pass@1. The paper does not report variability in these small differences across training runs or quantify the actual number of model calls, cumulative token usage, and data-curation time. Consequently, neither the stability of the search gains nor their cost-effectiveness relative to one-shot planning is established.
-
Explicit-plan supervision lacks further validation across tasks and training datasets. Its representation-level comparisons are limited to the 1k-trace study on VerilogEval-v1-Human, whereas the broader 6k-trace evaluation uses rollout-only targets. The specifications in the main studies come from VeriThoughts, and the scaling experiment in the appendix does not provide a comparison using an independent training dataset. Evaluations across VerilogEval tasks and CVDP provide cross-benchmark evidence, but do not establish whether the benefits of explicit-plan supervision depend on the task or training-data source.
-
The method’s effectiveness across student models has not been evaluated. The main experiments use Qwen2.5-Coder-7B-Instruct throughout, without comparisons across model families or parameter scales. The experiment using GPT-5.4-mini as the teacher retains the same student and lacks CoT and Plan-CoT baselines under that teacher. The current results therefore do not establish consistent improvements across base models or characterize the dependence of the gains on the teacher configuration.
Questions for authors
-
Could you compare the method with representative existing Verilog generation models under a consistent evaluation protocol and document differences in training data, model size, and inference budget? For comparisons between training pipelines, using the same student model and comparable training budgets where feasible would help establish the gains over existing approaches.
-
Could you construct code-only targets from the same Plan-MCTS samples and compare them with rollout-only and explicit-plan supervision? How does a plan-and-code target perform when the subsequent free-form reasoning is omitted? These comparisons would help separate the contributions of Verilog implementation quality, plan supervision, and reasoning supervision.
-
Could you report the actual data-curation costs of MCTS-SFT and compare them with CoT and Plan-CoT? The number of model calls, cumulative input and output tokens, and wall-clock time would be informative. Please also clarify which resources are matched by the stated generation budget and report variability across random initializations to characterize the stability of the search gains and their computational cost.
-
What motivated the omission of explicit plans from the 6k-trace training targets? Could you provide matched comparisons between explicit-plan and rollout-only supervision on the existing VerilogEval-v2 tasks to examine the effectiveness of plan supervision across tasks and at a larger data scale?
-
Could you reproduce the main comparisons on an independent training dataset with a different specification distribution or design complexity, using the same student and comparable training budgets? This would help assess whether the gains depend on the characteristics of VeriThoughts and distinguish transfer across evaluation benchmarks from applicability across training-data sources.
-
Could you evaluate the relative gains of Plan-MCTS, Plan-CoT, and conventional CoT training on students from different model families or parameter scales? Non-search baselines under the second teacher would also help characterize how teacher configurations and student models affect the results.
2 comments
Rebuttal - Reviewer n6cv (Part 2)
3. Curation cost and variability
We matched 1,000 retained traces per method and the per-specification quotas. The table includes all ten logged attempts per method on each of the 142 shared specifications, including attempts discarded before SFT.
| Method | Logged teacher activity | Input / output tokens | Yosys starts / testbench returns | Median min/spec |
|---|---|---|---|---|
| CoT | 1,417 completion replies | 0.647M / 2.384M | 1,185 / 1,140 | 8.47 |
| Plan-CoT | 1,420 planner starts, 1,122 rollout replies | 1.386M / 3.341M | 1,063 / 1,004 | 20.08 |
| Plan-MCTS | 4,232 candidate replies, 1,416 rollout replies | 4.525M / 11.054M | 1,346 / 1,255 | 26.03 |
Reply counts include responses with token records. Plan-CoT token totals cover accepted plans and their rollouts. Testbench returns come from the compile-and-simulate procedure, with simulation conditional on successful compilation. Exact retry and simulator-launch counts are unavailable.
We refit Plan-CoT and Plan-MCTS at SFT seeds 42 and 43 using each method's fixed 1,000-example dataset, a common split and LoRA recipe, and targets containing a structured plan, rollout reasoning, and RTL. On VerilogEval-v1 Human at 10 samples per task, Plan-CoT reached 55.13% pass@10 at both seeds, and Plan-MCTS reached 51.92% and 54.49%. The Plan-MCTS minus Plan-CoT gaps were −3.21 and −0.64 percentage points, with paired 95% task intervals of [−8.33, +2.56] and [−5.13, +4.49], respectively. The intervals resample tasks for fixed checkpoint pairs. The two fits per method probe training variation while holding curation fixed.
4. Explicit-plan supervision at 6k
The original targets came from the DeepSeek-R1 rollouts, and the DeepSeek-V3 plans were recorded separately in the search traces. We prepended the selected plan to each target, keeping the same 6,000 trajectories from 600 specifications and holding prompts, reasoning, and RTL fixed. Each VerilogEval-v2 evaluation used 20 samples per task across 156 tasks.
| VerilogEval-v2 | SFT target | Pass@1 | Pass@5 | Pass@10 | Pass@20 |
|---|---|---|---|---|---|
| Code Completion | Rollout only | 44.10% | 59.00% | 63.57% | 67.31% |
| Code Completion | Explicit plan | 45.16% | 58.16% | 62.03% | 64.10% |
| Spec-to-RTL | Rollout only | 41.73% | 58.16% | 63.20% | 66.67% |
| Spec-to-RTL | Explicit plan | 41.96% | 57.33% | 62.78% | 68.59% |
Explicit plans raised pass@1 by 1.06 percentage points on Code Completion and 0.23 on Spec-to-RTL. Rollout-only led at pass@5 and pass@10 on both tasks. At pass@20, rollout-only led by 3.21 points on Code Completion and explicit plans led by 1.92 points on Spec-to-RTL. The comparison therefore shows a trade-off across metrics. Paired task intervals for the 6k differences are pending.
6. Alternate student and second-teacher controls
We evaluated Llama 3.1 8B on VerilogEval-v1 Human (156 tasks, 10 samples each). The three reasoning-based models use 1,000 examples per method. The RTL-only model uses the 956 Plan-MCTS examples with available terminal RTL.
| Llama SFT arm | Rows | Pass@1 | Pass@5 | Pass@10 |
|---|---|---|---|---|
| CoT | 1,000 | 25.51% | 42.88% | 46.79% |
| Plan-CoT rollout | 1,000 | 23.97% | 40.76% | 48.08% |
| Plan-MCTS rollout | 1,000 | 24.04% | 44.11% | 50.64% |
| Plan-MCTS RTL-only | 956 | 22.50% | 37.54% | 41.67% |
CoT led at pass@1. Plan-MCTS led at pass@5 and pass@10, by 3.85 points over CoT and 2.56 over Plan-CoT at pass@10. Isolating the effect of RTL-only supervision requires training a rollout model on the same 956 examples.
With GPT-5.4-mini as teacher, we matched 5,390 sample IDs across 600 specifications for CoT, Thought-MCTS, and Plan-MCTS. All three Qwen students trained on terminal RTL only. We evaluated with 10 samples per task on VerilogEval-v1 Human.
| Teacher strategy | Pass@1 | Pass@5 | Pass@10 |
|---|---|---|---|
| CoT | 37.88% | 48.15% | 50.64% |
| Thought-MCTS | 36.35% | 46.34% | 48.08% |
| Plan-MCTS | 37.44% | 48.84% | 51.92% |
Plan-MCTS led at pass@5 and pass@10, including a 1.28-point lead over CoT at pass@10. CoT led at pass@1 by 0.44 points.
Conclusion
The student trained on MCTS-curated data led the student trained on a CodeV-R1 subset by 12.01 points at pass@10. With identical RTL, rollout reasoning gave the largest observed pass@10 gain. We thank the reviewer again for proposing controls that helped sharpen these conclusions.
Rebuttal - Reviewer n6cv (Part 1)
We thank the reviewer for the constructive suggestions on baselines, supervision, and generalization. We seek to address each question above through testing what the plan and rollout reasoning contribute, alongside comparisons across students and teachers.
Questions and answers
A1. With the same Qwen student and roughly 11,000 SFT examples per model, the student trained on MCTS-curated traces scored 56.66% at pass@10 versus 44.65% for the student trained on a CodeV-R1 subset.
A2. Adding rollout reasoning to the same RTL raised pass@10 by 5.13 percentage points, and including the plan alongside that reasoning shortened outputs by 604 tokens on average.
A3. Plan-MCTS recorded 15.58M teacher tokens versus 3.03M for CoT and 4.73M for Plan-CoT, and both paired task intervals in the two-seed comparison include zero.
A4. Explicit plans raised pass@1 by 1.06 percentage points on Code Completion and 0.23 on Spec-to-RTL, while rollout-only led at pass@5 and pass@10.
A6. Plan-MCTS led at pass@10 in both the Llama student and GPT-5.4-mini teacher comparisons, while CoT led at pass@1 in both.
Detailed responses
1. Comparison with existing Verilog models
We fine-tuned Qwen2.5-7B-Instruct on roughly 11,000 examples per model. One dataset contained MCTS-curated traces from VeriThoughts specifications. The other contained a subset of CodeV-R1 training examples. We evaluated both on VerilogEval-v1 Human (156 tasks, 20 samples each).
The MCTS-trained student led at all three k values. Its 12.01-percentage-point lead at pass@10 has a paired 95% task interval of [6.04, 17.99]. This comparison holds the base model and approximate SFT dataset size fixed. The optimizer step counts differ as shown, and the exact decoding limits for those evaluations are unverified.
Published VeriThoughts-7B scores 37.2% at pass@1 and 56.8% at pass@10 with a 1,024-token output cap. Our 11k model leads its pass@1 by 0.94 points and is within 0.14 points at pass@10. Our 15k model leads at all three k values, reaching 41.73%, 54.97%, and 59.60% at pass@1, pass@5, and pass@10. Full CodeV-R1 reports 81.7% pass@10 with 87,000 SFT examples, reinforcement learning, and a 16,384-token context. VeriThoughts · CodeV-R1
2. Isolating RTL, plans, and rollout reasoning
Of the 1,000 Plan-MCTS trajectories in the controlled study, 956 had terminal RTL available for all four target formats. We trained four Qwen2.5-7B-Instruct models on those same 956 examples, holding the prompts and final RTL fixed and adding the selected plan, rollout reasoning, both, or neither. Each model generated 10 samples per task on the 156 VerilogEval-v1 Human tasks.
Rollout reasoning gave the largest observed pass@10 gain over RTL alone, +5.13 percentage points (paired 95% task interval [−0.64, +10.90]). Adding just the plan to RTL gave +1.28 points [−4.49, +6.43]. Adding the plan to the rollout reduced mean response length from 3,523 to 2,919 tokens, with pass@1 up 1.47 points and pass@10 down 0.64 points [−6.41, +5.13]. All three pass@10 intervals include zero.