acceptodds
Under review as a conference paper at ICLR 2027

RTL-INTERACT: Training EDA-Interactive Models for Multi-Task RTL Design

Abstract

Large language models offer a promising route to register-transfer-level (RTL) design, but reliable implementations require checking and revision based on electronic design automation (EDA) tools' feedback. Existing RTL-specialized models often lack learned interaction with such EDA tools or rely on separate generation and repair components. We introduce \method, a multi-task post-training framework that trains a single RTL-specialized model to generate RTL, construct task-appropriate EDA checks, and revise implementations using tool feedback. The framework combines automatically constructed multi-task demonstrations with staged supervision from chain-of-thought to tool-integrated reasoning. We further introduce Pattern-Aware Policy Optimization (PAPO) for multi-task reinforcement learning. PAPO augments Group Relative Policy Optimization with correctness-gated rewards that favor concise successful checking while rewarding effective revision, and uses task grouping to preserve coverage across objectives. On CVDP, RTL-Interact-32B achieves 48.7% weighted Pass@1 and 68.9% Pass@5, surpassing its teacher model, DeepSeek-V3.2, while RTL-Interact-8B achieves 44.9% weighted Pass@1. Additionally, we show that PAPO can reduce average tool calls by 63.8% while improving Pass@1. Both models also outperform all RTL-specialized baselines on VerilogEval v2 and RTLLM v2, demonstrating the effectiveness of our method.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Code Generation, Verilog

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QasBOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents RTL-Interact, a post-training framework that trains a single model to generate RTL and to call EDA tools (simulation, lint, synthesis, and simulation-based equivalence checking) to check and revise its own designs. Starting from the CodeV-R1 dataset, DeepSeek-V3.2 is used to filter spec–RTL pairs and generate about 13.5K tool-use trajectories across three tasks: generation, completion of masked logic, and lint/synthesis optimization. The model is trained first with chain-of-thought SFT and then with tool-use SFT; a matched-budget study supports this ordering. It is then trained with RL using PAPO, a GRPO variant with two additions. The first is a correctness-gated reward that depends on the model’s self-checking pattern: 1.5 if the first self-check passes, 1.2 if it passes after revision, 0.8 without a successful self-check, and 0 if the final design is wrong. The second is fixed task proportions in each batch.

On CVDP’s 302 non-agentic problems, RTL-Interact-32B reaches 48.7% Pass@1 and 68.9% Pass@5, slightly above its tool-enabled teacher, and the 8B model reaches 44.9%. Both models are reported to beat all RTL-specialized baselines on CVDP, VerilogEval v2, and RTLLM v2. RL also reduces tool calls by 63.8% for the 8B model.

Critical strengths
  1. A practical unified model with useful controlled studies. One model handles several RTL tasks and uses EDA feedback itself, with no separate repair model or agent pipeline, which makes it easy to deploy. The design studies are well controlled. The SFT ordering comparison keeps the token budget fixed and shows the order matters for keeping tool use (TIR→CoT gives 0% tool use). The PAPO ablations keep data, initialization, and training steps fixed. The separate generation-vs-revision test (Table 5) shows that RL mainly improves revision from feedback. These experiments explain where the gains come from better than most papers in this area.

  2. Strong results against the teacher, with an efficiency gain. The 32B model slightly exceeds its tool-enabled teacher on CVDP, and RL improves accuracy while cutting tool calls and response length. That combination is useful in practice.

Critical weaknesses
  1. The contribution of each part of the PAPO reward isn’t shown. The reward has three separate terms: a bonus for passing the first self-check (+0.5), a smaller bonus for passing after revision (+0.2), and a penalty for correct answers without a successful self-check (−0.2). The only reward ablation removes all three at once (binary reward), and the authors note it “tests the whole reward design, not each rule separately.” So we don’t know which term matters. Does the gain come from rewarding efficient checking, from rewarding successful revision, or simply from penalizing unchecked answers? The reward values are also fixed without any sensitivity study. Since PAPO is the paper’s main methodological contribution, per-term ablations are needed to support it. There is also no comparison against a simple tool-call or length penalty, which would show whether the “pattern-aware” design adds anything beyond encouraging shorter trajectories.

  2. It’s unclear why PAPO helps when binary-reward RL barely does. Binary-reward RL improves the 8B model only from 39.5% (SFT) to 39.7%, while PAPO reaches 44.9%. This is a large and unusual gap, and the paper doesn’t explain it. One possible factor is truncation: the SFT model hits the length limit on 33.7% of problems, PAPO reduces this to 15.2%, and every truncated answer counts as a failure. If so, much of PAPO’s gain may come from shortening responses rather than from better RTL design or checking. A related concern is that the largest bonus goes to trajectories whose self-written testbench passes on the first try, so the model could earn it by writing weaker tests. The paper doesn’t check whether self-test quality changes after RL, even though tool calls drop by 64% while thinking length grows 2.8×.

Questions for authors
  1. Can you ablate each reward term separately, i.e., only the first-pass bonus, only the revision bonus, and only the unchecked penalty? Which term drives the improvement?
  2. How sensitive are the results to the reward values (+0.5 / +0.2 / −0.2)?
  3. Why does binary-reward RL barely improve over SFT? Was it tuned as carefully as PAPO? How much of PAPO’s gain remains when accuracy is computed only on non-truncated responses?
  4. How does PAPO compare against binary reward plus a simple tool-call or length penalty?
  5. Does the quality of the model’s self-written testbenches change after RL? Is there evidence that the first-pass bonus encourages weaker tests?
6 comments

We thank Reviewer QasB for thoughtful questions on reward design, binary RL, and self-test quality.

  • W1: The contribution of each part of the PAPO reward isn't shown. The reward has three separate terms: a bonus for passing the first self-check (+0.5), a smaller bonus for passing after revision (+0.2), and a penalty for correct answers without a successful self-check (-0.2). The only reward ablation removes all three at once (binary reward), and the authors note it "tests the whole reward design, not each rule separately." So we don't know which term matters.
  • Q1: Can you ablate each reward term separately, i.e., only the first-pass bonus, only the revision bonus, and only the unchecked penalty? Which term drives the improvement?

The three terms do not act as independent additive bonuses; instead, they establish a strict reward ordering across checking patterns to shape the model's verification behavior. Among correct trajectories, PAPO distinguishes three patterns: Immediate Self-check Resolution (ISR: passing the first checking round, reward 1.5), Feedback-driven Refinement Success (FRS: refining across multiple checking rounds until passing, reward 1.2), and Unsuccessful Self-checking (US: skipping checks or ending without a passing check, reward 0.8), while incorrect RTL always receives 0. Together, these terms enforce the strict ordering R_ISR (1.5) > R_FRS (1.2) > R_US (0.8) > 0, which relies on two essential boundaries:

  • Enforcing R_FRS > R_US (1.2 vs. 0.8) incentivizes the model to repair failed checks rather than submit unchecked or broken code. Zeroing both collapses R_FRS = R_US = 1.0, removing the training signal to keep fixing after the first checking round fails.

  • Enforcing R_ISR > R_FRS (1.5 vs. 1.2) rewards deeper initial planning and curbs redundant re-checks after passing. Zeroing both collapses R_ISR = R_FRS = 1.0, leaving redundant multi-round loops unpenalized.

  • Empirically, removing both boundaries in Binary RL (0, 0, 0) causes trajectories to drift toward US (31.6% → 41.0%), whereas PAPO (+0.5, +0.2, -0.2) sharply reduces US and lifts both ISR and Pass@1 across both 8B and 32B models (added to App. F, Table 11):

Model / Reward (CVDP Func.)ISRFRSUSPass@1
8B Stage-2 SFT27.9%40.4%31.6%39.5%
8B Binary RL (0, 0, 0)22.8%36.2%41.0%39.7%
8B PAPO (+0.5,+0.2,-0.2)37.1%39.2%23.7%44.9%
32B SFT → 32B PAPO31.6 → 36.3%40.8 → 42.2%27.6 → 21.5%42.5 → 48.7%
  • W1: The reward values are also fixed without any sensitivity study.
  • Q2: How sensitive are the results to the reward values (+0.5 / +0.2 / -0.2)?

(+0.5, +0.2, -0.2) was set a priori from the desired ordering R_ISR > R_FRS > R_US > 0 without test-set tuning. Due to the prohibitive compute cost of online multi-turn EDA RL, we did not run a grid search: a single 729-step 8B RL run requires 48 A100 GPUs for 110 hours (220 A100 GPU-days), so sweeping even 5 alternative configurations would consume 1,100 A100 GPU-days (26,400 GPU-hours). Any weights preserving this strict ordering provide the same directional incentives and may perform comparably or better.

  • W1: There is also no comparison against a simple tool-call or length penalty, which would show whether the "pattern-aware" design adds anything beyond encouraging shorter trajectories.
  • Q4: How does PAPO compare against binary reward plus a simple tool-call or length penalty?

With initialization, task grouping, and training steps matched, PAPO reaches 44.9% Pass@1 whereas Cost-Penalty RL collapses to 2.0%, proving PAPO’s reward is necessary. (Added to Sec. 4.3, Table 5.) Matching all RL settings with PAPO except the reward function, we train the Cost-Penalty RL baseline under R_x(τ) = V_x(y)·max(0.5001, 1 - 0.25 min(1, n_calls/N_max) - 0.25 min(1, L/L_max)). Key checkpoint results are shown below:

StepPAPOCost-Pen.PAPO Calls(Act.)Pen. Calls(Act.)PAPO LenPen. Len
10039.0%38.7%4.28 (99.5%)3.09 (98.5%)18.3k13.2k
15038.4%34.1%3.82 (99.2%)2.27 (86.4%)17.2k8.9k
20041.3%25.5%3.39 (99.0%)0.04 (2.5%)16.0k1.9k
35039.2%17.9%2.92 (98.6%)0.00 (0.3%)15.6k0.9k
50041.8%9.3%2.84 (98.7%)0.00 (0.3%)16.8k1.2k
70044.7%6.5%1.56 (98.3%)0.00 (0.0%)13.3k0.9k
72944.9%2.0%1.61 (98.5%)0.00 (0.0%)13.7k0.5k

The cost-penalty reward favors zero tool calls and minimal generation length (reward = 1.0), causing tool-call activation to collapse to 2.5% by Step 200 and generation length to 0.5k tokens. In contrast, PAPO maintains >98% tool-call activation and achieves 44.9% Pass@1.

  • W2: It's unclear why PAPO helps when binary-reward RL barely does. Binary-reward RL improves the 8B model only from 39.5% (SFT) to 39.7%, while PAPO reaches 44.9%. This is a large and unusual gap, and the paper doesn't explain it.
  • Q3: Why does binary-reward RL barely improve over SFT? Was it tuned as carefully as PAPO?

Despite identical RL tuning, Binary RL stalls near SFT because it neither rewards thorough verification nor penalizes wasteful or unverified behavior. Using the exact same 8B SFT initialization, training data, and RL settings as PAPO, all Binary RL checkpoints remain in 39.5%–40.1% Pass@1 (39.7% final vs. 39.5% SFT). Because every correct submission earns the exact same 1.0 reward regardless of whether it passed on the first self-check, looped through many redundant tool calls, or skipped verification altogether, the model receives no training signal to improve its checking behavior. As a result, submissions without a passing self-check rise (31.6% → 41.0%), while mean tool calls (4.66 → 4.90) and incomplete rollouts (60.2% → 59.4%) remain unimproved.

  • W2: One possible factor is truncation: the SFT model hits the length limit on 33.7% of problems, PAPO reduces this to 15.2%, and every truncated answer counts as a failure. If so, much of PAPO's gain may come from shortening responses rather than from better RTL design or checking.
  • Q3: How much of PAPO's gain remains when accuracy is computed only on non-truncated responses?

This is a misunderstanding: the concern assumes that hitting the length limit causes a rollout to fail, but the causal relationship can run in the opposite direction. When PAPO finds a correct RTL design early, it can finish before the limit, and when a baseline has not yet found a correct design, it may keep revising until it hits the limit. This is supported by our evaluation setting: even when a rollout hits the length limit, our evaluator extracts the last valid RTL candidate from the final tool call and checks it for correctness. Thus, when a truncated baseline rollout with a valid candidate fails, it is because even its last candidate fails the checks. To address Q3 directly, we also compare the three models on the same tasks using only non-truncated rollouts (222 tasks totally), shown in the following table:

CVDP Non-Truncated Metric8B SFT8B Binary8B PAPO
Non-truncated Tasks (N=222)54.6%51.8%56.6%

Together with Table 6's single-shot revision evaluation (0% truncation across all models, where 8B PAPO improves from 26.0% to 35.0%), these results confirm that PAPO's gains do not primarily come from reduced truncation.

  • W2: A related concern is that the largest bonus goes to trajectories whose self-written testbench passes on the first try, so the model could earn it by writing weaker tests. The paper doesn't check whether self-test quality changes after RL, even though tool calls drop by 64% while thinking length grows 2.8x.
  • Q5: Does the quality of the model's self-written testbenches change after RL? Is there evidence that the first-pass bonus encourages weaker tests?

Writing weaker self-written testbenches cannot game the first-pass bonus; self-written testbench quality improves rather than weakens after RL. The model's self-written testbenches only determine intermediate self-check outcomes, whereas final RL correctness is judged independently against the golden reference RTL. Passing buggy RTL through a weak self-test still receives zero RL reward, so the first-pass bonus is only awarded to genuinely correct code. Over 1,510 CVDP rollouts per model (302 × 5), (Appendix G, Table 12) reports Self-Check Precision (the share of self-test-passing trajectories that also pass the official benchmark testbench) for Completed trajectories (non-truncated, passing their final check) and Single-Round (ISR) trajectories (passing on the first checking round):

Self-Check Metric8B SFT8B PAPO32B SFT32B PAPO
Completed Self-Check Precision52.5%61.4%58.7%63.5%
Single-Round (ISR) Self-Check Precision71.7%73.9%70.2%78.1%

Dear Reviewers,

Please read the authors’ rebuttal and join the discussion. Kindly clarify whether your main concerns have been addressed and raise any follow-up questions promptly so the authors have time to respond.

Thank you for your time and effort!

Reviewer_6hpJOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RTL-INTERACT post-trains Qwen3-8B/32B to generate RTL, build EDA checks, and revise from tool feedback, using SFT distilled from DeepSeek-V3.2 followed by PAPO, a GRPO variant with correctness-gated rewards for concise checking and effective revision. Evaluation spans CVDP, VerilogEval v2, and RTLLM v2, with reward, grouping, and supervision ablations.

Critical strengths
  1. Consistent gains across benchmarks. Both sizes beat comparable RTL-specialized baselines in Pass@1 on CVDP, VerilogEval v2, and RTLLM v2, and the 32B model surpasses its tool-enabled teacher on CVDP.
  2. Better accuracy with fewer tool calls. On CVDP, RL lifts the 8B model's Pass@1 from 39.5% to 44.9% while cutting average tool calls from 4.7 to 1.7. Ablations credit both pattern-aware rewards and task grouping, a useful improvement given the inference cost of repeated EDA checks.
Critical weaknesses
  1. Efficiency gains are only partly quantified. Tool calls drop by 63.8%, but thinking text grows to 1.5× its SFT level. Overall response length falls, yet the paper reports no end-to-end latency, model inference or EDA execution cost, or cost per solved task. The results show fewer tool interactions, not net computational savings.

  2. PAPO's specific design is not shown to be necessary. The reward ablation shows pattern-aware rewards beat binary correctness, but there is no comparison with simpler cost-aware rewards such as a length or tool-call penalty. Whether the lower tool use and higher accuracy require PAPO's pattern design remains unclear.

Questions for authors
  1. What are the end-to-end inference time and total cost per solved task for SFT and PAPO, counting both model generation and EDA execution?

  2. With task grouping and training budget held fixed, does PAPO outperform correctness rewards with a simple length or tool-call penalty? [Request for additional experiments]

  3. Could PAPO's preference for single-round success push the model to skip necessary checks or stop refining too early? What evidence shows that fewer tool calls do not come at the expense of verification quality?

  4. Please report confidence intervals for the key gains, especially the 1.0-point improvement over TIR-only supervision and the 2.0-point CVDP Pass@1 margin over the teacher.

4 comments

We thank Reviewer 6hpJ for constructive questions on end-to-end cost, baselines, verification, and CIs.

  • W1: Efficiency gains are only partly quantified. Tool calls drop by 63.8%, but thinking text grows to 1.5x its SFT level. Overall response length falls, yet the paper reports no end-to-end latency, model inference or EDA execution cost, or cost per solved task. The results show fewer tool interactions, not net computational savings.
  • Q1: What are the end-to-end inference time and total cost per solved task for SFT and PAPO, counting both model generation and EDA execution?
  1. Fewer tool calls eliminate repeated code regeneration, reducing total tokens despite longer thinking. Generated tokens/rollout drop 28.8% despite 1.5x <think> length because cutting 63.8% tool calls (4.7→1.7) eliminates 8,775 non-<think> tokens (-73.0%) across tool-call requests and final RTL, primarily by avoiding repeated candidate RTL/TB regeneration across rounds.

  2. PAPO achieves substantial net reductions in both end-to-end latency and cost on matched Both-Solved tasks. Counting LLM inference and measured EDA time, PAPO cuts overall rollout time by 30.3%, and on the matched Both-Solved subset reduces rollout tokens by 27.9%, tool calls by 62.8%, and end-to-end time by 29.4% (added to App. H, Table 13; linked in Sec. 4.5):

CVDP Metric (N = 302)Stage-2 SFT (Step 0)PAPO RL (Step 729)Relative Change
Mean <think> Tokens7,18010,427+45.2%
Generated Tokens per Rollout (Figure 3)19,20813,680-28.8%
Tool-Call Request + Final-RTL Tokens12,0283,253-73.0%
Mean Tool Calls per Rollout4.71.7-63.8%
End-to-End Rollout Time (LLM + EDA)167.8 s116.9 s-30.3%
CVDP Pass@139.5%44.9%+13.7%
Generated Tokens per Rollout (Both-Solved)17,82512,846-27.9%
Mean Tool Calls per Rollout (Both-Solved)4.31.6-62.8%
End-to-End Rollout Time (LLM + EDA) (Both-Solved)155.3 s109.7 s-29.4%
  • W2: PAPO's specific design is not shown to be necessary. The reward ablation shows pattern-aware rewards beat binary correctness, but there is no comparison with simpler cost-aware rewards such as a length or tool-call penalty. Whether the lower tool use and higher accuracy require PAPO's pattern design remains unclear.
  • Q2: With task grouping and training budget held fixed, does PAPO outperform correctness rewards with a simple length or tool-call penalty? [Request for additional experiments]

With initialization, task grouping, and training steps matched, PAPO reaches 44.9% Pass@1 whereas Cost-Penalty RL collapses to 2.0%, proving PAPO’s reward is necessary. (Added to Sec. 4.3, Table 5.) Matching all RL settings with PAPO except the reward function, we train the Cost-Penalty RL baseline under R_x(τ) = V_x(y)·max(0.5001, 1 - 0.25 min(1, n_calls/N_max) - 0.25 min(1, L/L_max)); key checkpoint results are shown below:

StepPAPOCost-Pen.PAPO Calls(Act.)Pen. Calls(Act.)PAPO LenPen. Len
10039.0%38.7%4.28 (99.5%)3.09 (98.5%)18.3k13.2k
15038.4%34.1%3.82 (99.2%)2.27 (86.4%)17.2k8.9k
20041.3%25.5%3.39 (99.0%)0.04 (2.5%)16.0k1.9k
35039.2%17.9%2.92 (98.6%)0.00 (0.3%)15.6k0.9k
50041.8%9.3%2.84 (98.7%)0.00 (0.3%)16.8k1.2k
70044.7%6.5%1.56 (98.3%)0.00 (0.0%)13.3k0.9k
72944.9%2.0%1.61 (98.5%)0.00 (0.0%)13.7k0.5k
  • The resulting reward peaks at 1.0 when n_calls = 0 and L is minimal, favoring unchecked guesses over verified code. This collapses tool use (Steps 100→200: 98.5%→2.5%) and length (1.9k→0.5k, 2.0%). PAPO assigns 0 to incorrect code and differentiates correct rollouts (0.8 < 1.2 < 1.5) without taxing <think>, keeping >98% activation, 1.61 calls, and 44.9% Pass@1 (41.5% Round-1 recovery vs. 31.3% Binary).
  • Q3: Could PAPO's preference for single-round success push the model to skip necessary checks or stop refining too early? What evidence shows that fewer tool calls do not come at the expense of verification quality?
  1. PAPO prevents check-skipping (>98% tool use) and premature stopping after an initial self-check fail. Among correct rollouts, our three self-check patterns are: Immediate Self-check Resolution (ISR: passing the first checking round, reward 1.5), Feedback-driven Refinement Success (FRS: refining across multiple checking rounds until passing, reward 1.2), and Unsuccessful Self-checking (US: skipping checks or ending without a passing check, reward 0.8). Therefore, skipping checks loses the verification bonuses and earns only 0.8 (US) even when the final RTL is correct, which is strictly lower than FRS (1.2) and ISR (1.5). As a result, tool activation stays above 98% throughout training (Table 5). Likewise, once the first checking round (Round 1) fails, the 1.5 (ISR) reward is no longer reachable. Stopping immediately yields 0 if the bug remains or at best 0.8 (US), whereas continuing to refine with EDA feedback until a self-check passes earns 1.2 (FRS). Empirically (Table 12, added to Appendix G), Round-1 Recovery, which is the final benchmark pass rate among trajectories whose first checking round failed, rises to 41.5% on 8B (vs. 38.5% SFT and 31.3% Binary RL).

  2. Fewer tool calls do not reduce verification quality because terminal RL rewards are independently calculated against reference RTL. The model's self-written testbenches only determine intermediate self-checks, whereas RL correctness V_x(y) is gated by CodeV-R1's reference-based equivalence checker and benchmarks are scored by official hidden test suites. Weak self-tests that let buggy RTL pass still earn V_x(y) = 0 and zero reward. Over 1,510 CVDP rollouts per model (302 × 5), Table 12 (added to Appendix G) reports Self-Check Precision (share of self-test-passing trajectories that also pass the official benchmark) for both Completed trajectories (non-truncated, passing their final check) and Single-Round (ISR) submissions—confirming that even single-round ISR submissions do not rely on weaker checks:

CVDP Metric8B SFT8B Bin.8B PAPO32B SFT32B PAPO
Round-1 Recovery (after fail)38.5%31.3%41.5%41.3%46.3%
Completed Self-Check Precision52.5%46.9%61.4%58.7%63.5%
Single-Round (ISR) Self-Check Precision71.7%52.9%73.9%70.2%78.1%
  • Q4: Please report confidence intervals for the key gains, especially the 1.0-point improvement over TIR-only supervision and the 2.0-point CVDP Pass@1 margin over the teacher.

On CVDP (N=302) and RTLLM v2 (N=50) with n=5 rollouts/task (T=0.7), we resample per-task paired pass-rate differences 10,000 times for the two queried margins:

Comparison (Benchmark)BaselineOursPaired Δ [95% CI]p-value
CoT→TIR vs. TIR-only (900M, CVDP)28.5%29.5%+1.0 pp [-1.5%, +3.3%]0.4047
32B PAPO vs. DeepSeek-V3.2 (CVDP)46.7%48.7%+2.0 pp [-0.9%, +5.4%]0.2141
32B PAPO vs. DeepSeek-V3.2 (VE2 CC)67.9%80.3%+12.4 pp [+7.0%, +17.8%]< 0.0001
32B PAPO vs. DeepSeek-V3.2 (RTLLM v2)68.8%77.2%+8.4 pp [+3.1%, +13.7%]0.0018

For the other key RL gains, CVDP Pass@1 improvements are all significant in (App. E, Table 10): 8B PAPO vs. SFT (+5.4 pp [+2.9, +8.2], p<0.0001), 8B PAPO vs. Binary RL (+5.2 pp [+2.5, +8.0], p<0.0001), and 32B PAPO vs. SFT (+6.2 pp [+2.9, +9.5], p=0.0002).

Ref: [CodeV-R1] Zhu et al., arXiv:2505.24183 (2025).

Dear Reviewers,

Please read the authors’ rebuttal and join the discussion. Kindly clarify whether your main concerns have been addressed and raise any follow-up questions promptly so the authors have time to respond.

Thank you for your time and effort!

Dear Reviewer 6hpJ,

Thank you again for your constructive feedback and for recognizing our cross-benchmark gains, margin over the distillation teacher, and tool-call reduction. As the discussion period is nearing its end, we would like to kindly check whether our responses and revised manuscript have addressed your concerns.

In our response, we added end-to-end LLM + EDA latency measurements, token breakdowns, and matched Both-Solved subset cost comparisons confirming net computational savings (Section 4.5 and Appendix H, Table 13), alongside a controlled Cost-Penalty RL baseline under matched settings showing why simple call/length penalties trigger tool abandonment whereas PAPO reaches 44.9% Pass@1 (Section 4.3, Table 5). We also evaluated Round-1 failure recovery and Self-Check Precision against official hidden benchmarks to confirm that fewer tool calls do not weaken verification quality (Appendix G, Table 12), and reported paired bootstrap confidence intervals for all queried margins (Appendix E, Table 10). These additional experiments and analyses are included in the revised manuscript and highlighted in blue.

If you have any remaining questions, we would be happy to address them before the discussion ends. If our responses have resolved your concerns, we would sincerely appreciate it if you could consider updating your score.

Best regards, The Authors

Reviewer_feimOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a multi-task post-training approach to train LLMs to generate RTL code, leverage EDA tools, and modify hardware designs using feedback. The RTL-Interact approach trains with a new policy optimization method for multi-task RL and employs staged supervision. The paper shows improved correctness and fewer tool calls.

Critical strengths
  1. The method eliminates the generation, verification, and revision being in seperate components and combines into single model.
  2. The paper demonstrates strong benchmark coverage and multiple ablations.
Critical weaknesses
  1. There is insufficent establishment of the statistical robustness of the resuts. For instance, the results lack error bars and confidence intervals and results for multiple seeds.
  2. The paper demonstrates limited novelty as an ML research contribution, largely building off of prior work in post-training and multi-task RLVR.
Questions for authors
  1. Could the authors please report pass rates under a stronger independently generated testbench to assess strength of the results and see whether they capture edge cases?

  2. Could the authors provide confidence intervals as well as results on multiple seeds and inpdepedently trained checkpoints?

  3. Could the authors elaborate more on their methodological novelty as well as share application and relevance of results to the ML community, including discussion of application to ML accelerator design? Moreover, is there application of the proposed RL technique to domains beyond RTL?

3 comments

We thank Reviewer feim for recognizing our unified architecture and broad benchmark evaluations.

  • W1: There is insufficent establishment of the statistical robustness of the resuts. For instance, the results lack error bars and confidence intervals and results for multiple seeds.
  • Q2: Could the authors provide confidence intervals as well as results on multiple seeds and inpdepedently trained checkpoints?

All core accuracy gains are statistically significant (p <= 0.0032) with strictly positive 95% confidence intervals and ~2:1 task win-to-loss ratios across benchmarks and model scales:

Comparison (Benchmark)Baseline Pass@1 [95% CI]Ours Pass@1 [95% CI]Paired Δ [95% CI]Two-Sided p-valueWins / Losses / Ties
8B SFT vs. Qwen3-8B Base (CVDP)18.9% [15.6, 22.4]39.5% [35.0, 44.1]+20.6 pp [+16.9, +24.3]<0.0001154 / 21 / 127
8B PAPO vs. 8B SFT (CVDP)39.5% [35.0, 44.1]44.9% [40.3, 49.8]+5.4 pp [+2.9, +8.2]<0.000183 / 43 / 176
8B PAPO vs. Binary RL (CVDP)39.7% [35.1, 44.3]44.9% [40.3, 49.8]+5.2 pp [+2.5, +8.0]<0.000193 / 36 / 173
8B PAPO vs. w/o Task Grouping (CVDP)41.8% [37.0, 46.6]44.9% [40.3, 49.8]+3.1 pp [+1.1, +5.3]0.003275 / 42 / 185
32B PAPO vs. 32B SFT (CVDP)42.5% [37.6, 47.0]48.7% [43.8, 53.6]+6.2 pp [+2.9, +9.5]0.000288 / 45 / 169
8B PAPO vs. Qwen3-8B Base (RTLLM v2)45.2% [33.6, 56.8]77.6% [66.4, 86.4]+32.4 pp [+22.4, +42.4]<0.000132 / 4 / 14

Evaluation uncertainty is quantified via 10,000 paired task-level bootstrap resamples, while consistency across independently trained 8B and 32B models addresses training stability. Main-table Pass@1/Pass@5 already average n=5 rollouts/task (T=0.7; 302 CVDP, 50 RTLLM v2) for decoding variance; 10,000 paired task resamples yield the 95% CIs, Δ, p-values, and W/L/T above (added to App. E, Table 10). Since re-running 3-stage post-training across seeds is computationally prohibitive, following DeepSeek-R1 and CodeV-R1 we fix training seeds and verify robustness across two model scales and three benchmarks.

  • W2: The paper demonstrates limited novelty as an ML research contribution, largely building off of prior work in post-training and multi-task RLVR.
  • Q3: Could the authors elaborate more on their methodological novelty as well as share application relevance of results to the ML community, including discussion of application to ML accelerator design? Moreover, is there application of the proposed RL technique to domains beyond RTL?
  1. PAPO resolves multi-round tool-use credit assignment without a critic or PRM, breaking standard binary RLVR's plateau (39.5% → 39.7%, Table 4) to reach 44.9%. Traditional binary RLVR stalls because a multi-round loop or unchecked guess gets the same 1.0 reward as a 1-round verified fix. Gated by final correctness V_x(y) ∈ {0, 1}, PAPO partitions passing rollouts by binary self-checks (p_1, ..., p_n) into various granularity, i.e., Immediate Self-check Resolution (ISR: passing the first checking round, reward 1.5), Feedback-driven Refinement Success (FRS: refining across multiple checking rounds until passing, reward 1.2), and Unsuccessful Self-checking (US: skipping checks or ending without a passing check, reward 0.8). Paired with our CoT → TIR stage-order finding (reversing drops tool use 97.4% → 0.0%) and task-grouped batching, this yields a multi-round agentic RL recipe that can serve as a useful reference for the machine learning community.

  2. RTL-Interact directly automates and optimizes core ML hardware accelerator blocks with substantial physical PPA reductions. CVDP and RTLLM v2 cover systolic PEs, MACs, pipelined arithmetic, FIFOs, and arbiters; via Yosys (45nm Nangate) + OpenSTA across 919 training-pool optimization trajectories, our pipeline cuts cell count by 24.0%, area by 20.1%, and peak power by 14.3%, with all 919 trajectories passing reference-based simulation equivalence under the tested stimuli.

  3. PAPO can extend beyond RTL to other tool-interactive RLVR domains. Its reward uses only domain-agnostic interaction signals—tool calls n, self-check outcomes p_n, and an external verifier V_x(y)—making it adaptable to SWE agents (self-written unit tests vs. hidden CI), Lean 4 theorem proving, and SQL generation; cross-domain empirical validation is left for future work.

  • Q1: Could the authors please report pass rates under a stronger independently generated testbench to assess strength of the results and see whether they capture edge cases?

This stems from a misunderstanding: all reported pass rates use the original benchmarks' independent human-expert testbenches with thorough edge-case coverage, never model-written ones. (Clarified in Sec. 3.1 & App. A.3.) There are three kinds of testbenches in our work:

  1. Model-written self-check testbenches serve solely for intermediate model self-debugging and never score any benchmark or RL reward.

  2. Rule-based testbench to calculate RL rewards, using CodeV-R1’s reference-based equivalence verifier. During RL, each prompt pairs with its GitHub reference RTL and the verifier tests equivalence via extensive randomized stimuli, so passing a weak self-test cannot earn positive reward. We have emphasized this in Section 3.1 in the revised paper.

  3. Official hidden human-expert testbenches independently score all Pass@1/Pass@5 benchmarks. Benchmarks involve CVDP (Cocotb + Pytest), VerilogEval v2, and RTLLM v2 (Synopsys VCS); these testbenches cover clock/reset edges, corner cases, random vectors, and synthesis equivalence.

Ref: [CodeV-R1] Zhu et al., arXiv:2505.24183 (2025); [DeepSeek-R1] DeepSeek-AI, arXiv:2501.12948 (2025).

Dear Reviewers,

Please read the authors’ rebuttal and join the discussion. Kindly clarify whether your main concerns have been addressed and raise any follow-up questions promptly so the authors have time to respond.

Thank you for your time and effort!

Comments on the paper

4 comments

We thank Reviewer Kvhq for supportive comments and helpful questions on RL dynamics and protocol details.

  • W1: FRS requires multiple checks and a successful last check, but not prior failure or effective repair. If all successful same-problem rollouts share one reward multiplier, normalized advantages reduce to binary-reward advantages when variance is nonzero and the stabilizer is neglected. This conditional observation does not prove PAPO ineffective; it makes unreported pattern mixtures and genuine repair statistics important.
  • Q1: How often do correct same-problem rollouts mix patterns, and how many FRS trajectories actually fail, revise, and succeed? How do repair and self-check false-positive rates compare with matched binary-reward RL?

Empirically, the fraction of tasks where PAPO's normalized advantages degenerate to binary-reward advantages is very small. Under G = 8 rollouts across all 302 CVDP tasks (2,416 rollouts per model), among tasks with at least 2 passing rollouts, 76.5% (117/153, SFT) and 73.6% (128/174, PAPO) mix at least 2 distinct checking patterns (ISR, FRS, US), with 0.0% collapsing to all-8 ISR. Across all 4 terminal outcomes (ISR-pass, FRS-pass, US-pass, and Fail), 62.9% (190/302, SFT) and 63.2% (191/302, PAPO) span at least 2 outcomes with non-zero gradients; the remaining single-outcome groups consist of all-8 failures (32.8%, 99/302 for SFT; 35.1%, 106/302 for PAPO) and only 4.3% (13/302, SFT) / 1.7% (5/302, PAPO) all-8 single-pattern passes (which have zero gradient under any RL reward).

Furthermore, over 1,510 CVDP rollouts per model (302 × 5), Genuine Repair Share of FRS, which is defined as the share of FRS trajectories that fail at least one prior self-check before revising and passing the final self-check, reaches 74.6%–87.2% (81.9% on functional tasks for PAPO). On the same 1,510 rollouts, PAPO also outperforms matched Binary RL on both Round-1 Recovery (official benchmark pass rate after failing the first checking round) and self-check false-positive rate (the false-pass share among self-test passes, where lower is better):

CVDP Trajectory / Check Metric8B SFT8B Binary8B PAPO
Genuine Repair Share of FRS87.2%78.0%74.6%
Round-1 Recovery (after fail) ↑38.5%31.3%41.5%
Overall Self-Check False-Positive Rate (All) ↓47.6%51.5%46.7%
Completed Self-Check False-Positive Rate ↓47.5%53.1%38.6%
Single-Round (ISR) False-Positive Rate (All) ↓28.3%47.1%26.1%
  • W1: A matched correctness-gated call/length-cost control is absent.
  • Q2: With initialization, task composition, training budget, and validation-only tuning matched, does PAPO outperform a correctness-gated call/length-cost reward?

With initialization, task grouping, and training steps matched, PAPO reaches 44.9% Pass@1 whereas Cost-Penalty RL collapses to 2.0%, proving PAPO’s reward is necessary. (Added to Sec. 4.3, Table 5.) Matching all RL settings with PAPO except the reward function, we train the Cost-Penalty RL baseline under R_x(τ) = V_x(y)·max(0.5001, 1 - 0.25 min(1, n_calls/N_max) - 0.25 min(1, L/L_max)); key checkpoint results are shown below:

StepPAPOCost-Pen.PAPO Calls(Act.)Pen. Calls(Act.)PAPO LenPen. Len
10039.0%38.7%4.28 (99.5%)3.09 (98.5%)18.3k13.2k
15038.4%34.1%3.82 (99.2%)2.27 (86.4%)17.2k8.9k
20041.3%25.5%3.39 (99.0%)0.04 (2.5%)16.0k1.9k
35039.2%17.9%2.92 (98.6%)0.00 (0.3%)15.6k0.9k
50041.8%9.3%2.84 (98.7%)0.00 (0.3%)16.8k1.2k
70044.7%6.5%1.56 (98.3%)0.00 (0.0%)13.3k0.9k
72944.9%2.0%1.61 (98.5%)0.00 (0.0%)13.7k0.5k

The cost-penalty reward favors zero tool calls and minimal generation length (reward = 1.0), causing tool-call activation to collapse to 2.5% by Step 200 and generation length to 0.5k tokens. In contrast, PAPO maintains >98% tool-call activation and achieves 44.9% Pass@1.

  • W1: Truncation falls from 33.7% to 15.2%, leaving improved budget use as an alternative explanation.

This is not the main reason. Truncation falls because, when PAPO finds a correct RTL design early, it can finish before the limit, and when a baseline has not yet found a correct design, it may keep revising until it hits the limit. This is supported by our evaluation setting: even when a rollout hits the length limit, our evaluator extracts the last valid RTL candidate from the final tool call and checks it for correctness. Thus, when a truncated baseline rollout with a valid candidate fails, it is because even its last candidate fails the checks. We also compare the three models on the same tasks using only non-truncated rollouts (222 tasks totally), shown in the following table:

CVDP Non-Truncated Metric8B SFT8B Binary8B PAPO
Non-truncated Tasks (N=222)54.6%51.8%56.6%

Together with Table 6's single-shot revision evaluation (0% truncation across all models, where 8B PAPO improves from 26.0% to 35.0%), these results confirm that PAPO's gains do not primarily come from reduced truncation.

  • W2: Tool-enabled prompts also add reasoning/testing guidance, so the tool switch is not isolated.
  • Q2: Do benefits persist with identical reasoning prompts and only tool availability changed?

Within Tables 2–3, every Tool ✗ (App. I.1) and Tool ✓ (I.2) entry uses an identical prompt per setting (Table 4 uses I.2 throughout; Table 6 permits no tool calls), and running I.2 with Tool ✗ drops Qwen3-8B Pass@1 from 18.9% (I.1) to 17.6% (-1.3 pp), proving gains come from live tool execution rather than prompt text.

  • W2: Appendix C selects alpha on CVDP without specifying an independent validation split.
  • Q3: Which data select alpha, reward weights, and checkpoints?

Reward weights (+0.5, +0.2, -0.2) and step budgets (729 [8B], 499 [32B]) were fixed a priori without test-set tuning, and α=0.3 (App. C) also achieves top accuracy on held-out VerilogEval v2.

  • W2: Optimization PPA proxies, thresholds, and verification stimuli need specification.
  • Q3: Please specify the PPA metric/threshold, whether self-check and final-verification stimuli differ.

RL PPA rewards gate strictly on simultaneous Yosys post-mapping cell and wire reductions (n_cells < b_cells and n_wires < b_wires) plus golden-RTL equivalence. Only postmap_cells and postmap_wires are gated (excluding peak_power, arrival_time, and cell_area, which can be empty or noisy on combinational or clockless designs), with no extra % threshold; ties/increases give 0, and lint requires 0 Verilator warnings. Self-checks (p_j) run model TBs; RL V_x(y) calls CodeV-R1’s equivalence verifier against golden RTL, while benchmarks use official hidden suites (clarified in Sec. 3.1 & App. A.3).

  • W2: Small differences lack paired intervals or repeated-training variability.
  • Q3: Please report uncertainty for the main comparisons. Do end-to-end measurements confirm a resource benefit beyond reducing calls from 4.7 to 1.7?

Per-task 10k-bootstrap CIs (n=5 rollouts/task) confirm gains (p <= 0.0032), and GPU + EDA timing confirms end-to-end rollout savings (added to App. E, Table 10 & App. H, Table 13):

CVDP Paired Comparison (10k Bootstrap)Paired Δ [95% CI]p-val
8B PAPO vs. SFT+5.4 pp [+2.9, +8.2]<0.0001
8B PAPO vs. Binary RL+5.2 pp [+2.5, +8.0]<0.0001
8B PAPO vs. w/o task grouping+3.1 pp [+1.1, +5.3]0.0032
32B PAPO vs. SFT+6.2 pp [+2.9, +9.5]0.0002
CVDP Cost (GPU+EDA)SFT → PAPOChange (Δ)
LLM+EDA Rollout Time167.8s → 116.9s-30.3%
Gen. Tok. / Rollout19,208 → 13,680-28.8%

Ref: [CodeV-R1] Zhu et al., arXiv:2505.24183 (2025).

AuthorsOpenReview ↗

Paper Revision

Dear Reviewers and Area Chairs,

We sincerely thank all reviewers for your thoughtful, insightful, and constructive comments, which helped us significantly improve the clarity and rigor of our paper. We have uploaded a revised manuscript with all additions and modifications highlighted in blue. The main changes are summarized below:

(1) [Section 3.1] Clarified how source reference RTL is handled across task types during RL and explicitly distinguished CodeV-R1's terminal equivalence checker from model-authored intermediate self-check testbenches.

(2) [Section 4.3 & Table 5] Added a controlled comparison against a correctness-gated Cost-Penalty RL baseline (Table 5) and analyzed why monotonic penalties induce tool abandonment whereas PAPO preserves tool use and improves Pass@1.

(3) [Section 4.5] Added a pointer to Appendix H for the full end-to-end cost breakdown.

(4) [Appendix A.3] Added two clarifying paragraphs detailing the isolation of model self-checks, RL equivalence verification, and official benchmark testbenches, as well as optimization PPA gating criteria and post-hoc 45nm physical evaluation.

(5) [Appendix B] Added a cross-reference to the controlled Cost-Penalty RL comparison in Section 4.3 (Table 5).

(6) [Appendix E & Table 10] Added 10,000-resample paired task-level bootstrap significance tests (95% CIs, paired Δ, p-values, and win/loss/tie counts) across core benchmarks and additional comparisons.

(7) [Appendix F & Table 11] Added checking-pattern distributions (ISR, FRS, and US) on CVDP functional tasks before and after RL for both 8B and 32B models.

(8) [Appendix G & Table 12] Added evaluation of Round-1 failure recovery and model self-check precision against independent official benchmark verification on CVDP.

(9) [Appendix H & Table 13] Added end-to-end rollout latency (LLM + EDA) and token breakdowns (<think> vs. Tool-Call Request + Final-RTL Tokens) on CVDP, along with matched Both-Solved subset savings.

Dear Reviewers,

Please read the authors’ rebuttal and join the discussion. Kindly clarify whether your main concerns have been addressed and raise any follow-up questions promptly so the authors have time to respond.

Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.