- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper presents RTL-Interact, a post-training framework that trains a single model to generate RTL and to call EDA tools (simulation, lint, synthesis, and simulation-based equivalence checking) to check and revise its own designs. Starting from the CodeV-R1 dataset, DeepSeek-V3.2 is used to filter spec–RTL pairs and generate about 13.5K tool-use trajectories across three tasks: generation, completion of masked logic, and lint/synthesis optimization. The model is trained first with chain-of-thought SFT and then with tool-use SFT; a matched-budget study supports this ordering. It is then trained with RL using PAPO, a GRPO variant with two additions. The first is a correctness-gated reward that depends on the model’s self-checking pattern: 1.5 if the first self-check passes, 1.2 if it passes after revision, 0.8 without a successful self-check, and 0 if the final design is wrong. The second is fixed task proportions in each batch.
On CVDP’s 302 non-agentic problems, RTL-Interact-32B reaches 48.7% Pass@1 and 68.9% Pass@5, slightly above its tool-enabled teacher, and the 8B model reaches 44.9%. Both models are reported to beat all RTL-specialized baselines on CVDP, VerilogEval v2, and RTLLM v2. RL also reduces tool calls by 63.8% for the 8B model.
Critical strengths
-
A practical unified model with useful controlled studies. One model handles several RTL tasks and uses EDA feedback itself, with no separate repair model or agent pipeline, which makes it easy to deploy. The design studies are well controlled. The SFT ordering comparison keeps the token budget fixed and shows the order matters for keeping tool use (TIR→CoT gives 0% tool use). The PAPO ablations keep data, initialization, and training steps fixed. The separate generation-vs-revision test (Table 5) shows that RL mainly improves revision from feedback. These experiments explain where the gains come from better than most papers in this area.
-
Strong results against the teacher, with an efficiency gain. The 32B model slightly exceeds its tool-enabled teacher on CVDP, and RL improves accuracy while cutting tool calls and response length. That combination is useful in practice.
Critical weaknesses
-
The contribution of each part of the PAPO reward isn’t shown. The reward has three separate terms: a bonus for passing the first self-check (+0.5), a smaller bonus for passing after revision (+0.2), and a penalty for correct answers without a successful self-check (−0.2). The only reward ablation removes all three at once (binary reward), and the authors note it “tests the whole reward design, not each rule separately.” So we don’t know which term matters. Does the gain come from rewarding efficient checking, from rewarding successful revision, or simply from penalizing unchecked answers? The reward values are also fixed without any sensitivity study. Since PAPO is the paper’s main methodological contribution, per-term ablations are needed to support it. There is also no comparison against a simple tool-call or length penalty, which would show whether the “pattern-aware” design adds anything beyond encouraging shorter trajectories.
-
It’s unclear why PAPO helps when binary-reward RL barely does. Binary-reward RL improves the 8B model only from 39.5% (SFT) to 39.7%, while PAPO reaches 44.9%. This is a large and unusual gap, and the paper doesn’t explain it. One possible factor is truncation: the SFT model hits the length limit on 33.7% of problems, PAPO reduces this to 15.2%, and every truncated answer counts as a failure. If so, much of PAPO’s gain may come from shortening responses rather than from better RTL design or checking. A related concern is that the largest bonus goes to trajectories whose self-written testbench passes on the first try, so the model could earn it by writing weaker tests. The paper doesn’t check whether self-test quality changes after RL, even though tool calls drop by 64% while thinking length grows 2.8×.
Questions for authors
- Can you ablate each reward term separately, i.e., only the first-pass bonus, only the revision bonus, and only the unchecked penalty? Which term drives the improvement?
- How sensitive are the results to the reward values (+0.5 / +0.2 / −0.2)?
- Why does binary-reward RL barely improve over SFT? Was it tuned as carefully as PAPO? How much of PAPO’s gain remains when accuracy is computed only on non-truncated responses?
- How does PAPO compare against binary reward plus a simple tool-call or length penalty?
- Does the quality of the model’s self-written testbenches change after RL? Is there evidence that the first-pass bonus encourages weaker tests?
6 comments
- W2: It's unclear why PAPO helps when binary-reward RL barely does. Binary-reward RL improves the 8B model only from 39.5% (SFT) to 39.7%, while PAPO reaches 44.9%. This is a large and unusual gap, and the paper doesn't explain it.
- Q3: Why does binary-reward RL barely improve over SFT? Was it tuned as carefully as PAPO?
Despite identical RL tuning, Binary RL stalls near SFT because it neither rewards thorough verification nor penalizes wasteful or unverified behavior. Using the exact same 8B SFT initialization, training data, and RL settings as PAPO, all Binary RL checkpoints remain in 39.5%–40.1% Pass@1 (39.7% final vs. 39.5% SFT). Because every correct submission earns the exact same 1.0 reward regardless of whether it passed on the first self-check, looped through many redundant tool calls, or skipped verification altogether, the model receives no training signal to improve its checking behavior. As a result, submissions without a passing self-check rise (31.6% → 41.0%), while mean tool calls (4.66 → 4.90) and incomplete rollouts (60.2% → 59.4%) remain unimproved.
- W2: One possible factor is truncation: the SFT model hits the length limit on 33.7% of problems, PAPO reduces this to 15.2%, and every truncated answer counts as a failure. If so, much of PAPO's gain may come from shortening responses rather than from better RTL design or checking.
- Q3: How much of PAPO's gain remains when accuracy is computed only on non-truncated responses?
This is a misunderstanding: the concern assumes that hitting the length limit causes a rollout to fail, but the causal relationship can run in the opposite direction. When PAPO finds a correct RTL design early, it can finish before the limit, and when a baseline has not yet found a correct design, it may keep revising until it hits the limit. This is supported by our evaluation setting: even when a rollout hits the length limit, our evaluator extracts the last valid RTL candidate from the final tool call and checks it for correctness. Thus, when a truncated baseline rollout with a valid candidate fails, it is because even its last candidate fails the checks. To address Q3 directly, we also compare the three models on the same tasks using only non-truncated rollouts (222 tasks totally), shown in the following table:
| CVDP Non-Truncated Metric | 8B SFT | 8B Binary | 8B PAPO |
|---|---|---|---|
| Non-truncated Tasks (N=222) | 54.6% | 51.8% | 56.6% |
Together with Table 6's single-shot revision evaluation (0% truncation across all models, where 8B PAPO improves from 26.0% to 35.0%), these results confirm that PAPO's gains do not primarily come from reduced truncation.
- W2: A related concern is that the largest bonus goes to trajectories whose self-written testbench passes on the first try, so the model could earn it by writing weaker tests. The paper doesn't check whether self-test quality changes after RL, even though tool calls drop by 64% while thinking length grows 2.8x.
- Q5: Does the quality of the model's self-written testbenches change after RL? Is there evidence that the first-pass bonus encourages weaker tests?
Writing weaker self-written testbenches cannot game the first-pass bonus; self-written testbench quality improves rather than weakens after RL. The model's self-written testbenches only determine intermediate self-check outcomes, whereas final RL correctness is judged independently against the golden reference RTL. Passing buggy RTL through a weak self-test still receives zero RL reward, so the first-pass bonus is only awarded to genuinely correct code. Over 1,510 CVDP rollouts per model (302 × 5), (Appendix G, Table 12) reports Self-Check Precision (the share of self-test-passing trajectories that also pass the official benchmark testbench) for Completed trajectories (non-truncated, passing their final check) and Single-Round (ISR) trajectories (passing on the first checking round):
| Self-Check Metric | 8B SFT | 8B PAPO | 32B SFT | 32B PAPO |
|---|---|---|---|---|
| Completed Self-Check Precision | 52.5% | 61.4% | 58.7% | 63.5% |
| Single-Round (ISR) Self-Check Precision | 71.7% | 73.9% | 70.2% | 78.1% |
Dear Reviewers,
Please read the authors’ rebuttal and join the discussion. Kindly clarify whether your main concerns have been addressed and raise any follow-up questions promptly so the authors have time to respond.
Thank you for your time and effort!
We thank Reviewer QasB for thoughtful questions on reward design, binary RL, and self-test quality.
The three terms do not act as independent additive bonuses; instead, they establish a strict reward ordering across checking patterns to shape the model's verification behavior. Among correct trajectories, PAPO distinguishes three patterns: Immediate Self-check Resolution (
ISR: passing the first checking round, reward 1.5), Feedback-driven Refinement Success (FRS: refining across multiple checking rounds until passing, reward 1.2), and Unsuccessful Self-checking (US: skipping checks or ending without a passing check, reward 0.8), while incorrect RTL always receives 0. Together, these terms enforce the strict orderingR_ISR (1.5) > R_FRS (1.2) > R_US (0.8) > 0, which relies on two essential boundaries:Enforcing
R_FRS > R_US(1.2 vs. 0.8) incentivizes the model to repair failed checks rather than submit unchecked or broken code. Zeroing both collapsesR_FRS = R_US = 1.0, removing the training signal to keep fixing after the first checking round fails.Enforcing
R_ISR > R_FRS(1.5 vs. 1.2) rewards deeper initial planning and curbs redundant re-checks after passing. Zeroing both collapsesR_ISR = R_FRS = 1.0, leaving redundant multi-round loops unpenalized.Empirically, removing both boundaries in Binary RL (0, 0, 0) causes trajectories to drift toward
US(31.6% → 41.0%), whereas PAPO (+0.5, +0.2, -0.2) sharply reducesUSand lifts bothISRandPass@1across both 8B and 32B models (added to App. F, Table 11):(+0.5, +0.2, -0.2)was set a priori from the desired orderingR_ISR > R_FRS > R_US > 0without test-set tuning. Due to the prohibitive compute cost of online multi-turn EDA RL, we did not run a grid search: a single 729-step 8B RL run requires 48 A100 GPUs for 110 hours (220 A100 GPU-days), so sweeping even 5 alternative configurations would consume 1,100 A100 GPU-days (26,400 GPU-hours). Any weights preserving this strict ordering provide the same directional incentives and may perform comparably or better.With initialization, task grouping, and training steps matched, PAPO reaches 44.9% Pass@1 whereas Cost-Penalty RL collapses to 2.0%, proving PAPO’s reward is necessary. (Added to Sec. 4.3, Table 5.) Matching all RL settings with PAPO except the reward function, we train the Cost-Penalty RL baseline under
R_x(τ) = V_x(y)·max(0.5001, 1 - 0.25 min(1, n_calls/N_max) - 0.25 min(1, L/L_max)). Key checkpoint results are shown below:The cost-penalty reward favors zero tool calls and minimal generation length (reward = 1.0), causing tool-call activation to collapse to 2.5% by Step 200 and generation length to 0.5k tokens. In contrast, PAPO maintains >98% tool-call activation and achieves 44.9% Pass@1.