acceptodds
Under review as a conference paper at ICLR 2027

DiffRTL: Differential Reward Models for Synthesis-Efficient Reinforcement Learning of RTL Optimizers

Abstract

Training-time reinforcement learning is the standard route to endowing open-source language models with register-transfer-level (RTL) design ability, but rewarding power, performance, and area (PPA) requires logic synthesis per design, orders of magnitude slower than any other check in the loop. This synthesis bottleneck caps affordable training, and learned predictors of absolute PPA fall short as substitutes, since typical errors exceed the effect sizes of most optimizations. We observe that optimization consumes only the relative effect of a change, which is far easier to learn because paired comparisons cancel systematic errors. DiffRTL trains a differential reward model on verified transformation pairs manufactured by a paired data engine and uses a fast forward pass of the model as the PPA reward, corrected during training by periodic real-synthesis audits. Every reported number remains grounded in real simulation, formal equivalence checking, and synthesis. Spending 640 synthesis calls on audits alone, DiffRTL beats synthesis-rewarded training at six times the budget by 6.8 eff@1 points on Pluto, with matching gains on RTLLMv2, at a small fraction of the tool time. The differential formulation clearly surpasses an absolute predictor trained on identical data. Audits measure reward over-optimization exactly and suppress it throughout training.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Electronic Design Automation, RTL Code Generation, Large Language Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bh72OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DiffRTL replaces synthesis-based PPA rewards with a differential reward model, corrected through periodic synthesis audits. On Pluto, it achieves 53.6 eff@1 versus 46.8 for synthesis-rewarded GRPO, using 640 versus 4,000 online synthesis calls. It also reports gains on RTLLM v2 and RTL-OPT.

Critical strengths
  1. This paper addresses an important practical bottleneck: expensive synthesis feedback limits RL training.
Critical weaknesses
  1. Baseline coverage leaves the quality–efficiency tradeoff unclear. The synthesis baselines receive roughly 1.6/5/10 updates, versus 200 for DiffRTL. This is meaningful for online-budget efficiency, but a step-matched reference would clarify the effect of replacing real rewards. It is also unclear whether alternative budget allocations could strengthen the synthesis baselines.
  2. Overall cost savings and amortization need stronger evidence. Corpus construction costs 133k synthesis calls, while the headline savings concern online synthesis. Since the baselines share an SFT initialization derived from this corpus, common data costs and the reward model’s additional costs should be separated. Reuse of the trained reward model for additional policy-training runs is not demonstrated.
Questions for authors
  1. Could the authors report actual completed updates and partial-batch handling for each synthesis baseline, and provide a step-matched reference?
  2. Could the synthesis baselines improve by distributing their budgets over more updates?
  3. After separating shared SFT costs from reward-model costs, when does DiffRTL become cheaper overall? Can the same reward model support additional policy-training runs without rebuilding the corpus?
1 comment
AuthorsOpenReview ↗

Response to Reviewer bh72

We thank the reviewer for the comments. Tables R1 to R9 are in the general response.

W1. Baseline coverage leaves the tradeoff between quality and efficiency unclear.

We agree that Table 1 compares methods at a matched online budget, where the baselines complete far fewer updates. Table R1 reports the completed updates (2, 5, 10, and 20), and the revised Table 1 adds a policy-updates column.

Step-matched reference (Table R2). With identical updates, batch, and schedule, the two reward sources differ by at most 0.7 eff@1 points across six checkpoints. Each difference is within one standard deviation. At the 8×8 batch, the synthesis arm is nominally ahead at all three checkpoints. With three seeds, the comparison excludes a large per-update cost but cannot exclude a cost near one point. The Table 1 gap comes from the updates that each method can afford: 200 updates at 64×8 for DiffRTL and 10 for Synthesis@4000.

Scope of the reference. The 64×8 synthesis arm reaches 50 updates (about 22 h per seed), and the 200-update comparison uses an 8×8 batch.

Alternative budget allocations. See Q2.

W2. Overall cost savings and amortization need stronger evidence.

Cost separation (Table R4). The SFT warm start shared by all trained rows consumes 76,382 of the 133,000 corpus synthesis calls (57.4%). The remaining 56,618 calls (42.6%), 1,142 CPU·h of formal verification, 8.3 A100·h of edit generation, and 38.4 A100·h of reward-model training are specific to DiffRTL.

Reuse (Table R5). One reward-model checkpoint starts all 30 policy-training runs of the paper and this response without any corpus rebuild. The runs share the policy backbone and the instance pool, and differ in audit budget, ablation switch, or batch size. Corpus plus audits total 163,660 synthesis calls, or 5,455 calls per run when the entire corpus is charged to DiffRTL. For comparison, one synthesis-rewarded run of 50 updates consumes 19,936 calls (Table R2). The same checkpoint also trains a policy for a different objective (Table R9, 1 seed, preliminary), which is the only reuse beyond the paper's own configuration. Reuse across policy backbones is not tested.

The revised paper summarizes the cost ledger in Section 5.3, with the full ledger in Appendix J.2, and states that the headline savings concern online synthesis.

Q1. Completed updates, partial-batch handling, and a step-matched reference.

Completed updates (Table R1).

Budget (calls)6402,0004,0008,000
Completed updates251020

Synthesis@640 completes 2 updates but the second update prices only 37 to 39 of 64 instances. Synthesis@4000 seed 2 exhausts the budget during update 10.

Partial batches. When the budget ends inside an update, fully priced instances enter the update and unpriced instances contribute no gradient. A new update starts only if the remaining budget can price at least half the batch. Under this rule, Synthesis@8000 did not start update 21 with 8 to 99 calls left.

Step-matched reference. Table R2, summarized under W1.

Q2. Could the synthesis baselines improve by distributing their budgets over more updates?

Table R3 shows five allocations of the same 4,000-call budget with varying instances per update (G = 8 is fixed throughout).

Rows 2 to 5 use a shortened warm-up and constant peak learning rate. The 8×8 setting with the most updates (79) reaches 48.0, 1.2 points above the default 64×8 allocation (46.8). But the best reallocation remains 5.6 points below DiffRTL (53.6) while consuming 4,000 synthesis calls against 640 audit calls. The gap is not caused by how the budget is distributed.

Q3. When does DiffRTL become cheaper overall, and can the same reward model support additional policy-training runs?

  • Shared and reward-model costs. Separated in Table R4. Costs specific to DiffRTL are corpus synthesis (56,618 calls), formal verification (1,142 CPU·h), edit generation (8.3 A100·h), reward-model training (38.4 A100·h), and per-run audits (640 calls, 7.4 A100·h).
  • Break-even point. In CPU time, n = C_fix / (C_syn − 12.8), rounded up, with C_fix = 2,274 CPU·h at the seed-split accounting and 5,342 CPU·h when all corpus costs are charged to DiffRTL (formal verification included in both). Against the 64×8 run of 200 updates (about 80K calls, not run, parity assumed), n is 2 and 4. Against the measured 64×8 run of 50 updates (19,936 calls, eff@1 51.0), n is 6 and 14. Against the measured 8×8 run of 200 updates (9,983 calls, eff@1 50.5), n is 13 and 29. The final DiffRTL policy reaches 53.6. GPU time is outside this count (notes under Table R4).
  • Reuse. Yes, within the scope tested. The same checkpoint starts all 30 policy-training runs of the paper and this response, and one run with a new objective, with no corpus rebuild (Tables R5 and R9). The 30 runs share the policy backbone and the instance pool. Reuse across policy backbones is not tested.
Reviewer_AracOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses a learned reward model to cut synthesis calls in RL training. The model predicts PPA changes from code edits. Periodic synthesis checks give new labels to fix its predictions.

Critical strengths

The paper studies a useful reward hacking problem when using proxy reward model. Predicted rewards can go up while real circuit quality gets worse. The experiments show that updating the reward model with real synthesis results helps reduce this gap.

Critical weaknesses
  1. All results use Yosys + OpenSTA. Some gains in Yosys disappear under stronger commercial synthesis such as DC. So it is not clear how much of diffrtl's gain depends on this flow. Please test some outputs with Design Compiler.

  2. The reward model sees at most 120 lines of local context. This can miss info needed to judge if an edit hits the circuit's critical path. Table 2 only reports combined ranking quality, which does not establish accurate delay prediction. Please report separate results for area, delay, and power.

  3. The proxy reward model cuts synthesis calls in RL, but building and training it also costs time and compute. Its prediction errors also need repeated synthesis checks (reward hacking problem). After counting these costs, how much does diffrtl actually save over using synthesis directly?

Questions for authors
  1. Do the gains over baseline remain under dc?
  2. How accurate is the reward model on delay alone?
  3. How much does diffrtl save over using synthesis directly to reach the same optimization quality?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Arac (Part 1/2)

We thank the reviewer for the comments. Tables R1 to R9 are in the general response.

W1. All results use Yosys and OpenSTA. Please test some outputs with Design Compiler.

We have no Design Compiler (DC) license during the rebuttal period and cannot report DC results. We agree with the premise of the concern. Lu et al. (2026) report the same effect on earlier optimization benchmarks: differences that Yosys retains often vanish under DC.

The closest test available to us is the stock Yosys synth script (Table R8). This test also exposes a weakness of our original recipe. The original recipe F is the explicit pass list of Listing 1, which does not call the Yosys synth macro. F therefore lacks the wreduce, peepopt, alumacc, and share passes and is weaker than the Yosys default at the coarse stage. F′ replaces the coarse stage with the stock synth script, which adds these passes. Policies and the reward model stay trained under F. Both arms are evaluated zero-shot under F′ without any adaptation.

  • F′ lowers the PPA product of the base designs by 7.9% to 9.3%.
  • The order DiffRTL > Synthesis@4000 > SFT only holds on all three benchmarks under F′.
  • Absolute gains shrink. The RTLLM v2 improvement of DiffRTL drops from 27.4% to 21.8%. Of the best rewrites selected under F on Pluto, 76.3% keep a positive objective under F′, with a median gain ratio of 0.68. This figure follows the single rewrite selected under F for each task, whereas the 88% below is the ratio of eff@1 over all 16 samples.
  • Flow dependence is not specific to the learned reward. The F′/F gain ratio on RTLLM v2 is 80% for DiffRTL, 75% for Synthesis@4000, and 72% for SFT only. On Pluto all three methods retain about 88%. The shrinkage under F′ is comparable across methods, which indicates that it reflects the flow rather than the reward source.

What Table R8 shows and what it does not. Part of the gain reported under F is removed by passes that a standard flow runs, and the lead of DiffRTL over the synthesis-rewarded baseline remains. Table R8 does not settle the DC question. DC is stronger than F′, and gains are expected to shrink further. We make no claim about DC.

RTL-OPT. Among our three benchmarks, RTL-OPT is the one whose headroom was validated under DC by its authors: 35 of its 36 expert-optimized references remain better than the suboptimal versions under compile_ultra at a 1 ns clock (Lu et al., 2026). On this benchmark DiffRTL improves 26 designs under F and 21 under F′, against 18 and 14 for Synthesis@4000. This shows that the headroom of the benchmark is not a Yosys artifact. It does not show that our rewrites hold under DC.

Scope of the claim. The paper's claim is relative to a flow F (Section 3): a differential reward model with audits replaces most calls to F during RL. The method is not bound to Yosys. With a DC license, the corpus and audits would call DC instead, and the value of replacing online DC calls would be higher because license seats cap the number of parallel calls. The revised paper states the flow-dependence limitation in the main text, where Table R8 is also reported. The returned rewrites are in the anonymous repository, and RTL-OPT ships DC scripts, so a DC evaluation of our outputs needs no further artifact from us.

W2. The reward model sees at most 120 lines of local context. Please report area, delay, and power separately.

Table R6 reports the three metrics separately, and Tables R7a to R7d analyze delay. The data confirm the concern.

  • Delay is the weakest metric. In-family delay AUC is 0.731 against 0.858 for area and 0.852 for power.
  • Local context matters. Delay AUC is 0.763 when the slice contains the whole module and 0.688 otherwise (Table R7b). The whole module is visible for 72.8% of the Pluto base designs, 48.0% of RTLLM v2, and 66.7% of RTL-OPT (Table R7c). Modules that fit the slice also tend to be smaller, so this split is suggestive and does not isolate the effect of context.
  • The differential formulation still leads on delay. In-family delay AUC is 0.731 against 0.541 for the absolute formulation, and the lead holds in every split. Outside the in-family split, the absolute formulation is at chance or below on delay (0.506 and 0.449).
  • Within-group comparison is what the policy consumes. Eq. (6) standardizes rewards over the G rollouts of one design, so the update depends on how the reward model orders and spaces rewrites of the same base. Pairwise ranking AUC within sibling groups measures this property. A uniform optimism bias is measured separately by the audits (0.68 pp on delay before repair, Table R7d).

Every reported PPA value comes from real synthesis, where delay on RTLLM v2 improves by 5.0% for DiffRTL against 2.4% for Synthesis@4000 (Table 1 configurations, which are not step-matched). The revised paper adds per-metric AUC to Section 5.4 and lists local context as a limitation.

AuthorsOpenReview ↗

Response to Reviewer Arac (Part 2/2)

W3. Building and training the reward model costs time and compute, and prediction errors need repeated synthesis checks.

Table R4 lists every cost by ledger.

  • The repeated synthesis checks are the audits. The audits are the entire online budget of 640 calls and are already counted in B_train.
  • Costs specific to DiffRTL are 56,618 corpus synthesis calls (42.6% of the corpus), 1,142 CPU·h of formal verification, 8.3 A100·h of edit generation, 38.4 A100·h of reward-model training, and 640 calls plus 7.4 A100·h per training run.
  • The other 57.4% of the corpus feeds the SFT warm start shared by every trained row of Table 1.
  • DiffRTL trades GPU time for synthesis time. One DiffRTL update takes 1.35 min of wall clock against 26.4 min for one synthesis-rewarded update (notes under Table R1).
  • At equal updates the two arms reach equal quality within one standard deviation (Table R2), and the synthesis arm uses about 124 times more synthesis calls at 64×8. At 19,936 calls the synthesis arm is still 2.6 points below the final DiffRTL policy.
  • After the one-time costs are counted, the break-even run count depends on the synthesis-rewarded run used as comparator. It ranges from 2 to 13 at the seed-split accounting and from 4 to 29 when all corpus costs are charged to DiffRTL (table under Q3). The count uses CPU time and excludes the GPU time listed above. One checkpoint starts the 30 runs of Table R5.

Q1. Do the gains over the baseline remain under DC?

We cannot test DC without a license. Under the stock Yosys synth recipe F′, the lead of DiffRTL over Synthesis@4000 remains (Table R8).

BenchmarkLead under FLead under F′
Pluto eff@1 (points)6.85.8
RTLLM v2 PPA improvement (points)12.010.2
RTL-OPT improved designs87

Q2. How accurate is the reward model on delay alone?

Delay ranking quality is listed below (Tables R6 and R7a).

SplitDifferential, all pairsDifferential, above-floor pairsAbsolute, all pairs
in-family (AUC)0.7310.7820.541
cross-family (AUC)0.6760.7310.506
RTLOpt (pairwise accuracy)0.6020.6550.449

This result is consistent with a delay head that carries usable training signal. A policy trained on the same reward model with delay-weighted λ = (0.2, 0.6, 0.2) improves realized delay on RTLLM v2 by 9.2% against 6.7% for the uniform-weight policy selected with the same λ (Table R9, 1 seed, preliminary).

Q3. How much does DiffRTL save over direct synthesis at the same optimization quality?

At every step-matched checkpoint in Table R2 the two arms reach the same eff@1 within one standard deviation, so the step-matched ratio is also the equal-quality ratio. The synthesis arm uses about 124 times more synthesis calls than the DiffRTL arm at 64×8 and about 15.6 times more at 8×8. At 19,936 calls (50 updates at 64×8), the synthesis arm is still 2.6 points below the final DiffRTL policy (53.6 ± 1.4). A linear extrapolation of the last measured slope gives about 31K calls to match, which is a lower bound because the gain per update declines in both arms. If step-matched parity holds to 200 updates, the synthesis arm needs about 80K calls.

After the one-time costs are counted, the break-even run count is n = C_fix / (C_syn − 12.8) in CPU·h, rounded up (notes under Table R4).

Synthesis-rewarded comparatorCallsPluto eff@1n, seed-splitn, all corpus
64×8, 200 updates (not run, parity assumed)about 80Knot run24
64×8, 50 updates (measured)19,93651.0 ± 1.6614
8×8, 200 updates (measured)9,98350.5 ± 1.31329

The count uses CPU time only. At the seed-split accounting, DiffRTL also adds 38.4 and 8.3 A100·h once (reward-model training and edit generation) and 7.4 A100·h per run (Table R4). One checkpoint starts the 30 runs of Table R5 and one run with a new objective (Table R9, 1 seed, preliminary) without rebuilding the corpus.

Reviewer_xd93OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

the paper proposed RL framework for RTL design, where the core contribution is good diff reward modelling, leading to accuracy and efficiency improvement.

Critical strengths

use domain knowledge in RTL area, to better model reward model

Critical weaknesses

limited novelty in RL methodology or benchmarking contribution.

Questions for authors

if reward on action can be modelled, why not considered PPO where value estimation could also be possible?

1 comment
AuthorsOpenReview ↗

Response to Reviewer xd93

We thank the reviewer for the comments. Tables R1 to R9 are in the general response.

W1. Limited novelty in RL methodology or benchmarking contribution.

We claim neither a new policy-gradient algorithm nor a new benchmark. The contribution lies on the reward side of RL and has two parts.

  1. Differential formulation. The reward model prices the relative PPA effect of an edit instead of the absolute PPA of a design. With identical backbone and training pairs, the differential model exceeds the absolute model by 16.0 AUC points (Table 2). The lead holds on area, delay, and power separately (Table R6).
  2. Audited surrogate reward. The true objective is executable but expensive. Sparse synthesis audits on current-policy rollouts therefore measure reward over-optimization exactly and repair the reward model. Without audits the gap between predicted and realized improvement reaches 5.7 pp, and with audits the gap stays below 1 pp (Figure 3). Preference-based reward models cannot query their true objective and cannot be audited in this way.

New evidence supports the recipe. At matched updates, policies trained with the learned reward and with the synthesis reward are within one standard deviation (Table R2).

Q1. If the reward on actions can be modelled, why not PPO with value estimation?

In short, a critic cannot remove the synthesis call, and our objective is already the PPO clipped surrogate with a group-mean baseline in place of the critic.

  1. A value function cannot replace the reward. A value function is specific to the current policy and is trained from rewards, so PPO with a critic still needs one synthesis call per rollout. The reward model predicts the policy-independent effect of an edit and can therefore be trained offline, audited against synthesis, and reused across the 30 policy-training runs of the paper and this response (Table R5). This is the core difference.
  2. The objective is already PPO without a critic. Eq. (16) is the PPO clipped surrogate with ε = 0.2. The critic baseline is replaced by the mean reward of the G = 8 rollouts of the same design. With inner epochs = 1, the clip does not activate (Appendix D.2).
  3. A critic adds little in this setting. Each episode is one complete rewrite with one terminal reward, so no bootstrapping takes place. The group mean is a Monte Carlo estimate of the state value that a critic would have to learn.
  4. A fixed optimizer isolates the reward source. The synthesis-rewarded baseline follows ChipSeek-R1, which uses GRPO. The reward interface is independent of the optimizer, so a learned critic can be added without changing the framework.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for the constructive comments. New results are collected in Tables R1 to R9 and are cited by number in the individual responses. New training runs use three seeds unless marked otherwise, on the hardware of the paper. Every quality number comes from real simulation, formal equivalence checking, and synthesis.

Summary of new evidence.

  1. Step-matched parity (Table R2). At matched updates, policies trained with the synthesis reward and with the DiffRTL reward reach the same Pluto eff@1 within one standard deviation. The synthesis arm spends about 398 synthesis calls per update against 3.2 audit calls at the 64×8 batch (ratio 124:1) and about 50 against 3.2 at the 8×8 batch (ratio 15.6:1). The gap in Table 1 therefore comes from the number of updates that each method can afford at a given online budget.
  2. Budget reallocation (Table R3). Redistributing the 4,000-call budget over more updates gains at most 1.2 points. The best reallocation stays 5.6 points below DiffRTL and uses 4,000 synthesis calls against 640 audit calls.
  3. Cost and reuse (Tables R4, R5, and R9). The ledger separates shared costs from DiffRTL-specific costs and gives the break-even run count for each comparator. One reward-model checkpoint starts all 30 policy-training runs of the paper and this response without a corpus rebuild. These runs share the policy backbone and the instance pool. One further run trains a new objective (1 seed, preliminary).
  4. Limitations confirmed by the new data (Tables R6 to R8). Delay is the weakest metric of the reward model. Gains measured under F shrink under the stock Yosys synth script for every method, and the order of methods is unchanged. We have no Design Compiler result. The revised paper states the delay and flow limitations.

Two similarly named sets. RTL-OPT (Lu et al., 2026; 36 pairs) is a policy benchmark of Table 1. RTLOpt (120 triples released with CODMAS) serves only as the external test split of the reward model in Table 2.

Reviewer pointEvidence
bh72 W1, Q1: completed updates and step-matched referenceTables R1, R2
bh72 Q2: budget reallocationTable R3
Arac W3, Q3 and bh72 W2, Q3: full cost, break-even, reuseTables R4, R5, R9
Arac W2, Q2: per-metric accuracy and local contextTables R6, R7, R9
Arac W1, Q1: stronger synthesisTable R8
xd93 W1, Q1: novelty and choice of optimizerText response, Tables R2 and R5

Baselines and step-matched comparison (Tables R1 to R3)

Table R1. Synthesis-rewarded baselines: budget use and completed policy updates.

SettingBudget (calls)Billed calls per seedCompleted updatesLast update
Synthesis@640640637 / 639 / 6362Update 2 prices 37 to 39 of 64 instances
Synthesis@20002,0001,988 / 1,994 / 1,9825Prices 62 to 64 of 64 instances
Synthesis@40004,0003,964 / 4,000 / 3,95110Seed 2 exhausts the budget during update 10
Synthesis@8000 (Figure 2)8,0007,938 / 7,992 / 7,901208 to 99 calls remain after update 20
  • Schedule. All settings follow the 200-update cosine schedule of Table 12 with 5 warm-up updates and stop when the budget ends. Synthesis@640 stops inside the warm-up phase and Synthesis@2000 stops at the last warm-up update. The schedule is shared by the two arms: the DiffRTL arm follows the same schedule and reaches 46.5 ± 1.3 at 10 updates (Table R2). Table R3 removes this constraint for the baseline by shortening the warm-up and holding the learning rate at its peak.
  • Partial batch. When the budget ends inside an update, fully priced instances enter the GRPO update as usual. Unpriced instances receive r_fail = −1, which gives zero advantage under Eq. (6) and no policy gradient (Appendix D.2). A new update starts only if the remaining budget can price at least half the batch (32 of 64 instances at the 64×8 configuration). Under this rule, update 2 of Synthesis@640 started and priced 37 to 39 of 64 instances, and Synthesis@8000 did not start update 21 with 8 to 99 calls left.
  • Wall clock. One synthesis-rewarded update at 64×8 takes 26.4 min on average (median 24.1, p95 35.8). One DiffRTL update takes 1.35 min (4.5 h over 200 updates).

Table R2. Step-matched comparison on Pluto, eff@1 (%), mean ± std over three seeds. The two arms share updates, batch, schedule, warm start, and seeds. Only the reward source differs. The 10-update row reports the Table 1 Synthesis@4000 run. The 20-update row reports the Figure 2 Synthesis@8000 run. The 50-update row extends the synthesis-rewarded training beyond the original budget.

UpdatesBatchSynthesis reward: cumulative callsSynthesis reward: eff@1DiffRTL: cumulative audit callsDiffRTL: eff@1
1064×83,97246.8 ± 1.73246.5 ± 1.3
2064×87,94448.1 ± 1.66448.5 ± 1.3
5064×819,93651.0 ± 1.616051.1 ± 1.2
508×82,50647.7 ± 1.316047.5 ± 1.1
1008×85,00248.8 ± 1.532048.3 ± 1.7
2008×89,98350.5 ± 1.364049.8 ± 1.2
  • At every 64×8 checkpoint, one update costs about 398 synthesis calls against 3.2 audit calls, a ratio of about 124:1. At the 8×8 batch, one update costs about 50 synthesis calls against 3.2 audit calls, a ratio of about 15.6:1.
  • The 64×8 synthesis arm stops at 50 updates (about 22 h per seed). A 64×8 reference at 200 updates would need about 80K calls per seed.
  • The 8×8 runs use one eighth of the Table 12 batch, so absolute values are below Table 1. At 200 updates with the 8×8 batch, the RTLLM v2 PPA improvement is 24.6 ± 2.0% with synthesis reward and 23.1 ± 1.9% with DiffRTL.
  • At the 8×8 batch, the synthesis arm is nominally ahead at all three checkpoints (by 0.2, 0.5, and 0.7 points), and the margin grows with the number of updates. With three seeds, the comparison excludes a large per-update cost but cannot exclude a cost near one point. The 64×8 comparison, which shows tighter agreement, reaches 50 updates.

Table R3. Reallocation of the 4,000-call budget. Row 1 keeps the schedule of Table 12. Rows 2 to 5 use 1-update warm-up with constant peak learning rate (2-update warm-up for 8×8) instead of the cosine schedule of Table 12. Row 5 also raises the learning rate. G = 8 is fixed throughout.

Instances × GLRCompleted updateseff@1
64×8 (default allocation)5×10⁻⁶1046.8 ± 1.7
32×85×10⁻⁶2047.4 ± 1.4
16×85×10⁻⁶4047.6 ± 1.0
8×85×10⁻⁶7948.0 ± 1.2
16×82×10⁻⁵4047.6 ± 1.1

Rows 2 to 5 of Table R3 shorten the warm-up and hold the learning rate at its peak, and the best of these rows (8×8, 48.0) gains 1.2 points over the default allocation. But the best reallocation is still 5.6 points below DiffRTL (53.6) while consuming 4,000 synthesis calls against 640 audit calls.

Cost and reuse (Tables R4 and R5)

Table R4. Cost ledger. Shared costs feed the SFT warm start used by every trained row of Table 1.

ItemSynthesis callsCPU·hA100·hBorne by
Corpus synthesis, seeds used by the SFT set76,3821,528Shared
Corpus synthesis, remaining seeds56,6181,132DiffRTL
LLM edit generation11.1 / 8.3Shared / DiffRTL
Formal verification1,540 / 1,142Shared / DiffRTL
SFT training18.6Shared
Reward-model training38.4DiffRTL
Per run, DiffRTL: audits, RM inference and updates64012.87.4DiffRTL
Per run, DiffRTL: generation and backpropagation, 200 updates28.8DiffRTL
Per run, synthesis reward at 50 updates (measured)19,936399Baseline
  • Split rule. A seed is shared when at least one of the 3 × 10^4 SFT examples comes from the seed. Shared seeds number 10,247 of 18,400 and account for 57.4% of the corpus calls. Formal verification and LLM edit generation follow the same split.
  • Trade. DiffRTL spends more GPU time and less synthesis time than synthesis-rewarded training.
  • Common per-update cost. Generation and backpropagation cost the same per update in both arms, because the loop is identical apart from the reward source (Appendix D.3). The 28.8 A100·h row is therefore not specific to DiffRTL at matched updates, and the baseline row lists synthesis only.
  • Saving at equal quality. At every checkpoint in Table R2 the two arms reach the same eff@1 within one standard deviation, so the step-matched call ratio is also the equal-quality ratio. The synthesis arm uses about 124 times more synthesis calls than the DiffRTL arm at 64×8 and about 15.6 times more at 8×8. At 19,936 calls (50 updates at 64×8) the synthesis arm is still 2.6 points below the final DiffRTL policy. A linear extrapolation of the last measured slope gives about 31K calls to match. This is a lower bound, because the gain per update declines in both arms (Table R2). If step-matched parity holds to 200 updates, the synthesis arm needs about 80K calls.
  • Break-even. The count uses CPU time, the resource that synthesis consumes. The break-even run count is n = C_fix / (C_syn − 12.8), rounded up. C_fix is the one-time CPU time specific to DiffRTL, C_syn is the synthesis CPU time of one synthesis-rewarded run, and 12.8 CPU·h is the audit cost of one DiffRTL run. At the seed-split accounting, C_fix is 2,274 CPU·h: 1,132 of corpus synthesis plus 1,142 of formal verification, which doubles the DiffRTL-specific CPU time. When all corpus synthesis and all formal verification are charged to DiffRTL, C_fix is 5,342 CPU·h. The result depends on the synthesis-rewarded run used as comparator.
Synthesis-rewarded comparatorCallsC_syn (CPU·h)Pluto eff@1n, seed-splitn, all corpus
64×8, 200 updates (not run, parity assumed)about 80Kabout 1,600not run24
64×8, 50 updates (measured)19,93639951.0 ± 1.6614
8×8, 200 updates (measured)9,98320050.5 ± 1.31329
  • The final DiffRTL policy reaches 53.6 ± 1.4, above both measured comparators. The 30 runs of Table R5 exceed every entry of the table, with a narrow margin for the last cell.
  • GPU time is outside this count. At the seed-split accounting, DiffRTL adds 38.4 A100·h of reward-model training and 8.3 A100·h of edit generation once, and 7.4 A100·h per run. In wall clock, one DiffRTL run of 200 updates takes 4.5 h, against about 22 h for the 50-update synthesis-rewarded run at 64×8.

Table R5. Policy-training runs that start from one reward-model checkpoint.

SourceRunsAudit calls
Figure 2, five audit budgets from 500 to 4,000 calls, 3 seeds each1524,420
Table 3, four ablations that keep the reward model, 3 seeds each124,320
Rebuttal, 8×8 DiffRTL arm of Table R2, 3 seeds31,920
Total3030,660
  • The Table 1 seeds are the 640-call point of Figure 2. The four ablations are no audit, audit w/o RM update, no functional simulation gate, and audit interval N = 40.
  • No run rebuilds the corpus. Corpus plus audits total 163,660 synthesis calls, or 5,455 calls per run when the entire corpus is charged to DiffRTL. For comparison, one synthesis-rewarded run of 50 updates consumes 19,936 calls (Table R2). The same checkpoint also trains a policy for a new delay-weighted objective (Table R9, 1 seed, preliminary).
  • Scope of the reuse evidence. The 30 runs share the policy backbone, the instance pool, and the uniform objective. They differ in audit budget (Figure 2), ablation switch (Table 3), or batch size (Table R2). The 12 ablation runs are degraded configurations by design and are counted because they start from the same checkpoint. Table R9 is the only run with a different objective.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.