- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DiffRTL replaces synthesis-based PPA rewards with a differential reward model, corrected through periodic synthesis audits. On Pluto, it achieves 53.6 eff@1 versus 46.8 for synthesis-rewarded GRPO, using 640 versus 4,000 online synthesis calls. It also reports gains on RTLLM v2 and RTL-OPT.
Critical strengths
- This paper addresses an important practical bottleneck: expensive synthesis feedback limits RL training.
Critical weaknesses
- Baseline coverage leaves the quality–efficiency tradeoff unclear. The synthesis baselines receive roughly 1.6/5/10 updates, versus 200 for DiffRTL. This is meaningful for online-budget efficiency, but a step-matched reference would clarify the effect of replacing real rewards. It is also unclear whether alternative budget allocations could strengthen the synthesis baselines.
- Overall cost savings and amortization need stronger evidence. Corpus construction costs 133k synthesis calls, while the headline savings concern online synthesis. Since the baselines share an SFT initialization derived from this corpus, common data costs and the reward model’s additional costs should be separated. Reuse of the trained reward model for additional policy-training runs is not demonstrated.
Questions for authors
- Could the authors report actual completed updates and partial-batch handling for each synthesis baseline, and provide a step-matched reference?
- Could the synthesis baselines improve by distributing their budgets over more updates?
- After separating shared SFT costs from reward-model costs, when does DiffRTL become cheaper overall? Can the same reward model support additional policy-training runs without rebuilding the corpus?
Response to Reviewer bh72
We thank the reviewer for the comments. Tables R1 to R9 are in the general response.
W1. Baseline coverage leaves the tradeoff between quality and efficiency unclear.
We agree that Table 1 compares methods at a matched online budget, where the baselines complete far fewer updates. Table R1 reports the completed updates (2, 5, 10, and 20), and the revised Table 1 adds a policy-updates column.
Step-matched reference (Table R2). With identical updates, batch, and schedule, the two reward sources differ by at most 0.7 eff@1 points across six checkpoints. Each difference is within one standard deviation. At the 8×8 batch, the synthesis arm is nominally ahead at all three checkpoints. With three seeds, the comparison excludes a large per-update cost but cannot exclude a cost near one point. The Table 1 gap comes from the updates that each method can afford: 200 updates at 64×8 for DiffRTL and 10 for Synthesis@4000.
Scope of the reference. The 64×8 synthesis arm reaches 50 updates (about 22 h per seed), and the 200-update comparison uses an 8×8 batch.
Alternative budget allocations. See Q2.
W2. Overall cost savings and amortization need stronger evidence.
Cost separation (Table R4). The SFT warm start shared by all trained rows consumes 76,382 of the 133,000 corpus synthesis calls (57.4%). The remaining 56,618 calls (42.6%), 1,142 CPU·h of formal verification, 8.3 A100·h of edit generation, and 38.4 A100·h of reward-model training are specific to DiffRTL.
Reuse (Table R5). One reward-model checkpoint starts all 30 policy-training runs of the paper and this response without any corpus rebuild. The runs share the policy backbone and the instance pool, and differ in audit budget, ablation switch, or batch size. Corpus plus audits total 163,660 synthesis calls, or 5,455 calls per run when the entire corpus is charged to DiffRTL. For comparison, one synthesis-rewarded run of 50 updates consumes 19,936 calls (Table R2). The same checkpoint also trains a policy for a different objective (Table R9, 1 seed, preliminary), which is the only reuse beyond the paper's own configuration. Reuse across policy backbones is not tested.
The revised paper summarizes the cost ledger in Section 5.3, with the full ledger in Appendix J.2, and states that the headline savings concern online synthesis.
Q1. Completed updates, partial-batch handling, and a step-matched reference.
Completed updates (Table R1).
Synthesis@640 completes 2 updates but the second update prices only 37 to 39 of 64 instances. Synthesis@4000 seed 2 exhausts the budget during update 10.
Partial batches. When the budget ends inside an update, fully priced instances enter the update and unpriced instances contribute no gradient. A new update starts only if the remaining budget can price at least half the batch. Under this rule, Synthesis@8000 did not start update 21 with 8 to 99 calls left.
Step-matched reference. Table R2, summarized under W1.
Q2. Could the synthesis baselines improve by distributing their budgets over more updates?
Table R3 shows five allocations of the same 4,000-call budget with varying instances per update (G = 8 is fixed throughout).
Rows 2 to 5 use a shortened warm-up and constant peak learning rate. The 8×8 setting with the most updates (79) reaches 48.0, 1.2 points above the default 64×8 allocation (46.8). But the best reallocation remains 5.6 points below DiffRTL (53.6) while consuming 4,000 synthesis calls against 640 audit calls. The gap is not caused by how the budget is distributed.
Q3. When does DiffRTL become cheaper overall, and can the same reward model support additional policy-training runs?