- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Veri-eRank selects Verilog SFT data by comparing a base model with a reference model adapted on the full corpus. Each example’s code is split into AST spans weighted by instruction relevance, and the score is the drop in spectral entropy of these span representations after adaptation, averaged over resampled span views with a penalty on their variance. Selecting 20% of a 12k corpus, it beats six selection baselines and full-data SFT on three benchmarks with two 7-8B code models.
Critical strengths
- Consistent results with proper statistics. Gains hold on all six model-benchmark pairs over five seeds, and paired tests are significant against most baselines.
- Reasonable ablations. Removing instruction weights, span-level resampling or the variance term each costs 2-5 points, which supports the main design choices.
Critical weaknesses
The method is interesting and the results look good, but some controls are missing and a few claims are stronger than the evidence.
- No random baseline. A random 20% subset is a basic control for data selection, and it is often surprisingly competitive. Here random subsets are only used in the analysis in Section 3.5, not as a training baseline.
- No validation split. The paper sweeps the selection budget from 5% to 40% on the same test benchmarks and reports 20% as the best point. It is also unclear how the hyperparameters in Table 3 were chosen. Please clarify which choices were fixed beforehand and which were made after reviewing test performance.
- Full SFT comparison. On Qwen, the gain over Full SFT is only 1.05 points, and there is no significance test for this comparison. The same issue applies to Seed-Coder.
- Uneven baselines. SPICE uses a 0.5B proxy for Qwen while Veri-eRank uses the 7B model itself. This follows SPICE’s setup, but it is still not an equal comparison in model capacity. VeriSelector also performs much worse than the other baselines, so I would like to see some evidence that the reimplementation is faithful.
- Novelty and theory. The spectral score is closely related to Diff-eRank. The RTL-specific structural weighting and stability terms are new, but the theoretical result mainly shows that entropy goes down under the paper’s assumption. It does not really explain why examples with larger entropy reduction should be better training data.
Questions for authors
Q1. How does Veri-eRank compare with random 20% subsets trained under the same protocol and matched seeds? A clear margin over random selection would substantially strengthen the data-selection claim.
Q2. How were the 20% selection budget and the hyperparameters in Table 3 chosen? Which of these choices, if any, were made after inspecting performance on the downstream test benchmarks?
Q3. Is the improvement over Full SFT statistically significant under a paired test across the same five seeds?
2 comments
Official Response to Reviewer tBkf [2/2]
W3 & Q3: Paired Statistical Significance Against Full SFT
We thank the reviewer for prompting a direct statistical comparison with Full SFT. The improvement over Full SFT is statistically significant on Seed-Coder (p = 0.018). On Qwen, the observed mean gain is positive but does not reach statistical significance (p = 0.292). We will add these significance test results and the corresponding analysis to Appendix B.3.
| Backbone | Mean Gain vs. Full SFT (pp) | -value | |
|---|---|---|---|
| Qwen2.5-Coder-7B | +1.05 | 1.21 | 0.292 |
| Seed-Coder-8B | +1.98 | 3.88 | 0.018 |
Moreover, the primary goal of data selection at a 20% budget is data efficiency: matching full-corpus performance while reducing final-training data by 80%. As shown in Table 1, whereas other 20% selection baselines generally fall short of Full SFT, Veri-eRank maintains or modestly improves upon full-corpus performance across both backbones with lower cross-seed variance.
W4: Baseline Fairness and Implementation Fidelity
We sincerely appreciate the reviewer's careful attention to our implementation details.
-
SPICE: We additionally evaluated SPICE using a capacity-matched Qwen2.5-Coder-7B proxy. Veri-eRank still significantly outperforms SPICE-7B by 5.62 pp (), while SPICE incurs higher filtering overhead. On Seed-Coder-8B, SPICE already uses a matched 8B proxy, where Veri-eRank achieves a 2.24 pp gain.
-
VeriSelector: We reproduced VeriSelector based on the authors' official open-source implementation and carefully rechecked its data selection, structural feature extraction, training, and evaluation against the original paper. Implementation details are provided in Appendix B.2, and the shared training and evaluation scripts will be available in our anonymous repository.
W5: Theoretical Link to Training Value and Distinction from Diff-eRank
We sincerely thank the reviewer for carefully examining our theoretical motivation and raising this insightful point.
-
Distinction from Diff-eRank. Unlike Diff-eRank, which measures full-sequence representations for model-level evaluation, Veri-eRank incorporates instruction-conditioned AST spans and cross-view stability for sample-level selection. Our controlled ablations in Appendix Tables 9–11 confirm the importance of these designs, showing that Veri-eRank substantially outperforms the adapted sample-level Diff-eRank baseline across both backbones.
-
Empirical link to downstream training value. To verify the relationship between our theoretical analysis and downstream sample value, we supplemented a ranking experiment on equal-sized 20% subsets drawn from the High, Middle, and Low score tiers alongside random selection, reporting five-seed mean and standard deviation of Macro Pass@1 below.
| Subset Tier (20% Budget) | Qwen2.5-Coder-7B Macro Pass@1 (%) | Seed-Coder-8B Macro Pass@1 (%) |
|---|---|---|
| High (Veri-eRank) | ||
| Middle | ||
| Low | ||
| Random-20% |
Across both backbones, macro-average performance consistently follows the order High Middle Low, providing direct empirical evidence that our ranking signal effectively identifies training subsets with higher downstream utility. We will include this supplementary experiment in the revised appendix.
Official Response to Reviewer tBkf [1/2]
We sincerely thank Reviewer tBkf for the thoughtful assessment, generous encouragement, and constructive feedback. We deeply appreciate your recognition of our consistent performance gains across models and benchmarks, and we respectfully address each of your points below.
W1 & Q1: Random 20% Subset Baseline
We thank the reviewer for highlighting this essential control. To address this, we evaluated five randomly sampled 20% subsets per backbone under the same SFT protocol and five training seeds. Veri-eRank achieves higher mean Macro Pass@1 than all five sampled random subsets on both backbones. The subset-specific paired tests yield unadjusted p-values below 0.024 on Qwen and 0.018 on Seed-Coder. Across the five subsets, the average Random-20% performance is 47.12% on Qwen and 50.02% on Seed-Coder. The table below reports our default random subset per backbone, and we will include the results for the remaining subsets in the revised appendix.
W2 & Q2: Selection of the 20% Budget and Measurement Hyperparameters
We thank the reviewer for this important question regarding our experimental protocol. Table 3 lists the measurement configuration used in the reported experiments.
20% budget. The 20% budget follows prior work such as VeriSelector (Shen et al., 2026). Section 3.3 reports performance across budgets from 5% to 40%.
Core scoring parameters. We set the stability penalty coefficient β=0.5 before inspecting downstream benchmark results to balance the mean response and cross-view stability. Appendix Table 12 reports sensitivity to β. We use a single epoch for the M1 reference model to balance computation cost and model performance. We supplement the sensitivity analysis of M1 over varying training epochs in the table below, and this analysis will be included in the revised appendix.