acceptodds
Under review as a conference paper at ICLR 2027

Veri-eRank: Verilog Data Selection via Stable Structural Adaptation Responses

Abstract

Code language models are increasingly adapted for register-transfer-level (RTL) Verilog generation from natural-language specifications. Simulation-validated corpora provide abundant executable supervision, but functional correctness does not make every RTL example equally useful to a given model: an example's adaptation value depends on the model's current capabilities. Data selection should therefore identify examples that most effectively advance the model's specification-to-design ability. Existing methods, whether loss- or gradient-based, score candidates from signals aggregated over the full sequence, without explicitly assessing whether learning from an example strengthens the model's representations of the specification-relevant RTL structures. We introduce Veri-eRank, which ranks RTL examples by changes in specification-relevant structural representations from a base model to an adapted model. Veri-eRank represents each RTL program using AST-derived structural spans, including declarations, assignments, event controls, and control flow, and weights each span by its relevance to the specification. Specifically, Veri-eRank measures the spectral entropy of the structural representations before and after adaptation, and favors examples that reduce spectral entropy after adaptation. To avoid scores driven by a particular local observation, Veri-eRank evaluates resampled structural views and favors responses that are both strong and consistent. Theoretically, our analysis shows that preserving structural directions while reducing isotropic variation lowers spectral entropy after adaptation, motivating our spectral-response score. Under a 20% data budget, Veri-eRank attains the highest mean Pass@1 among the compared data selection methods in all six settings across two models and three RTL benchmarks. Five-seed macro-average Pass@1 reaches for Qwen2.5-Coder and for Seed-Coder, exceeding SPICE by 5.42 and 2.24 percentage points and Full SFT by 1.05 and 1.98 points, respectively.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
RTL Code Generation, Data Selection, Representation Learning, Instruction Tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tBkfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Veri-eRank selects Verilog SFT data by comparing a base model with a reference model adapted on the full corpus. Each example’s code is split into AST spans weighted by instruction relevance, and the score is the drop in spectral entropy of these span representations after adaptation, averaged over resampled span views with a penalty on their variance. Selecting 20% of a 12k corpus, it beats six selection baselines and full-data SFT on three benchmarks with two 7-8B code models.

Critical strengths
  • Consistent results with proper statistics. Gains hold on all six model-benchmark pairs over five seeds, and paired tests are significant against most baselines.
  • Reasonable ablations. Removing instruction weights, span-level resampling or the variance term each costs 2-5 points, which supports the main design choices.
Critical weaknesses

The method is interesting and the results look good, but some controls are missing and a few claims are stronger than the evidence.

  • No random baseline. A random 20% subset is a basic control for data selection, and it is often surprisingly competitive. Here random subsets are only used in the analysis in Section 3.5, not as a training baseline.
  • No validation split. The paper sweeps the selection budget from 5% to 40% on the same test benchmarks and reports 20% as the best point. It is also unclear how the hyperparameters in Table 3 were chosen. Please clarify which choices were fixed beforehand and which were made after reviewing test performance.
  • Full SFT comparison. On Qwen, the gain over Full SFT is only 1.05 points, and there is no significance test for this comparison. The same issue applies to Seed-Coder.
  • Uneven baselines. SPICE uses a 0.5B proxy for Qwen while Veri-eRank uses the 7B model itself. This follows SPICE’s setup, but it is still not an equal comparison in model capacity. VeriSelector also performs much worse than the other baselines, so I would like to see some evidence that the reimplementation is faithful.
  • Novelty and theory. The spectral score is closely related to Diff-eRank. The RTL-specific structural weighting and stability terms are new, but the theoretical result mainly shows that entropy goes down under the paper’s assumption. It does not really explain why examples with larger entropy reduction should be better training data.
Questions for authors

Q1. How does Veri-eRank compare with random 20% subsets trained under the same protocol and matched seeds? A clear margin over random selection would substantially strengthen the data-selection claim.

Q2. How were the 20% selection budget and the hyperparameters in Table 3 chosen? Which of these choices, if any, were made after inspecting performance on the downstream test benchmarks?

Q3. Is the improvement over Full SFT statistically significant under a paired test across the same five seeds?

2 comments
AuthorsOpenReview ↗

Official Response to Reviewer tBkf [1/2]

We sincerely thank Reviewer tBkf for the thoughtful assessment, generous encouragement, and constructive feedback. We deeply appreciate your recognition of our consistent performance gains across models and benchmarks, and we respectfully address each of your points below.

W1 & Q1: Random 20% Subset Baseline

We thank the reviewer for highlighting this essential control. To address this, we evaluated five randomly sampled 20% subsets per backbone under the same SFT protocol and five training seeds. Veri-eRank achieves higher mean Macro Pass@1 than all five sampled random subsets on both backbones. The subset-specific paired tests yield unadjusted p-values below 0.024 on Qwen and 0.018 on Seed-Coder. Across the five subsets, the average Random-20% performance is 47.12% on Qwen and 50.02% on Seed-Coder. The table below reports our default random subset per backbone, and we will include the results for the remaining subsets in the revised appendix.

BackboneMethodMacro Pass@1 (%)Δ\Delta Macro (pp)95% CI (pp)pp-value
Qwen2.5-Coder-7BRandom (20%)46.91±1.3146.91 \pm 1.31+3.93[1.47,6.38][1.47, 6.38]0.01130.0113
Qwen2.5-Coder-7BVeri-eRank50.83±1.1450.83 \pm 1.14———
Seed-Coder-8BRandom (20%)49.23±1.2149.23 \pm 1.21+3.83[1.99,5.67][1.99, 5.67]0.004480.00448
Seed-Coder-8BVeri-eRank53.06±0.5753.06 \pm 0.57———

W2 & Q2: Selection of the 20% Budget and Measurement Hyperparameters

We thank the reviewer for this important question regarding our experimental protocol. Table 3 lists the measurement configuration used in the reported experiments.

  1. 20% budget. The 20% budget follows prior work such as VeriSelector (Shen et al., 2026). Section 3.3 reports performance across budgets from 5% to 40%.

  2. Core scoring parameters. We set the stability penalty coefficient β=0.5\beta=0.5 before inspecting downstream benchmark results to balance the mean response and cross-view stability. Appendix Table 12 reports sensitivity to β\beta. We use a single epoch for the M1M_1 reference model to balance computation cost and model performance. We supplement the sensitivity analysis of M1M_1 over varying training epochs in the table below, and this analysis will be included in the revised appendix.

Reference DurationQwen Spearman ρ\rhoQwen Macro Pass@1 (%)Seed-Coder Spearman ρ\rhoSeed-Coder Macro Pass@1 (%)
0.5 Epochs0.89547.09±1.3147.09 \pm 1.310.91248.69±1.4448.69 \pm 1.44
1.0 Epoch1.00050.83±1.1450.83 \pm 1.141.00053.06±0.5753.06 \pm 0.57
2.0 Epochs0.80246.43±0.9146.43 \pm 0.910.84151.06±1.6651.06 \pm 1.66
  1. Remaining numerical and sampling parameters. The top-50 probability truncation, eigenvalue cutoffs, and resampling settings are implementation choices specified in Table 3. Appendix Tables 10–11 report ablations of the structural-view and resampling design.
AuthorsOpenReview ↗

Official Response to Reviewer tBkf [2/2]

W3 & Q3: Paired Statistical Significance Against Full SFT

We thank the reviewer for prompting a direct statistical comparison with Full SFT. The improvement over Full SFT is statistically significant on Seed-Coder (p = 0.018). On Qwen, the observed mean gain is positive but does not reach statistical significance (p = 0.292). We will add these significance test results and the corresponding analysis to Appendix B.3.

BackboneMean Gain vs. Full SFT (pp)t(4)t(4)pp-value
Qwen2.5-Coder-7B+1.051.210.292
Seed-Coder-8B+1.983.880.018

Moreover, the primary goal of data selection at a 20% budget is data efficiency: matching full-corpus performance while reducing final-training data by 80%. As shown in Table 1, whereas other 20% selection baselines generally fall short of Full SFT, Veri-eRank maintains or modestly improves upon full-corpus performance across both backbones with lower cross-seed variance.

W4: Baseline Fairness and Implementation Fidelity

We sincerely appreciate the reviewer's careful attention to our implementation details.

  1. SPICE: We additionally evaluated SPICE using a capacity-matched Qwen2.5-Coder-7B proxy. Veri-eRank still significantly outperforms SPICE-7B by 5.62 pp (p=0.0028p=0.0028), while SPICE incurs higher filtering overhead. On Seed-Coder-8B, SPICE already uses a matched 8B proxy, where Veri-eRank achieves a 2.24 pp gain.

  2. VeriSelector: We reproduced VeriSelector based on the authors' official open-source implementation and carefully rechecked its data selection, structural feature extraction, training, and evaluation against the original paper. Implementation details are provided in Appendix B.2, and the shared training and evaluation scripts will be available in our anonymous repository.

W5: Theoretical Link to Training Value and Distinction from Diff-eRank

We sincerely thank the reviewer for carefully examining our theoretical motivation and raising this insightful point.

  1. Distinction from Diff-eRank. Unlike Diff-eRank, which measures full-sequence representations for model-level evaluation, Veri-eRank incorporates instruction-conditioned AST spans and cross-view stability for sample-level selection. Our controlled ablations in Appendix Tables 9–11 confirm the importance of these designs, showing that Veri-eRank substantially outperforms the adapted sample-level Diff-eRank baseline across both backbones.

  2. Empirical link to downstream training value. To verify the relationship between our theoretical analysis and downstream sample value, we supplemented a ranking experiment on equal-sized 20% subsets drawn from the High, Middle, and Low score tiers alongside random selection, reporting five-seed mean and standard deviation of Macro Pass@1 below.

Subset Tier (20% Budget)Qwen2.5-Coder-7B Macro Pass@1 (%)Seed-Coder-8B Macro Pass@1 (%)
High (Veri-eRank)50.83±1.1450.83 \pm 1.1453.06±0.5753.06 \pm 0.57
Middle47.49±2.1047.49 \pm 2.1051.41±0.9951.41 \pm 0.99
Low46.38±1.4046.38 \pm 1.4049.14±1.6249.14 \pm 1.62
Random-20%46.91±1.3146.91 \pm 1.3149.23±1.2149.23 \pm 1.21

Across both backbones, macro-average performance consistently follows the order High >> Middle >> Low, providing direct empirical evidence that our ranking signal effectively identifies training subsets with higher downstream utility. We will include this supplementary experiment in the revised appendix.

Reviewer_1Gx8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Veri-eRank, a data selection method for RTL code generation. It scores training examples using AST structural information, instruction relevance, and changes in spectral entropy between a base model and a reference-adapted model, then selects high-scoring examples for SFT under a limited data budget. Experiments on two code models and three RTL benchmarks show improvements over existing data selection methods. However, the practical efficiency gains and methodological novelty remain insufficiently established.

Critical strengths
  1. The method incorporates domain-specific structural information. Rather than directly applying a generic data selection metric, the authors incorporate RTL AST structures, instruction-conditioned weighting, and stability across structural views into the scoring mechanism. This design is reasonably well motivated for RTL generation.
  2. The results demonstrate potential benefits under a limited data budget. With a 20% data budget, Veri-eRank improves macro-average Pass@1 from 49.78% to 50.83% on Qwen2.5-Coder-7B and from 51.08% to 53.06% on Seed-Coder-8B. These results suggest that a carefully selected subset can maintain or improve performance while reducing the amount of data used in final SFT.
Critical weaknesses
  1. Reference adaptation on the full dataset undermines the practical efficiency advantage. Veri-eRank first performs one epoch of reference adaptation using the entire candidate dataset, then scores the examples and finally restarts from the original base model to perform three epochs of SFT using only the selected 20%. This design raises a fundamental question: how much computational cost does the method actually save if supervised training on the full dataset is required before data selection? The claimed 80% data reduction applies only to the final SFT stage, not to the entire pipeline. Although the authors report the time required for reference adaptation and scoring, this does not provide a complete end-to-end comparison that includes final SFT. Moreover, the reference adaptation stage already exposes the model to the entire dataset. While this may be useful for estimating sample value, it weakens the practical appeal of the method as an efficient data selection strategy.
  2. The performance gains are relatively modest, and a random selection baseline is missing. With a 20% data budget, Pass@1 improves by only 1.05 percentage points on Qwen and 1.98 percentage points on Seed-Coder. These gains are not negligible, but their practical significance is uncertain given the additional costs of full-data reference adaptation, representation extraction, and sample ranking. More importantly, the paper does not report downstream performance when randomly selecting 20% of the data under the same training configuration. Consequently, it remains unclear how much of the improvement is attributable to Veri-eRank's ranking mechanism rather than changes in training dynamics caused by using a smaller dataset. For a paper whose central contribution is data selection, this is a significant experimental omission.
  3. The methodological novelty is limited, as the main contribution appears to be applying existing quantitative techniques to RTL data selection. Spectral entropy and effective rank are established mathematical tools, and the paper builds on prior work in spectral analysis and Diff-eRank. The authors combine these techniques with RTL AST structures, instruction relevance, and stability-based penalties to construct a domain-specific data selection pipeline. However, the paper does not introduce a new principle for learning data value, a novel training objective, or a new model optimization mechanism. Its contribution primarily lies in the domain-specific design of the scoring mechanism and its empirical evaluation, rather than a substantive methodological advance in machine learning. For a top-tier machine learning venue, this level of novelty may be insufficient.
Questions for authors
  1. After accounting for reference adaptation, sample scoring, and final SFT, how many GPU-hours does Veri-eRank actually save compared with Full SFT? Does it retain a clear efficiency advantage when achieving comparable performance?
  2. What is the Pass@1 performance of a randomly selected 20% subset under the same SFT configuration and five-seed evaluation protocol? If random selection performs similarly, how should the central contribution of Veri-eRank be repositioned?
  3. Beyond combining spectral entropy changes with RTL structural information, what new principle does this work establish for data selection? Can the method deliver comparable benefits in other code generation tasks, or are its gains primarily empirical and specific to the evaluated RTL benchmarks?
5 comments
AuthorsOpenReview ↗

Official Response to Reviewer 1Gx8 [1/3]

We sincerely thank Reviewer 1Gx8 for the time, thoughtful assessment, and constructive feedback. We deeply appreciate your recognition of our domain-specific structural design and empirical gains under a limited data budget, and we respectfully address each of your concerns below.

W1 & Q1: End-to-End GPU-Hour Accounting, Compute-Matched Efficiency, and the Role of Reference Adaptation

We thank the reviewer for raising this practical question regarding end-to-end computational cost and the role of full-corpus reference exposure. To address your concerns systematically, we compiled the complete wall-clock costs and conducted compute-matched Full SFT comparisons across four training durations: 0.60.6 epochs to match final-SFT gradient updates, 1.61.6 epochs to match total updates including reference adaptation, 3.03.0 epochs as the default full-corpus baseline, and approximately 4.04.0 epochs to match single-run end-to-end wall-clock time. In the table below, the single-run time for Veri-eRank sums the one-time selection overhead from Tables 7 and 8 and the final subset SFT time.

Training ProtocolQwen Time (1 Run / 5 Runs)Qwen Macro Pass@1 (%)Seed-Coder Time (1 Run / 5 Runs)Seed-Coder Macro Pass@1 (%)
Full SFT (0.6 ep)11m / 55m45.78±1.1145.78 \pm 1.1113m / 65m50.22±1.8750.22 \pm 1.87
Full SFT (1.6 ep)30m / 150m48.96±0.9348.96 \pm 0.9333m / 165m52.06±1.1352.06 \pm 1.13
Full SFT (3.0 ep)57m / 285m49.78±2.3949.78 \pm 2.3962m / 310m51.08±0.8451.08 \pm 0.84
Full SFT (∼4.0\sim 4.0 ep)73m / 365m48.80±0.9448.80 \pm 0.9489m / 445m50.06±1.9550.06 \pm 1.95
Veri-eRank (3.0 ep)73m / 117m50.83±1.1450.83 \pm 1.1489m / 141m53.06±0.5753.06 \pm 0.57

1. End-to-end GPU-hour savings and multi-run amortization. Veri-eRank reduces total backpropagation volume by nearly half in a single run and saves more than half of the total GPU-hours across repeated training runs. Even for a single run, our pipeline requires only 1.61.6 full-corpus epoch equivalents of gradient updates compared with 3.03.0 epochs for standard Full SFT. A single Veri-eRank run takes 73/89 minutes on Qwen/Seed-Coder, compared with 57/62 minutes for Full SFT. However, when the ranking is reused across five training runs, the total time decreases from 285/310 to 117/141 minutes, respectively. Each subsequent 20% SFT run cuts training time by roughly four-fifths as analyzed in Appendix A.10. Consequently, the one-time selection overhead breaks even at just two training runs. Across five runs, Veri-eRank saves approximately 2.80 GPU-hours on Qwen and 2.82 GPU-hours on Seed-Coder, respectively.

2. Efficiency and accuracy advantage under matched single-run compute. Even when restricted to a single run without amortization, Veri-eRank consistently outperforms Full SFT under both matched gradient updates and matched wall-clock time. At matched gradient updates of 0.60.6 or 1.61.6 epochs, Full SFT fails to reach comparable accuracy on either backbone. More importantly, when given a single-run wall-clock budget matched to the nearest minute (73/89 minutes on Qwen/Seed-Coder, approximately four epochs), Full SFT does not close the accuracy gap and instead degrades relative to 3.03.0 epochs, showing that allocating compute to structure-aware selection is more effective than extending full-corpus training.

3. Why one-epoch reference probing followed by subset retraining outperforms continuous full-corpus training. The performance degradation of four-epoch Full SFT above directly explains why exposing the model to the full dataset during reference adaptation does not diminish the value of subset selection. Exposing a model to all candidate examples is not equivalent to learning optimal representations from them, as multi-epoch training on the uncurated pool repeatedly updates parameters on redundant and noisy supervision. By using a single full-corpus epoch strictly as a diagnostic probe to identify examples with stable structural consolidation and then retraining a fresh base model M0M_0 exclusively on the selected 20% subset, Veri-eRank prevents multi-epoch noise accumulation while achieving higher accuracy and lower cross-seed variance. We will add this complete end-to-end cost and compute-matched analysis to Appendix B.5.

AuthorsOpenReview ↗

Official Response to Reviewer 1Gx8 [2/3]

W2 & Q2: Random-20% Subset Baseline, Smaller-Dataset Training Dynamics, and Statistical Significance Against Full SFT

We thank the reviewer for highlighting this essential control. Following your suggestion, we evaluated five independently sampled Random-20% subsets per backbone under the identical SFT configuration and five training seeds. We also conducted two-sided paired tt-tests against the first sampled subset (our default Random-20% baseline) and three-epoch Full SFT, as summarized below.

ComparisonBaseline Macro Pass@1Gain (pp)Paired pp
Qwen vs. Random (Default)46.91±1.3146.91 \pm 1.31+3.93+3.930.01130.0113
Qwen vs. Full SFT49.78±2.3949.78 \pm 2.39+1.05+1.050.2920.292
Seed-Coder vs. Random (Default)49.23±1.2149.23 \pm 1.21+3.83+3.830.004480.00448
Seed-Coder vs. Full SFT51.08±0.8451.08 \pm 0.84+1.98+1.980.0180.018

Across five random subsets, the average Macro Pass@1 is 47.12% (range: 45.86–48.40%) on Qwen and 50.02% (range: 48.90–51.48%) on Seed-Coder. Veri-eRank achieves higher mean performance than every sampled random subset.

1. Training dynamics under the 20% data budget. Training for three epochs on a 20% subset performs 461 gradient updates, compared with 2,304 updates for three-epoch Full SFT. In all runs, the cosine learning-rate schedule is scaled to complete a full decay cycle over each run's total steps. Random-20% and the step-matched 0.6-epoch Full SFT baseline in W1 achieve lower mean accuracy than three-epoch Full SFT on both backbones.

2. Veri-eRank's performance against Random-20% and Full SFT. Under the same 20% data budget and final-SFT protocol, Veri-eRank consistently achieves higher mean performance than random selection, supporting the effectiveness of our data-ranking mechanism. Under the same 20% data budget, competing baselines in Table 1 fall below Full SFT and trail Random-20% on Qwen. In paired tt-tests against three-epoch Full SFT, Veri-eRank's improvement is statistically significant on Seed-Coder, while on Qwen the mean gain is positive but does not reach statistical significance. Nonetheless, Veri-eRank fulfills the primary goal of 20%-budget selection by matching or modestly improving upon full-corpus performance with 80% less final-training data and lower cross-seed variance. We will add the default Random-20% baseline, the five-subset results, and these paired significance tests to Table 1, Section 3.2, and Appendix B.3.

AuthorsOpenReview ↗

Official Response to Reviewer 1Gx8 [3/3]

W3 & Q3: Methodological Novelty Beyond Existing Spectral Metrics, Core Data-Selection Principle, and Generalizability to Other Code Tasks

Spectral entropy and effective rank are established quantities, and we build on the paired spectral perspective of Diff-eRank. Diff-eRank, however, is a model-level diagnostic over whole sequences. Directly applying it to our task fails: in Table 9, sample-level Diff-eRank and unstructured all-code ΔH\Delta H trail Veri-eRank by 2.17–5.08 pp and fall below Random-20% on Qwen.

The principle Veri-eRank introduces is that an example is valuable when adaptation concentrates the representations of its specification-relevant substructures, and when that concentration holds across the program rather than on a single local span. Turning a spectral difference into a well-defined measure of this property requires three controls that existing spectral metrics lack:

Matched support. Both checkpoints are evaluated on the exact same token set, weights, and full-support center, so ΔH\Delta H reflects representation change rather than a changed observation set (removing pairing costs 5.64/3.26 pp, Table 10).

Fixed role composition. Resampled views draw whole AST spans within each role and rescale weights so every role retains its full-support mass πr\pi_r. Subsampling then probes within-role geometry without shifting role proportions across views (removing this constraint costs 3.72/2.30 pp, Table 10).

Cross-view consistency. Each view is a principal submatrix of one precomputed unit-trace Gram matrix, making entropies directly comparable across views without extra forward passes. Penalizing cross-view dispersion rules out local span outliers (+4.94/3.42 pp over the mean-only score, Table 11), while measuring structural representation change outperforms ranking by answer-token loss reduction (+5.83/2.77 pp, Table 9).

In principle, Veri-eRank extends naturally to other programming languages because Algorithm 1 decouples syntax parsing from geometric scoring. Once AST nodes are mapped to structural roles, weighting, role-preserving resampling, and robust ranking operate solely on aligned spans and hidden states, so adapting to Python, C++, or Rust only requires replacing the Tree-sitter grammar and the AST-to-role mapping. We will clarify this general selection principle in the revised manuscript and evaluate broader code-generation domains in future work.

Reviewer_Fs6qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Veri-eRank for selecting instruction–Verilog examples for fine-tuning. It compares a base model with a reference model adapted for one epoch on the full corpus, ranking examples by spectral-entropy reductions in instruction-weighted, AST-derived representations, penalized by variation across structural views. Final fine-tuning starts from the original base model using the selected subset. Experiments on two backbones and three benchmarks show improvements over six selection baselines at a 20% data budget and modest gains over full-data SFT. The paper includes theoretical analysis, five-seed experiments, and extensive component ablations.

Critical strengths
  1. A concrete domain-specific contribution. Building on spectral representation analysis such as Diff-eRank, the method combines instruction-weighted RTL observations with paired structural resampling for data selection. Ablations against loss reduction, unstructured spectral responses, and an adapted Diff-eRank baseline support the usefulness of these design choices.

  2. Consistent improvements within the evaluated setting. Veri-eRank achieves the highest mean Pass@1 among the compared selection methods in all six backbone–benchmark combinations. Five-seed results and substantial component ablations provide credible empirical evidence, including macro-average gains of 5.42 and 2.24 percentage points over SPICE.

Critical weaknesses
  1. The interpretation and robustness of the selection signal are insufficiently established. Theorem 1 shows that entropy decreases when a fixed structural component is preserved while isotropic background variation is reduced. It does not establish that actual adaptation satisfies this assumption, or that larger decreases identify examples with greater downstream training value. Importantly, all rankings use a reference model trained for exactly one epoch, without an empirical justification or sensitivity analysis for this duration. Since scores measure changes along this training trajectory, different checkpoints could favor different examples. The structural-view robustness analysis and repeated final-SFT runs do not address this dependence. The missing duration analysis therefore concerns a central determinant of the ranking, not merely a peripheral hyperparameter.

  2. The practical advantage over simpler alternatives remains unclear. Selection requires full-corpus reference adaptation, representation scoring, and spectral computations. Reported selection times include these operations, but end-to-end cost comparisons with full-data SFT are missing. Gains over full-data SFT are modest, at 1.05 and 1.98 percentage points, without a corresponding paired significance analysis. There is also no downstream random-20%-subset baseline or compute-matched direct-training comparison. Equal-epoch training gives full-data SFT substantially more updates, so the comparison does not isolate selection quality from training duration. These omissions make it difficult to assess whether the additional machinery provides a compelling accuracy–cost advantage.

Questions for authors
  1. Clarification and additional experiment: How was the one-epoch reference-adaptation duration chosen? Please compare a small number of shorter and longer durations, for example 0.5, 1, and 2 epochs, reporting selected-subset overlap, downstream performance, and selection cost. Is the advantage robust across reference-training seeds?

  2. Supporting evidence: Beyond the structure-preserving noise-reduction assumption, what evidence links larger spectral responses to greater training value, rather than simply stronger representation changes along the shared full-corpus training trajectory?

  3. Additional experiments and cost reporting: How does the method compare with random 20% subsets under the same SFT protocol and direct full-corpus training under a comparable total compute budget? Please report end-to-end costs, including reference adaptation, scoring, and final SFT.

  4. Secondary diagnostic evaluation: Can compilation failures and functional failures be reported separately? Pass@1 is appropriate for single-candidate correctness, but this breakdown would clarify what improves. Evaluation under a fixed multi-candidate sampling budget would further establish the scope of the gains.

3 comments
AuthorsOpenReview ↗

Official Response to Reviewer Fs6q [1/3]

We sincerely thank Reviewer Fs6q for the time, thoughtful assessment, and constructive feedback. We deeply appreciate your recognition of our domain-specific contribution and address each of your concerns below.

W1 & Q1: Sensitivity and Robustness of Reference Adaptation

We thank the reviewer for highlighting the central role of the reference trajectory in our ranking mechanism.

1. Rationale for the one-epoch reference duration.

The one-epoch duration was set beforehand as a principled baseline in Table 3 to ensure uniform exposure across the candidate corpus. Under this setting, every candidate example participates in exactly one gradient update. A shorter trajectory leaves part of the corpus unseen by direct parameter updates, whereas a longer trajectory begins to memorize redundant or noisy examples while increasing reference training overhead.

2. Sensitivity to reference training duration.

Although rankings remain positively correlated across reference durations, downstream performance is sensitive to the reference-training horizon. One epoch performs best among the tested settings. Selection costs cover reference adaptation and candidate scoring. The one-epoch costs are the original selection timings summarized in Tables 7 and 8.

Reference DurationQwen Overlap / Spearman ρ\rhoQwen Macro Pass@1 (%)Seed-Coder Overlap / Spearman ρ\rhoSeed-Coder Macro Pass@1 (%)Selection Cost (min, Qwen / Seed-Coder)
0.5 Epochs76.39% / 0.89547.09±1.3147.09 \pm 1.3180.26% / 0.91248.69±1.4448.69 \pm 1.4455 / 69
1.0 Epoch (Ours)100.0% / 1.00050.83±1.1450.83 \pm 1.14100.0% / 1.00053.06±0.5753.06 \pm 0.5762 / 76
2.0 Epochs68.58% / 0.80246.43±0.9146.43 \pm 0.9171.35% / 0.84151.06±1.6651.06 \pm 1.6685 / 100

3. Robustness across independent reference trajectories.

Our additional experiments with independent reference-training seeds confirm that both the selected subsets and downstream gains remain stable across trajectory initializations. We trained two additional one-epoch reference checkpoints per backbone with independent random seeds. Every resulting subset achieves downstream performance that surpasses Full SFT, random selection, and all six selection baselines in Table 1.

Reference CheckpointQwen Overlap / Spearman ρ\rhoQwen Macro Pass@1 (%)Seed-Coder Overlap / Spearman ρ\rhoSeed-Coder Macro Pass@1 (%)
M1(1)M_1^{(1)} (Ours)100.0% / 1.00050.83±1.1450.83 \pm 1.14100.0% / 1.00053.06±0.5753.06 \pm 0.57
M1(2)M_1^{(2)}85.23% / 0.95149.99±0.4849.99 \pm 0.4886.45% / 0.96052.14±0.5752.14 \pm 0.57
M1(3)M_1^{(3)}84.17% / 0.94650.04±0.8550.04 \pm 0.8583.84% / 0.95351.53±1.4851.53 \pm 1.48

W1 & Q2: Linking Spectral Consolidation to Downstream Training Value

We appreciate this insightful question regarding why spectral consolidation reflects downstream training utility rather than arbitrary representation drift. We address this concern in three progressive steps:

1. Theoretical motivation: separating learnable structures from arbitrary trajectory drift. The theorem provides a conditional motivation, while the score-tier experiments provide empirical support. Our scoring rule favors examples with both strong spectral concentration and cross-view stability. Along the shared reference trajectory, examples already mastered by the base model exhibit negligible spectral change, whereas noisy or overly complex examples fail to consolidate consistently across AST spans and induce large cross-view dispersion. By penalizing view dispersion alongside mean spectral reduction as analyzed in Appendices A.5 to A.9, Veri-eRank filters out both trivial and unstable trajectory drift to retain examples whose specification-relevant structures consolidate reliably.

2. Ablations: ruling out raw trajectory magnitude. If high scores merely reflected stronger changes along the shared trajectory, metrics that directly rank candidates by trajectory magnitude on the exact same base and reference checkpoints would perform comparably. However, our controlled ablations in Section 3.4 and Appendix B.6 show that ranking by answer-token loss reduction, unweighted all-code spectral change, full-sequence effective-rank difference in Table 9, or unpenalized mean spectral change in Table 11 substantially degrades downstream accuracy on both backbones. These results confirm that raw representation change along the shared trajectory is insufficient for effective selection.

AuthorsOpenReview ↗

Official Response to Reviewer Fs6q [2/3]

3. Direct empirical verification: score-quantile subset training. We further trained fresh base models on subsets drawn from different score tiers of the Veri-eRank ranking under the identical 20% budget and five-seed SFT protocol. As shown below, mean downstream accuracy increases monotonically from the low-score tier to the middle and top tiers on both backbones.

Score Tier (20% Budget)Qwen2.5-Coder-7B Macro Pass@1 (%)Seed-Coder-8B Macro Pass@1 (%)
Low Tier (Bottom 20%)46.38±1.4046.38 \pm 1.4049.14±1.6249.14 \pm 1.62
Random-20% (Default Subset)46.91±1.3146.91 \pm 1.3149.23±1.2149.23 \pm 1.21
Middle Tier (40%–60%)47.49±2.1047.49 \pm 2.1051.41±0.9951.41 \pm 0.99
High Tier (Top 20%, Veri-eRank)50.83±1.1450.83 \pm 1.1453.06±0.5753.06 \pm 0.57

W2 & Q3: Practical Accuracy–Cost Advantage over Random and Full-Corpus SFT

We thank the reviewer for emphasizing the importance of rigorous baseline controls and transparent end-to-end cost accounting.

1. Comparison with Random-20% subsets and paired significance tests.

We evaluated five independent Random-20% subsets per backbone, each trained with five matched seeds (25 runs per backbone). Their average Macro Pass@1 values were 47.12% on Qwen (range: 45.86–48.40%) and 50.02% on Seed-Coder (range: 48.90–51.48%). Veri-eRank outperforms all five random subsets in mean performance. Against the default random subset, the paired p-values are 0.0113 and 0.00448, respectively.

In two-sided paired tt-tests across the five seeds, the improvement over Full SFT is statistically significant on Seed-Coder with a +1.98 pp gain and p=0.018p = 0.018, while on Qwen the mean gain of +1.05 pp is positive with p=0.292p = 0.292. Nonetheless, Veri-eRank fulfills the primary goal of 20%-budget data selection, which is matching full-corpus performance while reducing final-training data by 80%. As shown in Table 1, whereas competing 20% selection baselines generally fall short of Full SFT, Veri-eRank maintains or modestly improves upon full-corpus performance across both backbones with lower cross-seed variance.

2. Compute-matched Full SFT and end-to-end efficiency.

We conducted compute-matched Full SFT experiments and compiled the complete end-to-end wall-clock costs to evaluate both data quality and overall efficiency. Specifically, we compared Veri-eRank against Full SFT trained for 0.60.6 epochs to match final-SFT gradient updates, 1.61.6 epochs to match total updates including reference adaptation, 3.03.0 epochs as the default full-corpus baseline, and approximately 4.04.0 epochs to match single-run end-to-end wall-clock time. In the table below, the single-run time for Veri-eRank sums the one-time selection overhead from Tables 7 and 8 and the final subset SFT time.

Training ProtocolQwen Time (1 Run / 5 Runs)Qwen Macro Pass@1 (%)Seed-Coder Time (1 Run / 5 Runs)Seed-Coder Macro Pass@1 (%)
Full SFT (0.6 ep)11m / 55m45.78±1.1145.78 \pm 1.1113m / 65m50.22±1.8750.22 \pm 1.87
Full SFT (1.6 ep)30m / 150m48.96±0.9348.96 \pm 0.9333m / 165m52.06±1.1352.06 \pm 1.13
Full SFT (3.0 ep)57m / 285m49.78±2.3949.78 \pm 2.3962m / 310m51.08±0.8451.08 \pm 0.84
Full SFT (∼4.0\sim 4.0 ep)73m / 365m48.80±0.9448.80 \pm 0.9489m / 445m50.06±1.9550.06 \pm 1.95
Veri-eRank (3.0 ep)73m / 117m50.83±1.1450.83 \pm 1.1489m / 141m53.06±0.5753.06 \pm 0.57

These results show that Veri-eRank outperforms Full SFT under both matched gradient updates and matched wall-clock time. Furthermore, we have to note that the one-time selection overhead is quickly amortized across repeated training runs. Because each subset SFT run reduces training time by roughly four-fifths as analyzed in Appendix A.10, our pipeline breaks even in total wall-clock time at just two runs and cuts the five-seed evaluation cost by more than half.

AuthorsOpenReview ↗

Official Response to Reviewer Fs6q [3/3]

Q4: Supplementary Diagnostic Evaluation

We appreciate this helpful suggestion to decompose the error modes and clarify the scope of our empirical gains.

We categorize evaluation outcomes into compilation failures, functional failures, other failures (including timeouts and unknown errors), and successful cases. All rates are micro-averaged over 262 tasks and reported as mean ± SD across five seeds. SPICE uses a 0.5B proxy on Qwen and the 8B backbone on Seed-Coder.

MethodCompile Fail (%)Functional Fail (%)Other Fail (%)Micro Pass@1 (%)
Full SFT (100%)11.60 ± 1.4437.86 ± 1.411.53 ± 0.5449.01 ± 1.96
Random-20% (Default Subset)13.44 ± 0.6339.54 ± 1.031.45 ± 0.4245.57 ± 1.22
SPICE (20%)13.13 ± 1.0339.47 ± 0.692.14 ± 0.8845.27 ± 0.79
Veri-eRank (20%)11.45 ± 0.9737.79 ± 1.291.37 ± 0.5149.39 ± 1.28
MethodCompile Fail (%)Functional Fail (%)Other Fail (%)Micro Pass@1 (%)
Full SFT (100%)9.08 ± 0.4237.63 ± 0.881.53 ± 0.7151.76 ± 0.44
Random-20% (Default Subset)10.08 ± 1.0338.02 ± 2.191.83 ± 0.7850.08 ± 1.54
SPICE (20%)8.70 ± 0.5039.08 ± 1.371.15 ± 0.6051.07 ± 0.63
Veri-eRank (20%)9.92 ± 0.6034.96 ± 0.881.98 ± 1.0653.13 ± 0.32

Veri-eRank achieves the lowest functional failure rate among the compared methods on both backbones, with a more pronounced reduction on Seed-Coder (37.63% to 34.96% relative to Full SFT). Complete results across all baselines will be included in the revised appendix.

For our primary evaluation, we focused on greedy Pass@1 because it directly isolates how structure-aware selection improves underlying specification-to-RTL representations without introducing sampling variance. Due to compute constraints during this period, we did not expand our experiments to multi-candidate decoding, and we will explore Pass@kk evaluation as a valuable direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.