- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces EvoRTL-Bench and RTLocating for locating RTL blocks affected by natural-language change requests. It combines semantic evidence with intent-conditioned dependency propagation and evaluates both localization and downstream editing.
Critical strengths
-
The benchmark addresses a useful hardware maintenance problem using real revisions and fine-grained labels.
-
Controlled ablations and downstream editing results provide evidence that the proposed localization method is useful.
Critical weaknesses
-
All data come from OpenTitan. The IP-disjoint split is useful, but transfer to independently developed repositories remains untested.
-
Ground-truth labels follow the reference commit. Other valid edit locations may be counted as errors, so localization scores may not fully reflect usefulness for editing.
-
The downstream evaluation covers only 30 requests and reports the best of up to four attempts. This leaves uncertainty about the gains in first-attempt editing.
Questions for authors
-
Could you provide results on another RTL repository using the model trained on EvoRTL-Bench? This would help establish the scope of generalization.
-
Could you inspect some high-ranked blocks outside the reference diff to check whether they are plausible alternative edit locations?
-
Could you report first-attempt success rates and confidence intervals for the improvement over SweRank?
3 comments
2. Inspection of highly ranked blocks outside the reference diff (W2, Q2)
We acknowledge this limitation of annotations derived from reference commits. As discussed in Appendix A.6 Data Limitations, the reference support set records the locations actually modified in a historical revision. These locations need not constitute the unique or minimal support set sufficient to satisfy the request. The same functional requirement may admit different RTL implementations and combinations of edits across blocks. Consequently, candidate blocks untouched by the reference diff may still be useful edit locations.
Following your suggestion, we examine highly ranked candidates outside the reference diff for four representative requests and construct feasible local alternatives for two of them:
- aon_timer: Fixing the enable signal at its definition or use site. The reference revision modifies the blocks at ranks 1, 2, and 7, with the block at rank 2 correcting the prescaler counter's enable condition. The alternative retains the reference edits at ranks 1 and 7, leaves the original code at rank 2 unchanged, and directly uses the corrected enable condition in the register-update block at rank 3, which is untouched by the reference diff. The set of modified blocks therefore changes from to . A local formal check confirms that the two fixes produce the same counter next state.
- kmac: Correcting endianness at upstream conversion or downstream use sites. The reference commit modifies the data and mask conversion conditions in the blocks at ranks 1 and 2. The alternative retains the original logic in these two blocks and adds an unconditional byte swap to both the data and mask paths in the downstream consumer blocks at ranks 3 and 4, which are untouched by the reference diff. Since two consecutive byte swaps restore the original order, the alternative produces the same corresponding outputs as the reference fix under both configuration values. The data and mask paths must be updated together.
- The other two requests concern wakeup-cause capture and output control in aon_timer, and spurious overflow reporting in uart when erroneous frames are filtered, respectively. For these two requests, we found no feasible local alternative among the remaining top-10 candidates examined.
The cases, code for the reference and alternative edits, and verification details have been added to Appendix A.7. These checks provide concrete evidence of the scope limitations of the reference annotations and more accurately illustrate the practical editing value of some highly ranked non-gold candidates.
3. First-attempt success rates and paired confidence intervals (W3, Q3)
We have used the experimental run logs to compute and report first-attempt results. For the single-turn LLM, the first attempt is the first model call. For Codex, it is the first complete agent run, which may include multiple internal searches, tool calls, and code edits. Success requires all observed outputs to match the reference RTL at every sampled cycle under the same testbench and stimuli.
| Editor | Full-IP | SweRank-Retriever | RTLocating | RT − Swe | Paired 95% CI |
|---|---|---|---|---|---|
| Single-turn LLM | 6/30 (20.0%) | 8/30 (26.7%) | 11/30 (36.7%) | +10.0 percentage points | [−3.3, 23.3] percentage points |
| Codex | 12/30 (40.0%) | 16/30 (53.3%) | 17/30 (56.7%) | +3.3 percentage points | [−10.0, 16.7] percentage points |
The task-paired outcomes are:
| Editor | Both succeed | Only RTLocating succeeds | Only SweRank succeeds | Both fail |
|---|---|---|---|---|
| Single-turn LLM | 7 | 4 | 1 | 18 |
| Codex | 14 | 3 | 2 | 11 |
The confidence intervals are computed using 10,000 paired bootstrap resamples over requests. In each resample, we draw 30 requests with replacement, preserve the pairing of each request's outcomes under the RTLocating and SweRank conditions, and recompute the difference in success rates. The 2.5th and 97.5th percentiles are used as the interval endpoints.
RTLocating has higher point estimates of first-attempt success rates for both editors, but the confidence intervals for both differences include zero. Thus, under these experimental conditions, the evidence is insufficient to establish a statistically significant improvement in editing success rate over SweRank.
Thank you for the thorough rebuttal. I appreciate the additional experiments.
The cross-repository evaluation on CVA6, Ibex, and Caliptra is convincing, and the examples in A.7 clarify the limits of the reference labels. The first-attempt results are also helpful. As the confidence intervals include zero, I’d suggest phrasing the downstream editing gains a bit more cautiously in the paper.
I remain positive about this work and will keep my score.
Thank you for recognizing the value of EvoRTL-Bench for real-world hardware maintenance and the evidence provided by our controlled experiments and downstream editing evaluation. Your suggestions regarding transfer to independent repositories, the value of alternative edit locations beyond the reference patch, and first-attempt reliability have helped us clarify the scope of our evaluation and add the corresponding experiments. The main updates to the revised manuscript are as follows:
Additional LLM localization baselines (Section 5.2, Table 2). We have added SweRankLLM (7B), adapted on the EvoRTL-Bench training set, and GPT-5.5 to rerank the candidate lists produced by both SweRank-Retriever and the full RTLocating system. LLM reranking improves localization quality for both front ends. Under the same GPT-5.5 reranking configuration, the RTLocating front end achieves a MAP of 0.6317, compared with 0.6028 for the SweRank-Retriever front end, while Accuracy@2R increases from 36.36% to 41.82%. These results show that RTLocating's localization gains persist after LLM reranking. Appendix D.1 also reports the additional online overhead of reranking. Taken together, the performance and cost results support using the full RTLocating system as a lightweight front end that provides candidates and localization hints to downstream LLMs, allowing hardware dependency modeling and LLM reasoning to complement each other.
New cross-repository generalization evaluation (Section 5.7, Table 5). We freeze the models trained on the source domain and evaluate them on 79 external requests adapted from HWE-Bench across CVA6, Ibex, and Caliptra. RTLocating achieves higher scores on all four reported metrics in each of the three repositories, with an overall MAP improvement of approximately 14.0% over SweRank-Retriever. These results provide direct experimental evidence of transfer to independent repositories.
New analysis of the benchmark's training utility (Section 5.9, Figure 6). Under a fixed training budget, we compare SweRank-Retriever, GraphCodeBERT, and CodeV-DS-6.7B using 0%, 25%, 50%, 75%, and 100% of the EvoRTL-Bench training tasks. For the nonzero fractions, we use stratified, nested subsets and three random seeds. With just 25% of the training tasks, all three models outperform their unadapted versions in MAP on both the source-domain test set and the external HWE-Bench tasks. More training data generally improves source-domain performance, while gains on the external tasks gradually plateau, providing further evidence of the dataset's training value across different models.
In addition, we have added examples of alternative edit locations outside the reference support set in Appendix A.7. In the following section, we address your comments individually.
1. Cross-repository generalization (W1, Q1)
Following your suggestion, we evaluate models trained on EvoRTL-Bench on RTL repositories independent of OpenTitan. Both SweRank-Retriever and the full RTLocating system use frozen source-domain checkpoints, with no fine-tuning on the target repositories. HWE-Bench constructs tasks from historical bug-fix PRs in multiple open-source hardware repositories, with CVA6, Ibex, and Caliptra being three of its source repositories. Our evaluation uses 79 tasks adapted from HWE-Bench for localization over pre-change RTL blocks: 29 from CVA6, 35 from Ibex, and 15 from Caliptra.
The overall results are shown below, with all metrics macro-averaged over requests:
RTLocating improves MAP by 0.0580 in absolute terms and approximately 14.0% in relative terms, while Accuracy@2R increases by 6.75 percentage points. In the per-repository comparisons, RTLocating achieves higher scores on all four metrics in each of the three repositories. These results indicate that the advantage of the source-domain-trained model transfers to the external RTL tasks included in this evaluation. The experimental setup and overall results have been added to Section 5.7 and Table 5; task adaptation details, reasons for task exclusion, and per-repository results are provided in Appendix D.6.