acceptodds
Under review as a conference paper at ICLR 2027

Where Does a Hardware Change Act? Intent-Conditioned Modification Support Recovery in Evolving RTL Designs

Abstract

Reliable hardware evolution requires identifying the register-transfer-level (RTL) locations affected by a natural-language change request. This is challenging because changes often span multiple locations coupled by hardware dependencies, beyond independent request–to–code matching. We formulate this task as -to-RTL localization and introduce EvoRTL-Bench, the first benchmark aligning natural-language change requests with fine-grained modification support sets from real RTL revisions. Our analysis shows that modification support sets are sparse yet structurally cohesive, exhibit non-random dependency patterns, and have sizes and topologies predictable from change requests. Motivated by these findings, we propose RTLocating, which fuses textual semantics, functional behavior, and architectural context into selective semantic seeds, then performs intent-conditioned impact propagation over directed, typed RTL dependencies with adaptive channel and scope selection. Counterfactual decision supervision trains these decisions using the support-recovery utility of alternative evidence allocations and propagation actions. On EvoRTL-Bench, RTLocating improves mean average precision by 17.3% relative to the strongest baseline. End-to-end editing experiments further indicate that its localization guidance improves downstream RTL editing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
RTL change localization, LLM-aided hardware design, hardware design automation, design evolution, graph-based reasoning, multi-positive retrieval, counterfactual supervision

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_W8dfOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces EvoRTL-Bench and RTLocating for locating RTL blocks affected by natural-language change requests. It combines semantic evidence with intent-conditioned dependency propagation and evaluates both localization and downstream editing.

Critical strengths
  1. The benchmark addresses a useful hardware maintenance problem using real revisions and fine-grained labels.

  2. Controlled ablations and downstream editing results provide evidence that the proposed localization method is useful.

Critical weaknesses
  1. All data come from OpenTitan. The IP-disjoint split is useful, but transfer to independently developed repositories remains untested.

  2. Ground-truth labels follow the reference commit. Other valid edit locations may be counted as errors, so localization scores may not fully reflect usefulness for editing.

  3. The downstream evaluation covers only 30 requests and reports the best of up to four attempts. This leaves uncertainty about the gains in first-attempt editing.

Questions for authors
  1. Could you provide results on another RTL repository using the model trained on EvoRTL-Bench? This would help establish the scope of generalization.

  2. Could you inspect some high-ranked blocks outside the reference diff to check whether they are plausible alternative edit locations?

  3. Could you report first-attempt success rates and confidence intervals for the improvement over SweRank?

3 comments

Thank you for recognizing the value of EvoRTL-Bench for real-world hardware maintenance and the evidence provided by our controlled experiments and downstream editing evaluation. Your suggestions regarding transfer to independent repositories, the value of alternative edit locations beyond the reference patch, and first-attempt reliability have helped us clarify the scope of our evaluation and add the corresponding experiments. The main updates to the revised manuscript are as follows:

  1. Additional LLM localization baselines (Section 5.2, Table 2). We have added SweRankLLM (7B), adapted on the EvoRTL-Bench training set, and GPT-5.5 to rerank the candidate lists produced by both SweRank-Retriever and the full RTLocating system. LLM reranking improves localization quality for both front ends. Under the same GPT-5.5 reranking configuration, the RTLocating front end achieves a MAP of 0.6317, compared with 0.6028 for the SweRank-Retriever front end, while Accuracy@2R increases from 36.36% to 41.82%. These results show that RTLocating's localization gains persist after LLM reranking. Appendix D.1 also reports the additional online overhead of reranking. Taken together, the performance and cost results support using the full RTLocating system as a lightweight front end that provides candidates and localization hints to downstream LLMs, allowing hardware dependency modeling and LLM reasoning to complement each other.

  2. New cross-repository generalization evaluation (Section 5.7, Table 5). We freeze the models trained on the source domain and evaluate them on 79 external requests adapted from HWE-Bench across CVA6, Ibex, and Caliptra. RTLocating achieves higher scores on all four reported metrics in each of the three repositories, with an overall MAP improvement of approximately 14.0% over SweRank-Retriever. These results provide direct experimental evidence of transfer to independent repositories.

  3. New analysis of the benchmark's training utility (Section 5.9, Figure 6). Under a fixed training budget, we compare SweRank-Retriever, GraphCodeBERT, and CodeV-DS-6.7B using 0%, 25%, 50%, 75%, and 100% of the EvoRTL-Bench training tasks. For the nonzero fractions, we use stratified, nested subsets and three random seeds. With just 25% of the training tasks, all three models outperform their unadapted versions in MAP on both the source-domain test set and the external HWE-Bench tasks. More training data generally improves source-domain performance, while gains on the external tasks gradually plateau, providing further evidence of the dataset's training value across different models.

In addition, we have added examples of alternative edit locations outside the reference support set in Appendix A.7. In the following section, we address your comments individually.

1. Cross-repository generalization (W1, Q1)

Following your suggestion, we evaluate models trained on EvoRTL-Bench on RTL repositories independent of OpenTitan. Both SweRank-Retriever and the full RTLocating system use frozen source-domain checkpoints, with no fine-tuning on the target repositories. HWE-Bench constructs tasks from historical bug-fix PRs in multiple open-source hardware repositories, with CVA6, Ibex, and Caliptra being three of its source repositories. Our evaluation uses 79 tasks adapted from HWE-Bench for localization over pre-change RTL blocks: 29 from CVA6, 35 from Ibex, and 15 from Caliptra.

The overall results are shown below, with all metrics macro-averaged over requests:

MethodMAPMRRRecall@2RAccuracy@2R
SweRank-Retriever0.41500.551546.18%30.38%
RTLocating0.47300.645451.63%37.13%

RTLocating improves MAP by 0.0580 in absolute terms and approximately 14.0% in relative terms, while Accuracy@2R increases by 6.75 percentage points. In the per-repository comparisons, RTLocating achieves higher scores on all four metrics in each of the three repositories. These results indicate that the advantage of the source-domain-trained model transfers to the external RTL tasks included in this evaluation. The experimental setup and overall results have been added to Section 5.7 and Table 5; task adaptation details, reasons for task exclusion, and per-repository results are provided in Appendix D.6.

2. Inspection of highly ranked blocks outside the reference diff (W2, Q2)

We acknowledge this limitation of annotations derived from reference commits. As discussed in Appendix A.6 Data Limitations, the reference support set SqS_q records the locations actually modified in a historical revision. These locations need not constitute the unique or minimal support set sufficient to satisfy the request. The same functional requirement may admit different RTL implementations and combinations of edits across blocks. Consequently, candidate blocks untouched by the reference diff may still be useful edit locations.

Following your suggestion, we examine highly ranked candidates outside the reference diff for four representative requests and construct feasible local alternatives for two of them:

  • aon_timer: Fixing the enable signal at its definition or use site. The reference revision modifies the blocks at ranks 1, 2, and 7, with the block at rank 2 correcting the prescaler counter's enable condition. The alternative retains the reference edits at ranks 1 and 7, leaves the original code at rank 2 unchanged, and directly uses the corrected enable condition in the register-update block at rank 3, which is untouched by the reference diff. The set of modified blocks therefore changes from {1,2,7}\{1,2,7\} to {1,3,7}\{1,3,7\}. A local formal check confirms that the two fixes produce the same counter next state.
  • kmac: Correcting endianness at upstream conversion or downstream use sites. The reference commit modifies the data and mask conversion conditions in the blocks at ranks 1 and 2. The alternative retains the original logic in these two blocks and adds an unconditional byte swap to both the data and mask paths in the downstream consumer blocks at ranks 3 and 4, which are untouched by the reference diff. Since two consecutive byte swaps restore the original order, the alternative produces the same corresponding outputs as the reference fix under both configuration values. The data and mask paths must be updated together.
  • The other two requests concern wakeup-cause capture and output control in aon_timer, and spurious overflow reporting in uart when erroneous frames are filtered, respectively. For these two requests, we found no feasible local alternative among the remaining top-10 candidates examined.

The cases, code for the reference and alternative edits, and verification details have been added to Appendix A.7. These checks provide concrete evidence of the scope limitations of the reference annotations and more accurately illustrate the practical editing value of some highly ranked non-gold candidates.

3. First-attempt success rates and paired confidence intervals (W3, Q3)

We have used the experimental run logs to compute and report first-attempt results. For the single-turn LLM, the first attempt is the first model call. For Codex, it is the first complete agent run, which may include multiple internal searches, tool calls, and code edits. Success requires all observed outputs to match the reference RTL at every sampled cycle under the same testbench and stimuli.

EditorFull-IPSweRank-RetrieverRTLocatingRT − SwePaired 95% CI
Single-turn LLM6/30 (20.0%)8/30 (26.7%)11/30 (36.7%)+10.0 percentage points[−3.3, 23.3] percentage points
Codex12/30 (40.0%)16/30 (53.3%)17/30 (56.7%)+3.3 percentage points[−10.0, 16.7] percentage points

The task-paired outcomes are:

EditorBoth succeedOnly RTLocating succeedsOnly SweRank succeedsBoth fail
Single-turn LLM74118
Codex143211

The confidence intervals are computed using 10,000 paired bootstrap resamples over requests. In each resample, we draw 30 requests with replacement, preserve the pairing of each request's outcomes under the RTLocating and SweRank conditions, and recompute the difference in success rates. The 2.5th and 97.5th percentiles are used as the interval endpoints.

RTLocating has higher point estimates of first-attempt success rates for both editors, but the confidence intervals for both differences include zero. Thus, under these experimental conditions, the evidence is insufficient to establish a statistically significant improvement in editing success rate over SweRank.

Thank you for the thorough rebuttal. I appreciate the additional experiments.

The cross-repository evaluation on CVA6, Ibex, and Caliptra is convincing, and the examples in A.7 clarify the limits of the reference labels. The first-attempt results are also helpful. As the confidence intervals include zero, I’d suggest phrasing the downstream editing gains a bit more cautiously in the paper.

I remain positive about this work and will keep my score.

Reviewer_jTKPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies RTL modification-support localization and introduces EvoRTL-Bench, a benchmark of 1,094 change instances derived from OpenTitan revision history. The authors also propose RTLocating, a supervised framework that ranks existing behavioral blocks in a pre-change RTL design given a natural-language change request. RTLocating combines textual, functional dataflow, and design-level structural evidence to produce an initial ranking, then propagates relevance along typed dependencies with request-adaptive channel weights and propagation depth. Offline ranking-based utility targets supervise evidence fusion and propagation decisions.

Critical strengths
  1. The paper introduces a fine-grained, multi-block RTL localization benchmark based on real revision history, with a 500-instance audit of functional filtering and request quality and structural analyses of modification supports. Controlled experiments also show that adaptive propagation improves mean average precision from 0.448 to 0.471 using the same initial block rankings.
Critical weaknesses
  1. EvoRTL-Bench derives normalized change requests from completed commit descriptions and uses the existing behavioral blocks modified or deleted in each reference revision as labels. This defines a valid historical localization task, but one observed implementation may not represent other valid ways to make the same change. New structures without pre-change counterparts are excluded from block ranking. Together with the single-project data and retrospective request construction, this leaves the benchmark's applicability to independently specified maintenance requests uncertain.
  2. The additional value of RTLocating's structural reasoning over alternative localization approaches is not fully established. The fixed-seed ablations demonstrate improvements over homogeneous and globally typed propagation, but the directed producer-to-consumer propagation has not been compared with a matched bidirectional alternative for changes that require tracing upstream logic. The main baseline comparison also omits SweRank's reranker and does not include an LLM or coding agent evaluated on the same block-level localization task. These gaps leave the accuracy–cost advantage over alternative approaches unclear.
Questions for authors
  1. How well do requests reconstructed from completed commits reflect maintenance requests written before implementation? How does the benchmark handle added logic, interface changes, and alternative valid implementations when the labels contain only pre-change blocks? Could the authors compare results using original commit descriptions or independently written requests, with explicit location hints controlled?
  2. The paper analyzes structural relationships among modified blocks using an undirected graph, while the proposed propagation follows directed producer-to-consumer dependencies. How often do missed support blocks require reverse traversal? Could the authors compare the current method with a bidirectional alternative using the same initial rankings and otherwise matched settings?
  3. Could the authors compare RTLocating with either the complete SweRank retrieve-and-rerank pipeline or an LLM-based RTL localization agent using the same pre-change design, change request, block-level output, and a comparable computational budget? Please also report recall and complete-support recovery at fixed top-5, top-10, and top-20, together with costs that distinguish reusable preprocessing from per-request computation.
  4. In the RTL editing experiment, SweRank-Retriever and RTLocating provide top-10 localization cues, whereas Full-IP does not. Please clarify the exact RTL context, prompts, and additional annotations provided in each condition. Do the guided conditions receive the full design with added localization cues, or a different subset of the code? How do input lengths, truncation rules, and total editing costs compare? Since the reported best attempt is selected using the reference post-change RTL, could the authors provide first-attempt and paired task-level results and explain how a final patch would be selected without that reference? How were the 30 editing tasks selected?
8 comments

Thank you for recognizing the value of our fine-grained benchmark of historical revisions, manual auditing, and controlled propagation comparisons with a fixed initial ranking. Your questions about request representativeness, propagation direction, fair comparisons of accuracy and cost, and the transparency of the editing evaluation have helped us further examine the method's scope and experimental setup. The main updates to the revised manuscript are as follows:

  1. Additional LLM localization baselines (Section 5.2, Table 2). We have added SweRankLLM (7B), adapted on the EvoRTL-Bench training set, and GPT-5.5 to rerank candidates from both SweRank-Retriever and the full RTLocating system. LLM reranking improves localization quality for both front ends. Under the same GPT-5.5 reranking configuration, RTLocating achieves a MAP of 0.6317, compared with 0.6028 for SweRank-Retriever, while Accuracy@2R increases from 36.36% to 41.82%. These results show that RTLocating's localization gains persist after LLM reranking. Appendix D.1 also reports the additional online overhead of reranking. Taken together, the performance and cost results support using the full RTLocating system as a lightweight front end that provides candidates and localization hints to downstream LLMs, allowing hardware dependency modeling and LLM reasoning to complement each other.

  2. New cross-repository generalization evaluation (Section 5.7, Table 5). We freeze the models trained on the source domain and evaluate them on 79 external requests adapted from HWE-Bench across CVA6, Ibex, and Caliptra. RTLocating achieves higher values on all four reported metrics in each of the three repositories, with an overall MAP improvement of approximately 14.0% over SweRank-Retriever.

  3. New analysis of the benchmark's training utility (Section 5.9, Figure 6). Under a fixed training budget, we compare three retrieval models using 0%, 25%, 50%, 75%, and 100% of the EvoRTL-Bench training tasks. For the nonzero fractions, we use stratified, nested subsets and three random seeds. With just 25% of the training tasks, all three models outperform their unadapted versions in MAP on both the source-domain test set and the external HWE-Bench tasks. More training data generally improves source-domain performance, while gains on the external tasks gradually plateau.

In addition, Appendices A.7 and D.7 now include examples of alternative edit locations and details of editing inputs and prompt templates, respectively.

Below, we address your specific questions and report the directionality, fixed-budget, and cost analyses you suggested.

1. Request representativeness and the scope of the pre-change candidate space (W1, Q1)

(1) Historical requests and maintenance requests written before implementation

To address your concerns about request provenance and potential implementation cues, we provide further clarification from three perspectives: the request-construction protocol, explicit identifier cues, and the source of the request wording.

Request normalization in EvoRTL-Bench organizes the behavioral changes and motivations described in historical commit messages into a consistent request format. It emphasizes the difference between pre-change and post-change behavior, functional context, and design intent, while reducing variation in writing style and information organization across commits. This provides clear, consistently structured task inputs for different localization methods. As described in Appendix A.3, the request-generation model can access only the commit title and body. It cannot access the reference diff, the list of changed files, RTL, ASTs, or support annotations. Its output must not contain file paths, line numbers, block IDs, diff syntax, or verbatim RTL snippets, although hardware identifiers already present in the original commit message may be retained. This protocol grounds normalization in historical descriptions and avoids generating localization hints directly from patches or support annotations. We further conduct the following two analyses to examine the effects of explicit implementation identifiers and request wording, respectively.

a. Analysis of explicit identifier cues

Retrospective descriptions may explicitly name signals or types in the implementation. We therefore group requests according to whether they contain user-defined identifiers from the reference support blocks. We extract identifiers, including signal names, struct fields, types, enum values, and macro names, from the pre-change reference support blocks and check for case-sensitive, whole-token matches in the requests. Comments, strings, numeric literals, and language keywords are excluded during extraction. Of the 110 test requests, 36 (32.73%) contain at least one such overlapping token, while 74 (67.27%) contain none.

Request groupMethodMAPMRRRecall@RRecall@2RAccuracy@RAccuracy@2R
With overlap (n=36)SweRank-Retriever0.51080.731446.98%51.86%15.74%19.44%
With overlap (n=36)RTLocating0.58360.828953.27%67.83%25.00%33.33%
Without overlap (n=74)SweRank-Retriever0.34790.508426.02%42.87%4.95%15.77%
Without overlap (n=74)RTLocating0.41590.571334.21%50.87%10.81%22.07%

Both methods achieve higher MRR and Accuracy@R in the overlap group, consistent with the interpretation that explicit identifier cues aid localization. In the group without overlap, RTLocating outperforms SweRank-Retriever on every metric. These group-level statistics show that the observed advantage is not confined to requests that directly name identifiers in the support blocks.

b. Comparison of request wording sources

To examine sensitivity to normalized wording, we keep the test set, model parameters, candidate sets, and reference support sets fixed, and apply the same input-length limits and truncation rules. The only change is to replace each normalized request with the corresponding child commit's original Git %B, namely its title and body.

Request conditionMethodMAPMRRRecall@2RAccuracy@2R
Normalized requestsSweRank-Retriever0.40120.581445.81%16.97%
Normalized requestsRTLocating0.47080.655656.42%25.76%
Original commit messagesSweRank-Retriever0.38470.577644.50%16.67%
Original commit messagesRTLocating0.45180.646052.85%24.24%

(2) Handling new logic and interface changes

The current task ranks continuous assignments or complete procedural blocks that already exist in the pre-change design. The reference annotations distinguish the existing support set SqS_q from addition events Aq+A_q^+ that have no pre-change counterpart.

Change typeAnnotation and evaluation treatment
Modifying or deleting an existing behavioral blockThe corresponding complete pre-change block is included in SqS_q.
Inserting logic within an existing behavioral blockThe pre-change block is included in SqS_q if the insertion location can be uniquely mapped to it.
Adding an independent behavioral block, module, or fileThe addition is recorded as an event in Aq+A_q^+ and is not included in the ranking of pre-change blocks.
Changes outside the candidate space, such as interface declarations or instance connectionsProvenance information is retained. If existing behavioral blocks are also modified, localization is evaluated on those pre-change blocks; declarations and connections themselves are not behavioral-block candidates.
Tasks with no evaluable pre-change support blocks or no reliable mappingThese tasks are excluded from the current block-level localization evaluation.

The detailed rules are provided in Appendix A.2 and Table 8; the scope limitations are discussed in Appendix A.6.

(3) Alternative valid implementations and the reference support set

We agree that SqS_q records the locations modified in one historical revision and need not be the unique or minimal support set sufficient to satisfy the request. The same functional requirement may admit different RTL implementations and combinations of edits across blocks. Consequently, candidate blocks untouched by the reference diff may still be useful edit locations. In Appendix A.7, we present four representative requests and qualitatively examine the editing value of highly ranked non-gold candidates and the limitations of the reference annotations:

  • aon_timer: While retaining the reference edits at ranks 1 and 7, the fix to the enable-signal definition at rank 2 can be moved to the register-update block at rank 3, which is untouched by the reference diff. A local formal check confirms that the two implementations produce the same counter next state.
  • kmac: The endianness fix can be moved from the upstream conversion blocks at ranks 1 and 2 to the downstream data- and mask-consuming blocks at ranks 3 and 4. Retaining the original upstream logic and adding byte swapping consistently to both downstream paths produces the same corresponding outputs as the reference fix.
  • The other two requests concern wakeup-cause capture and output control in aon_timer, and spurious overflow reporting in uart, respectively. We found no feasible local alternative among the remaining top-10 candidates examined for these requests.

(4) Transfer to independent repositories

To examine the external applicability of models trained on a single project, we add a cross-repository evaluation on HWE-Bench. HWE-Bench is a repository-level benchmark of real-world hardware bug repair for LLM agents, with tasks constructed from historical bug-fix PRs in multiple open-source hardware repositories. CVA6, Ibex, and Caliptra are three of its source repositories. Our evaluation uses 79 tasks adapted from HWE-Bench for localization over pre-change RTL blocks: 29 from CVA6, 35 from Ibex, and 15 from Caliptra. The overall results are shown below, with all metrics macro-averaged over requests:

MethodMAPMRRRecall@2RAccuracy@2R
SweRank-Retriever0.41500.551546.18%30.38%
RTLocating0.47300.645451.63%37.13%

RTLocating achieves an overall MAP of 0.4730, compared with 0.4150 for SweRank-Retriever, while Accuracy@2R increases from 30.38% to 37.13%. All four reported metrics are higher in each of the three repositories. These results indicate that the advantage of the source-domain-trained model transfers to the external RTL tasks included in this evaluation. The experimental setup and overall results are provided in Section 5.7 and Table 5, with task adaptation details and per-repository results in Appendix D.6.

Reviewer_yvEEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates Δ\DeltaSpec-to-RTL localization: given a pre-change RTL design and a natural-language change request, rank the design's behavioral blocks so that the blocks edited by the reference revision come first. It makes three contributions:

(1) EvoRTL-Bench. 1,094 instances mined from OpenTitan's Git history. GPT-4.1 rewrites commit messages into requests without seeing the diff, supports come from mapping diffs onto AST blocks of the parent design. And a 500-instance practitioner audit reports a 98.2% pass rate.

(2) Empirical study. Supports are sparse and more cohesive than module-matched and degree-matched random sets; A TF-IDF probe of the request predicts their size and spread.

(3) RTLocating. Textual, functional (DFG/GATv2) and architectural (DTG/GLIDE) scores are fused with query-dependent weights into seeds, which propagate over six typed dependency channels with query-conditioned channel and hop weights. These decisions are trained with "counterfactual decision supervision" derived from offline AP gains.

Critical strengths

(S1) A new task with traceable labels. The task differs from NL-to-RTL retrieval, since targets need not implement or mention the requested behavior; and from bug localization, since no faulty design or tests are assumed. Each construction step is explicit: per-diff-type mapping rules (Table 7), diff-blind request generation (App. A.3), and an audit that locks leakage judgments before annotators see the diff (Tables 11-13). The empirical study uses module-matched and degree-matched nulls, Holm correction and IP-cluster bootstraps. With the best Accuracy @2R2R at 25.76%, the benchmark leaves clear headroom for hardware agents.

(S2) Controls, not only end metrics, support the central claim. Starting from the same frozen seed, untyped propagation lowers MAP, while query-conditioned channel and scope reach 0.471 (Table 3). Permuting query representations within each IP costs 0.020 MAP (App. D.3), shuffled counterfactual targets drop MAP to 0.397 (Table 4), and permuted fusion weights lower seed MAP to 0.432. These results rule out added capacity, auxiliary-loss regularization and a fixed global channel preference as explanations. The system also leads every support-size and IP-size bin in Figure 3.

Critical weaknesses

(W1) Uncertainty is missing where the method-specific claims depend on it. All results come from one IP-disjoint split with 110 test queries and an unstated number of test IPs. Tables 2 and 4 report no dispersion or paired tests, and seed-level standard deviations do not capture IP sampling. The 0.07 MAP margin over SweRank is likely robust, but the paper-specific gains are small.

The editing study (n=30n = 30) separates RTLocating from SweRank by two full recoveries per editor. It reports the best of up to four attempts selected by recovery against the reference revision, and does not say how the requests were chosen.

(W2) No LLM-based localizer is evaluated, although the paper motivates the task through LLM hardware agents.

Every baseline in Table 2 is a lexical, embedding, or GNN retriever. SweRank's reranker is excluded (App. C.5.2), and LocAgent is cited but not evaluated.

Under Full-IP, Codex's localization failed in 7 of 30 requests (Figure 11), which suggests agentic localization is strong. This bears on significance rather than validity: RTLocating may be most useful as a first stage or as a source of hints for agents. An LLM reranking baseline over the top-kk candidates, plus set-level recall and precision of Codex's edited blocks against Sq\mathcal{S}_q, would establish its position.

Questions for authors

(1) How were the 30 editing requests chosen? Does the single-turn prompt include the full IP besides the top-10 blocks? Please report first-attempt outcomes, a paired RTLocating-vs-SweRank comparison, and whether failure attribution was blind to the condition.

(2) Do you have results for an LLM-based localizer, such as gpt-5.5 reranking the top-50 candidates, or Codex's edited blocks scored against Sq\mathcal{S}_q?

(3) App. A.3 allows identifiers that appear in the original commit message. What fraction of test requests name an identifier occurring in a support block, and how do the methods perform with and without such identifiers?

(4) OOF folds are grouped by design version (App. C.2), so the OOF encoders were typically trained on other revisions of the same IP, whereas test IPs are unseen. Are Train-OOF and Test seed statistics comparable, and do IP-grouped folds change the targets?

5 comments

Thank you for your positive assessment of the task definition, annotation traceability, and controlled experiments, and for specifically identifying areas for improvement in statistical uncertainty, LLM localization comparisons, and the transparency of the downstream editing evaluation. Your feedback has helped us clarify how to describe RTLocating's role in LLM-based hardware workflows more accurately. The main updates to the revised manuscript are as follows:

  1. Additional LLM localization baselines (Section 5.2, Table 2). We have added SweRankLLM (7B), adapted on the EvoRTL-Bench training set, and GPT-5.5 to rerank the candidate lists produced by both SweRank-Retriever and the full RTLocating system. LLM reranking improves localization quality for both front ends. Under the same GPT-5.5 reranking configuration, the RTLocating front end achieves a MAP of 0.6317, compared with 0.6028 for the SweRank-Retriever front end, while Accuracy@2R increases from 36.36% to 41.82%. These results show that RTLocating's localization gains persist after LLM reranking. Appendix D.1 also reports the additional online overhead of reranking. Taken together, the performance and cost results support using the full RTLocating system as a lightweight front end that provides candidates and localization hints to downstream LLMs, allowing hardware dependency modeling and LLM reasoning to complement each other.

  2. New cross-repository generalization evaluation (Section 5.7, Table 5). We freeze the models trained on EvoRTL-Bench and evaluate them on 79 external requests adapted from HWE-Bench across three hardware repositories: CVA6, Ibex, and Caliptra. RTLocating achieves a MAP of 0.4730, compared with 0.4150 for SweRank-Retriever, while Accuracy@2R increases from 30.38% to 37.13%. All four reported metrics are higher in each of the three repositories. Task adaptation details, reasons for task exclusion, and per-repository results are provided in Appendix D.6.

  3. New analysis of the benchmark's training utility (Section 5.9, Figure 6). Under a fixed training budget, we compare SweRank-Retriever, GraphCodeBERT, and CodeV-DS-6.7B using 0%, 25%, 50%, 75%, and 100% of the training tasks. For the nonzero fractions, we use stratified, nested subsets and three random seeds. With just 25% of the training tasks, all three models outperform their unadapted versions in MAP on both the EvoRTL-Bench test set and the external HWE-Bench tasks. More training data generally improves source-domain performance, while gains on the external tasks gradually plateau, providing further evidence of EvoRTL-Bench's training value across different models.

In addition, to address your specific questions about the editing experiments, we have added details of task selection, input context, and prompt templates in Appendix D.7.

In the following sections, we address each of your comments in turn.

1. Number of test IPs and uncertainty in the results (W1)

Our IP-disjoint test set contains 5 unseen IPs and 110 modification requests. The five IPs are aon_timer, kmac, otp_macro, rv_timer, and uart, covering four functional categories: timing and control, cryptographic computation, secure storage, and serial communication.

We agree that variability across training seeds and uncertainty from sampling test IPs provide different information. We therefore report the sample standard deviation across three random-seed runs (ddof=1) and perform 100,000 paired bootstrap resamples clustered by IP to assess between-method differences. Before computing paired differences, we average each method's results across the three seeds for each request. In each resample, we draw 5 IPs with replacement, retaining all requests from each selected IP and preserving the pairing between methods. An IP drawn multiple times contributes with the corresponding multiplicity. We then compute the mean metric difference over requests. The 95% intervals are given by the 2.5th and 97.5th percentiles of the resampled difference distribution.

MetricSweRank-Retriever: mean ± seed SDRTLocating: mean ± seed SDRTLocating − SweRankIP-clustered paired bootstrap 95% CI
MAP0.4012 ± 0.00140.4708 ± 0.003+0.0696[0.0434, 0.0977]
MRR0.5814 ± 0.00260.6556 ± 0.003+0.0742[0.0329, 0.1130]
Recall@2R0.458 ± 0.030.564 ± 0.003+0.106[0.0594, 0.163]
Accuracy@2R0.170 ± 0.050.258 ± 0.005+0.09[0.03, 0.152]

In this analysis, the intervals for all four metric differences lie above zero. Resampling entire IP clusters preserves the correlation among requests within the same IP, allowing the intervals to reflect performance differences as the composition of test IPs varies. However, there are still only 5 independent test IPs, and increasing the number of resamples does not increase the number of independent designs. We therefore interpret this as a sensitivity analysis over the current test IPs and discuss generalization alongside the additional external-repository evaluation (Section 5.7, Table 5).

2. Task selection, input context, and first-attempt results for downstream editing (W1, Q1)

How the tasks were selected. We select the 30 requests from the test set by setting fixed quotas based on IP and reference support set size, followed by random sampling within each stratum, to cover different functions, design sizes, and modification scales. The sample covers all 5 test IPs. In terms of reference support set size, it includes 8 single-block requests, 10 with 2–5 blocks, 10 with 6–10 blocks, and 2 with at least 11 blocks. The detailed distribution has been added to Appendix D.7.1 and Table 28.

Does the single-turn prompt also include the full IP? Under the RTLocating- and SweRank-Retriever-guided conditions, the single-turn LLM does not receive additional full-IP context. Its input includes the full request, the top 10 pre-change RTL blocks, declarations and interfaces of their enclosing modules, and block IDs, file paths, and line ranges. The editor returns replacement code for each block ID. The Full-IP condition provides the complete relevant pre-change RTL files, with edits specified by filename and original line range.

Thus, for the single-turn LLM, the comparison between Full-IP and the guided conditions involves differences in both context and patch representation, and measures the overall effects of these configurations. The RTLocating- and SweRank-Retriever-guided conditions use the same prompt format, context-organization rules, and editing budget, allowing a more direct comparison of the localization front ends.

First-attempt results and paired comparison. We have used the experimental run logs to compute and report first-attempt results. For the single-turn LLM, the first attempt is the first model call. For Codex, it is the first complete agent run, which may include multiple searches, tool calls, and code edits.

EditorFull-IPSweRank-RetrieverRTLocatingRT − SwePaired 95% CI
Single-turn LLM6/30 (20.0%)8/30 (26.7%)11/30 (36.7%)+10.0 percentage points[−3.3, 23.3] percentage points
Codex12/30 (40.0%)16/30 (53.3%)17/30 (56.7%)+3.3 percentage points[−10.0, 16.7] percentage points

The task-paired outcomes are as follows:

EditorBoth succeedOnly RTLocating succeedsOnly SweRank succeedsBoth fail
Single-turn LLM74118
Codex143211

The intervals are computed using 10,000 paired bootstrap resamples over tasks, preserving the pairing of each request's outcomes under the two conditions. RTLocating has higher point estimates of success rates for both editors, but the intervals for both differences include zero. Thus, under these experimental conditions, the evidence is insufficient to establish a statistically significant improvement in editing success rate over SweRank.

Was failure attribution blinded to the experimental condition? No. The evaluators knew the experimental conditions during retrospective attribution and applied the same criteria across all conditions. They examined the correspondence between the blocks the editor selected or attempted to modify and the reference support blocks. Missing relevant reference locations was categorized as a localization failure; identifying the relevant locations but failing to fully restore the tested behavior was categorized as an editing failure. Automated behavioral scoring should be distinguished from manual attribution here: all patches are evaluated using the same Verilator testbench and stimuli. Full recovery requires all observed outputs to match the reference revision at every sampled cycle. Recovery rates, success counts, and the paired intervals above are computed from simulation results and do not depend on the manually assigned failure categories.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.