acceptodds
Under review as a conference paper at ICLR 2027

RTLScope: Fine-Grained Uncertainty-Aware RTL Retrieval with Probabilistic Embeddings

Abstract

There is increasing deployment of LLM and agentic applications in the chip design industry, and fine-grained natural-language-to-RTL retrieval becomes essential as it grounds queries in the RTL code snippets that hardware-design workflows require. Such queries span abstraction levels and vary in detail, from local RTL operations to behavioral intent, and a single description often maps to multiple snippets that jointly implement a behavior; existing benchmarks, however, focus on RTL generation and leave this retrieval capability unmeasured. Because correctness is critical in chip design, measuring the uncertainty of retrieval is highly valuable. We introduce RTLScope, a repository-scale benchmark for fine-grained RTL retrieval built from practical SoC and other large-scale open-source designs; two complementary construction pipelines are built to obtain multi-level queries and one-to-many NL–RTL alignments. We further propose a probabilistic embedding method that learns alignment with a group-listwise multi-positive objective and quantifies query and retrieval uncertainty. With the proposed method, we achieve 15.0% and 15.2% increases on MRR and MAP over the strongest fine-tuned baseline. Together, the benchmark and method enable reliable RTL retrieval for specification-compliance analysis, agentic debugging, verification assistance, design comprehension, and retrieval-augmented hardware-design workflows.

Primary area:
datasets and benchmarks
Keywords:
RTL, Hardware design, Probabilistic embedding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_s5sNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces RTLSCOPE, a benchmark for fine-grained natural-language-to-RTL retrieval, along with a probabilistic embedding method for uncertainty-aware retrieval. The benchmark is constructed from open-source RTL repositories using LLM-assisted query generation and verification pipelines. The proposed method combines Gaussian embeddings with a multiplicity-aware loss to handle queries associated with multiple code snippets. Experiments report improvements over existing retrieval baselines and examine uncertainty through query masking.

Critical strengths

Fine-grained RTL retrieval is a relatively underexplored problem. The benchmark covers multiple query abstraction levels and one-to-many query–RTL alignments. The proposed retrieval method improves MRR from 0.617 to 0.709 and MAP from 0.571 to 0.658 over the strongest fine-tuned baseline.

Critical weaknesses

The benchmark relies heavily on LLM-generated queries and automatically constructed relevance labels.The paper does not clearly explain the train/test splitting strategy or whether test relevance annotations are independently validated. Round-trip verification of RTL functionality does not necessarily guarantee the correctness or completeness of individual query–RTL alignments. This raises concerns about the reliability of the reported retrieval metrics. The proposed method combines probabilistic embeddings, a multiplicity-aware listwise loss, and an auxiliary pair loss. However, the paper does not isolate their individual contributions. It is therefore unclear whether the improvements come from probabilistic representations or the modified training objectives. Furthermore, the uncertainty evaluation mainly relies on query masking, without directly evaluating whether predicted uncertainty correlates with retrieval errors.

Questions for authors

How are the training and test sets separated? Are test relevance labels independently verified by human experts, and how is the completeness of relevant RTL annotations ensured? Can authors report results using deterministic embeddings with the proposed multiplicity-aware loss, as well as ablations of the auxiliary pair loss and KL regularization? This would help establish which components drive the reported improvements. Does the predicted variance correlate with actual retrieval errors?

1 comment
  1. Training, validation, and test split: We use 80% for training, 10% for testing, and 10% for validation.

  2. Completeness and correctness: Correctness is a general concern for LLM-generated synthetic datasets, rather than a limitation unique to our benchmark. LLM-assisted supervision is also used in related RTL research: CircuitFusion generates sub-circuit summaries using GPT-4/GPT-4o, TopoRTL's released pipeline uses LLM APIs for functional-summary generation, and UniRTL incorporates existing RTL–summary datasets, including LLM-generated designs. Given the scarcity of RTL alignment data, our goal is to construct a useful, implementation-grounded dataset with explicit quality controls.

    Round-trip verification as a quality enhancement. Our synthesis pipeline grounds queries in inspected RTL snippets and organizes the descriptions into an implementation plan. An agent then regenerates the design from this plan alone, and the plan is accepted only when the regenerated design passes the specified verification checks against the original. This adds a behavioral quality check beyond directly relying on LLM-generated descriptions.

    Repository history provides an additional, developer-authored source of supervision. We retain sanitized human-authored commit messages and the corresponding RTL snippets. These records provide authentic descriptions of engineering changes, rather than solely LLM-generated labels. Together, revision-grounded queries and design-wide synthesis provide complementary coverage, and round-trip verification adds an additional quality-control step. Our aim is systematic grounding and validation of the dataset, not a claim of guaranteed 100% annotation accuracy.

    Completeness. We divide each RTL module into code chunks, and agents process each chunk sequentially. This provides systematic coverage of the source code during query generation. The coverage is enforced programmatically.

    *Ablations of the auxiliary pair loss and KL regularization We included the ablation study in appendix. Results show removing each term leads to performance degradation.

    LossMRRMAP
    PCME full0.7070.655
    PCME no pair0.6720.636
    PCME no KL0.6810.643
Reviewer_f5AkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

RTLScope introduces a repository-scale benchmark for natural-language-to-RTL retrieval, covering multiple levels of query abstraction and one-to-many query/code alignments. The authors construct data using revision-grounded query generation and round-trip-verified synthesis, and propose Gaussian probabilistic embeddings with a multiplicity-aware loss. On their benchmark, the method improves over the strongest fine-tuned baseline by 15.0% in MRR and 15.2% in MAP, while query variance is evaluated as an uncertainty signal.

Critical strengths
  1. The benchmark targets an important, under-measured task and represents multiple query abstraction levels and one-to-many code matches. This can enable systematic work on hardware-design retrieval.
  2. The probabilistic embedding approach addresses uncertainty alongside ranking, and the reported gains over a fine-tuned baseline are consistent across several retrieval metrics.
Critical weaknesses
  1. The benchmark's generated and synthesized queries may encode artifacts of the construction pipeline or favor the benchmark's own query styles. Without a substantial, independently authored and carefully split engineering-query test set, the reported gains do not yet support the intended claim about practical NL-to-RTL retrieval.
  2. The uncertainty evaluation primarily studies trends under controlled token masking and differences across query styles. Neither demonstrates calibration, failure detection, selective retrieval performance, or a usable abstention threshold on naturally occurring queries. This leaves the central uncertainty-aware reliability claim insufficiently validated.
Questions for authors
  1. Do predicted variances improve calibrated failure detection or risk-coverage on naturally occurring queries, compared with deterministic scores and simple confidence baselines? Without such evidence, what operational decision is the uncertainty estimate intended to support?
  2. Could the authors provide a comparison with other RTL-specific embedding techniques like CircuitFusion, UniRTL, TopoRTL, etc?
1 comment

Thank you for the comments and suggestions. We clarified the concerns as follows:

  1. Different problem focus from prior works: Thank you for pointing us to CircuitFusion, UniRTL, and TopoRTL. These works focus on learning circuit representations, whereas our work focuses on constructing a dedicated fine-grained NL–RTL retrieval dataset and estimating representation uncertainty for retrieval. CircuitFusion learns multimodal sub-circuit representations and retrieves similar known circuits to assist downstream design-quality prediction.UniRTL evaluates both performance prediction and retrieval; its natural-language code search matches a functional summary to its corresponding RTL design using a top-1 bitext-mining protocol, while its code-to-code task evaluates functional equivalence. TopoRTL emphasizes topology-aware circuit representations. Our task instead aligns a natural-language description with the group of repository snippets that jointly implements the described behavior.

    Dedicated dataset construction and uncertainty estimation are our two central contributions. We construct repository-scale retrieval data through revision-grounded query generation and round-trip-verified synthesis, covering multiple abstraction levels and granularities while preserving multiple positive snippets for one description. Our probabilistic embedding method then learns these one-to-many alignments with a multiplicity-aware objective and provides an input-dependent variance signal for representation uncertainty, in addition to ranking candidates. The problem we address is therefore how to construct and learn uncertainty-aware, one-to-many NL–RTL alignments in practical repositories, rather than how to improve general-purpose circuit representations for design-level prediction and matching.

  2. Meaning and intended use of uncertainty: Our predicted variance characterizes representation uncertainty: how broadly the model represents a query in the learned embedding space. We therefore evaluate whether it reflects differences in query specificity and responds to controlled information removal. PCME and CoPE also examine ambiguity and the relationship between predicted uncertainty and retrieval performance: PCME studies text masking and recall across uncertainty bins, while CoPE reports recall across query-uncertainty bins.[6][7] These analyses distinguish representation uncertainty from a calibrated probability of retrieval failure.

    The intended operational use is to support query refinement. For example, a high-variance query can prompt an engineer or agent to provide additional signal names, triggering conditions, or behavioral details before relying on a localized retrieval result. Our query-style and masking analyses provide evidence that variance responds to specificity and missing information, motivating this use as a query-refinement signal.

  3. Construction bias: Construction bias is a general concern for synthetic datasets, rather than a limitation unique to our benchmark. However, the possibility of generation artifacts does not by itself establish that the reported gains are driven by those artifacts. Our benchmark does not rely exclusively on LLM-authored queries. We use two complementary construction pipelines and retain sanitized, developer-authored commit messages from real open-source RTL repositories as a raw-query source. The revision-grounded pipeline uses these messages and the corresponding changed RTL statements to generate queries at multiple abstraction levels. Thus, the engineering context comes from actual implementation and repair activities.

LLM-assisted data construction is also used in the RTL representation-learning works highlighted by the reviewer. CircuitFusion generates sub-circuit summaries using GPT-4/GPT-4o, and TopoRTL's released pipeline generates functional summaries through LLM APIs. UniRTL incorporates existing RTL–summary datasets, including LLM-generated designs. This illustrates that synthetic generation is an established approach for addressing limited RTL data.

The second pipeline complements revision-local coverage through design-wide query synthesis. The generated descriptions provide controlled variation in abstraction and granularity. Its queries are grounded in inspected RTL snippets, and the resulting implementation plans are checked through round-trip regeneration and verification. Together, the benchmark combines developer-authored descriptions, revision-grounded queries, and design-wide synthesis, with multiple positive snippets preserved for each description where applicable.

Given the scarcity of RTL-related datasets, our benchmark provides a useful resource for training and evaluating retrieval models in LLM-assisted hardware-design applications.

Reviewer_qAHAOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a probabilistic retrieval system for hardware description languages that involves uncertainty quantification. The paper includes both benchmarking evaluations and a benchmark set. The key claims are that hardware description languages are different from generic coding languages in that the requirements on correctness are a lot higher, thus necessitating uncertainty quantification.

Critical strengths

I think the dataset that has been built for retrieval is potentially interesting, including the fine-grained tags proposed by the authors and the list of retrieval tasks.

Critical weaknesses

I have several concerns about the motivations stated in the paper. First, uncertainty quantification is by itself a relatively well-studied topic with a very significant body of literature, and useful for any retrieval task, including generic programming languages. More importantly, the distinction that the authors make between hardware description languages and generic coding languages aren't actually the case: descriptions span abstraction levels; a description is often implemented by multiple snippets distribute...across module boundaries are true even for generic programming languages also which then begs the question: why there is a need for a new or different approach. It would help if the authors evaluated their method on a general programming language as well: if the hypothesis is correct, the approach should be less helpful or less differentiated on top of the best methods for general programming languages.

Questions for authors

Can you please provide more details as to why a special approach is needed for RTL compared to an entire body of work on code retrieval for other programming languages. That body of work is so well developed that I would be surprised if there isn't a state-of-the-art method tailored, let's say, for C code that cannot be adapted to Verilog.

1 comment

Thanks for your constructive feedback. We clarify the concern as follows:

  1. Differences between software code and RTL: We agree that characteristics such as "descriptions span abstraction levels; a description is often implemented by multiple snippets..." can also be found in software code. To examine how distributed changes are in RTL and software, we conducted a commit analysis of OpenTitan and Vortex, as both open-source hardware projects contain software and hardware components. The statistics show that RTL implementation-file changes generally touch more files and contain more diff hunks than software changes. These observations support constructing a dataset that preserves distributed implementation evidence and multiple positive snippets.
MetricOpenTitan RTLOpenTitan SoftwareVortex RTLVortex Software
Mean files touched2.151.803.661.76
Mean diff hunks10.959.2119.4313.97
Multi-file commits42.29%29.45%60.87%25.22%
  1. Motivation Clarification: Our motivation has two parts: the scarcity of RTL training data and the need to characterize uncertainty in fine-grained retrieval.

    First, hardware design faces a scarcity of publicly available training data. As an indication of the much smaller public development footprint, GitHut reports that SystemVerilog and Verilog account for 0.288% and 0.065%, respectively, of pull-request activity among its top 50 languages in 2024 Q1: 0.353% combined, compared with 9.459% for C++ and 16.925% for Python—approximately 27 and 48 times the combined RTL share, respectively [1]. One contribution of our work is therefore a specialized dataset for repository-scale NL–RTL retrieval, designed to reflect realistic query scenarios. It covers multiple abstraction levels and granularities, and multiple positive snippets that jointly implement the described behavior. This supplies domain-specific training and evaluation data for adapting existing retrieval methods to RTL.

    It is true that there is a broad literature on uncertainty estimation; however, uncertainty estimation for RTL retrieval is underexplored. This is partly due to the limited availability of datasets constructed for this purpose. One of our contributions is a dataset that can be used for this purpose.

    Necessity of uncertainty estimation. Fine-grained RTL retrieval should accommodate queries ranging from precise implementation conditions to broad behavioral intent. A ranked list orders candidate snippets, but does not explicitly characterize how much implementation detail a query leaves unspecified. For example, “handle reset” may retrieve a wide range of snippets as it is ambiguous. When retrieval guides debugging, code modification, or verification, making this ambiguity visible can help an engineer or agent decide what additional detail to provide before acting on the retrieved evidence.

    Applicability of software retrieval models. Our model is built on CodeRankEmbed, introduced in CoRNStack (ICLR 2025), which reported state-of-the-art results across multiple software code retrieval benchmarks.[2] We evaluate both frozen CodeRankEmbed and a version fine-tuned on our RTL data. The fine-tuned baseline achieves an MRR of 0.617 and MAP of 0.571, while our method with multiplicity-aware learning achieves 0.709 and 0.658, respectively (Table 1 of the manuscript). These results demonstrate a benefit beyond the software retriever's RTL fine-tuning baseline. Our work builds on software retrieval advances to construct a specialized dataset, learn one-to-many NL–RTL alignments, and estimate representation uncertainty.

    Sources: [1] GitHut 2.0, pull requests, 2024 Q1: https://madnight.github.io/githut/#/pull_requests/2024/1 [2] Suresh et al., CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking, ICLR 2025: https://openreview.net/forum?id=iyJOUELYir

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.