- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces RTLSCOPE, a benchmark for fine-grained natural-language-to-RTL retrieval, along with a probabilistic embedding method for uncertainty-aware retrieval. The benchmark is constructed from open-source RTL repositories using LLM-assisted query generation and verification pipelines. The proposed method combines Gaussian embeddings with a multiplicity-aware loss to handle queries associated with multiple code snippets. Experiments report improvements over existing retrieval baselines and examine uncertainty through query masking.
Critical strengths
Fine-grained RTL retrieval is a relatively underexplored problem. The benchmark covers multiple query abstraction levels and one-to-many query–RTL alignments. The proposed retrieval method improves MRR from 0.617 to 0.709 and MAP from 0.571 to 0.658 over the strongest fine-tuned baseline.
Critical weaknesses
The benchmark relies heavily on LLM-generated queries and automatically constructed relevance labels.The paper does not clearly explain the train/test splitting strategy or whether test relevance annotations are independently validated. Round-trip verification of RTL functionality does not necessarily guarantee the correctness or completeness of individual query–RTL alignments. This raises concerns about the reliability of the reported retrieval metrics. The proposed method combines probabilistic embeddings, a multiplicity-aware listwise loss, and an auxiliary pair loss. However, the paper does not isolate their individual contributions. It is therefore unclear whether the improvements come from probabilistic representations or the modified training objectives. Furthermore, the uncertainty evaluation mainly relies on query masking, without directly evaluating whether predicted uncertainty correlates with retrieval errors.
Questions for authors
How are the training and test sets separated? Are test relevance labels independently verified by human experts, and how is the completeness of relevant RTL annotations ensured? Can authors report results using deterministic embeddings with the proposed multiplicity-aware loss, as well as ablations of the auxiliary pair loss and KL regularization? This would help establish which components drive the reported improvements. Does the predicted variance correlate with actual retrieval errors?
Training, validation, and test split: We use 80% for training, 10% for testing, and 10% for validation.
Completeness and correctness: Correctness is a general concern for LLM-generated synthetic datasets, rather than a limitation unique to our benchmark. LLM-assisted supervision is also used in related RTL research: CircuitFusion generates sub-circuit summaries using GPT-4/GPT-4o, TopoRTL's released pipeline uses LLM APIs for functional-summary generation, and UniRTL incorporates existing RTL–summary datasets, including LLM-generated designs. Given the scarcity of RTL alignment data, our goal is to construct a useful, implementation-grounded dataset with explicit quality controls.
Round-trip verification as a quality enhancement. Our synthesis pipeline grounds queries in inspected RTL snippets and organizes the descriptions into an implementation plan. An agent then regenerates the design from this plan alone, and the plan is accepted only when the regenerated design passes the specified verification checks against the original. This adds a behavioral quality check beyond directly relying on LLM-generated descriptions.
Repository history provides an additional, developer-authored source of supervision. We retain sanitized human-authored commit messages and the corresponding RTL snippets. These records provide authentic descriptions of engineering changes, rather than solely LLM-generated labels. Together, revision-grounded queries and design-wide synthesis provide complementary coverage, and round-trip verification adds an additional quality-control step. Our aim is systematic grounding and validation of the dataset, not a claim of guaranteed 100% annotation accuracy.
Completeness. We divide each RTL module into code chunks, and agents process each chunk sequentially. This provides systematic coverage of the source code during query generation. The coverage is enforced programmatically.
*Ablations of the auxiliary pair loss and KL regularization We included the ablation study in appendix. Results show removing each term leads to performance degradation.