- Rating
- 3
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a benchmark, named UnderRTL, to evaluate the quality of RTL representations. The benchmark has two main families: (1) Functional understanding and (2) PPA prediction. The benchmark contributes a considerable number of cases, including 7k+ designs, and evaluates 24+ methods.
Critical strengths
The benchmark contributes a considerable number of cases, including 7k+ designs, and evaluates 24+ methods. I appreciate the authors' efforts in curating the datasets, which I think is their most valuable contribution. The overview illustration is clear and makes it much easier for readers to follow. The experimental figures are beautiful, though hard to understand.
Critical weaknesses
The paper's writing needs significant improvement. I can tell the authors have already worked hard to refine the AI-generated text and improve the paper's structure and layout. But the paper is still painful to read.
The biggest problem is the motivation from the argument: "We argue that such a benchmark should provide linkage, comparability, and informativeness." These three features are hard to understand. The authors' central point seems to be that the RTL representation should support both functionality and PPA awareness. I think the only meaningful scenario that requires the RTL representation to be aware of both functionality and PPA is PPA-Aware Retrieval, while the other 5 tasks the paper introduced can be conducted separately, focusing only on either functionality or PPA. So the argument in the introduction section weakens the motivation.
I suggest the authors use Functional understanding and PPA awareness consistently in the paper. The words like functional invariance and implementation sensitivity seem to refer to the same thing. PPA awareness is much clearer than implementation awareness.
The authors also propose RQ1-4, which lack clear definitions and application scenarios, making them hard to understand.
Questions for authors
Q1: In the benchmark tasks, do other tasks, except PPA-Aware Retrieval, really require support for both functionality and PPA awareness? For example, do functional understanding tasks require PPA awareness? Conversely, do PPA prediction & Ranking require functionality awareness?
Q2: Can you provide a clear task formulation for each task, including input, model (supported methods), and output, together with evaluation metrics? Because I am confused about the current benchmarking formulation.
5 comments
Response to Reviewer L3iE (2/4)
Why separate tasks. Each task is designed to emphasize one capability, so that the capability can be measured with clear supervision and failures can be attributed precisely. For example, in practical design reuse, if a representation fails to retrieve, from a library of candidate designs, a circuit that both implements the required function and meets the desired quality, the separate tasks (Function-Aware Retrieval and PPA-Aware Retrieval) show whether the failure stems from recognizing the function or from distinguishing quality. Within each axis, the tasks progress from single circuits, to pairwise or group comparison, to query-driven retrieval.
Importantly, even tasks that emphasize one capability are built on linked data. To answer the reviewer's question in both directions:
- Functional tasks and implementation. Functional tasks do not ask the model to predict PPA, but Functional Equivalence uses implementation variants as its source of difficulty: a representation must recognize the same function across the very implementation changes that drive PPA differences.
- PPA tasks and function. PPA tasks do not output functional labels, but PPA Ranking deliberately holds function fixed within verified equivalence groups, so that ranking differences can be attributed to implementation rather than to function. Empirically, the two capabilities are related but not interchangeable: Equivalence AP correlates with Area/Power ranking (ρs = 0.70/0.59) but only weakly with WNS ranking (0.20), with opposite trends for graph and text models.
The three design principles. The table below states what each principle means, and how UnderRTL realizes it:
| Principle | Meaning | Realization in UnderRTL |
|---|---|---|
| Linkage (task design) | Both capabilities are evaluated on the same source circuits and their equivalent variants, so they can be directly compared | Equivalence pairs and PPA Ranking/Retrieval groups share one pool of verified equivalent variants |
| Comparability (evaluation) | All methods are evaluated in the same way, so score gaps reflect the representation ability | Shared data and splits; frozen encoders; same head and training setting per task |
| Informativeness (diagnosis) | The benchmark explains why and where models fail, not only how well they score | Controlled analyses of lexical reliance (RQ1), equivalence failure modes (RQ2), capability interplay (RQ3), and generalization gaps (RQ4) |
We have added this discussion to Appendix B.1, including the application scenarios that require both capabilities, the relation of each task to both axes, and a table summarizing the three design principles and their realization in UnderRTL.
Response to Reviewer L3iE (3/4)
W2: I suggest the authors use Functional understanding and PPA awareness consistently in the paper. The words like functional invariance and implementation sensitivity seem to refer to the same thing. PPA awareness is much clearer than implementation awareness.
A2: We thank the reviewer for the opportunity to further explain the terminology. We clarify the distinction between our two evaluation axes and their corresponding diagnostic properties.
Evaluation axes.
- Functional Understanding: By functional understanding, we mean whether an RTL representation captures what a circuit does, independent of how it is written. UnderRTL measures it at three levels: recognizing functional categories (Functional Classification), identifying the same function across different implementations (Functional Equivalence), and matching natural-language function intent (Function-Aware Retrieval).
- Implementation Awareness: By implementation awareness, we mean whether an RTL representation can anticipate the quality of the circuit that downstream implementation flows will produce from a given RTL description. In the RTL-to-GDSII flow, RTL coding choices, such as how comparisons, selections, and state updates are structured, propagate through physical implementation and determine the quality of the realized circuit. Because these effects are normally observed only after running time-consuming downstream tools, a key goal of learning-based RTL modeling is to shift this assessment left to the RTL stage. Implementation awareness therefore refers to sensitivity to cross-stage implementation quality, rather than to any single metric. PPA is the most widely used and directly quantifiable indicator of this quality, and UnderRTL instantiates it through synthesis-derived Area, Power, and timing (Slack, WNS, TNS) across multiple PDKs and clock constraints. We therefore use implementation awareness to name the capability and PPA to name its current evaluation targets.
Functional invariance vs. implementation sensitivity. These terms do not refer to the two axes themselves; they are two complementary properties examined on the same equivalent implementations-whether shared function remains recognizable, and whether implementation differences remain distinguishable. Consider two functionally equivalent adders, one ripple-carry and one carry-lookahead:
- Functional invariance asks whether the representation still recognizes them as implementing the same function.
- Implementation sensitivity asks whether it still distinguishes that one is faster and the other smaller.
A representation can possess both, but one does not imply the other: a representation that maps both adders to identical embeddings appears perfectly invariant yet has no implementation sensitivity. We examine exactly this interplay in Section 5.4.
The distinction between functional invariance and implementation sensitivity is stated in Section 3.1, and the rationale for the term implementation awareness is given in Appendix B.1.
Response to Reviewer L3iE (1/4)
We thank the reviewer for recognizing the value of UnderRTL curated dataset, comprehensive evaluation of diverse representation methods, and clear overview illustration.
Below, we explain why general-purpose RTL representations need both functional understanding and implementation awareness, and why linked evaluation provides insights beyond individual task scores (W1 & Q1, A1). We then distinguish the two evaluation axes from their specific diagnostic properties (W2, A2), and present the purpose and practical relevance of the four research questions, together with the input, model, output, and evaluation metric for each of the six tasks (W3 & Q2, A3).
A1: We thank the reviewer for the opportunity to further explain the motivation. The motivation for UnderRTL follows from the role of RTL representations in circuit design and optimization. We explain the complementary requirements of functional understanding and implementation awareness, followed by the rationale for separate tasks and linked evaluation.
Why RTL representations need both capabilities. RTL is the stage at which a circuit's function is fixed but its implementation is still being chosen: the same function admits many RTL implementations whose synthesized quality can differ substantially. For example, the three formally equivalent max-of-four circuits in Appendix E.4 (Fig. 13) have WNS values of -0.070, -0.110, and -0.250 ns under the same setting. Many RTL design activities therefore ask two questions at once-what does this design do, and how good is this implementation:
Moreover, recent RTL representations are pretrained once and reused across such tasks (e.g., CircuitFusion[1], DeepRTL2[2], TopoRTL[3], UniRTL[4]), so a single embedding is expected to serve both purposes.
Importantly, the two capabilities are distinct and can compete during representation learning. Functional understanding requires the information shared by equivalent implementations to remain recoverable, whereas implementation awareness requires the information that distinguishes them to be preserved. A representation can satisfy both by encoding shared function and implementation-specific differences together, but it may also gain one at the expense of the other-for example, a representation that maps all equivalent implementations to the same embedding would appear functionally consistent while losing the ability to distinguish their PPA. Whether existing representations achieve both cannot be inferred from evaluations on unrelated designs, and requires evaluating them on the same functionally related implementations. This is the central motivation of UnderRTL.
[1] CircuitFusion: Multimodal Circuit Representation Learning for Agile Chip Design. ICLR 2025.
[2] DeepRTL2: A Versatile Model for RTL-Related Tasks. ACL 2025.
[3] Topology Matters in RTL Circuit Representation Learning. ICLR 2026.
[4] UniRTL: Unifying Code and Graph for Robust RTL Representation Learning. ICML 2026.