acceptodds
Under review as a conference paper at ICLR 2027

Do RTL Representations Understand Circuits? A Diagnostic Benchmark for Functional Understanding and Implementation Awareness

Abstract

Learning-based methods increasingly assist register-transfer level (RTL) circuit design and optimization, making effective RTL representations an important foundation for downstream applications. Such representations should capture shared functional semantics across equivalent implementations while remaining sensitive to implementation differences relevant to design quality. However, existing evaluations of RTL representations assess functional behavior and implementation quality separately and on different designs, leaving it unclear how well representations capture each capability and how the two interact. In this paper, we introduce UnderRTL, a unified diagnostic benchmark for evaluating these two capabilities, termed functional understanding and implementation awareness. Built on 7,075 curated RTL designs, UnderRTL provides verified functional variants, synthesis-based PPA labels across multiple PDKs and constraints, and natural-language queries for six linked tasks. Functionally equivalent implementations bridge the two axes, enabling controlled analysis of functional invariance and implementation sensitivity. UnderRTL integrates 24 engineered-feature, text, graph, and multimodal methods under standardized evaluation protocols. Our experiments reveal task-dependent limitations in both capabilities, as well as weak transfer to unseen designs, motivating more robust and generalizable RTL representations.

Primary area:
datasets and benchmarks
Keywords:
Diagnostic Benchmark, RTL Representation Learning, Electronic Design Automation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_L3iEOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a benchmark, named UnderRTL, to evaluate the quality of RTL representations. The benchmark has two main families: (1) Functional understanding and (2) PPA prediction. The benchmark contributes a considerable number of cases, including 7k+ designs, and evaluates 24+ methods.

Critical strengths

The benchmark contributes a considerable number of cases, including 7k+ designs, and evaluates 24+ methods. I appreciate the authors' efforts in curating the datasets, which I think is their most valuable contribution. The overview illustration is clear and makes it much easier for readers to follow. The experimental figures are beautiful, though hard to understand.

Critical weaknesses

The paper's writing needs significant improvement. I can tell the authors have already worked hard to refine the AI-generated text and improve the paper's structure and layout. But the paper is still painful to read. The biggest problem is the motivation from the argument: "We argue that such a benchmark should provide linkage, comparability, and informativeness." These three features are hard to understand. The authors' central point seems to be that the RTL representation should support both functionality and PPA awareness. I think the only meaningful scenario that requires the RTL representation to be aware of both functionality and PPA is PPA-Aware Retrieval, while the other 5 tasks the paper introduced can be conducted separately, focusing only on either functionality or PPA. So the argument in the introduction section weakens the motivation.

I suggest the authors use Functional understanding and PPA awareness consistently in the paper. The words like functional invariance and implementation sensitivity seem to refer to the same thing. PPA awareness is much clearer than implementation awareness.

The authors also propose RQ1-4, which lack clear definitions and application scenarios, making them hard to understand.

Questions for authors

Q1: In the benchmark tasks, do other tasks, except PPA-Aware Retrieval, really require support for both functionality and PPA awareness? For example, do functional understanding tasks require PPA awareness? Conversely, do PPA prediction & Ranking require functionality awareness?

Q2: Can you provide a clear task formulation for each task, including input, model (supported methods), and output, together with evaluation metrics? Because I am confused about the current benchmarking formulation.

5 comments
AuthorsOpenReview ↗

Response to Reviewer L3iE (1/4)

We thank the reviewer for recognizing the value of UnderRTL curated dataset, comprehensive evaluation of diverse representation methods, and clear overview illustration.

Below, we explain why general-purpose RTL representations need both functional understanding and implementation awareness, and why linked evaluation provides insights beyond individual task scores (W1 & Q1, A1). We then distinguish the two evaluation axes from their specific diagnostic properties (W2, A2), and present the purpose and practical relevance of the four research questions, together with the input, model, output, and evaluation metric for each of the six tasks (W3 & Q2, A3).

W1 & Q1: The paper's writing needs significant improvement. I can tell the authors have already worked hard to refine the AI-generated text and improve the paper's structure and layout. But the paper is still painful to read. The biggest problem is the motivation from the argument: "We argue that such a benchmark should provide linkage, comparability, and informativeness." These three features are hard to understand. The authors' central point seems to be that the RTL representation should support both functionality and PPA awareness. I think the only meaningful scenario that requires the RTL representation to be aware of both functionality and PPA is PPA-Aware Retrieval, while the other 5 tasks the paper introduced can be conducted separately, focusing only on either functionality or PPA. So the argument in the introduction section weakens the motivation. In the benchmark tasks, do other tasks, except PPA-Aware Retrieval, really require support for both functionality and PPA awareness? For example, do functional understanding tasks require PPA awareness? Conversely, do PPA prediction & Ranking require functionality awareness?

A1: We thank the reviewer for the opportunity to further explain the motivation. The motivation for UnderRTL follows from the role of RTL representations in circuit design and optimization. We explain the complementary requirements of functional understanding and implementation awareness, followed by the rationale for separate tasks and linked evaluation.

Why RTL representations need both capabilities. RTL is the stage at which a circuit's function is fixed but its implementation is still being chosen: the same function admits many RTL implementations whose synthesized quality can differ substantially. For example, the three formally equivalent max-of-four circuits in Appendix E.4 (Fig. 13) have WNS values of -0.070, -0.110, and -0.250 ns under the same setting. Many RTL design activities therefore ask two questions at once-what does this design do, and how good is this implementation:

  • Design reuse and retrieval-augmented RTL generation retrieve existing modules that implement the required function and fit an area, power, or timing budget.
  • RTL optimization and candidate selection (e.g., choosing among LLM-generated rewrites) must keep a rewrite functionally equivalent while selecting the one with better PPA; a learned evaluator should therefore both recognize equivalence and rank implementation quality.
  • Early PPA estimation must generalize across functionally diverse designs, where the implemented function largely determines the scale and structure of the resulting circuit.

Moreover, recent RTL representations are pretrained once and reused across such tasks (e.g., CircuitFusion[1], DeepRTL2[2], TopoRTL[3], UniRTL[4]), so a single embedding is expected to serve both purposes.

Importantly, the two capabilities are distinct and can compete during representation learning. Functional understanding requires the information shared by equivalent implementations to remain recoverable, whereas implementation awareness requires the information that distinguishes them to be preserved. A representation can satisfy both by encoding shared function and implementation-specific differences together, but it may also gain one at the expense of the other-for example, a representation that maps all equivalent implementations to the same embedding would appear functionally consistent while losing the ability to distinguish their PPA. Whether existing representations achieve both cannot be inferred from evaluations on unrelated designs, and requires evaluating them on the same functionally related implementations. This is the central motivation of UnderRTL.

[1] CircuitFusion: Multimodal Circuit Representation Learning for Agile Chip Design. ICLR 2025.

[2] DeepRTL2: A Versatile Model for RTL-Related Tasks. ACL 2025.

[3] Topology Matters in RTL Circuit Representation Learning. ICLR 2026.

[4] UniRTL: Unifying Code and Graph for Robust RTL Representation Learning. ICML 2026.

AuthorsOpenReview ↗

Response to Reviewer L3iE (2/4)

Why separate tasks. Each task is designed to emphasize one capability, so that the capability can be measured with clear supervision and failures can be attributed precisely. For example, in practical design reuse, if a representation fails to retrieve, from a library of candidate designs, a circuit that both implements the required function and meets the desired quality, the separate tasks (Function-Aware Retrieval and PPA-Aware Retrieval) show whether the failure stems from recognizing the function or from distinguishing quality. Within each axis, the tasks progress from single circuits, to pairwise or group comparison, to query-driven retrieval.

Importantly, even tasks that emphasize one capability are built on linked data. To answer the reviewer's question in both directions:

  • Functional tasks and implementation. Functional tasks do not ask the model to predict PPA, but Functional Equivalence uses implementation variants as its source of difficulty: a representation must recognize the same function across the very implementation changes that drive PPA differences.
  • PPA tasks and function. PPA tasks do not output functional labels, but PPA Ranking deliberately holds function fixed within verified equivalence groups, so that ranking differences can be attributed to implementation rather than to function. Empirically, the two capabilities are related but not interchangeable: Equivalence AP correlates with Area/Power ranking (ρs = 0.70/0.59) but only weakly with WNS ranking (0.20), with opposite trends for graph and text models.

The three design principles. The table below states what each principle means, and how UnderRTL realizes it:

PrincipleMeaningRealization in UnderRTL
Linkage (task design)Both capabilities are evaluated on the same source circuits and their equivalent variants, so they can be directly comparedEquivalence pairs and PPA Ranking/Retrieval groups share one pool of verified equivalent variants
Comparability (evaluation)All methods are evaluated in the same way, so score gaps reflect the representation abilityShared data and splits; frozen encoders; same head and training setting per task
Informativeness (diagnosis)The benchmark explains why and where models fail, not only how well they scoreControlled analyses of lexical reliance (RQ1), equivalence failure modes (RQ2), capability interplay (RQ3), and generalization gaps (RQ4)

We have added this discussion to Appendix B.1, including the application scenarios that require both capabilities, the relation of each task to both axes, and a table summarizing the three design principles and their realization in UnderRTL.

AuthorsOpenReview ↗

Response to Reviewer L3iE (3/4)

W2: I suggest the authors use Functional understanding and PPA awareness consistently in the paper. The words like functional invariance and implementation sensitivity seem to refer to the same thing. PPA awareness is much clearer than implementation awareness.

A2: We thank the reviewer for the opportunity to further explain the terminology. We clarify the distinction between our two evaluation axes and their corresponding diagnostic properties.

Evaluation axes.

  • Functional Understanding: By functional understanding, we mean whether an RTL representation captures what a circuit does, independent of how it is written. UnderRTL measures it at three levels: recognizing functional categories (Functional Classification), identifying the same function across different implementations (Functional Equivalence), and matching natural-language function intent (Function-Aware Retrieval).
  • Implementation Awareness: By implementation awareness, we mean whether an RTL representation can anticipate the quality of the circuit that downstream implementation flows will produce from a given RTL description. In the RTL-to-GDSII flow, RTL coding choices, such as how comparisons, selections, and state updates are structured, propagate through physical implementation and determine the quality of the realized circuit. Because these effects are normally observed only after running time-consuming downstream tools, a key goal of learning-based RTL modeling is to shift this assessment left to the RTL stage. Implementation awareness therefore refers to sensitivity to cross-stage implementation quality, rather than to any single metric. PPA is the most widely used and directly quantifiable indicator of this quality, and UnderRTL instantiates it through synthesis-derived Area, Power, and timing (Slack, WNS, TNS) across multiple PDKs and clock constraints. We therefore use implementation awareness to name the capability and PPA to name its current evaluation targets.

Functional invariance vs. implementation sensitivity. These terms do not refer to the two axes themselves; they are two complementary properties examined on the same equivalent implementations-whether shared function remains recognizable, and whether implementation differences remain distinguishable. Consider two functionally equivalent adders, one ripple-carry and one carry-lookahead:

  • Functional invariance asks whether the representation still recognizes them as implementing the same function.
  • Implementation sensitivity asks whether it still distinguishes that one is faster and the other smaller.

A representation can possess both, but one does not imply the other: a representation that maps both adders to identical embeddings appears perfectly invariant yet has no implementation sensitivity. We examine exactly this interplay in Section 5.4.

The distinction between functional invariance and implementation sensitivity is stated in Section 3.1, and the rationale for the term implementation awareness is given in Appendix B.1.

Reviewer_Gf3wOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces UnderRTL, a diagnostic benchmark that connects the evaluation of functional understanding and implementation awareness through formally verified equivalent RTL implementations. The design examines whether representations capture shared functional semantics across implementations while preserving differences relevant to power, performance, and area (PPA). Built on 7,075 source RTL designs, the benchmark constructs functional annotations, equivalent and non-equivalent variants, synthesis-derived PPA labels across technologies and constraints, and natural-language queries for six tasks: functional classification, equivalence prediction, function-aware retrieval, PPA prediction, PPA ranking, and PPA-aware retrieval. The paper compares 24 methods using a unified downstream protocol with frozen representations and examines lexical dependence, structural information, equivalence errors, and transfer across environments and design distributions. Text models perform best on functional classification and retrieval, while engineered statistical features extracted directly from RTL code and circuit graphs remain competitive on several PPA prediction and ranking tasks. Associations between equivalence prediction and implementation ranking vary across PPA objectives and model families, and adaptation using target-environment labels does not guarantee direct transfer to larger designs.

Critical strengths

The paper introduces a diagnostic benchmark that connects functional understanding with implementation awareness through equivalent RTL implementations, providing a common evaluation basis for both capabilities. Equivalence prediction examines whether models recognize functionality preserved under implementation changes, while within-group PPA ranking and preference-aware retrieval assess whether they distinguish implementation quality. Establishing these tasks on circuits with explicit functional relationships enables analysis of the relationship between functional semantics and implementation information, constituting an important contribution beyond general RTL datasets and individual task evaluations.

The construction of UnderRTL includes concrete quality controls for functional annotations, equivalence relations, and derived instances. Functional classification comprises six coarse and thirty fine-grained categories, with multiple stages of review and a manual sample audit. Equivalence and non-equivalence labels are established through Yosys-based formal verification, and all variants, subgraphs, and queries inherit their source-circuit split. To mitigate edit-magnitude shortcuts introduced by different generation mechanisms, the paper matches transformation distances between positive and negative pairs, reducing the AUROC of a distance-feature classifier from 0.884 to 0.505. These measures improve label reliability and reduce the influence of incidental cues on equivalence evaluation.

The paper conducts systematic experiments covering text, graph, multimodal, and engineered-feature representations, revealing differences in information dependence and applicability. The engineered baselines distinguish Full features, which combine source-code and structural statistics, from Structure features, which retain only circuit-graph structural descriptors. This distinction enables assessment of the gains from learned representations over directly extracted statistical features. Identifier perturbations, structural-feature augmentation, and equivalence-error analyses further expose capability differences beyond model rankings. The appendix additionally examines matching thresholds, ranking-group selection, differences between combinational and sequential circuits, and register-count scaling, strengthening the diagnostic value of the benchmark.

Critical weaknesses

Some configuration details remain insufficient for complete reproduction of the model comparisons. The appendix describes pretraining principles and downstream protocols, but provides limited detail on the main training hyperparameters, input-length handling, and representation-extraction settings of the adapted models. Complete definitions of the engineered statistical features, together with the key configurations and initial-state assumptions used for formal equivalence checking, are also needed. These details are particularly important for reproducing the competitive feature baselines and core equivalence labels.

The correlation analysis in RQ3 characterizes associations between task scores, but provides a limited explanation of functional invariance and implementation sensitivity. Equivalence AP measures a classifier’s ability to recognize equivalence rather than directly characterizing representation stability under equivalent transformations. The seed-wise analysis supports the stability of some correlation directions, but the within-family samples are small, and WNS ranking is sensitive to equivalent-group selection. Analyses of error associations within the same groups or representative transformations would help explain the opposing trends across model families.

Additional error analysis would improve the interpretation of some PPA regression results. The paper acknowledges that the inverse transformation of logarithmic targets can amplify errors on heavy-tailed samples, but the overall comparison primarily uses R² in the original label space. For extremely negative values and large standard deviations, the influence of a small number of extreme predictions remains unclear. Extending the fixed-scale NRMSE already used in selected stratified analyses to the key regression comparisons would help assess the main model rankings.

Questions for authors

Could you clarify the open-source release plan and the scope of the materials to be released? Specifically, it would be helpful to provide the main configurations of the adapted models and prediction heads, complete definitions of the engineered features, key assumptions used for formal equivalence checking, and the data, precomputed PPA labels, and evaluation scripts. Please also explain whether researchers without commercial EDA tools will be able to reproduce the downstream evaluations using the released materials. Where feasible, representation-extraction time and GPU memory requirements for the main methods would help assess practical usage costs.

Could you further explain the relationship between equivalence prediction and PPA ranking through error associations within the same groups or representative transformation cases? Please also clarify how equivalence AP relates to functional invariance and the scope of the interpretation it supports.

For regression tasks with extremely negative R² values, could you additionally report MAE or fixed-scale NRMSE and examine the contribution of high-error samples to the aggregate results? Are the main model rankings and OOD-transfer conclusions consistent under these metrics?

RTL-OPT [1] also considers RTL implementations of different quality, functional verification, and PPA evaluation. Could you discuss its relationship to UnderRTL and clarify the additional insights provided by multitask diagnosis of frozen representations compared with code-optimization evaluation?

[1] Yao Lu, Shang Liu, Hangan Zhou, Wenji Fang, Qijun Zhang, and Zhiyao Xie. A New Benchmark for the Appropriate Evaluation of RTL Code Optimization. arXiv:2601.01765, 2026. https://arxiv.org/abs/2601.01765

5 comments
AuthorsOpenReview ↗

Response to Reviewer Gf3w (1/4)

We thank the reviewer for recognizing the value of our linked diagnostic framework for functional understanding and implementation awareness, the rigor of our benchmark construction, and the comprehensive evaluation of diverse representation methods.

We have addressed the concerns by clarifying model configurations, verification assumptions, and open-source plans (W1 & Q1, A1), explaining the interpretation of equivalence prediction and providing within-group error analysis (W2 & Q2, A2), supplementing regression and OOD evaluations with alternative metrics and error analysis (W3 & Q3, A3), and distinguishing UnderRTL from RTL-OPT (Q4, A4).

W1 & Q1: Some configuration details remain insufficient for complete reproduction of the model comparisons. The appendix describes pretraining principles and downstream protocols, but provides limited detail on the main training hyperparameters, input-length handling, and representation-extraction settings of the adapted models. Complete definitions of the engineered statistical features, together with the key configurations and initial-state assumptions used for formal equivalence checking, are also needed. These details are particularly important for reproducing the competitive feature baselines and core equivalence labels. Could you clarify the open-source release plan and the scope of the materials to be released? Specifically, it would be helpful to provide the main configurations of the adapted models and prediction heads, complete definitions of the engineered features, key assumptions used for formal equivalence checking, and the data, precomputed PPA labels, and evaluation scripts. Please also explain whether researchers without commercial EDA tools will be able to reproduce the downstream evaluations using the released materials. Where feasible, representation-extraction time and GPU memory requirements for the main methods would help assess practical usage costs.

A1: We thank the reviewer for the constructive suggestions on improving the reproducibility and accessibility of UnderRTL. Our benchmark aims to enable fair comparisons across diverse representation methods through standardized adaptation and evaluation, while preserving their core architectures and learning mechanisms.

For model configurations, UnderRTL integrates engineered-feature, graph, text, and multimodal methods. Adapted graph and multimodal models generally use 768-dimensional output embeddings, with native settings retained where appropriate. Text models use publicly available Hugging Face checkpoints and their corresponding representation-extraction procedures. Models requiring RTL pretraining are trained for 50 epochs with an effective batch size of 64, using gradient accumulation for memory-intensive models. All encoders are frozen during downstream evaluation and paired with MLP prediction heads sharing the same hidden-layer configuration. We have added Appendix D.5 with the main model configurations, input-length handling, representation-extraction settings, complete engineered-feature definitions, and shared training protocols.

For equivalence verification, we use Yosys to construct a miter for each circuit pair with shared primary inputs and compare their outputs using SAT-based induction. States not otherwise initialized in the formal model are initialized to zero, with no additional reset sequences imposed. Asynchronous circuits are converted to synchronous representations for verification. We have clarified these settings in Appendices C.3 to support reproducibility and interpretation of the benchmark labels.

For open-source release, we plan to release the raw and processed datasets, task labels, model configurations, and code for pretraining, representation extraction, and downstream evaluation upon acceptance. Importantly, the released data will include precomputed PPA labels, allowing researchers without commercial EDA tools to reproduce downstream evaluations and develop new models without rerunning synthesis and timing analysis. Our experiments were conducted on NVIDIA A800 80GB GPUs.

AuthorsOpenReview ↗

Response to Reviewer Gf3w (2/4)

W2 & Q2: The correlation analysis in RQ3 characterizes associations between task scores, but provides a limited explanation of functional invariance and implementation sensitivity. Equivalence AP measures a classifier's ability to recognize equivalence rather than directly characterizing representation stability under equivalent transformations. The seed-wise analysis supports the stability of some correlation directions, but the within-family samples are small, and WNS ranking is sensitive to equivalent-group selection. Analyses of error associations within the same groups or representative transformations would help explain the opposing trends across model families. Could you further explain the relationship between equivalence prediction and PPA ranking through error associations within the same groups or representative transformation cases? Please also clarify how equivalence AP relates to functional invariance and the scope of the interpretation it supports.

A2: We thank the reviewer for this suggestion. We have clarified the interpretation of Functional Equivalence AP and added a within-group case analysis in Appendix E.4 to examine how equivalence recognition and implementation-quality discrimination behave on the same RTL implementations.

1. Interpretation of Equivalence AP and functional invariance.

We use Equivalence AP as a task-level measure of functional invariance: it evaluates whether frozen representations support recognition of shared functionality across different RTL implementations while distinguishing non-equivalent circuits. It does not directly measure geometric invariance of embeddings. Equivalent RTL implementations can differ substantially in comparison, selection, and dependency structures, and these differences may encode information relevant to PPA. An embedding-distance criterion alone cannot distinguish functional differences from implementation-specific differences. Our evaluation therefore measures whether functional equivalence remains recoverable from the representations while implementation differences remain useful for PPA ranking. Accordingly, Equivalence AP characterizes functional consistency through downstream discrimination, rather than requiring equivalent implementations to have identical or nearby embeddings.

2. Within-group error analysis.

To examine how these two capabilities interact at the prediction level, we added a case study involving three formally verified equivalent RTL implementations. All three compute the maximum of four unsigned 4-bit inputs using different comparison and selection structures:

  • A: Encoded selection using comparison-generated selection codes (WNS = -0.070 ns).
  • B: Priority-based selection using casez (WNS = -0.110 ns).
  • C: Cascaded pairwise comparisons (WNS = -0.250 ns).

Their synthesized timing establishes the ground-truth order A > B > C. We compare equivalence predictions and WNS ranking decisions on the same A-C pair across three downstream seeds:

ModelEquivalence predictionWNS ranking
Graphormer (Graph)Correct (3/3)Incorrect (3/3)
Qwen3-Embedding (Text)Incorrect (3/3)Correct (3/3)
Qwen2.5-Coder (Text)Correct (3/3)Correct (3/3)

Graphormer consistently recognizes functional equivalence (mean probability 0.994) but reverses the timing order. Qwen3-Embedding shows the opposite pattern, failing equivalence recognition (mean probability 0.409) while correctly ranking WNS. Qwen2.5-Coder succeeds on both decisions (mean equivalence probability 0.540).

These contrasting outcomes demonstrate that functional equivalence recognition and implementation-quality discrimination can succeed or fail independently on the same circuit pair. The case provides direct evidence of distinct prediction behaviors that aggregate task scores alone do not reveal.

3. Interpretation of model-family differences.

The within-group observations complement the family-level correlations: models differ in how successfully they recover functional relationships and distinguish timing-relevant implementation differences. The three prediction patterns illustrate why stronger equivalence recognition does not necessarily imply better WNS ranking, and why the relationship between the two capabilities can differ across representation architectures.

We also examine the stability of these associations across random seeds and equivalent-group selection criteria. The opposing WNS correlation directions persist across seeds for graph models (0.702) and text models (-0.693). WNS ranking is more sensitive to group selection, with a model-ranking Spearman correlation of 0.490 between the first and last separation settings. Together, these analyses characterize the observed family-dependent associations and their sensitivity to evaluation conditions, while the new case study shows how divergent functional and timing decisions arise on identical implementations.

AuthorsOpenReview ↗

Response to Reviewer Gf3w (3/4)

W3 & Q3: Additional error analysis would improve the interpretation of some PPA regression results. The paper acknowledges that the inverse transformation of logarithmic targets can amplify errors on heavy-tailed samples, but the overall comparison primarily uses R2 in the original label space. For extremely negative values and large standard deviations, the influence of a small number of extreme predictions remains unclear. Extending the fixed-scale NRMSE already used in selected stratified analyses to the key regression comparisons would help assess the main model rankings. For regression tasks with extremely negative R2 values, could you additionally report MAE or fixed-scale NRMSE and examine the contribution of high-error samples to the aggregate results? Are the main model rankings and OOD-transfer conclusions consistent under these metrics?

A3: We thank the reviewer for this suggestion. We have extended the PPA regression analysis in Appendix E.6 with fixed-scale NRMSE and error-concentration measurements to examine the sources of extreme regression errors, the stability of model rankings, and the consistency of OOD generalization conclusions.

1. Extreme regression errors and their contribution.

We evaluate the original selected prediction heads without retraining or checkpoint reselection. NRMSE is normalized by the standard deviation of the corresponding IID training labels, while Top-1 SSE measures the largest-error sample's percentage contribution to the total squared error.

Our analysis reveals that several extremely negative R2 values are dominated by individual high-error predictions. For GraphCLIP's Area prediction, a single circuit contributes 99.926% and 99.408% of the total squared error in seeds 0 and 42, respectively, compared with 10.559% in seed 615, explaining the large variation across seeds. Similarly, the largest-error sample accounts for 99.770% and 99.474% of TNS squared error for GCN and GIN, respectively, averaged across three seeds.

These high-error cases are also diagnostically valuable. They reveal severe prediction failures that aggregate model rankings may conceal. Retaining them in UnderRTL enables the assessment of tail robustness and helps identify models prone to large PPA estimation errors, complementing average regression performance with a more detailed characterization of prediction reliability.

2. Model ranking consistency.

The main model rankings remain stable under fixed-scale NRMSE. Across three-seed averages, the Spearman correlations between R2 and negative NRMSE are 0.998, 1.000, and 1.000 for Area, Power, and TNS, respectively. Only TopoRTL and RTLCoder-DS exchange their positions among the leading Area models.

Fixed-scale NRMSE additionally provides interpretable error magnitudes, while Top-1 SSE reveals how strongly individual predictions influence the aggregate results. Switching to fixed-scale NRMSE largely preserves the model rankings.

3. OOD generalization consistency.

We further evaluate direct OOD transfer and five-/ten-shot adaptation using the same fixed-scale NRMSE, allowing prediction errors to be compared across IID and OOD distributions without changes in the normalization denominator.

For Area and Power, all eight representative models exhibit higher NRMSE under direct OOD transfer than under IID evaluation, confirming substantial cross-design degradation. Ten-shot adaptation reduces NRMSE for seven of eight models on both targets, although errors remain substantially above their IID levels. This agrees with the original R2 results, where seven of eight models retain negative scores after ten-shot adaptation.

For WNS, the behavior differs. Only one of eight models exhibits higher direct OOD NRMSE than IID NRMSE, and ten-shot adaptation improves five of eight models. Thus, negative OOD R2 values for WNS do not necessarily correspond to increased absolute prediction errors relative to IID, highlighting the influence of target-variance differences.

Overall, fixed-scale NRMSE confirms the main conclusions on model rankings and the difficulty of cross-design Area/Power prediction, while the error-concentration analysis identifies severe sample-level failures and the WNS results reveal a distinct distribution-shift pattern. The corresponding results and analyses have been added to Appendix E.6.

Reviewer_gkC3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces UnderRTL, a benchmark for evaluating frozen RTL representations along two axes: functional understanding and implementation awareness. It curates 7,075 source designs, constructs formally verified equivalent and non-equivalent variants, and obtains synthesis-derived PPA labels across six PDK–constraint settings. The six tasks cover functional classification, equivalence discrimination, functional retrieval, PPA prediction, PPA ranking within functionally equivalent implementation groups, and preference-conditioned retrieval. Under a shared downstream evaluation protocol, the benchmark compares 24 engineered-feature, text, graph, and multimodal methods.

Critical strengths
  1. Comparing PPA within formally equivalent groups is a useful way to test whether representations capture implementation differences while holding functionality fixed. I found the data controls convincing: source-level splits, deduplication, formal checks, and distance matching all address concrete concerns. The distance-only probe's AUROC drop from 0.884 to 0.505 is good evidence that simple edit-magnitude cues have been reduced.

  2. The comparison of 24 methods, including explicit-feature baselines and results averaged over three random seeds, gives the benchmark practical value. Lexical perturbation experiments and transfer tests on larger IP designs reveal limitations that the overall task-level results alone would not show.

Critical weaknesses

Section 3.2 (page 5) describes formally verified equivalence pairs and synthesis-derived PPA labels, but the conditions defining these labels are unclear. Appendices C.3–C.4 give the verification flow, tools, PDKs, and clock settings, but I could not find the reset/initial-state assumptions for sequential equivalence or the switching-activity assumptions for power. These choices determine what Ci ≡ Cj and the power rankings mean. Making these conditions explicit would help readers interpret and reproduce two central parts of the benchmark.

Questions for authors
  1. What reset, initial-state, and input assumptions define the equivalence labels in Section 3.2? What switching-activity assumptions define the power labels? Could these be specified alongside the verification and synthesis settings in Appendices C.3–C.4?

  2. D.4 describes subgraph pooling for WNS/TNS, and E.6 reports WNS results for 507 test circuits with Nreg = 0. For methods using register-centered subgraphs, what fallback is used? What I/O timing constraints define the labels for these combinational circuits?

  3. For Function-Aware Retrieval, have you audited whether candidates other than the designated source circuit also satisfy a query? If multiple candidates are relevant, how are they handled when computing MRR?

3 comments
AuthorsOpenReview ↗

Response to Reviewer gkC3 (1/2)

We thank the reviewer for recognizing the value of our linked diagnostic framework, rigorous benchmark construction, comprehensive evaluation of diverse methods, and analyses of representation limitations and generalization challenges.

We have addressed the concerns by clarifying the assumptions underlying equivalence verification and power estimation (W1 & Q1, A1), explaining the representation and timing evaluation of combinational circuits (Q2, A2), and clarifying the single-positive retrieval protocol, reporting a supplementary relevance audit, and discussing extensions to multi-positive annotations (Q3, A3).

W1 & Q1: Section 3.2 (page 5) describes formally verified equivalence pairs and synthesis-derived PPA labels, but the conditions defining these labels are unclear. Appendices C.3-C.4 give the verification flow, tools, PDKs, and clock settings, but I could not find the reset/initial-state assumptions for sequential equivalence or the switching-activity assumptions for power. These choices determine what Ci ≡ Cj and the power rankings mean. Making these conditions explicit would help readers interpret and reproduce two central parts of the benchmark. What reset, initial-state, and input assumptions define the equivalence labels in Section 3.2? What switching-activity assumptions define the power labels? Could these be specified alongside the verification and synthesis settings in Appendices C.3-C.4?

A1: We thank the reviewer for the opportunity to further clarify the assumptions underlying these labels.

For equivalence verification, we use Yosys to construct a miter for each circuit pair with shared primary inputs and compare their outputs using SAT-based induction. States not otherwise initialized in the formal model are initialized to zero, with no additional reset sequences imposed. Asynchronous circuits are converted to synchronous representations for verification.

For power estimation, we use PrimeTime's default switching-activity settings with activity propagation (vectorless power analysis). The power labels are computed as the sum of internal, switching, and leakage power. This provides a consistent basis for power comparisons across circuits under the same activity model.

We have clarified these settings in Appendices C.3-C.4 to support reproducibility and interpretation of the benchmark labels.

Q2: D.4 describes subgraph pooling for WNS/TNS, and E.6 reports WNS results for 507 test circuits with Nreg = 0. For methods using register-centered subgraphs, what fallback is used? What I/O timing constraints define the labels for these combinational circuits?

A2: We thank the reviewer for the opportunity to clarify how combinational circuits are handled in representation learning and timing evaluation.

For combinational circuits without registers, we adopt two strategies depending on the representation method.

  • Register-centered subgraph-based models: We treat the entire combinational circuit as a single subgraph, allowing these models to process circuits without registers using the same representation-extraction pipeline.
  • RIG-based models: we represent the circuit as a single-node RIG rather than an empty graph. Both mean and max pooling then reduce to the embedding of this node.

These strategies allow both types of models to handle combinational circuits without excluding them or changing their downstream prediction interfaces.

For combinational timing evaluation, we use a virtual clock to constrain input-to-output paths. Under the default Nangate45 setting, the virtual clock period is 0.5 ns, with maximum/minimum input and output delays of 0.05/0.01 ns and setup/hold uncertainties of 0.05/0.02 ns. WNS and TNS labels are then obtained through PrimeTime static timing analysis under these constraints, ensuring that combinational circuits have well-defined timing labels despite having no registers.

AuthorsOpenReview ↗

Response to Reviewer gkC3 (2/2)

Q3: For Function-Aware Retrieval, have you audited whether candidates other than the designated source circuit also satisfy a query? If multiple candidates are relevant, how are they handled when computing MRR?

A3: We thank the reviewer for the constructive suggestion and the opportunity to clarify our relevance annotation and evaluation protocol.

In our benchmark, we adopt a single-positive relevance protocol, where each query is constructed from the functional description of a source circuit, which serves as its sole annotated positive. Accordingly, MRR measures the retrieval rank of the designated source circuit, providing a consistent and reproducible basis for comparing different representations.

To examine multiple relevant candidates, we conducted a sampled audit. We randomly sampled five test queries and inspected the top-5 candidates retrieved by Qwen3-Embedding-8B under each of the three downstream seeds. Every candidate other than the designated source circuit was checked against the query at the RTL-code level and assigned to one of three mutually exclusive categories:

  • Fully satisfies: implements all functions explicitly required by the query.
  • Partial overlap: implements some required sub-functions but misses at least one explicit requirement.
  • No match: does not implement the required functions.
QueryRequested functionOther candidatesFully satisfiesPartial overlapNo match
TILE-VTXVertex fetch, four-corner tile construction, shared-vertex reuse120 (0.0%)4 (33.3%)8 (66.7%)
CDC-TIMERProgrammable timer, cross-clock synchronization, falling-edge interrupt150 (0.0%)15 (100.0%)0 (0.0%)
OP-ALUOpcode-selected arithmetic, registered result, valid flag131 (7.7%)12 (92.3%)0 (0.0%)
STORE-ALIGNStore-data alignment, byte mask gated by valid write requests120 (0.0%)12 (100.0%)0 (0.0%)
BLOCK-PARITYParameterized blocking, per-block even parity, parity-vector output120 (0.0%)5 (41.7%)7 (58.3%)
All-641 (1.6%)48 (75.0%)15 (23.4%)

Counts are occurrences of candidates other than the source circuit in the top-5 results across three seeds.

The audit shows that additional fully satisfying candidates are rare: only 1 of the 64 other candidates met all requirements of its query. This candidate was ranked below the designated source circuit, so treating it as an additional positive would not change MRR.

Most other top-ranked candidates share some sub-functions with the query but miss explicit requirements. Examples include a timer without cross-clock synchronization, an arithmetic unit without a result-valid output, and a store-alignment module whose byte mask is not gated by valid write requests. This indicates that the queries specify complete functional behavior rather than a broad circuit type, so the designated source circuit remains a well-defined target. It also shows that the task requires distinguishing complete functional matches from closely related circuits, which is the fine-grained functional understanding that Function-Aware Retrieval is designed to assess.

We also agree that extending relevance annotations is a valuable suggestion. One possible approach is to pool top-ranked candidates from multiple models and have domain experts assess their relevance to each query, enabling multi-positive or graded relevance labels. We will explore this direction in future work.

I thank the authors for their detailed response, which has resolved most of my concerns. I decide to maintain my recommendation for acceptance.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their insightful comments and constructive feedback. We are encouraged that reviewers recognized our linked diagnostic framework (gkC3, Gf3w), the value and rigor of our curated benchmark (L3iE, gkC3, Gf3w), the comprehensive evaluation of diverse methods (L3iE, gkC3, Gf3w), and the clear overview illustration (L3iE).

1. Key Rebuttal Highlights

I. Clarifying Motivation, Terminology, and Task Formulation

  • Motivation and design principles (L3iE W1&Q1): We explained why RTL representations need both capabilities: RTL is the stage at which a circuit's function is fixed but its implementation is still being chosen, and the same function admits many RTL implementations whose synthesized quality can differ substantially (example in Appendix E.4). Through design reuse, rewrite selection, and early PPA estimation, we showed that one embedding is expected to serve both purposes, and that the two capabilities can compete (e.g., mapping all equivalent implementations to one embedding is invariant yet blind to PPA), so both should be evaluated on the same functionally related implementations. We further explained that each task emphasizes one capability for precise failure attribution while sharing linked data (e.g., equivalence pairs vary implementation, while ranking groups fix function), and tabulated the meaning and realization of the three design principles.
  • Terminology (L3iE W2): We distinguished the evaluation axes from their diagnostic properties. Implementation awareness names the capability of anticipating downstream implementation quality, with PPA as its quantitative target, whereas functional invariance and implementation sensitivity are two complementary properties examined on the same equivalent implementations.
  • RQs and task formulations (L3iE W3&Q2): We clarified that the six tasks measure specific capabilities, while the four RQs diagnose what information supports them, where they fail, and how they generalize; we summarized each RQ and specified the input, output, and metric of each task.
  • Relation to RTL-OPT (Gf3w Q4): We compared the two benchmarks in evaluation goal, target, role of equivalence and PPA, and insights. RTL-OPT evaluates LLM-based RTL optimization, whereas UnderRTL targets RTL representation learning and diagnoses frozen representations along both capabilities; the two are complementary.

II. Clarifying Labels and Reproducibility

  • Label assumptions (gkC3 W1&Q1, Q2; Gf3w W1&Q1): We specified the input, initial-state, and reset assumptions for formal equivalence and the vectorless switching-activity setting for power. For combinational circuits without registers, we explained how subgraph- and RIG-based models represent them and how a virtual clock defines their timing labels.
  • Reproducibility (Gf3w W1&Q1): We added engineered-feature definitions and model configurations, and committed to releasing data, splits, precomputed PPA labels, and code, enabling evaluation without commercial EDA tools.

III. Deepening Diagnostic Analyses

  • Capability relationship (Gf3w W2&Q2): We clarified that Equivalence AP measures functional consistency at the task level rather than geometric invariance, and added a within-group case study: on the same equivalent pair, Graphormer recognizes equivalence but misranks WNS, while Qwen3-Embedding shows the opposite.
  • Regression errors and OOD transfer (Gf3w W3&Q3): We added fixed-scale NRMSE and error-concentration analyses, showing that several extreme negative R2 values are dominated by individual samples while model rankings remain stable. The OOD analysis confirms Area/Power degradation and attributes negative WNS R2 mainly to target-variance shift.
  • Retrieval relevance (gkC3 Q3): We clarified the single-positive protocol and audited sampled top-5 candidates: only 1 of 64 non-source candidates fully satisfied its query and ranked below the source, leaving MRR unchanged.

2. Revision Summary (marked in blue)

  • Sections 1–3.1 & Table 2: Refined design principles, added RTL-OPT discussion, and clarified terminology and task formulations (L3iE, Gf3w).
  • Appendix B.1: Added motivation, task–axis relations, and Tables 4–5 on design principles and RQs (L3iE).
  • Appendices C.3–C.4 & D.5: Specified label assumptions and added implementation details (gkC3, Gf3w).
  • Appendices E.4 & E.6.2: Added the within-group case study and regression/OOD error analyses (Gf3w).

3. Post-Rebuttal Feedback

We thank all reviewers for their responses; all hold positive ratings:

  • Reviewer gkC3: most concerns resolved; maintained acceptance.
  • Reviewer Gf3w: the additions address the main concerns; raised the rating from 3 to 4.
  • Reviewer L3iE: found our clarification useful; kept weak acceptance.

We once again thank the Area Chair and all reviewers for their time and valuable suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.