acceptodds
Under review as a conference paper at ICLR 2027

ChipFactory: Capturing the World of Chip Design at Scale

Abstract

Advancing AI for chip design requires models that can understand complex hardware, optimize designs, and assess their functional and implementation outcomes. Studying these capabilities calls for data that connect diverse design contexts and implementation choices with observable EDA backend feedback. However, previous work rarely combines design complexity, diversity, and backend feedback richness at scale. We present ChipFactory, a dataset of over 1 million design records covering complex hardware across diverse sources, abstraction levels, and implementation choices, with rich backend feedback from simulation through physical implementation. ChipFactory combines designs from generators and compilers, external datasets, human-designed accelerators, and software programs, relying on HLS together with AI agents to expand the latter two into design variants at scale. Reusable agent skills and an open harness let researchers build their own benchmarks from these design records. We instantiate 13 tasks spanning implementation prediction, behavioral prediction, ranking, and generation, and evaluate twelve LLMs, from state-of-the-art proprietary models to open-source and domain-specialized ones. Even the strongest model, Claude Fable 5.1, achieves an average score of only 58.4 out of 100, and the four smallest open-source models solve none of the generation problems. ChipFactory is open source to help the community expose the challenges models still face in reasoning across hardware representations and train specialized models for chip design.

Primary area:
datasets and benchmarks
Keywords:
AI for chip design, dataset, large language models, LLM agents, hardware design, high-level synthesis, electronic design automation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TtPFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a dataset of over 1 million design records to address the lack of complex, diverse, and complete data in digital chip design, specifically connecting diverse hardware design representations and implementation choices with observable EDA backend feedback. To complement this, the authors introduce an agent-driven data generation pipeline and a benchmark suite evaluating different LLMs across prediction, ranking, and generation tasks. By highlighting the performance gaps still exhibited by current language models, the work demonstrates how this dataset can serve as a foundation for training specialized models and advancing AI-driven EDA.

Critical strengths
  1. ChipFactory addresses an important gap in hardware-design datasets by combining large sample quantity with detailed design artifacts and backend implementation feedback. Its more than one million design records connect diverse hardware representations to information spanning RTL simulation, synthesis, placement, and routing. This linkage is particularly valuable because it enables research not only on code generation, but also on predicting implementation outcomes and studying trade-offs between design choices and hardware metrics. The dataset therefore provides a potentially useful foundation for developing and evaluating LLM-based hardware design tools.

  2. The paper introduces 13 tasks across implementation prediction, behavioral prediction, ranking, and generation, and evaluates 12 models spanning proprietary, open-source, and specialized systems. This provides a structured baseline across several distinct capabilities rather than focusing exclusively on code generation or a single prediction task. The benchmark's breadth can help researchers identify where current models perform well or struggle, compare approaches under a common evaluation framework, and guide future work on more capable hardware-design agents.

Critical weaknesses
  1. The dataset's scale and complexity come largely from agent-driven and from automated pragma and configuration sweeps. The paper does not establish that this pipeline results in functionally valid and non-redundant variants. There is no description of how behavioral preservation is checked between the original program and the agent's HLS product. If no tests exist, the testbench is also agent-written, so the design and the check may share the same interpretation of the intended behavior. Only 86,337 of the 1,006,725 records have retained RTL-simulation waveforms, and Table 8 reports 26,664 simulation failures with no breakdown of cause or stated denominator. Backend measurements remain valid for the implementation that was synthesized, but functionally incorrect variants undermine conclusions about trade-offs between supposedly equivalent designs. This is directly relevant to the ranking tasks (T08-T10), which compare variants of one base design. Additionally, the parameter spaces, sweep bounds, and filtering criteria are not described, and the paper does not quantify near-duplicates. The record count therefore does not show how much independent architectural diversity the dataset contains, and it is unclear how related variants are handled in train/test splits and benchmark construction.

  2. Each benchmark problem is evaluated in a single agent episode, which limits the ability to assess the variability and reproducibility of the reported results. Agent-based tasks may be sensitive to stochastic generation and differences in the trajectories explored during problem solving. Without repeated independent runs, variance estimates, or statistical significance analysis, it is difficult to determine whether observed performance differences between models reflect consistent advantages or run-specific outcomes. This limits the strength of the conclusions drawn from model comparisons. Reporting results across multiple independent episodes, together with appropriate measures of variability and statistical comparisons, would provide stronger evidence for the robustness of the benchmark findings.

Questions for authors
  1. Does the RTL LOC and hierarchy-depth measurement (App. A.1) account for verbose or boilerplate code emitted by HLS tools and generators (wrappers, FSMs, interface logic, library modules), or is it counted as-is?
  2. In the comparison with prior datasets, will all fields and artifacts listed as planned for ChipFactory be publicly released upon acceptance, or will some remain unavailable or restricted?
  3. Have the authors compared token consumption and execution costs across models? Can failures caused by output-format violations, invalid answers, or exhaustion of token, time, or turn budgets be distinguished from failures reflecting a model's underlying task-solving ability?
  4. Although the paper discusses chip design broadly, the dataset and benchmark appear to focus on digital circuits. Could the authors clarify the intended scope of ChipFactory and whether extending it to analog or mixed-signal design is a planned direction?
2 comments
AuthorsOpenReview ↗

Response to Reviewer TtPF (1/2)

We thank the reviewer for the careful review and constructive suggestions. In the revision, we described variant generation and functional verification in a new Appendix B.2, extended the simulation coverage with a breakdown of RTL simulation failures, and added a repeat-stability experiment and an analysis of cost and failure types.

W1.1 (parameter space, functional validity and non-redundancy of the variant-generation pipeline).

Parameter space. Appendix B.2 describes, for each source class, how variants are generated and which admission rules apply. Table 8 lists the parameters swept in each class, and Table 9 shows that agent-written design records are only 30.9% of the total.

Functional validity. Whether a variant keeps its function depends on how it was generated. Pragma sweeps change only how the HLS tool implements the program, not what the program computes. Compiler transformations preserve semantics by construction. Configuration sweeps over workload, data types and similar parameters change the function on purpose, and each configuration counts as a distinct design. Agent rewrites have no such guarantee, so they must pass testbench verification (see W1.2).

Non-redundancy. Of the variants whose normalized source or RTL is identical, we keep only one, and we merge pragma-only variants with identical synthesis resources into one. Variants with the same function but different implementations stay, because their QoR differs, and that difference is what the dataset is meant to provide.

Revised: Appendix B.2.

W1.2 (checking behavior preservation from the original program to the agent's HLS).

In ChipFactory, the software program is the golden functional reference. Expected outputs come from running the program, so agents write only test inputs, and every HLS design derived from the program must reproduce its outputs on hidden tests. The testbench must reach 100% branch coverage of the reference C/C++ code and detect every non-equivalent single-site mutant. Agent rewrites of human-designed accelerators are checked against the original design and its own testbench. Agent-written design records are 30.9% of all records (Table 9). Appendix B.2 (Functional verification and testbench coverage) gives the full protocol.

W1.3 (simulation coverage and failure causes).

We've updated Table 2, now 580,331 designs ran RTL simulation and 480,367 (47.7% of all design records) passed their testbenches. Table 12 states total / succeeded / failed numbers explicitly for every stage. RTL simulation is 580,331 / 480,367 / 99,964, so the failure rate is 17.2% of the 580,331 simulations that ran. Table 13 decomposes all 99,964 failures. Causes are read from per-pool log samples and scaled by each pool's failure count; no pool's causes are imputed from another's. A single missing depth pragma on an m_axi interface accounts for 61,645 failures (61.7%). Only 8.8% implicates the generated RTL itself: a C testbench returning nonzero (6.8%) or a testbench/RTL compile error (2.0%).

Revised: Table 2; Appendix B.3 (Tables 12 and 13).

W1.4 (independent design diversity, and related variants in train/test splits and benchmark construction).

We report base designs and design variants separately (Table 2, Table 10): about 1 million design variants come from 126,255 base designs, and the base designs reflect the diversity of independent designs. The supervised experiments in Section 5.3 report two test settings. In the first, unseen base designs, no variant of a test base design appears in the training set. In the second, held-out variants, the test variant itself is not used in training, but other variants of the same base design may be in the training set. The experiments in Appendices D.6 and D.7 also split by base design. In the benchmark, T01–T04 sample hierarchically by source, then base design, then variant (Appendix D.1).

AuthorsOpenReview ↗

Response to Reviewer TtPF (2/2)

W2 (a single episode per problem cannot show the stability of the results).

Within our compute and API budget, we fixed 8 problems per task (2 from each of the four sources) for repeat runs: 8 API models repeat each of T01–T14 for 5 rounds, 4,480 episodes in total (Appendix D.5, Table 19). The sample standard deviation of the five round scores of each model–task pair is 0.0–17.7 points. Averaged over the 14 tasks, each model's standard deviation across the 5 independent runs is only 0.7–3.6 points (the Avg column of Table 19), so the variation of a single run has little effect on a model's overall score.

The differences between models are stable across rounds. On the T01–T14 average, Claude ranks first and Gemini 3.8 Flash second in all 5 rounds, and each round's model ranking has a Kendall τ of 0.86–0.93 with the ranking of the main experiment (Appendix D.5).

Revised: Appendix D.5 (Table 19).

Q1 (whether the RTL LOC and hierarchy-depth measurement includes tool-generated boilerplate).

Apart from removing blank lines and file-level headers, we count the generated RTL as it is, because this code is a necessary part of the complete design (see Appendix A.1).

Q2 (public release of all fields and artifacts).

All data and artifacts described in the paper will be publicly released upon acceptance to support open research. We have reviewed the licenses of the included external datasets and generators/compilers to ensure that the data may be redistributed and used for model training.

Q3.1 (token consumption and execution cost across models).

Appendix D.4 compares, for the 12 models over all 3,560 problems of T01–T14, the mean input and output tokens per episode, the interaction time, the estimated API cost, and the CPU-server cost estimated from the interaction time. For example, Claude costs about USD 1.80 per problem in API fees and Gemini 3.8 Flash about USD 0.19. Claude's average score is about 13 points higher, at about 10 times the cost per problem. The smaller local models use one to two orders of magnitude fewer tokens per episode than the API models. The CPU-server cost, estimated with a Google Cloud 4 vCPU / 32 GB server at USD 0.228 per hour, totals about USD 318 over the 12 models, far below the total API cost (about USD 9,676).

Revised: Appendix D.4 (Table 17).

Q3.2 (separating format, invalid-answer and budget failures from task-solving failures).

Appendix D.4 (Table 18) divides the final outcome of each episode into two classes:

  • Valid answer: the final submission passes the format and task-constraint checks and is scored. A score above 0 earns credit, and a score of 0 is a wrong answer; a wrong answer reflects the model's own failure to solve the task.
  • Execution failure: no submission or an invalid one (format error or task-constraint violation), exhaustion of the token, turn, tool-call or context budget, or a timeout. All of these receive 0 points.

Of the 42,720 main-experiment episodes, 18,601 earn credit and 19,790 are wrong answers, while 4,329 (10%) are execution failures: 3,846 invalid or missing answers, 326 out-of-budget episodes and 157 timeouts. Most failures therefore come from valid but wrong answers, not from format or budget problems.

Revised: Appendix D.4 (Table 18).

Q4 (scope: digital designs, and analog or mixed-signal design as a direction).

ChipFactory currently focuses on digital designs. We now state this focus in the conclusion and list extending ChipFactory to analog and mixed-signal designs as future work.

Revised: Section 6 (Conclusion).

Reviewer_zXAyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents ChipFactory, a dataset with 1 million design records intended to provide higher design complexity, design diversity and backend feedback richness. It combines designs from generators and compilers, external datasets, human designed accelerators, and software programs, and utilizes HLS tools with AI agents to create a large number of design variants. A benchmark suite with 13 tasks was built and the performance of 12 (frozen) commercial and open-source models was evaluated.

Critical strengths

Creating more hierarchical designs. While using agentic AI with HLS to scale a dataset is quite common in several of the works that have been cited, using agents to refactor software programs into new base designs would help create hierarchical designs. LLM-based tools don’t do well in hierarchical designs and so this augmented dataset could help study the gaps in LLM-based tools.

Availability of the open source dataset. (Managing such a large dataset (15TB of data) could be a nightmare. Also labels generated by tools change very quickly, making this an even bigger problem to manage.)

Critical weaknesses

To better appreciate the value of the large size of the dataset, models should be finetuned with ChipFactory dataset and other prior work dataset and shown that the one tuned with ChipFactory is better than other datasets.

It seems that the dataset is being used to auto-create benchmarks for the downstream tasks. But does the large scale really matter given that the 13 tasks use the dataset to auto generate 3140 problems from about 0.36% of records? For benchmarks, I think quality, correctness, and diversity matter more than scale. The paper is weak on all three: under 9% of records are simulated, most variants are sweeps over few bases, and about 87% of the RTL is tool-generated.t is not clear what kinds of benchmarks can ChipFactory create that the existing tools cannot.

Having a large dataset is useful for training purposes. However, in this paper, only one experiment trains a model (Fmax prediction), and there is no comparison to other datasets.

The LogikBench citation is missing.

Questions for authors

Which downstream task does ChipFactory enable that existing datasets (CircuitNet 2.0/3.0, EDALearn, ForgeHLS, HLSyn, wa-hls4ml, HLSDataset, LogikBench) cannot? Is there a way to demonstrate it?

How was functional correctness verified? It seems that the testbenches were also generated by agents. In that case, how can you verify correctness?

How was testbench coverage validated for agent-written tests? What fraction of records have a test?

What percentage of base designs (not variants) actually go through place and route?

4 comments
AuthorsOpenReview ↗

Response to Reviewer zXAy (1/4)

We thank the reviewer for the detailed review and constructive suggestions. In the revision, we added training comparisons with an existing dataset and across training-data scales, extended the C/RTL simulation coverage, added statistics on design authorship and base-design coverage, and added the comparison with LogikBench.

W1 (no training comparison with other datasets).

We take HLS prediction as an example (predicting the latency and resource usage of an HLS design) and compare training on the HLSyn dataset with training on ChipFactory. We use two graph neural networks designed for HLS, HARP and Hierarchical-MoE (H-MoE), to predict the latency and the LUT, FF, DSP and BRAM usage reported by Vitis HLS. For HLSyn, we use the HLSyn data points included in ChipFactory, so that the tool version and flow match ChipFactory and the comparison is fair. Each run changes only the training data and uses the same test set, and no base design (kernel) in the test set appears in any training set. We use two metrics, each averaged over the five prediction targets, where higher is better: (1) weighted Kendall τ, the Kendall τ of the model's ranking of the design variants within each base design, averaged with weights given by each base design's number of design variants; (2) 20% pass rate, the percentage of predictions within 20% relative error of the true value, the same scoring as the numerical prediction tasks in the paper.

Training data#Design variants#Base designsWeighted τ, HARPWeighted τ, H-MoE20% pass rate, HARP20% pass rate, H-MoE
HLSyn25K21−0.0190.0053.1%3.6%
ChipFactory (V80 @ 300 MHz only, as HLSyn)210K14.9K0.4650.50118.8%24.9%
ChipFactory591K25.0K0.6030.61235.1%35.1%

Models trained only on HLSyn can hardly rank the design variants within unseen base designs. All HLSyn designs in ChipFactory run HLS with the same device and clock (AMD Alveo V80 at 300 MHz). Training only on the V80 @ 300 MHz subset of the ChipFactory training set raises weighted τ from between −0.02 and 0.01 to 0.46–0.50, and the 20% pass rate from 3.1–3.6% to 18.8–24.9%. With the full ChipFactory training set, which covers six devices and eight clock targets, the two metrics rise further to 0.60–0.61 and 35.1%. This experiment shows that the large scale and diverse coverage of ChipFactory help trained models generalize.

Revised: Section 5.3 (the new experiment) and Appendix D.7 (full setup and per-target results).

W2 (whether the large scale matters).

The scale of ChipFactory serves both model training and custom benchmark construction. The 3,140 evaluation problems of the initial submission (3,560 after revision) are one of its uses. To measure the training benefit of scale, we reuse the models and test set of the HLS prediction experiment above and train on random 10%, 25% and 50% subsets of the design variants in the 591K ChipFactory training set:

Fraction of ChipFactory10%25%50%100%
#Design variants59K148K295K591K
Weighted τ, HARP0.5440.5800.5950.603
Weighted τ, H-MoE0.5690.6010.6060.612
20% pass rate, HARP28.3%33.4%31.8%35.1%
20% pass rate, H-MoE30.6%33.8%34.3%35.1%

Both metrics of both models generally improve as the training data grows. A random 10% subset of ChipFactory alone is already far better than training on HLSyn (weighted τ: 0.544 vs. −0.019 and 0.569 vs. 0.005; 20% pass rate: 28.3% vs. 3.1% and 30.6% vs. 3.6%). The large scale and diverse coverage of ChipFactory help here.

Revised: Section 5.3 and Appendix D.7.

W3 (quality, correctness and diversity of the dataset).

We agree that dataset quality is critical. We reply to each point below.

W3.1 (under 9% of records are simulated).

We substantially extended the simulation coverage (Table 12, Appendix B.3). Of all 1,006,735 design records, 580,331 (57.6%) ran RTL simulation, and 480,367 (47.7%) passed it; in the initial submission this share was under 9%. Table 13 breaks down the causes of the simulation failures.

Revised: Appendix B.3 (Tables 12 and 13).

AuthorsOpenReview ↗

Response to Reviewer zXAy (2/4)

W3.2 (most variants are sweeps over a few bases).

Building several design variants for the same functional requirement is itself a common and useful practice in EDA: before tape-out, designers usually iterate over several implementations, compare their performance, power and area, and then choose one that meets the constraints. Sweeping parameters from a small number of base designs is therefore a common way to build EDA datasets. For example, the 41,596 records of HLSyn come from 42 kernels (about 1:990).

For comparison, we report the numbers of base designs and design variants separately (Table 10). The 1,006,735 design records of ChipFactory come from 126,255 base designs, about 8 variants per base on average. For the designs we generate ourselves (excluding external datasets), the ratio is 1:35.

Appendix B.2 (Variant Generation) describes how variants are generated for each of the four source classes, with Table 8 listing the parameters swept in each class, and it separates parameter sweeps, code transformations and agent rewrites.

Revised: Appendix B.2 (Table 8) and Appendix B.3 (Table 10).

W3.3 (about 87% of the RTL is tool-generated).

Table 8 of the submitted version classified 87% of the RTL as "generated", but this category mixed RTL from HLS compilation, agent generation and other tools, and did not separate design authorship from the generation path. The new Table 9 splits the designs into seven classes, agent-written RTL/HLS/HCL, human-written RTL/HLS/HCL, and human-written generators, and reports the count of each: 311,242 records (30.9%) are agent-written, 353,565 (35.1%) are human-written, and 341,928 (34.0%) are produced by human-written generators. Part of the human-designed accelerator designs come from agent rewrites of submodules and count as agent-written; the rest are complete modules extracted directly from the human designs and count as human-written.

Writing RTL by hand takes a long time, and most high-quality industrial RTL designs are private assets, so the human-written RTL that can be collected is very limited. ChipFactory already collects the publicly available human designs broadly (96,526 human-written RTL records).

In addition, many ChipFactory designs come from human-written generators. These generators encode accumulated architectural knowledge and optimization methods as reusable generation rules, and they automatically produce hardware implementations for different configurations, which saves writing each RTL/HLS design by hand. Developing and using hardware generators is itself a common design method in hardware. The RTL these tools produce also carries human design knowledge and is often of high quality.

Revised: Appendix B.2 (Table 9).

W4 (benchmarks that ChipFactory can build and existing tools cannot).

Please see Q1, where we compare the tasks supported by ChipFactory and by existing datasets and give concrete examples.

W5 (only one training experiment, Fmax prediction, with no comparison to other datasets).

Please see W1. We added a new HLS latency and resource prediction experiment with two graph neural networks (HARP and H-MoE) and compared it with training on HLSyn.

W6 (missing LogikBench citation).

We added the LogikBench citation and included LogikBench in the comparisons of Table 1, Figure 2 and Figures 9–11. LogikBench is a benchmark of 250 RTL designs, while ChipFactory is a dataset with a different purpose.

Revised: Table 1, Figure 2, Figures 9–11.

AuthorsOpenReview ↗

Response to Reviewer zXAy (3/4)

Q1 (downstream tasks that ChipFactory enables and existing datasets do not).

We compared the feedback stages that these datasets publicly release and, using the task definitions of Table 3, list below the tasks each dataset can build directly. Only ChipFactory can build all 14 tasks directly. The tasks that depend on simulation results (T05 signal prediction, T07 co-simulation latency prediction, and the generation tasks T11–T13, which need testbench verification) cannot be built directly from any of the other datasets, because none of them releases RTL simulation results. For example, CircuitNet 2.0/3.0 have no functional simulation feedback and contain no HLS designs. ForgeHLS and HLSyn provide only HLS synthesis results, with no RTL synthesis or post-route results. EDALearn provides only the implementation flow code, so users must run the flow themselves to obtain labels. LogikBench provides RTL synthesis results, but it releases neither multiple implementation variants of the same design nor post-route results, so it cannot directly build the tasks that need variant ranking (such as T08) or post-route metrics (such as T01 and T02).

DatasetAvailable feedback stagesTasks in Table 3
CircuitNet 2.0RTL synthesis, Placement, RoutingT01, T02, T08
CircuitNet 3.0RTL synthesis, RoutingT01, T02, T08
EDALearn— (no backend labels)—
ForgeHLSHLS synthesisT06, T14
HLSynHLS synthesisT06
wa-hls4mlHLS synthesis, RTL synthesisT06, T08–T10, T14
HLSDatasetHLS synthesis, RTL synthesis, RoutingT01–T04, T06, T08–T10, T14
LogikBenchRTL synthesis—
ChipFactoryC simulation, RTL simulation, HLS synthesis, RTL synthesis, Optimization, Placement, RoutingT01–T14

In addition, Figure 11 (Appendix A) surveys the feedback of 70 prior works across 8 stages. ChipFactory is the only work with available labels at every stage. It stores the source code, the per-stage results and the simulation artifacts linked together, so tasks can be built without rerunning EDA tools.

Q2 (verifying functional correctness when agents also write the testbenches).

Human-written RTL and testbenches do not, by themselves, guarantee functional correctness. In practice, verification confidence is established through testing and coverage analysis, with human experts defining verification targets and deciding when sufficient confidence has been gained. We apply the same standard to agent-generated designs and testbenches through coverage-driven testbench construction, following and extending the approaches of Zou et al. (2026, arXiv:2606.30949) and Ding et al. (2026, arXiv:2609.21157).

We treat each software program as the golden functional reference for the HLS designs derived from it, and we generate expected outputs by running the program rather than asking an agent to write them. For programs refactored from existing code, the reference is the original function, and the tests must reject a deliberately incorrect refactor, as in Zou et al. (2026). For programs written from a specification, we start with two independent agent sessions that implement the same specification, and we keep the program as the reference only when the two implementations agree on every test. Following Ding et al. (2026), the design agent develops against a public subset of the tests, while hidden tests determine correctness. We ask agents to improve the testbench until it achieves 100% branch coverage of the reference C/C++ code, as measured by GNU gcov, and detects every non-equivalent single-site mutant of the reference.

We apply the same verification flow to human-designed accelerators, using the original human-written design as the reference and its testbench as the starting test set. Where additional tests are needed, agents add inputs, and the original design supplies the expected outputs.

Agent-written code must pass C simulation against these tests, and we use the compiled RTL testbench from the C testbench for RTL co-simulation. This process establishes verification confidence through explicit coverage and mutation-testing targets.

Every agent-written testbench in ChipFactory is checked with this protocol, and each design record reports the outcome of every check.

Revised: Appendix B.2 (Functional verification and testbench coverage).

AuthorsOpenReview ↗

Response to Reviewer zXAy (4/4)

Q3 (validating the coverage of agent-written tests, and the fraction of records with a test).

How coverage is validated. Agent-written testbenches must reach 100% branch coverage of the reference C/C++ code (measured with GNU gcov) and detect every non-equivalent single-site mutant. We measure coverage on the reference code rather than on the agent's HLS, because a behavior that the HLS drops does not show up in the HLS code's own coverage (see Q2).

Fraction of records with a test. A record counts as tested when its flow runs a C or RTL simulation. All 408,127 records of the software programs and human-designed accelerators, the two classes we build ourselves, have tests. Designs from generators, compilers and external datasets keep the tests supplied by their sources, and 229,447 of these 598,608 records (38.3%) have tests. Overall, 637,574 of 1,006,735 records (63.3%) have a test.

Q4 (percentage of base designs that go through place and route).

We added Table 11 to Appendix B.3, which reports backend coverage per base design. Of all 126,255 base designs, 94,875 (75.1%) have at least one variant that completed placement, and 88,138 (69.8%) have at least one variant that completed routing. The three source classes whose code we generate ourselves all exceed 95% (placement / routing): 99.6% / 98.2% for generators and compilers, 96.8% / 96.8% for human-designed accelerators, and 96.1% / 95.8% for software programs. The overall share is pulled down mainly by external datasets (70.8% / 64.4%). Their code is included as released, and for 22,501 of their base designs every variant fails RTL synthesis, so these bases never reach backend implementation.

Revised: Appendix B.3 (Table 11).

Reviewer_mQDZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Research on AI for chip design lacks large and diverse hardware datasets with informative implementation feedback. The paper presents ChipFactory, a large dataset with over 1 million design records from multiple sources, including variants generated using AI agents and high-level synthesis. In addition, the agent skills can be reused to build new benchmarks. The evaluation of 12 LLMs on 13 tasks demonstrates the capabilities and limitations of the evaluated models.

Critical strengths
  • The paper addresses the shortage of large hardware datasets. The data construction effort is substantial.
  • Reusable agent skills facilitate the construction of new benchmarks.
Critical weaknesses
  • The benchmark focuses on prediction, ranking, and reconstruction tasks rather than chip design and optimization. It is unclear whether a model performing well on these tasks can help improve chip designs.
  • The paper reports source-specific dataset statistics but does not analyze model performance by source. Data from different sources may have different characteristics and lead to different performance.
Questions for authors
  • How many problems from each source are included in each task?
  • Could the authors provide a more detailed comparison of data from different sources and report model performance separately for each source? In the supervised prediction experiments, how well do the methods generalize or transfer across sources?
2 comments
AuthorsOpenReview ↗

Response to Reviewer mQDZ (1/2)

We thank the reviewer for the careful review and constructive suggestions. In the revision, we added an HLS code optimization task (T14), an analysis of model performance and problem counts by data source, and a cross-source generalization experiment for supervised prediction.

W1 (link between the tasks and chip-design optimization).

The existing tasks mainly evaluate how well models understand, predict and compare hardware behavior and implementation outcomes. In design exploration, predicting implementation metrics quickly and screening candidates helps reduce expensive backend tool runs. Prediction is also a main evaluation target of existing ML-for-EDA datasets, for example timing prediction in CircuitNet, design-quality prediction in HLSyn, and resource and latency prediction in wa-hls4ml.

We agree with the reviewer that prediction and reconstruction scores do not directly show optimization ability. We therefore added T14, an HLS code optimization task. The model may change code structure and pragmas, and it is scored by the throughput that HLS synthesis estimates, provided the design passes C simulation and meets the resource constraints. Claude Fable 5.1 scores highest (13.67/100, with valid optimizations on 32/40 problems), followed by Gemini 3.8 Flash (6.895/100, 15/40 problems). The strongest model also leads on optimization, and it still leaves a large gap to the reference throughput.

Revised:

  • Section 4.2 and Table 3: the T14 task definition, inputs, outputs and tools.
  • Sections 5.1–5.2 and Figure 5: T14 in the 14-task comparison, and a new results paragraph, HLS code optimization (T14).
  • Appendix C.2: problem construction, verification and scoring; Appendix D.1: budget limits; Appendix D.2: valid optimizations per model.

W2 (no analysis of model performance by data source).

We added 380 problems so that each of T01–T13 has equal problem counts across the four data sources (Table 14): T01–T04 and T08–T10 are unchanged, T05–T07 grow from 300 to 400 problems each, T11 grows from 200 to 240, and T12 and T13 from 100 to 120 each. On this basis, we added an analysis of model performance by data source (Appendix D.3). Within a single task, difficulty differs clearly across sources, and the hardest source changes from task to task. Because each source is harder on some tasks and easier on others, the four source averages of each model over the 14 tasks are close, differing by at most about 7.3 points.

  • The hardest source depends on the task: generators and compilers on T01 (10 of 12 models), external datasets on T08 (8 of the 11 models with nonzero scores), software programs on T11 (7 of 8), and human-designed accelerators on T14 (5 of 7).
  • The same source can go from easiest to hardest: Gemini 3.1 Pro scores highest on external datasets in T01 (58.6) but lowest on them in T08 (27.8).

Different sources therefore pose different challenges on different tasks, and with source-balanced sampling the overall benchmark results do not lean noticeably toward any one source.

Revised: Appendix D.3 (Benchmark results by design source).

Q1 (number of problems from each source in each task).

The revision adds 380 problems so that every task has an equal quota from each of the four sources. The per-source counts are below (the same as Table 14 in Appendix D.1). Each row gives the count for a single task, and each ranking problem contains 10 design variants.

TaskGenerators / compilersExternal datasetsHuman-designed acceleratorsSoftware programsTotal
T01–T04100100100100400
T05–T07100100100100400
T08–T102020202080
T1160606060240
T12–T1330303030120
T141010101040

Revised: Appendix D.1 (Table 14).

Q2.1 (comparing data across sources and reporting model performance per source).

Table 10 (Appendix B.3) compares the four sources by number of designs, number of base designs, and RTL complexity (lines, files, modules, ports and hierarchy depth). Software programs (14,409) and external datasets (16,315) contribute the most complex bases. External datasets also contain many simple bases (88,404), none of whose designs is both hierarchical and at least 100 RTL lines long.

Model performance by source is given in our response to W2 (Appendix D.3).

Revised: Appendix B.3 (Table 10).

AuthorsOpenReview ↗

Response to Reviewer mQDZ (2/2)

Q2.2 (cross-source generalization in the supervised prediction experiments).

In Appendix D.6 (Table 21), we added a cross-source post-route Fmax⁡F_{\max} prediction experiment over the four design sources. It compares within-source (train on all sources, test on unseen base designs of the same source) with leave-one-source-out (train on the other three sources, test on the source left out). On external datasets, software programs, and generators and compilers, leaving out the whole source lowers the rank correlation (Spearman ρ) of all four models (the RTL-level RTL-Timer and MasterRTL, and the netlist-level XGBoost and DE-HNN), and the frequency error grows by about 1.4–2.8×. Mean absolute error (MAE, MHz), for example:

ModelExternal (ID → OOD)Software (ID → OOD)Generators (ID → OOD)
RTL-Timer61.1 → 108.736.0 → 92.141.9 → 105.6
DE-HNN45.5 → 69.720.8 → 39.030.5 → 80.8

The ρ of the RTL-level models drops by 0.13–0.23, while that of the netlist-level models drops by only 0.04–0.10, so the netlist-level models transfer more robustly. Generalizing to a design source unseen in training is therefore harder than generalizing to unseen designs from the same source. This also shows the value of ChipFactory's source diversity: when a source is missing from the training data, the predictor degrades markedly on that source.

Revised: Appendix D.6.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Changes

We thank all reviewers for their time and constructive comments. As the legend at the top right of page 1 of the revised paper shows, we highlight our changes in color: blue (multiple reviewers), red (Reviewer mQDZ), green (Reviewer zXAy), orange (Reviewer TtPF), and grey (proofreading). The legend and the margin tags in the paper label the reviewers A, B and C; these are Reviewers mQDZ, zXAy and TtPF, respectively.

Summary of major changes:

For Dataset

(1) Added Appendix B.2 (Variant Generation), which describes the parameter sweep space (Table 8) and the functional-correctness verification flow. (Reviewers mQDZ, zXAy, TtPF)

(2) Added the comparison with LogikBench. (Reviewer zXAy)

(3) Added statistics on design authorship and base-design coverage (Appendix B.2, Table 9; Appendix B.3, Tables 10–11). (Reviewers zXAy, TtPF)

(4) Extended the C/RTL simulation coverage (Table 2; Appendix B.3, Table 12). (Reviewers zXAy, TtPF)

(5) Added a breakdown of RTL simulation failure causes (Appendix B.3, Table 13). (Reviewer TtPF)

For Benchmark

(6) Added T14, an HLS code optimization task (Sections 4–5, Table 3, Figure 5, Appendices C.2 and D.2). (Reviewer mQDZ)

(7) Added 380 problems so that every task has equal problem counts across the four data sources (Appendix D.1, Table 14), and added an analysis of model performance by data source (Section 5.2, Appendix D.3). (Reviewers mQDZ, TtPF)

(8) Added a repeat-stability experiment (Appendix D.5, Table 19). (Reviewer TtPF)

(9) Added an analysis of model cost and failure types (Appendix D.4, Tables 17–18). (Reviewer TtPF)

For Training

(10) Added a cross-source generalization experiment for supervised prediction (Appendix D.6, Table 21). (Reviewers mQDZ, TtPF)

(11) Added training comparisons with an existing dataset and across training-data scales (Section 5.3, Table 6; Appendix D.7, Table 22). (Reviewer zXAy)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.