- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents a dataset of over 1 million design records to address the lack of complex, diverse, and complete data in digital chip design, specifically connecting diverse hardware design representations and implementation choices with observable EDA backend feedback. To complement this, the authors introduce an agent-driven data generation pipeline and a benchmark suite evaluating different LLMs across prediction, ranking, and generation tasks. By highlighting the performance gaps still exhibited by current language models, the work demonstrates how this dataset can serve as a foundation for training specialized models and advancing AI-driven EDA.
Critical strengths
-
ChipFactory addresses an important gap in hardware-design datasets by combining large sample quantity with detailed design artifacts and backend implementation feedback. Its more than one million design records connect diverse hardware representations to information spanning RTL simulation, synthesis, placement, and routing. This linkage is particularly valuable because it enables research not only on code generation, but also on predicting implementation outcomes and studying trade-offs between design choices and hardware metrics. The dataset therefore provides a potentially useful foundation for developing and evaluating LLM-based hardware design tools.
-
The paper introduces 13 tasks across implementation prediction, behavioral prediction, ranking, and generation, and evaluates 12 models spanning proprietary, open-source, and specialized systems. This provides a structured baseline across several distinct capabilities rather than focusing exclusively on code generation or a single prediction task. The benchmark's breadth can help researchers identify where current models perform well or struggle, compare approaches under a common evaluation framework, and guide future work on more capable hardware-design agents.
Critical weaknesses
-
The dataset's scale and complexity come largely from agent-driven and from automated pragma and configuration sweeps. The paper does not establish that this pipeline results in functionally valid and non-redundant variants. There is no description of how behavioral preservation is checked between the original program and the agent's HLS product. If no tests exist, the testbench is also agent-written, so the design and the check may share the same interpretation of the intended behavior. Only 86,337 of the 1,006,725 records have retained RTL-simulation waveforms, and Table 8 reports 26,664 simulation failures with no breakdown of cause or stated denominator. Backend measurements remain valid for the implementation that was synthesized, but functionally incorrect variants undermine conclusions about trade-offs between supposedly equivalent designs. This is directly relevant to the ranking tasks (T08-T10), which compare variants of one base design. Additionally, the parameter spaces, sweep bounds, and filtering criteria are not described, and the paper does not quantify near-duplicates. The record count therefore does not show how much independent architectural diversity the dataset contains, and it is unclear how related variants are handled in train/test splits and benchmark construction.
-
Each benchmark problem is evaluated in a single agent episode, which limits the ability to assess the variability and reproducibility of the reported results. Agent-based tasks may be sensitive to stochastic generation and differences in the trajectories explored during problem solving. Without repeated independent runs, variance estimates, or statistical significance analysis, it is difficult to determine whether observed performance differences between models reflect consistent advantages or run-specific outcomes. This limits the strength of the conclusions drawn from model comparisons. Reporting results across multiple independent episodes, together with appropriate measures of variability and statistical comparisons, would provide stronger evidence for the robustness of the benchmark findings.
Questions for authors
- Does the RTL LOC and hierarchy-depth measurement (App. A.1) account for verbose or boilerplate code emitted by HLS tools and generators (wrappers, FSMs, interface logic, library modules), or is it counted as-is?
- In the comparison with prior datasets, will all fields and artifacts listed as planned for ChipFactory be publicly released upon acceptance, or will some remain unavailable or restricted?
- Have the authors compared token consumption and execution costs across models? Can failures caused by output-format violations, invalid answers, or exhaustion of token, time, or turn budgets be distinguished from failures reflecting a model's underlying task-solving ability?
- Although the paper discusses chip design broadly, the dataset and benchmark appear to focus on digital circuits. Could the authors clarify the intended scope of ChipFactory and whether extending it to analog or mixed-signal design is a planned direction?
2 comments
Response to Reviewer TtPF (2/2)
W2 (a single episode per problem cannot show the stability of the results).
Within our compute and API budget, we fixed 8 problems per task (2 from each of the four sources) for repeat runs: 8 API models repeat each of T01–T14 for 5 rounds, 4,480 episodes in total (Appendix D.5, Table 19). The sample standard deviation of the five round scores of each model–task pair is 0.0–17.7 points. Averaged over the 14 tasks, each model's standard deviation across the 5 independent runs is only 0.7–3.6 points (the Avg column of Table 19), so the variation of a single run has little effect on a model's overall score.
The differences between models are stable across rounds. On the T01–T14 average, Claude ranks first and Gemini 3.8 Flash second in all 5 rounds, and each round's model ranking has a Kendall τ of 0.86–0.93 with the ranking of the main experiment (Appendix D.5).
Revised: Appendix D.5 (Table 19).
Q1 (whether the RTL LOC and hierarchy-depth measurement includes tool-generated boilerplate).
Apart from removing blank lines and file-level headers, we count the generated RTL as it is, because this code is a necessary part of the complete design (see Appendix A.1).
Q2 (public release of all fields and artifacts).
All data and artifacts described in the paper will be publicly released upon acceptance to support open research. We have reviewed the licenses of the included external datasets and generators/compilers to ensure that the data may be redistributed and used for model training.
Q3.1 (token consumption and execution cost across models).
Appendix D.4 compares, for the 12 models over all 3,560 problems of T01–T14, the mean input and output tokens per episode, the interaction time, the estimated API cost, and the CPU-server cost estimated from the interaction time. For example, Claude costs about USD 1.80 per problem in API fees and Gemini 3.8 Flash about USD 0.19. Claude's average score is about 13 points higher, at about 10 times the cost per problem. The smaller local models use one to two orders of magnitude fewer tokens per episode than the API models. The CPU-server cost, estimated with a Google Cloud 4 vCPU / 32 GB server at USD 0.228 per hour, totals about USD 318 over the 12 models, far below the total API cost (about USD 9,676).
Revised: Appendix D.4 (Table 17).
Q3.2 (separating format, invalid-answer and budget failures from task-solving failures).
Appendix D.4 (Table 18) divides the final outcome of each episode into two classes:
- Valid answer: the final submission passes the format and task-constraint checks and is scored. A score above 0 earns credit, and a score of 0 is a wrong answer; a wrong answer reflects the model's own failure to solve the task.
- Execution failure: no submission or an invalid one (format error or task-constraint violation), exhaustion of the token, turn, tool-call or context budget, or a timeout. All of these receive 0 points.
Of the 42,720 main-experiment episodes, 18,601 earn credit and 19,790 are wrong answers, while 4,329 (10%) are execution failures: 3,846 invalid or missing answers, 326 out-of-budget episodes and 157 timeouts. Most failures therefore come from valid but wrong answers, not from format or budget problems.
Revised: Appendix D.4 (Table 18).
Q4 (scope: digital designs, and analog or mixed-signal design as a direction).
ChipFactory currently focuses on digital designs. We now state this focus in the conclusion and list extending ChipFactory to analog and mixed-signal designs as future work.
Revised: Section 6 (Conclusion).
Response to Reviewer TtPF (1/2)
We thank the reviewer for the careful review and constructive suggestions. In the revision, we described variant generation and functional verification in a new Appendix B.2, extended the simulation coverage with a breakdown of RTL simulation failures, and added a repeat-stability experiment and an analysis of cost and failure types.
W1.1 (parameter space, functional validity and non-redundancy of the variant-generation pipeline).
Parameter space. Appendix B.2 describes, for each source class, how variants are generated and which admission rules apply. Table 8 lists the parameters swept in each class, and Table 9 shows that agent-written design records are only 30.9% of the total.
Functional validity. Whether a variant keeps its function depends on how it was generated. Pragma sweeps change only how the HLS tool implements the program, not what the program computes. Compiler transformations preserve semantics by construction. Configuration sweeps over workload, data types and similar parameters change the function on purpose, and each configuration counts as a distinct design. Agent rewrites have no such guarantee, so they must pass testbench verification (see W1.2).
Non-redundancy. Of the variants whose normalized source or RTL is identical, we keep only one, and we merge pragma-only variants with identical synthesis resources into one. Variants with the same function but different implementations stay, because their QoR differs, and that difference is what the dataset is meant to provide.
Revised: Appendix B.2.
W1.2 (checking behavior preservation from the original program to the agent's HLS).
In ChipFactory, the software program is the golden functional reference. Expected outputs come from running the program, so agents write only test inputs, and every HLS design derived from the program must reproduce its outputs on hidden tests. The testbench must reach 100% branch coverage of the reference C/C++ code and detect every non-equivalent single-site mutant. Agent rewrites of human-designed accelerators are checked against the original design and its own testbench. Agent-written design records are 30.9% of all records (Table 9). Appendix B.2 (Functional verification and testbench coverage) gives the full protocol.
W1.3 (simulation coverage and failure causes).
We've updated Table 2, now 580,331 designs ran RTL simulation and 480,367 (47.7% of all design records) passed their testbenches. Table 12 states total / succeeded / failed numbers explicitly for every stage. RTL simulation is 580,331 / 480,367 / 99,964, so the failure rate is 17.2% of the 580,331 simulations that ran. Table 13 decomposes all 99,964 failures. Causes are read from per-pool log samples and scaled by each pool's failure count; no pool's causes are imputed from another's. A single missing depth pragma on an m_axi interface accounts for 61,645 failures (61.7%). Only 8.8% implicates the generated RTL itself: a C testbench returning nonzero (6.8%) or a testbench/RTL compile error (2.0%).
Revised: Table 2; Appendix B.3 (Tables 12 and 13).
W1.4 (independent design diversity, and related variants in train/test splits and benchmark construction).
We report base designs and design variants separately (Table 2, Table 10): about 1 million design variants come from 126,255 base designs, and the base designs reflect the diversity of independent designs. The supervised experiments in Section 5.3 report two test settings. In the first, unseen base designs, no variant of a test base design appears in the training set. In the second, held-out variants, the test variant itself is not used in training, but other variants of the same base design may be in the training set. The experiments in Appendices D.6 and D.7 also split by base design. In the benchmark, T01–T04 sample hierarchically by source, then base design, then variant (Appendix D.1).