- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a new benchmark, Silica-Bench, to evaluate LLM agents’ ability in long-horizon circuit design. This task is very suitable to evaluate a long-horizon agent’s ability of considering multi-objective trade-off, dividing a finite budget between exploration and validation, preserving requirements, etc. This is because this circuit design task is an open-ended task; it has continuously graded objectives; it has hard constraints on validity and functional requirements; and it has multiple time-consuming evaluation stages, where earlier stages are faster but less accurate. In this benchmark, agents receive a reference RTL design and a reference RTL-to-GDS flow. Then the agent needs to improve the RTL design and/or the flow configurations within a budget, and the agent needs to choose designs to nominate as final submissions. The benchmark contains 36 circuit designs from 8 domains. The experiments cover 6 model-harness systems. The experiments demonstrate that the LLM agents generally achieve valid designs, but making improvements is very challenging.
Critical strengths
1, The circuit design task is very suitable for evaluating LLM agents’ ability, and related long-horizon benchmarks are rare. So constructing such a benchmark is a meaningful contribution that might benefit both the ML community and the circuit design community. The tasks are realistic and very challenging.
2, I think the design of benchmark pipeline, evaluation protocol, and evaluation metrics are effective. The benchmark can evaluate LLM agents’ long-horizon ability in circuit design.
Critical weaknesses
1, The experiments exclude all four designs whose reference flows are longer than 3 hours. Including these designs can make the benchmark more long-horizon.
2, Paper writing needs some improvement. Some expressions are not very clear, and it takes some effort to understand some detailed information.
Questions for authors
1, Do agents have access to simulation tools or a public testbench that can help them check functional correctness?
2, There is no code or dataset. Considering this paper’s main contribution is a new benchmark, it is very important to release the benchmark dataset and related code. Will you publicly share the benchmark dataset and code?
4 comments
Response to Reviewer qBta (2)
4. Dataset and code release
Yes. The source and testing materials for all 36 designs are now available in the anonymous repository(https://anonymous.4open.science/r/ICLR2027-SILICA-127C/). The following links open concrete files:
| Material | What reviewers can inspect |
|---|---|
| Design index | All 36 design folders, DUT tops, package scope, and integration notes |
| AXI Virtual FIFO, PtahConv, BMB interconnect | Task descriptions, testing entry points, and links to reference RTL and test sources |
| AXI configuration, PtahConv configuration, BMB configuration | Functional parameters and dependencies for the named instances |
| RS encoder checker, rasterizer checker | Concrete observation and error-detection logic |
| Flow documentation | Design Compiler-to-Innovus handoff, external dependencies, and scope of the flow references |
| Settings, common helpers, synthesis, placement, CTS/routing | Five Tcl flow-reference files covering synthesis, placement, clocking, routing, extraction, and reporting |
The package retains third-party licence notices and documents portability changes, native test dependencies, and omitted inputs, including the r8051 firmware image. The flow references require licensed tools, technology libraries, frozen task constraints, and physical setup supplied separately.
This release makes the task definitions and checking logic inspectable. Replaying the full historical campaign additionally requires the agent harness, nomination and resource controllers, protected scoring service, and historical result/analysis artifacts; those are not bundled in this source package. The published documentation states these boundaries explicitly. Commercial binaries are not part of the release. You can download the ASAP 7 PDK from https://github.com/The-OpenROAD-Project/asap7 to access the relevant process library.
Thank you for recognizing the value of continuously graded engineering tasks with hard behavioral requirements; we hope the inclusion of all long tasks, clearer presentation, and released design and testing sources reinforce your support for SILICA-Bench as a shared research resource for the AI and EDA communities, and we welcome further discussion.
Thank you for your careful revisions and additional efforts! The inclusion of the long-horizon designs, clearer paper writing, and the release of the datasets are very helpful. I believe this benchmark could benefit the community. I will maintain my positive score of 3.
Thank you for recognizing the value of the expanded long-task evaluation and released materials; your support encourages us to keep improving SILICA-Bench as a community resource, and we would warmly welcome further discussion and feedback.
Response to Reviewer qBta (1)
Thank you for recognizing the value of a continuously graded engineering task with hard behavioral requirements. The revision includes all long-reference designs, makes the task and scoring sequence explicit, and clarifies search-time testing. A source package covering all 36 designs is now available in the anonymous repository(https://anonymous.4open.science/r/ICLR2027-SILICA-127C/).
1. Inclusion of the four long-reference designs
All four designs whose reference flows exceed three hours are included in the 36-design primary analysis. Their reference times and configured search allowances are:
Each system has 36 scheduled runs on these tasks: four designs, three Joint conditions, and three independent submissions. Their failure-inclusive EVQs range from 1.0835 to 1.3602 (Section 6.5; Table F.11). All six designs omitted from the original terminal comparison are now retained. One unscored cutoff outcome contributes zero analysis utility; no design is removed because of it (Appendix D.4). Additional summaries of the observed resource records show system-level median search times of 6.52–31.68 hours on these tasks. Thus, the long-task evidence includes actual sustained interaction as well as configured allowances; the saved-nomination analysis separately measures whether progress survives final delivery.
2. Clearer organization and terminology
The revised presentation follows the task's decision sequence:
This places the definitions before their results and makes each metric's interpretation explicit. For LLM evaluation, acceptance, delivered improvement, and retention of saved progress answer different questions about an agent's ability to complete an engineering workflow.
3. Simulation tools and public correctness testing
Yes, agents can run simulation during search using both Icarus Verilog and Synopsys VCS. They can construct their own tests, including differential checks against the visible reference RTL. Public interfaces include
run_check,register_candidate, andget_status, alongside public tool reports. Section 5.2 and Appendix A.3 now explicitly document both simulators as search-time capabilities. VCS is also used for the protected RTL and corresponding post-route netlist replay described in Appendix B.Access to a simulator is separate from access to the official assessment workload. Protected workloads, golden observations, and hidden assessment results were unavailable during the reported searches. An official public testbench is not guaranteed for every task.
The released source package(https://anonymous.4open.science/r/ICLR2027-SILICA-127C/rebuttal/README.md) now provides task-specific test sources, configurations, dependencies, and file lists for inspection. Its documentation distinguishes these published assessment materials from the original agent-visible search inputs. Making their checking logic public now does not change what agents could access during the reported experiments.