acceptodds
Under review as a conference paper at ICLR 2027

SILICA-BENCH: EVALUATING LONG-HORIZON LLM AGENTS THROUGH VERIFIED RTL-TO-GDS IMPROVEMENT

Abstract

Long-horizon LLM agents must turn a sequence of decisions into a reliable result: interpret imperfect feedback, preserve requirements, and divide a finite budget between exploration and validation. We introduce , which evaluates these abilities through physical improvement of digital circuits with a commercial RTL-to-GDS flow. Four properties make this a useful agent task: an open-ended, continuously graded power, performance, and area (PPA) objective with a fixed behavioral target; hard functional requirements; costly staged evidence over coupled RTL and flow choices; and 36 designs across eight domains and multiple scales. Agents receive reference RTL and a reference flow, work within a declared budget, and register what they deliver. Protected assessment binds RTL and post-route netlist replay to the nominated implementation, using local reconstruction or identity-matched reuse of its physical outputs. The protocol separates validity from improvement, early evidence from final quality, and discovery from delivery. A paired study of six model–harness systems and three edit authorities covers 240 nominations on 30 designs. Every configuration achieves at least 93% validity, yet no Joint system delivers a changed, verified gain above 5% on more than one third of tasks, and two Joint systems fall below unit failure-inclusive quality. Eleven of twelve valid regressions larger than 5% use the local reconstruction path. These results distinguish successful execution from reliable improvement, and make delivery of a verifiable engineering artifact an observable target for long-horizon agent evaluation. We will provide the complete dataset and test environment via an anonymous GitHub link during the rebuttal period.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, long-horizon decision-making, agent evaluation, tool use, RTL optimization, EDA-flow co-optimization, electronic design automation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_qBtaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new benchmark, Silica-Bench, to evaluate LLM agents’ ability in long-horizon circuit design. This task is very suitable to evaluate a long-horizon agent’s ability of considering multi-objective trade-off, dividing a finite budget between exploration and validation, preserving requirements, etc. This is because this circuit design task is an open-ended task; it has continuously graded objectives; it has hard constraints on validity and functional requirements; and it has multiple time-consuming evaluation stages, where earlier stages are faster but less accurate. In this benchmark, agents receive a reference RTL design and a reference RTL-to-GDS flow. Then the agent needs to improve the RTL design and/or the flow configurations within a budget, and the agent needs to choose designs to nominate as final submissions. The benchmark contains 36 circuit designs from 8 domains. The experiments cover 6 model-harness systems. The experiments demonstrate that the LLM agents generally achieve valid designs, but making improvements is very challenging.

Critical strengths

1, The circuit design task is very suitable for evaluating LLM agents’ ability, and related long-horizon benchmarks are rare. So constructing such a benchmark is a meaningful contribution that might benefit both the ML community and the circuit design community. The tasks are realistic and very challenging.

2, I think the design of benchmark pipeline, evaluation protocol, and evaluation metrics are effective. The benchmark can evaluate LLM agents’ long-horizon ability in circuit design.

Critical weaknesses

1, The experiments exclude all four designs whose reference flows are longer than 3 hours. Including these designs can make the benchmark more long-horizon.

2, Paper writing needs some improvement. Some expressions are not very clear, and it takes some effort to understand some detailed information.

Questions for authors

1, Do agents have access to simulation tools or a public testbench that can help them check functional correctness?

2, There is no code or dataset. Considering this paper’s main contribution is a new benchmark, it is very important to release the benchmark dataset and related code. Will you publicly share the benchmark dataset and code?

4 comments
AuthorsOpenReview ↗

Response to Reviewer qBta (1)

Thank you for recognizing the value of a continuously graded engineering task with hard behavioral requirements. The revision includes all long-reference designs, makes the task and scoring sequence explicit, and clarifies search-time testing. A source package covering all 36 designs is now available in the anonymous repository(https://anonymous.4open.science/r/ICLR2027-SILICA-127C/).

1. Inclusion of the four long-reference designs

All four designs whose reference flows exceed three hours are included in the 36-design primary analysis. Their reference times and configured search allowances are:

DesignReference flow (h)Search wall allowance (h)
RS decoder4.333834.6702
AXI Virtual FIFO7.277658.2208
PtahConv3.980631.8450
BMB interconnect5.473443.7875

Each system has 36 scheduled runs on these tasks: four designs, three Joint conditions, and three independent submissions. Their failure-inclusive EVQs range from 1.0835 to 1.3602 (Section 6.5; Table F.11). All six designs omitted from the original terminal comparison are now retained. One unscored cutoff outcome contributes zero analysis utility; no design is removed because of it (Appendix D.4). Additional summaries of the observed resource records show system-level median search times of 6.52–31.68 hours on these tasks. Thus, the long-task evidence includes actual sustained interaction as well as configured allowances; the saved-nomination analysis separately measures whether progress survives final delivery.

2. Clearer organization and terminology

The revised presentation follows the task's decision sequence:

  1. Sections 3–4: the supplied reference, behavioral target, resource contract, and permitted edits. RTL-only fixes the flow; Flow-only fixes RTL; Joint allows both. The flow conditions distinguish parameter, stage-command, and recipe interfaces.
  2. Section 5: the frozen nomination, protected assessment, accepted-only quality, and failure-inclusive utility. It explains checkpoint selection and restricted finalization, and distinguishes profile acceptance from physical signoff.
  3. Section 6 and Tables 1–3: delivered quality, permission comparisons, and measured nomination trajectories. Figures 1–3 illustrate the protocol; Figures 4–5 report empirical results. Appendix A consolidates notation and task contracts.

This places the definitions before their results and makes each metric's interpretation explicit. For LLM evaluation, acceptance, delivered improvement, and retention of saved progress answer different questions about an agent's ability to complete an engineering workflow.

3. Simulation tools and public correctness testing

Yes, agents can run simulation during search using both Icarus Verilog and Synopsys VCS. They can construct their own tests, including differential checks against the visible reference RTL. Public interfaces include run_check, register_candidate, and get_status, alongside public tool reports. Section 5.2 and Appendix A.3 now explicitly document both simulators as search-time capabilities. VCS is also used for the protected RTL and corresponding post-route netlist replay described in Appendix B.

Access to a simulator is separate from access to the official assessment workload. Protected workloads, golden observations, and hidden assessment results were unavailable during the reported searches. An official public testbench is not guaranteed for every task.

The released source package(https://anonymous.4open.science/r/ICLR2027-SILICA-127C/rebuttal/README.md) now provides task-specific test sources, configurations, dependencies, and file lists for inspection. Its documentation distinguishes these published assessment materials from the original agent-visible search inputs. Making their checking logic public now does not change what agents could access during the reported experiments.

AuthorsOpenReview ↗

Response to Reviewer qBta (2)

4. Dataset and code release

Yes. The source and testing materials for all 36 designs are now available in the anonymous repository(https://anonymous.4open.science/r/ICLR2027-SILICA-127C/). The following links open concrete files:

MaterialWhat reviewers can inspect
Design indexAll 36 design folders, DUT tops, package scope, and integration notes
AXI Virtual FIFO, PtahConv, BMB interconnectTask descriptions, testing entry points, and links to reference RTL and test sources
AXI configuration, PtahConv configuration, BMB configurationFunctional parameters and dependencies for the named instances
RS encoder checker, rasterizer checkerConcrete observation and error-detection logic
Flow documentationDesign Compiler-to-Innovus handoff, external dependencies, and scope of the flow references
Settings, common helpers, synthesis, placement, CTS/routingFive Tcl flow-reference files covering synthesis, placement, clocking, routing, extraction, and reporting

The package retains third-party licence notices and documents portability changes, native test dependencies, and omitted inputs, including the r8051 firmware image. The flow references require licensed tools, technology libraries, frozen task constraints, and physical setup supplied separately.

This release makes the task definitions and checking logic inspectable. Replaying the full historical campaign additionally requires the agent harness, nomination and resource controllers, protected scoring service, and historical result/analysis artifacts; those are not bundled in this source package. The published documentation states these boundaries explicitly. Commercial binaries are not part of the release. You can download the ASAP 7 PDK from https://github.com/The-OpenROAD-Project/asap7 to access the relevant process library.

Thank you for recognizing the value of continuously graded engineering tasks with hard behavioral requirements; we hope the inclusion of all long tasks, clearer presentation, and released design and testing sources reinforce your support for SILICA-Bench as a shared research resource for the AI and EDA communities, and we welcome further discussion.

Thank you for your careful revisions and additional efforts! The inclusion of the long-horizon designs, clearer paper writing, and the release of the datasets are very helpful. I believe this benchmark could benefit the community. I will maintain my positive score of 3.

Thank you for recognizing the value of the expanded long-task evaluation and released materials; your support encourages us to keep improving SILICA-Bench as a community resource, and we would warmly welcome further discussion and feedback.

Reviewer_vrReOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tests whether agents can improve circuit PPA within a fixed time budget. Agents edit RTL and tool settings. The submitted designs are checked and measured after routing. Most submissions pass the checks, but few achieve clear gains.

Critical strengths

The benchmark covers designs from several domains and measures results with commercial tools after routing. The experiments show a useful gap. Agents often produce valid outputs, but improving the reference design remains difficult.

Critical weaknesses
  1. The paper mainly reports the final PPA, so we learn little about what happens during optimization. Do agents improve steadily, or struggle until a late gain? Please show how PPA changes across edits on a few representative tasks, and explain which changes help or hurt.
  2. The longest tasks are especially relevant to the paper's long-horizon focus, but their results are incomplete. It would help to discuss what the available runs show. How did agents use the extra time, where did they struggle, and what prevented completion? This could add useful insights (better if the authors can complete these designs).
  3. The benchmark needs more checks on score reliability. Each runs only once per design, so we do not know how much scores vary across runs. Please repeat a few tasks to measure this variation. In addition, it would also help to test some high-scoring traces and outputs. This can show whether incorrect edits still receive high scores (benchmark hacking).
Questions for authors

Please see the weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer vrRe (1)

Thank you for asking what happens during optimization and whether the scores are reliable. The revision adds measured progress, complete long-task coverage, three independent submissions for every primary design–condition combination, and explicit acceptance diagnostics. Below we answer the process and reliability questions with these results and an additional long-task resource breakdown.

1. When do agents improve, and what helps or hurts?

Saved progress is now measured in Table 3 and Figure 5(b). Claude's design-balanced mean utility evolves through 1.0966, 1.2093, 1.2346, and 1.2536 at 25%, 50%, 75%, and final delivery. Qwen Max rises from 1.0367 at 75% to 1.0770 at delivery. A further count from the trajectory records shows that Qwen Max's final utility exceeds its 75% nomination in 179/324 runs (55.25%); this count concerns any recorded increase.

Progress is not always retained. Across 1,943 observed trajectories, 124 finish below their best of four saved nominations, including 55 with an absolute utility loss of at least 0.05 (Appendix G.3). This distinguishes continued improvement from the ability to deliver an already saved result.

The permission experiments also identify useful differences in the available action interface. With RTL fixed, moving between the F1 parameter interface and the F2 stage-command interface changes EVQ by +0.2326 for Claude, +0.1458 for GPT, and +0.0873 for Qwen Max; these three contrasts survive Holm correction over the 32 permission comparisons (Table 2; Appendix F.2). These are comparisons of edit interfaces, not a strictly nested expansion of the same action set.

Together, the analyses show when saved utility changes and which tested interfaces yield better aggregate outcomes. Looking at the vast majority of trajectories, we find that, in many cases, an increase in a design’s overall score is due to the model proactively coordinating synchronized, collaborative modifications to both the RTL and flow scripts—a process built upon repeated data collection and lessons learned. This also explains why allowing modifications to only one aspect yields less benefit than allowing simultaneous modifications to both. It further demonstrates that our test scenario is a real-world scenario that requires complex reasoning and critical thinking.

2. Long-task results and actual use of the available time

All four designs with reference flows longer than three hours (test duration more than 8 × 3 = 24 hours) are included, with three Joint conditions and three submissions per system. Their configured search allowances are 31.85–58.22 hours. The following table combines the outcome statistics in Table F.11 with an additional breakdown of the same campaign's resource records:

SystemAccepted / scheduledLong-task EVQMedian search time (h)Mean EDA requests
Claude32/361.360231.6851.14
GPT35/361.184225.1729.28
Kimi36/361.267613.5111.28
GLM34/361.08356.525.91
Qwen Max35/361.207112.567.00
Grok31/361.088614.6025.36

Resource summaries use 36 observed runs per system except GLM, which has 35; its unscored cutoff outcome remains in the 36-outcome EVQ denominator, while unavailable resource values are not imputed. EDA requests include public-tool requests and are not counts of completed full flows or reasoning steps.

These measurements establish actual multi-hour search and repeated tool interaction on the long tasks. They also show distinct operating patterns: Kimi reaches EVQ 1.2676 with a 13.51-hour median search, while GPT reaches 1.1842 with 25.17 hours. This is a descriptive comparison of deployed systems, not a causal claim that shorter search is better. Together with the saved-nomination results, these observations connect sustained tool interaction to a concrete evaluation question: whether an agent preserves progress and makes a reliable final choice under the resource contract. Duration alone is not interpreted as reasoning depth.

3. What limited or prevented completion?

We separate search stopping conditions from assessment failures. In the full primary campaign, GLM records 141 response-limit stops, Qwen Max 185 wall-limit stops, and GPT 215 normal completions (Tables F.14–F.16). Appendix H.1 separately categorizes functional failures, missing implementation evidence, and unavailable checks.

The revised long-task analysis retains all 216 scheduled runs, with 203 accepted outcomes. Thus, difficult cases remain visible in the result. The aggregate records do not identify every long-run bottleneck. Detailed event-level attribution would require the corresponding execution traces. Subsequently, we will publicly release all data and actual trajectories from each run of every design as open source to facilitate further research and analysis.

AuthorsOpenReview ↗

Response to Reviewer vrRe (2)

4. Run-to-run variability and ranking stability

Every primary design–condition combination now has three independent submissions. Each repetition aggregates 108 scheduled runs per system; the final column reports the median sample SD within those 108 combinations (Table F.10).

Systemr1 EVQr2 EVQr3 EVQMedian within-combination SD
Claude1.26181.26641.23400.0800
GPT1.12131.14341.11790.0285
Kimi1.15821.13861.08230.0740
GLM1.11351.09391.08400.0734
Qwen Max1.10401.09761.03220.0587
Grok1.08291.01901.03400.0935

Claude leads in all three repetitions and all leave-one-design-out comparisons; the GPT/Kimi order varies. Design-cluster intervals keep each design's conditions and repetitions together, and paired tests include multiplicity correction (Appendix G.1). This addresses variation in end-to-end agent outcomes. In commercial tools, the measurement errors caused by repeatedly reconstructing unchanged workpieces are negligible, and performance remains relatively stable even during repeated operations.

5. Functional integrity and high-scoring outputs

The functional gate has an observable effect on scoring. In the full campaign of 3,924 scheduled runs, 58 records have functional-fail but implementation-pass status and receive zero utility (Appendix H.1). Complete physical evidence therefore does not automatically earn a positive score. This count is over the broader campaign, which includes the primary 1,944-outcome comparison.

The assessment contract requires both the nominated RTL and corresponding post-route netlist to match protected reference observations. Interfaces, reset, legal inputs, output observations, latency, and measurement constraints are fixed (Appendices A–C). Finite-workload checking nevertheless has a defined coverage boundary. For all the designs in our benchmark, we have enhanced and refined their testbenches, resulting in testbench coverage of nearly 85% to 90% or higher for each design. Furthermore, the model has absolutely no visibility into the testbench that evaluates our final acceptance criteria during execution, making it virtually impossible for it to cheat by modifying functionality to achieve a higher score.

The released review package also exposes concrete checking implementations. For example, rs_encoder/tb/handshake_tb.sv checks framed output against expected values and enforces output stability under backpressure. rasterizer/tb/check_pixels.py computes an independent pixel/depth oracle and detects missing, extra, duplicate, and incorrect outputs. These materials make the intended observation contracts and failure conditions inspectable; their availability is separate from an execution audit of the accepted high-scoring candidates.

For physical integrity, Appendix H lists all explicit DRC failures and recomputes the scores under stricter requirements. Requiring observed DRC PASS retains Claude as the highest system at EVQ 1.2261, while exchanging the close GLM/Qwen point ordering. This reveals acceptance-profile sensitivity rather than treating a high scalar score as signoff evidence.

Regarding your request to review high-scoring RTL/netlist traces that had already passed verification, we analyzed only the subset of traces that ultimately failed functional testing. In some cases, the model incorrectly interpreted non-simplifiable logic; in others, the test suite generated by the model itself was incomplete, leading to the introduction of non-equivalent modifications. We have not yet observed any instances of models actively cheating during test bench testing; this area still requires further testing and research. Appendix B.2 also flags an inconsistent archived netlist-status field and excludes it as evidence of a pass. The rejection records and sensitivity analysis strengthen score interpretation. Subsequently, we will publicly release all data and actual trajectories from each run of every design as open source to facilitate further research and analysis.

Thank you for guiding us toward a clearer account of the optimization process and the reliability of its outcomes; we hope the measured trajectories, repeated runs, and long-task results strengthen your confidence in SILICA-Bench’s contribution to evaluating how reliably agents turn sustained optimization into delivered improvements, and we welcome further discussion.

Thank you again for your thoughtful suggestions on optimization behavior and reliability; we hope the new trajectories, long-task results, and repeated runs strengthen your confidence in SILICA-Bench’s evaluation of sustained engineering improvement, and would welcome your further thoughts.

Reviewer_uRpPOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SILICA-Bench, a benchmark for long-horizon LLM agents that must improve the post-route PPA of existing digital designs while preserving their behavior. It contains 36 designs from eight domains, using a commercial flow (Design Compiler + Innovus), the ASAP7 PDK, and a 200 ps clock. Agents receive reference RTL, a callable reference flow, and a wall-clock budget of 3× the reference flow time. Depending on their “edit authority” (RTL-only, Flow-only, or Joint), they can edit RTL, flow settings, or both. They must explicitly register (“nominate”) a candidate; only the last registration before the deadline is scored.

Protected assessment replays both the RTL and the post-route netlist against hidden workloads, then computes quality Q as the geometric mean of area, power, and delay-proxy ratios against the reference. The paper reports three metrics: validity rate (R), expected verified quality with failures counted as zero (EVQ), and the rate of changed designs improving by more than 5% (VIR). Six models were run in the Claude Code harness, plus RTL-only and Flow-only variants for Qwen, giving 240 paired nominations on 30 designs. Results: validity is at least 93% for every configuration, but no Joint system improves more than 10/30 designs by over 5%, and EVQ ranges from 0.927 to 1.069. Large regressions are concentrated in nominations that the evaluator rebuilt locally.

Critical strengths
  1. A realistic and carefully designed evaluation setting. The benchmark uses a commercial RTL-to-GDS flow (Design Compiler + Innovus) on real open-source designs from eight domains. Behavior is a hard constraint, and PPA quality is scored continuously. This is much closer to actual hardware engineering than most RTL benchmarks, which mainly check functional correctness. The evaluation is also well protected: both the RTL and the post-route netlist are replayed against hidden workloads, and only the artifact the agent explicitly registers is scored. This prevents agents from getting credit for favorable but unverified reports or for good candidates they never delivered.

  2. Clear separation of validity, quality, and improvement. The three metrics (validity rate, expected quality with failures counted as zero, and rate of real improvement) answer different questions, and the results show why that matters. The reporting is honest and detailed, including per-design results, failure cases, and limitations.

Critical weaknesses

1. Two of the three main claims are not supported by data.

The paper is built around three distinctions: a valid design is not necessarily improved, an early result is not the final result, and finding a good design is not the same as delivering it. Only the first is measured.

  • The paper never reports how often synthesis-level improvements disappear after place-and-route.
  • It never compares the best design an agent found against the one it actually submitted, so we don't know how often agents find a good design but fail to deliver it.

Only the final submission is scored, and the intermediate checkpoints appear only in illustrations. Since the submission protocol and staged evidence are presented as core parts of the benchmark, the paper should show that they change the conclusions.

2. The design source and selection criteria are not described. The paper mentions a “historical design guide” and “archived” coverage, and the design IDs (e.g., G2R10, E14) suggest an earlier collection, but it never says where the designs come from or why these 36 were chosen. This makes it hard to judge how representative the suite is, how much room for improvement each design has, and whether selection was biased toward designs where the flow ran smoothly. All designs are also popular public repositories, so contamination is likely.

Questions for authors
  1. How often does the best design an agent found differ from the one it actually submitted? Would system rankings change if scored by the best design found? This would directly test the claim that finding a good design is not the same as delivering it.

  2. How often do synthesis-level improvements disappear or reverse after place-and-route? This would directly test the claim that an early result is not the final result.

  3. Why use a 200 ps (5 GHz) clock that none of the designs meet? How do the rankings change under a realistic clock where the reference designs close timing?

  4. Where do the 36 designs come from, and what criteria were used to select them? The paper refers to a "historical design guide" and archived coverage, and the design IDs (e.g., G2R10, E14) suggest an earlier collection, but the source and selection process are not described. Were any candidate designs excluded, for example because the reference flow failed or coverage was insufficient?

  5. How much room for improvement do these designs have? What does a non-LLM baseline achieve, such as aggressive synthesis settings, a flow-parameter search, or an expert engineer? Without this, it is hard to judge whether 10/30 improvements is good or poor.

  6. What is the largest design the infrastructure can reliably support end to end? Can the six excluded long-running designs be re-run, and do the conclusions change when they are included? Since the excluded large designs show some of the largest gains (Table F.9), how does agent performance change with design size?

5 comments
AuthorsOpenReview ↗

Response to Reviewer uRpP (1)

Thank you for the detailed questions about what the benchmark measures and how its results should be interpreted. The revision adds measured delivery trajectories, three submissions per primary design–condition combination, and the full 36-design suite. We also report a non-LLM perturbation control and have released inspectable source and test materials for all 36 designs(https://anonymous.4open.science/r/ICLR2027-SILICA-127C/rebuttal/README.md). We answer your six questions below.

1. Best candidate found versus the candidate delivered

The saved nominations already provide direct evidence of delivery loss. Of 1,943 observed runs, 124 (6.38%) finish below their best of four saved nominations, including 55 (2.83%) with a loss of at least 0.05 utility. On the common 323 matched trajectories per system, with all 36 designs equally weighted:

SystemDelivered utilityBest of four saved nominationsMean gap
GPT1.12811.13160.0035
Kimi1.12711.15560.0285
GLM1.10101.13890.0379

The point ordering changes from GPT/Kimi/GLM at delivery to Kimi/GLM/GPT under retrospective selection (Table 3; Appendix G.3). Thus, selection of the final artifact changes the empirical picture even among candidates already saved in the nomination record.

The candidates represented by these four records are a subset of the scoreable candidates from each run. Under the same per-candidate assessments, additional candidates could reveal further delivery losses but cannot remove the recorded losses. Thus, the measured gap is a lower bound on the corresponding complete-history gap; the complete-history optimum and its system rankings remain unmeasured. The evaluator never exposes these hidden scores to help the agent select its delivery.

2. Do synthesis gains survive place-and-route?

In the primary campaign, physical quality is evaluated at the post-route endpoint under the fixed measurement policy. Synthesis or placement feedback alone is insufficient for scoring: the physical evidence must correspond to the delivered RTL–flow artifact and its completed implementation (Section 5.3; Appendix C).

Early-stage performance serves more as a guide for later performance, but it cannot be used to accurately rank results. Often, even when using the same script, good early-stage results do not guarantee better results later on. This is quite common. We therefore distinguish the methodological reason for a post-route endpoint from an empirical claim about how often stage rankings reverse. The measured checkpoint results address saved progress and delivery, not this separate stage comparison. We will then make all data and actual trajectories from each run of every design publicly available as open source to facilitate further research and analysis.

3. The 200 ps clock and timing realism

The original 200 ps value was an imposed optimization constraint; it was not evidence that the designs operated at 5 GHz. A uniform constraint across heterogeneous designs was insufficiently justified. The revision uses frozen per-design clocks, ranging from 140.1 to 1,826.2 ps, and reports their actual reference slack (Table D.2).

Appendix D.2 now documents how these targets were selected. For each design ii, we first characterize the reference under an aggressive high-frequency constraint and obtain a timing-derived reference frequency estimate f^i,probe\hat f_{i,\mathrm{probe}}. We set the target frequency 20% higher:

T_i^{\mathrm{clk}}=\frac{1}{1.2\hat f_{i,\mathrm{probe}}}.$$ The purpose is to give each design a reference-dependent optimization target with a fixed relative margin, instead of imposing the same absolute frequency across heterogeneous designs. The selected clock is then frozen for reference and candidate evaluation. This is a timing-derived calibration, not a silicon-measured maximum operating frequency; Table D.2 separately reports the realized reference slack under the selected clocks. The timing component has a defined relative interpretation, with superscript $0$ denoting the reference: $$D=T^{\mathrm{clk}}-s^{\mathrm{setup}},\qquad D^0-D=s^{\mathrm{setup}}-s^{0,\mathrm{setup}}$$ when the reference and candidate share the same clock. Thus, improving this component means improving setup slack under the same constraint. Clock, path groups, corners, and measurement policy remain fixed; agents cannot improve their score by relaxing them. **All 36 references still have negative setup slack.** The revision therefore measures relative optimization under a declared timing-pressure profile, not timing-closed operation or empirical $F_{\max}$. Appendix C states the measurement scope explicitly.
AuthorsOpenReview ↗

Response to Reviewer uRpP (2)

4. Design sources, selection, exclusions, and contamination

Appendix D documents the complete evaluation population: 36 designs across eight domains, with identifiers, size information, frozen reference PPA and clocks, reference costs, and historical workload-coverage records. Every fixed task remains in the revised primary analysis, including failed and cutoff outcomes. This removes complete-case filtering within the reported suite.

The anonymous repository makes the RTL and testing sources for all 36 designs inspectable, with documented portability changes. The design index enumerates the 36 design folders and DUT tops. Each design includes RTL, test sources, file lists, a README.md, and simulation.json describing its functional configuration and dependencies; third-party copyright and licence notices are retained. The package exposes both the implementations and their concrete checking logic.

This source transparency and complete-suite reporting address different parts of the provenance question. We describe the suite as a fixed collection spanning the stated domains and scales. After enhancing the testbench as much as possible, we selected circuit projects for which the testbench provided sufficiently high coverage (capable of detecting functional errors introduced by LLM modifications) as our benchmarks, which allowed us to enforce stricter control over correctness.

The task itself supplies reference RTL, so prior familiarity with that source is distinct from producing a better implementation under the frozen assessment contract. Public-source exposure may nevertheless help a model. Protected workloads and isolated scoring restrict access to hidden feedback during search, without establishing absence from pretraining. The assessment inputs now published for inspection were unavailable to agents during the reported searches; the release documentation distinguishes them from the historical agent-visible inputs (Appendix I).

5. Improvement headroom and non-LLM calibration

The revised results show substantial observed headroom relative to the supplied reference, spread across the suite. Counting the per-design averages of all nine scheduled runs in Table F.1, Claude and GPT each exceed unit utility on 34/36 designs. Claude exceeds 1.05 on 33/36; recomputing its mean after removing its highest-scoring design leaves EVQ 1.2259. These additional summaries of Table F.1 concern design averages, not claims that every run improves.

Claude's accepted-output geometric-mean ratios are A/A0=0.7960A/A^0=0.7960, P/P0=0.7062P/P^0=0.7062, and D/D0=0.9079D/D^0=0.9079, alongside failure-inclusive EVQ 1.2541 (Tables 1 and F.9). These component measurements make the nature of the achieved improvement visible; EVQ is not a percentage reduction in every PPA component.

We have additionally performed hundreds of non-LLM trials using local equivalence-preserving design perturbations and random flow-script modifications. The design-specific best observed scores were approximately:

DesignBest observed control score
AXI Virtual FIFO1.012
PtahConv1.008
BMB interconnect1.019

These additional results characterize the tested local exploration procedure.

Observed improvement headroom does not imply that a local search policy will find it: such a policy may miss useful structural or coordinated flow changes. Synopsys documents how RTL coding structure can affect datapath extraction and how feedback-guided changes can expose better synthesis opportunities (datapath analysis). Strong tool optimization therefore does not imply that the supplied reference is optimal. The useful benchmark question is whether a system can find, validate, and deliver such improvements through the existing tools.

These control results are per-design best-of-search observations and are reported separately from system-level mean EVQ. They calibrate this local perturbation policy. The broader flow-tuning literature studies materially different search policies and costs (METRICS2.1, ICCAD 2021).

AuthorsOpenReview ↗

Response to Reviewer uRpP (3)

6. Complete task coverage, supported scale, and size effects

All six previously excluded designs are now included. One GLM/PtahConv outcome remained unscored at the reporting cutoff; it receives zero analysis utility while the design remains included and its unavailable trajectory is left unfilled. Table D.5 isolates the effect of inclusion by comparing the 30-design subset and all 36 designs under the same revised protocol:

SystemSame-protocol 30 designsAll 36 designsFour reference flows >3 h
Claude1.24441.25411.3602
GPT1.13151.12751.1842
Kimi1.12391.12631.2676
GLM1.11401.09721.0835
Qwen Max1.06611.07801.2071
Grok1.04981.04531.0886

Including the six designs preserves the aggregate point ordering. The long-task stratum provides additional information: Kimi and Qwen Max exceed GPT's point estimate there. These are four designs, each with nine scheduled runs per system (Table F.11), not 36 independent designs.

The largest demonstrated reference is AXI Virtual FIFO: 139,707.57 μm² post-route cell area, with a 7.2776-hour reference flow (Tables D.1–D.3). An additional summary of its primary run records gives 54/54 accepted outcomes across six systems, three Joint conditions, and three submissions. BMB interconnect also has 54/54. This provides repeated end-to-end evidence at these scales; it does not identify the infrastructure's maximum capacity.

Performance varies with system and size. Across Small/Medium/Large strata, Claude's EVQs are 1.1385/1.3219/1.3992, GPT's 1.0746/1.1550/1.2089, and GLM's 1.0702/1.1418/1.0085 (Table F.12). Five systems have a higher Large than Small point estimate; GLM has a different pattern. The strata use the disclosed historical size proxy and contain 15/17/4 designs. We present this as descriptive size sensitivity.

We sincerely appreciate your detailed feedback, which substantially strengthened our experiments and their interpretation; we hope the new evidence clarifies SILICA-Bench’s contribution in distinguishing progress achieved during search from improvement actually delivered, and would be grateful for your favorable reconsideration and the opportunity to discuss any remaining concerns.

Reviewer_mrR8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SILICA-Bench, a benchmark for evaluating LLM agents on time-bounded optimization of digital circuits through a commercial RTL-to-GDS workflow. The central methodological contribution is a nomination-based evaluation protocol. The agent must register an immutable RTL-flow snapshot before the deadline. Only the latest completed registration is assessed. Protected evaluation then checks both RTL behavior and the corresponding post-route netlist on hidden workloads, verifies the completeness and identity of the physical implementation, and computes a normalized geometric-mean PPA score. The paper reports three primary metrics: validity rate, failure-inclusive expected verified quality (EVQ), and verified improvement rate (VIR).

The empirical study evaluates six model-harness systems under joint RTL-and-flow authority. One system is additionally evaluated under RTL-only and flow-only authority. After complete-case filtering, the main analysis contains 240 terminal nominations: eight configurations evaluated on 30 designs. The principal empirical finding is that high validity does not imply frequent improvement. All joint configurations achieve at least 28/30 valid nominations, but no joint configuration improves more than 10/30 tasks by over 5%. The paper also finds that edit authority has mixed effects and that large regressions are disproportionately associated with locally reconstructed implementations rather than reused public-flow outputs.

The evaluation protocol is thoughtful and unusually explicit about artifact identity, failure-inclusive scoring, and the difference between a candidate explored during a run and the artifact actually delivered. However, the current experiments do not yet convincingly demonstrate the paper’s central “long-horizon” claims, and the physical-validity profile is sufficiently relaxed that the term “verified physical improvement” risks overstating what is established. The absence of repeated agent runs, controlled reproducibility experiments, meaningful non-LLM baselines, and empirical checkpoint or stage-transition analyses further limits the strength of the conclusions.

Critical strengths
  1. The benchmark tests an important artifact-producing task. RTL-to-GDS optimization is a valuable setting for studying agents that must preserve hard requirements while making coupled decisions using costly evidence. The benchmark is more informative than a single-turn RTL-generation test.

  2. The paper introduces new evaluation method beyond the RTL-only test via matched dual replay. Checking both candidate RTL and the corresponding post-route netlist catches cases in which the source behaves correctly but the actual implementation does not. The three reported functional failures with complete physical outputs demonstrate why this gate matters.

Critical weaknesses

One of the core critical weakness is that the experiments do not necessarily establish the claimed long-horizon phenomena

The paper made three distinctions: 1) valid is not improved; 2) early evidence is not final quality; 3) a candidate found during search is not necessarily the artifact delivered.

The experiments convincingly demonstrate the first distinction, but provide little direct empirical evidence for the second and third.

Only the terminal nomination at (k=1) is scored. The 25%, 50%, and 75% checkpoints in the figures are conceptual rather than empirical. The paper does not report, for example, 1) nomination quality as a function of budget fraction; 2) how often a strong candidate was registered and then replaced by a worse terminal candidate; 3) the best public candidate found versus the terminal candidate delivered; 4) rank reversals between synthesis, placement, and post-route measurements; 5) whether more agent steps or longer evidence chains correlate with success; 5) performance under multiple announced budget levels.

Similarly, RQ3 asks whether budget spent becomes improvement delivered, but the analysis mainly compares average budget consumption with final EVQ across systems. This does not isolate a budget effect. A system consuming 99% of its budget and performing poorly only shows that elapsed time alone is not a success metric, which is nearly tautological. This matters because nomination semantics are a strong protocol contribution, but the paper currently demonstrates that the protocol could measure long-horizon behavior more than it demonstrates the behavior empirically.

Questions for authors
  1. Can the authors report actual 25%, 50%, 75%, and 100% nomination trajectories from the registration ledger?

  2. What are the rank correlations and reversal rates between synthesis estimates, placement results, and final post-route PPA?

  3. How sensitive are the results to repeated agent runs? Do the leading systems remain leading across seeds on even a small task subset?

3 comments
AuthorsOpenReview ↗

Response to Reviewer mrR8 (1)

Thank you for identifying the evidence needed to turn the protocol into an empirical study of long-horizon behavior. The revision now measures saved progress and delivery, repeats every primary design–condition combination, and includes all 36 designs. The primary campaign comprises six deployed model–harness systems, three Joint conditions, and three independent submissions: 1,944 scheduled runs. Sections 5–6 document the revised timing, resource, and finalization settings. We also report an additional non-LLM perturbation control below, separately from this primary campaign.

1. Actual checkpoint trajectories and delivery loss (Question 1)

Table 3 and Figure 5(b) now contain measured trajectories. At each checkpoint, we record the latest nomination completed by that time; the saved identities are subsequently assessed under the protected profile. Below we summarize these trajectories together with their retrospective best-of-four utility. We use the same 323 matched trajectories per system, average within each design, and weight all 36 designs equally.

System25%50%75%Final deliveryBest of four
Claude1.09661.20931.23461.25361.2576
GPT1.06241.11191.12641.12811.1316
Kimi1.02831.08391.12321.12711.1556
GLM1.03101.07741.10251.10101.1389
Qwen Max1.00671.01671.03671.07701.0981
Grok0.98671.01561.03931.04641.0751

All entries are failure-inclusive utility. “Best of four” first takes the maximum within each run, then applies the same averaging. The separate primary terminal analysis retains all 324 scheduled runs per system, including one unscored GLM cutoff outcome assigned zero analysis utility.

Final denotes the actual delivery event. Early completion is carried forward to subsequent checkpoints. A resource-triggered stop permits at most 60 seconds and one response to select an artifact frozen before the search cutoff, with no edits or new EDA calls; hidden scores are unavailable to the agent (Section 5.4; Appendix G.2). Thus, Final is not relabeled as a 100%-wall-time observation.

For each observed run, we measure

Ga=max⁡k∈{0.25,0.50,0.75,final}Ua,k−Ua,final.G_a=\max_{k\in\{0.25,0.50,0.75,\mathrm{final}\}}U_{a,k}-U_{a,\mathrm{final}}.

124/1,943 observed trajectories (6.38%) finish below their best saved nomination; 55/1,943 (2.83%) lose at least 0.05 utility (Appendix G.3). The latter is an absolute utility difference. These results directly establish that quality attained by a saved candidate can be lost at delivery. Retaining progress and selecting the final artifact are therefore measurable parts of the task, alongside finding an improved implementation.

The four-point comparison also has a useful conservative interpretation: under the same per-candidate assessments, adding other scoreable candidates from that run can only increase its retrospective best and delivery gap. The observed losses therefore do not depend on having recovered every candidate. However, this is a subset bound, not a statistical confidence bound or an estimate of the complete-history best. Retrospective selection also changes the GPT/Kimi/GLM point ordering; it is not an online selection policy or evidence that every ordering difference is significant.

2. Synthesis, placement, and post-route evidence (Question 2)

The contribution here is an explicit physical evaluation endpoint. Public synthesis and placement reports inform search; the score requires the nominated artifact's routed, extracted implementation and matching behavioral and physical evidence. An early estimate alone does not establish that this implementation was completed and delivered. The released source package(https://anonymous.4open.science/r/ICLR2027-SILICA-127C/rebuttal/flow/README.md) makes the stage interfaces inspectable: synthesis.tcl (https://anonymous.4open.science/r/ICLR2027-SILICA-127C/rebuttal/flow/synthesis.tcl) produces a mapped netlist, place.tcl (https://anonymous.4open.science/r/ICLR2027-SILICA-127C/rebuttal/flow/place.tcl) performs placement optimization, and cts_route.tcl (https://anonymous.4open.science/r/ICLR2027-SILICA-127C/rebuttal/flow/cts_route.tcl) performs clock-tree construction, routing, parasitic extraction, and post-route reporting. These are flow references.

Section 5.3 separates the endpoint definition from that empirical question. The new temporal trajectories establish delivery loss. We will subsequently make all data and actual trajectories from each run of each design available as open source to facilitate more thorough research and analysis.

AuthorsOpenReview ↗

Response to Reviewer mrR8 (2)

3. Repeated submissions and stability of the leading result (Question 3)

All 108 design–Joint-condition combinations per system now have three independent submissions. Claude leads in each repetition, with EVQs 1.2618, 1.2664, and 1.2340, and in every leave-one-design-out comparison. Its five primary pairwise contrasts also pass Holm correction across the 15 system comparisons (Tables F.5 and F.10).

We report close comparisons without overstating their ordering: GPT minus Kimi is 0.0012, with paired 95% interval [−0.0460, 0.0517]. Table F.10 reports within-combination run variability separately from the design-cluster uncertainty in Appendix G.1. These are independent submissions, not controlled provider sampling seeds.The 0.05 delivery-loss threshold is therefore descriptive.

4. Search activity and the role of budget

We agree that elapsed time versus final quality does not identify a causal budget effect. RQ3 now asks how nominated quality survives delivery (Section 6.4). We additionally report observable tool activity: after centering within design–condition groups, EDA-request count correlates with terminal utility at 0.403 for Claude and 0.460 for GPT, compared with 0.262 and 0.174 for search time (Table F.17).

The primary campaign records 24,491 EDA requests and 7,149 nominations, with approximately 10,181.9 cumulative search hours and 55,787.9 EDA CPU core-hours (summing Table F.14). Wall-time and CPU coverage are 1,943/1,944 and 1,929/1,944 runs, respectively. These are accumulated run-level costs, and search is accounted for separately from protected assessment.

These measurements describe tool interaction, not private reasoning depth or evidence-chain length. All checkpoints belong to the same announced budget. The configured resource contract and actual consumption are reported separately in Appendix F.5.

5. Non-LLM perturbation control and its interpretation

We also conducted hundreds of non-LLM trials involving local equivalence-preserving design perturbations and random flow-script modifications. The best observed scores for the three named designs were approximately:

DesignBest observed control score
AXI Virtual FIFO1.012
PtahConv1.008
BMB interconnect1.019

These results provide an empirical reference point for the tested local exploration procedure.

A plausible explanation is that the reference already benefits from substantial commercial synthesis optimization, while the final outcome depends on interacting placement, clocking, and routing decisions (Design Compiler; Innovus). Consequently, a local edit can leave a similar optimized implementation or introduce trade-offs that limit its final benefit.

The control strengthens the motivation for evaluating purposeful engineering search with costly feedback: the agent must identify useful changes, preserve behavior, and retain the benefit through implementation and delivery. We report these per-design best-of-search observations separately from the agents' failure-inclusive mean EVQ, because task coverage and aggregation differ. The comparison characterizes the tested local exploration procedure. Flow-only LLM runs retain their role as interface comparisons.

6. Meaning and limits of verification

“Verified” refers to the declared workload and implementation-evidence contract: protected RTL and corresponding netlist replay, plus complete, identity-matched physical evidence (Sections 5.1–5.3; Appendices B–C). Acceptance under this profile is separate from exhaustive equivalence or timing/physical signoff; Appendix B.2 also retains the qualification concerning the archived netlist-status field.

We quantify sensitivity to a stronger physical requirement. Requiring an observed DRC CLEAN PASS gives Claude EVQ 1.2261, still the highest, while the close Qwen/GLM point ordering changes (Tables H.3–H.4). This exposes which conclusions depend on the acceptance profile. The revised empirical contribution is the measured separation among acceptance, delivered utility, and retained progress, with each interpreted under its stated contract.

Thank you for helping us substantiate SILICA-Bench’s distinction between saved progress and final delivery; we hope the measured nomination trajectories and repeated runs strengthen your confidence in this contribution to long-horizon agent evaluation, and we would welcome further discussion of any remaining concerns.

Thank you again for your thoughtful scrutiny of the evaluation protocol; we hope the new trajectories and repeated evaluations clarify its contribution to measuring retained progress and final delivery, and would appreciate your reassessment and any further questions.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response to the reviewers——New changes (1)

We thank all four reviewers, the Senior Area Chairs, and the Program Chairs for helping strengthen the evidence and presentation of SILICA-Bench. The substantive manuscript revisions are marked in blue. The central evaluation question is whether an agent can improve a persistent engineering artifact while preserving its required behavior, use costly feedback to guide further decisions, and deliver the resulting benefit. The revision now supports this question with complete-suite repetitions, measured progress and delivery, explicit resource accounting, and inspectable task implementations.

1. Experimental breadth, scale, and repeated evaluation

The primary comparison covers all 36 designs, six deployed model–harness systems, three Joint conditions, and three independent submissions: 1,944 scheduled runs. The eight domains include bus/NoC/DMA, arithmetic/FEC, cryptography, memory/control, I/O/media, CPU, ML, and rasterization. Reference post-route cell area spans 82.45–139,707.57 μm² (Tables D.1–D.2).

The full study comprises 3,924 scheduled runs and 3,923 scored outcomes: 1,944 primary runs, 1,728 additional matched RTL-only/Flow-only runs across R0/F1–F3 for four systems, and 252 auxiliary Qwen Flash runs using an earlier task package. These three analysis populations are kept separate (Section 6.1; Appendix E).

The resource records quantify the extent of the primary search campaign:

Recorded primary-search quantityAggregateAvailable records / scheduled runs
SEARCH wall time≈10,181.9 run-hours1,943/1,944
EDA CPU consumption≈55,787.9 core-hours1,929/1,944
EDA requests24,4911,943/1,944
Nominations7,1491,943/1,944

These additional sums use the resource records summarized in Table F.14. They accumulate observed search-phase costs, including model/tool waits in wall time, and exclude protected assessment.

Failure-inclusive EVQ ranges from 1.0453 to 1.2541, with acceptance rates of 92.28%–99.07%. All six previously omitted designs are retained. The same-protocol 30/36-design comparison preserves the aggregate point ordering (Tables 1 and D.5). Claude leads in each repetition and every leave-one-design-out comparison, and its five primary pairwise contrasts pass Holm correction. Close comparisons remain uncertain: GPT minus Kimi is 0.0012, with paired 95% interval [−0.0460, 0.0517] (Tables F.5 and F.10).

2. Actual long tasks, saved progress, and final delivery

Reference full-flow times span 23.07 minutes to 7.28 hours. Configured allowances are 8Tiref8T_i^{\mathrm{ref}} for wall time and 8Ciref8C_i^{\mathrm{ref}} for EDA CPU time, with each run using a 16-core pool shared across at most two concurrent EDA jobs. Configured allowances and observed consumption are reported separately.

All four designs with reference flows exceeding three hours are included. Their search allowances are 31.85–58.22 hours, with 203 accepted outcomes among 216 scheduled runs. Additional resource summaries give system-level median search durations of 6.52–31.68 hours on these tasks. The long-task medians use observed records: 35 for GLM and 36 for each other system. Outcome denominators retain all 36 scheduled runs per system (Section 6.5; Table F.11).

Table 3 and Figure 5(b) report actual saved nominations at 25%, 50%, and 75% of the announced wall allowance, together with final delivery. Final is the delivery event; resource-triggered finalization only permits bounded selection among already frozen candidates. Hidden assessment scores never guide that selection.

Across 1,943 observed trajectories, 124 (6.38%) finish below their best of four saved nominations, including 55 (2.83%) with an absolute utility loss of at least 0.05. Retrospective saved-best selection changes the GPT/Kimi/GLM point ordering. This directly measures whether recorded progress survives delivery; it concerns saved nominations, rather than an exhaustive reconstruction of every explored candidate (Appendix G.3). Elapsed time documents interaction cost; the trajectory comparisons provide the evidence about retained progress.

AuthorsOpenReview ↗

General response to the reviewers——New changes (2)

3. Explicit timing conditions and verification scope

The uniform 200 ps constraint is replaced by frozen per-design clocks spanning 140.1–1,826.2 ps. Appendix D.2 now explains the selection procedure: a high-frequency reference probe supplies a timing-derived frequency estimate f^i,probe\hat f_{i,\mathrm{probe}}, from which we select

\qquad T_i^{\mathrm{clk}}=\frac{1}{f_{i,\mathrm{target}}}.$$ The aim is a reference-dependent, fixed relative challenge instead of imposing the same absolute clock on heterogeneous designs. The selected clock is frozen for reference and candidate evaluation, preventing improvement through clock relaxation; Table D.2 reports the realized reference slack. All 36 final references retain negative setup slack. Accordingly, $D_i=T_i^{\mathrm{clk}}-s_i^{\mathrm{setup}}$ measures constraint-relative timing improvement; timing-closed operation and rankings under a timing-closed profile remain unestablished. Acceptance follows the declared finite-workload and artifact-matched implementation contract, with the archive qualifications retained in Appendix B. Requiring observed DRC PASS still leaves Claude highest at EVQ **1.2261**, while changing the close GLM/Qwen ordering (Appendix H). Acceptance under this profile is distinct from physical signoff. ## 4. Improvement headroom and a named non-LLM reference Claude and GPT each exceed unit mean utility on **34/36 designs** (derived from Table F.1). For Claude's accepted outputs, the geometric-mean ratios are $A/A^0=0.7960$, $P/P^0=0.7062$, and $D/D^0=0.9079$ (Table F.9). These component measurements show the physical changes underlying the quality score. We also conducted hundreds of trials using local equivalent-design perturbations and random flow-script modifications on the following designs: | Tested design | Reference post-route cell area (μm²) | Best observed non-LLM score | | -------------- | ---------------------------: | --------------------------: | | [AXI Virtual FIFO](https://anonymous.4open.science/r/ICLR2027-SILICA-127C/rebuttal/axi_vfifo/README.md) | 139,707.57 | ≈1.012 | | [PtahConv](https://anonymous.4open.science/r/ICLR2027-SILICA-127C/rebuttal/ptahconv/README.md) | 21,944.15 | ≈1.008 | | [BMB interconnect](https://anonymous.4open.science/r/ICLR2027-SILICA-127C/rebuttal/bmb_ooo/README.md) | 9,505.50 | ≈1.019 | Areas are frozen-reference measurements from Table D.2; the scores are **per-design observed maxima from the additional local exploration**, reported separately from failure-inclusive mean EVQ. The links provide design definitions and functional configurations. This provides an empirical reference for the tested procedure. Strong commercial optimization and remaining headroom can coexist: [Synopsys' datapath analysis](https://www.synopsys.com/articles/techniques-for-improving-qor.html) explains how RTL organization affects the optimization opportunities visible to synthesis. This supports the interpretation that a local perturbation procedure need not exhaust available optimization opportunities; it is not a measured attribution of each trial. ## 5. Public sources and search-time testing **The [anonymous repository](https://anonymous.4open.science/r/ICLR2027-SILICA-127C/) (https://anonymous.4open.science/r/ICLR2027-SILICA-127C/) provides RTL and testing sources for all 36 designs**, functional configurations, dependencies, retained licence notices, and five Tcl flow-reference files: | Material | What reviewers can inspect | | ------------ | ---------- | | [Design index](https://anonymous.4open.science/r/ICLR2027-SILICA-127C/rebuttal/README.md) | All 36 instances and source-package scope| | [Flow documentation](https://anonymous.4open.science/r/ICLR2027-SILICA-127C/rebuttal/flow/README.md) | Synthesis, placement, routing and extraction interfaces and dependencies | | [RS encoder checker](https://anonymous.4open.science/r/ICLR2027-SILICA-127C/rebuttal/rs_encoder/tb/handshake_tb.sv) | Expected framed outputs and stability under backpressure| | [Rasterizer checker](https://anonymous.4open.science/r/ICLR2027-SILICA-127C/rebuttal/rasterizer/tb/check_pixels.py) | An independent pixel/depth oracle and output-error checks| The published assessment inputs were unavailable during the reported searches. Agents could use **Icarus Verilog and Synopsys VCS** and construct checks against the visible reference RTL (Section 5.2; Appendix A.3). The release documents portability changes and external dependencies; the full production harness, protected scoring service, and historical result/analysis artifacts are not included in this source package.
AuthorsOpenReview ↗

General response to the reviewers——New changes (3)

6. Significance for AI evaluation and engineering agents

SILICA-Bench jointly measures acceptance, continuous physical quality, and retention of progress under a fixed behavioral contract. This complements SWE-bench, RE-Bench, and Terminal-Bench with behavior-preserving physical optimization and artifact-bound post-route assessment. Many implementations can satisfy the same behavioral target; useful progress is continuously graded among accepted results.

For LLM4EDA, the permission experiments make the action interface an empirical variable. With RTL fixed, F2 stage-command editing exceeds F1 parameter editing by 0.2326/0.1458/0.0873 EVQ for Claude/GPT/Qwen Max, with all three contrasts passing Holm correction. All six J3−J2 intervals include zero (Table 2; Appendix F.2). Thus, the benchmark can reveal which systems benefit from which engineering interfaces, while evaluating physical outcomes through the same declared contract.

For practical engineering agents, persistent candidate identities connect decisions and checks to a concrete deliverable. Costly, delayed feedback makes experimentation and validation consequential; the final nomination rule makes retaining progress and committing the appropriate artifact measurable parts of performance. The observed delivery gaps give these design goals an empirical basis. The contribution is an evaluation, supported by repeated runs and explicit resource accounting, of whether agents convert tool interaction into accepted improvements that survive final delivery.

We sincerely thank all reviewers for their thoughtful feedback, and hope the expanded evidence and released materials strengthen your support for SILICA-Bench’s contribution to evaluating whether LLM agents can preserve required behavior, retain progress, and reliably deliver engineering improvements through sustained tool interaction; we warmly welcome further discussion of any remaining questions.

AuthorsOpenReview ↗

SILICA-Bench: Scientific Contribution and Commitment to a Lasting Community Benchmark

Dear Area Chair and Reviewers,

We sincerely thank you for your careful reading, constructive feedback, and continued discussion. Your suggestions have substantially strengthened the experiments, presentation, and accessibility of SILICA-Bench.

1. AI evaluation grounded in real engineering needs

SILICA-Bench connects long-horizon LLM evaluation with a concrete industrial need: improving existing chip designs while preserving their required behavior. Agents must make coupled RTL and flow decisions, use costly feedback, manage finite resources, and deliver an implementation that passes independent assessment. This provides a concrete setting for studying planning, tool use, constraint preservation, and reliable completion through engineering outcomes.

The central insight is that useful progress must survive final delivery. The revised primary study covers 36 designs across eight domains, six deployed systems, and 1,944 scheduled runs, with three independent submissions per system for each design–Joint-condition combination. All four designs with reference flows exceeding three hours are included. Among 1,943 observed trajectories, 124 finish below the best of their four recorded nominations (Table 3; Appendix G.3). Measuring this gap makes retaining progress and selecting the final artifact explicit parts of agent evaluation.

For the AI community, this brings an industrially motivated class of continuously graded, behavior-preserving optimization tasks into agent research. Our aim is reciprocal progress: engineering requirements expose concrete gaps in AI capabilities, while improved agents can be assessed through useful, measurable physical outcomes. Semiconductor design provides a demanding setting in which to develop this connection between AI research and advanced engineering practice.

2. Recognition and guidance from the reviewers

We appreciate the complementary strengths identified across the original reviews and subsequent discussion:

  • mrR8 described the nomination semantics as “a strong protocol contribution” in the original review.
  • uRpP highlighted “A realistic and carefully designed evaluation setting” in the original review, and subsequently acknowledged that most concerns had been addressed.
  • vrRe recognized the gap between functional validity and improvement: “The experiments show a useful gap.”
  • qBta, in the latest discussion, noted that the benchmark “could benefit the community”, following the expanded long-task evaluation, clearer writing, and release of materials.

These comments and the accompanying questions have helped us clarify both the scientific contribution and the evidence supporting it.

3. Commitment to a maintained and fair community benchmark

We are committed to developing SILICA-Bench into a mature, accessible, and sustained open benchmark. Building on the design, testing, and flow-reference sources already available in the anonymous repository, we will:

  • Maintain the benchmark implementation and testing entry points: complete the intended open-source release, document installation and evaluation procedures, provide worked examples, and state tool and technology dependencies clearly.
  • Build and maintain a dedicated website: provide a task catalogue, evaluation instructions, release history, and results organized by benchmark version.
  • Expand the tasks regularly: add circuits and testbenches, develop more demanding long-horizon workflows, and connect this evaluation methodology with other real engineering problems.
  • Conduct periodic, transparent evaluations of multiple LLM systems: use a declared benchmark version and fix tasks, protocols, budget rules, and assessment conditions within each comparison; disclose model, harness, and tool configurations; and report repeated runs, failures, and actual resource consumption so that comparisons remain fair and interpretable.

Our long-term aim is to help the community develop more capable and reliable LLM agents for consequential engineering work. We would be grateful for your favorable consideration of the revised contribution, and warmly welcome further questions, suggestions, and opportunities for scientific exchange. Thank you again for helping us strengthen both the study and its future value to the community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.