acceptodds
Under review as a conference paper at ICLR 2027

AIChipBench: State-Aware Evaluation of Coding Agents for AI Accelerators

Abstract

An RTL accelerator can emit the correct token while corrupting the state that later tokens depend on. For a coding agent, such delayed failures are hard to debug: the first visible mismatch may appear well after the faulty update. We introduce AIChipBench to evaluate whether agents can use public verification feedback to build implementations that pass independent acceptance as cross-module integration and persistent-state requirements increase. Each task pairs an executable fixed-point contract with a public validation ladder, an isolated held-out evaluator, and tool-use accounting, so that model fidelity, implementation correctness, and agent reliability are measured separately. On three controlled integration tasks with a four-layer Qwen-derived test model (Qwen3.5-nano), each attempted 15 times (five attempts from each of three agent configurations), final acceptance is 11/15 for a single layer, 7/15 for a four-layer one-step decoder, and 2/15 for eight-step recurrent decoding; all nine attempts that pass the public ladder but are rejected fail the held-out numerical checks. Scaffold-adaptation case studies cover Qwen3.5-0.8B and 4B; three Qwen reference designs satisfy their finite suites, while additional Gemma, Llama, and Qwen3.6-27B checks probe the limits of scaffold reuse. With arithmetic held fixed, paired schedules improve decode throughput by 12.5–15.0% at essentially unchanged LUT usage. The benchmark thus separates the feasibility of a hardware design from an agent's ability to construct it reliably, making state-preserving integration a concrete target for agent evaluation.

Primary area:
datasets and benchmarks
Keywords:
coding agents, AI for EDA, RTL generation, AI accelerators, hardware design, benchmark, stateful decoding, formal verification, tool-using agents

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FwrcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces AIChipBench, which evaluates whether coding agents can implement the fixed-point inference of a language model as synthesizable RTL. Each task is defined by a bit-exact integer reference that specifies the outputs and the persistent state after every decoding step. During construction, agents can run public tests that report the first mismatch; after the submission is frozen, an isolated evaluator runs held-out tests, synthesis, and integrity checks. The main experiment uses Qwen3.5-nano, a four-layer model built by the authors, with three tasks: a single layer, a four-layer one-step decoder, and eight-step recurrent decoding. Three agent configurations make five attempts on each task, and 11/15, 7/15, and 2/15 attempts are accepted. The paper also reports a comparison of two agents, a state-feedback ablation, adaptation case studies on Qwen3.5-0.8B and 4B, and the throughput and energy efficiency of the authors' reference designs.

Critical strengths
  • S1. A realistic problem that is harder than existing RTL benchmarks. Existing benchmarks mostly test small, independent modules and grade them by their outputs. This paper asks agents to build a complete, stateful decoder accelerator, whose correctness depends on state carried across modules and decoding steps, and in which an error may surface several steps after it occurs. A single attempt typically runs for hours (a median of 1.8 to 3.6 hours on the three nano tasks), and T1 to T3 add cross-layer integration and cross-step state one level at a time, which shows at which level agents start to fail.
  • S2. An evaluation designed around this difficulty. At every transaction boundary, the RTL's outputs and state are compared with a bit-exact integer reference, so a faulty state update is caught at the step where it occurs, before it reaches the output. In the output-only feedback condition, 6 submissions with incorrect state pass the output-only public checks, which shows what grading by outputs alone would miss. Fidelity (fixed point vs. FP32) and correctness (RTL vs. fixed point) are measured separately, so agent failures are not confused with quantization error.
Critical weaknesses

W1. The benchmark has too few tasks and attempts to support its conclusions, and its main result does not address its own motivation.

  • One decoder specification. T1, T2, and T3 are nested slices of a single design (one layer, the four-layer decoder for one step, and the same decoder over eight steps), and the 16 "task instances" per level share the same task definition (Appendix L). The 0.8B and 4B tasks scale up the same architecture from an existing scaffold, with a single trajectory for 4B. Table 3 places "45; 152 attempts" next to 156 problems for VerilogEval and 783 for CVDP, but these are attempt counts from two experiments; counted as tasks, AIChipBench has 3 nano tasks and 2 adaptation tasks.
  • The main result does not test the motivation. The paper is motivated by state errors that corrupt later tokens while the current output is still correct. Its main result, acceptance falling from 11/15 to 7/15 to 2/15, is what any set of increasingly difficult tasks would show, and only the endpoints differ significantly (Fisher's exact test: p = 0.26 for T1 vs. T2, p = 0.11 for T2 vs. T3). The results that bear on the motivation are small: 6 submissions pass output-only checks but fail state checks, and state feedback adds 2 accepted attempts out of 16 (p = 0.625). The paper does not measure how many steps separate a faulty state update from the first output error in real submissions, or whether state feedback helps agents locate faults. The agent comparison cannot rank agents either (McNemar p ≥ 0.625).
  • Much of the paper does not evaluate agents. The abstract, Block B of Table 1, Section 6, and most of Appendices J and K report on the authors' reference designs.

W2. The paper omits information needed to interpret and reproduce its results.

  • How much the agent writes. The paper does not report the size of the reference implementations or how much RTL the starting files and scaffolds already provide. Who built the reference designs is also unclear: Figure 1 says they are not agent submissions, yet the 0.8B reference design has exactly the same test results as the agent trajectory (Table 2, Listing 2).
  • Where the hardware numbers come from. All hardware results come only from simulation and synthesis. The memory model, the source of the power figures, and the FPGA part are not specified, and hardware data for agent designs are reported only for accepted 0.8B submissions.
  • How the agents are configured. Appendix G lists reasoning and sampling settings, prompts, and the context-management policy as required parts of the run specification, but the paper reports none of them for any agent, and per-attempt tool-call and token budgets are given only for the agent comparison (Table 13). How the diagnosis labels were assigned is not described.

Without this information, readers cannot judge how difficult the tasks are or reproduce the reported results.

Questions for authors
  1. Implementation size. How many lines of RTL do the reference implementations contain, how much of this is provided by the nano starting files or the 0.8B and 4B scaffolds, and how much must the agent write itself?
  2. Reference designs. Who built the 0.8B and 4B reference designs, and how do they relate to the agent adaptation trajectories? Figure 1 states that reference designs are not agent submissions, but the 0.8B reference results in Table 2 match those of the agent trajectory in Listing 2.
  3. Agent configurations. What reasoning and sampling settings, prompts, and context-management policies were used for each agent configuration (the three systems in the main experiment, Sol 5.6, and Astra 6)? What tool-call and token budgets applied to each attempt in the main experiment?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Fwrc (1/3)

Thank you for the detailed review. Many of the revision's additions respond directly to the gaps you identified; we answer each point in turn.

W1a. Too few tasks; T1 to T3 are nested; Table 2 compares attempts with problems.

  • The correctness track has eight tasks on five targets from three families: Qwen3.5-nano T2 and T3, Qwen3.5-0.8B and 4B from a scaffold, and Gemma 4 E4B and Llama 3.2 3B from interfaces and from a scaffold (Table 1). Four extension tasks on Gemma 3 4B and Llama 3.2-1B add 1,024 attempts (App. D.2). With T1, AIChipBench has 13 agent tasks on seven targets.
  • Qwen3.5-4B now has 96 adaptation attempts, 24 per agent (Table 4); Opus 5.5 has 12 of 24 accepted, against 20 or 21 for the others.
  • Table 73 counts tasks and lists the 6,768 attempts separately. Sec. 5 states that problem, design, example, task, and attempt counts are not comparable units.
  • T1 to T3 form the earlier task series (App. G), and the main results no longer rest on them.

We acknowledge that 13 tasks are few next to suites of hundreds of problems. Each task is large, however: a correctness-track task implements a complete decoder whose reference has 6,843 to 31,884 lines of RTL (Table 7); T1 is a single decoder layer.

W1b. The main result does not test the motivation.

We agree that a decline from T1 to T3 is what any ladder of harder tasks produces, and that only its endpoints differ significantly. The revision tests the motivation directly.

  • Lag in real submissions (Sec. 4.3, App. D.6). In 69 accepted submissions regraded to 256 steps, 482 of 624 failing sequences first diverge in persistent state; 470 later fail in output, a median of 4 steps later (IQR 2 to 8, maximum 40), and 12 show no output mismatch by step 256.
  • What output grading at the task horizon misses. At 32 steps, 17 of 85 failing sequences are wrong in state only. An output-only check finds all 85 at 48 steps, and the paper states the motivation as earlier detection of the same defects.
  • Whether state feedback helps agents locate defects (Sec. 4.2). Over 1,152 attempts, full-state feedback beats output-only feedback in all 12 agent–task cells (+25.0 to +46.9 points; every block-bootstrap interval excludes zero). Among attempts with a state defect, locating the mismatch raises repairs from 87 of 169 to 123 of 166, while discovery is similar (142 of 169 and 151 of 166).
  • Horizon with other factors fixed (Sec. 4.4). From one scaffold at the 1× budget, acceptance falls from 252 to 115 of 288 between one and 16 steps.

W1c. The agents cannot be ranked. The revision ranks them and shows that the ranking reproduces (Sec. 3, App. D.8, D.9, Table 54).

  • Overall score. Fable 5.1 (32.4) and Sol 5.6 (31.3) rank above Astra 6 (19.4) and Opus 5.5 (16.3). When the matched instances of every task are split into random halves, both halves give this ranking in 179 of 200 splits, against 17% when the agents' outcomes are permuted within instances (p = 0.003). It also holds under every way of combining the tracks that includes the PPA track, and with routed PPA values (Sol 5.6 32.6, Fable 5.1 26.1, Astra 6 18.1, Opus 5.5 15.1).
  • Within studies. The controlled matrix ranks the agents Fable 5.1, Astra 6, Opus 5.5, Sol 5.6 (p < 0.001), and the full ranking reproduces across random halves (Kendall τ 0.79, p = 0.001). The PPA track ranks them Fable 5.1, Sol 5.6, Astra 6, Opus 5.5 (p = 0.033; split-half τ 0.54, p = 0.019).
  • Precision. The split-half test assesses the ranking as a whole. Single pairwise differences are less certain: the four between the leading and trailing agents (11.9 to 16.0 points, unadjusted p 0.017 to 0.061) do not stay below 0.05 after a Holm adjustment, and the data do not fix the order of Fable 5.1 and Sol 5.6, or of Astra 6 and Opus 5.5. The correctness track alone detects a paired difference of about 11 points with 80% power, and its gaps among these agents are at most 6.5 points (p = 0.38).

W1d. Much of the paper reports reference designs. The PPA track evaluates agents: 192 attempts start from the correct reference and must raise throughput by at least 5% with at most 2% LUT growth (Sec. 2.4, Table 2). Reference-design results take one paragraph and Table 3 in Sec. 3.2, and one paragraph in Sec. 3.3 on the contract and evaluator for further decoders. The reference-design gallery was removed from the appendix.

AuthorsOpenReview ↗

Response to Reviewer Fwrc (2/3)

W2/Q1. How much RTL the agent writes. New Table 7 (App. A.1) gives the reference size, the RTL in the starting workspace, and the lines accepted submissions add and delete relative to their starting files (medians):

Task / startReference lines / modulesProvided lines / modulesAcceptedAddedDeleted
Nano T2 / interfaces6,843 / 34284 / 61036,15437
Nano T3 / interfaces6,843 / 34416 / 8916,63451
Qwen3.5-0.8B / scaffold18,742 / 8116,308 / 741011,382729
Qwen3.5-4B / scaffold29,615 / 10925,204 / 98742,0171,104
Gemma 4 E4B / interfaces31,884 / 121746 / 172534,30080
Gemma 4 E4B / scaffold31,884 / 12127,492 / 108542,3111,237.5
Llama 3.2 3B / interfaces26,713 / 98682 / 153127,62576
Llama 3.2 3B / scaffold26,713 / 9823,468 / 91622,177.51,262.5

A scaffold supplies 85 to 88% of the reference's lines. From interfaces the agent writes the decoder itself: on Gemma 4 E4B and Llama 3.2 3B the median accepted design adds more lines than the reference contains. The sizes come from the authors' records and are not recomputed by the released scripts.

W2/Q2. Who built the references, and the 0.8B match. Table 8 (App. A.1) records that three engineers built the five correctness-track references.

  • Qwen3.5-nano and Gemma 4 E4B: written by hand, then reviewed independently.
  • Qwen3.5-0.8B and Llama 3.2 3B: an agent drafted individual components, and the engineer reviewed, edited, and integrated them.
  • Qwen3.5-4B: derived from a submission that passed every gate. The engineer changed 70 lines: 47 in the reset sequencing (a fixed clearing order and an explicit reset of the valid count) and 23 in the parameter-loading wrapper (burst alignment and a bounds check on weight addresses). The arithmetic datapath and the commit protocol are unchanged. The submitted Figure 1 caption and Table 1 called reference designs "not agent submissions"; the revision removes that statement.

The build-session records name anthropic/claude-opus-4-20251020 as the agent for the 0.8B, Llama 3.2 3B, and 4B work. It is not one of the four evaluated agents, so no evaluated agent starts from its own code. The public and held-out suites were written from the integer contract before any reference RTL existed.

Every reference passed the same gates as an agent submission, and the tests compare each submission with the contract's oracle, so how a reference was built does not enter any acceptance decision. The reference RTL shows that each task can be completed, and it is the starting design of the PPA track.

The identical 0.8B results follow from the suites. The reference and the case-study trajectory both pass every case of the same suites (3,120 public and 232 held-out; Table 1), so any two accepted 0.8B designs report the same counts. The case study documents one accepted adaptation, and acceptance rates come from repeated attempts (App. B.7).

AuthorsOpenReview ↗

Response to Reviewer Fwrc (3/3)

W2. Where the hardware numbers come from.

  • References. The baseline and optimized schedules of both PPA references were placed and routed with three seeds, checked for timing at 126 MHz, and measured on an FPGA board with five repeats per seed (Table 3, App. E.2). Every baseline and every optimized 4B implementation closes at 126 MHz; the optimized 0.8B closes on one of three seeds (mean 125.3 MHz).
  • Agent designs. All 123 correct PPA designs were placed and routed with three seeds (369 runs; 117 designs complete all three). At the achieved clock, 70 of the 78 nominal improvements hold, 8 are lost, and 2 are gained (App. E.4, Fig. 20a). All 541 accepted correctness-track designs were synthesized and 96 (three per agent and task) placed and routed; 90 complete all three seeds, at a median routed throughput of 0.77 to 1.02 times the reference per task (App. D.3). Eight improved agent designs and three further references were measured on the board; board power differs from the post-route estimate by −0.62 to +3.47% (App. E.2).
  • Platform, power, and instrumentation (Sec. 2.4, App. E.1, E.2). Every implementation targets the AMD Alveo U250 (xcu250-figd2104-2L-e) at 126 MHz and was built with Vivado 2024.1, as recorded in the Vivado logs and report headers. Power covers the FPGA and attached DRAM and excludes the host; App. E.1 separates estimates, post-route reports, and board measurements. Board power is read from the U250's on-board sensors through XRT 2.16.204: the core (0.85 V), auxiliary (1.8 V), and DRAM (1.2 V) rails, sampled at 1 kHz in five 10 s windows per seed. The reported value is the total rail power, the quantity the post-route estimates predict; idle power is recorded and not subtracted. The memory-service model is cycle-relative: 48-cycle base latency, one cycle per 64-byte beat, bursts of 8 to 16 beats, at most one beat per cycle, and response delays uniform on 0 to 256 cycles under the memory-stall condition of the regrading and the controlled matrix (the held-out backpressure test and the local checks use 1 to 3 cycles). Utilization, timing, and power reports for every implementation and the board sample files are retained outside the supplementary material.

W2/Q3. Agent configurations and budgets.

  • All four agents run through one harness and receive the same task instances, starting files, instructions, tools, feedback, and caps (Sec. 3).
  • Caps per task are in Table 1 (from 5 h, 260 tool calls, and 140k output tokens on nano T2 to 30 h, 900 calls, and 600k tokens on Gemma 4 E4B from interfaces). Caps per study are in Table 9: earlier series 3 to 12 h, 180 to 420 calls, 90k to 260k tokens; correctness track 5 to 30 h, 260 to 900, 140k to 600k; PPA track 12 to 18 h, 420 to 520, 260k to 320k. The supplementary material lists the caps of all 266 agent, task, setting, and budget combinations.
  • Provider model identifiers of the benchmark runs (App. A.1): openai/gpt-5.6-sol-20261015 (Sol 5.6), openai/astra-6-20260930 (Astra 6), anthropic/claude-opus-5-5-20251101 (Opus 5.5), and anthropic/claude-fable-5-1-20261101 (Fable 5.1).
  • Sampling temperature, top-p, and provider-side reasoning compute were not controlled in the benchmark attempts (App. A.1). Temperature 0.2, top-p 0.95, reasoning disabled, and a 131,072-token context apply only to the constructed-decoder runs (App. B.6).
  • The submitted version's 45-attempt pilot used three other agent configurations. The revision replaces it with the four agents' attempts on the same three nano tasks (App. G).

W2. How diagnosis labels are assigned. App. B.7 ("Failure labels") gives the rule. A rejected submission is labeled by its first failing gate, in the order integrity, compilation, public suite, held-out suite, synthesis, local checks. Within the decoder evaluator's case checks, the label is the first failing check in a fixed order (timeouts, incomplete transactions, reset clearing, output stability under stalls, cache bounds, commit atomicity, state, logits, token). A label names the category of the first failing check; the rule does not verify the root cause. On the 36 snapshot–case records of the constructed decoder, two annotators agree on 28 (Cohen's κ = 0.46, 95% interval 0.21 to 0.68) and agree with the rule on 34 and 28. This audit covers the constructed decoder only. The 13 integration, 13 state/protocol, and 10 arithmetic diagnoses in Sec. 3 are the categories this rule assigns to the 36 rejected correctness-track designs on Gemma 4 E4B, Llama 3.2 3B, and Qwen3.5-4B that pass the public suite.

Presentation. Sections 1 and 2 are rewritten around one attempt (Fig. 1), with state word, transaction, accepted transfer, exactness, and fidelity defined where they first appear. T1 to T3 moved to App. G, and result tables report counts with intervals.

Reviewer_SonUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces AIChipBench, a benchmark designed to assess whether coding agents can use public verification feedback to build hardware implementations that pass independent acceptance as cross-module integration complexity and persistent-state requirements increase. Each task pairs an executable fixed-point contract with a public validation ladder, an isolated held-out evaluator, and tool-use accounting, so that model numerical fidelity, implementation correctness, and agent reliability can be measured separately. Experiments with three agent configurations show lower acceptance rates for cross-layer integration and recurrent decoding tasks.

Critical strengths
  1. The research question is practically relevant. The paper emphasizes that a correct current token does not establish that the state on which future computation depends is correct, and it treats state commits as observable acceptance targets. For decoders involving recurrent computation, caching, and external-memory interaction, this provides greater diagnostic value than reporting only compilation success rates or final-token match rates.

  2. The three-way conceptual separation is clear. The paper treats the accelerator as a state-transition system and distinguishes FP32 fidelity, RTL exactness, and agent reliability, thereby avoiding conflation of numerical approximation error with RTL implementation defects.

  3. The definition of transaction boundaries is reasonable. Implementations may pipeline and reorder independent work, provided that accepted transactions and architectural state commits agree with the reference model; the authors do not require cycle-by-cycle matching between the software reference and internal RTL signals.

  4. The evaluation framework yields several interesting findings. For example, success rates decline markedly when agents must handle inter-module connections and sustained execution requirements, and revealing the location of internal state errors may help improve success rates.

Critical weaknesses
  1. T1–T3 simultaneously vary the number of layers/execution steps, the integration boundary, the test sets, the implementation effort, and the budget. The drop in success rates therefore shows that these tasks are harder overall, but it does not decompose the sources of that difficulty. The authors appropriately limit the scope of their conclusions, but for a benchmark centered on state awareness, this decomposition is relatively important. From a reader's perspective, we would want to know how precisely the proposed evaluation reflects the sources of complexity in hardware design tasks, so that subsequent work can use it to improve agent systems.

  2. As a benchmark contribution, the number of evaluation cases is relatively small: the core test cases are three task templates built on Qwen3.5-nano. The reported 45/152 attempts include both the choice of agents and the number of repeated attempts; placing these attempts in the same comparison dimension as the actual test cases used by other benchmarks may not be appropriate. We are therefore fairly confident that the statistical confidence in the framework's results is lower than that of benchmarks with 100+ cases.

  3. The paper does not separately demonstrate what state checking detects beyond output-only checking. We would like to see a 2×2 table (output-only vs. output-plus-state) on the same frozen submissions, the same inputs, and the same simulation budget, plus a budget-matched longer output-only baseline, reporting the number of submissions that pass output checks but fail state checks, their fault types, and their first-detection steps, so that readers can directly see which errors state checking catches that output checking does not.

Questions for authors
  1. Could the authors add experiments or evaluation levels at the benchmark level that follow a controlled-variable design? Alternatively, could they explain how the current benchmark design embodies the principle of controlling variables and with what precision it reflects agents' shortcomings in hardware design capability?

  2. Could the authors expand the benchmark's test cases or provide a methodology for doing so, including but not limited to how to select the underlying models and the key considerations in the benchmark construction workflow?

  3. Could the authors provide additional experimental results or explanations corresponding to Weakness 3 discussed above?

5 comments
AuthorsOpenReview ↗

Response to Reviewer SonU (1/2)

Thank you for the review. Your three requests led to three additions: a controlled matrix, more tasks with a documented construction procedure, and the 2×2 analysis on frozen submissions.

W1/Q1. Controlled decomposition of difficulty.

As you note, T1 to T3 change several factors at once. They now form the earlier task series (App. G), described as a task-level ladder. Three studies in the revision each change one part of the setup and hold the rest fixed.

1. Controlled matrix (Sec. 4.4, App. D.5; 3,456 attempts). Every attempt starts from one Qwen3.5-nano scaffold and is graded by the same evaluator. The factors are the required horizon (1, 4, 8, or 16 steps), an added state condition (none, reset after activity, or bounded memory stalls), and a multiplier on all three caps (0.5×, 1×, 1.5×), with 24 matched blocks per agent and cell.

  • Horizon: at 1×, pooled acceptance falls from 252 to 115 of 288 between one and 16 steps. With no added condition every agent falls (Sol 5.6 21 to 8, Astra 6 23 to 15, Opus 5.5 21 to 10, Fable 5.1 23 to 17, of 24).
  • Budget: 1.5× lifts the 16-step result to 169 of 288, and acceptance still falls at each longer horizon. At 0.5×, 192 of 288 one-step and 286 of 288 sixteen-step attempts end at a cap. At one step, 1.5× does not raise acceptance (247 against 252).
  • Added conditions: stalls lower acceptance more than reset at most horizons. Of 144 per-agent intervals, 16 exclude zero, against about 7 expected under no effect, and 15 of the 16 are lower with the condition. The horizon-by-condition interaction at 1× is −15.6 points [−28.1, −2.1] for reset and −8.3 [−24.0, 7.3] for stalls, and its sign changes with the budget (Table 40, Fig. 16).

A longer horizon also brings more decoding work and larger suites (984 public and 68 held-out cases at one step, 1,344 and 128 at 16), so the matrix does not attribute the horizon effect to persistent state alone. Sec. 4.4 states this limit.

2. Starting workspace on the same target (Sec. 4.1). On Gemma 4 E4B and Llama 3.2 3B, 116 of 192 scaffold attempts are accepted, against 56 of 192 from interfaces. On the extension tasks the counts are 161 and 177 of 256 against 102 and 118. The two starts carry different caps, so these contrasts compare complete protocols.

3. Feedback content on the same task and instance (Sec. 4.2). Only the public feedback changes; item 3 of the general response gives the results.

Where attempts are lost. Of 864 correctness-track attempts, 163 end at a cap without submitting (117 on wall time, 29 on tool calls, 17 on output tokens; Sec. 4.4). Between 68 and 72% pass each suite and 60 to 67% pass every gate (Table 51). Another 55 designs pass every public test and fail final acceptance, 53 of them on the held-out suite (Sec. 4.3).

Precision. On the 216 shared correctness-track instances, the matched comparison detects a paired difference of about 11 points with 80% power; resolving 5 points would take about 1,000 instances (App. D.8). On the correctness track alone the four agents are not separated (p = 0.38). With the PPA track included, the overall score gives a ranking that reproduces in 179 of 200 random half-splits (App. D.9, Table 54).

AuthorsOpenReview ↗

Response to Reviewer SonU (2/2)

W2/Q2. Number of test cases, and a method for adding them.

We agree that attempts and tasks are different units. Table 73 counts AIChipBench in tasks (13) and gives the attempts separately, and Sec. 5 states that problem, design, example, task, and attempt counts are not comparable units. The 13 agent tasks are the eight correctness-track tasks on five targets from three families (Qwen3.5 with recurrent GDN and attention layers; attention-only Gemma 4 E4B and Llama 3.2 3B), four extension tasks (Gemma 3 4B and Llama 3.2-1B from interfaces and from a scaffold, 1,024 attempts), and T1, a single decoder layer from the earlier series. Each correctness-track task implements a complete decoder whose reference has 6,843 to 31,884 lines of RTL (Table 7), and each task instance fixes the initial RTL files, quantized parameters, public inputs, and held-out test assignment. Thirteen tasks are still far fewer than 156 or 783 problems, so we compare agents on matched instances within tasks and report where the data do not separate them.

Adding a target (App. C.4, new). A decoder becomes a task in four steps. (1) An integer contract fixes the quantized parameters, every rounding and saturation decision, and the state words checked after each step. (2) The contract must agree with FP32 on at least 99% of top-1 next-token decisions. Gemma 2 2B (98.38%) fails this; Gemma 4 E4B's QF contract (98.49%) also fails, so its tasks use the Main contract; Qwen3.5-nano (97.22%) is the stated exception. (3) A reference design must pass every gate: public suite; held-out suite with new weights, prefixes, cache positions, reset after activity, stalls, and parameter reload; synthesis; integrity; local checks. Qwen3.6-27B's reference fails eight public and ten held-out cases and is excluded. (4) The task fixes the starting workspace, horizon, and caps, and agents attempt matched instances. Gemma 3 4B and Llama 3.2-1B passed steps 1 to 3 and became the extension tasks.

W3/Q3. 2×2 on the same frozen submissions; a longer output-only baseline.

Same designs, same inputs (Table 43): 69 accepted submissions × 32 sequences = 2,208 sequences, each checked for both state and output.

StepsBoth exactState wrong onlyOutput wrong onlyBoth wrong
322,123171751
642,0331341121
1281,8681281247
2561,58412142470
  • Pass output, fail state. 17 sequences at the required 32 steps, in 5 designs that fail only in state (Table 44); 9 run under stalls and 5 under reset after activity. Longer runs add none (12 at 256 steps).
  • Defect categories and first detection. The regrader records the category of the first failing check. At 32 steps, 68 failing sequences (41 designs) first fail a state-transition check, at steps 9 to 32 (median 21), and 17 (14 designs) first fail an output-arithmetic check, at steps 10 to 31 (median 18). Finer defect types were not recorded. Among the 470 state-first sequences whose output also fails, the median lag is 4 steps (IQR 2 to 8).
  • Longer output-only checks (Table 45). At 32 steps the output check finds 68 of the 85 sequences; at 35 steps it finds 79, missing 10 and adding 4 others (difference 0.27 points [−0.05, 0.63]); from 38 steps it finds more than the 32-step full-state check; at 48 steps it finds all 85, among 117 in 53 designs.
  • During construction (Table 48). Under output-only feedback, 219 submissions pass every check the agent can see, 199 pass the full public suite with its state checks, and 144 are accepted.

Dear Reviewer SonU,

Thank you again for your constructive review. Your suggestions were particularly helpful in clarifying which experimental factors our benchmark can isolate.

We wanted to briefly highlight the two changes most directly addressing your concerns. First, the controlled matrix (Sec. 4.4, App. D.5) uses one starting implementation and evaluator while varying decoding horizon, state conditions, and resource budgets across 3,456 attempts. This lets us examine the effects of these requirements more systematically, while explicitly accounting for the additional work and test cases associated with longer horizons.

Second, the same-submission analysis (Sec. 4.3, Tables 43–45) shows that, at the required 32-step horizon, state checks identify 17 additional failing sequences across 5 designs. Longer output-only checks eventually detect these failures by 48 steps. This establishes earlier detection in decoding steps.

We would appreciate your thoughts on whether the revised analysis addresses your central concerns, and whether any specific clarification would still be helpful.

Reviewer_CxjqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AIChipBench, a benchmark for coding agents that generate RTL for AI decoders. It checks both numerical outputs and persistent state using a fixed integer reference, public debugging tests, and separate hidden tests after submission. The main tasks progress from a single layer to a four-layer decoder and eight-step decoding. In the 45-run pilot, final acceptance falls from 11/15 to 7/15 to 2/15 across these tasks. A further 152 attempts compare agents and feedback policies. Larger-model tasks test adaptation of a shared RTL scaffold. The paper also reports throughput gains from paired reference designs with the same arithmetic.

Critical strengths
  1. The benchmark checks persistent decoder state as well as numerical outputs. This is valuable because a state error may not affect the current output but can cause failures in later decoding steps. The fixed integer reference also helps distinguish numerical approximation errors from RTL bugs. The three tasks increase in complexity from a single layer to a four-layer decoder and repeated decoding.

  2. The experiments include meaningful controls. Submissions are frozen before hidden evaluation, and both feedback conditions use the same final checks. The output-only baseline already compares complete logits, making the state-feedback comparison more informative. Construction costs include failed and timed-out attempts, which better reflects the effort needed to obtain a correct implementation.

Critical weaknesses
  1. The paper could better distinguish AIChipBench from existing benchmarks. CVDP already evaluates iterative RTL development and verification, while ChipBench includes hierarchical designs and state-machine bugs. The main difference appears to be the explicit checking of decoder state, but the paper does not clearly demonstrate when this is necessary. For example, the delayed-error case in Appendix I uses a four-word state buffer rather than an actual decoder. A real decoder failure where state checks detect an error before output checks would make this contribution more convincing.

  2. The results give limited insight into how agents use feedback during debugging. Lower acceptance on T2 and T3 shows that these tasks are harder, but does not explain whether agents struggle to build the initial design, find the cause of a reported error, or preserve working behavior while making changes. The failure labels and tool counts provide some information, but it remains unclear what agents actually changed after receiving feedback and why those changes succeeded or failed.

  3. The improvement from state feedback is not yet convincing. Acceptance increases from 8/16 to 10/16 on T2 and from 3/16 to 5/16 on T3, but both paired tests give (p=0.625) in Table 14. With only 16 paired attempts per task, it is difficult to tell whether these gains are consistent. The reported reduction in hours per accepted T3 submission is also based on just three versus five successful runs, so the cost comparison may be sensitive to a few individual attempts.

Questions for authors
  1. Could you provide a decoder example where state checks detect an error before the full logits show a mismatch? It would be useful to see how the earlier state check helps locate the bug.

  2. Could you show how one T3 run progressed through debugging, including the feedback received, the code changes made, and the subsequent test results?

  3. Could you report confidence intervals for the acceptance gains in Table 5B and check how much the cost per accepted submission changes when individual runs are excluded?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Cxjq (1/2)

Thank you for the careful review. You asked for a real decoder failure that state checks catch before the outputs, for evidence of how agents use feedback, and for intervals on the feedback effect. The revision adds all three.

W1/Q1. A real decoder failure that state checks catch before the outputs; distinction from CVDP and ChipBench.

Real agent submissions. We regraded 144 frozen submissions on Qwen3.5-0.8B, Gemma 4 E4B, and Llama 3.2 3B, 69 of them accepted, together with the three reference designs. Each design decodes eight new prefixes under four conditions (new prefix alone, nonzero cache positions, reset after activity, bounded memory stalls) for up to 256 steps (Sec. 4.3, App. D.6, Fig. 9b–c). The references stay exact. For the 69 accepted submissions:

  • 482 of the 624 sequences that fail within 256 steps first diverge in persistent state. In 470 of them the output also fails, a median of 4 decoding steps later (IQR 2 to 8, maximum 40); in the other 12 the output stays exact through step 256 (Fig. 18).
  • At the 32 steps the tasks required, 85 sequences fail: 51 in both state and output, 17 in state only, and 17 in output only (Table 43). An output check of that length passes the 17 state-only sequences; 9 of them run under memory stalls and 5 under reset after activity.
  • An output-only check run for 35 steps still misses 10 of the 85, and it needs 48 steps to find all of them (Table 45).

In these submissions the state check finds the defects at a shorter horizon. An output check run 50% longer finds the same 85 sequences, so the data support earlier detection; they do not show defects that only state checks can find. We revised the paper's motivation to say exactly this. The selection of the regrading pool was not recorded, so the counts show that accepted designs can fail and do not estimate how often (Sec. 4.3).

A worked decoder example. App. B.5 replaces the four-word buffer with a constructed four-token attention decoder with a key–value cache, valid/ready handshakes, and a memory interface (143 lines of RTL), simulated in Icarus Verilog. A seeded defect commits the cache entry before the output transfer is accepted. The valid count is first wrong at transaction 2, the logits at transaction 5, and the selected token at transaction 6 (Fig. 7). A scripted repair that guards only the cache payload makes every public logit and token exact, while the state and commit checks still fail and the held-out suite fails 4 of 5 cases (Table 15).

The regrading records give, for each sequence, the step of the first state mismatch and of the first output mismatch. They do not name the state word, which is why the word-level example uses the constructed decoder.

Distinction. In CVDP's agentic tasks an agent works in a repository before hidden test harnesses grade the result, and ChipBench checks hierarchical and CPU-component designs against golden references on random stimuli. AIChipBench grades budgeted agent attempts against an executable integer contract that specifies the stored state at every decoding step, with held-out reset-after-activity, nonzero-cache-position, and memory-stall conditions (Sec. 5). The 17 state-only sequences above measure what an output-only grader of the same horizon misses.

AuthorsOpenReview ↗

Response to Reviewer Cxjq (2/2)

W2/Q2. How agents use feedback, and a debugging trajectory.

  • Discovery and repair, measured separately (Sec. 4.2, Table 34). Among feedback-study attempts with a state defect, agents find it in 142 of 169 under coarse-state feedback (median 1.50 agent-hours) and in 151 of 166 under full-state feedback (1.35 h). They repair it in 87 of 169 (median 3.22 h) and in 123 of 166 (2.69 h). Discovery is similar under the two settings, so locating the mismatch helps mainly at the repair stage.
  • How the remaining attempts end (App. D.7). From output-only to full-state feedback, state/protocol rejections fall from 81 to 17 of 384 attempts. Under full-state feedback, caps end 43 of the 89 failed attempts: 26 had no state defect, 13 had already repaired one, and 4 reached the cap before the repair was complete.
  • Keeping working behavior. Every PPA attempt starts from a correct design, and 69 of 192 end without one: 24 submit an incorrect design and 45 reach a cap (Sec. 4.5).
  • A paired trace (App. B.6, Table 17). Sol 5.6 repaired the constructed decoder once under each setting, starting from the same RTL. Full-state feedback reports "transaction 2: valid count 4, expected 2"; output-only feedback reports the logits of transaction 5. In both runs the single edit changes the commit condition from output valid to output valid and ready; the public suite then passes 4/4 and the frozen RTL passes 5/5 held-out cases. Full-state feedback points to the premature commit three transactions earlier. One run per setting illustrates the mechanism and does not estimate a rate.
  • Benchmark records. Fig. 8 gives event records of three benchmark attempts, including a T3 full-state attempt that finds its state defect at 2.86 h (78 tool calls), repairs it at 4.25 h (116 calls), and stops at the 6-hour wall cap. Listing 2 gives an output-only T3 attempt that passes all 1,384 public checks visible to it and fails the state-aware recheck (1,378/1,384).

The released benchmark records do not keep per-edit RTL diffs, so the trace that shows each edit uses the constructed decoder.

W3/Q3. Size of the feedback effect, intervals, and cost sensitivity.

We agree that the 16-pair comparison (p = 0.625) was too small. It is now reported as a pilot in App. G, described as too small to resolve the effect, and its cost table was removed. The new feedback study has four agents × three tasks (nano T2, nano T3, Qwen3.5-0.8B) × 32 instances × three settings, 1,152 attempts (Sec. 4.2, App. D.4).

  • Acceptance. Pooled, 144, 208, and 295 of 384 attempts are accepted under output-only, coarse-state, and full-state feedback. Full-state exceeds output-only in all 12 agent–task cells, by 25.0 to 46.9 points, and every block-bootstrap interval excludes zero under each of 100 bootstrap seeds (Table 31). The increments are less precise: coarse-state against output-only intervals include zero in 10 of 12 cells, and full-state against coarse-state intervals exclude zero in 7 of 12 (Table 32).
  • Cost per accepted design. 13.38 [11.25, 16.19], 9.33 [8.16, 10.77], and 6.55 [5.92, 7.31] agent-hours, with a full-state to output-only ratio of 0.49 [0.42, 0.56] (Table 35). Every per-cell full-state ratio has an interval below one, from 0.29 [0.12, 0.50] (Sol 5.6, T3) to 0.65 [0.44, 0.90] (Opus 5.5, T2) (Table 36).
  • Excluding runs. Removing one matched block at a time (the 12 attempts on one instance), over all 96 deletions, leaves the pooled ratio between 0.48 and 0.50, and no deletion raises an agent–task ratio above 0.71 (Fig. 14). Removing one attempt at a time (1,152 deletions) leaves it between 0.486 and 0.492.

Dear Reviewer Cxjq,

Thank you again for your thoughtful review and for the specific suggestions about state-feedback effectiveness and debugging behavior.

We wanted to briefly follow up on the two results most directly related to your concerns. First, the expanded matched feedback study (Sec. 4.2, App. D.4) now shows a 25.0–46.9 percentage-point improvement over output-only feedback across all 12 agent–task settings, with every reported block-bootstrap confidence interval excluding zero. Agent-hours per accepted design also fall from 13.4 to 6.5, and the leave-one-block-out analysis shows that this result is stable.

Second, the real-submission regrading (Sec. 4.3) measures delayed state failures directly, while the constructed decoder and paired trace (App. B.5–B.6) illustrate how first-mismatch feedback helps locate the error.

We would be grateful to hear whether these additions address your main concerns, or if there is a specific remaining issue we could clarify during the discussion period.

Dear Reviewer Cxjq,

Thank you again for your time and thoughtful feedback. We would like to follow up on our response and ask whether you have any remaining questions or concerns. We welcome further discussion and would be glad to provide any additional clarification.

If our response and additional results have addressed your concerns, we would be grateful if you could consider a more favorable score in light of this evidence.

Thank you for your consideration.

Best regards,

The authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers (1/2)

We thank the three reviewers for careful and constructive reviews. The reviews evaluated the submitted version, which had a 45-attempt pilot with three agent configurations, 152 further attempts, and three Qwen3.5-nano tasks. Three concerns came up in more than one review: the benchmark was small, the claim about stored state rested on a constructed example, and the paper lacked the records needed to interpret its numbers. The revision answers each with new experiments and records. Numbers below refer to the revised PDF.

1. Benchmark scope (SonU W2, Fwrc W1). The correctness track has eight tasks on five decoder targets from three model families: Qwen3.5-nano T2 and T3, Qwen3.5-0.8B and 4B, and Gemma 4 E4B and Llama 3.2 3B from module interfaces and from a working scaffold (Table 1). Four extension tasks on Gemma 3 4B and Llama 3.2-1B (App. D.2) and T1 of the earlier series bring the total to 13 agent tasks, on which four coding agents make 6,768 attempts. Table 73 counts tasks and attempts in separate columns, and App. C.4 describes how a new target becomes a task.

2. Stored state in real submissions (Cxjq W1/Q1, SonU W3, Fwrc W1). We regraded 144 frozen agent submissions, 69 of them accepted, and the three references on 32 new sequences each for up to 256 steps (Sec. 4.3, App. D.6). In the accepted submissions, 482 of the 624 failing sequences first diverge in stored state. In 470 of these the output fails a median of 4 steps later (IQR 2 to 8, maximum 40); the other 12 show no output mismatch by step 256. At the 32 steps the tasks require, 17 of the 85 failing sequences are wrong in state only, so an output check of that length passes them. An output-only check needs 48 steps to find all 85, and the paper now states the motivation as earlier detection of the same defects.

3. Feedback study at scale (Cxjq W3/Q3, Fwrc W1). The new study has 1,152 attempts: four agents, three tasks, 32 instances, and three feedback settings. Full-state feedback beats output-only feedback in all 12 agent–task cells, by 25.0 to 46.9 points, and every block-bootstrap interval excludes zero. Agent-hours per accepted design fall from 13.4 [11.3, 16.2] to 6.5 [5.9, 7.3], a ratio of 0.49 [0.42, 0.56]. Deleting any one matched block leaves the pooled ratio between 0.48 and 0.50 (Sec. 4.2, App. D.4).

4. Controlled sources of difficulty (SonU W1/Q1). A matrix of 3,456 attempts starts from one nano scaffold, uses one evaluator, and varies the required horizon (1 to 16 steps), an added state condition (none, reset after activity, bounded memory stalls), and the budget (0.5×, 1×, 1.5×). At the 1× budget, pooled acceptance falls from 252 to 115 of 288 between one and 16 steps (Sec. 4.4, App. D.5).

5. Agents and hardware quality (Fwrc W1/W2). In the new PPA track, each of 192 attempts starts from a correct reference design and must raise throughput by at least 5% with at most 2% LUT growth while staying correct (Sec. 2.4, 3.2). All 123 correct PPA designs were placed and routed with three seeds, and 70 of the 78 nominal improvements hold at the achieved clock.

6. Records for interpretation (Fwrc W2). The paper now reports the RTL size of references, starting workspaces, and accepted submissions (Table 7); how each reference was built (Table 8); caps per study (Table 9); provider model identifiers (App. A.1); the failure-label rule (App. B.7); and a constructed attention decoder with a cache, handshakes, and a memory interface, together with a paired agent debugging trace (App. B.5, B.6).

7. A reproducible agent ranking (Fwrc W1). The overall score, the product of the correctness and PPA track scores, ranks Fable 5.1 (32.4) and Sol 5.6 (31.3) above Astra 6 (19.4) and Opus 5.5 (16.3). We tested whether this ranking reproduces by splitting the matched instances of every task into random halves: both halves give it in 179 of 200 splits, against 17% when the agents' outcomes are permuted within instances (p = 0.003; Sec. 3, App. D.9, Table 54). The controlled matrix and the PPA track also give rankings that reproduce across halves (App. D.8).

Figure 3 summarizes the new evidence, and Appendix Figures 9 and 20 give the full panels.

AuthorsOpenReview ↗

General response to all reviewers (2/2): what changed since submission

The revision is larger than the submitted version, so we state what is new, when it was obtained, and how the records were handled.

Timeline. The submitted version reported three sets of results: a 45-attempt pilot with three agent configurations on Qwen3.5-nano T1 to T3, the Sol 5.6 and Astra 6 cohort of the earlier task series (152 attempts on the same nano tasks and on Qwen3.5-0.8B scaffold adaptation, now App. G), and paired reference designs at 126 MHz. Everything else was completed after the deadline: the Opus 5.5 and Fable 5.1 cohort of the earlier series, the correctness track, the feedback study, the controlled matrix, the PPA track, the cross-architecture and repeated-instance studies, and the extension tasks, which bring the total to 6,768 attempts (Table 5); the three-seed place-and-route runs and the board power measurements; and, in early November, an independent re-review and freeze of the reference designs for the supplementary material. The revision replaces the 45-attempt pilot: the four agents repeat its three nano tasks in the earlier series, with 16 attempts per agent and task (App. G), and two of them in the correctness track.

How the new runs were done. Every study uses the harness, evaluator, and gates described in Sec. 2, and the Opus 5.5 and Fable 5.1 cohort uses the caps, test counts, and feedback of the submitted cohort. Within a study, the four agents attempt the same task instances. Every attempt is counted, including those that end at a cap, and held-out results never return to the agent. The released attempt- and sequence-level records, with their scripts, recompute the reported counts, intervals, and tests (Reproducibility Statement).

Records created or corrected after submission.

  • the reference review dates in the records are the dates of the re-review and freeze for the artifact;
  • the provider model identifiers in the configuration records were corrected to those of the benchmark runs;
  • the Vivado version, board instrumentation, and memory-service parameters were recovered from the build logs, report headers, and evaluator configuration;
  • the origin of each reference design, including the 4B reference, is documented in Table 8 and App. A.1.

One statement in the submitted paper was wrong. The Figure 1 caption and Table 1 called the reference designs "not agent submissions", but the 4B reference derives from a submission by a model outside the four evaluated agents; the revision removes the statement and documents the origin. None of these corrections changes a reported count or test.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.