- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces AIChipBench, which evaluates whether coding agents can implement the fixed-point inference of a language model as synthesizable RTL. Each task is defined by a bit-exact integer reference that specifies the outputs and the persistent state after every decoding step. During construction, agents can run public tests that report the first mismatch; after the submission is frozen, an isolated evaluator runs held-out tests, synthesis, and integrity checks. The main experiment uses Qwen3.5-nano, a four-layer model built by the authors, with three tasks: a single layer, a four-layer one-step decoder, and eight-step recurrent decoding. Three agent configurations make five attempts on each task, and 11/15, 7/15, and 2/15 attempts are accepted. The paper also reports a comparison of two agents, a state-feedback ablation, adaptation case studies on Qwen3.5-0.8B and 4B, and the throughput and energy efficiency of the authors' reference designs.
Critical strengths
- S1. A realistic problem that is harder than existing RTL benchmarks. Existing benchmarks mostly test small, independent modules and grade them by their outputs. This paper asks agents to build a complete, stateful decoder accelerator, whose correctness depends on state carried across modules and decoding steps, and in which an error may surface several steps after it occurs. A single attempt typically runs for hours (a median of 1.8 to 3.6 hours on the three nano tasks), and T1 to T3 add cross-layer integration and cross-step state one level at a time, which shows at which level agents start to fail.
- S2. An evaluation designed around this difficulty. At every transaction boundary, the RTL's outputs and state are compared with a bit-exact integer reference, so a faulty state update is caught at the step where it occurs, before it reaches the output. In the output-only feedback condition, 6 submissions with incorrect state pass the output-only public checks, which shows what grading by outputs alone would miss. Fidelity (fixed point vs. FP32) and correctness (RTL vs. fixed point) are measured separately, so agent failures are not confused with quantization error.
Critical weaknesses
W1. The benchmark has too few tasks and attempts to support its conclusions, and its main result does not address its own motivation.
- One decoder specification. T1, T2, and T3 are nested slices of a single design (one layer, the four-layer decoder for one step, and the same decoder over eight steps), and the 16 "task instances" per level share the same task definition (Appendix L). The 0.8B and 4B tasks scale up the same architecture from an existing scaffold, with a single trajectory for 4B. Table 3 places "45; 152 attempts" next to 156 problems for VerilogEval and 783 for CVDP, but these are attempt counts from two experiments; counted as tasks, AIChipBench has 3 nano tasks and 2 adaptation tasks.
- The main result does not test the motivation. The paper is motivated by state errors that corrupt later tokens while the current output is still correct. Its main result, acceptance falling from 11/15 to 7/15 to 2/15, is what any set of increasingly difficult tasks would show, and only the endpoints differ significantly (Fisher's exact test: p = 0.26 for T1 vs. T2, p = 0.11 for T2 vs. T3). The results that bear on the motivation are small: 6 submissions pass output-only checks but fail state checks, and state feedback adds 2 accepted attempts out of 16 (p = 0.625). The paper does not measure how many steps separate a faulty state update from the first output error in real submissions, or whether state feedback helps agents locate faults. The agent comparison cannot rank agents either (McNemar p ≥ 0.625).
- Much of the paper does not evaluate agents. The abstract, Block B of Table 1, Section 6, and most of Appendices J and K report on the authors' reference designs.
W2. The paper omits information needed to interpret and reproduce its results.
- How much the agent writes. The paper does not report the size of the reference implementations or how much RTL the starting files and scaffolds already provide. Who built the reference designs is also unclear: Figure 1 says they are not agent submissions, yet the 0.8B reference design has exactly the same test results as the agent trajectory (Table 2, Listing 2).
- Where the hardware numbers come from. All hardware results come only from simulation and synthesis. The memory model, the source of the power figures, and the FPGA part are not specified, and hardware data for agent designs are reported only for accepted 0.8B submissions.
- How the agents are configured. Appendix G lists reasoning and sampling settings, prompts, and the context-management policy as required parts of the run specification, but the paper reports none of them for any agent, and per-attempt tool-call and token budgets are given only for the agent comparison (Table 13). How the diagnosis labels were assigned is not described.
Without this information, readers cannot judge how difficult the tasks are or reproduce the reported results.
Questions for authors
- Implementation size. How many lines of RTL do the reference implementations contain, how much of this is provided by the nano starting files or the 0.8B and 4B scaffolds, and how much must the agent write itself?
- Reference designs. Who built the 0.8B and 4B reference designs, and how do they relate to the agent adaptation trajectories? Figure 1 states that reference designs are not agent submissions, but the 0.8B reference results in Table 2 match those of the agent trajectory in Listing 2.
- Agent configurations. What reasoning and sampling settings, prompts, and context-management policies were used for each agent configuration (the three systems in the main experiment, Sol 5.6, and Astra 6)? What tool-call and token budgets applied to each attempt in the main experiment?
5 comments
Response to Reviewer Fwrc (2/3)
W2/Q1. How much RTL the agent writes. New Table 7 (App. A.1) gives the reference size, the RTL in the starting workspace, and the lines accepted submissions add and delete relative to their starting files (medians):
| Task / start | Reference lines / modules | Provided lines / modules | Accepted | Added | Deleted |
|---|---|---|---|---|---|
| Nano T2 / interfaces | 6,843 / 34 | 284 / 6 | 103 | 6,154 | 37 |
| Nano T3 / interfaces | 6,843 / 34 | 416 / 8 | 91 | 6,634 | 51 |
| Qwen3.5-0.8B / scaffold | 18,742 / 81 | 16,308 / 74 | 101 | 1,382 | 729 |
| Qwen3.5-4B / scaffold | 29,615 / 109 | 25,204 / 98 | 74 | 2,017 | 1,104 |
| Gemma 4 E4B / interfaces | 31,884 / 121 | 746 / 17 | 25 | 34,300 | 80 |
| Gemma 4 E4B / scaffold | 31,884 / 121 | 27,492 / 108 | 54 | 2,311 | 1,237.5 |
| Llama 3.2 3B / interfaces | 26,713 / 98 | 682 / 15 | 31 | 27,625 | 76 |
| Llama 3.2 3B / scaffold | 26,713 / 98 | 23,468 / 91 | 62 | 2,177.5 | 1,262.5 |
A scaffold supplies 85 to 88% of the reference's lines. From interfaces the agent writes the decoder itself: on Gemma 4 E4B and Llama 3.2 3B the median accepted design adds more lines than the reference contains. The sizes come from the authors' records and are not recomputed by the released scripts.
W2/Q2. Who built the references, and the 0.8B match. Table 8 (App. A.1) records that three engineers built the five correctness-track references.
- Qwen3.5-nano and Gemma 4 E4B: written by hand, then reviewed independently.
- Qwen3.5-0.8B and Llama 3.2 3B: an agent drafted individual components, and the engineer reviewed, edited, and integrated them.
- Qwen3.5-4B: derived from a submission that passed every gate. The engineer changed 70 lines: 47 in the reset sequencing (a fixed clearing order and an explicit reset of the valid count) and 23 in the parameter-loading wrapper (burst alignment and a bounds check on weight addresses). The arithmetic datapath and the commit protocol are unchanged. The submitted Figure 1 caption and Table 1 called reference designs "not agent submissions"; the revision removes that statement.
The build-session records name anthropic/claude-opus-4-20251020 as the agent for the 0.8B, Llama 3.2 3B, and 4B work. It is not one of the four evaluated agents, so no evaluated agent starts from its own code. The public and held-out suites were written from the integer contract before any reference RTL existed.
Every reference passed the same gates as an agent submission, and the tests compare each submission with the contract's oracle, so how a reference was built does not enter any acceptance decision. The reference RTL shows that each task can be completed, and it is the starting design of the PPA track.
The identical 0.8B results follow from the suites. The reference and the case-study trajectory both pass every case of the same suites (3,120 public and 232 held-out; Table 1), so any two accepted 0.8B designs report the same counts. The case study documents one accepted adaptation, and acceptance rates come from repeated attempts (App. B.7).
Response to Reviewer Fwrc (3/3)
W2. Where the hardware numbers come from.
- References. The baseline and optimized schedules of both PPA references were placed and routed with three seeds, checked for timing at 126 MHz, and measured on an FPGA board with five repeats per seed (Table 3, App. E.2). Every baseline and every optimized 4B implementation closes at 126 MHz; the optimized 0.8B closes on one of three seeds (mean 125.3 MHz).
- Agent designs. All 123 correct PPA designs were placed and routed with three seeds (369 runs; 117 designs complete all three). At the achieved clock, 70 of the 78 nominal improvements hold, 8 are lost, and 2 are gained (App. E.4, Fig. 20a). All 541 accepted correctness-track designs were synthesized and 96 (three per agent and task) placed and routed; 90 complete all three seeds, at a median routed throughput of 0.77 to 1.02 times the reference per task (App. D.3). Eight improved agent designs and three further references were measured on the board; board power differs from the post-route estimate by −0.62 to +3.47% (App. E.2).
- Platform, power, and instrumentation (Sec. 2.4, App. E.1, E.2). Every implementation targets the AMD Alveo U250 (xcu250-figd2104-2L-e) at 126 MHz and was built with Vivado 2024.1, as recorded in the Vivado logs and report headers. Power covers the FPGA and attached DRAM and excludes the host; App. E.1 separates estimates, post-route reports, and board measurements. Board power is read from the U250's on-board sensors through XRT 2.16.204: the core (0.85 V), auxiliary (1.8 V), and DRAM (1.2 V) rails, sampled at 1 kHz in five 10 s windows per seed. The reported value is the total rail power, the quantity the post-route estimates predict; idle power is recorded and not subtracted. The memory-service model is cycle-relative: 48-cycle base latency, one cycle per 64-byte beat, bursts of 8 to 16 beats, at most one beat per cycle, and response delays uniform on 0 to 256 cycles under the memory-stall condition of the regrading and the controlled matrix (the held-out backpressure test and the local checks use 1 to 3 cycles). Utilization, timing, and power reports for every implementation and the board sample files are retained outside the supplementary material.
W2/Q3. Agent configurations and budgets.
- All four agents run through one harness and receive the same task instances, starting files, instructions, tools, feedback, and caps (Sec. 3).
- Caps per task are in Table 1 (from 5 h, 260 tool calls, and 140k output tokens on nano T2 to 30 h, 900 calls, and 600k tokens on Gemma 4 E4B from interfaces). Caps per study are in Table 9: earlier series 3 to 12 h, 180 to 420 calls, 90k to 260k tokens; correctness track 5 to 30 h, 260 to 900, 140k to 600k; PPA track 12 to 18 h, 420 to 520, 260k to 320k. The supplementary material lists the caps of all 266 agent, task, setting, and budget combinations.
- Provider model identifiers of the benchmark runs (App. A.1):
openai/gpt-5.6-sol-20261015(Sol 5.6),openai/astra-6-20260930(Astra 6),anthropic/claude-opus-5-5-20251101(Opus 5.5), andanthropic/claude-fable-5-1-20261101(Fable 5.1). - Sampling temperature, top-p, and provider-side reasoning compute were not controlled in the benchmark attempts (App. A.1). Temperature 0.2, top-p 0.95, reasoning disabled, and a 131,072-token context apply only to the constructed-decoder runs (App. B.6).
- The submitted version's 45-attempt pilot used three other agent configurations. The revision replaces it with the four agents' attempts on the same three nano tasks (App. G).
W2. How diagnosis labels are assigned. App. B.7 ("Failure labels") gives the rule. A rejected submission is labeled by its first failing gate, in the order integrity, compilation, public suite, held-out suite, synthesis, local checks. Within the decoder evaluator's case checks, the label is the first failing check in a fixed order (timeouts, incomplete transactions, reset clearing, output stability under stalls, cache bounds, commit atomicity, state, logits, token). A label names the category of the first failing check; the rule does not verify the root cause. On the 36 snapshot–case records of the constructed decoder, two annotators agree on 28 (Cohen's κ = 0.46, 95% interval 0.21 to 0.68) and agree with the rule on 34 and 28. This audit covers the constructed decoder only. The 13 integration, 13 state/protocol, and 10 arithmetic diagnoses in Sec. 3 are the categories this rule assigns to the 36 rejected correctness-track designs on Gemma 4 E4B, Llama 3.2 3B, and Qwen3.5-4B that pass the public suite.
Presentation. Sections 1 and 2 are rewritten around one attempt (Fig. 1), with state word, transaction, accepted transfer, exactness, and fidelity defined where they first appear. T1 to T3 moved to App. G, and result tables report counts with intervals.
Response to Reviewer Fwrc (1/3)
Thank you for the detailed review. Many of the revision's additions respond directly to the gaps you identified; we answer each point in turn.
W1a. Too few tasks; T1 to T3 are nested; Table 2 compares attempts with problems.
We acknowledge that 13 tasks are few next to suites of hundreds of problems. Each task is large, however: a correctness-track task implements a complete decoder whose reference has 6,843 to 31,884 lines of RTL (Table 7); T1 is a single decoder layer.
W1b. The main result does not test the motivation.
We agree that a decline from T1 to T3 is what any ladder of harder tasks produces, and that only its endpoints differ significantly. The revision tests the motivation directly.
W1c. The agents cannot be ranked. The revision ranks them and shows that the ranking reproduces (Sec. 3, App. D.8, D.9, Table 54).
W1d. Much of the paper reports reference designs. The PPA track evaluates agents: 192 attempts start from the correct reference and must raise throughput by at least 5% with at most 2% LUT growth (Sec. 2.4, Table 2). Reference-design results take one paragraph and Table 3 in Sec. 3.2, and one paragraph in Sec. 3.3 on the contract and evaluator for further decoders. The reference-design gallery was removed from the appendix.