acceptodds
Under review as a conference paper at ICLR 2027

ChipReasoning: Benchmarking Evidence Discipline in Agentic Chip Implementation

Abstract

Engineering decisions in chip implementation depend on tool reports. Report selection is therefore part of the task: an agent must inspect the right evidence and keep its claims within the scope of that evidence. We introduce ChipReasoning, a benchmark comprising 170 cases organized into 85 counterfactual pairs across six chip-engineering domains. Each case provides six labeled but initially hidden report packets under a fixed inspection budget. The two cases in each pair share the same request and visible labels, but differ in one decision-critical packet. We evaluate both the inspection record and the submitted engineering disposition using a seven-component Process score. Significantly, across 11 configurations, Process scores range from 42.22 to 71.60, with 10 of 11 configurations outperforming the strongest deterministic, content-blind inspection baseline, which scores 50.41. While the leading configurations successfully inspect the changed packet in more than 97% of pairs, the highest both-correct pair rate remains only 45.9%. These results strongly emphasize that the remaining errors are concentrated in translating retrieved evidence into the final engineering commitment.

Primary area:
datasets and benchmarks
Keywords:
Chip, benchmark, agent

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_FFrhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ChipReasoning, a benchmark evaluating how LLMs select reports under a limited inspection budget and make engineering decisions from RTL, synthesis, and verification evidence. It addresses a meaningful problem. However, the authors need to explain how the reports and reference answers were validated and how the tasks reflect real workflows. Differences between CLI/API setups also raise concerns about comparison fairness.

Critical strengths
  1. The paper addresses a real problem in chip design: deciding when to proceed to the next stage often requires engineering experience or hands-on guidance.

  2. This engineering judgment is valuable in its own right, making it a worthwhile capability to evaluate in LLMs.

Critical weaknesses
  1. Unclear benchmark reliability. The reports and reference answers are constructed using rules, but how the final cases were independently validated is insufficiently explained. The authors should provide a clear validation procedure, using representative runs in established (commercial or open-source) EDA tools and expert checks of the reference decisions.

  2. Unclear practical relevance. The tasks mainly involve RTL, synthesis, and verification reports. Their connection to real chip-design workflows and broader implementation stages needs clarification.

  3. Uneven execution conditions. Models use different CLI/API setups, with differences in history handling and token limits that may affect the fairness of comparisons. Please report the actual token usage, interaction counts, and retry counts for each model so that comparability can be assessed.

Questions for authors
  1. How were the constructed reports and reference decisions validated? Please describe the procedure and provide representative EDA-tool results and expert checks supporting their correctness.

  2. Which real chip-design stages and decisions do the tasks represent? Why does the limited report-inspection budget reflect practical engineering constraints?

  3. How were execution conditions controlled across CLI/API setups? Please report actual token usage, interaction counts, and retry counts for each model, and explain how setup differences affect comparability.

Reviewer_fEuEOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

CHIPREASONING evaluates whether model-agent systems can select and use EDA report evidence to reach chip implementation decisions. It contains 170 cases in 85 counterfactual pairs across six engineering domains, each with a request, six hidden evidence packets, an inspection budget, and a five-way disposition space, built from four open RTL designs and six report classes. It introduces a Process score over the action log and a content-blind baseline, and evaluates 11 model configurations.

Critical strengths

A clean separation of evidence access from evidence use. The paper shows that systems frequently inspect the decision-critical packet yet still reach the wrong disposition, a failure mode that final-answer-only evaluation would hide.

Critical weaknesses

The motivation is not clear: the paper never establishes why this particular decision slice needs a benchmark. The stated target is "chip implementation decision," yet the benchmark removes tool execution and downstream consequences and asks only for a disposition label over frozen report fields; §5.4 concedes that dispositions follow "fixed predicates" over slack and constraint status. A real post-synthesis decision is a multi-round process anchored by observable consequences, so the paper must argue that reading frozen reports is a meaningful proxy for engineering judgment—and that the capability is not already covered by existing report-comprehension, verification, or EDA-execution benchmarks. Neither argument is made, and since reproducible tool execution is feasible here (open-source OpenSTA/OpenROAD with pinned libraries and constraints), the choice to freeze reports is a design decision whose cost and justification are left unstated.

The benchmark scale is too small and homogeneous to support its claims. All cases come from four open RTL designs sharing one report schema and one annotation rule set, under a single one-shot inspect-then-submit protocol. With 85 pairs and a manually reviewed case set, this is closer to a diagnostic probe than a benchmark: the headline result that most systems beat the content-blind baseline may be a schema-level rather than capability-level effect, and with no held-out schema, additional design, or protocol variation, the external validity of the ranking is untested.

Questions for authors

N.A

Reviewer_dSfqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ChipReasoning asks an agent to open evidence packets about a chip design under a budget and then commit to an engineering decision. Cases come in pairs that differ in a single packet, and a Process score grades both what was opened and what was decided. Across eleven model configurations, the strongest ones usually open the decisive packet but often decide wrong anyway, so the paper concludes that the main errors come after evidence access, in turning measurements into a correct engineering commitment.

Critical strengths
  • The core idea is worth something. Grading what the agent opened along with what it decided, on pairs that differ in one packet, changes what an agent benchmark asks instead of adding one more knob, and other hidden-evidence benchmarks could adopt it.
  • The sanity checks do their job. A fixed policy that opens the decisive packet but ignores its contents solves no pair, and the gap between Process and the answer-only score catches models citing packets they did not open. That last finding stands whether or not the gold labels are correct.
Critical weaknesses
  • My rating comes from the evidence, not the concept. The authors do not state when their diagnosis would stop working, and the obvious case sits in their own supplement: the gold decision is a fixed lookup on 29 hidden packet states, and the packets show that state as plainly named flags. This mapping is not given to the models, and only the author team reviewed the labels. The one outside check, a small panel of engineers, has no records in the supplement. So a post-access error may just mean the model didn't recover the authors' labeling rule, not a lapse in engineering judgment. Two cheap experiments would tell these apart, and neither is run: a program that opens the relevant packets and applies the stated rules, and a rerun of the top systems with those rules in the prompt. The only deterministic baseline ignores packet contents, so the paper has a floor but no ceiling.
  • The submitted artifact doesn't support several reported results. The Claude-Opus-4-8 row in Table A5, highlighted as the best non-reference value in its column, matches numbers logged for a different Claude model, and the authors' own notes call that row a proxy estimate, not a direct measurement. The paper doesn't say this. It also reports three runs per configuration and says all three are in the supplement, but only the first is there, with notes listing the other two as not yet scheduled. And seven pairs were dropped because models disagreed with the gold, which the paper describes only as an ambiguity audit.

Outside my rating: the supplementary files contain a home-directory path with a user name and other identifying traces, and the PDF lacks the ICLR running header.

Questions for authors
  1. Can you add the rule-applying baseline and the rules-in-prompt rerun? What matters is whether post-access errors persist once the rules are given. Blind labels from engineers outside the author team on a few dozen pairs would also show whether the gold holds up.
  2. Was the Claude-Opus-4-8 row in Table A5 measured on that model? If so, which run produced it?
  3. Where are the second and third runs behind Table 3?
Reviewer_vRt2OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Summary

  • The authors propose an LLM agent benchmark for a chip implementation decision-making task. LLM agents should decide which reports and packets to open, and need to decide whether to move on to the design, or whether it needs more evidence for the engineering requests.
Critical strengths

Strength

  • The authors define a task and domain that has seldom been addressed by open-source LLM agent benchmarks before.
  • They propose a process score that measures partial answers.
Critical weaknesses

Weakness

  • I am not sure how the constructed benchmark truly reflects real chip design workloads, and it feels more like an artificial benchmark. For instance, I do not understand why an LLM agent should have a fixed inspection budget when the number of packets is only six. Although the authors evaluate an all-packet setting, they should better justify whether the restricted setting captures a meaningful challenge encountered in practice.
  • The agent interface is restricted to selecting predefined report packets and submitting a judgment. Agentic harnesses like Claude Code or Codex enable the model to navigate files, formulate tool inputs, rerun analyses, and gather additional evidence. I think a more realistic setting should evaluate models inside such agentic harnesses instead of this restricted interface setting.
  • For the counterfactual pairs, the authors artificially change the evidence from the complete report, which makes me question the relevance to the real-world workload. The example shown by the authors seems to show that such counterfactual examples are too trivial (with no constraints loaded). I am not sure how much of these cases require chip-engineering reasoning, beyond simply identifying a lacking field, making “Request more evidence” action classification trivial.
  • For the process score, the authors did not explain the reasoning behind the weights of each component or justify their relative importance. While they include the sensitivity to the ranking in Section A.4, I believe justifying the relative magnitude between different components, or explaining how they chose these weights, is necessary.
Questions for authors

See the weakness section please.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.