- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces ChipReasoning, a benchmark evaluating how LLMs select reports under a limited inspection budget and make engineering decisions from RTL, synthesis, and verification evidence. It addresses a meaningful problem. However, the authors need to explain how the reports and reference answers were validated and how the tasks reflect real workflows. Differences between CLI/API setups also raise concerns about comparison fairness.
Critical strengths
-
The paper addresses a real problem in chip design: deciding when to proceed to the next stage often requires engineering experience or hands-on guidance.
-
This engineering judgment is valuable in its own right, making it a worthwhile capability to evaluate in LLMs.
Critical weaknesses
-
Unclear benchmark reliability. The reports and reference answers are constructed using rules, but how the final cases were independently validated is insufficiently explained. The authors should provide a clear validation procedure, using representative runs in established (commercial or open-source) EDA tools and expert checks of the reference decisions.
-
Unclear practical relevance. The tasks mainly involve RTL, synthesis, and verification reports. Their connection to real chip-design workflows and broader implementation stages needs clarification.
-
Uneven execution conditions. Models use different CLI/API setups, with differences in history handling and token limits that may affect the fairness of comparisons. Please report the actual token usage, interaction counts, and retry counts for each model so that comparability can be assessed.
Questions for authors
-
How were the constructed reports and reference decisions validated? Please describe the procedure and provide representative EDA-tool results and expert checks supporting their correctness.
-
Which real chip-design stages and decisions do the tasks represent? Why does the limited report-inspection budget reflect practical engineering constraints?
-
How were execution conditions controlled across CLI/API setups? Please report actual token usage, interaction counts, and retry counts for each model, and explain how setup differences affect comparability.