- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces DepthBenchCAD, a benchmark and evaluation framework for LLM-generated CAD programs. It studies how to allocate a limited evaluation budget across tasks, independent generations, and parameter edits. The authors use a three-level variance model to predict when deeper auditing improves evaluation reliability. They evaluate the framework across five LLM-based systems and two CAD environments.
Critical strengths
-
Useful evaluation problem: The paper highlights an important trade-off between auditing individual programs more thoroughly and evaluating more independent programs. This is particularly relevant for CAD programs, where correctness can change after parameter edits.
-
Detailed benchmark design: The authors describe a benchmark with 120 task templates and 44,160 audit records. They also use held-out tasks and expert validation of the automated judge. I appreciate that the paper reports both the benefits and limitations of the proposed allocation strategies.
Critical weaknesses
-
Benchmark availability and reproducibility: The paper states that the dataset, evaluation code, and reproduction scripts are included in anonymous supplementary materials. However, I could not locate an accessible supplementary attachment or repository link on OpenReview or in the PDF. This is a major concern for a Datasets and Benchmarks submission, since these resources are central to the paper's contribution. Without access to them, it is difficult to verify the results or assess the benchmark's usefulness for future research.
-
Limited practical validation: The proposed allocation methods do not consistently outperform fixed-depth baselines. While the edit-stratified strategy performs better in environment B, the fixed-depth baseline achieves the lowest average estimation error in environment A. The audit-depth predictions are also evaluated on only 20 comparisons across both environments. Furthermore, the evaluation uses 16 predefined edit states and a finite test pool, which limits how broadly the findings can be generalized. The additional calibration cost also raises questions about practical efficiency.
Questions for authors
-
Could the authors clarify where the dataset, evaluation code, and reproduction scripts can be accessed? Appendix G describes these resources, but I could not find an accessible supplementary attachment or repository link.
-
Could the authors provide further evidence that the audit-depth recommendations generalize to larger task pools or different CAD environments?
-
How sensitive are the allocation results to errors in the automated judge, particularly for S4 and S5, where some failures are missed?
-
Could the authors compare the allocation strategies under a single budget that includes both calibration and evaluation costs?