acceptodds
Withdrawn from ICLR 2027

DepthBenchCAD: When Does Deeper Auditing Yield More Reliable Conclusions?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generative CAD evaluation has increasingly moved beyond static geometry toward executability, parameter reasoning, and editability, yet how behavioral evidence should be allocated under a finite evaluation budget remains largely unresolved. We show that this omission creates a measurement paradox: auditing each generated program more deeply can reduce the reliability of model-level conclusions by sacrificing task and generation coverage. We formalize this problem as a three-level evidence-allocation framework over task templates, independent generations, and counterfactual edit states. DepthBenchCAD defines an audit-depth-invariant average failure-risk estimand, derives finite-population variance under nested sampling, and combines variance with measured execution costs to predict when additional auditing improves or harms model-level estimation. Across two CAD environments and five LLM-based generation systems, deeper auditing improves individual-program measurement. We find a clear answer to when deeper auditing is most valuable: when a program’s behavior varies substantially across edit states and generating new programs is expensive; when uncertainty is driven mainly by task differences or generation randomness, broader coverage gives more reliable model-level estimates. Calibration estimates predict the direction of most such audit-depth changes on held-out tasks. These results advance behavioral CAD evaluation from checking whether generated programs survive edits to determining how much and where to audit for reliable model-level conclusions under finite budgets.

Primary area:
datasets and benchmarks
Keywords:
generative CAD, model evaluation, evaluation methodology, benchmark design, behavioral evaluation, hierarchical variance, audit depth, budget allocation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_TPwpOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces DepthBenchCAD, a benchmark and evaluation framework for LLM-generated CAD programs. It studies how to allocate a limited evaluation budget across tasks, independent generations, and parameter edits. The authors use a three-level variance model to predict when deeper auditing improves evaluation reliability. They evaluate the framework across five LLM-based systems and two CAD environments.

Critical strengths
  1. Useful evaluation problem: The paper highlights an important trade-off between auditing individual programs more thoroughly and evaluating more independent programs. This is particularly relevant for CAD programs, where correctness can change after parameter edits.

  2. Detailed benchmark design: The authors describe a benchmark with 120 task templates and 44,160 audit records. They also use held-out tasks and expert validation of the automated judge. I appreciate that the paper reports both the benefits and limitations of the proposed allocation strategies.

Critical weaknesses
  1. Benchmark availability and reproducibility: The paper states that the dataset, evaluation code, and reproduction scripts are included in anonymous supplementary materials. However, I could not locate an accessible supplementary attachment or repository link on OpenReview or in the PDF. This is a major concern for a Datasets and Benchmarks submission, since these resources are central to the paper's contribution. Without access to them, it is difficult to verify the results or assess the benchmark's usefulness for future research.

  2. Limited practical validation: The proposed allocation methods do not consistently outperform fixed-depth baselines. While the edit-stratified strategy performs better in environment B, the fixed-depth baseline achieves the lowest average estimation error in environment A. The audit-depth predictions are also evaluated on only 20 comparisons across both environments. Furthermore, the evaluation uses 16 predefined edit states and a finite test pool, which limits how broadly the findings can be generalized. The additional calibration cost also raises questions about practical efficiency.

Questions for authors
  1. Could the authors clarify where the dataset, evaluation code, and reproduction scripts can be accessed? Appendix G describes these resources, but I could not find an accessible supplementary attachment or repository link.

  2. Could the authors provide further evidence that the audit-depth recommendations generalize to larger task pools or different CAD environments?

  3. How sensitive are the allocation results to errors in the automated judge, particularly for S4 and S5, where some failures are missed?

  4. Could the authors compare the allocation strategies under a single budget that includes both calibration and evaluation costs?

Reviewer_7gptOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DepthBenchCAD, an evaluation framework to measure the behavioral reliability of generative CAD systems. The novelty of DepthBenchCAD compared with the state of the art is that it studies how a fixed evaluation budget should be divided across task templates, independent program generations and counterfactual edit states, rather than assuming that deeper auditing is always better. Experiments on two CAD environments, 120 task templates and five LLM-based generation systems demonstrate that deeper auditing improves the assessment of individual programs but may reduce the reliability of model-level conclusions when it limits task or generation coverage.

Critical strengths
  • The paper focuses on the practically relevant problem of trading off deep auditing and broad evaluation coverage under a fixed budget.
Critical weaknesses
  • The study has a narrow scope, limited to a specific questions related to generative CAD and a small set of evaluation environments.
  • The problem formulation, assumptions, and target use cases are not specified clearly enough.
  • The technical contribution is a limited empirical study which samples task templates, independent generations, and edit states under a cost model.
  • Generalisability of the conclusions is unclear, as they mainly rely on empirical observations obtained with manually defined costs, budgets, and edit states.
  • The paper provides insufficient information about the actual CAD task specifications used in the benchmark.
Questions for authors

I do not have questions whose answers can materially affect my assessment.

Reviewer_PeLZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the evaluation of LLM-generated parametric CAD programs, where a program that builds correctly at its nominal parameters (analogous to passing visible unit tests) may still fail under counterfactual parameter edits (analogous to hidden test cases). Under a fixed evaluation budget, it formalizes the trade-off between running more hidden tests per program (k) and covering more task templates (q) or independent generations (g), using a three-level nested variance decomposition combined with a linear cost model. Across two CAD environments and five LLMs, the authors show that deeper auditing can increase model-level estimation error for systems with cheap generation while reducing it for systems with expensive generation, and that calibration-based estimates predict the direction of this effect in 19 of 20 held-out comparisons.

Critical strengths
  • The paper addresses the behavior of generated CAD programs under parameter edits, which nominal-geometry metrics do not capture, and formulates audit depth as a budget allocation problem with an explicit estimand (Sec. 3.1).
Critical weaknesses
  • (W1) Unverified cost values. Costs are reported only as normalized ratios (1/3/8/16/32 in Environment A, exactly doubled in B; Table 11), whose regularity is hard to reconcile with measured wall-clock means (Appendix D.3). Moreover, r_I = 1 for S1 would imply near-zero LLM generation time, since c_I already includes an initial build. Raw timings are recorded (Table 7) but never reported; the authors should provide them with their variability, explain the conversion to integer costs, and include a cost-sensitivity analysis.
  • (W2) Generation and execution do not share a budget. Code generation runs on remote APIs, whereas CAD execution runs locally (Table 11), so the two costs are incurred on different machines and, for APIs, in different units (money vs. compute time). Summing them into a single budget (Eq. 6) is not justified, and the central trade-off between generating more programs and auditing existing ones appears to be an artifact of serial execution at concurrency 1.
  • (W3) Unmotivated budget constraint. The test pools contain only 48 and 36 templates, and the authors fully audited all 44,160 records, showing that exhaustive evaluation is feasible at this scale. The budget C₀ = 1024 thus appears arbitrary, and no realistic setting in which it would bind is presented.
  • (W4) No practical gain. Fixed k = 16 attains the lowest mean J in Environment A (0.471 vs. 0.481–0.533 for calibrated allocations; Sec. 5.1), while full calibration costs 2.04–5.67× the formal budget (Table 13). The authors concede that they "do not interpret calibration-driven joint allocation as outperforming the best fixed depth on average."
  • (W5) Limited novelty. Eqs. (4), (6), and (A.6) are standard results of cost-optimal multistage sampling [1, 2]. The claimed "measurement paradox" is an expected consequence of this theory.
  • (W6) Near-circular validation. Both the "actual" and the predicted ΔJ are computed from the same variance formula, using test-pool and calibration-pool components respectively (Sec. 4.5). The 19/20 correct signs therefore mainly reflect the stability of variance estimates across splits, not predictive validity.
  • (W7) Under-specified data construction. The edit-state generator is described only through acceptance criteria (Tables 5–6). The paper does not state how task specifications, valid ranges, semantic contracts, and edited parameter values are produced, nor whether they were authored or verified by humans. The 120 templates are also variants of only 14 part families (Table 4), limiting task diversity.

[1] W. G. Cochran. Sampling Techniques, 3rd ed. Wiley, 1977, Ch. 10.

[2] S. L. Lohr. Sampling: Design and Analysis, 2nd ed. Brooks/Cole, 2010, Ch. 5.

Questions for authors
  • (Q1) Please report the raw per-run wall-clock times (mean, standard deviation, and median) for generation, nominal build, and edit execution for each system and environment, and explain how these were converted into the integer costs in Table 11. Were the Environment B costs measured independently, or set to twice the Environment A values?
  • (Q2) In what practical setting do API generation and local CAD execution draw from a single budget? Please repeat the allocation analysis under (a) monetary cost based on API pricing and (b) parallel execution in which generation and auditing overlap, and report whether the preferred audit depths change.
  • (Q3) What is the total cost of exhaustively auditing the test pool in your cost units, and what fraction of it does C₀ = 1024 represent? Can you show results on a larger template pool, or describe a concrete evaluation scenario in which a budget of this size would bind?
  • (Q4) Given that fixed k = 16 matches or outperforms the calibrated allocations, what procedure do you recommend to practitioners? Please report the break-even number of evaluations N_break (Eq. A.7) for each system to show when calibration pays off.
  • (Q5) Which components of the framework go beyond classical cost-optimal multistage sampling (e.g., Cochran, Sec. 10.6), whose optimum subsample size has the same form as Eq. (A.6)?
  • (Q6) Can you compute model-level MSE empirically from repeated subsampling of the test pool, rather than from Eq. (5), and re-evaluate the sign predictions against it? For how many of the 20 comparisons does a confidence interval on ΔJ exclude zero?
  • (Q7) How were the task specifications, valid ranges, semantic contracts, and edited parameter values produced (manually, by rules, or with an LLM), and were they verified by humans? What fraction of candidate states was rejected by the acceptance criteria?
Reviewer_QrhiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DepthBenchCAD, a budget-aware evaluation framework for generative CAD systems. It investigates whether deeper auditing of individual CAD programs necessarily leads to more reliable model-level evaluation under a fixed computational budget.

The authors formulate CAD evaluation as a three-level sampling problem involving task templates, independent generations, and counterfactual edit states. They define an audit-depth-invariant failure-risk estimand and derive a variance decomposition that separates uncertainty across these three levels. Based on the estimated variance components and execution costs, the framework optimizes the allocation of evaluation resources among task coverage, generation diversity, and audit depth.

The framework is evaluated across two CAD environments, 14 task families, and five LLM-based generation systems, using 44,160 audit records. The experiments show that deeper auditing reduces program-level estimation error but does not always improve model-level estimation accuracy. The proposed variance-and-cost model predicts the direction of audit-depth benefits in most held-out comparisons. The paper also examines calibration costs, cross-task-family transfer, confidence interval coverage, and automated-judge reliability.

Critical strengths
  1. Clear problem formulation. The paper identifies a practical trade-off between audit depth and evaluation coverage under a fixed computational budget. Distinguishing program-level measurement accuracy from model-level estimation reliability provides a useful perspective for evaluating generative CAD systems.

  2. Systematic empirical evaluation. The authors evaluate their framework across two CAD environments and five LLM-based generation systems, considering multiple audit depths, variance components, and calibration costs. The experiments provide empirical evidence of how different sources of uncertainty affect evaluation reliability, although the observations largely align with established statistical principles.

Critical weaknesses
  1. Limited methodological novelty. The proposed framework largely relies on established techniques, including hierarchical sampling, variance decomposition, and cost-constrained optimization. The paper mainly applies these existing principles to generative CAD evaluation, without introducing a substantially new statistical methodology.

  2. Limited new insights. The central finding that deeper auditing may reduce model-level estimation reliability under a fixed budget is largely predictable from classical sampling theory. Increasing repeated measurements within a program necessarily trades off against broader task and generation coverage. The experimental results primarily confirm this expected trade-off rather than reveal fundamentally new phenomena or mechanisms.

  3. Limited improvements over simple baselines. The proposed allocation strategies do not consistently outperform fixed-depth auditing. In the primary environment, fixed k = 16 achieves a lower average estimation error than the pooled, system-level, and stratified allocation strategies. This raises concerns about whether the additional optimization complexity provides sufficient practical benefits.

  4. High calibration costs and limited generalizability. The framework relies on variance and cost estimates obtained through calibration, which can introduce substantial additional evaluation overhead. Moreover, the experiments are restricted to a finite set of predefined CAD edit states and task families. It remains unclear whether the proposed allocation strategies can generalize effectively to different evaluation settings, edit distributions, or real-world CAD workflows.

Questions for authors
  1. Novelty beyond existing sampling theory. The proposed framework appears to be largely based on established principles of hierarchical sampling, variance decomposition, and cost-constrained optimization. Could the authors clarify which specific methodological or theoretical contributions go beyond directly applying existing statistical techniques? What new insights does the framework provide that cannot be readily derived from classical sampling theory?

  2. Practical advantages over fixed-depth baselines. In the primary environment, the proposed allocation strategies do not consistently outperform fixed-depth auditing, with fixed k = 16 achieving the lowest average estimation error among the compared strategies. Could the authors clarify under what conditions their method provides substantial improvements over strong fixed-depth baselines? Are these improvements statistically significant and consistent across different systems and evaluation budgets?

  3. Calibration costs and overall efficiency. The reported calibration costs are substantial relative to the formal evaluation budget. If calibration expenses are included in the total evaluation budget rather than accounted for separately, does the proposed framework still provide meaningful improvements in estimation accuracy? How many repeated evaluations would be required to justify the additional calibration overhead?

  4. Generalizability of the findings. The experiments rely on 16 predefined edit states per CAD template and a limited number of task families within two environments. How sensitive are the estimated variance components and recommended audit depths to the choice and distribution of edit states? Is there evidence that the conclusions remain valid under different edit distributions or evaluation settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.