- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CAD-Ladder is an annotation method and benchmark that produces five increasingly abstract descriptions (L1–L5) from execution-verified CadQuery programs through sequential rewriting. It audits whether the descriptions align with the geometry and preserve essential constraints, then uses repeated blind reconstruction to assess fidelity and consistency. The authors report higher semantic consistency than CADPrompt (77.9% vs. 42.5%) and better reconstruction after audit-guided revision and source-grounded reannotation. They also compare standalone LLMs and agents on T1 at L5, where delivery quality ranges from 49.7% to 95.1%.
Critical strengths
- The paper addresses a meaningful evaluation problem. When a description is incorrect or incomplete, it is difficult to tell whether a poor reconstruction reflects a limitation of the model or a problem with the instructions.
- The CADPrompt reannotation experiment provides convincing evidence that descriptions matter. Mean reconstruction IoU rises from 0.7748 to 0.9295 on the same 200 targets, with no increase in average description length.
- I like the fixed reference bank and T0–T5 groups as a way to study tasks ranging from exact reuse to increasingly demanding edits.
Critical weaknesses
- I find the novelty somewhat limited. Previous work has already examined the importance of description quality [1] and introduced staged annotation pipelines with descriptions at different levels of detail [2]. I see the main contribution as a new annotation and evaluation framework for existing CAD programs, with an analysis of semantic consistency and reconstruction.
- I would like to see independent validation of the semantic audit. Extracting facts from the source, checking constraints, and reconstructing geometry all rely on GPT-6 Astra. Using separate contexts helps limit information leakage, but it does not establish whether the extracted facts and judgments are correct. This makes the 77.9% consistency rate unreliable, particularly because it depends on automatically selected core facts.
- I am concerned that the main benchmark still uses descriptions that fail its own audit. At L5, 38 of the 172 audited T1 descriptions fail the semantic-consistency criterion, while only 17.4% meet the stricter exact-recovery criterion in Table 1. I do not think every useful description needs to support exact recovery, but the semantic failures are a concern given the paper's focus on annotation reliability. I am also not yet convinced that the improvements come from sequential rewriting. Could rewriting the 38 failed descriptions from source without audit feedback achieve gains comparable to audit-guided revision? This comparison would help clarify how much the audit adds beyond another rewrite.
- I would like to see a broader comparison of the reference-assisted systems. The 95.1% IoU in Table 5 is obtained on T1, where the evaluation programs differ from available reference programs through literal replacement. This is a useful retrieval-and-editing setting, but I find it difficult to compare these results directly with those of systems that generate without access to the reference bank.
[1] Govindarajan et al. "CADmium: Fine-Tuning Code Language Models for Text-Driven Sequential CAD Design" [2] Khan et al. "Text2CAD: Generating Sequential CAD Designs from Beginner-to-Expert Level Text Prompts"
Questions for authors
- How well does the semantic audit agree with blinded CAD-expert judgments, including the selection of source facts? Do the conclusions hold when the audit and reconstruction are repeated with another commercial model, such as Claude or Gemini?
- Do the system rankings hold after correcting the 38 failed L5 descriptions? How does generating L5 descriptions directly from the source compare with sequential rewriting?
- How do the systems compare when given access to the same reference bank, and do the findings extend beyond T1?
3 comments
Response to Reviewer ocwH (2/2)
Q2 - Revised descriptions rankings and direct generation
The system comparison uses one fixed description version for every evaluated system, including the same 38 descriptions that fail SCR. Any incorrect or omitted information is therefore part of the same shared input conditions, so the reported comparison is fair with respect to which descriptions the systems are given. The original DeepSeek-V4-Flash API used for the system experiments is no longer available to us. Repeating the entire comparison would therefore entail substantial rerunning costs in terms of both computation and time.
We added the direct-generation comparison on 30 matched targets. Existing sequential L5 descriptions and newly generated source-to-L5 descriptions were each reconstructed once by DeepSeek-V4.1-Flash in DeepSeek Harness under the same normalized pose-aligned scoring protocol:
| L5 construction | Mean IoU | Valid |
|---|---|---|
| Existing sequential | 0.9353 | 30/30 |
| Direct from source | 0.9570 | 30/30 |
Direct generation is slightly better in this comparison, achieving a 0.0216 higher mean IoU. Direct generation is therefore competitive in this test, but we would clarify that the ladder is an operational design for paired input representations. L1 retains annotated executable code; L2 removes executable syntax; L3 fuses operation-level instructions into parametric prose; L4 emphasizes quantified relative relations; and L5 compresses construction detail into a part-level specification. The ladder's benchmark value remains the availability of paired L1–L5 representations: a single directly generated L5 does not supply the other input conditions.
Q3 - Reference access and evidence beyond T1
For access to the same reference bank, the comparison established by our experiments is among the DSF reference-harness variants: with the backbone and bank fixed, delivery quality is 91.5% with flat retrieval, 93.5% with family-first retrieval, and 95.1% when the retrieved program's L1 annotation is also supplied. Thus, access to the same bank does not determine performance by itself; how references are retrieved and presented also matters. Table 5 therefore compares complete configurations under their stated resources, while the fixed-backbone ablations isolate the contribution of reference access: adding retrieval to save–run–inspect raises delivery quality from 73.3% to 91.5%, an 18.2-point gain.
Beyond T1, the fixed reference harness already achieves delivery quality of 90.5% on T3, 89.4% on T4, and 76.5% on T5. On T2, aligning the same outputs raises delivery quality from 15.6% to 88.2%. These results suggest that the harness can also handle relative computations and feature edits, while geometry rebuilding is more challenging and global pose is a distinct source of failure. The benchmark therefore reveals where a reference-assisted system succeeds and where it breaks down.
Reference-assisted systems should benefit most directly on T1, where literal changes preserve the existing construction. T2 requires correct transformation composition; T3 requires relative reasoning; T4 requires feature or Boolean edits; T5 places more weight on independent geometric planning. These strata describe the edits required relative to a fixed reference bank. For systems that do not access this bank, the relationship to a stored program is not part of their available information or generation procedure.
Taken together, our experiments provide complementary evidence: the common T1 tasks distinguish the evaluated system configurations; the fixed-backbone ablations measure the contribution of execution, inspection, and retrieval; and the reference-relation strata reveal how delivery changes with the required adaptation. These experiments support our central conclusion that CAD-Ladder both distinguishes systems and explains important dependencies of their performance.
Thank you for the clarifications and additional experiments. I appreciate the value of evaluating and improving text–geometry consistency of Text-to-CAD data, but I still find the novelty limited compared to existing approaches. I am also not fully convinced that sequential rewriting is beneficial, as generating the L5 description from source performs better in the new analysis. I will maintain my score.
Response to Reviewer ocwH (1/2)
We sincerely thank you for your careful assessment and for recognizing the importance of description reliability, the same-target CADPrompt evidence, and the reference-relation strata. We also appreciate your suggestions on independent validation, the role of description revisions, and fair system comparisons. Below, we clarify the contribution and address each concern with the available evidence and additional experiments.
W1 - Contribution relative to prior work
We agree that generating high-quality Text-to-CAD data and producing descriptions at multiple levels of detail have been studied before. Our research question is how to evaluate and improve the text–geometry consistency of Text-to-CAD data. Data generation alone does not establish that a text–geometry pair is a reliable training or evaluation example. CAD-Ladder makes this consistency an explicit object of measurement and correction. Source-grounded semantic auditing checks what the description asserts against the executed geometry and separately measures which core facts are missing. The resulting records locate specific problems for revision, and blind reconstruction tests whether the changed description better recovers the same target. This connects three questions that matter for dataset reliability: what is inconsistent or missing, how it can be corrected, and whether the correction improves downstream geometric recovery.
The same-target CADPrompt experiment demonstrates the relevance of this question to an existing human-authored dataset: replacing descriptions increases mean IoU from 0.7748 to 0.9295 on the same 200 targets, with slightly shorter descriptions on average. Thus, our contribution is a framework for evaluating and improving the consistency of existing text–geometry pairs, supported by measurable reconstruction effects. We also provide a benchmark for evaluating CAD models and agents: multilevel annotations enable comparisons across input representations, while controlled agent configurations and reference-relation partitions reveal how tool access, reference reuse, and different editing requirements affect CAD generation.
W3 - Audit feedback versus another source-only rewrite
We tested source-only regeneration on the same 38 targets using GPT-5.6 Sol, the same annotation model used for the audit-guided revisions. The new generation received the source programs and style/length constraints, but no original descriptions, revised texts, or audit feedback. Each new description was reconstructed once by DeepSeek-V4.1-Flash in DeepSeek Harness, using the same instructions, budgets, and normalized pose-aligned scorer.
Source-only regeneration improves mean IoU from 0.7984 to 0.9237, recovering much of the gain achieved by audit-guided revision (0.9540). Audit-guided revision shows only a modest additional advantage in this end-to-end comparison: 0.0303 in mean IoU and 36/38 versus 33/38 targets reaching 0.80. The audit identifies which existing descriptions need attention, locates specific conflicts and omissions, and supplies evidence for targeted correction. A successful source-only rewrite can produce a good replacement, but does not itself diagnose the reliability of the original description.
Q1 - Human calibration and validation with another reconstruction family
In the new calibration study, two CAD-proficient human annotators reviewed 50 CAD-Ladder L5 descriptions and 50 CADPrompt descriptions. They evaluated the same 1,609 frozen atomic statements against source evidence, with the original auditor's S/C/U labels and rationales hidden.
Three-class agreement between the two human annotators is Cohen's κ = 0.7888, with corpus-specific values of 0.7997 for CAD-Ladder and 0.7768 for CADPrompt. The SCR values obtained using each annotator's S/C/U labels are:
The calibration evaluates human judgments on the frozen atomic statements, with source-fact selection and the original core-retention scores held fixed. The resulting SCR decisions agree with the original auditor on 99 of 100 descriptions for each annotator, preserving the cross-corpus difference. This supports the stability of the description-level decisions under human atomic judgments.
We also repeated the reconstruction test with DeepSeek-V4.1-Flash on the same 38 original/revised description pairs. Mean IoU increased from 0.7984 to 0.9540, and the number of targets reaching IoU ≥ 0.80 increased from 25 to 36. This provides reconstruction evidence that the revision benefit transfers beyond the original model family.