acceptodds
Under review as a conference paper at ICLR 2027

CAD-Ladder: A Source-Grounded Benchmark for Text-to-CAD Models and Agents

Abstract

Generating CAD models by synthesizing executable code from natural-language descriptions is promising, but evaluation depends on whether each description and reference program denote the same geometry. Misaligned pairs can penalize faithful systems and obscure differences among models and agents. We introduce CAD-Ladder, a source-grounded annotation method and benchmark that derives five increasingly abstract descriptions through sequential code-to-language rewriting from execution-verified CadQuery programs. Semantic auditing checks whether the reference solid satisfies the description’s geometric constraints and estimates essential source-constraint retention. Repeated blind reconstruction measures geometric fidelity and inter-run agreement through deterministic geometric scoring. CAD-Ladder achieves a semantic consistency rate of 77.9%, compared with 42.5% for CADPrompt. Audit-guided revision of 38 descriptions that fail the semantic criterion raises their joint reconstruction pass rate from 55.3% to 73.7%. Applying CAD-Ladder's annotation principles to the same 200 CADPrompt targets increases mean reconstruction IoU from 0.7748 to 0.9295. The resulting benchmark separates standalone LLM and agentic configurations on T1 at L5, with delivery quality ranging from 49.7% to 95.1%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Text-to-CAD, datasets and benchmarks, semantic consistency, large language models, LLM agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ocwHOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

CAD-Ladder is an annotation method and benchmark that produces five increasingly abstract descriptions (L1–L5) from execution-verified CadQuery programs through sequential rewriting. It audits whether the descriptions align with the geometry and preserve essential constraints, then uses repeated blind reconstruction to assess fidelity and consistency. The authors report higher semantic consistency than CADPrompt (77.9% vs. 42.5%) and better reconstruction after audit-guided revision and source-grounded reannotation. They also compare standalone LLMs and agents on T1 at L5, where delivery quality ranges from 49.7% to 95.1%.

Critical strengths
  1. The paper addresses a meaningful evaluation problem. When a description is incorrect or incomplete, it is difficult to tell whether a poor reconstruction reflects a limitation of the model or a problem with the instructions.
  2. The CADPrompt reannotation experiment provides convincing evidence that descriptions matter. Mean reconstruction IoU rises from 0.7748 to 0.9295 on the same 200 targets, with no increase in average description length.
  3. I like the fixed reference bank and T0–T5 groups as a way to study tasks ranging from exact reuse to increasingly demanding edits.
Critical weaknesses
  1. I find the novelty somewhat limited. Previous work has already examined the importance of description quality [1] and introduced staged annotation pipelines with descriptions at different levels of detail [2]. I see the main contribution as a new annotation and evaluation framework for existing CAD programs, with an analysis of semantic consistency and reconstruction.
  2. I would like to see independent validation of the semantic audit. Extracting facts from the source, checking constraints, and reconstructing geometry all rely on GPT-6 Astra. Using separate contexts helps limit information leakage, but it does not establish whether the extracted facts and judgments are correct. This makes the 77.9% consistency rate unreliable, particularly because it depends on automatically selected core facts.
  3. I am concerned that the main benchmark still uses descriptions that fail its own audit. At L5, 38 of the 172 audited T1 descriptions fail the semantic-consistency criterion, while only 17.4% meet the stricter exact-recovery criterion in Table 1. I do not think every useful description needs to support exact recovery, but the semantic failures are a concern given the paper's focus on annotation reliability. I am also not yet convinced that the improvements come from sequential rewriting. Could rewriting the 38 failed descriptions from source without audit feedback achieve gains comparable to audit-guided revision? This comparison would help clarify how much the audit adds beyond another rewrite.
  4. I would like to see a broader comparison of the reference-assisted systems. The 95.1% IoU in Table 5 is obtained on T1, where the evaluation programs differ from available reference programs through literal replacement. This is a useful retrieval-and-editing setting, but I find it difficult to compare these results directly with those of systems that generate without access to the reference bank.

[1] Govindarajan et al. "CADmium: Fine-Tuning Code Language Models for Text-Driven Sequential CAD Design" [2] Khan et al. "Text2CAD: Generating Sequential CAD Designs from Beginner-to-Expert Level Text Prompts"

Questions for authors
  1. How well does the semantic audit agree with blinded CAD-expert judgments, including the selection of source facts? Do the conclusions hold when the audit and reconstruction are repeated with another commercial model, such as Claude or Gemini?
  2. Do the system rankings hold after correcting the 38 failed L5 descriptions? How does generating L5 descriptions directly from the source compare with sequential rewriting?
  3. How do the systems compare when given access to the same reference bank, and do the findings extend beyond T1?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ocwH (1/2)

We sincerely thank you for your careful assessment and for recognizing the importance of description reliability, the same-target CADPrompt evidence, and the reference-relation strata. We also appreciate your suggestions on independent validation, the role of description revisions, and fair system comparisons. Below, we clarify the contribution and address each concern with the available evidence and additional experiments.

W1 - Contribution relative to prior work

We agree that generating high-quality Text-to-CAD data and producing descriptions at multiple levels of detail have been studied before. Our research question is how to evaluate and improve the text–geometry consistency of Text-to-CAD data. Data generation alone does not establish that a text–geometry pair is a reliable training or evaluation example. CAD-Ladder makes this consistency an explicit object of measurement and correction. Source-grounded semantic auditing checks what the description asserts against the executed geometry and separately measures which core facts are missing. The resulting records locate specific problems for revision, and blind reconstruction tests whether the changed description better recovers the same target. This connects three questions that matter for dataset reliability: what is inconsistent or missing, how it can be corrected, and whether the correction improves downstream geometric recovery.

The same-target CADPrompt experiment demonstrates the relevance of this question to an existing human-authored dataset: replacing descriptions increases mean IoU from 0.7748 to 0.9295 on the same 200 targets, with slightly shorter descriptions on average. Thus, our contribution is a framework for evaluating and improving the consistency of existing text–geometry pairs, supported by measurable reconstruction effects. We also provide a benchmark for evaluating CAD models and agents: multilevel annotations enable comparisons across input representations, while controlled agent configurations and reference-relation partitions reveal how tool access, reference reuse, and different editing requirements affect CAD generation.

W3 - Audit feedback versus another source-only rewrite

We tested source-only regeneration on the same 38 targets using GPT-5.6 Sol, the same annotation model used for the audit-guided revisions. The new generation received the source programs and style/length constraints, but no original descriptions, revised texts, or audit feedback. Each new description was reconstructed once by DeepSeek-V4.1-Flash in DeepSeek Harness, using the same instructions, budgets, and normalized pose-aligned scorer.

DescriptionMean IoUIoU ≥\geq 0.80
Original0.798425/38
Source-only regeneration0.923733/38
Audit-guided revision0.954036/38

Source-only regeneration improves mean IoU from 0.7984 to 0.9237, recovering much of the gain achieved by audit-guided revision (0.9540). Audit-guided revision shows only a modest additional advantage in this end-to-end comparison: 0.0303 in mean IoU and 36/38 versus 33/38 targets reaching 0.80. The audit identifies which existing descriptions need attention, locates specific conflicts and omissions, and supplies evidence for targeted correction. A successful source-only rewrite can produce a good replacement, but does not itself diagnose the reliability of the original description.

Q1 - Human calibration and validation with another reconstruction family

In the new calibration study, two CAD-proficient human annotators reviewed 50 CAD-Ladder L5 descriptions and 50 CADPrompt descriptions. They evaluated the same 1,609 frozen atomic statements against source evidence, with the original auditor's S/C/U labels and rationales hidden.

Three-class agreement between the two human annotators is Cohen's κ\kappa = 0.7888, with corpus-specific values of 0.7997 for CAD-Ladder and 0.7768 for CADPrompt. The SCR values obtained using each annotator's S/C/U labels are:

CorpusOriginal auditorAnnotator AAnnotator B
CAD-Ladder L539/50 (78%)38/50 (76%)38/50 (76%)
CADPrompt21/50 (42%)21/50 (42%)21/50 (42%)

The calibration evaluates human judgments on the frozen atomic statements, with source-fact selection and the original core-retention scores held fixed. The resulting SCR decisions agree with the original auditor on 99 of 100 descriptions for each annotator, preserving the cross-corpus difference. This supports the stability of the description-level decisions under human atomic judgments.

We also repeated the reconstruction test with DeepSeek-V4.1-Flash on the same 38 original/revised description pairs. Mean IoU increased from 0.7984 to 0.9540, and the number of targets reaching IoU ≥\geq 0.80 increased from 25 to 36. This provides reconstruction evidence that the revision benefit transfers beyond the original model family.

AuthorsOpenReview ↗

Response to Reviewer ocwH (2/2)

Q2 - Revised descriptions rankings and direct generation

The system comparison uses one fixed description version for every evaluated system, including the same 38 descriptions that fail SCR. Any incorrect or omitted information is therefore part of the same shared input conditions, so the reported comparison is fair with respect to which descriptions the systems are given. The original DeepSeek-V4-Flash API used for the system experiments is no longer available to us. Repeating the entire comparison would therefore entail substantial rerunning costs in terms of both computation and time.

We added the direct-generation comparison on 30 matched targets. Existing sequential L5 descriptions and newly generated source-to-L5 descriptions were each reconstructed once by DeepSeek-V4.1-Flash in DeepSeek Harness under the same normalized pose-aligned scoring protocol:

L5 constructionMean IoUValid
Existing sequential0.935330/30
Direct from source0.957030/30

Direct generation is slightly better in this comparison, achieving a 0.0216 higher mean IoU. Direct generation is therefore competitive in this test, but we would clarify that the ladder is an operational design for paired input representations. L1 retains annotated executable code; L2 removes executable syntax; L3 fuses operation-level instructions into parametric prose; L4 emphasizes quantified relative relations; and L5 compresses construction detail into a part-level specification. The ladder's benchmark value remains the availability of paired L1–L5 representations: a single directly generated L5 does not supply the other input conditions.

Q3 - Reference access and evidence beyond T1

For access to the same reference bank, the comparison established by our experiments is among the DSF reference-harness variants: with the backbone and bank fixed, delivery quality is 91.5% with flat retrieval, 93.5% with family-first retrieval, and 95.1% when the retrieved program's L1 annotation is also supplied. Thus, access to the same bank does not determine performance by itself; how references are retrieved and presented also matters. Table 5 therefore compares complete configurations under their stated resources, while the fixed-backbone ablations isolate the contribution of reference access: adding retrieval to save–run–inspect raises delivery quality from 73.3% to 91.5%, an 18.2-point gain.

Beyond T1, the fixed reference harness already achieves delivery quality of 90.5% on T3, 89.4% on T4, and 76.5% on T5. On T2, aligning the same outputs raises delivery quality from 15.6% to 88.2%. These results suggest that the harness can also handle relative computations and feature edits, while geometry rebuilding is more challenging and global pose is a distinct source of failure. The benchmark therefore reveals where a reference-assisted system succeeds and where it breaks down.

Reference-assisted systems should benefit most directly on T1, where literal changes preserve the existing construction. T2 requires correct transformation composition; T3 requires relative reasoning; T4 requires feature or Boolean edits; T5 places more weight on independent geometric planning. These strata describe the edits required relative to a fixed reference bank. For systems that do not access this bank, the relationship to a stored program is not part of their available information or generation procedure.

Taken together, our experiments provide complementary evidence: the common T1 tasks distinguish the evaluated system configurations; the fixed-backbone ablations measure the contribution of execution, inspection, and retrieval; and the reference-relation strata reveal how delivery changes with the required adaptation. These experiments support our central conclusion that CAD-Ladder both distinguishes systems and explains important dependencies of their performance.

Thank you for the clarifications and additional experiments. I appreciate the value of evaluating and improving text–geometry consistency of Text-to-CAD data, but I still find the novelty limited compared to existing approaches. I am also not fully convinced that sequential rewriting is beneficial, as generating the L5 description from source performs better in the new analysis. I will maintain my score.

Reviewer_5QoMOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces an annotation method consisting of five sequential levels to address the code-to-language translation problem in CAD for more consistent generation. It also designs two metrics, semantic consistency rates and repeated blind reconstruction, to measure the code-language consistency and CAD reconstruction quality and shows improvements on CADPrompt. Finally, it benchmarks models from domain-specific LLMs to general coding agents with harness with the generated annotations.

Critical strengths

The paper is generally well written and structured. The visualization helps understand the pipeline.

Critical weaknesses

The motivation and positioning of the method are weak. The problem it attempts to address—text-code consistency in CAD—is merely a special case of generic text-to-code generation, rather than targeting a CAD-specific challenge. Furthermore, using LLMs as judges and for synthetic data generation is not novel. It is also unclear how this approach contrasts with existing baselines, if any. The narrative of the paper is confusing. If the focus is on introducing a benchmark, the authors should explain how it differs from existing CAD benchmarks and demonstrate its uniqueness so others are willing to use it, rather than comparing it with CADPrompt as if it were a method. Conversely, if the focus is on a new annotation method, it is unclear what the authors aim to demonstrate by benchmarking various coding agents on data generated by their own pipeline.

Questions for authors

How do the authors justify the design choices of L1–L5 in the annotation method? They appear largely heuristic. Are there any ablation studies to support them? How does CAD-Ladder differ from CADPrompt, and is a direct comparison between them fair? Could the authors provide further explanation or analysis as to why CAD-Ladder can improve the semantic consistency rate, reconstruction pass rate, and mean IoU? What key takeaway is Table 1 intended to demonstrate, and how does it specifically help answer RQ1? Why is only a single metric reported in Tables 3 and 4? Both consistency and reconstruction metrics should be included.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5QoM (1/2)

We thank you for your thoughtful review and constructive comments on the motivation, design choices, and experimental comparisons. Your questions have helped us clarify the relationship between the benchmark, its construction procedure, and the evidence supporting its value. Below, we respond to these concerns and explain what each experiment establishes.

W1 - Motivation and positioning

CAD-Ladder is a benchmark and a framework for making its conditioning descriptions inspectable. Its motivation is an evaluation-validity problem: if a text and its reference denote different solids, an executable, text-faithful prediction can receive a low reference-IoU score. In terms of the feasible set G(x)\mathcal{G}(x) of shapes satisfying a description x, the reference must belong to G(x)\mathcal{G}(x) before similarity to that reference can be interpreted as evidence of instruction following. A description may also admit several geometries; this is why conflict and information retention are measured separately.

CAD makes this issue concrete. A number must be attached to the correct component and measurement convention; a nominal sketch feature may disappear after a Boolean operation; and connectivity, hole counts, quantifiers, and relative placement must be checked against the resulting solid. Program execution alone does not establish these correspondences. Our contribution brings executable geometric evidence, frozen constraint-level records, paired descriptions, and traceable revision into one benchmark. The CADPrompt experiments establish the relevance of description quality beyond our own parts; the model/agent experiments show how the benchmark exposes sensitivity to representation and available tools.

Q1 - Design of the ladder and the new direct-generation comparison

The ladder is an operational design for paired input representations. L1 retains annotated executable code; L2 removes executable syntax; L3 fuses operation-level instructions into parametric prose; L4 emphasizes quantified relative relations; and L5 compresses construction detail into a part-level specification. These transitions expose different information to a generator while keeping the target fixed. The ladder is designed to provide paired input representations for evaluation; its value does not require sequential generation to outperform direct generation.

We added a comparison on 30 matched targets, using GPT-6 Astra to generate L5 descriptions directly from the source programs. Existing sequential L5 descriptions and newly generated source-to-L5 descriptions were reconstructed by DeepSeek-V4.1-Flash in DeepSeek Harness under the same normalized pose-aligned scoring protocol:

L5 constructionMean IoUValid
Existing sequential0.935330/30
Direct from source0.957030/30

Direct source-to-L5 generation is slightly better in this comparison, achieving a 0.0216 higher mean IoU. The ladder's benchmark value remains the availability of paired L1–L5 representations: a single directly generated L5 does not supply the other input conditions.

AuthorsOpenReview ↗

Response to Reviewer 5QoM (2/2)

Q2 - Fairness and mechanisms compared with CADPrompt

The two comparisons in the paper have distinct purposes. Table 3 measures the corpora under the same semantic-audit policy; their target distributions differ, so this is a corpus characterization. Table 4 holds all 200 CADPrompt target geometries fixed and changes their descriptions under matched reconstruction and scoring conditions, directly testing whether the annotation principles transfer beyond CAD-Ladder's chosen parts. On these same targets, mean IoU rises from 0.7748 to 0.9295, and IoU ≥\geq 0.90 increases from 108/200 to 158/200. Mean description length decreases from 90.0 to 85.8 words. Thus, the improvement cannot be explained by selecting easier parts or simply making descriptions longer.

The mechanisms are geometric, not merely stylistic. Appendix G shows how assigning dimensions to the correct component and explicitly describing a subtractive recess changes what the reconstructor builds. It also retains a counterexample in which dimensions are preserved but an L-shaped relation becomes a T-shaped relation. The new 38-target DeepSeek experiment provides a further positive transfer result for the revised descriptions, with mean IoU increasing by 0.1556. Together, the matched studies support the practical value of checking and improving text–geometry alignment.

DescriptionTargetsMean IoUIoU ≥\geq 0.80
Original380.798425/38 (65.8%)
Revised380.954036/38 (94.7%)

Q3 - Interpretation of Table 1 and the complementary metrics

Table 1 addresses the first part of RQ1: how reliable the descriptions are as CAD specifications. Reliability requires both consistency of the information stated and retention of the information needed to characterize the source geometry. These are different properties: a description can contain few false statements while leaving important features unspecified. Table 1 makes this distinction concrete. Coverage remains above 90% and conflict among decidable statements stays between 1.5% and 3.1%, whereas assessed core retention decreases from 97.3% at L1 to 75.3% at L5. The key contrast in this reported profile is therefore between largely compatible stated constraints and increasing omission of core information. At L5, 134 of 172 descriptions pass SCR (77.9%), but only 17.4% satisfy the stricter requirement of zero decided conflict and complete core retention. Thus, most descriptions satisfy the joint low-conflict and retention criterion, while substantially fewer preserve every assessed core fact without contradiction. Table 1 therefore answers how reliable the descriptions are with a measured qualification, rather than a blanket claim that all generated descriptions are correct.

Tables 3 and 4 emphasize different metric families because they answer different questions. Table 3 asks how the existing CAD-Ladder and CADPrompt descriptions compare under the same semantic-audit policy, and reports coverage, conflict, core retention, and SCR. Adding a cross-corpus reconstruction comparison would compare different target geometries and text distributions: it would not fairly isolate the effect of annotation quality. The reconstruction comparison is therefore reported in Table 4, where the target geometries are held fixed. Table 4 asks whether replacing descriptions using our source-grounded annotation procedure improves geometric recovery. It holds all 200 CADPrompt targets, the reconstructor, and the scoring protocol fixed, changing only the descriptions. Mean IoU increases from 0.7748 to 0.9295, and the number reaching IoU ≥\geq 0.90 rises from 108 to 158, without an increase in average description length. A semantic audit of the reannotated descriptions would provide additional information about which constraints improved. We have not performed that additional audit, because the matched reconstruction improvement already directly supports the conclusion tested in Table 4: the replacement descriptions improve recoverability of the same target geometry.

Reviewer_eFjcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that Text-to-CAD benchmarks can contain text–code pairs that do not describe the same geometry. Such pairs penalize systems that faithfully follow the text. It proposes CAD-Ladder, built as follows:

  • Corpus. Starting from execution-verified CadQuery programs in BenchCAD, an LLM rewrites each program into five description levels, from L1 (the program annotated in place) to L5 (an abstract natural-language specification). Tasks are split into T0–T5 according to their relation to the nearest memory/reference program.
  • Validation of descriptions.
    • A two-context semantic audit: a source context extracts frozen constraints and core facts (K_core), and a separate judgment context labels each constraint as S, C, or U. This yields the Conflict rate, the essential-fact retention rate (R_ess), and a pass criterion (SCR).
    • Two blind reconstructions per description, scored by pose-aligned 64³ voxel IoU.

Reported results:

  • SCR: 77.9% at L5, vs. 42.5% for CADPrompt.
  • CADPrompt re-annotation: applying the paper's annotation principles to 200 CADPrompt targets raises reconstruction IoU from .7748 to .9295.
  • Audit-guided revision: revising 38 failing descriptions raises the reconstruction pass rate from 21/38 to 28/38.
  • System comparison: standalone LLMs, domain models (CAD-Coder, CADFusion, Cadrille, etc.), commercial coding agents (Claude Code, Codex), and an in-house harness (DSF with tools and optional memory) are compared using fixed-frame Q_delivery. On T1/L5, delivery quality ranges from 49.7% to 95.1%.
Critical strengths

S1. It identifies a real, under-measured problem. Treating whether a description and its reference program describe the same geometry as an explicit measurement target is valuable. It brings the "fix the benchmark labels" idea behind SWE-bench Verified and EvalPlus to CAD. If validated, the idea matters beyond CAD for code-generation benchmarks in general. This is the main reason I see the paper as worth revising rather than rejecting outright.

S2. A carefully designed, information-isolated audit protocol. The source context and the judgment context are kept separate. Constraints and judgments are frozen before K_core is revealed, and "once frozen, constraints cannot be deleted, weakened, split, or supplemented" (p.15), which limits post-hoc rationalization. S and C labels both require positive evidence, and the quantifier and numeric-tolerance rules are explicit (App. C.2). In principle, the audit can be checked independently.

Critical weaknesses

W1. The description-quality evidence comes from a closed, uncalibrated LLM loop. Every description-quality number comes from a single LLM auditor (GPT-6 Astra). The same model performs the blind reconstructions, and Codex 6 Astra is among the evaluated systems. The paper does not say which model rewrote the L1–L5 corpus or who performed the 38 revisions. The authors acknowledge that the estimates "have not been calibrated against a blinded, expert-adjudicated subset" (App. H). If the rewriting model shares the auditor's family, the auditor may simply prefer its own style, and the gap with CADPrompt may partly reflect a preference for LLM-written, precision-preserving specifications over human task requests .

W2. Pose invariance differs between validation and scoring. The audit ignores global translation and rotation (p.4), and reconstruction takes the best of 24 axis-aligned rotations (Eq. 8). The benchmark, however, scores in a fixed frame (p.23), while L4/L5 deliberately remove absolute placement. A "validated" L5 description therefore may not specify the coordinate frame that scoring depends on. On T2, delivery is 15.6% fixed-frame vs. 88.2% aligned. The drops across levels and the zig-zag curves may partly reflect unspecified pose rather than modeling ability. How translation is handled, and whether the voxel grid is shared or normalized per shape (which could hide scale errors), are not specified.

Questions for authors
  1. Which LLM rewrote the main L1–L5 corpus, and who (human or LLM, which model) carried out the 38 revisions? Are they in the same family as the auditor and reconstructor (GPT-6 Astra)?
  2. Please have at least 2 CAD-proficient annotators blindly label a stratified subset of at least 100 descriptions (CAD-Ladder L5 and CADPrompt) for S/C/U and K_core essentiality and retention. Report inter-annotator κ, the auditor's precision and recall on C labels, and SCR agreement. Also rerun Tables 1 and 3 with an auditor from a different model family. Does the CAD-Ladder vs. CADPrompt gap keep its direction under human judgment?
  3. Is voxelization done on a shared grid or a per-shape normalized grid? Does Eq. 8 also align translation? Do the L4/L5 texts specify a coordinate frame?
1 comment
AuthorsOpenReview ↗

Response to Reviewer eFjc

We sincerely thank you for your careful review and for recognizing the value of measuring text–geometry consistency. Your questions about model provenance, human calibration, and geometric scoring have helped us strengthen the evidence and clarify the evaluation protocol. We address each point below.

Q1 - Model provenance and transfer of revision gains

The main corpus used Claude Fable 5 for its LLM rewriting stages; GPT-5.6 Sol revised the 38 descriptions, and GPT-6 Astra performed the original semantic audit and CAD-Ladder blind reconstruction. Thus, the main rewriting model is from a different family, while the revision and original validation models share a family.

To test whether the revision benefit transfers beyond that family, we reconstructed the original and revised descriptions of the same 38 targets with DeepSeek-V4.1-Flash in DeepSeek Harness. Each description received one reconstruction in a separate context, with matched instructions, budgets, reference geometry, and the same normalized pose-aligned scorer.

The paired mean gain is +0.1556 IoU, with 25 targets improving, 10 unchanged, and 3 declining. This provides geometric evidence that the benefit of the revised descriptions transfers to another reconstruction family.

DescriptionTargetsMean IoUIoU ≥\geq 0.80
Original380.798425/38 (65.8%)
Revised380.954036/38 (94.7%)

Q2 - Human calibration and the cross-corpus difference

The difference retains its direction in the new human calibration when core retention is held fixed. Two CAD-proficient annotators reviewed 50 CAD-Ladder L5 and 50 CADPrompt descriptions. They evaluated the same 1,609 frozen atomic statements against source evidence, with the original auditor's S/C/U labels and rationales hidden. The fixed statements provide identical evaluation units for computing agreement and C-label precision/recall.

Three-class agreement between the two human annotators is Cohen's κ\kappa = 0.7888, with corpus-specific values of 0.7997 for CAD-Ladder and 0.7768 for CADPrompt. The original auditor's C-label results against each annotator are:

Human referenceC precisionC recall
Annotator A75.81% (47/62)54.65% (47/86)
Annotator B75.41% (46/61)59.74% (46/77)

On the 1,558 statements where both annotators agree in S/C, precision/recall is 77.59%/61.64%. The SCR values obtained using each annotator's S/C/U labels are:

CorpusOriginal auditorAnnotator AAnnotator B
CAD-Ladder L539/50 (78%)38/50 (76%)38/50 (76%)
CADPrompt21/50 (42%)21/50 (42%)21/50 (42%)

Although the auditor and human annotators differ on some atomic judgments, their description-level SCR decisions agree on 99 of 100 cases. SCR applies thresholds to conflict and core retention, so differences in individual atomic judgments need not change a description's pass/fail outcome. With core retention held fixed, both human annotations preserve the higher SCR of CAD-Ladder L5 relative to CADPrompt.

Q3 - Normalization alignment and coordinate frames

The implementation independently centers each complete shape and uniformly scales it by its own longest bounding-box extent before voxelization on a common 64$^3$ unit-cube lattice. The lattice is shared in normalized coordinates, rather than world coordinates. Translation is therefore removed by preprocessing, and uniform scale errors are also removed.

We would clarify the scope of Eq. 8 on page 4: it is introduced for reconstruction-based description validation, including the CAD-Ladder L5 and CADPrompt studies, rather than as the scoring definition for all subsequent T1–T5 system evaluations. In this evaluation pipeline, translation is already accounted for by per-shape centering before any rotation search. L4/L5 do not prescribe a unique global coordinate frame, so the reconstruction check does not assess reproduction of an unspecified global position.

The additional aligned result for T2 serves a specific diagnostic purpose. T2 includes global transformations relative to the reference program, so reporting fixed-orientation and pose-aligned scores for the same outputs distinguishes shape recovery from reference-relative orientation. The change from 15.6% to 88.2% shows how strongly the T2 score depends on orientation. Thus, translation is normalized out before voxel scoring, while the T2 before/after-alignment comparison deliberately exposes orientation sensitivity. These two uses are consistent with their different evaluation purposes: checking whether a description supports the intended geometry and diagnosing the behavior of a system on transformation tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.