- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes an analysis to isolate the effect of disclosing the "intent" behind a CAD design in a CAD prompt. To do so they build a dataset with 223k CAD shapes labeled by gpt-5.2, requiring it to include the "intent" in the prompt, and compare the finetuning of a 4B LLM on such data vs data without "intent" and claim to observe that the one fine tuned with the intent actually achieves better accuracy on test data with no intent in the prompts, and that the resulting model has a lower invalidity ratio than four general-purpose LLMs.
Critical strengths
The paper poses a reasonable and interesting hypothesis: that adding intent in the training data of a generative CAD model is a substantial information addition that can improve CAD generation capabilities, even when test prompts do not contain it. Characterizing the properties that improve the quality of CAD training data is an effective strategy to improve CAD models' generative power. If the hypothesis is confirmed empirically this would represent a meaningful contribution to the AI for CAD literature.
Critical weaknesses
The paper does not isolate what it claims to isolate.
- "Program normalization" and "description normalization" are never defined and it is hard to understand what their method is comparing to. Also, it is not clear how the unnormalized programs and descriptions are built for shapes without a construction history.
- It is not clear how the descriptions without intent are built, since the prompts in the Appendix always generate the intent paragraphs.
- Confounders are not isolated: the proposed re-annotation changes several things together and the claimed gains might come from the annotator, the length of the annotations, the fact that there are more geometrical details, or the fact that the descriptions copy the style of the Text2CAD-Bench prompts. Indeed, by construction of the prompt, the "design intent" paragraph often restates geometry.
This is important because the effect of the intent is the main claim of the paper, and the current setup does not allow attributing the gains to it.
Anyways, the empirical evidence is very weak.
- No variance or confidence intervals, while some gains are very tiny and possibly not significant.
- A usual failure mode in CAD evals is that metrics are computed and averaged over valid outputs, which might surface a bias: a variant can look better if it fails on harder parts. This happens here too, e.g., adding intent raises the invalidity ratio on L1 Seq while lowering CD.
- I am afraid some 3D shapes might live in the intersection between the training and test data, and no deduplication against Text2CAD-Bench is reported.
- The evaluation setup employed in Table 3 for the specialized text-to-CAD models is very different from the one used to evaluate IntentCAD, since their numbers are copied from Wang et al.. Also, the LLMs are allowed up to three retries and it is not clear if IntentCAD is too.
- There is no comparison with the same model fine-tuned on existing datasets (e.g., CADFS or Text-to-CadQuery) with a matched budget.
- Results are restricted to one single model and to the two easiest tiers of Text2CAD-Bench.
- In general, the claimed wins are far from being evident and are, at least, overstated: even under their own protocol, IntentCAD ranks 2nd to 4th on CD and IoU on 3 of 4 splits.
This is important because, even setting aside the first point, the results do not support the claims in the abstract.
Questions for authors
My main concerns require new experiments and a substantial rewrite, so I do not expect a rebuttal to be enough to change my recommendation. Still, the following questions could affect my assessment.
Clarifications
- Are you positive you are not training on shapes present in Text2CAD-Bench? Did you run any deduplication (exact or near-duplicate) between your training sources and the benchmark?
- How are the descriptions without intent built? With a separate prompt, or by dropping the first two paragraphs of the descriptions with intent?
- How are the unnormalized programs and descriptions built for shapes without a construction history (e.g., ABC or the industrial parts)?
- Some invalidity ratios are not multiples of 0.5, which they should be with 200 prompts per split: the unnormalized row in Table 2 and most LLM rows in Table 3. How were they computed?
- Does IntentCAD also get up to three retries, like the LLMs in Table 3? Which decoding settings do you use?
- Regarding the human reviews, how many regeneration rounds did you perform, how large was each sample, and what was the error rate?
- Why only the L1 and L2 tiers of Text2CAD-Bench? L3 also includes parts with complex topology, which seems in scope for your claims.
- Will the dataset be released, and how many of the 223K parts come from private sources?
Missing experiments
These are, in my opinion, the minimum required to support the claims of the paper:
- Ablations to isolate the impact of including "intent" in the prompt, e.g., replacing the intent paragraph with a neutral or paraphrased paragraph of the same length.
- Confidence intervals over multiple seeds, and metrics that count invalid outputs as failures instead of averaging over valid outputs only.
- A comparison with the same model fine-tuned on existing datasets with a comparable training budget.