- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper argues that point-cloud-to-CAD reconstruction is judged by the wrong criteria: programs are scored on whether they execute and produce a similar shape, yet a CAD program is only useful if its parameters are right, and the authors show these criteria diverge sharply — models whose programs almost always execute recover the exact ground-truth parameter set only rarely. To make parameter precision measurable, they introduce GeomCAD, a procedurally generated corpus of executable CadQuery programs balanced across four generative operations (extrude, revolve, sweep, loft), with every literal stored on an integer grid so that exact parameter match is well defined, and with a surface mix far richer than prior program-paired corpora. The accompanying GeomCAD-Eval protocol pairs exact and graded parameter recovery with standard shape-similarity metrics and a failure-mode decomposition of program validity. A reference model trained on the corpus (a CAD-Recode-style point-conditioned LLM) outperforms zero-shot and fine-tuned baselines on every operation, yet still recovers only a minority of ground-truth parameter values. The corpus, evaluation code, and checkpoints are released.
Critical strengths
- Making exact parameter recovery measurable, and showing that it diverges from the metrics the field actually reports. The integer-grid design is what makes PEM well defined at all — with float literals, any comparison would need a tolerance, as the Zero-to-CAD discussion notes. The headline observation, that models executing on over 90% of inputs match the full parameter set on almost none of them, is useful to have on record: it suggests that recent shape-metric progress says less about dimensional correctness than is commonly assumed, and it gives follow-up work a concrete target to optimize against.
- The corpus is large and more diverse than prior program-paired data, and the release engineering is careful. Four operations at roughly a million samples each, with about a third of faces neither planar nor cylindrical, should make it usable as training data and not only as a test set. On the release side — pinned OCCT container, mesh round-trip audit, split-versus-pool checks as a release gate, versioned scoring — the main claims can be recomputed by a reader rather than taken on trust, which is not something every benchmark paper manages.
Critical weaknesses
- Robustness to realistic input noise is never tested. All point clouds are sampled from ideal CAD surfaces. If the end goal is scanning a real object and recovering how it was made, the input comes from a scanner, and scanner point clouds always carry noise, outliers, and partial coverage. The evaluation never enters this regime: the encoder sees clean 8,192-point clouds, and no noisy variant of the splits exists. This matters more here than for shape-similarity benchmarks, because the paper's central claim is about precision at 0.1% grid resolution — a metric suite built around exactness is only convincing if the exactness survives realistic sensing conditions. Even a small noise-level sweep on a subset would substantially strengthen the paper.
- Every sample is a single-operation program, and it is unclear whether the evaluation survives composition. Real CAD programs are sequences of operations, and once operations compose, construction ambiguity grows quickly: materially different operation sequences can produce the same solid, so strict literal-sequence matching (PEM) — and to a lesser degree multiset recall (rPEM) — becomes increasingly ill-defined exactly where the task becomes realistic. The only real-data evidence in the paper is zero-shot transfer to DeepCAD, which is extrude-only and shares sketch provenance with the training pool ("only partly external", as the authors themselves put it). Whether GeomCAD-Eval remains meaningful for multi-operation reconstruction is an open question, and it bounds how far the benchmark's framing as a measure of progress toward useful CAD reconstruction can currently reach.
- PEM is meaningful only where the ground-truth program is identifiable from shape, and the paper's own results show this fails on a quarter of the benchmark: loft PEM is 0.0 even for CAD-Exact joint at TVF@5% 94.8, because free-form sections admit multiple valid parametrizations. The flagship metric therefore cannot distinguish "wrong parameters" from "a different but equally valid parametrization" exactly where the geometry is richest. The graded rPEM variants mitigate this, but the headline framing ("exact parameter recovery remains largely unsolved") still leans on strict exact match, and the abstract's 2.7% figure inherits the caveat.
Questions for authors
- Have you considered an equivalence-aware variant of PEM — e.g., crediting predictions whose executed solid matches GT within tolerance even when the literal sequence differs, or canonicalizing section parametrizations for loft? How would the headline ≤2.7% change under such a metric? This seems important for interpreting PEM as a measure of reconstruction quality rather than of parametrization luck.
- Do the Tier-1/Tier-3 picture and the executability–exactness gap survive realistic sensing conditions (noise, partial views, varying density)? Is a noisy variant of the splits planned?
- Real CAD programs are sequences of operations. Does the integer-grid PEM remain well defined under operation-order permutations once programs compose extrude/revolve/sweep/loft, and do you see GeomCAD extending to that setting?
Follow-up before discussion deadline
Dear Reviewer KAf1,
Thank you again for the careful review. Since our earlier replies, the revised PDF has added:
Noise-robustness sweep (§F.5, W1): a 500-sample Gaussian-noise sweep. Validity drops from 94.2% to 51.0%, aggregate TVF@5% from 73.6% to 7.6%, and strict PEM from 4/500 to 0/500; the rPEM margin over a shuffled-reference baseline remains positive across the sweep (+26.5 pp on clean inputs, +4.2 pp at σ = 0.02).
Multi-operation pilot (§F.6, W2/Q3): a 1,497-sample zero-shot pilot on two-type Boolean-union compositions. 76.6% of CAD-Exact (joint) predictions execute, aggregate TVF@5% is 22.4%, and 0.0% match the reference literal sequence; the executability–exactness gap persists under composition.
PEM-eq10 metric (§3.5, §E.1, Table 3, W3/Q1): a TVF-gated, order-invariant, tolerance-based literal recovery variant (±10 grid units) addressing your parametrization equivalence question. For CAD-Exact (joint), the TVF-gated recovery rate reaches 27.4% on loft and 21.5% on revolve.
If these additions address your concerns, we would be grateful if you would consider updating your assessment before the discussion deadline.
Respectfully,
The authors