- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Small changes in CAD construction sequence parameters can cause topological differences in the final model. When these changes cause holes not to pierce the intended faces, shells to become disjoint or the formation of internal voids, these errors are “deal breaker” for the usability of the geometry for downstream processes.
Topology Matters studies these deal breaking problems, comparing results from 5 well known CAD generation models. They discover that 10% of standard dataset used for academic CAD generation experiments (DeepCAD) cannot be rebuilt with the original topology, when parameters are quantized to 8 bits of precision (two or four times the resolution used in many academic papers).
Results are additionally reported stratified by a coarse complexity measure and construction sequence novelty. Finally prompts the effectiveness of prompts which request a change to the genus of the solid are investigated.
Critical strengths
It’s very interesting to see a paper which focuses on better metrics for existing works rather than just promoting a new architecture. The ML CAD community doesn’t do enough of this kind of thing.
Changes to the betti numbers of a solid indicate serious problems in the generation. Other topological comparisons like face and edge counts and Weisfeiler Lehman graph hash are too strict.
I found the section which compared the fidelity metric between novel and covered sequences interesting.
Critical weaknesses
The biggest weakness of the paper is the treatment of non-watertight generation results. The DeepCAD network is known to be dreadful for generating non-watertight geometry. . Admittedly auto-encoding is better than generation, but the authors still find more than 10% of generations are invalid. The way these invalid generations are combined with the other metrics completely changes the rankings, with DeepCAD scoring best on the Sig Fidelity metric and third best when invalidity is taken into account. I believe what is happening here is that the invalidity effectively filters the generated data distribution and reduces the number of complex bodies. This modified distribution is then much easier to achieve a high Sig Fidelity score on.
One way to compare across all methods is to consider only the set of examples for which all methods create watertight solids. This might restrict the number of complex solids too much, but it has the advantage that the remaining distribution is within the generation capability of all models.
184> We assign a topological signature only to watertight welded meshes; invalid or non-watertight samples are excluded from topology-based comparisons
I don’t feel this stressed how dependent the result are on how the non-watertight bodies are taken into account.
Many papers report a “valid ratio” which uses the Open Cascade function BRepCheck_Analyzer.IsValid(). This is rather stricter than watertightness of the output mesh. From the CAD perspective, if your solid doesn’t meet this bar then the data cannot be used. Looks like the authors didn’t use this function. The betti number results would be more convincing to me if used on generations which already meet the BRepCheck_Analyzer.IsValid() bar.
DeepCAD and SkexGen use latent vectors or codebook codes at the auto-encoder bottleneck. The other methods use text. Text to cad is much harder task. Appendix G makes this clear. It would be much better if this was in the main paper. It’s a detail which is critical to understanding the results in Section 4.1. If there isn’t space then make this clearer in Evaluated Models (line 162).
The authors use a complexity score based on betti numbers rather than face count. I think face count is a better measure of the perceived complexity of a B-Rep body.
The main goals of this paper are to convince the reader that metrics based on betti number are valuable. In my experience, the original DeepCAD network was considered to be a rather easy benchmark to beat, but these metrics imply it’s actually quite competitive. Given this disconnect between my personal experience and my “vibe checks” based on examining images of 1000s of DeepCAD generations, I am a little concerned that the new metric isn’t working as expected. I remain convinced that disjoint shells and incorrect genus are measuring real problems which real designers care about. Perhaps the problem is that all these generative algorithms are not product ready. It would be interesting to see if Astra hits this out of the park.
Questions for authors
Why evaluate the betti numbers from the mesh? You can extract the number of shells and lumps from the OCC B-Rep. Then use the Euler formular to find the genus. See
Partial entity structure: a compact non-manifold boundary representation based on partial topological entities, Sang-Hun Lee and Kunwoo Lee
for details of the extended Euler formula
3 comments
Thank you for the extra results
Thank you for the extra results. These are very interesting.
Evaluating the betti numbers from the mesh Thank you. Yes. This makes sense. Good to see the lump count is in agreement with your mesh based values.
Signature Fidelity on common valid subset
Thank you for the extra numbers. These clearly show that DeepCAD still does very well with the Signature Fidelity metric. Your experiment proves that the gains don't come from filtering more complex examples from the distribution.
I'm still really confused by the results though. My experience is that DeepCAD is a rather easy baseline to beat. Probably the mismatch is because most papers show DeepCAD generation, while your results are for auto-encoding. That would make the comparison with SkexGen sensible.
To compare correctly with the text based intermediate representations one would need to do something like converting to point cloud, creating the latent from the point cloud and then decoding. That's probably too much to ask for, although it might be an interesting experiment for you to try. I would expect to see the Signature Fidelity fall off a cliff when the DeepCAD latent isn't passed through directly.
I'm happy with the paper as long as it's made very clear that the DeepCAD and SkexGen results are auto-encoding and so can't be directly compared with the other methods.
I'm even surprised that DeepCAD beats SkexGen, but maybe it's because quantization of the codebooks introduce topological changes.
Further clarification of evaluation settings and an additional text-conditioned experiment
We sincerely thank you for this insightful comment. We agree that DeepCAD and SkexGen are evaluated under reconstruction-based settings, whereas Text2CAD, CAD-Coder, and Cadrille-SFT use text-conditioned generation. Specifically, the former receive ground-truth latent representations or shape codes, while the latter generate CAD from expert-level textual descriptions. Therefore, their absolute Signature Fidelity scores should not be directly compared as measures of generation capability.
Additionally, we would also like to mention that our goal is to provide a unified topological evaluation protocol across different CAD modeling paradigms, rather than comparing performance across different paradigms. We will clarify these distinctions in the revised manuscript.
Following your suggestion, we conducted an additional experiment using an alternative pipeline to further investigate this issue. We used Text2CAD to generate CAD command sequences from expert-level descriptions, converted these sequences into DeepCAD's input format, and then processed them using DeepCAD's official construction pipeline.
| Model / Pipeline | Signature Fidelity |
|---|---|
| DeepCAD (reconstruction-based) | 88.85% |
| Text2CAD (text-conditioned) | 80.37% |
| Text2CAD sequence + DeepCAD construction | 79.56% |
The alternative pipeline achieves 79.56%, substantially lower than DeepCAD's reconstruction result and close to Text2CAD's 80.37%. This further illustrates why reconstruction-based and text-conditioned results should be interpreted separately. We will make this distinction explicit throughout the revised manuscript.
We thank you again for this valuable suggestion. We hope our response addresses your concerns and clarifies the distinction between the two evaluation settings. We would be glad to provide further discussion or clarification if any questions remain.
Official Comment by Authors
We sincerely thank you for recognizing the importance of metric-focused research in the CAD generation community. We would like to address your questions individually as follows.
Q1: Why evaluate the betti numbers from the mesh?
A1: We emphasize that our goal is to evaluate global topological fidelity, rather than to advocate for a particular representation or extraction algorithm. Betti numbers characterize intrinsic topological properties and can be computed using either mesh-based or B-Rep-based representations. Thus, our mesh-based pipeline is an implementation choice.
Following your suggestion, we implemented an independent B-Rep-based extractor using OCC solid/lump information, shell classification, and the extended Euler–Poincaré relation. Across all 50,751 watertight-valid samples, the two approaches agree on 50,574 (99.65%).
This result demonstrates strong agreement between the two extraction approaches. We therefore retain the mesh-based pipeline, and use the B-Rep-based results as an independent cross-validation of our topological measurements.
W2: The generation paradigms of the evaluated models are not sufficiently clarified
A2: We sincerely appreciate the insightful suggestion and will add the corresponding explanation to the manuscript.
W3: Could you evaluate Betti fidelity using only generations that pass OpenCascade's BRepCheck_Analyzer.IsValid() check?
A3: Thanks for your suggestion, and we applied
BRepCheck_Analyzer.IsValid()as a stricter validity criterion and reevaluated the results. We report the individual coverage rates(valid rate) under the combined generation-success, watertightness, and OCC-validity criteria in the table below. To ensure a fair comparison, we evaluated all five models on their common valid subset, consisting of 5,106 out of 7,925 samples (64.43%). The results are as follows:These results show that DeepCAD retains the highest Signature Fidelity even under the stricter OCC validity criterion and on an identical evaluation subset. We will include these additional results in the revised manuscript.