- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates how to identify the root cause of computer-aided design (CAD) errors and guide the corresponding repair. The authors first propose a system to read the dependency graph before and after edits and utilize an LLM to generate an executable causal model. Then, counterfactual perturbations are performed to give explanations and validate them in the Fusion 360.
Critical strengths
- The research on CAD error location and interpretation in this paper is quite meaningful and has practical value.
- This paper is easy to follow.
- The author validated the proposed method in real Fusion 360, enabling verification of the explanations provided by the method.
Critical weaknesses
- The novelty of this paper is limited. Causal models and counterfactual perturbations have already been extensively studied, and beyond these established components, the paper offers no other substantive methodological or theoretical innovations.
- The experimental evaluation is limited and does not yet provide sufficiently convincing evidence of the proposed method’s effectiveness. In particular, the small number of baselines makes it difficult to establish its advantages over existing approaches.
- In Equation 7, if the perturbation of variable r causes changes in other root variables as well, then the resulting counterfactual explanation might be inaccurate. This is particularly important in identifying the root cause of CAD errors. The author did not clarify whether the mutual influence among the root variables was isolated when generating the counterfactual explanation.
- The description of the executable causal model is not clear enough. The differences between the model mentioned in this paper and the general structural causal model have not been clearly explained. In the "Executable causal models", the role of the intermediate variable I has not been demonstrated.
- The experimental setup is not clearly described. The criteria for selecting CAD error cases and the number of cases are not explicitly stated. Moreover, the existing experiments do not demonstrate the effectiveness of the verification feedback.
Questions for authors
- As the number of learning-acting cycles increases, both the search computation and validation feedback of the proposed method also increase. How to determine whether the performance improvement comes from the feedback mechanism?
- Could the authors briefly introduce the main types of CAD errors? This would help readers without domain expertise better understand the proposed approach.