- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This submission introduces PhysDefect, a benchmark for 3D anomaly localization that combines physics-based defect synthesis with a projector-stereo scanner simulator. The benchmark models seventeen mechanisms spanning deformation, removal, and addition, and evaluates point-cloud methods across three scanning conditions and several normal-reference-bank sizes. The paper also uses a paired simulation of native Anomaly-ShapeNet defects to study the effect of observation changes.
Critical strengths
- Well-motivated benchmark direction. The motivation that existing synthetic 3D anomalies are predominantly local and clean is compelling. Treating defect formation and the acquisition process as connected sources of variation is a useful framing.
- Broad and coherent synthesis design. The three families and seventeen mechanisms cover a wider range of morphology than standard local displacement pipelines. Carrying geometry and material state between events is an interesting capability, especially for stress-corrosion-type chains.
- Useful controlled-observation protocol. The easy, medium, and hard scans and the paired ShapeNet study make it possible to study how acquisition changes interact with method behavior rather than treating the point cloud as a noise-free input.
- Promising reproducibility practices. The manuscript specifies random seeds, generation records, scanner parameters, annotations, and several numerical checks.
Critical weaknesses
- Physical realism is asserted but not externally validated. Table 7 establishes internal numerical behavior (for example, a finer-grid pitting reference and an analytical capillary corner), not whether the generated morphology, severity, and co-occurrence of mechanisms resemble industrial defects. Similarly, scanner calibration relies on three aggregate statistics from released point clouds. The paper needs evaluation on measured scans with known defects, or at minimum a carefully documented comparison of real and synthetic defect morphology, size, localization, and scanner statistics. A train-on-PhysDefect, test-on-real transfer experiment would directly support the practical premise.
- The evaluation metric does not measure the full detection problem. The main P-AUROC is computed only for scans with both point labels, and undefined or failed predictions are omitted. Artifacts, outliers, and mixed pixels remain model inputs but are excluded from scoring, while the 400 normal test scans per level are not used to assess object-level false alarms. This can substantially overstate robustness when defects are occluded or acquisition artifacts are common. Please report object-level AUROC and false-positive metrics on normal objects, a localization metric such as AU-PRO, and per-mechanism/per-level visibility, exclusion, and failure rates.
- Baseline comparisons are not controlled tightly enough for a benchmark claim. The compared rows differ in more than the underlying method: point counts, pretraining, modality selection, normalization, and registration procedures vary; Simple3D is only evaluated at K = 1. The appendix explicitly notes that feature and registration effects are entangled. Consequently, Table 4 cannot isolate detector quality, and the strong PTP baseline may partly reflect the chosen alignment protocol. Please provide a standardized input and registration pipeline where possible, retain released method settings where not, and report sensitivity to alignment quality and random seeds.
- The paper lacks ablations that attribute gains or difficulty to the proposed physical components. The three scan levels jointly vary coverage, stereo visibility, noise, multipath, and post-processing. Likewise, the paper does not compare its physics-based synthesis with analytic or procedural synthesis while matching geometry, defect area, and severity. Controlled factorial ablations are needed to show which scanner components and which physical mechanisms make the benchmark harder or more realistic, rather than simply different.
- Dataset composition and aggregation need clearer reporting. The 100 evaluation geometries are selected from 34,372 eligible CAD meshes, but the manuscript does not provide the exact category split, selection protocol, mechanism-wise sample counts, or severity distributions in the main paper. The metric macro-averages only defined geometry-mechanism cells, and Table 11 shows that effective sample sets differ by method. Please release and summarize these distributions, report uncertainty intervals, and show results under a common valid-scan subset or an explicit treatment of missing and failed cases.
- Some quantitative details appear inconsistent or insufficiently explained. Table 2 lists the medium local thickness ratio as 0.028, whereas Table 10 lists 0.057 for the medium configuration. If these values use different geometry sets, point samplings, or processing stages, that distinction should be explicit. The definition and selection of the label threshold tau and the severity controller should also be made reproducible for each mechanism.
Questions for authors
- What measured-defect or real-scan evidence supports the claim that the simulator closes a meaningful sim-to-real gap?
- How do the conclusions change when all normal test scans and scanner artifacts are included in a false-positive evaluation?
- Can the authors provide a severity-matched comparison with at least one existing geometric synthesis method and isolate scanner components one at a time?
- Why do the reported medium thickness ratios differ between Tables 2 and 10, and what exact data subset and processing stage does each table use?
3 comments
Response to Reviewer YwyD(2)
Point 3: Baseline configurations, reference-bank size, and alignment sensitivity
We thank the reviewer for highlighting the need to clarify baseline configurations, reference-bank sizes, and registration conditions. We will report each method’s input point count, pretraining source, input modalities, normalization, and registration settings so that the complete configurations underlying the results can be inspected. For Simple3D, we added evaluations at K=1, 4, and 16, keeping the test scans, registration cache, feature extraction settings, and 5% coreset sampling ratio fixed, with nested sets of normal references. On the common scorable subset of 3,101 anomalous scans spanning 98 CAD geometries in the medium tier, CAD-macro P-AUROC is 0.7187, 0.7456, and 0.7602, respectively. Increasing K from 1 to 16 improves P-AUROC by 4.15 percentage points, with a CAD-clustered bootstrap 95% confidence interval of [3.57, 4.79] percentage points. Regarding registration, the main evaluation applies independent random rotations and translations to normal references and test scans, then estimates alignment using only the observed point clouds; ground-truth transformations are not supplied to the detectors. Measurement noise, acquisition artifacts, and surface-processing errors are introduced during scan generation to simulate practical acquisition conditions. To quantify sensitivity to residual alignment errors, we additionally selected 16 CAD geometries, retained their scanning errors, and used the simulator’s known common coordinate frame as a diagnostic reference. We then applied controlled rotational or translational offsets to the test clouds and scored them without re-registration. On the common scorable subset of 497 anomalous scans in the medium tier, PTP achieves a CAD-macro P-AUROC of 0.8074 after estimated registration and 0.8333 in the known common coordinate frame. Adding rotational offsets of 1° and 5° yields 0.7362 and 0.5657, respectively; a translational offset of 0.5% of the reference cloud’s bounding-box diagonal yields 0.6587. These results quantify the effects of reference-bank size on Simple3D and alignment quality on PTP. The supplementary material will provide complete results across K values, valid-sample and failure counts, and alignment-sensitivity curves.
Point 4: Controlled synthesis and scanner ablations
We thank the reviewer for highlighting the need for controlled ablations of physical synthesis and scanning components. We added two experiments. First, we compared PhysDefect and AF3AD on the same CAD geometries under ideal observation and medium simulated scanning, covering 99 CAD geometries, 17 mechanisms, 3,053 matched pairs, and 11 detector configurations. We matched total vertex displacement for 1,300 topology-preserving pairs and a surface-deviation proxy for 1,753 material-addition or removal pairs, with relative budget differences within 10%. An additional analysis constrained the visible anomalous-point fraction and defect depth. For impact defects, PhysDefect yields P-AUROCs averaging 6.27 and 5.80 percentage points below AF3AD under ideal observation and simulated scanning, respectively. After additionally constraining defect extent and depth, 9/11 configurations retain this direction under both observation conditions. For bending defects, P-AUROC instead increases by an average of 1.55 and 0.71 percentage points, showing that detection difficulty depends on the mechanism. Second, we fixed defect geometry across 8 CAD geometries and 3 mechanisms, separately disabling measurement noise, artifacts, or surface reconstruction, or halving the scanning viewpoints. For example, disabling measurement noise changes the P-AUROCs of PTP, PatchCore-FPFH, and Simple3D by −0.40, +1.53, and +2.68 percentage points, respectively. The CAD-clustered bootstrap 95% confidence intervals for all 12 scanning-component differences include zero, so these results do not establish a consistent directional effect of any component across all algorithms. These controlled comparisons nevertheless quantify sensitivity to individual perturbations, informing algorithm selection and validation under practical acquisition conditions. The shared supplementary material provides examples across the four synthesis-by-observation conditions and individual scanning interventions, together with results by mechanism and confidence intervals.
Response to Reviewer YwyD(3)
Point 5: Dataset composition and statistical aggregation
We thank the reviewer for highlighting the need to clarify dataset composition and statistical aggregation. We compiled identifiers, sources, and categories for all 100 CAD geometries, together with sample counts and realized severity distributions for all 17 mechanisms. The geometries comprise 34 models from MCB-A, 33 from MCB-B, and 33 from SimJEB, retaining 39 original category labels. Selection first filters for watertight, SDF-compatible models with at most 200,000 triangular faces, then applies deterministic farthest-point selection over geometric descriptors within each source to cover diverse shapes. The dataset contains 3,213 anomalous and 2,000 normal instances, with 148–198 anomalous instances per mechanism. Severity statistics are computed from the final generated outputs: we report the median and 10th/90th percentiles of maximum geometric deviation for each mechanism, alongside anomalous vertex fractions and, where applicable, anomalous surface-area fractions. For example, the median anomalous vertex fraction is 2.65% for impact defects and 41.74% for vibration-induced deformation, reflecting the distinction between localized defects and extensive deformation. To address differences in valid samples across methods, we added comparisons on common scorable subsets containing 3,111, 3,078, and 3,083 scans in the easy, medium, and hard tiers, respectively, each covering 99 CAD geometries. On the medium common subset, PTP achieves a P-AUROC of 0.7958 with a CAD-clustered bootstrap 95% confidence interval of [0.7781, 0.8135], while Simple3D achieves 0.7252 with an interval of [0.7040, 0.7467]. The shared supplementary material provides category and mechanism distributions, severity distributions, the complete asset list, and valid-sample, exclusion, and failure counts for each method, making dataset composition, aggregation denominators, and uncertainty directly inspectable.
Point 6: Thickness statistics and reproducible annotation
We thank the reviewer for identifying the inconsistency in the thickness statistics and requesting clearer annotation rules. Our audit found that the values of 0.028 in Table 2 and 0.057 in Table 10 lack records that uniquely identify their input samples and processing stages. We will therefore remove these unverifiable values from the revision and explicitly distinguish high-density scan calibration from diagnostics on downsampled detector inputs. For the latter, we added reproducible statistics on the same set of 32 normal scans spanning 8 CAD geometries. Each cloud contains 4,096 points. We estimate the local standard deviation along the surface normal using 24-point neighborhoods, take its median over the flattest 20% of regions, and divide by the cloud’s median nearest-neighbor distance. The median local thickness ratios are 0.193160 for the full scanning pipeline and 0.193223 with surface reconstruction disabled. The input manifest, random seed, and computation code accompany the supplementary material. For annotation, the geometric anomaly score is the absolute signed distance from the generated surface to the normal reference surface in normalized coordinates, with a shared threshold of τ=0.002; a positive binary label additionally requires an attributable defect-event contribution at that location. The nominal severity targets are 0.006–0.06 for maximum deviation and 0.0005–0.30 for anomalous fraction. We record each mechanism’s fraction definition, parameters, retry count, and acceptance outcome for every sample. Controller-stage measurements and final geometric annotation statistics are reported separately so that the target criteria and realized output distributions can be independently verified.
Response to Reviewer YwyD(1)
Point 1: Evidence from measured defects and real-data evaluation
We thank the reviewer for highlighting the need to validate PhysDefect against measured defects and real detection outcomes. We added two experiments: fitting measured defect geometry and evaluating synthetic supervision on real data. First, we impacted a 100 × 100 × 0.1 mm copper foil and a 40 × 40 × 40 mm aluminum industrial heat sink with a 28 mm solid steel ball, then captured the defects using a FreeScan Combo 3D scanner. With fixed defect locations and equal numbers of candidate evaluations, we compared PhysDefect with AF3AD, NormAS, PA3AD, and GLFM, measuring point-to-surface RMSE on held-out scan points excluded from fitting. For the isolated copper-foil dent, RMSEs ranged from 0.024 to 0.038 mm, with PhysDefect achieving 0.0323 mm. For the heat sink, PhysDefect achieved the lowest RMSE of 0.4131 mm, reducing error by 13.5% relative to the best competing method under unconstrained displacement and by 26.1% under matched total vertex displacement. Second, we generated training anomalies from 48 normal templates using four physical material-removal mechanisms and evaluated on 1,206 real Real3D-AD test samples, keeping the DGCNN architecture, training budget, and three random seeds fixed across conditions. Compared with a normal-only replay control, micro P-AUROC increased from 0.5003 ± 0.0245 to 0.5995 ± 0.0149, where ± denotes the standard deviation across seeds. Real anomalous test samples were not used for parameter selection. These experiments provide evidence of fitting measured defect morphology and improving real point-level localization through synthetic supervision, respectively. The supplementary material presents photographs, scan overlays from identical viewpoints, copper-foil cross sections, and complete fitting and transfer results.
Point 2: Detection metrics, coverage, and failures
We thank the reviewer for highlighting the importance of false alarms on normal objects in practical inspection. The original P-AUROC measures defect localization within a scan; distinguishing normal from defective objects requires a separate evaluation. We added O-AUROC using the 400 normal scans in each scanning tier, defining the object score as the 99th percentile of anomaly scores over all input points, including acquisition artifacts. For fixed-threshold evaluation, we set the threshold to the 95th percentile of object scores from 80 normal scans spanning 20 CAD geometries, then evaluated normal and anomalous scans from the remaining 80 geometries. In the medium tier, PTP and Simple3D achieve O-AUROCs of 0.6483 and 0.6689, respectively, whereas their P-AUROCs on the common valid subset are 0.7958 and 0.7252, reversing their relative ranking. At the threshold calibrated on normal scans, PTP has a normal-object false-positive rate of 4.69% and an anomalous-object detection rate of 5.05%. These results demonstrate that localizing observed defects and identifying defective objects are distinct evaluation objectives. We therefore report O-AUROC, normal-object false-positive rates, and the corresponding detection rates alongside P-AUROC. We also added AUPRO with an FPR integration limit of 0.3: PTP and Simple3D achieve 0.4722 and 0.4045, respectively, in the medium tier. Fully unobservable defects account for 0.68%, 0.90%, and 1.40% of anomalous samples in the easy, medium, and hard tiers, respectively. Among the 3,613 test scans in the hard tier, PTP and Simple3D have 38 and 52 inference failures, respectively. The shared supplementary material provides complete results across methods and scanning tiers, statistics on unobservable defects, unscorable cases, and failures by mechanism and tier, and comparisons on the common scorable subset with confidence intervals.