- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposed a candidate-selection protocol, BasinMark, for structure-alone provenance auditing of AI generated crystals following relaxation and standardization. BasinMark selects generated candidates according to keyed structural fingerprints and audits sets of crystal structures by counting each unique fingerprint class only once.
The paper shows that exact same-basin relaxation may erase coordinate perturbations and removing cell-shape bins may improve fingerprint-label survival with no distortion of distinct class labels in two benchmark datasets. Under a key-independent null, the deduplicated class auditing provides proper false-positive control. In addition, collections with identical fingerprints remain indistinguishable, which allows watermark evidence to be copied or reused across datasets while limiting provenance claims to fingerprint-class resolution.
Critical strengths
Studying the sets of crystal structures after relaxation and crystallographic standardization, instead of using only the AI generator's raw output, is of practical relevance. Removing cell-shape bins substantially improves watermark survival through relaxation without distorting distinct fingerprint class labels. Proposition 2 explains why duplicate fingerprints do not provide fresh independent evidence under a key-independent collection and ideal random-function labels. The naive 6.4% rejection rate for WyckoffLLM-vanilla illustrates the practical consequences of ignoring collisions.
Critical weaknesses
The entirety of the proposed watermarking pipeline is not tested together. In effect, the paper uses separate experiments to evaluate fingerprint survival and detection, in which the joint post-deposition power remains unknown. Thus, the reported detection rates are not collectively reflective of the reliability of BasinMark in detecting watermarks after the full generation, selection, and deposition workflow. There lacks supportive evidence for the central practical claim in this paper.
One noticeable weakness is that there exist inconsistencies between implementation and simulation designs in empirical validation, ranging from the released code and available detailed results in GitHub. (i) The released reuse simulation uses the secret key to retain only green fingerprints before constructing the adversary’s observations. This differs from the stated threat model. The code is absent to generate results of attack rates reported in Figure 2e. (ii) The WyCryst negative controls available in the GitHub report 1% rejections for both unmarked and shuffled-key collections at (m=8) at a nominal 0.1% threshold, which contradicts to the statement that unmarked-key false-positive rates remain at most 0.5%. (iii) Executing the released function on seven equally populated classes returns 7.75, which exceeds the theoretical bound of 7. This is an inconsistent reporting with respect to the bound . This mismatch complicates the interpretation of the diversity results.
Questions for authors
-
Algorithm 1 relaxes candidates before watermark selection, while the survival experiments examine fingerprint changes across relaxation, and the limitations section states that joint post-deposition power remains unmeasured. Such mixed messages make the paper hard to follow; it is unclear whether the reported detection rates validate fully the intended workflow. Confusion arises: What processing occurs after selection, and which results demonstrate detection after that processing?
-
The released reuse routine uses the secret key to retain only green fingerprints before constructing the observations. Genuine selected outputs can include non-green fallback selections, so this filtering evaluates an easier attack than observing actual outputs without the key. Which script produced Figure 2e? Did those experiments retain fallback outputs without filtering observations by their true green status? The original script and per-trial results are absent so it is impossible to verify the reported attack rates.
-
The WyCryst results available in GitHub show 2/200 (1%) rejections for both unmarked and shuffled-key controls at (m=8) whereas S2.7 states that unmarked-key rates remain at most 0.5%. Such summary available in the GitHub discounts the anomaly using a pooled multiple-testing correction, which in fact addresses a different question from per-audit false-positive calibration. How do you reconcile these results? What calibration evidence accounts for the reused keys and overlapping collections?
-
Proposition 2 defines , but the implementation reports a bias-corrected population collision estimator. For seven equally populated classes, it returns 7.75 exceeding the theoretical bound of 7 from if deemed as the quantity from Proposition 2. Which quantity is intended in the empirical results? Would using the proposition’s definition change any diversity or feasibility conclusions?
5 comments
The revision provides additional evidence on detection after downstream processing. However, the revised framework makes substantive changes to the original selection procedure, and the motivation and consequences of these changes need further clarification.
The original method selects the lowest-energy green candidate within each batch, falling back to the batch minimum only when no green candidate exists. The revision introduces an energy cap, which can exclude existing green candidates, and then introduces pooled selection across batches. Adding independent downstream processing does not itself require either change. The revision lacks the evidence for the unrestricted selector alongside the capped and pooled selectors, and insufficient explanation of the detection–quality trade-off that motivates each change.
Another weakness concerns the implementation of pooled selection. There is a clear discrepancy between the revised pooled selection (selecting distinct green fingerprint classes), and the implementation (selecting individual candidate records). Selecting candidates is consistent with the original method, and the final audit deduplicates their fingerprints. The revision is unclear on whether the pooled-selection description assumes distinct candidate fingerprints and how duplicate classes are handled when that assumption does not hold.
In conclusion, the fundamental weakness of the proposed method is not well addressed.
We thank the reviewer for the response. The matched evidence below addresses both the unrestricted baseline and whether the record/class discrepancy affects the reported experiments.
Independent downstream processing does not require an energy cap or pooling. These are additional design choices motivated by selection quality: the cap limits selection energy, while pooling allows interchangeable output slots to share candidates under energy constraints. Their consequences should be evaluated against the original unrestricted rule.
The following archived comparison holds the no-cell fingerprint, marking probability gamma = 1/2, candidate draws, keys, downstream processing, and deduplicated binomial auditor fixed, with nominal alpha = 0.001. Each trial selects 32 structures from 256 candidates. The downstream procedure independently re-relaxes the selected coordinates with CHGNet-r2SCAN, standardizes the primitive cell, and writes and rereads a five-decimal CIF. Using gamma = 1/2 throughout isolates the selection changes from the revised default gamma = 5/8.
| Generator | Selector | Detection after processing | Excess energy (meV/atom) | Stable yield |
|---|---|---|---|---|
| MatterGen | Original unrestricted fixed batches | 500/500 | 32.1 | 74.0% |
| MatterGen | Fixed capped batches | 491/500 | 21.8 | 77.0% |
| MatterGen | Pooled, slot and mean caps | 500/500 | 16.2 | 86.3% |
| CrystalFormer | Original unrestricted fixed batches | 500/500 | 400.0 | 22.7% |
| CrystalFormer | Fixed capped batches | 12/500 | 56.7 | 31.8% |
| CrystalFormer | Pooled, slot and mean caps | 455/500 | 49.3 | 30.7% |
"Original" denotes the submitted unrestricted selection rule, evaluated with the same no-cell fingerprint as the other rows to isolate selection effects. Excess energy is the producer ranking energy relative to the cheapest 32 candidates in each trial. Stable yield uses archived DFT hull energy <=0.10 eV/atom for MatterGen and archived phonon-stability labels for CrystalFormer, divided by all emissions. These source/reference labels are withheld during selection; they are not new stability tests of the deposited structures. The 500 trials vary keys and candidate draws within each fixed cohort.
The original selection rule retains detection under this tested processing: 500/500 for both generators. The CrystalFormer 12/500 result belongs to the capped variant. Capping reduces excess energy from 400.0 to 56.7 meV/atom but sharply reduces detection; pooling recovers 455/500 detections at 49.3 meV/atom. Compared with unrestricted selection, pooling sacrifices detection while reducing ranking energy and increasing source-label stable yield from 22.7% to 30.7%. Compared with capped selection, stable yield decreases from 31.8% to 30.7%. These results support a measured detection-quality trade-off, not a uniform improvement across all metrics. We will add the unrestricted baseline and this controlled comparison to the manuscript.
Regarding duplicate handling, the implemented pooled selector maximizes green candidate-record count subject to its energy constraints. For each proposed count, it considers the cheapest required green records and non-green records. It does not enforce distinct fingerprint classes during selection. We will correct the manuscript's use of "classes" in describing this operation.
We checked both candidate cohorts underlying the table: each contains 384 records with 384 distinct producer fingerprints. Thus, the record/class distinction does not change the selections in these experiments. For general inputs with duplicate classes, multiple selected records may share a producer fingerprint. The final audit always counts distinct deposited fingerprints once; repeated deposited fingerprints provide no additional class evidence. Maximizing green records does not guarantee distinct output fingerprints or maximize distinct green classes. Any interpretation of green-emission counts as distinct-class evidence must state the relevant distinctness assumption.
The evidence therefore separates the original selector's downstream survival from the quality constraints introduced by the extensions, and establishes the scope of the implementation discrepancy. We will make the record-level selection objective and its relationship to class-level evidence explicit.
We respectfully ask you to reconsider the soundness assessment and overall score in light of this controlled comparison and the verified duplicate analysis. If you consider the fundamental weakness unresolved, please identify the specific remaining claim or missing comparison so that we can address it directly.
Thank you for checking the implementation against the paper. Each of the three discrepancies the reviewer found has a concrete fix in the revised code, data and text, and the end-to-end question now has a measured answer.
In W1 and Q1 the reviewer was concerned that the submission measured survival and detection separately, so it was unclear what processing happens after selection and which results show detection after it. The revision measures them together. Selection uses producer coordinates; the auditor sees only independently processed deposits. Section 4.4 and Table 1 report that chain: the producer selects 32 of 256 relaxed candidates, a depositor independently re-relaxes those coordinates (variable-cell CHGNet-r2SCAN), standardizes the primitive cell and writes a five-decimal CIF, and the auditor reads the files. Detection is 499 of 500 on MatterGen and 476 of 500 on CrystalFormer, against 491 and 12 for fixed bounded batches. On two held-out CrystalFormer cohorts that share no classes with earlier data, under a frozen protocol, ordinary processing gives 498 of 500 on both. A 1% strain before re-relaxation gives 492 and 493, and 496 and 497 with an optional producer preview. Every result in the paper is labeled by its input geometry and endpoint. The archived MatterGen DFT comparison (495 of 500) is labeled on its own because the starting coordinates of those DFT runs were not recorded.
In W2(i) and Q2 the reviewer found that the released reuse routine used the secret key to keep only green fingerprints before building the observer's inputs, an easier attack than the stated threat model, and that the script and records behind Figure 2e were missing. Both are fixed. The filtering routine is gone. The replacement driver gives the observer every emitted record in order, including non-green fallbacks, and only its public fingerprint. Only the producer and the auditor get the key. At ten observations the observer saw 2,000 emissions per generator, with six MatterGen and three CrystalFormer fallbacks, each in a different trial. Replay-allowed collections rejected in 194 and 197 of 200 trials, and the nine fallback trials account for all nine misses. Excluding exposed record IDs gives 0 of 200 for both. Figure 2e and Table 8 are rebuilt from these trials. The per-trial observation records (ordered IDs, fallback positions, decisions) and the plotting script are in the anonymous archive.
W2(ii) and Q3 pointed out that the WyCryst controls in the repository reject 2 of 200 (1.0%) at m = 8 while Appendix S2.7 stated at most 0.5%, and that the pooled multiple-testing correction answers a different question than per-audit calibration. We agree on both points. The text now reports the 2 of 200 counts for each control, and we no longer use pooled corrections to describe per-audit calibration. Fresh controls that hold the batches fixed and draw two new keys per trial give 15 of 20,000 for unmarked minimum-energy selection and 10 of 20,000 for an independent key on marked outputs, against an attainable discrete null of 0.054% at the nominal 0.1% level. Where keys or collections are shared, the paper states the pairing and reports uncertainty over the independent units (Appendix S2.7).
In W2(iii) and Q4 the reviewer noticed that the released function returns 7.75 for seven equally populated classes, above the bound n_eff <= K = 7 of Proposition 3 (Proposition 2 in the submission). The function returned a bias-corrected population estimator under that name. The API now reports the observed quantity of Proposition 3 as n_eff and the population estimator under a separate name, with regression tests for balanced and duplicate-heavy samples. The recomputed observed values are 1,490.1 for 1,495 MatterGen classes and 981.1 for 985 CrystalFormer classes, so the diversity conclusions stand. Rejection decisions use the distinct-class count K and were never affected. The seven-class example now sits next to Proposition 3.
Because the selectors of Section 4.4 differ from the fixed batches of Algorithm 1, Appendix S3.2 adds Proposition A, which writes the fixed-batch green count as a Poisson-binomial over the affordable classes in each block. With no fitted parameter it predicts producer-stage power of 3.8%, 1.9%, 3.2% and 98.3% on four archived cohorts, against observed 2.8%, 1.6%, 4.4% and 99.0%. Proposition B covers the pooled selector; its supply condition held in all 2,000 trials. Proposition C states the two-view audit calibration with a proof, checked against exhaustive enumeration in the test suite.
The reviewer's code checks made the empirical claims easier to verify. Every discrepancy identified is resolved in the released code and data, and the main claim now has a direct end-to-end measurement. We believe the revision answers each point with measurements in the paper, and we respectfully ask the reviewer to reconsider overall score by increasing.