- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposed MARCO to embed watermarks into protein structures using an encoder and decoder while keeping the underlying protein model frozen. It combines geometric losses, adversarial training, and simulated attacks to preserve structure quality and watermark recovery. Experiments cover six protein models, several removal attacks, and transfer to models fine-tuned on watermarked data. The paper also claims certified robustness and reliable source attribution.
Critical strengths
Not changing the base model is a useful property. The distance and torsion losses provide a concrete way to account for protein geometry.
Critical weaknesses
-
The proof is for a different decoder from the one used in the experiments. Appendix B, Eq. (21), defines g by adding noise to the input, running decoder D, and taking a majority vote. Theorem 1 proves that g gives the same answer after small changes to the input. But Algorithm 2 and Section 4.3 use D directly. D can give a wrong answer even when the majority vote is correct. Training D with noise does not close this gap. Figure 11 tests random noise, which also cannot show that the decoder works for every allowed change. The claimed guarantee therefore does not follow for the method being evaluated.
-
The claimed 0.1% false-positive bound is wrong even under the paper's own assumptions. Section 4.3 says that its detection rule, z >= 3, gives a false-positive rate no higher than 0.1%. With 64 bits, this rule accepts at least 44 matching bits. Under the assumed independent random matches, the false-positive rate is actually about 0.1845%, almost twice the claimed limit. Meeting the 0.1% limit under this assumption requires at least 45 matches. Every result with exactly 44 matches would then change from a successful detection to a failed detection. The paper does not report how many results would change, so its detection rates do not show how well MARCO works at the claimed 0.1% false-positive limit. There is also no evidence that the independent random-match assumption holds for the trained decoder on clean structures. Table 7's small negative control cannot establish that limit: even zero errors in 100 samples gives a 95% confidence upper limit of about 2.95%.
-
The tests do not show that every model trained on watermarked data will keep the watermark. The abstract refers to any such model, and Appendix A, Eq. (19), requires every output to carry the watermark. Section D.2 tests selected fine-tuning settings with three source and three student architectures. These results cannot establish a claim about every model and every output. Even a student that closely matches the training distribution could occasionally produce an output without a detectable watermark. The paper gives no argument that rules this out. Its main claim about watermark transfer needs to be limited to what the evidence supports.
-
Eq. (20) cannot hold together with the claimed reliable detection for a fixed watermark. The equation requires every classifier to have nearly the same average output on clean and watermarked structures. Now choose a classifier that returns 1 when MARCO detects the watermark and 0 otherwise. Reliable detection means its average output is near 1 on watermarked structures and near 0 on clean structures. Their difference is therefore near 1, although Eq. (20) requires it to be near 0. Restricting the claim to attackers without the watermark or decoder could avoid this conflict, but the equation contains no such restriction. The lower Frechet distance in Section D.4 only shows improvement in that metric. It does not show that classifiers cannot distinguish the structures. Thus, the paper's definition of covertness is incompatible with its detection goal as written.
-
The paper dismisses attacks that remove the watermark without showing that they destroy the proteins' usefulness. Section D.1 acknowledges that strong scrubbing and DP-SGD can weaken or remove the watermark, but calls the resulting models “functionally unusable” because structural and sequence similarity drop. Higher RMSD and lower BLOSUM62 scores show that outputs differ more from the reference. They do not establish that the proteins can no longer perform their intended function. Under Section 3, a successful attack must evade detection while preserving both structural integrity and biological function. The reported similarity losses are relevant evidence of reduced fidelity, but the paper does not explain which task-relevant utility criteria or thresholds they violate. It therefore remains unclear whether these attacks fail the stated utility-preservation requirement. The authors should justify the “functionally unusable” conclusion using such criteria and, where necessary, additional validation, or limit the conclusion to the measured degradation in structural and sequence fidelity.
Questions for authors
- Do the experiments use D directly or take a majority vote as in g? If they use g, how many noise samples are used, and how is uncertainty in the vote accounted for?
- What are the detection rates when the required number of matching bits increases from 44 to 45? What evidence from clean structures supports the claimed false-positive rate of at most 0.1%?
- What task-relevant utility criteria support the conclusion that the attacked models are functionally unusable, and how do the reported RMSD and BLOSUM62 values relate to those criteria?
2 comments
Thank you for addressing some of my concerns. However, the incorrect FPR bound, missing detection rates at the corrected threshold, and insufficient evidence for the claimed loss of biological utility remain unresolved. The proposed changes to the claims are also not reflected in a revised manuscript. I therefore maintain my original score.
Thank you for recognizing the value of keeping the base PGM frozen and incorporating distance and torsion constraints. We clarify the implementation, theoretical scope, and empirical claims below.
1. Certified robustness and the deployed decoder
The experiments and Algorithm 2 use the base decoder D. Appendix B establishes certifiability: Equation (21) constructs the smoothed classifier gi from Di, and Theorem 1 guarantees that gi is unchanged for any perturbation δ satisfying ∥δ∥2<r, subject to its probability condition. Thus, the theorem applies to optional smoothed certification, rather than every prediction of D.
Standard deployment uses D and requires no randomized smoothing. When a formal certificate is needed, the verifier invokes gi and estimates its class probability through repeated noisy evaluations of Di. We will revise Algorithm 2 and Appendix B to distinguish these modes and describe Figure 11 as empirical robustness to random perturbations, rather than certified radii.
2. Detection threshold and false-positive rate
Our detector follows the hypothesis-testing framework widely used in generative-model watermarking. Kirchenbauer et al. (2023), for example, count green-list tokens and compute a z-score under a binomial null, deriving an interpretable false-positive probability rather than estimating it from a large negative dataset. MARCO applies the same principle to a multi-bit signature: under the null that an unrelated decoded bit matches the registered bit with probability 0.5, the match count follows Binomial(64,0.5). Table 7 is therefore an empirical specificity check, while the false-positive probability follows from the null distribution.
3. Scope of radioactivity
MARCO shows consistently strong radioactivity in a comprehensive evaluation covering all nine combinations of three heterogeneous source PGMs (RFdiffusion, ESMFold, and Chroma) and three pirate architectures (FrameDiff, FrameFlow, and FoldFlow2). Despite their architectural differences, every combination achieves BA ≥0.90 and DR ≥0.99 under the reported rule. Dilution and adaptive fine-tuning further test persistence when pirate data or training changes. Together, these results provide strong evidence of reliable transfer across the evaluated extraction settings, rather than dependence on one source-pirate pair.
4. Definition of covertness
Equation (20) should be qualified by the adversary's capabilities. Under our threat model, only the authorized verifier can access the decoder and secret watermark parameters. Covertness therefore asks whether an adversary without this information can distinguish clean from watermarked structures using observable geometric or biophysical features. Statistical indistinguishability is widely used to evaluate covertness; we will revise Equation (20) to state this criterion explicitly under the threat model.
5. Utility after watermark-removal attacks
Protein-design literature provides operational criteria for usability. RFdiffusion defines in-silico success as predicted C-alpha RMSD below 2 angstroms to the designed backbone and, for motif scaffolding, motif RMSD below 1 angstrom (Watson et al., 2023). Large-scale analysis also shows that protein pairs with TM-score below 0.5 are mostly in different folds (Xu and Zhang, 2010). We will therefore classify an attacked output as failing standard PGM designability when refolded C-alpha RMSD is at least 2 angstroms, failing motif preservation when motif RMSD is at least 1 angstrom, and likely changing fold when TM-score is below 0.5.
BLOSUM62 measures whether substitutions resemble those in conserved regions of related protein families (Henikoff and Henikoff, 1992); increasing sequence divergence is also associated with increasing structural divergence (Chothia and Lesk, 1986). A large RMSD combined with substantially lower normalized BLOSUM62 therefore indicates loss of the original conformation and evolutionarily plausible sequence, strongly suggesting reduced design utility and possible functional damage. Because our normalization is study-specific, no universal BLOSUM62 cutoff is supported; we use it as corroborating evidence and base the “unusable” designation on the structural thresholds above.