- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a whitebox parameter watermark for MoE LLMs. The watermark uses multibit paired signals that distribute across experts and layer, signal carriers are allocated evenly across experts and selected within each expert using statistics from a small calibration set. Moreover, the method does not require fine tuning or training. Detection is straight forward by comparing original weight with the watermarked model's weight, the procedure first takes a sign vote per pair, decodes each bit by majority, then compares statistic with a threshold to decided whether its watermarked. Experiment on OLMoE Qwen GPT-OSS demonstrated the performance and robustness of the method.
Critical strengths
The balanced carrier allocation is a well-motivated, MoE-specific design and ablation shows its effectiveness.
The method is lightweight, it only requires value change of a small set of the parameters.Despite the simplicity,it still demonstrated great detectability compared to baselines. The method is also robust to attacks and have small effect on model performance.
Critical weaknesses
Limited novelty. in cited work (Uchida, Lin 2024), they have proposed paired difference, majority voting for identification. Moreover, EmMark (https://arxiv.org/abs/2402.17938), which is not cited, proposed a framework with similar procedures for dense networks: importance-based allocation, additive signature, verification by comparison to original model.
Questions for authors
Could the authors compare SparseMark with EmMark if we transfer the method of EmMark to MoE models?
On OLMoE, routing statistics only exclude 3 of 1,024 experts, could the authors test additional configurations that vary this allocation and see how it works?
Neural Mark showed am abnormal PPL in Table 1 for OLMoE and Qwen, but normal value for GPT-OSS, could the authors explain this discrepancy?
2 comments
Response to Reviewer rKYj (cont.)
Q2. Why use a 32-token eligibility threshold?
R2: We retain expert when , where is its set of routed calibration tokens. On OLMoE, only 3 of 1,024 experts receive fewer than 32 tokens. This is therefore an observed count for this calibration set, not the number of experts that the method is designed to exclude.
Targeted quantization. The threshold sweep below compares two targeted transformations: 4-bit quantization of the 880 highest-frequency experts and 3-bit quantization of the 192 highest-frequency experts. Values are WER (%).
| Threshold (tokens) | Experts excluded | 880 experts, 4-bit | 192 experts, 3-bit |
|---|---|---|---|
| 0, 16 | 0 | 97.40 ± 0.90 | 88.75 ± 1.74 |
| 32, 48 | 3 | 98.05 ± 0.39 | 90.62 ± 1.66 |
| 64 | 7 | 97.53 ± 0.81 | 89.69 ± 1.96 |
| 128 | 9 | 97.27 ± 0.55 | 89.49 ± 1.81 |
| 256 | 21 | 97.07 ± 0.28 | 88.98 ± 1.54 |
| 512 | 52 | 95.31 ± 1.95 | 88.83 ± 0.92 |
| 1,024 | 108 | 95.70 ± 0.68 | 88.52 ± 2.08 |
Note. Thresholds 0 and 16 exclude no experts, and thresholds 32 and 48 exclude the same three experts; rows are merged because the resulting carrier sets are identical.
The 32/48 row has the highest observed mean under both transformations. Moving to 64 excludes four more experts without improving either mean.
Q3. Why does NeuralMark affect perplexity differently across backbones?
R3: We respectfully note that this issue was already investigated in Section 4.2 and Appendix E.3 through an sweep over:
.
The sweep shows that NeuralMark’s high PPL on OLMoE (minimum 406.00) and Qwen1.5-MoE (minimum 27.29) is not caused by evaluating only an unsuitable value of .
The cross-backbone discrepancy is instead explained by the treatment of internal activations in the MoE SwiGLU blocks. At its native setting , NeuralMark introduces relatively large LoRA-based weight perturbations. GPT-OSS-20B applies built-in activation clamping to its MoE SwiGLU gating/intermediate activations, with a threshold of . This provides a numerical buffer that prevents the perturbations from propagating into extreme internal outliers and preserves normal PPL ().
By contrast, OLMoE and Qwen1.5-MoE use standard unclamped SwiGLU projections. The same perturbation scale can therefore cascade through sparse layers into severe activation outliers, causing teacher-forced PPL to increase sharply. This architectural explanation is consistent with the -sweep results.
Planned manuscript changes
- Section 5, opening weight-based methods paragraph: distinguish shared primitives from the routing-aware allocation rule and add the EmMark citation. In Appendix A.1, immediately after Table 4, distinguish the controlled allocation ablation from the direct comparison.
- Appendix E.2, after Table 22: describe the MoE adaptation, verifier inputs, and comparison budget. Appendix E.3, before Appendix F: report the four transformation results separately from the existing clean cross-backbone comparison.
- New Appendix B.4, immediately after B.3: add the carrier-row stability table and the targeted-quantization grid. Cross-reference it at the eligibility definition in Appendix D.1.
- Section 4.2, after Table 1, and Appendix E.3, after the NeuralMark strength sweep: distinguish paired changes from absolute PPL and document the carrier and adaptation scope used on each backbone.
We sincerely hope that our clarifications and planned revisions have thoroughly addressed your concerns, and if so, we would be deeply grateful if you would kindly consider a more favorable evaluation of our work.
Response to Reviewer rKYj
We thank Reviewer rKYj for the constructive feedback. Before addressing the individual questions, we clarify the distinction between generic watermarking primitives and our architectural contribution.
Q1. What is new, and how does SparseMark compare with prior work?
R1: SparseMark is built upon three novel, tightly coupled algorithmic designs for robust parameter watermarking in sparse Mixture-of-Experts (MoE) models. First, it leverages calibration-time routing statistics during enrollment to enforce balanced carrier allocation across eligible experts, preventing carrier concentration within a small subset of experts. Second, keyed block ordering mixes carrier–reference pairs across experts and MoE layers before partitioning them into codeword blocks, making the block assignment depend on the secret key rather than public routing or response scores. Third, for a fixed embedding-energy budget, SparseMark applies antisymmetric paired updates to each carrier–reference pair, maximizing the shift in pair contrast while preserving the pair sum to strengthen the embedded signal against parameter perturbations and removal attacks. These design choices distinguish SparseMark from several related parameter-space approaches. Uchida et al. (2017) exploit differences between parameter coordinates, yet their differential construction primarily compensates for embedding-induced energy variations. In contrast, SparseMark uses antisymmetric differential updates to enlarge the pair contrast between a carrier and its reference coordinate under a fixed squared-update budget, thereby improving recovery margins against erasure and quantization noise. Li et al. (2024) achieve robust verification by aggregating votes across repeated verifier evaluations on perturbed parameters, whereas SparseMark aggregates unit sign votes across disjoint carrier–reference pairs within a single static candidate checkpoint without requiring model forward passes. Finally, while EmMark (Zhang & Koushanfar, 2024) selects watermark coordinates via importance scores, SparseMark combines within-expert gate-weighted response selection with balanced expert quotas and keyed cross-layer block spreading, preventing the watermark structure from being exposed or erased through public-score targeting.
Allocation rule experiments validate the superior robustness of our method over EmMark. Given EmMark’s public response ranking, zeroing out the down-projections of 32 experts compromises 86.82% of pairs under EmMark’s global response selection, compared to only 3.12% under SparseMark’s allocation rule. Consequently, the watermark extraction rate (WER; higher is better) drops to 71.56% with EmMark’s rule, whereas SparseMark maintains 100.00% WER (see Section 4.5 and Appendix A.1, Tables 4 and 6).
Direct comparison with EmMark, as suggested by the reviewer. Unlike SparseMark and our baselines (NeuralMark, PathMark, ELLMark), which are blind schemes, EmMark is non-blind: verification requires comparing the suspect model against the full pre-watermarked checkpoint M0, incurring prohibitive storage and I/O overhead per query at LLM scale. Moreover, EmMark embeds watermarks via discrete integer additions and subtractions, a mechanism exclusively tailored to quantized weight grids. Coupled with its assumption of neither subsequent pruning nor fine-tuning, this design cannot operate on floating-point MoE checkpoints throughout a full post-training lifecycle.
Notwithstanding these fundamental differences in watermarking paradigms, we adapted EmMark to a 256-bit MoE setting and evaluated it alongside SparseMark on OLMoE. The table reports WER (%) under this unified evaluation protocol.
ΔPPL is +0.0011 ± 0.0020 for the adapted EmMark and −0.0004 ± 0.0048 for SparseMark. SparseMark has the higher observed mean WER in these four conditions.