SparseMark: Routing-Aware Differential Parameter Watermarking for Mixture-of-Experts LLMs

Parameter watermarking for sparse mixture-of-experts (MoE) language models faces a placement challenge: evidence concentrated in a few experts is vulnerable to localized damage, while embedding must preserve model utility. We introduce SparseMark, a post-training multi-bit watermark that distributes evidence across experts and limits the influence of individual parameter pairs during recovery. SparseMark combines balanced expert allocation with response-based carrier selection, encodes a secret codeword through antisymmetric parameter updates, and spreads each bit across experts using keyed block ordering. Verification recovers the codeword from stored baseline contrasts through unit sign voting, without model forward passes. Controlled experiments show that balanced allocation improves recovery under targeted expert damage at comparable utility cost, while unit voting outperforms mean aggregation under pruning at fixed embedding energy. SparseMark achieves exact clean recovery of 256-bit codewords across three MoE backbones with small measured changes in perplexity. Detailed OLMoE evaluations demonstrate watermark retention under the evaluated pruning, quantization, and fine-tuning settings, while identifying limited recovery margin under feedback-guided attacks after quantization. These findings support distributed carrier placement and bounded pairwise influence as design principles for parameter watermarking in sparse MoE models.