Moment Transport Poisoning for Online Test-Time Adaptation

Online test-time adaptation (TTA) updates a pretrained model on unlabeled test data, allowing poisoned inputs to influence benign predictions through shared normalization and model updates. We propose Moment Transport Poisoning (\method), a grey-box attack that targets pre-BatchNorm activation statistics using a fixed source model. MTP maximizes standardized shifts in channel means and log-variances and combines this moment objective with non-ground-truth entropy through gradient-normalized projected updates. Our analysis connects moment displacement to feature distortion under mixed-batch normalization and to adaptation-state changes, and derives sufficient conditions for increased benign risk. Experiments on CIFAR-10-C, CIFAR-100-C, and four ImageNet-C subsets show that MTP outperforms DIA and RTTDP baselines in mean benign error. Five-seed evaluations on both CIFAR-C datasets confirm a mean advantage at both perturbation budgets. Ablations show that moment transport contributes more than the entropy objective alone, while their combination achieves the highest benign error. Elevated error during subsequent benign-only adaptation and increased held-out probe loss further demonstrate effects beyond the poisoned batch.