HOW DO WEIGHT AND ACTIVATION SPARSITY INTERACT?

A multiply-accumulate (MAC) contributes to inference cost only if both its weight and activation operand are nonzero, yet the two are almost always regularized independently. We test whether jointly regularizing both, with a shared per-element-normalized squared-Hoyer measure, causes them to reinforce or compete, and find the answer is conditional rather than universal. On ResNet-18/CIFAR-10, joint (additive) training pushes both sparsities above what either solo penalty reaches alone (weight sparsity $86.75\%$ vs. $86.08\%$; activation sparsity $67.72\%$ vs. $63.08\%$; $n=5$ seeds), and the resulting effective-MAC fraction beats a naive independence estimate at a $10.5\times$ reduction and $97.0\%$ of baseline accuracy – though at matched compute a solo penalty alone reaches statistically indistinguishable accuracy, so the gain is a different sparsity split, not a proven accuracy advantage. A theoretically appealing coupled (product) objective instead underperforms the additive sum by $14.1$ accuracy points at matched compute, substantially a convergence-speed artifact: the gap shrinks by more than half after doubling the training budget and closes to $2.2$ points by 50 epochs. Reinforcement replicates in most but not all of six further settings (MNIST, CIFAR-100, SVHN, MobileNetV2, and structured sparsity) – reversing on MNIST, an order of magnitude stronger under structured sparsity, and turning outright bimodal on SVHN – and the same conditional pattern holds under an entirely different, discrete weight-sparsity mechanism on four independent production edge-AI models, where the interaction is complementary, antagonistic, sign-flipping, or accuracy-improving depending on the model. Critically, the coupled objective's failure is a fact about Hoyer specifically, not product-form coupling in general: a density-aligned surrogate closes a $14.2$-point accuracy gap to $0.3$ points, statistically indistinguishable from additive. We report this as evidence that weight and activation sparsity interact dynamically and conditionally rather than independently.