SPLIT: Structure-Preserving Budget Allocation for Visual Anomaly Detection

Industrial anomaly detectors increasingly exploit multi-layer features from frozen backbones, but conventional layer-wise heads entangle detector budget with feature structure: a head's width is fixed by the backbone's feature dimension, so trainable budget can only be changed by changing which layers are consumed. We introduce SPLIT ( Structure-Preserving budget aLlocation for vIsual anomaly deTection), a latent reparameterization of multi-layer anomaly heads whose latent dimension controls trainable budget at a fixed feature structure. Separating the two exposes a budget-dependent budget-sharing trade-off: under a matched total budget, a single shared latent code concentrates a scarce budget and wins at the smallest budgets, while per-layer codes remove cross-layer latent coupling at the cost of more budgets—take over once each head receives a workable width. The asymmetry between backbones is consistent with their cross-layer representation structure, and feature composition remains a third, independent factor. Across five industrial benchmarks and two frozen backbones, SPLIT is competitive with prior work, using only 0.002M latent parameters on DINOv3 and 0.008M on CLIP.