Measuring Class Expansion Capacity in Frozen Representations

How many new classes can a frozen representation support without updating its feature extractor? We introduce frozen-representation class expansion — an extractor is trained on B base classes, frozen permanently, and then used to recognise further classes from newly stored statistics alone — and define expansion capacity as the largest label space served at a stated accuracy and memory budget. Measured on the arriving classes alone, capacity is much lower than on the pooled label space (65.8±6.8 vs. 88.2±3.1 classes at 50% accuracy), and it grows with the number of base classes even as closed-set accuracy falls. Second-order estimation, not prototype multiplicity, is the dominant lever (+16.4 points against at most +1.1). This motivates a readout combining shared whitening with a low-rank class-specific residual, which holds the accuracy–storage frontier at its own budget in nine of ten representation–dataset settings and on a 300-class stress test, and matches or beats full per-class covariance at 2.6–5.3% of its memory on every pretrained extractor, including at native resolution on Food-101. Whitening instead by the base features' uncentred second moment adds 1.7 to 4.3 points on backbones trained on the base classes, partly through the directions they learned to separate them, a gain whose sign a measured specialisation index predicts in 14 of 15 settings; it does not help pretrained ViTs. The prototype and depth findings are largely specific to backbones trained from scratch on few classes, at 32 pixels and at native resolution alike.