The Floor That Barely Falls: Untrained Encoders as the Operative Null for Concept-Discovery Metrics

Concept-discovery methods are scored with NMI, MIG, DCI and relatives, and often against an implicit floor of zero: of $8$ sampled papers reporting one of these metrics, none reports a chance level. That a floor exists is known: Carbonneau et al. (2022) checked, and found most of these metrics do score near zero on a random representation — all but DCI under a lasso regressor. We measure a strictly larger floor, for a different reason. Their random baseline is noise independent of the factors; ours is an untrained encoder applied to the real inputs, which carry information about the factors that generated them, and that difference is the whole paper. On our $K=4$ cell the untrained-encoder floor is $0.116$ NMI at $240$ held-out samples, in the decomposition run of the same cell whose canonical estimate is $0.115$. Adjusted mutual information (Vinh et al., 2010) removes $0.081$ of it. Under the equal-frequency binning these metrics use the correction is affine, so it cannot reorder methods — but it rescales every difference by about $1.09$, which matters against a fixed equivalence margin. What it leaves, $0.035$, is the operative quantity, and it barely shrinks: $0.030$ at $1200$ samples, a $1.16\times$ change across a fivefold rise in evaluation data while the removable part falls $5.6\times$. Matching inflates it only at small samples: in a Gaussian reference that reproduces this floor, 90% of it is population alignment at $240$ samples and 99% at $1200$, and cross-fitting the matching agrees. Its size is a property of the configuration, and not in the direction a reader would guess: coarsening the discretisation from $10$ bins to $3$ shrinks the total floor $2.3\times$ while raising this component from 30% of it to 85%. It has to be measured per cell rather than looked up. Doing so changes conclusions, including ours: on a tabular benchmark we build, one cell places no task-supervised method above its null at the sample size we ship it at, and the gain a task label plus its architecture buys over that architecture's own untrained state differs by an order of magnitude across architectures (a post-hoc largest-to-smallest ratio of $12.5\times$ whose interval is informative only at its lower end, $4\times$) — and the smallest gain clears its floor only narrowly (under Holm, not under plain Bonferroni). It also disqualifies an equivalence margin this literature would call tight: $\delta =0.05$ NMI is $1.7$–$2.2\times$ the largest excess over chance among the methods it is used to separate, and $3.3$–$25\times$ their between-method range. Read against measured chance, concept labels still beat every architecture we tested, in every cell that resolves.