Prediction Is Not Description: An External-Label Audit of Sparse Autoencoders

Sparse autoencoders (SAEs) are evaluated almost entirely from inside the model. On those metrics they are hard to tell apart from random directions, and simple supervised baselines beat them. Yet that verdict conflates two properties: whether a direction separates a concept, and whether it can be described. Telling them apart needs a criterion the model never saw. MIMIC-IV admissions carry ICD-9 codes that human coders assign from the chart for billing, with no automated process involved. We train vanilla and JumpReLU SAEs on Gemma-2-2B activations from 50,000 discharge summaries, then audit them against a fixed 46-code panel on held-out notes, beside six alternatives: a general-purpose SAE, covariance- matched random directions, PCA, lexical n-grams, and two directions fitted on the codes themselves. Grounding against the codes fixes a common scale of correlation strength, on which the SAEs reach |rpb| = 0.86 against 0.43 for the random directions. Three tests then measure separate properties along that scale. Description and ablation both put the two domain-trained SAEs first. At matched correlation a blind judge recovers the code from an SAE feature’s unsupervised explanation 62–75% of the time, and from a direction fitted on that code only 17–38%, at or below the random floor. Ablation raises loss more on notes carrying the code for every domain-specific SAE target, and for no random direction. The general-purpose SAE, applied zero-shot, falls below both on every test it enters. Prediction reverses the order: the lexical n-grams classify the codes better than either SAE. Prediction and description are separate properties of a direction, and it is description that the sparse decomposition supplies.