The PFAD Not Taken: Measuring Polysemanticity via Feature Affine Decomposition

Polysemanticity is central to mechanistic interpretability, yet existing evaluations rely primarily on top-activating examples and qualitative inspection. These approaches are sensitive to dataset coverage and miss distribution-wide functional behavior. We model polysemanticity as functional heterogeneity: a feature is polysemantic when it behaves like different semantically meaningful functions on different subsets of the data. We introduce PFAD (Polysemanticity via Feature Affine Decomposition), a method that decomposes a scalar feature into a hard-routed mixture of affine experts in a vision-language embedding space, interpreting each component through aligned text-image retrieval. We further propose the Effective Decomposition Rank (EDR), a redundancy-aware measure of how many active, non-redundant semantic components explain a feature. We evaluate PFAD on controlled benchmarks with known mixtures and apply it to real vision features. PFAD recovers known component structure and reveals polysemantic behavior invisible to top-activation inspection. Our results suggest that polysemanticity is better understood as distribution-wide functional heterogeneity rather than exemplar diversity.