BioInvariant: Evaluating Appropriate Biological Invariance and Sensitivity in Genomic Foundation Models

Genomic foundation models are increasingly used as general-purpose DNA encoders, but they are still evaluated primarily through downstream accuracy. Predictive accuracy alone does not reveal whether representation geometry reflects known biological structure or is dominated by simpler sequence statistics such as GC content. We introduce BioInvariant, a counterfactual evaluation framework for frozen genomic representations. BioInvariant tests two complementary properties: appropriate invariance under transformations that preserve biological meaning, and appropriate sensitivity under perturbations that disrupt biological meaning, always relative to composition-matched controls. We evaluate DNABERT-2, Nucleotide Transformer, HyenaDNA, and Caduceus across four axes: reverse-complement consistency, coding consequence, clinical variant signal, and regulatory motif disruption. Raw embedding similarity can be misleading: Caduceus attains the highest raw reverse-complement similarity, but its reverse-complement-specific signal nearly disappears after GC-matched correction; DNABERT-2 and Nucleotide Transformer show stronger corrected specificity. Sensitivity results are also axis-dependent, and no model is consistent across all axes. Additional audits show that context length, GC shortcuts, background perturbations, pooling, and control position substantially affect interpretation; in particular, HyenaDNA's apparent motif sensitivity depends on where the control substitution is placed, and only DNABERT-2's motif effect is significant for both upstream and downstream controls. BioInvariant therefore complements downstream benchmarks with a representation-level criterion: whether frozen genomic encoders move for biologically appropriate reasons under controlled, matched counterfactuals.