Minimal Nuisance, Maximal Fidelity: Latent-State-Conditioned Invariance in Domain Generalisation

Generalisation across physical systems remains difficult because domains differ in equipment, operating regimes, and system dynamics. As a result, models often rely on system-specific shortcuts rather than learning transferable process behaviour. Existing invariant representation learning methods typically enforce global domain invariance, or condition only on coarse task labels, potentially erasing useful within-label behavioural structure. We argue that the appropriate target is *latent-state-conditioned nuisance invariance* (LaSNI), where domain-specific variation should be removed only after conditioning on the underlying behavioural state. We formalise this target with $z\perp d\mid s$, where $z$ is the learned representation, $d$ the domain, and $s$ an unobserved latent behavioural state. Since $s$ is unobserved, we propose *Local Behavioural Nuisance Suppression* (LBNS), a source-only surrogate that constructs dynamic same-label neighbourhoods, estimates a local behavioural reference, and suppresses cross-domain variation in the residual rather than the full representation. Across six real-world tasks and 30 leave-domain-out folds, LBNS achieves the strongest aggregate transfer and lowest average domain recoverability across all five objectives. An independent descriptor diagnostic shows that its latent neighbours are more behaviourally coherent than random same-label samples in every fold and dataset, including under cross-domain restriction. These results support conditioning nuisance suppression more finely than the observed class when labels hide heterogeneous behaviour.