Innovation SIGReg: Regularizing Transition Residuals for JEPA World Models

Gaussian regularization helps joint-embedding predictive architectures (JEPAs) resist collapse, or loss of distinctions between inputs. Yet the marginal embedding distribution pooled over trajectories need not be Gaussian, so enforcing this target can compete with linear state decoding. Innovation SIGReg applies Sketched Isotropic Gaussian Regularization to persistence-adjusted transition residuals, with a weighted raw-embedding anchor to constrain location and scale. Linear–Gaussian dynamics motivate the target: true transition noise can remain Gaussian while state distributions change, although learned residuals need not recover that noise. On Reacher, a robot-arm benchmark, mean linear angle-decoding $R^2$ at temporal stride 64 rises from $0.537\pm 0.092$ to $0.787\pm 0.125$ across ten runs per method, and the weakest-coordinate score rises from $0.217$ to $0.669$. Normalization alone and rescaled temporal differences do not reproduce these gains at the tested settings. On Two-Room navigation, unguarded residual regularization produces both strong and poor planners. An anchor guard improves mean success over marginal SIGReg by $4.00$ percentage points in a prespecified first-cohort comparison and by $5.67$ points in an exploratory second-cohort comparison; fixed and scheduled anchor increases perform comparably in the first. Detaching current-batch statistics yields $97.50\pm 1.00\%$ success versus $76.00\pm 17.39\%$ with differentiable statistics, although three runs per configuration leave reliability unresolved. A separate Reacher ablation reverses this comparison: detachment reduces $R^2$ from $0.421$ to $0.010$, while detached running averages perform best. These findings show that both the regularization target and the handling of its statistics matter for downstream decoding and planning.