Learning a Forward Operator from Cross-Modal Second-Order Statistics

We consider the linear forward model $y=Ax+ϵ$, $A\in {\mathbb{R}}^{C\times N}$, $C<N$, with the high-dimensional source $x\in {\mathbb{R}}^N$ unobserved during training; we learn $A$ from matching the $y$-covariance ${\Sigma }_y\in {\mathbb{R}}^{C\times C}$ together with a given covariance ${\Sigma }^{\mathrm{ref}}\in {\mathbb{R}}^{P\times P}$ for $\Pi x$, where $\Pi$ aggregates $x$ into $P<N$ dimensions. The two covariances of different dimensions are linked through the pseudo-inverse of $A$: With $B:=\Pi A^{†}$, $A$ can be learned under the covariance-matching objective $B{\Sigma }_y{B}^T\approx {\Sigma }^{\mathrm{ref}}$. We parameterise $A$ in low-rank (LoRA) form: $A=A_0+U{V}^T$, with $A_0$ a known baseline. Our theoretical analysis addresses three questions relevant to training under this objective: identifiability of the matching set in $B$-space, the size of the derivative of $A\mapsto A^{†}$ through which the gradient passes, and conditions under which training keeps $A$ close to a warm-start $A_0$. Identifiability tells us how the set of $B$ that best fits the references sits in ${\mathbb{R}}^{P\times C}$. A single reference leaves a smooth manifold $\mathcal{S}$ whose dimension depends on the regime. Several references, under simultaneous-diagonalisation conditions on the reference covariances, reduce the matching set ${\mathcal{S}}_K$, everywhere on ${\mathbb{R}}^{P\times C}$, to $2^{\min (P,C)}$ isolated solutions, mutually Frobenius-separated, which differ from one another only by sign flips of the rows in the whitened coordinates of ${\Sigma }^{\mathrm{ref}}$. The gradient bound tells us flow through $A^{†}$ stays controlled while $U{V}^T$ is small relative to ${\sigma }_{\min }(A_0)$; the controlled scale tightens as rank grows, so rank has a usable ceiling. The trajectory result tells us that, starting from $A_0$, gradient flow stays nearest the same solution as $B(A_0):=\Pi A_0^{†}$. The other solutions are loss-equivalent and Frobenius-far from $B(A_0)$, so confinement keeps $A$ a small correction to $A_0$ throughout training, consistent with the LoRA premise (low-rank adaptation of pre-trained networks, Hu et al. (2022)).