PhysLensDiff: Valid-by-Construction Lensing Emulation, and Why Physics Penalties Fail

Simulators obey their equations but are slow; the generative models that emulate them are fast but usually carry no guarantee that their samples obey those equations, especially far from the training data. We study the two standard fixes on strong gravitational lensing, where validity can be checked exactly. The soft fix adds the lens equation to the training loss as a penalty. It lowers the violation, but by darkening the images: we prove that the penalty always rewards darkening, and observe that the samples lose their content and stop following the requested lens, in lensing and in divergence-free flows. Reweighting or normalising the penalty does not help. The hard fix avoids this. PhysLensDiff generates the source, the lens and the dark-matter substructure, then ray-traces them through a differentiable lens layer, so every sample is exactly valid for any requested lens, at about 0.1% of generation time. Unconstrained image generators, even when given the full lens, remain far from valid out of distribution. Our with/without-substructure pairs train dark-matter classifiers that are more accurate on held-out simulated systems than the best alternative from image generators.