The paper claims 4 main contributions, however, I see multiple problems with all of them:
(i) the claim that the new simulator is fast: this is not benched properly against SOTA simulators.
(ii) for contribution 2, it’s not clear at all what is meant by the text in the introduction, but it’s also not clear that it’s useful, since other codes actually solve the lens equation analytically for arbitrary pixellated sources and non-parametric lenses at arbitrary resolution for any combination. It’s customary to check against established codes when proposing new ones. Moreover, there already exists many 2-sample tests or coverage tests to check the validity of emulated samples when training emulators (e.g. PQMass, 2402.04355), and looking at modelling residuals is not a new contribution.
(iii) Showing that soft optimization has pathological failure modes: this is not surprising nor novel, especially in strong lensing where this specific failure mode is known, just like the fact that it is known that since the lens equation is non-linear, it has perfectly degenerate solutions (e.g. a circular source with no lens mass)
(iv) this was already possible to do with analytical simulations that include substructure fields, it’s not clear what the new contribution is here. Training a classifier to distinguish simulated strongly lensed systems with or without subhalos is something that was also already done in astrophysics.
Given those, I do not think the paper would have sufficiently substantiated contributions and demonstrated usefulness to be published in a tailored domain journal like JOSS or an astrophysics publication.
But here my main and most fundamental criticism is that this submission has no demonstrated interest to the general ICML community beyond the limited domain application of strong lensing.
Below I describe the weaknesses I see with the work in its current form in more details:
Weaknesses:
-
Limited interest to to the general ICML community, as the only demonstrated application is strong lensing, a very niche field in astrophysics.
-
Time improvement claimed of 0.1% of generation time, but not compared to actual specific simulation from a specific existing simulator, and it’s not explained what hardware or configurations are used (CPU? GPU? Which? Batched?). Some of those numbers are hidden in appendices and are only compared with FlowLensing, which is by very far not the most commonly used SOTA lensing code. The Introduction cites Lenstronomy as a ‘slow code’ and other codes built on it as raytracing being slow, but actually modern GPU-based simulators (e.g written in PyTouch or JAX) demonstrated to be much faster, e.g. SLSim or Caustics, and the paper doesn’t benchmark against them.
-
Insufficient review of the literature (omission of important other codes like PyAutoLens, insufficient review for Strong lensing with ML that misses or miss-represent many works).
-
Accuracy on lensed image generation or forward model gradients is not demonstrated when compared to those baseline codes, which any new raytracing code would need to demonstrate, even if it’s a traditional solver.
-
Scientific interest of the emulator is not demonstrated clearly when compared to other existing software.
-
I have a hard time seeing how the maths derived is relevant for anything more than strong lensing (this links back to weakness 1), yet all results derived are already established in the field of strong lensing (e.g. prop 1).
-
The writing is very poor. Many sentences are very convoluted and make very little sense. For example: ‘whose unit scale is measured by regressing shipped deflections on solved ones’ and ‘and the gap is invisible exactly where emulators are most useful: away from the training data.’ And ‘trade a little exactness for that convenience’ and ‘the residual alone can be satisfied by degenerating the sample’ and ‘it lies on a linear manifold recovered to 0.08% RMS’. Also this sentence is very obscure for a general reader: ‘an exactly solved residual with a convergence certificate, anchored on real data between a matched-strength wrong lens and an image that ignores the requested lens’
-
Modeling choices are hard-baked in the framework, and are hard to understand from a scientific perspective. There was no justification provided for why these choices are sufficient or appropriate for the analysis of what data (which telescope?):
-
In addition to there being no physical size details about source FOV assumptions, the source prior and pixellation choice (64x64 pixels) are not appropriate for most modern data. This is insufficient for most modern strong lensing observations, especially for the scientific purpose of substructure analysis. It’s known (e.g. Stacy et al. 2025 and Nightingale and Dye, 2015) that insufficient source resolution will bias substructure analysis. Also, if a collection of or a single Sersic is assumed for the source model, then the source can be described much more accurately with a sum of analytic profiles which the ray tracer can just directly evaluate with no need to interpolate on a grid. The chosen framework just introduces the unnecessary problems and systematics associated with a pixellated parametrization for no gain.
In addition, it is hardly justified to train a diffusion model that then requires 100 diffusion steps at sampling time for each source generated when the source profile parameters can just be sampled from a prior and the brightness evaluated directly for every simulation.
-
Subhalo fields are 128x128. This is hardly justified for the source resolution chosen. The substructure appendix is very hard to understand. It gives no context for non-expert about what the quoted numbers mean, and the phrasing is very obtuse (e.g. there is clearly a lack of units here and I’m not sure what ‘real axions’ are : Fitting a log–log slope to the radial power spectrum of the substructure residual over k = 3–25 gives−2.27,−1.92 and−1.83 for real axion, CDM and WDM systems, against−3.99,−3.53 and −3.78 for ours. ) Overall to me this sounds like the authors should have concluded they did a poor job of training a generative model over their substructure maps, but no quality assessment of this is made anywhere. Also, there are no details about FOV generated to avoid sub halo fields adding global convergences, how the field of view is handled, what profiles are assumed, range of masses for WDM, etc… Figure 7 lacks any of these very important scientific details that would allow practitioners to understand what is being done.
-
Also, for modern lens modelling, 3-parameters macro-models are clearly insufficient. Modern modelling tools use at minimum EPL profiles +external shear + multiples (which are essential to include in case of substructure modelling to avoid biases O'Riordan and Vegetti, 2024)
Dear Reviewers,
Thank you for your thoughtful reviews. The authors have provided detailed responses to the concerns raised.
I would appreciate it if you could revisit the author responses and clarify whether your main concerns have been adequately addressed, particularly which issues remain critical to your overall assessment.
Given the differences in the initial recommendations, I also encourage you to consider and discuss one another's key arguments to help clarify the remaining disagreements and develop a more consistent understanding of the paper's merits and limitations.
Please feel free to update your ratings if your assessments have changed following the discussion.
Thank you for your continued contributions.