acceptodds
Under review as a conference paper at ICLR 2027

Learning the specimen structure in single-molecule localization microscopy

Abstract

Single-molecule localization microscopy (SMLM) reconstructs three-dimensional organic specimens stained by fluorophores - molecules that emit light under laser excitation - beyond the diffraction limit by localizing active emitters in thousands of frames and accumulating their positions into a point cloud. However, conventional methods process short windows of frames independently, ignoring that all frames share the same underlying structure. In this paper, we propose a method that learns the structure of the specimen and uses it to improve emitter localization. We train a neural network on simulated data to detect and localize emitters, while simultaneously fitting a distribution over emitters to the localizations it predicts on real frames. We use this distribution to adapt the simulator throughout training, hence feeding information from the whole acquisition back into individual localizations. We jointly estimate the point spread function, the background, and the emitter photophysics via the same procedure, enabling self-calibration. After training, we obtain a network for detecting and localizing emitters, and a fitted spatial density that directly provides a continuous super-resolved representation of the specimen. On synthetic benchmarks, learning the specimen structure reduces localization errors compared with the same network trained with uniformly distributed emitters. On real acquisitions, the fitted density reveals fine specimen structure, showing smooth filaments and clearly defined nuclear pore rings.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Single Molecule Localization Microscopy, SMLM, Deep Learning, Inverse Problems, Self-calibration, Self-supervised

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FBmtOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper adapts the simulator to the acquisition being reconstructed, during training. A neural posterior over the emitter configuration is trained on simulated frames (OT-relaxed NPE, with per-detection uncertainty), and a second head estimates the shared imaging parameters (PSF, background). A parametric emitter distribution (a flow for the spatial density, a small model for the photon counts, and a binomial for the count), together with a Gaussian over the imaging parameters, is fit to the posterior's predictions on the real acquisition and fed back into the simulator through a defensive mixture whose weight λ is ramped from 0 to 0.95. The specimen structure learned from the whole acquisition is thus fed back into every frame's localization. The fitting objectives are derived from an empirical-Bayes variational bound (Appendix A). Experiments cover the SMLM 2016 Challenge (synthetic, high and low SNR) and three real acquisitions, against DECODE, LiteLoc, SHOT and LUNAR.

Critical strengths
  1. The problem this paper attacks is real. Per-frame methods discard the fact that all frames image the same specimen, and exploiting that structure across the acquisition has remained open since Cox et al. 2012 could not scale to realistic field sizes.
  2. The external-calibration ablation gives the clearest evidence for the core claim. With the same network and calibration, learning only the spatial density and the count changes RMSE_ax from 33.4 to 19.3 nm, RMSE_lat from 29.0 to 22.5 nm, and Jaccard from 0.888 to 0.912 (high SNR), and from 47.0 to 41.5 / 57.6 to 48.9 (low SNR). This is a controlled comparison of exactly the proposed change. The variance is larger with three seeds (±2.7, ±4.7), so part of the gain is low-confidence.
  3. The method is unified: emitters, photophysics, count, PSF and background are all latents fit by the same procedure. The pooling asymmetry (arithmetic for the per-block emitters, logarithmic with the Greenberg correction for the shared PSF) is justified by the Appendix A bound, and I agree with the reasoning.
  4. Self-calibration extends LUNAR (which learns the PSF only, while the simulator photophysics stay uniform, per Table 1's own classification) to the background and the photophysics. The acquisition-only setting is a useful test of this.
Critical weaknesses
  1. The structure contribution is not decoupled from self-calibration. The Ours-structure vs Ours-fixed comparison learns both the spatial density and the count, so "structure" is doing double duty. More importantly, LUNAR, which learns the PSF without learning the structure, already nearly matches Ours in the acquisition-only high-SNR setting (Jaccard 0.836 vs 0.830, E3D 0.657 vs 0.653). The current results do not let us tell how much of the acquisition-only gain comes from the learned structure versus from the learned imaging model. Please add (a) density-only and count-only ablations, and (b) the missing cell: uniform structure with learned PSF and background, in the acquisition-only setting.
  2. The real-data results entangle structure quality with the number of detected localizations. Your method detects far fewer localizations than the others (11.0M vs 22.4M on tubulin; 0.41M vs 1.93M on Nup96). Neither metric directly rewards recovery of the full emitter set: a sparser reconstruction can inflate RSP, and FRC and RSP can disagree, as Nup107 shows, where your method has the best FRC (11.8 nm) and the worst RSP (0.779) at the same time. Without ground truth, the real-data advantage is hard to attribute, and I would like §5.2 (and, ideally, the abstract) to acknowledge this. The RSP reference is the pixel-wise temporal standard deviation rather than the SQUIRREL mean image. Appendix F.2 gives a reasonable justification; the main text does not mention the modification, and a numerical RSP comparison against the standard mean-image reference would show the ranking is not an artifact of this choice.
  3. I could not find a quantitative evaluation of Contribution 3, the fitted density as reconstruction. q_φ(r) is a flow fit to noisy per-frame predictions and is smoothed by construction, so "smooth filaments and clearly defined rings" rests on visual inspection; a quantitative check is needed to support the accuracy claim. On the Challenge data the true emitter distribution is known, so a Wasserstein distance to the true density or a skeleton comparison would be a natural check and is currently absent.
  4. The comparison population depends on a threshold value that is only in the code. SNR_min "defines the population the network learns to detect" (Appendix C.1): it sets the target set for the synthetic metrics and the learned detection sensitivity that drives the real-data detection counts. The paper does not state its value (it is in the supplementary config) and does not ablate it.
  5. Stability of the self-reinforcing loop and a derivation mismatch. The loop rests on "we have not observed a collapse", and the componentwise mixture used in practice (Appendix C.2) deviates from the mixture in Eq. (2) used by the derivation. Does Lemma 1, and the proposal correction in Eq. (6), still hold under the componentwise mixture? The assumptions (fixed specimen, static background, one global PSF, one dye family) also exclude drift, field-dependent aberrations and live cells, which are common in practice and should be discussed, especially since a single global photophysics model over 10⁴ frames is in tension with photobleaching.
Questions for authors
  1. What is the gain from structure alone, decoupled from self-calibration (uniform structure + learned PSF, acquisition-only setting), and what do the density-only and count-only ablations show?
  2. What value of SNR_min was used, and how sensitive are the results to it?
  3. Can you give a quantitative evaluation of q_φ(r) against the known ground-truth distribution on the Challenge data, and report wall-clock and GPU cost for your method and the baselines (100k iterations, batch 128, two posteriors, 200 Sinkhorn iterations per sample, flow matching, 10-step ODE sampling)?
  4. Why is RMSE_ax 42.4 nm for your method versus 35.6 nm for LUNAR in the acquisition-only high-SNR setting, and does the method hold up with added drift or a decaying background?
3 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for their feedback.

Q1 a). What is the gain from structure alone, decoupled from self-calibration We have added a new ablation study in Appendix H, Table 7. As requested, we isolated the effect of learnt vs uniform spatial density, in a setting where the count, the photophysics, the PSF and the background are all learnt. This experiment shows that the benefit of the learnt structure is not tied to the external calibration: in the acquisition-only setting, it still improves E3D from 0.605 to 0.653 at high SNR, and from 0.450 to 0.521 at low SNR. Without it, self-calibration alone falls behind LUNAR at high SNR (0.605 vs 0.657).

Q1 b). What do the density-only and count-only ablations show? They show that the spatial density does most of the work: learning qϕ(r)q_\phi(r) alone already improves over a uniform density, while learning qϕ(K)q_\phi(K) alone does not beat a uniform prior p0(K)p_0(K). We detail these results in our answer Q3 a) to reviewer x3e6.

Q2. What value of SNRmin was used, and how sensitive are the results to it?

We use SNRmin = 8 in all experiments, which we now state explicitly in Appendix C. We study its influence in a new Table 6 (Appendix E), reproduced here for convenience (Ours-structure, external calibration):

SNRminRecall (high / low SNR)Precision (high / low SNR)E3D (high / low SNR)
50.958 / 0.8380.944 / 0.9330.793 / 0.574
8 (as in the manuscript)0.955 / 0.7860.954 / 0.9470.796 / 0.560
120.950 / 0.6570.962 / 0.9720.798 / 0.476

In short, at high SNR, almost all emitters are above the tested thresholds, and the results are almost independent of SNRmin. At low SNR, SNRmin controls the precision-recall trade-off: higher values increase precision but lower recall, as the network learns to ignore dim emitters. Hence, SNRmin should be adjusted to the SNR of each acquisition.

Q3 a). Can you give a quantitative evaluation of qϕ(r)q_\phi(r) against the known ground-truth distribution on the Challenge data?

Unfortunately, the 2016 SMLM Challenge do not release the ground-truth structure. We simulated a new acquisition with the same imaging setup and a known density, and assessed the reconstruction of the structure with Wasserstein distances, KL divergences and total variation. We describe everything in a new Section 5.2. In short, our learnt density scored best on all but one metric, showing a better overall capacity on this task than other methods, and suggesting that it goes beyond a simple re-interpretation of classical point clouds. We detail this experiment in our answers to reviewers gngG (Q2 a) and x3e6 (Q2).

Q3 b). Report wall-clock and GPU cost for your method and the baselines We report training times in a new Table 8 (Appendix J). In short, for the same input, our training step is among the fastest, but on large real acquisitions, our training is about an order of magnitude slower. We detail these results in our answers to reviewers x3e6 (Q4 c) and gngG (W2 b).

AuthorsOpenReview ↗

Response 2/2

Q4 a). Why is RMSE_ax 42.4 nm for your method versus 35.6 nm for LUNAR in the acquisition-only high-SNR setting?

We agree with the reviewer that this result is intriguing, as the ranking is reversed compared to the low SNR setting. We have investigated it, and so far our hypothesis is a difference in PSF model: our blind PSF has no blur term, while LUNAR's PSF model has one. A model error produces a bias that does not shrink with the number of photons. At high SNR, it dominates the error; at low SNR, noise drowns it out. This hypothesis is consistent with our ablation study: with the beads PSF instead of the learnt one, our RMSE_ax drops from 42.4 to 17.6 nm (Table 7).

Unfortunately, we did not have time to redo all our runs with a blur term during the short rebuttal period, but we will update our acquisition-only results in the final version. We warmly thank the reviewer for this question, which pointed us to a limitation of our PSF model. Finally, this does not affect our main claim: the learnt structure still improves the results in this setting (see Q1 a).

Q4 b). Does the method hold up with added drift or a decaying background? Our model accounts for neither. Drift would blur the fitted density, and hence weaken the learnt structure, while a decaying background would bias our static background estimate. More generally, as pointed out by the reviewer, our assumptions also exclude other common effects (photobleaching, field-dependent aberrations, live cells). We believe our framework can be extended to relax these assumptions (e.g. by learning a drift trajectory or a smooth time-dependent background), but we did not have time to test this during the short rebuttal period. We have added these limitations to our Conclusion and Discussion section.

W2 a). Without ground truth, the real-data advantage is hard to attribute We share the reviewer's concern: without ground truth, FRC and RSP are imperfect proxies, and to the best of our knowledge, no metric can yet fully assess reconstructions on real acquisitions. This limitation is already discussed in our Conclusion, and we have clarified it in Section 5.3 (formerly 5.2). Accordingly, we have clarified our claims on real data in the Abstract, Introduction and Conclusion, and the Abstract now acknowledges the absence of ground truth.

W2 b). The RSP reference is the pixel-wise temporal standard deviation rather than the mean image

That is correct. As discussed by Culley et al. (2018), the mean image can be unreliable when it is dominated by background or out-of-focus fluorescence: RSP would then measure agreement with the background rather than with the specimen. As a picture is worth a thousand words, we have added a new Figure 5 (Appendix F), which shows the mean and the standard deviation of each dataset.

For completeness, we report below the RSP against the mean image. The ranking is not preserved, but the results are uninformative, as they reflect agreement with the background rather than with the specimen. We still report them here for completeness:

TubulinNup107Nup96
DECODE0.7430.3210.619
LiteLoc0.4920.2900.447
SHOT0.5760.3390.461
LUNAR0.4910.2630.476
Ours0.3440.2780.497

We have clarified the main text accordingly, as requested.

W5. Does Lemma 1, and the proposal correction in Eq. (6), still hold under the componentwise mixture?

For Lemma 1, yes: it holds for any θ\theta, ϕ\phi and ψ\psi, as it does not involve the simulation proposal. For Eq. (6), yes in theory: the componentwise mixture leaves the proposal over the PSF and the background unchanged. In practice, however, we approximate this proposal by qψˉ(c)q_{\bar\psi}(c) (Appendix D.1), which becomes increasingly accurate as λ\lambda gets close to 1. On a similar topic, we have studied the impact of the defensive mixture in our new Table 5, and results do not collapse even without it (λmax⁡=1\lambda_{\max}=1). See our answer Q3 b) to reviewer x3e6.

Dear Reviewer,

As the discussion period ends in 24 hours, we would be keen to hear if our responses address your concerns. Following your suggestions, we have added the ablation of the learnt structure decoupled from self-calibration (Table 7), the evaluation of the learnt density against a known ground truth (Section 5.2, Table 2), the value and sensitivity of SNR_min (Table 6), and the training costs of all methods (Table 8). We have also clarified our claims on real data and discussed the limitations of our method in the Conclusion.

If our responses resolve your concerns, we would appreciate it if you could take them into account in your assessment. Thank you again for your consideration.

Reviewer_x3e6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adapts an SMLM simulator to a single acquisition by jointly training neural posteriors on simulations and fitting emitter and imaging-parameter distributions to predictions on real frames. A learned spatial density feeds acquisition-wide structure back into frame-level localization and serves as a continuous reconstruction. PSF, background, and photophysics are also estimated. Validation includes high-density 3D astigmatic SMLM Challenge data at two SNR levels and three experimental acquisitions. The calibrated structure/count adaptation reduces localization RMSE; acquisition-only results are competitive but not uniformly superior. I have reviewed this manuscript myself; no AI assistance was used. Real-data evidence consists of point-cloud FRC/RSP and qualitative density reconstructions.

Critical strengths
  1. Acquisition-wide adaptation is useful and empirically supported. Under external calibration, Ours-structure improves high-SNR lateral/axial RMSE from 29.0/33.4 to 22.5/19.3 nm and E3D from 0.725 to 0.796 relative to Ours-fixed (Table 1). At low SNR, RMSE also improves, although Jaccard does not. This is meaningful evidence for the combined spatial/count adaptation.
  2. The unified acquisition-specific framework learns several nuisance quantities and provides a continuous representation. The paper compares four relevant baselines under two calibration regimes and acknowledges mixed real-data metrics and feedback-loop limitations. This makes the application contribution credible and potentially useful.
Critical weaknesses
  1. Independent validation of the learned structure is incomplete. Table 2 evaluates localization point clouds, whereas Figure 3 shows the learned density against baseline point clouds. Smoother images therefore do not establish better structural fidelity. Appendix F.2 splits localizations for FRC, but does not establish that model fitting is independent across halves; a shared acquisition-trained prior could induce correlated errors. RSP uses a low-resolution temporal-SD reference, which cannot validate nanoscale geometry.
  2. The source and reliability of adaptation gains remain insufficiently isolated. Ours-structure learns both spatial density and emitter count. Full acquisition-only experiments further change photophysics, PSF, and background. Appendix A's variational PSF target and its proposal-corrected marginal-evidence target are distinct, and Eq. (40) introduces a further proposal approximation. Their practical effects are not quantified. These gaps limit confidence in the central mechanism.
Questions for authors
  1. Clarification/validation: Were FRC halves processed with independently trained spatial densities and calibration models? Please clarify shared training and, if shared, report independent-half training with comparable data budgets.
  2. Validation: Can the learned density itself be quantitatively evaluated on synthetic ground truth, with matched rendering or post-processing for baselines? Please assess both recovery of true fine structures and suppression of false structures.
  3. Attribution: Please separate spatial-only and count-only adaptation under external calibration, and quantify calibration-component effects in the full method. How sensitive are results to the mixture cap and uncertainty-dependent density gate?
  4. Theory/implementation: Under what conditions does replacing pi_theta in Eq. (13) by pi_eta in Eq. (21) preserve the intended optimization? How does replacing the mixture proposal by stop-gradient q_psi in Eq. (40) affect calibration or stability? Please also specify axial scaling used for reported nm errors and per-acquisition compute cost.
4 comments
AuthorsOpenReview ↗

Response 1/3

We thank the reviewer for their feedback.

Q1. Were FRC halves processed with independently trained spatial densities and calibration models

The reviewer is right: in the previous Table 2 (now Table 3), every uiPSF model was fitted, and every method was trained, on the full acquisition instead of on each half separately, thus breaking the independence assumption of FRC. To fix it, we now fit uiPSF twice --- once on each half --- and similarly train every method twice, to respect the assumption of FRC and report valid numbers in Table 3. All methods use the same two halves with their usual training protocole, so methods use comparable data budgets. The updated Table 3 has kept the same ranking, showing only a new tie on FRC for NPC-Nup107. We thank the reviewer for this important finding, and we have clarified the main text on that.

Q2. Can the learned density itself be quantitatively evaluated on synthetic ground truth?

Yes, we have added this experiment in a new Section 5.2, which we describe in our answer Q2 a) to reviewer gngG. We have reported both the forward KL, which penalizes missed structures, and the reverse KL, which penalizes invented ones. We reproduce Table 2 here for convenience:

MethodW1 (nm)W2 (nm)KL fwd.KL rev.TV
DECODE (KDE)68.0103.81.4393.6760.685
LiteLoc (KDE)80.0116.61.7554.4260.754
SHOT (KDE)81.9125.81.4303.6520.682
LUNAR (KDE)76.0116.51.7334.2390.726
Ours-fixed (KDE)80.6133.01.4913.7570.691
Ours (KDE)71.8120.81.3133.2760.651
Ours (learnt density)63.1115.71.2682.2450.576

Our fitted density outperforms all others on every metric except W2, where it is second, showing that it offers in general superior specimen reconstruction performance than point-cloud-based baselines.

AuthorsOpenReview ↗

Response 2/3

Q3 a). Please separate spatial-only and count-only adaptation under external calibration, and quantify calibration-component effects in the full method.

We have added a complete ablation study in a new Appendix H (Table 7), whose E3D is reproduced here for convenience:

SpatialCountHigh SNRLow SNR
UniformReference0.6590.461
LearntReference0.7810.492
UniformUniform (Ours-fixed)0.7250.521
UniformLearnt0.7020.502
LearntLearnt (Ours-structure)0.7960.560

In short, learning qϕ(r)q_\phi(r) alone already improves over a uniform density, while learning qϕ(K)q_\phi(K) alone does not beat the uniform prior p0(K)p_0(K). Our interpretation is that the count only determines how many emitters there are, not where they are: a learnt structure can concentrate them to create high and low density regions, as in the data, whereas a uniform density spreads them evenly everywhere. We have clarified this in the manuscript (Appendix H).

Regarding calibration, the PSF explains most of the gap between the external-calibration and acquisition-only settings:

PSFPhotophysicsHigh SNRLow SNR
BeadsReference0.7960.560
BeadsLearnt0.8080.584
LearntReference0.6580.520
LearntLearnt (acquisition-only)0.6530.521

Learning the photophysics improves the results with the beads PSF, and does not hurt with the learnt PSF. We have also added our method with the beads PSF and learnt photophysics to Table 1, which corresponds to how we expect it to be used when beads are available. It scores best or on par with the best on every metric.

Regarding the background, we found that learning it only moves it slightly away from our initial estimate (Appendix E): over the 54 runs of our ablation study (Table 7), the median relative deviation is 4% (high SNR) and 14% (low SNR). Such offset is below half the per-pixel noise standard deviation of the camera noise. Unfortunately, we did not have time to run an ablation with a fixed background with multiple seeds and add it to Table 7 during the short rebuttal period.

Q3 b). How sensitive are results to the mixture cap and uncertainty-dependent density gate?

Regarding the mixture cap, we have added results on the 2016 Challenge in a new Table 5 (Appendix D.2), reproduced here for convenience:

λmax⁡\lambda_{\max}E3D, high SNRE3D, low SNR
0 (Ours-fixed)0.7250.521
0.80.7830.522
0.95 (as in the manuscript)0.7960.560
1 (no cap)0.7960.545

We found that results improve as λmax⁡\lambda_{\max} increases, but plateau beyond 0.95. Hence, we set \lambda_\max to 0.95 as it maintains performance while preserving diversity in the simulations for free. We have clarified the role of \lambda_\max in Appendix D.2.

We added a density gate in our fitted density to prevent it from resolving unrealistically thin structures, and to help the convergence of our method in a few cases. We ran additional experiments on the 2016 Challenge: disabling the gate does not change the E3D of the neural posterior, while a 2x stricter gate (i.e. the gate is set from twice the predicted uncertainty) lowers it:

Density gateE3D, high SNRE3D, low SNRFinest density scale (px), high / low SNR
Default (as in the manuscript)0.7960.5600.17 / 0.24
Disabled0.7970.5600.10 / 0.10
2x stricter0.7880.5350.34 / 0.58

We have also clarified its role in our manuscript.

AuthorsOpenReview ↗

Response 3/3

Q4. a) Under what conditions does replacing pi_theta in Eq. (13) by pi_eta in Eq. (21) preserve the intended optimization?

In a few words, πη\pi_\eta is a substitute for πθ\pi_\theta that only needs the estimated posteriors qη(c∣yi)q_\eta(c \mid y_i), whereas πθ\pi_\theta requires the image likelihood of every real block. Unlike πθ\pi_\theta, it does not inherit the variational gap of qθq_\theta (its distance to the true emitter posterior), but it relies on the PSF head and on the emitter proposal. Hence, the substitution preserves the intended optimization under two conditions: the PSF head qηq_\eta is accurate, and the emitter proposal p~λ(x)\tilde p^\lambda(x) is close to qϕ(x)q_\phi(x), which holds once λ\lambda gets close to 1.

Q4. b) How does replacing the mixture proposal by stop-gradient q_psi in Eq. (40) affect calibration or stability?

Once λ\lambda reaches 0.95, the proposal is 95% qψq_\psi, so the approximation is nearly exact, and in practice, none of our runs diverged. The stop-gradient comes from the correction of Greenberg et al. (2019): the division is only here to cancel the simulation prior contained in the neural posterior, and is not meant to be an objective with respect to ψ\psi.

Q4. c) Please also specify axial scaling used for reported nm errors and per-acquisition compute cost.

Regarding axial scaling, before computing the metrics of Sage et al. (2019), we replace their wobble correction by a full affine transformation fitted by RANSAC, for all methods. It corrects global shifts and scalings, in particular along the zz axis, whose scale is unknown without a pre-calibrated PSF. FRC and RSP only use lateral positions, hence are independent of this scale. We have clarified Appendix F on this matter.

Regarding compute cost, we have added a new Appendix J, whose Table 8 is reproduced here for convenience (one RTX PRO 6000 Blackwell GPU, approximately half the throughput of an H100):

MethodStep time (ms)Challenge (h)Tubulin (h)NPC-Nup107 (h)NPC-Nup96 (h)
DECODE2094.54.54.54.5
LiteLoc4381.51.51.51.5
SHOT36310.510.510.510.5
LUNAR15511.34.76.02.9
Ours2517.781.233.142.5

For the same input, our method is among the fastest per training step. However, this observation may be unfair to other methods in practice, since they can train on small simulated patches, whatever the size of the acquisition. By contrast, our second objective needs to process real frames at full size, and for simplicity, our current implementation simulates frames of the same size for the first objective. Thus, in practice, our training time grows with the field of view, while other methods' do not, which results in training times longer by an order of magnitude on real, large acquisitions. As our implementation is a proof of concept, we have not focused on optimizing our code yet: training on random crops, for instance, would likely speed it up, and we will study this in the coming weeks.

W1. Figure 3 shows the learned density against baseline point clouds Regarding Figure 4 (formerly Figure 3), we have added a new Figure 8 in Appendix I for an apples-to-apples comparison: it shows our point cloud next to those of LUNAR and SHOT, and our density next to KDEs of their localizations.

Dear Reviewer,

As the discussion period ends in 24 hours, we would be keen to hear if our responses address your concerns. Following your suggestions, we have added corrected FRC results with independently trained halves (Table 3), the evaluation of the learnt density against a known ground truth (Section 5.2, Table 2), spatial-only, count-only and calibration ablations (Table 7), the sensitivity to the mixture cap (Table 5) and to the density gate (in our response), the training costs (Table 8), and side-by-side point clouds and densities on real data (App. I).

If our responses resolve your concerns, we would appreciate it if you could take them into account in your assessment. Thank you again for your consideration.

Reviewer_gngGOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a novel approach for Single-molecule localization microscopy (SMLM), where sparsely activated fluorescent molecules spread across multiple time frames are used to reconstruct 3D sample structure that surpasses diffraction limit of resolution. The proposed method leverages common underlying specimen structure, introducing learnable emitter distribution qϕ(x)q_{\phi}(x) to describe spatial structures, activation number and statistical features of lighting, along with a neural posterior qθ(x∣y)q_{\theta}(x | y) trained on simulation data to localize molecules frame-by-frame. A sequential optimization method is utilized to fit emitter distribution on real captured frames, which is then used to adjust simulation data. A self-calibration process is also proposed to estimate posterior and distributions of PSF, background and emitter photophysics. The latent spatial density that is learned during training can offer continuous super-resolved representations as well, rather than just point cloud. Comprehensive quantitative and qualitative evaluations on synthetic benchmark and real data have shown distinct advantage of the proposed method, comparing to existing ones.

Critical strengths
  • The paper has offered a novel viewpoint to better utilize common specimen structure that had been overlooked by previous works. This fits in a broader inspirational paradigm that harnesses latent structure shared by numerous observations in order to improve performance on local inverse problems. A complete pipeline with clear mathematical definitions and derivations is estabilished, along with a framework that connects structure learning, localization, and imaging system.

  • Ample evaluation results and stunning reconstruction performances show strong evidence of the proposed method. Table 1 has demonstrated clear advantage of the method in terms of quantitaive metrics, achieving the best or second-best places in all (but one) experiment settings. Figure 1 and 3 are convincing evidence of its 3D reconstruction abilities in both super-resolution accuracy and least visual artifacts.

Critical weaknesses
  • Evaluations on real dataset, specifically Table 2, show limited progress. The metrics, FRC and RSP, have shown different patterns on different evaluation sets, with the proposed method showing inferior performances in either cases. For figure 3, the visulization is a comparison between point clouds and spatial density qϕ(r)q_{\phi}(r), while table 2 uses point clouds. Admittedly, lack of GT data in SMLM makes things hard to quantify. Yet, a clearer and more direct evaluation on how close the learned spatial density compared to real structure would be preferred for the circumstance.

  • The computational complexity and theoretical limits can be discussed more thoroughly. Since the proposed method is a per-acquisition optimization, compared to other methods that train on a dataset and process all samples using the same network, computational cost needs to be clarified. As for theoretical analysis, whether the learned structure prior introduces localization bias-variance trade-off is worth discussing.

Questions for authors
  • How does the learned structure prior affect localization bias and uncertainty? Is it possible to show error distributions rather than just RMSE? How sensitive are other components to posterior uncertainty?
  • Would it be possible to further validate actual reconstruction accuracies of spatial density? E.g., in synthetic data, does the learned spatial density align with emitter spatial distribution? Could we establish similar evaluations in real data?
  • What is the theoretical bound for localization accuracy, when given PSF or PSF unknown? What is the physical limit of the proposed method?
3 comments
AuthorsOpenReview ↗

Response 1/2

We thank the reviewer for their feedback.

Q1 a). How does the learned structure prior affect localization bias and uncertainty? Is it possible to show error distributions rather than just RMSE?

We have added Figures 6 and 7 in a new Appendix G, which show the distributions of localization errors and predicted uncertainties against zz, with and without the learnt structure.

In short, the bias stays small compared to the spread of the errors, and the learnt structure does not strongly amplify it. Regarding uncertainty, we have found that the estimated lateral errors match the real ones, whereas axial uncertainty is underestimated (see Figure 7 in the updated version of the article). We attribute this behavior to having two objectives, as already discussed in the original version of the paper. We have added an additional discussion of this phenomenon in the Conclusion section.

Q1 b). How sensitive are other components to posterior uncertainty?

We understand "other components" as the other learnt parts of our simulator, i.e. the PSF, the background, the emitter count and the photophysics.

On the one hand, for the PSF and the background, Eq. 6 pools evidence over the whole acquisition, i.e. over ~10^4 frames and ~10^5 emitters on the Challenge datasets. We observe that their predicted uncertainty quickly goes down, reaching below 10^-6 for each Zernike coefficient in the PSF model.

On the other hand, the count and the photophysics can suffer from error propagation, like the spatial density. They are however much more constrained --- the count is a single scalar, and the photophysics is a 1D distribution --- and we found no collapsing issue in this direction. We measured this over all runs in the new ablation Table 7 where the photophysics is learnt: the median of our learnt photon-count distribution is within 5% of the ground truth at high SNR, and 16% at low SNR.

We also tested the sensitivity of the learnt photophysics to posterior uncertainty by re-training models with artificially increased posterior STDs, and the median barely moves:

Posterior stdHigh SNRLow SNR
1x (normal)0.97 x GT median0.97 x GT median
2x0.94 x GT median0.93 x GT median
3x0.96 x GT median0.92 x GT median

Q2 a). Would it be possible to further validate actual reconstruction accuracies of spatial density?

Yes, we have added this experiment in a new Section 5.2. We simulated an acquisition of 12 random filaments (new Figure 3), trained all methods on it, and compared the true emitter density to our learnt density and to KDEs of each method's localizations (with bandwidths chosen by cross-validation, as explained in the main text). See Table 2 in the new version of the paper, reproduced here for convenience (lower is better):

MethodW1 (nm)W2 (nm)KL fwd.KL rev.TV
DECODE (KDE)68.0103.81.4393.6760.685
LiteLoc (KDE)80.0116.61.7554.4260.754
SHOT (KDE)81.9125.81.4303.6520.682
LUNAR (KDE)76.0116.51.7334.2390.726
Ours-fixed (KDE)80.6133.01.4913.7570.691
Ours (KDE)71.8120.81.3133.2760.651
Ours (learnt density)63.1115.71.2682.2450.576

Our method outperforms all others on every metric except W2, where it is second. This highlights that our density representation goes beyond the usual point-cloud representation regarding direct structure reconstruction.

Q2 b). Could we establish similar evaluations in real data?

We share the reviewer's concern: without ground truth, FRC and RSP are imperfect proxies, and unfortunately our results with them are mixed (Table 3, formerly Table 2). As discussed in our Conclusion, this calls for new metrics for real acquisitions. We have clarified our claims on real data in the Abstract, Introduction and Conclusion.

AuthorsOpenReview ↗

Response 2/2

Q3 a). What is the theoretical bound for localization accuracy, when given PSF or PSF unknown?

With a known PSF and no prior, the bound is the Cramér-Rao lower bound (CRLB), which holds for unbiased estimators. For a single emitter at the focal plane on a single frame, in the setting of the high-SNR Challenge, the CRLB is around 25 nm along all 3 axes. By contrast, our method uses 3 frames and the learnt structure as a prior, hence can go below it: it reaches 22.5 nm laterally (x and y axes) and 19.3 nm axially (z axis).

When the PSF is unknown, the CRLB also exists, but gets much harder to compute, and we did not have time to compute it during the short rebuttal period.

Q3 b). What is the physical limit of the proposed method?

Two quantities cannot be recovered:

  • The axial scale: without calibration, we can recover zz only up to a scale factor. Converting it to nanometers requires an axial calibration (Section 4).
  • The SNRmin threshold, which controls the existence of an emitter. A dim emitter is still there, just undetectable: "is there an emitter?" only has an answer once we set a threshold.

For SNRmin, all methods set one in their own way, either implicitly through their training priors or by a threshold on detection scores, a minimal photon count, etc. In other words, SNRmin indirectly sets the precision-recall trade-off. In our method, SNRmin is an explicit threshold on the SNR of each emitter (Eq. 43 in Appendix E), which we set to 8.

W1. The visualization is a comparison between point clouds and spatial density.

We have completed Figure 3 (now Figure 4) with a new Figure 8 in Appendix I. It shows our point cloud next to those of LUNAR and SHOT, and our learnt density next to KDEs of their point clouds.

W2 a). The proposed method is a per-acquisition optimization.

The reviewer is right: our method needs retraining for every acquisition. However, DECODE, LiteLoc and SHOT also need retraining whenever the PSF, the photophysics or the emitter density change. In practice, these rarely remain constant between two acquisitions: a small change in the imaging buffer alters the photophysics, and a different coverslip position changes the PSF aberrations. Overall, we believe that not having to calibrate these parameters in advance is worth the retraining, even if our training is currently slower than that of other methods (see W2 b). We have clarified this in the Conclusion.

W2 b). Computational cost needs to be clarified.

We report training times in a new Table 8, and discuss compute costs in a new Appendix J. In short, for the same input, our training step is the second fastest (251 ms, behind DECODE at 209 ms). However, our second objective processes real frames at full size, whereas other methods train on small simulated frames: thus, on real acquisitions with large field of views (like 256x256 frames of the Tubulin dataset), our method is about an order of magnitude slower to train. We have not yet tried to reduce this cost, e.g. by training on random crops or only on regions of interest. See also our answers to Reviewers x3e6 (Q4 c) and FBmt (Q3 b).

Dear Reviewer,

As the discussion period ends in 24 hours, we would be keen to hear if our responses address your concerns. Following your suggestions, we have added an evaluation of the learnt density against a known ground truth (Section 5.2, Table 2), the distributions of localization errors and uncertainties (App. G), the training costs (Table 8), and side-by-side point clouds and densities on real data (App. I). We have also clarified our claims on real data and discussed the limitations of our method in the Conclusion.

If our responses resolve your concerns, we would appreciate it if you could take them into account in your assessment. Thank you again for your consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global comment

We thank all reviewers for their in-depth reviews, which helped us substantially improve our manuscript. Over the past 10 days, we re-trained 62 models, for a total of 470 H100-equivalent hours. Our main changes are:

  • A direct evaluation of the learnt density against a known ground truth on a new synthetic acquisition (Section 5.2, Figure 3, Table 2).
  • A complete ablation study of each learnt component of our simulator (Appendix H, Table 7).
  • Corrected FRC results on real data, with every method retrained independently on two halves of each dataset (Table 3).
  • Sensitivity studies of the mixture cap λmax⁡\lambda_{\max} (Table 5) and of SNRmin (Table 6).
  • Error and uncertainty distributions (Appendix G), training costs (Appendix J), and side-by-side point clouds and densities on real data (Appendix I).
  • Clarified claims on real data in the Abstract, Introduction and Conclusion
  • An extended Conclusion, with the limitations raised by the reviewers.

All changes are in blue in the revised manuscript. We hope that our answers address all the reviewers' points, and remain available for any further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.