- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses encoder overfitting in autoencoder-based heterogeneous cryo-EM reconstruction. Methods such as cryoDRGN could in principle amortize conformation inference, but their encoders overfit to noise in the training images and generalize poorly to unseen particles, preventing training on a subset of a large dataset and inferring the rest. The authors propose CryoNOO, a training objective added to cryoDRGN. The reconstruction model’s own rendered projection of each training particle serves as a denoised view. Fresh Gaussian noise is added to this view, and a VICReg loss encourages the encoder to produce consistent latents for the original and re-noised images. The added components are used only during training, so inference is unchanged. The paper evaluates the method on CryoBench-derived synthetic datasets and on CAHRA data. It reports three finding: (i) better reconstruction of held-out particles, including training on 10% of a 1M particle dataset and inferring the remainder; (ii) placement of particles from a held-out conformation at the expected location in latent space; and (iii) conformation inference from the learned latents that is competitive with cryoSBI.
Critical strengths
-
A well-motivated, simple fix for a documented failure mode, with the right controls. Encoder memorization in cryoDRGN-style models is a known bottleneck, and the CryoNOO’s fix requires only an expander head and one additional loss on top of an unmodified cryoDRGN (Sec. 3.2). The expander is discarded after training and inference is identical to cryoDRGN. Crucially, the ablations rule out the trivial explanations: adding noise to the original images, invariance-only training, and SimSiam all underperform the full objective (Table 3), so the gain is attributable to the combination of model-generated denoised views and the full VICReg objective.
-
A practically meaningful use case demonstrated at scale, with consistent gains across settings. The 100K to 1M amortization experiment (Table 1) matches a real workflow, gains over cryoDRGN-100K appear on all five CryoBench datasets (Table 6) and on experimental CAHRA data including HQ to LQ transfer (Table 7), and the latent space analyses (Table 9-11) corroborate the mechanism rather than just the endpoint.
Critical weaknesses
-
The gains are not cleanly attributed to the claimed mechanism, and no variability is reported.
- Sec. 3.2 states gradients are not stopped through X̂. VICReg therefore also updates the decoder, so the method is not a teacher-student set up and encoder-side and decoder-side effects are entangled. No stop-gradient ablation is given.
- The ablations do not isolate the paper’s key insight “reconstruction model as denoiser”, namely that the decoder-rendered projection serves as a denoised view. The direct control is to keep the full VICReg objective but construct views from the raw image instead (X vs. X + ε, no rendering). Neither existing variant in Table 3 provides this control. “CryoDRGN + Noise Aug” (0.328) appears to apply noise augmentation without VICReg (Sec. 4.4 is not explicit), which removes both components at once. “Only Invariance” (0.325, identical to the cryoDRGN baseline) ablates VICReg’s regularizers while keeping X̂.
- Table 7 is hard to reconcile with an encoder-generalization claim. On CAHRA HQ→HQ, CryoNOO improves strongly on training particles. CryoDRGN’s test metrics are at or above its train metrics on ADH, Aldolase and PKM2, so there is no generalization gap to close. Those single-protein sets also have no conformational heterogeneity. Part of the gains may come from other pipeline differences or from a decoder-side regularization effect.
-
The headline practical claims outrun the evidence.
- Large-scale amortization is shown only on synthetic data, at SNR 0.1 (App. A), which is less noisy than the SNR 0.01 setting that also appears in Table 4. On the one heterogeneous real dataset (“Combined”), test gains are marginal (Table 7a: FSC 0.335→0.356, CD 1.132→1.120, vIoU 0.530→0.538), and CD is slightly worse in HQ→LQ (1.074→1.095, Table 7b).
- HQ→LQ is described as validating the subset-train/infer-rest pipeline (App. C.1). But LQ and HQ particles come from the first and last 30 frames of each movie (App. C.1.1). If these are the same physical particles, this tests dose/noise robustness rather than generalization to unseen particles.
- Unseen-conformation generalization (Sec. 4.2, Fig. 3) is shown only qualitatively, for three held-out states on one dataset, and each held-out state is directly flanked by training states, making this a mild interpolation test. The visual criterion is also not matched: the “expected location” boxes for CryoNOO cover roughly a quarter of the latent disk, whereas those for cryoDRGN are small patches on the ring. Moreover, cryoDRGN already generalizes poorly to unseen particles of seen conformations on IgG-1D (test CD 4.570, Table 6), so the scattering of held-out-conformation particles in Fig. 3 does not by itself isolate the unseen-conformation effect. The 2D projection method for the 8-D latent is not stated.
- The real-data noise handling is heuristic. A factor of 10× over the estimated σ is recommended (App. A), whereas the robustness claim in Sec. 4.4 rests on 0.5×–2× on synthetic data with known σ.
Questions for authors
- No table reports variability across runs, and per-image FSC is computed from one image per conformation (App. E). Please report mean ± std over at least three seeds for the main comparisons (Tables 1, 2, 6, 7), and the test-set size for Table 2, so that small differences (e.g., IgG-RL and Spike-MD in Table 6; CryoNOO vs. cryoSBI in Table 2) can be assessed.
- Table 3 is evaluated on test per-image FSC, and its chosen configuration matches the IgG-1D test result in Table 6. The CAHRA noise factor of 10 was also selected based on observed performance (App. A). Were these choices made on the test split? If so, please re-select them on a held-out validation split and report the resulting test performance.
- Add controls on synthetic data: (i) VICReg between two noisy copies of the real image with no decoder-rendered view; (ii) a stop-gradient on X̂; (iii) an oracle using clean ground-truth projections plus fresh noise. Please also report latent variance across noise draws of the same clean image for cryoDRGN vs CryoNOO.
- Are the HQ and LQ particles in HQ→LQ the same physical particles? If so, please evaluate on genuinely held-out particles.
Thank you for the response and for running the controls. The new experiments address my main concerns well. The no-reprojection and matched-noise variants isolate the contribution of the rendered view, the oracle shows little headroom is left in this setting, and the stop-gradient variant both resolves the encoder/decoder entanglement question and improves the method. The multi-seed results in Tables B, C, and F show the main gaps are well outside run-to-run variability. The noise-factor answer is also satisfying: the 10x factor was selected on two separate experimental datasets rather than on the CAHRA evaluation data, and the Aldolase sweep shows a flat plateau from 10 to 50. The clarification that HQ and LQ particles are not the same physical particles validates HQ->LQ as a genuine unseen-particle test. Overall the rebuttal strengthens the paper, and I will raise my rating from weak reject to weak accept.