acceptodds
Under review as a conference paper at ICLR 2027

Contrastive Image-Metadata Pre-Training for Materials Transmission Electron Microscopy

Abstract

The transmission electron microscope facilitates the highest-resolution imaging of any instrument ever created, and its limiting factor is no longer spatial resolution but dose efficiency. Low electron doses avoid sample damage but produce noisy images for which, unlike in classical computer vision, there is no ground truth. Autonomous materials experimentation poses a related problem, since closed-loop instruments need representations grounded in the microscope state at acquisition. Both demand representations grounded in how an image was acquired. We release 7,330 paired high-angle annular dark-field scanning-TEM (HAADF-STEM) images and their seven-dimensional acquisition metadata, and propose Contrastive Image-Metadata Pre-training (CIMP), a CLIP-style encoder that aligns the two modalities and reaches 84.4% Top-1 cross-modal retrieval on a held-out split. All seven parameters are individually recoverable from the frozen visual embedding through a linear probe, and we use the embedding to condition a metadata-conditioned style-transfer model that re-renders experimental images under different acquisition parameters. Virtually scaling dwell time and beam current of low-dose images turns this model into a physics-informed denoiser; in a blind user study, experimental microscopists prefer it over the current state-of-the-art denoiser for STEM imagery on 70.2% of trials.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
electron microscopy, materials science, contrastive learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_D5TcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents CIMP, a contrastive image–metadata pretraining framework for learning associations between electron microscopy images and acquisition parameters. The authors curate a dataset of 7,330 HAADF-STEM images, each paired with seven acquisition parameters, and use contrastive learning to align the two modalities. The learned representations are then used for parameter-controlled style transfer and low-dose image denoising.

Critical strengths

None

Critical weaknesses
  1. The proposed CIMP framework relies on seven acquisition parameters, which may not always be retained in exported image files. Is the method still applicable to legacy images or external datasets with incomplete metadata, and would missing metadata affect its performance?

  2. The linear probe can restore the collected parameters, but this is not sufficient to prove that the model has learned the cross-instrument universal physical laws. It is recommended to conduct cross-instrument tests to verify its generalization ability.

  3. Although paired evaluation on gold-on-carbon samples is provided, the structural fidelity of the generated images remains insufficiently validated. The authors should include simulated data with known ground truth or paired experiments on additional materials to assess whether denoising leads to detail loss or hallucinated structures.

  4. SHINE is a related self-supervised electron microscopy denoising method. The authors should provide a comparison with SHINE.

  5. The authors should include additional ablation studies to assess the respective contributions of the acquisition parameters and CIMP pretraining.

Questions for authors

The proposed CIMP framework relies on seven acquisition parameters, which may not always be retained in exported image files. Is the method still applicable to legacy images or external datasets with incomplete metadata, and would missing metadata affect its performance?

Reviewer_3VgcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes contrastive pretraining of HAADF-STEM images with their acquisition metadata. The authors collect a dataset of 7,330 images, each paired with seven microscope acquisition parameters. After contrastive pretraining, they use the learned embedding space to condition a style-transfer GAN that translates images between different acquisition settings. Translating low-dose images to high-dose acquisition settings allows the model to act as a denoiser. They evaluate this approach using experimental acquisition sweeps, a single paired low-/high-dose image comparison, and a user study in which microscopists prefer the proposed method over an adapted STEM denoising baseline in 70.2% of judgments.

Critical strengths

S1. The dataset addresses a real gap, as experimental microscopy images are rarely published together with detailed acquisition metadata. The collection of 7,330 images across different specimens and acquisition conditions could provide a useful resource for future work.

S2. The approach of using acquisition metadata to condition a style-transfer GAN is intuitive and well motivated. The resulting ability to translate between acquisition settings, particularly from low-dose to high-dose images without paired training data, seems valuable for practical microscopy workflows.

S3. The paper is generally well written and clearly explains its contributions and methodology. The experimental acquisition-parameter sweeps and physical-response checks provide useful evidence that the model learns meaningful relationships between acquisition settings and image appearance.

S4. The blinded user study with five experimental microscopists provides valuable evidence for the practical usefulness of the denoising results. The reported preference for CIMP over the adapted Thornley et al. baseline is encouraging, particularly given the use of real experimental images.

Critical weaknesses

W1. The contribution of contrastive pretraining is not established. The main downstream application of CIMP is to condition the style-transfer GAN and provide an embedding-alignment loss, but the paper does not establish that contrastive pretraining is necessary for either component. A comparison with a GAN conditioned directly on the acquisition parameters, using an appropriate metadata-aware discriminator, would be important to isolate the contribution of CIMP. An additional ablation separating CIMP-based conditioning from the embedding-alignment loss would clarify which component contributes to the results.

W2. The denoising evaluation mainly compares against adapted versions of Thornley et al. and Noise2Atom, but does not sufficiently detail how these baselines were adapted and tuned for the new dataset. This is particularly relevant given the severe artifacts reported for the Thornley et al. baseline, whose original method includes specific measures to suppress such artifacts. Other relevant restoration approaches, particularly Lobato et al. (2024), are omitted. Furthermore, the style-transfer evaluation lacks simple physics-based baselines, such as affine intensity transformations for gain/offset or Poisson-based noise models for dwell time.

W3. The quantitative denoising comparison in Table 4 appears to use only a single low-/high-dose image pair from one gold-on-carbon field of view, which is insufficient to establish general denoising performance across materials and acquisition conditions. While the larger dataset does not contain additional matched acquisition pairs, a broader evaluation using simulated STEM images with known clean references or additional controlled experimental acquisitions would substantially strengthen the results. Such a benchmark would also allow for more reliable comparisons with existing denoisers.

W4. LPIPS and KID rely on perceptual features developed for natural images, but the paper does not explain how these metrics are adapted to single-channel STEM images or justify their suitability for the modality. It is unclear whether they adequately capture the preservation of scientifically important structures, such as atomic columns and defects. This is especially relevant because LPIPS plays an important role in the denoising comparisons and KID is estimated on very small image sets.

W5. The paper interprets the linear recoverability of acquisition parameters as evidence of a factored, physics-aligned representation, but independent linear probes do not establish that the individual factors are disentangled. Moreover, the qualitative convergence-angle experiment illustrates that the model can learn correlations in the acquisition data rather than the isolated physical effect of a parameter. The results demonstrate acquisition-aware representations, but do not fully support the stronger interpretation.

Questions for authors

Q1. How were the low- and high-dose images used in Table 4 registered, and what residual spatial misalignment remains between them? Were possible specimen changes due to repeated beam exposure considered when computing the pixel-wise metrics?

Q2. How exactly was the Thornley et al. model adapted to the single-channel HAADF-STEM dataset, and which architecture, training settings, and hyperparameters were used? Were its original measures for avoiding checkerboard artifacts retained, and was comparable tuning performed across methods?

Q3. Could the authors report an overall preference confidence interval that accounts for repeated judgments from the same raters and images? It would also be interesting to see whether preferences differ across input noise levels or acquisition settings, particularly given the reported observation that Thornley et al. can perform better on higher-signal images.

Q4. Beyond image–metadata retrieval and linear probing, is there evidence that CIMP features are useful for other downstream microscopy tasks? It would also be interesting to evaluate whether retrieval and metadata recovery remain effective on held-out specimens or acquisition sessions rather than only randomly held-out images.

Q5. The paper motivates the method partly through real-time microscope operation, but does not report the actual inference latency of the complete denoising pipeline. What processing speed does the model achieve at typical experimental image resolutions, and how does it compare with the adapted baselines?

Q6. What gallery size is used for the reported retrieval accuracies? The results appear to be multiples of 1/64, raising the question of whether retrieval is evaluated within batches of 64 rather than against the full validation set of 733 images.

Reviewer_CvAgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the lack of ground truth in low-dose STEM imaging by proposing CIMP, which aligns raw micrographs with acquisition metadata using contrastive learning. The authors introduce a paired dataset of 7.3k real experimental images and validate the representation across linear probing, GAN-based restoration/denoising, and an operator user study.

Critical strengths
  1. The dataset itself is a solid contribution. Curating and releasing 7k+ paired, multi-parameter experimental STEM micrographs is genuinely useful for the community, especially given how rarely such data is shared.
  2. The overall validation is thorough and well-designed, particularly the linear probing analysis to confirm parameter recoverability and the blind preference study with real microscopists.
Critical weaknesses
  1. Is CIMP embeddings actually necessary over raw metadata? The generator is conditioned on CIMP embeddings via FiLM. However, there is no direct baseline showing what happens if you simply condition the network directly on the raw acquisition parameters (or a basic MLP projection of them). An ablation here is critical to justify why contrastive pretraining is needed in the first place.

  2. Physical/crystallographic fidelity at the atomic scale: While the GAN produces visually cleaner images, visual appeal doesn't necessarily mean crystallographic accuracy. Does the model risk introducing subtle hallucinations or shifting atomic column positions (e.g., picometer-scale drifts)? It would help if the authors could discuss or quantify lattice fidelity. Additionally, since some baselines perform quite poorly (e.g., Noise2Atom failing significantly), could the authors clarify how these baselines were tuned to ensure a fair comparison?

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.